/system/bin/mediaserver crash at system startup

[andreasgal]

/system/bin/mediaserver crash at system startup

[shianyow]

I found a way to reproduce this issue anytime, simply kill pid of "com.android.phone".

The "com.android.phone" will be restarted when been killed, and running "com.android.phone" will trigger some function calls at mediaserver which caused mediaserver crash. During system start "com.android.phone" was executed after /system/bin/mediaserver", it explains why mediaserver always crash during system startup.

By this discovery, I can attach gdbserver to pid of "mediaserver", then kill pid of "com.android.phone" to debug this issue by gdb.

The crash point android_atmoic_add() was called by several different functions in mediaserver. When crash case occured, the back trace sequence is as below:

#0  android_atomic_add (increment=1, ptr=0xa008)
    at system/core/include/cutils/atomic-arm.h:175
#1  0xa811b010 in android::SharedBuffer::acquire (this=0x1)
    at frameworks/base/libs/utils/SharedBuffer.cpp:97
#2  0xa811c216 in android::getEmptyString () at frameworks/base/libs/utils/String8.cpp:133
#3  0xa811c636 in android::String8::String8 (this=0x1)
    at frameworks/base/libs/utils/String8.cpp:239
#4  0xa90343a4 in android::AudioParameter::AudioParameter (this=0x40567a78, 
    keyValuePairs=...) at frameworks/base/media/libmedia/AudioSystem.cpp:845
#5  0xa9a08448 in android::AudioPolicyManager::getParamFromPolicy(android::String8 const&)
    ()
   from /home/sywu/B2G/glue/gonk/out/target/product/galaxys2/system/lib/libaudiopolicy.so
#6  0xa9a08448 in android::AudioPolicyManager::getParamFromPolicy(android::String8 const&)
    ()
   from /home/sywu/B2G/glue/gonk/out/target/product/galaxys2/system/lib/libaudiopolicy.so
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

It's working fine when been called another sequence as below:

#0  android_atomic_add (increment=1, ptr=0xa020)
    at system/core/include/cutils/atomic-arm.h:175
#1  0xa811b010 in android::SharedBuffer::acquire (this=0x1)
    at frameworks/base/libs/utils/SharedBuffer.cpp:97
#2  0xa811ccdc in getEmptyString (this=0x62c08)
    at frameworks/base/libs/utils/String16.cpp:251
#3  android::String16::String16 (this=0x62c08)
    at frameworks/base/libs/utils/String16.cpp:310
#4  0xa8214392 in android::BpBinder::BpBinder (this=0x62bd8, handle=3, 
    __in_chrg=<value optimized out>, __vtt_parm=<value optimized out>)
    at frameworks/base/libs/binder/BpBinder.cpp:93
#5  0xa821d43c in android::ProcessState::getStrongProxyForHandle (
    this=<value optimized out>, handle=3)
    at frameworks/base/libs/binder/ProcessState.cpp:222
#6  0xa821b81e in android::unflatten_binder (proc=..., in=<value optimized out>, 
    out=0x40667c64) at frameworks/base/libs/binder/Parcel.cpp:242
#7  0xa821b864 in android::Parcel::readStrongBinder (this=0x40667dd0)
    at frameworks/base/libs/binder/Parcel.cpp:960
#8  ......

Some things need to be further investigated:

1. One thing strange, when the crash case happens, the gdb back trace shows "corrupt stack?". Will it be related to this issue?

2. Is it related to un-thumb mode?
   There are 3 kinds of android_atomic_add() implementation in "atomic-arm.h".
   Current B2G was using the 2nd case (the __ARM_HAVE_LDREX_STREX case).

[shianyow]

Sorry, back trace for crash case should be as below. The ptr=0xa9a081dd in android_atomic_add() is invalid(not at 4 byte boundary, which caused SIGBUS), and the value of "stream" at back trace no.6 became messed after calling "libaudiopolicy.so".

Unfortunately "libaudiopolicy.so" is proprietary library from S2 stock firmware, I cannot trace into it to know what really happened.

Could be compatibility issue between "libaudiopolicy.so"(from S2) and other libraries(from Android AOSP).

#0  android_atomic_add (increment=1, ptr=0xa9a081dd)
    at system/core/include/cutils/atomic-arm.h:175
#1  0xa811b010 in android::SharedBuffer::acquire (this=0x1)
    at frameworks/base/libs/utils/SharedBuffer.cpp:97
#2  0xa811c666 in android::String8::setTo (this=0x40667a7c, other=...)
    at frameworks/base/libs/utils/String8.cpp:297
#3  0xa90343d6 in operator= (other=<optimized out>, this=<optimized out>)
    at frameworks/base/include/utils/String8.h:360
#4  android::AudioParameter::AudioParameter (this=0x40667a78, keyValuePairs=...)
    at frameworks/base/media/libmedia/AudioSystem.cpp:848
#5  0xa9a08448 in android::AudioPolicyManager::getParamFromPolicy(android::String8 const&)
    ()
   from /home/sywu/B2G/glue/gonk/out/target/product/galaxys2/system/lib/libaudiopolicy.so
#6  0xa8d2c0a6 in android::AudioPolicyService::getStrategyForStream (
    this=<optimized out>, stream=-1449098787)
    at frameworks/base/services/audioflinger/AudioPolicyService.cpp:354
#7  0xa9035456 in android::AudioSystem::getStrategyForStream (
    stream=android::AudioSystem::VOICE_CALL)
    at frameworks/base/media/libmedia/AudioSystem.cpp:677
#8  ...

Below is a quick work around fix. Not recommend to use it at this moment, unless you are working on something blocked by mediaserver crash.

diff --git a/services/audioflinger/AudioPolicyService.cpp b/services/audioflinger/AudioPolicyService.cpp
index f24e08e..0bf4712 100644
--- a/services/audioflinger/AudioPolicyService.cpp
+++ b/services/audioflinger/AudioPolicyService.cpp
@@ -351,7 +351,8 @@ uint32_t AudioPolicyService::getStrategyForStream(AudioSystem::stream_type strea
     if (mpPolicyManager == NULL) {
         return 0;
     }
-    return mpPolicyManager->getStrategyForStream(stream);
+    //return mpPolicyManager->getStrategyForStream(stream);
+    return 0;
 }

[shianyow]

When replacing with "libaudiopolicy.so" from CyanogenMod 7.1 for SGS2, this crash issue disappeared. And I can hear dialing tone in Phone.js which couldn't before. It's a better temporary solution than previous work around fix.

I'll take a look at how this library was built by CM.

[jamesho86]

sounds great!

[shianyow]

CM 7.1 for SGS2 also pulling "libaudiopolicy.so" from stock firmware, instead of building from source.
My S2 socket firmware version is ZSKI3, I think the issue only happened with specific version of "libaudiopolicy.so".
If that's the case, we should put specific version of "libaudiopolicy.so" to source tree, instead of pulling by extract-files.sh.

[joneschrisg]

Nice work @shianyow!

Unfortunately, we can't host the proprietary blobs ourselves :(. According to https://github.com/CyanogenMod/android_vendor_cyanogen/blob/gingerbread/CHANGELOG.mkdn, CM 7.1 is based on gingerbread 2.3.7, while our codebase is ~2.3.3. This was hypothesized to be part of what caused the problems with wifi. Since we're all on 2.3.4 firmware now, it's probably time to rebase our code on that.

[joneschrisg]

Actually, that's wrong ... b2g is based on 2.3.5 right now. So looks like our blobs are just too old.

Maybe the best solution here would be to download CM blobs we need as part of the config process (but not commit them to our repo). That would make the config-galaxy-s2 step a lot simpler and more reliable, and wouldn't require having a phone to build b2g (good for automated builds).

[shianyow]

Downloading CM blobs in config process sounds a good idea. :)

[shianyow]

Will do it this way later on.

[joneschrisg]

Great! Thanks for taking this.

Based on what you found, it looks like the blobs we need are only in the CM images. So we're probably going to have to download, inflate, mount, then extract what we need.

[shianyow]

For Android build, current B2G still use kernel from stock firmware, we still need to get blobs from real device, because they could be different depending on stock firmware version. For example, we are maintaining several versions(UHKG7, UHKI2, ZSKI3, GWK74, XWKI4...) in extract-files.sh and they require different proprietary libraries.

How about this?

1. Close issue #56 by extracting only "libaudiopolicy.so" from CM 7.1 as a solution.
2. Create another issue to extract all blobs from CM 7.1 instead of real device. I will try first if we can stay at 2.3.5 and using blobs from CM7.1. If not, we might need to migrating to 2.3.7, or need to update Linux kernel.

[joneschrisg]

Yes, that's a good plan. We can also grab blobs from an older version of CM.

[andreasgal]

Make sure the blobs are downloaded during make flash. We can't put them into our repository.

[joneschrisg]

Yes, as part of the configure process. It's important not to commit the blobs.

[shianyow]

Now "libaudiopolicy.so" will be extracted from CM 7.1 in config-galaxy-s2 process.

[joneschrisg]

\o/