Crash rates spike from <0.5% to ~8% on GooglePlay after updating to Air SDK 50.2.4.1

[mohossama]

Crash rates spike from <0.5% to ~8% on GooglePlay after updating to Air SDK 50.2.4.1

[ajwfrost]

Hi

Looking at the backtrace, this is a crash happening during the garbage collection process. But the nature of that process is such that we can't tell what object is being cleaned up.. it's likely that the error happens where an object is being processed which still has a hanging reference open to an object which itself had been cleaned up previously...

It was quite some jump between versions there. Can I check whether anything else changed or is the runtime update the only change that was made?

Also just wondering whether you use any ANEs, and in particular whether you maintain any references to ActionScript objects from within the extension objects (within the C or Java code, if you are writing these yourselves)?

If you have any other crash logs it may be useful, sometimes we'd spot the object that's being corrupted if the crash is timing-related.

thanks

[mohossama]

Hi thanks for the followup

Can I check whether anything else changed or is the runtime update the only change that was made?

We needed to update some ANEs with the Air runtime update to 50.x. Here's a list of the ones that were updated, just in case anything rings a bell. They were mostly native dependency updates as far as I can tell:

extensionID>com.appsflyer.adobeair</extensionID>

<extensionID>com.ironsource.adobeair</extensionID>
<extensionID>com.ironsource.adobeair.admob</extensionID>
<extensionID>com.ironsource.adobeair.facebook</extensionID>
<extensionID>com.ironsource.adobeair.fbaudiencenetwork</extensionID>
<extensionID>com.ironsource.adobeair.unityads</extensionID>
<extensionID>com.ironsource.adobeair.unityadssdk</extensionID>
<extensionID>com.ironsource.adobeair.applovin</extensionID>
<extensionID>com.ironsource.adobeair.applovinsdk</extensionID>

in particular whether you maintain any references to ActionScript objects from within the extension objects (within the C or Java code, if you are writing these yourselves)?

We do maintain a few ANEs that we also touched with this update. I don't think we introduced new AS3 code in them, but we can review them. I just want to make sure I understood correctly what is meant by maintaining a reference to AS object from within the native code (Java in our case). Do you mean like taking the FREOject from a function parameter and keeping that reference somewhere? Does retaining a reference of the FREContext object in the Java code also count?

If you have any other crash logs it may be useful

Here are some more samples that look different that were correlated to the same crash by GooglePlay. Is that what you meant?

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 6660 >>> air.com.... <<<

backtrace:
  #00  pc 0x0000000000868d98  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #01  pc 0x00000000008664a4  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #02  pc 0x00000000007bed64  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #03  pc 0x000000000086610c  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #04  pc 0x0000000000865518  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #05  pc 0x0000000000865634  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #06  pc 0x00000000008641b4  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #07  pc 0x0000000000863f78  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #08  pc 0x0000000000871250  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #09  pc 0x0000000000785f88  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #10  pc 0x0000000000771b48  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #11  pc 0x000000000040a810  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #12  pc 0x000000000028be58  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #13  pc 0x000000000028bdf4  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #14  pc 0x000000000028b4cc  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #15  pc 0x000000000028b53c  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #16  pc 0x000000000028b270  /data/app/~~nhVtyb9EFxkbwLnZnSdu-A==/air.com....-b-XQcELpna0ScQGmMLLbsg==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #17  pc 0x00000000000c163c  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+204)
  #18  pc 0x0000000000054930  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)

 *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 8172 >>> air.com.... <<<

backtrace:
  #00  pc 0x0000000001372708  /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #01  pc 0x000000000086fe04  /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #02  pc 0x0000000000867b1c  /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #03  pc 0x00000000008678e8  /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #04  pc 0x0000000000867800  /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #05  pc 0x00000000002789e8  /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #06  pc 0x000000000027c228  /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #07  pc 0x000000000027f698  /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/lib/arm64/libCore.so (BuildId: 0941d34cf16b317780fd8fcb701e876875cc8447)
  #08  pc 0x000000000004a084  /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/oat/arm64/base.odex (art_jni_trampoline+116)
  #09  pc 0x00000000005ba6b0  /apex/com.android.art/lib64/libart.so (nterp_helper+4016)
  #10  pc 0x000000000036197c  /data/app/~~6_Hihc-gYiAm3y1ZZth70A==/air.com....-9lTosdzsD9i4oDLAwz4zow==/oat/arm64/base.vdex (com.adobe.air.customHandler.handleMessage+20)
  #11  pc 0x0000000000a336f8  /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (android.os.Handler.dispatchMessage+168)
  #12  pc 0x0000000000a36f98  /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (android.os.Looper.loopOnce+1000)
  #13  pc 0x0000000000a36b08  /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (android.os.Looper.loop+1112)
  #14  pc 0x0000000000771210  /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (android.app.ActivityThread.main+2432)
  #15  pc 0x000000000033f080  /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+640)
  #16  pc 0x00000000003884dc  /apex/com.android.art/lib64/libart.so (_jobject* art::InvokeMethod<(art::PointerSize)8>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jobject*, _jobject*, unsigned long)+1588)
  #17  pc 0x0000000000387e98  /apex/com.android.art/lib64/libart.so (art::Method_invoke(_JNIEnv*, _jobject*, _jobject*, _jobjectArray*) (.__uniq.165753521025965369065708152063621506277)+32)
  #18  pc 0x00000000003336a8  /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (art_jni_trampoline+120)
  #19  pc 0x00000000009a9048  /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run+136)
  #20  pc 0x00000000009b320c  /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (com.android.internal.os.ZygoteInit.main+3548)
  #21  pc 0x000000000033f080  /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+640)
  #22  pc 0x00000000004e1ea8  /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeWithVarArgs<_jmethodID*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, std::__va_list)+728)
  #23  pc 0x000000000057b930  /apex/com.android.art/lib64/libart.so (art::JNI<true>::CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, std::__va_list)+156)
  #24  pc 0x00000000000bead0  /system/lib64/libandroid_runtime.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+120)
  #25  pc 0x00000000000cadf8  /system/lib64/libandroid_runtime.so (android::AndroidRuntime::start(char const*, android::Vector<android::String8> const&, bool)+852)
  #26  pc 0x0000000000002568  /system/bin/app_process64 (main+1300)
  #27  pc 0x0000000000083198  /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init+96)

Please let me know if you need more info

[mohossama]

@ajwfrost sorry for the direct mention, not my intention to spam you, but this question from my last comment is quite important and I fear it may have gotten lost in the wall of text. It would really help our investigation if you can confirm what you meant by "maintain references to ActionScript objects from within the extension objects"

in particular whether you maintain any references to ActionScript objects from within the extension objects (within the C or Java code, if you are writing these yourselves)?

[...] Do you mean like taking the FREOject from a function parameter and keeping that reference somewhere? Does retaining a reference of the FREContext object in the Java code also count?

[ajwfrost]

Hi - thanks for the ping, I'd not actually seen that update earlier, got lost in the mass of notifications :-(

But yes, keeping any reference to an FREObject would be a bad idea, ActionScript references should only be used within the single function call (unless they're also pinned via an ActionScript property elsewhere). I think Adobe had tried to ensure that references got held open but I would still think it depends a bit on how/when the garbage collector works...

We can check those other crash logs to see if they give more information, thanks!

[mohossama]

Thanks for the quick response! We will inspect the Java code in the ANEs we have the source code for to see if we can identify such reference-holding patterns.