SIGSEGV in mono_type_get_name_full

[TimBurik]

Android framework version

net8.0-android, net9.0-android

Affected platform version

.NET 8.0.403, .NET 9.0.100

Description

Ever since switching from Xamarin.Android to .NET8, and recently to .NET9, we see a crash group which appears in all our releases in GooglePlay Console. Originally we were suspected that this crash group might be related to the "out-of-memory", but we don't see any other alloc-related issues originating from other places.
Interestingly, only a small percentage of reports GooglePlay Console considers to be a "user-perceived" once. We are suspecting that these crashes might be happening when one of our crash monitoring systems (AppCenter, Sentry) is collecting information.

We have a lot of similar crash reports: from different architectures (armeabi/arm64), with different levels of recursion, with different realloc implementations, but always from the same place of origin.
crash_symbolicated_example1.txt
crash_symbolicated_example2.txt
crash_symbolicated_example3.txt
crash_symbolicated_example4.txt

Steps to Reproduce

Unfortunately, we haven't found a reliable way to reproduce it manually, but it has a pretty big counter in production (around 0.5% affected sessions)

Did you find any workaround?

No workaround found yet

Relevant log output

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 2783 >>> <app.bundle.id> <<<

backtrace:
  #00  pc 0x0000000000061a28  /apex/com.android.runtime/lib64/bionic/libc.so (je_arena_ralloc_no_move+88)
  #01  pc 0x0000000000061f2c  /apex/com.android.runtime/lib64/bionic/libc.so (je_arena_ralloc+160)
  #02  pc 0x0000000000059740  /apex/com.android.runtime/lib64/bionic/libc.so (je_realloc+236)
  #03  pc 0x0000000000053934  /apex/com.android.runtime/lib64/bionic/libc.so (realloc+88)
  #04  pc 0x00000000001ddba8  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (monoeg_realloc+108)
  #05  pc 0x00000000001e0720  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (monoeg_g_string_append+105)
  #06  pc 0x000000000020d820  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_recurse+457)
  #07  pc 0x000000000020d740  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_recurse+388)
  #08  pc 0x000000000020daf0  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_recurse+494)
  #09  pc 0x000000000020daf0  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_recurse+494)
  #10  pc 0x000000000020d514  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_full+557)
  #11  pc 0x000000000023cd18  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (ves_icall_System_RuntimeType_getFullName_raw+5131)
  #12  pc 0x000000000000b2d0 

[jonpryor]

Unfortunately, in all of the attached crash_symbolicated*.txt files and in the Relevant log output section, the crash isn't within mono_type_get_name_full, but instead rooted by realloc(3):

  #00  pc 0x0000000000061a28  /apex/com.android.runtime/lib64/bionic/libc.so (je_arena_ralloc_no_move+88)
  #01  pc 0x0000000000061f2c  /apex/com.android.runtime/lib64/bionic/libc.so (je_arena_ralloc+160)
  #02  pc 0x0000000000059740  /apex/com.android.runtime/lib64/bionic/libc.so (je_realloc+236)
  #03  pc 0x0000000000053934  /apex/com.android.runtime/lib64/bionic/libc.so (realloc+88)

The caller in all cases is monoeg_realloc(), which is g_realloc(), which calls realloc(3).

I don't know how wee can protect against a crash within realloc(3).

My guess is that the RAM on these devices is going bad, or a cosmic ray is altering RAM contents. (See also, Effect of Cosmic Rays on Computer Memories.) The error described doesn't otherwise make any sense at all.

[TimBurik]

Hello @jonpryor
We were also initially thinking about memory corruption or issues with the realloc, but ultimately discarded this idea with the following reasoning:

1. The crash group which contains (mono_type_get_name_recurse+457) in stack trace has by far the biggest count in our latest release (262 events vs 164 for the top 2);
2. We don't see any crash groups, related to the memory allocation/reallocation/deallocation, but not related to the mono_type_get_name_full. If memory was corrupted - there would have been many reports from different places, it feels;
3. I've just went through the full list of crash reports for the last 60 days, and I've found one more very similar stack trace:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 18223 >>> <app-bundle-id> <<<

backtrace:
  #00  pc 0x00000000000fe6b4  /apex/com.android.runtime/lib64/bionic/libc.so (pthread_mutex_trylock+8)
  #01  pc 0x000000000005f0f8  /apex/com.android.runtime/lib64/bionic/libc.so (je_arena_dalloc_small+412)
  #02  pc 0x000000000005b77c  /apex/com.android.runtime/lib64/bionic/libc.so (arena_dalloc_no_tcache+172)
  #03  pc 0x0000000000057f20  /apex/com.android.runtime/lib64/bionic/libc.so (je_free+1532)
  #04  pc 0x000000000020d828  /data/app/~~RL6hdI-QsPXoCMvL9q104w==/<app-bundle-id>-Cffzh5p9axDpbK9h8aVXDQ==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_recurse+458)
  #05  pc 0x000000000020d620  /data/app/~~RL6hdI-QsPXoCMvL9q104w==/<app-bundle-id>-Cffzh5p9axDpbK9h8aVXDQ==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_recurse+445)
  #06  pc 0x000000000020db18  /data/app/~~RL6hdI-QsPXoCMvL9q104w==/<app-bundle-id>-Cffzh5p9axDpbK9h8aVXDQ==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_recurse+494)
  #07  pc 0x000000000020d514  /data/app/~~RL6hdI-QsPXoCMvL9q104w==/<app-bundle-id>-Cffzh5p9axDpbK9h8aVXDQ==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_full+557)
  #08  pc 0x000000000023cd18  /data/app/~~RL6hdI-QsPXoCMvL9q104w==/<app-bundle-id>-Cffzh5p9axDpbK9h8aVXDQ==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (ves_icall_System_RuntimeType_getFullName_raw+5131)
  #09  pc 0x000000000000a218 

4. This means that the only two crash groups related to the memory allocation we had in last 60 days, both originating from two neighboring lines in source code:
   https://github.com/dotnet/runtime/blob/d2cdcdd69391bbfde1a8fb4aa275a6cc393ca65b/src/mono/mono/metadata/class.c#L457-L458

[TimBurik]

We may have found the root cause of those crashes in our code base, although not the exact place.

We have a custom implementation of ANR monitor for Android in our implementation, which uses a custom Java.Lang.Thread implementation, which periodically posts to the Handler(Looper.MainLooper) to detect if UI thread is responsive. In case ANR is detected, we collect information about the items located in the queue of our custom SynchronizationContext implementation.

We suspect that crash might be happening within is information collection. Starting from delegates and their states we recursively traverse the object hierarchy, collecting property values and type names (hence the connection to the native mono_type_get_name_full). In rare cases we observe some kind of loops: while collecting information about Delegate we analyze the value of Target, which turns out to be a Delegate too, whose Target property is also a Delegate,... and so on, probably infinitely. So most likely at some point in this infinite loop the app is running out of memory while trying to get the type of the next delegate, which is likely the reason of the SIGSEGV in realloc.

Either way, since we disabled this custom ANR monitor the crash groups had disappeared completely.