Random app crashes on publish in AOT mode

[rolfbjarne]

From @jdarwood007 on Tue, 31 Dec 2024 19:55:39 GMT

Description

Random app crashes on publish in AOT mode

I am randomly getting app crashes in my app, only in release mode and only after sending to the App store and running in TestFlight on a physical device. Making this very difficult to get debugging information.

The problem seems to go away randomly as well. I can sometimes get the error to go away after a few times of cleaning and building the solution. It is inconsistent in what I do to fix it. However I have determined that even with no code changes in my app, after a few times of rebuilding and cleaning that I can send the exact same code up and have it not crash.

I got a crash report and had all symbols stripped. So I've added this to my .csproj

		<NoSymbolStrip>true</NoSymbolStrip>

This resulted in me being able to get this stack trace:

0   libsystem_kernel.dylib        	       0x1bc94fbbc __pthread_kill + 8
1   libsystem_pthread.dylib       	       0x1dd4bc844 pthread_kill + 208
2   libsystem_c.dylib             	       0x18c7786ac abort + 124
3   MyApp                         	       0x102869720 sigabrt_signal_handler.cold.1 + 48
4   MyApp                         	       0x1027dc6e4 sigabrt_signal_handler + 196
5   libsystem_platform.dylib      	       0x1dd4a4d48 _sigtramp + 52
6   libsystem_pthread.dylib       	       0x1dd4bc844 pthread_kill + 208
7   libsystem_c.dylib             	       0x18c7786ac abort + 124
8   MyApp                         	       0x102699d88 mono_log_write_os_log + 408
9   MyApp                         	       0x10268f4b4 monoeg_g_logv + 172
10  MyApp                         	       0x10268f5fc monoeg_g_log + 28
11  MyApp                         	       0x1027b48e4 load_aot_module + 4260
12  MyApp                         	       0x1026f04e8 mono_assembly_request_load_from + 1108
13  MyApp                         	       0x1026f0004 mono_assembly_request_open + 472
14  MyApp                         	       0x1026f1ca8 mono_assembly_open + 80
15  MyApp                         	       0x1026326b4 xamarin_assembly_preload_hook + 452
16  MyApp                         	       0x1026f1f54 invoke_assembly_preload_hook + 76
17  MyApp                         	       0x1026ef760 mono_assembly_request_byname + 1012
18  MyApp                         	       0x1027be9d4 load_image + 280
19  MyApp                         	       0x1027b487c load_aot_module + 4156
20  MyApp                         	       0x1027b93a0 mono_aot_get_method + 316
21  MyApp                         	       0x1027a7a60 jit_compile_method_with_opt + 564
22  MyApp                         	       0x1027ac034 mono_jit_runtime_invoke + 480
23  MyApp                         	       0x102753ae8 mono_runtime_invoke_checked + 148
24  MyApp                         	       0x10270e174 create_exception_two_strings + 592
25  MyApp                         	       0x10270def8 mono_exception_from_name_two_strings_checked + 148
26  MyApp                         	       0x1026ebf74 mono_runtime_init_checked + 712
27  MyApp                         	       0x1027ab910 mini_init + 6660
28  MyApp                         	       0x1027b1b60 mono_jit_init_version + 20
29  MyApp                         	       0x102631bd8 xamarin_bridge_initialize + 164
30  MyApp                         	       0x102632ac4 xamarin_main + 648
31  MyApp                         	       0x102825a00 main + 64
32  dyld                          	       0x1070c44d0 start + 444

It seems the app is hitting mono_exception_from_name_two_strings_checked early on in the startup of my app, as indicated the code is in the main and xamarin triggered the faulty code.

I hooked up the device to a Mac mini and was able to get the console to present the following error message

error: Failed to load AOT module 'Microsoft.Maui.Controls' while running in aot-only mode because a dependency cannot be found or it is out of date.

This seems to back the results of what I am experiencing in that the crash is occurring randomly and that cleaning fixes it, by possibly removing that out of date dependency.

To resolve the crashes, I have done the following, all sometimes work and at other times does not resolve the issue.

1. Build > Clean Solution
2. Delete the bin and obj folders
3. Close Visual Studio, Delete the contents of C:\Users%me%\AppData\Local\Temp\Xamarin and /Users/%me%/Library/Caches/Xamarin/XMA/SDKs (on mac)
4. Close and shutdown windows and mac. Return next day.

I don't have any project to provide, as it is random and by luck I feel, will just work for a bit. I've seen this on a few projects I work with and the resolution has always been the same, try to clean things and rebuild until it works.

I have noticed that more often than not, the problem seems to exist after I deployed a the app to the device via TestFlight , if I plug it into the mac and deploy the app in debug mode, unplug device, clean, rebuild and publish the app into TestFlight and install the TestFlight version, it will crash.

Steps to Reproduce

No response

Link to public reproduction project repository

No response

Version with bug

9.0.21 SR2.1

Is this a regression from previous behavior?

Not sure, did not test other versions

Last version that worked well

Unknown/Other

Affected platforms

iOS

Affected platform versions

Any

Did you find any workaround?

No response

Relevant log output

Copied from original issue dotnet/maui#26890

[rolfbjarne]

From @mattleibow on Sat, 04 Jan 2025 14:18:54 GMT

@rolfbjarne thoughts?

[rolfbjarne]

From @mattleibow on Sat, 04 Jan 2025 14:22:56 GMT

@jdarwood007 not sure if it matters, but are you using the interpreter? Have you set the <UseInterpreter>true</UseInterpreter> in your csproj?

[rolfbjarne]

From @mattleibow on Sat, 04 Jan 2025 14:24:32 GMT

I found these issues:

• [MacCatalyst] Failure to run in release mode (arm64) maui#24404
• MAUI App crash at startup in net9.0.100-preview.6.24328.19 maui#24025
• Crash on startup with iOS Release maui#24618

Might be unrelated, might not be.

[rolfbjarne]

From @jdarwood007 on Sat, 04 Jan 2025 15:44:23 GMT

@mattleibow I do not have it defined in my .csproj, so no. Although a quick read on the documentation seems that this may be the reason because I use generics. I consume a REST API returning JSON results and it uses a standardized return with a property that has a defined type (so my return is something like StandardResponse<Student>).

I did enable <MauiEnableXamlCBindingWithSourceCompilation>true</MauiEnableXamlCBindingWithSourceCompilation> during testing of this issue and realized I had unsafe bindings and even got errors to indicate that I needed to use JsonSerializerContext. I never realized these issues as calls worked as expected with no problems in debug mode when initially written.

However, I can get it to build and run successfully randomly and even have releases. My thoughts here are that something on the Visual Studio or Mac OS side is being held and retained during the build from a previous build and causing this. It is possible that a file is being cached improperly, a file stat call is returning cached results or a file is not recognized as needing to be rebuilt.

For the linked issues, the first issue mentions dotnet clean. I came to the same conclusion. At some points, doing that resolves the issue. But the source of the problem remains a mystery. Why does not a 'build' not properly do this?
The first issue does say that the interpreter is enabled by default on debug builds. I am using a M1 Mac mini, so this is helpful and seems close to my issue.

The second issue was resolved with an update, but the change message indicates a different function than the one I'm seeing crashing. That is we are seeing mono_runtime_init_checked called and then it reaches and exception mono_exception_from_name_two_strings_checked. Indicating the failure during mono_runtime_init_checked was reached.

The third linked issue isn't relevant, as their crash is caused by code they directly wrote, whereas from what I can tell, my issue occurs early on in the startup process, before my code can even execute.

[rolfbjarne]

I hooked up the device to a Mac mini and was able to get the console to present the following error message

error: Failed to load AOT module 'Microsoft.Maui.Controls' while running in aot-only mode because a dependency cannot be found or it is out of date.

This seems to back the results of what I am experiencing in that the crash is occurring randomly and that cleaning fixes it, by possibly removing that out of date dependency.

Yes, this certainly looks like a problem with incremental builds, where we don't correctly identify what needs to be rebuilt and what doesn't (which is why a clean build works: then everything will be rebuilt).

The annoying part here is that these incremental build issues can be rather hard to track down, because we need very specific instructions in order to reproduce them: provide a test project (or your actual project if that's easier for you), and then something like:

1. (start from a clean slate)
2. Build project in VS by clicking the "build" button
3. Edit file X.cs at line 123 to say "Hello World"
4. Build project again in VS by clicking the "build" button
5. Run on device.

[jdarwood007]

The app is an internal app used for the business. I tested on another of our internal apps, currently on .net 8, and even after upgrading to .net 9 and doing testing, it worked flawlessly. Yet the first app (in this report) gave me trouble for weeks with random crashes. I even tried to mess with various build settings to determine if it was something during the linker doing something and accidentally stripping out a needed component.

Unfortunately, building in dev and release mode has worked fine for the past week now. I don't have a build to send to test flight to verify if its still doing it. Oddly enough though, when I would build for release, it would sometimes work, but then sending it up to testflight and back down to the device would crash.

Are there additional debugging steps I can try to do in order to produce more verbose information? Can I get the compiler to indicate what libraries/components it determines will not need to be updated and thus are skipped in the build output log? I don't see any from the output myself.

[rolfbjarne]

I even tried to mess with various build settings to determine if it was something during the linker doing something and accidentally stripping out a needed component.

Any problems with the linker would be consistent - i.e. cleaning wouldn't fix it.

Are there additional debugging steps I can try to do in order to produce more verbose information? Can I get the compiler to indicate what libraries/components it determines will not need to be updated and thus are skipped in the build output log? I don't see any from the output myself.

Kind of, if you get an MSBuild binlog (https://github.com/xamarin/xamarin-macios/wiki/Diagnosis#binary-build-logs), it will typically be possible to deduce this by inspecting the binlog. However, the build process is rather complex, so it's not trivial. If you can get such a binlog, we could have a look and see if we find anything useful (note that figuring out that library X wasn't rebuilt correctly is only half the picture - we also need to know why library X should have been rebuilt, which means we need to know exactly what you did in your project).

[jdarwood007]

I received the crash again today while building. The error message this time was

error: Failed to load AOT module 'SQLitePCLRaw.core' while running in aot-only mode because a dependency cannot be found or it is out of date.

This is interesting as I didn't touch anything related to the SQLite code or package between the last release and the one I am working on. I'm running the latest 2.1.10 for that Nuget package. The only code changes between my last two releases are XAML and some code behind for a component and page.

As a test, I deleted everything in the folder on the mac located in: /Users/jeremy/Library/Caches/Xamarin/mtbs/builds/MyApp/
After I did this, I did a build and publish, app works in TestFlight.
The mac is a M1 located directly on the network with a wired 1GB link between it and my Windows machine.

So the issue is related directly to something on the CreateIpa process I believe. I will try next time to selectively delete files on the mac side to see if I can't limit it down more. If you have suggestions let me know.

I was able to generate the bin log for the build and CreateIpa process. Does the Bin log include what is occurring on the mac during its build?
Is there a way to send you this file?

[rolfbjarne]

I was able to generate the bin log for the build and CreateIpa process. Does the Bin log include what is occurring on the mac during its build?
Is there a way to send you this file?

If you want to share privately, you can put it on a file-sharing service (say Dropbox), and then share a link to it ([email protected]).

[microsoft-github-policy-service]

Hi @rolfbjarne. Due to inactivity, we will be closing this issue. Please feel free to re-open this issue if the issue persists. For enhanced visibility, if over 7 days have passed, please open a new issue and link this issue there. Thank you.

[LordBenjamin]

@rolfbjarne can you reopen please?

Bumping this as I can reproduce the same stack trace (or something very similar!) and it's causing major disruption for our customer, who is waiting for us to deliver functional changes that are blocked by AOT issues that have essentially appeared overnight with no obvious change to our csproj / packages / etc.

[rolfbjarne]

@LordBenjamin if your problem goes away with a clean rebuild, then it's something else, and I would suggest creating a new issue where we can properly track it.

[LordBenjamin]

@LordBenjamin if your problem goes away with a clean rebuild, then it's something else, and I would suggest creating a new issue where we can properly track it.

The problem does not go away with a clean rebuild. It only goes away if I delete the SDK cache folder on the Mac (as per @jdarwood007 comment above) and then re-pair the Mac from Visual Studio.

[jdarwood007]

@LordBenjamin, perhaps you could help me track it down. I don't have any planned releases of my app right now, so its bug fixes only that would trigger me to build a new version.

On a recent build, I did last month, I got the following error:

error: Failed to load AOT module 'System.Security.Cryptography' while running in aot-only mode because a dependency cannot be found or it is out of date.

I deleted on the mac cache folder: ./obj/Release/net9.0-ios/ios-arm64/nativelibraries/
Rebuilt and success. However I still had my iPhone connected from getting that error, so I might have messed up the testing results.

A week later I had to send another bug fix up and I tried and got another error:

error: Failed to load AOT module 'ZXing.Net.MAUI.Controls' while running in aot-only mode because a dependency cannot be found or it is out of date.

I am using that library, but I know it hasn't changed. However, I think my visual studio did have an update. So perhaps related?

Using what I learned from the last test I tried to delete: ./obj/Release/net9.0-ios/ios-arm64/stripped/ZXing.Net.MAUI.Controls.dll. However, it still crashed.

So then I tried to delete the following:

• ./obj/Release/net9.0-ios/iossimulator-arm64/stripped/ZXing.Net.MAUI.Controls.dll
• ./obj/Release/net9.0-ios/iossimulator-arm64/linked/ZXing.Net.MAUI.Controls.dll

This one was weird as Visual Studio started to indicate a file was missing. I did a clean solution and rebuilt it. The application still crashed.

I then tried to delete all ZXing.Net.MAUI.Controls with something like find /obj/Release -iname 'ZXing.Net.MAUI.Controls* -delete` and then I found out a ./C:/ as well which seemed to mirror my windows system. Odd. I deleted that as well.

I rebuilt the app and then it was successful and didn't crash. However, I had to rebuild again after reverting as I did some library updates and something in the community toolkit was breaking the app and didn't want to load pages. I reverted and didn't do any tracing into why. I did a clean solution during that.

That is as far as I have got with tracking down at least what files are being held over that is causing this. I am not certain if what I am doing is even helping. I figured at least if we can find the location of the files that are not being rebuilt, that the build process can be looked at for those files and determine if it is. It's possible just the nativelibraries folder is the problem and deleting just the affected file in it would help.

I am trying to understand the file structure as it comes down to the Mac. As I said there was a "C" folder representing the C:\Users\jeremy\source\Repo\App as it matches on the Windows system. There is also a separate bin and obj folder. Which I would expect. Inside the bin/obj folders, we have the expected net9-ios folder, and inside there is which device I am building against, typically it is just iossimulator-arm64. Then inside there, I saw the stripped, linked, and nativelibraries. Which order they are built in I don't know yet. However the nativelibraries has an aot-output folder, which leads me to believe that the files inside there are the ones that have gone through the aot process.

Unless we can get some better tools to help us along with the build process to output some additional information, I think I am left with trying to slowly track down which directory has the affected files and fix the issue once deleted. From there, hopefully, I can provide information to help the devs understand why the build process does not think the file needs to be rebuilt when it does.

Hopefully, as well we can figure out why a clean solution isn't cleaning up this.

[berga0d]

Same problem here