CREATEMULTIDEXMAINDEXCLASSLIST Issue

[brad-dixon]

Description

The below error occurs for me during both a debug and release build on my local machine ...

/Users/{user}/{company}/{project}/Droid/CREATEMULTIDEXMAINDEXCLASSLIST: Error: Can't write [/private/var/folders/zx/cg5_6s3930q1f4t896hrk7t00000gn/T/9rugjo7k.1ft.jar] (Can't read [/Users/{user}/{company}/{project}/Droid/obj/Debug/lp/78/jl/bin/classes.jar(;;;;;;!META-INF/MANIFEST.MF)] (Duplicate zip entry [classes.jar:android/arch/lifecycle/LiveData$1.class])) ({project}.Droid)

... and this only seems to happen when Xamarin.Essentials is added. If I remove it, the project builds without issue.

There is another issue open here ...

xamarin/XamarinComponents#579

... but I do not have Xamarin.Google.Guava installed on this project so apologies if this is already open elsewhere.

Basic Information

Given it looks related, the project has multidex turned on due to the inclusion of a package that was added around 12 months ago.

The project is targeting Android 9.0 (API Level 28) but also tried on 8.1 and still got the same error.

Happy to provide more information where appropriate.

Apologies if this isn't linked to Essentials but it's the only thing I currently have to go on.

Version

Visual Studio version information attached.

Version Info.pdf

[jamesmontemagno]

It could be that you have some libraries already added in and now this puts you over the multi-dex limit. In your project settings simply turn on multi-dex.

[brad-dixon]

Thanks James. Yes, understood but you may have missed in my description, multidex was turned on 12 months ago just before we went live with the app. Another package pushed it over the top, hence why it’s been on for so long.

[jamesmontemagno]

Ah, i see that comment now in there. Is it using the D8/R8 settings?

[jamesmontemagno]

@Redth any insight into this.... seems like a weird multi-dex issue... perhaps different dependencies from other nugets.

[Redth]

This isn't really specific to essentials, it's just that you're hitting the max # of methods in the main dex file for the android app. One way to help mitigate this is to enable proguard to link away unused bits of native java code to bring the overall dex count down.

I'd try proguard and try again.

[brad-dixon]

@Redth, yeah, was kinda wondering if there was a limit to multidex. Not used to these ceilings that exist in Android. Will try a few things and feed back to the both of you.

[brad-dixon]

@Redth @jamesmontemagno time for a stiff drink I think! It's lucky it's Friday afternoon here where I am.

Not having any luck, after reading through this issue ... https://github.com/xamarin/xamarin-android/blob/master/Documentation/guides/D8andR8.md ... and the instructions on how to apply the d8/r8, I tried the following combinations with the associated results.

Multi-Dex = On, Dex Compiler = dx, Code Shrinker = proguard

...
  Task "Desugar"
...
    Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 5
    	at org.objectweb.asm.ClassReader.a(Unknown Source)
    	at org.objectweb.asm.ClassReader.a(Unknown Source)
    	at org.objectweb.asm.ClassReader.a(Unknown Source)
    	at org.objectweb.asm.ClassReader.b(Unknown Source)
    	at org.objectweb.asm.ClassReader.accept(Unknown Source)
    	at org.objectweb.asm.ClassReader.accept(Unknown Source)
    	at com.google.devtools.build.android.desugar.Desugar.desugarClassesInInput(Desugar.java:391)
    	at com.google.devtools.build.android.desugar.Desugar.desugarOneInput(Desugar.java:326)
    	at com.google.devtools.build.android.desugar.Desugar.desugar(Desugar.java:280)
    	at com.google.devtools.build.android.desugar.Desugar.main(Desugar.java:584)
    /Library/Frameworks/Mono.framework/External/xbuild/Xamarin/Android/Xamarin.Android.Common.targets(2732,3): error : java.lang.ArrayIndexOutOfBoundsException :  5
  Done executing task "Desugar" -- FAILED.
Done building target "_CompileToDalvikWithDx" in project "MyProject.Droid.csproj" -- FAILED.
Target "_CleanGetCurrentAndPriorFileWrites" skipped. Previously built unsuccessfully.
Done building project "MyProject.Droid.csproj" -- FAILED.

Multi-Dex = On, Dex Compiler = d8, Code Shrinker = proguard

The logs here are a little more verbose and is far too much to drop in to this thread. Bottom line, a lot of checking on classes but no distinct and called out errors, that I can see. I really need an expert to decipher a lot of what I am seeing, bottom line however, the below conclusion to the build ...

  Done executing task "Proguard" -- FAILED.
Done building target "_CompileToDalvikWithD8" in project "MyProject.Droid.csproj" -- FAILED.
Target "_CleanGetCurrentAndPriorFileWrites" skipped. Previously built unsuccessfully.
Done building project "MyProject.Droid.csproj" -- FAILED.

Multi-Dex = On, Dex Compiler = d8, Code Shrinker = r8

...
  Task "R8"
      processing ClassesZip, JavaLibrariesToEmbed...
...
    R8 : warning : Missing class: com.samsung.android.fingerprint.FingerprintManager$EnrollFinishListener
    R8 : warning : Missing class: com.samsung.android.fingerprint.FingerprintIdentifyDialog$FingerprintListener
    R8 : warning : Missing class: com.samsung.android.fingerprint.IFingerprintClient$Stub
    R8 : error : Compilation can't be completed because some library classes are missing.
    Compilation failed
    The command exited with code 1.
  Done executing task "R8" -- FAILED.
Done building target "_CompileToDalvikWithD8" in project "MyProject.Droid.csproj" -- FAILED.
Target "_CleanGetCurrentAndPriorFileWrites" skipped. Previously built unsuccessfully.
Done building project "MyProject.Droid.csproj" -- FAILED.

As you can imagined, I've Googled (sorry, Binged) the bejesus out of the above but nothing has jumped out and helped me.

I'm fairly unclear at this point if I've hit some kind of hard ceiling, the errors don't point it to being that. I've had a look through the nuget packages being included and there's virtually nothing I can remove without having to rework the entire project ... not to mention I inherited this bad boy from a previous developer.

The inclusions I made for Essentials was nothing more than Geolocation and Connectivity and @jamesmontemagno based on what I read here ... https://montemagno.com/optimizing-xamarin-apps-libraries-with-the-linker/ ..., and my limited understanding of the linker, Essentials has been fully "linkerized" so should the compiler not be stripping everything else away that's not being used? Having said that, it may very well be doing that but is still pushing me over the edge. I hope my understanding is correct.

If you guys have no other suggestions, do you think it's worth closing this and throwing it over the fence to the Xamarin.Android team?

Again, thanks for your help, massively appreciated.

[jamesmontemagno]

Thanks for the write up, it is hard to know without knowing all the dependencies and seeing what is in there. Assumption is that we have a few support libraries in there. Even though it is fully linker safe there is still native code that gets called and that adds to the dex count at the and of the day (even if it is just 1 method in Essentials).

@JonDouglas and @jonathanpeppers may be able to help out, but perhaps if we had a sample or knew what nugets you had we could figure it out. We can share that privately and start up a thread.

[brad-dixon]

@jamesmontemagno, that is more than a fair request.

Is this a good start?

"CarouselView.FormsPlugin" version="4.4.3"
"FastAndroidCamera" version="2.0.0"
"Geo" version="0.14.1"
"GoogleGson" version="2.8.5"
"Microsoft.AppCenter" version="2.0.0"
"Microsoft.AppCenter.Analytics" version="2.0.0"
"Microsoft.AppCenter.Auth" version="2.0.0"
"Microsoft.AppCenter.Crashes" version="2.0.0"
"Microsoft.Azure.Mobile.Client" version="3.1.0"
"Microsoft.Azure.Mobile.Client.SQLiteStore" version="3.1.0"
"Microsoft.Bcl" version="1.1.10"
"Microsoft.Bcl.Build" version="1.0.21"
"Microsoft.CSharp" version="4.4.0"
"Microsoft.Data.Edm" version="5.8.3"
"Microsoft.Data.OData" version="5.8.3"
"Microsoft.Data.Services.Client" version="5.8.3"
"Microsoft.IdentityModel.Clients.ActiveDirectory" version="3.14.0"
"Microsoft.Net.Http" version="2.2.29"
"Microsoft.NETCore.Platforms" version="2.0.0"
"Microsoft.Win32.Primitives" version="4.3.0"
"NETStandard.Library" version="2.0.0"
"Newtonsoft.Json" version="10.0.3"
"PCLCrypto" version="2.0.147"
"PDFTron.Android" version="6.8.5"
"PDFTron.Android.FAB" version="1.0.0"
"PDFTron.Android.PageCropper" version="1.0.0"
"PDFTron.Android.Tools" version="6.8.4"
"PInvoke.BCrypt" version="0.5.97"
"PInvoke.Kernel32" version="0.5.97"
"PInvoke.NCrypt" version="0.5.97"
"PInvoke.Windows.Core" version="0.5.97"
"Plugin.CurrentActivity" version="1.0.1"
"Plugin.Fingerprint" version="1.4.6"
"Plugin.Permissions" version="2.0.1"
"Rg.Plugins.Popup" version="1.1.4.168"
"SQLite.Net.Core-PCL" version="3.1.1"
"SQLitePCLRaw.bundle_green" version="1.1.8"
"SQLitePCLRaw.core" version="1.1.8"
"SQLitePCLRaw.lib.e_sqlite3.android" version="1.1.8"
"SQLitePCLRaw.provider.e_sqlite3.android" version="1.1.8"
"System.AppContext" version="4.3.0"
"System.Collections" version="4.3.0"
"System.Collections.Concurrent" version="4.3.0"
"System.ComponentModel.EventBasedAsync" version="4.3.0"
"System.ComponentModel.TypeConverter" version="4.3.0"
"System.Console" version="4.3.0"
"System.Diagnostics.Debug" version="4.3.0"
"System.Diagnostics.Tools" version="4.3.0"
"System.Diagnostics.Tracing" version="4.3.0"
"System.Dynamic.Runtime" version="4.3.0"
"System.Globalization" version="4.3.0"
"System.Globalization.Calendars" version="4.3.0"
"System.IO" version="4.3.0"
"System.IO.Compression" version="4.3.0"
"System.IO.Compression.ZipFile" version="4.3.0"
"System.IO.FileSystem" version="4.3.0"
"System.IO.FileSystem.Primitives" version="4.3.0"
"System.Linq" version="4.3.0"
"System.Linq.Expressions" version="4.3.0"
"System.Linq.Queryable" version="4.3.0"
"System.Net.Http" version="4.3.3"
"System.Net.Primitives" version="4.3.0"
"System.Net.Requests" version="4.3.0"
"System.Net.Sockets" version="4.3.0"
"System.Numerics.Vectors" version="4.5.0"
"System.ObjectModel" version="4.3.0"
"System.Reflection" version="4.3.0"
"System.Reflection.Extensions" version="4.3.0"
"System.Reflection.Primitives" version="4.3.0"
"System.Resources.ResourceManager" version="4.3.0"
"System.Runtime" version="4.3.0"
"System.Runtime.Extensions" version="4.3.0"
"System.Runtime.Handles" version="4.3.0"
"System.Runtime.InteropServices" version="4.3.0"
"System.Runtime.InteropServices.RuntimeInformation" version="4.3.0"
"System.Runtime.Numerics" version="4.3.0"
"System.Runtime.Serialization.Formatters" version="4.3.0"
"System.Runtime.Serialization.Primitives" version="4.3.0"
"System.Security.Cryptography.Algorithms" version="4.3.0"
"System.Security.Cryptography.Encoding" version="4.3.0"
"System.Security.Cryptography.Primitives" version="4.3.0"
"System.Security.Cryptography.X509Certificates" version="4.3.1"
"System.Spatial" version="5.8.3"
"System.Text.Encoding" version="4.3.0"
"System.Text.Encoding.Extensions" version="4.3.0"
"System.Text.RegularExpressions" version="4.3.0"
"System.Threading" version="4.3.0"
"System.Threading.Tasks" version="4.3.0"
"System.Threading.Timer" version="4.3.0"
"System.Xml.ReaderWriter" version="4.3.0"
"System.Xml.XDocument" version="4.3.0"
"System.Xml.XmlDocument" version="4.3.0"
"Validation" version="2.4.15"
"WindowsAzure.Storage" version="8.4.0"
"Xam.iTextSharpLGPL" version="0.1.0"
"Xam.Plugin.Geolocator" version="4.0.1"
"Xam.Plugin.Media" version="3.0.1"
"Xam.Plugin.Version" version="1.0.0.0"
"Xam.Plugins.Forms.RoundedBoxView" version="1.0.0.7"
"Xam.Plugins.Settings" version="2.5.8"
"Xamarin.Android.Arch.Core.Common" version="1.1.1.1"
"Xamarin.Android.Arch.Core.Runtime" version="1.1.1.1"
"Xamarin.Android.Arch.Lifecycle.Common" version="1.1.1.1"
"Xamarin.Android.Arch.Lifecycle.Extensions" version="1.1.1.1"
"Xamarin.Android.Arch.Lifecycle.LiveData" version="1.1.1.1"
"Xamarin.Android.Arch.Lifecycle.LiveData.Core" version="1.1.1.1"
"Xamarin.Android.Arch.Lifecycle.Runtime" version="1.1.1.1"
"Xamarin.Android.Arch.Lifecycle.ViewModel" version="1.1.1.1"
"Xamarin.Android.Support.Animated.Vector.Drawable" version="28.0.0.1"
"Xamarin.Android.Support.Annotations" version="28.0.0.1"
"Xamarin.Android.Support.AsyncLayoutInflater" version="28.0.0.1"
"Xamarin.Android.Support.Collections" version="28.0.0.1"
"Xamarin.Android.Support.Compat" version="28.0.0.1"
"Xamarin.Android.Support.Constraint.Layout" version="1.1.2"
"Xamarin.Android.Support.Constraint.Layout.Solver" version="1.1.2"
"Xamarin.Android.Support.CoordinaterLayout" version="28.0.0.1"
"Xamarin.Android.Support.Core.UI" version="28.0.0.1"
"Xamarin.Android.Support.Core.Utils" version="28.0.0.1"
"Xamarin.Android.Support.CursorAdapter" version="28.0.0.1"
"Xamarin.Android.Support.CustomTabs" version="28.0.0.1"
"Xamarin.Android.Support.CustomView" version="28.0.0.1"
"Xamarin.Android.Support.Design" version="28.0.0.1"
"Xamarin.Android.Support.DocumentFile" version="28.0.0.1"
"Xamarin.Android.Support.DrawerLayout" version="28.0.0.1"
"Xamarin.Android.Support.Fragment" version="28.0.0.1"
"Xamarin.Android.Support.Interpolator" version="28.0.0.1"
"Xamarin.Android.Support.Loader" version="28.0.0.1"
"Xamarin.Android.Support.LocalBroadcastManager" version="28.0.0.1"
"Xamarin.Android.Support.Media.Compat" version="28.0.0.1"
"Xamarin.Android.Support.Print" version="28.0.0.1"
"Xamarin.Android.Support.SlidingPaneLayout" version="28.0.0.1"
"Xamarin.Android.Support.SwipeRefreshLayout" version="28.0.0.1"
"Xamarin.Android.Support.Transition" version="28.0.0.1"
"Xamarin.Android.Support.v4" version="28.0.0.1"
"Xamarin.Android.Support.v7.AppCompat" version="28.0.0.1"
"Xamarin.Android.Support.v7.CardView" version="28.0.0.1"
"Xamarin.Android.Support.v7.MediaRouter" version="28.0.0.1"
"Xamarin.Android.Support.v7.Palette" version="28.0.0.1"
"Xamarin.Android.Support.v7.RecyclerView" version="28.0.0.1"
"Xamarin.Android.Support.Vector.Drawable" version="28.0.0.1"
"Xamarin.Android.Support.VersionedParcelable" version="28.0.0.1"
"Xamarin.Android.Support.ViewPager" version="28.0.0.1"
"Xamarin.Build.Download" version="0.4.7"
"Xamarin.Essentials" version="1.1.0"
"Xamarin.Forms" version="3.4.0.1008975"
"ZXing.Net.Mobile" version="2.3.1"
"ZXing.Net.Mobile.Forms" version="2.3.1"


PDFTron is a monster and it's what pushed us to use Multi-Dex originally.

[jonathanpeppers]

Multi-Dex = On, Dex Compiler = d8, Code Shrinker = r8

You should not need multidex if you have a code shrinker. You may be hitting an issue I saw when using r8+multidex: dotnet/android@34ee473

But let's recap...

What settings are for Debug and which ones Release?

• Debug - you want fast builds, you sacrifice APK size / startup time for a faster "developer loop".
• Release - you want your APK to be small & startup fast, at the tradeoff of a longer build. Hopefully you are running these on CI, anyway.

So Debug these are the settings that make sense:

• Linker is off
• Code Shrinker is off
• Multidex is on--only if your app hits the dex limit without it

Release:

• Linker is on
• Code Shrinker is on -- if you want to use it, you may need proguard rules
• Multidex should be off if you are using the Code Shrinker (should keep you under the dex limit)

Also 🤦‍♂

There is an issue in VS 2019 16.1, where this dropdown won't let you "unselect" a code shrinker:

If you hit this, you can edit the csproj file manually to workaround. Just remove <AndroidLinkTool> completely.

Questions

• After you configure your app as prescribed above for Debug/Release? does it work?
• Does your app work without the new d8/r8 settings? (Using dx and either no code shrinker or proguard)

Hope that helps!

[brad-dixon]

@jonathanpeppers thanks for the clarification and the targeted approach for the settings.

The good news is, I've managed to get everything building and there's no doubt that this has been a learning experience for me.

There's no bad news as such but something that definitely came into play but was a little more hidden away was the "Desugar" step. If it's set in .csproj, that setting doesn't appear (at least I couldn't find it) when you open up the project properties in VS 2019 for Mac. I added it after reading your blog to see if it would help me. Of course, it didn't, at least, it didn't when needing to play with the other key build properties.

When I made the settings you suggested above, "Desugar" often got in the way. I had it set but it didn't want to play with the settings you had provided in your last suggestion.

Good news though, I do have "None" however as an option for the Code Shrinker setting so Windows is clearly different in that regard.

Now, before, I go on, I think the actual root cause of my issue was something to do with me needing to up the target version of the build to accommodate Essentials along with Xamarin.Android.Arch.Lifecycle.LiveData being somewhat out of whack when moving to target 9.0 ...!

Then when @jamesmontemagno and @Redth mentioned proguard and d8/r8, I made changes to the project to accommodate the new settings and then boom, different errors started to appear (not through anyone's fault, it's just that the goalposts had changed). The one thing that I thought I was sure of though is that Multidex was very much a requirement but as it turns out and as you already know Jonathan, it's not.

So I got curious and today and I performed a build on all of the different combinations of the 5 major settings that had come into the mix throughout this whole thing, ignoring those that were called out in your d8/r8 blog post. This was the result ...

As you can see, multiple different combinations allowed for a successful build but more interestingly, no Multidex in a lot of those cases. You can also see that Multidex on with Desugar on was never successful, so I was always up against it given I thought Multidex was a must for this project.

I figure you guys are all over that stuff already but for anyone else who comes across this thread, they may very well find it useful.

Refinement of the APK size or build speed is not currently something I am concerned with. Just getting the thing to build again was my biggest challenge and now I have a whole heap of options from a settings perspective, noting that Debug and Release will typically be different, which is nothing new.

So thanks to you guys for helping me through this. I very much appreciate it.

Thanks again!!!

[jonathanpeppers]

@brad-dixon <AndroidEnableDesugar> is a setting you should probably just remove, as Google has moved support to d8 for this.

What is doing? It "desugars" Java 8+ code so that it will run on Android. It removes the "syntactic sugar".

If you hit an error saying something like "dx failed, XYZ is not supported", then your app uses a Java library using Java 8+ features. You would be better off just using <AndroidDexTool>d8</AndroidDexTool> instead in that case. It supports desugaring by default.

@brad-dixon if there is a combination here, that seems like it should work. Can you post a bug on xamarin-android with a build log?