[META] Experimental network features

[mmomjian]

Experimental network features

Features

There are some network features that have been added to the Immich mobile app through the years that interact with various libraries and background tasks in unexpected ways. These features include:

• HTTP Basic Auth for server URL
• Self-signed SSL certs
• Mutual TLS
• Custom HTTP proxy headers

Limitations

Typically, these network features work fine in the normal library view, but cause issues with:

• video playback
• foreground asset upload
• background asset upload
• asset download
• App crashing

Solutions

Thanks to the recent implementation of auto-switching server URLs in #14437, many of these features are no longer needed for a smooth Immich experience. There are other options, including the use of a VPN or a real SSL cert.

Handling known issues

We have updated the FAQ (link will work once #15228 is merged). This issue can be used to track any known issues with these experimental network features. These issues will not be a priority for the dev team, but can be improved by the community if possible.

Please do not use this discussion for comments like "same" or +1". This is designed only to keep track of current known limitations to these experimental features, or discuss potential solutions and future patches.

[rovo89]

The final sentences sound quite intimidating, so I'm wondering: Is it allowed to discuss analysis of root causes and potential solutions here? Or where would that go?

[mmomjian]

The final sentences sound quite intimidating, so I'm wondering: Is it allowed to discuss analysis of root causes and potential solutions here? Or where would that go?

If the discussion is relevant to potential future PRs or improvements, or the actual underlying issue, that's fine. Often these topics turn into people repetitively saying "I want this feature too", that is what we are trying to avoid

[ckuyehar]

I propose that we use the discussions area to discuss a specific feature. If you start a discussion, mention it in the discussion and ask @mmomjian to update the top comment so that it links to the discussion.

The moment, two or more features are discussed in this single thread, it's going to get chaotic.

[bo0tzz]

It is already know that these features do not work well, so we should avoid rehashing "it's broken" over and over again.
I would propose instead to keep this thread focussed on very specific discussion of why things are not working (eg referencing specific code paths and such) that might result in a PR. That way things should not get chaotic, and there should be no need for more separate issues or discussions (which is chaotic in its own way).

[rovo89]

As per #14845 (comment), "app crashes" should be added to the list of limitations. Happens on Android whenever I navigate to a video, e.g. by clicking on it in the timeline or when swiping through it. I don't even need to click the "play" button to provoke the crash.

[rovo89]

As far as analysis is concerned, I'm copying my findings from #5553 (comment) which is closed meanwhile. My setup uses mTLS (client certificates) and a Let's Encrypt server certificate, I'm using Android.

---

A major problem seems to be the many levels of abstraction and different packages used. Flutter (using Dart), Jetpack, Android framework... that make it hard to follow the call chain and pass the required information down to the places where connections are actually used.

For most parts of the app, Immich uses HttpSSLCertOverride to set the client certificate and allow self-signed certificates (for the Immich server host only), which is set globally here. But I think that only applies to places where Dart's standard HttpClient class is used. Under the hood, I think it makes adjustments to OpenSSL's SSLCertContext which is wrapped by Dart's SecurityContext. My understanding is that the native HTTP client isn't involved at all here, but OpenSSL is used directly.

The video player seems to use the Android native HTTP client which is created here. DefaultHttpDataSource comes from here and uses HttpURLConnection. IIUC, Android uses a customized OpenJDK with their own handlers, in this case com.android.okhttp.HttpsHandler. This seems to be the place where they set SSL options, which uses HttpsURLConnection.getDefaultSSLSocketFactory(). I haven't tracked it down completely yet, but I think SSLContext.init() is where a client certificate would be specified.

I have just written this down to understand the current flow and maybe get corrections from someone. Next step would be finding a way to pass the client certificate to the appropriate place without breaking abstraction... Maybe it would be sufficient to set call HttpsURLConnection.setDefaultSSLSocketFactory() in Immich code?

For completeness, the download library has some code to accept untrusted certificates which calls exactly that method with an accept-all trust manager, indicating that it could profit from this approach as well.

[ckuyehar]

Custom HTTP proxy headers

• I think this issue refers to using the "custom proxy headers" in the Immich mobile app.
• I use this feature and I have no issue with this.

Can someone refer an issue related to this one?

[mmomjian]

Custom HTTP proxy headers

• I think this issue refers to using the "custom proxy headers" in the Immich mobile app.

• I use this feature and I have no issue with this.

Can someone refer an issue related to this one?

It's possible there's no known issues. However we still consider it an experimental feature. Glad it's working well!

[rovo89]

As for the crash in the video player, here is the trace with a debug build:

E/ExoPlayerImplInternal(10356): Playback error
E/ExoPlayerImplInternal(10356):   androidx.media3.exoplayer.ExoPlaybackException: Source error
E/ExoPlayerImplInternal(10356):       at androidx.media3.exoplayer.ExoPlayerImplInternal.handleIoException(ExoPlayerImplInternal.java:737)
E/ExoPlayerImplInternal(10356):       at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:709)
E/ExoPlayerImplInternal(10356):       at android.os.Handler.dispatchMessage(Handler.java:105)
E/ExoPlayerImplInternal(10356):       at android.os.Looper.loopOnce(Looper.java:232)
E/ExoPlayerImplInternal(10356):       at android.os.Looper.loop(Looper.java:317)
E/ExoPlayerImplInternal(10356):       at android.os.HandlerThread.run(HandlerThread.java:85)
E/ExoPlayerImplInternal(10356):   Caused by: androidx.media3.datasource.HttpDataSource$InvalidResponseCodeException: Response code: 400
E/ExoPlayerImplInternal(10356):       at androidx.media3.datasource.DefaultHttpDataSource.open(DefaultHttpDataSource.java:401)
E/ExoPlayerImplInternal(10356):       at androidx.media3.datasource.StatsDataSource.open(StatsDataSource.java:87)
E/ExoPlayerImplInternal(10356):       at androidx.media3.exoplayer.source.ProgressiveMediaPeriod$ExtractingLoadable.load(ProgressiveMediaPeriod.java:1085)
E/ExoPlayerImplInternal(10356):       at androidx.media3.exoplayer.upstream.Loader$LoadTask.run(Loader.java:450)
E/ExoPlayerImplInternal(10356):       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
E/ExoPlayerImplInternal(10356):       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
E/ExoPlayerImplInternal(10356):       at java.lang.Thread.run(Thread.java:1012)
D/AndroidRuntime(10356): Shutting down VM
E/AndroidRuntime(10356): FATAL EXCEPTION: main
E/AndroidRuntime(10356): Process: app.alextran.immich.debug, PID: 10356
E/AndroidRuntime(10356): java.lang.ClassCastException: androidx.media3.datasource.HttpDataSource$InvalidResponseCodeException cannot be cast to java.lang.Error
E/AndroidRuntime(10356): 	at me.albemala.native_video_player.NativeVideoPlayerViewController.onPlayerError(NativeVideoPlayerViewController.kt:137)
E/AndroidRuntime(10356): 	at androidx.media3.exoplayer.ExoPlayerImpl.lambda$updatePlaybackInfo$16(ExoPlayerImpl.java:2111)
E/AndroidRuntime(10356): 	at androidx.media3.exoplayer.ExoPlayerImpl$$ExternalSyntheticLambda27.invoke(D8$$SyntheticClass:0)
E/AndroidRuntime(10356): 	at androidx.media3.common.util.ListenerSet$ListenerHolder.invoke(ListenerSet.java:342)
E/AndroidRuntime(10356): 	at androidx.media3.common.util.ListenerSet.lambda$queueEvent$0(ListenerSet.java:226)
E/AndroidRuntime(10356): 	at androidx.media3.common.util.ListenerSet$$ExternalSyntheticLambda1.run(D8$$SyntheticClass:0)
E/AndroidRuntime(10356): 	at androidx.media3.common.util.ListenerSet.flushEvents(ListenerSet.java:248)
E/AndroidRuntime(10356): 	at androidx.media3.exoplayer.ExoPlayerImpl.updatePlaybackInfo(ExoPlayerImpl.java:2174)
E/AndroidRuntime(10356): 	at androidx.media3.exoplayer.ExoPlayerImpl.handlePlaybackInfo(ExoPlayerImpl.java:2008)
E/AndroidRuntime(10356): 	at androidx.media3.exoplayer.ExoPlayerImpl.lambda$new$1$androidx-media3-exoplayer-ExoPlayerImpl(ExoPlayerImpl.java:348)
E/AndroidRuntime(10356): 	at androidx.media3.exoplayer.ExoPlayerImpl$$ExternalSyntheticLambda10.run(D8$$SyntheticClass:0)
E/AndroidRuntime(10356): 	at android.os.Handler.handleCallback(Handler.java:991)
E/AndroidRuntime(10356): 	at android.os.Handler.dispatchMessage(Handler.java:102)
E/AndroidRuntime(10356): 	at android.os.Looper.loopOnce(Looper.java:232)
E/AndroidRuntime(10356): 	at android.os.Looper.loop(Looper.java:317)
E/AndroidRuntime(10356): 	at android.app.ActivityThread.main(ActivityThread.java:8787)
E/AndroidRuntime(10356): 	at java.lang.reflect.Method.invoke(Native Method)
E/AndroidRuntime(10356): 	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:591)
E/AndroidRuntime(10356): 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:871)
I/Process (10356): Sending signal. PID: 10356 SIG: 9

So it crashes here because HttpDataSource.InvalidResponseCodeException is a java.lang.Exception, not a java.lang.Error. The common super-class for both is java.lang.Throwable, and indeed changing the cast and the onError() parameter to Throwable solves the crash. The throbber (animated circle spinning while loading) is simply shown indefinitely and I can navigate to other images.

I'll send a PR for that to both upstream and the Immich fork.
EDIT: It's only in the code that was added to move to Exoplayer.

[ckuyehar]

For completeness, the download library has some code to accept untrusted certificates which calls exactly that method with an accept-all trust manager, indicating that it could profit from this approach as well.

The background_downloader does NOT allow self-signed certificates in release mode. I previously mentioned this in #15188.

However... even if you were to trust your CA and import your client certificate in your mobile device, the background_downloader will still fail. Why? The package background_downloader v8.9.0 doesn't support mutual TLS.

[rovo89]

The background_downloader does NOT allow self-signed certificates in release mode. I previously mentioned this in #15188.

I wasn't planning to use their debug feature. I mentioned it because it shows that their implementation seems to consider whatever is dictated by HttpsURLConnection.setDefaultSSLSocketFactory(). Calling the latter from Immich code should therefore have the same effect, and I have hopes that this approach would also allow to pass client certificates. See KeyManager in SSLContext.init().

[rovo89]

Yeah, my proof of concept for Android is working. 🥳 I added the following in ImmichApp.kt:

val keyStore = KeyStore.getInstance("PKCS12")
val clientCertificateContent = getAssets().open("mykey.pfx")
keyStore.load(clientCertificateContent, "mypassword".toCharArray())

val keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm())
keyManagerFactory.init(keyStore, null)

val sslContext = SSLContext.getInstance("TLS")
sslContext.init(keyManagerFactory.keyManagers, null, SecureRandom())
HttpsURLConnection.setDefaultSSLSocketFactory(sslContext.socketFactory)

I successfully tested videos and image downloads. That's for mTLS only. I see no problem accepting any server certificate, the question is whether that would need to be restricted to only the Immich server host like here. But maybe that could be done by checking the hostnames in the certificate... not sure.

Any comments on whether such an approach would be acceptable? Of course, the imported key would need to be used instead of assets (was just easier for testing) and it would have to be reloaded when the key is changed. Also no idea how I can create a dedicate native "component" for this stuff, I might need some assistance with that once the general idea has been confirmed. And sorry, no idea about iOS.

[ckuyehar]

I successfully tested videos and image downloads.

🥳 congrats! awesome job.

I see no problem accepting any server certificate, the question is whether that would need to be restricted to only the Immich server host like here.

Immich should get out of the business of making that decision and defer that to Android/iOS. Let the OS perform the necessary checks.

Any comments on whether such an approach would be acceptable?

I do have some thoughts... Instead of maintaining/creating another certificate store within the Immich mobile app (abbrev IMA)... IMA should select the certificate from the OS. This will ensure we're using the OS certificate stores (trusted CAs and user) instead of a custom store within IMA. This will also allow IMA (when the time is right) to depreciate "allow self-signed certificates" because the CA would be trusted on the mobile device and a "self-signed" error would never appear.

In other words you should be able to trust the CA, uncheck "allow self-signed certificates" in IMA, select your client certificate in the Android User Store and login to IMA, download images and play videos without issue.

[mertalev]

cc @etnoy if you have any thoughts on this

[bo0tzz]

Immich should get out of the business of making that decision and defer that to Android/iOS. Let the OS perform the necessary checks.

Flutter has its own certificate handling and doesn't use the system's. That's the whole reason we're here in the first place.

[rovo89]

Exactly. It's neither my idea nor my preferred approach to upload the certificate in the Immich app, I'm just following the path that was already taken.

[rovo89]

dart-lang/sdk#50435 sounds like no real solution is planned for the HTTP client in Dart, they suggest using package:cronet_http. No idea about that, I'm not familiar with Dart. Indeed it feels a bit clunky having to upload the certificates again that I already configured in the OS, but it's a one-time thing, so it's acceptable for me as a user with a special setup.

I also thought about adding the self-signed certificate to the trust store, which I would anyway do in the OS, but using it from there has the same limitations as above and I'm not sure if yet another upload button in Immich would be worth it. So the simple switch sounds like an acceptable compromise to me.