[BUG] No remote movie can be played on Immich Mobile using a Self Signed CA and a proxy in front of IMMICH server/microservices

[amitrea]

The bug

Please NOTICE
This is NOT an issue of transcoding. Please read well the thread.
This is SSL issue related to flutter videoplayer that IMMICH mobile app is using to play remote videos on Android mobile.
Trusted Self signed certificates (registered in Android) should be valid also for playing videos as it is for images or other API calls to IMMICH server.

I cleanly install Immich mobile on two Android phones (Samsung & Huawei - no relevance as long as it worked on previous versions of Immich Mobile 1.89.0).
On one phone I had it already installed, version 1.89.0. I cleanup cache and data for Immich Mobile App and then re-authenticated.
On the second phone I installed for the first time.
Movies don't play on both mobile devices in Android Immich App.

All images and movies previews looks OK. Seeing a full picture works. Playing an existing movie doesn't work. It is stuck on loading the stream. I tried random movies and have the same issue.

Previous to version 1.89.0 I could play movies. I remember that I didn't tried to play a movie with version 1.89.0, but for sure with the previous versions.

In Immich Mobile logs I could not find any INFO, WARNING or ERROR.
On the server side (server or microservices) no INFO, WARNING or ERROR.

On Immich Web (mobile or desktop browser) playing movies works fine.

The OS that Immich Server is running on

Fedora 38

Version of Immich Server

v1.90.1

Version of Immich Mobile App

v1.90.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

N/A (Everything works fine on Web)

Your .env content

N/A (Everything works fine on Web)

Reproduction steps

Please see in the description.

Additional information

Please note I am using my own PKI. I have installed the private root CA into Android (User Certificates), so it's in trust store of Android. I checked this in every browser I have on my mobile to see if my ROOT CA certificate is considers, and indeed no more SSL warnings/errors.

In Immich App I enabled Self-Signed cert. All functionality except videoplayer flutter plugin is working fine.
It seems that the videoplayer plugin uses its own SSLSocket.

I saw the error message in Caddy: Unknown certificate but only when I try to play a movie. For the rest of Immich API calls I do not get SSL error.

For more info please check a bit the code for videoplayer flutter plugin here.

In my opinion Immich should trust as well the certificates user adds in Android trust store and probably no need to specify "Allow Self Signed Certificates". This option is a security issue - to trust every Self Signed Certificate. This could be an option only in a DEV environment.

[alextran1502]

Did you change anything relating in the transcoding video settings?

[amitrea]

Hi @alextran1502

Yes, target resolution is 4k. It was the same before, when I was able to play them from Immich Android app.

However, I highlight it again, from browser it's fine.

[alextran1502]

Did you change the codec? Can you post the screenshot of transcoding settings?

[amitrea]

Sure.

[alextran1502]

Can you try revert the transcoding settings to default, then record and upload a new videos and see if it works on your phone?

[amitrea]

I suppose you want to identify where is the issue.

I will record a video, upload and then play. Can you remind me what was the default resolution? 1440p or 1024p?

[alextran1502]

The default is 720p.There is also a button to reset to default at the bottom left corner

[amitrea]

Test 1:
By mistake, I was a bit focus on making the movie and forgot to reset the resolution, still 4k.

I opened immich Android app and uploaded. Playing is fine.
I deleted locally from my phone Gallery and tried again. It doesn't work.

Test 2:
I changed the resolution to 720p. The same as above, playing OK until you delete from local Gallery.

Test 3:
Change resolution to original. The same as above.

My conclusion, maybe yours is different: As long the the movie is on my local Gallery it works, if I delete the streaming from server doesn't work.

Can you please try on your side as well?

[amitrea]

I also reset all values from Transcoding section to default. Did the same as in previous tests and the same behaviour.
It doesn't work if I delete it from local Gallery.

[alextran1502]

What are the movie settings on your phone's camera?

[amitrea]

4k

[alextran1502]

Do you see any logs in the server container when you are trying to stream the video on the mobile app?

[amitrea]

As said in the description no logs on server side (server or microservices) and on mobile app logs.

By the way, I mentioned that the issue is on two distinct Android phones: Huawei Mate Pro and a Samsung A somthing (do not remember).

If I login to immich from browsers on both phones it plays the videos OK.

[tomikonieczny]

Mobile app (android S23 Ultra) v. 1.90.2 build.2114

Transcoding OFF

It looks like there is an issue with playing HEVC media, H.264 is fine.

[amitrea]

I understand that the application (web or mobile) streams the transcoded video file and play it on the device.
So, in this case does not matter if the video was recorded in H.265 because the transcoded video, as one can see in the transcoding settings I attached above, is H.264.

In your case maybe would be a different issue!
When you say "H.264 is fine", it plays the video from mobile disk or only from remote (Immich)?
It is working to me as well when I have the mp4 locally but when I delete it from my Phone Gallery (not through Immich App)it doesn't play the remote one.

My videos are encoded in HEVC (camera calls it Efficient Video Format) and the video player that Immich Android App is using, does't have any issue to play it as long as it is on local disk. I checked the format of my video with `ffmpeg -i video_1.mp4'.

Something is wired - at least an INFO, better an exception !

[tomikonieczny]

It is working to me as well when I have the mp4 locally but when I delete it from my Phone Gallery (not through Immich App)it doesn't play the remote one.

Yes, I've just recreated exact same case. When HEVC file is locally available player works otherwise doesn't.

Logs from immich_server:
[Nest] 7 - 12/08/2023, 10:05:41 PM ERROR [ErrorInterceptor] Failed to serve file
[Nest] 7 - 12/08/2023, 10:05:41 PM ERROR [ErrorInterceptor] Error: Request aborted

I've been logged in by external IP (port 2283 and 5432 are forwarded to local IP). I've decided to log in by local IP and then HEVC files started been playing.. Relogin helped? Missing forwarded ports to local IP?

[amitrea]

OK, I tested also connecting with Immich mobile app using directly the FQDN behind the proxy (http://immich-server.svc.local:3001/api) and started being playing the videos.
I use latest Caddy v2 proxy behind all my services, with SSL stop on proxy. The URL is https://immich.svc.local). The authentication is done through an OpenID implementation.

As mentioned already everything is working fine through the proxy when accessing the application through browser, using the last URL provided above.

So, the bug is in Immich Mobile App or in a dependency library (flutter videoplayer plugin).

...

Later on after a bit of investigation:

Please note I am using my own PKI. I have installed the private root CA into Android (User Certificates), so it's in trust store of Android. I checked this in every browser I have on my mobile to see if my ROOT CA certificate is considers, and indeed no more SSL warnings/errors.

In Immich App I enabled Self-Signed cert. All functionality except videoplayer flutter plugin is working fine.
It seems that the videoplayer plugin uses its own SSLSocket.

I saw the error message in Caddy: Unknown certificate but only when I try to play a movie. For the rest of Immich API calls I do not get SSL error.

For more info please check a bit the code for videoplayer flutter plugin here.

In my opinion Immich should trust as well the certificates user adds in Android trust store and probably no need to specify "Allow Self Signed Certificates". This option is a security issue - to trust every Self Signed Certificate. This could be an option only in a DEV environment.

I will amend the description with my findings.

[Liujun3712]

I found the same issue when I changed the transcode setting to HEVC and re-transcode all the videos today. All the video just cannot playback.
Now, I am trying to use VP9 codec and re-transcode all.

Version of Immich Server
v1.90.2

Version of Immich Mobile App
v1.90.0 in iOS

[amitrea]

This is NOT an issue of transcoding. Please read well the thread.
Please open another issue for your issue.

This is SSL issue related to flutter videoplayer that IMMICH mobile app is using to play videos on Android mobile.
Trusted Self signed certificates (registered in Android) should be valid also for playing videos as it is for images or other calls to IMMICH server.

[br4yd]

I can confirm this bug. I run my server on a Synology NAS behind their proxy server using Tailscale to connect from the outside because no public IPv4 (provider only serves DS_Lite). I had everything running via HTTP but eventually switched to HTTPS using a self-signed cert (can only be self-signed because as mentioned it's running inside a Tailscale network and is therefore not reachable from the outside). Since switching to that self-signed certificate I can't playback videos or live photos on mobile. I can however play them back on web.

[mmanjos]

I'm seeing the same thing on my environment:

• Immich v1.92.0 on docker
• nginx on the same node that runs the docker container
• SSL certificate used by nginx (server.crt and server.key below) is a valid certificate from my own self-hosted CA. The CA root is imported onto the android phone and other services respect the certificates that are signed by it without errors.

nginx is configured like this:

server {
  listen      443 ssl;
  server_name photos.my.domain.invalid;
  root /var/www/html;
  index index.html index.htm;
  client_max_body_size 5000M;

  ssl_certificate       /etc/ssl/private/server.crt; # this is a cert signed by my home root CA that validates server_name above
  ssl_certificate_key   /etc/ssl/private/server.key;

  location / {
    client_max_body_size 5000M;
    proxy_pass http://localhost:2283; # this is immich running on docker 0.0.0.0:2283->3001/tcp
    proxy_set_header Host              $http_host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_http_version 1.1;
    proxy_set_header   Upgrade    $http_upgrade;
    proxy_set_header   Connection "upgrade";
    proxy_redirect off;
  }
}

I've tried a handful of different reverse proxy configs, but based on the nginx logs and the behavior of the app, it feels like it's failing at the TLS session whenever a video is requested for playback on the phone (i.e. no actual HTTP request makes it into the nginx logs - it just fails silently)

[jrasm91]

AFAIK the image and video players in flutter don't use the http interceptor that has been added to solve the issue with self-signed certificates elsewhere.

[mertalev]

You're right, feat/exoplayer is the relevant branch. I merged it into main and did some cleanup.

[squanbo]

I encountered the same issue when using a reverse proxy. I am using a self-signed certificate and have enabled the 'Allow self-signed certificates' option in the settings. When opening a video on the Android client, the app crashes.
`ApiException 400: TLS/SSL communication failed: GET /partners (Inner exception: HandshakeException: Handshake error in client (OS Error:
CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:393)))

#0 _SecureFilterImpl._handshake (dart:io-patch/secure_socket_patch.dart:99)
#1 _SecureFilterImpl.handshake (dart:io-patch/secure_socket_patch.dart:143)
#2 _RawSecureSocket._secureHandshake (dart:io/secure_socket.dart:920)
#3 _RawSecureSocket._tryFilter (dart:io/secure_socket.dart:1049)
`

[rovo89]

Of course I'd like to have support for remote videos, but even if this doesn't get added immediately, at least the crash should be fixed... I assume this might also affect other situations where the connection failed.

[yotis1982]

Here is my workaround, I hope this helps a bit.

I can confirm it has nothing to do with transcoding, as already mentioned above.

I have my own PKI CA (using certstrap app), using Caddy as reverse proxy and having the self-signed certificated by my CA.
In Immich, if I try to login with the https reverse proxy URL, I cannot (it throws an error) unless I enable "Allow self-signed SSL certificates". If I enable this setting in Immich mobile app, I can login but the videos do not load (basically just a static image and cannot "play" them).

What I did (and this is on iOS ecosystem, but I think will work on Android) to overcome this issue, is that I installed the PKI CA certificate on iOS. I just attached the CA certificate to an email send to myself and once I open the attachment the iOS is asking if I want to install the profile attached to the CA (with warnings that this is insecure because self-signed, etc..)
Once install, in the Setting > Certificate Trust Settings I need to enable the trust for my CA certificate.

Then Immich app allow me to login to the https reverse proxy URL without enabling "allow self-signed SSL certificates".
Videos work fine on Immich app after going through the above.

Not an expert here, but I think the player has a problem with "self-signed SSL certificates" without a CA certificate trusted (even if this CA is also self-signed").

Hope this helps!

[phipz]

This issue is still open and persisting.

For me, on Immich iOS:
Video playback via the HTTP port works.
Video playback via the HTTPS reverse proxy does not work, while Login/Browsing works.

Allow-self-signed SSL Certificates is on.

My SSL certificate is from Let's Encrypt for my public domain, while I dial in with the local IP address (hence an SSL error for the hostname mismatch, which should normally be skipped with the above setting turned ON).

[ckuyehar]

@phipz - i just created a guide documenting how to securely setup TLS reverse proxy on the Internet for anywhere Immich mobile app use.

you can review these docs, https://github.com/ckuyehar/immich/blob/ckuyehar-docs-updates/docs/docs/guides/remote-access.md and https://github.com/ckuyehar/immich/blob/ckuyehar-docs-updates/docs/docs/administration/reverse-proxy-tls.md

please note: this doesn't resolve the issue - this merely works around the existing capabilities of Immich today

[rovo89]

Remote video playback and image download are failing for me on Android because I have mTLS configured (with a proper Let's Encrypt certificate). My wife is getting more and more annoyed about this and said she would rather pay for Google Photos than continue to live with these limitations (especially the video)... which I don't want, so I started looking into this myself.

A major problem seems to be the many levels of abstraction and different packages used. Flutter (using Dart), Jetpack, Android framework... that make it hard to follow the call chain and pass the required information down to the places where connections are actually used.

For most parts of the app, Immich uses HttpSSLCertOverride to set the client certificate and allow self-signed certificates (for the Immich server host only), which is set globally here. But I think that only applies to places where Dart's standard HttpClient class is used. Under the hood, I think it makes adjustments to OpenSSL's SSLCertContext which is wrapped by Dart's SecurityContext. My understanding is that the native HTTP client isn't involved at all here, but OpenSSL is used directly.

The video player seems to use the Android native HTTP client which is created here. DefaultHttpDataSource comes from here and uses HttpURLConnection. IIUC, Android uses a customized OpenJDK with their own handlers, in this case com.android.okhttp.HttpsHandler. This seems to be the place where they set SSL options, which uses HttpsURLConnection.getDefaultSSLSocketFactory(). I haven't tracked it down completely yet, but I think SSLContext.init() is where a client certificate would be specified.

I have just written this down to understand the current flow and maybe get corrections from someone. Next step would be finding a way to pass the client certificate to the appropriate place without breaking abstraction... Maybe it would be sufficient to set call HttpsURLConnection.setDefaultSSLSocketFactory() in Immich code?

For completeness, the download library has some code to accept untrusted certificates which calls exactly that method with an accept-all trust manager, indicating that it could profit from this approach as well.

[alextran1502]

@rovo89 what are the differences of your setup vs a more traditional reverse proxy setup with Let's Encrypt generating SSL certificate? I am asking because the route that most users take doesn't run into this issue

[rovo89]

I use client certificates (aka. mTLS) for all the stuff hosted on my home server. If a device doesn't present its valid certificate, the connection is canceled already during the handshake. That gives me some peace of mind because I don't need to care much about the security of all the stuff I host, I can be very sure that this first line of defense won't even let attackers see a password prompt or anything. And since my browser is configured to automatically choose the client certificate for everything under the my domain, it's transparent for me.

A few versions back, support for uploading client certificates was added to the mobile app, so I take it that such a setup isn't completely out of focus, but it's not 100% complete. And with two kids, watching the videos my partner took on her phone is a rather frequent thing, which brings it close to a show-stopper for us.

[rovo89]

Oh, and I do use a wildcard certificate from Let's Encrypt for my domain and subdomains for all the hosted services. The server certificate isn't the problem, the client certificate is (which is using a simple self-operated CA, but that shouldn't matter).

[amigthea]

@rovo89 what are the differences of your setup vs a more traditional reverse proxy setup with Let's Encrypt generating SSL certificate? I am asking because the route that most users take doesn't run into this issue

I was worried when I migrated to immich because of this mTLS issue, but I have the scenario you described (let's encrypt for server certificates and mTLS, with self signed CA, for client) and I can play video remotely with no problem. Is the OP problem related only to mTLS and self signed server certificate?

[rovo89]

Huh, that's strange. Are you on Android as well? For me the app crashes after a few seconds whenever I try to watch a video that's not on my device (either taken by my partner or deleted locally). Before the switch to the new video player, it just had a black screen.

Unfortunately there are multiple scenarios described in this issue, making it a bit hard to follow. Multiple issues have been reported in the past months, but I think they have all been closed as duplicates of this one.

[ktm-91]

I was worried when I migrated to immich because of this mTLS issue, but I have the scenario you described (let's encrypt for server certificates and mTLS, with self signed CA, for client) and I can play video remotely with no problem. Is the OP problem related only to mTLS and self signed server certificate?

I have only server certificate (no mTLS), generated by Let's Encrypt, and I cannot play videos from the Android app (since a few version updates it crashes when I try)

[amigthea]

Huh, that's strange. Are you on Android as well? For me the app crashes after a few seconds whenever I try to watch a video that's not on my device (either taken by my partner or deleted locally). Before the switch to the new video player, it just had a black screen.

Unfortunately there are multiple scenarios described in this issue, making it a bit hard to follow. Multiple issues have been reported in the past months, but I think they have all been closed as duplicates of this one.

android 14, oneplus 9 pro, Immich app v.1.230.0
of course tested on a video that's on my backup server only

I have only server certificate (no mTLS), generated by Let's Encrypt, and I cannot play videos from the Android app (since a few version updates it crashes when I try)

that's really weird, I agree with @rovo89, those all seems different scenarios

[ryan77627]

Huh, that's strange. Are you on Android as well? For me the app crashes after a few seconds whenever I try to watch a video that's not on my device (either taken by my partner or deleted locally). Before the switch to the new video player, it just had a black screen.

Same here, using a standard reverse proxy setup through Caddy with the exception being a custom CA (self-hosted), no mTLS. I am trying to get some logs, but I cannot get any to show up in the built-in logger. When I have some time (moving for school currently) I'll try to get the logs through logcat directly since this is an entire app crash.

[amigthea]

errata corrige
I tried to backup a new video, delete it from my phone, and play it remotely over mTLS: the app crash after a brief loading screen, the same is happening with the old video, must have messed up the first try somehow

[amitrea]

First, the issue is related to having your own PKI for services exposed behind a proxy.

Android app is trying to play a video uploaded into Immich, and deleted locally to save space on your mobile. See this comment.
If you have the video on your phone you will not see this bug.

I am using a reverse proxy (Caddy) to terminate SSL/TLS connections, which offloads the encryption and decryption processes from a part of my backend services including Immich.

I am not exposing the services outside, on internet, so no need to maintain both a PKI infrastructure and Lent's encrypt periodic renewal (cert manager or something else).

I imported as trusted certificate the self signed root certificate on all devices that wants to use the services behind the proxy.

I strongly think that the bug is in flutter videoplayer, as I investigated a bit in detail at that time.
Someone, already commented here that indeed this could be the problem.

Other comments to previous comments:

Would be interesting to see how one can use mTLS with Let's Encrypt certificates.
Let's Encrypt certificates support the "TLS Client Authentication" feature, allowing them to be used for client authentication in theory. However, this is often not the most suitable option for most scenarios. It's important to ensure that your system doesn't indiscriminately accept any certificate issued by Let's Encrypt. In such cases, it may be more effective to use your own private Certificate Authority (CA).
Also Let's Encrypt issue certificates with fix profiles which typically ins not suitable for client authentication.

The mTLS is more used in context of secure communication between two or more trusting services (service mesh). Of course you can use also between your client devices (apps, or browsers) and a service or proxy.

[alextran1502]

I think this issue is related to different HTTP clients used in the video player. So ideally, the way to fix this issue is to pass the HTTP client with the certificate to all requests, including the video player.

@rovo89 For wife-approval-factor, maybe you can set up a VPN for her?

[ryan77627]

@amitrea Ah, apologies. I thought the Native Player was supposed to fix the self-managed PKI issues we had. I've been loosely following this issue for the past 7-ish months, but haven't been keeping up to date. I'd prefer to just use my own certs, but I could provision public certificates and use them if need be. Anyways, @alextran1502 I have some logs, I do not know how useful they are. Seems the app crash stems from ExoPlayer running into a source issue (stemming from the TLS stuff). App crashes presumably because it cannot handle whatever ExoPlayer returns to it (again, hard to tell because the installed app seems to be minified. If there are debug builds somewhere let me know).

Immich crash:

FATAL EXCEPTION: main
Process: app.alextran.immich, PID: 28260
java.lang.ClassCastException: h0.q cannot be cast to java.lang.Error
	at E5.b.k0(Unknown Source:35)
	at j0.i0.C1(Unknown Source:2)
	at j0.i0.c0(Unknown Source:0)
	at j0.g0.b(Unknown Source:4)
	at f0.n$c.a(Unknown Source:17)
	at f0.n.h(Unknown Source:16)
	at f0.n.a(Unknown Source:0)
	at f0.m.run(Unknown Source:6)
	at f0.n.f(Unknown Source:67)
	at j0.i0.f2(Unknown Source:422)
	at j0.i0.n1(Unknown Source:192)
	at j0.i0.s1(Unknown Source:0)
	at j0.i0.m0(Unknown Source:0)
	at j0.Y.run(Unknown Source:4)
	at android.os.Handler.handleCallback(Handler.java:991)
	at android.os.Handler.dispatchMessage(Handler.java:102)
	at android.os.Looper.loopOnce(Looper.java:232)
	at android.os.Looper.loop(Looper.java:317)
	at android.app.ActivityThread.main(ActivityThread.java:8787)
	at java.lang.reflect.Method.invoke(Native Method)
	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:591)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:871)

ExoPlayer dump uploaded in file.
10_01-10-26-46_521.log

[amitrea]

Dear all,

At the time I identified this issue, with the investigation I made, I stopped using the Android application. After a while, seeing not too much interest for it, I stopped following this issue.

Today was an avalanche of messages and was curious 🤨 . Only after @ryan77627 replied, I read a bit what happened here and saw somebody already merged a fix for the Android player.

I then said to download the Android App again and set it back to connect to my Immich instance through the Caddy proxy, using this time the "https://" schema and the same private PKI (with the root certificated added to my android as trusted).

It seems that now I can play remote videos from Immich server, videos that are not locally on my mobile and using https with my own/private PKI.

Thank you to the people that worked on the fix, finally :)

Thank you @yan77627. Your message made me decide to test the app again, so no need to apologise. 🤨

@alextran1502 and the team, you can consider closing this issue as well, form my point of view.

If other issues, as other mentioned here, then maybe they need to create a new issue.

If you want to close it I am happy to do it. Just please let me know.

[alextran1502]

Closing this issue per the OP's request. If others are interested. Please help us open a new issue with your setup

[ktm-91]

Dear all,

At the time I identified this issue, with the investigation I made, I stopped using the Android application. After a while, seeing not too much interest for it, I stopped following this issue.

Today was an avalanche of messages and was curious 🤨 . Only after @ryan77627 replied, I read a bit what happened here and saw somebody already merged a fix for the Android player.

I then said to download the Android App again and set it back to connect to my Immich instance through the Caddy proxy, using this time the "https://" schema and the same private PKI (with the root certificated added to my android as trusted).

It seems that now I can play remote videos from Immich server, videos that are not locally on my mobile and using https with my own/private PKI.

Thank you to the people that worked on the fix, finally :)

Thank you @yan77627. Your message made me decide to test the app again, so no need to apologise. 🤨

@alextran1502 and the team, you can consider closing this issue as well, form my point of view.

If other issues, as other mentioned here, then maybe they need to create a new issue.

If you want to close it I am happy to do it. Just please let me know.

Sorry but I don't understand... I just tried to uninstall and reinstall the Android app, but the bug is still there: when I try to play a video present on Immich server, through a TLS reverse proxy (HAProxy in my case), the app crashes. It's the same bug present since more than a year ago.
I'm using the latest versions of both the Android app and the Immich server

[amigthea]

Today was an avalanche of messages and was curious 🤨 . Only after @ryan77627 replied, I read a bit what happened here and saw somebody already merged a fix for the Android player.

it still crash for me, using both TLS and mTLS through nginx

[ckuyehar]

It seems that now I can play remote videos from Immich server, videos that are not locally on my mobile and using https with my own/private PKI.

@amitrea

• What version of the mobile app are you using?
• Try downloading an image that doesn't exist on your phone. Success?

[bo0tzz]

We're now tracking any unusual networking things like this in #15230.

[amitrea]

It seems that now I can play remote videos from Immich server, videos that are not locally on my mobile and using https with my own/private PKI.

@amitrea

• What version of the mobile app are you using?
• Try downloading an image that doesn't exist on your phone. Success?

I am using Android app version 1.124.0.build.173
Server version: 1.123.0