[BUG] 502 Bad Gateway

[SadRobot11]

The bug

My Immich server throws 502 Bad Gateway error after running for some time. It works fine after server restart (compose down - compose up or when applying a new update), but after some hours usually the next day, I get the error both on mobile and web (mobile also throws backup error if trying to sync new images). This has been a plague for many months, probably since I started using Immich this year.

When checking immich_proxy logs I see the following error message there:
2023/09/16 17:24:39 [error] 48#48: *47 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: , request: "POST /api/oauth/config HTTP/1.1", upstream: "http://bridge_router_ip:3001/oauth/config", host: "my immich domain"

So I can see that it attempts to connect upstream to a wrong address, the correct one should be 172.18.0.8:3001 as far as I know.

I am using Nginx Proxy Manager, but I don't see any issues in the setup there. I am also using a two router setup: one is connected to internet in the bridge mode (has bridge_router_ip), and the other is servicing the local network while connected to the bridge router for internet.

The OS that Immich Server is running on

Unraid 6.11.5

Version of Immich Server

v1.78.0

Version of Immich Mobile App

v1.78.1 build.102

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
    logging:
      driver: none
    volumes:
      - tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

###################################################################################
# Database
###################################################################################

# NOTE: The following four database variables support Docker secrets by adding a *_FILE suffix to the variable name
# See the docker-compose documentation on secrets for additional details: https://docs.docker.com/compose/compose-file/compose-file-v3/#secrets
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=postgres
DB_DATABASE_NAME=immich

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich_redis

# REDIS_URL will be used to pass custom options to ioredis.
# Example for Sentinel
# {"sentinels":[{"host":"redis-sentinel-node-0","port":26379},{"host":"redis-sentinel-node-1","port":26379},{"host":"redis-sentinel-node-2","port":26379}],"name":"redis-sentinel"}
# REDIS_URL=ioredis://eyJzZW50aW5lbHMiOlt7Imhvc3QiOiJyZWRpcy1zZW50aW5lbDEiLCJwb3J0IjoyNjM3OX0seyJob3N0IjoicmVkaXMtc2VudGluZWwyIiwicG9ydCI6MjYzNzl9XSwibmFtZSI6Im15bWFzdGVyIn0=

# Optional Redis settings:

# Note: these parameters are not automatically passed to the Redis Container
# to do so, please edit the docker-compose.yml file as well. Redis is not configured
# via environment variables, only redis.conf or the command line

# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_USERNAME=
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION=/mnt/user/photos/immich


###################################################################################
# Typesense
###################################################################################
TYPESENSE_API_KEY=some-random-text
# TYPESENSE_ENABLED=false
# TYPESENSE_URL uses base64 encoding for the nodes json.
# Example JSON that was used:
# [
#      { "host": "typesense-1.example.net", "port": "443", "protocol": "https" },
#      { "host": "typesense-2.example.net", "port": "443", "protocol": "https" },
#      { "host": "typesense-3.example.net", "port": "443", "protocol": "https" },
# ]
# TYPESENSE_URL=ha://WwogIHsgImhvc3QiOiAidHlwZXNlbnNlLTEuZXhhbXBsZS5uZXQiLCAicG9ydCI6ICI0NDMiLCAicHJvdG9jb2wiOiAiaHR0cHMiIH0sCiAgeyAiaG9zdCI6ICJ0eXBlc2Vuc2UtMi5leGFtcGxlLm5ldCIsICJwb3J0IjogIjQ0MyIsICJwcm90b2NvbCI6ICJodHRwcyIgfSwKICB7ICJob3N0IjogInR5cGVzZW5zZS0zLmV4YW1wbGUubmV0IiwgInBvcnQiOiAiNDQzIiwgInByb3RvY29sIjogImh0dHBzIiB9Cl0=

###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

# DISABLE_REVERSE_GEOCODING=false
# REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>[email protected]</i><br>Password: <i>demo</i>"
####################################################################################

PUBLIC_LOGIN_PAGE_MESSAGE=

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

####################################################################################
# Alternative API's External Address - Optional
#
# This is an advanced feature used to control the public server endpoint returned to clients during Well-known discovery.
# You should only use this if you want mobile apps to access the immich API over a custom URL. Do not include trailing slash.
# NOTE: At this time, the web app will not be affected by this setting and will continue to use the relative path: /api
# Examples: http://localhost:3001, http://immich-api.example.com, etc
####################################################################################

#IMMICH_API_URL_EXTERNAL=http://localhost:3001

###################################################################################
# Immich Version - Optional
#
# This allows all immich docker images to be pinned to a specific version. By default,
# the version is "release" but could be a specific version, like "v1.59.0".
###################################################################################

#IMMICH_VERSION=

Reproduction steps

Everything works after restart of the server, but after some undetermined amount of hours (usually about a day), I get 502 Bad Gateway when connecting to my Immich server either from the mobile app or from web.

Additional information

No response

[alextran1502]

Hello, do you see the same issue if you use the local IP to access the instance?

[SadRobot11]

Hello, do you see the same issue if you use the local IP to access the instance?

Yes I do, the same 502 Bad Gateway error when trying to go to immich proxy port directly in the browser. The immich proxy log has slightly different error message (only request field is different):

2023/09/17 10:56:04 [error] 41#41: *25 connect() failed (111: Connection refused) while connecting to upstream, client: 172.21.0.1, server: , request: "GET /api HTTP/1.1", upstream: "http://bridge_router_ip:3001/api", host: "my immich domain"

[alextran1502]

I converted the issue into a discussion since I don't think this is Immich's bug.

What is the docker engine version of your machine?

[SadRobot11]

@jrasm91 after restarting the Immich stack everything works for a while, and then the Immich proxy replaces the Ip of Immich server to Ip of my bridge router, and of course can't connect to the server anymore. I have provided my configuration for Immich stack in the original post, I can't figure out how to solve this problem and why is it happening, that's why I am here asking for help. I appreciate your help and thankful for your time, and hate to ask for more, but I am totally stuck.

P. S. I don't see evidence of Immich server not running, nothing suspicious pops up in the server logs.

[jrasm91]

I have never seen something like this happen before. There are lots of people using immich with the default confusion and this has not come up before. Can you explain in now detail your setup? Do you use docker compose? What is your network setup like for the containers? Do you have any custom DNS or other infrastructure in place?

[SadRobot11]

@jrasm91 I am on Unraid 6.11.5 and I use docker compose. My other containers including nginx proxy manager are on a custom bridge network. I use one router/modem in bridge mode (so doesn't really act as a router), which is connected to internet, and another router which is servicing the home network. I do not use any custom DNS in my network, if relevant my DNS provider is Cloudflare.

The IP which replaces the immich_server IP in immich_proxy is the IP of my bridged router/modem. My home network is on 192.168.1.0, including the Unraid server, the bridged router is however on 192.168.100.0 (this subnet was default for that device).

The reason for using a router in bridged mode, is that the bridged router is 5G capable, but doesn't have enough functionality, so I have to use another router for my network (which is not 5G capable).

[jrasm91]

I'm not really sure what to do here, this is a bit outside of my expertise/knowledge. If the main issue is a stale ip causes a networking problem, perhaps it is worth looking into assigning a static ip to the immich-server container.

[SadRobot11]

I am all in for a workaround at this point!
Since your advice, I have tried to specify a network in the compose file and specify IP for immich_server within that subnet, but it didn't work:

services:
  immich_server:
    <-->
    networks:
      network:
        ipv4_address: 172.24.0.8

<-->

networks:
  network:
    driver: bridge
    ipam:
      config:
        - subnet: 172.24.0.0/16
          gateway: 172.24.0.1

(actually every service defaulted to connecting to IP of my bridged router, some pattern here).

Should I specify static IP for every service in the same subnet then? Or is this done in some other way?

[SadRobot11]

I really appreciate the great tips from @jrasm91, thank you very much! Unfortunately I do not posses the required knowledge to solve the issue still. I will be very grateful if you could explain to me how do it.

[manuelkirsch]

you can try adding following environment configs to the proxy, worked for me:

immich-proxy:
container_name: immich_proxy
image: ghcr.io/immich-app/immich-proxy:release
environment:
- IMMICH_WEB_URL=http://immich-web:3000
- IMMICH_SERVER_URL=http://immich-server:3001
- IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003
ports:
- 2233:8080

[dsm1212]

I resolved the connection refused messages I think you are seeing, but I'm not sure they are causing the 502's. My issue was that I was using a host name resolved by my router in the nginx proxy config. The problem was that the immich proxy could be entered either with the host ip or the container ip. That can lead to a CSRF error and connection refusd. So I put the nginx server (swag) in the same docker network as the immich proxy and then in my proxy config used the container name for the immich proxy. That will always resolve to the container ip. All the connection refused messages disappeared.

[SadRobot11]

Thank you! I ended up resolving the issue by getting a new router so that I would not have to utilize a bridged router at all. (Luckily did not have to pay for it.) And now Immich also does not use a separate proxy container, so probably I would be fine with the old setup as well.

[nikos445]

I managed to fix timeouts and 502 Bad Gateway in immich upload (cli commands) with the following configuration
+ 5 replicas of ingress-nginx pods

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: immich-server
  namespace: immich
  generation: 1
  labels:
    k8slens-edit-resource-version: v1
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-body-size: '0'
    nginx.ingress.kubernetes.io/proxy-buffer-size: 256k
    nginx.ingress.kubernetes.io/proxy-buffers-number: '100'
    nginx.ingress.kubernetes.io/proxy-max-temp-file-size: 30024m
status:
  loadBalancer:
    ingress:
      - ip: x.x.x.1
      - ip: x.x.x.2
      - ip: x.x.x.3
      - ip: x.x.x.4
      - ip: x.x.x.5
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - example.org
      secretName: immich-cert
  rules:
    - host: example.org
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: immich-server
                port:
                  number: 3001

I know it's overkill but after uploading 20.000 photos and videos I revert it back to normal sizes.
I had videos exceeding 10 GB (8k quality)... over default 1024m limit of buffer size

[HVR88]

This still happens regularly when hosting on Unraid.

In addition, the docker log for the container repeats an error that immich is unable to read/find a .immich file under the encoded_video path. The file is there, set with the same permissions as every other file/folder in the immich hierarchy.

Restarting the full docker stack fixes the issue. No 502 and no errors in log.