Unable to run job handler (faceDetection/face-detection) with remote IML

[ilisei]

The bug

Good afternoon, the server has been running since version 98. After on version 117, I decided to completely reinstall it from scratch.
After deploying in the same configuration as it was with the complete removal of the previous one, everything worked. But I saw in the logs that periodically there is no connection to the remote server. But I recognized the faces.
Today it no longer recognizes, but in the logs it gives out such a picture.
Link between the primary and remote server via VPN.

The OS that Immich Server is running on

Ubuntu 22.04.5 LTS

Version of Immich Server

v1.117.0

Version of Immich Mobile App

none

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

LOCAL SERVER

# WARNING: Make sure to use the docker-compose.yml of the current release:                           
#                                                                                                    
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml                   
#                                                                                                    
# The compose file on main may not be compatible with the latest release.                            
#                                                                                                    
                                                                                                     
name: immich                                                                                         
                                                                                                     
services:                                                                                            
  immich-server:                                                                                     
    container_name: immich_server                                                                    
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}                               
    # extends:                                                                                       
    #   file: hwaccel.transcoding.yml                                                                
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated tran
scoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, e
dit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /mnt/external_library:/mnt/external_library
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: true

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2d1463258f2764328496376f5d965f20c6a67f66ea2b06dc42af351f
75248792
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23
190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system
, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="
$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command
='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count i
s $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s 
      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", p
ublic, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB"
, "-c", "wal_compression=on"]
    restart: always


REMOTE MACHINE LEARNING

version: "3.8"

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#


services:
  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    restart: always
    ports:
      - 3003:3003
    healthcheck:
      disable: false
volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/e
nvironment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/immich_upload
#UPLOAD_LOCATION=./library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

IMMICH_LOG_LEVEL=log

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: ht
tps://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=Etc/UTC
TZ="Asia/Almaty"
# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=****

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=****
DB_DATABASE_NAME=immich

Reproduction steps

1. Reboot Server, wait 5 min
2. Start Facial Recognition

Relevant log output

LOCAL SERVER
[Nest] 8  - 10/09/2024, 3:05:54 PM   ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.5}},"recognition":{"modelName":"buffalo_l"}}}' failed with status 500: Internal Server Error
[Nest] 8  - 10/09/2024, 3:05:54 PM   ERROR [Microservices:JobService] Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.5}},"recognition":{"modelName":"buffalo_l"}}}' failed with status 500: Internal Server Error
    at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:22:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26)
    at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:226:52)
    at async /usr/src/app/dist/services/job.service.js:163:36
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 8  - 10/09/2024, 3:05:54 PM   ERROR [Microservices:JobService] Object:
{
  "id": "3a39a9ad-f7a6-49a0-88b2-0105a8cb9712"

REMOTE 

[10/09/24 10:05:54] ERROR    Exception in ASGI application                                           
                                                                                                     
                             ╭─────── Traceback (most recent call last) ───────╮                     
                             │ /usr/src/app/main.py:147 in predict             │                     
                             │                                                 │                     
                             │   144 │   text: str | None = Form(default=None) │                     
                             │   145 ) -> Any:                                 │                     
                             │   146 │   if image is not None:                 │                     
                             │ ❱ 147 │   │   inputs: Image | str = await run(l │                     
                             │   148 │   elif text is not None:                │                     
                             │   149 │   │   inputs = text                     │                     
                             │   150 │   else:                                 │                     
                             │                                                 │                     
                             │ /usr/src/app/main.py:188 in run                 │                     
                             │                                                 │                     
                             │   185 │   if thread_pool is None:               │                     
                             │   186 │   │   return func(*args, **kwargs)      │                     
                             │   187 │   partial_func = partial(func, *args, * │                     
                             │ ❱ 188 │   return await asyncio.get_running_loop │                     
                             │   189                                           │                     
                             │   190                                           │                     
                             │   191 async def load(model: InferenceModel) ->  │                     
                             │                                                 │ 
                             │ /usr/local/lib/python3.11/concurrent/futures/th │
                             │ read.py:58 in run                               │
                             │                                                 │
                             │ /usr/src/app/main.py:147 in <lambda>            │
                             │                                                 │
                             │   144 │   text: str | None = Form(default=None) │
                             │   145 ) -> Any:                                 │
                             │   146 │   if image is not None:                 │
                             │ ❱ 147 │   │   inputs: Image | str = await run(l │
                             │   148 │   elif text is not None:                │
                             │   149 │   │   inputs = text                     │
                             │   150 │   else:                                 │
                             │                                                 │
                             │ /usr/src/app/models/transforms.py:52 in         │
                             │ decode_pil                                      │
                             │                                                 │
                             │   49 def decode_pil(image_bytes: bytes | IO[byt │
                             │   50 │   if isinstance(image_bytes, Image.Image │
                             │   51 │   │   return image_bytes                 │
                             │ ❱ 52 │   image: Image.Image = Image.open(BytesI │
                             │      bytes) else image_bytes)                   │
                             │   53 │   image.load()                           │
                             │   54 │   if not image.mode == "RGB":            │
                             │   55 │   │   image = image.convert("RGB")       │
                             │                                                 │
                             │ /opt/venv/lib/python3.11/site-packages/PIL/Imag │
                             │ e.py:3498 in open                               │
                             │                                                 │
                             │   3495 │   for message in warning_messages:     │
                             │   3496 │   │   warnings.warn(message)           │
                             │   3497 │   msg = "cannot identify image file %r │
                             │ ❱ 3498 │   raise UnidentifiedImageError(msg)    │
                             │   3499                                          │
                             │   3500                                          │
                             │   3501 #                                        │
                             ╰─────────────────────────────────────────────────╯
                             UnidentifiedImageError: cannot identify image file 
                             <_io.BytesIO object at 0x7f4ddefc4d10>

Additional information

No response

[bo0tzz]

Is the remote ML instance on 1.117.0 as well?

[ilisei]

Is the remote ML instance on 1.117.0 as well?

Yes

[mertalev]

Thumbnail generation for these assets probably failed. You can try refreshing thumbnails for the affected assets through the multi-select options. Once the jobs are finished and you confirm no errors in the logs, you can click "missing" for face detection and smart search in the job panel.

[ilisei]

Thanks, I'll try the other day

[ilisei]

Good afternoon, while I found that these were CR2 files. But after a couple of days everything was decided by itself.

[LelouchKirito01]

Hello ilisei! I'm having very similar issues with Immich with exact same error:

[Nest] 7 - 01/31/2025, 9:33:00 PM ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): PostgresError: pgvecto.rs: Bad literal.
INFORMATION: hint = Bad character with ascii 0x31.
[Nest] 7 - 01/31/2025, 9:33:00 PM ERROR [Microservices:JobService] PostgresError: pgvecto.rs: Bad literal.
INFORMATION: hint = Bad character with ascii 0x31.
at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:788:26)
at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:474:6)
at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
at Socket.emit (node:events:524:28)
at addChunk (node:internal/streams/readable:561:12)
at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
at Readable.push (node:internal/streams/readable:392:5)
at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
[Nest] 7 - 01/31/2025, 9:33:00 PM ERROR [Microservices:JobService] Object:
{
"id": "bab1fdf0-a7aa-4aaa-b11f-9ea73bfafbdf",
"source": "upload"
}

But in my case, I don't have CR2 files, it happens with every photo, it doesn't matter if it is uploaded by the backup mobile function or manually from the web app. I have same version (v1.123) on the "immich_server" contianer and the "immich_machine_learning" container on my PC (which manage machine learning remotely). I'm starting to run out of ideas. Was your problem solved by doing nothing after a couple of days? Or do you at least know what the problem was, even if you did nothing?
Thank you!

[mertalev]

You should update your remote machine learning container to the latest version.

[LelouchKirito01]

Yes, it is already updated to the lastest version.

[ilisei]

Hi, I honestly forgot that there was a problem) I actually the main problem was that there was not enough capacity on the main server where immich was. That's why I had errors after I reduced the number of queues to 1. The problem was solved. Now moved to another server, there is more capacity and such problems were no more.