Race condition between Seahub and MySQL

[Kreeblah]

I use the following (redacted) Docker compose file for my Seafile installation. It's very similar to the official one:

services:
  seafile-db:
    image: mariadb:10.11
    container_name: seafile-mysql
    environment:
      - MYSQL_ROOT_PASSWORD=SOME_PASSWORD  # Required, set the root's password of MySQL service.
      - MYSQL_LOG_CONSOLE=true
      - MARIADB_AUTO_UPGRADE=1
    volumes:
      - /opt/seafile-mysql:/var/lib/mysql  # Required, specifies the path to MySQL data persistent store.
    networks:
      - seafile-net

  memcached:
    image: memcached:1.6.18
    container_name: seafile-memcached
    entrypoint: memcached -m 256
    networks:
      - seafile-net
          
  seafile:
    image: seafileltd/seafile-mc:11.0-latest
    container_name: seafile
    ports:
      - "80:80"
    volumes:
      - /opt/seafile-data:/shared   # Required, specifies the path to Seafile data persistent store.
    environment:
      - DB_HOST=seafile-db
      - DB_ROOT_PASSWD=SOME_PASSWORD  # Required, the value should be root's password of MySQL service.
      - TIME_ZONE=Etc/UTC  # Optional, default is UTC. Should be uncomment and set to your local time zone.
      - [email protected] # Specifies Seafile admin user, default is '[email protected]'.
      - SEAFILE_ADMIN_PASSWORD=ADMIN_PASSWORD     # Specifies Seafile admin password, default is 'asecret'.
      - SEAFILE_SERVER_LETSENCRYPT=false   # Whether to use https or not.
      - SEAFILE_SERVER_HOSTNAME=seafile.mydomain.com # Specifies your host name if https is enabled.
    depends_on:
      - seafile-db
      - memcached
    networks:
      - seafile-net

networks:
  seafile-net:

I also have systemd set up to launch it:

[Unit]
Description=Seafile container
Requires=network-online.target docker.service
After=network-online.target docker.service

[Service]
Restart=always
ExecStart=/usr/bin/docker compose -f /root/seafile/docker-compose.yml up
ExecStop=/usr/bin/docker compose stop seafile; /usr/bin/docker compose stop memcached; /usr/bin/docker compose stop seafile-db

[Install]
WantedBy=default.target

However, about half the time when I start the service, Seahub fails to start because MySQL isn't quite ready by the time it checks. It doesn't seem to retry, either. The specific error I see is:

Starting seafile server, please wait ...
Seafile server started

Done.

Starting seahub at port 8000 ...
Error:Seahub failed to start.
Please try to run "./seahub.sh start" again
Traceback (most recent call last):
  File "/scripts/start.py", line 94, in <module>
waiting for mysql server to be ready: %s (2003, "Can't connect to MySQL server on 'seafile-db' ([Errno 111] Connection refused)")
    main()
  File "/scripts/start.py", line 80, in main
    call('{} start'.format(get_script('seahub.sh')))
  File "/scripts/utils.py", line 70, in call
    return subprocess.check_call(*a, **kw)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/opt/seafile/seafile-server-11.0.12/seahub.sh start' returned non-zero exit status 1.

In those cases, I either need to stop the service and start it again, or run seahub.sh within the container manually in order to successfully start Seahub.

[freeplant]

We are re-designing the docker image for version 12.0. We will keep the issue in mind during the process.

If you still have the problem at that time, just let me know.

[SkywalkerSpace]

To troubleshoot the problem, when seahub.sh failed to start, please send the seahub.log at that moment.

By the way, please make sure the docker is the latest version.

[Kreeblah]

To troubleshoot the problem, when seahub.sh failed to start, please send the seahub.log at that moment.

By the way, please make sure the docker is the latest version.

Unfortunately, there's not much going on there. I just checked to make sure Docker and my Docker images are up to date and restarted the service a few times until this happened, and this is all that's in seahub.log:

2024-11-15 16:04:21,592 [INFO] xmlschema:1274 include_schema Include schema from 'file:///usr/local/lib/python3.10/dist-packages/xmlschema/schemas/XSD_1.1/xsd11-extra.xsd'
2024-11-15 16:04:38,221 [INFO] xmlschema:1274 include_schema Include schema from 'file:///usr/local/lib/python3.10/dist-packages/xmlschema/schemas/XSD_1.1/xsd11-extra.xsd'
2024-11-15 16:04:38,328 [INFO] xmlschema:1274 include_schema Include schema from 'file:///usr/local/lib/python3.10/dist-packages/xmlschema/schemas/XSD_1.1/xsd11-extra.xsd'
2024-11-15 16:04:38,496 [INFO] xmlschema:1274 include_schema Include schema from 'file:///usr/local/lib/python3.10/dist-packages/xmlschema/schemas/XSD_1.1/xsd11-extra.xsd'

Meanwhile, much like the Docker log above, the Docker logs for this are showing that it can't connect to the DB:

Starting seafile server, please wait ...
Seafile server started

Done.

Starting seahub at port 8000 ...
Error:Seahub failed to start.
Please try to run "./seahub.sh start" again
Traceback (most recent call last):
  File "/scripts/start.py", line 94, in <module>
waiting for mysql server to be ready: %s (2003, "Can't connect to MySQL server on 'seafile-db' ([Errno 111] Connection refused)")
    main()
  File "/scripts/start.py", line 80, in main
    call('{} start'.format(get_script('seahub.sh')))
  File "/scripts/utils.py", line 70, in call
    return subprocess.check_call(*a, **kw)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/opt/seafile/seafile-server-11.0.13/seahub.sh start' returned non-zero exit status 1.

[Kreeblah]

I just updated to the 12.0.4 container, and this is still sometimes happening. This time, the Docker logs show this:

Starting seafile server, please wait ...
Seafile server started

Done.

Starting seahub at port 8000 ...
Error:Seahub failed to start.
Please try to run "./seahub.sh start" again
Traceback (most recent call last):
  File "/scripts/start.py", line 94, in <module>
waiting for mysql server to be ready: mysql is not ready
    main()
  File "/scripts/start.py", line 80, in main
    call('{} start'.format(get_script('seahub.sh')))
  File "/scripts/utils.py", line 70, in call
    return subprocess.check_call(*a, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/opt/seafile/seafile-server-12.0.4/seahub.sh start' returned non-zero exit status 1.

[pr4xx]

I got the same problem when upgrading from latest version to 12.0.4. My installation is broken since then. One of the visible errors is this line when starting the container:

/scripts/bootstrap.py:159: SyntaxWarning: invalid escape sequence '\ '

  call('''sed -i -e '/def validate_mysql_user_host(self, host)/a \ \ \ \ \ \ \ \ return host' {}'''

I looked into it for a bit and I am not sure why the bootstrap script tries to change something in another static container file?
However, even when removing the calls from the bootstrap script, the server does not come up.

[samuelsson]

I get the same error, but there is more information when running the container with the up command compared to the docker logs. The lines about connection aborted, if that is relevant or not. No logs in seahub.log.

This is a completely fresh install of 12.0.4 and it worked fine at first, but after a couple of reboots it now fails.

seafile            | Starting seafile server, please wait ...
seafile            | Seafile server started
seafile            |
seafile            | Done.
seafile            |
seafile            | Starting seahub at port 8000 ...
seafile-mysql      | 2024-12-01 12:07:39 7 [Warning] Aborted connection 7 to db: 'seahub_db' user: 'seafile' host: '172.18.0.4' (Got an error reading communication packets)
seafile-mysql      | 2024-12-01 12:07:39 10 [Warning] Aborted connection 10 to db: 'seahub_db' user: 'seafile' host: '172.18.0.4' (Got an error reading communication packets)
seafile            | Error:Seahub failed to start.
seafile            | Please try to run "./seahub.sh start" again
seafile            | Traceback (most recent call last):
seafile            |   File "/scripts/start.py", line 94, in <module>
seafile            |     main()
seafile            | waiting for mysql server to be ready: mysql is not ready
seafile            |   File "/scripts/start.py", line 80, in main
seafile            |     call('{} start'.format(get_script('seahub.sh')))
seafile            |   File "/scripts/utils.py", line 70, in call
seafile            |     return subprocess.check_call(*a, **kw)
seafile            |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
seafile            |   File "/usr/lib/python3.12/subprocess.py", line 413, in check_call
seafile            |     raise CalledProcessError(retcode, cmd)
seafile            | subprocess.CalledProcessError: Command '/opt/seafile/seafile-server-12.0.4/seahub.sh start' returned non-zero exit status 1.