netbox container unhealthy post v3.4.1 upgrade from v3.3.9 #11362

davama · 2023-01-02T22:03:36Z

davama
Jan 2, 2023

Was encountering similar issues as here:
#11355

but before the upgrade, i deleted those duplicate name device objects (via gui) and recreated them.

Then the upgrade/migration went without much issue.
Today I noticed that my background tasks was queued up and didnt understand why.

When i looked at the container i noticed that it said unhealthy but dont see any interesting errors when i start the containers..

[+] Running 2/0
[+] Running 4/4tbox-postgres     Created                                                                                                                                                                                              0.1s
 ⠿ Container netbox-postgres      Created                                                                                                                                                                                             0.1s
[+] Running 5/6tbox-redis-cache   Created                                                                                                                                                                                             0.1s
 ⠿ Container netbox-postgres      Started                                                                                                                                                                                            12.2s
 ⠿ Container netbox-redis-cache   Started                                                                                                                                                                                            12.2s
 ⠿ Container netbox-redis         Started                                                                                                                                                                                            12.2s
 ⠿ Container netbox               Waiting                                                                                                                                                                                           107.3s
 ⠿ Container netbox-worker        Created                                                                                                                                                                                             0.1s
 ⠿ Container netbox-housekeeping  Created                                                                                                                                                                                             0.1s
container for service "netbox" is unhealthy
netbox-redis         | 1:C 02 Jan 2023 21:46:26.849 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
netbox-redis         | 1:C 02 Jan 2023 21:46:26.849 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
netbox-redis         | 1:C 02 Jan 2023 21:46:26.850 # Configuration loaded
netbox-redis         | 1:M 02 Jan 2023 21:46:26.851 * monotonic clock: POSIX clock_gettime
netbox-redis         | 1:M 02 Jan 2023 21:46:26.852 * Running mode=standalone, port=6379.
netbox-redis         | 1:M 02 Jan 2023 21:46:26.852 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
netbox-redis         | 1:M 02 Jan 2023 21:46:26.852 # Server initialized
netbox-postgres      | 
netbox-redis         | 1:M 02 Jan 2023 21:46:26.852 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
netbox-redis         | 1:M 02 Jan 2023 21:46:26.853 * Reading RDB base file on AOF loading...
netbox-redis         | 1:M 02 Jan 2023 21:46:26.854 * Loading RDB produced by version 7.0.5
netbox-redis         | 1:M 02 Jan 2023 21:46:26.854 * RDB age 5890626 seconds
netbox-redis         | 1:M 02 Jan 2023 21:46:26.854 * RDB memory usage when created 0.82 Mb
netbox-redis         | 1:M 02 Jan 2023 21:46:26.854 * RDB is base AOF
netbox-redis         | 1:M 02 Jan 2023 21:46:26.854 * Done loading RDB, keys loaded: 0, keys expired: 0.
netbox-redis         | 1:M 02 Jan 2023 21:46:26.854 * DB loaded from base file appendonly.aof.1.base.rdb: 0.000 seconds
netbox-redis         | 1:M 02 Jan 2023 21:46:27.535 * DB loaded from incr file appendonly.aof.1.incr.aof: 0.681 seconds
netbox-redis         | 1:M 02 Jan 2023 21:46:27.535 * DB loaded from append only file: 0.682 seconds
netbox-redis         | 1:M 02 Jan 2023 21:46:27.535 * Opening AOF incr file appendonly.aof.1.incr.aof on server start
netbox-redis         | 1:M 02 Jan 2023 21:46:27.535 * Ready to accept connections
netbox-redis         | 1:M 02 Jan 2023 21:47:27.093 * 10000 changes in 60 seconds. Saving...
netbox-redis         | 1:M 02 Jan 2023 21:47:27.094 * Background saving started by pid 13
netbox-postgres      | PostgreSQL Database directory appears to contain a database; Skipping initialization
netbox-postgres      | 
netbox-postgres      | 2023-01-02 21:46:26.984 UTC [1] LOG:  starting PostgreSQL 14.5 on x86_64-pc-linux-musl, compiled by gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219, 64-bit
netbox-postgres      | 2023-01-02 21:46:26.984 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
netbox-postgres      | 2023-01-02 21:46:26.984 UTC [1] LOG:  listening on IPv6 address "::", port 5432
netbox-redis-cache   | 1:C 02 Jan 2023 21:46:26.869 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
netbox-postgres      | 2023-01-02 21:46:26.985 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
netbox-postgres      | 2023-01-02 21:46:27.002 UTC [21] LOG:  database system was shut down at 2023-01-02 21:46:09 UTC
netbox-postgres      | 2023-01-02 21:46:27.038 UTC [1] LOG:  database system is ready to accept connections
netbox-redis-cache   | 1:C 02 Jan 2023 21:46:26.869 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
netbox-redis         | 13:C 02 Jan 2023 21:47:27.131 * DB saved on disk
netbox-redis         | 13:C 02 Jan 2023 21:47:27.132 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
netbox-redis         | 1:M 02 Jan 2023 21:47:27.195 * Background saving terminated with success
netbox-redis-cache   | 1:C 02 Jan 2023 21:46:26.869 # Configuration loaded
netbox-redis-cache   | 1:M 02 Jan 2023 21:46:26.870 * monotonic clock: POSIX clock_gettime
netbox-redis-cache   | 1:M 02 Jan 2023 21:46:26.872 * Running mode=standalone, port=6379.
netbox-redis-cache   | 1:M 02 Jan 2023 21:46:26.872 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
netbox-redis-cache   | 1:M 02 Jan 2023 21:46:26.872 # Server initialized
netbox-redis-cache   | 1:M 02 Jan 2023 21:46:26.872 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
netbox-redis-cache   | 1:M 02 Jan 2023 21:46:26.873 * Loading RDB produced by version 7.0.5
netbox-redis-cache   | 1:M 02 Jan 2023 21:46:26.873 * RDB age 17 seconds
netbox-redis-cache   | 1:M 02 Jan 2023 21:46:26.873 * RDB memory usage when created 0.92 Mb
netbox-redis-cache   | 1:M 02 Jan 2023 21:46:26.873 * Done loading RDB, keys loaded: 3, keys expired: 0.
netbox-redis-cache   | 1:M 02 Jan 2023 21:46:26.873 * DB loaded from disk: 0.000 seconds
netbox-redis-cache   | 1:M 02 Jan 2023 21:46:26.873 * Ready to accept connections
netbox               | 🧬 loaded config '/etc/netbox/config/configuration.py'
netbox               | 🧬 loaded config '/etc/netbox/config/extra.py'
netbox               | 🧬 loaded config '/etc/netbox/config/logging.py'
netbox               | 🧬 loaded config '/etc/netbox/config/plugins.py'
netbox               | 💡 Superuser Username: administrator, E-Mail: [email protected]
netbox               | ✅ Initialisation is done.
netbox               | ⏳ Waiting for control socket to be created... (1/10)
netbox               | 2023/01/02 21:46:50 [warn] 7#7 Unit is running unprivileged, then it cannot use arbitrary user and group.
netbox               | 2023/01/02 21:46:50 [info] 7#7 unit 1.27.0 started
netbox               | 2023/01/02 21:46:50 [info] 22#22 discovery started
netbox               | 2023/01/02 21:46:50 [notice] 22#22 module: python 3.10.4 "/usr/lib/unit/modules/python3.10.unit.so"
netbox               | 2023/01/02 21:46:50 [info] 7#7 controller started
netbox               | 2023/01/02 21:46:50 [notice] 7#7 process 22 exited with code 0
netbox               | 2023/01/02 21:46:50 [info] 24#24 router started
netbox               | 2023/01/02 21:46:50 [info] 24#24 OpenSSL 3.0.2 15 Mar 2022, 30000020
netbox               | ⚙️ Applying configuration from /etc/unit/nginx-unit.json
netbox               | 2023/01/02 21:46:51 [info] 28#28 "netbox" prototype started
netbox               | 2023/01/02 21:46:51 [info] 29#29 "netbox" application started
netbox               | ✅ Unit configuration loaded successfully
netbox               | 2023/01/02 21:46:54 [notice] 7#7 process 20 exited with code 0
netbox               | 2023/01/02 21:49:37 [info] 150#150 "netbox" application started

These are the other container versions:

# docker exec -it netbox-redis redis-server -v
Redis server v=7.0.5 sha=00000000:0 malloc=jemalloc-5.2.1 bits=64 build=8fdc9f58f913def3
# docker exec -it netbox-redis-cache redis-server -v
Redis server v=7.0.5 sha=00000000:0 malloc=jemalloc-5.2.1 bits=64 build=8fdc9f58f913def3
# docker exec -it netbox-postgres psql -V
psql (PostgreSQL) 14.5
# docker exec -it netbox grep "^VERSION" netbox/settings.py
VERSION = '3.4.1'

I do find ineresting that my netbox container takes almost 2min (107.3s) to start... so not sure if that is the reason I get unhealthy state.
Dont even know if the unhealthy state has anything to do with the background tasks not being processed ...

What can i try/check?
Any input is much appreciated!!

thanks,
Dave

Answered by davama

Jan 3, 2023

figured it out

The healthcheck on the netbox service was not functioning, which caused the worker and housekeeper container to NOT run either.

Since the healthcheck was curl -f http://localhost:8080/apii/, I had to change my env variable ALLOWED_HOST from

ALLOWED_HOSTS='netbox.domain.net'
to
ALLOWED_HOSTS='netbox.domain.net localhost'

And since we have http_proxy environment variables set, i had to set no_proxy=localhost env variable.

Very odd that only NOW did I have to do this change.

Either-way, all is well in the world.

Thanks,
Dave

View full answer

davama · 2023-01-03T14:14:45Z

davama
Jan 3, 2023
Author

figured it out

The healthcheck on the netbox service was not functioning, which caused the worker and housekeeper container to NOT run either.

Since the healthcheck was curl -f http://localhost:8080/apii/, I had to change my env variable ALLOWED_HOST from

ALLOWED_HOSTS='netbox.domain.net'
to
ALLOWED_HOSTS='netbox.domain.net localhost'

And since we have http_proxy environment variables set, i had to set no_proxy=localhost env variable.

Very odd that only NOW did I have to do this change.

Either-way, all is well in the world.

Thanks,
Dave

0 replies

anders-m · 2023-01-30T12:54:45Z

anders-m
Jan 30, 2023

I have the same problem, which file did you change the ALLOWED_HOST in, I cant find it anywhere?

0 replies

davama · 2023-01-30T13:10:08Z

davama
Jan 30, 2023
Author

it’s an environment variable, so technically docker-compose.yml. If you are not using docker then change your configuration.py file

0 replies

SteffenBernd · 2023-02-03T10:30:53Z

SteffenBernd
Feb 3, 2023

I´m using Docker-compose and after the i did a new pulll to update i have the same error with container unhealthy.
where exactly did you configure the allowed hosts?
i cant find anything like this in the docker-compose.yml

1 reply

davama Feb 8, 2023
Author

I have all my env variables defined in an .env file.

Similar to what they have here: https://github.com/netbox-community/netbox-docker/tree/release/env

SteffenBernd · 2023-02-08T06:49:29Z

SteffenBernd
Feb 8, 2023

hmmm i´ve resolved the issue by reinstalling the netbox docker-compose and imported the DB....
and i´ve checked out the .env files in the env folder but there is no ALLOWED_HOSTS to be seen.

0 replies

razaahmed · 2023-02-25T20:59:49Z

razaahmed
Feb 25, 2023

I am running into the same issue here. still not clear. I am using docker-compose. please clarify how exactly to resolve this issue


 ⠿ Container netbox-docker-redis-1                 Started                                                                                                                                                             5.9s
 ⠿ Container netbox-docker-netbox-1                Waiting                                                                                                                                                            95.3s
 ⠿ Container netbox-docker-netbox-worker-1         Created                                                                                                                                                             0.2s
 ⠿ Container netbox-docker-netbox-housekeeping-1   Created                                                                                                                                                             0.1s
container for service "netbox" is unhealthy

1 reply

SteffenBernd Feb 27, 2023

as i said i reinstalled it and imported the DB.

enriquevm51 · 2023-04-07T02:44:34Z

enriquevm51
Apr 7, 2023

Hello, I had to comment out all health check lines about netbox in docker-compose.yml.

Previously I tried to set the curl validation with the external port without success.

I installed a fresh virtual machine with Ubuntu 22.04 LTS, Docker and Docker-Compose and the standard default installation (git clone, docker-compose.override.yml, pull and up); and the issue still appeared.

7 replies

kawalex May 31, 2023

Hello,
I had the same issue and find out it was because the no_proxy in netbox docker did not have localhost while it was on the server. to resolve this, I edited docker-compose.override.yml file and add:
version: '3.4'
services:
netbox:
environment:
- no_proxy=localhost,
Maybe it will help !

enriquevm51 May 31, 2023

that's it!
Thank you so much!

lbertini Sep 26, 2023

The answer of @kawalex worked for me, thanks!

version: '3.4'

services:
netbox:
ports:
- 8000:8080
environment:
- no_proxy=localhost

djodyssey1127 Oct 30, 2023

worked for me

environment:

no_proxy=localhost

djoreilly Apr 30, 2025

To fix dependency failed to start: container netbox-docker-netbox-1 is unhealthy changed docker-compose.override.yml:

version: '3.4'
services:
  netbox:
    ports:
      - 8000:8080
    environment:
      - no_proxy=localhost

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

netbox container unhealthy post v3.4.1 upgrade from v3.3.9 #11362

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 7 comments 9 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

netbox container unhealthy post v3.4.1 upgrade from v3.3.9 #11362

Replies: 7 comments · 9 replies

davama Jan 3, 2023 Author

davama Jan 30, 2023 Author

davama Feb 8, 2023 Author

Replies: 7 comments 9 replies

davama
Jan 3, 2023
Author

davama
Jan 30, 2023
Author

davama Feb 8, 2023
Author