nebula-sync is killing pihole

[heffneil]

I would have pihole fail overnight every night for a week. I have now disabled nebula-sync and the problem didn't occur last night.

These are two new installs and not really being uses I am trying to get a robust setup.

Please see this write-up here:

pi-hole/docker-pi-hole#1684 (comment)

nebula-sync logs:

This is after I just restarted the container this morning:

https://pastebin.com/csY0ftsT

Docker compose:

nebula-sync:
image: ghcr.io/lovelaze/nebula-sync:latest
container_name: nebula-sync
restart: always
environment:
- PRIMARY=http://172.16.0.125|superheff
- REPLICAS=http://172.16.0.126|superheff
- FULL_SYNC=true
- CRON=*/5 * * * *
- TZ=America/New_York

[heffneil]

It died again since starting it back up about 3 hours ago:

2025-01-16 12:45:00.451 EST [50/T406] INFO: [✓] FTLCONF_webserver_api_password is used
2025-01-16 12:45:00.451 EST [50/T406] INFO: [✓] FTLCONF_dns_upstreams is used
2025-01-16 12:45:00.468 EST [50/T405] INFO: Restarting FTL: Teleporter (ZIP) import
2025-01-16 12:45:00.469 EST [50M] INFO: Asked to terminate by "/usr/bin/pihole-FTL no-daemon" (PID 50, user pihole UID 100)
2025-01-16 12:45:00.490 EST [50/T280] INFO: Terminating timer thread
2025-01-16 12:45:00.564 EST [50/T277] INFO: Terminating database thread
2025-01-16 12:45:00.693 EST [50/T278] INFO: Terminating GC thread
2025-01-16 12:45:00.721 EST [50M] INFO: Finished final database update
2025-01-16 12:45:00.721 EST [50M] INFO: Waiting for threads to join
2025-01-16 12:45:00.721 EST [50M] INFO: Thread dns-client (2) is idle, terminating it.
2025-01-16 12:45:00.721 EST [50M] INFO: Thread ntp-client (4) is idle, terminating it.
2025-01-16 12:45:00.721 EST [50M] INFO: All threads joined
2025-01-16 12:45:00.723 EST [50M] INFO: Stored 2 API sessions in the database

2025-01-16T11:55:00-05:00 INF Syncing Teleporters...
2025-01-16T11:55:00-05:00 INF Invalidating sessions...
2025-01-16T11:55:00-05:00 INF Sync complete
2025-01-16T12:00:00-05:00 INF Running full sync replicas=1
2025-01-16T12:00:00-05:00 INF Authenticating clients...
2025-01-16T12:00:00-05:00 INF Syncing Teleporters...
2025-01-16T12:00:00-05:00 ERR Sync failed error="sync teleporters: http://172.16.0.126/api/teleporter: Post "http://172.16.0.126/api/teleporter\": EOF"
2025-01-16T12:05:00-05:00 INF Running full sync replicas=1
2025-01-16T12:05:00-05:00 INF Authenticating clients...
2025-01-16T12:05:00-05:00 INF Syncing Teleporters...
2025-01-16T12:05:00-05:00 INF Invalidating sessions...
2025-01-16T12:05:00-05:00 INF Sync complete
2025-01-16T12:10:00-05:00 INF Running full sync replicas=1
2025-01-16T12:10:00-05:00 INF Authenticating clients...
2025-01-16T12:10:00-05:00 INF Syncing Teleporters...
2025-01-16T12:10:00-05:00 INF Invalidating sessions...
2025-01-16T12:10:00-05:00 INF Sync complete
2025-01-16T12:15:00-05:00 INF Running full sync replicas=1
2025-01-16T12:15:00-05:00 INF Authenticating clients...
2025-01-16T12:15:00-05:00 INF Syncing Teleporters...
2025-01-16T12:15:00-05:00 INF Invalidating sessions...
2025-01-16T12:15:00-05:00 INF Sync complete
2025-01-16T12:20:00-05:00 INF Running full sync replicas=1
2025-01-16T12:20:00-05:00 INF Authenticating clients...
2025-01-16T12:20:00-05:00 INF Syncing Teleporters...
2025-01-16T12:20:00-05:00 ERR Sync failed error="sync teleporters: http://172.16.0.126/api/teleporter: unexpected status code: 400"
2025-01-16T12:25:00-05:00 INF Running full sync replicas=1
2025-01-16T12:25:00-05:00 INF Authenticating clients...
2025-01-16T12:25:00-05:00 INF Syncing Teleporters...
2025-01-16T12:25:00-05:00 INF Invalidating sessions...
2025-01-16T12:25:00-05:00 INF Sync complete
2025-01-16T12:30:00-05:00 INF Running full sync replicas=1
2025-01-16T12:30:00-05:00 INF Authenticating clients...
2025-01-16T12:30:00-05:00 INF Syncing Teleporters...
2025-01-16T12:30:00-05:00 INF Invalidating sessions...
2025-01-16T12:30:00-05:00 INF Sync complete
2025-01-16T12:35:00-05:00 INF Running full sync replicas=1
2025-01-16T12:35:00-05:00 INF Authenticating clients...
2025-01-16T12:35:00-05:00 INF Syncing Teleporters...
2025-01-16T12:35:00-05:00 ERR Sync failed error="sync teleporters: http://172.16.0.126/api/teleporter: Post "http://172.16.0.126/api/teleporter": EOF"
2025-01-16T12:40:00-05:00 INF Running full sync replicas=1
2025-01-16T12:40:00-05:00 INF Authenticating clients...
2025-01-16T12:40:00-05:00 INF Syncing Teleporters...
2025-01-16T12:40:00-05:00 ERR Sync failed error="sync teleporters: http://172.16.0.126/api/teleporter: unexpected status code: 400"
2025-01-16T12:45:00-05:00 INF Running full sync replicas=1
2025-01-16T12:45:00-05:00 INF Authenticating clients...
2025-01-16T12:45:00-05:00 INF Syncing Teleporters...
2025-01-16T12:45:05-05:00 ERR Sync failed error="sync teleporters: http://172.16.0.126/api/teleporter: Post "http://172.16.0.126/api/teleporter\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
2025-01-16T12:50:00-05:00 INF Running full sync replicas=1
2025-01-16T12:50:00-05:00 INF Authenticating clients...
2025-01-16T12:50:05-05:00 ERR Sync failed error="authenticate: http://172.16.0.126/api/auth: Post "http://172.16.0.126/api/auth\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
2025-01-16T12:55:00-05:00 INF Running full sync replicas=1
2025-01-16T12:55:00-05:00 INF Authenticating clients...
2025-01-16T12:55:05-05:00 ERR Sync failed error="authenticate: http://172.16.0.126/api/auth: Post "http://172.16.0.126/api/auth\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
2025-01-16T13:00:00-05:00 INF Running full sync replicas=1
2025-01-16T13:00:00-05:00 INF Authenticating clients...
2025-01-16T13:00:05-05:00 ERR Sync failed error="authenticate: http://172.16.0.126/api/auth: Post "http://172.16.0.126/api/auth\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

[lovelaze]

Hi @heffneil

It's hard to tell what's wrong here.

In your case with a "full sync" it's really just making a GET /teleporter to your primary and then a POST /teleporter with the previous response. Most likely it's not related to nebula-sync at all, but it's also strange that your sync seems to fail from time to time.

I would verify that:

• the piholes are on the same versions
• the primary does not contain broken/invalid config

[heffneil]

Thanks for the reply! I believe the versions are the same I will confirm. I made sure to pull new containers before trying to sync. I know you're project isn't pihole but is there some config checker? With my configuration being so new I can't imagine I broke something already!

[heffneil]

I have a serious, and perhaps stupid, question:

does someone run nebula sync on all nodes? Or just the master and it replicates down to the replicas?

[lovelaze]

does someone run nebula sync on all nodes? Or just the master and it replicates down to the replicas?

The intended way of running it is to let 1 single pihole be primary and then sync its configuration to X replicas.

[heffneil]

Got it so me running it on the other nodes would be a problem and a mistake

[lovelaze]

It might not ruin things but it doesn't make sense. It would be the equivalent of exporting all pihole configurations at the same time (perhaps) and restoring them in a random order 😄