Unconditional panic() on storage-init failure at startup causes crash-loop instead of graceful degrade/retry

[jagatsingh]

Describe the bug

main.initializeStorage() (main.go:105) calls panic(err) unconditionally when
store.Initialize() fails at startup, for any underlying error — including a transient
network/DNS failure to the configured storage.type: postgres backend. Combined with a
restart: unless-stopped/always-style supervisor (Docker, systemd, k8s), this turns a
temporary storage-connectivity blip into a tight crash-loop that never lets Gatus reach the
point where it would actually run and report on the endpoints it's configured to monitor.

This is the same crash signature as #1633 (panic: EOF at main.initializeStorage/
main.go:105, fixed for that specific case by bumping lib/pq in #1644) — but the underlying
issue is broader than any one driver bug: any transient failure to reach the storage backend at
startup is fatal, by design (confirmed in #1395: "Gatus is built with fail fast in mind"). This
report is about a different trigger that isn't fixed by a driver bump.

What do you see?

Setup: storage.type: postgres, storage.path pointing at a Postgres/TimescaleDB container by
Docker Compose service name. When that container is removed from the Docker network (not
merely stopped/restarted — a genuinely different condition than a normal "connection refused"),
Docker's embedded DNS resolver (127.0.0.11:53) returns a malformed/unusual response rather than
NXDOMAIN, which the Go resolver surfaces as server misbehaving:

panic: dial tcp: lookup timescaledb-v2 on 127.0.0.11:53: server misbehaving

goroutine 1 [running]:
main.initializeStorage(0x1c611bc80000)
	/app/main.go:105 +0x1766
main.main()
	/app/main.go:34 +0xc5

lib/pq propagates this raw dial error out of store.Initialize(), and main.go panics on it
unconditionally. Under restart: unless-stopped, this repeated 206 times over ~19 hours
(224 occurrences of the panic line in the container logs) until the storage container was
restored — Gatus never once reached the point of running its configured checks (including, in
our case, a TCP check on that same host, which would have alerted on exactly this outage if
Gatus had ever gotten past initializeStorage).

Full write-up (self-inflicted circular dependency on our end, but the panic-vs-graceful-degrade
question is the part we think is worth raising upstream):
https://github.com/jagatsingh/trading_infrastructure/blob/main/docs/timescaledb-v2-removal-incident.md

What do you expect to see?

Ideally, store.Initialize() failing at startup should not be an unconditional, un-retried
panic(). A few options that would all help:

• Retry storage-connect with backoff for some bounded window before giving up, rather than
  panicking on the first attempt.
• Log the fatal error clearly and os.Exit(1) instead of panic(), so it's visually obvious
  this is a startup-config problem rather than an internal Gatus bug (the current panic/stack
  trace reads like an application crash, not "your storage backend is unreachable").
• (Bigger ask, understand if out of scope) an option to fall back to an in-memory store with a
  loud warning if the configured persistent store can't be reached at startup, rather than
  refusing to start at all — since a monitoring tool arguably has more value running
  degraded than not running.

Any of the above would have meant Gatus kept running and alerting throughout the incident above
instead of crash-looping silently for ~19 hours.

Steps to reproduce

1. docker compose with two services: a Postgres/TimescaleDB container (db) and Gatus
   configured with storage.type: postgres, storage.path: postgres://...@db:5432/gatus....
2. Start the stack; confirm Gatus is running normally.
3. docker rm -f db (a full remove, not docker stop) while Gatus keeps running.
4. Wait for Gatus's own restart: unless-stopped policy (or manually restart the Gatus
   container).
5. Observe panic: dial tcp: lookup db on 127.0.0.11:53: server misbehaving at
   main.initializeStorage/main.go:105, repeating on every restart until db exists again.

Version

v5.36.0

Additional information

• Confirmed via source read at the v5.36.0 tag that this is not a bug in the per-check TCP
  code path (client.CanCreateNetworkConnection already returns (false, nil) gracefully on any
  dial error, DNS failures included) — the panic is specifically in the storage-initialization
  path at startup.
• Our fix on our end is to stop pointing Gatus's own storage at a container one of its own
  checks monitors (removes the circular dependency entirely) — filing this because the
  unconditional-panic-on-storage-init-failure behavior itself seems like it could bite anyone
  whose storage backend has any transient startup-time connectivity hiccup, not just our specific
  circular-dependency case.

[thejdubb02]

Confirming the root cause and raising the design question before anyone writes code.

The panic is unconditional: main.initializeStorage (main.go:105) calls panic(err) on any error from store.Initialize(), so a permanent misconfiguration and a transient failure to reach the backend are treated identically. Under a restart-always supervisor the transient case becomes a crash-loop that never reaches the monitoring loop, so Gatus cannot report on the very outage that took it down.

I know the stated position is fail-fast (#1395), and that is the right default for a bad config: if storage.type or the credentials are wrong, looping forever hides the mistake. The question is only whether a transient connectivity failure at startup should be distinguishable from a permanent one.

One option that keeps fail-fast intact: a bounded startup retry, a small fixed number of attempts with a short backoff, then panic as today. A permanent error (bad DSN, auth failure, unknown driver) still fails within seconds because every attempt fails the same way, while a transient DNS or network blip gets a few seconds to resolve before Gatus gives up. No indefinite degrade, no serving checks against a backend that was never reachable.

Before I put up a PR: would you accept a bounded-retry approach here, and if so do you have a preference on the cap and whether it should be configurable (say STORAGE_INITIALIZE_RETRIES with a sane default), or would you rather keep the unconditional panic and treat this as operator responsibility? Happy to implement whichever direction you prefer.