Skip to content

Commit 6d6f5e8

Browse files
FlyM1ssclaude
andauthored
fix(deploy): widen health-check window past the truth-layer cold start (#321)
The #318/#320 image starts healthy but the deploy's health check gave up before gunicorn bound :8000, taking prod down on a false negative. entrypoint.sh builds the XBRL truth-layer DuckDB store BEFORE gunicorn forks (~45s observed on the droplet), on top of collectstatic + Playwright verify. The old check waited `sleep 30` + 3x5s (~45s) then failed -- and the journal shows all 3 probes (08:26:16/:22/:27) ran BEFORE gunicorn even listened (08:26:30). The container was fine; the window just raced the store build. Widen to `sleep 10` + 24x5s (~130s, ~2.8x the measured cold start) and succeed as soon as /health/ answers. Also dump `journalctl -n 80` on failure so future startup regressions are diagnosable from the deploy log directly. Not touching gunicorn: the "Control server error: Permission denied" (gunicorn 25.x control socket defaulting to /app/gunicorn.ctl under the root-owned CWD) is non-fatal -- f86604d ran the identical gunicorn/config/user and served fine. Silencing that control socket is a separate, lower-priority follow-up. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent aad7e3d commit 6d6f5e8

1 file changed

Lines changed: 15 additions & 7 deletions

File tree

.github/workflows/backend-deploy.yml

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -220,20 +220,28 @@ jobs:
220220
exit 1
221221
}
222222
223-
echo "Waiting 30s for service to start..."
224-
sleep 30
225-
226-
echo "Checking health endpoint..."
227-
for i in 1 2 3; do
223+
# Cold start is dominated by entrypoint.sh building the XBRL truth-layer
224+
# DuckDB store BEFORE gunicorn forks (~45s observed), on top of collectstatic
225+
# + Playwright verification. `systemctl is-active` only proves the container
226+
# launched, not that gunicorn is listening -- so poll /health/ across a window
227+
# comfortably above the measured cold start (~130s) instead of a fixed ~45s
228+
# wait, which raced the store build and failed the deploy even though the
229+
# container was healthy. Succeeds as soon as the endpoint answers.
230+
echo "Waiting 10s before first health probe..."
231+
sleep 10
232+
233+
echo "Checking health endpoint (up to ~130s for cold start)..."
234+
for i in $(seq 1 24); do
228235
if curl -sf -m 10 http://localhost:8000/health/; then
229236
echo ""
230-
echo "Deployment verification successful!"
237+
echo "Deployment verification successful (attempt $i)!"
231238
exit 0
232239
fi
233-
echo "Attempt $i/3 failed, retrying in 5s..."
240+
echo "Attempt $i/24 failed, retrying in 5s..."
234241
sleep 5
235242
done
236243
237244
echo "ERROR: Health check failed"
238245
systemctl --user status "$SYSTEMD_UNIT" || true
246+
journalctl --user -u "$SYSTEMD_UNIT" --no-pager -n 80 || true
239247
exit 1

0 commit comments

Comments
 (0)