feat: HMAC-signed ingest completion webhooks (#14)

[leo-aa88]

Closes #14.

Optional callback_url on POST /v1/ingestions so consumers get a signed terminal-state POST instead of only polling.

When it fires

After a batch worker job is marked done or failed. Payload status is succeeded, partial (done with error_count > 0), or failed. job_id is the ingestion job id when present. Tail jobs and sync POST /v1/ingestions/lines do not send callbacks.

Webhook failures are logged and do not change ingest status. Polling still works.

Signing

HMAC-SHA256 over the raw JSON body. Header:

X-Raglogs-Signature: sha256=<hex>

The signature is not inside the JSON. Per-key whsec_… is generated at raglogs keys create and stored on api_keys.webhook_secret (Alembic 0006). Delivery loads it via api_key_id in the worker payload. Fallback is WEBHOOK_SECRET when auth is off, OIDC, or the key has a null secret. The API bearer token is never used for HMAC.

Retries: jittered exponential backoff (WEBHOOK_MAX_RETRIES, WEBHOOK_TIMEOUT). 5xx and connect errors retry; other 4xx do not.

Tests

Unit tests cover HMAC verify/tamper, URL scheme checks, mocked httpx retry counts, fail-open after exhausted retries, and worker invocation on done/failed.

  

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[cursor]

Code review — PR #42 (issue #14)

must-fix

None

HMAC-SHA256 is over the raw JSON body with X-Raglogs-Signature: sha256=<hex> (not a signature field). Per-key whsec_… is used rather than the argon2 API key; WEBHOOK_SECRET is the fallback; logs use host/job_id/error only. 5xx/429/connect retry with wait_exponential_jitter; other 4xx do not; deliver_callback / maybe_deliver_ingest_callback catch and leave ingest status unchanged. Unit tests cover sign/verify/tamper, retry counts, fail-open, and worker done/failed invocation. Alembic 0006 is new; 0004/0005 are untouched. Settings are unprefixed (WEBHOOK_SECRET, WEBHOOK_MAX_RETRIES, WEBHOOK_TIMEOUT). file: and empty hosts 422. Tail/push-lines not firing is documented.

should-fix

• src/worker/runner.py:163 (process_one → maybe_deliver_ingest_callback), before src/db/session.py:39 session.commit(): delivery (including WEBHOOK_MAX_RETRIES extra attempts and per-attempt WEBHOOK_TIMEOUT) runs while the worker session is still open. claim_next_job only flushes running; nothing commits until run_worker exits with get_db(). Concrete failure: GET /v1/ingestions/jobs/{id} keeps returning pending for the whole retry budget (default 6×10s plus backoff), so the documented poll fallback cannot see the terminal state while delivery is in progress. A worker kill during that window rolls back the ingest rows and the job stays pending, so the next claim re-ingests. HTTP errors are still fail-open (they do not raise); the bug is commit ordering, not the retry policy. Deliver after the session commits (or commit the terminal status first, then POST).

nice-to-have

• src/core/ingestion/webhooks.py:151 (_post_callback): status < 400 is treated as success. httpx does not follow redirects by default, so a 301/302 (trailing slash, http→https) is logged as webhook_delivered even though the JSON never landed. Treat only 2xx as success.
• src/api/routes/ingestions.py:369 (create_ingestion): mode=tail still stores callback_url and returns 202. Operators can set a callback and never get a POST. Reject or ignore-with-warning on tail (push/tail not firing is otherwise correctly documented).

Verdict

must-fix count: 0
should-fix count: 1

[cursor]

Code review — PR #42 (issue #14), round 2

Previous should-fix status

• commit-before-deliver: fixed. process_one now calls db.commit() then maybe_deliver_ingest_callback after both the done try path and the failed except path (src/worker/runner.py). Tests assert call order ["commit", "deliver"] on done (test_commits_terminal_status_before_webhook_delivery) and failed (test_callback_invoked_on_failed). Pollers can see terminal status during webhook retries; a crash in that window cannot re-claim the job.

must-fix

None

should-fix

None

nice-to-have

None

Verdict

must-fix count: 0
should-fix count: 0