Skip to content

Reverse proxy turns the gateway's 413 into a dropped connection #291

Description

@crtahlin

Found by running the provenance-smasher suite against the dev gateway.

Symptom

An oversized request body gets a transport error instead of an HTTP status:

POST https://provenance-gateway.dev.datafund.io/api/v1/stamps/   (5 MB JSON)
  -> httpx.ReadError, connection dropped

POST https://provenance-gateway.dev.datafund.io/api/v1/data/     (11 MB body)
  -> httpx.ReadError, connection dropped

The gateway itself behaves correctly

Speaking to the container directly, bypassing the proxy:

POST http://127.0.0.1:8000/api/v1/stamps/   (5 MB JSON)
  -> 413 {"detail":"Request body too large. Maximum size for JSON is 1048576 bytes."}

POST http://127.0.0.1:8000/api/v1/data/     (11 MB body)
  -> 402 payment required

So body_limit middleware works and returns a clear, documented status. That response never reaches an external caller — the proxy drops the connection instead.

The likely mechanism is the upstream responding and closing before the request body has been fully read, which the proxy surfaces as a reset rather than relaying the early response.

Why it matters

  • A client cannot distinguish "your body is too large" from a network failure, so the actionable error is lost.
  • It is untestable from outside: the smasher suite asserts 413/422 and gets an exception. Three of its security tests fail for this reason, and they are asserting correct expectations.
  • It only affects callers going through the public hostname — i.e. all real ones.

Suggested fix

Give the proxy its own body limit matching the gateway's, so it returns 413 itself rather than failing the connection. In deploy/Caddyfile:

request_body {
    max_size 10MB
}

Sized to MAX_UPLOAD_SIZE_MB, which is currently 10. Worth checking the JSON limit (1 MB) separately, since the two differ and a single proxy-level cap cannot express both — the proxy limit should be the larger of the two so the gateway keeps deciding the finer-grained cases.

Note

This is a regression from introducing the reverse proxy, not a pre-existing gateway defect. Before the migration the gateway was reached through a different proxy, so it is worth confirming whether the old path behaved the same before assuming this is new.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingp2Medium priority — planned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions