Fix endless 5xx responses leading to pages

[segiddins]

The root cause here is that rails was returning
responses to the reverse proxy where the total
size of all the headers was greater than 4k. This
would lead to an
upstream sent too big header while reading response header from upstream
error in nginx, resulting in a default 502 bad
gateway response.

It took us dozens of hours of searching to find
the root cause because our nginx log parsing was
throwing away all of the error logs, and I only
saw it when directly tailing the logs from the
container in k8s. I have since fixed the log pipeline so the errors will show up as errors, with the error message.

As to how the rails app was returning so many
header bytes:
The Redirector middleware generates a 301
whenever it sees a host that does not match a
predefined list. Notably, api.rubygems.org
points to prod, but is not in that list.
The 301 includes a Location header with the
correct host, while maintaining the rest of the
path and query string. This means that the Location
header could contain up to the nginx limit of almost
8k bytes. In combination with the other headers returned
by the rails app, this was sufficient to exceed the
default limit of 4k bytes.

Verified that this fixes the 502s on staging by manually applying.

Tested via curl http://localhost:8080/versions/$(printf 'HelloWorld%.0s' {1..395})/abcdef -H 'X-Forwarded-Proto: https' -H 'X-Edge-Proto: https' -H 'Host: indexs.rubygems.org' -H 'Accept: sam/test' -v

Signed-off-by: Samuel Giddins [email protected]

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.06%. Comparing base (cecf79b) to head (10cd26c).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #5392   +/-   ##
=======================================
  Coverage   97.06%   97.06%           
=======================================
  Files         451      451           
  Lines        9391     9391           
=======================================
  Hits         9115     9115           
  Misses        276      276           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[simi]

Are those valid requests (according to app logic) with such a huge headers?

[segiddins]

They are valid HTTP, yes, as is the response.

[martinemde]

Is there any chance we would instead disable buffering so that the response is sent to the client (fastly) synchronously without buffering the entire response and potentially writing files to disk?

https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering

[segiddins]

No, buffering (of the headers) is necessary.