add: use SO_REUSEPORT on platform supporting it

[mkleczek]

DISCLAIMER:
This commit was authored entirely by a human without the assistance of LLMs.

Fixes #4694

Stacked on top of #4702 as it is not enough to start a new instance, it is also necessary not to fail in-flight requests on the old instance.

[steve-chavez]

This is failing all tests, I'd suggest to put these type of PRs as draft.

[mkleczek]

This is failing all tests, I'd suggest to put these type of PRs as draft.

Hmm... worked before latest push. Will switch to draft and fix.

[mkleczek]

This is failing all tests, I'd suggest to put these type of PRs as draft.

Hmm... worked before latest push. Will switch to draft and fix.

@steve-chavez - it looks like the issue is that on my machine PostgREST startup is fast enough so that it loads the schema cache before it accepts any requests. Here in CI the new instance fails with 503 because it didn't yet load the schema cache.

The question: is there any particular reason why we return 503, instead of simply not start listening on the socket until schema cache is loaded? The way we have it right now means we can't really support zero-downtime upgrades because once we start the new instance but before it loads the schema cache, some clients will get 503.

[steve-chavez]

The question: is there any particular reason why we return 503, instead of simply not start listening on the socket until schema cache is loaded?

We were aiming to have requests wait instead of 503, this waiting does happen during schema cache reload but not on startup; we discussed this on #4129. Would it be better to not listen on the socket? How would clients behave in this case?

[develop7]

They would get a "connection refused" error, which means nobody there and is imo more confusing that any 5xx error. UX-wise I would prefer some waiting to a presumably hard fail any day.

[mkleczek]

We were aiming to have requests wait instead of 503, this waiting does happen during schema cache reload but not on startup; we discussed this on #4129. Would it be better to not listen on the socket? How would clients behave in this case?

They would get a "connection refused" error, which means nobody there and is imo more confusing that any 5xx error. UX-wise I would prefer some waiting to a presumably hard fail any day.

I am not convinced, see below.

This is a complex topic so let's dig into it a little more. The startup sequence right now is:

1. Postgrest is not running.
2. Clients get connection refused
3. Postgrest starts listening on a socket
4. Clients get 503
5. Postgrest loaded schema cache
6. Normal traffic

The alternatives are:
a - blocking during schema cache loading

1. Postgrest is not running.
2. Clients get connection refused
3. Postgrest starts listening on a socket
4. Clients are blocked and potentially timeout getting some network error
5. Postgrest loaded schema cache
6. Normal traffic

b - listening on a socket only after schema cache loaded

1. Postgrest is not running.
2. Clients get connection refused
3. Postgrest loaded schema cache and starts listening
4. Normal traffic

So from the point of view of the clients (they don't know when Postgrest was started), we have 3 alternatives:

1. connection refused -> 503 -> normal
2. connection refused -> blocked/time out -> normal
3. connection refused -> normal

I am not sure what value clients get from the first two options comparing to the third one. Diagnostics and readiness checks should be done using admin server anyway.

In case of SO_REUSEPORT the situation is even worse if we start listening early. We have the following situation: instance 1 is running, instance 2 is started. From the point of view of clients:

1. Today: normal traffic -> some clients get 503 (both instances are listening only one is ready) -> normal traffic
2. Blocking: normal traffic -> some clients blocked/time out (both instances are listening one is blocking) -> normal traffic
3. Not listening: just normal traffic (there are no disruptions at all because all requests are handled by instance 1 until instance 2 is ready and starts listening).

So the first two options cause disruptions whereas the third one is fully zero-downtime and transparent to the clients.

My take on it would be:

1. Start admin server as early as possible.
2. Load schema cache.
3. Start listening on main socket.

This would require splitting binding from listening on the main socket (ie. we need to bind without listening first so that we can pass the socket to the admin server).

@steve-chavez @develop7 thoughts?

[mkleczek]

Dependent on resolving (or having a workaround to) yesodweb/wai#853

[steve-chavez]

So the first two options cause disruptions whereas the third one is fully zero-downtime and transparent to the clients.
My take on it would be:

Agree, sounds much better.

[wolfgangwalther]

Conflicted in the changelog.

[steve-chavez]

Suggested change

	:author: `mkleczek <https://github.com/mkleczek>`_

We've been doing this for almost all how-tos:

• https://docs.postgrest.org/en/v14/how-tos/sql-user-management-using-postgres-users-and-passwords.html
• https://docs.postgrest.org/en/v14/how-tos/create-soap-endpoint.html
• ..

[steve-chavez]

Let's put a heading and anchor here so we can link it from other places

Suggested change

	On operating systems that support ``SO_REUSEPORT``, you can start multiple
	.. _reuseport:
	SO_REUSEPORT
	~~~~~~~~~~~~~
	On operating systems that support ``SO_REUSEPORT``, you can start multiple

[steve-chavez]

Why did we increase the default? Could this be done for particular tests instead?

[mkleczek]

It was somewhat flaky on my machine. Can roll it back if needed.

[steve-chavez]

Yes, let's try that

[steve-chavez]

I'd suggest putting all these paragraphs under the reuseport section, otherwise it's kinda hard to hunt them down.

[steve-chavez]

Added this paragraph on my suggestion: https://github.com/PostgREST/postgrest/pull/4703/changes#r3470538404

Can be deleted from here if you agree

[steve-chavez]

Ditto here, maybe like:

Suggested change

	When :ref:`server-reuseport` is enabled on an operating system that supports
	``SO_REUSEPORT``, you can start multiple PostgREST instances on the same
	:ref:`server-host` and ``server-port``. For example, two PostgREST processes
	can use the same configuration:
	.. code:: ini
	server-host = "127.0.0.1"
	server-port = 3000
	server-reuseport = true
	New connections are then distributed by the operating system between the
	running PostgREST processes. This can be used to start a replacement process
	before stopping the old one, or to run several PostgREST processes behind one
	port.
	If ``server-reuseport`` is disabled, starting another PostgREST process on
	the same host and port will fail with the usual address-in-use error.
	.. _server-reuseport:
	server-reuseport
	----------------
	=============== =================================
	**Type** Bool
	**Default** false
	**Reloadable** N
	**Environment** PGRST_SERVER_REUSEPORT
	**In-Database** `n/a`
	=============== =================================
	Enables ``SO_REUSEPORT`` on the TCP server socket. This allows multiple
	PostgREST processes to bind to the same :ref:`server-host` and
	:ref:`server-port` when the operating system supports it.
	Enabling this setting on an operating system that does not support
	``SO_REUSEPORT`` is a configuration error. PostgREST will fail to start
	instead of falling back to a normal TCP socket.
	This setting does not apply when :ref:`server-unix-socket` is used.
	.. _server-reuseport:
	server-reuseport
	----------------
	=============== =================================
	**Type** Bool
	**Default** false
	**Reloadable** N
	**Environment** PGRST_SERVER_REUSEPORT
	**In-Database** `n/a`
	=============== =================================
	Enables ``SO_REUSEPORT`` on the TCP server socket. This allows multiple
	PostgREST processes to bind to the same :ref:`server-host` and
	:ref:`server-port` when the operating system supports it.
	For example, two PostgREST processes can use the same configuration
	.. code:: ini
	server-host = "127.0.0.1"
	server-port = 3000
	server-reuseport = true
	New connections are then distributed by the operating system between the
	running PostgREST processes. This can be used to start a replacement process
	before stopping the old one, or to run several PostgREST processes behind one
	port.
	Use a different ``admin-server-port`` for each instance. Admin ports are not shared
	between instances:
	- Readiness checks always target one specific PostgREST
	- Give each instance a different :ref:`admin-server-port`, otherwise the new instance will fail to start.
	Enabling this setting on an operating system that does not support
	``SO_REUSEPORT`` is a configuration error. PostgREST will fail to start
	instead of falling back to a normal TCP socket.
	This setting does not apply when :ref:`server-unix-socket` is used.

[steve-chavez]

Included this in https://github.com/PostgREST/postgrest/pull/4703/changes#r3470538404. To have everything in one place.

[steve-chavez]

This doesn't look related to this feature?

[mkleczek]

This doesn't look related to this feature?

Not directly but I added it because it is important in this case: we have multiple PostgREST processes running at the same time and it is easy to target the wrong one with health checks.

[steve-chavez]

Right, make sense. But it feels a bit out of place here. I believe it should go inside the server-reuseport section in the config.

[wolfgangwalther]

So let me see whether I understand the problem this tries to hint at: I turn on server-reuseport. I set server-host at the default of !4. This automatically applies to admin-server-host as well, I think. I now accidentally set the admin-server-port to the same value for both instances. According to the note further up, I would expect this to fail, because the same port for the admin server is used.

But it's not, because it's using a different interface. So I run two admin servers on the same port, but on different interfaces. Now, things start to break.

Is this what you had in mind?

If yes, I feel like it fits right in here. But it should be more framed as an exception to the above rule ("admin servers on the same port will fail to start").

If not.. please elaborate.