Missing tests - recovery tests

[steve-chavez]

Currently recovery tests are done manually, it'd be great to have them as automated tests.

These are the main scenarios:

(the connection recovery worker is referred as just "worker")

1. postgrest started with a pg connection, then pg becomes unavailable

• worker starts only after a request to postgrest, postgrest should respond with a {"details":"no connection to the server\n","message":"Database client error. Retrying the connection."}
  • if db-channel-enabled=true, worker starts immediately, not necessary to prove this in tests though.
• pg becomes available, postgrest succeeds reconnecting, reloads the schema cache and responds with 200
• if db-load-guc-config=true, it should also re-read the in-db config.
  • test with an ALTER ROLE postgrest_test_authenticator SET pgrst.db_schemas = 'public'; and try a GET /public_consumers which should give a 404 if the in-db config isn't re-read.

2. unavailable pg, postgrest started

• worker starts immediately, postgrest should respond with a 503 {"message":"Database connection lost. Retrying the connection."}
  • Bug: if db-channel-enabled=true, postgrest doesn't reply and curl gives Connection refused. This must be because of the mvarConnectionStatus MVar, it doesn't happen on 1 though.
• pg becomes available, postgrest succeeds reconnecting, reloads the schema cache and responds with 200
• if db-load-guc-config=true, it should also re-read the in-db config.
  • Same test as 1

3. SIGUSR1 - NOTIFY reload schema

• when these are done, no running requests using pg connections must be interrupted
• when postgrest has a pg connection, both SIGUSR1 and NOTIFY will reload the schema cache
  • if db-load-guc-config=true, it should also re-read the in-db config.
  • ensure SIGUSR1 starts the worker when db-channel-enabled=true(got it to lock before, and worker was not starting, so this must be ensured)
• when postgrest loses the connection, and db-channel-enabled=false(only SIGUSR1)
  • SIGUSR1 starts the worker, only one can run at a time. Ensured by refIsWorkerOn, this can be confirmed by doing several SIGUSR1 and just noting one Attempting to reconnect to the database in 1 seconds... message. If refIsWorkerOn is removed, there will be several Attempting to reconnect to the database in 1 seconds... mesagges.
    • Not sure how to test this, maybe count the number of threads?
  • pg becomes available, postgrest succeeds reconnecting, reloads the schema cache and responds with 200
• when postgrest loses the connection, and db-channel-enabled=true
  • ensure the listener recovers, e.g. doing a NOTIFY 'reload cache/load config' should work after recovery.

[steve-chavez]

For the admin port health check(#2092), we'd also need to test:

• Starting with a down connection, /health must reply with 503
• Starting with a healthy connection, /health must reply with 200.. then a disconnection happens.. /health must reply with 503.. connection up again.. /health must reply with 200 again.

With db-channel-enabled as True and False for both cases.

[wolfgangwalther]

Dropping an idea for my own future reference here:

To allow breaking / unbreaking the pgrst <-> pg connection, we can create an individual symlink to the pg socket for each test-case - and then rename that accordingly. If renamed to something else this will break the connection. This will allow us to keep the PG server up - should be a lot faster than starting and stopping all the time. And it will not prevent us from running the io tests in parallel down the road.

[steve-chavez]

At startup, when the schema cache load fails(could be because a statement_timeout=1 or REVOKEd privileges on pg_catalog) we try to reload it on an endless loop.

This is done here:

postgrest/src/PostgREST/Workers.hs

Lines 93 to 94 in e13d912

	SCOnRetry ->
	work

Is not simple to remove the autoreload of the schema cache because the connection can be lost in the connection worker (which can be because of DNS error, too many clients error from pg, etc).

If we remove the autoreload we might have a regression like #1685.

Somehow joining the schema cache loading process on the connection worker and retrying it with exponential backoff(instead of endless loop) could be a solution.

[steve-chavez]

Optional: A recovery test for EMFILE could be added as well: #2158 (comment)

[steve-chavez]

Metrics(#2129) is required to test #1094 and #2364 (pool protection).

Related PostgREST/postgrest-docs#557