[fix] Full disk when db down

[zamentur]

Problem

When db is down (for example due to oem_killer), synapse write a tons of stacktrace about not being able to join database, this fill quickly the disk.

Solution

Reduce logs level

PR Status

Tested on a prod, but need to be validated...

• Code finished and ready to be reviewed/tested
• The fix/enhancement were manually tested (if applicable)

Automatic tests

Automatic tests can be triggered on https://ci-apps-dev.yunohost.org/ after creating the PR, by commenting "!testme", "!gogogadgetoci" or "By the power of systemd, I invoke The Great App CI to test this Pull Request!". (N.B. : for this to work you need to be a member of the Yunohost-Apps organization)

[zamentur]

!testme

[yunohost-bot]

Test it, book it, worm it, technologic. 🤖 🎶

[Josue-T]

Discussed today at PRLanta, and the issue currently with this PR is that we need to ensure that fail2ban still work after this change.

The different possibility to go forward on this would be:

• Do some testing with fail2ban and see how we can adapt the log level to have the log less verbose and keep fail2ban working. Maybe we can have a log level to INFO for the authentication component while having the other component a log level to ERROR.
• Change the approach of this issue and ensure that synapse is not working when the database is crashed. We can do this by declaring the dependency between synapse and postgresql with the systemd unit.

[Gredin67]

We could add WantedBy=postgresql

Avec BindsTo= ou Requires=, ça ne le fait pas automatiquement (je crois). Par contre [Install] UpheldBy= peut être intéressant
Sinon [Install] WantedBy= est moins forçant ([Install] UpheldBy= forcerait synapse à être lancé tant que postgresql tourne, alors que [Install] WantedBy= ne fait qu’essayer de démarrer synapse à chaque démarrage de postgresql)

[Josue-T]

I think what we need here is to ensure that synapse don't start or is running if postgresql is not running. But in my opinion having a relations in the sense that synapse start when postgresql start is not really what we would like. For instance when we are upgrading synapse and we stop synapse, we don't what that synapse is started just because postgresql is restarted.

But I agree that finding the good setting here might be not so easy.

[Gredin67]

OK so I think

After=network.target
Requires=postgresql.service

would solve our issue on sans-nuage.fr : shut down synapse when postgresql fails
@zamentur