`demo` machine root disk was full

[benoit74]

Its occupation has grown very quickly:

Culprit was probably /var/log/syslog which occupied 11.7GB despite weekly rotation and compression.

No clue what caused it to grow so fast.

I cleared docker left-overs, reconfigured logrotate to compress all syslog files (not keeping last one uncompressed) and ran logrotate.

[benoit74]

Disk consumption was not back to normal so I investigated and found a "better" culprit

> ls -lah /var/lib/docker/containers/32ae9cac67c24adee043bfa43b7e1c4b598b85c5d6e82f55b951fd2e4aae1d56/32ae9cac67c24adee043bfa43b7e1c4b598b85c5d6e82f55b951fd2e4aae1d56-json.log
-rw-r----- 1 root root 14G Jul  6 12:45 /var/lib/docker/containers/32ae9cac67c24adee043bfa43b7e1c4b598b85c5d6e82f55b951fd2e4aae1d56/32ae9cac67c24adee043bfa43b7e1c4b598b85c5d6e82f55b951fd2e4aae1d56-json.log

This is the multi-proxy Docker log file.

I restarted the multi-proxy systemd unit, it cleared the log file but it failed to restart.

I rebooted the machine and it did not helped.

I cleared caddy_data volume and multi-proxy was back to "normal".

I then had to restart the demo-watcher.service to resurrect premium-preppers and compsci demo.

Looks like caddy internal files and maybe others were corrupted due to disk exhaustion.

[benoit74]

Same problem again. Surprisingly existing demos continue to work well. We will need to investigate and mitigate the root cause.

[benoit74]

syslog is occupying again a lot of disk space.

Don't get why logrotate did not automatically rotated the file, so I've set verbose mode on logrotate service.

Will also need to discuss with @rgaudin why we have so many logs, not sure it is that appropriate or needed.

[benoit74]

logrotate is rotating files only once a week. So we should ensure to not generate enough logs to fill the disk in few hours when uncompressed.