Unbound container log file size

[rhcarvalho]

Hi! Thanks for Dokku and its plugins, including this one! This is essentially the same as dokku/dokku-mongo#114, this time with hopefully more context and possible solutions.

Feel free to open a new ticket if there is more information or if you are having a similar issue and think Dokku should handle it (and have an idea as to how).

Originally posted by @josegonzalez in #114

Like the OP, we had an issue with container logs growing over time to unreasonable sizes (also in the order of GB). AFAICT these are stdout logs, not "MongoDB internal files" as dokku/dokku-mongo#114 (comment) might have influenced that issue in the wrong direction.

We've had to manually intervene and truncate the log file during maintenance windows (using sudo truncate -s 0 /path/to/logfile) in the past, but that's not a proper long-term solution.

Details

$ docker inspect dokku.mongo.my_db | jq '.[0] | {logDriver:.HostConfig.LogConfig.Type, logOpts:.HostConfig.LogConfig.Config, logPath:.LogPath}'
{
  "logDriver": "json-file",
  "logOpts": {},
  "logPath": "/var/lib/docker/containers/da641389ce75c5782db93b4f77ebeeef0256262535ff350ea1bb98e1217c3eea/da641389ce75c5782db93b4f77ebeeef0256262535ff350ea1bb98e1217c3eea-json.log"
}
$ sudo du -sh /var/lib/docker/containers/da641389ce75c5782db93b4f77ebeeef0256262535ff350ea1bb98e1217c3eea/da641389ce75c5782db93b4f77ebeeef0256262535ff350ea1bb98e1217c3eea-json.log
804M    /var/lib/docker/containers/da641389ce75c5782db93b4f77ebeeef0256262535ff350ea1bb98e1217c3eea/da641389ce75c5782db93b4f77ebeeef0256262535ff350ea1bb98e1217c3eea-json.log

Perhaps the best way to explain the problem is to show what Dokku already supports for containers of Dokku apps: https://dokku.com/docs/deployment/logs/?h=log#docker-log-retention

Using dokku logs:set --global max-size 20m, one can limit all Dokku-app containers to retain logs for at most up to 20 MB. We use that feature in our Dokku apps. That configuration, however, doesn't apply to MongoDB containers created by this plugin.

I suspect this issue applies to all Dokku plugins that create containers (any container using the json-file log driver and producing logs will lead to a large log file overtime, unless some global Docker config on the host would prevent it). For instance, I found dokku/dokku-postgres#155 which dates back to 2018 and hints at exactly the same problem with the Postgres container (and was closed by the OP on the next day without explanation).

Possible solutions

• Respect the global config for log retention
• Add a plugin-specific log retention config
• Add a generic way to pass extra Docker flags/config when starting the database container
  • --log-driver flag
• Update the Dokku Log Management documentation to call attention to this problem and suggest updating the default Docker daemon config

Known workarounds

• Configure the default Docker log driver on the host to limit the log size and/or enable rotation
  • https://docs.docker.com/engine/logging/configure/#configure-the-default-logging-driver
  • https://docs.docker.com/engine/logging/drivers/json-file/#options
• Manually truncate the log file (e.g. using a cronjob), not recommended

[josegonzalez]

Moved it over to the dokku-datastore plugin repo as this affects all datastores.

There isn't a way to do this today.

[rhcarvalho]

Thanks @josegonzalez !

Another possible workaround (a variation of one I suggested earlier) is to configure the Docker daemon to use the journald driver, then configure journald log storage limits and rotation.

I acknowledge each team or individual might have different preferences depending on the host OS and their own experiences, so perhaps starting with a hint in the Dokku installation guide, commenting on the need for configuring the Docker daemon for production, might be a good first step?