journald versions incompatibilities

[jhoareau]

Hi Segment team @achille-roussel @jnjackins ,

We've been encountering an issue with the logger not posting to CloudWatch after having upgraded our ECS host instances to Ubuntu 18.04. After some investigation, we realised that journald inside the Docker image could not read the journal from our host instance:

root@f0298aa93076:/# journalctl | tail
Journal file /run/log/journal/0caac9ee81044a2e884eba8299ce5bba/user-1000.journal uses an unsupported feature, ignoring file.

Also, we've had a slight issue with the Readme that says that we should be using /run/log/journald as a mounting point but, in Ubuntu 18.04, the default logging folder is /var/log/journald, so we had to change the source mount point for the bind mount.

A solution could be to update systemd running inside the instance from systemd sources: systemd/systemd#3875 - that operation could be done in your Dockerfile.

Cheers!

[achille-roussel]

Plenty has changed in systemd and we’ve been dealing with tons of problems on our own attempt to upgrade to 18.04, so I would say that I’m not surprised.

Would it work if you mounted the host /var/log/journald as /run/log/journald in the container?

If you want to contribute to fixing the README with updated instructions for 18.04 I’ll be happy to merge it. Thanks for reporting anyway!

[jhoareau]

We have mounted this as that, and we encounter the error I've posted in the issue, the journald version incompatibility :)

[kapralVV]

@jhoareau @achille-roussel

We've been encountering an issue with the logger not posting to CloudWatch after having upgraded our ECS host instances to Ubuntu 18.04.

In Dockerfile for ecs logs you may find the following comment

We need a go compiler that's based on an image with libsystemd-dev installed

The code is complied using Ubuntu 16.04 (and its version of libsystemd-dev)

So to make it works on Ubuntu 18.04 it's required to re-compile ecs-logs using it
I did it and it's OK now. I can share Dockerfile I used for re-compilation .