Skip to content

fix: truncate overly long container log lines from pod logs - #395

Merged
ilya-lesikov merged 1 commit into
werf:mainfrom
Sharvash:fix/381-truncate-long-log-lines
Jun 16, 2026
Merged

fix: truncate overly long container log lines from pod logs#395
ilya-lesikov merged 1 commit into
werf:mainfrom
Sharvash:fix/381-truncate-long-log-lines

Conversation

@Sharvash

Copy link
Copy Markdown
Contributor

kubedog now caps a single container log line at 64 KiB when it reads Pod logs from the Kubernetes API, truncating anything longer instead of buffering it without limit, so one oversized line can no longer freeze log tracking.

Previously followContainerLogs accumulated each log line into a buffer unbounded until it saw a newline, then handed the whole line to the logging pipeline. When a line was very long this broke in one of two ways:

  • the logging library logboek stalls while rendering the huge line; the consumer stops draining the buffered ContainerLogChunk channel (capacity 1000), the channel fills up, the blocking send stops the log-stream reader goroutine, and the whole tracking pipeline freezes until the context times out (the hang reported in Hangs on very long container log lines in nelm release install output #381 for nelm release install output)
  • a stream that never emits a newline grows the buffer without bound, with no upper limit on memory use

The read loop already pulls log data in 64 KiB chunks into a line buffer. This adds a containerLogLineLengthLimit of 64 KiB (the size of that read buffer) and enforces it per line: once a line reaches the limit kubedog stops appending bytes but keeps scanning until the newline, so log parsing stays in sync, and a flag carried across chunk reads records that the line was cut. When the line is flushed it gets a short marker before the timestamp/message split, so the marker lands in the message body:

 [truncated by kubedog: line exceeded 65536 bytes]

The line is truncated, not dropped - the visible part is still shown and only the overflow is discarded. Reading with a limit through the Kubernetes API itself does not help here: PodLogOptions.LimitBytes caps the total bytes of the whole stream, not the length of an individual line, so it cannot express "trim each line".

followContainerLogs is the single place every tracker reads container logs — the Job tracker has no log loop of its own and reuses the Pod tracker — so this also covers #158, where a long line in a job's pod output hangs job tracking. The underlying problem lives in the logging library (werf/logboek#73); this change is a guard on the kubedog side so a single misbehaving container cannot freeze tracking regardless of the renderer.

Verified with go build ./..., go vet and golangci-lint clean, and the read-loop logic checked against a standalone replica across boundary cases: a line exactly at the limit is not marked, one byte over it is, the truncation flag is carried across chunk boundaries, a stream with no newline keeps memory bounded at the limit, a multi-byte rune cut mid-character does not panic, and multiple lines in one chunk are split correctly.

Closes #381
Refs #158

kubedog now caps a single container log line at 64 KiB when it reads Pod logs from the Kubernetes API, truncating anything longer instead of buffering it without limit, so one oversized line can no longer freeze log tracking.

Previously `followContainerLogs` accumulated each log line into a buffer unbounded until it saw a newline, then handed the whole line to the logging pipeline. When a line was very long this broke in one of two ways:
- the logging library `logboek` stalls while rendering the huge line; the consumer stops draining the buffered `ContainerLogChunk` channel (capacity 1000), the channel fills up, the blocking send stops the log-stream reader goroutine, and the whole tracking pipeline freezes until the context times out (the hang reported in werf#381 for `nelm release install` output)
- a stream that never emits a newline grows the buffer without bound, with no upper limit on memory use

The read loop already pulls log data in 64 KiB chunks into a line buffer. This adds a `containerLogLineLengthLimit` of 64 KiB (the size of that read buffer) and enforces it per line: once a line reaches the limit kubedog stops appending bytes but keeps scanning until the newline, so log parsing stays in sync, and a flag carried across chunk reads records that the line was cut. When the line is flushed it gets a short marker before the timestamp/message split, so the marker lands in the message body:
```
 [truncated by kubedog: line exceeded 65536 bytes]
```

The line is truncated, not dropped — the visible part is still shown and only the overflow is discarded. Reading with a limit through the Kubernetes API itself does not help here: `PodLogOptions.LimitBytes` caps the total bytes of the whole stream, not the length of an individual line, so it cannot express "trim each line".

`followContainerLogs` is the single place every tracker reads container logs — the Job tracker has no log loop of its own and reuses the Pod tracker — so this also covers werf#158, where a long line in a job's pod output hangs job tracking. The underlying problem lives in the logging library (werf/logboek#73); this change is a guard on the kubedog side so a single misbehaving container cannot freeze tracking regardless of the renderer.

Verified with `go build ./...`, `go vet` and `golangci-lint` clean, and the read-loop logic checked against a standalone replica across boundary cases: a line exactly at the limit is not marked, one byte over it is, the truncation flag is carried across chunk boundaries, a stream with no newline keeps memory bounded at the limit, a multi-byte rune cut mid-character does not panic, and multiple lines in one chunk are split correctly.

Closes werf#381
Refs werf#158

Signed-off-by: Alexey Gorovenko <sharvashinho@gmail.com>
@ilya-lesikov

Copy link
Copy Markdown
Member

I tried to make all of this actually efficient one time and it was harder than I expected, so I didn't finish it.

If this small fix fixes the hangs of werf/nelm on really long log lines, then it's good, I'm going to merge it.

But can you first manually check if it works, let's say you have a pod and there is a 100 megabyte log line in it, does nelm/werf still hangs on it when trying to display it?

@Sharvash

Copy link
Copy Markdown
Contributor Author

I manually checked the long-line case with nelm.

Test setup: a pod emitted a single 100 MB log line, and I compared OLD kubedog without this fix against NEW kubedog with this fix.

Results:

  • OLD: nelm release install completed, but peak RSS memory grew to ~1.37 GiB and terminal output expanded to 419,465 lines because the full 100 MB log cell was repeatedly re-rendered.
  • NEW: install completed in 13s, peak RSS memory stayed at ~72 MiB, output was 303 lines, and the truncation marker appeared exactly once:
    [truncated by kubedog: line exceeded 65536 bytes]

So for nelm, the fix works: the pathological memory/output blowup is eliminated. It did not deadlock in my run, but the old behavior was clearly not efficient/safe for a 100 MB log line.

I did not run an empirical werf test in this pass because that would require replacing kubedog-for-werf-helm separately. However, the fix is in kubedog's shared pod log reader (pkg/tracker/pod/tracker.go), before logs are passed upward to nelm/werf consumers. Since werf goes through nelm ReleaseInstall, which uses this kubedog pod tracking path, werf should be protected by the same lower-level fix by construction.

Separate note from testing: standalone kubedog multitrack currently crashes before reaching log streaming when tracking a Job. This is unrelated to this PR: pkg/tracker/job/feed.go passes nil as informerFactory, and then the Job tracker dereferences it while starting the informer.

There is related historical context in #385, which was closed and later addressed on the werf side via werf/werf#7298. I did not find a separate open kubedog issue specifically for this standalone kubedog multitrack CLI path.

@ilya-lesikov
ilya-lesikov merged commit 2c00b08 into werf:main Jun 16, 2026
1 of 2 checks passed
@ilya-lesikov

Copy link
Copy Markdown
Member

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hangs on very long container log lines in nelm release install output

2 participants