fix: stop extension goroutine/memory leak and post-runtimeDone hang (#48)#49
Open
thecraftman wants to merge 3 commits into
Open
fix: stop extension goroutine/memory leak and post-runtimeDone hang (#48)#49thecraftman wants to merge 3 commits into
thecraftman wants to merge 3 commits into
Conversation
…ut (#48) On high-invocation-volume functions the extension's sandbox memory grew across warm invocations until it hit the Lambda memory ceiling, after which the extension stopped completing its post-runtimeDone work and the invocation was killed at the function timeout with a full-window `extensionOverhead` span, even though the runtime finished in a few hundred ms. Root cause: the flush path used axiom-go IngestEvents, which streams events through an io.Pipe fed by a background zstd encoder goroutine. When a flush stalled or was cancelled mid-flight, that goroutine blocked forever in Encoder.Close -> PipeWriter.Write, leaking itself and its ~4MB compression buffer on every affected flush. The flush also ran synchronously in the NextEvent loop with context.Background(), so a stalled ingest held the sandbox open until the function timeout. Fix: - Encode a gzip NDJSON body in memory and send it via Ingest(io.Reader) instead of the streaming IngestEvents. There is no io.Pipe and no background encoder goroutine, so nothing can be stranded. Verified with a goroutine-leak guard: 0 stranded zstd/pipe goroutines after 50 cancelled flushes (previously ~2 per flush). net/http rewinds the *bytes.Reader for the shutdown retry path. - Bound each flush by the invocation deadline (flushContext, overridable via AXIOM_FLUSH_TIMEOUT) so a slow or stalled ingest can no longer hold the sandbox open until the function timeout. - Bound the retry-putback buffer (maxBufferedEvents, overridable via AXIOM_MAX_BUFFERED_EVENTS): on a sustained ingest outage the oldest events are dropped and the backing array is released, so memory can't grow without bound. - Add flusher unit tests and a goroutine-leak regression guard. Note: the underlying defect also exists in axiom-go's IngestEvents/getBody; this change routes around it entirely from the extension. A separate upstream fix is recommended so other axiom-go consumers don't hit the same leak. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
golangci-lint (latest, v2.12.2) fails on main with goconst issues in server/server.go and server/server_test.go that predate this branch — the last three CI runs on main are red for the same reason. Because the Test and Build jobs are gated on Lint (needs: lint), the failures also prevented this PR's tests from running at all. Extract the repeated event field/value literals (function, type, record, requestId) into constants and use them in both files. No behavior change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
100k buffered events at typical Lambda log-line sizes could occupy tens to hundreds of MB — the safety cap itself could exhaust a 128MB function. 10k (~10 flush batches of backlog) keeps the worst-case buffer small relative to the smallest memory configurations while still riding out transient ingest failures. Still overridable via AXIOM_MAX_BUFFERED_EVENTS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This should fix the Axiom-extension-caused memory ceiling and extensionOverhead=260s behavior we saw. If approved, I'd be quick to report back if this solves this timeout error we are seeing on Lambdas, even if you decide to put this on a new version number. @islameldigwi @thecraftman |
islameldigwi
approved these changes
Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On high-invocation-volume functions, sandbox memory grew steadily across warm invocations until it hit the Lambda memory ceiling. Once at the ceiling the extension stopped completing its post-
runtimeDonework:platform.reportshowed the invocation killed at the function timeout with a single full-windowextensionOverheadspan, even thoughplatform.runtimeDonereportedstatus=successin a few hundred ms. The leak was proportional to invocations, reproduced at any memory size, and reset only on fresh sandboxes — and the consuming process was the Go extension, not the runtime.Root cause
The flush path used axiom-go
IngestEvents, which streams events through anio.Pipefed by a background zstd encoder goroutine. When a flush stalled or was cancelled mid-flight, that goroutine blocked forever inEncoder.Close -> PipeWriter.Write, leaking itself plus its ~4 MB compression buffer on every affected flush. Compounding it, the flush ran synchronously in theNextEventloop withcontext.Background(), so a stalled ingest held the sandbox open until the function timeout (the full-windowextensionOverhead).Reproduced with a goroutine-leak harness driving 200 cancelled flushes: +400 goroutines (~2 per flush), each stranded in
zstd.(*Encoder).Close → io.(*pipe).write.Tests
flusherunit tests: success clears buffer, error requeues, requeue is bounded (drops oldest), context cancellation aborts the flush and requeues, empty buffer is a no-op.TestFlushDoesNotStrandStreamingEncoder: regression guard asserting nozstd/io.Pipegoroutines survive repeated cancelled flushes.gofmtclean ·go vetclean ·go test -race ./...clean · cross-arch builds (linux/amd64 + arm64) OK.