OOM events silently lost when breadcrumb serialization exceeds 5 MB headroom

[spawnia]

Problem Statement

When PHP hits memory_limit and the Sentry breadcrumb buffer contains large entries (e.g. SQL queries with bindings), the OOM event is silently discarded. The SDK's 5 MiB headroom (sentry-php#1633) is insufficient when serializing the event requires more memory than is available after OOM recovery.

Measured: serializing 100 breadcrumbs × ~100 KB each requires ~15 MB of working memory. With only 5 MB headroom, the secondary OOM during json_encode is caught by ErrorHandler::invokeListeners() (catch (\Throwable) {}) and silently discarded.

This makes the failure intermittent — events are captured when breadcrumbs happen to be small at the time of OOM, but lost when they've accumulated large entries.

Root Cause

With max_breadcrumbs=100 (default) and SQL bindings enabled (common in Laravel), the breadcrumb buffer accumulates serializable data proportional to query complexity. When OOM occurs:

1. ErrorHandler::handleFatalError() frees $reservedMemory (16 KiB) and raises memory_limit by 5 MiB
2. FatalErrorListenerIntegration attempts to capture the exception
3. PayloadSerializer::serialize() → json_encode() needs to allocate a contiguous buffer for the output — this requires ~15 MB for 100 breadcrumbs with large metadata
4. The allocation fails (only ~5 MiB available), triggering a secondary OOM
5. The secondary OOM is caught by ErrorHandler::invokeListeners() at ErrorHandler.php line 496 and silently discarded
6. The original OOM event is lost with no indication

Reproduction

Tested with sentry/sentry 4.29.0, PHP 8.3. Run php reproduction.php — output is EVENT LOST.

reproduction.php

<?php

declare(strict_types=1);

require_once __DIR__ . '/vendor/autoload.php';

use Sentry\Breadcrumb;
use Sentry\Event;
use Sentry\Integration\FatalErrorListenerIntegration;
use Sentry\SentrySdk;
use Sentry\State\Scope;
use Sentry\Transport\Result;
use Sentry\Transport\ResultStatus;
use Sentry\Transport\TransportInterface;

final class RecordingTransport implements TransportInterface
{
    public function send(Event $event): Result
    {
        file_put_contents(__DIR__ . '/oom_result.txt', 'EVENT CAPTURED');

        return new Result(ResultStatus::success(), $event);
    }

    public function close(?int $timeout = null): Result
    {
        return new Result(ResultStatus::success());
    }
}

final class AppState
{
    /** @var array<string> */
    public static array $data = [];
}

@unlink(__DIR__ . '/oom_result.txt');

\Sentry\init([
    'dsn' => 'http://key@localhost/1',
    'max_breadcrumbs' => 100,
    'transport' => new RecordingTransport(),
    'integrations' => [new FatalErrorListenerIntegration()],
]);

// Fill 100 breadcrumbs with ~100KB each (simulates SQL queries with bindings).
// Serializing the full event requires ~15MB of working memory.
SentrySdk::getCurrentHub()->configureScope(static function (Scope $scope): void {
    for ($i = 0; $i < 100; $i++) {
        $bindings = [];
        for ($j = 0; $j < 200; $j++) {
            $bindings["param_{$j}"] = str_repeat('x', 512);
        }
        $scope->addBreadcrumb(new Breadcrumb(
            Breadcrumb::LEVEL_INFO,
            Breadcrumb::TYPE_DEFAULT,
            'db.sql.query',
            'SELECT * FROM t WHERE id IN (' . implode(', ', array_fill(0, 50, '?')) . ')',
            $bindings,
        ));
    }
});

// Fill persistent memory to near the limit (simulates DI container, models, caches).
// These survive the fatal error — they are NOT freed.
ini_set('memory_limit', '24M');
$limit = 24 * 1024 * 1024;
$sizes = [64, 128, 256, 512, 1024, 2048, 4096, 8192];
$counter = 0;
while (memory_get_usage(true) < $limit - 128 * 1024) {
    AppState::$data[] = str_repeat('P', $sizes[$counter % count($sizes)]);
    $counter++;
}

register_shutdown_function(static function (): void {
    if (file_exists(__DIR__ . '/oom_result.txt')) {
        echo "RESULT: EVENT CAPTURED\n";
    } else {
        echo "RESULT: EVENT LOST — OOM error silently swallowed\n";
    }
});

// Trigger OOM. Top-level allocation fails, freeing nothing.
// After fatal: ~5MB headroom. Needed for serialization: ~15MB. → secondary OOM, discarded.
$trigger = str_repeat('Z', 256 * 1024);

Suggested Fix

When handleFatalError() detects an OOM (by checking the error message), clear or heavily truncate the breadcrumb buffer on the current scope before attempting capture. This reduces the serialization payload to well under 5 MiB regardless of what accumulated during normal execution.

Alternatively, the serialization path could catch OOM specifically and retry with a stripped-down event (no breadcrumbs, minimal context) — a degraded report is better than no report.

Workaround

Set max_breadcrumbs to a low value (e.g. 20) in your Sentry config. This keeps useful debugging context while keeping the serialization payload within the headroom budget.

Environment

• sentry/sentry: 4.29.0 (also confirmed on 4.19.1)
• PHP: 8.3
• memory_limit: 512M (production; reproduction uses 24M for speed)
• max_breadcrumbs: 100 (default)
• sql_bindings: true (Laravel default via sentry-laravel)

[stayallive]

The PHP SDK also keeps a memory buffer (on top of Laravel) and we have tried to make this more reliable already: getsentry/sentry-php#1633 (also ported to 4.x).

I am wondering if both those measures have no effect because Laravel is handling the error and we therefore don't detect it as an OOM event and thus don't take the measure in the PHP SDK to free up our own buffer and perform an additional memory increase.

It's very hard to test and create a definitive test case for this, so many variables 😄

Happy to try and find a more stable solution though. I think the first thing to do is make sure the PHP SDK is aware it is an OOM event and can take it's own measures. If that is not enough we could see if we can limit or drop added event data to make it's capture more reliable.