Skip to content

Bound the fatal-signal crash report with a watchdog so it cannot hang forever - #1108

Open
takano32 wants to merge 1 commit into
pharo-project:pharo-12from
takano32:fix/crash-handler-watchdog
Open

takano32 wants to merge 1 commit into
pharo-project:pharo-12from
takano32:fix/crash-handler-watchdog

Conversation

@takano32

Copy link
Copy Markdown
Contributor

Problem

When a fatal signal is raised from inside the allocator — most importantly glibc abort() on detected heap corruption, which raises SIGABRT while still holding the malloc arena mutex — the VM's crash reporter deadlocks and the process hangs forever.

doReport() (src/unix/debugUnix.c) is not async-signal-safe: it fopen()s the crash-dump file, which calls malloc(). From the signal handler, that malloc() blocks on the arena mutex the aborting thread already holds → deadlock. The result on this VM is the worst possible outcome for debugging: the main thread sits in futex_wait, crash.dmp is 0 bytes, stderr produces nothing, and in CI the job only dies on a pipeline timeout. Every heap-corruption bug in the VM is thereby rendered undiagnosable.

(Found while debugging a real heap overflow on aarch64: each occurrence wedged with an empty crash.dmp and had to be kill -9'd.)

Fix

Make the fatal-signal report self-terminating. doReport() now arms an alarm() watchdog with a dedicated, async-signal-safe SIGALRM handler (write(2) + _exit) before it touches the allocator, and disarms it once the report completes:

  • SIGALRM is otherwise routed to the crash handler (see installErrorHandlers), so the watchdog handler is installed for the duration and the previous disposition is restored afterwards — the SIGUSR1 diagnostic-dump path is unchanged.
  • A reentrancy guard turns a crash inside the reporter into an immediate abort() rather than unbounded recursion.

This intentionally does not rewrite reportStackState() to be fully async-signal-safe (a much larger change). It guarantees the process always terminates, and in the common case — a fatal signal that does not originate in the allocator — the full crash dump is still produced exactly as before.

Verification

  • Mechanism (deterministic): a standalone reproduction using the VM's exact handler flags (SA_NODEFER | SA_SIGINFO, empty sa_mask) with a fatal handler that blocks forever (pause() loop, standing in for the malloc deadlock) is terminated by the alarm watchdog after the timeout, exiting via the watchdog handler — instead of hanging.
  • No regression: the patched VM boots and evaluates (3+4 → 7); the change adds code only around the crash path and leaves normal execution untouched.

Scope note

The 30-second timeout is a safety bound, not a normal-path cost (a healthy report finishes in well under a second and cancels the alarm). Only a genuinely deadlocked/hung report hits it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Dszsb5ALkx1ytH8atZPvEV

The crash reporter is not async-signal-safe: doReport() fopen()s the crash
file, which allocates. When the fatal signal was raised from inside the
allocator itself -- e.g. glibc abort() on detected heap corruption, which
holds the arena mutex -- that allocation deadlocks on the mutex the aborting
thread already holds. The process then wedges forever in the signal handler
(main thread in futex_wait, an empty 0-byte crash.dmp, a stuck stderr), so a
heap bug produces no diagnostics at all and, in CI, only a pipeline timeout.

Make the handler self-terminating: doReport() arms an alarm() watchdog with a
dedicated async-signal-safe SIGALRM handler (write(2) + _exit) before touching
the allocator, and disarms it when the report completes. SIGALRM is otherwise
routed to the crash handler, so the watchdog handler is installed for the
duration and the previous disposition restored afterwards (keeping the SIGUSR1
diagnostic-dump path unchanged). A reentrancy guard turns a crash inside the
reporter into an immediate abort instead of unbounded recursion.

This does not make reportStackState() itself async-signal-safe (a larger
change); it guarantees the process always terminates and, in the common case
where the report path does not deadlock, the full dump is still produced.

Verified: a standalone reproduction using the VM's exact handler flags
(SA_NODEFER|SA_SIGINFO, empty sa_mask) with a handler that blocks forever is
terminated by the alarm watchdog after the timeout; the patched VM boots and
evaluates normally.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dszsb5ALkx1ytH8atZPvEV
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant