`RuntimeError: Attempting to use a closed pool` on DeepSeek-v3-16b torchtitan on Blackwell + torch nightly + CUDA 13.4

[syed-ahmed]

Bug description

🐛 Describe the bug

After bumping torch nightly 2.14.0a0+gite9d5fe4 → 2.14.0a0+gitf877a09), DeepSeek-v3-16b pretraining run crashes during the first torch.compile invocation (the flex-attention document block mask, _compiled_create_block_mask). The inductor async-compile SubprocPool worker exits and every subsequent kernel-compile submit then raises RuntimeError: Attempting to use a closed pool.

Reproducer:

torchrun --nnodes 2 --nproc-per-node 4 -m torchtitan.train \
  --module deepseek_v3 --config deepseek_v3_16b \
  --compile.enable --compile.components model,loss \
  --training.steps 50 --training.local_batch_size 4 --training.seq_len 4096 \
  --parallelism.expert_parallel_degree 8 \
  --dataloader.dataset c4_test

• Hardware: 2 nodes × 4 GPUs (Blackwell GB200 NVL, arm64). Also reproduces on a single node × 8 GPUs; not platform-specific.

Ablation (which settings cause / avoid the bug):

• --compile.enable --compile.components model,loss → crash (this reproducer).
• --compile.no-enable → still crashes, because torchtitan compiles the flex-attention block mask unconditionally on step 1.
• TORCHINDUCTOR_COMPILE_THREADS=1 (force in-process compilation, no SubprocPool subprocess worker) → passes end-to-end (all 50 steps, exit 0) at baseline throughput. This is the key isolating signal: the Triton kernels themselves compile and run correctly; the defect is specifically in the async SubprocPool compile-worker path (worker bootstrap/fork/IPC) under the new toolchain, not in the model graph or codegen.

Error logs

[titan] Compiling the loss function with torch.compile
[titan] Training starts at step 1
 torch/_inductor/compile_worker/subproc_pool.py:338] SubprocPool unclean exit
  File ".../torchtitan/models/attention.py", ... return _compiled_create_block_mask(*args, **kwargs)
  File "torch/_dynamo/eval_frame.py", line 1205, in compile_wrapper
  File "torch/_inductor/compile_fx.py", line 1132, in _compile_fx_inner
    raise InductorError(e, currentframe()).with_traceback(...)
  File "torch/_inductor/graph.py", line 2991, in _compile_to_module
    async_compile.triton(subs_name, src_code)
  File "torch/_inductor/async_compile.py", line 495, in triton
  File "torch/_inductor/compile_worker/subproc_pool.py", line 309, in submit
    self._send(MsgHeader.JOB, job_id, job_data)
  File "torch/_inductor/compile_worker/subproc_pool.py", line 315, in _send
    raise RuntimeError("Attempting to use a closed pool")
torch._inductor.exc.InductorError: RuntimeError: Attempting to use a closed pool
-> torch.distributed.elastic.multiprocessing.errors.ChildFailedError (exitcode 1, all ranks)

This issue was drafted with assistance from the claude-opus-4-8 AI model.

Versions

• torch: 2.14.0a0+gitf877a09 (nightly) — previously-good: 2.14.0a0+gite9d5fe4
• CUDA: 13.4.0.019 — previously-good: 13.3.1.008
• triton: 3.8.0+git43422b04
• torchtitan: 0.2.2
• Python: 3.12; OS: Ubuntu 24.04 (arm64); driver 580.173.02

[tianyu-l]

cc @drisspg