Track cross-process JIT compilation stampede on cold cache misses

[goog00]

Context

Follow-up to #193.

The in-process JIT cache uses OnceCell to compile each kernel only once per process. The persistent disk cache can be shared across processes, but compilation is not deduplicated across process boundaries.

When multiple processes sharing the same cache directory cold-start concurrently — commonly with one process per GPU — they can all miss the same key, spawn tileiras, and compile the same kernel.

This is correct: atomic temporary-file + rename writes ensure that the cache converges on one identical entry. However, the first parallel launch wastes N−1 compiles. Subsequent starts hit the populated cache.

Decision for #193

Merge without a per-key cross-process compilation lock and track the limitation here.

A blocking flock was left out because:

• A slow or hung lock holder could block every waiter.
• Waiters cannot reliably distinguish active compilation from a stuck compiler.
• File-lock semantics and reliability can vary across NFS deployments, where shared caches are likely to live.
• The stampede affects only a cold cache and converges correctly afterward.

Users can avoid the initial stampede by warming the shared cache with one process before launching parallel workers. This workaround is documented in cutile-book/guide/jit-compilation.md.

Follow-up

Revisit cross-process compilation coordination if cold-start stampedes become a significant real-world cost.

Any future solution should:

• Where cross-process coordination is supported, allow only one process to compile a missing cache entry while the others reuse its result.
• Document whether this coordination is supported on network filesystems such as NFS.
• Where coordination is unavailable, fall back to redundant compilation rather than fail a kernel launch or block indefinitely.

[elibol]

Two ideas worth tracking here, based on how other systems handle persistent JIT caches:

1. A minimum compile time threshold for persisting (JAX defaults to 1 second). Only persist compiles that were expensive enough to be worth it. The frontend residual is about 40 to 60 ms and tileiras dominates a cold compile, so this keeps the cache focused on the expensive kernels. It also shrinks the cost of the cold start stampede, which is what this issue tracks.

2. Per shard eviction (Mesa moved to this in 23.1). Give each of the 256 shards a budget of capacity/256 and evict within the shard that was just written. Deterministic, no probabilistic trigger, no full tree scan. A simplification path if the current amortization ever becomes hard to maintain.