Long-context KeyDiff KV-cache compaction crashes with EXC_BAD_ACCESS in cactus::kvcompress::rerope_recent_int8 (multi-turn session, ~900 decode tokens)

[Abusoss]

Summary

cactus_complete (streaming) reliably crashes the host process with EXC_BAD_ACCESS (code=2) inside cactus::kvcompress::rerope_recent_int8, called from cactus::engine::Model::compress_kv_cache_keydiff, on the second turn of a multi-turn conversation once decode has produced ~900 tokens on top of an ~11.4k-character prompt. No Python exception is raised — the process is killed at the OS level (zsh: illegal hardware instruction when run without a debugger). This appears to be in the rolling KeyDiff KV-cache compaction path added in #686 ("Long-context: rolling KeyDiff KV-cache compaction + chunked prefill").

This is 100% reproducible on our machine across multiple runs with an identical two-turn session.

Environment

• Repo / runtime version: <cactus commit hash or release tag you built/downloaded — run pip show cactusor checkweights//manifest.json/ yourcactus --version if applicable>
• OS: macOS <your macOS version, e.g. 15.x>
• Chip: <exact Apple Silicon chip, e.g. Apple M3 Pro — from "About This Mac">, arm64
• Python: 3.12 (venv at .../cactus/venv), calling the engine via ctypes (_ctypes.cpython-312-darwin.so)
• Model bundle: gemma-4-e2b-it-cq4-apple-uncalibrated (Gemma 4 E2B, CQ4, Apple, uncalibrated), loaded via cactus_init(model_path, None, False)
• Integration: custom FastAPI OpenAI-compatible bridge server (cactus_agent_server.py) calling cactus_reset() + cactus_complete() (streaming, with a token callback) from a Python threading.Thread per request, one request per HTTP call, inference_lock serializes calls to a single loaded model handle.

Steps to Reproduce

1. Load gemma-4-e2b-it-cq4-apple-uncalibrated via cactus_init.
2. Turn 1: call cactus_reset() then cactus_complete() (streaming) with a 2-message conversation that renders to ~7,644 chars of messages_json, tools_json present (20 tool defs, ~19,633 chars), options={"auto_handoff": false, "enable_thinking_if_supported": true, "max_tokens": 64000}. Generation completes successfully after 849 decode tokens. No crash.
3. Turn 2, same session (no reset in between other than the cactus_reset() the bridge issues before every completion call): call cactus_reset() then cactus_complete() again, now with a 4-message conversation (prior turn included) rendering to ~11,422 chars of messages_json, same tools/options. Generation proceeds normally — heartbeats confirm ~25 tokens/sec of clean decode — until ~900 decode tokens, at which point the process receives EXC_BAD_ACCESS and dies immediately, mid-stream, with no error returned to the caller.

We reproduced this 3 times in a row with lldb attached and it crashed at effectively the same point in the same code path every time (once at the tail end of a long recursive stack that we've since ruled out as a separate, already-mitigated issue — see "What we've ruled out" below).

Crash details (from lldb -- $(which python) cactus_agent_server.py)

Process stopped
* thread #7, stop reason = EXC_BAD_ACCESS (code=2, address=0x448424000)
    frame #0: 0x0000000107c21440 libcactus_engine.dylib`cactus::kvcompress::rerope_recent_int8(signed char*, float*, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, double, double) + 1104
libcactus_engine.dylib`cactus::kvcompress::rerope_recent_int8:
->  0x107c21440 <+1104>: ldr    s1, [x17], #0x4
    0x107c21444 <+1108>: ldr    q2, [x26, #0x70]
    0x107c21448 <+1112>: tbl.16b v1, { v1 }, v2
    0x107c2144c <+1116>: scvtf.4s v1, v1, #0x18

(lldb) bt
* thread #7, stop reason = EXC_BAD_ACCESS (code=2, address=0x448424000)
  * frame #0: 0x0000000107c21440 libcactus_engine.dylib`cactus::kvcompress::rerope_recent_int8(...) + 1104
    frame #1: 0x0000000107c11cf4 libcactus_engine.dylib`cactus::engine::Model::compress_kv_cache_keydiff(cactus::kvcompress::Params const&) + 1948
    frame #2: 0x0000000107c0c6d0 libcactus_engine.dylib`cactus::engine::Model::decode(std::__1::vector<unsigned int, ...> const&, float, float, unsigned long, std::__1::basic_string<...> const&, float*, float, float) + 1828
    frame #3: 0x0000000107c80ce0 libcactus_engine.dylib`cactus_complete + 8924
    frame #4: 0x000000019a0d0050 libffi.dylib`ffi_call_SYSV + 80
    frame #5: 0x000000019a0d95b8 libffi.dylib`ffi_call_int + 1220
    frame #6: 0x0000000106e9d890 _ctypes.cpython-312-darwin.so`_ctypes_callproc + 776
    frame #7: 0x0000000106e96a10 _ctypes.cpython-312-darwin.so`PyCFuncPtr_call + 216
    frame #8: 0x000000010075d944 Python`_PyObject_MakeTpCall + 124
    frame #9: 0x0000000100851464 Python`_PyEval_EvalFrameDefault + 38756
    frame #10: 0x0000000100760a5c Python`method_vectorcall + 368
    frame #11: 0x0000000100925534 Python`thread_run + 136
    frame #12: 0x00000001008ba430 Python`pythread_wrapper + 48
    frame #13: 0x00000001859a1c58 libsystem_pthread.dylib`_pthread_start + 136

The faulting instruction is a NEON 128-bit load (ldr q2, [x26, #0x70]) immediately followed by a table lookup (tbl.16b) inside rerope_recent_int8, which strongly suggests x26 is pointing at a buffer that is either undersized, misaligned, or has already been freed/moved relative to what the loop bound / stride computation in compress_kv_cache_keydiff expects for this position count.

What we've ruled out

We initially hit a different crash on the same two-turn scenario: EXC_BAD_ACCESS in cactus_metal_plan_build at the bottom of a ~7,440-frame-deep recursive stack inside CactusGraph::execute, which was overflowing the default (~512KB–2MB) stack of the worker thread created via Python's threading.Thread. We mitigated that specific crash on our side by calling threading.stack_size(16 * 1024 * 1024) before starting any threads, which resolved it completely (confirmed via multiple repro runs with no more cactus_metal_plan_build crashes). The rerope_recent_int8 crash above is a separate issue that only became visible once the stack-overflow crash was out of the way — it happens on a normal, shallow (~13 frame) call stack, so it isn't a stack-size problem.

Suspected location to investigate

• cactus::kvcompress::rerope_recent_int8 (buffer indexing/stride at the NEON tbl/load site around the reported offset +1104/+1108)
• cactus::engine::Model::compress_kv_cache_keydiff, specifically how it computes the recent-token window bounds passed down into rerope_recent_int8 for a KV cache that has already undergone rolling KeyDiff compaction on a prior turn (i.e. does the recent-window arithmetic account for a cache that was already compacted once, or does it assume the pre-compaction/original position indices?)
• Given this reproduces specifically on turn 2+ of a session (i.e. after at least one prior compaction cycle has already run), it may be specific to repeated/rolling compaction rather than a single compaction pass — worth checking whether a single very long single-turn generation that crosses the same token count without a prior cactus_reset()/turn boundary also reproduces it, or only the multi-turn case does.

Question for maintainers

Is there any way to disable KeyDiff KV-cache compaction at runtime via options_json (e.g. something like a kv_cache_compression/compress_kv flag)? We checked docs/cactus_engine.md's options table and didn't find one documented — if one exists but is undocumented, that would be a usable workaround for us in the meantime. If not, is pinning to the commit immediately before #686 currently the only way to avoid this on long-context, multi-turn use?

Happy to provide the full runtime bundle manifest, a minimal repro script (no FastAPI, just direct cactus_init/cactus_reset/cactus_complete calls replaying the same two-turn message history), or register/memory dumps around 0x448424000 if useful — let me know what's most useful to attach.

[clumsylad21]

I investigated this and opened #792 with a fix.

I wasn't able to reproduce the reporter's exact Gemma/FastAPI two-turn crash end-to-end, but I was able to isolate the unsafe cache state that leads to the out-of-bounds access.

The key finding is that Metal sliding-window KV caches use current_seq_len as a logical ring-buffer cursor. Unlike a normal contiguous cache, this value can legitimately grow beyond the number of physical rows allocated in the cache.

For example, in the regression setup I observed:

current_seq_len = 600
max_seq_len     = 517
sink_size       = 4

For this cache, only 512 physical sliding-window rows are addressable:

max_seq_len - sink_size - 1 = 512

However, Model::compress_kv_cache_keydiff() was passing the logical current_seq_len directly as the upper bound to the sliding-layer re-RoPE path. That could make rerope_recent_int8() process rows beyond the physical INT8/scales buffers, which is consistent with the reported crash inside that function.

I initially verified this by forcing the sliding append path to CPU, where current_seq_len stays physically bounded; the failure disappeared. Restoring the Metal ring behavior showed the logical cursor exceeding physical capacity as expected.

The fix in #792 keeps the Metal logical cursor unchanged, but bounds the re-RoPE range to the physical cache span:

min(current_seq_len, max_seq_len - sink_size - 1)

I also added a graph-managed INT8 regression test that exercises the Metal sliding ring together with global KeyDiff compaction/shrink and repeated re-RoPE cycles on macOS ARM64.

@Abusoss, if you're able to test #792 against the original workload, that would be very helpful to confirm it resolves the exact end-to-end repro you reported.
@ncylich, Could you please review it?