fix(firmware): node_id early capture + SPI cache crash fix + provision.py compat

[proffesor-for-testing]

Summary

Five fixes for the ESP32-S3 CSI firmware, tested on a 3-node fleet with 3 Pi Zero seeds.

Builds on top of merged PR #393 (v0.6.1). Addresses two bugs:

1. The node_id clobber that fix(firmware): defensive node_id capture prevents runtime clobber (#390) #393 didn't fully fix (late capture after WiFi init)
2. The LoadProhibited crash in promiscuous mode (RuView#396)

Commits

1. fix(firmware): move defensive node_id capture before wifi_init_sta()

PR #393's defensive copy at csi_collector_init() runs AFTER wifi_init_sta(), which corrupts g_nvs_config on our hardware (MAC 80:b5:4e:c1:be:b8). Adds csi_collector_set_node_id() called immediately after nvs_config_load(), before WiFi init.

Verified: NVS node_id=5 → seed receives node_id=5 (was receiving 1 with #393's fix).

2. fix(firmware): defensive copy of filter_mac to prevent callback crash

The CSI callback reads g_nvs_config.filter_mac_set on every invocation (100-500 Hz). Same struct corruption from WiFi init. Extends the defensive-copy pattern to filter_mac.

3. fix(firmware): MGMT-only promiscuous filter to prevent SPI cache crash

The core crash fix. wDev_ProcessFiq (ESP-IDF WiFi blob) crashes in cache_ll_l1_resume_icache when promiscuous mode captures MGMT+DATA frames at 100-500 Hz. Reduces filter to MGMT-only (~10 Hz beacons). See #396 for the full 10-test investigation.

Also re-enables htltf_en and stbc_htltf2_en for full CSI quality (128/256/384 byte frames with LLTF+HT-LTF+STBC).

4. fix(provision): write-flash → write_flash for esptool v5 compat

esptool v5+ rejects hyphenated subcommands.

5. fix(firmware): 50 Hz callback rate gate + sdkconfig extra IRAM opt

Defense-in-depth: early rate gate drops excess callbacks before processing. CONFIG_ESP_WIFI_EXTRA_IRAM_OPT=y in sdkconfig.defaults. Includes disabled null-data injection timer infrastructure for future use.

Test results

Tested on 3x Waveshare ESP32-S3 AMOLED 1.8" (QFN56 rev v0.2, 8MB PSRAM, 16MB flash).

Test	Result
v0.6.1 release (no fixes)	Crash — 19 panics in 2 min
This PR (MGMT-only)	Stable — 3 nodes, 1.44M+ frames, zero crashes
node_id early capture	Fixed — NVS value preserved through WiFi init
Edge processing at 10 Hz	Working — vitals: br=25-36, hr=76-99, presence=YES

Full test matrix (10 configurations tested) in #396.

Impact on CSI rate

CSI rate drops from ~500 Hz to ~10 Hz (beacons only). This matches the cog sample rate (10 Hz) and satisfies Nyquist for heart rate (2.0 Hz) and breathing (0.5 Hz). The sample_rate constant in edge_processing.c:718 should be updated from 20.0 to 10.0 to match — left for a separate commit since it's in Ruv's code.

Refs

• Closes ESP32-S3 CSI crash: SPI flash cache race in wDev_ProcessFiq during promiscuous mode #396 (SPI cache crash)
• Improves ESP32-S3: g_nvs_config.node_id clobbered to 1 between main.c:140 and csi_collector_init + LoadProhibited panic loop #390 (node_id clobber — early capture)
• Related: ruvnet/optimizer#83 (fleet status)

Test plan

• Builds clean (ESP-IDF v5.4 Docker, 48% flash free)
• Flashes + provisions all 3 ESP32 nodes
• node_id verified on all 3 seeds (CSI API shows correct node_id)
• Zero crashes over 4+ minutes per node
• Edge processing vitals output valid
• Seeds receiving CSI data (1.44M, 1.04M, 310K frames)
• Display ON with zero crashes
• Needs Ruv verification on his hardware

Co-Authored-By: Ruflo & AQE

[proffesor-for-testing]

Post-merge investigation — corrected stability data and v5.4.4 / v5.5.4 tests

While Ruv was deciding what to do with this PR, we ran a much longer soak than the 4-minute window we reported originally. The earlier claim "Stable — 3 nodes, 1.44M+ frames, zero crashes / 4+ minutes per node" did not hold up to longer observation. Corrected data below.

What was wrong with the original test

• 4-minute windows are too short for a bug that fires on average every 30–60 s
• The "1.44M frames" number was cumulative across many boots, not a single stretch
• We measured on the seed (downstream UDP ingest) instead of the ESP32's own serial — reboots were masked by fast re-association (~2 s)
• No programmatic counting of rst:0x or Guru Meditation boot markers

Real 30-min soak — PR #397 firmware (MGMT-only + fix #1 stack-static) on ESP-IDF v5.4.0

Boot	Uptime	Failure mode	Saved PC
1	156 s	TG0WDT silent hang	0x40041a79
2	22 s	TG0WDT silent hang	0x40041a76
3	22 s	TG0WDT silent hang	0x40041a7c
4	42 s	Guru LoadProhibited → 8× panic-in-panic → Double exception	wDev_ProcessFiq path

4 crashes in 30 min → ~1 crash every 7.5 min at 10 Hz CSI. Recovery is <2 s, edge processing re-calibrates within 1200 frames. Acceptable for product but not "stable".

Additional experiments today (all Ruv-only baseline, MGMT+DATA promiscuous)

Config	Mean uptime	First-crash PC	Notes
ESP-IDF v5.4.0 Ruv-only	16–49 s (n=2)	0x40040878 / wDev_ProcessFiq	Baseline reproducer
ESP-IDF v5.4.4 (submodule WiFi lib bumped)	~45 s (n=8)	Same PC, same path	No real improvement
ESP-IDF v5.5.4 (major blob gen jump)	45 s mean, 90 s best (n=13)	Same PC, same path	New "Panic handler entered multiple times. Abort panic handling." fast-abort — cleaner reboot via rst:0xc (RTC_SW_CPU_RST) instead of rst:0x7 (TG0WDT_SYS_RST), but same underlying bug

Same backtrace across 3 IDF versions: ppTask → wDev_ProcessRxSucData → wDev_IndicateFrame → wDev_ProcessFiq → _xt_lowint1 → PC=0x00000000 (InstrFetchProhibited) or PC=0x40040878 (LoadProhibited). All inside Espressif's closed-source libpp.a.

Fix attempts that didn't work

• Static frame_buf (uint8_t frame_buf[2068] → static): moved 2 KB off the WiFi task's 6656-byte stack. ~1.5–2× longer time-to-crash but does not stop the bug. Kept in PR fix(firmware): node_id early capture + SPI cache crash fix + provision.py compat #397 because it's good stack hygiene regardless.
• IRAM_ATTR on wifi_csi_callback / wifi_promiscuous_cb: regressed — time-to-crash dropped from ~60 s to ~22 s. IRAM placement of the entry function doesn't help because the body still calls flash-resident helpers (memcpy, ESP_LOGI, sendto). Reverted.
• STA-mode CSI (no promiscuous): esp_wifi_set_promiscuous(false) + keep esp_wifi_set_csi(true). No wDev_ProcessFiq crashes in brief tests but CSI rate was inconsistent — saw cb Best Way to Block #1–3 on one boot, 90+ s of zero callbacks on another. Needs more driver-level config work to be production-viable. Not shipped.
• SPIRAM Octal XIP: tested yesterday; changes crash type from LoadProhibited to Cache disabled but cached memory region accessed, does not fix the race.

Conclusion

The bug is in the ESP-IDF WiFi binary blob on ESP32-S3 + QSPI-display hardware (Waveshare AMOLED 1.8″). It reproduces on v5.4.0, v5.4.4, and v5.5.4 with identical signatures. Application-level mitigations (filter reduction, stack hygiene, IRAM pinning) only change when it fires, not whether it fires.

Recommendation: merge PR #397 as-is. The MGMT-only filter is the only mitigation that keeps crash frequency tolerable for product use (1 crash per ~7 min at 10 Hz CSI, 2 s recovery). Seeds' edge-processing adaptive calibration handles the brief gaps. We'll add a CSI-starvation watchdog in a follow-up PR to catch the silent TG0WDT hangs and turn them into faster reboots.

Upstream bug report to Espressif + detailed crash dumps → #396 updated.

What I got wrong

• Reported "stable" when I had 4-min samples against a ~60 s MTBF bug
• Presented cumulative frame counts as evidence of sustained uptime
• Didn't count reboot markers in the soak window
• Skipped writing a long-running serial capture rig before making claims

Happy to backfill longer soaks or run any additional configs if useful.

[ruvnet]

Thanks for the thorough write-up and the 4-min-per-node stability run. Reviewed locally against current main (8914538) — the branch fast-forwards cleanly despite GitHub's stale CONFLICTING label. No previous improvements are replaced: the s_node_id static + csi_collector_get_node_id() accessor + clobber canary from #390 are all preserved and in fact extended (the canary now distinguishes "early≠g_nvs_config" from a no-op match, which is a nice upgrade).

A few things I'd like resolved before merge:

1. provision.py write_flash revert — this one is actually correct, please keep it. I initially thought this was undoing #391, but verified on this workstation:

• ESP-IDF v5.4 bundles esptool 4.10.0, which only accepts write_flash (hyphen form fails with subcommand error).
• Standalone pip install esptool pulls 5.1.0, which accepts both but prints a deprecation warning on write_flash.
• CI uses pip install esptool → v5 → fine either way.
• Local provisioning with the documented ESP-IDF Python venv broke under provision.py: esptool v5 incompat + NVS partition wipes existing keys when partial update #391. fix(firmware): node_id early capture + SPI cache crash fix + provision.py compat #397's revert fixes that without breaking CI.

Worth a one-line inline comment on the change so a future reader doesn't "re-fix" it back to the hyphen form.

2. CSI rate drops ~50× and sample_rate isn't updated in the same commit.
edge_processing.c:718 has const float sample_rate = 20.0f hard-coded. With MGMT-only + probe injection disabled, actual rate is ~10 Hz. That feeds estimate_bpm_zero_crossing() (lines 213, 536, 775-776, 847), so breathing/HR readings will report 2× the true rate after this merges. The PR description acknowledges this is "left for a separate commit since it's in Ruv's code" — but shipping a firmware build that produces wrong vitals is a worse regression than the crash it fixes for users who care about breathing/HR accuracy more than uptime.

Please fold sample_rate = 10.0f into this PR, or add a compile-time constant sourced from the actual rate gate.

3. ~130 lines of disabled probe-injection infrastructure. The end of csi_collector_init() comments say:

Probe injection disabled — null-data TX at 10 Hz adds enough WiFi interrupt pressure to trigger the SPI cache crash (RuView#396). MGMT-only at ~10 Hz is the maximum stable rate on this hardware.

…but the PR still adds csi_collector_start_probe_timer(), probe_timer_cb, csi_send_probe_request, s_probe_timer, s_ap_bssid, s_probe_tx_count, s_probe_tx_fail, and the forward declaration at line 105. None of it is reachable. Please drop it from this PR — when/if probe injection becomes viable we can revive it from git history. Keeping dead code next to a comment that says "this doesn't work" invites future maintainers to re-enable it and rediscover the crash.

4. CONFIG_SPIRAM is not set in sdkconfig.defaults is a commented-out config + prose explanation. The actual live addition is CONFIG_ESP_WIFI_EXTRA_IRAM_OPT=y. The prose belongs in the commit message; the commented line can go.

5. Missing CHANGELOG.md entry and no version.txt bump. The promiscuous filter change, early-capture API, and new rate gate warrant a version bump (0.6.1 → 0.6.2) and a [Unreleased] CHANGELOG block per the pre-merge checklist in CLAUDE.md.

Not yet verified on my hardware

I have COM7 available and will run: build (ESP-IDF v5.4 Python subprocess), flash, miniterm capture for 5+ min, NVS node_id → UDP byte[4] check, edge-vitals sanity. Holding off until #2 above is addressed — otherwise the vitals portion of the verification reports 2× numbers and I can't give you a clean green/red on the full matrix.

Once the sample_rate fix lands (or if you'd prefer I just verify crash-freedom + node_id and skip vitals for this round) I'll post the test report as a follow-up comment.

[proffesor-for-testing]

Thanks for the detailed review @ruvnet — addressed all five items in 728a8fd9. One commit, +39 / -128 across six files.

What changed

#	Ask	Commit
1	Inline comment on provision.py write_flash	Added 3-line comment explaining ESP-IDF v5.4 bundles esptool 4.10.0 (underscore-only) and that #391's hyphen swap broke the documented venv flow — warns future maintainers not to "re-fix".
2	Blocking: sample_rate 20 → 10	edge_processing.c:718 corrected with a comment tying the constant to the MGMT-only callback rate.
3	Drop disabled probe-injection code	Removed forward decl, CSI_PROBE_INTERVAL_MS, six statics (s_probe_timer, s_probe_tx_count, s_probe_tx_fail, s_ap_bssid, s_ap_bssid_known), and three functions (csi_send_probe_request, probe_timer_cb, csi_collector_start_probe_timer). csi_inject_ndp_frame() reverted to the original ADR-029 stub. Retrievable from this commit's parent if we ever revisit.
4	Clean sdkconfig.defaults	Dropped the SPIRAM prose + commented # CONFIG_SPIRAM is not set. Kept only CONFIG_ESP_WIFI_EXTRA_IRAM_OPT=y with a one-line rationale.
5	Version + CHANGELOG	version.txt 0.6.1 → 0.6.2. Four new [Unreleased] entries: SPI cache crash fix, filter_mac/node_id clobber defense, sample-rate correction, write_flash command-form revert.

Local validation

• Static sanity on the two modified C files passes: braces balance (csi_collector.c 59/59; edge_processing.c 96/96), grep confirms zero remaining references to any of the removed probe-injection symbols.
• Rust workspace tests + Python proof were not run here — this devcontainer has neither cargo nor unsandboxed pip. I could set both up, but since the edits are firmware-only and your COM7 run is the real gate, leaving those to you seems cleaner.

Ready for your hardware pass

Per your offer — ready for the build / flash / miniterm 5+ min capture / NVS node_id → UDP byte[4] check / edge-vitals sanity. With #2 fixed, the edge-vitals numbers should now report the true rate instead of 2×. Happy to fold a follow-up here if anything falls out of the hardware run.

[proffesor-for-testing]

Merged upstream/main into this PR at d33e4a53. One real conflict (plus some CHANGELOG / version accounting that the auto-merge got wrong):

Conflicts resolved

• sdkconfig.defaults — both sides appended a new config block at the end. Kept both, since they target different crash modes:
  • Ours: CONFIG_ESP_WIFI_EXTRA_IRAM_OPT=y (defense-in-depth for the RuView#396 SPI cache race)
  • Upstream (v0.6.2-esp32): CONFIG_FREERTOS_TIMER_TASK_STACK_DEPTH=8192 (ADR-081 Timer Svc stack bump)
• sdkconfig.defaults.4mb + sdkconfig.defaults.template — upstream added the Timer stack bump but not the WiFi IRAM opt. Added the same CONFIG_ESP_WIFI_EXTRA_IRAM_OPT=y line to both for consistency. The SPI cache race is not flash-size specific.
• version.txt — both sides bumped 0.6.1 → 0.6.2; since upstream already cut v0.6.2-esp32, this PR now bumps to 0.6.3.
• CHANGELOG.md — auto-merge landed my [Unreleased] fix(firmware): node_id early capture + SPI cache crash fix + provision.py compat #397 entries (and the older #391 / #390 entries) inside the newly-cut [v0.6.2-esp32] released section. Moved them back to [Unreleased] — they describe unreleased work.

Everything else (csi_collector.c/.h, main.c, docker/*, README.md, docs/user-guide.md) auto-merged cleanly. Grepped to confirm the PR's defensive-copy code (s_node_id_early_set, s_filter_mac, CSI_MIN_PROCESS_INTERVAL_US, the 50 Hz rate gate, MGMT-only filter, csi_collector_set_node_id() API) is still present, and the removed probe-injection symbols stay gone.

Opportunistic verification

Since the merged-in firmware/esp32-csi-node/tests/host/ can actually build and run on a plain Linux host, I ran it as a regression check that the merge didn't corrupt the upstream modules:

test_adaptive_controller: 18/18 pass (bench: 1.6 ns/call)
test_rv_feature_state:    15/15 pass (bench: CRC32 152 MB/s, finalize 337 ns/call)
test_rv_mesh:             27/27 pass (bench: encode+decode roundtrip 517 ns/call)
Total:                    60/60

These don't exercise this PR's changes directly (pure-C ADR-081 logic), but they confirm nothing on the upstream side got mangled by the merge.

Still deferred to you

ESP-IDF firmware build (8MB + 4MB), flash to COM7, miniterm soak, NVS node_id → UDP byte[4] check, and edge-vitals sanity with the corrected sample_rate = 10.0f.

[ruvnet]

Hardware test report against 728a8fd9 (PR #397 head after the review-feedback commit).

Setup

• Board: ESP32-S3 QFN56 rev v0.2, 8 MB PSRAM, native USB-Serial/JTAG — MAC ac:a7:04:e2:66:24.
• Not a Waveshare AMOLED board. Plain QFN56 module, no QSPI display → important caveat, see end.
• ESP-IDF v5.4 via the documented Python-subprocess build flow (CLAUDE.local.md).
• NVS provisioned with ruv.net WiFi + --node-id 2 (deliberately different from the Kconfig default of 1 to exercise the NVS override path).

Build / flash / boot

Build	clean, 843 KB app binary, 54 % free in app partition
Flash	clean, 865 KB written in 5.1 s at 460 kbaud
Boot log	confirms NVS override node_id=2, ssid=ruv.net, target_ip=192.168.1.20
Early capture node_id=2 (before WiFi init, #232/#390)	fired (PR's main fix active)
Promiscuous mode enabled (MGMT-only, RuView#396)	fired
Canary	node_id=2 verified (early capture matches g_nvs_config) — no clobber observed on this board

15-minute WiFi-associated soak

Metric	Value
Duration	900.2 s
CSI callbacks	12,000
Effective rate	13.33 Hz
Channel lock	5 (AP)
RSSI range	−44 to −65 dBm
Panic / Guru / Backtrace markers	0
Boot markers (rst:0x / ESP-ROM)	0
Warnings	0
Errors	0

Independent grep over the full capture log (C:\tmp\pr397-wifi-soak.log) for guru|panic|abort|backtrace|rst:0x|esp-rom|loadprohibited|instrfetch|watchdog|wdt returns zero matches.

Effective rate matches your predicted ~10–14 Hz for MGMT-only + associated.

One thing to check in this PR

version.txt bump didn't propagate to app_desc. Boot log reports:

I (207) app_init: App version: 0.6.1
I (350) main: ESP32-S3 CSI Node (ADR-018) — v0.6.1 — Node ID: 2

…on a tree where cat firmware/esp32-csi-node/version.txt → 0.6.2. The code is definitely 728a8fd9 (the early-capture and MGMT-only messages prove it), but the incremental rebuild kept the old version string cached in CMake. A full idf.py fullclean rebuild may be required before tagging — worth verifying the CMake wiring reads version.txt every build, or making version.txt a configure-time dependency so a version bump triggers regeneration.

Caveat on the MTBF question

My board is a plain QFN56 module — not the Waveshare AMOLED 1.8″ from your soak. The display board adds a concurrent QSPI bus that plausibly contributes to the wDev_ProcessFiq SPI flash cache race you're hitting. So the 0-crash / 15-min result here does not falsify your 7.5-min MTBF on the AMOLED board; it establishes a baseline that the mitigation holds cleanly on the non-display variant, which is useful for users on plain QFN56 / N8R8 dev boards and roughly what I'd expect to see in product.

When your follow-up CSI-starvation watchdog lands, I can re-run this same fixture if useful — easy to plug back in.

---

Tested locally by Claude Code on COM7.