diff --git a/README.md b/README.md index ce30f98..3a64924 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,10 @@ # Electronica DT workshop: heap exhaustion on nRF9151 + nuSIM -A joint workshop by [Exein](https://www.exein.io) and [Deutsche Telekom IoT](https://iot.telekom.com) at Electronica. The Exein Runtime Security Monitor runs on a Nordic nRF9151 DK with Deutsche Telekom's integrated [nuSIM](https://iot.telekom.com/global/iot-connectivity/isim). Press **Button 2**, the device burst-allocates past 60% of its heap, the monitor reports `heap_low` over LTE, and a **Heap Exhaustion** incident shows up on [runtime.exein.io](https://runtime.exein.io). +A joint workshop by [Exein](https://www.exein.io) and [Deutsche Telekom IoT](https://iot.telekom.com) at Electronica. The Exein Runtime Security Monitor runs on a Nordic nRF9151 DK with Deutsche Telekom's integrated [nuSIM](https://iot.telekom.com/global/iot-connectivity/isim). Each of the four buttons triggers one incident type: the monitor reports it over LTE, and the incident shows up on [runtime.exein.io](https://runtime.exein.io) (**Log Event**, **Heap Exhaustion**, **Free Stack Below Critical Threshold**, **Memory Leak**). ``` -Button 2 ─▶ heap burst ─▶ HEAP_LOW policy ─▶ POST /zephyr/threats (modem TLS, LTE-M) ─▶ incident +Button 1-4 ─▶ error log / heap burst / deep stack / slow leak ─▶ LOG / HEAP_LOW / STACK_LOW / MEM_LEAK policy + ─▶ POST /zephyr/threats (modem TLS, LTE-M) ─▶ incident ``` **Try it from your browser:** [**exein-io.github.io/electronica-dt-workshop**](https://exein-io.github.io/electronica-dt-workshop/) flashes the latest release onto the DK over USB (Chrome/Edge on Linux, macOS, Windows or Android, nothing to install, nuSIM-safe), with a serial terminal and the registration token field. Without the browser: [Flash a release with nrfutil](#b-with-nrfutil-windows-macos-linux). @@ -115,7 +116,7 @@ Use this whenever you prefer Nordic's tools or can't use Chrome/Edge. **Never** nrfutil device reset ``` `ERASE_RANGES_TOUCHED_BY_FIRMWARE` is mandatory: never a full erase. With several boards connected, add `--serial-number ` (`nrfutil device list`). -4. Watch the log in a serial terminal that asserts DTR (115200 baud, the J-Link's first COM port / tty; e.g. PuTTY, or `python3 -m serial.tools.miniterm --dtr 1 115200`): a new board logs `registered with the platform; token cleared`, then press **Button 2**. +4. Watch the log in a serial terminal that asserts DTR (115200 baud, the J-Link's first COM port / tty; e.g. PuTTY, or `python3 -m serial.tools.miniterm --dtr 1 115200`): a new board logs `registered with the platform; token cleared`, then press the buttons (see *Run the demo*). `settings.hex` is the same settings page the web flasher writes (`0x80000`). The board clears the token after one registration attempt; a board that already has an identity ignores it, so reflashing never creates a new agent. @@ -146,17 +147,19 @@ net: L4 connected ← LTE up (LED1 solid; blin time_sync: wall clock set from pool.ntp.org registration: client identity present in the credential store heartbeat: sending system info … POST status 204 -workload: Press Button 2 for a heap exhaustion (HEAP_LOW) incident +workload: Buttons: 1 Log Event, 2 Heap Exhaustion, 3 Stack Overflow Risk, 4 Memory Leak ``` -Press **Button 2** (nothing else running in between): -``` -workload: BTN2: burst-allocated 24 x 512 B (expect HEAP_LOW) -dispatcher: POST 1 record(s) … POST status 204 -``` -→ runtime.exein.io: **Heap Exhaustion**, severity high, "heap used 12 kB, threshold 10 kB" (60% of the 16 KB heap = 9831 B). +Press one button at a time and wait for its incident before the next one; violations in the same 5 s reporting interval are grouped into one incident. + +| Button | What the board does | Log | Incident on runtime.exein.io | +|---|---|---|---| +| **1** | Logs the same error 5 times | ` workload: sensor read failed: -5` (5×) | **Log Event**, medium, *discovery*: **one** incident showing the message, not five | +| **2** | Burst-allocates 24 × 512 B (past 60% of the 16 KB heap), frees after 2 s | `BTN2: burst-allocated 24 x 512 B …` | **Heap Exhaustion**, high, *impact* | +| **3** | Starts thread `stack_hog`, which recurses to ~85% of its 1 KB stack and stays there 20 s | `BTN3: stack_hog near its stack limit for 20 s …` | **Free Stack Below Critical Threshold**, high, *impact*, naming thread `stack_hog`; repeats every interval while the thread lives | +| **4** | Leaks 160 × 32 B, one every 100 ms (16 s), below the exhaustion threshold; frees after 2 s | `BTN4: leaking 160 x 32 B over 16 s` … `BTN4: leaked …` | **Memory Leak**, medium, *impact*, once the growth has lasted 8 leak windows; usually one more right after the memory is freed | -**Platform support (2026-10):** `/zephyr/threats` accepts only `heap_low`. Every other finding (`heap_drop`, `stack_low`, `log`, …) gets the **whole batch** rejected with `400 unsupported finding`, so the blob is built with `POLICY_HEAP_LOW` only. +Every alert batch ends with `dispatcher: … POST status 204`. A button pressed again while its previous run is still active is ignored. nuSIM health check over AT (the app's AT host shares the log UART; any terminal with DTR asserted, e.g. `python3 -m serial.tools.miniterm --dtr 1 115200`): `AT%XICCID` → your nuSIM's ICCID, `AT+CEREG?` → `0,1`/`0,5`, `AT+COPS?` → `26201`. @@ -165,7 +168,7 @@ nuSIM health check over AT (the app's AT host shares the log UART; any terminal ## How it fits together - **`app/src/main.c`**: `nrf_modem_lib_init()`, platform CA into the modem key store (LTE must be off), LTE via the modem's Zephyr net_if (`conn_mgr`), `runtime_zephyr_start()` (waits for IPv4, SNTP, SystemInfo), `workload_start()`. A small credential backend keeps CA + client identity in the modem; nothing goes to TF-M ITS. LED1 tracks L4 connectivity. -- **`app/src/app_workload.c`**: Button 2 → 24 × 512 B burst on the system heap → freed again. +- **`app/src/app_workload.c`**: one trigger per button. Button 1 → 5 × `LOG_ERR`; Button 2 → 24 × 512 B heap burst, freed again; Button 3 → `stack_hog` thread at ~85% stack; Button 4 → 160 × 32 B slow leak, freed again. - **`app/prj.conf`**: monitor feature set (frozen into the blob), modem TLS, platform endpoint, AT host. - **Monitor as a binary Zephyr module:** see [How the Exein runtime library is configured](#how-the-exein-runtime-library-is-configured). - **Rebuilding the blob** (internal; needed after changing any monitor option in `prj.conf`): build `app/` against the `exein-runtime-zephyr` sources (with a `platform-protobuf` checkout at `../protobuf`, branch `zephyr-agent-schema`, plus the issue #3 patch applied), then package it with upstream's script and commit `dist/runtime-zephyr/*` as `runtime-zephyr/`: @@ -202,10 +205,12 @@ Every option the library code itself reads is compiled into the `.a`. Changing t ```conf CONFIG_RUNTIME_MONITOR=y -CONFIG_POLICY_HEAP_LOW=y # the only finding prod accepts -CONFIG_POLICY_P2=n # memory-leak trend (rejected by the platform today) -CONFIG_POLICY_STACK_LOW=n # stack high-water (rejected) -CONFIG_POLICY_LOG=n # error logs → alerts (rejected) +CONFIG_POLICY_HEAP_LOW=y # Button 2: Heap Exhaustion +CONFIG_POLICY_P2=y # Button 4: Memory Leak (trend over leak windows) +CONFIG_POLICY_STACK_LOW=y # Button 3: Free Stack Below Critical Threshold +CONFIG_POLICY_LOG=y # Button 1: Log Event (every LOG_ERR) +CONFIG_THREAD_STACK_WARN_PCT=80 # stack_hog peaks at ~85%; the default 90 misses it +CONFIG_MEM_LEAK_WINDOW_EVENTS=16 # 8 windows x 16 heap events; Button 4 makes 160 CONFIG_THREAD_COLLECT_PERIOD_MS=2000 CONFIG_ANALYZER_PERIOD_MS=1000 # analyzer drains the event buffer every second CONFIG_MONITOR_EVENT_BUF_SIZE=16384 @@ -265,7 +270,7 @@ rc = runtime_zephyr_start(&info, &conn, NULL); // NULL = thresholds from Kconfi 4. Send SystemInfo. 5. Start the collector, analyzer and dispatcher threads. -Then `workload_start()` arms Button 2. +Then `workload_start()` arms Buttons 1-4. ### What to change where @@ -274,7 +279,7 @@ Then `workload_start()` arms Button 2. | Endpoint, ports, connect timeout | `prj.conf` connection block | App rebuild | | Platform host, token, custom id | settings page (web flasher or `scripts/settings-hex.py`) | Nothing: flash the page | | Policies, periods, buffer sizes | `prj.conf` monitor block | **Blob rebuild** (see "Rebuilding the blob" above), then app | -| HEAP_LOW threshold only | pass `runtime_zephyr_thresholds` to `runtime_zephyr_start()` | App rebuild | +| HEAP_LOW / STACK_LOW thresholds, leak windows | pass `runtime_zephyr_thresholds` to `runtime_zephyr_start()` | App rebuild | | Where credentials are stored | `creds` backend in `main.c` | App rebuild | `runtime-zephyr/` also contains upstream's `runtime_monitor.conf`, which is the TLS-on-app-core (mbedTLS) variant. This app doesn't use it: `prj.conf` follows upstream's `runtime_monitor_ncs.conf` (TLS in the modem). @@ -284,7 +289,8 @@ Then `workload_start()` arms Button 2. - **Flashing failed halfway:** don't unplug, power off or reset the board, flash again right away. On the nRF9151 debug access locks at every reset until the firmware opens it; without a bootloader nothing opens it, and the only way back in is ERASEALL, which destroys the nuSIM. - **`NW reg status: 90` (UICC_FAIL):** built without the Redtea TF-M (`overlay-nusim.conf` missing), or the nuSIM ITS is erased (check `0xF8000`; restoring needs a full backup taken before, otherwise DT has to re-provision). - **`no connectivity before registration (-116)`:** LTE attach slower than `CONFIG_RUNTIME_MONITOR_NET_CONNECT_TIMEOUT` (180 s). -- **Incidents POST `400`:** an unsupported finding in the batch; see *Platform support*. +- **Incidents POST `400`:** the platform rejected a finding in the batch (the whole batch is dropped). All four finding types are accepted on platform.exein.io since 2026-10-23. +- **Log Event incidents without pressing Button 1:** the LOG policy reports *every* `LOG_ERR` on the board, including the system's own (modem, LTE). - **No AT response:** DTR must be asserted (picocom `--noreset`, minicom and raw `stty` fail silently). ## Contacts diff --git a/app/prj.conf b/app/prj.conf index dd33861..ff93dca 100644 --- a/app/prj.conf +++ b/app/prj.conf @@ -22,13 +22,15 @@ CONFIG_ZBUS_MSG_SUBSCRIBER_NET_BUF_STATIC_DATA_SIZE=192 # --- Runtime security monitor: built-in policies only (no LLEXT/syscall hooks) --- # Feature set + tunables below are frozen into the blob (README: "Rebuilding the blob"). CONFIG_RUNTIME_MONITOR=y -# platform.exein.io /zephyr/threats accepts only heap_low today; any other -# finding (heap_drop, stack_low, log, ...) gets the whole batch rejected with -# 400 "unsupported finding". Re-enable once the platform takes them. +# One policy per button (app_workload.c); platform.exein.io accepts all four. CONFIG_POLICY_HEAP_LOW=y -CONFIG_POLICY_P2=n -CONFIG_POLICY_STACK_LOW=n -CONFIG_POLICY_LOG=n +CONFIG_POLICY_P2=y +CONFIG_POLICY_STACK_LOW=y +CONFIG_POLICY_LOG=y +# Button 3's stack_hog peaks at ~85% of its stack; the default 90 misses it. +CONFIG_THREAD_STACK_WARN_PCT=80 +# Button 4 leaks 160 x 32 B: 16-event windows x 8 = 128 events, 5 KB growth. +CONFIG_MEM_LEAK_WINDOW_EVENTS=16 CONFIG_THREAD_COLLECT_PERIOD_MS=2000 # Analyzer drains the heap event buffer every second; 5000 ms with a small # buffer drops FREE events and wedges the HEAP_LOW live-bytes tally. @@ -79,7 +81,7 @@ CONFIG_FLASH_MAP=y # --- Custom id nRF9151-DK_ in SystemInfo (main.c) --- CONFIG_HWINFO=y -# --- Button 2 triggers the heap burst; LED1 shows LTE state --- +# --- Buttons 1-4 trigger the four incidents; LED1 shows LTE state --- CONFIG_GPIO=y # --- Resources --- diff --git a/app/src/app_workload.c b/app/src/app_workload.c index 2509708..eb41e03 100644 --- a/app/src/app_workload.c +++ b/app/src/app_workload.c @@ -1,10 +1,14 @@ /* - * Demo workload for the Exein Runtime Security Monitor: heap exhaustion. + * Demo workload for the Exein Runtime Security Monitor: one button per + * incident type. * - * Button 2 burst-allocates past 60% of the 16 KB system heap, so the - * HEAP_LOW policy fires and the monitor posts a heap_low finding over LTE; - * the platform turns it into a "Heap Exhaustion" incident. The memory is - * held for HOLD_MS, then freed, so the demo is repeatable. + * Button 1 5 identical LOG_ERR -> Log Event (one incident) + * Button 2 burst past 60% of the heap -> Heap Exhaustion + * Button 3 thread "stack_hog" at ~90% -> Free Stack Below Critical Threshold + * Button 4 slow sustained heap growth -> Memory Leak + * + * Every trigger cleans up after itself (memory freed, thread exits), so the + * demo is repeatable. */ #include @@ -15,6 +19,22 @@ LOG_MODULE_REGISTER(workload, LOG_LEVEL_INF); +/* ── Button 1: Log Event ─────────────────────────────────────────── */ + +/* Identical errors in one reporting interval become a single incident. */ +#define LOG_REPEAT 5 + +static void log_burst_work(struct k_work *w) +{ + ARG_UNUSED(w); + for (int i = 0; i < LOG_REPEAT; i++) { + LOG_ERR("sensor read failed: -5"); + } +} +static K_WORK_DEFINE(log_burst, log_burst_work); + +/* ── Button 2: Heap Exhaustion ───────────────────────────────────── */ + /* The analyzer checks HEAP_LOW once per CONFIG_ANALYZER_PERIOD_MS, after * replaying the alloc/free events of that period: memory freed within the * same period never shows up. Holding it for two periods always does. @@ -48,23 +68,153 @@ static void heap_burst_work(struct k_work *w) } static K_WORK_DEFINE(heap_burst, heap_burst_work); -static const struct gpio_dt_spec btn2 = GPIO_DT_SPEC_GET(DT_ALIAS(sw1), gpios); -static struct gpio_callback btn2_cb; +/* ── Button 3: Stack Overflow Risk ───────────────────────────────── */ + +/* A dedicated thread recurses until STACK_HOG_PCT of its stack is used, then + * stays alive for STACK_HOG_HOLD_S so the collector (THREAD_COLLECT_PERIOD_MS) + * samples it, then exits. Must stay above CONFIG_THREAD_STACK_WARN_PCT and + * below 100: a real overflow halts the board. Its own thread, because the + * high-water mark of the system workqueue would stay high forever. + */ +#define STACK_HOG_SIZE 1024 +#define STACK_HOG_PCT 90 +#define STACK_HOG_HOLD_S 20 -static void btn2_pressed(const struct device *d, struct gpio_callback *cb, uint32_t pins) +K_THREAD_STACK_DEFINE(hog_stack, STACK_HOG_SIZE); +static struct k_thread hog_thread; +static atomic_t hog_busy; + +static size_t stack_used(void) { - k_work_submit(&heap_burst); /* k_malloc is not ISR-safe */ + const struct k_thread *t = k_current_get(); + volatile char here; + + return t->stack_info.start + t->stack_info.size - (uintptr_t)&here; } -void workload_start(void) +__attribute__((noinline)) +static void deep_recurse(size_t target) +{ + volatile char pad[64]; + + pad[0] = 0; + if (stack_used() < target) { + deep_recurse(target); + } + pad[63] = pad[0]; +} + +static void stack_hog_entry(void *p1, void *p2, void *p3) +{ + ARG_UNUSED(p1); + ARG_UNUSED(p2); + ARG_UNUSED(p3); + + /* One frame of headroom: the check runs before the next 64 B pad. */ + deep_recurse(k_current_get()->stack_info.size * STACK_HOG_PCT / 100 - 96); + LOG_WRN("BTN3: stack_hog near its stack limit for %d s (expect STACK_LOW)", + STACK_HOG_HOLD_S); + k_sleep(K_SECONDS(STACK_HOG_HOLD_S)); + atomic_clear(&hog_busy); +} + +static void stack_hog_work(struct k_work *w) +{ + ARG_UNUSED(w); + if (!atomic_cas(&hog_busy, 0, 1)) { + return; /* previous stack_hog still running */ + } + k_thread_create(&hog_thread, hog_stack, K_THREAD_STACK_SIZEOF(hog_stack), + stack_hog_entry, NULL, NULL, NULL, 8, 0, K_NO_WAIT); + k_thread_name_set(&hog_thread, "stack_hog"); +} +static K_WORK_DEFINE(stack_hog, stack_hog_work); + +/* ── Button 4: Memory Leak ───────────────────────────────────────── */ + +/* One unfreed block every LEAK_PACE_MS, so every leak window + * (CONFIG_MEM_LEAK_WINDOW_EVENTS heap events) grows. LEAK_N blocks span more + * than the 8 windows the policy needs and exceed MEM_LEAK_MIN_NET_GROWTH + * (4096 B), while staying below the 60% HEAP_LOW threshold of the 16 KB heap. + * Freed again after HOLD_MS. + */ +#define LEAK_N 160 +#define LEAK_SIZE 32 +#define LEAK_PACE_MS 100 + +static void *leaked[LEAK_N]; +static int leak_n; +static bool leak_busy; + +static void leak_free_work(struct k_work *w) +{ + ARG_UNUSED(w); + while (leak_n > 0) { + k_free(leaked[--leak_n]); + } + leak_busy = false; +} +static K_WORK_DELAYABLE_DEFINE(leak_free, leak_free_work); + +static void leak_step_work(struct k_work *w); +static K_WORK_DELAYABLE_DEFINE(leak_step, leak_step_work); + +static void leak_step_work(struct k_work *w) { - if (!gpio_is_ready_dt(&btn2)) { - LOG_ERR("button 2 not ready"); + ARG_UNUSED(w); + if (leak_n < LEAK_N && (leaked[leak_n] = k_malloc(LEAK_SIZE)) != NULL) { + leak_n++; + k_work_schedule(&leak_step, K_MSEC(LEAK_PACE_MS)); return; } - gpio_pin_configure_dt(&btn2, GPIO_INPUT); - gpio_pin_interrupt_configure_dt(&btn2, GPIO_INT_EDGE_TO_ACTIVE); - gpio_init_callback(&btn2_cb, btn2_pressed, BIT(btn2.pin)); - gpio_add_callback(btn2.port, &btn2_cb); - LOG_INF("Press Button 2 for a heap exhaustion (HEAP_LOW) incident"); + LOG_WRN("BTN4: leaked %d x %d B, freeing in %d ms (expect MEM_LEAK)", + leak_n, LEAK_SIZE, HOLD_MS); + k_work_schedule(&leak_free, K_MSEC(HOLD_MS)); +} + +static void leak_start_work(struct k_work *w) +{ + ARG_UNUSED(w); + if (leak_busy) { + return; /* previous leak still running or held */ + } + leak_busy = true; + LOG_WRN("BTN4: leaking %d x %d B over %d s", LEAK_N, LEAK_SIZE, + LEAK_N * LEAK_PACE_MS / 1000); + k_work_schedule(&leak_step, K_NO_WAIT); +} +static K_WORK_DEFINE(leak_start, leak_start_work); + +/* ── Buttons ─────────────────────────────────────────────────────── */ + +static const struct gpio_dt_spec btns[] = { + GPIO_DT_SPEC_GET(DT_ALIAS(sw0), gpios), + GPIO_DT_SPEC_GET(DT_ALIAS(sw1), gpios), + GPIO_DT_SPEC_GET(DT_ALIAS(sw2), gpios), + GPIO_DT_SPEC_GET(DT_ALIAS(sw3), gpios), +}; +/* k_malloc and thread creation are not ISR work: hand off to the workqueue. */ +static struct k_work *const actions[] = { &log_burst, &heap_burst, &stack_hog, &leak_start }; +static struct gpio_callback btn_cb[ARRAY_SIZE(btns)]; + +static void btn_pressed(const struct device *d, struct gpio_callback *cb, uint32_t pins) +{ + ARG_UNUSED(d); + ARG_UNUSED(pins); + k_work_submit(actions[cb - btn_cb]); +} + +void workload_start(void) +{ + for (int i = 0; i < ARRAY_SIZE(btns); i++) { + if (!gpio_is_ready_dt(&btns[i])) { + LOG_ERR("button %d not ready", i + 1); + continue; + } + gpio_pin_configure_dt(&btns[i], GPIO_INPUT); + gpio_pin_interrupt_configure_dt(&btns[i], GPIO_INT_EDGE_TO_ACTIVE); + gpio_init_callback(&btn_cb[i], btn_pressed, BIT(btns[i].pin)); + gpio_add_callback(btns[i].port, &btn_cb[i]); + } + LOG_INF("Buttons: 1 Log Event, 2 Heap Exhaustion, 3 Stack Overflow Risk, 4 Memory Leak"); } diff --git a/app/src/app_workload.h b/app/src/app_workload.h index 1a69d3e..798f132 100644 --- a/app/src/app_workload.h +++ b/app/src/app_workload.h @@ -1,7 +1,8 @@ #ifndef APP_WORKLOAD_H #define APP_WORKLOAD_H -/* Wires Button 2 to a heap burst that triggers a HEAP_LOW incident. */ +/* Wires Buttons 1-4 to Log Event, Heap Exhaustion, Stack Overflow Risk and + * Memory Leak triggers. */ void workload_start(void); #endif /* APP_WORKLOAD_H */ diff --git a/app/src/main.c b/app/src/main.c index 7c6a9a2..74e887b 100644 --- a/app/src/main.c +++ b/app/src/main.c @@ -1,9 +1,8 @@ /* * Exein Runtime Security Monitor on the nRF9151 DK. * - * Runs the monitor (precompiled runtime-zephyr blob, HEAP_LOW policy: the - * only finding the platform accepts today) beside a button-driven demo - * workload (see app_workload.c). The monitor registers with the Exein platform over LTE and + * Runs the monitor (precompiled runtime-zephyr blob: HEAP_LOW, MEM_LEAK, + * STACK_LOW and LOG policies) beside a button-driven demo workload (see app_workload.c). The monitor registers with the Exein platform over LTE and * posts incidents there; TLS runs in the modem, credentials live in the * modem's key store under SEC_TAG. */ diff --git a/runtime-zephyr/zephyr/blobs/cortex-m33-tfm/libruntime_zephyr.a b/runtime-zephyr/zephyr/blobs/cortex-m33-tfm/libruntime_zephyr.a index 1db297e..84ff790 100644 Binary files a/runtime-zephyr/zephyr/blobs/cortex-m33-tfm/libruntime_zephyr.a and b/runtime-zephyr/zephyr/blobs/cortex-m33-tfm/libruntime_zephyr.a differ diff --git a/runtime-zephyr/zephyr/module.yml b/runtime-zephyr/zephyr/module.yml index 182657b..bd923ea 100644 --- a/runtime-zephyr/zephyr/module.yml +++ b/runtime-zephyr/zephyr/module.yml @@ -15,7 +15,7 @@ build: # this entry, with the blob's real path and sha256, into the delivered module. blobs: - path: cortex-m33-tfm/libruntime_zephyr.a - sha256: "3de0328cc089ca1566a57319e4eb9884a26d58a8ecf0432f3f139f36c2883f25" + sha256: "7c62568ebae8f413f9f32daa35d506733b8aabbe0b7ba01388201ca765e397da" type: lib version: "ec8d57d-dirty" license-path: LICENSE