Adopt the eeprom.json schema: firmware port, Sevalla hosting, schema-driven settings UI, MAVLink-ready backend

[dakejahl]

Summary

Planning issue for adopting the version-aware EEPROM schema (am32-firmware/AM32#295) across ARK32: port the schema + tooling to ARK-Electronics/ARK32, host the ARK32 eeprom.json from our firmware releases, rebuild the configurator settings UI generatively from the schema, and put a backend seam in place so the device transport can later swap between USB MSP passthrough and MAVLink (USB/UDP). Findings come from a deep audit of all three codebases (this repo, the ARK32 firmware fork, and the QGC implementation in mavlink/qgroundcontrol#14013). Direction is decided — see Decisions at the bottom.

Approach: incremental refactor, not a rebuild. The overhaul (#3) already produced a solid, tested protocol core (packages/am32-core + web/node/sim/cli packages, ~20k LOC, 519 tests, layer boundaries enforced by tsconfig/ESLint/CI gates). The only remaining hand-built area is the settings UI (~1,900 LOC with all display metadata as template attributes in one file) — exactly the part the schema replaces. Rebuilding from scratch would discard 20k LOC of proven code to replace 2k LOC of UI. Meanwhile the firmware-side port is nearly free (layouts are byte-identical), and QGC's implementation turns out to be only partially schema-driven — this configurator would be the first consumer to use the schema's groups for fully generative UI construction.

Findings

1. Firmware port (ARK-Electronics/ARK32) — nearly free

• The EEPROM struct layout on ark-release is byte-for-byte identical to the eeprom_schema branch: 55 fields, same offsets, uint8_t buffer[192], EEPROM_VERSION 3 on both. validate_layout() passes against ark-release and eeprom.json needs zero field edits. The entire eeprom.h diff is formatting (tabs vs spaces, pointer style, comments).
• Real work items: generate_eeprom.py emits 4-space/uint8_t* style while make check_format (clang-format, PX4 style) wants tabs/uint8_t *, so the byte-exact sync check fails on style alone; the workflows filter on main (ours is ark-release); the publish workflow's tag glob v[0-9]+.[0-9]+ won't match v3.0-ark-style tags; $id/meta/SCHEMA_URL need the ARK rebrand.
• The upstream publish workflow's upload transport is deliberately stubbed (pending the am32.ca deploy mechanism) — we don't need it at all: our schema ships as a release asset instead (see phase 0/1).
• Release pipeline (ci: publish release and nightly hex assets for the ARK32 configurator ARK32#69): CICD_build.yml publishes AM32_<TARGET>_<MAJOR.MINOR[-tag]>.hex to GitHub Releases on tags plus a rolling nightly prerelease; no manifest/index exists and no external upload — the configurator resolves assets by filename convention. The schema rides this exact channel as one more release asset.
• Bonus unification targets: scripts/build_factory_image.py, hwci/hwci/settings.py (EEPROM_FIELDS), and the DroneCAN param table (Src/DroneCAN/DroneCAN.c) are three more hand-maintained mirrors of the layout that could later be generated from or validated against eeprom.json.
• DroneCAN param get/set and firmware update over CAN exist in firmware source but are compiled only for the _CAN-suffixed SITL target today; CAN hardware targets live on unmerged branches (relevant to Validate the configurator against CAN ESCs (ARK 12S CAN ESC) #9).
• Gap to track: eeprom.json declares eepromVersion 4 (CAN block, offsets 176–191) but firmware EEPROM_VERSION is still 3 on every branch — the v4 bump needs coordinating with upstream when the CAN ESC ships.

2. The schema itself (am32-firmware/AM32#295)

• Inc/eeprom.json (924 lines): eepromVersions (layout revisions 2/3/4 with sizes), 55 fields (offset/size/type, display name, description, unit, raw vs display scaling with factor/offset/decimals, enum values with descriptions, disabledValue sentinels, readOnly/hidden), 10 ordered groups with group-level version gating, and meta. Version-awareness is two composable mechanisms: availability gates (minEepromVersion/minFirmwareVersion) and versions reinterpretation overrides keyed "eeprom:N+" / "firmware:X.Y+" (e.g. timingAdvance changes range+scaling entirely at eeprom v3).
• generate_eeprom.py keeps hand-edited eeprom.h as the source of truth: validates the layout (gapless, non-overlapping, in-bounds, contiguous nested structs) and byte-compares a regenerated header (--check, wired into CI and make eeprom-check); --write regenerates the header from the schema.
• Upstream hosting is already live: https://am32.ca/eeprom serves the schema today — via the MinIO-backed server/routes/eeprom.ts mechanism this codebase inherited. We are not using that path (see Decisions).

3. QGC implementation (mavlink/qgroundcontrol#14013) — reference only (no QGC work in this plan)

• The QGC UI is not schema-generated. Sections and control types are hand-coded QML; the schema supplies ranges/units/decimals/enum options/sentinels and the availability predicate. The schema's groups are parsed but never consumed and description is never rendered. A generative configurator UI goes further: groups + type + metadata construct the whole page with zero per-field source code.
• The portable design is now precisely documented for the TS port: version resolution (eeprom version = data[1], fw version = data[3]/data[4] — read from the EEPROM blob itself; pick the single highest applicable override per axis, eeprom then firmware, non-cumulative, re-resolve on every read), conversion math (display = raw*factor + offset; raw = clamp(round(inverse)) — round, don't truncate), dirty tracking compared in raw byte space, majority-match as a per-setting histogram with an exemption list (directionReversed).
• QGC bugs/hazards to fix rather than replicate in our port: display bounds are derived before version overrides are applied (pwmFrequency renders 0–255 instead of 8–48/8–144); multi-byte types silently degrade to uint8; majority ties crown the lowest raw value; edits fan out to every selected ESC, so touching directionReversed arms a broadcast write that would clobber alternating CW/CCW props; QGC fetches am32.ca/eeprom.json while the publish target is am32.ca/eeprom.
• MAVLink transport: ESC_EEPROM (id 292, development dialect). Read = MAV_CMD_REQUEST_MESSAGE(param1=292, param2=escIndex|255) followed by an unsolicited stream (no correlation/timeouts). Write = fire-and-forget full 48-byte image plus a write_mask of dirty bytes only, header bytes 0–4 always masked off, GCS source IDs required (mavlink-router drops frames otherwise). The PX4 side exists (feat(dshot): Extended Telemetry and EEPROM support PX4/PX4-Autopilot#26263) but is DShot-wire only and caps at 48 bytes / uint32[2] mask, so the v4 CAN block is unreachable end-to-end today, and CAN ESCs would additionally need an FC-side DroneCAN bridge.

4. This repo — state after the overhaul (#3)

• The protocol layer is in good shape and should be kept: 5-method Transport seam with three proven implementations (am32-web, am32-node, am32-sim), a single construction site in useEscSession, typed errors, injectable clock, property tests on the codec, and an FC/ESC simulator the CLI smoke-tests against. Boundaries are enforced (core tsconfig has no DOM types; ESLint forbids the app from importing below the session layer).
• The one remaining hand-built area is the settings UI (~1,930 LOC across 4 files): all display metadata lives as template attributes in pages/configurator.vue (ranges, factors, offsets, the one enum, switch/radio arrays, 9 inline disable predicates, isInEEpromVersion checks, semver literals v2.16–v2.19) — exactly the data the schema provides. SettingField.vue is already a generic type-discriminated renderer and largely survives. The wire layer (eeprom/layout.ts + codec) is version-gated and name-keyed, so schema-driven rendering needs no protocol change.
• Bug found during the audit: SettingFieldGroup reads switch/radio values from firstValidEscData while SettingField reads selectedEscInfo[0], and writes go to the selection — deselect ESC fix(serial): harden ESC enumeration timeouts and RX drain #1 and checkboxes/radios display fix(serial): harden ESC enumeration timeouts and RX drain #1's values while writing to the others. Every inline :disabled predicate shares the asymmetry. This dies with the rewrite.
• Firmware policy is hiding in components: the 2.19 TIMING_ADVANCE auto-migration and the blank-EEPROM heuristic in SerialDevice.vue belong in am32-core, where the CLI and simulator get them too.
• Inherited upstream cruft worth pruning (~2,700+ LOC, independent of everything else): the admin/sponsors surface (pages/admin/** — login, sponsor CRUD, user management — plus /api/admin/*, /api/sponsors, image uploads, cookie-session auth, and the Prisma Sponsor/Session/User models that are the database's only use), the KISS-ULTRA/unlocker/am32-tools catalog wiring, netlify.toml + unused @netlify/* deps, and dead files (src/settings.ts is 0 bytes, run.ts mirror script, server/api/download.ts, modals/FlashHex.vue). Also: the ci.yml push trigger targets master, so push-CI never runs on ark-release.
• Extra input for the field-by-field audit: AM32_CONFIGURATOR_COMPARISON.md / AM32_REVIEW_FOLLOWUPS.md (untracked notes in the local QGC checkout) — a tri-source comparison of all 40 settings across firmware, this configurator, and QGC.

Plan

Keep the platform (Nuxt SPA + Nitro + am32-core), rebuild only the settings page generatively, and stage MAVLink behind a backend interface. Phases are independently shippable:

Phase 0 — port the schema to ARK32 firmware (S, ~½–1 day)

• Cherry-pick the three eeprom_schema commits onto ark-release.
• Resolve the style collision: pipe the generated header through clang-format before comparing (keeps --check meaningful under our formatting), or exempt Inc/eeprom.h from check_format.
• Retarget workflows: branch filters main → ark-release; drop the stubbed am32.ca upload step entirely.
• Rebrand $id/meta; point SCHEMA_URL (used by --url checks) at our hosted URL.
• Attach eeprom.json as a release asset alongside the hex files in CICD_build.yml — on tagged releases and the rolling nightly prerelease (nightly firmware may carry schema changes ahead of a release). Zero secrets, rides ci: publish release and nightly hex assets for the ARK32 configurator ARK32#69's pipeline.

Phase 1 — host the ARK32 schema (S, ~½–1 day)

• server/routes/eeprom.ts: replace the MinIO lookup — fetch the eeprom.json asset from the latest ARK-Electronics/ARK32 GitHub release, cache server-side with a short TTL (same pattern as server/utils/github-files.ts), and serve it at <configurator-domain>/eeprom with sensible cache headers. No MinIO dependency.
• Optional ?channel=nightly to serve the nightly prerelease's schema for users running nightly firmware.
• Vendor the current schema in-repo as the offline/PWA fallback (also the dev-mode source).

Phase 2 — schema-driven settings UI (M–L, ~1½–2 weeks)

• New schema module in am32-core (pure, tested): parse + ajv-validate, availability gates, version-override resolution, raw↔display conversion, dirty-in-raw-space, majority-match — porting the QGC algorithms with the bugs listed above fixed. Derive the runtime wire layout from the schema so a new field needs no app release; validate fetched schemas with the same layout invariants as generate_eeprom.py (gapless/non-overlapping/in-bounds) and cross-check against the vendored copy (new fields OK, moved offsets suspicious) so a bad hosted schema can't cause wrong-offset writes. Read-back verification on write stays as the last line of defense.
• Generative renderer: groups → sections, type + metadata → widget (slider/combo/toggle/melody), description → help text, disabledValue → sentinel-toggle pattern; pages/configurator.vue metadata deleted; SettingField.vue becomes schema-fed.
• ARK32-only compatibility gate: on connect, verify the ESC is running ARK32 firmware (target-name allowlist from the .file_name region the app already reads for asset resolution, plus a firmware-major sanity check) and block non-ARK firmware with a clear message pointing at ARK32 releases. Cuts the supported-firmware matrix to what we actually test and steers users onto ARK firmware.
• Schema extensions needed for full parity — propose upstream as feat(eeprom): add version-aware eeprom.json schema synced with eeprom.h am32-firmware/AM32#295 follow-ups so the format doesn't fork: declarative dependency predicates (disabledWhen/visibleWhen; 9 inline predicates here, 3 in QGC), a widget hint (RTTTL melody editor), optional unit-format string, and a story for defaults (the schema currently has no default key at all; we carry per-product defaults in factory/ARK_4IN1_F051_eeprom_defaults.json).
• Fix the firstValidEscData/selectedEscInfo[0] inconsistency; move the 2.19 migration and blank-EEPROM heuristic into am32-core.
• Fetch the hosted schema with a TTL cache (IndexedDB) + bundled fallback + ajv validation.

Phase 3 — cleanup track (S, ~1 day, can run anytime)

• Delete the admin/sponsors surface: pages/admin/**, api/admin/**, api/sponsors.ts, the uploads route, cookie-session auth utils, and the landing-page sponsor carousel.
• With that gone the database has no remaining users — remove the Prisma stack entirely (schema, migrations, generated client, server/utils/database.ts, the prisma migrate deploy step in scripts/start.mjs, and the DATABASE_URL requirement on Sevalla).
• Remove netlify leftovers (netlify.toml, @netlify/* deps), dead files (src/settings.ts, run.ts + src/fetch-and-upload-releases.ts, server/api/download.ts, modals/FlashHex.vue), and the KISS-ULTRA/unlocker/am32-tools catalog wiring.
• Fix the ci.yml push trigger (master → ark-release).
• Stretch: with the schema and firmware both GitHub-backed, MinIO has no remaining callers — drop useMinio/the storage plugin/Redis presigned-URL caches, and move per-board default EEPROM images to release assets (the factory pipeline already builds *.eeprom.bin).

Phase 4 — MAVLink backend (M–L, postponed to a later session; scoped here for the full picture)

• Extract an EscBackend interface from Am32Session's public surface; useEscSession.connect takes a DeviceDescriptor (MSP-serial, MAVLink-serial, MAVLink-UDP-via-bridge) instead of a SerialPort. The schema UI then binds only to backend + schema, making the transport swappable.
• MavlinkEscBackend speaking ESC_EEPROM (id 292) per the portable spec in findings §3. USB MAVLink works in-browser via Web Serial. Settings-only initially: ESC firmware update over MAVLink (MAVFTP) is blocked on FC-side support, and CAN ESCs additionally need an FC-side DroneCAN esc_eeprom bridge (PX4 work, out of scope here). This backend is what makes the QGC feature unnecessary if that PR never lands.
• UDP path — local bridge, since browsers can open neither UDP sockets nor local processes:
  • Distribution: a bridge subcommand on the existing ark32 CLI, riding release-cli.yml's per-OS binaries (Ubuntu/Windows/macOS) on GitHub Releases.
  • Install UX: the web app probes the bridge's loopback WebSocket; on failure it shows an install prompt with per-OS download links to the release assets.
  • Programmatic start/stop: the installer registers a custom URL protocol (e.g. ark32://bridge), which is the only browser-permitted way to launch a local process on demand; after launch the page controls it over the loopback WebSocket (start/stop/status of MAVLink sessions, clean shutdown, idle auto-exit). Security: bind loopback only, validate Origin against our domain, pairing token on first connect so arbitrary sites can't drive it.

Decisions (2026-08-01)

• UI: replace only the settings UI with schema-driven rendering; keep the platform and am32-core. No app rebuild.
• Hosting: serve /eeprom from the latest ARK32 GitHub release asset with caching — no MinIO dependency (likely unconfigured in prod anyway; GitHub release is the source of truth regardless).
• Schema strategy: keep the EEPROM layout matched with upstream AM32 to avoid gotchas, but the configurator always pulls our ARK32-hosted schema, and compatibility is restricted to ARK32 firmware (gate on connect) to reduce maintenance burden and steer users onto ARK firmware.
• Prune: sponsors/admin surface goes, including the Prisma/DB/auth stack it drags in.
• QGC PR: reference material only — no QGC work is part of this plan.
• MAVLink: postponed to a separate effort; phase 4 above is the scoped plan. Local UDP bridge is acceptable, distributed via GitHub release assets (Ubuntu/Windows), launchable/controllable from the web app via protocol handler + loopback WebSocket.
• Remaining minor item: split phase 0 into its own issue on ARK-Electronics/ARK32 when that work starts.

[dakejahl]

UI against the simulator

Follow-up from discussion: long-term we also want the web configurator UI to drive a simulated FC + ESCs (not only ark32 --sim / the test harness). Ideal DX would be something like ./run.sh --sim or a Connect → Simulator path, so protocol/UI work does not require hardware.

How it fits this issue

Phase 4 already defines the right seam: extract EscBackend from Am32Session’s public surface and have useEscSession.connect take a DeviceDescriptor instead of a bare SerialPort. Simulator should be one more descriptor on that same surface, not a special-case stack in the Nuxt app.

Roughly:

• Serial MSP — today’s path (WebSerialTransport / node-serial + current session)
• Serial MAVLink / UDP MAVLink — MavlinkEscBackend (UDP in-browser via the local bridge)
• Simulator — same MSP-passthrough backend (or a thin sim-backed backend) over SimTransport / createSimHarness, optionally with fault knobs for repro

Transport stays “bytes only.” MSP vs MAVLink are backends; sim is primarily a pipe + FC/ESC model under the MSP backend (what am32-sim already is for CLI/tests). The UI and CLI both talk only to EscBackend, so Connect can offer Simulator next to USB serial and MAVLink without forking settings/flash UX.

Why not ./run.sh --sim alone today

am32-sim is deliberately excluded from the client bundle (no Nuxt alias; ESLint blocks app imports). That stays correct until there is an explicit product path:

1. After EscBackend extract — e.g. dev-only or gated descriptor that constructs the MSP backend with a sim transport (may require a careful, non-shipped bundle path for am32-sim), or
2. Via the same local bridge as UDP MAVLink — browser talks WebSocket to ark32 bridge, bridge runs the sim harness in Node. Keeps sim out of the browser bundle entirely and reuses bridge install/security work.

Either way belongs with Phase 4 (or a small follow-on once the backend interface exists), not as an ad-hoc flag on run.sh before the connect surface is descriptor-based.

Suggested acceptance sketch (when we take it)

• UI can connect without Web Serial / hardware and exercise enumerate, settings read/write, and (where the backend allows) flash against sim
• Same session events/log surface as real hardware so the configurator pages do not branch on “isSim”
• Optional: expose a subset of --fault / --escs / --fc for debugging from the Connect UI or query params
• CLI ark32 --sim remains the fast headless gate; UI-sim is for integration of schema UI + multi-backend connect, not a replacement for assert-cli-sim.sh