Three sections:
- §1–§7: Architecture
- §8–§22: MQTT contract
- §23–§24: Boot + Seed
Diagrams: see readme.md.
Why. SCADA shops already speak MQTT. AsyncAPI describes the topics. Picking a non-MQTT bus would mean re-teaching every integrator we work with.
All telemetry flows over MQTT (HiveMQ CE). AsyncAPI documents the topics.
Why. A gateway adds a service we'd have to deploy, secure, observe, and recover. Service URLs are known per-deployment; the SPA can call them directly. Per-service auth is repetitive but cheap.
Frontend SPA calls backend services directly. Each service owns its own auth, CORS, rate limiting.
Why. Commissioning a device or swapping a connection should not require redeploying the gateway and HMI. They re-fetch from device-api and pick up the change. The two endpoints separate two concerns: /asyncapi is the messaging contract (wire shape), /topology is deployment state (what exists).
| Consumer | Endpoint(s) | What it gets |
|---|---|---|
ems-industrial-gateway, dlr-operating-envelope |
GET /asyncapi |
Channels + schemas + x-protocol-source (per-device connection + binding for the gateway to dial) + x-enum-values |
ems-hmi |
GET /asyncapi + GET /topology/view + GET /topology/sld.svg |
Messaging contract; sanitized topology projection (per §22); generated SLD SVG (regenerated on every topology change) |
platform-api, commissioning tooling |
GET /topology |
Full DTM including gateway-only fields (connection.host/port/unit_id, per-measurement binding) |
No static config files distributed. Exponential backoff on startup.
Why. Rust where memory safety on edge hardware matters. Python where the ML ecosystem lives. TypeScript where the web ecosystem lives. Standardizing on one would force at least one team to fight their toolchain.
- Rust — protocol handling (
industrial-fixtures,industrial-gateway,dlr-pst-sim) - Python — ML / agents /
dlr-operating-envelope - TypeScript —
device-api,hmi
Why. Operators run the stack themselves. Kubernetes is too much to learn for a single-site EMS. CFN covers AWS-connected sites; ISO covers air-gapped defense forward. Same image, different wrapper.
| Path | Trigger | Mechanism |
|---|---|---|
| Cloud CFN | aws_partition ∈ {standard, govcloud} |
Per-order CloudFormation YAML download from delivery portal; operator runs aws cloudformation create-stack locally |
| Air-gapped ISO | aws_partition == none or deployment_context == defense_forward |
Self-contained image with DTM, SSH key, ems_mode baked in |
No Kubernetes. No IaC on operator side. platform-api composes the per-order CFN template on demand and uploads to S3 as orders/{id}/ems-stack.yaml. Each YAML bakes deployment_uuid, dtm_url, ems_mode directly.
Why. Hand-drawing the SVG per deployment rots the moment topology changes. Asking HMI to lay out the diagram from buses[] data forces HMI to own electrical-drawing logic (orthogonal routing, symbol placement, label collision) that already exists in edp-api's drawing pipeline. Splitting the responsibilities cleanly: edp-api owns layout, device-api owns serving + runtime regeneration, HMI owns animation + binding to live data.
Electrical bus topology is encoded in a buses[] block in the DTM. edp-api computes buses[] from the sizing payload, emits it alongside devices, and also emits a runtime SVG artifact with every device's device_id bound to a stable SVG element ID. IEC 61850 alignment: buses[].id = ConnectivityNode, buses[].type = VoltageLevel domain, buses[].members[] = Terminal references, port = Terminal name. Bus bars are not devices — no device_id, no MQTT topics.
edp-api is internal infrastructure; customer browsers never call it. The SVG is served to HMI from GET /topology/sld.svg on ems-device-api — same origin as /topology/view and /asyncapi. On every POST /topology or §21 mutation, device-api regenerates the SVG from edp-api (or its cached copy) alongside the spec and view projection. HMI fetches /topology/sld.svg at boot and again on system/topology_changed, stores the SVG string in a React Context, and overlays live MQTT measurements onto the SVG element IDs. Particle flow along energized buses animates via SVG <animateMotion>; particle direction is driven by the sign of kW on the bus's source-side device. Reduced-motion replaces particles with a single static arrowhead at the bus midpoint.
| Commercial | Defense / sovereign | |
|---|---|---|
| Document + Vector | Aurora | Aurora |
| Time series | Tiger Cloud (URL param) | Aurora pg_partman |
| Graph | Neo4j Aura (URL param) | Neptune |
| FTS | Aura native | AOSS |
| Customer params | 2 connection URLs (Tiger, Aura) | zero |
| Idle cost | Aurora $0 + Tiger $0-30 + Aura $65 = $65-95/mo | $480/mo |
| Customer ops | sign up at Tiger + Aura, paste 2 URLs | nothing |
Why. Defense and sovereign deployments physically can't depend on non-AWS SaaS — FedRAMP authorization, supply-chain review, and GovCloud-region constraints rule out Tiger Cloud and Neo4j Aura. Commercial deployments face none of those constraints and benefit from cheaper, feature-richer managed vendors. The variant is gated by aws_partition / deployment_context from §5: govcloud partition or defense_forward context → defense column; everything else → commercial column. App code is identical across variants — abstract TimeseriesClient and GraphClient shims swap implementations at boot from the cfg.yml-injected connection details. Aurora, S3, and the ISO path (§5) are unaffected. |
Why. Everything a device emits vs everything sent to a device. Collapsing "measurement / status / state / calculation / alarm" into one family removes the gray zone — they're all just things the device emits.
measurements/— everything a device emits (sensor reads, computed values, health, alarms, command acks)commands/— everything sent to a device
system/ carries broker-internal events (topology changed, etc.) — not a family. Boolean/enum readings are measurements with unit=none. Derived quantities (DLR dynamic rating) are measurements.
Why. Wildcards are predictable when every topic of a kind has the same number of slots. The parser switches on one slot. Broker ACL hooks match on slot positions without payload introspection.
sites/{site_id}/devices/{device_id}/measurements/{measurement}/{unit} # 6 segments
sites/{site_id}/devices/{device_id}/commands/{verb}/{target}/{unit} # 7 segments
system/{event_type} # 2 segment
Parser switches on slot 4 (family ∈ {measurements, commands}). Unitless measurements use literal none in the unit slot. Verb enum: [set, reset, clear, start, stop, enable, disable].
Why. Operator at 3am should be able to read the unit without looking it up. Enum-lock at codegen time prevents degC vs celsius divergence — build fails if anyone tries.
volts | amps | watts | vars | voltamperes | watt_hours | hertz |
celsius | percent | watts_per_m2 | meters_per_second |
bar | liters_per_minute | none
Lowercase, plural where natural (volts not volt), short compounds (watts_per_m2 not watts_per_meter_squared), vars for reactive power. No SI-prefixed duplicates (watt_hours only). Adding a unit is a spec change → patch-level bump → system/topology_changed → consumers re-fetch.
Why. One place to put the scaling math. Three consumers downstream stay simple. The spec records the math for regulatory audit.
Industrial protocols carry raw integers with vendor scaling. Gateway applies scale and offset; MQTT carries the engineering value. Scaling metadata lives in AsyncAPI under x-source per channel:
x-source:
protocol: modbus_tcp
server_unit_id: 1
register_type: holding
address: 3000
data_type: int16
word_order: high_low
scale: 0.1
offset: 0Gateway code is codegenned from the spec.
Why. Quality information is rare and recoverable. Don't pay wire bytes for it on every sample. Industrial historians that need it run the time-join once in an adapter.
Every sample is {ts, value}. No quality, no source flags, no correlation IDs. Quality lives in a separate status measurement, joined by timestamp at query time (LATERAL join of status ≤ sample.ts). ts is RFC3339/ISO8601 string with Z suffix — ingests natively into Postgres TIMESTAMPTZ.
Why. Four shapes cover every channel. Float for sensors, boolean for two-state things, enum for labeled states, trigger for commands with no payload.
FloatSample { ts, value: number }
BooleanSample { ts, value: boolean }
EnumSample { ts, value: string } # channel schema sets value.enum
TriggerSample { ts } # commands only — reset, clear, fire-and-forget
EnumSample channel sets value.enum from the template values: block (e.g. ["AUTO", "MANUAL", "RUNPQ"]). Gateway translates raw register integers to the label before publish; HMI renders the label it receives. No int-to-string logic in any consumer.
Verb → schema:
| Verb | Schema |
|---|---|
| set | Float / Boolean / Enum |
| start, stop, enable, disable | Boolean (persistent state) |
| reset, clear | Trigger |
Why. Without templates, every deployment re-fights the naming bikeshed and downstream codegen has nothing concrete to target. Templates are PR-gated so the argument happens once at template-design time, then the rest of the system codegens against a known shape.
Why opinionated, not vendor-agnostic. ARCNODE owns the inside of every container we ship and the integration of every BESS we support. Vendor-agnostic templates would force codegen and HMI to generalize against "any device might look like anything" — the wrong abstraction for a product that owns the inside of its containers. Opinionated templates let codegen targets be specific and HMI views be designed against known shapes.
Measurements, commands, units, and protocol bindings live in device templates (e.g. bess_module, dlr_sensor). The DTM references templates by bare slug; it does not redefine measurements per deployment. Templates are PR-gated; canonical home is edp-api/device_templates/. Git history is the version source.
Module types. Three roots:
compute_module— arcnode-fab (GPU servers, NVLink, DLC pumps, plate heat exchangers)grid_module— arcnode-fab (AC switchgear, transformer, PCS, metering relays)bess_module— BYO (Tesla Megapack/Megablock, CATL EnerOne)
For arcnode-fab containers, templates encode hardware design end-to-end. For BESS, templates encode the supported vendor's protocol surface and register map, authored by the ARCNODE team.
Hierarchy. Each template declares a contains: block listing direct child templates (required and scalable). Containment chains arbitrarily; depth is unbounded. The DTM expresses this as a parent-chain tree: every device has an optional parent: device_id pointer. Topic addressing stays flat — device_id in the topic is the leaf, not a path; depth lives in the parent chain, walked by HMI/analyst codegen at render time. IEC 61850 alignment carried as optional metadata (iec_61850.logical_nodes, iec_61850_ref per measurement).
Example tree (BESS-shaped; identical pattern for compute and grid):
bess_module_1 template: bess_module, parent: null
├── bess_rack_1 template: bess_rack, parent: bess_module_1
│ ├── bess_bms_1 template: bess_bms, parent: bess_rack_1
│ ├── bess_inverter_1 template: bess_inverter, parent: bess_rack_1
│ └── bess_cell_1..N template: bess_cell, parent: bess_rack_1
├── bess_rack_2
└── ...
Each level declares its own measurements/commands. Module-level rollups (whole-module SOC), mid-level rollups (per-rack spread), and leaf reads (per-cell voltage) are independent channels, all flat in the topic, related only by parent links in the DTM.
IEC 61850 reference per measurement. Every measurement in every template carries an iec_61850_ref (e.g. MMXU.W for active power, ZBAT.BatChaSt for battery state of charge, XCBR.Pos.stVal for breaker position). This is the standards-canonical taxonomy for the measurement's semantic role and is the basis on which HMI resolves which measurement populates which SLD field, which Reading carries AC vs DC voltage labeling (Rule 3.7), etc. iec_61850_ref is required on every measurement; PR-gated at template-authoring time.
Bounds and thresholds per measurement. Every numeric measurement carries bounds: {min, max, nominal} (the physically valid range) and thresholds: {warn_min, warn_max, alarm_min, alarm_max} (the deployment-defaults for alarm derivation and chart MIN/MAX threshold lines). Bounds and thresholds are required for type=float; omitted for type=bool / type=enum. Per-deployment threshold overrides flow through DTM extra_measurements: per §14 "Per-deployment customization."
Distribution. edp-api embeds a templates_used: Record<templateRef, DeviceTemplate> map in every DTM payload, containing exactly the templates referenced by devices. Downstream consumers receive everything they need in one POST.
Per-deployment customization. Per-device extra_measurements: escape hatch in the DTM. Use sparingly.
User-defined template authoring is out of scope — new templates land via PR to edp-api/device_templates/. Device-level instantiation (using an existing template at a new device_id) is in scope — see §21.
Why. The customer wants to call it "Pile A DC Bus". The code wants voltage_dc. A display name override in the DTM keeps both right without breaking topic stability.
Topic, code, and spec use the canonical name (e.g. voltage_dc). The DTM carries a display_name override per site, device, and measurement, propagated to AsyncAPI as x-display-name and rendered by the HMI. Renaming a label is a metadata-only change → patch bump → re-fetch but no resubscribe. Resolution order: DTM override → template default → humanized canonical.
Why. Operator readability — UUIDs make mosquitto_sub -t '#' unreadable. Immutability — renaming an ID would invalidate every subscriber.
^[a-z][a-z0-9_]{0,62}[a-z0-9]$. Display names are mutable; identifiers are not. Uniqueness: site_id unique within deployment, device_id unique within site, measurement unique within template.
Why. Consumers always need to re-fetch the spec to be safe. A diff carried in the event would lie or go stale. The version in the event is an audit breadcrumb, not a computation input.
AsyncAPI spec uses semver. Patch = metadata fix; minor = additive (new channel, new device); major = breaking (rename, remove, unit change). system/topology_changed carries only {ts, version} — consumers re-fetch the full spec and diff client-side. Consumers always re-fetch unconditionally.
Why. Retain on commands is the dangerous default — explicitly off so a setpoint replay does not unexpectedly spin a BESS.
| Family | QoS | Retain | Why |
|---|---|---|---|
| measurements | 0 | true | High-rate telemetry; new subscriber sees latest immediately |
| commands | 1 | false | Imperative-at-time; no replay on broker restart |
| system | 1 | false | Transient triggers; no fire-on-new-connect |
Why. Device codegen runs on the most constrained hardware (Rust on ESP32 / Pi). Writing the spec from the device's viewpoint makes their code path literal. HMI and analyst invert during their own codegen — they have CPU to spare.
action: send = device publishes; action: receive = device subscribes. Topology-update events appear as action: receive since devices subscribe to them. Non-device consumers (hmi, analyst-server, platform-api) invert during their own codegen. family (slot 4) is orthogonal to action: action distinguishes publisher from subscriber, family distinguishes data direction. MQTT bindings (QoS, retain) are inferred from family per §18; per-channel bindings.mqtt overrides only for exceptions.
Why. Broker is per-deployment private (per §5). Blast radius is one site. Roles, mTLS, and JWT are real work; do them when the threat model demands.
Single MQTT username/password from template-secrets.env. Topic ACLs open. Roles, mTLS, JWT deferred.
Why. Operators commission, repair, and replace equipment in production. Forcing every change through a redeploy is operational friction without a safety benefit — the spec-regen and topology_changed mechanism already handles consumer reconciliation correctly. Template authoring stays PR-gated because adding a new measurement vocabulary item is an engineering change; template instantiation is a deployment-state change.
Adding, removing, or updating a device — instantiating an existing template at a new device_id, swapping a connection, retitling a display_name, deleting a device — is a device-api operation, not a redeployment. Each successful mutation:
- Validates referential integrity (template ref, parent acyclic,
bus.members[]consistency) - Persists a new versioned topology row (audit reconstruction)
- Regenerates the AsyncAPI spec
- Bumps semver per §17
- Publishes
system/topology_changed { ts, version }exactly once
POST /topology (full DTM replacement) is the same regen-and-broadcast applied wholesale. Transactional CRUD groups multiple changes into one broadcast.
Scope:
| Operation | Allowed dynamically | Notes |
|---|---|---|
| Add device using existing template | yes | Must reference a template in templates_used or bundled set |
| Remove device | yes | Cascades to children when parent; refuses if referenced by bus.members[].device_id until bus is updated |
Update display_name |
yes | Patch bump |
Update connection (host/port/unit_id) |
yes | Patch bump |
Update parent (re-parent) |
yes | Validates parent chain acyclic |
Add/remove extra_measurements on a device |
yes | Minor bump (additive) |
| Introduce a new template at runtime | no | PR-gated (§14) |
| Modify a template | no | PR-gated (§14) |
ems-hmi is read-only against /asyncapi + /topology/view — not a CRUD client and not a /topology consumer. /topology is invoked by platform-api (bulk delivery), commissioning workflows, and integrator/operator tooling; gateways consume /asyncapi (x-protocol-source carries everything they need).
Why. HMI needs deployment-state information (device hierarchy, bus topology, per-measurement metadata) but must never see gateway-only infrastructure (Modbus IPs, register addresses, function codes). AsyncAPI describes messages, not deployment state — stuffing topology into x-* extensions there confuses two abstractions. A dedicated projection endpoint keeps each contract clean.
GET /topology/view returns a sanitized projection of the DTM with the same shape conventions as /topology but with gateway-only fields removed and per-template measurement metadata inlined where HMI needs it.
| Block | Included | Stripped |
|---|---|---|
deployment_uuid, ems_mode, sizing_params |
✓ | — |
devices |
device_id, display_name, parent, template, blocking |
connection.host/port/unit_id, extra_measurements[*].binding, extra_measurements[*].publisher |
buses[] |
bus_id, type, members[] |
— |
templates_used |
per-measurement: unit, type, display_name_default, iec_61850_ref, bounds, thresholds, poll_rate_hz, values (for enums) |
binding, publisher |
HMI consumes this view to (a) walk parent chains for the module browser and rollups, (b) drive routing (/devices/:device_id resolves through template dispatch), (c) populate per-measurement chart thresholds (Rule 3.6) and Reading tone derivation, (d) sim the random-walk inside bounds in demo mode, and (e) resolve IEC 61850 refs → SLD field via the static IEC_TO_SLD_FIELD table.
/topology/view, /topology, and /topology/sld.svg (per §6) are three projections of the same persisted DTM. All three regenerate together on POST /topology and on §21 CRUD mutations; all three bump version per §17.
Why. dtm.json is config, not data: small (<100 KB), immutable per deployment, frozen at delivery. It belongs in the delivery bundle alongside docker-compose.yaml and the env files, not behind a runtime fetch. Treating it as a mounted file removes a runtime dependency on object storage at device-api boot, drops the minio container from the air-gapped variant, and removes LocalStack from the dev inner loop.
Day-1 boot uses one mechanism: read a JSON file at the path set via the BOOT_DTM_PATH env var. Same code path across cloud, ISO, dev, CI — only how the file lands at that path varies. Env-var contract matches the DOCUMENT_URL pattern: defaults to unset/skip, deployments set it explicitly.
Behavior:
BOOT_DTM_PATH |
Topology table | Action |
|---|---|---|
| set | empty | read + parse + validate + seed |
| set | populated | read + skip seed (don't overwrite operator changes) |
| unset | empty | log + skip (graceful empty start) |
| unset | populated | log + skip |
Read happens unconditionally when path is set — surfaces file-side issues at boot rather than at the next restart.
Failure modes (all fatal when BOOT_DTM_PATH is set):
| Failure | Detection |
|---|---|
| Env var set + file missing/unreadable | fs |
| Invalid JSON | JSON parser |
Fails Zod Dtm validation |
Zod |
templates_used slug unknown to bundled catalog |
Same check as POST /topology |
| DB write fails | Fatal with restart-loop (Docker restart policy) |
Env var unset = graceful empty start. POST /topology and the §21 CRUD endpoints are the canonical mutation surfaces.
Delivery per context (same code, different stager):
| Context | How BOOT_DTM_PATH is set + file lands |
|---|---|
| Cloud (CFN + EC2 + docker-compose) | Compose service block sets BOOT_DTM_PATH=/app/dtm.json and bind-mounts /opt/arcnode/dtm.json (staged by UserData via curl https://arcnode-public/orders/<id>/dtm.json) read-only |
| On-prem ISO | Same compose contract; ISO bake step writes the file to /opt/arcnode/dtm.json |
| Dev (docker-compose) | Same compose contract with a fixture file at dev-fixtures/dtm.json; or omit the env var to skip |
| CI / smoke | Env var unset → skip; tests POST DTMs via §21 endpoints |
Idempotency: device-api never overwrites a populated topology table from a stale read. To re-seed, clear the table (e.g., a migration step before redeploy) or use §21 CRUD endpoints.
Operator mutation: POST /topology and the §21 CRUD endpoints write directly to the topology document store. Re-mounting a new dtm.json and restarting device-api does not overwrite operator state — boot-read is gated on empty-table.
Out of scope: runtime fetch from object storage; bundled default DTM inside the image; env-var inline JSON; Parameter Store / Secrets Manager.
Why. EC2 already has every persistence secret in its environment at compose-up time. An init container that exits 0 fits docker-compose's lifecycle — compose blocks dependent services until it returns. Platform-api stays pure (provisioning + handing over secrets); the EMS team owns seed logic. No new Lambda, no new IAM perms.
Seeded slices (one time-series variant per deployment):
| Slice | Source | Tool |
|---|---|---|
| Vector | Public S3 corpus | curl + psql against Aurora pgvector |
| Graph | Public S3 corpus | curl + cypher-shell (Aura) or Neptune loader API (defense) |
| Time series | Public S3 SQL dump | curl + psql against Tiger Cloud or Aurora pg_partman |
Not seeded by §24:
- Document — DTM is loaded by
ems-device-apiboot per §23. - AOSS index — auto-populated by Graphiti during the graph seed step (defense variant only).
Idempotency: init container checks if the target slice is non-empty before seeding. Re-running docker compose up after a healthy seed is a no-op (ms exit). Clearing a slice (manual operator action) gets re-seeded on the next compose-up.
Trust boundary: seed scripts live in the EMS repo and execute inside the customer's EC2 instance.
Cloud: chat + code: bedrock.anthropic.claude-sonnet-4-6 (commercial) embedding: bedrock.amazon.titan-embed-text-v2 (commercial, 1024d)
Airgapped (24 GB VRAM tier): chat: ollama.gemma4:26b (Apache 2.0, 26B-A4B MoE) code: ollama.qwen3-coder:30b (Apache 2.0) embedding: ollama.qwen3-embedding:4b (Apache 2.0, 1024d via Matryoshka)
Why gemma4:26b for chat. The RAG path feeds long context into the chat model. On the 2×12 GB VRAM tier gemma4:26b runs 3–7× faster than qwen3.6:35b and degrades only 32% from 32k→128k context (qwen3.6 degrades 65%, collapsing to ~9 tok/s). Qwen3.6's one real edge is agentic coding — moot here, since qwen3-coder:30b owns the code slot. Chat does not need to be the coding champion.
- Two pre-built vector dumps (airgapped and cloud); one schema; one seed pipeline.
EmbeddingProviderabstraction added to seed and MCP server.- 1024-dim cutover requires re-seeding existing corpus.
- Larger airgapped tiers (H100+) get separate model picks in a future ADR if/when a customer needs them.
- Cohere Embed v4 on Bedrock: 5× more expensive than Titan, no quality win for our corpus.
- Bedrock Marketplace Qwen3-Embedding for single DB: ~$400–1500/mo always-on instance hours; cheaper to ship two artifacts.
- Llama 3.3 70B airgapped: doesn't fit 24 GB VRAM tier; no upside over Qwen3.6 at sizes that do fit.
- Qwen3.6-35B-A3B as airgapped chat: 3–7× slower than gemma4:26b on the 2×12 GB tier and degrades 65% at 128k context; coding advantage is irrelevant for the chat slot.
Status: Accepted Date: 2026-05-15
Analyst Agent needs market prices, weather, and energy news as tool inputs. Banner mock used Permutable AI, OpenWeather, Yes Energy. Permutable and Yes Energy are sales-gated enterprise contracts unsuited to demo-stage and ARCNODE-procured economics.
Three external sources, all free, no sales cycle:
get_lmp→ gridstatus.io (direct ISO endpoints)get_weather→ OpenWeather free tierget_energy_news→ feedparser over curated RSS list:- Reuters Energy, Bloomberg Energy, OilPrice.com, S&P Global Commodity Insights all publish feeds. Aggregate and done.
A market-data-poller sidecar pulls on a schedule (LMP 5 min, weather 15 min, news 10 min) and writes to TimescaleDB alongside site telemetry. Agent tools query the store, not the upstream APIs.
- Agent chat loop has no synchronous outbound HTTP.
- Upstream outages decoupled from agent availability.
- Rate limits scoped to one poller, not per agent message.
- Air-gapped deployments disable the poller. agent code unchanged.
- Mockup and website update. code or markdown find and replace. nothign implmnented yet.
- readme.md — topology, sequence, deployment diagrams
- AsyncAPI 3.0
- Sparkplug B — SCADA vocabulary reference
- edp-api
manifest_client.py— S3 fetch pattern