TeslaMate core roadmap: modular data sources, then Rust

[JakobLichterfeld]

What this is

This documents the direction @brianmay and I are taking the TeslaMate core, and collects input on the open design questions. It is documentation and brainstorming — not a vote, and not a call for contributions.

Two things came up in a closed PR (#5557) and in the older go-forward thread (#3416), and both belong somewhere visible:

• Architecture: decoupling data acquisition from data processing, so TeslaMate consumes additional APIs natively and can run streaming-only, without polling.
• Language: moving the core from Elixir to Rust.

We're implementing both. What is genuinely open — and what this thread is for — clusters in three areas: the event contract, the fidelity gap between Fleet Telemetry and old streaming, and what happens to the web UI. The open questions are marked as such below; the rest is decided and documented here so it can be reasoned about.

Ownership — please read before replying

Review capacity is the bottleneck in this project. A half-agreed contract implemented twice consumes it faster than anything else. That is why this section exists, and why it is strict.

This work touches the heart of TeslaMate: the state machine that decides what a drive is, what a charge is, and what ends up in your database. It cannot be assembled from independent pull requests.

• @brianmay and I are doing the implementation. No one else is assigned, and we are not looking to assign anyone.
• Please do not open pull requests against this work. PRs that anticipate the design — the event contract, the source abstraction, the Rust port — will be closed, regardless of their quality. If you believe you have something we should look at, ask here first; do not lead with code.
• This applies to the work described in this thread. Normal contributions elsewhere — docs, dashboards, bug fixes, features outside the core pipeline — are unaffected and welcome as always.
• Design input in this thread is wanted and will shape the result. Input and implementation are not the same thing here.

If you want to help concretely:

• tell us which source you run today (owner API, MyTeslaMate, Teslemetry, self-hosted proxy) and what breaks for you,
• report which Fleet Telemetry fields your vehicle actually delivers, and at what interval,
• weigh in on the open questions below — especially if you have run a streaming-only setup in production.

---

Data sources: modular sources, one core pipeline

To be precise about where we stand today: TeslaMate already works against the Fleet API and Fleet Telemetry — but only by way of a detour. It speaks exactly one wire protocol, the owner-API shape, and alternative backends are reached by pointing TESLA_API_HOST, TOKEN and TESLA_WSS_HOST at a third party (MyTeslaMate, Teslemetry) or a self-hosted proxy that translates them back into that shape.

That detour works, and it will keep working. But it has a price:

• users depend on a translating middleman for something that should be a first-class source,
• the data is capped at whatever the old streaming format can express — new Fleet Telemetry fields cannot surface through an owner-API-shaped bottleneck,
• and TeslaMate still polls vehicle_data even when the source underneath is push-only.

The target shape, which @brianmay sketched in #5557: a core process receives a stream of events, splits them into drives, charges and so on, and persists them. Which source runs is a matter of configuration:

• an owner-API source doing today's polling/streaming, trying not to keep the car awake,
• a Fleet Telemetry source, natively, streaming only, no polling,
• a proxy/third-party source, so current setups keep working unchanged.

To set expectations: sources are an internal structuring device, not a plugin API. Which sources exist, and what the contract between them and the core looks like, is a maintainer decision and will stay one. We're documenting the contract so the design can be reasoned about, not so it can be extended from outside.

Today acquisition and interpretation are entangled in Vehicle, which is why "streaming only" and "additional API" are hard rather than merely new. The polling loop is not a property of the owner API — it's a property of our architecture.

The deliverable of this step is the event contract: what an event is, which guarantees a source must provide, and what the core may assume. The refactor is just how we get there.

Open — input wanted

• Event vocabulary: what is the minimal set? (position/telemetry sample, charge sample, vehicle state change, sleep/wake, …)
• Fidelity: the contract should express what Fleet Telemetry can deliver but old streaming cannot. Where exactly do the two diverge in practice?
• Guarantees: may a source deliver out-of-order or duplicate events, or must sources normalize before handing over?
• Multiple sources per vehicle: worth supporting simultaneously? If so, what is the merge/precedence policy?
• Policy placement: does "don't keep the car awake" belong in the source or in the core?
• Backfill: how should gaps (car offline, source restarted) be reconciled?

---

Language: Rust for the core

The core moves to Rust. This decision is made; this thread is not the place to relitigate the language choice, and replies arguing Elixir vs. Rust in the abstract will not move it. What is genuinely open is listed below — most importantly the web UI question.

The reasons, in the order that matters to us:

• Compile-time guarantees. The bugs that hurt users most are state-machine bugs — a drive that never ends, a charge attributed to the wrong session, activity data surviving a transition it shouldn't. A type system that makes illegal states unrepresentable removes a class of these before any test runs.
• Dependency health. We already run on a community fork (tortoise311) because the original MQTT client died. The fork is maintained and we're fine today — the concern is structural: the pool of people maintaining infrastructure libraries in this ecosystem is small enough that one person losing interest moves us again. This was already @brianmay's concern in #3416.
• Deployment. A single static binary, smaller images, no BEAM runtime to ship.

Elixir/OTP has served this project well, and its supervision model is a good fit for "many vehicles, many long-lived processes". This is not a complaint about Elixir. It's a decision about where the core is easiest to keep correct over the next years.

What it costs

The Phoenix LiveView web UI is the largest single item. Settings, geofences, import and sign-in have no drop-in Rust equivalent. The realistic options are: keep the UI as a separate Elixir service talking to the same database, rewrite it against an HTTP API exposed by the Rust core, or replace it with something else entirely. This is open, and it's the question we'd most like input on.

Beyond that: migration ownership once Ecto is no longer the writer, and CI build times.

What stays stable for you

For everyone running TeslaMate, these hold across all of it:

• your data and your upgrade path — migrations ship with the release and run automatically, as they always have,
• the MQTT topic layout and payloads, our documented integration point and the basis of the Home Assistant and Node-RED setups — the characterization suite in Phase 1 asserts exactly this, so the promise is enforced by tests rather than good intentions,
• the Grafana dashboards we ship — and since the schema underneath them will change, this is an explicit commitment: every schema change ships with the updated dashboards in the same release,
• the amd64 and arm64 Docker images,
• configuration and env-var compatibility, or a documented migration path where something has to change.

What is not a stable interface: the database schema

The PostgreSQL schema is internal, and it is worth being blunt about this. It has never been a documented interface, and this work will not make it one. A restructuring of this size is exactly when the freedom to reshape storage is needed, and we intend to use it: expect substantial schema changes, not cosmetic ones.

Concretely: if you read the tables directly — custom Grafana panels, scripts, third-party or commercial tools built on today's schema — expect your integration to break during this work. Not might: will. There is no compatibility promise, no deprecation window, no advance notice beyond the migrations themselves, and we will not slow down, stage, or route the redesign around anything that depends on the current table shapes. Issues asking us to restore a schema detail or to coordinate changes with external tools will be closed. Your data is safe — migrations carry it forward — but every assumption about how it is stored is up for renegotiation, and nobody outside this repository is part of that negotiation. The supported surfaces are the dashboards we ship and MQTT. Everything else is and always was at your own risk.

If something we haven't listed here would break your setup, say so now, not after the migration.

---

Sequencing

The safety net comes before the surgery, and the architecture change stays decoupled from the language change — a rewrite that also changes the architecture has no reference implementation to test against.

Phase 1 — a characterization suite at the outer boundary. Before any code moves, the current behaviour gets pinned down: a recorded sequence of Tesla API payloads goes in, and the resulting persisted data and MQTT messages are asserted, with the vehicle state machine taken to ~100% coverage — the point of the number is that the suite must reliably catch behavioural deviations, not that a metric looks good. Crucially, the cases are stored as data rather than as code, and nothing in them refers to an internal function.

That boundary already exists — TeslaMate.VehicleCase.start_vehicle/3 drives exactly such a sequence through the API mock today. What's missing is coverage, the move from Elixir literals to data fixtures, and assertions against real persisted rows and messages instead of mock call assertions.

This suite is what makes the following phases safe, and it stays valid across both of them: whatever happens inside, identical API payloads must keep producing identical results.

One caveat we'll handle explicitly: pinning current behaviour also pins current bugs. Cases we believe are wrong get marked as such and fixed in their own commits — never silently, in the middle of a refactor.

Phase 2 — source abstraction in Elixir. Introduce the source/event boundary and make the event contract explicit, incrementally, with the Phase 1 suite green at every step. Delivers native streaming-only support without a rewrite. Event-level fixtures — the ones that can only exist once the contract does — are written here, at the new boundary, alongside the outer suite.

Phase 3 — Rust. Port module by module behind the same contract (strangler pattern). Both fixture sets are the gate: the same files drive both cores and the outputs are diffed. Anything the Rust core cannot reproduce does not replace the Elixir path. How the two runtimes coexist during the port — NIFs, a separate process, or running both cores side by side and diffing offline — is deliberately left open here and will be decided in the Phase 3 issues; the single-binary deployment benefit arrives at the end of this phase, not during it.

Users will not be moved onto the Rust core silently. How the switch is offered — an opt-in image, a beta tag, or both cores running against the same data and diffed before anything ships — will be settled there too, and announced before it happens.

Each phase stands on its own. Phase 1 is worth having whatever happens afterwards, Phase 2 removes the bottleneck regardless of the language, and Phase 3 is gated by both.

---

Where your input goes, and how this gets tracked

The questions marked Open above — the event contract, the fidelity gap between Fleet Telemetry and old streaming, and what happens to the web UI — are the ones where a good answer changes what we build. Corrections to the "what stays stable" list are equally welcome.

There are no dates attached to any of this, and regular development continues throughout: bug fixes, dashboards and features ship as usual, and none of this work blocks a release. Once the design questions have settled, we'll cut the work into issues that link back to this thread — those issues carry scope and acceptance criteria, not deadlines. Hard commitments made in this document — dashboards shipping together with schema changes, MQTT compatibility asserted by the suite — become acceptance criteria in those issues, so they can't silently slip. The issues are the tracking surface; the reasoning lives here. This thread stays open for the design discussion and will not be used as a task list.

What this thread is not: a place to claim implementation work. See "Ownership" above.

cc for visibility: @adriankumpf, @cwanja, @DrMichael, @Dulanic, @swiffer, @tobiasehlert — no action needed, this is a heads-up so you read it here first.

---

🤖 Drafted with Claude Code (Opus 5 high and Fable 5 high) — sponsored by Claude for Open Source

[JakobLichterfeld]

@jlestel, @Bre77 — you operate the largest proxy-based setup we know of, so two of the open questions above are ones you have real-world data on:

• Fidelity: where does the owner-API-shaped streaming format actually cap what Fleet Telemetry delivers to your users? Which fields do you drop or downsample because the old format can't express them?
• Guarantees: what do you see in practice — out-of-order delivery, duplicates, gaps after reconnects?

To be clear on scope: the proxy path keeps working as described above, and nothing here asks you to change anything. We're after your operational experience for the event contract design.

[Bre77]

Teslemetry never touched the Owners API, so i actually have limited knowledge of the "old way". However i understand Fleet Telemetry a lot, and Tesla does have some enforcemented minimums, but so does Teslemetry (cost control). The contract for this is full documented at this living JSON https://api.teslemetry.com/fields.json

I am currently going through a process to document the undocumented enums based on fleet observations.

You may also find https://github.com/Teslemetry/tesla-protocol useful, can easily add rust bindings.

Ill try answer the other questions when i can validate my memory.

[Bre77]

The following is a partially AI generated answer to your questions based on my real-world data.

Q1 (format fidelity): we don't operate an owner-shaped stream at all — our SSE is a native 239-field envelope, so nothing is dropped because an old format can't express it. What we do transform: numeric rounding to catalogued scale to reduce floating point waste (91 fields), invalid values become null for the 10 nullable fields and are omitted otherwise, protobuf type tags and unknown-enum raw integers aren't exposed, and units stay native. Rate policy lives in the vehicle config we send (interval/delta/resend) — 89 of 239 fields default slower than their documented minimum (for cost saving), and users can request lower through our UI. There's no second sampler after ingest.

Q2 (delivery guarantees): measured from a fixed 7-day production window — out-of-order arrivals are real but rare (4 inversions in ~473k publishes, ≈8 per million, seconds-scale lateness); zero exact duplicates across all record types (the only repeated IDs are the deliberate connect/disconnect pairing); reconnect gaps are hard to prove but I do believe the vehicle sends a buffer of untransmitted events when it reconnects (connectivity related, NOT sleep). We do cache all events server side so that clients can get the latest state even if they were not listening.

[JakobLichterfeld]

Thank you — I checked your figures against fields.json and they match exactly: 239 fields, 10 nullable, 89 with a default interval slower than the documented minimum, 91 carrying a scale. That level of precision is rare in this kind of thread, and it changes what we can design against.

What the catalogue settles, and one thing it makes urgent.

Every field defaults to resend_interval_seconds: 86400, and 118 of them additionally suppress on minimum_delta. So an unchanging value is not repeated for up to 24 hours. That means the stream alone cannot bring a consumer to a correct state: a source starting cold, or reconnecting, could be missing current values for a day. Your server-side cache isn't a convenience for us, it's a correctness requirement — any source we build has to perform a state resync on startup and after reconnect rather than just subscribing. We hadn't written that down, and it's now a hard requirement in the contract.

Working assumption, and the question that decides it. Interval defaults spread across 1s (115 fields), 10s (51), 30s (4) and 60s (69), with rate policy living in the vehicle config and no second sampler after ingest. Our working assumption is therefore that partial, field-level updates have to be the primitive in our event contract, with the owner API's atomic snapshot as the special case. But fields.json describes configuration, not the envelope — so this hinges on the first question below, and we're treating it as an assumption rather than a conclusion until you answer it.

Three follow-ups, unchanged in substance:

1. Are events batched into an envelope with one shared timestamp, or does each field carry its own?
2. On reconnect, do buffered events keep their original timestamps, and how far back can that buffer reach? That number sets the grace window for our drive/charge segmentation — the difference between delaying segmentation and making segments revisable.
3. Does the resync path expose the full cached state for a vehicle in one call, and is it scoped the same way as the stream?

Three small things that would help everyone consuming the catalogue, if they're cheap on your side:

• A machine-readable unit key. scale is there, but the unit itself only exists as prose inside description ("in kWh", "in kW"), so it can't be parsed reliably.
• Enum values. 51 of the 239 fields are "type": "enum" with no value list — BMSState is typical: type, description, nothing else. Your fleet-observation work on this is the single most valuable thing in the catalogue for us, and a fifth of the fields depend on it.
• A change signal. fields.json currently serves without ETag, Last-Modified or Cache-Control, and the per-field version is the Tesla firmware version, not the document's. For a living document that leaves full-file diffing as the only way to notice a change.

On semantics that vary by firmware: DoorState documents that two doors were swapped before 2024.44.32. That's a good example of something our contract has to be able to carry — either the source normalizes it, or the event carries enough context for the core to. If you know of others like it, they're worth listing.

On Teslemetry/tesla-protocol: noted, and useful. To be accurate for anyone else reading: it publishes TypeScript and Python bindings today, not Rust — but the definitions are there under Apache-2.0 and Rust can be generated from them with prost/tonic, which is what Phase 3 would do.

One fairness point on the original post. Calling third-party access a "detour" describes how TeslaMate consumes it today, not what you offer. Your stream isn't owner-shaped at all; we simply don't speak it yet. That gap is the thing this work exists to close.

---

🤖 Drafted with Claude Code (Opus 5 high) — sponsored by Claude for Open Source

[magrathean-uk]

Just my two cents guys, chose Go instead of Rust for this solution. As much as prefer rust, Go is better for this case IMHO. Again, I am not telling what, just adding some advice.

[brianmay]

I think we need to rethink our testing strategy.

Currently testing occurs by pushing inputs into a Vehicle and looking at the outputs. And maybe the key problem here is that Vehicle is too complicated. The tests are fragile, changes to Vehicle is likely to break the tests, including tests that are not related to the change. And if the tests break it is difficult to find out what went wrong. Even AI models can struggle and go around in circles (seen this happen with my refactoring PR).

I consider these tests integration tests just because of the large scope of code being tested. We probably do need to have some integration tests. But we shouldn't rely entirely on them. We should ideally have more unit tests then integration tests.

Possible things to think about:

• Split code up into functions that can be easily tested using proper unit tests. i.e. stateless pure functions that always return the same result for the same input, and should be easy to test.

• Test driven design. i.e. write the test for the function first, then update the function until the test succeeds.

Not sure how practical this will be. I often start new projects with good intentions and then get stuck along the way, e.g. thin API wrappers are not good candidates for unit testing IMHO. If we do encounter problems, we should probably discuss them.

[JakobLichterfeld]

Agreed on the destination, and I think your diagnosis is right — but I'd separate two things that are getting conflated, because once they're apart I don't think we actually disagree.

Fragility comes from coupling, not from scope. Today's tests aren't brittle because a lot of code runs; they're brittle because they assert on mock call sequences — LogMock and PubSubMock messages in a particular order. That ties every test to the internal call structure, which is exactly why an unrelated change to Vehicle breaks them and why the failure tells you nothing. A test that drives the same path but asserts only on persisted rows and published MQTT messages has far less to break on. So the Phase 1 suite as described is a fix for the problem you're naming, not more of it — provided we hold the line that nothing in it refers to an internal function.

The pure functions don't exist yet, and extracting them is the refactor. That's the ordering argument: unit tests written against a new decomposition can't tell us the decomposition preserved behavior — they test the new shape against itself. Only an outer net catches "drives now end one sample later than they used to". So characterization first isn't an alternative to the pyramid, it's how we get to the pyramid without breaking people's data on the way.

Where you're right and Phase 1 has no answer: diagnosability. When a golden-master case fails you get "row 47 differs" and no idea which transition caused it. Partial mitigations are small single-purpose fixtures instead of long drives, and a differ that reports the first diverging event index. But the real answer is yours — the unit layer is the diagnostic layer. The two suites do different jobs: the outer net proves behavior was preserved, unit tests tell us where it broke.

Two things I'd change in the plan on the back of this:

1. The outer suite should be as thin as it can be while still being behavior-complete — a net, not the target test strategy. I'd previously written ~100% coverage into Phase 1; on reflection that target belongs to the unit layer after extraction, because reaching it through broad end-to-end cases builds precisely the slow, hard-to-localize suite you're warning about. Phase 1 pins behavior; Phase 2 moves the coverage number inward as functions come out.
2. Phase 2 gets an explicit design goal, not just a structural one: the segmentation logic comes out as stateless functions — same input, same output — and those are written test-first. The event contract is new code, so TDD applies to it cleanly.

On your caveat about thin API wrappers: I think that maps exactly onto the source layer, and it's a reason the split helps. Sources are I/O adapters and deserve contract tests against recorded payloads, not unit tests. The core, once it's pure functions over an event stream, is the part that unit-tests well. Put differently, the source abstraction is what creates the seam your pyramid needs — the two pieces of this work aren't independent.

And yes, let's discuss the moment we get stuck rather than pushing through. The "AI goes in circles" symptom you saw is worth taking seriously as a signal about the suite itself: if a change can't be localized from a failure, that's the test design telling us something, not the tooling.

---

🤖 Drafted with Claude Code (Opus 5 high) — sponsored by Claude for Open Source

[brianmay]

I have had several attempts at writing Tesla owners API clients in rust.

old now, but still might have some good ideas: https://github.com/brianmay/fake_luxary_api/blob/main/fla_client/src/lib.rs

still current: https://github.com/brianmay/robotica-rust/blob/main/robotica-tokio/src/services/tesla/api.rs

I haven't looked at these in a while. If I did I imagine I might want to have words with my past self as to why I did various things :-)

The new version has support for logging metrics using opentelemetry (see init code https://github.com/brianmay/robotica-rust/blob/main/robotica-backend/src/logging.rs). Good metrics/remote logging is probably a good idea. Although I do find keeping it up-to-date and working can be painful (all relevant crates have to be updated at the same time; and recently AI incorrectly told me the TLS configuration wasn't required which broke all logging). And I never remember to look at the logged data.

[JakobLichterfeld]

Thanks — I went through fake_luxary_api, and there's more in it for us than the client code itself.

fla_server is a server that already speaks the owner-API shape over HTTP and handles streaming. What it does is simulate: simulate <id> driving|charging|idle generates plausible state from a mode, rather than replaying a recorded payload sequence. For the characterization suite we need reproducibility rather than plausibility, so the replay part isn't there — but adding a "play back this fixture file" mode to something whose endpoints and streaming plumbing already work is far less work than building that from scratch. I'd treat it as the base rather than write a replay server ourselves.

The bigger find is fla_common/src/types.rs together with the "Type Errors" section of your README. charger_phases declared as String and arriving as integer 1 is exactly the class of knowledge a project pays for twice if nobody writes it down — and you did. For the Rust port that's a head start on the owner-API types, and it's worth mining before we write our own.

It also settles a question about the test strategy: mismatches like that only surface against real recorded payloads. A simulator produces well-formed data by construction, which is why it can't be the safety net — but it's good for exercising paths that are hard to record, like a car that starts charging exactly on a poll boundary. Two different jobs, both useful.

For the two API clients, I'd treat them the same way: not as a starting structure, since the source abstraction makes the architecture different anyway, but as a record of the Tesla-specific behavior you already hit — the auth and refresh flow, how the stream behaves on reconnect, and the places where responses don't match documentation. Worth reading before we write the owner-API source, not after.

One practical note: the repo is GPL-3.0 and TeslaMate is AGPL-3.0. Those combine cleanly under §13 of both, and since you hold the copyright it's your call regardless — but worth settling explicitly before anything moves across rather than afterwards.

On observability, I'd like to turn it into a decision rather than leave it as a good idea, and your last line is the reason: keeping it working is painful and you never look at the data. That's telemetry without a consumer, and it's the default outcome unless the consumer is named up front.

Two things shape what makes sense for us. TeslaMate is self-hosted, so there is no central endpoint and for privacy reasons there shouldn't be one — nothing gets exported anywhere by default. What we can rely on is that every user already runs Grafana, so an opt-in Prometheus endpoint plus a shipped dashboard needs no new infrastructure and has a consumer on day one.

More importantly, the consumer that pays for itself is support. Our recurring cost is bug reports nobody can reproduce — "my drive was split", "this charge is missing". What resolves those isn't a dashboard, it's structured diagnostics a user can attach to an issue: which events arrived, which transitions followed, why a drive was closed. That argues for instrumenting the core's decisions rather than HTTP plumbing, and for output a person can read and paste.

Your maintenance experience argues for one more thing: a thin internal facade for emitting, so a crate version bump doesn't reach into the call sites. If OTel becomes painful we swap the facade, not the core.

So: decision-level diagnostics first because they pay for themselves in triage, opt-in Prometheus second into the Grafana users already have, no export anywhere by default. Small enough to stay maintained, which is the failure mode you're describing.

None of this needs touching before the event contract settles — I'm flagging it as intent, not as work starting.

---

🤖 Drafted with Claude Code (Opus 5 high) — sponsored by Claude for Open Source

[brianmay]

I updated the license of the fake_luxary_api to AGPL 3 o greater. Was purely an oversight. Nobody else has contributed changes or even used this project (to the best of my knowledge), so I think I am safe to make license changes.

[brianmay]

Name? Teslamate V5? Perhaps the first and one of the most important decisions we will need to make...

Need to be something distinct so we don't get confused when referencing old and new code.

[JakobLichterfeld]

Agreed that it matters, and I'd split it into two questions, because I think only one of them is about version numbers.

What users install. Next release is 4.1.0 — that's already what's in the tree. The Rust core landing is a major bump, so 5.0.0. That also lines up with what the discussion says about the schema: since the schema is explicitly not an interface, reshaping it isn't a breaking change by our own definition, so the whole source abstraction can ship across 4.x. The major is reserved for the language and runtime change, not the architecture change.

What we call the code. This is the part a version number handles badly, and I think it's what you're actually after. During a strangler port both cores exist side by side for a long time, and 4.x releases will already contain new-architecture code — so "the V5 code" is ambiguous exactly when we need it not to be, and it stops meaning anything the day 5.0 ships.

What works is naming the component rather than the release: the Rust core gets a crate and directory name and is referred to by it, and the current one is "the Elixir core". Unambiguous in a PR title, in an issue, and in a file path, and still correct in three years. The discussion post already drifts into this without meaning to — "the core", "the Elixir path".

I'd keep that name boring and descriptive rather than a codename. A codename costs every new contributor a lookup, and it eventually stops matching what the thing does.

One caution on reserving 5.0.0 publicly now: the post deliberately carries no dates, and pinning a number to an undated milestone means either 4.x runs for a very long time, or we end up needing 5.0 for something else and the mapping breaks. I'm happy either way — but if we say it out loud, we should be comfortable with 4.x being where everything happens for a while.

---

🤖 Drafted with Claude Code (Opus 5 high) — sponsored by Claude for Open Source

[kdjkdjkdj]

Hi @JakobLichterfeld, @brianmay — greetings from Augsburg, and from near Stuttgart as of tomorrow.

One note upfront: this is written with help from Claude 5, since my English is nowhere near as fluent as my Swabian.

I run a self-hosted TeslaMate whose data comes from Fleet Telemetry streaming: my own receiver, no proxy in between, in production since mid-June. Since you asked for actual field delivery, here is one day of measurements: 24 drives, one Supercharger session, ~15.5k records.

57 of 59 configured fields are delivered. Never seen: EstBatteryRange, ScheduledDepartureTime.

Configured intervals bound the rate, they are not a delivery promise:

field	configured	observed median	observed min
PackCurrent	5 s	5.0 s	1 s
Location	10 s	10.0 s	1 s
Gear	1 s	8.0 s	1 s
OutsideTemp	60 s	82.5 s	12 s
HvacACEnabled	60 s	70.0 s	6 s

Fields that rarely change simply arrive rarely. Conversely almost anything can burst down to ~1 s. A source can promise "not faster than N", not "a sample every N".

Fidelity: the gap to old streaming is categorical, not gradual. Charging is absent from the old format entirely — DCChargingPower, ChargeState, ChargerVoltage, ChargeAmps, ChargeLimitSoc have no equivalent. That is why a streaming-only setup still had to poll during charges until I fed those fields in separately.

Backfill: reconciliation has to be idempotent and keyed on something stable, not on "what happened since last time". After an offline gap the car replays buffered data, and a naive scan creates duplicate sessions. Marking reconstructed ranges explicitly was the only thing that made repeated scans safe.

Policy placement: treating the feed as active based on freshness rather than on the connection being up removed a whole class of bugs — a connected stream delivering nothing is indistinguishable from a live one at socket level.

Happy to post the full 57-field table with observed intervals if that helps the contract.

[JakobLichterfeld]

Split out as #5586, and the reason it's worth flagging here: it is not a Fleet Telemetry problem.

calculate_energy_used/1 assumes every charge row in a process is shaped the same way. When phase detection can't run — the guard is n > 15, so any session with 15 or fewer rows — every row that does carry charger_phases produces NULL and is filtered out of the sum. On my own database, which runs Owner streaming plus vehicle_data and has never seen Fleet Telemetry: 70 of 70 such processes have no charge_energy_used at all, 70 of 686 processes overall.

So the pattern from @kdjkdjkdj's report generalises past the source that surfaced it. His mixed-row case is what made us look; the short-session case was already there, on the path almost everyone runs.

To be precise about what the production data does and doesn't show: it proves the short-session mode deterministically, and it does not independently confirm the mixed-row mode — the used < added cases in my database are spread across categories including unmixed ones, so they have another cause that isn't in scope for that issue.

For the contract discussion this is a second, distinct class from #5583. That one is about building a record from a single message instead of from known state. This one is about an aggregate over rows that silently assumes they're homogeneous — and it survives any change to how records are written, because it runs afterwards. Worth remembering when we get to the segmentation logic: a projection that fills gaps correctly still leaves aggregates that discard rows they can't interpret.

---

🤖 Drafted with Claude Code (Opus 5 high) — sponsored by Claude for Open Source

[Bre77]

Side note: "A session does not begin with a complete state" is a problem Teslemetry solves, we send a full copy of cached telemetry on new SSE connections.

[kdjkdjkdj]

is_resend … we know it doesn't mark snapshots; we don't yet know it reliably marks a replayed buffer. Since it's the primitive the backfill requirement leans on, that's the one thing left to confirm if a reconnect ever lands in a capture window.

It landed. The vehicle does buffer and replay — and is_resend stays false throughout.

Before the numbers, the honest framing: this basis is thin and I built it specifically to answer your question, not from accumulated operation. It was 34 °C here today and a couple of the runs are sloppy for that reason — one I had to repeat because the car was still preconditioning and would not suspend. Read it as one clean observation, not as a survey.

Waiting for a natural disconnect didn't work

My first idea was a tunnel: a real radio outage while telemetry is flowing, no intervention needed. I drove through the Kappelbergtunnel twice. No disconnect at all — 1,228 records over that hour, event-time gaps of at most 5 s, no arrival gap over 30 s, arrival−event offset never above 2 s. It has good cellular coverage, so it is not a usable test.

The two gaps already sitting in my capture were no better. Both (40 min and 895 min) are the car going to sleep — no telemetry was generated during them, so there was nothing to buffer and is_resend: false afterwards says nothing at all. That is precisely why your question was still open.

So I staged one

With the car awake, parked at home and streaming at a ~3 s cadence, I took the receiver down for exactly 180 seconds:

13:50:01Z   docker stop  fleet-telemetry
13:53:01Z   docker start fleet-telemetry
13:53:08Z   vehicle reconnects (socket_connected)

stop/start, not a recreate — same container, so the capture survives. I did this while parked, not while driving: my own feed-failure monitor watches drive windows and would have taken a staged outage for a real one and re-pushed the config.

Result

burst at arrival 13:53:08Z	57 records at once
event times within it	13:50:01 → 13:53:08, monotonically increasing
records whose event time falls in the outage window	55 of 55 delivered — nothing lost
arrival − event	up to 187 s, then immediately back to 0
IsResend	false on all 57
snapshot in the burst	none

Raw records (VIN removed). The car was parked, so only PackVoltage/PackCurrent were still changing.

// first three of the burst — arrival identical, event time three minutes old
{"arrival":"2026-08-04T13:53:08Z","data":{"CreatedAt":"2026-08-04T13:50:01Z","IsResend":false,"PackVoltage":373.2799916565418}}
{"arrival":"2026-08-04T13:53:08Z","data":{"CreatedAt":"2026-08-04T13:50:02Z","IsResend":false,"PackCurrent":-0.7000000104308128}}
{"arrival":"2026-08-04T13:53:08Z","data":{"CreatedAt":"2026-08-04T13:50:07Z","IsResend":false,"PackCurrent":-0.7000000104308128}}

// last of the burst — the backlog has caught up with the present
{"arrival":"2026-08-04T13:53:08Z","data":{"CreatedAt":"2026-08-04T13:53:08Z","IsResend":false,"PackCurrent":-0.7000000104308128}}

// live again, offset back to zero
{"arrival":"2026-08-04T13:53:11Z","data":{"CreatedAt":"2026-08-04T13:53:11Z","IsResend":false,"PackVoltage":372.85999166592956}}
{"arrival":"2026-08-04T13:53:13Z","data":{"CreatedAt":"2026-08-04T13:53:13Z","IsResend":false,"PackCurrent":-0.7000000104308128}}

What follows

1. The flag cannot carry the backfill requirement. Fleet Telemetry recovers the gap perfectly — complete, ordered, with correct created_at. But nothing on the record says it is history. A consumer holding record n has no way to tell replay from live from the record.

2. Replay is detectable only by comparing created_at against your own arrival clock — which means the contract has to require that the consumer keeps an arrival clock, and the one-second resolution of created_at limits how sharp that comparison can get.

3. On the MQTT path it is not detectable at all, because created_at is discarded there. My TeslaMate received three minutes of stale state as "now" and could not know. Parked, that was harmless. Mid-drive the same burst would inject three minutes of stale positions as the current location. That makes it the fourth thing that path structurally cannot carry, after created_at, is_resend and event atomicity — and unlike the others it is not degraded data, it is silently wrong data.

4. The vehicle picks a recovery strategy per situation and does not declare which:

reconnect after …	behaviour	observations
vehicle sleep	full 57-field snapshot within 0–22 s, no replay	3 of 3
receiver outage while awake	complete replay, no snapshot	1 of 1

Both are sensible choices. But "did I just receive the present or the past?" is answerable only by inspecting the payload shape, and only in hindsight. If you are declaring source capabilities anyway, this looks like one worth declaring explicitly: does this source replay after a gap, or resynchronise?

What this cannot show

• One staged outage of 180 s, car parked. I don't know the buffer depth, and I don't know whether a longer gap, or one during driving, behaves the same. If the buffer has a limit, my test was well inside it.
• The snapshot-versus-replay contrast rests on 3 + 1 observations.
• I briefly thought snapshots aligned with state transitions — three fell within seconds of a drive boundary. The afternoon's driving refuted it: two of five snapshots sat mid-drive. Mentioning it so nobody builds on a pattern that isn't there.

Still owed from my side: the per-field table on both clocks. That needs a day with real driving in the capture window; the capture is still running.

---

If I have made a reasoning error anywhere in how I gathered or read this, I would rather hear it than not. And if there is other data that would help — different fields, a longer staged outage, a gap during driving, or anything else that only someone running their own receiver can produce — tell me what to record and I will run it.

---

🤖 Drafted with Claude Code (Opus 5)

[JakobLichterfeld]

@Bre77 — that closes the initial-state question for one source and turns it into a capability difference rather than a universal problem. A provider that ships cached telemetry on every SSE connect means the consumer never needs a warm-up on that path, and a consumer talking directly to fleet-telemetry always does. That's exactly the kind of thing the source declares rather than the core assumes.

[JakobLichterfeld]

@kdjkdjkdj — staging the outage was the right call after the tunnel didn't cooperate, and the result settles the question. Three things I'd add, one of which may change what the flag means.

TeslaMate already implements the comparison you're describing. vehicle.ex:660 computes stale_stream_data? against the last full response and discards frames that lose the comparison, logging Received stale stream data. So the arrival-versus-event check isn't something we need to invent for the contract — it exists, and it only covers the one source that carries a timestamp. That is now the second mechanism we've found in this shape, after merge/3: the right behavior is implemented once, bound to the Owner path, and absent everywhere else. It's becoming the argument for the source abstraction rather than an example of it.

Your point 3 is the one with teeth. Three minutes of stale positions arriving as current location isn't degraded data, it's wrong data, and drive segmentation would act on it. Parked it's harmless; mid-drive it would produce a fabricated position history. Worth stating plainly in the capability list: a source that cannot expose event time cannot be trusted with segmentation, full stop.

On is_resend, I think we may both have had the wrong model of it. Teslemetry's field catalogue gives every field a resend_interval_seconds, defaulting to 86400 — an unchanged value is re-sent after 24 hours. If is_resend marks that — "this is a periodic repeat of a value that hasn't changed" — rather than "this is replayed history, then false throughout your burst is not a gap in the protocol, it's the flag doing its actual job. The two concepts got conflated, most likely by me.

That's testable with the capture you already have running, at no extra cost: over a multi-day window, does IsResend: true ever appear, and if so, on which fields and at roughly what spacing? A hit at ~24-hour intervals on slow-moving fields would confirm the reading. If the flag never appears at all in a week of data, that's informative too — it would mean nothing sets it and it's useless either way.

One methodological caveat you didn't list. docker stop closes the socket cleanly. A radio blackout doesn't — the vehicle sees a dead connection rather than a graceful close, and may well choose a different recovery. Your result establishes what happens after a clean disconnect; whether an unclean one also replays is still open. Not a criticism of the test, just the boundary of what it proves.

On your recovery-strategy table: with @Bre77's note there are now three observed behaviors, not two — replay after an outage, snapshot after sleep, and provider-side cached state on connect. So the capability isn't a yes/no. Something closer to: on reconnect, does this source replay the gap, resynchronize current state, or neither? A source that does neither forces the consumer to warm up, which is where you started.

What would help next, in the order I'd value it:

1. IsResend: true anywhere in a multi-day window — settles what the flag means, costs you nothing.
2. A staged outage during driving, if you're willing to disable your feed monitor for it. That's where the wrong-data case actually bites, and it would show whether segmentation sees a coherent replay or an incoherent one.
3. Buffer depth: a longer outage — twenty minutes rather than three — to find out whether the replay is bounded and what happens when it is.

And noted on the snapshot-versus-state-transition pattern being refuted by your own afternoon. Reporting a hypothesis you disproved is more useful than most confirmations.

---

🤖 Comment drafted with Claude Code (Opus 5 high) — sponsored by Claude for Open Source

[swiffer]

Thanks for putting this together so clearly — the combination of modular sources + explicit event contract, sequenced properly before the Rust port, feels like the right long-term direction. The emphasis on characterization first and keeping the architecture change decoupled from the language change is especially appreciated; that kind of discipline is rare and valuable on a project that holds people’s historical driving/charging data.

I’d like to offer concrete help on the database side and the official Grafana dashboards once the shape of the schema changes becomes clearer. Happy to review migration plans, help keep the shipped dashboards in sync, or assist with any performance-related adjustments that surface.

A few design questions that would help me (and probably others who maintain custom panels or care about long-term reporting) understand the intended scope:

1. Are the expected schema changes primarily driven by the input-side restructuring (cleaner event model, better support for partial/field-level Fleet Telemetry updates, multi-source merging, backfill semantics, etc.), or is there also an intention to reshape storage in ways that enable additional or improved Grafana views / new analytical perspectives on the same data?

2. Is the rewrite mainly focused on project maintainability, correctness of the state machine, and removing architectural bottlenecks, or is “enabling richer views on the gathered data” also an explicit goal?

3. Consequently: is restructuring / extending the official dashboards around any new schema capabilities considered in-scope for this work, or is the commitment limited to “keep the existing dashboards working after every migration”?

4. On the storage design itself — is long-term reporting performance (query patterns over multi-year position/charge/drive history, especially on modest hardware) being taken into account while the schema is open for change, or is that considered a separate concern that can be addressed later?

No pressure on answers; just trying to understand where the boundary sits so any help I offer stays aligned with the actual goals.

Happy to stay quiet on the implementation side and only chime in on design / dashboard / migration topics as they surface.

[brianmay]

I can't speak on behalf of @JakobLichterfeld so what I say may not be correct.

I imagine in the process of doing a rewrite we will come across aspects, e.g. in the database design - that we consider to be broken. And ask why on earth did we ever do it that way. Allowing schema changes, when required, would allow us to fix the problem, rather then be forced to carry around legacy baggage.

I would like to at least try to make any schema changes to the Elixir code too. So that they can be both use the same database. But we won't know if that is practical or not until we get more solid detail.

[JakobLichterfeld]

@swiffer — thank you, and yes to all three areas you offered. Really love your dashboard and DB magic.

Let me answer the four questions directly, because the boundary you're asking about sits in two different places.

1. What drives the schema changes. Primarily the input side: a field-level event model, partial updates, multi-source merging, backfill semantics. What I do want is for the resulting shape not to foreclose better views — but enabling them is not what drives the cut.

2. Is "richer views" an explicit goal? Not of this work — it's the step after it. The goals here are maintainability, correctness of the state machine, and removing the single-wire-protocol bottleneck. Designing storage around views nobody has specified yet would couple two decisions and leave neither validatable, which is the same mistake as coupling the architecture change to the language change. The order is: get the model right, then design what can now be shown. That second step is real and intended, not a polite deferral — and it is where your input weighs most.

3. Dashboards: a standing commitment, not a one-off. Expect the schema to change repeatedly — not once during this work and then settle, but as an ongoing part of development, whenever the model turns out to be wrong. The commitment is per-change and permanent: the shipped dashboards keep working, and every schema change ships with its dashboard updates in the same release. That is what keeping them in sync means, and it does not expire.

What is not committed is restructuring the dashboards around new capabilities — that belongs to the step in answer 2, and it's the part I'd rather not decide alone.

4. Long-term reporting performance. This is never "too late" — what changes is the price. Every access pattern we get wrong and correct later is another migration, and every migration costs each user a maintenance window. Multi-year position tables on modest hardware are unforgiving, and they are exactly the ones you don't want to rewrite twice.

So the ask isn't "hurry before we lock it", it's "the earlier it lands, the fewer migrations it costs the people running this". And it's knowledge I don't have: which queries hurt today, on what hardware, over what history depth, and which access patterns the current shape fights against. You and the people maintaining custom panels are the ones who can answer that.

---

@brianmay — agreed, and I'd go further. Making the schema changes in the Elixir code too isn't only about not carrying legacy baggage forward; it's a precondition for the test strategy. Phase 3 gates the Rust core by driving both implementations from the same fixtures and diffing the outputs. If the two ever diverge on schema, that comparison stops being possible and the port loses its safety net. A shared database isn't a nicety during the transition — it's what makes a strangler port verifiable at all.

Which also answers where the schema work happens: in Elixir, now, incrementally, with the same migration discipline we already have. Not as a big-bang reshape at the language boundary.

---

Where I'd start: the charging area

Since you asked where the boundary sits, here is a concrete first candidate. It's first because the debt is on record rather than a matter of taste — #5586 exists because the code assumed something the schema permits it to violate.

• charges is a wide mixed table. AC columns (charger_phases, charger_voltage, charger_actual_current) sit in the same row shape as DC columns (fast_charger_*). Nothing expresses that a session is one or the other, and nothing prevents a process from containing both kinds of row. The homogeneity assumption that charge_energy_used drops charge rows when phase detection fails: under-reported or NULL on short and mixed sessions #5586 turned on is enforced nowhere.

• Phase semantics live in a bare multiplier. The √3 voltage correction in determine_phases/1 returns 1.732 as a "phase count". It is applied and then discarded — the stored charger_phases keeps whatever the vehicle reported, and the correction survives only inside the resulting energy figure. You cannot look at a stored charge_energy_used and tell which multiplier produced it. That's physics hidden in a number.

• charger_power is a smallint. It was created as a float in 2019 and narrowed in 20200410112005_database_efficiency_improvements to save space. That trade deserves a deliberate second look while the schema is open: the resulting quantisation is systematic, and it lands precisely on the fallback branch that a charging session uses when phase detection fails. That's a re-cut question, not a bugfix question.

• Energy exists only as a process aggregate. There is no per-interval record, so an energy figure is not reconstructable or auditable after the fact. Every failure mode in charge_energy_used drops charge rows when phase detection fails: under-reported or NULL on short and mixed sessions #5586 would have been visible in the data rather than only in the code if the integration were recorded per interval.

The fix now going in #5592 is a small preview of the seam. It touches the Elixir integration, a Grafana panel that had replicated the same logic, and a database constraint — because a stored phase count of 0 was read one way by is_nil(charger_phases) in Elixir and another way by charger_phases >= 1 in the dashboard, so the same row produced two different answers. That divergence is the kind of thing a re-cut should make structurally impossible rather than something we keep in sync by hand. It is also a worked example of what @brianmay described: the schema change lands in Elixir, with the dashboards migrated in the same release.

---

🤖 Review drafted with Claude Code (Opus 5 high) — sponsored by Claude for Open Source