Skip to content

feat(laqn): restore live ERG REST data source as LAQN-ERG#9

Open
ruaraidhd wants to merge 1 commit into
mainfrom
fix/laqn-erg-live-adapter
Open

feat(laqn): restore live ERG REST data source as LAQN-ERG#9
ruaraidhd wants to merge 1 commit into
mainfrom
fix/laqn-erg-live-adapter

Conversation

@ruaraidhd
Copy link
Copy Markdown
Contributor

Problem

The default LAQN source fetches hourly data from the openair RData feed (londonair.org.uk/r_data/<SITE>_<YEAR>.RData). That's the right choice for bulk/backfill — it's ~600× faster than the old ERG REST path, which is why d2a0535 switched to it. But the RData files are regenerated only ~daily and their newest rows lag real time by a day or more.

That makes the default LAQN source unusable for live polling: a consumer requesting a short trailing window (e.g. "last 2 hours") gets nothing recent back. The switch to RData silently removed LAQN's live path, because the source has no separate notion of "latest" vs "archive" — there's one fetch_data, and it became the lagged one.

This bit a downstream live dashboard (Argus), whose 15-minute poller calls download("LAQN", …, last=<short window>) and, since the switch, has been getting only day-old data — so LAQN dropped out of the live view entirely. The upstream ERG API is healthy and current (hourly data within a few hours of real time); it just isn't wired to a data fetcher anymore (the default source uses it for metadata only).

Fix

Restore the ERG REST hourly-data path (recovered from d2a0535^) as a separate, non-primary LAQN-ERG source, mirroring the existing AURN-SOS-live / AURN-RData-backfill split:

  • LAQN (unchanged) — openair RData. Stays primary; bulk/backfill keep their speed.
  • LAQN-ERG (new, primary: False) — ERG REST Data/Site endpoint, current to within a few hours. Callers that need live data opt in explicitly.

Metadata is shared between both (the ERG JSON API). LAQN-ERG also exposes fetch_latest (last-4h max per site/measurand), matching the SOS sources.

Tests

13 new tests in tests/test_laqn.py:

  • Fetcher — standard schema, source_network=LAQN, FINE→PM2.5, CO in mg/m3 / others ug/m3, UTC timestamps, empty-@Value rows dropped, unmapped species dropped, hits the ERG Data/Site endpoint, site codes upper-cased, no-data warning.
  • RegistrationLAQN-ERG registered as non-primary network source; default LAQN untouched.

Verification

Ran the deterministic blast-radius locally — 248 passed, 0 failed: test_conformance, test_registry, test_packaging, test_laqn, test_lmam, test_find_sites, test_api, test_sos, test_networks_portals_api (i.e. everything that exercises source registration, enumeration, conformance, and the registry reload-list cycles). The slow/integration (live-network, API-key) tests weren't run locally — that's for CI.

Follow-ups (not in this PR)

  • Version bump + changelog entry at release time.
  • Docs source-listing refresh to mention LAQN-ERG.
  • Downstream: Argus sets aeolus_source = "LAQN-ERG" for live LAQN polling once a release carrying this ships (keeping LAQN/RData for backfill).
  • This is a manual pre-figuring of the v0.5 network/backend split, where latest:/archive: backends and a FETCH_LATEST capability make "a network's live path" a first-class, testable thing — so this class of silent regression becomes a failing test at change time.

🤖 Generated with Claude Code

The default LAQN source fetches from the openair RData feed (londonair.org.uk/r_data/) — fast for bulk/backfill but regenerated only ~daily, so its newest rows lag real time by a day or more. That makes it unusable for live polling: a consumer requesting a short trailing window gets nothing recent. A downstream live dashboard (Argus) silently lost its current LAQN feed when the default switched to RData in d2a0535.

This restores the ERG REST hourly-data path as a separate, non-primary LAQN-ERG source, mirroring the AURN-SOS-live / AURN-RData-backfill split. The default LAQN source is unchanged (still RData) so bulk/backfill keeps its speed; callers needing current data opt into LAQN-ERG. Metadata is shared (ERG JSON API). 13 new tests cover the fetcher and registration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant