feat(laqn): restore live ERG REST data source as LAQN-ERG#9
Open
ruaraidhd wants to merge 1 commit into
Open
Conversation
The default LAQN source fetches from the openair RData feed (londonair.org.uk/r_data/) — fast for bulk/backfill but regenerated only ~daily, so its newest rows lag real time by a day or more. That makes it unusable for live polling: a consumer requesting a short trailing window gets nothing recent. A downstream live dashboard (Argus) silently lost its current LAQN feed when the default switched to RData in d2a0535. This restores the ERG REST hourly-data path as a separate, non-primary LAQN-ERG source, mirroring the AURN-SOS-live / AURN-RData-backfill split. The default LAQN source is unchanged (still RData) so bulk/backfill keeps its speed; callers needing current data opt into LAQN-ERG. Metadata is shared (ERG JSON API). 13 new tests cover the fetcher and registration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The default
LAQNsource fetches hourly data from the openair RData feed (londonair.org.uk/r_data/<SITE>_<YEAR>.RData). That's the right choice for bulk/backfill — it's ~600× faster than the old ERG REST path, which is whyd2a0535switched to it. But the RData files are regenerated only ~daily and their newest rows lag real time by a day or more.That makes the default
LAQNsource unusable for live polling: a consumer requesting a short trailing window (e.g. "last 2 hours") gets nothing recent back. The switch to RData silently removed LAQN's live path, because the source has no separate notion of "latest" vs "archive" — there's onefetch_data, and it became the lagged one.This bit a downstream live dashboard (Argus), whose 15-minute poller calls
download("LAQN", …, last=<short window>)and, since the switch, has been getting only day-old data — so LAQN dropped out of the live view entirely. The upstream ERG API is healthy and current (hourly data within a few hours of real time); it just isn't wired to a data fetcher anymore (the default source uses it for metadata only).Fix
Restore the ERG REST hourly-data path (recovered from
d2a0535^) as a separate, non-primaryLAQN-ERGsource, mirroring the existingAURN-SOS-live /AURN-RData-backfill split:LAQN(unchanged) — openair RData. Stays primary; bulk/backfill keep their speed.LAQN-ERG(new,primary: False) — ERG RESTData/Siteendpoint, current to within a few hours. Callers that need live data opt in explicitly.Metadata is shared between both (the ERG JSON API).
LAQN-ERGalso exposesfetch_latest(last-4h max per site/measurand), matching the SOS sources.Tests
13 new tests in
tests/test_laqn.py:source_network=LAQN,FINE→PM2.5, CO inmg/m3/ othersug/m3, UTC timestamps, empty-@Valuerows dropped, unmapped species dropped, hits the ERGData/Siteendpoint, site codes upper-cased, no-data warning.LAQN-ERGregistered as non-primary network source; defaultLAQNuntouched.Verification
Ran the deterministic blast-radius locally — 248 passed, 0 failed:
test_conformance,test_registry,test_packaging,test_laqn,test_lmam,test_find_sites,test_api,test_sos,test_networks_portals_api(i.e. everything that exercises source registration, enumeration, conformance, and the registry reload-list cycles). Theslow/integration(live-network, API-key) tests weren't run locally — that's for CI.Follow-ups (not in this PR)
LAQN-ERG.aeolus_source = "LAQN-ERG"for live LAQN polling once a release carrying this ships (keepingLAQN/RData for backfill).latest:/archive:backends and aFETCH_LATESTcapability make "a network's live path" a first-class, testable thing — so this class of silent regression becomes a failing test at change time.🤖 Generated with Claude Code