v2.0.4: scheduler retry upgrade, local storage flag, Uzbek translations - #83
Open
Victor1Ja wants to merge 31 commits into
Open
v2.0.4: scheduler retry upgrade, local storage flag, Uzbek translations#83Victor1Ja wants to merge 31 commits into
Victor1Ja wants to merge 31 commits into
Conversation
indexed-db.service.spec.ts imported LocalStorageService from a file that no longer exists; invalidlocation.page.spec.ts imported SchoolnotfoundPage from invalidlocation.page, which exports InvalidLocationPage. Both broke compilation of the whole karma suite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…re type Replaces the 3-retries-every-15-min cap (~45 min of coverage in a 4-hour window) with a policy that retries until the test completes or the window ends, paced by why the attempt failed: - No network: retry every minute. These retries generate no test traffic, and getNetInfo() now gets a navigator.onLine pre-check so offline ticks don't hit the IP-info services either. - Test failed with network up: exponential backoff min(60s * 1.2^n, 10 min), since each attempt consumes real bandwidth. The semaphore now persists retryAttempts (total tries in the slot), backoffLevel (the backoff exponent, reset when the network comes back after a no-network failure) and lastFailReason. getNetInfo() exceptions - which previously escaped decide() and skipped rescheduling entirely - are now caught and treated as no-network. Plan: project-memory/plans/0005-scheduler-retry-upgrade-v2.0.4.md (giga repo) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measurements that fail the realtime upload already get queued in IndexedDB and re-sent by the periodic sync, but on the backend they were indistinguishable from realtime uploads. The payload now carries: - upload_failed: false on realtime uploads; the copy queued in IndexedDB is saved with true, so the sync delivers it flagged without any sync changes. - scheduled_slot: 'A' | 'B' | 'C' for slot tests, 'startup' for the daily launch test, null for manual runs. - scheduled_at: the originally planned run time (the semaphore keeps it separately from choice, which moves with every retry). The sync payload also stops leaking IndexedDB bookkeeping fields (id, status, createdAt); the old code stripped isSynced, which measurement records never had. Plan: project-memory/plans/0006-local-test-storage-flag-v2.0.4.md (giga repo) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The app bundled a modified copy of @m-lab/ndt7 0.0.6 (2022) under
src/assets/js/ndt/. The copy differed from upstream in two ways: it added
client_name=giga-meter to the metadata sent to M-Lab, and it removed the
Node-only polyfills (require('ws') etc.) that broke webpack builds.
Upstream 0.1.5 makes both edits unnecessary: the polyfills are gone and
config.metadata is the official way to send client_name. So this switches
to the package:
- @m-lab/ndt7 bumped 0.0.6 -> ^0.1.5 (it was already a dependency, unused).
- measurement-client imports the package and passes
metadata: { client_name: 'giga-meter', client_version: app_version }.
- The worker files ship from node_modules via an angular.json assets glob;
runtime paths are unchanged.
- Vendored ndt7.js and both workers deleted (NDT5 legacy files left alone).
- src/types/ndt7.d.ts declares the untyped package.
Error propagation is structurally identical between the copy and 0.1.5
(same throw on locate fetch failure, same 'Could not understand response'
string), so the locate-error retry classification keeps working; specs
cover the config passed to ndt7.test and the retry classification.
Also brings 0.1.5's improved timeouts: 10s to connect + 12s of test after
connecting, instead of 12s total.
Plan: project-memory/plans/0007-ndt7-npm-package-migration.md (giga repo)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
indexed-db.service.spec.ts imported LocalStorageService from a file that no longer exists; invalidlocation.page.spec.ts imported SchoolnotfoundPage from invalidlocation.page, which exports InvalidLocationPage. Both broke compilation of the whole karma suite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Silences the optimization-bailout warning the same way electron already is. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Applies 35 revised strings from 'Translations GigaMeter_UZB and Rus.xlsx'
(186 rows keyed by dotted path). Most are terminology and typo fixes:
'ID' -> 'IDsi' agreement, 'ulanish' -> 'internet' where the English says
connectivity, 'Maktabinggiz' -> 'Maktabingiz', and strings that were left
in English ('ISP', 'Open Database License', 'Dashboard') now translated.
releaseNotes.2.0.2.title is deliberately NOT applied: the sheet holds the
old translation concatenated with a reworded one ('... tajribasi - Yaxshilangan
...'), which would render as a doubled title. Left at its current value
pending confirmation from the translator.
Verified: all 186 sheet keys resolve against the JSON (including the dotted
version keys under releaseNotes and the array items), key set unchanged, and
every {{placeholder}} and HTML tag still matches en.json.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Standalone Node probe (no build, no Electron) covering the release v2.0.4 item-3 research: network interfaces, gateway, DNS, VPN inference, Wi-Fi, network stats/connections, OS, CPU, disk, memory and elevation. Two passes to flag volatile attributes, per-call timing, and JSON (raw + redacted) + CSV outputs. Probe outputs are gitignored: the raw dump contains SSIDs, MACs, internal IPs and the Windows username. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Make probe-system-info.js requirable (exports main), tag output files with the runtime (node vs electron-<version>), honor PROBE_OUT_DIR, and add electron-probe-main.js to run the probe on Electron's embedded Node — Artefacto 2 of plan 0008, unpackaged variant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Automates the 8-step manual checklist of plan 0010 against a real backend,
replacing the dead Protractor scaffold (`ng e2e` pointed at Protractor 7 and
there was no runnable e2e at all).
Covered, in serial over one install (the checklist is a linear walk and the
state accumulates): clean start -> school registration -> first ndt7 test
(real, against M-Lab) -> realtime upload -> manual test -> scheduled slot ->
post-measurement UI -> restart persistence.
The plan 0006 fields (upload_failed, scheduled_slot, scheduled_at) are checked
in the DB column, not just in the POST payload (e2e/playwright/db.ts queries
Postgres through the container, so no new dependency).
Notable bits:
- Waiting for a real slot would take hours, so step 6 injects an expired
semaphore for slot A and lets the scheduler's 60s tick pick it up, as it
would in production. scheduledTesting has to be enabled first or
getSemaphore() wipes the semaphore on every tick.
- Fixtures live here, not in giga-meter-backend: its seed-runner.ts and Spain
seed only exist on the develop line, which tied the suite to that branch.
e2e/seed/{seed.js,seed-spain.sql} are mounted into the container instead, so
the suite runs against develop and staging alike.
- The compose sets DIRECT_DATABASE_URL: some branches declare `directUrl` in
the Prisma datasource and the backend dies with P1012 without it.
Verified green (5 passed) against both backend lines: develop and staging,
each with the plan 0006 migration applied.
`npm run e2e:stg` runs the same spec against the real Azure staging, skipping
the DB assertions. Not exercised yet: it writes real data to a shared
environment.
Plan: project-memory/plans/0010-e2e-happy-path-test-rc-2.0.4.md (giga repo)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Item 2 of the v2.0.4 release plan. Mirrors matomo.service.ts: everything is wrapped in try/catch so a failure here can never take the app down, and with no configuration the service simply does nothing. posthog-js is bundled rather than loaded from a CDN the way Matomo is. This is a desktop app that spends real time offline — that is literally what it measures — so the SDK needs to queue events until connectivity returns, and bundling avoids pulling a remote script in under Electron's CSP. Configuration (placeholders for now, real keys still to be pasted in): - posthogKeyDev/Stg/Prod + posthogHost in _environment.prod.ts, with placeholders in the .example. The project API key is the Sentry-DSN equivalent: public and embedded in the build, never a personal API key. - An empty key disables PostHog for that mode, so an unconfigured build sends nothing. That is also what keeps the e2e suite from talking to a real project. Privacy choices worth flagging: - Identity is the school's giga id — the same identifier already travelling with every measurement. No school name, Windows username, install path or IP. - autocapture and capture_pageview are off; only explicit events are sent. Pageviews are captured manually because autocapture does not see hash routes. - Session replay is off behind a flag. It records the user's screen and the scope is still being defined by Shilpa's research; in schools that is a privacy call, not a technical one. Funnel events: app_started, registration_completed, measurement_uploaded, measurement_queued_offline, measurements_synced, measurements_sync_failed. The last three close the loop on the plan 0006 offline flag. Verified: tsc clean, build OK, karma shows the same 18 failures / 40 passes with and without this change (pre-existing baseline, no regressions), and the Playwright e2e suite stays green (5 passed). Plan: project-memory/plans/0004-release-v2.0.4-2026-08.md (giga repo) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follow-up to the PostHog integration: the desktop shell now reports the one thing neither the renderer nor the backend can see — the auto-update lifecycle. A machine that fails to update stops sending measurements, so it silently vanishes from the app_version adoption queries. The main process gets no PostHog SDK or key of its own. posthog-node was implemented first and dropped: - Most of what it added was already covered. Main-process errors go to Sentry (@sentry/node), and version adoption comes from app_version on every measurement (project-memory/technical/APP_VERSION_ADOPTION_QUERIES.md). - posthog-node does not persist its queue between sessions; posthog-js does, in localStorage. On a machine that spends hours offline, adding the node SDK would have made shell telemetry the *less* reliable half, and it forced a bounded flush on quit that risked hanging app shutdown. So the main process emits `desktop_update_downloaded` / `desktop_update_failed` on a `telemetry-event` IPC channel (exposed through the preload as electronAPI.onTelemetryEvent) and the renderer publishes them with the key it already has. Events are dropped if the window is not alive, which is acceptable: update failures already reach Sentry independently. tsc clean on both the main process and the renderer. Plan: project-memory/plans/0004-release-v2.0.4-2026-08.md (giga repo) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both branches implemented PostHog independently (feat/posthog is not even based on the 2.0.4 RC). Conflicts resolved to keep one implementation, per Victor: Configuration - Per-environment project API keys (prod/dev/stg) with a session-recording flag that stays off unless explicitly enabled, from this branch. - The EU host default from feat/posthog is kept: without it posthog-js falls back to US, and school data should not leave the EU by omission. - Persistence is 'localStorage+cookie' from feat/posthog: localStorage holds the offline queue, the cookie keeps the device's anon id stable if the renderer storage is cleared. - init() now also bails out on the placeholder key from the example config, so a half-configured build sends nothing instead of sending to a bogus project. Main process - No posthog-node: electron/src/analytics.ts and the dependency are dropped, and app_launched / app_quit join the auto-update events on the existing 'telemetry-event' IPC channel. posthog-js persists its queue in localStorage and survives restarts offline, which posthog-node would not — and this app is offline a lot. - app_quit is emitted before myCapacitorApp.cleanup(), while the window is still alive; if the process dies before it flushes, the queued event ships on the next launch. - The 'posthog-identity' IPC channel is gone with posthog-node: identity now lives entirely in the renderer. Identity - Both models are kept: identify() by GigaID (this branch) and the school as a PostHog *group* (feat/posthog). The group is what allows counting devices per school instead of collapsing them into a single person. Events kept from both sides: app_started, registration_completed, measurement_uploaded, measurement_queued_offline, measurements_synced, measurements_sync_failed (this branch), app_launched, app_quit (feat/posthog), plus the auto-update pair and manual $pageview. Verified: ng build green, tsc clean on app sources and electron main, karma shows the same 18 pre-existing "should create" failures as before the merge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reverts b1b65e5 and c9b1796, restoring the vendored ndt7 client under src/assets/js/ndt/. The npm package is a black box from the app's point of view, and the next commit needs to reach into the client itself: ndt7 exposes no server-side wall clock, so capturing one means editing discoverServerURLs. Owning the file again is the cheapest way to do that without carrying a patched fork of the package. Restores: ndt7.js and both workers, @m-lab/ndt7 back to ^0.0.6, the angular.json assets glob and allowedCommonJsDependencies entry, and the vendored import in measurement-client. Drops src/types/ndt7.d.ts, which only existed to type the package. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measurements are stamped with Date.now(), so the timestamp is only as trustworthy as the clock of the machine running the test - and on these machines it frequently is not. ndt7 has no wall clock of its own: the measurement messages the server sends during a test carry only ElapsedTime, relative to the start of the test, and the browser WebSocket API does not expose the headers of the handshake response. The one server clock within reach is the Date header of the locate service response, which discoverServerURLs already fetches. Date is CORS-safelisted, so it reads cross-origin with no change on M-Lab's side. ndt7.js reads that header, returns it as ServerTime alongside the URLs and hands it to the download/upload complete callbacks. It is null when the header is missing or unparseable, and on the config.server path, which never contacts the locate service. measurement-client stores it on the record as serverTimestamp, preferring the download leg, and upload.service sends it as server_timestamp (ISO 8601, or null). The existing timestamp field is untouched: this is an extra reference point, not a replacement. Backend counterpart: unicef/giga-meter-backend PR adding the server_timestamp column. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The probe scripts mixed Spanish into console output, probe metadata and the CSV the tracking spreadsheet is built from. Translate all of it so the research artefacts are shareable with the wider team. Text-only change: console messages, `group`/`attr`/`requiresAdmin` labels, the availability/volatility values (`sí`/`parcial`/`vacío` -> `yes`/`partial`/ `empty`) and the CSV header. The row keys were renamed alongside their CSV consumer (`disponible`/`ejemplo`/`volatil`/`notas` -> `available`/`sample`/ `volatile`/`notes`). No logic changes. Note: the generated CSV/JSON columns and the `group` values change, so a table already pasted into the UNICEF spreadsheet needs regenerating. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Research probe: network & device info retrievable from the Windows client (plan 0008)
Fills the device columns the backend has accepted since giga-meter-backend#353 but that nothing ever populated — every row landed with NULL device_name, device_model, device_manufacturer, app_build_number and sdk_version — and adds the network context and Wi-Fi diagnosis from research plan 0008. Main process: - `device-context.ts` captures the volatile context the ticket asked for and no column covered: DNS servers, default gateway, connection type, VPN inference, IP family, rx/tx counters, CPU load, available memory and free disk. - The `get-wifi-connections` handler now explains an empty result. On Windows 11 24H2+ `netsh wlan` returns nothing while Location services are off, so the list arrives empty on a machine that IS on Wi-Fi. The handler classifies why (no_adapter | wlan_service_off | location_disabled | unknown, from a registry read plus adapter and service checks) and recovers the SSID through the ungated Get-NetConnectionProfile fallback, tagging it `ssid_source: 'nlm'`. Both extra calls only run on the empty path. - `get-device-identity` returns hostname, model and manufacturer, cached per app run. Cost. The research measured networkInterfaces (~1.1 s), cpu (~1.7 s) and diskLayout (~2.1 s) — far too slow per measurement — so everything derived from them is computed once and cached under the default gateway, which recomputes when the machine changes network. The per-measurement calls are the cheap ones and run concurrently, keeping the added time inside the 1.5 s budget the plan set. Every capture fails soft: a locked-down PowerShell policy yields null fields, never a failed measurement. `app_build_number` is the short commit, baked by generate-build-mode.js (which already runs before every Electron build), overridable with GIGA_METER_BUILD_NUMBER for a pipeline that has its own id, falling back to the app version outside a git checkout. Neither package.json carries a build counter, so the commit is the only value that distinguishes two builds of one version. The same generator now also bakes the speed-test SDK versions so `sdk_version` cannot drift from the dependency that shipped. Renderer: `DeviceContextService` wraps the IPC calls, failing soft outside Electron so the web build and the unit tests keep working; the measurement client captures the context before the test and the upload service maps it onto the payload. Queued offline measurements carry the fields automatically — the whole built payload is what goes into IndexedDB. Note: the attribute list is the research proposal, still pending Vipul's confirmation against the tracking spreadsheet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The code comments pointed at plan numbers from a personal knowledge base that nobody outside its owner can open, so the references were dead weight in a public repo. Same facts, stated on their own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Bump version in package.json and package-lock.json for both Electron and main app to 2.0.4. - Introduce `isRegistering` flag in ConfirmschoolPage to prevent duplicate school registrations during the confirmation process. - Update confirmSchool method to handle registration state and prevent multiple submissions. - Add new Playwright test for verifying that multiple taps on the confirmation button do not create duplicate registrations. - Minor adjustments to loading behavior and button states in the UI to improve user experience during registration.
fix: one school registration per confirmation, and bump to 2.0.4
feat: capture network/device context and diagnose empty Wi-Fi reads (plan 0008)
…stamp The base gained the device/network context capture (#86) and the duplicate registration fix (#87) while this branch was reverting the ndt7 npm package back to the vendored copy. Both touch measurement-client.service.ts and upload.service.ts, so the two sides are combined here rather than one replacing the other: - measurement-client.service.ts / upload.service.ts merged cleanly: the vendored ndt7 import and the serverTimestamp capture from this branch, the DeviceContextService injection, the context capture and the device columns from the base. - measurement-client.service.spec.ts conflicted and is resolved by hand. The TestBed block keeps this branch's serverTimestamp coverage; the integration block keeps the base's DeviceContextService mock and its two retry tests, now spying on the vendored ndt7 object instead of the npm one. The base's metadata assertion is dropped: `metadata` no longer exists in testConfig, because the vendored ndt7.js hardcodes the client name and version. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The target branch refactored this registration callback from .subscribe() to async/await, so the original one-line change no longer applied. Took the refactored block wholesale. On this base the fix is also no longer a rename: 'schoolId' is already written a few lines above from the route param, which carries the very same school_id, and StorageService goes through localStorage.setItem, so both sides store the identical string. What was left was a write to 'school_id' that nothing in the app reads. Removed it rather than renaming it onto a key that is already correct. Verified with ng build. Assisted-by: Claude Opus 5
Remove the dead school_id storage write
feat: revert to vendored ndt7 and capture M-Lab's server clock
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three of the v2.0.4 items, plus a build fix the branch needed.
1. Scheduler retry upgrade (
20d725d)Scheduled tests retried 3 times, 15 minutes apart — about 45 minutes of coverage inside a 4-hour window. If connectivity came back later in the window, the slot was lost.
The app now retries for the whole window, paced by why the attempt failed:
getNetInfo()also gets anavigator.onLinepre-check, so offline ticks stop hitting the IP-info services.min(60s × 1.2^n, 10 min), since each attempt costs real bandwidth. ~30 attempts per window instead of ~240 at a flat minute.The semaphore now persists
retryAttempts,backoffLevel(reset when the network returns after a no-network failure) andlastFailReason. Pre-2.0.4 semaphores without these fields are handled.Bug found along the way: when fully offline,
getNetInfo()throws — its geojs fallback fails too. That exception escapeddecide()and was swallowed bywatch(), so the no-network path never reached the rescheduling logic at all. It retried every minute by accident, without counting attempts. Now caught and treated as no-network.schedule.service.spec.tshad only a "should be created" and was rewritten: 13 specs covering the fixed offline pace, the 1.2^n growth, the cap, the recovery reset, success, window expiry, the clamp to the window end, and legacy semaphores.2. Local test storage flag (
ae5d6ba)Measurements whose realtime upload failed were already queued in IndexedDB and re-sent by the periodic sync, but once they reached the backend they were indistinguishable from realtime uploads. The payload now carries:
upload_failed—falseon realtime uploads; the copy queued in IndexedDB is saved withtrue, so the sync delivers it already flagged and needed no changes of its own.scheduled_slot—'A' | 'B' | 'C'for slot tests,'startup'for the daily launch test,nullfor manual runs.scheduled_at— the originally planned run time (the semaphore keeps it separately fromchoice, which moves with every retry).The sync payload also stops leaking IndexedDB bookkeeping fields (
id,status,createdAt) — the old code strippedisSynced, which these records never had.Backend counterpart: unicef/giga-meter-backend#347. The columns are additive, so this can ship before or after it.
3. Uzbek translations (
f405083)35 revised strings from the reviewed translation sheet (186 rows keyed by dotted path):
ID→IDsiagreement,ulanish→internetwhere the English says connectivity,Maktabinggiz→Maktabingiz.ISP,Open Database License,Dashboard) now translated.{{placeholder}}and HTML tag still matchesen.json.releaseNotes.2.0.2.titleis deliberately not applied: the sheet holds the old translation concatenated with a reworded one (... tajribasi - Yaxshilangan ...), which would render as a doubled title. Left at its current value, pending confirmation from the translator.4. Spec import fixes (
1211351)indexed-db.service.spec.tsimported aLocalStorageServicefrom a file that no longer exists, andinvalidlocation.page.spec.tsimportedSchoolnotfoundPagefrom a page that exportsInvalidLocationPage. Both broke compilation of the whole karma suite, so nothing could be tested on this branch until they were fixed.Testing
schedule.service.spec.ts: 13/13 green.should createfailures remain (TestBed setups missing providers, in components untouched here — they were hidden before, since the suite didn't compile). No new failures.ng lintis broken repo-wide (.eslintrc.jsoncan't loadplugin:@angular-eslint/recommended-extra) — pre-existing, not addressed here.Still to do before the release
locate.measurementlab.net, reconnect) and end-to-end for the storage flag.