feat: Support tracking issue impact over time (#156) by duncanleo · Pull Request #177 · foldaway/mrtdown-data

duncanleo · 2026-03-21T09:48:22Z

Summary

Overhauls mrtdown-data into the canonical reviewed data repository for MRTDown, with typed packages, deterministic file-backed tooling, and a migrated append-only issue dataset.

What Changed

Migrated canonical data into the new data/ layout:
- static entities under data/{station,line,service,operator,town,landmark}
- issues under data/issue/YYYY/MM/<issue_id>/
- append-only evidence.ndjson and impact.ndjson per issue
Introduced the monorepo package structure:
- @mrtdown/core for schemas and shared period/state helpers
- @mrtdown/fs for file-backed repositories and writers
- @mrtdown/cli for creation, validation, manifest, listing, show, and repair tooling
- @mrtdown/triage for LLM-assisted evidence triage and replay utilities
Removed the old API/database runtime from this repo; runtime serving and Postgres import now belong in mrtdown-site.
Added GitHub Pages publishing for generated manifests and downloadable data archives.
Added Changesets/npm publishing workflow for shared packages.
Added architecture docs describing the canonical-data vs runtime-data split and the two-repo mrtdown-data / mrtdown-site model.
Replayed and normalized issue impact data, including repairs for empty impacts, degraded-service extraction, recurring maintenance periods, and legacy issue normalization.

Breaking Changes

This repo no longer ships the old Hono/API/DuckDB runtime.
Consumers should use the new packages and/or the generated GitHub Pages data artifacts instead of importing from the previous src/api, src/db, or legacy schema paths.

Review Notes

This is a large structural migration. The most important review areas are:

package boundaries and exports
canonical data layout compatibility with mrtdown-site
CLI validation behavior
generated manifest/archive workflow
issue impact replay correctness

Validation

TODO before merge:

npm ci
npm run build
npm run test
npm run cli -- -- validate
Confirm Pages artifact generation with npm run cli -- -- manifest and npm run cli -- -- pages-index

Fixes #156

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Add one-off collapse tooling that clusters maintenance/infra issues by slug, type, and overlapping or adjacent time ranges, then merges into deterministic canonical issues. Apply the merge to current duplicate groups and persist audit artifacts for traceability. Made-with: Cursor

Drop the temporary collapse scripts and generated reports now that the one-off maintenance/infra merge has been applied to the data files. Made-with: Cursor

Update multilingual render text and translation source metadata for evidence rows that were rewritten from verifiable source articles, and keep deduped entries consistent with the revised text. Made-with: Cursor

Ensure service-impact claims without time hints are anchored to the evidence timestamp so impact events are not dropped, while still allowing maintenance/infra metadata-only cause updates to persist without requiring service periods. Made-with: Cursor

Add a deterministic relative-date tool with RRULE-style weekday inputs and tighten extraction guidance so planned weekend constraints become explicit timed claims instead of being dropped as advisory text. Also bump the OpenAI SDK to the latest workspace version. Made-with: Cursor

Persist regenerated impact events from the latest empty-impact replay run, including recovered weekend maintenance windows and related issue updates that now produce concrete events. Made-with: Cursor

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Delete the three CCL no-mobile-signals issue bundles to simplify the current cleanup pass and reduce unresolved issue validation failures. Made-with: Cursor

Strengthen extraction guidance for single-platform degraded-service evidence, replay the affected bucket 2/3 issues, and update the temporary validation report to reflect the reduced failure count and remaining buckets. Made-with: Cursor

Remove the temporary validation findings report from version control so it stays local-only and out of committed changes. Made-with: Cursor

Prevent `mrtdown-cli show` from crashing when derived service/facility state omits a `causes` array by defaulting to an empty list before rendering. Made-with: Cursor

Capture the source article's station-by-station rollout schedule in evidence and align impact periods so facilities are modeled in phased windows rather than one concurrent multi-year window. Made-with: Cursor

duncanleo · 2026-05-03T05:57:06Z

@codex review

chatgpt-codex-connector · 2026-05-03T06:01:20Z

💡 Codex Review

mrtdown-data/packages/triage/src/util/ingestContent/index.ts

Line 20 in e29fd65

const DATA_DIR = resolve(import.meta.dirname, '../../../../data');

Point ingestContent to the repository data directory

ingestContent resolves DATA_DIR with four .. segments from packages/triage/src/util/ingestContent, which lands at <repo>/packages/data instead of <repo>/data; that directory does not exist in this repo, so the first repository read (via FileStore.listDir in triage flow) will throw ENOENT and break webhook ingestion. This is production-impacting because ingestViaWebhook imports and executes this code path for every incoming message.

mrtdown-data/packages/fs/src/repo/issue/IssueRepository.ts

Lines 177 to 180 in e29fd65

    
           'impact.scopeItems.serviceId', 
        
           'impact.scopeItems.stationId', 
        
           'impact.scopeItems.fromStationId', 
        
           'impact.scopeItems.toStationId',

Query against actual impact event fields in issue search

The Fuse search keys target impact.scopeItems.*, but IssueRepository.get() stores impact data under impactEvents with event-specific fields (serviceScopes, entity.serviceId, etc.), so these clauses never match and searches by affected services/stations silently fail. This degrades FindIssuesTool matching and can cause triage to miss existing related issues.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

duncanleo self-assigned this Mar 21, 2026

duncanleo changed the title ~~feat: Support tracking issue impact over time~~ feat: Support tracking issue impact over time (#156) Mar 21, 2026

duncanleo force-pushed the duncanleo/data-overhaul branch 13 times, most recently from ca15f1a to 965bee3 Compare March 22, 2026 16:43

duncanleo had a problem deploying to github-pages March 22, 2026 17:24 — with GitHub Actions Failure

duncanleo temporarily deployed to github-pages March 22, 2026 17:25 — with GitHub Actions Inactive

duncanleo temporarily deployed to github-pages March 22, 2026 18:05 — with GitHub Actions Inactive

duncanleo temporarily deployed to github-pages March 22, 2026 18:09 — with GitHub Actions Inactive

duncanleo temporarily deployed to github-pages March 22, 2026 18:12 — with GitHub Actions Inactive

duncanleo temporarily deployed to github-pages March 23, 2026 06:28 — with GitHub Actions Inactive

duncanleo temporarily deployed to github-pages March 23, 2026 06:29 — with GitHub Actions Inactive

duncanleo temporarily deployed to github-pages March 23, 2026 06:35 — with GitHub Actions Inactive

duncanleo temporarily deployed to github-pages March 23, 2026 16:03 — with GitHub Actions Inactive

duncanleo force-pushed the duncanleo/data-overhaul branch from 0b0ff52 to 78cdc9d Compare March 23, 2026 16:07

duncanleo temporarily deployed to github-pages March 23, 2026 16:08 — with GitHub Actions Inactive

duncanleo force-pushed the duncanleo/data-overhaul branch from 2e64332 to 46a53bb Compare April 4, 2026 10:27

duncanleo temporarily deployed to github-pages April 4, 2026 10:28 — with GitHub Actions Inactive

duncanleo temporarily deployed to github-pages April 4, 2026 10:29 — with GitHub Actions Inactive

duncanleo and others added 26 commits May 2, 2026 21:43

chore(data): reconcile rebased issue updates

b5b9b10

docs(changeset): rebased release

b212f25

Version Packages (alpha) (#210)

10de246

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

triage: add empty-impact replay targeting

0603c07

data: replay empty impact issues

f0ebde8

triage: tune gpt-5.4-mini extractor evals

ffb450a

chore: remove one-off collapse tooling artifacts

8319e17

Drop the temporary collapse scripts and generated reports now that the one-off maintenance/infra merge has been applied to the data files. Made-with: Cursor

data: align edited evidence translations with source-backed text

e1b8fe1

Update multilingual render text and translation source metadata for evidence rows that were rewritten from verifiable source articles, and keep deduped entries consistent with the revised text. Made-with: Cursor

build(deps.triage): add 'luxon'

dcfb78a

build(deps.triage): bump 'openai' to v6.34.0

7e7a58e

data: replay latest empty-impact re-extraction batch

571db9c

Persist regenerated impact events from the latest empty-impact replay run, including recovered weekend maintenance windows and related issue updates that now produce concrete events. Made-with: Cursor

feat(triage): add gpt-5.4-mini pricing

d7bec95

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

data: remove CCL mobile-signal issue records

4a90d66

Delete the three CCL no-mobile-signals issue bundles to simplify the current cleanup pass and reduce unresolved issue validation failures. Made-with: Cursor

chore: keep temporary findings report untracked

4ef0c98

Remove the temporary validation findings report from version control so it stays local-only and out of committed changes. Made-with: Cursor

fix(cli): guard missing causes in show output

220c0b4

Prevent `mrtdown-cli show` from crashing when derived service/facility state omits a `causes` array by defaulting to an empty list before rendering. Made-with: Cursor

data: add phased screen-door upgrade windows for 2025-08-01 issue

da88682

Capture the source article's station-by-station rollout schedule in evidence and align impact periods so facilities are modeled in phased windows rather than one concurrent multi-year window. Made-with: Cursor

triage: use none reasoning for claim extraction

d6da691

data: recover 2011-12-14 circle line disruption

bf87aa2

data: normalize remaining legacy issue impacts

8198158

cli: add issue repair and id tooling

3fa20da

docs: update two-repo architecture plan

57da025

fix: restore canonical ingest workflow

e29fd65

duncanleo force-pushed the duncanleo/data-overhaul branch from f2d526d to e29fd65 Compare May 3, 2026 04:51

duncanleo deployed to github-pages May 3, 2026 04:51 — with GitHub Actions View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support tracking issue impact over time (#156)#177

feat: Support tracking issue impact over time (#156)#177
duncanleo wants to merge 102 commits intomainfrom
duncanleo/data-overhaul

duncanleo commented Mar 21, 2026 •

edited

Loading

Uh oh!

duncanleo commented May 3, 2026

Uh oh!

chatgpt-codex-connector Bot commented May 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

duncanleo commented Mar 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What Changed

Breaking Changes

Review Notes

Validation

Uh oh!

duncanleo commented May 3, 2026

Uh oh!

chatgpt-codex-connector Bot commented May 3, 2026

💡 Codex Review

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

duncanleo commented Mar 21, 2026 •

edited

Loading