feat: script for detecting unstable producer urls

[ianktc]

Summary

is_producer_url_unstable is used to identify feeds whose urls may change over time due to a date or a version baked into it. The feed url therefore stops being accessible the moment the agency publishes again with a different version or date. Only active and inactive feeds are scanned, and deprecated feeds can be scanned if a flag is added.

This PR adds scripts/identify_unstable_urls.py, which scans producer URLs for that
evidence and applies the field to what it finds. It also adds the is_producer_url_unstable field to the realtime feed schema in case of future adoption.

Usage

The script never writes on a plain run. The intended workflow is scan, read the CSV, then
apply:

# 1. Dry run. Prints per-rule hit counts and one line per feed, writes nothing.
python scripts/identify_unstable_urls.py

# 2. Same scan, but dump the reviewable CSV.
python scripts/identify_unstable_urls.py --report unstable_producer_urls.csv

# 3. Read the CSV. Then, once satisfied, write the field.
python scripts/identify_unstable_urls.py --apply

Other options: --data-type {gtfs,gtfs-rt,all} to narrow the catalog, --include-deprecated
to also scan deprecated feeds (raises the count from 86 to 126), and --weak to print the
signals that are deliberately never flagged, for manual review.

Redaction rules — Tokens to ignore (allow-list)

Blanked to a sentinel before any matching runs, so they cannot contribute a hit.

Rule	Matches	Why it's stable
uuid	8-4-4-4-12 hex	CKAN dataset/resource keys
hex_id	exactly 32 hex	ArcGIS item ids, Wix/hibu site keys, opendatasoft file ids. Deliberately not 40 — that's a git SHA
azure_service_version	sv=YYYY-MM-DD	Azure storage API version. The freshest-looking date in a SAS URL and it never moves
azure_signature	sig=…	SAS signature blob
credential	api_key/token/key/subscription-key=…	A 40-hex Mecatran apiKey is indistinguishable from a pinned commit; 8 feeds were mis-flagged before this
api_version_segment	/vN/ as a whole segment	API version, not feed version
api_version_segment_dotted	/vN.0/ as a whole segment	A zero minor is decorative — data.waltti.fi serves 11 feeds from /v1.0/
transitfeeds_id	/p/<agency>/<id>/latest	TransitFeeds numeric feed key
drive_file_id	?id=<25+ chars>	Google Drive file id

Detection rules — Token to trigger hit

Setis_producer_url_unstable to "True"

Rule	Catches
iso_date	YYYYMMDD / - / _ / . separated, plus optional HHMM/HHMMSS
delimited_year	standalone year 2010–2029, delimiter-anchored both sides
wordpress_upload_folder	wp-content/uploads/[sites/N/]YYYY/MM/
dated_version_tag	2020.0401.1 style release tags
dotted_version	v2.8, ?v=0.16.0, v1.0.0
locale_date	day-first 03.10.2025, 06-07-2021, 5.12.21
month_abbreviation_with_digits	3-letter month only fused to digits: coquimbo10feb16
version_query_parameter	&version=20250220
epoch_timestamp	10-digit Unix seconds
month_name	full month names, 6 languages
dotnet_ticks	18-digit DateTime.Ticks
labelled_year	2025Winter, Spring2025, Q1-CY2025, CY2026
shared_access_signature_window	SAS st=/se= expiry
cache_buster	refresh=/nocache=/timestamp= + 8+ hex
pinned_git_sha	40 hex as a path segment
large_version_counter	vNN+ — GTFS-V126
year_month	bare YYYYMM — gtff-202311

[emmambd]

Someone else on the team is probably better positioned to review this one than me. Are there specific requirements you wanted to discuss?

[ianktc]

Someone else on the team is probably better positioned to review this one than me. Are there specific requirements you wanted to discuss?

I saw comments on the other pr #1629 so I'll refine the detection rules so we don't catch false positives. I'll bring it up to the team after I can finalize them with you

[emmambd]

@ianktc Sounds good - looking at the "Detection rules" here, there may be some that I mistook as a false positive