Refactor update.sh script flags and enhance documentation

CHANGELOG.md

@@ -5,6 +5,20 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/).
 
+## [Unreleased]
+
+### Changed
+
+- Unify the update-script flag scheme. `./scripts/update.sh` (no args) now updates all sources incrementally from each source's last checkpoint. Source restriction lives on `--source`; source-scoping flags (`--titles`, `--days`, `--from`, `--to`) live at the top level instead of per-source prefixes. Old prefixes (`--ecfr-titles`, `--ecfr-all`, `--ecfr-skip-highlights`, `--fr-days`, `--fr-from`, `--fr-to`, `--usc-force`, `--usc-skip-highlights`) are removed; running them prints a migration hint and exits 1. `update-ecfr.sh --all` similarly removed in favor of `--force`.
+- Add `--skip-search`, `--dry-run`, and consistent `--force` semantics to all four scripts.
+- `update-fr.sh` now persists a JSON checkpoint at `downloads/fr/.fr-state.json` (`{ lastRun, lastDate }`). Default invocations resume from `lastDate`. Bootstrap (no checkpoint) errors with a hint requiring `--from` or `--days`, since FR has no inherent "all".
+- eCFR/USC bootstrap (missing `.ecfr-titles-state.json` / `.usc-release-point`) now logs the bootstrap explicitly and falls back to a full first-run automatically.
+- When `--force` runs against all three sources, the orchestrator now performs a single full `deploy.sh --search-docker` reindex at the end instead of three per-source incremental indexes.
+
+### Fixed
+
+- `--verbose` / `-v` on `update.sh` and the three sub-scripts now actually works. Previously the orchestrator parsed it and forwarded `--verbose` to each sub-script, but the sub-scripts rejected it as "Unknown option" and exited 1. Each sub-script now accepts `-v` / `--verbose` and threads `--verbose` through to its `convert-*` CLI invocation.
+
 ## [1.25.0]
 
 ### Added

CLAUDE.md

@@ -19,19 +19,22 @@ lexbuild/
 │   └── api/         # @lexbuild/api — Data API (Hono, SQLite, Meilisearch proxy)
 ├── scripts/
 │   ├── deploy.sh              # Production deploy (code, content, or full remote pipeline)
-│   ├── update.sh              # Unified incremental content update (all sources)
-│   ├── update-ecfr.sh         # eCFR incremental update (auto-detects changed titles)
-│   ├── update-fr.sh           # FR incremental update (date-range based)
-│   ├── update-usc.sh          # USC incremental update (release point detection)
+│   ├── update.sh              # Unified content update orchestrator (incremental by default)
+│   ├── update-ecfr.sh         # eCFR sub-script (change-detection via API metadata)
+│   ├── update-fr.sh           # FR sub-script (checkpoint-based date window)
+│   ├── update-usc.sh          # USC sub-script (release-point detection)
 │   ├── ecfr-changed-titles.ts # eCFR change detection helper (API metadata vs checkpoint)
 │   ├── setup-secrets.sh       # Initialize ~/.lexbuild-secrets on VPS
 │   └── .deploy.env.example    # Template for .deploy.env (VPS_HOST config)
 ├── downloads/
 │   ├── usc/
-│   │   └── xml/     # Full USC XML files (usc01.xml ... usc54.xml) — gitignored
+│   │   ├── xml/                       # Full USC XML files (usc01.xml ... usc54.xml) — gitignored
+│   │   └── .usc-release-point         # USC checkpoint (latest OLRC release point ID)
 │   ├── ecfr/
-│   │   └── xml/     # Full eCFR XML files (ECFR-title1.xml ... ECFR-title50.xml) — gitignored
-│   └── fr/          # FR XML + JSON files (YYYY/MM/doc-number.xml/.json) — gitignored
+│   │   ├── xml/                       # Full eCFR XML files (ECFR-title1.xml ... ECFR-title50.xml) — gitignored
+│   │   └── .ecfr-titles-state.json    # eCFR checkpoint (per-title amendment dates)
+│   └── fr/                            # FR XML + JSON files (YYYY/MM/doc-number.xml/.json) — gitignored
+│       └── .fr-state.json             # FR checkpoint ({ lastRun, lastDate })
 ├── fixtures/
 │   ├── fragments/   # Small synthetic XML snippets for unit tests
 │   └── expected/    # Expected output snapshots for integration tests
@@ -129,18 +132,36 @@ pnpm turbo build:api --filter=@lexbuild/api        # Production build
 ./scripts/deploy.sh --search-docker --source fr   # Incremental: index one source into existing volume
 ./scripts/deploy.sh --search-docker-seed          # Seed Docker volume from VPS (recover after volume loss)
 
-# Incremental content updates (from monorepo root)
-# Search indexing runs locally in Docker, not on the VPS — each update script's
-# final step delegates to `deploy.sh --search-docker --source <name>`.
-./scripts/update.sh                                # All sources incrementally
-./scripts/update.sh --source ecfr                  # One source
-./scripts/update.sh --skip-deploy                  # Local only
-./scripts/update-ecfr.sh                           # eCFR only (auto-detects changed titles)
-./scripts/update-ecfr.sh --titles 1,17             # Specific eCFR titles
-./scripts/update-fr.sh --days 3                    # FR last 3 days
-./scripts/update-usc.sh                            # USC (checks for new release point)
+# Content updates (from monorepo root)
+# Default behavior is incremental from each source's checkpoint. Search indexing
+# runs locally in Docker; each sub-script delegates to `deploy.sh --search-docker
+# --source <name>` after the local pipeline.
+./scripts/update.sh                                # All sources, incremental from checkpoints
+./scripts/update.sh --source fr                    # One source, incremental
+./scripts/update.sh --source ecfr,fr               # Multi-source, incremental
+./scripts/update.sh --source ecfr --titles 1,17    # eCFR titles 1, 17 only (skip change-detection)
+./scripts/update.sh --source fr --days 7           # FR last 7 days
+./scripts/update.sh --source usc --force           # USC full redownload + reconvert
+./scripts/update.sh --force --from 2026-01-01      # All sources, full rebuild (FR force requires --from)
+./scripts/update.sh --skip-deploy                  # Local only (no rsync, no search)
+./scripts/update.sh --skip-search                  # Rsync content/nav, but skip search reindex
+./scripts/update.sh --deploy-only                  # Push existing local output + reindex
+./scripts/update.sh --dry-run                      # Print plan, exit 0
+./scripts/update.sh -v / --verbose                 # Pass --verbose through to each convert step
 ```
 
+Sub-scripts (`update-ecfr.sh`, `update-fr.sh`, `update-usc.sh`) accept the same flag grammar
+minus `--source`. Run any of them with `--help` for the full list.
+
+**Checkpoints** (gitignored, in `downloads/<source>/`):
+- `usc/.usc-release-point` — latest OLRC release point ID (plain text).
+- `ecfr/.ecfr-titles-state.json` — per-title `latestAmendedOn` snapshot.
+- `fr/.fr-state.json` — `{ lastRun, lastDate }`. Default `update-fr.sh` uses `lastDate` as `--from`.
+
+If a checkpoint is missing, eCFR/USC bootstrap into a full first-run automatically. FR bootstrap
+errors with a hint; you must specify `--from YYYY-MM-DD` or `--days N` because FR has no inherent
+"all" (decades of documents).
+
 See `packages/cli/CLAUDE.md` for full command options. See `apps/astro/CLAUDE.md` for content pipeline scripts.
 
 ### CI / Release
@@ -272,6 +293,9 @@ Note: identifiers use `/us/cfr/` (content type) not `/us/ecfr/` (data source). B
 
 ## Common Pitfalls
 
+- **macOS ships bash 3.2 + `set -u` crashes empty-array expansion**: `/usr/bin/env bash` resolves to `/bin/bash` 3.2.57 on macOS. `printf '%s\n' "${arr[@]}"` and `"${arr[@]}"` at call sites fire `bash: arr[@]: unbound variable` when the array is empty. Use the bash 3.2-safe pattern: `printf '%s\n' "${arr[@]+"${arr[@]}"}"` (and same at call sites). Affects every script in `scripts/` since they all use `set -euo pipefail`.
+- **`set -u` + `$2` in value-taking case arms**: a case arm like `--foo) some_helper "hint $2" ;;` crashes with `bash: $2: unbound variable` if the user runs `script --foo` (no value) — *before* the helper's error-handling runs. Use `${2:-<placeholder>}`. The migration helpers in `scripts/update.sh` are the live example.
+- **macOS BSD sed vs GNU sed brace blocks**: `sed -n '2,/^$/{ s/^# //; p }'` (multi-command brace block) breaks on BSD sed with "extra characters at the end of p command". Use `awk 'NR==1{next} /^$/{exit} {sub(/^# ?/, ""); print}'` for cross-platform multi-line extraction. The update scripts use this pattern for `--help`.
 - **Hono v4 HTTPException bypasses middleware catch blocks**: In Hono v4, `HTTPException` is intercepted at the compose layer before middleware try-catch runs. Use `app.onError()` (not middleware) to handle `HTTPException`. The Data API configures this in `apps/api/src/app.ts`.
 - **USC snapshot fixtures use a `lexbuild@__VERSION__` placeholder**: `fixtures/expected/*.md` store the generator field as `lexbuild@__VERSION__`. The snapshot test (`packages/usc/src/snapshot.test.ts`) normalizes the live `generator: "lexbuild@X.Y.Z"` line to the same placeholder before comparison via a `normalizeGenerator()` helper, so version bumps do NOT churn the fixtures — no sed bump needed on release. (Historical note: before this normalization landed, the changesets Version Packages PR would fail CI because it bumped `package.json` but not the fixtures; the documented recovery was either to update fixtures in that PR or do a fully manual version bump.)
 - **Changeset `major` with lockstep versioning bumps ALL packages**: All 6 published packages are in the `fixed` array. A `major` changeset on any one package bumps every package to the next major (e.g., 1.x → 2.0.0). Prefer `minor` or `patch` unless all packages genuinely have breaking changes.

README.md

@@ -131,18 +131,36 @@ lexbuild enrich-fr --from 2000-01-01
 
 ### Incremental Updates
 
-Update scripts handle change detection, download, convert, and deploy in one command:
+A single script handles change detection, download, convert, and deploy across every source. The default is incremental from each source's last checkpoint:
 
 ```bash
-# Update all sources (auto-detects changes)
+# Painless: update everything from each source's checkpoint
+./scripts/update.sh
+
+# Restrict to one source (or several)
+./scripts/update.sh --source fr
+./scripts/update.sh --source ecfr,fr
+
+# Source-scoping flags
+./scripts/update.sh --source ecfr --titles 1,17    # eCFR titles 1, 17 only
+./scripts/update.sh --source fr --days 7           # FR last 7 days
+
+# Force a full redownload + reconvert
+./scripts/update.sh --source usc --force
+./scripts/update.sh --force --from 2026-01-01      # All sources (FR force requires --from)
+
+# Local only (no rsync to VPS, no search reindex)
 ./scripts/update.sh --skip-deploy
 
-# Update individual sources
-./scripts/update-ecfr.sh --skip-deploy
-./scripts/update-fr.sh --days 3 --skip-deploy
-./scripts/update-usc.sh --skip-deploy
+# Preview without running
+./scripts/update.sh --dry-run
+
+# Verbose convert output
+./scripts/update.sh -v
 ```
 
+Each source has a checkpoint in `downloads/<source>/`. eCFR/USC bootstrap automatically into a full first-run if their checkpoint is missing; FR errors with a hint and requires `--from`. Run `./scripts/update.sh --help` (or any sub-script with `--help`) for the full grammar.
+
 `update-usc.sh` and `update-ecfr.sh` convert every granularity in one parse using the `--granularities` flag (see below), so the convert step no longer scales with the number of output granularities.
 
 ---

apps/astro/src/content/docs/cli/commands.md

@@ -127,17 +127,21 @@ lexbuild convert-fr --all
 
 ## Update Scripts
 
-For routine updates, wrapper scripts handle the full pipeline (detect changes, download, convert, generate artifacts, deploy):
+A single orchestrator handles change detection, download, convert, and deploy across all sources. Default is incremental from each source's last checkpoint:
 
 ```bash
-./scripts/update.sh                      # All sources
-./scripts/update.sh --source ecfr        # One source
-./scripts/update-ecfr.sh --skip-deploy   # eCFR, local only
-./scripts/update-fr.sh --days 3          # FR, last 3 days
-./scripts/update-usc.sh                  # USC, checks release point
+./scripts/update.sh                                # All sources, incremental from checkpoints
+./scripts/update.sh --source fr                    # One source
+./scripts/update.sh --source ecfr,fr               # Multi-source
+./scripts/update.sh --source ecfr --titles 1,17    # eCFR titles 1, 17 only
+./scripts/update.sh --source fr --days 7           # FR last 7 days
+./scripts/update.sh --source usc --force           # USC full redownload + reconvert
+./scripts/update.sh --skip-deploy                  # Local pipeline only
+./scripts/update.sh --dry-run                      # Print plan, exit 0
+./scripts/update.sh -v                             # Pass --verbose through to each convert step
 ```
 
-Each script auto-detects what changed and only processes updates. `update-usc.sh` and `update-ecfr.sh` convert all granularities in one call using `--granularities` (see above), so the convert step parses the XML once per title rather than once per granularity. See [Incremental Updates](/docs/guides/bulk-download#incremental-updates) for details.
+`update-usc.sh` and `update-ecfr.sh` convert all granularities in one call using `--granularities` (see above), so the convert step parses the XML once per title rather than once per granularity. See [Incremental Updates](/docs/guides/bulk-download#incremental-updates) for details on checkpoints and bootstrap behavior.
 
 ## Getting Help
 

apps/astro/src/content/docs/guides/bulk-download.md

@@ -143,24 +143,43 @@ lexbuild convert-fr --from 2026-03-01
 
 ### Update Scripts
 
-For streamlined incremental updates, wrapper scripts handle the full pipeline (detect changes, download, convert, generate artifacts, deploy):
+A single orchestrator handles change detection, download, convert, and deploy across every source. Default is incremental from each source's last checkpoint:
 
 ```bash
-# All sources — auto-detects changes, downloads, converts, deploys
+# All sources, incremental from each source's checkpoint
 ./scripts/update.sh
 
-# Individual sources
-./scripts/update-ecfr.sh               # Only changed titles (via API metadata)
-./scripts/update-fr.sh --days 3         # Last 3 days
-./scripts/update-usc.sh                 # Checks OLRC release point
+# Restrict to specific sources
+./scripts/update.sh --source fr
+./scripts/update.sh --source ecfr,fr
 
-# Local only (no VPS deploy)
+# Source-scoping
+./scripts/update.sh --source ecfr --titles 1,17    # eCFR titles 1, 17 only
+./scripts/update.sh --source fr --days 7           # FR last 7 days
+
+# Force a full redownload + reconvert
+./scripts/update.sh --source usc --force
+./scripts/update.sh --force --from 2026-01-01      # All sources (FR force requires --from)
+
+# Local only (no VPS deploy, no search reindex)
 ./scripts/update.sh --skip-deploy
+
+# Preview without running
+./scripts/update.sh --dry-run
+
+# Verbose convert output
+./scripts/update.sh -v
 ```
 
-The eCFR script compares `latestAmendedOn` dates from the eCFR API against a local checkpoint to detect which titles have new amendments. The USC script checks for new OLRC release points. The FR script uses date-range filtering.
+Checkpoints live in `downloads/<source>/`:
+
+- **eCFR** (`.ecfr-titles-state.json`) snapshots each title's `latestAmendedOn` date. The script compares against the live eCFR API to detect which titles have new amendments.
+- **USC** (`.usc-release-point`) stores the latest OLRC release point ID; the pipeline runs only when the API returns a newer one.
+- **FR** (`.fr-state.json`) stores `lastRun` and `lastDate`. Default invocations use `lastDate` as the `--from` argument and update `lastDate` to today after a successful run.
+
+If a checkpoint is missing, eCFR/USC bootstrap into a full first-run automatically. FR has no inherent "all" (decades of documents), so a missing checkpoint requires explicit `--from YYYY-MM-DD` or `--days N`.
 
-All converters use `writeFileIfChanged()` internally, so unchanged sections keep their original file timestamps. This means downstream tools (Shiki highlighting, Meilisearch indexing) automatically skip reprocessing unchanged content.
+All converters use `writeFileIfChanged()` internally, so unchanged sections keep their original file timestamps. Downstream tools (Shiki highlighting, Meilisearch indexing) automatically skip reprocessing unchanged content.
 
 ## Output Granularity
 

apps/astro/src/content/docs/project/changelog.md

@@ -12,6 +12,14 @@ For the complete changelog, see [CHANGELOG.md on GitHub](https://github.com/chri
 
 ## Recent Releases
 
+### Unreleased
+
+- Unified update-script flag scheme. `./scripts/update.sh` (no args) now updates all sources incrementally from each source's last checkpoint. Source restriction via `--source`; source-scoping flags (`--titles`, `--days`, `--from`, `--to`) live at the top level. Old prefixes (`--ecfr-titles`, `--fr-days`, `--usc-force`, etc.) are removed and print migration hints
+- New `--skip-search`, `--dry-run`, and consistent `--force` semantics across all four scripts
+- New FR checkpoint at `downloads/fr/.fr-state.json` (`{ lastRun, lastDate }`). Default `update-fr.sh` resumes from `lastDate`. Missing checkpoint errors with a hint requiring `--from` or `--days`
+- eCFR/USC bootstrap (missing checkpoint) now logs explicitly and runs a full first-run automatically
+- Fixed: `--verbose` / `-v` on the update scripts now actually works — sub-scripts accept the flag and pass `--verbose` through to their `convert-*` CLI step (previously exited 1 with "Unknown option")
+
 ### 1.17.2
 
 - Added **Data API** (`apps/api/`) -- Hono-based REST API serving U.S. legal content from SQLite with Meilisearch search proxy