Initial release of Raincloud (v0.1.0)

Raincloud is a client-reproducible pipeline for building a curated catalog
of 249 public datasets as analytics-ready Parquet + Vortex files. See
README.md for the user-facing overview, AGENTS.md for the architecture,
SKILLS.md for procedural playbooks, and CHANGELOG.md for the v0.1.0
release contents.

License: Apache-2.0.

Author: mprammer

.agents/README.md

@@ -0,0 +1,24 @@
+# .agents
+
+Everything an AI coding agent needs to operate inside this repo. The `.claude → .agents` symlink at the repo root means tooling that follows either naming convention (Claude Code, Codex, Cursor, …) sees the same files.
+
+## What's here
+
+| Path | Role |
+|---|---|
+| [`settings.json`](settings.json) | **Tracked** allow-list of safe, read-only Bash / git / WebFetch / pipeline commands. A fresh-clone agent gets these pre-approved so it doesn't burn turns on permission prompts. Side-effecting stages (build / fetch / extract / convert / tighten_variant) are intentionally *not* pre-approved here. |
+| `settings.local.json` | **Gitignored** per-machine override — additional permissions specific to the local agent's session. Don't commit. |
+| [`skills/`](skills/) | 16 invokable skills following the [Agent Skills](https://agentskills.io) standard — wrappers around pipeline entrypoints (`/raincloud-build`, `/raincloud-fetch`, `/raincloud-status`, `/raincloud-validate-manifest`, `/raincloud-list-datasets`, …) and procedural playbooks (`/raincloud-add-dataset`, `/raincloud-add-handler`, `/raincloud-debug-build`, …). See [`skills/README.md`](skills/README.md). |
+| [`context/`](context/) | Symlinks back to the repo-root canonical docs (`AGENTS.md`, `SKILLS.md`, `README.md`, `sources.schema.md`) so each `SKILL.md` can pull authoritative guidance via a stable relative path without copying. |
+| `scheduled_tasks.lock` | Gitignored — agent-runtime state. |
+
+## Where to start (fresh-clone agent)
+
+1. Read [`../AGENTS.md`](../AGENTS.md) (auto-loaded via `../CLAUDE.md → AGENTS.md` symlink) for the invariants.
+2. Run `python -m scripts.pipeline.status --fast --missing-only` to verify the env.
+3. Run `python -m scripts.pipeline.validate_manifest` to confirm the manifest is well-formed.
+4. Browse [`skills/README.md`](skills/README.md) for the catalog of invokable commands.
+
+## Adding or editing a skill
+
+See [`skills/README.md`](skills/README.md#adding-or-editing-a-skill) — frontmatter schema, the `disable-model-invocation` flag for destructive skills, and the `allowed-tools` pre-approval pattern.

.agents/context/AGENTS.md

@@ -0,0 +1 @@
+../../AGENTS.md

.agents/context/README.md

@@ -0,0 +1 @@
+../../README.md

.agents/context/SKILLS.md

@@ -0,0 +1 @@
+../../SKILLS.md

.agents/context/sources.schema.md

@@ -0,0 +1 @@
+../../sources.schema.md

.agents/settings.json

@@ -0,0 +1,53 @@
+{
+  "$comment": "Tracked agent settings — safe, read-only defaults shared across fresh clones. settings.local.json sits next to this file (gitignored) and overrides per-machine. Side-effecting pipeline stages (build, fetch, extract, convert, tighten_variant) are intentionally NOT pre-approved here — those skills carry disable-model-invocation: true and confirmation lives in AGENTS.md.",
+  "permissions": {
+    "allow": [
+      "Bash(python -m scripts.pipeline.status *)",
+      "Bash(python -m scripts.pipeline.status)",
+      "Bash(python -m scripts.pipeline.validate_manifest *)",
+      "Bash(python -m scripts.pipeline.validate_manifest)",
+      "Bash(python -m scripts.pipeline.list_datasets *)",
+      "Bash(python -m scripts.pipeline.list_datasets)",
+      "Bash(python -m scripts.pipeline.docs *)",
+      "Bash(python -m scripts.pipeline.docs)",
+      "Bash(python -c *)",
+      "Bash(python3 -c *)",
+      "Bash(.venv/bin/python -c *)",
+      "Bash(.venv/bin/python -m scripts.pipeline.status *)",
+      "Bash(.venv/bin/python -m scripts.pipeline.validate_manifest *)",
+      "Bash(.venv/bin/python -m scripts.pipeline.docs *)",
+      "Bash(uv sync*)",
+      "Bash(uv run python -m scripts.pipeline.status *)",
+      "Bash(uv run python -m scripts.pipeline.validate_manifest *)",
+      "Bash(uv run python -m scripts.pipeline.docs *)",
+      "Bash(git status*)",
+      "Bash(git diff*)",
+      "Bash(git log*)",
+      "Bash(git show*)",
+      "Bash(git branch*)",
+      "Bash(git remote*)",
+      "Bash(git ls-files*)",
+      "Bash(git rev-parse*)",
+      "Bash(git blame*)",
+      "Bash(ls *)",
+      "Bash(wc *)",
+      "Bash(du -sh*)",
+      "Bash(df -h*)",
+      "Bash(file *)",
+      "Bash(head -*)",
+      "Bash(tail -*)",
+      "Bash(jq *)",
+      "WebFetch(domain:github.com)",
+      "WebFetch(domain:api.github.com)",
+      "WebFetch(domain:raw.githubusercontent.com)",
+      "WebFetch(domain:agentskills.io)",
+      "WebFetch(domain:code.claude.com)",
+      "WebFetch(domain:docs.astral.sh)",
+      "WebFetch(domain:duckdb.org)",
+      "WebFetch(domain:arrow.apache.org)",
+      "WebFetch(domain:parquet.apache.org)",
+      "WebFetch(domain:vortex.dev)",
+      "WebSearch"
+    ]
+  }
+}

.agents/skills/README.md

@@ -0,0 +1,65 @@
+# skills
+
+Project-local skills for AI coding agents (Claude Code, plus other tools that follow the [Agent Skills](https://agentskills.io) open standard, e.g. Codex). Each `<skill-name>/SKILL.md` is invokable as `/<skill-name>` and Claude can also load it automatically when its `description` matches the user's intent (unless `disable-model-invocation: true` is set).
+
+`.agents/` is the canonical directory; `.claude → .agents` is a symlink at the repo root, so Claude Code (which reads `.claude/skills/`) and tooling that reads `.agents/skills/` see the same files.
+
+## Script wrappers
+
+Wrappers around `python -m scripts.pipeline.<module>`. Side-effecting ones set `disable-model-invocation: true` so Claude won't auto-trigger destructive work; read-only ones are model-invocable.
+
+| Skill | Wraps | Purpose |
+|---|---|---|
+| `/raincloud-build` | `scripts.pipeline.build` | Full pipeline (fetch → … → convert) for one or more slugs. |
+| `/raincloud-fetch` | `scripts.pipeline.fetch` | Download raw bytes only. |
+| `/raincloud-extract` | `scripts.pipeline.extract` | Unpack archives into `_workdir/`. |
+| `/raincloud-convert` | `scripts.pipeline.convert` | Stage 7 — emit sibling `.vortex` per spec opt-in. |
+| `/raincloud-hydrate` | `scripts.pipeline.hydrate` | Stage 8 (optional, opt-in) — dereference a slug's URL column into a sibling parquet under `parquet-hydrated/`. Side-effecting (outbound HTTP); safety-filter-gated; `disable-model-invocation: true`. |
+| `/raincloud-docs` | `scripts.pipeline.docs` | Regenerate derived docs. *(model-invocable — regen is mostly idempotent.)* |
+| `/raincloud-tighten-variant` | `scripts.pipeline.tighten_variant` | In-place JSON → VARIANT promotion. |
+| `/raincloud-status` | `scripts.pipeline.status` | Per-slug filesystem state (raw / workdir / parquet / vortex / variant-pending). *(read-only, model-invocable.)* |
+| `/raincloud-validate-manifest` | `scripts.pipeline.validate_manifest` | Static checks for `sources.json` — JSON Schema + handler-registry / slug-uniqueness / fetch-auth cross-checks. *(read-only, model-invocable.)* |
+| `/raincloud-list-datasets` | `scripts.pipeline.list_datasets` | Filter/list slugs by family / handler / license / fetch-type / reader / vortex / regex. *(read-only, model-invocable.)* |
+
+## Procedural playbooks (model-invocable)
+
+These guide multi-step procedures from [`SKILLS.md`](../context/SKILLS.md). Default frontmatter — Claude can pull them up automatically when the user's request matches.
+
+| Skill | When to use |
+|---|---|
+| `/raincloud-add-dataset` | Adding a new dataset to `sources.json` and producing its first build. |
+| `/raincloud-add-handler` | Writing a new transform handler under `scripts/pipeline/handlers/`. |
+| `/raincloud-add-kaggle-tos` | Adding a Kaggle dataset gated behind a one-time ToS click-through. |
+| `/raincloud-promote-variant` | Picks the right path (in-place vs new-build) for JSON → VARIANT. |
+| `/raincloud-debug-build` | Diagnostic checklist for a failing build — isolate which stage broke. |
+| `/raincloud-large-build` | Run a memory- or runtime-heavy build safely (caps, nohup, logging). *(side-effecting — `disable-model-invocation: true`.)* |
+| `/raincloud-remove-dataset` | Remove a dataset from the manifest and clean up its outputs. *(destructive — `disable-model-invocation: true`.)* |
+
+## Supporting context
+
+`../context/` holds symlinks back to the repo-root canonical docs:
+
+- [`AGENTS.md`](../context/AGENTS.md) — invariants and architecture for AI agents.
+- [`SKILLS.md`](../context/SKILLS.md) — playbooks (the source for the procedural skills above).
+- [`README.md`](../context/README.md) — user-facing project overview.
+- [`sources.schema.md`](../context/sources.schema.md) — `sources.json` schema reference.
+
+Each `SKILL.md` references these via relative paths (`../../context/X.md`) so the agent pulls authoritative guidance without copying.
+
+## Adding or editing a skill
+
+```text
+my-skill/
+├── SKILL.md           # required — frontmatter + instructions
+├── reference.md       # optional — detailed reference loaded only when needed
+└── scripts/           # optional — bundled scripts the skill can execute
+    └── helper.py
+```
+
+Reference: <https://code.claude.com/docs/en/skills>. Frontmatter fields used here:
+
+- `name` — slug; matches the directory name.
+- `description` — front-load the key use case (truncated at 1,536 chars in the listing).
+- `argument-hint` — autocomplete hint for `/<skill> <args>`.
+- `disable-model-invocation` — `true` for side-effecting skills so Claude won't auto-trigger them.
+- `allowed-tools` — pre-approve specific tool patterns when the skill is active (e.g. `Bash(python -m scripts.pipeline.build *)`).

.agents/skills/raincloud-add-dataset/SKILL.md

@@ -0,0 +1,45 @@
+---
+name: raincloud-add-dataset
+description: Walk through adding a new dataset to sources.json and producing its first build. Use when the user wants to onboard a new upstream source, add a slug to the manifest, or extend the catalog.
+argument-hint: [<proposed-slug>] [<upstream-url>]
+---
+
+Guide the user through adding a new dataset entry. Reference: [SKILLS.md "Adding a new dataset"](../../context/SKILLS.md#adding-a-new-dataset), [sources.schema.md](../../context/sources.schema.md).
+
+Steps:
+
+1. **Identify the upstream.** Confirm with the user:
+   - A stable public URL (prefer the publisher's canonical endpoint over a mirror).
+   - The license — must permit redistribution-of-derivatives. Check SPDX ID and `source_url`.
+   - Approximate row count (used for `expect.rows`; can be `null` on first build).
+
+2. **Append a `DatasetSpec` to `sources.json`** using the Python load-edit-dump pattern from [AGENTS.md](../../context/AGENTS.md#safe-ways-to-edit-sourcesjson) — never `sed`. Start from [`examples/minimal_spec.json`](../../../examples/minimal_spec.json) (every field present with placeholder values) rather than typing one from scratch. Minimal direct-HTTP shape:
+
+   ```jsonc
+   {
+     "slug": "my-dataset",
+     "short_name": "My Dataset",
+     "full_name": "My Dataset (publisher attribution)",
+     "description": "One-line summary.",
+     "family": "direct",
+     "license": { "spdx": "CC0-1.0", "source_url": "...", "redistribution_permitted": true, "attribution_required": false },
+     "fetch":     { "type": "http", "urls": ["https://..."], "auth": null },
+     "extract":   { "type": "passthrough" },
+     "parse":     { "reader": "csv", "options": { "delimiter": "," } },
+     "transform": { "handler": "tighten_types", "params": {} },
+     "write":     { "output": "my-dataset.parquet", "compression": "zstd", "row_group_size_rows": 1048576, "statistics": true },
+     "expect":    { "rows": 123456 }
+   }
+   ```
+
+   Pick `family` from existing values (`direct`, `kaggle-upstream`, `nyc-tlc`, `public-bi`, `uci`) — do not invent new ones without discussing.
+
+3. **Validate the manifest.** Invoke `/raincloud-validate-manifest` — sub-second check that the new entry has the right shape, the handler resolves, the slug is unique, and `fetch.type`/`fetch.auth` agree. Catches typos before paying for a fetch.
+
+4. **Run the first build.** Invoke `/raincloud-build <slug> --loose` (the `--loose` is essential when the row count is a guess). If `expect.rows` was wrong, update the manifest with the actual count once the build succeeds.
+
+5. **Regenerate docs.** Invoke `/raincloud-docs` (or just `/raincloud-docs datasets columns_parquet coverage_parquet`).
+
+6. **Optionally opt into Vortex.** If the dataset's types are vortex-compatible, add `"convert": { "vortex": true }` to the spec and run `/raincloud-convert <slug>`.
+
+If the upstream needs unpacking, set `extract.type` accordingly and pick a parser — see existing specs in `sources.json` for shapes. If the source has nested JSON or row-level processing, you'll likely need a custom handler — see `/raincloud-add-handler`. If it's a Kaggle dataset behind ToS acceptance, see `/raincloud-add-kaggle-tos`.

.agents/skills/raincloud-add-handler/SKILL.md

@@ -0,0 +1,43 @@
+---
+name: raincloud-add-handler
+description: Walk through writing a new transform handler under scripts/pipeline/handlers/. Use when the default tighten_types/identity paths can't produce the right shape — row-level JSON parsing, streaming to avoid OOM, multi-output splitting, or VARIANT-from-the-start.
+argument-hint: [<handler-name>]
+---
+
+Guide the user through adding a transform handler. Reference: [SKILLS.md "Adding a new transform handler"](../../context/SKILLS.md#adding-a-new-transform-handler) and ["Writing a streaming handler"](../../context/SKILLS.md#writing-a-streaming-handler).
+
+When this is the right pattern: the default `tighten_types` / `identity` paths can't produce the right shape — e.g. the source needs row-level JSON parsing, streaming to avoid OOM, multi-output splitting (one upstream → many slugs), or VARIANT columns emitted from the start.
+
+Steps:
+
+1. **Create the handler** at `scripts/pipeline/handlers/<name>.py`. Signature:
+
+   ```python
+   def <name>(spec: dict, parsed: list[tuple[Path, pa.Table | None]], **params
+              ) -> list[tuple[str, pa.Table]]:
+       ...
+   ```
+
+   - `parsed` contains one `(path, table)` tuple per parsed file; `table` is `None` when `parse.reader = "custom"`.
+   - Return `[(output_slug, table), ...]` — one tuple per output parquet. Multi-output handlers emit several slugs from one source (see `glove_split`, `osm_pbf_split`, `stack_exchange_split`).
+   - **Streaming handlers** (write direct to parquet, bypass the write stage) return `[]`. Copy [`examples/streaming_handler.py.tmpl`](../../../examples/streaming_handler.py.tmpl) as the starting point — it has the `outputs_root()` / `duckdb_connect()` / cleanup wiring already shaped — and study `factbook_variant_parse`, `wikipedia_variant_parse`, `lichess_pgn_parse` for upstream-shape variations.
+
+2. **Register** in `scripts/pipeline/handlers/__init__.py`:
+
+   ```python
+   from .<name> import <name>
+   _REGISTRY = {
+       ...
+       "<name>": <name>,
+   }
+   ```
+
+3. **Wire it into the manifest** — set `"transform": { "handler": "<name>", "params": { ... } }` on the relevant spec.
+
+4. **Run the test suite** — `pytest` checks that every handler in `_REGISTRY` imports and that there are no orphans (registered but unreferenced) or unregistered handler names in the manifest. Sub-second; catches typos before paying for a fetch.
+
+5. **Build the dataset** — invoke `/raincloud-build <slug> --loose` to validate end-to-end.
+
+Style: handlers stay short (most under 150 lines). Reuse `duckdb_connect`, `outputs_root`, `spec_field` from `scripts.pipeline.spec` rather than reimplementing. Don't shoehorn a new shape into `tighten_types` or `identity` — write a dedicated handler.
+
+For VARIANT-from-the-start handlers, use DuckDB's `CAST(to_json(col) AS VARIANT)` inside `COPY ... TO PARQUET`. `duckdb_connect(db_path)` applies `storage_compatibility_version=v1.5.0` automatically. See `factbook_variant_parse` (1-column) and `wikipedia_variant_parse` (multi-column with typed siblings).

.agents/skills/raincloud-add-kaggle-tos/SKILL.md

@@ -0,0 +1,38 @@
+---
+name: raincloud-add-kaggle-tos
+description: Add a Kaggle dataset gated behind a one-time ToS click-through. Use when a Kaggle URL returns 403 from the API and the user needs the manifest+workflow pattern that documents the click-through gate cleanly.
+argument-hint: [<proposed-slug>] [<kaggle-url>]
+---
+
+Guide the user through adding a Kaggle dataset that requires a one-time click-through ToS acceptance on the Kaggle web UI before the API will serve downloads. Reference: [SKILLS.md "Adding a Kaggle dataset gated behind ToS acceptance"](../../context/SKILLS.md#adding-a-kaggle-dataset-gated-behind-tos-acceptance).
+
+**Diagnostic note:** an HTTP 403 from a Kaggle fetch can also indicate a wrong slug. Double-check the Kaggle URL before reaching for this pattern.
+
+Steps:
+
+1. **Add the spec** as a normal `fetch.type: "kaggle"` entry but mark it as gated and leave `expect.rows: null`:
+
+   ```jsonc
+   "fetch": {
+     "type": "kaggle",
+     "urls": ["https://www.kaggle.com/datasets/<owner>/<dataset>"],
+     "auth": "kaggle",
+     "requires_interactive_accept": true,
+     "notes": "Kaggle gates this dataset behind a one-time click-through ToS acceptance."
+   },
+   "expect": { "rows": null, "notes": "Row count populated after the first successful build." }
+   ```
+
+   Use the [Python load-edit-dump pattern](../../context/AGENTS.md#safe-ways-to-edit-sourcesjson) — never `sed`.
+
+2. **Confirm Kaggle creds are set up:** `~/.kaggle/kaggle.json` with `chmod 600`, and the project synced via `uv sync --extra kaggle`.
+
+3. **Try the first build via `/raincloud-build <slug> --loose`.** Pre-flight will print `kaggle (ToS-gated): ...`. Expect a 403 on the first try. The error message will point at the exact Kaggle URL the user must visit.
+
+4. **Tell the user to click Download once** in a browser signed into Kaggle. The 403 handler is generic — it triggers whether or not `requires_interactive_accept` is set, so forgetting the flag still yields a useful error; the flag only improves the up-front announcement.
+
+5. **Re-run `/raincloud-build <slug> --loose`.** It should succeed.
+
+6. **Update `expect.rows`** in the manifest with the actual count and **drop `expect.notes`** (now that the row count is known).
+
+7. **Regenerate docs** via `/raincloud-docs`.