feat: cellar — a searchable archive of Meta's shipped client source

cellar downloads a WhatsApp Web (or Messenger, Facebook, Instagram) client
revision, parses it with the oxc AST, and writes one file per module named after
the module — so the archive is something `grep -r` and an ordinary editor work on
directly. On top of that it offers bundle management, filtered diffs between
revisions, module lookup and search, and dependency/dependent graphs, from a CLI
and from an MCP server sharing one operations layer.

Two questions motivate it, neither answerable by grepping the bundle:

  - "What uses this module?" Minified call sites reference dependencies
    positionally (d[3]), never by name, so the only record of the reverse edge is
    the dependency arrays, which the indexer inverts.
  - "What changed since last release?" Meta serves only the current bundle and the
    URLs 404 once a rollout moves on. Historical revisions come from btarchive,
    which is what makes release-to-release diffing possible at all.

Design notes, each of which is a mistake made and corrected against real bundles:

  - AST, not regex. Brace-counting through string, template and regex literals
    silently truncates modules, and a truncated module reads as a spurious diff on
    the next revision. Exports resolve through the factory's actual parameter
    binding, so a minified `q.sendStanza = …` is found where `e\.(\w+)=` is not.
  - Deterministic. Sorted lists, BTreeMaps, and a serial filename-assignment pass
    over sorted names behind the parallel parse, so re-indexing a bundle produces
    byte-identical JSON.
  - Diagnostics, not silence. Parse failures, unresolved dependencies and skipped
    diffs are counted in manifest.json rather than dropped, so "no such module" and
    "we failed to extract it" never look alike.
  - Sources are re-printed from the AST. A module shipped on one 200 KB line makes
    a greppable directory useless and a line diff meaningless; byte identity is
    kept separately as rawSha256 so change detection stays exact.
  - Variants are kept. About a quarter of modules ship more than one distinct
    definition; all are written out and change detection hashes the whole set,
    since otherwise a new build variant reads as no change and a shift in which
    copy won reads as a change that never happened.
  - btarchive needs a self-consistent browser request: Meta's edge rejects a
    Chrome User-Agent that omits Sec-Fetch-* with a 400. NAVIGATION_HEADERS records
    the measured status codes.

Prior art, credited in the README: ProtoCocktail's wa-diff-analyzer, whose
empirically-accumulated module filter ruleset is ported here as the `default`
filter; whatspec, which solves the adjacent problem of extracting a typed
protocol IR from the same bundles and whose approach shaped this one; and Meta's
own meta-code-verify, which documents the btarchive endpoint in practice.

104 tests. Builds on stable; CI runs fmt, clippy -D warnings, the suite, an MSRV
job, a guard that no test reaches the network, and a check that the Claude and
Codex skills remain one file.

Author: Rajeh Taher

.claude/skills/cellar/SKILL.md

@@ -0,0 +1,136 @@
+---
+name: cellar
+description: Look things up in Meta's shipped client source (WhatsApp Web, Messenger, Facebook, Instagram) — find where a feature is implemented, discover unreleased features by diffing two client revisions, read any module's source, and trace what depends on what. Use when asked about WhatsApp Web internals, a WA protocol detail, what changed between two WhatsApp releases, where some behaviour lives in the client, or any question answerable from Meta's minified bundle. Triggers on "WhatsApp Web source", "what changed in this revision", "how does WhatsApp implement X", "which module does Y", "btarchive", "bundle diff", "WAWeb...", "WASmax...", "WAM event".
+---
+
+# cellar
+
+`cellar` keeps an archive of Meta's shipped client bundles, one directory per
+client revision, exploded so that **every module is its own ordinary file**. That
+last part is the point: once a query points you at a module, you can read and grep
+those files with your normal tools instead of going back through the CLI.
+
+## Before anything else
+
+```bash
+cellar bundle list
+```
+
+Nothing in the archive means nothing to search. Adding a bundle takes several
+minutes and gigabytes — say so before starting one, and never start two at once.
+
+```bash
+cellar bundle add --rev latest          # fetch and index the current revision
+cellar bundle add --rev 1030882912      # a specific historical revision
+```
+
+Meta serves historical revisions from `btarchive`, which is what makes
+release-to-release diffing possible at all — the live bundle URLs 404 the moment a
+rollout moves on.
+
+## Pick the right question
+
+**"Where is X implemented?"** — search, then read the files.
+
+```bash
+cellar module search --source 'addon_type|addonType' --name '^WAWeb' -C 2
+cellar grep 'disappearing_mode' --filter default
+```
+
+Narrow with `--name` whenever you can. It cuts the candidate set in the index
+before any file is opened, which is the difference between opening forty files and
+a hundred thousand. Then read the `path` each hit reports with your own file tools.
+
+**"What changed in this release?"** — diff, overview first.
+
+```bash
+cellar diff --no-hunks                       # the two newest stored revisions
+cellar diff whatsapp-1030882912 whatsapp-1030947950 --format json
+```
+
+Read the summary before the changes. `added` and `removed` modules are where new
+features appear; `modified` with new dependencies or new exports is where existing
+ones grow. Changes that are pure transpiler churn are counted as `noiseOnly` and
+withheld — pass `--include-noise` only if you suspect the noise rules are wrong.
+
+**"What uses this?"** — the graph, not grep.
+
+```bash
+cellar graph WAWebSendMsgStanza --direction dependents --depth 2
+cellar graph --match '^WAWebNewsletter' --direction deps --depth 1 --format mermaid
+```
+
+Grep genuinely cannot answer this. Minified call sites reference modules
+positionally (`d[3]`), never by name, so the only record of "who uses this" is the
+dependency arrays the indexer inverted.
+
+**"Show me this module."**
+
+```bash
+cellar module show WAWebSendMsgStanza
+cellar module show WAWebSendMsgStanza --path-only    # then read it directly
+```
+
+## Filters decide what you are looking at
+
+A bundle holds ~170k modules, most of them React components and code for the other
+Meta products that share the bundle. Every operation takes `--filter`:
+
+| filter | keeps | use for |
+| --- | --- | --- |
+| `default` | ~11% — the protocol surface | diffing releases; the default |
+| `all` | everything | when you suspect the filter hid your answer |
+| `protocol` | only positively-identified protocol modules | a tight, high-signal diff |
+| `schemas` | `.pb` / `.proto` / `.graphql` | new protobuf fields, new persisted queries |
+| `wam` | WAM analytics events and enums | earliest traces of unreleased features |
+
+`default` **hard-excludes** `.pb` and `.graphql` modules. If you are looking for a
+new protobuf field, use `--filter schemas` — otherwise you will conclude, wrongly,
+that nothing changed.
+
+Check an unfamiliar filter before trusting it:
+
+```bash
+cellar filter test default --bundle latest
+```
+
+Built-ins are read-only. To adjust one, fork it:
+
+```bash
+cellar filter fork default my-filter
+cellar filter show my-filter > /tmp/f.json     # edit, then:
+cellar filter set my-filter --from /tmp/f.json
+```
+
+Precedence is `hardExclude`, then `include`, then `exclude`; anything unmatched
+takes `defaultVerdict`. With `excludeDependentsOfExcluded`, a module that depends
+on an excluded one is excluded too — unless `include` claimed it.
+
+## Reading results honestly
+
+- **Diagnostics are not decoration.** `cellar bundle info <bundle>` reports parse
+  failures and unresolved dependencies. A surprising diff with non-zero
+  `chunkParseFailures` means modules are genuinely missing from the index, not that
+  they were removed from the client.
+- **Absent is not zero.** Truncated results say `truncated: true`; a skipped diff
+  says `hunksOmitted` and why. Never read a short list as a complete one.
+- **Variants are real.** About a quarter of modules ship in more than one form.
+  `variants` lists the others, written beside the primary as
+  `<name>~alt-<hash>.js`. A variant appearing or disappearing between revisions is
+  itself a rollout signal.
+- **Stored sources are re-printed, not byte-exact.** Modules are pretty-printed so
+  grep and diff work line by line. Identity is tracked separately by `rawSha256`,
+  so change detection is still exact.
+
+## MCP
+
+The same operations are available as MCP tools (`cellar mcp`), named
+`bundle_*`, `filter_*`, `module_get`, `module_search`, `diff`, `graph`. Prefer them
+when available; fall back to the CLI otherwise. Both return identical JSON.
+
+## Reporting findings
+
+Cite the module name and its path, and quote the lines you are relying on. "New
+`WAWebFooBar` module, `modules/WAWebFooBar.js:41`, sends `<iq type='set'
+xmlns='w:foo'>`" is a finding. "It looks like they added foo support" is a guess —
+if that is all you have, say so.

.codex/skills/cellar

@@ -0,0 +1 @@
+../../.claude/skills/cellar

.github/workflows/ci.yml

@@ -0,0 +1,78 @@
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  CARGO_TERM_COLOR: always
+  CARGO_INCREMENTAL: 0
+
+jobs:
+  check:
+    name: fmt + clippy + test
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    # Read-only: the job builds and tests, and `actions/checkout` writes the token
+    # into .git/config where every dependency build script could read it.
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          # Stable, deliberately: nothing here needs nightly, and requiring it
+          # would be an adoption barrier for a library other people build on.
+          toolchain: stable
+          components: rustfmt, clippy
+
+      - uses: Swatinem/rust-cache@v2
+
+      - run: cargo fmt --all -- --check
+
+      - run: cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
+
+      - run: cargo test --workspace --locked
+
+      # No test may reach the network: `bundle add` is the only operation that
+      # does, and it is exercised against local fixtures. This asserts the
+      # endpoints stay confined to cellar-fetch, so CI can never start depending
+      # on Meta's uptime.
+      - name: no network outside cellar-fetch
+        run: |
+          if grep -rn 'facebook\.com\|web\.whatsapp\.com' \
+              --include='*.rs' crates/*/src \
+              | grep -v '^crates/cellar-fetch/src/lib.rs' \
+              | grep -v 'archive_url_is_revision_and_platform'; then
+            echo "network endpoints must stay inside cellar-fetch"
+            exit 1
+          fi
+
+      # The two agent skills are one file behind a symlink; a copy would drift.
+      - name: skill is a single source of truth
+        run: |
+          test -L .codex/skills/cellar
+          diff -q .claude/skills/cellar/SKILL.md .codex/skills/cellar/SKILL.md
+
+  msrv:
+    name: msrv (1.95)
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          toolchain: "1.95"
+      - uses: Swatinem/rust-cache@v2
+        with:
+          shared-key: msrv
+      # The declared floor is a promise; check it instead of hoping.
+      - run: cargo check --workspace --all-targets --locked

.gitignore

@@ -0,0 +1,25 @@
+# Build artifacts
+target/
+**/*.rs.bk
+*.pdb
+
+# A local archive, if someone points CELLAR_HOME at the checkout. Bundles run to
+# gigabytes and must never be committed.
+.cellar/
+bundles/
+
+# Profiling
+*.profraw
+*.profdata
+flamegraph.svg
+perf.data*
+
+# IDE / editor
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db