docs: add the journey components, page templates, lint scripts and the llms.txt generator

Author: amankrx

.github/styles/config/vocabularies/TraceMachina/accept.txt

@@ -318,3 +318,11 @@ subtree
 hardlink
 multiplicatively
 SELinux
+repoint
+inlines
+permalinked
+gitignored
+inlined
+protobufs
+arg
+rustfmt

.gitignore

@@ -2,6 +2,7 @@ bazel-*
 !tools/bazel-retry.sh
 target/
 nativelink-test/fuzz/target/
+.turbo/
 .vscode/
 .idea/
 .zed

AGENTS.md

@@ -0,0 +1,168 @@
+# AGENTS.md
+
+A machine-readable map of the NativeLink repository for AI coding agents and new
+contributors. NativeLink is a high-performance remote build cache and execution
+platform (Remote Execution API), written in Rust.
+
+Full documentation: https://docs.nativelink.com. Two entry points for machine
+readers, both generated from the docs navigation so neither can drift from the
+sidebar: https://docs.nativelink.com/llms.txt is the link index, one line per
+page in reading order, and https://docs.nativelink.com/llms-full.txt is the same
+corpus with page bodies inlined. Human contributor guide:
+[`CONTRIBUTING.md`](CONTRIBUTING.md).
+
+## The four roles
+
+NativeLink is four cooperating roles that speak the Remote Execution API:
+
+- **CAS**: content-addressable storage for build inputs/outputs, keyed by digest.
+- **Action Cache (AC)**: maps an action's digest to its cached result.
+- **Scheduler**: queues actions and matches them to workers by platform property.
+- **Workers**: execute actions and stream results back to the CAS.
+
+A single binary can run any combination of these, configured in JSON5.
+
+## Crate map (where things live)
+
+| Crate | Owns |
+| ----- | ---- |
+| `nativelink` (root, `src/bin/nativelink.rs`) | The server binary; wires config to running services. |
+| `nativelink-config` | JSON5 config schema. Stores in `src/stores.rs`, server/worker in `src/cas_server.rs`. Source of truth for the config reference. |
+| `nativelink-service` | gRPC services: CAS, AC, Execution, Capabilities, ByteStream, Worker API, BEP, health, fetch/push. |
+| `nativelink-store` | Every store implementation: filesystem, memory, redis, S3/R2/GCS/Azure/OCI/Mongo, and the composition stores (`fast_slow`, `shard`, `size_partitioning`, `compression`, `dedup`, `existence_cache`, `verify`, `ref`). |
+| `nativelink-scheduler` | Scheduler internals: matching engine, awaited-action DB, worker registry, state manager, retries. |
+| `nativelink-worker` | Worker: action execution, input materialization, sandboxing/namespaces, directory cache. |
+| `nativelink-util` | Shared utilities: `fs`/`fs_util`, `evicting_map`, `action_messages`, digests, and the OTLP metrics (`metrics.rs`). |
+| `nativelink-proto` | Generated protobufs (REAPI + NativeLink extensions). |
+| `nativelink-metric` / `nativelink-macro` | The `#[metric]` component-metrics derive system and proc macros. |
+| `nativelink-error` | The `Error` type and `ResultExt`. |
+| `nativelink-test` / `nativelink-redis-tester` | Test harness (`nativelink_test`) and Redis test tooling. |
+
+## Where to change what
+
+- **Add or change a config field**: edit the relevant spec in
+  `nativelink-config/src/stores.rs` or `cas_server.rs`; the config reference is
+  generated from the doc comments, so document the field there.
+- **Add a store type**: implement `StoreDriver` in `nativelink-store/src`, add
+  its spec to `nativelink-config/src/stores.rs`, and wire it in the store factory.
+- **Add or change a gRPC service**: `nativelink-service/src/*_server.rs`.
+- **Change scheduling/matching/retries**: `nativelink-scheduler/src`.
+- **Change action execution / worker behavior**:
+  `nativelink-worker/src/running_actions_manager.rs`.
+- **Add an OTLP metric**: declare and record it in
+  `nativelink-util/src/metrics.rs`; record it at the call site; regenerate the
+  metrics reference. Do not leave a metric declared-but-never-recorded: the
+  generated reference has a column for exactly that, and it will say so.
+
+## Changing X: source of truth, and the doc that must follow
+
+When a change lands, this table says what else has to move with it. The rule is
+that the source of truth is always in the repo, and the doc either regenerates
+from it or cites it, never restates it by hand. Every path in the right-hand
+column is a real page under `web/apps/docs/content/docs/`.
+
+| You changed | Source of truth | Doc that must follow |
+| ----------- | --------------- | -------------------- |
+| A config field or its doc comment | `nativelink-config/src/{stores,cas_server}.rs` | `reference/nativelink-config/*`; regenerate with `gen:config-reference`, never hand-edit |
+| A store's behavior or defaults | `nativelink-store/src/*` | the backend page under `how-to/stores/`, plus `reference/nativelink-config/store-overview` if the composition model moved |
+| A metric name, type, or label | `nativelink-util/src/metrics.rs` and its call sites | `reference/metrics` (regenerate with `gen:metrics-reference`) and `operate/observability` |
+| A CLI flag or `NL_*` env var | the binary's arg parsing | `reference/cli-and-env` |
+| A gRPC service or the set of services a server exposes | `nativelink-service/src/*_server.rs` | `configuration/servers-and-services`, which names every service as the config spells it |
+| Scheduler matching or platform-property semantics | `nativelink-scheduler/src` | `remote-execution/platform-properties`, and `explanations/architecture` if the model changed rather than the mechanics |
+| Worker sandboxing or input materialization | `nativelink-worker/src/running_actions_manager.rs` | `explanations/architecture` and `operate/security-hardening`; the second one states what the sandbox is *not* |
+| The licence header on a file, or `LICENSE` | the headers themselves | `reference/oss-and-enterprise`, which is the only place the licence split is explained |
+| The canonical production config | `deployment-examples/`, `nativelink-config/examples/` | `operate/production-config`; the snippets are lifted from there, not invented |
+| A page's URL | `web/apps/docs/content/docs/**/meta.json` | add a redirect in `web/apps/docs/next.config.mjs` |
+
+## Build, test, verify
+
+NativeLink builds with both Bazel and Cargo.
+
+```bash
+bazel test //...                                  # all tests (first run 10-20 min)
+bazel test //nativelink-store/tests:s3_store_test # one target
+bazel build //nativelink:nativelink               # the server binary
+cargo test -p nativelink-store                    # a single crate with cargo
+```
+
+Run the built server against a config:
+
+```bash
+bazel run //nativelink:nativelink -- ./path/to/config.json5
+```
+
+Example configs live in `nativelink-config/examples/` and runnable deployments in
+`deployment-examples/` (docker-compose, including a multi-worker set) and
+`integration_tests/`.
+
+## The docs, and their gates
+
+The docs site is a [Fumadocs](https://fumadocs.dev) app at `web/apps/docs`. Pages
+are MDX under `content/docs/`, and `meta.json` in each directory controls both the
+sidebar order and which pages are published.
+
+```bash
+cd web && bun install
+bun run --filter docs dev     # local docs server; regenerates first
+bun run --filter docs build   # what CI builds
+bunx biome check --write .    # lint and format
+
+bun --filter @nativelink/docs lint:snippets  # JSON5 snippets against the config schema
+bun --filter @nativelink/docs lint:anchors   # explicit anchors on headings and FAQ entries
+```
+
+Six things a docs change has to satisfy:
+
+- **Vale and typos** run in pre-commit. New product nouns go in
+  `.github/styles/config/vocabularies/TraceMachina/accept.txt`.
+- **Biome** formats and lints the TypeScript and the MDX components.
+- **Every nav entry resolves.** A `meta.json` entry naming a page that doesn't
+  exist is a broken sidebar link, and `gen:llms` fails loudly on one.
+- **Moved URLs redirect.** Agents cache URLs longer than humans keep bookmarks;
+  a 404 on an old path is a regression.
+- **`lint:snippets` passes.** Every key in a JSON5 config snippet has to exist in
+  the generated config reference. A snippet that is deliberately wrong (showing
+  a mistake, or a foreign tool's config) opts out with
+  `{/* lint-snippets: ignore */}` above it.
+- **`lint:anchors` passes.** Every heading and every `<Accordion>` carries an
+  explicit anchor, so rewording a heading cannot silently repoint a citation.
+  `lint:anchors --fix` writes the anchor Fumadocs would have derived anyway,
+  which means running it never moves an existing link.
+
+Four things under `web/apps/docs` are generated. Regenerate them; never hand-edit:
+
+| Generated | From | Command |
+| --------- | ---- | ------- |
+| `content/docs/reference/nativelink-config/*` | the `nativelink-config` crate, via `build-schema` | `gen:config-reference` |
+| `content/docs/reference/metrics.mdx` | `nativelink-util/src/metrics.rs` and its call sites | `gen:metrics-reference` |
+| `content/docs/reference/changelog.md` | the repository-root `CHANGELOG.md` | `gen:changelog` |
+| `public/llms.txt`, `public/llms-full.txt` | the navigation and page frontmatter | `gen:llms` |
+
+The last two are gitignored and rebuilt by `dev` and `build`, so they cannot be
+committed in a stale state.
+
+Page structure follows four archetypes (tutorial, how-to, explanation,
+reference), with a template for each in `web/apps/docs/templates/`. The
+conventions those templates encode:
+
+- A narrative page never inlines an exhaustive field list. It explains the
+  fields that carry a decision and links the generated reference for the rest.
+- Pages on the reading path (Getting started, Remote execution, Configuration,
+  How-to guides, plus Why NativeLink before and Operate after) open with
+  `<Prerequisites>`, so a reader landing cold from a search knows what the page
+  assumes.
+- Tutorials and how-tos close with `<VerifyBlock>`: a checkable claim, not "it
+  should work now".
+- Behavioural claims carry `<SourceLink>`, permalinked to a release tag rather
+  than to `main`. The pinned ref lives in `web/apps/docs/lib/source-ref.ts`; bump
+  it in the same change that regenerates the reference for a new release.
+- Headings carry an explicit `[#anchor]`, and `<Accordion>` entries an `id`, so
+  a citation to a specific claim keeps resolving after the prose is reworded.
+
+## Conventions
+
+- Pre-commit runs rustfmt, `typos`, and (for docs) `vale`. Write to pass them.
+- The config reference under `web/apps/docs/content/docs/reference/nativelink-config`
+  is autogenerated via the `build-schema` binary; regenerate, never hand-edit.
+- Prefer generating docs/reference from a code source of truth over hand-writing,
+  to prevent drift.

web/apps/docs/.gitignore

@@ -8,3 +8,9 @@ next-env.d.ts
 # Generated from the repository-root CHANGELOG.md by scripts/gen-changelog.mjs
 # (runs as part of `dev` and `build`).
 content/docs/reference/changelog.md
+
+# Generated from the navigation and page frontmatter by scripts/gen-llms.mjs
+# (runs as part of `dev` and `build`). Committing them would create a second
+# source of truth for the sidebar, which is the thing they exist to prevent.
+public/llms.txt
+public/llms-full.txt

web/apps/docs/app/layout.tsx

@@ -12,9 +12,9 @@ import "./globals.css";
 export const metadata: Metadata = {
   title: {
     default: "NativeLink Docs",
-    template: "%s — NativeLink Docs",
+    template: "%s | NativeLink Docs",
   },
-  description: "Documentation for NativeLink — high-performance remote build cache & execution.",
+  description: "Documentation for NativeLink, a high-performance remote build cache and execution service.",
   metadataBase: new URL("https://docs.nativelink.com"),
 };
 

web/apps/docs/components/min-version.tsx

@@ -0,0 +1,31 @@
+import { cn } from "@nativelink/ui";
+
+interface MinVersionProps {
+  /** The first release in which this works, without the leading `v`. */
+  v: string;
+  className?: string;
+}
+
+/**
+ * Inline "needs at least this release" badge.
+ *
+ * Put one next to any field, flag, or command that does not exist in every
+ * supported release. Readers land on these pages from search engines and from
+ * agent caches, on whatever version they happen to be running, so "which
+ * version is this?" has to be answerable on the page itself.
+ */
+export function MinVersion({ v, className }: MinVersionProps) {
+  return (
+    <span
+      data-min-version={v}
+      className={cn(
+        "ml-1 inline-flex items-center rounded-full border border-brand/40",
+        "bg-brand-soft/40 px-2 py-0.5 align-middle font-mono text-[0.7rem]",
+        "font-semibold text-brand",
+        className,
+      )}
+    >
+      {v}+
+    </span>
+  );
+}

web/apps/docs/components/next-step.tsx

@@ -0,0 +1,80 @@
+import { cn } from "@nativelink/ui";
+import type * as React from "react";
+
+interface NextStepProps {
+  /** Where the reader goes next. */
+  href: string;
+  /** The destination's name, as the reader will see it in the sidebar. */
+  title: string;
+  /**
+   * `next` continues along the reading path; `aside` is a useful detour that does
+   * not advance the reader's position. Rendering them differently keeps the
+   * main path obvious when a page offers both.
+   */
+  kind?: "next" | "aside";
+  /** Why the reader would go there: one sentence, not a description. */
+  children: React.ReactNode;
+  className?: string;
+}
+
+/**
+ * The journey handoff.
+ *
+ * A page that ends without telling the reader where to go next has dumped
+ * information on them. This component renders that handoff the same way on
+ * every page, so the path through the docs is visible rather than implied.
+ */
+export function NextStep({
+  href,
+  title,
+  kind = "next",
+  children,
+  className,
+}: NextStepProps) {
+  const isNext = kind === "next";
+  return (
+    <a
+      href={href}
+      data-next-step={kind}
+      className={cn(
+        "my-4 flex items-start gap-3 rounded-xl border px-5 py-4 no-underline",
+        "transition-colors",
+        isNext
+          ? "border-brand/40 bg-brand-soft/30 hover:border-brand/70"
+          : "border-border bg-surface-elevated hover:border-border-strong",
+        className,
+      )}
+    >
+      <svg
+        width="18"
+        height="18"
+        viewBox="0 0 24 24"
+        fill="none"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+        className={cn(
+          "mt-1 shrink-0",
+          isNext ? "text-brand" : "text-muted-foreground",
+        )}
+        aria-hidden="true"
+      >
+        {isNext ? (
+          <path d="M5 12h14m-6-6 6 6-6 6" />
+        ) : (
+          <path d="M5 12h14m-6-6 6 6-6 6" opacity="0.55" />
+        )}
+      </svg>
+      <span className="flex-1">
+        <span className="block font-mono text-[0.7rem] font-semibold uppercase tracking-[0.12em] text-muted-foreground">
+          {isNext ? "Next" : "Sideways"}
+        </span>
+        <span className="block font-semibold text-foreground">{title}</span>
+        <span className="mt-0.5 block text-sm text-muted-foreground">
+          {children}
+        </span>
+      </span>
+    </a>
+  );
+}

web/apps/docs/components/prerequisites.tsx

@@ -0,0 +1,38 @@
+import { cn } from "@nativelink/ui";
+import type { ReactNode } from "react";
+
+interface PrerequisitesProps {
+  /**
+   * What the page assumes the reader already has, written as the state they
+   * are in rather than the page they read: "a cache serving hits", not "the
+   * quickstart". Link to the page that gets them there when there is one.
+   */
+  children: ReactNode;
+  className?: string;
+}
+
+/**
+ * States what a page assumes before its first step.
+ *
+ * Readers arrive mid-corpus from search and from agent caches, not only from
+ * the page before. A page in an ordered section therefore has to say what it
+ * assumes. Without that, a reader who lands on remote execution with no
+ * working cache follows correct instructions to a broken result and blames
+ * the product.
+ */
+export function Prerequisites({ children, className }: PrerequisitesProps) {
+  return (
+    <div
+      data-prerequisites
+      className={cn(
+        "my-6 rounded-xl border border-border bg-surface-elevated px-5 py-3",
+        className,
+      )}
+    >
+      <p className="my-0 font-mono text-xs font-semibold uppercase tracking-[0.12em] text-brand">
+        Before you start
+      </p>
+      <div className="mt-1 text-sm text-muted-foreground [&>p]:my-0">{children}</div>
+    </div>
+  );
+}