diff --git a/.github/styles/config/vocabularies/TraceMachina/accept.txt b/.github/styles/config/vocabularies/TraceMachina/accept.txt index b6e5b8201..a5ad239b4 100644 --- a/.github/styles/config/vocabularies/TraceMachina/accept.txt +++ b/.github/styles/config/vocabularies/TraceMachina/accept.txt @@ -318,3 +318,11 @@ subtree hardlink multiplicatively SELinux +repoint +inlines +permalinked +gitignored +inlined +protobufs +arg +rustfmt diff --git a/.gitignore b/.gitignore index a2f94dc0c..fa8b9fbad 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,7 @@ bazel-* !tools/bazel-retry.sh target/ nativelink-test/fuzz/target/ +.turbo/ .vscode/ .idea/ .zed diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..baecdc095 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,168 @@ +# AGENTS.md + +A machine-readable map of the NativeLink repository for AI coding agents and new +contributors. NativeLink is a high-performance remote build cache and execution +platform (Remote Execution API), written in Rust. + +Full documentation: https://docs.nativelink.com. Two entry points for machine +readers, both generated from the docs navigation so neither can drift from the +sidebar: https://docs.nativelink.com/llms.txt is the link index, one line per +page in reading order, and https://docs.nativelink.com/llms-full.txt is the same +corpus with page bodies inlined. Human contributor guide: +[`CONTRIBUTING.md`](CONTRIBUTING.md). + +## The four roles + +NativeLink is four cooperating roles that speak the Remote Execution API: + +- **CAS**: content-addressable storage for build inputs/outputs, keyed by digest. +- **Action Cache (AC)**: maps an action's digest to its cached result. +- **Scheduler**: queues actions and matches them to workers by platform property. +- **Workers**: execute actions and stream results back to the CAS. + +A single binary can run any combination of these, configured in JSON5. + +## Crate map (where things live) + +| Crate | Owns | +| ----- | ---- | +| `nativelink` (root, `src/bin/nativelink.rs`) | The server binary; wires config to running services. | +| `nativelink-config` | JSON5 config schema. Stores in `src/stores.rs`, server/worker in `src/cas_server.rs`. Source of truth for the config reference. | +| `nativelink-service` | gRPC services: CAS, AC, Execution, Capabilities, ByteStream, Worker API, BEP, health, fetch/push. | +| `nativelink-store` | Every store implementation: filesystem, memory, redis, S3/R2/GCS/Azure/OCI/Mongo, and the composition stores (`fast_slow`, `shard`, `size_partitioning`, `compression`, `dedup`, `existence_cache`, `verify`, `ref`). | +| `nativelink-scheduler` | Scheduler internals: matching engine, awaited-action DB, worker registry, state manager, retries. | +| `nativelink-worker` | Worker: action execution, input materialization, sandboxing/namespaces, directory cache. | +| `nativelink-util` | Shared utilities: `fs`/`fs_util`, `evicting_map`, `action_messages`, digests, and the OTLP metrics (`metrics.rs`). | +| `nativelink-proto` | Generated protobufs (REAPI + NativeLink extensions). | +| `nativelink-metric` / `nativelink-macro` | The `#[metric]` component-metrics derive system and proc macros. | +| `nativelink-error` | The `Error` type and `ResultExt`. | +| `nativelink-test` / `nativelink-redis-tester` | Test harness (`nativelink_test`) and Redis test tooling. | + +## Where to change what + +- **Add or change a config field**: edit the relevant spec in + `nativelink-config/src/stores.rs` or `cas_server.rs`; the config reference is + generated from the doc comments, so document the field there. +- **Add a store type**: implement `StoreDriver` in `nativelink-store/src`, add + its spec to `nativelink-config/src/stores.rs`, and wire it in the store factory. +- **Add or change a gRPC service**: `nativelink-service/src/*_server.rs`. +- **Change scheduling/matching/retries**: `nativelink-scheduler/src`. +- **Change action execution / worker behavior**: + `nativelink-worker/src/running_actions_manager.rs`. +- **Add an OTLP metric**: declare and record it in + `nativelink-util/src/metrics.rs`; record it at the call site; regenerate the + metrics reference. Do not leave a metric declared-but-never-recorded: the + generated reference has a column for exactly that, and it will say so. + +## Changing X: source of truth, and the doc that must follow + +When a change lands, this table says what else has to move with it. The rule is +that the source of truth is always in the repo, and the doc either regenerates +from it or cites it, never restates it by hand. Every path in the right-hand +column is a real page under `web/apps/docs/content/docs/`. + +| You changed | Source of truth | Doc that must follow | +| ----------- | --------------- | -------------------- | +| A config field or its doc comment | `nativelink-config/src/{stores,cas_server}.rs` | `reference/nativelink-config/*`; regenerate with `gen:config-reference`, never hand-edit | +| A store's behavior or defaults | `nativelink-store/src/*` | the backend page under `how-to/stores/`, plus `reference/nativelink-config/store-overview` if the composition model moved | +| A metric name, type, or label | `nativelink-util/src/metrics.rs` and its call sites | `reference/metrics` (regenerate with `gen:metrics-reference`) and `operate/observability` | +| A CLI flag or `NL_*` env var | the binary's arg parsing | `reference/cli-and-env` | +| A gRPC service or the set of services a server exposes | `nativelink-service/src/*_server.rs` | `configuration/servers-and-services`, which names every service as the config spells it | +| Scheduler matching or platform-property semantics | `nativelink-scheduler/src` | `remote-execution/platform-properties`, and `explanations/architecture` if the model changed rather than the mechanics | +| Worker sandboxing or input materialization | `nativelink-worker/src/running_actions_manager.rs` | `explanations/architecture` and `operate/security-hardening`; the second one states what the sandbox is *not* | +| The licence header on a file, or `LICENSE` | the headers themselves | `reference/oss-and-enterprise`, which is the only place the licence split is explained | +| The canonical production config | `deployment-examples/`, `nativelink-config/examples/` | `operate/production-config`; the snippets are lifted from there, not invented | +| A page's URL | `web/apps/docs/content/docs/**/meta.json` | add a redirect in `web/apps/docs/next.config.mjs` | + +## Build, test, verify + +NativeLink builds with both Bazel and Cargo. + +```bash +bazel test //... # all tests (first run 10-20 min) +bazel test //nativelink-store/tests:s3_store_test # one target +bazel build //nativelink:nativelink # the server binary +cargo test -p nativelink-store # a single crate with cargo +``` + +Run the built server against a config: + +```bash +bazel run //nativelink:nativelink -- ./path/to/config.json5 +``` + +Example configs live in `nativelink-config/examples/` and runnable deployments in +`deployment-examples/` (docker-compose, including a multi-worker set) and +`integration_tests/`. + +## The docs, and their gates + +The docs site is a [Fumadocs](https://fumadocs.dev) app at `web/apps/docs`. Pages +are MDX under `content/docs/`, and `meta.json` in each directory controls both the +sidebar order and which pages are published. + +```bash +cd web && bun install +bun run --filter docs dev # local docs server; regenerates first +bun run --filter docs build # what CI builds +bunx biome check --write . # lint and format + +bun --filter @nativelink/docs lint:snippets # JSON5 snippets against the config schema +bun --filter @nativelink/docs lint:anchors # explicit anchors on headings and FAQ entries +``` + +Six things a docs change has to satisfy: + +- **Vale and typos** run in pre-commit. New product nouns go in + `.github/styles/config/vocabularies/TraceMachina/accept.txt`. +- **Biome** formats and lints the TypeScript and the MDX components. +- **Every nav entry resolves.** A `meta.json` entry naming a page that doesn't + exist is a broken sidebar link, and `gen:llms` fails loudly on one. +- **Moved URLs redirect.** Agents cache URLs longer than humans keep bookmarks; + a 404 on an old path is a regression. +- **`lint:snippets` passes.** Every key in a JSON5 config snippet has to exist in + the generated config reference. A snippet that is deliberately wrong (showing + a mistake, or a foreign tool's config) opts out with + `{/* lint-snippets: ignore */}` above it. +- **`lint:anchors` passes.** Every heading and every `` carries an + explicit anchor, so rewording a heading cannot silently repoint a citation. + `lint:anchors --fix` writes the anchor Fumadocs would have derived anyway, + which means running it never moves an existing link. + +Four things under `web/apps/docs` are generated. Regenerate them; never hand-edit: + +| Generated | From | Command | +| --------- | ---- | ------- | +| `content/docs/reference/nativelink-config/*` | the `nativelink-config` crate, via `build-schema` | `gen:config-reference` | +| `content/docs/reference/metrics.mdx` | `nativelink-util/src/metrics.rs` and its call sites | `gen:metrics-reference` | +| `content/docs/reference/changelog.md` | the repository-root `CHANGELOG.md` | `gen:changelog` | +| `public/llms.txt`, `public/llms-full.txt` | the navigation and page frontmatter | `gen:llms` | + +The last two are gitignored and rebuilt by `dev` and `build`, so they cannot be +committed in a stale state. + +Page structure follows four archetypes (tutorial, how-to, explanation, +reference), with a template for each in `web/apps/docs/templates/`. The +conventions those templates encode: + +- A narrative page never inlines an exhaustive field list. It explains the + fields that carry a decision and links the generated reference for the rest. +- Pages on the reading path (Getting started, Remote execution, Configuration, + How-to guides, plus Why NativeLink before and Operate after) open with + ``, so a reader landing cold from a search knows what the page + assumes. +- Tutorials and how-tos close with ``: a checkable claim, not "it + should work now". +- Behavioural claims carry ``, permalinked to a release tag rather + than to `main`. The pinned ref lives in `web/apps/docs/lib/source-ref.ts`; bump + it in the same change that regenerates the reference for a new release. +- Headings carry an explicit `[#anchor]`, and `` entries an `id`, so + a citation to a specific claim keeps resolving after the prose is reworded. + +## Conventions + +- Pre-commit runs rustfmt, `typos`, and (for docs) `vale`. Write to pass them. +- The config reference under `web/apps/docs/content/docs/reference/nativelink-config` + is autogenerated via the `build-schema` binary; regenerate, never hand-edit. +- Prefer generating docs/reference from a code source of truth over hand-writing, + to prevent drift. diff --git a/web/apps/docs/.gitignore b/web/apps/docs/.gitignore index 02bf83b1b..b4f57d38e 100644 --- a/web/apps/docs/.gitignore +++ b/web/apps/docs/.gitignore @@ -8,3 +8,9 @@ next-env.d.ts # Generated from the repository-root CHANGELOG.md by scripts/gen-changelog.mjs # (runs as part of `dev` and `build`). content/docs/reference/changelog.md + +# Generated from the navigation and page frontmatter by scripts/gen-llms.mjs +# (runs as part of `dev` and `build`). Committing them would create a second +# source of truth for the sidebar, which is the thing they exist to prevent. +public/llms.txt +public/llms-full.txt diff --git a/web/apps/docs/app/layout.tsx b/web/apps/docs/app/layout.tsx index d6738f77c..d828be693 100644 --- a/web/apps/docs/app/layout.tsx +++ b/web/apps/docs/app/layout.tsx @@ -12,9 +12,9 @@ import "./globals.css"; export const metadata: Metadata = { title: { default: "NativeLink Docs", - template: "%s — NativeLink Docs", + template: "%s | NativeLink Docs", }, - description: "Documentation for NativeLink — high-performance remote build cache & execution.", + description: "Documentation for NativeLink, a high-performance remote build cache and execution service.", metadataBase: new URL("https://docs.nativelink.com"), }; diff --git a/web/apps/docs/components/min-version.tsx b/web/apps/docs/components/min-version.tsx new file mode 100644 index 000000000..e8f59c558 --- /dev/null +++ b/web/apps/docs/components/min-version.tsx @@ -0,0 +1,31 @@ +import { cn } from "@nativelink/ui"; + +interface MinVersionProps { + /** The first release in which this works, without the leading `v`. */ + v: string; + className?: string; +} + +/** + * Inline "needs at least this release" badge. + * + * Put one next to any field, flag, or command that does not exist in every + * supported release. Readers land on these pages from search engines and from + * agent caches, on whatever version they happen to be running, so "which + * version is this?" has to be answerable on the page itself. + */ +export function MinVersion({ v, className }: MinVersionProps) { + return ( + + {v}+ + + ); +} diff --git a/web/apps/docs/components/next-step.tsx b/web/apps/docs/components/next-step.tsx new file mode 100644 index 000000000..5de3fb179 --- /dev/null +++ b/web/apps/docs/components/next-step.tsx @@ -0,0 +1,80 @@ +import { cn } from "@nativelink/ui"; +import type * as React from "react"; + +interface NextStepProps { + /** Where the reader goes next. */ + href: string; + /** The destination's name, as the reader will see it in the sidebar. */ + title: string; + /** + * `next` continues along the reading path; `aside` is a useful detour that does + * not advance the reader's position. Rendering them differently keeps the + * main path obvious when a page offers both. + */ + kind?: "next" | "aside"; + /** Why the reader would go there: one sentence, not a description. */ + children: React.ReactNode; + className?: string; +} + +/** + * The journey handoff. + * + * A page that ends without telling the reader where to go next has dumped + * information on them. This component renders that handoff the same way on + * every page, so the path through the docs is visible rather than implied. + */ +export function NextStep({ + href, + title, + kind = "next", + children, + className, +}: NextStepProps) { + const isNext = kind === "next"; + return ( + + + + + {isNext ? "Next" : "Sideways"} + + {title} + + {children} + + + + ); +} diff --git a/web/apps/docs/components/prerequisites.tsx b/web/apps/docs/components/prerequisites.tsx new file mode 100644 index 000000000..e06cf2ece --- /dev/null +++ b/web/apps/docs/components/prerequisites.tsx @@ -0,0 +1,38 @@ +import { cn } from "@nativelink/ui"; +import type { ReactNode } from "react"; + +interface PrerequisitesProps { + /** + * What the page assumes the reader already has, written as the state they + * are in rather than the page they read: "a cache serving hits", not "the + * quickstart". Link to the page that gets them there when there is one. + */ + children: ReactNode; + className?: string; +} + +/** + * States what a page assumes before its first step. + * + * Readers arrive mid-corpus from search and from agent caches, not only from + * the page before. A page in an ordered section therefore has to say what it + * assumes. Without that, a reader who lands on remote execution with no + * working cache follows correct instructions to a broken result and blames + * the product. + */ +export function Prerequisites({ children, className }: PrerequisitesProps) { + return ( +
+

+ Before you start +

+
{children}
+
+ ); +} diff --git a/web/apps/docs/components/source-link.tsx b/web/apps/docs/components/source-link.tsx new file mode 100644 index 000000000..b6cd5f63b --- /dev/null +++ b/web/apps/docs/components/source-link.tsx @@ -0,0 +1,68 @@ +import { SOURCE_REF, sourceUrl } from "@/lib/source-ref"; +import { cn } from "@nativelink/ui"; + +interface SourceLinkProps { + /** Repo-relative path, for example `nativelink-config/src/stores.rs`. */ + file: string; + /** Symbol to name in the link text, for example `FastSlowSpec`. */ + symbol?: string; + /** + * Set when `file` names a directory rather than a file, so the link points + * at GitHub's tree view. Directory basenames repeat across the repo + * (`examples/`, `metrics/`), so pass `symbol` too when the last path segment + * alone would not say which one. + */ + dir?: boolean; + className?: string; +} + +/** + * A doc-to-source permalink. + * + * Every reference entry and every claim about runtime behaviour should carry + * one, so a reader can check the prose against the code that implements it. + * The `data-` attributes make the link machine-readable: an agent reading the + * page can follow the same path to the source that a human can. + * + * Prefer this over a hand-written GitHub URL even for a link that is only + * pointing at an example config. A hand-written URL names a ref inline, which + * means it is pinned to `main` (drifts silently) or to a tag that nothing + * bumps when `SOURCE_REF` moves. + */ +export function SourceLink({ file, symbol, dir, className }: SourceLinkProps) { + const path = file.replace(/^\/+|\/+$/g, ""); + const base = path.split("/").pop() ?? path; + const label = symbol ?? (dir ? `${base}/` : base); + return ( + + + {label} + + ); +} diff --git a/web/apps/docs/components/verify-block.tsx b/web/apps/docs/components/verify-block.tsx new file mode 100644 index 000000000..b011f1701 --- /dev/null +++ b/web/apps/docs/components/verify-block.tsx @@ -0,0 +1,56 @@ +import { cn } from "@nativelink/ui"; +import type * as React from "react"; + +interface VerifyBlockProps { + /** Overrides the default heading when a page needs a more specific one. */ + title?: string; + children: React.ReactNode; + className?: string; +} + +/** + * The "you did it right if..." box. + * + * Every tutorial step and every how-to ends with one of these. Without it a + * reader who followed the instructions has no way to distinguish "it worked" + * from "it appeared to work", which, for a build cache, is the difference + * between a fast build and a silently cold one. + * + * `data-verify-block` is deliberate: it makes the claim machine-detectable, so + * CI can assert that no tutorial or how-to page ships without one, and an agent + * reading the page can tell instructions apart from their success criteria. + */ +export function VerifyBlock({ + title = "You did it right if", + children, + className, +}: VerifyBlockProps) { + return ( +
+

+ + {title} +

+
{children}
+
+ ); +} diff --git a/web/apps/docs/lib/source-ref.ts b/web/apps/docs/lib/source-ref.ts new file mode 100644 index 000000000..bdb4e05ea --- /dev/null +++ b/web/apps/docs/lib/source-ref.ts @@ -0,0 +1,22 @@ +/** + * Where doc-to-source links point. + * + * Pinned to a release tag, never to `main`. A permalink that drifts is worse + * than no link at all: a reader who follows it lands on code that no longer + * matches the prose around it, and has no way to tell. Bump this in the same + * change that regenerates the config reference for a new release. + */ +export const SOURCE_REF = "v1.6.5"; + +export const REPO_URL = "https://github.com/TraceMachina/nativelink"; + +/** + * Permalink to a repo-relative path at the pinned ref. + * + * GitHub serves files under `blob/` and directories under `tree/`. It does + * redirect between the two, but a link that only works because of a redirect + * is one platform change away from a 404, so callers say which they mean. + */ +export function sourceUrl(file: string, kind: "blob" | "tree" = "blob"): string { + return `${REPO_URL}/${kind}/${SOURCE_REF}/${file.replace(/^\/+/, "")}`; +} diff --git a/web/apps/docs/mdx-components.tsx b/web/apps/docs/mdx-components.tsx index 1c48b6e67..3df683b6f 100644 --- a/web/apps/docs/mdx-components.tsx +++ b/web/apps/docs/mdx-components.tsx @@ -1,7 +1,12 @@ import { Callout } from "@/components/callout"; import { ConfigVersionSwitcher } from "@/components/config-version-switcher"; import { Mermaid } from "@/components/mermaid"; +import { MinVersion } from "@/components/min-version"; +import { NextStep } from "@/components/next-step"; +import { Prerequisites } from "@/components/prerequisites"; +import { SourceLink } from "@/components/source-link"; import { Steps } from "@/components/steps"; +import { VerifyBlock } from "@/components/verify-block"; import { Accordion, Accordions } from "fumadocs-ui/components/accordion"; import { Callout as FumadocsCallout } from "fumadocs-ui/components/callout"; import { Tab, Tabs } from "fumadocs-ui/components/tabs"; @@ -23,6 +28,14 @@ export function getMDXComponents(components?: MDXComponents): MDXComponents { Mermaid, Accordions, Accordion, + // The journey components. Every page on the reading path opens with + // and closes with ; every tutorial and how-to + // proves itself with ; behavioural claims carry a . + Prerequisites, + NextStep, + VerifyBlock, + SourceLink, + MinVersion, ...components, }; } diff --git a/web/apps/docs/package.json b/web/apps/docs/package.json index a0e19f38c..37cdaf96d 100644 --- a/web/apps/docs/package.json +++ b/web/apps/docs/package.json @@ -26,12 +26,18 @@ }, "private": true, "scripts": { - "build": "node scripts/gen-changelog.mjs && next build", + "build": "node scripts/gen-changelog.mjs && node scripts/gen-llms.mjs && next build", "clean": "rm -rf .next .turbo .source", - "dev": "node scripts/gen-changelog.mjs && next dev --port 3001", + "dev": "node scripts/gen-changelog.mjs && node scripts/gen-llms.mjs && next dev --port 3001", "gen:changelog": "node scripts/gen-changelog.mjs", "gen:config-reference": "node scripts/gen-config-reference.mjs", + "gen:llms": "node scripts/gen-llms.mjs", "lint": "biome check .", + "lint:anchors": "node scripts/lint-anchors.mjs", + "lint:boundaries": "node scripts/lint-boundaries.mjs", + "lint:links": "node scripts/lint-links.mjs", + "lint:navigation": "node scripts/lint-navigation.mjs", + "lint:snippets": "node scripts/lint-snippets.mjs", "start": "next start --port 3001", "typecheck": "tsc --noEmit" }, diff --git a/web/apps/docs/scripts/gen-llms.mjs b/web/apps/docs/scripts/gen-llms.mjs new file mode 100644 index 000000000..2007ab028 --- /dev/null +++ b/web/apps/docs/scripts/gen-llms.mjs @@ -0,0 +1,306 @@ +#!/usr/bin/env node +// Generate `public/llms.txt` and `public/llms-full.txt` from the navigation. +// +// Two files, one job: give a machine reader the same corpus a human gets from +// the sidebar, in the same order, without asking it to crawl. +// +// /llms.txt the link index: every published page, one line each, in +// reading order, following https://llmstxt.org/ +// /llms-full.txt the same order with page bodies inlined, so an agent can +// fetch once instead of following 70 links +// +// Both are generated from `meta.json` and page frontmatter. That is the whole +// point: a hand-maintained index is a second source of truth, and a second +// source of truth is a page that is wrong six weeks from now. Add a page to a +// `meta.json` and it appears here; delete one and it disappears; reword a +// description in frontmatter and this file says the same thing the page does. +// +// Usage, from web/: +// bun --filter @nativelink/docs gen:llms +// or directly: +// node scripts/gen-llms.mjs +// +// Runs as part of `dev` and `build`, after `gen:changelog`, because the +// changelog page is itself generated. + +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +const here = dirname(fileURLToPath(import.meta.url)); +const docsRoot = join(here, ".."); +const contentDir = join(docsRoot, "content/docs"); +const publicDir = join(docsRoot, "public"); + +const SITE = "https://docs.nativelink.com"; +const REPO = "https://github.com/TraceMachina/nativelink"; + +/** Pages a machine reader should know about that are not part of the docs nav. */ +const APPENDIX = [ + { + section: "For agents", + title: "AGENTS.md", + url: `${REPO}/blob/main/AGENTS.md`, + description: + "the repository map for an agent working on the code: what each crate owns, " + + "where to change what, and which doc has to follow each kind of change.", + }, + { + section: "Optional", + title: "NativeLink on GitHub", + url: REPO, + description: "source, issues, releases.", + }, + { + section: "Optional", + title: "Remote Execution API", + url: "https://github.com/bazelbuild/remote-apis", + description: "the protocol NativeLink implements.", + }, +]; + +const PREAMBLE = `> A remote build cache and remote execution service in one Rust binary, for +> build systems that speak the Remote Execution API: Bazel, Buck2, Siso, +> Pants, BuildStream, and CMake via recc. The cache, the scheduler and the +> worker are the same executable started with different configuration. + +These docs are ordered as a path, not a catalogue: get a cache serving hits, +then put workers behind the misses, then own the config file, then reach for +the how-to guides. Each section assumes the previous one is working, and every +page on the path opens by stating what it assumes. An agent reading this file +in order learns the same sequence a human learns from the sidebar. + +Getting started, then Remote execution, then Configuration, then How-to guides. +Why NativeLink comes before that path, and Operate comes after it. Concepts, +Reference and Contribute are outside the ordering: enter them from a search or +a link at any point.`; + +const GENERATED_NOTE = `This file is generated from the navigation (\`meta.json\`) and page frontmatter +by \`web/apps/docs/scripts/gen-llms.mjs\`, so it cannot drift from the sidebar.`; + +// The provenance comment generated pages open with. Matching the comment form +// rather than the bare word keeps a page that merely mentions the marker in +// prose from being treated as generated. +const GENERATED_MARKER = "{/* AUTOGENERATED"; + +// -------------------------------------------------------------------------- + +function readJson(path) { + return JSON.parse(readFileSync(path, "utf8")); +} + +/** Frontmatter is small and well-formed here; a full YAML parser is overkill. */ +function parseFrontmatter(source, path) { + if (!source.startsWith("---\n")) { + throw new Error(`${path} has no frontmatter`); + } + const end = source.indexOf("\n---", 3); + if (end === -1) throw new Error(`${path} has an unterminated frontmatter block`); + const bodyStart = source.indexOf("\n", end + 1) + 1; + const data = {}; + let key = null; + for (const line of source.slice(4, end).split("\n")) { + const match = /^([A-Za-z_][\w-]*):\s*(.*)$/.exec(line); + if (match) { + key = match[1]; + data[key] = match[2].trim(); + } else if (key && line.trim()) { + // A folded continuation line, which frontmatter descriptions use freely. + data[key] = `${data[key]} ${line.trim()}`.trim(); + } + } + for (const k of Object.keys(data)) { + data[k] = data[k].replace(/^["'](.*)["']$/, "$1"); + } + return { data, body: source.slice(bodyStart) }; +} + +/** Locate the file backing a nav entry, whichever extension it uses. */ +function pageFile(dir, name) { + for (const ext of [".mdx", ".md"]) { + const candidate = join(dir, name + ext); + if (existsSync(candidate)) return candidate; + } + return null; +} + +function isDirectory(dir, name) { + return existsSync(join(dir, name)) && existsSync(join(dir, name, "meta.json")); +} + +/** + * Walk a directory's `meta.json` in order and return its pages. + * + * Nav order is the reading order, so this walk *is* the journey: whatever comes + * out of it is what both generated files say, in the order the sidebar says it. + */ +function collectPages(dir, urlPrefix, out) { + const meta = readJson(join(dir, "meta.json")); + const pages = meta.pages ?? []; + // A folder's index page is the folder's own link in the sidebar, so + // meta.json does not list it; it still comes first in reading order. + if (!pages.includes("index") && pageFile(dir, "index")) pages.unshift("index"); + for (const entry of pages) { + if (entry.startsWith("---")) continue; // a sidebar separator, handled by the caller + if (isDirectory(dir, entry)) { + collectPages(join(dir, entry), `${urlPrefix}/${entry}`, out); + continue; + } + const file = pageFile(dir, entry); + if (!file) { + throw new Error( + `${join(dir, "meta.json")} lists "${entry}", but no such page exists. ` + + "Either the nav entry is stale or the page was never created.", + ); + } + const source = readFileSync(file, "utf8"); + const { data, body } = parseFrontmatter(source, file); + out.push({ + url: entry === "index" ? urlPrefix || "/" : `${urlPrefix}/${entry}`, + title: data.title ?? entry, + description: data.description ?? "", + body, + generated: source.includes(GENERATED_MARKER) || body.includes("Generated by scripts/"), + }); + } + return out; +} + +/** + * The whole corpus, as a flat list of sections in nav order. + */ +function buildSections() { + const rootMeta = readJson(join(contentDir, "meta.json")); + const sections = []; + + for (const entry of rootMeta.pages ?? []) { + if (entry.startsWith("---")) continue; // a sidebar separator carries no pages + if (isDirectory(contentDir, entry)) { + // The section's own sidebar title names it, so llms.txt and the sidebar + // cannot disagree. + const title = readJson(join(contentDir, entry, "meta.json")).title ?? entry; + sections.push({ + title, + pages: collectPages(join(contentDir, entry), `/${entry}`, []), + }); + continue; + } + const file = pageFile(contentDir, entry); + if (!file) throw new Error(`content/docs/meta.json lists "${entry}", but no such page exists`); + const { data, body } = parseFrontmatter(readFileSync(file, "utf8"), file); + sections.push({ + title: null, // the root page is the front door, not a section + pages: [ + { + url: "/", + title: data.title ?? entry, + description: data.description ?? "", + body, + generated: false, + }, + ], + }); + } + + for (const extra of APPENDIX) { + const section = sections.find((s) => s.title === extra.section); + if (section) section.pages.push({ ...extra, external: true }); + else sections.push({ title: extra.section, pages: [{ ...extra, external: true }] }); + } + + return sections; +} + +// -------------------------------------------------------------------------- + +function renderIndex(sections) { + const lines = ["# NativeLink", "", PREAMBLE, "", GENERATED_NOTE, "", "The full corpus with page bodies is at /llms-full.txt.", ""]; + + for (const section of sections) { + if (section.title) lines.push(`## ${section.title}`, ""); + for (const page of section.pages) { + const url = page.external ? page.url : `${SITE}${page.url}`; + lines.push(`- [${page.title}](${url}): ${page.description}`); + } + lines.push(""); + } + + return `${lines.join("\n").replace(/\n{3,}/g, "\n\n").trimEnd()}\n`; +} + +/** + * Strip the machinery a reader does not need: the archetype pointer, the lint + * escape hatches. Everything else, components included, is left verbatim, + * because a lossy transcription of a page is a third source of truth. + */ +function cleanBody(body) { + return body + .replace(/^\{\/\*\s*archetype:[^]*?\*\/\}\s*$/gm, "") + .replace(/^\{\/\*\s*lint-snippets:[^]*?\*\/\}\s*$/gm, "") + .replace(/^\{\/\*\s*AUTOGENERATED[^]*?\*\/\}\s*$/gm, "") + .replace(/\n{3,}/g, "\n\n") + .trim(); +} + +function renderFull(sections) { + const lines = [ + "# NativeLink: full documentation corpus", + "", + PREAMBLE, + "", + GENERATED_NOTE, + "", + `Every page below is delimited by a heading of the form \`# ${SITE}/\`,`, + "giving that page's canonical URL. The delimiter is the absolute URL rather", + "than the path because a bare `# /path` line is indistinguishable from a shell", + "comment naming a file, and several pages contain exactly that.", + "", + "Page bodies are reproduced verbatim, MDX components included, so nothing is", + "lost in transcription; links inside them are site-relative and resolve against", + `${SITE}. The generated reference pages (the configuration`, + "reference, the metrics reference and the changelog) are linked rather than", + "inlined: they are long, they change on a different cadence from the prose, and", + "they are better fetched on demand.", + "", + ]; + + for (const section of sections) { + for (const page of section.pages) { + if (page.external) continue; + lines.push("", `# ${SITE}${page.url}`, ""); + lines.push(`**${page.title}**: ${page.description}`); + lines.push(""); + if (page.generated) { + lines.push( + `_Generated reference page, not inlined. Fetch it from ${SITE}${page.url}._`, + ); + } else { + lines.push(cleanBody(page.body)); + } + } + } + + return `${lines.join("\n").replace(/\n{4,}/g, "\n\n\n").trimEnd()}\n`; +} + +// -------------------------------------------------------------------------- + +function main() { + const sections = buildSections(); + + const index = renderIndex(sections); + const full = renderFull(sections); + + writeFileSync(join(publicDir, "llms.txt"), index); + writeFileSync(join(publicDir, "llms-full.txt"), full); + + const pages = sections.flatMap((s) => s.pages).filter((p) => !p.external); + const inlined = pages.filter((p) => !p.generated).length; + console.log( + `Wrote public/llms.txt (${pages.length} pages, ${Math.round(index.length / 1024)} KB) and ` + + `public/llms-full.txt (${inlined} bodies inlined, ${pages.length - inlined} generated pages linked, ` + + `${Math.round(full.length / 1024)} KB).`, + ); +} + +main(); diff --git a/web/apps/docs/scripts/lint-anchors.mjs b/web/apps/docs/scripts/lint-anchors.mjs new file mode 100644 index 000000000..492d088b5 --- /dev/null +++ b/web/apps/docs/scripts/lint-anchors.mjs @@ -0,0 +1,253 @@ +#!/usr/bin/env node +// Anchor lint: every heading and every FAQ entry carries an explicit anchor. +// +// Fumadocs derives a heading's id from its text when no id is given. That is +// convenient and it is exactly the problem: the id is a function of the prose, +// so rewording a heading silently repoints every link that cited it. Humans +// notice a dead in-page link and scroll; agents cache the URL, follow it, land +// at the top of the page, and quote whatever is there. +// +// `/agents` promises machine readers that "a citation to a specific claim keeps +// resolving". This lint is what makes that promise structural rather than +// aspirational: an explicit `[#slug]` on every heading and an `id=` on every +// `` pins the anchor to the author's intent instead of to the +// current wording, so a heading can be rewritten without breaking a citation. +// +// Usage, from web/: +// bun --filter @nativelink/docs lint:anchors +// bun --filter @nativelink/docs lint:anchors --fix +// or directly: +// node scripts/lint-anchors.mjs [--fix] +// +// `--fix` appends the anchor Fumadocs would have generated anyway, so running +// it never changes where an existing link lands; it only freezes it. +// +// Generated pages are skipped: their anchors are the generator's business, and +// hand-editing them would be overwritten on the next regeneration. +// +// Exit code is 1 if any heading or accordion is missing an explicit anchor. +// +// The markdown-parsing imports below (`github-slugger`, `remark-parse`, +// `remark-mdx`, `unified`, `unist-util-visit`) are not listed in this package's +// dependencies. They arrive hoisted into `web/node_modules` as transitive +// dependencies of Fumadocs, which owns the versions this script has to agree +// with anyway; it reuses `remarkHeading` from `fumadocs-core`, so parsing the +// tree with a different remark than Fumadocs uses would be the actual bug. +// Declaring them here would pin a second copy free to drift from that. If a +// Fumadocs upgrade ever drops one, this script fails loudly at import rather +// than silently mis-slugging, which is the failure mode worth having. + +import { readFileSync, readdirSync, statSync, writeFileSync } from "node:fs"; +import { dirname, join, relative } from "node:path"; +import { fileURLToPath } from "node:url"; +import { remarkHeading } from "fumadocs-core/mdx-plugins"; +import Slugger from "github-slugger"; +import remarkMdx from "remark-mdx"; +import remarkParse from "remark-parse"; +import { unified } from "unified"; +import { visit } from "unist-util-visit"; + +const here = dirname(fileURLToPath(import.meta.url)); +const docsRoot = join(here, ".."); +const contentDir = join(docsRoot, "content/docs"); + +// The provenance comment generated pages open with. Matching the comment form +// rather than the bare word keeps a page that merely mentions the marker in +// prose from being treated as generated. +const GENERATED_MARKER = "{/* AUTOGENERATED"; +const FIX = process.argv.includes("--fix"); + +// -------------------------------------------------------------------------- + +function walk(dir, out = []) { + for (const name of readdirSync(dir)) { + const full = join(dir, name); + if (statSync(full).isDirectory()) walk(full, out); + else if (name.endsWith(".mdx")) out.push(full); + } + return out; +} + +/** + * Blank out the frontmatter without changing the line count, so every node + * position still points at the right line of the original file. + */ +function blankFrontmatter(source) { + if (!source.startsWith("---")) return source; + const end = source.indexOf("\n---", 3); + if (end === -1) return source; + const close = source.indexOf("\n", end + 1); + const head = source.slice(0, close); + return head.replace(/[^\n]/g, " ") + source.slice(close); +} + +const processor = unified() + .use(remarkParse) + .use(remarkMdx) + .use(remarkHeading, { generateToc: false }); + +/** + * Parse a page and return what carries an anchor and what does not. + * + * Heading ids come from Fumadocs' own `remarkHeading`, so the id this lint + * writes is byte-for-byte the id the site serves today. + */ +function inspect(source) { + const tree = processor.parse(blankFrontmatter(source)); + processor.runSync(tree); + + const headings = []; + const accordions = []; + const taken = new Set(); + const totals = { headings: 0, accordions: 0 }; + + visit(tree, "heading", (node) => { + totals.headings += 1; + const id = node.data?.hProperties?.id; + if (id) taken.add(id); + // `remarkHeading` strips `[#id]` off the trailing text node when it finds + // one, so an explicit anchor is exactly the case where the raw line ends + // in `[#…]`, so read the source line rather than the mutated node. + headings.push({ line: node.position.start.line, endLine: node.position.end.line, id }); + }); + + visit(tree, "mdxJsxFlowElement", (node) => { + if (node.name !== "Accordion") return; + totals.accordions += 1; + const attrs = new Map(); + for (const attr of node.attributes ?? []) { + if (attr.type !== "mdxJsxAttribute") continue; + attrs.set(attr.name, typeof attr.value === "string" ? attr.value : null); + } + if (attrs.has("id")) { + if (attrs.get("id")) taken.add(attrs.get("id")); + return; + } + accordions.push({ + line: node.position.start.line, + endLine: node.position.end.line, + title: attrs.get("title") ?? "", + }); + }); + + return { headings, accordions, taken, totals }; +} + +/** A slug that does not collide with anything already claimed on the page. */ +function uniqueSlug(text, taken) { + const slugger = new Slugger(); + let slug = slugger.slug(text) || "section"; + let n = 2; + while (taken.has(slug)) slug = `${slugger.slug(text) || "section"}-${n++}`; + taken.add(slug); + return slug; +} + +// -------------------------------------------------------------------------- + +function main() { + const files = walk(contentDir).sort(); + const problems = []; + let fixedFiles = 0; + let fixedAnchors = 0; + let checkedHeadings = 0; + let checkedAccordions = 0; + let skipped = 0; + + for (const file of files) { + const rel = relative(docsRoot, file); + const source = readFileSync(file, "utf8"); + if (source.includes(GENERATED_MARKER)) { + skipped += 1; + continue; + } + + let report; + try { + report = inspect(source); + } catch (error) { + problems.push({ file: rel, line: 1, message: `could not parse: ${error.message}` }); + continue; + } + + checkedHeadings += report.totals.headings; + checkedAccordions += report.totals.accordions; + + const lines = source.split("\n"); + const edits = []; + + for (const heading of report.headings) { + const raw = lines[heading.line - 1]; + if (/\[#[^\]]+\]\s*$/.test(raw)) continue; + if (heading.endLine !== heading.line) { + problems.push({ + file: rel, + line: heading.line, + message: "heading spans multiple lines; add the anchor by hand", + }); + continue; + } + if (!heading.id) { + problems.push({ file: rel, line: heading.line, message: "heading has no derivable id" }); + continue; + } + edits.push({ index: heading.line - 1, next: `${raw.replace(/\s+$/, "")} [#${heading.id}]` }); + problems.push({ + file: rel, + line: heading.line, + message: `heading has no explicit anchor (would be \`[#${heading.id}]\`)`, + }); + } + + for (const accordion of report.accordions) { + const raw = lines[accordion.line - 1]; + const id = uniqueSlug(accordion.title, report.taken); + if (!/\stitle=("[^"]*"|\{[^}]*\})/.test(raw)) { + problems.push({ + file: rel, + line: accordion.line, + message: "accordion has no id and no single-line title to derive one from", + }); + continue; + } + edits.push({ + index: accordion.line - 1, + next: raw.replace(/(\stitle=(?:"[^"]*"|\{[^}]*\}))/, `$1 id="${id}"`), + }); + problems.push({ + file: rel, + line: accordion.line, + message: `accordion has no explicit id (would be \`id="${id}"\`)`, + }); + } + + if (FIX && edits.length > 0) { + for (const edit of edits) lines[edit.index] = edit.next; + writeFileSync(file, lines.join("\n")); + fixedFiles += 1; + fixedAnchors += edits.length; + } + } + + console.log( + `Checked ${checkedHeadings} heading(s) and ${checkedAccordions} accordion(s) across ` + + `${files.length - skipped} page(s); skipped ${skipped} generated page(s).`, + ); + + if (FIX) { + console.log(`Pinned ${fixedAnchors} anchor(s) across ${fixedFiles} page(s).`); + return; + } + + if (problems.length === 0) { + console.log("Every heading and accordion carries an explicit anchor."); + return; + } + + console.error(`\n${problems.length} missing anchor(s):`); + for (const p of problems) console.error(` ${p.file}:${p.line} ${p.message}`); + console.error("\nRun `bun --filter @nativelink/docs lint:anchors --fix` to pin them."); + process.exitCode = 1; +} + +main(); diff --git a/web/apps/docs/scripts/lint-boundaries.mjs b/web/apps/docs/scripts/lint-boundaries.mjs new file mode 100644 index 000000000..e64de0a5c --- /dev/null +++ b/web/apps/docs/scripts/lint-boundaries.mjs @@ -0,0 +1,135 @@ +#!/usr/bin/env node +// Boundary lint: the open-source docs stay self-contained. +// +// These docs make a promise: everything described here can be run by anyone +// from the source in this repository, with no account and nothing to buy. That +// promise is easy to make once and hard to keep, because the natural place to +// mention a hosted offering is exactly where a self-hosted path gets tedious, +// and each individual mention reads as helpful. +// +// The policy is not "never mention the hosted product". It is: +// +// - Exactly one page, reference/oss-and-enterprise.mdx, compares the two and +// is allowed to sell. +// - Any other page may LINK to that page, and may state a factual boundary +// ("this is not implemented in the open-source distribution"). +// - No other page may carry a call to action: no sign-up, no pricing, no +// trial, no demo, no sales contact. +// +// So this lint looks for calls to action rather than for the words "cloud" or +// "enterprise", which have ordinary technical meanings this project uses +// constantly and which a naive grep would drown in. +// +// Escape hatch, on the line before the offending line: +// {/* lint-boundaries: ignore () */} +// Use it for a genuine false positive and say why. It is not a way to add a +// call to action with extra steps. +// +// Usage, from web/: +// bun --filter @nativelink/docs lint:boundaries +// or directly: +// node scripts/lint-boundaries.mjs +// +// Exit code is 1 if any page outside the allowed one carries a call to action. + +import { readFileSync, readdirSync, statSync } from "node:fs"; +import { dirname, join, relative } from "node:path"; +import { fileURLToPath } from "node:url"; + +const here = dirname(fileURLToPath(import.meta.url)); +const docsRoot = join(here, ".."); +const contentDir = join(docsRoot, "content/docs"); + +// The one page allowed to compare the open-source project with the hosted +// product, and to link outward for it. +const ALLOWED = new Set(["content/docs/reference/oss-and-enterprise.mdx"]); + +const IGNORE = /\{\/\*\s*lint-boundaries:\s*ignore/; + +const RULES = [ + { + name: "sign-up call to action", + re: /\b(sign\s?up|signup|create (?:a |an )?(?:free )?account|get started free|start (?:your )?free trial|try it free)\b/i, + }, + { + name: "pricing or plan language", + re: /\b(free tier|paid tier|pricing page|per-seat|per seat|enterprise plan|pro plan|billing)\b/i, + }, + { + name: "sales or demo contact", + re: /\b(contact (?:our )?sales|talk to (?:an? )?(?:expert|sales)|book a demo|schedule a demo|request a demo)\b/i, + }, + { + name: "hosted-product marketing link", + re: /https?:\/\/(?:www\.)?nativelink\.com\/(?:pricing|signup|sign-up|contact|demo|dashboard|app)\b/i, + }, + { + name: "upsell framing", + re: /\b(upgrade to (?:the )?(?:cloud|enterprise|pro)|available (?:only )?(?:on|in) (?:nativelink )?cloud\b)/i, + }, +]; + +// -------------------------------------------------------------------------- + +function walk(dir, out = []) { + for (const name of readdirSync(dir)) { + const full = join(dir, name); + if (statSync(full).isDirectory()) walk(full, out); + else if (name.endsWith(".mdx")) out.push(full); + } + return out; +} + +function main() { + const files = walk(contentDir).sort(); + const problems = []; + let checked = 0; + let allowed = 0; + + for (const file of files) { + const rel = relative(docsRoot, file).replace(/\\/g, "/"); + if (ALLOWED.has(rel)) { + allowed += 1; + continue; + } + checked += 1; + + const lines = readFileSync(file, "utf8").split("\n"); + for (let i = 0; i < lines.length; i += 1) { + if (i > 0 && IGNORE.test(lines[i - 1])) continue; + for (const rule of RULES) { + const hit = lines[i].match(rule.re); + if (hit) { + problems.push({ + rel, + line: i + 1, + message: `${rule.name}: \`${hit[0]}\``, + }); + } + } + } + } + + if (problems.length === 0) { + console.log( + `lint:boundaries: ${checked} pages carry no call to action (${allowed} page exempt).`, + ); + return; + } + + console.error( + `lint:boundaries: ${problems.length} call(s) to action outside the one allowed page:\n`, + ); + for (const { rel, line, message } of problems) { + console.error(` ${rel}:${line} ${message}`); + } + console.error( + "\nThese docs promise that everything in them can be run from this repository\n" + + "with no account. Comparisons with the hosted product belong on\n" + + "reference/oss-and-enterprise.mdx; every other page may link to it and state a\n" + + "factual boundary, but may not sell.\n", + ); + process.exit(1); +} + +main(); diff --git a/web/apps/docs/scripts/lint-links.mjs b/web/apps/docs/scripts/lint-links.mjs new file mode 100644 index 000000000..f81e65499 --- /dev/null +++ b/web/apps/docs/scripts/lint-links.mjs @@ -0,0 +1,346 @@ +#!/usr/bin/env node +// Link lint: every internal link resolves to a page that exists, at an anchor +// that exists. +// +// A 404 is loud. The failures this lint exists for are the quiet ones: a link +// to a page that still exists but whose section was renamed, so the browser +// silently lands at the top and the reader never learns they were sent +// somewhere specific. Humans notice they are in the wrong place and scroll. +// Agents do not; they follow the URL, take the top of the page, and quote it +// as the answer to a question it does not answer. +// +// So this checks three things, in increasing order of how easy they are to +// break without noticing: +// +// 1. `/some/page` resolves to a real .mdx file (or to a configured redirect). +// 2. `/some/page#some-anchor` resolves to an anchor that page actually +// declares. +// 3. `#local-anchor` inside a page resolves on that same page. +// +// It deliberately does NOT check external links. Those fail for reasons +// outside this repository (rate limits, transient outages, sites that block +// CI), and a lint that goes red for reasons a contributor cannot fix is a +// lint people learn to ignore. External reachability belongs in a scheduled +// job, not in the PR gate. +// +// Links to a served file rather than to a page (`/llms.txt`, an image, a +// downloadable config) are skipped. Whether those exist is decided by the +// build and by `public/`, not by anything in content/docs, so this lint has no +// way to answer the question and no business guessing. +// +// Generated pages get checked but not enforced, for the same reason: a broken +// link on a generated page is a bug in the generator, and failing an unrelated +// PR for it teaches contributors that this lint is somebody else's problem. +// They are reported separately, as warnings, so the bug is visible to whoever +// owns the generator. +// +// Usage, from web/: +// bun --filter @nativelink/docs lint:links +// or directly: +// node scripts/lint-links.mjs +// +// Exit code is 1 if any internal link is unresolvable. + +import { readFileSync, readdirSync, statSync } from "node:fs"; +import { dirname, join, relative } from "node:path"; +import { fileURLToPath } from "node:url"; +import { remarkHeading } from "fumadocs-core/mdx-plugins"; +import remarkMdx from "remark-mdx"; +import remarkParse from "remark-parse"; +import { unified } from "unified"; +import { visit } from "unist-util-visit"; + +const here = dirname(fileURLToPath(import.meta.url)); +const docsRoot = join(here, ".."); +const contentDir = join(docsRoot, "content/docs"); + +// Routes that exist but are not backed by an .mdx file in content/docs. +const EXTRA_ROUTES = new Set(["/"]); + +// A final path segment with an extension is a served file, not a page; see +// the header. The extension must start with a letter so a version segment like +// `/reference/v0.62` is still treated as a route. +const SERVED_FILE = /\/[^/]+\.[a-z][a-z0-9]{1,4}$/i; + +// Same marker lint-anchors uses to recognise a page it must not rewrite. +// The provenance comment generated pages open with. Matching the comment form +// rather than the bare word keeps a page that merely mentions the marker in +// prose from being treated as generated. +const GENERATED_MARKER = "{/* AUTOGENERATED"; + +// -------------------------------------------------------------------------- + +function walk(dir, out = []) { + for (const name of readdirSync(dir)) { + const full = join(dir, name); + if (statSync(full).isDirectory()) walk(full, out); + else if (name.endsWith(".mdx") || name.endsWith(".md")) out.push(full); + } + return out; +} + +/** + * content/docs/a/b.mdx -> /a/b ; content/docs/a/index.mdx -> /a + * + * `.md` counts too. Fumadocs serves both, and `reference/changelog.md` is a + * real page; indexing only `.mdx` reported every link to it as a dead link. + */ +function routeFor(file) { + const rel = relative(contentDir, file).replace(/\\/g, "/"); + const noExt = rel.replace(/\.mdx?$/, ""); + const route = noExt.endsWith("/index") + ? noExt.slice(0, -"/index".length) + : noExt === "index" + ? "" + : noExt; + return `/${route}`.replace(/\/$/, "") || "/"; +} + +function blankFrontmatter(source) { + if (!source.startsWith("---")) return source; + const end = source.indexOf("\n---", 3); + if (end === -1) return source; + const close = source.indexOf("\n", end + 1); + const head = source.slice(0, close); + return head.replace(/[^\n]/g, " ") + source.slice(close); +} + +const processor = unified() + .use(remarkParse) + .use(remarkMdx) + .use(remarkHeading, { generateToc: false }); + +// Plain `.md` pages are Markdown, not MDX. `reference/changelog.md` is +// assembled from upstream release notes and contains `` comments and +// bare `<` characters that are a syntax error under remarkMdx. They are still +// real pages that real links point at, so they get parsed, just not as MDX. +const mdProcessor = unified().use(remarkParse).use(remarkHeading, { generateToc: false }); + +/** + * Anchors a page offers, and internal links a page makes. + * + * Heading ids come from Fumadocs' own `remarkHeading`, so what this collects + * is what the site actually serves, including on generated pages, where the + * anchors are the generator's and there is no explicit `[#slug]` to read. + */ +function inspect(source, { mdx = true } = {}) { + const p = mdx ? processor : mdProcessor; + const tree = p.parse(blankFrontmatter(source)); + p.runSync(tree); + + const anchors = new Set(); + const links = []; + + visit(tree, "heading", (node) => { + const id = node.data?.hProperties?.id; + if (id) anchors.add(id); + }); + + visit(tree, "link", (node) => { + links.push({ href: node.url, line: node.position.start.line }); + }); + + visit(tree, "mdxJsxFlowElement", collectJsx); + visit(tree, "mdxJsxTextElement", collectJsx); + + function collectJsx(node) { + for (const attr of node.attributes ?? []) { + if (attr.type !== "mdxJsxAttribute") continue; + if (attr.name === "id" && node.name === "Accordion") { + if (typeof attr.value === "string") anchors.add(attr.value); + } + if (attr.name !== "href") continue; + if (typeof attr.value !== "string") continue; + links.push({ href: attr.value, line: node.position.start.line }); + } + } + + return { anchors, links }; +} + +/** + * The `redirects()` table out of next.config.mjs, without importing it. + * + * Importing the config would be the obvious thing, and it is the wrong thing: + * the first line of that file is `createMDX()`, so an import drags in + * fumadocs-mdx, which compiles the MDX config through esbuild's + * platform-specific native binary and writes to `.source/`. A link lint that + * cannot run unless the whole MDX toolchain is installed and matches the host + * architecture is a link lint that gets skipped. + * + * The `redirects()` body is pure data (array literals, `Object.entries`, + * spreads) and closes over nothing in module scope, so it can be lifted out + * by brace matching and evaluated on its own. If that extraction ever stops + * working, every redirected route reports as an unresolved link, which is a + * loud failure pointing straight here rather than a quiet one. + */ +function redirectTable(configSource) { + const start = configSource.indexOf("async redirects()"); + if (start === -1) throw new Error("no `async redirects()` in next.config.mjs"); + const open = configSource.indexOf("{", start); + let depth = 0; + let end = -1; + for (let i = open; i < configSource.length; i += 1) { + if (configSource[i] === "{") depth += 1; + else if (configSource[i] === "}" && --depth === 0) { + end = i; + break; + } + } + if (end === -1) throw new Error("unbalanced braces in redirects()"); + // Repo-local file, and the body is data: no imports, no closures. + const list = new Function(`"use strict";${configSource.slice(open + 1, end)}`)(); + if (!Array.isArray(list) || list.length === 0) { + throw new Error("redirects() did not evaluate to a non-empty array"); + } + return list; +} + +/** + * Redirect sources from next.config.mjs, so a link to a page that moved is + * correct rather than merely tolerated. + * + * Patterns with a `:param` segment become a prefix match: `/old/:slug*` + * accepts anything under `/old/`. If the config cannot be read we carry on + * with an empty table rather than failing: a missing redirect surfaces as an + * unresolved link, which is the more useful error. + */ +async function loadRedirects() { + const exact = new Set(); + const prefixes = []; + try { + const list = redirectTable(readFileSync(join(docsRoot, "next.config.mjs"), "utf8")); + for (const { source } of list) { + if (typeof source !== "string") continue; + const param = source.indexOf("/:"); + if (param === -1) exact.add(source.replace(/\/$/, "") || "/"); + else prefixes.push(source.slice(0, param)); + } + } catch (error) { + console.warn(` note: could not read next.config.mjs redirects (${error.message})`); + console.warn(" every link to a page that moved will report as unresolved."); + } + return { exact, prefixes }; +} + +// -------------------------------------------------------------------------- + +async function main() { + const files = walk(contentDir).sort(); + const pages = new Map(); + + for (const file of files) { + const source = readFileSync(file, "utf8"); + let report; + try { + report = inspect(source, { mdx: file.endsWith(".mdx") }); + } catch (error) { + console.error(`${relative(docsRoot, file)}:1 could not parse: ${error.message}`); + process.exit(1); + } + pages.set(routeFor(file), { + file, + generated: source.includes(GENERATED_MARKER), + ...report, + }); + } + + const redirects = await loadRedirects(); + const problems = []; + const warnings = []; + let checked = 0; + + const isRedirected = (route) => + redirects.exact.has(route) || + redirects.prefixes.some((p) => route === p.replace(/\/$/, "") || route.startsWith(p)); + + for (const [, page] of pages) { + const rel = relative(docsRoot, page.file); + // Generated pages are reported but do not fail the run; see the header. + const report = page.generated ? warnings : problems; + + for (const { href, line } of page.links) { + // External, mail, and protocol-relative links are not this lint's job. + if (/^[a-z][a-z0-9+.-]*:/i.test(href) || href.startsWith("//")) continue; + // A served file, not a page; see the header. + if (SERVED_FILE.test(href.split("#")[0])) continue; + + checked += 1; + const hash = href.indexOf("#"); + const path = hash === -1 ? href : href.slice(0, hash); + const anchor = hash === -1 ? "" : href.slice(hash + 1); + + // A bare `#anchor` is a link within this page. + if (path === "") { + if (anchor && !page.anchors.has(anchor)) { + report.push({ rel, line, message: `no anchor \`#${anchor}\` on this page` }); + } + continue; + } + + if (!path.startsWith("/")) { + report.push({ + rel, + line, + message: `relative link \`${href}\`: internal links must be absolute from the docs\n root. If you meant a reference-style link, the brackets are \`[text][${href}]\`, not \`[text](${href})\`.`, + }); + continue; + } + + const target = path.replace(/\/$/, "") || "/"; + const targetPage = pages.get(target); + + if (!targetPage) { + if (EXTRA_ROUTES.has(target) || isRedirected(target)) continue; + // The most common shape of this failure is worth naming outright. + const near = [...pages.keys()].find((r) => r.endsWith(target) || target.endsWith(r)); + report.push({ + rel, + line, + message: near + ? `\`${target}\` does not exist. Did you mean \`${near}\`?` + : `\`${target}\` does not exist, and no redirect covers it`, + }); + continue; + } + + if (anchor && !targetPage.anchors.has(anchor)) { + report.push({ + rel, + line, + message: `\`${target}\` exists but declares no anchor \`#${anchor}\``, + }); + } + } + } + + if (warnings.length > 0) { + console.warn(`lint:links: ${warnings.length} unresolved link(s) on generated pages.`); + console.warn( + "These do not fail the build: the fix belongs in the generator, not in\n" + + "the page, and not in whichever PR happened to run this next.\n", + ); + for (const { rel, line, message } of warnings) { + console.warn(` ${rel}:${line} ${message}`); + } + console.warn(""); + } + + if (problems.length === 0) { + console.log(`lint:links: ${checked} internal links across ${pages.size} pages resolve.`); + return; + } + + console.error(`lint:links: ${problems.length} unresolved of ${checked} internal links:\n`); + for (const { rel, line, message } of problems) { + console.error(` ${rel}:${line} ${message}`); + } + console.error( + "\nEvery internal link must resolve to a page that exists, at an anchor that\n" + + "exists. If a page moved, add a redirect in next.config.mjs; the redirect\n" + + "table counts as resolution, so old published URLs stay valid.\n", + ); + process.exit(1); +} + +await main(); diff --git a/web/apps/docs/scripts/lint-navigation.mjs b/web/apps/docs/scripts/lint-navigation.mjs new file mode 100644 index 000000000..0e41f957c --- /dev/null +++ b/web/apps/docs/scripts/lint-navigation.mjs @@ -0,0 +1,183 @@ +#!/usr/bin/env node +// Navigation lint: no page on the reading path is a dead end, and no page off +// it claims to be on it. +// +// The docs are ordered as a path (cache, then execution, then configuration, +// then how-to guides) and that ordering only exists if every page on it says +// what comes next. A page without a `` is where the path stops for +// the reader who reached it, regardless of how well the sidebar is organised. +// +// A page may offer more than one ``, but only one of them is the +// path. The rest are marked `kind="aside"` and render as optional detours, so +// the reader is never asked to pick blind between two equal-looking doors. So +// what this counts is *primary* next steps: exactly one per page. +// +// The inverse matters as much. A `` block tells the reader what +// the page assumes they already have, which only makes sense on a page that +// sits somewhere in the sequence. Reference, concept and contribute pages are +// entered from a search or a link rather than from the page before, and they +// carry none. +// +// So this checks three things: +// +// 1. Every non-index page on the path has a primary ``. +// 2. No page has two primary ``s. +// 3. No page outside the path sections has a ``. +// +// Section index pages are exempt from the first rule: they are the fan-out +// for their section, and the list of links IS the "what next". +// +// Fenced code blocks are stripped before anything is counted, so the page that +// documents these components for contributors does not trip the lint by +// showing what they look like. +// +// Usage, from web/: +// bun --filter @nativelink/docs lint:navigation +// or directly: +// node scripts/lint-navigation.mjs +// +// Exit code is 1 if any page is a dead end or states prerequisites it should not. + +import { readFileSync, readdirSync, statSync } from "node:fs"; +import { basename, dirname, join, relative } from "node:path"; +import { fileURLToPath } from "node:url"; + +const here = dirname(fileURLToPath(import.meta.url)); +const docsRoot = join(here, ".."); +const contentDir = join(docsRoot, "content/docs"); + +// The provenance comment generated pages open with. Matching the comment form +// rather than the bare word keeps a page that merely mentions the marker in +// prose from being treated as generated. +const GENERATED_MARKER = "{/* AUTOGENERATED"; + +// The sections that form the reading path, in order, plus the two that sit +// either side of it. Every page in these wants a `` and may open with +// ``. +const PATH_SECTIONS = [ + "use-cases", + "getting-started", + "remote-execution", + "configuration", + "how-to", + "operate", +]; + +// -------------------------------------------------------------------------- + +function walk(dir, out = []) { + for (const name of readdirSync(dir)) { + const full = join(dir, name); + if (statSync(full).isDirectory()) walk(full, out); + else if (name.endsWith(".mdx")) out.push(full); + } + return out; +} + +function sectionOf(file) { + const rel = relative(contentDir, file).replace(/\\/g, "/"); + const slash = rel.indexOf("/"); + return slash === -1 ? "" : rel.slice(0, slash); +} + +// Fenced blocks and inline code are prose about the components, not uses of +// them. contribute/docs.mdx shows a `` in a fence to explain +// the convention, and it should not be held to it. +function stripCode(source) { + return source.replace(/^```[\s\S]*?^```/gm, "\n").replace(/`[^`\n]*`/g, "``"); +} + +// `` is an optional detour. Everything else is the +// path. Attributes may wrap across lines, so match up to the closing `>` of +// the opening tag rather than to end of line. +function countNextSteps(source) { + const opens = source.match(/]*>/g) ?? []; + let primary = 0; + let asides = 0; + for (const tag of opens) { + if (/\bkind\s*=\s*["'{]?\s*aside/.test(tag)) asides += 1; + else primary += 1; + } + return { primary, asides }; +} + +function main() { + const files = walk(contentDir).sort(); + const problems = []; + const seenSections = new Set(); + let checked = 0; + + for (const file of files) { + const rel = relative(docsRoot, file).replace(/\\/g, "/"); + const raw = readFileSync(file, "utf8"); + if (raw.includes(GENERATED_MARKER)) continue; + const source = stripCode(raw); + + const section = sectionOf(file); + if (section) seenSections.add(section); + + const onPath = PATH_SECTIONS.includes(section); + const isIndex = basename(file) === "index.mdx"; + + const { primary } = countNextSteps(source); + const prerequisites = source.match(/: this page is on the reading path, so it is " + + "where the path stops for whoever lands here " + + '(a page with only `kind="aside"` steps offers detours and no road)', + }); + } + + if (primary > 1) { + const detail = + "one of these is the path and the rest want " + '`kind="aside"`, or the reader picks blind'; + problems.push({ + rel, + message: `${primary} primary elements: ${detail}`, + }); + } + + if (!onPath && prerequisites > 0) { + problems.push({ + rel, + message: ` on a page in \`${section || "the docs root"}\`, which is not on the reading path: it would tell the reader they are partway through a sequence they are not in`, + }); + } + } + + // A renamed or removed section would otherwise make this lint quietly stop + // checking the pages it was written for. + for (const section of PATH_SECTIONS) { + if (!seenSections.has(section)) { + problems.push({ + rel: "scripts/lint-navigation.mjs", + message: `PATH_SECTIONS names \`${section}\`, which has no pages`, + }); + } + } + + if (problems.length === 0) { + console.log(`lint:navigation: ${checked} pages, no dead ends, no misplaced prerequisites.`); + return; + } + + console.error(`lint:navigation: ${problems.length} problem(s):\n`); + for (const { rel, message } of problems) { + console.error(` ${rel} ${message}`); + } + console.error( + "\nPages on the reading path end with exactly one primary so the\n" + + 'path continues; further links go as `kind="aside"`. Pages off the path\n' + + "carry no , because they are entered from a search or a\n" + + "link rather than from the page before.\n", + ); + process.exit(1); +} + +main(); diff --git a/web/apps/docs/scripts/lint-snippets.mjs b/web/apps/docs/scripts/lint-snippets.mjs new file mode 100644 index 000000000..963a4f157 --- /dev/null +++ b/web/apps/docs/scripts/lint-snippets.mjs @@ -0,0 +1,309 @@ +#!/usr/bin/env node +// Anti-drift lint for the configuration snippets embedded in the docs. +// +// The configuration reference cannot drift from the binary; it is generated +// from the same Rust types the binary deserializes. Prose pages are a different +// matter: a how-to that shows `evict_bytes` keeps showing `evict_bytes` long +// after the field is renamed, and nothing fails. Readers copy the snippet, the +// binary rejects it with `unknown field`, and the docs look like a liar. +// +// This lint closes that gap without needing a Rust toolchain. It reads the +// *generated* configuration reference (the one that provably matches the +// binary), harvests every field name and every tagged-variant key out of it, +// and then checks that every key used in every JSON5 snippet under +// `content/docs/` appears there. +// +// Usage, from web/: +// bun --filter @nativelink/docs lint:snippets +// or directly: +// node scripts/lint-snippets.mjs +// +// Escapes, for the snippets that legitimately are not NativeLink config: +// - Put `{/* lint-snippets: ignore */}` on the line before the fence. +// - Keys nested under a free-form map (`properties`, `platform_properties`, +// `env`, …) are skipped automatically; those are operator-chosen names. +// +// Exit code is 1 if any snippet uses a key the reference does not know. + +import { readFileSync, readdirSync, statSync } from "node:fs"; +import { dirname, join, relative } from "node:path"; +import { fileURLToPath } from "node:url"; + +const here = dirname(fileURLToPath(import.meta.url)); +const docsRoot = join(here, ".."); +const contentDir = join(docsRoot, "content/docs"); +const REFERENCE = join(contentDir, "reference/nativelink-config/index.mdx"); + +/** Keys under these parents are operator-chosen, not config fields. */ +const FREEFORM_PARENTS = new Set([ + "properties", + "platform_properties", + "supported_platform_properties", + "property_modifications", + "env", + "environment", + "labels", + "annotations", + "const_labels", + "additional_environment", + "upload_action_result", + "headers", +]); + +/** + * Keys that are real config but cannot be harvested from the reference tables, + * because they are map *keys* rather than struct fields. Keep this list short + * and justified; every entry is a hole in the lint. + */ +const KNOWN_MAP_KEYS = new Set([ + // `services.experimental_*` and store names are user-chosen; the entries below + // are the fixed map keys the schema documents in prose rather than as fields. + "main", // conventional instance_name in every example + + // Real fields the generated reference does not mention anywhere, because + // `ExperimentalCloudObjectSpec` renders as a stub; the provider-specific + // halves of that spec never reach a props table or an example. Every entry + // here is a hole in the lint AND a gap in the reference; the fix is to teach + // `scripts/lib/schema-to-mdx.mjs` to expand that spec, after which these + // should be deleted. + "sas_url", // nativelink-config/src/stores.rs:1233 +]); + +const IGNORE_MARKER = "lint-snippets: ignore"; + +// -------------------------------------------------------------------------- + +function walk(dir, out = []) { + for (const name of readdirSync(dir)) { + const full = join(dir, name); + const st = statSync(full); + if (st.isDirectory()) walk(full, out); + else if (name.endsWith(".mdx") || name.endsWith(".md")) out.push(full); + } + return out; +} + +/** Every field name and tagged-variant key the generated reference documents. */ +function harvestKnownKeys(referencePath) { + const text = readFileSync(referencePath, "utf8"); + const known = new Set(KNOWN_MAP_KEYS); + + // Props-table rows: `| \`field\` | type | … |` + for (const m of text.matchAll(/^\|\s*`([a-z0-9_]+)`\s*\|/gm)) known.add(m[1]); + // Tagged-enum variant headings: "### \`fast_slow\`" + for (const m of text.matchAll(/^###\s+`([a-z0-9_]+)`\s*$/gm)) known.add(m[1]); + // Keys used in the reference's own JSON5 examples. Those examples come from + // the doc comments on the same Rust structs the tables come from, so they are + // as authoritative as the tables, and they are the only place some specs get + // documented at all (see the stub harvest below). + for (const block of text.matchAll(/^\s*```json5?\n([\s\S]*?)^\s*```\s*$/gm)) { + for (const m of block[1].matchAll(/^\s*"([a-z0-9_]+)"\s*:/gm)) known.add(m[1]); + } + // Some specs are rendered as a stub ("## ExperimentalCloudObjectSpec" whose + // whole body is "See [`experimental_cloud_object_store`](#…) for details") + // because the schema transform cannot flatten them into a props table. Their + // fields exist only as inline code in the narrative section the stub points + // at, so harvest that section. Scoped deliberately: harvesting inline code + // from the whole reference would make this lint pass on anything. + for (const stub of text.matchAll(/^##\s+\w+\s*\n+See \[`([a-z0-9_]+)`\]\([^)]*\) for details/gm)) { + const start = text.search(new RegExp(`^###\\s+\`${stub[1]}\`\\s*$`, "m")); + if (start === -1) continue; + const rest = text.slice(start + 1); + const end = rest.search(/^#{2,3}\s/m); + const section = end === -1 ? rest : rest.slice(0, end); + for (const m of section.matchAll(/`([a-z][a-z0-9_]{2,})`/g)) known.add(m[1]); + } + + if (known.size < 50) { + throw new Error( + `Harvested only ${known.size} keys from ${relative(docsRoot, referencePath)}: ` + + "the reference format changed and this lint would produce nothing but false " + + "positives. Fix the harvester before trusting a green run.", + ); + } + return known; +} + +/** Fenced json5/json blocks in an MDX file, with their starting line. */ +function extractSnippets(text) { + const snippets = []; + const lines = text.split("\n"); + for (let i = 0; i < lines.length; i++) { + const fence = /^\s*```(json5|json)\b/.exec(lines[i]); + if (!fence) continue; + const prev = lines + .slice(Math.max(0, i - 3), i) + .join("\n"); + const ignored = prev.includes(IGNORE_MARKER); + let j = i + 1; + const body = []; + while (j < lines.length && !/^\s*```\s*$/.test(lines[j])) body.push(lines[j++]); + snippets.push({ startLine: i + 2, lang: fence[1], body: body.join("\n"), ignored }); + i = j; + } + return snippets; +} + +/** Strip comments and string literals so brace/key scanning cannot be fooled. */ +function stripNoise(src) { + let out = ""; + let i = 0; + while (i < src.length) { + const c = src[i]; + if (c === '"' || c === "'") { + const quote = c; + out += " "; + i++; + while (i < src.length && src[i] !== quote) { + if (src[i] === "\\") i++; + i++; + } + i++; + continue; + } + if (c === "/" && src[i + 1] === "/") { + while (i < src.length && src[i] !== "\n") i++; + continue; + } + if (c === "/" && src[i + 1] === "*") { + i += 2; + while (i < src.length && !(src[i] === "*" && src[i + 1] === "/")) { + if (src[i] === "\n") out += "\n"; + i++; + } + i += 2; + continue; + } + out += c; + i++; + } + return out; +} + +/** + * Bare object keys in a JSON5 fragment, with the stack of parent keys at the + * point each appears. Works on fragments (no parse, no balanced-brace + * requirement), so elided examples still get checked. + */ +function scanKeys(src) { + const clean = stripNoise(src); + const found = []; + const stack = []; + let pendingKey = null; + let line = 1; + + const keyRe = /([A-Za-z_][A-Za-z0-9_]*)\s*:/y; + for (let i = 0; i < clean.length; i++) { + const c = clean[i]; + if (c === "\n") { + line++; + continue; + } + if (c === "{" || c === "[") { + stack.push(pendingKey); + pendingKey = null; + continue; + } + if (c === "}" || c === "]") { + stack.pop(); + pendingKey = null; + continue; + } + if (/[A-Za-z_]/.test(c)) { + keyRe.lastIndex = i; + const m = keyRe.exec(clean); + if (m) { + const parents = stack.filter(Boolean); + found.push({ key: m[1], line, parents }); + pendingKey = m[1]; + i = keyRe.lastIndex - 1; + continue; + } + // Not a key; skip the whole identifier so `true`/`null` don't re-trigger. + while (i < clean.length && /[A-Za-z0-9_]/.test(clean[i])) i++; + i--; + } + } + return found; +} + +function braceBalance(src) { + const clean = stripNoise(src); + let curly = 0; + let square = 0; + for (const c of clean) { + if (c === "{") curly++; + else if (c === "}") curly--; + else if (c === "[") square++; + else if (c === "]") square--; + if (curly < 0 || square < 0) return { ok: false, why: "closes a brace it never opened" }; + } + if (curly !== 0) return { ok: false, why: `${curly > 0 ? curly : -curly} unmatched \`{\`/\`}\`` }; + if (square !== 0) return { ok: false, why: `${square > 0 ? square : -square} unmatched \`[\`/\`]\`` }; + return { ok: true }; +} + +// -------------------------------------------------------------------------- + +function main() { + const known = harvestKnownKeys(REFERENCE); + const files = walk(contentDir).filter((f) => !f.includes("reference/nativelink-config/")); + + const problems = []; + let checked = 0; + let skipped = 0; + + for (const abs of files) { + const rel = relative(docsRoot, abs).split("\\").join("/"); + const text = readFileSync(abs, "utf8"); + for (const snip of extractSnippets(text)) { + if (snip.ignored) { + skipped++; + continue; + } + checked++; + + const balance = braceBalance(snip.body); + if (!balance.ok) { + problems.push({ + file: rel, + line: snip.startLine, + message: `snippet is not brace-balanced: ${balance.why}`, + }); + continue; + } + + for (const { key, line, parents } of scanKeys(snip.body)) { + if (parents.some((p) => FREEFORM_PARENTS.has(p))) continue; + if (known.has(key)) continue; + problems.push({ + file: rel, + line: snip.startLine + line - 1, + message: `\`${key}\` is not a field in the generated configuration reference${ + parents.length ? ` (under ${parents.map((p) => `\`${p}\``).join(" › ")})` : "" + }`, + }); + } + } + } + + console.log( + `Checked ${checked} config snippet(s) across ${files.length} page(s) against ` + + `${known.size} known keys${skipped ? `; skipped ${skipped} marked ignore` : ""}.`, + ); + + if (problems.length === 0) { + console.log("No drift found."); + return; + } + + console.error(`\n${problems.length} problem(s):`); + for (const p of problems) console.error(` ${p.file}:${p.line} ${p.message}`); + console.error( + "\nIf a snippet is deliberately not NativeLink config, put " + + `\`{/* ${IGNORE_MARKER} */}\` on the line before its fence.`, + ); + process.exitCode = 1; +} + +main(); diff --git a/web/apps/docs/templates/README.md b/web/apps/docs/templates/README.md new file mode 100644 index 000000000..26cbc28bc --- /dev/null +++ b/web/apps/docs/templates/README.md @@ -0,0 +1,31 @@ +# Page templates + +Four archetypes, one file each. Copy the one that matches what you're writing +into `content/docs/`, keep the structure, replace the content. + +These files live outside `content/docs/` on purpose: `source.config.ts` +publishes everything under that directory, and a template is not a page. + +| Archetype | Use it when the reader wants to | File | +|---|---|---| +| Tutorial | learn by doing, on a guaranteed happy path | `tutorial.mdx` | +| How-to | accomplish one specific task they already have | `how-to.mdx` | +| Explanation | understand why something is shaped the way it is | `explanation.mdx` | +| Reference | look up an exact fact | `reference.mdx` | + +Three rules cut across all four. + +**Reading path vs lookup surface.** Narrative pages never inline an +exhaustive field list. They explain the fields that carry a decision and link +the generated reference for the rest. The reference is exhaustive so the +narrative doesn't have to be. + +**Layer 1 → 2 → 3.** Layer 1 orients and shows the whole thing. Layer 2 is +the working detail. Layer 3 is the edge cases, the troubleshooting, and the +handoff. A beginner stops after layer 1 and is not lost; an expert skips to +layer 2 and is not patronised. + +**Every page on the reading path states what it assumes.** Use +``. A reader who lands cold from a search result should be told +what the page assumes and sent to the page that provides it if they don't have +it. diff --git a/web/apps/docs/templates/explanation.mdx b/web/apps/docs/templates/explanation.mdx new file mode 100644 index 000000000..bd63d851e --- /dev/null +++ b/web/apps/docs/templates/explanation.mdx @@ -0,0 +1,59 @@ +--- +title: How the thing actually works +description: The concept, not the procedure. +--- + +{/* archetype: explanation. Understanding-oriented. This page exists to make + the reader's model correct, so that every how-to afterwards makes sense. + No steps. If you're writing steps, it's a tutorial or a how-to. */} + +{/* ── Layer 1: the mental model ──────────────────────────────────────── */} + +**Who this is for:** the reader who wants to understand rather than do. +**What you'll have at the end:** a model accurate enough to predict the +system's behaviour. **Time:** honest estimate. + +The model in three to five sentences, up front, before any elaboration. A +reader should be able to stop here and still be better off than when they +arrived. + + B[concept] +`} /> + +{/* ── Layer 2: why it's shaped this way ──────────────────────────────── */} + +## Why it works this way + +The narrative: the tradeoffs, the alternatives that were rejected, and what +would break if it were done the usual way instead. Facts alone don't build +a model; a reader who knows *why* can reason about cases this page doesn't +cover, and that's the whole point of an explanation page. + +Where a claim is about actual behaviour, link the code: +. + +{/* ── Layer 3: the exact truth, and the questions ────────────────────── */} + +## Where the exact truth lives + +Field-level and value-level facts belong in the reference, and this page +links there rather than restating them. That link is not a cop-out: it's what +keeps this page true a year from now. + +## Common questions + + + + + The answer, short. + + + + +## What's next + + + Usually a how-to that this model makes tractable. + diff --git a/web/apps/docs/templates/how-to.mdx b/web/apps/docs/templates/how-to.mdx new file mode 100644 index 000000000..d673f2f9e --- /dev/null +++ b/web/apps/docs/templates/how-to.mdx @@ -0,0 +1,73 @@ +--- +title: Do the specific thing +description: Phrased as the task the reader already knows they have. +verified: v1.6.5 +--- + +{/* archetype: how-to. Task-oriented. The reader arrived with the task + already in mind; do not explain the subsystem to them. Complete working + artifact first, then only the parts that carry a decision. */} + +{/* ── Layer 1: orientation + the complete artifact ───────────────────── */} + +**Who this is for:** the reader with this exact task. **What you'll have at +the end:** the task, done. **Time:** honest estimate. + + + A config file you can edit. See [Configuration](/configuration). + + +## The configuration + +The whole thing, working, copy-pasteable. Not a fragment with an ellipsis in +it; a reader who has to reconstruct the surrounding braces will get it +wrong, and the whole point of a how-to is that they don't have to think. + +```json5 +{ + stores: [ + { + name: "CAS_MAIN", + // the complete, runnable thing + }, + ], +} +``` + +{/* ── Layer 2: the parts that matter ─────────────────────────────────── */} + +## What to change + +Only the fields carrying a decision, each one saying what the decision *is* +and what happens either way. For everything else, link the generated +reference rather than restating it; that's what keeps this page correct when +the field list changes. + +| Field | What it decides | Reference | +|---|---|---| +| `some_field` | the tradeoff, in a clause | [reference](/reference/nativelink-config) | + +The implementation is in +if you want to see exactly what the field does. + + + How the reader knows it worked: a metric that moved, a log line, a + response they can curl for. + + +{/* ── Layer 3: when it doesn't work ──────────────────────────────────── */} + +## Troubleshooting + +Real failure modes only, in symptom-first order. The reader is here because +something failed; they're scanning for their symptom, not reading prose. + +| Symptom | Cause | Fix | +|---|---|---| +| what they observe | what's actually wrong | what to do | + +## What's next + + + Where this task usually leads. + diff --git a/web/apps/docs/templates/reference.mdx b/web/apps/docs/templates/reference.mdx new file mode 100644 index 000000000..8dd76c4ef --- /dev/null +++ b/web/apps/docs/templates/reference.mdx @@ -0,0 +1,39 @@ +--- +title: Exhaustive list of the things +description: What this reference covers, in one line. +--- + +{/* archetype: reference. Information-oriented, exhaustive, no narrative. + A reader here has a specific question and wants an answer, not prose. + + IF THIS PAGE IS GENERATED: the generator owns everything below the intro + partial. Do not hand-edit it; edit the source of truth in the repo and + regenerate. Keep the banner. */} + +{/* Generated pages carry this banner verbatim: */} + + + This page is generated from the source. Edits here are overwritten. To + change it, change and + regenerate. + + +Two or three sentences of orientation at most: what's in this reference, how +it's ordered, and where the narrative version lives for a reader who wanted +that instead. Then stop writing prose. + +## `entry_name` + + + +One-line definition. Type, default, and constraints as a list or table. +Exactly one anchor per entry, stable across releases, because these are the +URLs that get cited. + +Source: + +## `next_entry_name` + +Same shape. Alphabetical, or protocol order for protocol references. Never +"grouped by how the author thinks about it"; the reader arrived knowing the +name they're looking for. diff --git a/web/apps/docs/templates/tutorial.mdx b/web/apps/docs/templates/tutorial.mdx new file mode 100644 index 000000000..641f98692 --- /dev/null +++ b/web/apps/docs/templates/tutorial.mdx @@ -0,0 +1,96 @@ +--- +title: The thing they will have built +description: One sentence, written as the outcome, not as the topic. +verified: v1.6.5 +--- + +{/* archetype: tutorial. Learning-oriented, a guaranteed happy path. + Options and alternatives do not belong here; they belong in a how-to. + If you find yourself writing "you could also", stop and move it. */} + +{/* ── Layer 1: orientation ───────────────────────────────────────────── */} + +**Who this is for:** the specific reader, in their own terms. **What you'll +have at the end:** the concrete artifact or capability, not the topic +covered. **Time:** an honest estimate. + + + What the reader must already have, as the state they are in, with a link to + the page that gets them there. See [Getting started](/getting-started). + + +## What you'll build + +One sentence describing the end state, then a diagram of it. Show the shape +before explaining the parts; a reader who can see where the pieces go +follows the steps far better than one assembling a mental model from +instructions. + + B[NativeLink] +`} /> + +{/* ── Layer 2: the working detail ────────────────────────────────────── */} + +## Before you start + +Pinned versions, not "a recent version". A tutorial that worked once should +work again, and a reader who fails at step 6 needs to be able to rule this +out. + +- NativeLink +- Whatever else, with its version + +## Steps + + + +
  • + +### One imperative sentence + +The command, alone, copy-pasteable: + +```bash +nativelink ./config.json5 +``` + +Then what they should see. Every step shows its expected output; this is +what turns a failure into "step 3 was wrong" instead of "it doesn't work". + +``` +[INFO] serving on 0.0.0.0:50051 +``` + +
    • + +
  • + +### The next one + +Same shape. One command, one expected output. If a step needs three +paragraphs of justification, the justification belongs in an explanation page +that this step links. + +
    • + +     + + + The concrete, checkable claim: a number that changed, a line in a log, a + file that now exists. Not "it should work now". A reader must be able to + tell success from silence. + + +{/* ── Layer 3: handoff ───────────────────────────────────────────────── */} + +## What's next + + + What it gets them, in the same voice as the rest of the page. + + + + For the reader whose next question isn't the next page on the path. +