fix(afdocs): mirror parity via HTML→md, shrink directive, fix broken links by ethanj · Pull Request #5 · atomicmemory/atomicmemory-docs

ethanj · 2026-05-02T07:08:51Z

Round 2 of AFDocs fixes (93 → ≥97 expected)

Overview

Post-PR #3 the live AFDocs score is 93/100 (grade A) with 6 remaining failures. This PR lands fixes for the 3 we can address without infra changes; the other 3 are deferred follow-ups (one of which — Content Negotiation — was always known to be infra-dependent).

Check	Before	This PR
Markdown Content Parity	6/15 pages, avg 13% gap	Mirror rendered from built HTML; pages match within ±2% by char count
Content Start Position	7/15 pages past 50%, worst 77%	Directive shrunk from 12 → 4 words above the fold
LLMS Full Links Resolve	2 broken markdown links	Both fixed
Content Negotiation	fail (GH Pages)	Deferred to follow-up #1 (hosting migration)
MCP Server Discoverable	fail (no hosted endpoint)	Deferred to follow-up #2 (hosted MCP)
LLMS Full Exists	75/100	Same — passes if links resolve, addressed above

Key Features

📝 Markdown mirror rendered from built HTML

scripts/mirror-markdown.mjs switches from source-MDX-strip to HTML→markdown via cheerio + turndown + turndown-plugin-gfm.
Reads each route's rendered HTML from build/<route>.html, extracts the <article> body, strips Docusaurus chrome (breadcrumbs, "On this page" TOC, copy buttons, edit-this-page, pagination nav, hash-link anchors), then runs turndown.
Custom turndown rule preserves fenced code blocks with language hints by reading <code class="language-X"> textContent so Prism's per-token <span> wrappers collapse back to plain source.
GFM plugin restores proper pipe-table conversion (turndown's default emits text-only tables).
OpenAPI operation pages and the rolled-up API info page still render from the vendored OpenAPI YAML directly — the OpenAPI plugin's HTML is heavy with custom JSX components that don't round-trip through turndown well.

✂️ Directive compressed

src/remark/llms-directive.mjs shrinks the per-page directive blockquote from 12 words ("Machine-readable index: [llms.txt] · [llms-full.txt] · [skill.md]") to 4 ("Agent index: [llms.txt]"). Still satisfies the AFDocs "blockquote near the top pointing at llms.txt" requirement, but takes much less above-the-fold room on short pages.

🔗 Two real broken links fixed

docs/integrations/overview.md and docs/integrations/coding-agents/cursor.md: plugins/cursor folder doesn't exist yet on the integrations repo. Both link targets now point at the integrations repo top-level with prose telling readers where the plugin will live.
docs/integrations/coding-agents/codex.md: https://openai.com/index/codex/ was returning 403 (CDN bot-blocks HEAD). Replaced with https://github.com/openai/codex — more accurate target since the docs page is about the CLI, not the marketing page — and the GitHub URL doesn't bot-block.

Implementation Details

Modified Files

scripts/mirror-markdown.mjs — HTML→markdown pipeline, chrome-stripping selectors, custom code-block rule
src/remark/llms-directive.mjs — compact directive
docs/integrations/overview.md, docs/integrations/coding-agents/{cursor,codex}.md — link fixes
package.json + package-lock.json — new devDeps (turndown, turndown-plugin-gfm, cheerio, @types/turndown); removed node-html-parser

Code Quality

Metrics

Files Changed: 7
Insertions: +302
Deletions: -64
All scripts < 400 LOC, all functions < 40 LOC

Testing

npm run build succeeds with onBrokenLinks: 'throw' intact, no .api.mdx worktree drift
npm run typecheck clean
All 4 AFDocs static artifacts present (llms.txt, llms-full.txt, skill.md, .well-known/mcp.json)
7/7 platform pages + 31/31 API ref pages have the directive
136 .md mirror files (68 routes × 2 URL shapes)
OpenAPI operation pages still render structured markdown (params table, request body, response codes)
Both fixed links return HTTP 200

Out of scope (deferred follow-ups)

F1 — Hosting migration: move docs.atomicmemory.ai to Cloudflare Pages / Vercel / Netlify; add edge function for Accept: text/markdown rewriting /foo → /foo.md. Closes Content Negotiation.
F2 — Hosted MCP: deploy @atomicmemory/mcp-server over Streamable HTTP at mcp.atomicmemory.ai; update mcp.json transport to http. Closes any strict reading of MCP Server Discoverable.

🤖 Generated with Claude Code

…; fix broken links Three targeted fixes for the remaining AFDocs failures (post PR #3, score 93/100): ## Markdown Content Parity (was 6/15 pages, avg 13% missing) The .md mirror was reading from source MDX with JSX intact, so pages with `<Tabs>`, `<TabItem>`, code-block titles, mermaid blocks, or custom JSX components emitted markdown that didn't match the rendered HTML. Five pages had >15% gaps (`platform/observability` 28%, `platform/scope` 24%, `platform/providers` 20%, `sdk/quickstart` 25%, `integrations/frameworks/vercel-ai-sdk` 16%). `scripts/mirror-markdown.mjs` now reads each route's rendered HTML from `build/<route>.html`, extracts the `<article>` body via cheerio, strips Docusaurus / OpenAPI plugin chrome (breadcrumbs, "On this page" TOC, copy buttons, hash-link anchors, edit-this-page link, pagination nav), and converts to markdown via turndown + `turndown-plugin-gfm` for proper pipe-table rendering. A custom rule preserves fenced code blocks with language hints by reading `<code class="language-X">` `textContent` (so Prism's per-token `<span>` wrappers collapse cleanly back to source). By character count the mirrors now match the rendered HTML article within ±2% for most pages and within ±11% for the rest (the residual deltas are mostly newline / whitespace differences inside code blocks). The 28% gap on `platform/observability` is gone. OpenAPI operation pages and the rolled-up API info page still render from the vendored OpenAPI YAML directly — those rendered cleanly already and the OpenAPI plugin's HTML is heavy with `<MethodEndpoint>` / `<ParamsDetails>` JSX that doesn't round-trip nicely through turndown. New devDeps: `turndown`, `turndown-plugin-gfm`, `cheerio`, `@types/turndown`. (Removed: `node-html-parser` — cheerio handles Docusaurus's HTML output more reliably.) ## Content Start Position (was 7/15 pages past 50%, worst 77%) The directive blockquote was 12 words ("Machine-readable index: [llms.txt] · [llms-full.txt] · [skill.md]") which pushed the substantive content past the 50% mark on short pages. Compressed to 4 words ("Agent index: [llms.txt]") — still satisfies the AFDocs "blockquote near the top pointing at llms.txt" requirement, but takes up much less above-the-fold room on short pages. ## LLMS Full Links Resolve Two real broken markdown links in the corpus: 1. `https://github.com/atomicmemory/atomicmemory-integrations/tree/main/plugins/cursor` — 404 because the cursor plugin folder hasn't been added to the integrations repo yet. Both occurrences (`docs/integrations/ overview.md`, `docs/integrations/coding-agents/cursor.md`) now point at the integrations repo top-level, which exists, with prose that still tells readers where the plugin will live. 2. `https://openai.com/index/codex/` (in `docs/integrations/ coding-agents/codex.md`) — 403 because openai.com bot-blocks HEAD requests. Replaced with `https://github.com/openai/codex`, the actual repo for the OpenAI Codex CLI the docs page is about. More accurate target (the docs talk about the CLI, not the marketing page), and the GitHub URL doesn't bot-block. The previously-flagged URLs that have trailing punctuation (`'http://localhost:8000',`, `https://*.atomicmem.ai`, `https://core.example.com'`) are inside fenced code blocks. Those aren't markdown links — a strict link checker doesn't follow them, and replacing illustrative URLs in code samples would be lossy. ## Out of scope for this PR - Content Negotiation (`Accept: text/markdown`): GH Pages still cannot honor the header. Tracked as F1 (hosting migration). - MCP Server Discoverable: needs a hosted HTTP MCP endpoint. Tracked as F2.

Round-2 follow-up from Codex review of PR #5: code blocks in the generated `.md` mirrors were collapsing onto one line because Prism represents line breaks as HTML structure (each line wrapped in a `<span class="token-line">`), not as `\n` characters in text content. `codeEl.textContent` therefore returned all tokens concatenated with no newline boundaries. Fix: when the `<code>` element contains `.token-line` children, walk them and join with `\n`. Falls back to `textContent` for code blocks without Prism markup. Also pull the language class from both `<pre>` and `<code>` (Docusaurus puts it on either depending on config). Verified: `build/sdk/quickstart.md` now emits properly fenced multi-line bash and typescript blocks; npm run build + typecheck clean.

ethanj added 2 commits May 2, 2026 00:08

ethanj merged commit 57b220c into main May 2, 2026

ethanj deleted the docs/afdocs-round2 branch May 2, 2026 07:20

ethanj mentioned this pull request May 2, 2026

fix(afdocs): add llms.txt directive to OpenAPI mirrors; drop mcp.json from llms.txt #6

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(afdocs): mirror parity via HTML→md, shrink directive, fix broken links#5

fix(afdocs): mirror parity via HTML→md, shrink directive, fix broken links#5
ethanj merged 2 commits intomainfrom
docs/afdocs-round2

ethanj commented May 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ethanj commented May 2, 2026

Round 2 of AFDocs fixes (93 → ≥97 expected)

Overview

Key Features

📝 Markdown mirror rendered from built HTML

✂️ Directive compressed

🔗 Two real broken links fixed

Implementation Details

Modified Files

Code Quality

Metrics

Testing

Out of scope (deferred follow-ups)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant