docs: migrate documentation from MkDocs to Fern

[lbliii]

📋 Summary

Adds a Fern Docs build under fern/ alongside the existing MkDocs site. Production target is docs.nvidia.com/nemo/datadesigner with a floating-latest pointer (latest.yml symlink) at v0.5.8. All concept, recipe, plugin, dev-note, and tutorial pages are migrated to MDX with the NVIDIA theme; tutorial notebooks render via <NotebookViewer> with captured outputs (text, DataFrames, inline images).

🔄 Changes

✨ Added

• fern/ Fern Docs build: docs.yml, fern.config.json, NVIDIA theme CSS, version-pinned nav (v0.5.8.yml), 64 MDX pages organized into Concepts, Tutorials, Recipes, Plugins, Code Reference, Dev Notes, plus landing.
• Floating-latest version layout: versions/latest.yml is a Unix symlink to v0.5.8.yml; same MDX backs both slug: latest and slug: v0.5.8. New releases re-target the symlink (recipe in fern/README.md).
• Custom MDX components: Authors, BadgeLinks, CustomCard, CustomFooter, ExpandableCode, MetricsTable, NotebookViewer, Tag, TrajectoryViewer — plus paired CSS in fern/styles/.
• Dev Notes blog kit: 11-author registry (.authors.yml + typed authors-data.ts), each post now uses <Authors ids={[...]} />; deep-research-trajectories renders 31 turns / 50 tool calls via <TrajectoryViewer> driven by typed example data; benchmark tables in text-to-sql/rqa/structured-outputs-from-nemotron/search-agent use <MetricsTable> with auto best-value highlighting.
• Notebook docs pipeline: fern/scripts/ipynb-to-fern-json.py converts .ipynb → fern/components/notebooks/*.{json,ts}, auto-stripping the leading Colab badge cell; new make targets generate-fern-notebooks and generate-fern-notebooks-with-outputs drive the .py → executed .ipynb → Fern JSON+TS chain.
• Python API reference via Fern libraries: pointing at packages/data-designer-config/src/data_designer/config; output (fern/code-reference/) is gitignored and regenerated locally with fern docs md generate.
• fern/README.md maintainer guide: prereqs, first-time setup, versioning recipe, folder layout.

🔧 Changed

• Makefile docs section pins to Python 3.13 via DOCS_PYTHON ?= 3.13 (default Python untouched) so pyarrow resolves wheels on machines running Python 3.14+; certifi-routed CA bundle (DOCS_CERTS) fixes SSL on python.org-installer Pythons; convert-execute-notebooks loops per-file with || failed=... so one notebook's missing API key doesn't kill the chain; generate-fern-notebooks reads from docs/notebooks/ (executed, outputs preserved) when present, falling back to docs/colab_notebooks/ otherwise.
• .gitignore adds fern/code-reference/ (Fern libraries output).
• docs/colab_notebooks/*.ipynb regenerated through convert-execute-notebooks → generate-colab-notebooks so they reflect the latest source.

🧪 Testing

• fern check passes (0 errors, 1 unrelated theme contrast warning)
• make generate-fern-notebooks runs idempotently end-to-end; auto-detects docs/notebooks/ source
• make generate-fern-notebooks-with-outputs produces real LLM outputs for tutorials 1–4 (12.3k captured cells across DataFrame HTML / plain text / inline image MIME types); 5–6 fall back to existing snapshots without OPENROUTER_API_KEY
• fern docs dev boots locally; spot-checked landing, Dev Notes index, tutorials, recipes, deep-research-trajectories
• make test not run (docs-only change; no Python source touched)

✅ Checklist

• Follows commit message conventions (docs: prefix, imperative, ≤72 char subject)
• Commits are signed off (DCO)
• Architecture docs updated — N/A, docs-only

🤖 Generated with Claude Code

[github-actions]

Thank you for your submission! We ask that you all sign our Developer Certificate of Origin before we can accept your contribution. You can sign the DCO by adding a comment below using this text:

---

I have read the DCO document and I hereby sign the DCO.

---

1 out of 2 committers have signed the DCO.
✅ (kirit93)[https://github.com/kirit93]
❌ @lbliii
_{You can retrigger this bot by commenting recheck in this Pull Request. Posted by the DCO Assistant Lite bot.}

[github-actions]

Docs preview: https://be7a8a93.dd-docs-preview.pages.dev

Notebook tutorials are placeholder-only in previews.

[github-actions]

PR #581 Review — docs: migrate documentation from MkDocs to Fern

Summary

This PR stands up a parallel Fern Docs build under fern/ alongside the existing MkDocs site, targeting docs.nvidia.com/nemo/datadesigner. It migrates all concept/recipe/plugin/dev-note/tutorial pages to MDX, adds 10 custom MDX components (rendered in Fern's sandbox), ships a notebook→Fern JSON converter, and pins the docs toolchain to Python 3.13 so pyarrow wheels resolve. The change is docs-only: no touches to packages/ or src/. 195 files changed, the great majority auto-generated (MDX pages, notebook JSON/TS pairs, static assets) — so the review focuses on the new machinery (Makefile, converter script, TSX components, Fern config) rather than the migrated prose content.

Scope is large but well-bounded. The diff is too large for gh pr diff to return in a single pass, so this review inspects the critical non-asset files and samples the rest.

Findings

Correctness & bugs

• fern/versions/_nav_order.yml points at a non-existent directory. All 82 entries reference ./versions/latest/pages/..., but the directory on disk is versions/v0.5.8/pages/... (with versions/latest.yml being a symlink, not a directory). Either this file is stale metadata from a migration script and should be removed, or the paths need to be rewritten to v0.5.8/pages/.... No other config (docs.yml, v0.5.8.yml) appears to consume it, which suggests it's dead. Either way it's confusing — please delete it or document what reads it.

• fern/components/BadgeLinks.tsx still contains scaffold placeholders (your-org/your-repo, your-package). If this component is actually rendered on a page, it will ship the placeholders to production; if it isn't used, delete the component so nothing picks it up accidentally. grep for <BadgeLinks across fern/versions/ before merging.

• NotebookViewer.tsx custom markdown renderer is fragile. The \^@BR\^@ sentinel in renderMarkdown (components/NotebookViewer.tsx:222-226) will collide if any notebook markdown cell literally contains that string. It also doesn't handle fenced code blocks, tables, or nested lists. Notebook markdown is usually simple, so this is a latent risk rather than a current break — but the failure mode (silent corruption) is unpleasant. Consider swapping for a proper markdown lib (Fern's bundler almost certainly ships one) or narrowing the collision risk with a more distinctive sentinel like a � escape.

• Include.tsx also hand-rolls a markdown parser (lines 624+). Two independent mini-parsers in one PR is a maintenance smell — if a bug shows up in one, you'll fix it twice. Same recommendation: adopt a tested library.

• ExpandableCode.tsx copy-button handler captures btn.textContent across an async setTimeout. If React re-renders the button during the 1500 ms window (e.g. route change), restoring orig can produce stale content. Minor, but easy to fix by using a local ref or deriving the original text from props instead of reading the live node.

• fern/versions/latest.yml is a Unix symlink (target v0.5.8.yml, no trailing newline). Windows developers cloning the repo without core.symlinks=true will see a plain text file containing the string v0.5.8.yml rather than a resolved symlink, and Fern will fail with a confusing error. This is likely acceptable given that Fern/docs contributors all work on macOS/Linux, but the fern/README.md versioning section should at least call out the Windows caveat.

Security

• NotebookViewer.tsx injects output cells of format: "html" via dangerouslySetInnerHTML without sanitization (components/NotebookViewer.tsx:493-497). Those are pandas DataFrame HTML reprs today, which is safe — but the source is whatever runs in docs/notebook_source/*.py at pipeline time. If a future notebook ever renders LLM-generated HTML or user-controlled content, this is an XSS sink. Since fern/components/notebooks/*.json is checked in, review happens at commit time, so this is a soft constraint rather than a live vulnerability. Worth a comment in the converter documenting the trust model.

• Include.tsx markdown renderer escapes &<>" but not ' (line 514-520). Given content comes from build-time imports of MDX files in this repo, it's fine today — but the component is documented as reusable (“Copy to your repo's …”), so callers who hand it untrusted content will get partial escaping. Consider including ' or documenting the trusted-input assumption.

• No secrets or tokens appear in the diff. The Adobe DTM launch-*.min.js script loaded via docs.yml is the standard NVIDIA analytics bundle — expected.

Makefile / build pipeline

• DOCS_PYTHON ?= 3.13 and the DOCS_CERTS certifi routing are well-commented and clearly motivated. The per-notebook loop with || failed=... is a real UX improvement over the prior "one failure kills the chain" behavior.

• Cleanup runs unconditionally after the loop: rm -rf docs/notebook_source/artifacts and rm -f docs/notebook_source/*.csv (Makefile:498-499). If no notebooks ran at all (say, uv failed upstream), this still fires — harmless given the paths, but the mv docs/notebook_source/*.ipynb … line before it (Makefile:497) will silently do nothing on total failure and leave behind any partially-executed files in docs/notebook_source/. Minor; current behavior is probably fine.

• generate-fern-notebooks shells out to the converter in a loop inside a single @... recipe. If any single invocation fails, make will stop — but the auto-detection (if ls docs/notebooks/*.ipynb) fails silently to colab_notebooks/ on any non-zero exit, not just "no files." Consider -f docs/notebooks/1-the-basics.ipynb for a more specific probe.

fern/scripts/ipynb-to-fern-json.py

• Clean, typed, SPDX-headered, respects the project's from __future__ import annotations convention. Good.

• Manual sys.argv parsing (if "-o" in args: idx = args.index("-o")) instead of argparse. Trivial, but argparse would give you --help, error messages, and type-checking for free.

• No tests. The converter has real logic (colab-badge detection regex, output-type dispatch, HTML/plain-text selection). A couple of unit tests against a fixture .ipynb would pay for themselves the next time someone tweaks extract_outputs. Not blocking for a docs PR, but worth a follow-up.

• get_language treats language: "python3" as "python" but any other value passes through verbatim ("julia-1.9" → "julia-1.9"). If Pygments doesn't have a lexer for the unnormalized name, highlight_code returns None and the fallback renders unescaped source into dangerouslySetInnerHTML… wait, no: it's source_html ?? escapeHtml(cell.source), so the fallback is escaped. Good.

Fern config (docs.yml, fern.config.json)

• Redirect rules are thorough (Sphinx index.html → slug, foo.html → foo, both versioned and unversioned). Order is correct — :path*/index.html must precede :path*.html, which it does.

• versions: registers both latest and v0.5.8 slugs that point to the same content through the symlink. Nice trick, well documented in fern/README.md.

• fern.config.json pins [email protected]. Pinning is good for reproducibility; make sure CI (or whoever publishes) uses the same version.

Style / minor

• SPDX headers on NotebookViewer.tsx, Authors.tsx, CustomCard.tsx, etc. say (c) 2024 NVIDIA but new files created in 2026 — project convention seems to accept either, but worth aligning.

• Tag.tsx uses React.ReactNode in its signature without a type-only import; other components in the same directory do import type { ReactNode } from "react". Inconsistent. Either works under Fern's JSX runtime, but pick one.

• Many components inline style={{ padding: "0.75rem 1rem", ... }} objects for error states (IncludeError, NotebookViewerError). Those would be better in styles/ CSS files alongside the other styling, matching the pattern already established for non-error cells.

• ExpandableCode.tsx and NotebookViewer.tsx both implement copy-to-clipboard with near-identical logic. Pull into a shared useCopyButton helper or component once a third caller appears.

• fern/components/CustomCard.tsx uses Tailwind utility classes (block p-6 rounded-lg border …). This assumes Fern's bundler ships Tailwind. It apparently does, but worth confirming via fern docs dev that these render as intended, since other components in the PR use plain class names + CSS files.

Test coverage

• Testing section in the PR body is honest: fern check passes; make generate-fern-notebooks* runs idempotently; fern docs dev renders. make test appropriately skipped (no Python runtime code touched). Good.

Verdict

Approve with follow-ups. The migration is well-structured, the build tooling is solid, and the custom components are scoped, typed, and commented. The main blockers to clean up before merge:

1. Resolve fern/versions/_nav_order.yml — delete it or fix the latest/pages vs v0.5.8/pages discrepancy.
2. Confirm BadgeLinks.tsx is either used (with real values substituted) or removed.
3. Document or narrow the XSS trust boundary on NotebookViewer's HTML output-cell rendering.

Nothing in this PR touches production Python code, so the blast radius of issues is confined to the docs site. Ship it after (1) is resolved; (2) and (3) can be fast-follows.

[aschilling-nv]

General Fern feedback

[aschilling-nv]

Suggested change

	experimental:
	mdx-components:
	- ./components
	experimental:
	mdx-components:
	- ./components
	basepath-aware: true