Skip to content

i18n.md

Latest commit

 

History

History
356 lines (287 loc) · 19 KB

i18n.md

File metadata and controls

356 lines (287 loc) · 19 KB
Raw

Internationalization (i18n)

three.ws localizes its UI the way LobeHub does: a single source-of-truth catalog is translated incrementally by an LLM, the output is committed as static JSON, and the browser swaps copy at runtime from those static files. There is no per-request translation call, no runtime cost, and full SEO/caching of each locale's JSON.

The one difference from LobeHub: their copy was already keyed (greenfield i18next). Ours lives inline in 570+ static HTML pages, so we add an extraction step that derives the catalog from annotated markup.

i18n:annotate ──▶ i18n:extract ──▶ public/locales/en.json ──▶ i18n:translate ──▶ public/locales/<lang>.json
 (auto data-i18n)     (build)        (source of truth)        (LLM, glossary-locked)    (committed, static)
        │                                                                                      │
   or hand-annotate                                                              runtime: /i18n.js swaps the DOM

1. Annotate copy

You almost never do this by hand. npm run i18n:annotate scans static HTML and injects the right data-i18n* attribute on every user-visible copy element, so a whole page becomes translatable in one command.

npm run i18n:annotate -- --pages="pages/pricing.html"          # dry-run, preview keys
npm run i18n:annotate -- --pages="pages/pricing.html" --apply  # write attrs + wire /i18n.js
npm run i18n:annotate -- --apply                               # every htmlExtract entry

The annotator (scripts/i18n-annotate.mjs) is built to be safe on hand-authored HTML:

  • Byte-surgical. It parses only to locate elements (via node-html-parser source offsets) and splices data-i18n="…" into the original string. Untouched bytes are preserved exactly (no whole-file reserialization), so diffs are minimal and reviewable, and stripping the inserted attributes reproduces the original file byte-for-byte.
  • Idempotent. An element that already carries an annotation is left alone; re-runs only pick up newly-added copy.
  • Conservative. Only leaf copy elements with real letters are touched. script/style/svg/pre/code/template/textarea, dynamic {{…}}/${…} copy, translate="no" / data-no-i18n opt-outs, and anything nested inside an element it already chose are skipped. <title>, meta description, and og/twitter tags are annotated too.
  • Self-wiring. With --apply it also injects <script type="module" src="/i18n.js"> if the page lacks it, and the runtime then auto-mounts a floating <lang-switcher> on any page that doesn't place its own, so a freshly-annotated page gets a working language picker with zero layout edits.

Keys are namespaced by page (pricing.plans_h, what_is.hero). To annotate by hand, or to understand what the tool emits, the attributes are:

Attribute Applies to Example
data-i18n="key" textContent <span data-i18n="common.take_tour">Take the tour</span>
data-i18n-html="key" innerHTML (copy with inline markup) <li data-i18n-html="home.bullet_earn"><strong>Earn</strong> USDC…</li>
data-i18n-attr="attr:key;…" element attributes <meta data-i18n-attr="content:home.meta_desc" content="…">

Keys are dot-namespaced (home.hero_h, common.language). Group by surface.

2. Extract the source catalog

npm run i18n:extract          # scan annotated HTML → public/locales/en.json
npm run i18n:extract -- --force   # overwrite values from current HTML (after copy edits)
npm run i18n:extract -- --prune   # drop keys no element references anymore

Existing values are preserved by default, so re-running never clobbers a reviewed string. Collisions (same key, different English in two files) fail the command.

3. Translate

npm run i18n:translate                     # translate only missing keys, every locale (incremental)
npm run i18n:translate -- --locale=es      # one locale
npm run i18n:translate -- --force          # retranslate everything
npm run i18n:translate -- --dry-run        # show what would translate (also refreshes manifest)
npm run i18n:translate -- --concurrency=8  # widen the chunk pool for a bulk run
npm run i18n                               # extract + translate

Configuration lives in .i18nrc.json (schema: scripts/i18n-rc.schema.json). Backend is selected by provider, overridable per run with --provider= / --model=. Env-var names match api/_lib/chat-models.js, so a key that already powers /chat works here too. .env and .env.local are read automatically - an exported variable still wins, so a one-off GEMINI_API_KEY=… npm run i18n:translate overrides the file.

provider free? credentials notes
vertex (default) credits GOOGLE_CLOUD_PROJECT + GCP auth billed to the platform's GCP credits; no free-tier quota wall
gemini ✅ free GEMINI_API_KEY / GOOGLE_API_KEY https://aistudio.google.com/apikey - free tier, best CJK
groq ✅ free GROQ_API_KEY https://console.groq.com/keys - Llama 3.3 70B, fast
openrouter ✅ free OPENROUTER_API_KEY https://openrouter.ai/keys - use a :free model
nvidia ✅ free NVIDIA_API_KEY https://build.nvidia.com - free NIM credits
openai paid OPENAI_API_KEY (+ OPENAI_BASE_URL) any OpenAI-compatible endpoint
anthropic paid ANTHROPIC_API_KEY Claude Messages API

Vertex takes GCP credentials three ways, in order: GCP_SERVICE_ACCOUNT_JSON, the Cloud Run / GCE metadata server, then the gcloud CLI. On a developer machine the first two are absent, so an authenticated gcloud auth login is all it needs. If none of the three resolve, the run aborts rather than degrading (see Never silently English below).

An OPENROUTER_API_KEY in the environment is automatically added as a failover rung behind whichever provider is primary, so a single 429 wall does not end a run.

Leave modelName unset to use the provider's default model, or pin one (e.g. "modelName": "meta-llama/llama-3.3-70b-instruct:free" for OpenRouter). Free tiers rate-limit, so concurrency defaults to 2 and the translator backs off and retries on HTTP 429 (honoring Retry-After). A Vertex run billed to our own project has far more headroom - pass --concurrency=8 for a bulk job like generating a locale from scratch (~550 chunks) instead of editing the config every other lane shares.

Never silently English

When a translation call fails, the pipeline bakes the English source into the key so the catalog stays complete and the runtime degrades gracefully. That is right for the occasional value a model can't render as valid JSON, and very wrong for a missing credential, which fails identically on every key: the run would march through the whole catalog rewriting it into English and exit 0, with lint green and the language still offered in the picker. So a credential/config error is classified separately - it is never retried, never baked, and aborts the run with the one line that says what to fix. Everything already translated is kept (the pipeline persists after each chunk) and a re-run resumes.

i18n:backfill (below) exists to clean up catalogs damaged by this before the guard existed.

Quality note: Llama 3.3 70B (Groq) is solid for Latin-script languages. For CJK/Arabic/Hindi, Gemini (free tier or Vertex) is noticeably better - switch per run with no code change. Whatever you pick, npm run i18n:lint catches any model that drops a placeholder, a glossary term, or a tag.

Glossary lock (the important part)

Terms in doNotTranslate — $THREE, the contract address, x402, USDC, Solana, IBM watsonx.ai, etc. - plus every {{placeholder}}, HTML tag, and HTML entity are masked to opaque sentinels before the text reaches the model and restored after. The model literally never sees them, so they come back byte-for-byte and can never be translated, reordered, or hallucinated around. $THREE is the only coin this platform references; this guarantees no machine translation can change that.

Entities are masked for a reason that is easy to miss. A value harvested from a data-i18n-html element is raw markup, and the runtime puts it back through innerHTML. So &lt; is source syntax, not typography: a model that helpfully decodes

<code>&lt;agent-3d&gt;</code>     <!-- a printed code sample -->

into

<code><agent-3d></code>          <!-- a live custom element -->

has just instantiated a 3D avatar in the middle of the SDK docs and swallowed the copy that followed. Masking makes the round trip byte-exact; the lint rule below catches any that predate it.

4. Lint (build gate)

npm run i18n:lint

Validates every committed locale against the source: no missing/empty keys, no {{placeholder}} drift, no dropped glossary terms, no stale keys, and no markup drift. Locales not yet generated are skipped (not failures). Needs no API key - safe in CI.

Markup drift compares the set of HTML tags in the translation against the source. It fires when a translation grows a tag the English never had (an escaped tag that got decoded into a live element, or one the model invented) or loses one it should have kept. It deliberately stays quiet on cosmetic entities: &amp;, &mdash; and &#39; decode to the same glyph the reader sees either way, so flagging them would bury the failures that actually break a page.

Repair (fix lint failures in place)

npm run i18n:repair                 # every locale
npm run i18n:repair -- --locale=he  # one locale

Glossary terms, {{placeholders}}, tags, and entities are masked to sentinels the model must reproduce verbatim. A small model occasionally omits a sentinel mid-sentence, so the term never gets restored and the key fails lint. --repair re-translates only the lint-failing keys (dropped glossary term, placeholder drift, or markup drift), one at a time, retrying each until it passes its own integrity check. A key that never converges after four attempts is baked to English (lint-clean, and the runtime already falls back to English for anything missing). Idempotent: a clean catalog reports nothing to repair. Takes the same --provider= / --model= overrides as i18n:translate.

Backfill (find copy that silently stayed in English)

npm run i18n:backfill                    # report only
npm run i18n:backfill -- --apply         # re-translate every locale
npm run i18n:backfill -- --apply --locale=de

i18n:translate only fills keys that are missing. That interacts badly with the English bake above: when the backend is unreachable (an expired credential, a quota wall), whole runs bake English into the locale, and because those keys now exist, every later run skips them. Nothing reports it. Lint stays green, the key count is complete, and the language picker still offers the language while the page reads as English. This is how Punjabi ended up 99.5% English while looking fully generated.

i18n:backfill compares each locale against the source, clears the values that are byte-identical to English, and re-runs the translator so they come back as real copy. Matching English is not on its own proof of a failure, so it only touches strings carrying two or more real words once glossary terms, markup, URLs and numbers are discounted. A single word (Avatar, 3D), a brand string (three.ws 3D API), a version (OpenAPI 3.1), a copyright line, or a fragment that is mostly markup stays identical in every language, correctly; clearing those would burn a translation call per run to get the same bytes back. Locales are processed worst-first, and it refuses to start while another translation run is live, since two runs rewriting the same catalogs drop each other's keys. Only languages listed in manifest.json are repaired, so a locale nobody can select yet never takes capacity from one visitors are reading; pass --all-locales to include the rest.

Clearing a key is safe at every instant: the runtime resolves a missing or empty value against the English catalog, so a locale mid-backfill renders exactly what it rendered before. Progress is saved per chunk, so an interrupted run keeps what it finished and re-running resumes.

Run it after any incident where the translation backend was down, and any time a locale's word count looks right but the page still reads in English.

5. Runtime

src/i18n.js (shipped at the stable URL /i18n.js) auto-runs on any page that includes it:

<script type="module" src="/i18n.js"></script>
<lang-switcher></lang-switcher>
<!-- accessible language picker -->

It detects the locale (?lang= → localStorage → navigator.languages → default), fetches /locales/<code>.json once, rewrites the annotated DOM, sets <html lang>/dir (RTL-aware), and persists the choice. Missing translations fall back to English, so partial locales degrade gracefully. window.threewsI18n exposes { t, setLocale, getLocale, initI18n }.

The <lang-switcher> reads public/locales/manifest.json, which lists only locales whose catalog is complete, so it never offers a language that would fall back to English. Presence of the file is not enough: writeManifest() counts untranslated (empty) values via untranslatedCount() and holds back any catalog that still has them, reporting what it skipped:

• manifest: 48 locale(s) listed, 1 held back until complete: as (3 untranslated keys)

So a locale that is mid-translation is simply absent from the switcher until the run finishes, instead of shipping as a half-translated page.

Adding a language

Add the code (and display name) to outputLocales / localeNames in .i18nrc.json (plus rtlLocales for a right-to-left script), run npm run i18n:translate -- --locale=<code>, review the output, and commit public/locales/<code>.json. The manifest and switcher pick it up automatically.

Only ship a locale once its catalog is complete. A run that dies partway (expired credentials, an exhausted provider) still leaves a catalog file with the full key skeleton and empty string values for everything it never reached. That file looks finished by key count, and because the runtime falls back to English per key, the page renders as a half-translated mix rather than failing loudly. Verify before committing:

npm run i18n:translate -- --lint --locale=<code>   # flags empty values + glossary drops
node -e "const f=require('./public/locales/<code>.json'),e=require('./public/locales/en.json');
const n=o=>Object.values(o).reduce((a,s)=>a+Object.keys(s).length,0);
let m=0;for(const s of Object.values(f))for(const v of Object.values(s))if(!String(v).trim())m++;
console.log('keys',n(f),'of',n(e),'| empty',m)"

Re-running i18n:translate is the fix: missingKeys() treats an empty string as untranslated, so a resumed run tops up exactly the gaps and costs nothing for the keys already done. For a catalog that is complete but trips the glossary/placeholder/markup lint, npm run i18n:repair -- --locale=<code> re-translates only the failing keys.

Shipping is gated on completeness rather than on your memory: every run rewrites public/locales/manifest.json and omits any locale with untranslated keys, so a half-built catalog is never offered in the switcher. That is what makes a from-scratch locale safe to build incrementally across several sessions.

Building the locales that have never been generated

outputLocales is deliberately longer than the shipped set. To work through the backlog:

scripts/i18n-generate-queue.sh          # every incomplete locale, 5 at a time
scripts/i18n-generate-queue.sh 3        # narrower pool
scripts/i18n-generate-queue.sh 5 "ca sr my"   # a specific list, in order

Each locale is ~18k keys / ~430 chunks, so this is an hours-long job, not a minutes-long one. It picks locales most-complete-first (closest to shipping ships soonest), skips any locale another process is already translating (two writers on one catalog would lose one set of writes), and is resumable: the translator persists after every chunk, so a killed queue loses at most one chunk.

Concurrent agents share this worktree. Before starting a bulk run, check ps -eo args | grep i18n- - a locale being written by someone else's backfill must not also be written by you.

SEO: hreflang alternates

Copy swaps client-side (the LobeHub model), and each locale is reachable at a distinct URL via ?lang=<code>, which detectLocale() honors. Search engines are told those translated URLs exist two ways, both driven off the manifest so they track annotation automatically:

  1. Sitemap alternates. npm run build:pages writes public/locales/localized-pages.json (every registered page whose HTML carries a data-i18n annotation). The dynamic core sitemap (api/sitemap/[type].js) reads it and emits an <xhtml:link rel="alternate" hreflang> per locale, plus x-default, for exactly those routes.
  2. On-page alternates. The runtime injects <link rel="alternate" hreflang> tags into <head> for the current path (default locale and x-default at the bare URL, others at ?lang=xx), so a page crawled directly advertises its translations even without the sitemap.

<html lang>/dir are set to the active locale on every swap. A build-time prerender that bakes each locale into static HTML (eliminating the first-paint English flash) is the natural next step; the catalogs are static and ready.

Tests

tests/i18n.test.js covers extraction, glossary round-trips, lint rules, runtime resolution/fallback, and catalog merge/prune. tests/i18n-markup.test.js pins the markup guarantees: entities survive a round trip byte-for-byte, an escaped tag that got decoded into a live element fails lint, and cosmetic entities do not. tests/i18n-backfill.test.js pins the rules for clearing silently-English values out of a shipped catalog.