From a54b348d407118fc0f1d4fed8543b719e8c99d9a Mon Sep 17 00:00:00 2001 From: Samuel Stroschein <35429197+samuelstroschein@users.noreply.github.com> Date: Tue, 25 Aug 2026 17:37:24 +0000 Subject: [PATCH 1/2] refactor(sdk): remove obsolete SQLite WASM path --- .changeset/remove-sdk-sqlite-wasm.md | 5 + docs/architecture.md | 6 +- docs/getting-started.md | 2 +- docs/introduction.md | 4 +- docs/unpacked-project.md | 14 +- docs/version-control.md | 2 +- docs/write-tool.md | 2 +- packages/sdk/README.md | 8 +- packages/sdk/benchmark/README.md | 74 --- .../sdk/benchmark/common-operations.bench.ts | 228 --------- packages/sdk/package.json | 2 - packages/sdk/src/database/jsonbPlugin.test.ts | 213 --------- packages/sdk/src/database/jsonbPlugin.ts | 280 ----------- packages/sdk/src/database/lixDialect.test.ts | 19 + packages/sdk/src/database/schema.ts | 44 -- packages/sdk/src/import-export/importFiles.ts | 7 +- .../sdk/src/lix-plugin/applyChanges.test.ts | 150 ------ packages/sdk/src/lix-plugin/applyChanges.ts | 171 ------- .../src/lix-plugin/detectConflicts.test.ts | 286 ------------ .../sdk/src/lix-plugin/detectConflicts.ts | 62 --- .../src/lix-plugin/inlangLixPluginV1.test.ts | 439 ------------------ .../sdk/src/lix-plugin/inlangLixPluginV1.ts | 132 ------ packages/sdk/src/lix-plugin/merge.test.ts | 133 ------ .../resolveConflictBySelecting.test.ts | 188 -------- packages/sdk/src/project/README_CONTENT.ts | 6 +- .../project/loadProjectFromDirectory.test.ts | 2 +- .../src/project/loadProjectFromDirectory.ts | 5 +- .../project/saveProjectToDirectory.test.ts | 3 +- .../sdk/src/project/saveProjectToDirectory.ts | 2 +- packages/sdk/src/project/snapshot.ts | 2 +- .../src/query-utilities/selectBundleNested.ts | 3 +- .../website-v2/src/content/landingpage.md | 4 +- pnpm-lock.yaml | 19 - 33 files changed, 57 insertions(+), 2460 deletions(-) create mode 100644 .changeset/remove-sdk-sqlite-wasm.md delete mode 100644 packages/sdk/benchmark/README.md delete mode 100644 packages/sdk/benchmark/common-operations.bench.ts delete mode 100644 packages/sdk/src/database/jsonbPlugin.test.ts delete mode 100644 packages/sdk/src/database/jsonbPlugin.ts delete mode 100644 packages/sdk/src/lix-plugin/applyChanges.test.ts delete mode 100644 packages/sdk/src/lix-plugin/applyChanges.ts delete mode 100644 packages/sdk/src/lix-plugin/detectConflicts.test.ts delete mode 100644 packages/sdk/src/lix-plugin/detectConflicts.ts delete mode 100644 packages/sdk/src/lix-plugin/inlangLixPluginV1.test.ts delete mode 100644 packages/sdk/src/lix-plugin/inlangLixPluginV1.ts delete mode 100644 packages/sdk/src/lix-plugin/merge.test.ts delete mode 100644 packages/sdk/src/lix-plugin/resolveConflictBySelecting.test.ts diff --git a/.changeset/remove-sdk-sqlite-wasm.md b/.changeset/remove-sdk-sqlite-wasm.md new file mode 100644 index 0000000000..81ec34ccd9 --- /dev/null +++ b/.changeset/remove-sdk-sqlite-wasm.md @@ -0,0 +1,5 @@ +--- +"@inlang/sdk": major +--- + +Remove the obsolete SQLite WASM dependency, public schema initializer, and special handling for unsupported legacy database artifacts. The SDK database API uses Lix through Kysely's PostgreSQL query compiler. diff --git a/docs/architecture.md b/docs/architecture.md index 503bc989ee..9438dffa64 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -4,7 +4,7 @@ Inlang is the open-format TMS (translation management system) for software teams ``` ┌─────────────────────────────────────────────┐ -│ Storage (SQLite + version control via Lix) │ +│ Storage and version control (Lix) │ ├─────────────────────────────────────────────┤ │ Data Model (Bundle, Message, Variant) │ ├─────────────────────────────────────────────┤ @@ -14,9 +14,9 @@ Inlang is the open-format TMS (translation management system) for software teams ## Storage -An `.inlang` project is canonically a single binary file: a SQLite database with version control via [lix](https://lix.dev). Like `.sqlite` for relational data, `.inlang` packages localization data into one file that tools can share. +An `.inlang` project is canonically a portable snapshot backed by [Lix](https://lix.dev). It packages localization data and project files into one file that tools can share. -For Git repositories, the binary file can be unpacked into a directory of plain files so changes can be reviewed alongside code. The packed file is the canonical format; the unpacked directory is the Git-friendly representation. +For Git repositories, the packed file can be unpacked into a directory of plain files so changes can be reviewed alongside code. The packed file is the canonical format; the unpacked directory is the Git-friendly representation. Inlang defines the localization format and TMS surface. Lix provides the underlying versioning, history, review, change proposals, rollback, and merging infrastructure. diff --git a/docs/getting-started.md b/docs/getting-started.md index 3aea75c781..8358040e06 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -6,7 +6,7 @@ Store translations in your repo as a vendor-neutral file format, so developers, Use inlang when localization data needs to be shared across tools, teams, automations, or coding agents. If you only need an app runtime with a couple of translation files, your current i18n setup may already be enough. -An `.inlang` project is canonically a single binary file: a SQLite database with version control via [lix](https://lix.dev). Inlang defines the localization format and TMS surface. Lix provides the underlying versioning, history, review, change proposals, and rollback infrastructure. +An `.inlang` project is canonically a portable snapshot backed by [Lix](https://lix.dev). Inlang defines the localization format and TMS surface. Lix provides the underlying storage, versioning, history, review, change proposals, and rollback infrastructure. For Git repositories, the file can be unpacked into a directory of plain files. The packed file is the canonical format; the unpacked directory is the Git-friendly representation. diff --git a/docs/introduction.md b/docs/introduction.md index 71bc51f746..9505ee559b 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -6,7 +6,7 @@ Inlang is the open-format TMS (translation management system) for software teams Store translations in your repo as a vendor-neutral file format, so developers, translators, CI, translation tools, and AI agents can read and update the same localization source of truth. -An `.inlang` project is canonically a single binary file: a SQLite database with version control via [lix](https://lix.dev). Like `.sqlite` for relational data, `.inlang` packages localization data into one file that tools can share. +An `.inlang` project is canonically a portable snapshot backed by [Lix](https://lix.dev). It packages localization data and project files into one file that tools can share. The `@inlang/sdk` is the reference implementation for reading and writing `.inlang` projects. @@ -16,7 +16,7 @@ Inlang defines the localization format and TMS surface. [Lix](https://lix.dev) p Messages, variants, and locale data live in the `.inlang` database. External translation files such as `messages/en.json` are compatibility files outside `project.inlang/`, connected through plugins. -For Git repositories, the binary file can be unpacked into a directory of plain files so changes can be reviewed alongside code. The packed file is the canonical format; the unpacked directory is the Git-friendly representation. +For Git repositories, the packed file can be unpacked into a directory of plain files so changes can be reviewed alongside code. The packed file is the canonical format; the unpacked directory is the Git-friendly representation. The SDK has two main parts: diff --git a/docs/unpacked-project.md b/docs/unpacked-project.md index 0bbabde414..d908833017 100644 --- a/docs/unpacked-project.md +++ b/docs/unpacked-project.md @@ -2,7 +2,7 @@ ## What is an unpacked project? -An unpacked project is the Git-friendly representation of an `.inlang` file. The canonical `.inlang` format is a single binary file: a SQLite database with version control via [lix](https://lix.dev). The unpacked directory exists so changes can be reviewed alongside code. +An unpacked project is the Git-friendly representation of an `.inlang` file. The canonical `.inlang` format is a portable snapshot backed by [Lix](https://lix.dev). The unpacked directory exists so changes can be reviewed alongside code. Messages, variants, and locale data live in the `.inlang` database. In unpacked Git projects, `settings.json` is the only tracked project file by default; translation files such as `messages/en.json` live outside `project.inlang/` and are connected through plugins. @@ -21,8 +21,8 @@ project.inlang/ | | Packed (`.inlang` file) | Unpacked (directory) | | ---------------- | --------------------------------- | ------------------------------------- | -| **Format** | Canonical single binary file | Git-friendly directory representation | -| **Git-friendly** | No (binary) | Yes (diffable, mergeable) | +| **Format** | Canonical portable snapshot | Git-friendly directory representation | +| **Git-friendly** | Limited | Yes (diffable, mergeable) | | **Portable** | Yes (one file to share) | No | | **Use case** | Sharing, backups, tools like Fink | Storing in git repos | @@ -32,12 +32,12 @@ project.inlang/ Most codebases use git for version control. Developers want their translations co-located with their code — not in a separate system. -### Git doesn't handle binary files well +### Packed snapshots are difficult to review in Git -An `.inlang` file is binary. Git can store binary files, but you lose: +Git can store a packed `.inlang` snapshot, but it is not organized for review: -- **Readable diffs** — Binary changes show as "file changed", not what changed -- **Merge conflict resolution** — Git can't merge binary files +- **Readable diffs** — Snapshot changes do not map cleanly to project files +- **Merge conflict resolution** — Snapshot-level conflicts are difficult to resolve - **Code review** — Teammates can't review translation changes in PRs An unpacked project solves this for the project configuration. The generated `.gitignore` keeps `settings.json` in Git and ignores generated/cache files. Translation files are stored outside `project.inlang/` according to plugin configuration. diff --git a/docs/version-control.md b/docs/version-control.md index 925638d27c..b89d2628f7 100644 --- a/docs/version-control.md +++ b/docs/version-control.md @@ -4,7 +4,7 @@ Inlang uses [Lix](https://lix.dev) for versioning, history, review, change propo Inlang defines the localization format and TMS surface. Lix provides the version-control layer underneath the `.inlang` project, so localization changes can be reviewed and merged without turning a vendor database into the source of truth. -An `.inlang` project is canonically a single binary file. For repositories, it can be unpacked into plain files so localization changes can be reviewed alongside code. The packed file is the canonical format; the unpacked directory is the repository-friendly representation. +An `.inlang` project is canonically a portable snapshot. For repositories, it can be unpacked into plain files so localization changes can be reviewed alongside code. The packed file is the canonical format; the unpacked directory is the repository-friendly representation. Use this layer for: diff --git a/docs/write-tool.md b/docs/write-tool.md index 46c3d12a88..3fb82345d8 100644 --- a/docs/write-tool.md +++ b/docs/write-tool.md @@ -6,7 +6,7 @@ This guide walks through building a tool that flags missing translations. By the Tools read and write translations through the `.inlang` format via the CRUD API. Because plugins handle conversion at the boundary, your tool works with any translation format — JSON, XLIFF, i18next, etc. — without parsing each one directly. -An `.inlang` project is canonically a single binary file. In Git repositories, it is often unpacked into a directory; `loadProjectFromDirectory()` loads that Git-friendly representation. +An `.inlang` project is canonically a portable snapshot. In Git repositories, it is often unpacked into a directory; `loadProjectFromDirectory()` loads that Git-friendly representation. If a `project.inlang/` directory already exists, load it with `loadProjectFromDirectory()`. If your tool is generating a new localization project from scratch, start with `newProject()` and save the packed file with `project.toBlob()`; see [Getting Started](/docs/getting-started) for a runnable create-save-reload example. diff --git a/packages/sdk/README.md b/packages/sdk/README.md index e2e1dadb1b..9c1ff114ad 100644 --- a/packages/sdk/README.md +++ b/packages/sdk/README.md @@ -22,9 +22,9 @@ The inlang SDK is the reference implementation for reading and writing `.inlang` `.inlang` files are designed to become the open standard for localization data and make i18n tools work together. Build editors, CLIs, runtimes, agents, and plugins on the same shared project format instead of inventing another file structure. -An `.inlang` project is canonically a single binary file: a SQLite database with version control via [lix](https://lix.dev). Like `.sqlite` for relational data, `.inlang` packages localization data into one file that tools can share. +An `.inlang` project is canonically a portable snapshot backed by [Lix](https://lix.dev). It packages localization data and project files into one file that tools can share. -For Git repositories, the binary file can be unpacked into a directory of plain files so changes can be reviewed alongside code. The packed file is the canonical format; the unpacked directory is the Git-friendly representation. +For Git repositories, the packed file can be unpacked into a directory of plain files so changes can be reviewed alongside code. The packed file is the canonical format; the unpacked directory is the Git-friendly representation. `.inlang` is the canonical project format. Plugins import and export formats like JSON, ICU MessageFormat v1, i18next, and XLIFF for compatibility with existing translation files and runtimes. Version control via lix adds file-level history, merging, and change proposals to `.inlang` projects. @@ -68,7 +68,7 @@ For coding agents: ## Getting Started > [!Note] -> Inlang files are single binary files. They can be unpacked and [stored as directories](#unpacked-inlang-files-directories) when you want to review changes in Git. The packed file remains the canonical format. +> Inlang projects are portable snapshots. They can be unpacked and [stored as directories](#unpacked-inlang-files-directories) when you want to review changes in Git. The packed file remains the canonical format. ### Installation @@ -259,7 +259,7 @@ await project.settings.set(settings) > [!NOTE] > Unpacked inlang files are the Git-friendly representation of packed `.inlang` files. > -> Git can store binary files, but plain-file review and merge workflows work better with the unpacked directory. **If you don't intend to store the inlang file in git, use the packed binary file.** +> Git can store packed snapshots, but plain-file review and merge workflows work better with the unpacked directory. **If you don't intend to store the inlang file in git, use the packed file.** > > Unpacked inlang files are not portable. They depend on plugins and do not persist [version control via lix](https://lix.dev/) data. diff --git a/packages/sdk/benchmark/README.md b/packages/sdk/benchmark/README.md deleted file mode 100644 index d9cc8efbdc..0000000000 --- a/packages/sdk/benchmark/README.md +++ /dev/null @@ -1,74 +0,0 @@ -# Lix 0.10 in-memory performance - -This benchmark models the common database operations used by Paraglide JS and -Sherlock. It compares the Lix 0.10 in-memory message store with the former -SQLite-WASM store through the same public inlang Kysely APIs. - -## Workload - -- 500 bundles -- two locales per bundle (`en`, `de`) -- 1,000 messages and 1,000 variants total -- Paraglide: read the complete nested project for compilation -- Sherlock: read one nested bundle, update one variant, and insert one bundle - with two translations -- Project load: restore the populated inlang snapshot into a fresh in-memory - Lix and close it - -## Results - -The table reports the mean latency averaged across two benchmark runs. Lower is -better. The delta is Lix latency divided by SQLite-WASM latency. - -| Operation | Lix 0.10 memory | SQLite-WASM | Delta | -| -------------------------------- | -------------: | ----------: | ------------: | -| Paraglide full nested read | 22.053 ms | 50.556 ms | 2.29x faster | -| Sherlock point nested read | 0.110 ms | 0.165 ms | 1.50x slower | -| Sherlock variant update | 3.045 ms | 0.025 ms | 121.32x slower | -| Sherlock nested insert | 4.440 ms | 0.289 ms | 15.39x slower | -| Load populated in-memory project | 140.645 ms | n/a | n/a | - -The two individual Lix means were: - -| Operation | Run 1 | Run 2 | -| ----------------- | ---------: | ---------: | -| Full nested read | 20.146 ms | 23.960 ms | -| Point nested read | 0.104 ms | 0.116 ms | -| Variant update | 3.051 ms | 3.039 ms | -| Nested insert | 4.195 ms | 4.686 ms | -| Project load |134.250 ms |147.040 ms | - -The bulk Paraglide query benefits from a single flat join and JavaScript -reconstruction. Lix 0.10's native `INSERT ... RETURNING` and `DEFAULT VALUES` -support remove the v0.9 compatibility readbacks, bringing nested insert latency -down substantially. Point writes still pay a higher Lix engine round-trip and -change-recording cost than SQLite-WASM. - -## Environment - -- Linux 7.0.0-22-generic, x86_64 -- AMD EPYC-Genoa, 16 vCPUs -- Node.js 22.22.1 -- pnpm 10.23.0 -- Vitest 3.2.4 / Tinybench -- `@lix-js/sdk` 0.10.0 - -## Reproduce - -From the repository root: - -```sh -pnpm --filter @inlang/sdk bench -``` - -Vitest runs each operation for 1.5 seconds after a 300 ms warmup. Project load -runs for 2 seconds after the same warmup. - -## Persistence note - -The v0.10 Node native in-memory backend cannot export or import an opaque Lix -snapshot; the SDK explicitly reports that memory snapshots are browser-only. -The migration therefore serializes the current inlang entities and project -files into a portable JSON blob and restores them into a fresh in-memory Lix. -This preserves current project state but not Lix branch/history metadata, and -legacy v0.4 SQLite `.inlang` blobs require a separate migration path. diff --git a/packages/sdk/benchmark/common-operations.bench.ts b/packages/sdk/benchmark/common-operations.bench.ts deleted file mode 100644 index cb9bfce7ac..0000000000 --- a/packages/sdk/benchmark/common-operations.bench.ts +++ /dev/null @@ -1,228 +0,0 @@ -import { afterAll, beforeAll, bench } from "vitest"; -import { CamelCasePlugin, Kysely } from "kysely"; -import { openLix, type Lix, type LixBatchStatement } from "@lix-js/sdk"; -import { - createDialect, - createInMemoryDatabase, - type SqliteWasmDatabase, -} from "sqlite-wasm-kysely"; -import { v7 } from "uuid"; -import { - applySchema, - type InlangDatabaseSchema, -} from "../src/database/schema.js"; -import { JsonbPlugin } from "../src/database/jsonbPlugin.js"; -import { initDb } from "../src/database/initDb.js"; -import { registerInlangSchemas } from "../src/database/registerSchemas.js"; -import { selectBundleNested } from "../src/query-utilities/selectBundleNested.js"; -import { insertBundleNested } from "../src/query-utilities/insertBundleNested.js"; -import { projectToBlob } from "../src/project/snapshot.js"; -import { loadProjectInMemory } from "../src/project/loadProjectInMemory.js"; -import { humanId } from "../src/human-id/human-id.js"; - -const BUNDLE_COUNT = 500; -const LOCALES = ["en", "de"] as const; - -let lix: Lix; -let lixDb: Kysely; -let sqlite: SqliteWasmDatabase; -let sqliteDb: Kysely; -let populatedBlob: Blob; -let mutationCounter = 0; - -beforeAll(async () => { - lix = await openLix(); - await registerInlangSchemas(lix); - lixDb = initDb({ lix }); - await seedLix(lix); - - sqlite = await createInMemoryDatabase({ readOnly: false }); - sqlite.createFunction({ name: "uuid_v7", arity: 0, xFunc: () => v7() }); - sqlite.createFunction({ name: "human_id", arity: 0, xFunc: () => humanId() }); - applySchema({ sqlite }); - sqliteDb = new Kysely({ - dialect: createDialect({ database: sqlite }), - plugins: [new CamelCasePlugin(), new JsonbPlugin({ database: sqlite })], - }); - await seedKysely(sqliteDb); - - await lix.executeBatch([ - { - sql: "INSERT INTO lix_file (path, content) VALUES ($1, $2)", - params: [ - "/settings.json", - new TextEncoder().encode( - JSON.stringify({ - baseLocale: "en", - locales: [...LOCALES], - modules: [], - }) - ), - ], - }, - ]); - populatedBlob = await projectToBlob(lix); -}, 120_000); - -afterAll(async () => { - await lixDb.destroy(); - await lix.close(); - await sqliteDb.destroy(); - sqlite.close(); -}); - -bench( - "paraglide: select 500 bundles / 1,000 messages nested — Lix 0.10 memory", - async () => { - await selectBundleNested(lixDb).execute(); - }, - { time: 1_500, warmupTime: 300 } -); - -bench( - "paraglide: select 500 bundles / 1,000 messages nested — SQLite WASM baseline", - async () => { - await selectBundleNested(sqliteDb).execute(); - }, - { time: 1_500, warmupTime: 300 } -); - -bench( - "sherlock: select one nested bundle — Lix 0.10 memory", - async () => { - await selectBundleNested(lixDb) - .where("bundle.id", "=", "bundle-250") - .executeTakeFirst(); - }, - { time: 1_500, warmupTime: 300 } -); - -bench( - "sherlock: select one nested bundle — SQLite WASM baseline", - async () => { - await selectBundleNested(sqliteDb) - .where("bundle.id", "=", "bundle-250") - .executeTakeFirst(); - }, - { time: 1_500, warmupTime: 300 } -); - -bench( - "sherlock: update one variant pattern — Lix 0.10 memory", - async () => { - mutationCounter++; - await lixDb - .updateTable("variant") - .set({ pattern: [{ type: "text", value: `edited-${mutationCounter}` }] }) - .where("id", "=", "variant-250-en") - .execute(); - }, - { time: 1_500, warmupTime: 300 } -); - -bench( - "sherlock: update one variant pattern — SQLite WASM baseline", - async () => { - mutationCounter++; - await sqliteDb - .updateTable("variant") - .set({ pattern: [{ type: "text", value: `edited-${mutationCounter}` }] }) - .where("id", "=", "variant-250-en") - .execute(); - }, - { time: 1_500, warmupTime: 300 } -); - -bench( - "sherlock: insert bundle with two translations — Lix 0.10 memory", - async () => { - mutationCounter++; - await insertBundleNested(lixDb, nestedBundle(`lix-new-${mutationCounter}`)); - }, - { time: 1_500, warmupTime: 300 } -); - -bench( - "sherlock: insert bundle with two translations — SQLite WASM baseline", - async () => { - mutationCounter++; - await insertBundleNested( - sqliteDb, - nestedBundle(`sqlite-new-${mutationCounter}`) - ); - }, - { time: 1_500, warmupTime: 300 } -); - -bench( - "paraglide/sherlock: load populated in-memory project — Lix 0.10 memory", - async () => { - const project = await loadProjectInMemory({ blob: populatedBlob }); - await project.close(); - }, - { time: 2_000, warmupTime: 300 } -); - -async function seedLix(target: Lix) { - const statements: LixBatchStatement[] = []; - for (let index = 0; index < BUNDLE_COUNT; index++) { - const bundleId = `bundle-${index}`; - statements.push({ - sql: "INSERT INTO bundle (id, declarations) VALUES ($1, $2)", - params: [bundleId, []], - }); - for (const locale of LOCALES) { - const messageId = `message-${index}-${locale}`; - statements.push({ - sql: 'INSERT INTO message (id, "bundleId", locale, selectors) VALUES ($1, $2, $3, $4)', - params: [messageId, bundleId, locale, []], - }); - statements.push({ - sql: 'INSERT INTO variant (id, "messageId", matches, pattern) VALUES ($1, $2, $3, $4)', - params: [ - `variant-${index}-${locale}`, - messageId, - [], - [{ type: "text", value: `Message ${index} (${locale})` }], - ], - }); - } - } - await target.executeBatch(statements); -} - -async function seedKysely(db: Kysely) { - for (let index = 0; index < BUNDLE_COUNT; index++) { - await insertBundleNested(db, nestedBundle(`bundle-${index}`, index)); - } -} - -function nestedBundle(id: string, index = mutationCounter) { - return { - id, - declarations: [], - messages: LOCALES.map((locale) => { - const messageId = id.startsWith("bundle-") - ? `message-${index}-${locale}` - : `${id}-message-${locale}`; - return { - id: messageId, - bundleId: id, - locale, - selectors: [], - variants: [ - { - id: id.startsWith("bundle-") - ? `variant-${index}-${locale}` - : `${id}-variant-${locale}`, - messageId, - matches: [], - pattern: [ - { type: "text" as const, value: `Message ${index} (${locale})` }, - ], - }, - ], - }; - }), - }; -} diff --git a/packages/sdk/package.json b/packages/sdk/package.json index 97ee0c88be..630bf5b188 100644 --- a/packages/sdk/package.json +++ b/packages/sdk/package.json @@ -31,7 +31,6 @@ "typecheck": "npm run env-variables && tsc --noEmit", "test": "npm run env-variables && tsc --noEmit && vitest run", "test:watch": "vitest", - "bench": "vitest bench --run benchmark", "lint": "eslint ./src", "format": "prettier ./src --write", "clean": "rm -rf ./dist ./node_modules" @@ -43,7 +42,6 @@ "@lix-js/sdk": "0.12.3", "@sinclair/typebox": "^0.31.17", "kysely": "^0.28.12", - "sqlite-wasm-kysely": "0.3.0", "uuid": "^14.0.0" }, "devDependencies": { diff --git a/packages/sdk/src/database/jsonbPlugin.test.ts b/packages/sdk/src/database/jsonbPlugin.test.ts deleted file mode 100644 index c6d1c5255a..0000000000 --- a/packages/sdk/src/database/jsonbPlugin.test.ts +++ /dev/null @@ -1,213 +0,0 @@ -import { Kysely } from "kysely"; -import { createDialect, createInMemoryDatabase } from "sqlite-wasm-kysely"; -import { test, expect } from "vitest"; -import { JsonbPlugin } from "./jsonbPlugin.js"; - -test("parsing and serializing of jsonb should work", async () => { - type MockSchema = { - foo: { - id: string; - data: Record; - }; - }; - const database = await createInMemoryDatabase({ - readOnly: false, - }); - - database.exec(` - CREATE TABLE foo ( - id TEXT PRIMARY KEY, - data BLOB NOT NULL - ) strict; - `); - - const db = new Kysely({ - dialect: createDialect({ - database, - }), - plugins: [new JsonbPlugin({ database })], - }); - - const foo = await db - .insertInto("foo") - .values({ - id: "mock", - data: { - data: "baz", - }, - }) - .returningAll() - .executeTakeFirstOrThrow(); - - expect(foo).toEqual({ - id: "mock", - data: { - data: "baz", - }, - }); -}); - -test("upserts should be handled", async () => { - type MockSchema = { - foo: { - id: string; - data: Record; - }; - }; - const database = await createInMemoryDatabase({ - readOnly: false, - }); - - database.exec(` - CREATE TABLE foo ( - id TEXT PRIMARY KEY, - data BLOB NOT NULL - ) strict; - `); - - const db = new Kysely({ - dialect: createDialect({ - database, - }), - plugins: [ - new JsonbPlugin({ - database, - }), - ], - }); - - const foo = await db - .insertInto("foo") - .values({ - id: "mock", - data: { - bar: "baz", - }, - }) - .returningAll() - .executeTakeFirstOrThrow(); - - expect(foo).toEqual({ - id: "mock", - data: { - bar: "baz", - }, - }); - - const updatedFoo = { - id: "mock", - data: { - bar: "baz", - baz: "qux", - }, - }; - - const updatedFooResult = await db - .insertInto("foo") - .values(updatedFoo) - .returningAll() - .onConflict((oc) => oc.column("id").doUpdateSet(updatedFoo)) - .executeTakeFirstOrThrow(); - - expect(updatedFooResult).toEqual(updatedFoo); -}); - -test("storing json as text is supposed to fail to avoid heuristics if the json should be stored as blob or text", async () => { - type MockSchema = { - foo: { - id: string; - data: Record; - }; - }; - const database = await createInMemoryDatabase({ - readOnly: false, - }); - - database.exec(` - CREATE TABLE foo ( - id TEXT PRIMARY KEY, - data TEXT NOT NULL - ) strict; - `); - - const db = new Kysely({ - dialect: createDialect({ - database, - }), - plugins: [new JsonbPlugin({ database })], - }); - - await expect(() => - db - .insertInto("foo") - .values({ - id: "mock", - data: { - bar: "baz", - }, - }) - .returningAll() - .executeTakeFirstOrThrow() - ).rejects.toThrowErrorMatchingInlineSnapshot( - `[SQLite3Error: SQLITE_CONSTRAINT_DATATYPE: sqlite3 result code 3091: cannot store BLOB value in TEXT column foo.data]` - ); -}); - -test("normalizes variants when messages is a JSON string", async () => { - type MockSchema = { - foo: { - id: string; - messages: string | any[]; - }; - }; - const database = await createInMemoryDatabase({ - readOnly: false, - }); - - database.exec(` - CREATE TABLE foo ( - id TEXT PRIMARY KEY, - messages TEXT NOT NULL - ) strict; - `); - - const db = new Kysely({ - dialect: createDialect({ - database, - }), - plugins: [new JsonbPlugin({ database })], - }); - - const rawMessages = JSON.stringify([ - { - selectors: "[]", - variants: JSON.stringify([ - { - matches: "[]", - pattern: '[{"type":"text","value":"x"}]', - }, - ]), - }, - ]); - - const foo = await db - .insertInto("foo") - .values({ - id: "mock", - messages: rawMessages, - }) - .returningAll() - .executeTakeFirstOrThrow(); - - expect(foo.messages).toEqual([ - { - selectors: [], - variants: [ - { - matches: [], - pattern: [{ type: "text", value: "x" }], - }, - ], - }, - ]); -}); diff --git a/packages/sdk/src/database/jsonbPlugin.ts b/packages/sdk/src/database/jsonbPlugin.ts deleted file mode 100644 index e4244a4a64..0000000000 --- a/packages/sdk/src/database/jsonbPlugin.ts +++ /dev/null @@ -1,280 +0,0 @@ -import { - OperationNodeTransformer, - sql, - ValueListNode, - ValueNode, - ValuesNode, - type KyselyPlugin, - type PluginTransformQueryArgs, - type PluginTransformResultArgs, - type QueryResult, - type RootOperationNode, - type UnknownRow, - OnConflictNode, -} from "kysely"; -import type { SqliteWasmDatabase } from "sqlite-wasm-kysely"; - -export class JsonbPlugin implements KyselyPlugin { - #serializeJsonTransformer = new SerializeJsonbTransformer(); - #database: SqliteWasmDatabase; - - constructor(args: { database: SqliteWasmDatabase }) { - this.#database = args.database; - } - - /** - * For an outgoing query like insert or update, the JSON - * values are transformed into `jsonb` function calls when - * executed against the database. - */ - transformQuery(args: PluginTransformQueryArgs): RootOperationNode { - if ( - args.node.kind === "InsertQueryNode" || - args.node.kind === "UpdateQueryNode" - ) { - const result = this.#serializeJsonTransformer.transformNode(args.node); - return result; - } - return args.node; - } - - /** - * For incoming query results, the JSON binaries are parsed - * into JSON objects. - */ - async transformResult( - args: PluginTransformResultArgs - ): Promise> { - for (const row of args.result.rows) { - for (const key in row) { - const value = row[key]; - // Always try to decode JSONB binary payloads. - if (value instanceof ArrayBuffer || ArrayBuffer.isView(value)) { - try { - const res = this.#database.exec(`SELECT json(?)`, { - returnValue: "resultRows", - bind: [value as any], - }); - - row[key] = JSON.parse(res[0] as any); - } catch { - // it's not a json binary - } - continue; - } - - // Only parse JSON text for known JSON columns to avoid - // coercing nested string values that look like JSON. - if (JSON_COLUMNS.has(key) && typeof value === "string") { - try { - row[key] = JSON.parse(value); - } catch { - // leave as-is if parsing fails - } - } - - if (key === "messages" && Array.isArray(row[key])) { - row[key] = normalizeMessages(row[key]); - } else if (key === "variants" && Array.isArray(row[key])) { - row[key] = normalizeVariants(row[key]); - } - } - } - return args.result; - } -} - -const JSON_COLUMNS = new Set([ - "declarations", - "selectors", - "matches", - "pattern", - // jsonArrayFrom results in nested bundle queries - "messages", - "variants", -]); - -function parseJsonIfString(value: T): T | unknown { - if (typeof value !== "string") { - return value; - } - try { - return JSON.parse(value); - } catch { - return value; - } -} - -function normalizeVariants(variants: any[]) { - return variants.map((variant) => { - if (variant && typeof variant === "object") { - return { - ...variant, - matches: parseJsonIfString(variant.matches), - pattern: parseJsonIfString(variant.pattern), - }; - } - return variant; - }); -} - -function normalizeMessages(messages: any[]) { - return messages.map((message) => { - if (message && typeof message === "object") { - const rawVariants = Array.isArray(message.variants) - ? message.variants - : parseJsonIfString(message.variants); - const normalizedVariants = Array.isArray(rawVariants) - ? normalizeVariants(rawVariants) - : rawVariants; - return { - ...message, - selectors: parseJsonIfString(message.selectors), - variants: normalizedVariants, - }; - } - return message; - }); -} - -class SerializeJsonbTransformer extends OperationNodeTransformer { - protected override transformOnConflict(node: OnConflictNode): OnConflictNode { - return super.transformOnConflict({ - ...node, - updates: node.updates?.map((updateItem) => { - if (updateItem.kind !== "ColumnUpdateNode") { - return updateItem; - } - return { - kind: "ColumnUpdateNode", - column: updateItem.column, - // @ts-expect-error - we know that the value is a ValueNode - value: this.transformValue(updateItem.value), - }; - }), - }); - } - - protected override transformValue(node: ValueNode): ValueNode { - const { value } = node; - const serializedValue = maybeSerializeJson(value); - if (value === serializedValue) { - return node; - } - // @ts-expect-error - we know that the node is a ValueNode - return sql`jsonb(${serializedValue})`.toOperationNode(); - } - /** - * Transforms the value list node by replacing all JSON objects with `jsonb` function calls. - */ - protected override transformValueList(node: ValueListNode): ValueListNode { - return super.transformValueList({ - ...node, - values: node.values.map((listNodeItem) => { - if (listNodeItem.kind !== "ValueNode") { - return listNodeItem; - } - // @ts-expect-error - we know that the node is a ValueNode - const { value } = listNodeItem; - const serializedValue = maybeSerializeJson(value); - - if (value === serializedValue) { - return listNodeItem; - } - return sql`jsonb(${serializedValue})`.toOperationNode(); - }), - }); - } - - /** - * Why this function is needed or why this works remains a mystery. - */ - override transformValues(node: ValuesNode): ValuesNode { - return super.transformValues({ - ...node, - values: node.values.map((valueItemNode) => { - if (valueItemNode.kind !== "PrimitiveValueListNode") { - return valueItemNode; - } - - // change valueItem to ValueListNode - return { - kind: "ValueListNode", - values: valueItemNode.values.map( - (value) => - ({ - kind: "ValueNode", - value, - }) as ValueNode - ), - } as ValueListNode; - }), - }); - } -} - -function maybeSerializeJson(value: any): any { - if ( - // binary data - value instanceof ArrayBuffer || - // uint8array, etc - ArrayBuffer.isView(value) || - value === null || - value === undefined - ) { - return value; - } else if (typeof value === "object" || Array.isArray(value)) { - return JSON.stringify(value); - } - return value; -} - -// The code here didn't work https://github.com/opral/inlang/issues/132#issuecomment-2339321910 -// but would be the "right" solution to avoid heuristics which column might or might not be a json column -// // modifies the query in place for readability and performance -// function mapQuery( -// node: InsertQueryNode, -// jsonColumns: TableSchema -// ): InsertQueryNode { -// // if the query is not an insert query, we don't need to do anything -// if (node.into === undefined) { -// return node; -// } -// // if the table is not in the schema that has json columns, we don't need to do anything -// const columnsWithJson = jsonColumns[node.into.table.identifier.name]; -// if (columnsWithJson === undefined) { -// return node; -// } -// // find the indexes of the values that need to be transformed -// // SQL query: INSERT INTO table (col1, col2) VALUES (val1, val2) -// const indexesThatNeedToBeTransformed: [number, string][] = []; -// for (const [i, col] of node.columns?.entries() ?? []) { -// const jsonType = columnsWithJson[col.column.name]; -// if (jsonType !== undefined) { -// indexesThatNeedToBeTransformed.push([i, jsonType]); -// } -// } -// const values = structuredClone(node.values); -// for (const [i, jsonType] of indexesThatNeedToBeTransformed) { -// if ( -// // top level values node that should contain a list of values -// node.values?.kind !== "ValuesNode" && -// // the node we are interested in must be a value node -// // @ts-expect-error - we know that the node is a ValuesNode with values -// (node.values as ValuesNode).values?.[i].kind !== "ValueNode" -// ) { -// throw new Error("Unexpected node structure"); -// } -// const serializedJson = JSON.stringify(node.values.values[0].values[i]); -// // @ts-expect-error - we know that the node is a ValuesNode with values -// values.values[0].values[i] = -// jsonType === "jsonb" -// ? sql`jsonb(${serializedJson})`.toOperationNode() -// : sql`json(${serializedJson})`.toOperationNode(); -// } - -// return { -// ...node, -// values, -// }; -// } diff --git a/packages/sdk/src/database/lixDialect.test.ts b/packages/sdk/src/database/lixDialect.test.ts index 9a0c7d02c8..34cbc896a5 100644 --- a/packages/sdk/src/database/lixDialect.test.ts +++ b/packages/sdk/src/database/lixDialect.test.ts @@ -4,6 +4,25 @@ import { initDb } from "./initDb.js"; import { registerInlangSchemas } from "./registerSchemas.js"; import { selectBundleNested } from "../query-utilities/selectBundleNested.js"; +test("compiles Kysely parameters with PostgreSQL placeholders", async () => { + const lix = await openLix(); + const db = initDb({ lix }); + + const compiled = db + .selectFrom("bundle") + .select("id") + .where("id", "=", "example") + .compile(); + + expect(compiled.sql).toBe( + 'select "id" from "bundle" where "id" = $1' + ); + expect(compiled.parameters).toEqual(["example"]); + + await db.destroy(); + await lix.close(); +}); + test("executes Kysely reads, writes, defaults, and transactions on Lix", async () => { const lix = await openLix(); await registerInlangSchemas(lix); diff --git a/packages/sdk/src/database/schema.ts b/packages/sdk/src/database/schema.ts index 43853adf36..3b29c53bfa 100644 --- a/packages/sdk/src/database/schema.ts +++ b/packages/sdk/src/database/schema.ts @@ -1,54 +1,10 @@ import type { Generated, Insertable, Selectable, Updateable } from "kysely"; -import type { SqliteWasmDatabase } from "sqlite-wasm-kysely"; import { Declaration, Pattern, VariableReference, } from "../json-schema/pattern.js"; -export function applySchema(args: { sqlite: SqliteWasmDatabase }) { - const foreignKeyActivated: any = args.sqlite.exec("PRAGMA foreign_keys", { - returnValue: "resultRows", - }); - if ( - // first row that is returned - // first column of the first row - // is equal to 0, then foreign keys are disabled - foreignKeyActivated[0][0] === 0 - ) { - args.sqlite.exec("PRAGMA foreign_keys = ON", { - returnValue: "resultRows", - }); - } - - args.sqlite.exec(` -CREATE TABLE IF NOT EXISTS bundle ( - id TEXT PRIMARY KEY DEFAULT (human_id()), - declarations BLOB NOT NULL DEFAULT (jsonb('[]')) -) strict; - -CREATE TABLE IF NOT EXISTS message ( - id TEXT PRIMARY KEY DEFAULT (uuid_v7()), - bundle_id TEXT NOT NULL, - locale TEXT NOT NULL, - selectors BLOB NOT NULL DEFAULT (jsonb('[]')), - FOREIGN KEY (bundle_id) REFERENCES bundle(id) ON DELETE CASCADE -) strict; - - -CREATE TABLE IF NOT EXISTS variant ( - id TEXT PRIMARY KEY DEFAULT (uuid_v7()), - message_id TEXT NOT NULL, - matches BLOB NOT NULL DEFAULT (jsonb('[]')), - pattern BLOB NOT NULL DEFAULT (jsonb('[]')), - FOREIGN KEY (message_id) REFERENCES message(id) ON DELETE CASCADE -) strict; - -CREATE INDEX IF NOT EXISTS idx_message_bundle_id ON message (bundle_id); -CREATE INDEX IF NOT EXISTS idx_variant_message_id ON variant (message_id); - `); -} - export type InlangDatabaseSchema = { bundle: BundleTable; message: MessageTable; diff --git a/packages/sdk/src/import-export/importFiles.ts b/packages/sdk/src/import-export/importFiles.ts index c165a836ac..b2908f98ad 100644 --- a/packages/sdk/src/import-export/importFiles.ts +++ b/packages/sdk/src/import-export/importFiles.ts @@ -21,10 +21,9 @@ const variantReferenceKey = ( const INSERT_BATCH_SIZE = 500; /** - * Kysely uses one column list for every row in a multi-row insert. SQLite's - * default-value handling for omitted NOT NULL JSON columns is not reliable - * when rows with different optional-column shapes are mixed, so keep each - * shape in its own insert statement. + * Kysely uses one column list for every row in a multi-row insert. Keep rows + * with different optional-column shapes in separate statements so omitted + * columns receive their schema defaults. */ async function insertInBatchesByShape(args: { rows: readonly T[]; diff --git a/packages/sdk/src/lix-plugin/applyChanges.test.ts b/packages/sdk/src/lix-plugin/applyChanges.test.ts deleted file mode 100644 index efff7b32fb..0000000000 --- a/packages/sdk/src/lix-plugin/applyChanges.test.ts +++ /dev/null @@ -1,150 +0,0 @@ -// eslint-disable-next-line @typescript-eslint/ban-ts-comment -// @ts-nocheck - -import { test, expect } from "vitest"; -import { loadProjectInMemory } from "../project/loadProjectInMemory.js"; -import { newProject } from "../project/newProject.js"; -import type { Change, NewChange } from "@lix-js/sdk"; -import { applyChanges } from "./applyChanges.js"; -import { loadDatabaseInMemory } from "sqlite-wasm-kysely"; -import { initDb } from "../database/initDb.js"; -import type { Bundle } from "../database/schema.js"; - -test.skip("it should be able to delete", async () => { - const project = await loadProjectInMemory({ - blob: await newProject(), - }); - - const changes: NewChange[] = [ - { - id: "1", - operation: "create", - file_id: "mock", - plugin_key: "mock", - type: "bundle", - meta: { id: "mock" }, - value: { - id: "mock", - declarations: [], - } satisfies Bundle, - }, - { - id: "2", - parent_id: "1", - operation: "update", - file_id: "mock", - plugin_key: "mock", - type: "bundle", - meta: { - id: "mock", - }, - value: { - id: "mock", - declarations: [], - } satisfies Bundle, - }, - { - id: "3", - parent_id: "2", - operation: "delete", - file_id: "mock", - plugin_key: "mock", - meta: { - id: "mock", - }, - type: "bundle", - value: undefined, - }, - ]; - - await project.db - .insertInto("bundle") - .values({ - id: "mock", - }) - .execute(); - - const dbFile = await project.lix.db - .selectFrom("file") - .selectAll() - .where("path", "=", "/db.sqlite") - .executeTakeFirstOrThrow(); - - const dbFileAfter = await applyChanges({ - lix: project.lix, - file: dbFile, - changes: changes as Change[], - }); - - const db = initDb({ - sqlite: await loadDatabaseInMemory(dbFileAfter.fileData), - }); - - const bundles = await db.selectFrom("bundle").selectAll().execute(); - - expect(bundles).toHaveLength(0); -}); - -test.skip("it should be able to upsert (insert & update)", async () => { - const project = await loadProjectInMemory({ - blob: await newProject(), - }); - - const changes: NewChange[] = [ - { - id: "1", - parent_id: undefined, - operation: "create", - file_id: "mock", - plugin_key: "mock", - type: "bundle", - meta: { id: "mock" }, - value: { - id: "mock", - declarations: [], - } satisfies Bundle, - }, - { - id: "2", - parent_id: "1", - operation: "update", - file_id: "mock", - plugin_key: "mock", - type: "bundle", - meta: { - id: "mock", - }, - value: { - id: "mock", - declarations: [], - } satisfies Bundle, - }, - ]; - - await project.db - .insertInto("bundle") - .values({ - id: "mock", - }) - .execute(); - - const dbFile = await project.lix.db - .selectFrom("file") - .selectAll() - .where("path", "=", "/db.sqlite") - .executeTakeFirstOrThrow(); - - const dbFileAfter = await applyChanges({ - lix: project.lix, - file: dbFile, - changes: changes as Change[], - }); - - const db = initDb({ - sqlite: await loadDatabaseInMemory(dbFileAfter.fileData), - }); - - const bundles = await db.selectFrom("bundle").selectAll().execute(); - - expect(bundles).toHaveLength(1); -}); diff --git a/packages/sdk/src/lix-plugin/applyChanges.ts b/packages/sdk/src/lix-plugin/applyChanges.ts deleted file mode 100644 index 774d42d544..0000000000 --- a/packages/sdk/src/lix-plugin/applyChanges.ts +++ /dev/null @@ -1,171 +0,0 @@ -// eslint-disable-next-line @typescript-eslint/ban-ts-comment -// @ts-nocheck - -import { - getLeafChange, - type Change, - type LixPlugin, - type LixReadonly, -} from "@lix-js/sdk"; -import { contentFromDatabase, loadDatabaseInMemory } from "sqlite-wasm-kysely"; -import { initDb } from "../database/initDb.js"; -import type { Kysely } from "kysely"; -import type { InlangDatabaseSchema } from "../database/schema.js"; - -export const applyChanges: NonNullable = async ({ - lix, - file, - changes, -}) => { - if (file.path?.endsWith("db.sqlite") === false) { - throw new Error( - "Unimplemented. Only the db.sqlite file can be handled for now." - ); - } - - // todo make transactional - - const sqlite = await loadDatabaseInMemory(file.data); - const db = initDb({ sqlite }); - - // the award for the most inefficient deduplication goes to... - const leafChanges = [ - ...new Set( - await Promise.all( - changes.map(async (change) => { - const leafChange = await getLeafChange({ change, lix }); - // enable string comparison to avoid duplicates - return JSON.stringify(leafChange); - }) - ) - ), - ].map((v) => JSON.parse(v)); - - // changes need to be applied in order of foreign keys to avoid constraint violations - // 1. bundles - // 2. messages - // 3. variants - const applyOrder: Record = { - bundle: 1, - message: 2, - variant: 3, - }; - - // future optimization potential here but sorting in one go - const orderedLeafChanges = [...leafChanges].sort((a, b) => { - const orderA = applyOrder[a.type]; - const orderB = applyOrder[b.type]; - - if (orderA === undefined || orderB === undefined) { - throw new Error( - `Received an unknown entity type: ${a.type} && ${ - b.type - }. Expected one of: ${Object.keys(applyOrder)}` - ); - } - - return orderA - orderB; - }); - for (const leafChange of orderedLeafChanges) { - // deletion - if (leafChange.value === undefined) { - await db - .deleteFrom(leafChange.type as "bundle" | "message" | "variant") - .where("id", "=", leafChange.meta?.id) - .execute(); - continue; - } - - // upsert the value - const value = leafChange.value as any; - - try { - await db - .insertInto(leafChange.type as "bundle" | "message" | "variant") - .values(value) - .onConflict((c) => c.column("id").doUpdateSet(value)) - .execute(); - } catch (e) { - // 787 = SQLITE_CONSTRAINT_FOREIGNKEY - if (e instanceof Error && (e as any)?.resultCode === 787) { - await handleForeignKeyViolation({ change: leafChange, lix, db }); - } else { - throw e; - } - } - } - return { fileData: contentFromDatabase(sqlite) }; -}; - -/** - * Handles foreign key violations e.g. a change - * doesn't exist in the target database but is referenced - * by an entity. - */ -async function handleForeignKeyViolation(args: { - change: Change; - lix: LixReadonly; - db: Kysely; -}) { - const lastKnown = async ( - type: "bundle" | "message" | "variant", - id: string - ) => - await args.lix.db - .selectFrom("change") - .selectAll() - // heuristic that getting the last bundle value is fine - // and using created_at is fine too. if the change is undesired - // , a user can revert it with version control via lix - .orderBy("created_at", "desc") - .where("type", "=", type) - .where((eb) => eb.ref("value", "->>").key("id"), "=", id) - .where("operation", "in", ["create", "update"]) - // TODO shouldn't throw. The API needs to be able to - // report issues back to the app without throwing and potentially failing - // to apply 1000 changes because 1 change is invalid - // same requirement as in inlang, see https://github.com/opral/inlang/issues/213 - .executeTakeFirstOrThrow(); - - if (args.change.type === "message") { - const lastKnownBundle = await lastKnown( - "bundle", - args.change.value?.bundleId - ); - await args.db - .insertInto("bundle") - .values(lastKnownBundle.value as any) - .execute(); - } else if (args.change.type === "variant") { - const lastKnownMessage = await lastKnown( - "message", - args.change.value?.messageId - ); - // getting the bundle too out of precaution - const lastKnownBundle = await lastKnown( - "bundle", - lastKnownMessage.value?.bundleId - ); - await args.db - .insertInto("bundle") - .values(lastKnownBundle.value as any) - // the bundle exists, so we can ignore the conflict - .onConflict((c) => c.doNothing()) - .execute(); - await args.db - .insertInto("message") - .values(lastKnownMessage.value as any) - .execute(); - await args.db - .insertInto(args.change.type as "bundle" | "message" | "variant") - .values(args.change.value as any) - .onConflict((c) => c.column("id").doUpdateSet(args.change.value as any)) - .execute(); - } - // re-execute applying the change - await args.db - .insertInto(args.change.type as "bundle" | "message" | "variant") - .values(args.change.value as any) - .onConflict((c) => c.column("id").doUpdateSet(args.change.value as any)) - .execute(); -} diff --git a/packages/sdk/src/lix-plugin/detectConflicts.test.ts b/packages/sdk/src/lix-plugin/detectConflicts.test.ts deleted file mode 100644 index d5824fa326..0000000000 --- a/packages/sdk/src/lix-plugin/detectConflicts.test.ts +++ /dev/null @@ -1,286 +0,0 @@ -// eslint-disable-next-line @typescript-eslint/ban-ts-comment -// @ts-nocheck - -import { test, expect } from "vitest"; -import { inlangLixPluginV1 } from "./inlangLixPluginV1.js"; -import { - newLixFile, - openLixInMemory, - type Change, - type NewChange, -} from "@lix-js/sdk"; - -test.skip("a create operation should not report a conflict given that the change does not exist in target", async () => { - const targetLix = await openLixInMemory({ blob: await newLixFile() }); - const sourceLix = await openLixInMemory({ blob: await newLixFile() }); - const changes = await sourceLix.db - .insertInto("change") - .values([ - { - id: "1", - parent_id: undefined, - operation: "create", - file_id: "mock", - plugin_key: "mock", - type: "mock", - value: { id: "change 1" }, - }, - ]) - .returningAll() - .execute(); - const conflicts = await inlangLixPluginV1.detectConflicts!({ - sourceLix, - targetLix, - leafChangesOnlyInSource: changes, - }); - expect(conflicts).toHaveLength(0); -}); - -test.todo( - "it should report deletions as a conflict if the parent of the target and source are not identical", - async () => { - const targetLix = await openLixInMemory({ blob: await newLixFile() }); - await targetLix.db - .insertInto("change") - .values([ - { - id: "1", - parent_id: undefined, - operation: "create", - file_id: "mock", - plugin_key: "mock", - type: "mock", - value: { - id: "change 1", - }, - }, - ]) - .execute(); - - const sourceLix = await openLixInMemory({ blob: await targetLix.toBlob() }); - - const changesNotInTarget: NewChange[] = [ - { - id: "2", - parent_id: "1", - operation: "delete", - file_id: "mock", - plugin_key: "mock", - type: "mock", - value: undefined, - }, - ]; - - await sourceLix.db - .insertInto("change") - .values(changesNotInTarget) - .execute(); - - const conflicts = await inlangLixPluginV1.detectConflicts!({ - sourceLix, - targetLix, - leafChangesOnlyInSource: changesNotInTarget as Change[], - }); - expect(conflicts).toHaveLength(1); - expect(conflicts[0]?.change_id).toBe("1"); - expect(conflicts[0]?.conflicting_change_id).toBe("2"); - throw new Error("The parent is identicak, fix this test"); - } -); - -test.skip("it should report an UPDATE as a conflict if leaf changes are conflicting", async () => { - const targetLix = await openLixInMemory({ blob: await newLixFile() }); - const sourceLix = await openLixInMemory({ blob: await targetLix.toBlob() }); - - const commonChanges: NewChange[] = [ - { - id: "12s", - parent_id: undefined, - operation: "create", - file_id: "mock", - plugin_key: "mock", - type: "mock", - value: { - id: "change 12s", - }, - }, - ]; - - const changesOnlyInTarget: NewChange[] = [ - { - id: "3sd", - parent_id: "12s", - operation: "update", - file_id: "mock", - plugin_key: "mock", - type: "mock", - value: { - id: "change 3sd", - }, - }, - ]; - - const changesOnlyInSource: NewChange[] = [ - { - id: "2qa", - parent_id: "12s", - operation: "update", - file_id: "mock", - plugin_key: "mock", - type: "mock", - value: { - id: "change 2qa", - }, - }, - ]; - - await sourceLix.db - .insertInto("change") - .values([...commonChanges, ...changesOnlyInSource]) - .execute(); - - await targetLix.db - .insertInto("change") - .values([...commonChanges, ...changesOnlyInTarget]) - .execute(); - - const conflicts = await inlangLixPluginV1.detectConflicts!({ - leafChangesOnlyInSource: changesOnlyInSource as Change[], - sourceLix: sourceLix, - targetLix: targetLix, - }); - - expect(conflicts).toHaveLength(1); -}); - -/** - * If the common ancestor is the leaf change of the target, then the - * source change are (likely) not a conflict because no update has - * been made to the target change that could conflict with updates - * in the source. - */ -test.skip("it should NOT report an UPDATE as a conflict if the common ancestor is the leaf change of the target", async () => { - const targetLix = await openLixInMemory({ blob: await newLixFile() }); - const sourceLix = await openLixInMemory({ blob: await targetLix.toBlob() }); - - const commonChanges: NewChange[] = [ - { - id: "12s", - parent_id: undefined, - operation: "create", - file_id: "mock", - plugin_key: "mock", - type: "mock", - value: { - id: "change 12s", - }, - }, - ]; - - const changesOnlyInTarget: NewChange[] = [ - { - id: "3sd", - parent_id: "12s", - operation: "update", - file_id: "mock", - plugin_key: "mock", - type: "mock", - value: { - id: "change 3sd", - }, - }, - { - id: "23a", - parent_id: "3sd", - operation: "update", - file_id: "mock", - plugin_key: "mock", - type: "mock", - value: { - id: "change 23a", - }, - }, - ]; - - const changesOnlyInSource: NewChange[] = []; - - await sourceLix.db - .insertInto("change") - .values([...commonChanges, ...changesOnlyInSource]) - .execute(); - - await targetLix.db - .insertInto("change") - .values([...commonChanges, ...changesOnlyInTarget]) - .execute(); - - const conflicts = await inlangLixPluginV1.detectConflicts!({ - leafChangesOnlyInSource: changesOnlyInSource as Change[], - sourceLix: sourceLix, - targetLix: targetLix, - }); - - expect(conflicts).toHaveLength(0); -}); - -test.skip("it should NOT report a DELETE as a conflict if the parent of the target and source are identical", async () => { - const targetLix = await openLixInMemory({ blob: await newLixFile() }); - await targetLix.db - .insertInto("change") - .values([ - { - id: "12s", - parent_id: undefined, - operation: "create", - file_id: "mock", - plugin_key: "mock", - type: "mock", - value: { - id: "change 12s", - }, - }, - ]) - .execute(); - - const sourceLix = await openLixInMemory({ blob: await targetLix.toBlob() }); - - const changesNotInTarget: NewChange[] = [ - { - id: "3sd", - parent_id: "12s", - operation: "delete", - file_id: "mock", - plugin_key: "mock", - type: "mock", - value: undefined, - }, - ]; - - const changesNotInSource: NewChange[] = [ - { - id: "2qa", - parent_id: "12s", - operation: "update", - file_id: "mock", - plugin_key: "mock", - type: "mock", - value: { - id: "2qa", - }, - }, - ]; - - await sourceLix.db.insertInto("change").values(changesNotInTarget).execute(); - - await targetLix.db.insertInto("change").values(changesNotInSource).execute(); - - const conflicts = await inlangLixPluginV1.detectConflicts!({ - sourceLix, - targetLix, - leafChangesOnlyInSource: changesNotInTarget as Change[], - }); - - expect(conflicts).toHaveLength(1); - expect(conflicts[0]?.change_id).toBe("2qa"); - expect(conflicts[0]?.conflicting_change_id).toBe("3sd"); -}); diff --git a/packages/sdk/src/lix-plugin/detectConflicts.ts b/packages/sdk/src/lix-plugin/detectConflicts.ts deleted file mode 100644 index 2999160335..0000000000 --- a/packages/sdk/src/lix-plugin/detectConflicts.ts +++ /dev/null @@ -1,62 +0,0 @@ -// eslint-disable-next-line @typescript-eslint/ban-ts-comment -// @ts-nocheck - -import { - getLowestCommonAncestor, - getLeafChange, - type LixPlugin, - type NewConflict, -} from "@lix-js/sdk"; - -export const detectConflicts: LixPlugin["detectConflicts"] = async ({ - sourceLix, - targetLix, - leafChangesOnlyInSource, -}) => { - const result: NewConflict[] = []; - for (const change of leafChangesOnlyInSource) { - const lowestCommonAncestor = await getLowestCommonAncestor({ - sourceChange: change, - sourceLix, - targetLix, - }); - - if (lowestCommonAncestor === undefined) { - // no common parent, no conflict. must be an insert - continue; - } - - const leafChangeInTarget = await getLeafChange({ - change: lowestCommonAncestor, - lix: targetLix, - }); - - if (lowestCommonAncestor.id === leafChangeInTarget.id) { - // no conflict. the lowest common ancestor is - // the leaf change in the target. aka, no changes - // in target have been made that could conflict with the source - continue; - } - - const hasDiff = - JSON.stringify(change.value) !== JSON.stringify(leafChangeInTarget.value); - - if (hasDiff === false) { - // TODO we have two different changes that yielded the same snapshot, - // lix or the plugin need to change the parents of the target change - // to both the source and the target change. users likely want to - // see that two "branches" led to the same snapshot - continue; - } - - // naive raise any snapshot difference as a conflict for now - // more sophisticated conflict reporting can be incrementally added - result.push({ - change_id: leafChangeInTarget.id, - conflicting_change_id: change.id, - reason: - "The snapshots of the change do not match. More sophisticated reasoning will be added later.", - }); - } - return result; -}; diff --git a/packages/sdk/src/lix-plugin/inlangLixPluginV1.test.ts b/packages/sdk/src/lix-plugin/inlangLixPluginV1.test.ts deleted file mode 100644 index 50c7590987..0000000000 --- a/packages/sdk/src/lix-plugin/inlangLixPluginV1.test.ts +++ /dev/null @@ -1,439 +0,0 @@ -// eslint-disable-next-line @typescript-eslint/ban-ts-comment -// @ts-nocheck - -import { test, expect, describe } from "vitest"; -import { inlangLixPluginV1 } from "./inlangLixPluginV1.js"; -import { type DiffReport } from "@lix-js/sdk"; -import { newProject } from "../project/newProject.js"; -import { loadProjectInMemory } from "../project/loadProjectInMemory.js"; -import { contentFromDatabase } from "sqlite-wasm-kysely"; -import type { Variant } from "../database/schema.js"; - -describe.skip("plugin.diff.file", () => { - test("insert of bundle", async () => { - const neuProject = await loadProjectInMemory({ blob: await newProject() }); - await neuProject.db - .insertInto("bundle") - .values({ - id: "1", - }) - .execute(); - const path = "/db.sqlite"; - const diffReports = await inlangLixPluginV1.diff.file!({ - old: undefined, - neu: { - id: "uuid", - path, - data: contentFromDatabase(neuProject._sqlite), - metadata: {}, - }, - }); - expect(diffReports).toEqual([ - { - type: "bundle", - operation: "create", - old: undefined, - neu: expect.objectContaining({ id: "1" }), - } satisfies DiffReport, - ]); - }); - - // reanble with declarations - test.todo("update of bundle", async () => { - const oldProject = await loadProjectInMemory({ blob: await newProject() }); - await oldProject.db - .insertInto("bundle") - .values([ - { - id: "1", - }, - { - id: "2", - }, - ]) - .execute(); - const neuProject = await loadProjectInMemory({ blob: await newProject() }); - await neuProject.db - .insertInto("bundle") - .values([ - { - id: "1", - }, - { - id: "2", - }, - ]) - .execute(); - - const diffReports = await inlangLixPluginV1.diff.file!({ - old: { - id: "uuid", - path: "/db.sqlite", - data: contentFromDatabase(oldProject._sqlite), - metadata: {}, - }, - neu: { - id: "uuid", - path: "/db.sqlite", - data: contentFromDatabase(neuProject._sqlite), - metadata: {}, - }, - }); - - expect(diffReports).toEqual([ - { - meta: { - id: "1", - }, - type: "bundle", - operation: "update", - old: { id: "1" }, - neu: { id: "1" }, - } satisfies DiffReport, - ]); - }); - - test("insert of message", async () => { - const neuProject = await loadProjectInMemory({ blob: await newProject() }); - await neuProject.db - .insertInto("bundle") - .values({ id: "unknown" }) - .execute(); - await neuProject.db - .insertInto("message") - .values({ - id: "1", - bundleId: "unknown", - locale: "en", - }) - .execute(); - const diffReports = await inlangLixPluginV1.diff.file!({ - old: undefined, - neu: { - id: "uuid", - path: "/db.sqlite", - data: contentFromDatabase(neuProject._sqlite), - metadata: {}, - }, - }); - expect(diffReports).toEqual( - expect.arrayContaining([ - { - type: "message", - operation: "create", - old: undefined, - neu: { - id: "1", - bundleId: "unknown", - selectors: [], - locale: "en", - }, - } satisfies DiffReport, - ]) - ); - }); - - test("update of message", async () => { - const oldProject = await loadProjectInMemory({ blob: await newProject() }); - await oldProject.db - .insertInto("bundle") - .values({ id: "unknown" }) - .execute(); - await oldProject.db - .insertInto("message") - .values([ - { - id: "1", - bundleId: "unknown", - locale: "en", - }, - { - id: "2", - bundleId: "unknown", - locale: "en", - }, - ]) - .execute(); - const neuProject = await loadProjectInMemory({ blob: await newProject() }); - await neuProject.db - .insertInto("bundle") - .values({ id: "unknown" }) - .execute(); - await neuProject.db - .insertInto("message") - .values([ - { - id: "1", - bundleId: "unknown", - locale: "de", - }, - { - id: "2", - bundleId: "unknown", - locale: "en", - }, - ]) - .execute(); - const diffReports = await inlangLixPluginV1.diff.file!({ - old: { - id: "uuid", - path: "/db.sqlite", - data: contentFromDatabase(oldProject._sqlite), - metadata: {}, - }, - neu: { - id: "uuid", - path: "/db.sqlite", - data: contentFromDatabase(neuProject._sqlite), - metadata: {}, - }, - }); - expect(diffReports).toEqual( - expect.arrayContaining([ - { - meta: { - id: "1", - }, - type: "message", - operation: "update", - old: { - id: "1", - bundleId: "unknown", - selectors: [], - locale: "en", - }, - neu: { - id: "1", - bundleId: "unknown", - selectors: [], - locale: "de", - }, - } satisfies DiffReport, - ]) - ); - }); - test("insert of variant", async () => { - const neuProject = await loadProjectInMemory({ blob: await newProject() }); - await neuProject.db - .insertInto("bundle") - .values({ id: "bundle1" }) - .execute(); - await neuProject.db - .insertInto("message") - .values({ id: "1", bundleId: "bundle1", locale: "en" }) - .execute(); - - await neuProject.db - .insertInto("variant") - .values({ - id: "1", - messageId: "1", - pattern: [{ type: "text", value: "hello world" }], - matches: [], - }) - .execute(); - const diffReports = await inlangLixPluginV1.diff.file!({ - old: undefined, - neu: { - id: "uuid", - path: "/db.sqlite", - data: contentFromDatabase(neuProject._sqlite), - metadata: {}, - }, - }); - expect(diffReports).toEqual( - expect.arrayContaining([ - { - type: "variant", - operation: "create", - old: undefined, - neu: { - id: "1", - messageId: "1", - pattern: [{ type: "text", value: "hello world" }], - matches: [], - }, - } satisfies DiffReport, - ]) - ); - }); - test("update of variant", async () => { - const oldProject = await loadProjectInMemory({ blob: await newProject() }); - await oldProject.db - .insertInto("bundle") - .values({ id: "bundle1" }) - .execute(); - await oldProject.db - .insertInto("message") - .values({ id: "1", bundleId: "bundle1", locale: "en" }) - .execute(); - await oldProject.db - .insertInto("variant") - .values([ - { - id: "1", - messageId: "1", - pattern: [{ type: "text", value: "hello world" }], - matches: [], - }, - { - id: "2", - messageId: "1", - pattern: [{ type: "text", value: "hello world" }], - matches: [], - }, - ]) - .execute(); - const neuProject = await loadProjectInMemory({ blob: await newProject() }); - await neuProject.db - .insertInto("bundle") - .values({ id: "bundle1" }) - .execute(); - await neuProject.db - .insertInto("message") - .values({ id: "1", bundleId: "bundle1", locale: "en" }) - .execute(); - await neuProject.db - .insertInto("variant") - .values([ - { - id: "1", - messageId: "1", - pattern: [{ type: "text", value: "hello world from Berlin" }], - matches: [], - }, - { - id: "2", - messageId: "1", - pattern: [{ type: "text", value: "hello world" }], - matches: [], - }, - ]) - .execute(); - const diffReports = await inlangLixPluginV1.diff.file!({ - old: { - id: "uuid", - path: "/db.sqlite", - data: contentFromDatabase(oldProject._sqlite), - metadata: {}, - }, - neu: { - id: "uuid", - path: "/db.sqlite", - data: contentFromDatabase(neuProject._sqlite), - metadata: {}, - }, - }); - expect(diffReports).toEqual([ - { - meta: { - id: "1", - }, - operation: "update", - type: "variant", - old: { - id: "1", - messageId: "1", - pattern: [{ type: "text", value: "hello world" }], - matches: [], - }, - neu: { - id: "1", - messageId: "1", - pattern: [{ type: "text", value: "hello world from Berlin" }], - matches: [], - }, - } satisfies DiffReport, - ]); - }); - - // https://github.com/opral/lix/issues/33 - test("it should generate changes after the first change", async () => { - const project = await loadProjectInMemory({ blob: await newProject() }); - - const initialChanges = await project.lix.db - .selectFrom("change") - .selectAll() - .execute(); - expect(initialChanges.length).toEqual(0); - - await project.db - .insertInto("bundle") - .values({ - id: "1", - }) - .execute(); - - // FIXME: how to await inlang sdk persisting the inlang db to lix? - await new Promise((resolve) => setTimeout(resolve, 500)); - - await project.lix.settled(); - - const changes = await project.lix.db - .selectFrom("change") - .selectAll() - .execute(); - - expect(changes.length).toBe(1); - expect(changes[0]?.value?.id).toBe("1"); - expect(changes[0]?.operation).toBe("create"); - }); -}); - -describe.skip("plugin.diff.variant", () => { - test("old and neu are the same should not report a diff", async () => { - const old: Variant = { - id: "1", - matches: [], - messageId: "5", - pattern: [{ type: "text", value: "hello world" }], - }; - const neu: Variant = { - id: "1", - matches: [], - messageId: "5", - pattern: [{ type: "text", value: "hello world" }], - }; - const diff = await inlangLixPluginV1.diff.variant({ old, neu }); - expect(diff).toEqual([]); - }); - - test("old and neu are different should yield a diff report", async () => { - const old: Variant = { - id: "1", - matches: [], - messageId: "5", - pattern: [{ type: "text", value: "hello world" }], - }; - const neu: Variant = { - id: "1", - matches: [], - messageId: "5", - pattern: [{ type: "text", value: "hello world from Berlin" }], - }; - const diff = await inlangLixPluginV1.diff.variant({ old, neu }); - expect(diff).toEqual([ - { - meta: { id: "1" }, - operation: "update", - type: "variant", - neu, - old, - } satisfies DiffReport, - ]); - }); - - test("old is undefined and neu is defined should return a diff report for the new value", async () => { - const old = undefined; - const neu: Variant = { - id: "1", - matches: [], - messageId: "5", - pattern: [{ type: "text", value: "hello world" }], - }; - const diff = await inlangLixPluginV1.diff.variant({ old, neu }); - expect(diff).toEqual([ - { operation: "create", type: "variant", neu, old } satisfies DiffReport, - ]); - }); -}); diff --git a/packages/sdk/src/lix-plugin/inlangLixPluginV1.ts b/packages/sdk/src/lix-plugin/inlangLixPluginV1.ts deleted file mode 100644 index 0306c40031..0000000000 --- a/packages/sdk/src/lix-plugin/inlangLixPluginV1.ts +++ /dev/null @@ -1,132 +0,0 @@ -// eslint-disable-next-line @typescript-eslint/ban-ts-comment -// @ts-nocheck - -import type { DiffReport, LixPlugin } from "@lix-js/sdk"; -import { loadDatabaseInMemory } from "sqlite-wasm-kysely"; -import { initDb } from "../database/initDb.js"; -import { applyChanges } from "./applyChanges.js"; -import { detectConflicts } from "./detectConflicts.js"; -import type { Bundle, Message, Variant } from "../database/schema.js"; - -export const inlangLixPluginV1: LixPlugin<{ - bundle: Bundle; - message: Message; - variant: Variant; -}> = { - key: "inlang-lix-plugin-v1", - glob: "*", - applyChanges, - detectConflicts, - diff: { - // TODO does not account for deletions - file: async ({ old, neu }) => { - // can only handle the database for now - if (neu === undefined || neu.path?.endsWith("db.sqlite") === false) { - return []; - } - const result: DiffReport[] = []; - const oldDb = old - ? initDb({ sqlite: await loadDatabaseInMemory(old.data) }) - : undefined; - - const newDb = neu - ? initDb({ - sqlite: await loadDatabaseInMemory(neu.data), - }) - : undefined; - - const newProjectBundles = await newDb - ?.selectFrom("bundle") - .selectAll() - .execute(); - - const newProjectMessages = await newDb - ?.selectFrom("message") - .selectAll() - .execute(); - const newProjectVariants = await newDb - ?.selectFrom("variant") - .selectAll() - .execute(); - - for (const bundle of newProjectBundles ?? []) { - const oldBundle = await oldDb - ?.selectFrom("bundle") - .selectAll() - .where("id", "=", bundle.id) - .executeTakeFirst(); - result.push( - ...(await inlangLixPluginV1.diff.bundle({ - old: oldBundle, - neu: bundle, - })) - ); - } - for (const message of newProjectMessages ?? []) { - const oldMessage = await oldDb - ?.selectFrom("message") - .selectAll() - .where("id", "=", message.id) - .executeTakeFirst(); - - result.push( - ...(await inlangLixPluginV1.diff.message({ - old: oldMessage, - neu: message, - })) - ); - } - for (const variant of newProjectVariants ?? []) { - const oldVariant = await oldDb - ?.selectFrom("variant") - .selectAll() - .where("id", "=", variant.id) - .executeTakeFirst(); - result.push( - ...(await inlangLixPluginV1.diff.variant({ - old: oldVariant, - neu: variant, - })) - ); - } - - return result; - }, - bundle: ({ old, neu }) => diffSnapshot({ old, neu, type: "bundle" }), - message: ({ old, neu }) => diffSnapshot({ old, neu, type: "message" }), - variant: ({ old, neu }) => diffSnapshot({ old, neu, type: "variant" }), - }, -}; - -function diffSnapshot({ - old, - neu, - type, -}: { - old?: Bundle | Message | Variant; - neu?: Bundle | Message | Variant; - type: "bundle" | "message" | "variant"; -}): DiffReport[] { - if (old === undefined && neu) { - return [{ type, old, neu, operation: "create" } satisfies DiffReport]; - } else if (old !== undefined && neu === undefined) { - return [{ type, old, neu, operation: "delete" } satisfies DiffReport]; - } else if (old && neu) { - const hasDiff = JSON.stringify(old) !== JSON.stringify(neu); - if (hasDiff) { - return [ - { - type, - meta: { - // id is required for deletions - id: old.id ?? neu.id, - }, - operation: "update", - old, - neu, - } satisfies DiffReport, - ]; - } - } - return []; -} diff --git a/packages/sdk/src/lix-plugin/merge.test.ts b/packages/sdk/src/lix-plugin/merge.test.ts deleted file mode 100644 index ae8c7cff71..0000000000 --- a/packages/sdk/src/lix-plugin/merge.test.ts +++ /dev/null @@ -1,133 +0,0 @@ -// eslint-disable-next-line @typescript-eslint/ban-ts-comment -// @ts-nocheck - -import { merge, type NewChange } from "@lix-js/sdk"; -import { test, expect } from "vitest"; -import { loadProjectInMemory } from "../project/loadProjectInMemory.js"; -import { newProject } from "../project/newProject.js"; -import { inlangLixPluginV1 } from "./inlangLixPluginV1.js"; -import type { NewBundle, NewMessage, NewVariant } from "../database/schema.js"; - -test.skip("it should update the variant to the source's value", async () => { - const target = await loadProjectInMemory({ blob: await newProject() }); - const source = await loadProjectInMemory({ blob: await target.toBlob() }); - - const dbFile = await target.lix.db - .selectFrom("file") - .select("id") - .where("path", "=", "/db.sqlite") - .executeTakeFirstOrThrow(); - - const commonChanges: NewChange[] = [ - { - id: "d92cdc2e-74cc-494c-8d51-958216272a17", - parent_id: undefined, - type: "bundle", - file_id: dbFile.id, - plugin_key: inlangLixPluginV1["key"], - operation: "create", - value: { - id: "even_hour_mule_drum", - } satisfies NewBundle, - commit_id: "c8ad005b-a834-4ca3-84fb-9627546f2eba", - }, - { - id: "24f74fec-fc8a-4c68-b31c-d126417ce3af", - parent_id: undefined, - type: "message", - file_id: dbFile.id, - plugin_key: inlangLixPluginV1["key"], - operation: "create", - value: { - id: "c2684c3d-3e14-47e4-96b7-8d33579bb7e9", - bundleId: "even_hour_mule_drum", - locale: "en", - selectors: [], - } satisfies NewMessage, - commit_id: "c8ad005b-a834-4ca3-84fb-9627546f2eba", - }, - { - id: "aaf0ec32-0c7f-4d07-af8c-922ce382aef1", - parent_id: undefined, - type: "variant", - file_id: dbFile.id, - plugin_key: inlangLixPluginV1["key"], - operation: "create", - value: { - id: "6a860f96-0cf3-477c-80ad-7893d8fde852", - messageId: "c2684c3d-3e14-47e4-96b7-8d33579bb7e9", - matches: [], - pattern: [ - { - type: "text", - value: "My Awesome Todo App", - }, - ], - } satisfies NewVariant, - commit_id: "c8ad005b-a834-4ca3-84fb-9627546f2eba", - }, - ]; - - const changesOnlyInTarget: NewChange[] = []; - const changesOnlyInSource: NewChange[] = [ - { - id: "01c059f9-8476-4aaa-aa6d-53ea3158b374", - parent_id: "aaf0ec32-0c7f-4d07-af8c-922ce382aef1", - type: "variant", - file_id: dbFile.id, - plugin_key: inlangLixPluginV1["key"], - operation: "update", - value: { - id: "6a860f96-0cf3-477c-80ad-7893d8fde852", - messageId: "c2684c3d-3e14-47e4-96b7-8d33579bb7e9", - matches: [], - pattern: [ - { - type: "text", - value: "My app is buggy", - }, - ], - } satisfies NewVariant, - meta: { - id: "6a860f96-0cf3-477c-80ad-7893d8fde852", - }, - commit_id: "df455c78-b5ed-4df0-9259-7bb694c9d755", - }, - ]; - - await source.lix.db - .insertInto("change") - .values([...commonChanges, ...changesOnlyInSource]) - .execute(); - - await target.lix.db - .insertInto("change") - .values([...commonChanges, ...changesOnlyInTarget]) - .execute(); - - await merge({ sourceLix: source.lix, targetLix: target.lix }); - - await target.lix.settled(); - - // need to reload project to re-initialize the state - const mergedProject = await loadProjectInMemory({ - blob: await target.toBlob(), - }); - - const changes = await mergedProject.lix.db - .selectFrom("change") - .selectAll() - .execute(); - - expect(changes).lengthOf(4); - - const variant = await mergedProject.db - .selectFrom("variant") - .selectAll() - .where("id", "=", "6a860f96-0cf3-477c-80ad-7893d8fde852") - .executeTakeFirstOrThrow(); - - expect(variant).toEqual( - expect.objectContaining(changesOnlyInSource[0]?.value) - ); -}); diff --git a/packages/sdk/src/lix-plugin/resolveConflictBySelecting.test.ts b/packages/sdk/src/lix-plugin/resolveConflictBySelecting.test.ts deleted file mode 100644 index 17900d7954..0000000000 --- a/packages/sdk/src/lix-plugin/resolveConflictBySelecting.test.ts +++ /dev/null @@ -1,188 +0,0 @@ -// eslint-disable-next-line @typescript-eslint/ban-ts-comment -// @ts-nocheck - -import { test, expect } from "vitest"; -import { newProject } from "../project/newProject.js"; -import { loadProjectInMemory } from "../project/loadProjectInMemory.js"; -import { - isInSimulatedCurrentBranch, - resolveConflictBySelecting, -} from "@lix-js/sdk"; -import { contentFromDatabase } from "sqlite-wasm-kysely"; - -test.skip("it should resolve a conflict with the selected change", async () => { - const project = await loadProjectInMemory({ blob: await newProject() }); - - const dbFile = await project.lix.db - .selectFrom("file") - .select("id") - .where("path", "=", "/db.sqlite") - .executeTakeFirstOrThrow(); - - const changes = await project.lix.db - .insertInto("change") - .values([ - { - id: "e59b2fc6-612d-4087-9790-03b2ab724274", - author: "Anonymous", - file_id: dbFile.id, - plugin_key: "inlang-lix-plugin-v1", - operation: "create", - type: "bundle", - value: { - id: "even_hour_mule_drum", - }, - created_at: "2024-08-28 20:58:04", - }, - { - id: "c0c9e3f3-0eba-4233-813f-e1901401a653", - author: "Anonymous", - file_id: dbFile.id, - plugin_key: "inlang-lix-plugin-v1", - operation: "create", - type: "message", - value: { - id: "c2684c3d-3e14-47e4-96b7-8d33579bb7e9", - bundleId: "even_hour_mule_drum", - locale: "en", - selectors: [], - }, - created_at: "2024-08-28 20:58:04", - }, - { - id: "b92a634f-1e25-410f-af5a-e8702cc92ef0", - author: "Anonymous", - file_id: dbFile.id, - plugin_key: "inlang-lix-plugin-v1", - operation: "create", - type: "variant", - value: { - id: "6a860f96-0cf3-477c-80ad-7893d8fde852", - messageId: "c2684c3d-3e14-47e4-96b7-8d33579bb7e9", - matches: [], - pattern: [ - { - type: "text", - value: "My Awesome Todo App", - }, - ], - }, - created_at: "2024-08-28 20:58:04", - }, - { - id: "samuels-change", - author: "Samuel", - parent_id: "b92a634f-1e25-410f-af5a-e8702cc92ef0", - file_id: dbFile.id, - plugin_key: "inlang-lix-plugin-v1", - operation: "update", - type: "variant", - value: { - id: "6a860f96-0cf3-477c-80ad-7893d8fde852", - messageId: "c2684c3d-3e14-47e4-96b7-8d33579bb7e9", - matches: [], - pattern: [ - { - type: "text", - value: "My awesome lix app", - }, - ], - }, - meta: { - id: "6a860f96-0cf3-477c-80ad-7893d8fde852", - }, - created_at: "2024-08-28 21:29:05", - }, - { - id: "peters-change", - author: "Peter", - parent_id: "b92a634f-1e25-410f-af5a-e8702cc92ef0", - file_id: dbFile.id, - plugin_key: "inlang-lix-plugin-v1", - operation: "update", - type: "variant", - value: { - id: "6a860f96-0cf3-477c-80ad-7893d8fde852", - messageId: "c2684c3d-3e14-47e4-96b7-8d33579bb7e9", - matches: [], - pattern: [ - { - type: "text", - value: "You are building an app", - }, - ], - }, - meta: { - id: "6a860f96-0cf3-477c-80ad-7893d8fde852", - }, - created_at: "2024-08-28 21:29:35", - }, - ]) - .returningAll() - .execute(); - - // --------------- - // Todo inlang sdk should auto write file to lix and differ must be - // fault tolerant. without this code, the differ captures an old - // change again - - for (const change of changes) { - await project.db - .insertInto(change.type as any) - .values(change.value as any) - .onConflict((c) => c.doNothing()) - .execute(); - } - await project.lix.db - .updateTable("file") - .set({ data: contentFromDatabase(project._sqlite) }) - .where("path", "=", "/db.sqlite") - .execute(); - // --------------- - - const conflicts = await project.lix.db - .insertInto("conflict") - .values([ - { - change_id: "samuels-change", - conflicting_change_id: "peters-change", - reason: "", - }, - ]) - .returningAll() - .execute(); - - const changesInCurrentBranchBefore = await project.lix.db - .selectFrom("change") - .selectAll() - .where(isInSimulatedCurrentBranch) - .execute(); - - expect(changesInCurrentBranchBefore.map((c) => c.id)).toEqual([ - changes[0]?.id, - changes[1]?.id, - changes[2]?.id, - "samuels-change", - ]); - - await resolveConflictBySelecting({ - lix: project.lix, - conflict: conflicts[0]!, - selectChangeId: "peters-change", - }); - - await project.lix.settled(); - - const changesInCurrentBranch = await project.lix.db - .selectFrom("change") - .selectAll() - .where(isInSimulatedCurrentBranch) - .execute(); - - expect(changesInCurrentBranch.map((c) => c.id)).toEqual([ - changes[0]?.id, - changes[1]?.id, - changes[2]?.id, - "peters-change", - ]); -}); diff --git a/packages/sdk/src/project/README_CONTENT.ts b/packages/sdk/src/project/README_CONTENT.ts index 45866c4b2c..bdc223c226 100644 --- a/packages/sdk/src/project/README_CONTENT.ts +++ b/packages/sdk/src/project/README_CONTENT.ts @@ -13,7 +13,7 @@ This is an [unpacked (git-friendly)](https://inlang.com/docs/unpacked-project) i Purpose: - This folder is the Git-friendly representation of an \`.inlang\` project. -- The canonical \`.inlang\` format is a single binary file; this directory is the unpacked version for Git. +- The canonical \`.inlang\` format is a portable snapshot; this directory is the unpacked version for Git. - This folder stores project configuration and plugin cache data. - Translation files live outside this folder and are referenced from \`settings.json\`. @@ -44,9 +44,9 @@ Translation files (like \`messages/en.json\`) live **outside** this folder and a ## What is inlang? -[Inlang](https://inlang.com) is an open project file format for localization. An \`.inlang\` project is canonically a single binary file: a SQLite database with version control via [lix](https://lix.dev). Like \`.sqlite\` for relational data, \`.inlang\` packages localization data into one file that tools can share. +[Inlang](https://inlang.com) is an open project file format for localization. An \`.inlang\` project is canonically a portable snapshot backed by [Lix](https://lix.dev). It packages localization data and project files into one file that tools can share. -For Git repositories, that binary file can be unpacked into a directory of plain files. The packed file is the canonical format; this directory is the Git-friendly representation. +For Git repositories, that packed file can be unpacked into a directory of plain files. The packed file is the canonical format; this directory is the Git-friendly representation. Use inlang when multiple tools, teams, automations, or agents need to use the same localization data. The \`@inlang/sdk\` is the reference implementation for reading and writing \`.inlang\` projects. diff --git a/packages/sdk/src/project/loadProjectFromDirectory.test.ts b/packages/sdk/src/project/loadProjectFromDirectory.test.ts index 5057dbd3ec..6500a11105 100644 --- a/packages/sdk/src/project/loadProjectFromDirectory.test.ts +++ b/packages/sdk/src/project/loadProjectFromDirectory.test.ts @@ -483,7 +483,7 @@ describe("it should keep files between the inlang directory and lix in sync", as const files = await selectLixFiles(project.lix); - expect(files.length).toBe(6 + 1 /* the db.sqlite file */); + expect(files.length).toBe(6); const filesByPath = files.reduce((acc, file) => { acc[file.path] = new TextDecoder().decode(file.content); diff --git a/packages/sdk/src/project/loadProjectFromDirectory.ts b/packages/sdk/src/project/loadProjectFromDirectory.ts index 5b823f284a..98162fb042 100644 --- a/packages/sdk/src/project/loadProjectFromDirectory.ts +++ b/packages/sdk/src/project/loadProjectFromDirectory.ts @@ -326,10 +326,7 @@ async function syncLixFsFiles(args: { async function checkLixState(currentLixState: FsFileState) { // go through all files in lix and check there state const filesInLix = ( - await args.lix.execute( - "SELECT path, content FROM lix_file WHERE path NOT LIKE $1", - ["%db.sqlite"] - ) + await args.lix.execute("SELECT path, content FROM lix_file") ).rows.map((row) => ({ path: row.get("path") as string, content: row.value("content").asBytes()!, diff --git a/packages/sdk/src/project/saveProjectToDirectory.test.ts b/packages/sdk/src/project/saveProjectToDirectory.test.ts index fd57adf5f4..6247649256 100644 --- a/packages/sdk/src/project/saveProjectToDirectory.test.ts +++ b/packages/sdk/src/project/saveProjectToDirectory.test.ts @@ -21,7 +21,7 @@ test("it should throw if the path doesn't end with .inlang", async () => { ).rejects.toThrowError("The path must end with .inlang"); }); -test("it should overwrite all files to the directory except the db.sqlite file", async () => { +test("it should write project files to the directory", async () => { const mockFs = Volume.fromJSON({ "/foo/bar.inlang/settings.json": JSON.stringify({ baseLocale: "en", @@ -55,7 +55,6 @@ test("it should overwrite all files to the directory except the db.sqlite file", // this test should be updated for files that should NOT // be contained in the directory in the future expect(files).toContain("settings.json"); - expect(files).not.toContain("db.sqlite"); expect(updatedSettings.baseLocale).toBe("en"); expect(updatedSettings.locales).toEqual(["en", "fr", "mock"]); }); diff --git a/packages/sdk/src/project/saveProjectToDirectory.ts b/packages/sdk/src/project/saveProjectToDirectory.ts index 4d483f4583..21dd0ef71b 100644 --- a/packages/sdk/src/project/saveProjectToDirectory.ts +++ b/packages/sdk/src/project/saveProjectToDirectory.ts @@ -126,7 +126,7 @@ export async function saveProjectToDirectory(args: { // write all files to the directory for (const file of files) { - if (file.path.endsWith("db.sqlite") || file.path === "/project_id") { + if (file.path === "/project_id") { continue; } const p = path.join(args.path, file.path); diff --git a/packages/sdk/src/project/snapshot.ts b/packages/sdk/src/project/snapshot.ts index 48c59d50ef..5a9ea5a6e3 100644 --- a/packages/sdk/src/project/snapshot.ts +++ b/packages/sdk/src/project/snapshot.ts @@ -73,7 +73,7 @@ export async function restoreProjectBlob(lix: Lix, blob: Blob): Promise { parsed = JSON.parse(await blob.text()); } catch (cause) { throw new Error( - "The project uses the legacy Lix SQLite format, which Lix 0.9 cannot open in memory.", + "The project uses a legacy binary Lix format that this SDK cannot open in memory.", { cause } ); } diff --git a/packages/sdk/src/query-utilities/selectBundleNested.ts b/packages/sdk/src/query-utilities/selectBundleNested.ts index 992283d561..023ca0e414 100644 --- a/packages/sdk/src/query-utilities/selectBundleNested.ts +++ b/packages/sdk/src/query-utilities/selectBundleNested.ts @@ -9,8 +9,7 @@ import type { /** * Select bundles with nested messages and variants. * - * Lix uses DataFusion, which does not provide SQLite's json_group_array() - * helper. A flat left join keeps this to one engine round-trip and the small + * A flat left join keeps this to one Lix engine round-trip and the small * reconstruction below preserves the established SDK result shape. */ export const selectBundleNested = (db: Kysely) => { diff --git a/packages/website-v2/src/content/landingpage.md b/packages/website-v2/src/content/landingpage.md index a5b4e882c6..292b1cefcf 100644 --- a/packages/website-v2/src/content/landingpage.md +++ b/packages/website-v2/src/content/landingpage.md @@ -25,9 +25,9 @@ Inlang is the open-format TMS (translation management system) for software teams Store translations in your repo as a vendor-neutral file format, so developers, translators, CI, translation tools, and AI agents can read and update the same localization source of truth. -An `.inlang` project is canonically a single binary file: a SQLite database with version control via [lix](https://lix.dev). Like `.sqlite` for relational data, `.inlang` packages localization data into one file that tools can share. +An `.inlang` project is canonically a portable snapshot backed by [Lix](https://lix.dev). It packages localization data and project files into one file that tools can share. -For Git repositories, the binary file can be unpacked into a directory of plain files so changes can be reviewed alongside code. The packed file is the canonical format; the unpacked directory is the Git-friendly representation. The `@inlang/sdk` is the reference implementation for reading and writing `.inlang` projects. +For Git repositories, the packed file can be unpacked into a directory of plain files so changes can be reviewed alongside code. The packed file is the canonical format; the unpacked directory is the Git-friendly representation. The `@inlang/sdk` is the reference implementation for reading and writing `.inlang` projects. `.inlang` is the canonical open format for localization. Plugins import and export formats like JSON, ICU MessageFormat v1, i18next, and XLIFF for compatibility with existing translation files and runtimes. diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 18edfe378f..fbe7932465 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -848,9 +848,6 @@ importers: kysely: specifier: ^0.28.12 version: 0.28.14 - sqlite-wasm-kysely: - specifier: 0.3.0 - version: 0.3.0(kysely@0.28.14) uuid: specifier: ^14.0.0 version: 14.0.0 @@ -4072,10 +4069,6 @@ packages: resolution: {integrity: sha512-10s/u/Main1RGO+jjzK+mgC/zh1ls1CEnq3Dujr03TwvzLg+j4FAohOmlYkQj8KQOj1vGR9cuB9F8tVBTwVGVA==} hasBin: true - '@sqlite.org/sqlite-wasm@3.48.0-build4': - resolution: {integrity: sha512-hI6twvUkzOmyGZhQMza1gpfqErZxXRw6JEsiVjUbo7tFanVD+8Oil0Ih3l2nGzHdxPI41zFmfUQG7GHqhciKZQ==} - hasBin: true - '@standard-schema/spec@1.1.0': resolution: {integrity: sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w==} @@ -10283,11 +10276,6 @@ packages: sprintf-js@1.1.3: resolution: {integrity: sha512-Oo+0REFV59/rz3gfJNKQiBlwfHaSESl1pcGyABQsnnIfWOFt6JNj5gCog2U6MLZ//IGYD+nA8nI+mTShREReaA==} - sqlite-wasm-kysely@0.3.0: - resolution: {integrity: sha512-TzjBNv7KwRw6E3pdKdlRyZiTmUIE0UttT/Sl56MVwVARl/u5gp978KepazCJZewFUnlWHz9i3NQd4kOtP/Afdg==} - peerDependencies: - kysely: '*' - srvx@0.10.0: resolution: {integrity: sha512-NqIsR+wQCfkvvwczBh8J8uM4wTZx41K2lLSEp/3oMp917ODVVMtW5Me4epCmQ3gH8D+0b+/t4xxkUKutyhimTA==} engines: {node: '>=20.16.0'} @@ -14811,8 +14799,6 @@ snapshots: '@sqlite.org/sqlite-wasm@3.46.0-build2': {} - '@sqlite.org/sqlite-wasm@3.48.0-build4': {} - '@standard-schema/spec@1.1.0': {} '@storybook/addon-actions@8.6.14(storybook@8.6.14(prettier@3.6.2))': @@ -22779,11 +22765,6 @@ snapshots: sprintf-js@1.1.3: {} - sqlite-wasm-kysely@0.3.0(kysely@0.28.14): - dependencies: - '@sqlite.org/sqlite-wasm': 3.48.0-build4 - kysely: 0.28.14 - srvx@0.10.0: {} stackback@0.0.2: {} From b84d7861d78e868325a0883e462225b9f707b240 Mon Sep 17 00:00:00 2001 From: Samuel Stroschein <35429197+samuelstroschein@users.noreply.github.com> Date: Tue, 25 Aug 2026 17:41:22 +0000 Subject: [PATCH 2/2] chore(sdk): release SQLite cleanup as patch --- .changeset/remove-sdk-sqlite-wasm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.changeset/remove-sdk-sqlite-wasm.md b/.changeset/remove-sdk-sqlite-wasm.md index 81ec34ccd9..93070dfdc4 100644 --- a/.changeset/remove-sdk-sqlite-wasm.md +++ b/.changeset/remove-sdk-sqlite-wasm.md @@ -1,5 +1,5 @@ --- -"@inlang/sdk": major +"@inlang/sdk": patch --- Remove the obsolete SQLite WASM dependency, public schema initializer, and special handling for unsupported legacy database artifacts. The SDK database API uses Lix through Kysely's PostgreSQL query compiler.