RFC: A standard database-access interface (Driver seam) for MoonBit

[Lfan-ke]

Background

MoonBit already has a dozen database packages — oboard/morm, mizchi/sqlite, jaredzhou/moonpg, mattn/postgres, bikallem/mariadb, and more — and no interface between them.

And

Each one invents its own Value, Row, error type, and execute/query surface.

But

So an ORM or query builder is welded to a single backend, a new backend can reuse nobody's pooling, decoding, or transaction machinery, and there is no shared type for anything to interoperate through. Go solved exactly this with the database/sql / database/sql/driver split: a small contract that backends implement, and a thick layer written once above it.

Therefore

This RFC proposes that split for MoonBit — adapted to no reflection and sealed enums by making the value model trait-based rather than a closed enum — and asks whether the sync, zero-dependency subset belongs in moonbitlang/core. A driver-backed reference implementation already exists (links at the end); this issue is the design discussion before any code is proposed for core.

Proposed interface

A closed value vocabulary as the internal interchange type, and open ToParam/FromValue traits as the public surface. The Driver trait is the whole backend contract.

pub(all) enum Value {                    // interchange vocabulary; drivers match exhaustively
  Null; Bool(Bool); Int(Int); Int64(Int64); Double(Double); Text(String); Blob(Bytes)
} derive(Eq)

pub(open) trait ToParam   { to_param(Self) -> Value }                       // bind side (open)
pub(open) trait FromValue { from_value(Value) -> Self raise DbError }       // read side (open)
// blanket impls carry NULL:  impl[T:ToParam] ToParam for T?;  impl[T:FromValue] FromValue for T?

pub(open) trait QueryExecutor {          // a Conn, a Pool, or a Tx — all substitutable
  execute(Self, String, params? : Array[&ToParam]) -> ExecResult raise DbError
  query(Self, String, params? : Array[&ToParam]) -> Array[Row] raise DbError
}
pub(open) trait Driver : QueryExecutor { // the out-of-tree backend contract, ~6 methods
  begin(Self) -> Unit raise DbError
  commit(Self) -> Unit raise DbError
  rollback(Self) -> Unit raise DbError
  close(Self) -> Unit
  capabilities(Self) -> Capabilities = _
}

pub(all) struct Row { columns : Array[String]; values : Array[Value] }
pub fn Row::get[T : FromValue](Self, Int) -> T raise DbError               // typed, checked
pub(all) struct ExecResult { rows_affected : Int64; last_insert_id : Int64 }

pub(all) suberror DbError {
  ConnectError(String); QueryError(kind~ : ErrorKind, code~ : String, msg~ : String)
  TypeError(String); NoRows; Closed
}
pub(all) enum ErrorKind {                // portable classes; raw vendor code stays a String
  UniqueViolation; ForeignKeyViolation; NotNull; Check
  SerializationFailure; ConnectionLost; Timeout; Syntax; Other
}

Layered above the seam, written once, generic over Driver: Pool[D]; a closure-scoped transaction(f) that commits on Ok and rolls back on any raise (MoonBit has no Drop, so this is the primary transaction API); Tx : QueryExecutor so query code runs unchanged inside a transaction; and optional capability traits Streamer/Cursor (large result sets), Preparer/Stmt (prepared handles), and Tx::savepoint. open is not a trait method — connection setup is backend-specific; the seam begins at a live handle, which is what keeps it transport-agnostic (native TCP, native C-FFI, wasm host imports).

Params are always a positional array at the seam; the placeholder token (? vs $1) is read from Capabilities, not baked in. NULL is Value::Null and Option[T], never a sentinel. IndexedDB is an explicit non-goal — it has no SQL surface — and belongs to a sibling KV trait; the browser's SQL story is a wasm-SQLite impl Driver.

Everything in this section is the proposed target surface. The reference implementation currently ships a smaller, concrete first cut of it, and the exact built-versus-proposed gap is spelled out under Reference implementation below — nothing here is claimed as already-built beyond what that section lists.

Async

The standard seam is sync (raise), so it is zero-dependency and compiles on native, wasm, and js; native drivers run their async runtime internally behind it. A parallel native-only AsyncDriver trait (in a sub-package that deps moonbitlang/async) mirrors the six I/O methods for callers who want true async concurrency. MoonBit has no colour polymorphism, so the two trait definitions are unavoidable — but the entire value model (Value, Row, ToParam/FromValue, DbError, Capabilities, TxOptions, pool and transaction logic) is colour-neutral and shared verbatim. Only the method colours differ.

The reference implementation stressed this boundary and surfaced a limit worth stating plainly. An FFI driver (sqlite, blocking C) satisfies the sync seam completely, transactions included. A wire-protocol driver cannot: moonbitlang/async is async-only and exposes no re-entrant sync bridge — its one public entry, run_async_main, tears down its socket fds on return, so a synchronous execute/query can only work by reconnecting per call, which cannot hold a transaction across calls, and begin/commit/rollback on the sync adapter must therefore raise. The honest division of labour that falls out of this: the sync Driver is the portable, FFI-complete floor and the autocommit face of network drivers, while the native-only AsyncDriver is where transactional wire-protocol drivers actually live. That makes AsyncDriver not optional polish but a first-class half of the design — and it is exactly why it must stay a native-only sibling and never fold into an all-target core.

Why a standard package

The value is interop, and interop only exists if there is one shared type. If moonorm decodes moondb.Row and moon-postgres produces it, they compose with no adapter. A custom Uuid that implements moondb.ToParam binds on every backend at once. A Pool written against moondb.Driver pools every backend. None of this is possible while each package owns its own Value/Row/error. This is the same reason database/sql, java.sql, and DB-API live in their respective standard libraries rather than as competing third-party crates.

Extensibility guarantees

• New backend: implement Driver (+ optional capability traits) in a separate package. Map columns to the 7 Value cases, errors to ErrorKind, quirks to Capabilities. Zero edits to the standard package; Pool, transaction, and every query layer work unchanged.
• New value type (Decimal, Uuid, Timestamp, JSON): implement ToParam + FromValue in a separate package. Zero edits to the standard package and zero edits to any driver — the type rides existing Text/Blob. This is what a closed Value enum cannot do: adding a case there would force a release plus a match-arm in every driver and break every downstream match. The closed enum is the portable floor; the open traits are the unbounded ceiling.
• Backward-compatible growth: new capabilities are new optional traits and new defaulted Capabilities fields — existing drivers keep compiling (Go's additive property, without Go's invisible runtime type-asserts, because capabilities are an inspectable value).
• A zero-dependency MockDriver in the standard package proves the contract is implementable on every target and serves as every query layer's test double.

Prior art

• Go database/sql/driver — the two-layer split (thin driver contract, thick shared layer); closed 7-type driver.Value; symmetric Valuer/Scanner. We keep the split, drop the reflection.
• Rust sqlx — Encode/Decode/Type as an open value model, no closed value enum, NULL via Option; the direct inspiration for ToParam/FromValue. We drop the GAT-heavy ValueRef (own values at the row boundary) and the async-only, drop-cancel model.
• Python DB-API 2.0 — the portable exception category tree (our ErrorKind); a warning against five fragmented paramstyles (we fix one positional convention) and duck typing (we use real traits).
• JDBC / ADO.NET — interface-as-contract + registry-as-discovery; a warning about unsound NULL (wasNull/DBNull) and wide mandatory interfaces (we use a tiny core + optional capability traits, not SQLFeatureNotSupportedException stubs).
• JS/wasm (Kysely) — the {sql, positional params} → {rows, insert_id, affected} data boundary; placeholders as compiler output; IndexedDB kept out of the SQL driver.

Placement: the sync value-model subset belongs in core

We propose the sync, zero-dependency subset — Value, Row, ToParam/FromValue, Driver, DbError, Capabilities, ExecResult — for moonbitlang/core; AsyncDriver, Pool, and concrete drivers stay as separate packages.

This subset is what core is for: zero dependencies, all three targets (native/wasm/js), small, and stable. It is the single shared vocabulary that lets independent drivers and query layers interoperate — the role database/sql/driver plays in Go's standard library. core membership is also the only thing that anoints one seam over competing third-party ones; without it, fragmentation is the status-quo failure. The parts that stay outside are exactly the parts that cannot be all-target or stable: AsyncDriver depends on the native-only moonbitlang/async, and Pool and the concrete drivers need to move on their own semver. That boundary is a feature — core holds a frozen seam while the ecosystem iterates.

The honest constraint is that a seam should be frozen only after a real driver has stressed it. A reference implementation accompanies this proposal — a query layer over the seam, an FFI sqlite driver, and native-wire postgres and mysql drivers, each tested against a real database in CI (SQLite, PostgreSQL 16, MySQL 8) — so the seam is proposed driver-backed, not on paper. That same exercise is what surfaced the async boundary described above — the proposal reflects what a real driver ran into, not a whiteboard guess.

To be precise about the gap between this proposal and the prototype: the shipped moondb is a smaller first cut. Its Driver is six concrete methods — execute/query take (String, Array[Value]) directly, plus begin/commit/rollback/close; its DbError is four cases (ConnectError / QueryError(String) / TypeError / Closed); Value, Row (with concrete typed accessors), ExecResult, and an in-memory MockDriver are as described above. The sqlite, postgres, and mysql drivers each implement that concrete Driver against a real database in CI. The open ToParam/FromValue traits, the QueryExecutor split, Capabilities, the ErrorKind-tagged QueryError, Pool, and the closure transaction shown in the proposal above are the intended evolution — not yet built, and exactly what this RFC asks maintainers to weigh before the surface is frozen. The prototype exists to prove the seam is real and driver-backed, not to present the final surface.

Reference implementation

All packages are public, Apache-2.0, and green in CI against real databases. They live in one repository — Lfan-ke/moonorm — as independently-published packages (the ORM at the root, the interface and drivers as sub-packages), each with its own moon.mod and mooncakes version:

• moondb — the interface package itself, plus the in-memory MockDriver: https://github.com/Lfan-ke/moonorm/tree/master/db
• moon-sqlite — FFI driver over the SQLite amalgamation (the one C-touching package): https://github.com/Lfan-ke/moonorm/tree/master/drivers/sqlite
• moon-postgres — pure-MoonBit PostgreSQL v3 wire driver, tested against PostgreSQL 16 in CI: https://github.com/Lfan-ke/moonorm/tree/master/drivers/postgres
• moon-mysql — pure-MoonBit MySQL/MariaDB wire driver, tested against MySQL 8 and MariaDB 10.11/11 in CI: https://github.com/Lfan-ke/moonorm/tree/master/drivers/mysql
• moonorm — a query/ORM layer written entirely against the seam, to show a consumer composes with any driver: https://github.com/Lfan-ke/moonorm

Question for maintainers: is the sync value-model subset acceptable for core, and is there anything you would change before it is frozen under core's stability guarantees?