diff --git a/README.md b/README.md index cc383b7..2ddd214 100644 --- a/README.md +++ b/README.md @@ -1,49 +1,117 @@ # lsmdb -An LSM-tree based relational database built in Rust. +An LSM-tree relational database in Rust for write-heavy, single-node workloads with durable storage and MVCC semantics. -## Current Status +## Why lsmdb -- Initial project skeleton is in place. -- Storage modules are scaffolded (WAL, MemTable, SSTable, Manifest, Compaction, Engine). -- Transaction modules are scaffolded (MVCC). -- Relational modules are scaffolded (Catalog, SQL, Planner, Executor). -- Server and CLI modules are scaffolded. -- Docs, integration test files, and benchmark entry points are scaffolded. -- Core engine behavior is not fully implemented yet. +`lsmdb` is designed for developers who want a modern storage architecture with a small, hackable Rust codebase. -## Layout +- **Write-optimized architecture**: WAL + MemTable + SSTable pipeline minimizes random write cost on SSD. +- **Durability first**: WAL replay, manifest tracking, checksum validation, and recovery-focused tests. +- **MVCC transactions**: snapshot-isolation behavior with conflict detection. +- **SQL pipeline included**: parser, validator, planner, and executor for a practical SQL subset. +- **Flexible usage**: run embedded in a Rust service or expose over the built-in TCP server protocol. -- `src/` database core library code -- `tests/integration/` end-to-end and subsystem integration tests +## What it uses + +- **Language/runtime**: Rust + Tokio +- **Storage stack**: WAL, MemTable, SSTable, Manifest, background flush/compaction +- **Concurrency/transactions**: MVCC (timestamps, snapshots, GC) +- **SQL stack**: lexer, parser, AST, validator, logical/physical planning, executor +- **Ops tooling**: CLI client, admin utility, diagnostics bundle, release gate checks + +See: +- `docs/architecture.md` +- `docs/sql_subset.md` +- `docs/testing.md` +- `docs/release_gate.md` + +## How this benefits developers + +- **Simple local integration**: use embedded mode directly in Rust without external DB process. +- **Deterministic behavior**: explicit test coverage around recovery, MVCC, and SQL semantics. +- **Operational visibility**: health/readiness/admin status payloads and diagnostics bundle support. +- **Fast iteration**: modular code structure (`storage`, `mvcc`, `sql`, `planner`, `executor`, `server`). + +## Current status + +Core storage, SQL execution, and integration tests are implemented and actively exercised. + +Known limitations for production adoption are tracked in `team_issues/` and release gate criteria in `docs/release_gate.md`. + +## Use it now + +### Option A: Embedded (recommended today) + +Use `lsmdb` directly inside your Rust app: + +```rust +use lsmdb::catalog::Catalog; +use lsmdb::executor::ExecutionSession; +use lsmdb::mvcc::MvccStore; + +let store = MvccStore::open_persistent("./data")?; +let catalog = Catalog::open(store.clone())?; +let mut session = ExecutionSession::new(&catalog, &store); +// parse -> validate -> plan -> execute SQL using the session +``` + +Reference: `tests/sql_persistence.rs`. + +### Option B: Server + CLI + +`lsmdb` includes server components and a custom wire protocol (`src/server/*`) plus a CLI client (`tools/lsmdb-cli/`). + +- Start a server from your application by calling `start_server_with_options(...)`. +- Connect using the CLI: + +```bash +cargo run --bin lsmdb-cli -- --addr 127.0.0.1:7878 +``` + +For authenticated/TLS modes, use `--user`/`--password` or `--token`, and optionally `--tls-ca-cert`. + +> Note: PostgreSQL wire compatibility is not implemented yet. Existing Postgres drivers/ORMs will not work directly. + +### Quickstart: run a local server now + +Use the runnable example at `examples/server_quickstart.rs`. + +1) Start server: + +```bash +cargo run --example server_quickstart +``` + +2) Connect from another terminal: + +```bash +cargo run --bin lsmdb-cli -- --addr 127.0.0.1:7878 +``` + +The example uses `MvccStore::open_persistent("./data")`, so data is persisted in `./data` across restarts. + +## Project layout + +- `src/` core database code +- `tests/integration/` integration scenarios - `tests/bench/` benchmark entry points -- `tools/lsmdb-cli/` CLI client -- `docs/` architecture and component docs +- `tools/lsmdb-cli/` interactive client +- `tools/lsmdb-admin/` config + diagnostics commands +- `docs/` architecture, SQL subset, testing, release gate -## Testing +## Build and test - library tests: `cargo test --lib --locked` - integration suite gate: `./tools/ci/run_integration_tests.sh` - full test run: `cargo test --locked` -- details: `docs/testing.md` -## Release Gate +## Operations and release readiness - release readiness criteria: `docs/release_gate.md` -- local critical blocker check: `./tools/release/check_critical_blockers.sh /` +- local blocker check: `./tools/release/check_critical_blockers.sh /` - CI release gate workflow: `.github/workflows/release-gate.yml` ## Collaborate -You can collaborate on this repository to help build a production-capable database. -Feature contributions are welcome across storage, transactions, SQL, performance, and tooling. - -## Target Features - -- Durable WAL with crash recovery -- MemTable flush pipeline to SSTables -- Leveled and tiered compaction -- MVCC snapshot isolation -- SQL subset with planning and execution -- Network server and interactive CLI -- Benchmarks and observability +Contributions are welcome across storage, SQL, transactions, server protocol, testing, and operability. diff --git a/examples/server_quickstart.rs b/examples/server_quickstart.rs new file mode 100644 index 0000000..960abb2 --- /dev/null +++ b/examples/server_quickstart.rs @@ -0,0 +1,32 @@ +use std::net::SocketAddr; +use std::sync::Arc; +use std::time::Duration; + +use lsmdb::catalog::Catalog; +use lsmdb::mvcc::MvccStore; +use lsmdb::server::{ServerOptions, start_server_with_options}; + +#[tokio::main] +async fn main() -> Result<(), Box> { + let bind_addr: SocketAddr = "127.0.0.1:7878".parse()?; + + // Persistent MVCC store (creates ./data if it does not exist). + let store = Arc::new(MvccStore::open_persistent("./data")?); + let catalog = Arc::new(Catalog::open((*store).clone())?); + + // Insecure local-dev mode. Do not use this mode for production. + let options = ServerOptions::insecure_for_local_dev(); + let _server = start_server_with_options(bind_addr, catalog, store, options).await?; + + println!("lsmdb server listening on {}", bind_addr); + println!("Connect from another terminal:"); + println!(" cargo run --bin lsmdb-cli -- --addr {}", bind_addr); + println!("Press Ctrl+C to stop."); + + loop { + tokio::time::sleep(Duration::from_secs(3600)).await; + } + + #[allow(unreachable_code)] + Ok(()) +}