Skip to content

Enhance server diagnostics, resource limits, and SQL conformance#54

Merged
ganeshwhere merged 19 commits intomainfrom
staging
Mar 18, 2026
Merged

Enhance server diagnostics, resource limits, and SQL conformance#54
ganeshwhere merged 19 commits intomainfrom
staging

Conversation

@ganeshwhere
Copy link
Copy Markdown
Owner

This pull request introduces significant improvements to release management, testing infrastructure, documentation, and some targeted code quality and correctness updates. The main themes are: establishing a robust release gating process (both automated and manual), enhancing test registration and conformance reporting, expanding documentation for operational and developer workflows, and making minor code correctness fixes.

Release Management and CI/CD Enhancements

  • Introduced a comprehensive release gate workflow (.github/workflows/release-gate.yml) that enforces build, test, integration, and critical issue checks before a release can proceed. This is documented in detail in docs/release_gate.md and supported by a new release sign-off issue template (.github/ISSUE_TEMPLATE/release-signoff.md). [1] [2] [3]
  • Refactored the main CI workflow (.github/workflows/ci.yml) to split quality checks, unit tests, and integration tests into separate jobs, added caching, and ensured coverage on both main and staging branches.

Testing Infrastructure and Documentation

  • Explicitly registered all integration test files in Cargo.toml to prevent silent skips, and documented the integration target policy in docs/testing.md. [1] [2]
  • Added documentation for the SQL conformance suite and its machine-readable compatibility report artifact, improving release and compatibility tracking. [1] [2]

Operational Documentation

  • Added detailed docs for config preflight validation (docs/config_preflight.md) and diagnostics bundle generation (docs/diagnostics_bundle.md), describing commands, output, exit codes, and redaction policies for operational readiness. [1] [2]
  • Updated the main README.md with testing and release gate instructions for quick reference.

MVCC and Catalog Improvements

  • Expanded MVCC documentation to clarify durability modes, commit durability contracts, and crash behavior, reflecting recent improvements in persistent MVCC support. [1] [2] [3]
  • Fixed minor correctness and ergonomics issues in the catalog and executor code, including error message improvements and correct key handling. [1] [2] [3] [4] [5] [6]

Dependency Updates

  • Added tokio-rustls and rustls-pemfile as new dependencies in preparation for future secure transport features.

Release Management and CI/CD

  • Added a release gate GitHub Actions workflow and supporting documentation to enforce production readiness, including both automated and manual sign-off procedures. (.github/workflows/release-gate.yml, docs/release_gate.md, .github/ISSUE_TEMPLATE/release-signoff.md) [1] [2] [3]
  • Refactored CI workflow to split linting, unit tests, and integration tests into separate jobs, improved caching, and expanded branch coverage. (.github/workflows/ci.yml)

Testing Infrastructure

  • Explicitly registered all integration test files in Cargo.toml and documented the policy to prevent silent test skips. (Cargo.toml, docs/testing.md) [1] [2]
  • Added SQL conformance suite documentation and machine-readable compatibility reporting. (docs/sql_conformance.md, docs/sql_subset.md) [1] [2]

Operational Documentation

  • Added documentation for config preflight validation and diagnostics bundle generation, including command usage, output formats, and exit codes. (docs/config_preflight.md, docs/diagnostics_bundle.md) [1] [2]
  • Updated README.md with new testing and release gate instructions. (README.md)

Code Quality and Correctness

  • Improved MVCC and catalog error handling, clarified documentation on durability, and fixed minor bugs in executor and filter logic. (src/catalog/mod.rs, src/executor/delete.rs, src/executor/filter.rs, docs/mvcc.md) [1] [2] [3] [4] [5] [6] [7] [8] [9]

Dependencies

  • Added tokio-rustls and rustls-pemfile to dependencies for future secure transport support. (Cargo.toml)

ganeshwhere and others added 19 commits March 8, 2026 05:14
- add cross-field config validation for memtable sizing and flush timing

- add startup diagnostics model with stable key=value rendering

- document config check usage and expected outputs in docs
- add lsmdb-admin binary with config check and diagnostics bundle commands

- capture redacted config, storage snapshot, optional logs, and CRC checksums

- register the new binary in Cargo and document bundle format/exit codes
- extend request/response protocol with health, readiness, and admin status payloads

- track runtime connection/uptime counters and expose MVCC metrics through admin status

- add CLI meta-commands and integration coverage for new operational endpoints
…s fixes

- add fixture-driven SQL conformance tests and compatibility report generation

- document conformance workflow and link subset coverage from SQL docs

- fix catalog/planner/projection/WAL/MVCC edge cases discovered during suite integration
- add open_persistent/with_storage_engine constructors for persistent MVCC startup

- persist committed MVCC state into StorageEngine and restore commit timeline on restart

- add restart durability tests for MVCC state and SQL/catalog execution path

- document in-memory vs durable MVCC modes in docs
feat(mvcc): add durable store mode backed by storage engine
- formalize durability and acknowledgment points in MVCC lifecycle docs

- add crash-before-ack, rollback, and uncommitted restart recovery tests

- track recovered MVCC keys/versions metrics after durable store restart

- preserve atomic commit visibility by reverting in-memory state on persistence failure
feat(mvcc): harden durable commit and recovery boundaries
- register integration suites under tests/integration as explicit Cargo test targets

- add CI integration gate script that verifies target registration before running cargo test --tests

- split CI into quality, unit, and integration jobs on main and staging

- document reliable local test commands in README and docs/testing.md

- harden compaction/planner integration coverage so hidden regressions surface consistently
- add a dedicated release-gate GitHub workflow for build, unit, integration, and blocker checks

- add a critical high-priority blocker script that fails releases when risk-class issues remain open

- document versioned release readiness criteria and operator-facing gate usage

- add engineering/ops release sign-off issue template for manual approvals

- wire README release-gate entry points for local and CI usage
- remove unused planner imports in runtime code while keeping test-only imports scoped to tests

- drop unreachable CLI loop statement to keep build output warning-free for touched files

- re-validate baseline with cargo check, cargo test --lib, and integration gate
build: finalize green baseline checks
- add config-backed server limits and structured error payloads for busy and resource-limit responses

- reject excess connections, oversized request frames, and multi-statement requests deterministically

- add executor and MVCC guardrails for scan, sort, join, and query-result cardinality

- expose runtime rejection counters in admin status and extend overload integration coverage
feat(server): enforce runtime resource limits and backpressure
…quotas

- add statement deadlines and cooperative cancellation across executor and MVCC scan paths
- expose active statement listing and cancel requests in the server protocol and CLI
- enforce per-identity concurrent query quotas and query result byte limits
- add integration coverage for timeout, cancellation, quota rejection, and transaction cleanup
@ganeshwhere ganeshwhere merged commit e704b97 into main Mar 18, 2026
4 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant