(m)TLS to PG

.claude/rules/naming.md

@@ -0,0 +1,50 @@
+---
+paths:
+  - "**/*.rs"
+  - "**/*.sh"
+  - "**/justfile"
+---
+
+## Names
+
+A reader should understand what a name holds or does without reading its definition. Choose the
+name that a new contributor understands on first reading, even if it is longer. A long name costs
+a few characters, and an unclear name makes every reader look up its definition.
+
+The examples below are illustrative, not taken from a real file, so they stay true when the code
+changes.
+
+## Rules
+
+- **Name what the value means, not its type or its literal value.** From `TIMEOUT`, a reader
+  learns what kind of value it is, but not what it limits. From `CONNECT_TIMEOUT`, a reader learns
+  that it limits a connection attempt.
+- **Name a function for its result or its effect.** A function that returns something is named
+  for what it returns (`pending_requests`, `latest_offset`). A function that changes something
+  starts with a verb (`create_topic`, `remove_member`). Avoid a bare noun such as `topic` or
+  `request` for a function that creates or sends something.
+- **Name a test for the behaviour it requires.** `expired_session_removes_member` states the rule
+  that the test checks. `test_session` or `session_case_2` does not.
+- **Spell words out.** Write `partition`, `message`, `configuration`, not `p`, `msg`, `cfg`. Terms
+  and abbreviations that the Kafka documentation uses are fine: `acks`, `isr`, `api`.
+- **Avoid generic words.** `data`, `info`, `result`, `value`, `item`, `helper`, `tmp`, `first`,
+  `each` say nothing about the content. Add what the thing is: `first_attempt`,
+  `records_per_batch`.
+- **Name a type for what it is.** A type that only holds data is named for that data
+  (`PartitionOffsets` holds a partition's offsets). A type with behaviour is named so the
+  behaviour shows (`TemporaryDirectory` deletes itself when it is dropped).
+- **Don't wrap a type in a new type to carry extra values.** If a function returns a
+  `Connection` and some data about it, return the `Connection` and a type named for the data,
+  such as `(Connection, Handshake)`. A name like `ConnectionWithHandshake` reads as a kind of
+  `Connection`, and hides that it bundles separate values.
+- **Make a boolean read as a statement.** `is_empty`, `port_is_free`, `include_defaults`.
+- **Use one name for one thing.** If the code already has a name for a concept, use that name
+  everywhere, and do not invent a synonym. Do not give two different things the same name.
+- **A short name is fine only in a short scope.** A closure parameter or a loop index that is used
+  on the next line can be `n` or `line`. A value that lives for more than a few lines needs a full
+  name.
+
+## The Test
+
+Read the name alone, without its type and without the code around it. If you need to read the
+definition to understand what it is, rename it.

.claude/rules/smoke-tests.md

@@ -0,0 +1,42 @@
+---
+paths:
+  - "nisshi-smoke-test/**"
+---
+
+## What a smoke test checks
+
+The smoke suite checks that the real Kafka CLI tools work against Nisshi as a user runs them. A
+tool's exit code and printed text are all that the user sees, so a smoke test reads that text. The
+tools write the text for people, though. A test that reads more of it than it needs breaks when a
+Kafka release changes the wording, and it checks the tool instead of the broker.
+
+## Rules
+
+- **Check only the facts that the test's doc comment states.** Check the exit code, and read from
+  the output only the values that the test is about. Leave the rest of the output alone: an empty
+  stdout or a whole printed line is a fact about the tool, not the broker.
+- **Check what the broker controls, not what the client chooses.** The Java client's partitioner
+  decides which partition a key goes to, and the tool decides how to format a line. Check a fact
+  that holds whatever the client chose, such as the total of the partitions' offsets instead of
+  each partition's offset. The broker numbers the records within each partition, so check that
+  numbering: each partition's offsets start at 0 and have no gap or repeat.
+- **Test the broker's logic in `nisshi-broker/tests/it`.** A detailed check of broker behaviour,
+  such as how it stores compressed batches, belongs in the integration tests, which send typed
+  requests and check typed responses. Keep a detailed check in the smoke suite only if no other
+  test can reach that behaviour, such as a process restart, several producers at once, or a topic
+  that is deleted and created again.
+- **Make a reader check the layout that it depends on.** A reader that splits a table at
+  whitespace finds each column by its position. It must compare the table's heading with the
+  heading it expects, and panic with the output if they differ. A new Kafka release then fails
+  with a clear message instead of a wrong value. The reader returns `None` only if the layout
+  matches and the value is not in the table.
+- **Name every string that a reader or a test matches in a tool's output.** Put it in a constant
+  at the top of that tool's module in `src/kafka_cli/`, so the module shows in one place what it
+  expects the tool to print. Put the names of the Kafka exceptions that the tools print in
+  `src/kafka_cli.rs`. Write a command-line argument as a literal where the command is built,
+  because the harness writes it and does not read it.
+- **Give each command that has a reader its own `Output` type.** A `KafkaCli` method whose output
+  a reader reads returns `Output<Marker>`, where `Marker` is an empty enum named after the method,
+  such as `DescribeTopic` for `describe_topic`. The reader is a method of `impl Output<Marker>`.
+  The compiler then rejects a reader called on another command's output. A method whose output no
+  reader reads returns `Output`.

.claude/settings.json

@@ -0,0 +1,16 @@
+{
+  "hooks": {
+    "PostToolUse": [
+      {
+        "matcher": "Write|Edit",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "cargo fmt --all 2>/dev/null || true",
+            "statusMessage": "Formatting with cargo fmt..."
+          }
+        ]
+      }
+    ]
+  }
+}

.config/nextest.toml

@@ -0,0 +1,21 @@
+# CI selects this profile with NEXTEST_PROFILE=ci; local runs keep the
+# default profile. A test still running after 5 minutes is killed rather
+# than holding the runner until the job times out. The slowest test,
+# new_cg::pg::consumer_next_action_32c, takes about 62 s.
+[profile.ci]
+slow-timeout = { period = "60s", terminate-after = 5 }
+
+# The settings of the smoke suite (nisshi-smoke-test). Each Kafka CLI call has
+# its own timeout, and `slow-timeout` stops a test that hangs anywhere else.
+[profile.smoke]
+fail-fast = false
+# Each test starts Kafka CLI JVMs, so memory limits how many tests can run.
+test-threads = 4
+slow-timeout = { period = "60s", terminate-after = 5 }
+
+# S3 runs only the topic lifecycle tests, which check that a broker on S3
+# stores and serves topics and records. Every other engine runs every test that
+# its feature compiles.
+[profile.smoke-s3]
+inherits = "smoke"
+default-filter = "test(/^topic_lifecycle::/) | test(=every_test_file_is_declared)"

.devcontainer/Dockerfile

@@ -13,4 +13,4 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
-FROM rust:1.85
+FROM rust:1.98@sha256:a8a5f0a1e5fe7dfe1d352591e4a1c7dd2c08fd70475cae872cf3458ba0df0546

.devcontainer/devcontainer.json

@@ -1,5 +1,5 @@
 {
-  "name": "tansu",
+  "name": "nisshi",
 
   "dockerFile": "Dockerfile",
 
@@ -15,7 +15,7 @@
     "3000": { "label": "grafana" },
     "5432": { "label": "postgresql" },
     "9090": { "label": "prom" },
-    "9092": { "label": "tansu" }
+    "9092": { "label": "nisshi" }
   },
 
   "features": {

.dockerignore

@@ -1,6 +1,4 @@
-/target
-/work-dir
-/logs
-/.git
-perf.data
-*.log
+# Allowlist: the Dockerfile only packages a prebuilt binary (see Dockerfile).
+*
+!dist/
+!LICENSE

.github/CODEOWNERS

@@ -0,0 +1,3 @@
+# See https://docs.github.com/en/repositories/managing-your-repositories-settings-and-security/customizing-your-repository/about-code-owners
+
+* @shortishly @sgamelin @solace-wkourlas @solace-aross

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,29 @@
+---
+name: Bug report
+about: Report something that isn't working as expected
+title: ""
+labels: bug
+assignees: ""
+---
+
+**Describe the bug**
+A clear description of what's wrong.
+
+**To reproduce**
+Steps to reproduce the behavior, including relevant configuration (storage
+backend, feature flags, `.env` settings).
+
+**Expected behavior**
+What you expected to happen instead.
+
+**Environment**
+- Nisshi version or commit:
+- Storage backend (postgres / libsql / s3 / memory / slatedb):
+- OS:
+
+**Logs**
+Relevant log output, with any sensitive data (credentials, hostnames)
+redacted.
+
+**Additional context**
+Anything else that's relevant.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea or improvement
+title: ""
+labels: enhancement
+assignees: ""
+---
+
+**What problem does this solve?**
+Describe the problem or limitation you're running into.
+
+**Describe the solution you'd like**
+What you'd want to happen.
+
+**Describe alternatives you've considered**
+Any alternative solutions or workarounds you've tried.
+
+**Additional context**
+Anything else that's relevant — related issues, prior art in other Kafka
+implementations, etc.

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,11 @@
+## What changes, and why?
+
+<!-- What a user or operator sees differently, and why. Link an issue if one exists. -->
+
+## Upgrade impact
+
+<!-- What an operator must do when upgrading or rolling back. "None" is fine. -->
+
+## How was this tested?
+
+<!-- Command(s) run, or why testing doesn't apply. -->

.github/dco.yml

@@ -0,0 +1,2 @@
+# CNCF dco2 App config (read from the default branch only): hide the "Set DCO to pass" override button.
+allowOverrideAction: false

.github/dependabot.yml

@@ -1,12 +1,70 @@
 version: 2
+# Each entry sets `commit-message`, so Dependabot titles its PRs
+# `build(deps): bump ...`. pr-title.yml requires a Conventional Commits
+# title, and Dependabot's default `Bump ...` title fails it.
 updates:
+  # The 7-day cooldown gives an upstream CVE or a yanked release time to
+  # surface before Dependabot opens a bump PR.
   - package-ecosystem: "github-actions"
     directory: "/"
     schedule:
-      interval: "daily"
+      interval: "weekly"
     target-branch: "main"
+    commit-message:
+      prefix: "build"
+      include: "scope"
+    cooldown:
+      default-days: 7
   - package-ecosystem: "cargo"
     directory: "/"
     schedule:
-      interval: "daily"
+      interval: "weekly"
     target-branch: "main"
+    commit-message:
+      prefix: "build"
+      include: "scope"
+    ignore:
+      # Pinned to 1.1.0 in Cargo.toml: buoyant_kernel 0.22.2 depends on
+      # "^1", which resolves to 1.2.0 by default, but 1.2.0 renamed the
+      # parse_column_name macro to column_name_segments, breaking the
+      # version deltalake 0.32.4 actually builds against. Dependabot can't
+      # see that constraint and will keep proposing 1.2.x, which fails CI.
+      # Drop this once deltalake bumps its buoyant_kernel dependency.
+      - dependency-name: "buoyant_kernel_derive"
+        versions: [">1.1.0"]
+      # rsasl 2.3.1 (the latest release) requires digest 0.10 for its SCRAM
+      # key derivation, and sha2 0.11 depends on digest 0.11 instead, so the
+      # two cannot both appear as the Sha256/Sha512 types
+      # nisshi-storage::alter_user_scram_credentials passes into
+      # rsasl::mechanisms::scram::tools::derive_keys. Dependabot cannot see
+      # that constraint and will keep proposing 0.11.x, which fails to
+      # build. Drop this once rsasl ships a release built against digest
+      # 0.11.
+      - dependency-name: "sha2"
+        versions: [">0.10.9"]
+    cooldown:
+      default-days: 7
+  # The Dockerfile's alpine base and .devcontainer's rust base are pinned to
+  # exact digests. A digest pin only stays current if something deliberately
+  # bumps it, so both get the same Dependabot coverage the other ecosystems
+  # above already have, rather than freezing in place indefinitely.
+  - package-ecosystem: "docker"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    target-branch: "main"
+    commit-message:
+      prefix: "build"
+      include: "scope"
+    cooldown:
+      default-days: 7
+  - package-ecosystem: "docker"
+    directory: "/.devcontainer"
+    schedule:
+      interval: "weekly"
+    target-branch: "main"
+    commit-message:
+      prefix: "build"
+      include: "scope"
+    cooldown:
+      default-days: 7

.github/scripts/ci-changes.sh

@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+#
+# Decides whether a pull request needs the Rust jobs in ci.yml,
+# dependencies.yml and codeql.yml. Reads the PR's changed paths, one per
+# line, on stdin and writes `rust=true` or `rust=false` to $GITHUB_OUTPUT
+# (stdout when unset).
+#
+# `rust=false` only when every path is on the skip list below: CI config
+# that none of those three workflows reads, and prose. Anything else,
+# including an empty list, means `rust=true`, so an unexpected path runs
+# the full suite rather than skipping it. The three workflows themselves,
+# and the scripts and SQL their jobs run, are excluded from the skip list so
+# a change to them is exercised by the jobs that use them.
+#
+# The `changes` jobs run the PR base's copy of this script, after its
+# ci-changes.test.sh. workflow-lint.yml runs the PR head's test.
+#
+set -euo pipefail
+
+skippable() {
+  case "$1" in
+    .github/workflows/ci.yml | .github/workflows/dependencies.yml | .github/workflows/codeql.yml) return 1 ;;
+    .github/workflows/scripts/* | .github/scripts/*) return 1 ;;
+    .github/*) return 0 ;;
+    docs/*.md) return 0 ;;
+    */*) return 1 ;;
+    *.md | LICENSE | NOTICE) return 0 ;;
+    *) return 1 ;;
+  esac
+}
+
+rust=false
+seen=0
+while IFS= read -r path || [ -n "$path" ]; do
+  [ -n "$path" ] || continue
+  seen=$((seen + 1))
+  if ! skippable "$path"; then
+    rust=true
+    echo "runs Rust jobs: $path"
+    break
+  fi
+done
+[ "$seen" -gt 0 ] || { rust=true; echo "no changed paths listed, running everything"; }
+
+echo "rust=$rust" >> "${GITHUB_OUTPUT:-/dev/stdout}"