Add private eth_call corpus mode to the RPC benchmark workflow

.github/workflows/run-rpc-benchmarks.yml

@@ -71,12 +71,17 @@ on:
         default: ""
       tool_config:
         description: >-
-          Tool-specific config as JSON. flood: {"tests":"eth_call eth_getBalance","rates":"10 100 500","duration":30,"deep_check":false,"label":"","extra_args":""}.
-          ethcallchaos: {"ref":"","corpus_db":"","rate":50,"parallel":8,"duration":300,"leaderboard_top":50,"api_port":5000} (empty ref = pinned commit default).
-          jsonbench: {"ref":"","mode":"","benchmark_config":"","compare_config":"config/compare/defaults.yaml","rps":"","duration":"","vus":"","concurrency":5,"timeout":30,"validate_schema":false,"html_report":true,"fail_on_diff":false,"max_fail_rate_pct":1,"deep_check":false,"extra_args":""}. deep_check (benchmark mode): after the timed load, replay every workload request once and store raw responses as deep-check-<client>.jsonl in the artifact; diff them across single-client runs offline with scripts/rpc-bench/deep-check-compare.py to catch wrong/partial/malformed results that k6 checks (has-a-response only) miss.
-          jsonbench benchmark_config: a curated workload — realistic-mix-head | ethcall-contracts-head | new-state-methods-head (bare name or repo-relative path); empty = a generated read mix. Its client list is rewritten to the node(s) here; rps/vus/duration override the workload when set (else the config's own values apply). Benchmarks report from k6's summary.json (no Prometheus).
-          jsonbench-sweep: {"clients":"nethermind geth reth","rps_list":"100 250 500","snapshot_block":"25490000","state_layout":"flat","benchmark_config":"config/benchmark/ethcallchaos-percategory-validated.yaml","ref":"","duration":"60s","iso_configs":"","iso_duration":"20s"}. Cross-client sweep run SEQUENTIALLY (one node up at a time). clients accepts ctype@image variants (e.g. nethermind@nethermindeth/nethermind:master) for same-client version A/B; iso_configs (space-separated single-scenario configs) runs each scenario isolated, empty = mixed only; benchmark_config is the mixed workload; ref/duration = json-bench commit + mixed load duration (empty ref = pinned default). In sweep mode the top-level single-client inputs (client, reference_client, snapshot_block, state_layout, docker_image) are superseded by these tool_config keys.
-          Leave empty for defaults.
+          Tool config as JSON — empty is fine for defaults. Common recipes:
+          (1) cross-client sweep: {"clients":"nethermind reth","rps_list":"50 100","duration":"60s"} —
+          clients take an optional @image (nethermind@nethermindeth/nethermind:master reth@ghcr.io/paradigmxyz/reth:v2.2.0), run one at a time;
+          (2) PRIVATE corpus sweep (latency + response parity on every eth-call-corpus*.jsonl.gz found on the runner; call contents never reach logs/artifacts — see README "Private eth_call corpus"):
+          {"eth_call_corpus":true,"clients":"nethermind reth","rps_list":"10 100","duration":"120s"} — first client is the parity baseline;
+          (3) single-node jsonbench: {"benchmark_config":"ethcall-contracts-head","rps":"50","duration":"60s"} (add "eth_call_corpus":true to use the default private corpus instead of the config's calls);
+          (4) flood: {"tests":"eth_call eth_getBalance","rates":"10 100 500","duration":30};
+          (5) ethcallchaos: {"rate":50,"parallel":8,"duration":300}.
+          All jsonbench keys: ref/mode/benchmark_config/compare_config/rps/duration/vus/concurrency/timeout/validate_schema/html_report/fail_on_diff/max_fail_rate_pct/deep_check/eth_call_corpus/extra_args.
+          All sweep keys: clients/rps_list/duration/snapshot_block/state_layout/benchmark_config/ref/iso_configs/iso_duration/eth_call_corpus/corpus_dir/corpus_glob (filename filter, e.g. a single corpus file)/corpus_requests (absolute requests per corpus cell, replaces duration)/corpus_passes (requests as a multiple of the corpus record count)/max_corpus_records (raise the 10k parity guard rail).
+          In sweep mode the single-client inputs above (client, reference_client, snapshot_block, state_layout, docker_image) are superseded by tool_config.
         required: false
         default: ""
       node_config:
@@ -128,6 +133,7 @@ jobs:
       health_timeout: ${{ steps.resolve.outputs.health_timeout }}
       node_cpuset: ${{ steps.resolve.outputs.node_cpuset }}
       node_memory: ${{ steps.resolve.outputs.node_memory }}
+      eth_call_corpus: ${{ steps.resolve.outputs.eth_call_corpus }}
       tool_config: ${{ steps.resolve.outputs.tool_config }}
     steps:
       - name: Resolve configuration
@@ -209,6 +215,23 @@ jobs:
             echo "tool_config is not valid JSON: ${tool_config}"
             exit 1
           fi
+          if ! echo "${tool_config}" | jq -e 'type == "object"' >/dev/null 2>&1; then
+            echo "tool_config must be a JSON object."
+            exit 1
+          fi
+          # Private eth_call corpus mode (jsonbench/jsonbench-sweep benchmark only): call contents
+          # stay on the runner; the artifact carries aggregate summaries and parity counts only.
+          eth_call_corpus="$(echo "${tool_config}" | jq -r 'if .eth_call_corpus == true then "true" else "false" end')"
+          if [[ "${eth_call_corpus}" == "true" ]]; then
+            case "${benchmark_tool}" in
+              jsonbench|jsonbench-sweep) ;;
+              *) echo "eth_call_corpus requires benchmark_tool=jsonbench or jsonbench-sweep."; exit 1;;
+            esac
+            if [[ "${comparison}" == "true" ]]; then
+              echo "eth_call_corpus is benchmark-only; leave reference_client=none (a corpus sweep compares its clients itself)."
+              exit 1
+            fi
+          fi
           [[ -z "${node_config}" ]] && node_config="{}"
           if ! echo "${node_config}" | jq -e . >/dev/null 2>&1; then
             echo "node_config is not valid JSON: ${node_config}"
@@ -280,6 +303,8 @@ jobs:
           network="$(nc '.network')";               [[ -z "${network}" ]] && network="mainnet"
           jsonrpc_modules="$(nc '.jsonrpc_modules')"
           [[ -z "${jsonrpc_modules}" ]] && jsonrpc_modules="Eth,Subscribe,Trace,TxPool,Web3,Proof,Net,Parity,Health,Rpc,Debug"
+          # Corpus runs only ever issue eth_call — serve the minimal module set.
+          [[ "${eth_call_corpus}" == "true" ]] && jsonrpc_modules="Eth"
           health_min="$(nc '.health_timeout_minutes')"; [[ -z "${health_min}" ]] && health_min="30"
           node_cpuset="$(nc '.cpuset')"
           node_memory="$(nc '.memory')"
@@ -401,6 +426,7 @@ jobs:
             echo "health_timeout=${health_timeout}"
             echo "node_cpuset=${node_cpuset}"
             echo "node_memory=${node_memory}"
+            echo "eth_call_corpus=${eth_call_corpus}"
             echo "tool_config<<TOOLCFG"
             echo "${tool_config}"
             echo "TOOLCFG"
@@ -506,6 +532,8 @@ jobs:
           LAYOUT_FLAGS: ${{ needs.resolve.outputs.layout_flags }}
           ADDITIONAL_FLAGS: ${{ needs.resolve.outputs.additional_flags }}
           HEALTH_TIMEOUT: ${{ needs.resolve.outputs.health_timeout }}
+          # Same raised cap as corpus sweep cells, so both dispatch modes agree (empty = script default).
+          RPC_GAS_CAP: ${{ needs.resolve.outputs.eth_call_corpus == 'true' && '1000000000000' || '' }}
           DOTTRACE: ${{ needs.resolve.outputs.dottrace }}
           NODE_CPUSET: ${{ needs.resolve.outputs.node_cpuset }}
           NODE_MEMORY: ${{ needs.resolve.outputs.node_memory }}
@@ -590,6 +618,7 @@ jobs:
           jb_to="$(get '.timeout')";                     [[ -n "${jb_to}" ]] && export JB_TIMEOUT="${jb_to}"
           export JB_VALIDATE_SCHEMA="$(getb '.validate_schema')"
           export JB_DEEP_CHECK="$(getb '.deep_check')"
+          export JB_ETH_CALL_CORPUS="$(getb '.eth_call_corpus')"
           jb_html="$(getb '.html_report')";              [[ -n "${jb_html}" ]] && export JB_HTML_REPORT="${jb_html}"
           export JB_FAIL_ON_DIFF="$(getb '.fail_on_diff')"
           # getb: '// empty' would swallow an explicit 0 (jq treats it as truthy-false).
@@ -649,6 +678,11 @@ jobs:
           set -euo pipefail
           cfg="${TOOL_CONFIG}"; [[ -z "${cfg}" ]] && cfg='{}'
           get() { echo "${cfg}" | jq -r "$1 // empty"; }
+          # For booleans '// empty' would swallow an explicit false — test null instead.
+          getb() { echo "${cfg}" | jq -r "$1 | if . == null then \"\" else tostring end"; }
+          export JB_ETH_CALL_CORPUS="$(getb '.eth_call_corpus')"
+          corpus_dir="$(get '.corpus_dir')";               [[ -n "${corpus_dir}" ]] && export CORPUS_DIR="${corpus_dir}"
+          corpus_glob="$(get '.corpus_glob')";             [[ -n "${corpus_glob}" ]] && export CORPUS_GLOB="${corpus_glob}"
           export CLIENTS="$(get '.clients')";              [[ -z "${CLIENTS}" ]] && export CLIENTS="nethermind geth reth"
           export RPS_LIST="$(get '.rps_list')";            [[ -z "${RPS_LIST}" ]] && export RPS_LIST="100 250 500"
           export SNAPSHOT_BLOCK="$(get '.snapshot_block')";[[ -z "${SNAPSHOT_BLOCK}" ]] && export SNAPSHOT_BLOCK="25490000"
@@ -658,6 +692,11 @@ jobs:
           export JB_DURATION="$(get '.duration')";         [[ -z "${JB_DURATION}" ]] && export JB_DURATION="60s"
           export ISO_CONFIGS="$(get '.iso_configs')"
           export ISO_DURATION="$(get '.iso_duration')";    [[ -z "${ISO_DURATION}" ]] && export ISO_DURATION="20s"
+          # Size corpus cells by request count rather than wall time (rate is still rps_list).
+          corpus_requests="$(get '.corpus_requests')"; [[ -n "${corpus_requests}" ]] && export CORPUS_REQUESTS="${corpus_requests}"
+          corpus_passes="$(get '.corpus_passes')";     [[ -n "${corpus_passes}" ]] && export CORPUS_PASSES="${corpus_passes}"
+          # Parity holds every record in memory; raising this is a deliberate act, not a default.
+          max_records="$(get '.max_corpus_records')"; [[ -n "${max_records}" ]] && export RPC_BENCH_MAX_CORPUS_RECORDS="${max_records}"
           export STATE_ROOT="${STATE_DIR}/sweep"
           ./scripts/rpc-bench/run-rpc-sweep.sh
 
@@ -687,6 +726,8 @@ jobs:
         env:
           CLIENT: ${{ needs.resolve.outputs.client }}
           DOTTRACE: ${{ needs.resolve.outputs.dottrace }}
+          # Corpus runs print match COUNTS only — log lines could quote private call data.
+          ETH_CALL_CORPUS: ${{ needs.resolve.outputs.eth_call_corpus }}
         shell: bash
         run: |
           set -euo pipefail
@@ -711,21 +752,21 @@ jobs:
           exception_found="false"
           if [[ -s "${exc_matches}" ]]; then
             exception_found="true"
-            echo "::warning::Exception lines detected in node log (first 40):"
-            head -n 40 "${exc_matches}" || true
+            echo "::warning::Exception lines detected in node log ($(wc -l < "${exc_matches}" | tr -d ' ') line(s))"
+            [[ "${ETH_CALL_CORPUS}" != "true" ]] && { head -n 40 "${exc_matches}" || true; }
           fi
 
           invalid_block_found="false"
           if grep -qEi "invalid[[:space:]_-]*block" "${clean}"; then
             invalid_block_found="true"
-            echo "::warning::Invalid block lines detected in node log (first 40):"
-            grep -Ein "invalid[[:space:]_-]*block" "${clean}" | head -n 40 || true
+            echo "::warning::Invalid block lines detected in node log"
+            [[ "${ETH_CALL_CORPUS}" != "true" ]] && { grep -Ein "invalid[[:space:]_-]*block" "${clean}" | head -n 40 || true; }
           fi
 
           for pattern in "Unhandled" "Fatal" "ERROR"; do
             if grep -qi "${pattern}" "${clean}"; then
-              echo "::warning::Severe log pattern '${pattern}' detected in node log (first 10):"
-              grep -in "${pattern}" "${clean}" | head -n 10 || true
+              echo "::warning::Severe log pattern '${pattern}' detected in node log ($(grep -ci "${pattern}" "${clean}") line(s))"
+              [[ "${ETH_CALL_CORPUS}" != "true" ]] && { grep -in "${pattern}" "${clean}" | head -n 10 || true; }
             fi
           done
 
@@ -777,7 +818,7 @@ jobs:
           done
 
       - name: Upload benchmark results
-        if: always()
+        if: always() && needs.resolve.outputs.eth_call_corpus != 'true'
         uses: actions/upload-artifact@v7
         with:
           name: rpcbench-results-${{ needs.resolve.outputs.benchmark_tool }}
@@ -793,6 +834,24 @@ jobs:
           retention-days: 30
           if-no-files-found: warn
 
+      # Corpus runs publish an allowlist-staged artifact instead: sanitized aggregate summaries,
+      # counts-only parity reports, and our own generated markdown — never node logs, raw tool
+      # output, or per-call files.
+      - name: Stage private corpus results
+        id: stage-corpus-results
+        if: always() && needs.resolve.outputs.eth_call_corpus == 'true'
+        shell: bash
+        run: python3 scripts/rpc-bench/corpus_results.py stage "${OUT_DIR}" "${RUNNER_TEMP}/rpcbench-corpus-results"
+
+      - name: Upload private corpus results
+        if: always() && needs.resolve.outputs.eth_call_corpus == 'true' && steps.stage-corpus-results.outcome == 'success'
+        uses: actions/upload-artifact@v7
+        with:
+          name: rpcbench-results-${{ needs.resolve.outputs.benchmark_tool }}
+          path: ${{ runner.temp }}/rpcbench-corpus-results/
+          retention-days: 30
+          if-no-files-found: error
+
       - name: Upload dotTrace snapshot
         if: always() && needs.resolve.outputs.dottrace == 'true'
         uses: actions/upload-artifact@v7

scripts/rpc-bench/README.md

@@ -143,7 +143,7 @@ the workflow's defensive-cleanup step).
 
 | Input | Meaning |
 |---|---|
-| `benchmark_tool` | `flood`, `ethcallchaos`, or `jsonbench`. |
+| `benchmark_tool` | `flood`, `ethcallchaos`, `jsonbench`, or `jsonbench-sweep`. |
 | `client` | `nethermind` (default), `geth`, or `reth` — the node under test. |
 | `reference_client` | `none` (default) or a client to compare against (see comparison mode). |
 | `snapshot_block` | Same-block snapshot set tag (`/mnt/sda/<client>-<tag>`); empty = expb snapshot (nethermind) / `25490000` (geth/reth, comparisons). |
@@ -278,6 +278,78 @@ the corpus DB is how you constrain the workload. Corpus resolution order:
 the `corpus-v1` release asset of `kamilchodola/EthCallChaos`) → a DB committed
 in the tool repo → fresh evolution from scratch.
 
+## Private `eth_call` corpus (`tool_config.eth_call_corpus: true`)
+
+For call sets that must not appear in GitHub logs or artifacts (e.g. shared by a
+third party): the corpus lives only on the runner, and runs publish **aggregate
+numbers and parity counts only**. This is a logging/artifact boundary, not a
+defense against the runner itself — anything executing on the VM (trusted
+images, this repo's scripts) can read the corpus there.
+
+The boundary covers call *contents*, not the **filename**: everything after the
+`eth-call-corpus-` prefix becomes the scenario label, which appears in the step
+summary, the parity table, artifact paths, and `summaries.manifest`. That is
+deliberate — scenarios have to be told apart — so name files by workload shape,
+never after anything sensitive.
+
+Two operational limits worth knowing before capturing: `corpus_parity.py`
+enforces `MAX_CORPUS_RECORDS = 10_000`, and the k6 fixture scales with record
+count (~142 MB for 497 records, since eth_call records with state overrides run
+to hundreds of KB each). Large captures need sampling down to a representative
+subset, not a raised cap.
+
+**Corpus files** (JSON Lines, one `{"method":"eth_call","params":[...]}` per
+line, extra fields ignored, optionally gzipped) go to the runner at
+`/mnt/sda/expb-data/rpc-bench/eth-call-corpus[-<label>].jsonl.gz`. A
+`jsonbench-sweep` with `eth_call_corpus:true` discovers **every**
+`eth-call-corpus*.jsonl.gz` there and runs each as its own scenario;
+single-node `jsonbench` uses the default `eth-call-corpus.jsonl.gz` only.
+`corpus_dir` (sweep tool_config) overrides the directory.
+
+**Sizing a cell by request count.** By default a corpus cell runs for `duration` at each
+`rps_list` rate. `corpus_requests` (absolute) or `corpus_passes` (a multiple of that
+corpus's record count) instead size the cell by how many requests it should issue: the
+rate is unchanged and the length is derived as `ceil(count / rps)`, since k6's
+constant-arrival-rate executor holds the rate. `corpus_passes: 5` on a 50k corpus at
+`rps_list: "500"` is 250,000 requests over 500s. Note this is *draws with replacement*,
+not a guarantee every record is visited — coverage is `N x (1 - (1 - 1/N)^requests)`.
+
+**What a corpus sweep does per client:** one k6 latency cell per corpus per
+`rps_list` entry (the corpus replaces the workload's `calls:`; rendered as a
+JSON-array fixture because json-bench's JSONL reader caps lines at ~64 KiB),
+then one full-corpus replay via `corpus_parity.py` while the node is still up.
+The **first client in `clients` is the parity baseline**; every later client's
+responses are compared byte-for-byte against it, and any defect or mismatch
+fails the job. Calls the baseline client rejects with a JSON-RPC error are
+recorded as error outcomes (captured corpora legitimately contain calls that
+fail at the pinned head, e.g. explicit `gasPrice` with an underfunded sender);
+both clients rejecting a call counts as agreement (`both_rpc_errors`), a
+one-sided rejection as divergence. Corpus cells raise start-node's uniform
+`RPC_GAS_CAP` from 1e9 to 1e12 so the corpus's explicit multi-billion `gas`
+fields are not clamped into artificial failures. Clients, images (`ctype@image`), rates, and duration are all
+free-form — pick rates the node can sustain, and mind that latency numbers from
+a cold node at low rps are indicative, not steady-state.
+
+**How contents stay off GitHub:** the json-bench container's output goes to a
+VM-scratch file instead of the job log; per-call k6 outputs, deep-check, and
+HTML reports are disabled or left in scratch; the published `summary.json` is
+rewritten by `corpus_results.py sanitize` to a fixed numeric schema; parity
+reports contain counters and client labels only; node logs are scanned for the
+usual Exception / invalid-block / shutdown gates but print **counts only** and
+are deleted (sweep) or excluded from upload; the artifact is assembled by
+`corpus_results.py stage`, which copies nothing but validated `summary.json`,
+`parity.json`, and generated markdown. Failures print category + counts (e.g.
+`rpc_error=3`), never request or response bytes — raw detail stays on the
+runner in `<scratch>/jsonbench/` for SSH diagnosis until the next run wipes it.
+
+Example — 4-way private comparison, 3 rates, both corpora, one dispatch:
+
+```json
+{"eth_call_corpus": true,
+ "clients": "nethermind@nethermindeth/nethermind:master nethermind@nethermindeth/nethermind:some-pr-branch reth@ghcr.io/paradigmxyz/reth:v2.2.0 reth@ghcr.io/paradigmxyz/reth:latest",
+ "rps_list": "1 10 100", "duration": "120s"}
+```
+
 ## dotTrace flow (goal #3)
 
 `dottrace=true` (requires `client=nethermind`) uses the same mechanism as expb's
@@ -334,5 +406,8 @@ The `reproducible-benchmarks` self-hosted runner must provide:
 | `stop-node.sh` | Graceful stop → collect logs + dotTrace → **verify snapshot unchanged** → tear down (per instance via `NODE_ENV_FILE`). |
 | `run-flood.sh` | Install flood + Vegeta, run the selected tests (load or `--equality`), report. |
 | `run-ethcallchaos.sh` | Clone/build/run EthCallChaos in an SDK container, scrape its API. |
+| `corpus_parity.py` | Private corpus replay: capture a baseline client's responses (VM-local), diff later clients against it, emit counts-only reports. |
+| `corpus_results.py` | Sanitize k6 summaries to a fixed numeric schema and stage only validated aggregate files for the corpus artifact. |
+| `prepare-eth-call-corpus.py` | Convert a JSONL(.gz) corpus into the JSON-array fixture json-bench consumes. |
 | `run-jsonbench.sh` | Clone/build json-bench's runner image, adapt the workload config to the node(s), run `benchmark` (summary.json metrics, no Prometheus) or `compare`, report. |
 | `cleanup.sh` | Guarded defensive cleanup (stale containers, leftover mounts, scratch). |