Skip to content

Add namespace-filtered delivery, block DB retention, and USDC demo orchestration - #185

Open
kushnireyal wants to merge 2 commits into
mainfrom
evm-gateway-demo
Open

Add namespace-filtered delivery, block DB retention, and USDC demo orchestration#185
kushnireyal wants to merge 2 commits into
mainfrom
evm-gateway-demo

Conversation

@kushnireyal

@kushnireyal kushnireyal commented May 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Completes the EVM gateway changes for the USDC multi-replica performance demo,
on top of #190 (notification-based tx completion).

  • Namespace-filtered delivery: each gateway replica skips blocks from other
    namespaces, preventing O(1/N) throughput degradation as replica count grows.
  • Block DB retention: TruncateBlocks prunes SQLite to a configurable window
    (default 10k blocks), called every 1k delivered blocks to keep storage bounded
    during long replay runs.
  • SQLite scratch-image fix: PRAGMA temp_store=MEMORY so WAL operations work
    in the release image built from scratch.
  • Demo orchestration (scripts/run-demo.sh): single command from a developer's
    Mac to sync the Ansible collection on the staging control node, generate testdata,
    and stream the workload in real time; exposed via make run-demo. New
    --reset-fabricx flag does a full backend teardown + start + cert refresh +
    redeploy for reproducible measurements on a clean chain.
  • USDC deployment runbook (integration/perf/USDC_deployment.md): staging
    topology, testdata steps, run commands, output format, troubleshooting, and a
    "Resetting between runs" guidance section.
  • Perf test extensions: configurable worker counts via env vars, TPS stability
    stats, latency percentiles, sent/committed/inflight counters.

@kushnireyal kushnireyal changed the title Add namespace-filtered delivery, block DB retention, and USDC demo orchestration [WIP] Add namespace-filtered delivery, block DB retention, and USDC demo orchestration May 25, 2026
@kushnireyal
kushnireyal force-pushed the evm-gateway-demo branch 2 times, most recently from 6e19fa8 to eeffae7 Compare May 25, 2026 09:42
@kushnireyal
kushnireyal requested a review from ale-linux May 25, 2026 10:49
@kushnireyal
kushnireyal marked this pull request as ready for review May 25, 2026 10:49
@kushnireyal kushnireyal changed the title [WIP] Add namespace-filtered delivery, block DB retention, and USDC demo orchestration Add namespace-filtered delivery, block DB retention, and USDC demo orchestration May 25, 2026
@kushnireyal
kushnireyal force-pushed the evm-gateway-demo branch 13 times, most recently from 3ed44f4 to 8484281 Compare May 26, 2026 06:59
@kushnireyal
kushnireyal removed the request for review from ale-linux May 26, 2026 13:04
@kushnireyal
kushnireyal marked this pull request as draft May 26, 2026 13:04
@kushnireyal
kushnireyal changed the base branch from main to chase May 26, 2026 16:46
@kushnireyal
kushnireyal force-pushed the evm-gateway-demo branch 7 times, most recently from f3d362b to 209b0d1 Compare May 28, 2026 06:46
kushnireyal added a commit that referenced this pull request May 30, 2026
@kushnireyal
evm-gateway-demo: clean port from chase + SDK baebbe88 + PR #185 revi…
a70dcbd 
…ew items

SDK upgrade (relative to chase):
- Bump fabric-x-sdk to baebbe88 (persistent streams + native ArMA Send→Recv ordering)
- Add context.Context arg to NewSubmitter calls (breaking API change)
- Drop custom ArMASubmitter: SDK now handles ordering correctly
- Switch perf test to NewFabricXTestHarnessWithFactory (fabric-x staging)
- Add FABX_CONFIG_PATH env support (Ansible sets per-replica path)
- Add PERF_PROCESSING_WORKERS / PERF_SUBMITTING_WORKERS env vars

Ported from PR #185:
- Dockerfile: mkdir /data && chmod 777 /data (busybox has no /data)
- endorser/versioned_db_wrapper.go: NewSnapshot(0) resolves to latest committed
  block (was reading genesis state for "latest")
- gateway/api/models.go: gasPrice nil fix for EIP-1559 receipts (was breaking
  ethers.js and other clients)
- integration/test_helpers.go: waitUntilSynced in NewFabricXTestHarnessWithFactory
  (60s timeout) so tests don't start before gateway is synced
- integration/perf/replay_json_dataset_test.go: require.NoError for harness setup
  (fail fast instead of letting test panic later in unrelated code)

Code review items addressed:
- replay_json_dataset_test.go: switch polling from TransactionByHash to
  TransactionReceipt; check receipt.Status (1=success, 0=EVM revert) for
  correct failure accounting
- gateway/app/app.go: default SyncTimeout 5m → 60s
- gateway/storage/store.go + gateway/core/chain.go: parameterize TruncateBlocks
  frequency via Store.BlockTruncationInterval (was hardcoded 1000)

Tooling:
- scripts/run-demo.sh: add --submitting-workers, --processing-workers flags;
  quiet-mode uses ssh -T instead of -tt (no PTY CR/LF corruption) and surfaces
  section headers, replica progress, and demo result lines
kushnireyal added a commit that referenced this pull request May 30, 2026
@kushnireyal
evm-gateway-demo: clean port from chase + SDK baebbe88 + PR #185 revi…
9ffc382 
…ew items

SDK upgrade (relative to chase):
- Bump fabric-x-sdk to baebbe88 (persistent streams + native ArMA Send→Recv ordering)
- Add context.Context arg to NewSubmitter calls (breaking API change)
- Drop custom ArMASubmitter: SDK now handles ordering correctly
- Switch perf test to NewFabricXTestHarnessWithFactory (fabric-x staging)
- Add FABX_CONFIG_PATH env support (Ansible sets per-replica path)
- Add PERF_PROCESSING_WORKERS / PERF_SUBMITTING_WORKERS env vars

Ported from PR #185:
- Dockerfile: mkdir /data && chmod 777 /data (busybox has no /data)
- endorser/versioned_db_wrapper.go: NewSnapshot(0) resolves to latest committed
  block (was reading genesis state for "latest")
- gateway/api/models.go: gasPrice nil fix for EIP-1559 receipts
- integration/test_helpers.go: waitUntilSynced in NewFabricXTestHarnessWithFactory
  (60s timeout) so tests don't start before gateway is synced
- integration/perf/replay_json_dataset_test.go: require.NoError for harness setup
- endorser/testimpl/balance_priming_statedb.go: SetSender now actually sets
  senderAddr and balanceSlot; GetState only intercepts the specific balance slot
  (was intercepting ALL contract storage and returning primeValue, which made
  _paused/_blacklisted/etc. read as non-zero and revert every transfer)
- endorser/testimpl/balance_priming_executor.go: recover real sender from tx
  and pass to SetSender (was passing zero address); convert vm.ErrExecutionReverted
  into ExecutionResult{Status:201} so reverts commit and the test can detect them

Code review items addressed:
- replay_json_dataset_test.go: switch polling from TransactionByHash to
  TransactionReceipt; check receipt.Status (1=success, 0=EVM revert) for
  correct failure accounting — this also exposed the priming bug above
- gateway/app/app.go: default SyncTimeout 5m → 60s
- gateway/storage/store.go + gateway/core/chain.go: parameterize TruncateBlocks
  frequency via Store.BlockTruncationInterval (was hardcoded 1000)

Tooling:
- scripts/run-demo.sh: add --submitting-workers, --processing-workers flags;
  quiet-mode uses ssh -T instead of -tt (no PTY CR/LF corruption) and surfaces
  section headers, replica progress, and demo result lines

Docs:
- integration/perf/USDC_deployment.md: trim stale troubleshooting entries and
  duplicate command examples; collapse output-format section to essentials
kushnireyal added a commit that referenced this pull request May 30, 2026
@kushnireyal
evm-gateway-demo: clean port from chase + SDK baebbe88 + PR #185 revi…
710e1f9 
…ew items

SDK upgrade (relative to chase):
- Bump fabric-x-sdk to baebbe88 (persistent streams + native ArMA Send→Recv ordering)
- Add context.Context arg to NewSubmitter calls (breaking API change)
- Drop custom ArMASubmitter: SDK now handles ordering correctly
- Switch perf test to NewFabricXTestHarnessWithFactory (fabric-x staging)
- Add FABX_CONFIG_PATH env support (Ansible sets per-replica path)
- Add PERF_PROCESSING_WORKERS / PERF_SUBMITTING_WORKERS env vars

Ported from PR #185:
- Dockerfile: mkdir /data && chmod 777 /data (busybox has no /data)
- endorser/versioned_db_wrapper.go: NewSnapshot(0) resolves to latest committed
  block (was reading genesis state for "latest")
- gateway/api/models.go: gasPrice nil fix for EIP-1559 receipts
- integration/test_helpers.go: waitUntilSynced in NewFabricXTestHarnessWithFactory
  (60s timeout) so tests don't start before gateway is synced
- integration/perf/replay_json_dataset_test.go: require.NoError for harness setup

Code review items addressed:
- gateway/app/app.go: default SyncTimeout 5m → 60s
- gateway/storage/store.go + gateway/core/chain.go: parameterize TruncateBlocks
  frequency via Store.BlockTruncationInterval (was hardcoded 1000)

Tooling:
- scripts/run-demo.sh: add --submitting-workers, --processing-workers flags;
  quiet-mode uses ssh -T instead of -tt (no PTY CR/LF corruption) and surfaces
  section headers, replica progress, and demo result lines

Docs:
- integration/perf/USDC_deployment.md: trim stale troubleshooting entries and
  duplicate command examples; collapse output-format section to essentials

Not ported (blocked):
- TransactionReceipt + receipt.Status check in perf test polling. Switching to
  receipt.Status reveals that every USDC transfer reverts at EVM level — both
  with chase's intercept-all priming and with the PR #185 targeted-priming fix.
  This is pre-existing (chase's TransactionByHash polling silently ignored
  status). Reverted to chase behavior for now; needs separate investigation
  of why USDC transfers revert. Details in review-pr-185.md.
kushnireyal added a commit that referenced this pull request May 30, 2026
@kushnireyal
evm-gateway-demo: clean port from chase + SDK baebbe88 + PR #185 revi…
0c9f49b 
…ew items

SDK upgrade (relative to chase):
- Bump fabric-x-sdk to baebbe88 (persistent streams + native ArMA Send→Recv ordering)
- Add context.Context arg to NewSubmitter calls (breaking API change)
- Drop custom ArMASubmitter: SDK now handles ordering correctly
- Switch perf test to NewFabricXTestHarnessWithFactory (fabric-x staging)
- Add FABX_CONFIG_PATH env support (Ansible sets per-replica path)
- Add PERF_PROCESSING_WORKERS / PERF_SUBMITTING_WORKERS env vars

Ported from PR #185:
- Dockerfile: mkdir /data && chmod 777 /data (busybox has no /data)
- endorser/versioned_db_wrapper.go: NewSnapshot(0) resolves to latest committed
  block (was reading genesis state for "latest")
- gateway/api/models.go: gasPrice nil fix for EIP-1559 receipts
- integration/test_helpers.go: waitUntilSynced in NewFabricXTestHarnessWithFactory
  (60s timeout) so tests don't start before gateway is synced
- integration/perf/replay_json_dataset_test.go: require.NoError for harness setup

Code review items addressed:
- gateway/app/app.go: default SyncTimeout 5m → 60s
- gateway/storage/store.go + gateway/core/chain.go: parameterize TruncateBlocks
  frequency via Store.BlockTruncationInterval (was hardcoded 1000)

Tooling:
- scripts/run-demo.sh: add --submitting-workers, --processing-workers flags;
  quiet-mode uses ssh -T instead of -tt (no PTY CR/LF corruption) and surfaces
  section headers, replica progress, and demo result lines

Docs:
- integration/perf/USDC_deployment.md: trim stale troubleshooting entries and
  duplicate command examples; collapse output-format section to essentials

Not ported (blocked):
- TransactionReceipt + receipt.Status check in perf test polling. Switching to
  receipt.Status reveals that every USDC transfer reverts at EVM level — both
  with chase's intercept-all priming and with the PR #185 targeted-priming fix.
  This is pre-existing (chase's TransactionByHash polling silently ignored
  status). Reverted to chase behavior for now; needs separate investigation
  of why USDC transfers revert. Details in review-pr-185.md.
@kushnireyal
kushnireyal force-pushed the evm-gateway-demo branch 6 times, most recently from 04b772a to 17c1170 Compare June 1, 2026 08:09
@ale-linux
ale-linux force-pushed the chase branch 3 times, most recently from 581d206 to ee82baa Compare June 1, 2026 10:25
Base automatically changed from chase to main June 1, 2026 10:43
@kushnireyal
kushnireyal force-pushed the evm-gateway-demo branch 6 times, most recently from 3d9571a to 0dd5e22 Compare June 3, 2026 09:33
@ale-linux
Notification-based transaction completion system
2efed97 
This commit introduces a notification-based architecture for tracking
transaction completion as an alternative to the traditional block-based
synchronization approach. The new system uses a pending transaction cache and
notification dispatcher to provide lower latency transaction tracking,
particularly beneficial for performance testing scenarios.

The Gateway now uses a BatchSubmitter for asynchronous endorsement submission
with optional notification support, and the integration test infrastructure
supports both traditional synchronization and notification-based modes. The
replay performance test demonstrates the new notification system with a
TxCompletionTracker for monitoring transaction completion, and includes a
hybrid mode that uses the synchronizer for initial catchup before switching to
notifications for ongoing updates.

This architecture maintains backward compatibility while enabling more
efficient transaction processing patterns for high-throughput scenarios.

Signed-off-by: Alessandro Sorniotti <aso@zurich.ibm.com>
@kushnireyal
kushnireyal force-pushed the evm-gateway-demo branch 2 times, most recently from 709ea35 to 5b2ea17 Compare June 4, 2026 07:07
@kushnireyal
evm-gateway-demo: multi-replica demo infra, namespace filtering, bloc…
d9e4444 
…k DB retention

- Add namespace filtering to Chain (NewChain gains namespace param)
- Add SQLite PRAGMA temp_store=MEMORY to fix crash on scratch images
- Add TruncateBlocks to storage.Store with configurable retention window
- Wire cfg.Network.Namespace and cfg.Gateway.SyncTimeout through app.go
- Add run-demo.sh Ansible-based demo runner script
- Add USDC_deployment.md multi-replica deployment guide
- Extend replay_json_dataset_test.go with wrap-around support
- Add chain_test.go namespace-filter tests and store_test.go truncation tests

Signed-off-by: Eyal Kushnir <eyal.kushnir@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ale-linux ale-linux ale-linux requested changes

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kushnireyal @ale-linux