test(systemtests): add appside mempool e2e test #580

cloudgray · 2025-09-02T11:16:30Z

Description

Summary

This PR implements comprehensive end-to-end testing for appside mempool functionality, adding robust test infrastructure and extensive test cases to verify mempool behavior across different transaction types and node configurations.

Closes #581

System Tests Directory Structure

clients/ - Multi-client support infrastructure
- Ethereum client integration for EVM transaction handling
- Cosmos client integration for native Cosmos transaction support
mempool/ - Dedicated mempool test suite implementation
- Comprehensive test cases using structured test suite pattern
- Deterministic testing with nonce and baseFee tracking

Test Mechanism

2.1 Persistent Node Architecture

Single node startup for multiple test cases execution
evmd nodes remain running throughout entire test suite
Test suite maintains deterministic state by tracking:
- Account nonces across all test cases
- BaseFee progression for optimal gas pricing

2.2 Common Verification Framework

Action-based test structure: Each test case broken into smaller
verification steps
Inter-test validation: Between every test case, suite performs:
- Verification that expected pending transactions are actually pending
  (or committed)
- Confirmation that expected queued transactions are properly queued
  on target nodes only
Final state verification: After all actions complete, suite waits for
pending transactions to commit

Test Coverage Checklist

Multi-node transaction broadcasting - All test cases execute with transactions broadcast across multiple nodes
Ethereum-only transaction scenarios:
- Queued transactions remain unbroadcast until promotion conditions met
- Pending transactions broadcast immediately (including queue-to-pending promotions)
- Same account transactions sent to different nodes handled correctly
Cosmos-only transaction scenarios - Native Cosmos transaction mempool behavior
Mixed transaction scenarios - Ethereum + Cosmos transactions broadcasted together
Transaction type mixing - LegacyFeeTx and DynamicFeeTx combined scenarios

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

tackled an existing issue or discussed with a team member
left instructions on how to review the changes
targeted the main branch

tests/systemtests/clients/ethclient.go

tests/systemtests/suite/query.go

…s=0stake

aljo242 · 2025-09-10T21:38:39Z

mempool/mempool.go

-	if m.HasEventBus() {
+	if m.eventBus != nil {


why did we make this change? seems out of scope for this

Oh, It seems that when I merge conflict from main branch, something wrong happens.
I'll rollback that changes.

fixed! (d089213)

vladjdk

Overall, lovely test setup! I think this will make it significantly easier to write E2E tests in the future, which is something I applaud you for doing.

I left a few comments, mostly regarding the separation of Pending and Committed as end-states, as transactions get gossiped differently in the consensus phase (commit) vs. the mempool phase (unconfirmed).

vladjdk · 2025-09-12T18:19:29Z

tests/systemtests/clients/types.go

+	Queued  map[string]map[string]*RPCTransaction `json:"queued"`
+}
+
+type RPCTransaction struct {


We have this type here. Let's use that instead of copying.

For this, maybe this PR (#600) should be merged first.

You can see this line (go.mod)

This is temporary fix for resolving Issue #601.
But after temporary fix, when I refer to some cosmos/evm package (to refer to rpctypes.RPCTransaction), I faced build error with dependency problem like below.

# github.com/cosmos/evm/x/vm/types ../../x/vm/types/opcodes_hooks.go:11:5: undefined: vm.OpCodeHooks ../../x/vm/types/config.go:37:15: undefined: vm.ExtendActivators ../../x/vm/types/logs.go:122:23: log.BlockTimestamp undefined (type *"github.com/ethereum/go-ethereum/core/types".Log has no field or method BlockTimestamp) FAIL github.com/cosmos/evm/tests/systemtests [build failed] FAIL github.com/cosmos/evm/tests/systemtests/clients [build failed] FAIL github.com/cosmos/evm/tests/systemtests/mempool [build failed] FAIL github.com/cosmos/evm/tests/systemtests/suite [build failed]

Maybe this is because of temporary replacing cosmos/evm package.

So, we need to merge PR #600 and remove replacement of cosmos/evm pacakges.

vladjdk · 2025-09-12T18:20:00Z

tests/systemtests/clients/types.go

+	PrivKey       *ethsecp256k1.PrivKey
+}
+
+type TxPoolResult struct {


Ditto, same file as the RPCTransaction.

ditto (#580 (comment))

vladjdk · 2025-09-12T18:22:40Z

tests/systemtests/go.mod

 	cosmossdk.io/systemtests v1.3.0
-	github.com/ethereum/go-ethereum v1.15.5
+	github.com/cometbft/cometbft/v2 v2.0.0-rc1


Why are we using V2 and RCs here?

It is because cosmossdk.io/systemtests v1.3.0 is dependent with cometbft v2.0.0-rc-1.

Actually, I tried to downgrade systemtests to v1.2.1 that is dependent with cometbft v0.38.17.

But I got issue.
I remember that some flags currently being used do not work.

vladjdk · 2025-09-12T18:24:10Z

tests/systemtests/mempool/interface.go

+type TestSuite interface {
+	// Test Lifecycle
+	BeforeEach(t *testing.T)
+	AfterEach(t *testing.T)


Suggested change

AfterEach(t *testing.T)

AfterEachCase(t *testing.T)

applied (e566f7a)

vladjdk · 2025-09-12T18:24:21Z

tests/systemtests/mempool/interface.go

+	// Test Lifecycle
+	BeforeEach(t *testing.T)
+	AfterEach(t *testing.T)
+	JustAfterEach(t *testing.T)


Suggested change

JustAfterEach(t *testing.T)

AfterEachAction(t *testing.T)

applied (e566f7a)

vladjdk · 2025-09-12T19:12:05Z

tests/systemtests/mempool/test_replacement.go

+				func(s TestSuite) {
+					tx1, err := s.SendCosmosTx(t, s.Node(0), "acc0", 0, s.BaseFeeX2(), big.NewInt(1))
+					require.NoError(t, err, "failed to send tx")
+					_, err = s.SendEthTx(t, s.Node(1), "acc0", 0, s.BaseFee(), nil)


ditto (#580 (comment))

applied (58e97d8)

vladjdk · 2025-09-12T19:12:48Z

tests/systemtests/mempool/test_replacement.go

+				func(s TestSuite) {
+					_, err := s.SendEthTx(t, s.Node(0), "acc0", 0, s.BaseFee(), nil)
+					require.NoError(t, err, "failed to send tx")
+					tx2, err := s.SendCosmosTx(t, s.Node(1), "acc0", 0, s.BaseFeeX2(), big.NewInt(1))


ditto (#580 (comment))

applied (58e97d8)

vladjdk · 2025-09-12T19:12:55Z

tests/systemtests/mempool/test_replacement.go

+				func(s TestSuite) {
+					tx1, err := s.SendEthTx(t, s.Node(0), "acc0", 0, s.BaseFeeX2(), big.NewInt(1))
+					require.NoError(t, err, "failed to send tx")
+					_, err = s.SendCosmosTx(t, s.Node(1), "acc0", 0, s.BaseFee(), nil)


ditto (#580 (comment))

applied (58e97d8)

vladjdk · 2025-09-12T19:13:58Z

tests/systemtests/mempool/test_replacement.go

+			},
+		},
+		{
+			name: "legacy should always not tx replace dynamic fee tx",


Suggested change

name: "legacy should always not tx replace dynamic fee tx",

name: "legacy should never replace dynamic fee tx",

applied! (58e97d8)

vladjdk · 2025-09-12T19:17:42Z

tests/systemtests/suite/query.go

+}
+
+// CheckPendingOrCommitted checks if the given tx is either pending or committed within the timeout duration
+func (s *SystemTestSuite) CheckPendingOrCommitted(


Let's do CheckPending and CheckCommitted as two separate calls as they function differently in Comet.

Agree!, I'll handle this comment tomorrow!

I just removed checking for committed tx from CheckPendingOrCommitted and made checking operation concurrent logic. (23d361d)

It is because, lets say there are tx1, tx2, tx3, for some test cases, 3 txs are committed in different block.
But for the other cases, tx1 and tx2 can be committed in one block. and tx3 in the other. In this case, if I check pending txs sequentially, we succeed to check tx1 but fail to check tx2 because tx2 can be already committed.

To solve this situation, I modified this verification method as concurrent function.

FYI, we don't have to add CheckCommitted. Checking commit status is already conducted in AfterEachCase. (ref)

tests/systemtests/suite/test_helpers.go

+		go func(tx *TxInfo) { //nolint:gosec // Concurrency is intentional for parallel tx checking
+			defer wg.Done()
+			err := s.CheckTxPending(tx.DstNodeID, tx.TxHash, tx.TxType, time.Second*30)
+			if err != nil {
+				mu.Lock()
+				errors = append(errors, fmt.Errorf("tx %s is not pending or committed: %v", tx.TxHash, err))
+				mu.Unlock()
+			}
+		}(txInfo)


cloudgray added 28 commits August 28, 2025 21:39

test(mempool): setup e2e test suite for mempool testing

6e76883

WIP

8b68d43

test(mempool): refactor test

7ae8b16

test(mempool): refactor system test suite

6700051

test(systemtest): refactor mempool test

3681758

Merge branch 'main' into teest/appside-mempool-e2e

b41028a

Merge branch 'main' into teest/appside-mempool-e2e

d8368f1

test(systemtests): add cosmosClient

87ec45e

test(systemtests): refactor mempool test

51cb60f

test(systemtests): refactor mempool test

2d41eeb

test(systemtests): refactor mempool test

a4d24e2

test(systemtests): add txpool content verification to mempool tests

f552c38

test(systemtests): fix mempool test suite

173541c

test(systemtests): fix mempool test suite

adc5533

test(systemtests): fix mempool tests

6a3b03e

test(systemtests): fix mempool test suite

6d8d240

test(systemtests): fix mempool test suite

14ef6a9

test(systemtests): fix mempool test suite

9fbe143

test(systemtests): add mempool test cases

48de3af

test(systemtests): add comments

e33650a

test(systemtests): chore: enable replacement tests

a4fe745

chore: modify names of test cases

25e65d4

test(systemtests): enhance helper functions

8f5d4a4

test(systemtests): fix mempool test

06fc10a

chore(systemtests): change test name and add readme.md

ac74377

Merge branch 'main' into teest/appside-mempool-e2e

230e422

chore: update CHANGELOG.md

de46b7f

fix(proto): restore removed data field during merge

bf6db75

github-advanced-security bot found potential problems Sep 9, 2025

View reviewed changes

tests/systemtests/clients/ethclient.go Fixed Show fixed Hide fixed

tests/systemtests/clients/ethclient.go Fixed Show fixed Hide fixed

tests/systemtests/suite/query.go Fixed Show fixed Hide fixed

tests/systemtests/suite/query.go Fixed Show fixed Hide fixed

test(systemtests): refactor

2ecece2

cloudgray added 4 commits September 9, 2025 14:41

test(systemtests): fix map iteration

58d1e7d

chore(systemtests): fix build tag

e6085fd

test(systemtests): add test case for mempool

1c55fb9

test(systemtests): fix map iteration

a55ad3d

cloudgray marked this pull request as ready for review September 9, 2025 08:13

cloudgray requested review from a team as code owners September 9, 2025 08:13

cloudgray requested review from vladjdk and aljo242 September 9, 2025 08:13

cloudgray and others added 3 commits September 10, 2025 14:18

test(systemtests): refactor and add test case for --minimum-gas-price…

d139dd7

…s=0stake

test(systemtests): enhance validation

ffbb84d

Merge branch 'main' into teest/appside-mempool-e2e

57a8d10

aljo242 reviewed Sep 10, 2025

View reviewed changes

cloudgray added 4 commits September 11, 2025 13:26

fix: restore removed changes from merge

d089213

fix: restore removed changes from merge

f4fe8af

fix: restore removed changes from merge

88d36bc

fix: restore removed changes from merge

95c6dfd

vladjdk requested changes Sep 12, 2025

View reviewed changes

vladjdk self-assigned this Sep 12, 2025

cloudgray added 5 commits September 15, 2025 21:51

chore(systemtests): rename test suite hooks

e566f7a

fix(systemtests): fix test cases

58e97d8

Merge branch 'main' into teest/appside-mempool-e2e

5d2b2fd

chore: modify mempool e2e test README.md

14a4cf8

test(systemtests): enhance post check of mempool test

23d361d

github-advanced-security bot found potential problems Sep 17, 2025

View reviewed changes

tests/systemtests/suite/test_helpers.go Fixed Show fixed Hide fixed

chore: apply codeQL feedback

11c6152

github-advanced-security bot found potential problems Sep 17, 2025

View reviewed changes

	name: "legacy should always not tx replace dynamic fee tx",
	name: "legacy should never replace dynamic fee tx",

test(systemtests): add appside mempool e2e test #580

Are you sure you want to change the base?

test(systemtests): add appside mempool e2e test #580

Conversation

cloudgray commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Summary

System Tests Directory Structure

Test Mechanism

Test Coverage Checklist

Author Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vladjdk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloudgray Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloudgray Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloudgray Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloudgray commented Sep 2, 2025 •

edited

Loading

cloudgray Sep 15, 2025 •

edited

Loading

cloudgray Sep 15, 2025 •

edited

Loading

cloudgray Sep 17, 2025 •

edited

Loading