Skip to content

test: add go-test-json baseline support for execution-specs suite - #289

Open
arner wants to merge 1 commit into
hyperledger:mainfrom
arner:baseline-eth
Open

test: add go-test-json baseline support for execution-specs suite#289
arner wants to merge 1 commit into
hyperledger:mainfrom
arner:baseline-eth

Conversation

@arner

@arner arner commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Adds a go test -json parser to cmd/baseline (ParseGoTestJSON) and wires up a new eth-tests-execution-specs suite, mirroring the existing OZ Hardhat mocha-json baseline: known failures tracked per-subtest in testdata/eth_known_failures.json instead of a whole-file skip list.

Trims testdata/execution_specs_tests.skip down to only the one file that panics (confirmed via a real run: an unrecovered panic crashes the whole go test process, unsafe for the per-subtest baseline model to represent) -- the other 9 known-bad files now run and their failures are tracked in the baseline instead.

Wires testdata/execution-specs-tests fixture caching, a real (non-suppressed) baseline-check gate, and per-PR execution into the execution-specs-tests CI job.

Adds a go test -json parser to cmd/baseline (ParseGoTestJSON) and wires up a
new eth-tests-execution-specs suite, mirroring the existing OZ Hardhat
mocha-json baseline: known failures tracked per-subtest in
testdata/eth_known_failures.json instead of a whole-file skip list.

Trims testdata/execution_specs_tests.skip down to only the one file that
panics (confirmed via a real run: an unrecovered panic crashes the whole go
test process, unsafe for the per-subtest baseline model to represent) --
the other 9 known-bad files now run and their failures are tracked in the
baseline instead.

Wires testdata/execution-specs-tests fixture caching, a real (non-suppressed)
baseline-check gate, and per-PR execution into the execution-specs-tests CI
job.

Signed-off-by: Arne Rutjes <arne123@gmail.com>
@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 78.37838% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 25.17%. Comparing base (aaad3b9) to head (b32dbaf).

Files with missing lines Patch % Lines
cmd/baseline/main.go 23.07% 10 Missing ⚠️
cmd/baseline/baseline.go 90.16% 3 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #289      +/-   ##
==========================================
+ Coverage   24.80%   25.17%   +0.37%     
==========================================
  Files          74       74              
  Lines       10736    10803      +67     
==========================================
+ Hits         2663     2720      +57     
- Misses       7842     7849       +7     
- Partials      231      234       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

Copy link
Copy Markdown

✅ OZ Hardhat Compatibility — 84.1% passing (4755/5652)

No regressions, no stale entries.

Full breakdown — 4755 passed, 896 failed, 1 skipped (5652 total, 84.1% passing)

By suite

  • access: 354/578 passing (61%)
  • crosschain: 10/15 passing (67%)
  • finance: 0/2 passing (0%)
  • governance: 77/185 passing (42%)
  • metatx: 0/12 passing (0%)
  • proxy: 194/252 passing (77%)
  • token: 1637/1852 passing (88%)
  • utils: 2483/2755 passing (90%)

Expected failures by cause (896)

  • max-code-size: 212
  • eth_signTypedData_v4: 165
  • execution reverted: 152
  • insufficient-funds: 94
  • fixed-timestamp: 68
  • gas-stub: 36
  • hardhat_impersonateAccount: 29
  • Expected transaction to be reverted with custom error 'AccessManagerNotScheduled', but it reverted with custom error 'AccessManagerUnauthorizedAccount': 27
  • fixed-block-number: 17
  • Expected transaction to be reverted without a reason, but it didn't revert: 9
  • eth_getProof: 6
  • evm_setAutomine: 6
  • personal_sign: 5
  • Expected transaction to be reverted with custom error 'FailedCall', but it reverted with custom error 'TimelockUnexpectedOperationState': 4
  • expected 0 to equal 864000.: 4
  • expected 1000180 to equal 1785339388.: 4
  • expected false to be true: 3
  • hardhat_setStorageAt: 3
  • Error in "RoleGranted" event: Error in the 4th argument assertion: expected 1000000 to equal 1785339216.: 2
  • Error in "RoleGranted" event: Error in the 4th argument assertion: expected 1000000 to equal 1786548816.: 2
  • Expected transaction to be reverted with custom error 'AccessManagerNotScheduled', but it reverted with a different custom error: 2
  • Expected transaction to be reverted with custom error 'FailedCall', but it didn't revert: 2
  • expected 0 to equal 10.: 2
  • expected 1000000 to equal 1785339208.: 2
  • expected 1086400 to equal 1785425629.: 2
  • expected 1786203210 to equal 1786203209.: 2
  • expected 518400 to equal 172800.: 2
  • expected Result(2) [ 10000n, 0n ] to deeply equal [ 0n, 10000n ]: 2
  • expected Result(2) [ 17n, 9983n ] to deeply equal [ 15n, 9985n ]: 2
  • expected Result(2) [ 42n, 9958n ] to deeply equal [ 40n, 9960n ]: 2
  • expected Result(2) [ 5000n, 5000n ] to deeply equal [ 0n, 10000n ]: 2
  • Error in "RoleGranted" event: Error in the 4th argument assertion: expected 1000000 to equal 1785339217.: 1
  • Error in "RoleGranted" event: Error in the 4th argument assertion: expected 1345600 to equal 1785684816.: 1
  • Error in "RoleGranted" event: Error in the 4th argument assertion: expected 1345600 to equal 1785684817.: 1
  • Expected transaction NOT to be reverted: 1
  • Expected transaction to be reverted with custom error 'AccessManagerUnauthorizedCall', but it didn't revert: 1
  • Expected transaction to be reverted with custom error 'TimelockUnexecutedPredecessor', but it reverted with custom error 'TimelockUnexpectedOperationState': 1
  • Expected transaction to be reverted with panic code 0x01 (Assertion error), but it reverted with a custom error: 1
  • expected '0x1685e618cda7b11fca7b27c7bd4d36980ba…' to equal '0xd95ed73cb4091f22c77fc6b8c487f5ff168…': 1
  • expected 0 to equal 36000.: 1
  • expected 0 to equal 950400.: 1
  • expected 1259200 to equal 1785598419.: 1
  • expected 1345600 to equal 1785684816.: 1
  • expected 1345600 to equal 1785684817.: 1
  • expected 1786203210 to equal 0.: 1
  • expected 36000 to equal 4919.: 1
  • expected 4919 to equal 0.: 1
  • expected 9941 to equal 9958.: 1
  • expected Result(2) [ 10000n, 0n ] to deeply equal [ 9953n, 47n ]: 1
  • expected Result(2) [ 10000n, 10000n ] to deeply equal [ 9962n, 10038n ]: 1
  • expected Result(2) [ 10n, 9990n ] to deeply equal [ 0n, 10000n ]: 1
  • expected Result(2) [ 42n, 9958n ] to deeply equal [ 0n, 10000n ]: 1
  • expected [ false, '0' ] to deeply equal [ true, '0' ]: 1
  • failed to revert Store to block 9584: snapshot for block 9584 not found: 1
  • invalid argument 1: json: cannot unmarshal hex string of odd length into Go value of type common.Hash: 1
  • out of gas: 1
  • transaction already pending: 1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant