Skip to content

Conversation

@LouisTsai-Csie
Copy link
Collaborator

@LouisTsai-Csie LouisTsai-Csie commented Nov 26, 2025

🗒️ Description

  • Supported Command: fill &execute

  • Modes:

    • -m benchmark (under compute/ folder)
    • -m stateful (under stateful/ folder)
  • Flags

    • --gas-benchmark-values (specify the block gas limit)
    • --fixed-opcode-count (specify the opcode count)
  • Options

    • -m repricing (run only the subset of benchmark tests for quick testing)

Issue Description
When running benchmark tests with the --fixed-opcode-count flag, the current behavior incorrectly restricts execution to only the benchmark cases marked with the repricing marker.

However, repricing is meant to be an independent option that limits execution to a subset of the benchmark suite.
Using --fixed-opcode-count should not imply repricing mode.

Fix Applied in This PR
This PR adjusts the logic so that:

  • Running with --fixed-opcode-count executes the full benchmark suite: Four passes are executed (including contract_balance with values 0 and 1):
fill -v tests/benchmark/compute/instruction/test_account_query.py::test_selfbalance --fixed-opcode-count 10 --clean -m benchmark
  • Running with -m repricing executes only the repricing subset: Two passes are executed (only contract_balance = 0)
fill -v tests/benchmark/compute/instruction/test_account_query.py::test_selfbalance --fixed-opcode-count 10 --clean -m repricing

Add a sanity check CI workflow for fixed opcode count feature.

🔗 Related Issues or PRs

None

✅ Checklist

  • All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
    uvx tox -e static
  • All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
  • All: Considered adding an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • All: Set appropriate labels for the changes (only maintainers can apply labels).
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.
  • Tests: For PRs implementing a missed test case, update the post-mortem document to add an entry the list.
  • Ported Tests: All converted JSON/YML tests from ethereum/tests or tests/static have been assigned @ported_from marker.

Cute Animal Picture

Put a link to a cute animal picture inside the parenthesis-->

@LouisTsai-Csie LouisTsai-Csie self-assigned this Nov 26, 2025
@LouisTsai-Csie LouisTsai-Csie added C-refactor Category: refactor A-test-benchmark Area: Tests Benchmarks—Performance measurement (eg. `tests/benchmark/*`, `p/t/s/e/benchmark/*`) labels Nov 26, 2025
@LouisTsai-Csie LouisTsai-Csie marked this pull request as ready for review November 27, 2025 05:07
Comment on lines +253 to +257
if self.fixed_opcode_count is not None and self.code_generator is None:
pytest.skip(
"Cannot run fixed opcode count tests without a code generator"
)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, the fixed opcode count feature is supported only by benchmark tests that use the code generator. If a test does not support this feature, it should be skipped.

There are two possible ways to filter such tests:

We could skip them during collection via pytest_collection_modifyitems, but this is more complex because it requires determining whether each test is a benchmark test and whether it uses the code generator.

Therefore, I chose to skip the unsupported tests directly in benchmark test wrapper instead.

@LouisTsai-Csie LouisTsai-Csie force-pushed the refactor-repricing-marker branch from 7945e85 to 79e7a7b Compare November 27, 2025 05:14
@LouisTsai-Csie LouisTsai-Csie force-pushed the refactor-repricing-marker branch from 2110a2c to b54883f Compare November 28, 2025 06:26
@codecov
Copy link

codecov bot commented Nov 28, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.08%. Comparing base (f23e4ab) to head (b54883f).
⚠️ Report is 2 commits behind head on forks/osaka.

Additional details and impacted files
@@             Coverage Diff              @@
##           forks/osaka    #1810   +/-   ##
============================================
  Coverage        86.08%   86.08%           
============================================
  Files              743      743           
  Lines            44076    44076           
  Branches          3891     3891           
============================================
  Hits             37941    37941           
  Misses            5657     5657           
  Partials           478      478           
Flag Coverage Δ
unittests 86.08% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@LouisTsai-Csie LouisTsai-Csie marked this pull request as draft December 1, 2025 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-test-benchmark Area: Tests Benchmarks—Performance measurement (eg. `tests/benchmark/*`, `p/t/s/e/benchmark/*`) C-refactor Category: refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant