tests/benchmarks: writing and running cage-rtt benchmarks#830
Open
stupendoussuperpowers wants to merge 4 commits intomainfrom
Open
tests/benchmarks: writing and running cage-rtt benchmarks#830stupendoussuperpowers wants to merge 4 commits intomainfrom
tests/benchmarks: writing and running cage-rtt benchmarks#830stupendoussuperpowers wants to merge 4 commits intomainfrom
Conversation
stupendoussuperpowers
requested review from
JustinCappos,
Yaxuan-w and
rennergade
Contributor
End-to-End Test ReportTest PreviewTest Report Deterministic TestsSummary
Test Results by Category
Fail TestsSummary
Test Results by Category
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Member
|
Do we have some example data with this benchmark suite? |
Contributor
Author
|
It's in the README. Sample outputs: lind@232affd4dc4d:~/lind-wasm$ ./scripts/benchrunner.py fs_read imfs_grate.fs_read.grate
Running: /home/lind/lind-wasm/tests/benchmarks/fs_read.c
Running: /home/lind/lind-wasm/tests/benchmarks/imfs_grate.fs_read.grate
TEST PARAM LINUX (ns) LIND (ns) GRATE (ns) ITERATIONS
---- ----- ---------- ----------- ------------ ----------
Read 1 240 451 (1.879) 1102 (4.592) 1000000
Read 1024 252 461 (1.829) 1102 (4.373) 1000000
Read 4096 267 475 (1.779) 1149 (4.303) 1000000
Read 10240 389 586 (1.506) 1348 (3.465) 1000lind@232affd4dc4d:~/lind-wasm$ ./scripts/benchrunner.py -o result.json fs_read imfs_grate.fs_read.grate && jq . result.json
Running: /home/lind/lind-wasm/tests/benchmarks/fs_read.c
Running: /home/lind/lind-wasm/tests/benchmarks/imfs_grate.fs_read.grate
{
"Read": {
"1": {
"grate": "1043",
"lind": "448",
"linux": "237",
"loops": "1000000"
},
"1024": {
"grate": "1038",
"lind": "457",
"linux": "250",
"loops": "1000000"
},
"4096": {
"grate": "1088",
"lind": "469",
"linux": "265",
"loops": "1000000"
},
"10240": {
"grate": "1273",
"lind": "712",
"linux": "383",
"loops": "1000"
}
}
} |
Contributor
|
I think the grate benchmarks are using the wrong register_handler API post #798 |
rennergade
reviewed
Feb 26, 2026
Contributor
There was a problem hiding this comment.
- collect_tests globs * without filtering by .c suffix — will try to compile README.md, imfs.h, etc. and os._exit(1)
- Bare except: catches KeyboardInterrupt/SystemExit — should be except Exception:
Contributor
There was a problem hiding this comment.
this always reports failure?
Contributor
Author
|
As discussed with @rennergade, instead of having grates duplicates in this benchmarks folder, I'll update the runner to just fetch them from the example grates repo like a quasi-package manager of sorts. |
Contributor
Author
|
Updated this to use the example grates repo to source grates. Added a PR to example grates repo as well to make sure it's set up to be used like that. Also update the test runner script for some better error handling (e.g. skip if compilation/run fails, print debug logs, etc.) |
Contributor
End-to-End Test ReportTest PreviewTest Report Deterministic TestsSummary
Test Results by Category
Fail TestsSummary
Test Results by Category
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
rennergade
reviewed
Feb 26, 2026
| // Child | ||
| if (pid == 0) { | ||
| close(sv[0]); | ||
| char buf[msg_size]; |
Contributor
There was a problem hiding this comment.
char buf[msg_size] is a VLA on the child stack in uds_dgram. For KiB(32) that's 32KB on the stack, which is fine, but in uds_stream (line 75) the parent uses malloc while the child in uds_dgram uses VLA. Inconsistent — the commented-out VLAs (// char buf[msg_size]) suggest this was partially migrated.
rennergade
reviewed
Feb 26, 2026
|
|
||
| // Emits one benchmark row in the format: | ||
| // <test>\t<param>\t<loops>\t<avg_ns> | ||
| inline void emit_result(char *test, int param, long long average, int loops) { |
Contributor
There was a problem hiding this comment.
emit_result is declared inline in the .c file but has external linkage via the header declaration. This can cause linker issues. Remove the inline keyword.
rennergade
reviewed
Feb 26, 2026
rennergade
reviewed
Feb 26, 2026
|
|
||
| def run_lind(wasm_paths, res, platform): | ||
| """Run lind-boot with one or more wasm paths.""" | ||
| cmd = ["sudo", "lind-boot"] + wasm_paths |
Contributor
There was a problem hiding this comment.
probably worth calling lind_run?
rennergade
reviewed
Feb 26, 2026
| return rel.name | ||
|
|
||
|
|
||
| def parse_output(res, output, platform): |
Contributor
There was a problem hiding this comment.
parse_output silently swallows parse errors. param is stored as int key in the dict but JSON serialization will convert it to a string key. When reading back, param keys won't match. Consider using string keys consistently.
rennergade
reviewed
Feb 26, 2026
| run_native(native_path, res) | ||
| elif test.suffix == ".grate": | ||
| print("Running: ", test) | ||
| status = run_grate_test(test.with_suffix(""), res) |
Contributor
There was a problem hiding this comment.
run_grate_test returns None on success (no explicit return), so the if not status check on line 332 will always print "Failed to compile. Skipping." even on success.
rennergade
requested changes
Feb 26, 2026
Contributor
rennergade
left a comment
rennergade
left a comment
There was a problem hiding this comment.
added some more comments.
probably want a .gitignore
Should think about factoring out the imfs code
Contributor
End-to-End Test ReportTest PreviewTest Report Deterministic TestsSummary
Test Results by Category
Fail TestsSummary
Test Results by Category
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Contributor
Author
|
Addressed all the comments. Not sure about the parse_output one, since it does have a try/except block which logs when the output is malformed along with the output. |
rennergade
previously approved these changes
Feb 27, 2026
Contributor
End-to-End Test ReportTest PreviewUnified Test Report grate harness
Cases
wasm harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
Test Results by Category
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
rennergade
previously approved these changes
Mar 4, 2026
- Add scripts/benchrunner.py as the test runner. - Add tests/benchmarks folder with benchmark source files. - Add tests/benchmarks/README.md to add details about the benchmarking test suite. - Address all comments on the PR. - Add a support for test descriptions. - Add support for .csv output.
stupendoussuperpowers
force-pushed
the
regular-benches
branch
from
41f2dd6 to
b7a9c59
Compare
Contributor
End-to-End Test ReportTest PreviewUnified Test Report grate harness
Cases
wasm harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
Test Results by Category
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Yaxuan-w
reviewed
Mar 11, 2026
|
|
||
| ### IMFS Tests | ||
| Example: `imfs_read.c` | ||
| These call IMFS functions directly to estimate in-memory FS overhead |
Member
There was a problem hiding this comment.
Do we need to prepare the imfs_grate wasm binary before running the test? If so, what's the step? ie: we need to pull from example repo manually or..?
I also didn't find imfs_read.c in this PR..
Contributor
Author
There was a problem hiding this comment.
Need to update the readme, we are not doing independent imfs tests right now.
Member
|
I tried to run the runner locally and it works well. I see there's a |
Contributor
Author
All grates are fetched from the lind-wasm-example-grates repo using a git shallow clone, the grates/ folder within the benchmarks folder is equivalent to As previously discussed, this was done so that we don't need to manually make changes to a grate we might want to benchmark in two separate places i.e. once in the example-grates repo and then once in this benchmarks folder. It therefore should not be committed. |
Member
JustinCappos
left a comment
JustinCappos
left a comment
There was a problem hiding this comment.
Need to board a flight. Will try to look more later
Comment on lines
+10
to
+12
| def repo_root() -> Path: | ||
| """Return repo root (scripts/..).""" | ||
| return Path(__file__).resolve().parent.parent |
Member
There was a problem hiding this comment.
Does this do what the comment says? Doesn't it resolve to ../.. from the script itself instead?
Contributor
Author
There was a problem hiding this comment.
Will change the doc string it's a bit confusing.
It's supposed to mean => benchrunner.py is in the scripts/ folder (Path(__file__).parent), the root of the lind-wasm repo is scripts/..
| str(GRATES_REPO_DIR), | ||
| ] | ||
| ) | ||
| run_cmd(["git", "-C", str(GRATES_REPO_DIR), |
Member
There was a problem hiding this comment.
The way in which you checkout things here seems odd to me. Might these be better off in the same repo or with them as cargo packages with explicit dependenciet?
Contributor
Author
There was a problem hiding this comment.
We need to fetch grates from somewhere and right now we use the lind-wasm-example-grates repo as the single source of truth for them. So we need to do quasi-package manager things for this. Ideally I guess we'd have each grate set up as a crates.io entry or as a packaged release on github for C grates, but none of that is set up currently.
Comment on lines
+127
to
+132
| if status: | ||
| existing = [ | ||
| line.strip() | ||
| for line in status.stdout.decode("utf-8").splitlines() | ||
| if line.strip() | ||
| ] |
Member
There was a problem hiding this comment.
I'm not that sure what you are doing here. What is considered existing? Does a directory existing count? Does it need to be bit for bit identical with all files, etc.?
Contributor
Author
There was a problem hiding this comment.
We use sparse checkouts to grab only the grate that was requested, this just check if our git config has already registered this folder for that sparse checkout.
If not, it adds it to the list
Latest changes are pulled regardless, on L140
Comment on lines
+146
to
+147
| add_sparse_path(f"examples/{grate_name}") | ||
| repo_path = GRATES_EXAMPLES_DIR / grate_name |
Member
There was a problem hiding this comment.
Seems like this should be a constant both places, right?
Contributor
Author
There was a problem hiding this comment.
repo_path will be grates/imfs-grate or grates/geteuid-grate etc, depending on which grate is being fetched. The only constant is the base path encoded in GRATES_EXAMPLES_DIR
|
|
||
| #define LOOP_COUNT 1000000 | ||
|
|
||
| int main(int argc, char *argv[]) { |
Member
There was a problem hiding this comment.
There is a lot of duplication across the tests. Would you be better off having them be more modular and included instead of duplicating all of the main, etc. code?
Contributor
Author
There was a problem hiding this comment.
I tried to trim down some of these tests, right now I do not want to over-engineer helpers around these benchmarks so I've tried to keep each benchmark as standalone as possible. Can look into building something more sophisticated as we add more tests.
stupendoussuperpowers
requested review from
JustinCappos,
Yaxuan-w and
rennergade
Contributor
End-to-End Test ReportTest PreviewUnified Test Report grate harness
Cases
wasm harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
Test Results by Category
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The previous data measured and reported in #718 was done for the most part with a hacky and temporary script that needed manual intervention at various points.
This PR adds all the tests used in the Performance Tracker (+ IPC tests) with a runner.
Changes:
tests/benchmarkfolder which contains source code for benchmarking testsbenchrunner.pyto compile, run, and aggregate the results to either print to the terminal or store as ajsonREADME.mddetailing how these tests are run, and how to add new tests