Gather all data from benchmarks #10224
Draft
+339
−355
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Akirathan
added
the
CI: No changelog needed
Akirathan
changed the title
Emits JS sources in the target directory. Debugger friendly.
|
Since there are no artifacts for Oracle GraalVM benchmark jobs, I had to parse the data from job outputs. 70cb785 adds all these results that were gathered via |
Frizi
approved these changes
Jun 13, 2024
5 tasks
mergify bot
pushed a commit
that referenced
this pull request
Feb 13, 2025
- Related to recent concerns about Table performance, we don't measure many kinds of operations. - This PR tries to address that by adding benchmarks for: - `Column.cast` operation with a few (not exhaustive) combinations of source/target types - unary operations: - `not` on `Boolean` column (which is supposed to be O(1)), - `text_length` on text columns (including a fixed length column that _could_ in the future be specialized to be O(1)), - `hour`/`year` extracting values from date/time columns. - Also adds a few more statistics to store in the XML report generated from benchmarks to allow for easier comparison. - These will be superseded by #10224 if it gets finished.
This was referenced Feb 17, 2025
This reverts commit 70cb785.
|
Remove the json files (results) in 326312c. They are not useful - they are in the old schema and their label names are incorrect. Moreover, we will not use enterprise benchmark results anytime soon anyway. |
Let's stick with the already working Python project for now.
|
The experimental JS project |
# Conflicts: # build.sbt # lib/java/benchmarks-common/src/main/java/org/enso/interpreter/bench/BenchmarkResultProcessor.java # lib/java/benchmarks-common/src/main/java/org/enso/interpreter/bench/BenchmarksRunner.java # lib/java/benchmarks-common/src/main/java/org/enso/interpreter/bench/ReportItem.java
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #9355
Pull Request Description
Gather benchmarking data in a new, JSON format. Previously, we gathered only one
scorevalue per result. Now, everything that is provided by JMH framework is gathered.The produced
bench-results.jsoncan be simply read locally. An example from few dummy runs is:{ "schema" : "results_schema.json", "results" : [ { "label" : "org.enso.interpreter.bench.benchmarks.semantic.VectorBenchmarks.averageOverVector", "timestamp" : "2024-06-10T12:33:54.026187718", "score" : 0.0018004264683645476, "scoreError" : "NaN", "samples" : 1, "min" : 0.0018004264683645476, "max" : 0.0018004264683645476, "mean" : 0.0018004264683645476, "stdDev" : "NaN", "warmupIterations" : 1, "warmupMillis" : 1000, "measureIterations" : 1, "measureMillis" : 1000, "commitId" : "e3322694913312d01f70c8af6b8b06ed78cfa002", "branch" : "wip/akirathan/9355-bench-res-json", "configuration" : { "osName" : "Linux", "osArch" : "amd64", "osVersion" : "6.5.0-35-generic", "vmName" : "OpenJDK 64-Bit Server VM", "vmVersion" : "21.0.2+13-jvmci-23.1-b30", "vmVendor" : "GraalVM Community", "jdkVersion" : "21.0.2" } }, { "label" : "org.enso.interpreter.bench.benchmarks.semantic.VectorBenchmarks.averageOverArray", "timestamp" : "2024-06-10T12:34:44.508761575", "score" : 0.0016180807887320298, "scoreError" : "NaN", "samples" : 1, "min" : 0.0016180807887320298, "max" : 0.0016180807887320298, "mean" : 0.0016180807887320298, "stdDev" : "NaN", "warmupIterations" : 1, "warmupMillis" : 1000, "measureIterations" : 1, "measureMillis" : 1000, "commitId" : "e3322694913312d01f70c8af6b8b06ed78cfa002", "branch" : "wip/akirathan/9355-bench-res-json", "configuration" : { "osName" : "Linux", "osArch" : "amd64", "osVersion" : "6.5.0-35-generic", "vmName" : "OpenJDK 64-Bit Server VM", "vmVersion" : "21.0.2+13-jvmci-23.1-b30", "vmVendor" : "GraalVM Community", "jdkVersion" : "21.0.2" } }, { "label" : "org.enso.interpreter.bench.benchmarks.semantic.VectorBenchmarks.averageOverArrayProxy", "timestamp" : "2024-06-10T12:34:44.53703994", "score" : 0.002464619691933143, "scoreError" : "NaN", "samples" : 1, "min" : 0.002464619691933143, "max" : 0.002464619691933143, "mean" : 0.002464619691933143, "stdDev" : "NaN", "warmupIterations" : 1, "warmupMillis" : 1000, "measureIterations" : 1, "measureMillis" : 1000, "commitId" : "e3322694913312d01f70c8af6b8b06ed78cfa002", "branch" : "wip/akirathan/9355-bench-res-json", "configuration" : { "osName" : "Linux", "osArch" : "amd64", "osVersion" : "6.5.0-35-generic", "vmName" : "OpenJDK 64-Bit Server VM", "vmVersion" : "21.0.2+13-jvmci-23.1-b30", "vmVendor" : "GraalVM Community", "jdkVersion" : "21.0.2" } }, { "label" : "org.enso.interpreter.bench.benchmarks.semantic.VectorBenchmarks.averageOverArrayProxyNew", "timestamp" : "2024-06-10T12:34:44.546709051", "score" : 0.0020222642668386607, "scoreError" : "NaN", "samples" : 1, "min" : 0.0020222642668386607, "max" : 0.0020222642668386607, "mean" : 0.0020222642668386607, "stdDev" : "NaN", "warmupIterations" : 1, "warmupMillis" : 1000, "measureIterations" : 1, "measureMillis" : 1000, "commitId" : "e3322694913312d01f70c8af6b8b06ed78cfa002", "branch" : "wip/akirathan/9355-bench-res-json", "configuration" : { "osName" : "Linux", "osArch" : "amd64", "osVersion" : "6.5.0-35-generic", "vmName" : "OpenJDK 64-Bit Server VM", "vmVersion" : "21.0.2+13-jvmci-23.1-b30", "vmVendor" : "GraalVM Community", "jdkVersion" : "21.0.2" } } ] }Important Notes
Checklist
Please ensure that the following checklist has been satisfied before submitting the PR:
Scala,
Java,
TypeScript,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.