Add reproducible benchmark baselines and CI checks

[ajevans99]

Summary

Supersedes #171 with a clean implementation based on current main.

• Adds JSONSchemaBenchmarks for schema construction, validation, and all four output levels across Poll, OpenAPI-fragment, and draft 2020-12 workloads.
• Adds checksum-pinned, opt-in fetching for the canada, citm_catalog, and twitter reference corpus while preserving offline local benchmarks.
• Commits the complete 66-file Ubuntu p90 baseline set and verifies coverage before any regression check, preventing package-benchmark from silently accepting partial baselines.
• Adds an Ubuntu workflow that reports wall-clock, CPU, and malloc tables, enforces deterministic malloc thresholds, writes job summaries, and uploads baseline artifacts.
• Uses filesystem-safe benchmark identifiers so package-benchmark can load the thresholds it writes.
• Uses the supported thresholds check operation and handles SwiftPM's improvement marker without accepting regressions or other plugin failures.
• Extends formatting coverage to the isolated benchmark package.

Timing calibration

Static wall-clock and CPU thresholds are informational on GitHub's shared hosted runners. Repeated runs of the same commit exceeded the initial 10% tolerance, with timing deviations reaching 73%, while allocation counts remained stable. CI therefore reports timing for trend analysis and gates malloc counts with the configured 10% relative plus 10-allocation absolute tolerance. Suspected timing regressions should be reproduced on a dedicated runner.

The final enforcement workflow passed three consecutive runs on ubuntu-24.04 (run 32418739341, attempts 1-3), including both benchmark targets and the fetched reference corpus.

Validation

• swift test
• make format
• Both benchmark targets list successfully with 66 total cases.
• OrderedJSON and JSONSchema benchmark smoke runs complete.
• Baseline coverage detects all missing files and accepts the committed complete 66-file set.
• Threshold wrapper rejects allocation regressions and accepts only package-benchmark's explicit improvement marker.
• Metric selection was verified against a deliberately impossible timing baseline while malloc thresholds remained unchanged.

Tracking

• Supersedes Add benchmark baselines and CI checks #171
• Infrastructure for Benchmark and optimize OrderedJSON parser/serializer #162
• Infrastructure for Benchmark and optimize JSONSchema validation hot paths #164

[github-actions]

✅ No public API breaking changes introduced

Compared this PR's head against origin/main using
swift package diagnose-api-breaking-changes. No breakages
detected in JSONSchema, JSONSchemaBuilder, or
JSONSchemaConversion.

_{This check shows only changes this PR introduces. Cumulative
breakages since the last release tag are reviewed at release time.}

[Copilot]

Pull request overview

Adds reproducible OrderedJSON and JSONSchema benchmarks, pinned optional corpora, committed p90 baselines, and CI enforcement.

Changes:

• Adds schema construction, validation, output, and serialization workloads.
• Adds checksum-pinned corpus fetching and baseline coverage checks.
• Adds Ubuntu CI reporting, threshold checks, artifacts, and formatting coverage.

Reviewed changes

Copilot reviewed 83 out of 84 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Summary
Makefile	Formats benchmark sources.
Benchmarks/Scripts/fetch-reference-corpus.sh	Fetches checksum-pinned reference corpora.
Benchmarks/Scripts/check-thresholds.sh	Runs benchmark threshold checks.
Benchmarks/Scripts/check-baseline-coverage.sh	Moderate unresolved issue (3 votes): listing failures can be masked by process substitution, allowing incomplete baselines to pass.
Benchmarks/README.md	Documents benchmark usage and CI baselines.
Benchmarks/Package.swift	Adds the JSONSchema benchmark target.
Benchmarks/Package.resolved	Updates dependency resolution metadata.
Benchmarks/OrderedJSONBenchmarks/OrderedJSONBenchmarks.swift	Adds workloads, thresholds, and optional corpus loading.
Benchmarks/JSONSchemaBenchmarks/Resources/poll.schema.json	Adds Poll schema fixture.
Benchmarks/JSONSchemaBenchmarks/Resources/poll.instance.json	Adds Poll instance fixture.
Benchmarks/JSONSchemaBenchmarks/Resources/openapi-fragment.schema.json	Adds OpenAPI schema fixture.
Benchmarks/JSONSchemaBenchmarks/Resources/openapi-fragment.instance.json	Adds OpenAPI instance fixture.
Benchmarks/JSONSchemaBenchmarks/Resources/draft2020-12-schema.schema.json	Adds draft 2020-12 schema fixture.
Benchmarks/JSONSchemaBenchmarks/Resources/draft2020-12-schema.instance.json	Adds draft 2020-12 instance fixture.
Benchmarks/JSONSchemaBenchmarks/JSONSchemaBenchmarks.swift	Moderate unresolved issue (4 votes): unresolved meta-schema references make draft 2020-12 validation and output baselines measure invalid-reference short circuits.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.wide-object.OrderedJSON.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.wide-object.JSONEncoder.sortedKeys.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.users-array.OrderedJSON.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.users-array.JSONEncoder.sortedKeys.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.twitter.OrderedJSON.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.twitter.JSONEncoder.sortedKeys.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.small.OrderedJSON.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.small.JSONEncoder.sortedKeys.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.poll-instance.OrderedJSON.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.poll-instance.JSONEncoder.sortedKeys.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.deep-array.OrderedJSON.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.deep-array.JSONEncoder.sortedKeys.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.citm_catalog.OrderedJSON.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.citm_catalog.JSONEncoder.sortedKeys.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.canada.OrderedJSON.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.serialize.canada.JSONEncoder.sortedKeys.p90.json	Adds serialization p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.roundtrip.wide-object.OrderedJSON.p90.json	Adds round-trip p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.roundtrip.users-array.OrderedJSON.p90.json	Adds round-trip p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.roundtrip.twitter.OrderedJSON.p90.json	Adds round-trip p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.roundtrip.small.OrderedJSON.p90.json	Adds round-trip p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.roundtrip.poll-instance.OrderedJSON.p90.json	Adds round-trip p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.roundtrip.deep-array.OrderedJSON.p90.json	Adds round-trip p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.roundtrip.citm_catalog.OrderedJSON.p90.json	Adds round-trip p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.roundtrip.canada.OrderedJSON.p90.json	Adds round-trip p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.wide-object.OrderedJSON.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.wide-object.JSONSerialization.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.wide-object.JSONDecoder.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.users-array.OrderedJSON.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.users-array.JSONSerialization.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.users-array.JSONDecoder.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.twitter.OrderedJSON.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.twitter.JSONSerialization.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.twitter.JSONDecoder.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.small.OrderedJSON.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.small.JSONSerialization.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.small.JSONDecoder.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.poll-instance.OrderedJSON.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.poll-instance.JSONSerialization.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.poll-instance.JSONDecoder.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.deep-array.OrderedJSON.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.deep-array.JSONSerialization.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.deep-array.JSONDecoder.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.citm_catalog.OrderedJSON.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.citm_catalog.JSONSerialization.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.citm_catalog.JSONDecoder.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.canada.OrderedJSON.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.canada.JSONSerialization.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/OrderedJSONBenchmarks.parse.canada.JSONDecoder.p90.json	Adds parsing p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.validate.poll.Schema.validate.p90.json	Adds validation p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.validate.openapi-fragment.Schema.validate.p90.json	Adds validation p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.validate.draft2020-12-schema.Schema.validate.p90.json	Adds validation p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.output.poll.verbose.p90.json	Adds output p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.output.poll.flag.p90.json	Adds output p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.output.poll.detailed.p90.json	Adds output p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.output.poll.basic.p90.json	Adds output p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.output.openapi-fragment.verbose.p90.json	Adds output p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.output.openapi-fragment.flag.p90.json	Adds output p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.output.openapi-fragment.detailed.p90.json	Adds output p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.output.openapi-fragment.basic.p90.json	Adds output p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.output.draft2020-12-schema.verbose.p90.json	Adds output p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.output.draft2020-12-schema.flag.p90.json	Adds output p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.output.draft2020-12-schema.detailed.p90.json	Adds output p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.output.draft2020-12-schema.basic.p90.json	Adds output p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.construct.poll.Schema.init.p90.json	Adds construction p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.construct.openapi-fragment.Schema.init.p90.json	Adds construction p90 baseline.
Benchmarks/Baselines/JSONSchemaBenchmarks.construct.draft2020-12-schema.Schema.init.p90.json	Adds construction p90 baseline.
.gitignore	Ignores benchmark artifacts and fetched corpora.
.github/workflows/format.yml	Extends formatting coverage.
.github/workflows/benchmarks.yml	Adds Ubuntu benchmark CI, reports, and artifacts.

Suppressed comments (2)

Makefile:23

• -not -path only filters which paths are printed; find still descends into Benchmarks/.build. After any benchmark build, make format therefore scans the entire SwiftPM build/dependency tree on every invocation. Prune that directory with -path 'Benchmarks/.build' -prune -o before the name filters.
  Makefile:17
• -not -path only filters which paths are printed; find still descends into Benchmarks/.build. After any benchmark build, make format therefore scans the entire SwiftPM build/dependency tree on every invocation. Prune that directory with -path 'Benchmarks/.build' -prune -o before the name filters.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.