feat: add performance-testing package by JoshuaKGoldberg · Pull Request #3024 · flint-fyi/flint

JoshuaKGoldberg · 2026-06-24T11:51:40Z

PR Checklist

Addresses an existing open issue: fixes ⚡️ Performance: Create a comparison generator setup #2627
That issue was marked as status: accepting prs
Steps in CONTRIBUTING.md were taken

Overview

Adds a new packages/performance-testing that includes a basic harness for generating different cases of (files to lint) x (rules to enable).

Measurements

Hyperfine 1.20.0 on an Apple Silicon Mac, Node 24.13.1, against ESLint 10.8.0 and typescript-eslint 8.66.0:

┌───────────────────────┬───────┬───────────────────────┬───────┐
│ eslint                │ files │ flint                 │ rules │
├───────────────────────┼───────┼───────────────────────┼───────┤
│ '649.1 ms ± 165.5 ms' │ 2     │ '843.0 ms ±  31.9 ms' │ 1     │
│ '807.2 ms ±  41.2 ms' │ 2     │ '855.9 ms ±  31.3 ms' │ 119   │
│ '802.3 ms ±  25.3 ms' │ 2     │ '826.9 ms ±  32.9 ms' │ 272   │
│ '954.6 ms ±  37.1 ms' │ 256   │ '1.030 s ±  0.043 s'  │ 1     │
│ '1.264 s ±  0.043 s'  │ 256   │ '1.182 s ±  0.029 s'  │ 119   │
│ '1.408 s ±  0.035 s'  │ 256   │ '1.292 s ±  0.032 s'  │ 272   │
│ '2.092 s ±  0.039 s'  │ 1024  │ '1.929 s ±  0.056 s'  │ 1     │
│ '2.678 s ±  0.063 s'  │ 1024  │ '2.414 s ±  0.048 s'  │ 119   │
│ '3.215 s ±  0.045 s'  │ 1024  │ '2.959 s ±  0.041 s'  │ 272   │
└───────────────────────┴───────┴───────────────────────┴───────┘

tl;dr: we're slower than ESLint for a small number of files and rules, then quickly get progressively faster at bigger scales of files and/or rules.

❤️‍🔥

netlify · 2026-06-24T11:51:45Z

✅ Deploy Preview for flint-fyi ready!

Name	Link
🔨 Latest commit	`6aaf8f1`
🔍 Latest deploy log	https://app.netlify.com/projects/flint-fyi/deploys/6a8a4c02b0bef60008ba25ad
😎 Deploy Preview	https://deploy-preview-3024--flint-fyi.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

changeset-bot · 2026-06-24T11:51:46Z

⚠️ No Changeset found

Latest commit: 6aaf8f1

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

lishaduck · 2026-06-25T01:21:16Z

JoshuaKGoldberg · 2026-06-27T20:31:34Z

┌───────┬───────┬───────────────────────┬──────────────────────┐
│ files │ rules │ eslint                │ flint                │
├───────┼───────┼───────────────────────┼──────────────────────┤
│ 2     │ 1     │ '771.6 ms ±  39.8 ms' │ '1.727 s ±  0.031 s' │
│ 2     │ 119   │ '852.0 ms ±   7.1 ms' │ '2.202 s ±  0.058 s' │
│ 2     │ 274   │ '876.8 ms ±   6.2 ms' │ '2.213 s ±  0.021 s' │
│ 256   │ 1     │ '1.095 s ±  0.013 s'  │ '1.966 s ±  0.037 s' │
│ 256   │ 119   │ '1.412 s ±  0.056 s'  │ '2.572 s ±  0.024 s' │
│ 256   │ 274   │ '1.551 s ±  0.009 s'  │ '2.775 s ±  0.010 s' │
└───────┴───────┴───────────────────────┴──────────────────────┘

Something is definitely wrong on the Flint side 😂, we're not ~1/2 the speed of ESLint. Nooope.

lishaduck · 2026-06-27T22:58:44Z

Something is definitely wrong on the Flint side 😂, we're not ~1/2 the speed of ESLint. Nooope.

You sure?

(closer look pins ESLint@1:19 and Flint@1:41 which feels reasonabler)

JoshuaKGoldberg · 2026-06-28T01:10:32Z

The perf comparisons should be running equivalent rules. I'm wondering if it's because Flint rules are so much more type-heavy, it's not a fair comparison?

lishaduck · 2026-06-28T01:39:52Z

I'm wondering if it's because Flint rules are so much more type-heavy, it's not a fair comparison?

What if we just compare some not-type-aware lints Flint<->core ESLint and some type-aware lints typescript-eslint<->Flint so it's comparable? Flint'll still lose against raw ESLint probably, not hopefully not TSESLint.

lishaduck · 2026-06-28T01:40:49Z

I'm hoping we can get a roughly unbiased comparison per-axis like vlt: https://github.com/vltpkg/benchmarks

Resolves conflicts from the comparisons -> rule-data package rename.

Pairs each Flint rule with exactly one ESLint rule instead of whichever mapping happened to sort last, so rule counts match on both sides. Points both configs at src/**/*.ts with the same ignores, declares the plugins the generated ESLint configs import, and runs Flint with --cache-ignore so Hyperfine's repeat runs do not measure a warm cache against a cold one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

codecov · 2026-08-19T04:29:20Z

Codecov Report

❌ Patch coverage is 94.87179% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...sting/src/creators/files/createESLintConfigFile.ts	90.00%	1 Missing and 1 partial ⚠️
...es/performance-testing/src/creators/files/rules.ts	92.00%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

Adds the return types and module ordering ESLint wants, marks the package private so the package-json rules stop asking for exports, swaps two Array constructors for Array.from, and disables performance/loopAwaits where the sequencing is deliberate. Knip treats the package the way it already treats e2e, since its dependencies are consumed by the ESLint configs it writes into generated cases rather than by its own source. Also records measurements in the README and enables the 1024-file cases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

JoshuaKGoldberg · 2026-08-19T13:54:38Z

aaaand we're faster again. The performance measurements were very flawed. Added to the PR description. Phew!

Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-01a01a5f-14c9-7385-a5b9-f93a5dc2f150

michaelfaith · 2026-08-19T22:16:23Z

✅ All modified and coverable lines are covered by tests.

We seem to have a gap in our test coverage measuring. There are no tests in this PR, yet codecov thinks everything's covered...

michaelfaith

This is really cool. Nice work. I'm particularly interested in building on this to include more plugins. For instance, i'd love to be able to use this to measure the perf impact of moving the markdown language to use the Sätteri parser.

Just a bit of feedback from my side. Also it'd be nice to add some unit test coverage.

Co-authored-by: michael faith <michaelfaith@users.noreply.github.com>

michaelfaith

Looks good! 👍 Just a bunch of string -> function describe titles (though these could be skipped and done when we formally adopt the vitest rule (I thought we already had...)), and the failing unit tests (due to cross platform path seps)

Co-authored-by: michael faith <michaelfaith@users.noreply.github.com>

JoshuaKGoldberg · 2026-08-23T01:17:23Z

What if we just compare some not-type-aware lints Flint<->core ESLint and some type-aware lints typescript-eslint<->Flint so it's comparable? Flint'll still lose against raw ESLint probably, not hopefully not TSESLint.

I'm happy to report we're now winning overall! Whew. But it would be interesting to maybe add this as a separate issue, I think - in case we get curious?

[WIP] feat: add performance comparisons package

c407a90

vercel Bot deployed to Preview June 24, 2026 11:54 View deployment

JoshuaKGoldberg mentioned this pull request Jun 27, 2026

🚀 Feature: Add type aware-ness to getStaticValue #3050

Open

2 tasks

JoshuaKGoldberg added 2 commits June 27, 2026 16:24

Merge branch 'main'

7b24685

--skip-formatting

6153d4c

vercel Bot deployed to Preview June 27, 2026 20:53 View deployment

JoshuaKGoldberg and others added 2 commits August 18, 2026 19:42

Merge branch 'main' into perf-comparisons

d1963eb

Resolves conflicts from the comparisons -> rule-data package rename.

JoshuaKGoldberg changed the title ~~[WIP] feat: add performance-testing package~~ feat: add performance-testing package Aug 19, 2026

JoshuaKGoldberg commented Aug 19, 2026

View reviewed changes

Comment thread packages/performance-testing/src/creators/files/createESLintConfigFile.ts Outdated

JoshuaKGoldberg commented Aug 19, 2026

View reviewed changes

Comment thread packages/performance-testing/README.md Outdated

JoshuaKGoldberg commented Aug 19, 2026

View reviewed changes

Comment thread rules.txt Outdated

chore: apply performance testing review feedback

51b7e2e

Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-01a01a5f-14c9-7385-a5b9-f93a5dc2f150

JoshuaKGoldberg force-pushed the perf-comparisons branch from 333c913 to 51b7e2e Compare August 19, 2026 14:14

JoshuaKGoldberg marked this pull request as ready for review August 19, 2026 14:18

This was referenced Aug 19, 2026

perf(core): validate out-of-lint-set cache dependencies by timestamp #3253

Open

perf(core): walk each file's AST once for all rules #3254

Draft

perf(typescript-language): stop tsserver formatting discarded log messages #3256

Draft

michaelfaith reviewed Aug 19, 2026

View reviewed changes

JoshuaKGoldberg and others added 6 commits August 21, 2026 11:48

initial points

04a80b5

Apply suggestions from code review

ed391cc

Co-authored-by: michael faith <michaelfaith@users.noreply.github.com>

lil fix: ordering

bfb15ab

added some basic unit tests

48373f0

Merge branch 'main' into perf-comparisons

990bb81

fix: get performance-testing past knip and flint after merging main

89bdba0

JoshuaKGoldberg requested a review from michaelfaith August 21, 2026 18:17

michaelfaith approved these changes Aug 21, 2026

View reviewed changes

JoshuaKGoldberg and others added 2 commits August 22, 2026 16:55

Apply suggestions from code review

b0e55f0

Co-authored-by: michael faith <michaelfaith@users.noreply.github.com>

path.join in writeStructure

5f2c1ef

JoshuaKGoldberg added 3 commits August 22, 2026 21:19

also normalize in writeFile.test.ts

dd0863d

I thought I removed the results section...

feddd2e

Corrected Knip config

6aaf8f1

Uh oh!

Conversation

JoshuaKGoldberg commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Checklist

Overview

Measurements

Uh oh!

netlify Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for flint-fyi ready!

Uh oh!

changeset-bot Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

lishaduck commented Jun 25, 2026

Uh oh!

JoshuaKGoldberg commented Jun 27, 2026

Uh oh!

lishaduck commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JoshuaKGoldberg commented Jun 28, 2026

Uh oh!

lishaduck commented Jun 28, 2026

Uh oh!

lishaduck commented Jun 28, 2026

Uh oh!

codecov Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JoshuaKGoldberg commented Aug 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelfaith commented Aug 19, 2026

Uh oh!

michaelfaith left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelfaith left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JoshuaKGoldberg commented Aug 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JoshuaKGoldberg commented Jun 24, 2026 •

edited

Loading

netlify Bot commented Jun 24, 2026 •

edited

Loading

changeset-bot Bot commented Jun 24, 2026 •

edited

Loading

lishaduck commented Jun 27, 2026 •

edited

Loading

codecov Bot commented Aug 19, 2026 •

edited

Loading