Skip to content

fix: OOM error #36

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

fix: OOM error #36

wants to merge 8 commits into from

Conversation

SgtPooki
Copy link

@SgtPooki SgtPooki commented Jun 16, 2025

Fixes #34
Built on top of #35 -- I couldn't get project to compile without this

You can run the test with npm run unittest -- -g 'does not run out of memory'

Note that there is also a lot of memory being consumed by fsExtra in CodeBlockExtractor when reading an entire file into memory. We could speed things up and use much less memory if we used some sort of readline approach to reading the markdown code blocks, and emitted them with an async iterator as they pop up.

I went ahead with the CodeBlockExtractor change.. it speeds things up a bit and should allow for much larger files to be read a little more safely, but can remove from this PR if desired. check the stats from hyperfine run below.

NOTE that in the below hyperfine tests, the memory test removed so test speeds are a little more accurate.

> hyperfine --parameter-list branch fix/dont-use-chai-object-prototype-modifications,test/show-OOM-error --setup "git switch {branch} && npm run build" "npm run unittest -- -g 'TypeScriptDocsVerifier'" --runs 50
Benchmark 1: npm run unittest -- -g 'TypeScriptDocsVerifier' (branch = fix/dont-use-chai-object-prototype-modifications)
  Time (mean ± σ):     13.722 s ±  0.228 s    [User: 20.098 s, System: 2.675 s]
  Range (min … max):   13.451 s … 14.373 s    50 runs
 
Benchmark 2: npm run unittest -- -g 'TypeScriptDocsVerifier' (branch = test/show-OOM-error)
  Time (mean ± σ):     10.674 s ±  0.213 s    [User: 14.952 s, System: 1.856 s]
  Range (min … max):   10.411 s … 11.318 s    50 runs
 
Summary
  npm run unittest -- -g 'TypeScriptDocsVerifier' (branch = test/show-OOM-error) ran
    1.29 ± 0.03 times faster than npm run unittest -- -g 'TypeScriptDocsVerifier' (branch = fix/dont-use-chai-object-prototype-m [20m22.831s]

How I changed the solution to #30 (previously fixed by #33):

Wrap all functions that should be able to be wrapped in an IIFE without error in a named IIFE, and to ensure line number accuracy for errors, recognize that if the code block is wrapped.

It's possible that someone may want to completely prevent this behavior for some reason, so I would propose a <!-- ts-docs-verifier:no-wrap --> or // ts-docs-verifier:no-wrap pragma in order to allow for escaping this new behavior if desired.

@SgtPooki
Copy link
Author

Will update and remove changes from #35

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug: Out-of-Memory error when parsing many snippets with compileSnippets
1 participant