Skip to content

Fix O(N^2) serialization in analysis step - #235

Open
rhyschappell wants to merge 1 commit into
HannesStark:mainfrom
rhyschappell:main
Open

Fix O(N^2) serialization in analysis step#235
rhyschappell wants to merge 1 commit into
HannesStark:mainfrom
rhyschappell:main

Conversation

@rhyschappell

Copy link
Copy Markdown
Contributor

In Analyze.run_parallel, tasks are submitted as ex.submit(self.compute_metrics, i). Submitting a bound method pickles self. That is, the entire Analyze object, including self.data.predict_set, which holds one path per design, gets pickled for each task. This means O(n_designs^2) serialization time, which is expensive in the low 10s of thousands of designs range, and blows up in the high 10s of thousands/low 100s of thousands range.

This PR pickles self once per worker instead of once per task.

I tested this change on a 50k design run and see a per-task pickle size drop from ~9MB to 16 bytes. On a 100k design run this takes the estimated analysis time from ~18hrs to ~1.5hrs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant