Use separate Bundler download and installation workers

[joshuay03]

What was the end-user or developer problem that led to this PR?

Bundler uses one worker pool for gem downloads and installations. A slow download occupies a slot that could install an already downloaded gem, while a slow native extension build occupies a slot that could download another gem.

What is your fix for the problem, implemented in this PR?

Use separate download and installation worker pools connected to a shared completion queue. Both pools retain the existing BUNDLE_JOBS size, and compact-index metadata concurrency remains unchanged. This lets network I/O overlap installation without changing Bundler's existing concurrency settings.

Serialize on-demand remote spec lookups so concurrent default-gem downloads cannot race while refreshing shared source state.

Scope update

Add BUNDLE_DOWNLOAD_JOBS and BUNDLE_METADATA_JOBS, and make BUNDLE_JOBS installation-only. Downloads default to 3× installation jobs, capped at 8; metadata defaults to download jobs.

Across five alternating cold-cache Docker builds per variant, median install time improved by 8.5% (66.0s versus 72.1s) and mean install time improved by 9.2% (67.2s versus 74.0s).

Those statements are outdated because the original benchmark changed both the scheduler and its concurrency settings. With concurrency matched to master, review measurements retained a 7.6% reduction from the original 9.5% reduction. My benchmark host reported nproc as 11, and I do not have a measurement that isolates the settings. This revision therefore keeps the scheduler and race fix while removing the new settings and the narrower BUNDLE_JOBS semantics.

Make sure the following tasks are checked

• Describe the problem / feature
• Write tests for features and bug fixes
• Write code to solve the problem
• Make sure you follow the current code style and write meaningful commit messages without tags

[hsbt]

I reproduced this locally and would like to narrow the scope before merging.

On a 10-core macOS arm64 box, cold cache, 94-gem lockfile: at the default settings the split is a wash, 28.57s median before vs 28.68s after (n=6, alternating). With BUNDLE_JOBS=2 I do get your number, -7.9%. But when I pin BUNDLE_DOWNLOAD_JOBS=2 and BUNDLE_METADATA_JOBS=10 on your branch so the concurrency matches master and only the scheduler differs, I still get -7.6% out of that -9.5%. Almost all of the win is the two-pool rewrite in parallel_installer.rb, not the new settings.

So I would like to take the scheduler change and the @remote_spec_for_mutex race fix, and drop BUNDLE_DOWNLOAD_JOBS, BUNDLE_METADATA_JOBS, and the BUNDLE_JOBS narrowing. Being fast out of the box is worth more to users than a knob most of them will never find, and the numbers say the scheduler rewrite already gets us there. The new defaults also lower download and metadata concurrency from processor_count to 8 on machines with 9 or more cores, so as they stand they add a setting and a regression at the same time.

Does that work for you, or do you have a measurement that isolates the settings from the scheduler change? What is nproc on your benchmark host?

[joshuay03]

That works for me. nproc on my benchmark host was 11. I do not have a measurement that isolates the settings from the scheduler change. I will update the PR to address the feedback and narrow its scope as requested.

[hsbt]

👍 Thanks!