Add optional support for building native extensions of path sources - #9820
Add optional support for building native extensions of path sources#9820akiellor wants to merge 5 commits into
Conversation
7dab41b to
2e1090d
Compare
|
This PR is an attempt to tackle this issue. Feedback is welcome 🙏. |
15b702a to
70a12e4
Compare
This adds a new `build_path_extensions` setting for local path gems with native extensions. When enabled, Bundler compiles those extensions out of tree under `BUNDLE_PATH`, keeps the checkout untouched, and puts the built extension directory ahead of the gem's own load paths so `bundle exec` and `require` can load the compiled artifacts. The feature supports per-gem opt in via `build_path_extensions.<gem>`, reuses existing build flags from `build.<gem>`, avoids rebuilding when nothing changed, and rebuilds when native extension sources or build flags change. It also surfaces native extension build output for path sources and preserves the existing failure mode when a required extension has not been built or has gone stale.
70a12e4 to
8da90ad
Compare
|
@hsbt, do you have any feedback on this proposed change? IMHO, it would meaningfully improve the developer experience of working with gems with native extensions in a monorepo. Thanks for all your efforts on rubygems. |
| def missing_extensions?(spec) | ||
| return false unless extension_dir_for(spec) | ||
|
|
||
| spec.missing_extensions? |
There was a problem hiding this comment.
spec.missing_extensions? is forced to false for Bundler::Source::Path by FixPathSourceMissingExtensions, which rubygems_ext.rb prepends whenever Gem.rubygems_version < 3.5.22. Ruby 3.2 ships RubyGems 3.4.19, so build_path_extensions builds nothing there, and the new specs will fail on the system-rubygems-bundler (ruby-3.2) job.
There was a problem hiding this comment.
Interesting.
I've reimplemented the feature to avoid missing_extensions? to drive if the extension should be built.
An alternative would have been to version gate this feature and report an error if is attempted to be used on an old version of rubygems.
Your feedback on a preferred approach is welcome 🙏
| SharedHelpers.filesystem_access(@extension_build_dir, :create) do |path| | ||
| FileUtils.rm_rf(path) | ||
| FileUtils.mkdir_p(path) | ||
| FileUtils.cp_r("#{@spec.full_gem_path}/.", path) |
There was a problem hiding this comment.
When the gem directory contains BUNDLE_PATH -- gem "app", path: "." with bundle config set path vendor/bundle -- the staging directory sits inside the copy source, and FileUtils.cp_r recurses into itself until Errno::ENAMETOOLONG. That writes a deep tree into the checkout this feature promises never to touch.
There was a problem hiding this comment.
Good catch. I've added a spec and fix to avoid this problem.
…hin the source tree (vendor/bundle)
2 participants
What was the end-user or developer problem that led to this PR?
Bundler did not build native extensions for gems sourced from a local
:path, which is a common setup in monorepos. That meant path-based gems that relied on compiled code could not be used unless developers built those extensions manually in their checkout. In practice, this made local development in monorepos inconsistent with out-of-tree gems, brokebundle exec/requirefor those gems, and added manual setup steps that made the developer experience worse.What is your fix for the problem, implemented in this PR?
This adds a new
build_path_extensionssetting for local path gems with native extensions. When enabled, Bundler compiles those extensions out of tree underBUNDLE_PATH, keeps the checkout untouched, and puts the built extension directory ahead of the gem's own load paths sobundle execandrequirecan load the compiled artifacts.The feature supports per-gem opt in via
build_path_extensions.<gem>, reuses existing build flags frombuild.<gem>, avoids rebuilding when nothing changed, and rebuilds when native extension sources or build flags change. It also surfaces native extension build output for path sources and preserves the existing failure mode when a required extension has not been built or has gone stale.Make sure the following tasks are checked