Add optional support for building native extensions of path sources

This adds a new `build_path_extensions` setting for local path gems with native extensions. When enabled, Bundler compiles those extensions out of tree under `BUNDLE_PATH`, keeps the checkout untouched, and puts the built extension directory ahead of the gem's own load paths so `bundle exec` and `require` can load the compiled artifacts.

The feature supports per-gem opt in via `build_path_extensions.<gem>`, reuses existing build flags from `build.<gem>`, avoids rebuilding when nothing changed, and rebuilds when native extension sources or build flags change. It also surfaces native extension build output for path sources and preserves the existing failure mode when a required extension has not been built or has gone stale.

Author: akiellor

lib/bundler/cli/pristine.rb

@@ -46,6 +46,14 @@ def run
 
             next if git_sources.include?(source)
             git_sources << source
+          when Source::Path
+            extension_dir = source.extension_dir_for(spec)
+            unless extension_dir
+              Bundler.ui.warn("Cannot pristine #{gem_name}. Gem is sourced from local path.")
+              next
+            end
+
+            FileUtils.rm_rf extension_dir
           else
             Bundler.ui.warn("Cannot pristine #{gem_name}. Gem is sourced from local path.")
             next

lib/bundler/man/bundle-config.1

@@ -76,6 +76,10 @@ The following is a list of all configuration keys and their purpose\. You can le
 .IP "\(bu" 4
 \fBbin\fR (\fBBUNDLE_BIN\fR): If configured, \fBbundle binstubs\fR will install executables from gems in the bundle to the specified directory\. Otherwise it will create them in a \fBbin\fR directory relative to the Gemfile directory\. These executables run in Bundler's context\. If used, you might add this directory to your environment's \fBPATH\fR variable\. For instance, if the \fBrails\fR gem comes with a \fBrails\fR executable, \fBbundle binstubs\fR will create a \fBbin/rails\fR executable that ensures that all referred dependencies will be resolved using the bundled gems\.
 .IP "\(bu" 4
+\fBbuild_path_extensions\fR (\fBBUNDLE_BUILD_PATH_EXTENSIONS\fR): Whether Bundler should build the native extensions of gems sourced from a local path\. Off by default, since building them means compiling code out of a directory Bundler does not own\. It can also be set for a single gem, as \fBbuild_path_extensions\.<gem>\fR (\fBBUNDLE_BUILD_PATH_EXTENSIONS__<GEM>\fR), which takes precedence over the global setting\.
+.IP
+Extensions are built out of tree: the gem is copied into a directory under \fBBUNDLE_PATH\fR and compiled there, so that nothing is ever written into your checkout, and the build directory is put ahead of the gem's own \fBlib\fR on the load path\. A rebuild happens whenever the size or modification time of any file under an extension's directory changes, or when the flags configured through \fBbuild\.<gem>\fR change\. Run \fBbundle pristine <gem>\fR to force one\.
+.IP "\(bu" 4
 \fBcache_all\fR (\fBBUNDLE_CACHE_ALL\fR): Cache all gems, including path and git gems\. This needs to be explicitly before bundler 4, but will be the default on bundler 4\.
 .IP "\(bu" 4
 \fBcache_all_platforms\fR (\fBBUNDLE_CACHE_ALL_PLATFORMS\fR): Cache gems for all platforms\.

lib/bundler/man/bundle-config.1.ronn

@@ -122,6 +122,19 @@ learn more about their operation in [bundle install(1)](bundle-install.1.html).
    `rails` executable, `bundle binstubs` will create a `bin/rails` executable
    that ensures that all referred dependencies will be resolved using the
    bundled gems.
+* `build_path_extensions` (`BUNDLE_BUILD_PATH_EXTENSIONS`):
+   Whether Bundler should build the native extensions of gems sourced from a
+   local path. Off by default, since building them means compiling code out of
+   a directory Bundler does not own. It can also be set for a single gem, as
+   `build_path_extensions.<gem>` (`BUNDLE_BUILD_PATH_EXTENSIONS__<GEM>`), which
+   takes precedence over the global setting.
+
+   Extensions are built out of tree: the gem is copied into a directory under
+   `BUNDLE_PATH` and compiled there, so that nothing is ever written into your
+   checkout, and the build directory is put ahead of the gem's own `lib` on the
+   load path. A rebuild happens whenever the size or modification time of any
+   file under an extension's directory changes, or when the flags configured
+   through `build.<gem>` change. Run `bundle pristine <gem>` to force one.
 * `cache_all` (`BUNDLE_CACHE_ALL`):
    Cache all gems, including path and git gems. This needs to be explicitly
    before bundler 4, but will be the default on bundler 4.

lib/bundler/rubygems_ext.rb

@@ -223,6 +223,19 @@ def load_paths
       full_require_paths
     end
 
+    alias_method :rg_full_require_paths, :full_require_paths
+    def full_require_paths
+      @bundler_full_require_paths ||= begin # rubocop:disable Naming/MemoizedInstanceVariableName
+        paths = rg_full_require_paths
+
+        if source.respond_to?(:path?) && source.path? && paths.include?(extension_dir)
+          [extension_dir] + (paths - [extension_dir])
+        else
+          paths
+        end
+      end
+    end
+
     alias_method :rg_extension_dir, :extension_dir
     def extension_dir
       # following instance variable is already used in original method

lib/bundler/settings.rb

@@ -8,6 +8,7 @@ class Settings
 
     BOOL_KEYS = %w[
       auto_install
+      build_path_extensions
       cache_all
       cache_all_platforms
       clean

lib/bundler/source/path.rb

@@ -75,9 +75,10 @@ def name
 
       def install(spec, options = {})
         using_message = "Using #{version_message(spec, options[:previous_spec])} from #{self}"
+        using_message += " with native extensions" if missing_extensions?(spec)
         using_message += " and installing its executables" unless spec.executables.empty?
         print_using_message using_message
-        generate_bin(spec, disable_extensions: true)
+        generate_bin(spec, disable_extensions: !missing_extensions?(spec), build_args: options[:build_args])
         nil # no post-install message
       end
 
@@ -119,6 +120,16 @@ def expanded_original_path
         @expanded_original_path ||= expand(original_path)
       end
 
+      def extension_dir_for(spec)
+        return unless build_extensions?(spec)
+
+        @extension_dirs ||= {}
+        @extension_dirs[spec.full_name] ||= Bundler.install_path.join(
+          "extensions", Gem::Platform.local.to_s, Gem.extension_api_version,
+          "#{spec.full_name}-#{extension_digest(spec)}"
+        ).to_s
+      end
+
       private
 
       def expanded_path
@@ -172,6 +183,10 @@ def load_spec_files
             # consider that for activation and never makes sense to ignore it.
             spec.ignored = false
 
+            if extension_dir = extension_dir_for(spec)
+              spec.extension_dir = extension_dir
+            end
+
             # Validation causes extension_dir to be calculated, which depends
             # on #source, so we validate here instead of load_gemspec
             validate_spec(spec)
@@ -207,6 +222,55 @@ def load_spec_files
         index
       end
 
+      def build_extensions?(spec)
+        return false unless path?
+        return false if spec.extensions.empty?
+
+        per_gem = Bundler.settings["build_path_extensions.#{spec.name}"]
+        return per_gem unless per_gem.nil?
+
+        Bundler.settings[:build_path_extensions] || false
+      end
+
+      def missing_extensions?(spec)
+        return false unless extension_dir_for(spec)
+
+        spec.missing_extensions?
+      end
+
+      def extension_digest(spec)
+        gem_dir = spec.full_gem_path
+        digest = SharedHelpers.digest(:SHA256).new
+        digest << Bundler.settings["build.#{spec.name}"].to_s
+
+        extension_source_files(spec).each do |file|
+          digest << file.delete_prefix("#{gem_dir}/") << "\0"
+          stat = File.stat(file)
+          digest << "#{stat.size}-#{stat.mtime.to_i}-#{stat.mtime.nsec}" << "\0"
+        end
+
+        digest.hexdigest[0, 12]
+      end
+
+      def extension_source_files(spec)
+        files = spec.extensions.flat_map do |extension|
+          dir = File.dirname(extension)
+          base = dir == "." ? spec.full_gem_path : File.join(spec.full_gem_path, dir)
+          Gem::Util.glob_files_in_dir("**/*", base)
+        end
+
+        files << spec.loaded_from.to_s
+        files.uniq.select {|file| File.file?(file) }.sort
+      end
+
+      def extension_build_dir(spec)
+        return unless extension_dir_for(spec)
+
+        Bundler.bundle_path.join(
+          "cache", "bundler", "path_extensions", File.basename(extension_dir_for(spec))
+        ).to_s
+      end
+
       def relative_path(path = self.path)
         if path.to_s.start_with?(root_path.to_s)
           return path.relative_path_from(root_path)
@@ -235,7 +299,8 @@ def generate_bin(spec, options = {})
           env_shebang: false,
           disable_extensions: options[:disable_extensions],
           build_args: options[:build_args],
-          bundler_extension_cache_path: extension_cache_path(spec)
+          bundler_extension_cache_path: extension_cache_path(spec),
+          extension_build_dir: extension_build_dir(spec)
         )
         installer.post_install
       rescue Gem::InvalidSpecificationException => e

lib/bundler/source/path/installer.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require_relative "../../rubygems_gem_installer"
+require_relative "../../vendored_fileutils"
 
 module Bundler
   class Source
@@ -18,6 +19,7 @@ def initialize(spec, options = {})
           @build_args         = options[:build_args] || Bundler.rubygems.build_args
           @gem_bin_dir        = "#{Bundler.rubygems.gem_dir}/bin"
           @disable_extensions = options[:disable_extensions]
+          @extension_build_dir = options[:extension_build_dir]
           @bin_dir = @gem_bin_dir
         end
 
@@ -34,8 +36,39 @@ def post_install
           run_hooks(:post_install)
         end
 
+        def build_extensions
+          return super unless @extension_build_dir
+
+          stage_extension_sources
+
+          real_spec = @spec
+          @spec = staged_spec(real_spec)
+          begin
+            super
+          ensure
+            @spec = real_spec
+            FileUtils.rm_rf(@extension_build_dir)
+          end
+        end
+
         private
 
+        def stage_extension_sources
+          SharedHelpers.filesystem_access(@extension_build_dir, :create) do |path|
+            FileUtils.rm_rf(path)
+            FileUtils.mkdir_p(path)
+            FileUtils.cp_r("#{@spec.full_gem_path}/.", path)
+          end
+        end
+
+        def staged_spec(spec)
+          staged = spec.dup
+          staged.source = nil
+          staged.full_gem_path = @extension_build_dir
+          staged.extension_dir = spec.extension_dir
+          staged
+        end
+
         def run_hooks(type)
           hooks_meth = "#{type}_hooks"
           return unless Gem.respond_to?(hooks_meth)