Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: incremental rebuild make failed modules and dependencies #9393

Merged
merged 3 commits into from
Feb 20, 2025
Merged

refactor: incremental rebuild make failed modules and dependencies #9393

merged 3 commits into from
Feb 20, 2025

Conversation

jerrykingxyz
Copy link
Contributor

@jerrykingxyz jerrykingxyz commented Feb 19, 2025 •
edited
Loading

Summary

  1. Make will not always rebuild all of failed module and failed dependencies when recompile. The MakeArtifact will save make_failed_module and make_failed_dependencies as statistical data, which are only used to find failed module and failed dependencies as indexes.
struct MakeArtifact {
// data
-  pub make_failed_module: IdentifierSet,
-  pub make_failed_dependencies: HashSet<DependencyId>,
// statistical data
+  pub make_failed_module: IdentifierSet,
+  pub make_failed_dependencies: HashSet<DependencyId>,
...
}
  1. Add factorize_info to all of ModuleDependency and ContextDependency, and FactorizeInfo will save *_dependencies and diagnostics like Module.BuildInfo (PS. no need to save those info if factorize success).
pub enum FactorizeInfo {
  Success,
  Failed {
    related_dep_ids: Vec<DependencyId>,
    file_dependencies: HashSet<ArcPath>,
    context_dependencies: HashSet<ArcPath>,
    missing_dependencies: HashSet<ArcPath>,
    #[cacheable(with=Skip)]
    diagnostics: Vec<Diagnostic>,
  },
}

trait ModuleContext {
  ..
  fn factorize_info(&self) -> &FactorizeInfo;
  fn factorize_info_mut(&mut self) -> &mut FactorizeInfo;
}
  1. Modified and deleted files will rebuild the affected make_failed_dependencies instead of all make_failed_dependencies.
impl Cutout {
  fn cutout_artifact(..) {
    match param {
        ..
        MakeParam::RemovedFiles(files) {
          ..
          for dep_id in &artifact.make_failed_dependencies {
            let dep = module_graph
              .dependency_by_id(dep_id)
              .expect("should have dependency");
            let info = FactorizeInfo::get_from(dep).expect("should have factorize info");
            if info.depends_on(&files) {
              force_build_deps.insert(*dep_id);
            }
          }
        }
    }
  }
}

Checklist

  • Tests updated (or not required).
  • Documentation updated (or not required).

@github-actions github-actions bot added the team The issue/pr is created by the member of Rspack. label Feb 19, 2025
@netlify Netlify
Copy link

netlify bot commented Feb 19, 2025 •
edited
Loading

✅ Deploy Preview for rspack canceled.

Name Link
🔨 Latest commit 4542440
🔍 Latest deploy log https://app.netlify.com/sites/rspack/deploys/67b6edbdc783d400084fb99a

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 19, 2025 •
edited
Loading

📝 Benchmark detail: Open

Name Base (2025-02-19 a81bd20) Current Change
10000_big_production-mode_disable-minimize + exec 36.9 s ± 476 ms 37.7 s ± 1.22 s +2.17 %
10000_development-mode + exec 1.76 s ± 20 ms 1.7 s ± 18 ms -3.82 %
10000_development-mode_hmr + exec 672 ms ± 7 ms 663 ms ± 6.6 ms -1.35 %
10000_production-mode + exec 2.23 s ± 54 ms 2.19 s ± 145 ms -1.61 %
10000_production-mode_persistent-cold + exec 2.37 s ± 98 ms 2.3 s ± 33 ms -2.77 %
10000_production-mode_persistent-hot + exec 1.63 s ± 48 ms 1.62 s ± 36 ms -0.94 %
arco-pro_development-mode + exec 1.76 s ± 60 ms 1.72 s ± 51 ms -2.30 %
arco-pro_development-mode_hmr + exec 386 ms ± 3.5 ms 376 ms ± 1.9 ms -2.50 %
arco-pro_production-mode + exec 3.67 s ± 70 ms 3.53 s ± 170 ms -3.84 %
arco-pro_production-mode_generate-package-json-webpack-plugin + exec 3.66 s ± 258 ms 3.6 s ± 282 ms -1.76 %
arco-pro_production-mode_persistent-cold + exec 3.71 s ± 205 ms 3.57 s ± 96 ms -3.99 %
arco-pro_production-mode_persistent-hot + exec 2.3 s ± 79 ms 2.36 s ± 241 ms +2.77 %
arco-pro_production-mode_traverse-chunk-modules + exec 3.56 s ± 131 ms 3.52 s ± 154 ms -1.03 %
large-dyn-imports_development-mode + exec 1.99 s ± 47 ms 1.96 s ± 61 ms -1.42 %
large-dyn-imports_production-mode + exec 2.04 s ± 31 ms 2.04 s ± 38 ms -0.30 %
threejs_development-mode_10x + exec 1.52 s ± 7.6 ms 1.53 s ± 17 ms +0.70 %
threejs_development-mode_10x_hmr + exec 774 ms ± 5.1 ms 791 ms ± 40 ms +2.30 %
threejs_production-mode_10x + exec 5.15 s ± 62 ms 5.21 s ± 349 ms +1.06 %
threejs_production-mode_10x_persistent-cold + exec 5.2 s ± 92 ms 5.18 s ± 36 ms -0.35 %
threejs_production-mode_10x_persistent-hot + exec 4.44 s ± 201 ms 4.42 s ± 55 ms -0.46 %
10000_big_production-mode_disable-minimize + rss memory 8666 MiB ± 41 MiB 8673 MiB ± 53.5 MiB +0.08 %
10000_development-mode + rss memory 647 MiB ± 23.7 MiB 681 MiB ± 19 MiB +5.30 %
10000_development-mode_hmr + rss memory 1326 MiB ± 157 MiB 1334 MiB ± 174 MiB +0.60 %
10000_production-mode + rss memory 617 MiB ± 8.92 MiB 661 MiB ± 31.4 MiB +7.13 %
10000_production-mode_persistent-cold + rss memory 724 MiB ± 23 MiB 760 MiB ± 31.7 MiB +5.00 %
10000_production-mode_persistent-hot + rss memory 690 MiB ± 28.4 MiB 734 MiB ± 22.7 MiB +6.44 %
arco-pro_development-mode + rss memory 566 MiB ± 42.3 MiB 593 MiB ± 39.3 MiB +4.75 %
arco-pro_development-mode_hmr + rss memory 654 MiB ± 47.4 MiB 662 MiB ± 50.6 MiB +1.22 %
arco-pro_production-mode + rss memory 716 MiB ± 21.3 MiB 733 MiB ± 25.4 MiB +2.28 %
arco-pro_production-mode_generate-package-json-webpack-plugin + rss memory 729 MiB ± 38.1 MiB 719 MiB ± 19.7 MiB -1.30 %
arco-pro_production-mode_persistent-cold + rss memory 798 MiB ± 61 MiB 790 MiB ± 26.1 MiB -0.98 %
arco-pro_production-mode_persistent-hot + rss memory 646 MiB ± 20.1 MiB 667 MiB ± 30.1 MiB +3.34 %
arco-pro_production-mode_traverse-chunk-modules + rss memory 724 MiB ± 42.3 MiB 722 MiB ± 27.8 MiB -0.18 %
large-dyn-imports_development-mode + rss memory 636 MiB ± 1.7 MiB 652 MiB ± 10 MiB +2.53 %
large-dyn-imports_production-mode + rss memory 519 MiB ± 4.18 MiB 526 MiB ± 9.64 MiB +1.41 %
threejs_development-mode_10x + rss memory 550 MiB ± 14.7 MiB 558 MiB ± 10.1 MiB +1.36 %
threejs_development-mode_10x_hmr + rss memory 1138 MiB ± 114 MiB 1098 MiB ± 219 MiB -3.51 %
threejs_production-mode_10x + rss memory 837 MiB ± 31.6 MiB 845 MiB ± 5.72 MiB +0.98 %
threejs_production-mode_10x_persistent-cold + rss memory 930 MiB ± 31.9 MiB 924 MiB ± 67.7 MiB -0.64 %
threejs_production-mode_10x_persistent-hot + rss memory 794 MiB ± 26.3 MiB 817 MiB ± 35.9 MiB +2.84 %

@codspeed-hq CodSpeed HQ
Copy link

codspeed-hq bot commented Feb 19, 2025 •
edited
Loading

CodSpeed Performance Report

Merging #9393 will not alter performance

Comparing jerry/make (4542440) with main (b46f13c)

🎉 Hooray! codspeed-rust just leveled up to 2.7.2!

A heads-up, this is a breaking change and it might affect your current performance baseline a bit. But here's the exciting part - it's packed with new, cool features and promises improved result stability 🥳!
Curious about what's new? Visit our releases page to delve into all the awesome details about this new version.

Summary

✅ 7 untouched benchmarks

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 20, 2025 •
edited
Loading

📝 Benchmark detail: Open

Name Base (2025-02-20 2101c1b) Current Change
10000_big_production-mode_disable-minimize + exec 37.5 s ± 434 ms 37.5 s ± 830 ms -0.04 %
10000_development-mode + exec 1.73 s ± 37 ms 1.69 s ± 7 ms -2.24 %
10000_development-mode_hmr + exec 674 ms ± 4.7 ms 671 ms ± 21 ms -0.49 %
10000_production-mode + exec 2.21 s ± 43 ms 2.16 s ± 53 ms -2.13 %
10000_production-mode_persistent-cold + exec 2.35 s ± 168 ms 2.34 s ± 80 ms -0.45 %
10000_production-mode_persistent-hot + exec 1.59 s ± 22 ms 1.63 s ± 68 ms +2.08 %
arco-pro_development-mode + exec 1.75 s ± 105 ms 1.76 s ± 137 ms +0.34 %
arco-pro_development-mode_hmr + exec 385 ms ± 3.1 ms 375 ms ± 0.55 ms -2.61 %
arco-pro_production-mode + exec 3.57 s ± 201 ms 3.52 s ± 140 ms -1.33 %
arco-pro_production-mode_generate-package-json-webpack-plugin + exec 3.65 s ± 103 ms 3.61 s ± 217 ms -0.95 %
arco-pro_production-mode_persistent-cold + exec 3.69 s ± 248 ms 3.57 s ± 198 ms -3.28 %
arco-pro_production-mode_persistent-hot + exec 2.33 s ± 131 ms 2.31 s ± 153 ms -0.98 %
arco-pro_production-mode_traverse-chunk-modules + exec 3.58 s ± 113 ms 3.58 s ± 145 ms -0.26 %
large-dyn-imports_development-mode + exec 1.98 s ± 26 ms 1.96 s ± 34 ms -0.90 %
large-dyn-imports_production-mode + exec 2.04 s ± 53 ms 2.04 s ± 129 ms -0.04 %
threejs_development-mode_10x + exec 1.51 s ± 24 ms 1.56 s ± 43 ms +2.65 %
threejs_development-mode_10x_hmr + exec 776 ms ± 34 ms 811 ms ± 25 ms +4.56 %
threejs_production-mode_10x + exec 5.2 s ± 394 ms 5.23 s ± 176 ms +0.58 %
threejs_production-mode_10x_persistent-cold + exec 5.27 s ± 332 ms 5.24 s ± 98 ms -0.58 %
threejs_production-mode_10x_persistent-hot + exec 4.44 s ± 275 ms 4.44 s ± 44 ms 0.00 %
10000_big_production-mode_disable-minimize + rss memory 8671 MiB ± 16.8 MiB 8692 MiB ± 54.1 MiB +0.23 %
10000_development-mode + rss memory 658 MiB ± 28.2 MiB 654 MiB ± 19.7 MiB -0.60 %
10000_development-mode_hmr + rss memory 1266 MiB ± 241 MiB 1253 MiB ± 282 MiB -1.03 %
10000_production-mode + rss memory 645 MiB ± 23.5 MiB 629 MiB ± 27.6 MiB -2.40 %
10000_production-mode_persistent-cold + rss memory 736 MiB ± 20.5 MiB 733 MiB ± 30.1 MiB -0.33 %
10000_production-mode_persistent-hot + rss memory 728 MiB ± 27.3 MiB 734 MiB ± 22.2 MiB +0.84 %
arco-pro_development-mode + rss memory 580 MiB ± 34.2 MiB 571 MiB ± 33.3 MiB -1.46 %
arco-pro_development-mode_hmr + rss memory 648 MiB ± 49.5 MiB 639 MiB ± 55.6 MiB -1.40 %
arco-pro_production-mode + rss memory 730 MiB ± 44.8 MiB 715 MiB ± 30.4 MiB -2.12 %
arco-pro_production-mode_generate-package-json-webpack-plugin + rss memory 740 MiB ± 22.9 MiB 722 MiB ± 18 MiB -2.43 %
arco-pro_production-mode_persistent-cold + rss memory 811 MiB ± 34.9 MiB 792 MiB ± 43.9 MiB -2.38 %
arco-pro_production-mode_persistent-hot + rss memory 671 MiB ± 24 MiB 669 MiB ± 34.1 MiB -0.40 %
arco-pro_production-mode_traverse-chunk-modules + rss memory 736 MiB ± 26.8 MiB 707 MiB ± 28.3 MiB -3.93 %
large-dyn-imports_development-mode + rss memory 652 MiB ± 3.22 MiB 644 MiB ± 4.55 MiB -1.16 %
large-dyn-imports_production-mode + rss memory 537 MiB ± 9.03 MiB 524 MiB ± 8.62 MiB -2.48 %
threejs_development-mode_10x + rss memory 550 MiB ± 25.2 MiB 557 MiB ± 8.16 MiB +1.29 %
threejs_development-mode_10x_hmr + rss memory 1091 MiB ± 142 MiB 1154 MiB ± 47.9 MiB +5.78 %
threejs_production-mode_10x + rss memory 825 MiB ± 47.2 MiB 837 MiB ± 14.2 MiB +1.45 %
threejs_production-mode_10x_persistent-cold + rss memory 931 MiB ± 39.6 MiB 939 MiB ± 41.1 MiB +0.87 %
threejs_production-mode_10x_persistent-hot + rss memory 811 MiB ± 46.3 MiB 808 MiB ± 29.2 MiB -0.39 %

@jerrykingxyz jerrykingxyz changed the title refactor: make failed modules and dependencies support incremental refactor: incremental rebuild make failed modules and dependencies Feb 20, 2025
@jerrykingxyz jerrykingxyz enabled auto-merge (squash) February 20, 2025 09:39
@jerrykingxyz jerrykingxyz merged commit 6c8b2c4 into main Feb 20, 2025
31 checks passed
@jerrykingxyz jerrykingxyz deleted the jerry/make branch February 20, 2025 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@h-a-n-a h-a-n-a h-a-n-a approved these changes

Assignees
No one assigned
Labels
team The issue/pr is created by the member of Rspack.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jerrykingxyz @h-a-n-a