Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: slow module detection for webpack #75368

Merged
merged 9 commits into from
Jan 28, 2025
Merged

Conversation

gaojude
Copy link
Contributor

@gaojude gaojude commented Jan 27, 2025

Problem

During development, certain modules can take an unexpectedly long time to compile, significantly impacting build performance. Currently, developers have no visibility into which modules are causing these slowdowns, making it difficult to optimize their codebase.

Solution

This PR introduces a new experimental feature that detects and reports modules that are slow to compile during webpack builds. When enabled, it:

  • Tracks build time for each module
  • Reports modules that exceed a configurable time threshold
  • Shows a dependency tree visualization of slow modules
  • Helps developers identify performance bottlenecks in their build process

Configuration

Enable in next.config.js:

module.exports = {
  experimental: {
    slowModuleDetection: {
      buildTimeThresholdMs: 500,  // Time threshold to consider a module "slow"
    }
  }
}

Example Output

image

Implementation Details

  • Implements a new webpack plugin (SlowModuleDetectionPlugin) that hooks into the compilation process
  • Uses webpack's module graph to build a dependency tree of slow modules
  • Provides clear, actionable output to help developers optimize their builds
  • Adds comprehensive error handling with detailed error messages
  • Includes e2e tests to ensure reliability

Limitations

  • To ensure accurate measurements, we will set the parallelism option to 1 in the Webpack configuration (see Webpack documentation) when the feature is enabled. This may significantly slow down development performance, but it will help surface issues proportionally to their normal duration.
  • Currently only supports webpack builds. Turbopack support will be added in a future PR
  • Only tracks module build time, not other compilation steps
  • Thresholds may need tuning based on project size and complexity

https://linear.app/vercel/issue/NDX-86/slow-module-detection

Copy link
Contributor Author

gaojude commented Jan 27, 2025

This stack of pull requests is managed by Graphite. Learn more about stacking.

@gaojude gaojude force-pushed the jude/webpack-slow-module branch from 6951bb6 to dd6f379 Compare January 27, 2025 16:54
@ijjk
Copy link
Member

ijjk commented Jan 27, 2025

Stats from current PR

Default Build (Increase detected ⚠️)
General Overall increase ⚠️
vercel/next.js canary vercel/next.js jude/webpack-slow-module Change
buildDuration 17.9s 15.8s N/A
buildDurationCached 14.8s 12.4s N/A
nodeModulesSize 392 MB 392 MB ⚠️ +101 kB
nextStartRea..uration (ms) 427ms 426ms N/A
Client Bundles (main, webpack)
vercel/next.js canary vercel/next.js jude/webpack-slow-module Change
5306-HASH.js gzip 54 kB 53.9 kB N/A
8276.HASH.js gzip 169 B 168 B N/A
8377-HASH.js gzip 5.46 kB 5.46 kB N/A
bccd1874-HASH.js gzip 52.9 kB 52.9 kB
framework-HASH.js gzip 57.5 kB 57.5 kB N/A
main-app-HASH.js gzip 240 B 241 B N/A
main-HASH.js gzip 34.5 kB 34.4 kB N/A
webpack-HASH.js gzip 1.71 kB 1.71 kB N/A
Overall change 52.9 kB 52.9 kB
Legacy Client Bundles (polyfills)
vercel/next.js canary vercel/next.js jude/webpack-slow-module Change
polyfills-HASH.js gzip 39.4 kB 39.4 kB
Overall change 39.4 kB 39.4 kB
Client Pages
vercel/next.js canary vercel/next.js jude/webpack-slow-module Change
_app-HASH.js gzip 193 B 193 B
_error-HASH.js gzip 193 B 193 B
amp-HASH.js gzip 512 B 510 B N/A
css-HASH.js gzip 343 B 342 B N/A
dynamic-HASH.js gzip 1.84 kB 1.84 kB
edge-ssr-HASH.js gzip 265 B 265 B
head-HASH.js gzip 363 B 362 B N/A
hooks-HASH.js gzip 393 B 392 B N/A
image-HASH.js gzip 4.59 kB 4.58 kB N/A
index-HASH.js gzip 268 B 268 B
link-HASH.js gzip 2.35 kB 2.35 kB N/A
routerDirect..HASH.js gzip 328 B 328 B
script-HASH.js gzip 397 B 397 B
withRouter-HASH.js gzip 323 B 326 B N/A
1afbb74e6ecf..834.css gzip 106 B 106 B
Overall change 3.59 kB 3.59 kB
Client Build Manifests
vercel/next.js canary vercel/next.js jude/webpack-slow-module Change
_buildManifest.js gzip 748 B 747 B N/A
Overall change 0 B 0 B
Rendered Page Sizes
vercel/next.js canary vercel/next.js jude/webpack-slow-module Change
index.html gzip 523 B 523 B
link.html gzip 537 B 538 B N/A
withRouter.html gzip 518 B 519 B N/A
Overall change 523 B 523 B
Edge SSR bundle Size
vercel/next.js canary vercel/next.js jude/webpack-slow-module Change
edge-ssr.js gzip 129 kB 129 kB N/A
page.js gzip 210 kB 210 kB N/A
Overall change 0 B 0 B
Middleware size
vercel/next.js canary vercel/next.js jude/webpack-slow-module Change
middleware-b..fest.js gzip 671 B 664 B N/A
middleware-r..fest.js gzip 155 B 156 B N/A
middleware.js gzip 31.3 kB 31.3 kB N/A
edge-runtime..pack.js gzip 844 B 844 B
Overall change 844 B 844 B
Next Runtimes
vercel/next.js canary vercel/next.js jude/webpack-slow-module Change
app-page-exp...dev.js gzip 386 kB 386 kB
app-page-exp..prod.js gzip 131 kB 131 kB
app-page-tur..prod.js gzip 144 kB 144 kB
app-page-tur..prod.js gzip 140 kB 140 kB
app-page.run...dev.js gzip 373 kB 373 kB N/A
app-page.run..prod.js gzip 128 kB 128 kB
app-route-ex...dev.js gzip 39.4 kB 39.4 kB
app-route-ex..prod.js gzip 25 kB 25 kB
app-route-tu..prod.js gzip 25 kB 25 kB
app-route-tu..prod.js gzip 24.8 kB 24.8 kB
app-route.ru...dev.js gzip 41 kB 41 kB
app-route.ru..prod.js gzip 24.8 kB 24.8 kB
dist_client_...dev.js gzip 326 B 326 B
dist_client_...dev.js gzip 318 B 318 B
pages-api-tu..prod.js gzip 8.81 kB 8.81 kB
pages-api.ru...dev.js gzip 11.5 kB 11.5 kB
pages-api.ru..prod.js gzip 8.8 kB 8.8 kB
pages-turbo...prod.js gzip 21.6 kB 21.6 kB
pages.runtim...dev.js gzip 31.3 kB 31.3 kB N/A
pages.runtim..prod.js gzip 21.6 kB 21.6 kB
server.runti..prod.js gzip 73.7 kB 73.7 kB
Overall change 1.26 MB 1.26 MB
build cache Overall increase ⚠️
vercel/next.js canary vercel/next.js jude/webpack-slow-module Change
0.pack gzip 2.1 MB 2.1 MB ⚠️ +687 B
index.pack gzip 74.8 kB 75.3 kB ⚠️ +563 B
Overall change 2.18 MB 2.18 MB ⚠️ +1.25 kB
Diff details
Diff for main-HASH.js

Diff too large to display

Diff for app-page-exp..ntime.dev.js
failed to diff
Diff for app-page.runtime.dev.js
failed to diff
Diff for pages.runtime.dev.js

Diff too large to display

Commit: ebf2d78

@gaojude gaojude changed the title Turbopack: use raw_read_dir to create fewer FileSystemPaths (#75237) slow module detection for webpack Jan 27, 2025
@gaojude gaojude changed the title slow module detection for webpack feat: slow module detection for webpack Jan 27, 2025
@gaojude gaojude force-pushed the jude/webpack-slow-module branch 2 times, most recently from 791cfdb to 25ead27 Compare January 27, 2025 17:01
@vercel vercel deleted a comment from ijjk Jan 27, 2025
@gaojude gaojude marked this pull request as ready for review January 27, 2025 17:07
/**
* Configuration options for the slow module detection webpack plugin.
*/
slowModuleDetectionWebpack?: {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel we should have a more generic like performance or anything similar to group our experimental perf features. That will allow us to easily extend in the future

{
   performance: {
      slowModuleDetection: {}  
   }
}

When users enabling it and running next.js with turbopack, we can log a warning saying this is not supported in turbopack yet. In way we can keep the configuration atomic and streamlined, and once turbopack is supported we only need to drop the warning

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect we'll move this into the existing logging options, so having a top-level experimental flag for now seems fine.

But I think we should rename it to slowModuleDetection and then add it to the turbopack warnings list here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we could present it in a different way for instance inside the devtool or overlay, then they dont have to belong to logging options. It's more like a functionality regardless the way of presentation. I lean on moving it to a new option group like performance that potentially has more extension in the future

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While it's experimental, I don't think we need to worry about creating a subconfiguration for it. But I agree once we land it, we'll want to come up with something.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea sg, be careful that the logging is currently in the stable config. so I guess putting on top is fine. "Slow module" can be one of the perspectives of this feature. Maybe we can name it to moduleSize. Webpack has a performance option that we could learn from it for naming.

{
  performance: {
    moduleSize: boolean | RegExp // for filtering out the unnecessary changes. 
  },
};

e.g. if you only want to view js or css this RegExp filter of moduleSize can be useful

Copy link
Contributor Author

@gaojude gaojude Jan 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the feedback, @huozhi and @ztanner.

As discussed, we’ll revisit the configuration schema once this feature is planned as stable. (@huozhi also suggested that a RegExp filter might be necessary, but I’ll hold off on making that decision until we’ve gathered more feedback from users experimenting with the feature.)

I’ll also remove "webpack" from the name, as suggested by @ztanner.

@ijjk
Copy link
Member

ijjk commented Jan 27, 2025

Tests Passed

@gaojude gaojude force-pushed the jude/webpack-slow-module branch from ab39854 to 8b53be3 Compare January 27, 2025 20:39
@gaojude gaojude requested a review from ztanner January 27, 2025 20:40
@gaojude gaojude force-pushed the jude/webpack-slow-module branch from c5adfaa to 09ac82b Compare January 27, 2025 22:16
@gaojude gaojude requested a review from sokra January 27, 2025 22:19
@gaojude gaojude requested a review from sokra January 28, 2025 02:07
@gaojude gaojude force-pushed the jude/webpack-slow-module branch from 3a5d74d to d127bfe Compare January 28, 2025 14:57
@gaojude gaojude requested review from ztanner and huozhi January 28, 2025 15:36
@gaojude gaojude merged commit 31c047d into canary Jan 28, 2025
133 checks passed
@gaojude gaojude deleted the jude/webpack-slow-module branch January 28, 2025 15:56
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 12, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants