Add support for rustdoc mergeable cross-crate info parts #16167
Conversation
rustbot
added
A-build-execution
notriddle
force-pushed
the
notriddle/cci
branch
4 times, most recently
from
002bb4d to
204f234
Compare
This is an unstable feature that we designed to fix several performance problems with the old system: 1. You couldn't easily build crate docs in hermetic environments. This doesn't matter for Cargo, but it was one of the original reasons to implement the feature. 2. We have to build all the doc resources in their final form at every step, instead of delaying slow parts (mostly the search index) until the end and only doing them once. 3. It requires rustdoc to take a lock at the end. This reduces available concurrency for generating docs.
notriddle
force-pushed
the
notriddle/cci
branch
from
204f234 to
9925bdf
Compare
There was a problem hiding this comment.
Thanks for creating this draft PR! I didn't know about RFC 3662 until now, so may need more time to catch up.
From my quick skimming through that, it seem prettty aligned with the WIP new build-dir layout in #15010 (one unit per independent directory), as well as the fine-grained locking in #4282.
Not sure if we want to pair them together. It may depend on the timeline of the stabilization of each unstable feature.
| let mut arg = if build_runner.bcx.gctx.cli_unstable().rustdoc_mergeable_info { | ||
| // toolchain resources are written at the end, at the same time as merging | ||
| OsString::from("--emit=invocation-specific,dep-info=") | ||
| } else { | ||
| // if not using mergeable CCI, everything is written every time | ||
| OsString::from("--emit=toolchain-shared-resources,invocation-specific,dep-info=") |
There was a problem hiding this comment.
Another reason we need to stabilize rustdoc's --emit flag 😬
There was a problem hiding this comment.
Yeah, I know. I've been working on that, and will try to get it stabilized before trying to get this stabilized.
There was a problem hiding this comment.
I wonder how this CCI affects scrape-examples workflow on rustdoc side, or it doesn't?
(Probably more a question in the rustodc tracking issue)
3 participants
Part of rust-lang/rust#130676
This is an unstable feature that we designed to fix several performance problems with rustdoc's current implementation:
The old system couldn't support building different crates' docs in a hermetic environment and then merge them at the end. Cargo doesn't do this, but other build systems want to be able to, which was one of the original motivations for proposing mergeable cross-crate info to rustdoc.
The old system has to build all the doc resources in their final form at every step, instead of delaying slow parts (mostly the search index) until the end and only doing them once. By adding this feature to Cargo and rustdoc, we should be able to make back some of the perf problems caused by the suffix tree-based search engine.
The old system requires rustdoc to take a lock while it generates cross-crate info. This reduces available concurrency for generating docs. The new system only has to synchronize at the very end.
A test case for this feature has been added to the automatic test suite.