redesign stage 0 std #119899
Open
redesign stage 0 std #119899
+289
−327
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rustbot
added
S-waiting-on-review
onur-ozkan
added
S-waiting-on-author
This comment has been minimized.
This comment has been minimized.
rustbot
added
A-testsuite
onur-ozkan
force-pushed
the
redesign-stage0-std
branch
from
9fcee5c to
066ce06
Compare
This comment has been minimized.
This comment has been minimized.
onur-ozkan
force-pushed
the
redesign-stage0-std
branch
from
066ce06 to
472c50c
Compare
This comment has been minimized.
This comment has been minimized.
onur-ozkan
force-pushed
the
redesign-stage0-std
branch
2 times, most recently
from
ce81474 to
b688ffa
Compare
This comment has been minimized.
This comment has been minimized.
onur-ozkan
force-pushed
the
redesign-stage0-std
branch
6 times, most recently
from
5f1747d to
00e59f0
Compare
|
This PR modifies If appropriate, please update |
|
@rustbot ready r? bootstrap |
rustbot
added
S-waiting-on-review
onur-ozkan
force-pushed
the
redesign-stage0-std
branch
from
00e59f0 to
8ac271c
Compare
|
@rustbot author (currently stage 2 std is copied from stage 1 and this behaviour should change with the beta std change) |
rustbot
removed
the
S-waiting-on-review
|
Rebased. Do you still plan to test this PR @jieyouxu? I've rebased it multiple times and it's exhausting a lot, so I would like to hold rebasing again until everyone has finished reviewing and testing. |
|
Yes I'm still testing (but progress has been slow), rebasing is not necessary. EDIT: trying to get the testing done by today. |
|
Sorry for the wait, but local testing looks good to me (ignore the two problems previously, consider them resolved as they were also discussed in the review thread). |
Local manual testing (Round 1)Known and intentional omissions
TestsUniform platform (build == host == target)Cherry-picked platform: Check flow
|
| Kind | |
|---|---|
| Description | Minimal library profile, download CI rustc unspecified, check library with no explicit stage |
| Invocation | ./x check library |
| Outcome | OK; a local stage 1 rustc was built because PR modifies compiler/ tree |
To reproduce
Setup
$ rm -rf build/
$ tee config.toml <<EOF
profile = "library"
EOFInvocation
$ ./x check libraryAssessment
Watching output. Observe that stage1 rustc was fully built instead of downloading and using CI rustc.
Teardown
$ rm -rf build/profile = "library" + download-rustc = false + ./x check library
| Kind | |
|---|---|
| Description | Library profile, force disable CI rustc, check library with no explicit stage |
| Invocation | ./x check library |
| Outcome | OK; builds stage 1 compiler, checks stage 1 library incl. bench example |
To reproduce
Setup
$ rm -rf build/
$ tee config.toml <<EOF
profile = "library"
[rust]
download-rustc = false
EOFInvocation
$ ./x check libraryAssessment
Stage 1 rustc built, stage 1 library artifacts checked
Teardown
$ rm -rf build/profile = "library" + ./x check library --stage=0
| Kind | |
|---|---|
| Description | Minimal library profile, check at stage 0 |
| Invocation | ./x check library --stage=0 |
| Outcome | OK; no-op as expected |
To reproduce
Setup
$ rm -rf build/
$ tee config.toml <<EOF
profile = "library"
EOFInvocation
$ ./x check library --stage=0Assessment
Ok, no-op.
Teardown
$ rm -rf build/profile = "library" + ./x check library --stage=1
| Kind | |
|---|---|
| Description | Minimal library profile; check at explicit stage 1 |
| Invocation | ./x check library --stage=1 |
| Outcome | OK; builds stage 1 rustc, checks stage 1 library artifacts |
To reproduce
Setup
$ rm -rf build/
$ tee config.toml <<EOF
profile = "library"
EOFInvocation
$ ./x check library --stage=1Assessment
Builds stage 1 rustc, checks stage 1 library artifacts
Teardown
$ rm -rf build/profile = "library" + ./x check library --stage=2
| Kind | |
|---|---|
| Description | Minimal library profile, check at explicit stage 2 |
| Invocation | ./x check library --stage=2 |
| Outcome | OK; builds stage 1/2 rustc, stage 1 library, checks stage 2 library |
To reproduce
Setup
$ rm -rf build/
$ tee config.toml <<EOF
profile = "library"
EOFInvocation
$ ./x check library --stage=2Assessment
Builds stage 1 rustc, stage 1 library, stage 2 rustc.
Checks stage 2 library.
Teardown
$ rm -rf build/profile = "library" + ./x check std
| Kind | |
|---|---|
| Description | Minimal library profile, check with std path |
| Invocation | ./x check std |
| Outcome | Builds stage 1 rustc; Checks stage 1 std |
To reproduce
Setup
$ rm -rf build/
$ tee config.toml <<EOF
profile = "library"
EOFInvocation
$ ./x check stdAssessment
Builds stage 1 rustc, checks stage 1 std
Teardown
$ rm -rf build/profile = "library" + ./x check core
| Kind | |
|---|---|
| Description | Minimal library profile, check core path |
| Invocation | ./x check core |
| Outcome | OK; builds stage 1 rustc, checks stage 1 core |
To reproduce
Setup
$ rm -rf build/
$ tee config.toml <<EOF
profile = "library"
EOFInvocation
$ ./x check coreAssessment
Builds stage 1 rustc, checks stage 1 core
Teardown
$ rm -rf build/profile = "library" + ./x check alloc
| Kind | |
|---|---|
| Description | Library profile, check alloc path |
| Invocation | ./x check alloc |
| Outcome | OK; builds stage 1 rustc, checks stage 1 alloc |
To reproduce
Setup
$ rm -rf build/
$ tee config.toml <<EOF
profile = "library"
EOFInvocation
$ ./x check allocAssessment
Builds stage 1 rustc, checks stage 1 alloc
Teardown
$ rm -rf build/Build flow
profile = "compiler" + ./x build library --stage=0
| Kind | |
|---|---|
| Description | Compiler profile, explicit stage 0 library |
| Invocation | ./x build library --stage=0 |
| Outcome | OK; no-op |
To reproduce
Setup
$ rm -rf build/
$ tee config.toml <<EOF
profile = "compiler"
EOFInvocation
$ ./x build library --stage=0Assessment
- No-op.
-build/host/stage0/bin/rustc -vVis beta rustc.
Teardown
$ rm -rf build/profile = "compiler" + ./x build library --stage=1
| Kind | |
|---|---|
| Description | Compiler profile, explicit stage 1 library |
| Invocation | ./x build library --stage=1 |
| Outcome | OK; builds stage 1 rustc/library, rustc binary OK |
To reproduce
Setup
$ rm -rf build/
$ tee config.toml <<EOF
profile = "compiler"
EOFInvocation
$ ./x build library --stage=1Assessment
- Builds stage 1 rustc, stage 1 library.
build/host/stage1/bin/rustc -vVOK.
Teardown
$ rm -rf build/profile = "compiler" + ./x build library --stage=2
| Kind | |
|---|---|
| Description | Minimal compiler profile, build explicit stage 2 library |
| Invocation | ./x build library --stage=2 |
| Outcome | OK; builds stage 1 rustc/library, builds stage 2 rustc, uplifts stage 2 rustc, stage 2 rustc OK |
To reproduce
Setup
$ rm -rf build/
$ tee config.toml <<EOF
profile = "compiler"
EOFInvocation
$ ./x build library --stage=2Assessment
- Builds stage 1 rustc, stage 1 library, stage 2 rustc, uplift stage 2 library.
build/host/stage2/bin/rustc -vVOK
Teardown
$ rm -rf build/Test flow
COMPILETEST_FORCE_STAGE0=1 ./x test run-make --stage=0
| Kind | |
|---|---|
| Description | Compiler profile, forced stage 0 run-make |
| Invocation | COMPILETEST_FORCE_STAGE0=1 ./x test run-make --stage=0 |
| Outcome | OK; expected amount of failures |
To reproduce
Setup
$ rm -rf build/
$ tee config.toml <<EOF
profile = "compiler"
EOFInvocation
$ COMPILETEST_FORCE_STAGE0=1 ./x test run-make --stage=0Assessment
OK, amount of failure is expected.
Teardown
$ rm -rf build/profile = "compiler" + ./x test --stage=2
| Kind | |
|---|---|
| Description | Compiler profile, full explicit stage 2 tests |
| Invocation | ./x test --stage=2 |
| Outcome | OK |
To reproduce
Setup
$ rm -rf build/
$ tee config.toml <<EOF
profile = "compiler"
EOFInvocation
$ ./x test --stage=2 --skip=tests/debuginfoNOTE: debuginfo tests were failing because of Python stuff, unrelated to change.
Assessment
Seems okay.
Teardown
$ rm -rf build/Build cache invalidation and spurious rebuilds
profile = "compiler" + ./x test mir-opt + ./x test mir-opt
| Kind | |
|---|---|
| Description | Compiler profile, test library twice |
| Invocation | ./x test mir-opt, ./x test mir-opt |
| Outcome | OK; no library rebuilds |
To reproduce
Setup
$ rm -rf build/
$ tee config.toml <<EOF
profile = "compiler"
EOFInvocation
$ `./x test mir-opt`Assessment
OK, no unnecessary library rebuilds
Teardown
$ rm -rf build/--keep-stage-std
profile = "compiler" + ./x build library --stage=1 + ./x build library --keep-stage-std=1
| Kind | |
|---|---|
| Description | Compiler profile, build stage 1 library, modify rustc, keep stage 1 std |
| Invocation | ./x build library --stage=1, ./x build library --keep-stage-std=1 |
| Outcome | OK; stage 1 library not rebuilt |
To reproduce
Setup
$ rm -rf build/
$ tee config.toml <<EOF
profile = "compiler"
EOFInvocation
$ ./x build library --stage=1
$ echo "// meow" >> compiler/rustc_driver/src/lib.rs
$ ./x build library --keep-stage-std=1Assessment
Stage 1 library not rebuilt
Teardown
$ rm -rf build/
$ git checkout -- .Cross-compile (build == host)
Cherry-picked sample:
- Build platform:
x86_64-unknown-linux-gnu - Host platform:
x86_64-unknown-linux-gnu - Target platform:
wasm32-unknown-unknown
| Kind | |
|---|---|
| Description | Compiler profile, cross build stage 1 to wasm32-u-u |
| Invocation | ./x build library |
| Outcome | OK; cross build successful |
To reproduce
Setup
Download [wasi-sdk 25 wasi-sdk-25.0-arm64-linux.tar.gz] and extract to ../wasi-sdk-24.0-x86_64-linux/.
$ rm -rf build/
$ tee config.toml <<EOF
profile = "compiler"
[build]
target = [
"wasm32-unknown-unknown",
]
[target."wasm32-unknown-unknown"]
wasi-root = "../wasi-sdk-24.0-x86_64-linux/"
EOFInvocation
$ ./x build libraryAssessment
OK; stage 1 library cross to wasm32-u-u.
Teardown
$ rm -rf build/
|
I think the merge CI will run the most comprehensive and detailed tests on this. |
|
Many things are not tested by CI, e.g. AFAIK we still do not test whether things like Miri work properly in stage 1 (might be stage 0 after this PR?). CI typically only tests stage 2, involving at least 2 compiler builds, which does not match local development builds. |
|
(I will also run a bunch of try builds here to check that they work and how long they take). @bors try @rust-timer queue |
This comment has been minimized.
This comment has been minimized.
rustbot
added
the
S-waiting-on-perf
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Apr 2, 2025
redesign stage 0 std ### Summary This PR changes how bootstrap builds the stage 1 compiler by switching to precompiled stage 0 standard library instead of building the in-tree one. The goal was to update bootstrap to use the beta standard library at stage 0 rather than compiling it from source (see the motivation at rust-lang/compiler-team#619). Previously, to build a stage 1 compiler bootstrap followed this path: ``` download stage0 compiler -> build in-tree std -> compile stage1 compiler with in-tree std ``` With this PR, the new path is: ``` download stage0 compiler -> compile stage1 compiler with precompiled stage0 std ``` This also means that `cfg(bootstrap)`/`cfg(not(bootstrap))` is no longer needed for library development. ### Building "library" Since stage0 `std` is no longer in-tree `x build/test/check library --stage 0` is now no-op. The minimum supported stage to build `std` is now 1. For the same reason, default stage values in the library profile is no longer 0. Because building the in-tree library now requires a stage1 compiler, I highly recommend library developers to enable `download-rustc` to speed up compilation time. <hr> If you encounter a bug or unexpected results please open a topic in the [#t-infra/bootstrap](https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap) Zulip channel or create a [bootstrap issue](https://github.com/rust-lang/rust/issues/new?template=bootstrap.md). (Review thread: https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Review.20thread.3A.20stage.200.20redesign.20PR/with/508271433) ~~Blocked on rust-lang#122709
Signed-off-by: onur-ozkan <[email protected]>
|
I can confirm that if you push while a try build is running, there won't be a "success" notification, and then the timing run will never start. |
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Apr 3, 2025
redesign stage 0 std ### Summary This PR changes how bootstrap builds the stage 1 compiler by switching to precompiled stage 0 standard library instead of building the in-tree one. The goal was to update bootstrap to use the beta standard library at stage 0 rather than compiling it from source (see the motivation at rust-lang/compiler-team#619). Previously, to build a stage 1 compiler bootstrap followed this path: ``` download stage0 compiler -> build in-tree std -> compile stage1 compiler with in-tree std ``` With this PR, the new path is: ``` download stage0 compiler -> compile stage1 compiler with precompiled stage0 std ``` This also means that `cfg(bootstrap)`/`cfg(not(bootstrap))` is no longer needed for library development. ### Building "library" Since stage0 `std` is no longer in-tree `x build/test/check library --stage 0` is now no-op. The minimum supported stage to build `std` is now 1. For the same reason, default stage values in the library profile is no longer 0. Because building the in-tree library now requires a stage1 compiler, I highly recommend library developers to enable `download-rustc` to speed up compilation time. <hr> If you encounter a bug or unexpected results please open a topic in the [#t-infra/bootstrap](https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap) Zulip channel or create a [bootstrap issue](https://github.com/rust-lang/rust/issues/new?template=bootstrap.md). (Review thread: https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Review.20thread.3A.20stage.200.20redesign.20PR/with/508271433) ~~Blocked on rust-lang#122709
|
We can just benchmark the existing build: @rust-timer build 5f2f74f But the second try build will be useful anyway, to see the cached CI duration :) |
This comment has been minimized.
This comment has been minimized.
|
Finished benchmarking commit (5f2f74f): comparison URL. Overall result: ✅ improvements - BENCHMARK(S) FAILEDBenchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf. @bors rollup=never ❗ ❗ ❗ ❗ ❗
❗ ❗ ❗ ❗ ❗ Instruction countThis is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.
Max RSS (memory usage)Results (primary 2.1%)This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.
CyclesResults (primary 2.8%, secondary 2.9%)This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.
Binary sizeThis benchmark run did not return any relevant results for this metric. Bootstrap: missing data |
rustbot
removed
the
S-waiting-on-perf
|
☀️ Try build successful - checks-actions |
Hmm, yes, a good reminder that I'll need to modify the bootstrap benchmark to be compatible with the new stage scheme. Now going to try some more try builds: @bors try |
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Apr 3, 2025
redesign stage 0 std ### Summary This PR changes how bootstrap builds the stage 1 compiler by switching to precompiled stage 0 standard library instead of building the in-tree one. The goal was to update bootstrap to use the beta standard library at stage 0 rather than compiling it from source (see the motivation at rust-lang/compiler-team#619). Previously, to build a stage 1 compiler bootstrap followed this path: ``` download stage0 compiler -> build in-tree std -> compile stage1 compiler with in-tree std ``` With this PR, the new path is: ``` download stage0 compiler -> compile stage1 compiler with precompiled stage0 std ``` This also means that `cfg(bootstrap)`/`cfg(not(bootstrap))` is no longer needed for library development. ### Building "library" Since stage0 `std` is no longer in-tree `x build/test/check library --stage 0` is now no-op. The minimum supported stage to build `std` is now 1. For the same reason, default stage values in the library profile is no longer 0. Because building the in-tree library now requires a stage1 compiler, I highly recommend library developers to enable `download-rustc` to speed up compilation time. <hr> If you encounter a bug or unexpected results please open a topic in the [#t-infra/bootstrap](https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap) Zulip channel or create a [bootstrap issue](https://github.com/rust-lang/rust/issues/new?template=bootstrap.md). (Review thread: https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Review.20thread.3A.20stage.200.20redesign.20PR/with/508271433) ~~Blocked on rust-lang#122709 try-job: dist-x86_64-linux try-job: dist-x86_64-msvc
|
⌛ Trying commit 34d5ce5 with merge 2d73d7b4868980a1500b60c43d1ce8a7fa0f294c... |
|
The job Click to see the possible cause of the failure (guessed by this bot) |
|
💔 Test failed - checks-actions |
bors
added
S-waiting-on-author
Something really weird is going on when running the stage0 dist tests using the following: |
|
☔ The latest upstream changes (presumably #137869) made this pull request unmergeable. Please resolve the merge conflicts. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR changes how bootstrap builds the stage 1 compiler by switching to precompiled stage 0 standard library instead of building the in-tree one. The goal was to update bootstrap to use the beta standard library at stage 0 rather than compiling it from source (see the motivation at rust-lang/compiler-team#619).
Previously, to build a stage 1 compiler bootstrap followed this path:
With this PR, the new path is:
This also means that
cfg(bootstrap)/cfg(not(bootstrap))is no longer needed for library development.Building "library"
Since stage0
stdis no longer in-treex build/test/check library --stage 0is now no-op. The minimum supported stage to buildstdis now 1. For the same reason, default stage values in the library profile is no longer 0.Because building the in-tree library now requires a stage1 compiler, I highly recommend library developers to enable
download-rustcto speed up compilation time.If you encounter a bug or unexpected results please open a topic in the #t-infra/bootstrap Zulip channel or create a bootstrap issue.
(Review thread: https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Review.20thread.3A.20stage.200.20redesign.20PR/with/508271433)
Blocked on #122709try-job: dist-x86_64-linux
try-job: dist-x86_64-msvc