Remove `Nonterminal` and `TokenKind::Interpolated` #124141

nnethercote · 2024-04-18T23:28:21Z

A third attempt at this; the first attempt was #96724 and the second was #114647.

rustbot · 2024-04-18T23:28:25Z

Some changes occurred in src/tools/rustfmt

cc @rust-lang/rustfmt

ijackson · 2024-05-10T15:28:11Z

❤️ @nnethercote for working on this. Thank you! I'm not sure if there's a way for me to help, as someone who doesn't really know much about the compiler innards, but please LMK if you think of something.

nnethercote · 2024-05-10T20:45:22Z

@ijackson: thanks! I'm curious why you are interested in this change, given that it's a compiler internals rearrangement?

nnethercote · 2024-05-10T20:47:31Z

@ijackson: Oh, I see, you are interested in #67062 being fixed. Unfortunately my current thoughts are that this PR alone won't be enough to fix that issue, though it's a necessary stepping stone.

ijackson · 2024-05-10T22:40:41Z

@ijackson: Oh, I see, you are interested in #67062 being fixed. Unfortunately my current thoughts are that this PR alone won't be enough to fix that issue, though it's a necessary stepping stone.

Right. It seems ... quite nontrivial. So, thanks.

oriongonza · 2024-05-12T21:49:53Z

After this is done TokenKind will become Copy right?

nnethercote · 2024-05-12T22:35:11Z

After this is done TokenKind will become Copy right?

Yes.

Instead of using AST pretty printing. This is a step towards removing `token::Interpolated`, which will eventually (in rust-lang#124141) be replaced with a token stream within invisible delimiters. This changes (improves) the output of the `stringify!` macro in some cases. This is allowed. As the `stringify!` docs say: "Note that the expanded results of the input tokens may change in the future. You should be careful if you rely on the output." Test changes: - tests/ui/macros/stringify.rs: this used to test both token stream pretty printing and AST pretty printing via different ways of invoking of `stringify!` (i.e. `$expr` vs `$tt`). But those two different invocations now give the same result, which is a nice consistency improvement. This removes the need for the `c2!` macro. - tests/ui/macros/trace_faulty_macros.rs: there is some sub-optimal spacing in the printing of `A { a : a, b : 0, c : _, .. }`, which will be fixed in the next commit. The spacing of `1+1` improves -- it now matches the formatting in the source code. - tests/ui/proc-macro/*: minor improvements where small differences between `INPUT (DISPLAY)` output and `DEEP-RE-COLLECTED (DISPLAY)` output disappear.

…, r=<try> Print `token::Interpolated` with token stream pretty printing. This is a step towards removing `token::Interpolated` (rust-lang#124141). It unavoidably changes the output of the `stringify!` macro, generally for the better. r? `@petrochenkov`

nnethercote · 2024-05-17T06:47:27Z

#125174 carves off a piece of this PR so it can be merged separately.

Add tests for `-Zunpretty=expanded` ported from stringify's tests This PR adds a new set of tests for the AST pretty-printer. Previously, pretty-printer edge cases were tested by way of `stringify!` in [tests/ui/macros/stringify.rs](https://github.com/rust-lang/rust/blob/1.78.0/tests/ui/macros/stringify.rs), such as the tests added by rust-lang@419b269 and rust-lang@527e2ea. Those tests will no longer provide effective coverage of the AST pretty-printer after rust-lang#124141. `Nonterminal` and `TokenKind::Interpolated` are being removed, and a consequence is that `stringify!` will perform token stream pretty printing, instead of AST pretty printing, in all of the `stringify!` cases including $:expr and all other interpolations. This PR adds 2 new ui tests with `compile-flags: -Zunpretty=expanded`: - **tests/ui/unpretty/expanded-exhaustive.rs** — this test aims for exhaustive coverage of all the variants of `ExprKind`, `ItemKind`, `PatKind`, `StmtKind`, `TyKind`, and `VisibilityKind`. Some parts could use being fleshed out further, but the current state is roughly on par with what exists in the old stringify-based tests. - **tests/ui/unpretty/expanded-interpolation.rs** — this test covers tricky macro metavariable edge cases that require the AST pretty printer to synthesize parentheses in order for the printed code to be valid Rust syntax. r? `@nnethercote`

Rollup merge of rust-lang#125236 - dtolnay:expandtest, r=nnethercote Add tests for `-Zunpretty=expanded` ported from stringify's tests This PR adds a new set of tests for the AST pretty-printer. Previously, pretty-printer edge cases were tested by way of `stringify!` in [tests/ui/macros/stringify.rs](https://github.com/rust-lang/rust/blob/1.78.0/tests/ui/macros/stringify.rs), such as the tests added by rust-lang@419b269 and rust-lang@527e2ea. Those tests will no longer provide effective coverage of the AST pretty-printer after rust-lang#124141. `Nonterminal` and `TokenKind::Interpolated` are being removed, and a consequence is that `stringify!` will perform token stream pretty printing, instead of AST pretty printing, in all of the `stringify!` cases including $:expr and all other interpolations. This PR adds 2 new ui tests with `compile-flags: -Zunpretty=expanded`: - **tests/ui/unpretty/expanded-exhaustive.rs** — this test aims for exhaustive coverage of all the variants of `ExprKind`, `ItemKind`, `PatKind`, `StmtKind`, `TyKind`, and `VisibilityKind`. Some parts could use being fleshed out further, but the current state is roughly on par with what exists in the old stringify-based tests. - **tests/ui/unpretty/expanded-interpolation.rs** — this test covers tricky macro metavariable edge cases that require the AST pretty printer to synthesize parentheses in order for the printed code to be valid Rust syntax. r? `@nnethercote`

Instead of using AST pretty printing. This is a step towards removing `token::Interpolated`, which will eventually (in rust-lang#124141) be replaced with a token stream within invisible delimiters. This changes (improves) the output of the `stringify!` macro in some cases. This is allowed. As the `stringify!` docs say: "Note that the expanded results of the input tokens may change in the future. You should be careful if you rely on the output." Test changes: - tests/ui/macros/stringify.rs: this used to test both token stream pretty printing and AST pretty printing via different ways of invoking of `stringify!` (i.e. `$expr` vs `$tt`). But those two different invocations now give the same result, which is a nice consistency improvement. This removes the need for all the `c2*` macros. The AST pretty printer now has more thorough testing thanks to rust-lang#125236. - tests/ui/proc-macro/*: minor improvements where small differences between `INPUT (DISPLAY)` output and `DEEP-RE-COLLECTED (DISPLAY)` output disappear.

petrochenkov · 2024-05-23T20:01:24Z

It's great to see that enum InvisibleOrigin allows to migrate the parser to delimited groups relatively simply, with just the maybe_whole to maybe_reparse_metavar_seq replacement.

Of course it prevents a lot of interesting stuff like reparsing expr as pat and similar, like it would work in a purely token-based model, but all that can be carefully introduced later, when it's possible to do backward compatibly.

petrochenkov · 2024-05-23T20:02:23Z

How hard would it be to get this to a perf run?
(With or without the NtExpr/NtLiteral stuff.)

petrochenkov · 2024-05-23T20:03:04Z

Blocked on #125174.
@rustbot blocked

…try> Remove `NtExpr` and `NtLiteral` The next part of rust-lang#124141. r? `@petrochenkov`

…etrochenkov Remove `NtExpr` and `NtLiteral` The next part of rust-lang#124141. r? `@petrochenkov`

This is temporarily needed for `x doc compiler` to work. They can be removed once the `Nonterminal` is removed (rust-lang#124141).

…etrochenkov Remove `NtExpr` and `NtLiteral` The next part of rust-lang#124141. r? `@petrochenkov`

`NtBlock` is the last remaining variant of `Nonterminal`, so once it is gone then `Nonterminal` can be removed as well.

They are no longer needed. This does slightly worsen the error message for a single test, but that test contains code that is so badly broken that I'm not worried about it.

These are no longer needed now that `Nonterminal` is gone.

rustbot · 2025-04-02T05:29:38Z

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

Some changes occurred to the CTFE machinery

cc @RalfJung, @oli-obk, @lcnr

Some changes occurred in compiler/rustc_sanitizers

cc @rust-lang/project-exploit-mitigations, @rcvalle

Some changes occurred in compiler/rustc_codegen_ssa

cc @WaffleLapkin

nnethercote · 2025-04-02T05:33:00Z

#133436, #137517, #138083, and #138478 have all been merged, removing all Nonterminal variants except for NtBlock. The remaining commits in this PR now remove NtBlock plus some cleaning up.

@bors try @rust-timer queue

bors · 2025-04-02T05:34:12Z

⌛ Trying commit 1830245 with merge df3277b...

…Kind-Interpolated, r=<try> Remove `Nonterminal` and `TokenKind::Interpolated` A third attempt at this; the first attempt was rust-lang#96724 and the second was rust-lang#114647. r? `@ghost`

bors · 2025-04-02T07:40:55Z

☀️ Try build successful - checks-actions
Build commit: df3277b (df3277bc867c119b5867d21eb2431db22e13c439)

rust-timer · 2025-04-02T09:21:02Z

Finished benchmarking commit (df3277b): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

	mean	range	count
Regressions ❌ (primary)	1.1%	[0.3%, 2.0%]	10
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.5%	[-1.8%, -0.1%]	65
Improvements ✅ (secondary)	-0.7%	[-2.5%, -0.1%]	48
All ❌✅ (primary)	-0.3%	[-1.8%, 2.0%]	75

Max RSS (memory usage)

Results (primary 0.8%, secondary 4.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.8%	[1.6%, 2.1%]	2
Regressions ❌ (secondary)	4.0%	[3.0%, 4.9%]	2
Improvements ✅ (primary)	-1.2%	[-1.2%, -1.2%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.8%	[-1.2%, 2.1%]	3

Cycles

Results (primary -0.0%, secondary 7.6%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.3%	[1.3%, 1.3%]	1
Regressions ❌ (secondary)	7.6%	[4.0%, 11.2%]	10
Improvements ✅ (primary)	-1.3%	[-1.3%, -1.3%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.0%	[-1.3%, 1.3%]	2

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 774.398s -> 774.46s (0.01%)
Artifact size: 365.94 MiB -> 365.54 MiB (-0.11%)

petrochenkov · 2025-04-02T12:51:02Z

compiler/rustc_ast/src/tokenstream.rs

-                DelimSpacing::new(Spacing::JointHidden, spacing),
-                Delimiter::Invisible(InvisibleOrigin::FlattenToken),
-                TokenStream::token_alone(token::Lifetime(ident.name, is_raw), ident.span),
-            ),


I'm thinking about implications of this change.
The flattening operation was performed was performed in

AST lowering

Attribute lowering (related to the recent attribute changes)

cfg_eval too, but that should not be relevant to NtIdent/NtLifetime

This means NtIdent/NtLifetime can now appear in HIR, and then encoded to metadata from there and then decoded.
Which is good? Because now we preserve both inner and outer spans of these tokens in cross-crate scenarios, while previously we lost the outer spans.

Is code working with HIR and lowered attributes ready to encounter the new tokens?
The MetaItem parsing logic was ready, I expect the new HIR attribute parsing logic replacing it to also be ready.
Declarative macro expansion logic working on tokens decoded from metadata should be ready.
Besides that nobody should parse tokens after HIR lowering.

NtIdent/NtLifetime still cannot appear in proc macros though, the logic in fn from_internal in proc macro server is independent from flatten_token, but equivalent to it.
It also keeps and uses only the inner spans and wraps NtLifetime into a group.

So overall this should be ok, I think.

Was the flattening change included into the previous crater run?
It may affect cross-crate edition checks.

It was. This was the revision tested by the crater run, and one of its parents was the flattening removal.

petrochenkov · 2025-04-02T12:52:54Z

r=me after answering #124141 (comment).
@rustbot author

rustbot · 2025-04-02T12:52:56Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

nnethercote · 2025-04-02T23:29:35Z

I will wait a bit before merging this. I want to see if there is any fallout from #138478 first, because that was a big and complicated change.

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Apr 18, 2024

nnethercote marked this pull request as draft April 18, 2024 23:28

nnethercote mentioned this pull request Apr 18, 2024

Start removing Nonterminal #114647

Closed

nnethercote force-pushed the rm-Nonterminal-and-TokenKind-Interpolated branch 2 times, most recently from 42a623a to c133e16 Compare April 23, 2024 03:51

petrochenkov self-assigned this Apr 28, 2024

nnethercote force-pushed the rm-Nonterminal-and-TokenKind-Interpolated branch from c133e16 to 7aef5db Compare May 16, 2024 10:50

This comment has been minimized.

Sign in to view

nnethercote mentioned this pull request May 17, 2024

Print token::Interpolated with token stream pretty printing. #125174

Merged

This comment was marked as resolved.

Sign in to view

dtolnay mentioned this pull request May 18, 2024

Add tests for -Zunpretty=expanded ported from stringify's tests #125236

Merged

rustbot removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 23, 2024

nnethercote mentioned this pull request Mar 17, 2025

Remove NtExpr and NtLiteral #138478

Merged

This comment was marked as resolved.

Sign in to view

bors added a commit to rust-lang-ci/rust that referenced this pull request Mar 24, 2025

Auto merge of rust-lang#138478 - nnethercote:rm-NtExpr-NtLiteral, r=<…

2f5db75

…try> Remove `NtExpr` and `NtLiteral` The next part of rust-lang#124141. r? `@petrochenkov`

nnethercote mentioned this pull request Mar 26, 2025

Removal of NtTy resulted in regression #138975

Open

bors added a commit to rust-lang-ci/rust that referenced this pull request Mar 28, 2025

Auto merge of rust-lang#138478 - nnethercote:rm-NtExpr-NtLiteral, r=p…

8eca53e

…etrochenkov Remove `NtExpr` and `NtLiteral` The next part of rust-lang#124141. r? `@petrochenkov`

bors added a commit to rust-lang-ci/rust that referenced this pull request Mar 28, 2025

Auto merge of rust-lang#138478 - nnethercote:rm-NtExpr-NtLiteral, r=p…

de2d1da

…etrochenkov Remove `NtExpr` and `NtLiteral` The next part of rust-lang#124141. r? `@petrochenkov`

bors added a commit to rust-lang-ci/rust that referenced this pull request Mar 28, 2025

Auto merge of rust-lang#138478 - nnethercote:rm-NtExpr-NtLiteral, r=p…

4a67d7d

…etrochenkov Remove `NtExpr` and `NtLiteral` The next part of rust-lang#124141. r? `@petrochenkov`

jdonszelmann pushed a commit to jdonszelmann/rust that referenced this pull request Mar 31, 2025

Increase recursion_limit in numerous crates.

d6bd24c

This is temporarily needed for `x doc compiler` to work. They can be removed once the `Nonterminal` is removed (rust-lang#124141).

bors added a commit to rust-lang-ci/rust that referenced this pull request Apr 2, 2025

Auto merge of rust-lang#138478 - nnethercote:rm-NtExpr-NtLiteral, r=p…

70dab5a

…etrochenkov Remove `NtExpr` and `NtLiteral` The next part of rust-lang#124141. r? `@petrochenkov`

nnethercote added 4 commits April 2, 2025 16:07

Remove NtBlock, Nonterminal, and TokenKind::Interpolated.

bb495d6

`NtBlock` is the last remaining variant of `Nonterminal`, so once it is gone then `Nonterminal` can be removed as well.

Impl Copy for Token and TokenKind.

4d8f757

Remove TokenStream::flattened and InvisibleOrigin::FlattenToken.

4c0cbae

They are no longer needed. This does slightly worsen the error message for a single test, but that test contains code that is so badly broken that I'm not worried about it.

Remove recursion_limit increases.

1830245

These are no longer needed now that `Nonterminal` is gone.

nnethercote force-pushed the rm-Nonterminal-and-TokenKind-Interpolated branch from c70491b to 1830245 Compare April 2, 2025 05:29

rustbot added A-query-system Area: The rustc query system (https://rustc-dev-guide.rust-lang.org/query.html) PG-exploit-mitigations Project group: Exploit mitigations labels Apr 2, 2025

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 2, 2025

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 2, 2025

petrochenkov reviewed Apr 2, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

nnethercote commented Apr 18, 2024 •

edited

Loading

rustbot commented Apr 18, 2024

ijackson commented May 10, 2024

nnethercote commented May 10, 2024

nnethercote commented May 10, 2024

ijackson commented May 10, 2024

oriongonza commented May 12, 2024

nnethercote commented May 12, 2024

This comment has been minimized.

nnethercote commented May 17, 2024 •

edited

Loading

This comment was marked as resolved.

petrochenkov commented May 23, 2024 •

edited

Loading

petrochenkov commented May 23, 2024

petrochenkov commented May 23, 2024

This comment was marked as resolved.

rustbot commented Apr 2, 2025

nnethercote commented Apr 2, 2025

This comment has been minimized.

bors commented Apr 2, 2025

bors commented Apr 2, 2025

This comment has been minimized.

rust-timer commented Apr 2, 2025

petrochenkov Apr 2, 2025

petrochenkov Apr 2, 2025

nnethercote Apr 2, 2025

petrochenkov commented Apr 2, 2025

rustbot commented Apr 2, 2025

nnethercote commented Apr 2, 2025

Remove Nonterminal and TokenKind::Interpolated #124141

Are you sure you want to change the base?

Remove Nonterminal and TokenKind::Interpolated #124141

Conversation

nnethercote commented Apr 18, 2024 • edited Loading

rustbot commented Apr 18, 2024

ijackson commented May 10, 2024

nnethercote commented May 10, 2024

nnethercote commented May 10, 2024

ijackson commented May 10, 2024

oriongonza commented May 12, 2024

nnethercote commented May 12, 2024

This comment has been minimized.

nnethercote commented May 17, 2024 • edited Loading

This comment was marked as resolved.

petrochenkov commented May 23, 2024 • edited Loading

petrochenkov commented May 23, 2024

petrochenkov commented May 23, 2024

This comment was marked as resolved.

rustbot commented Apr 2, 2025

nnethercote commented Apr 2, 2025

This comment has been minimized.

bors commented Apr 2, 2025

bors commented Apr 2, 2025

This comment has been minimized.

rust-timer commented Apr 2, 2025

Overall result: ❌✅ regressions and improvements - please read the text below

Instruction count

Max RSS (memory usage)

Cycles

Binary size

petrochenkov Apr 2, 2025

Choose a reason for hiding this comment

petrochenkov Apr 2, 2025

Choose a reason for hiding this comment

nnethercote Apr 2, 2025

Choose a reason for hiding this comment

petrochenkov commented Apr 2, 2025

rustbot commented Apr 2, 2025

nnethercote commented Apr 2, 2025

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

nnethercote commented Apr 18, 2024 •

edited

Loading

nnethercote commented May 17, 2024 •

edited

Loading

petrochenkov commented May 23, 2024 •

edited

Loading