Temporary lifetime extension for blocks

[dianne]

NB: The cleanups and refactors from this are currently being split into smaller PRs.

• cleanup: merge RvalueScopes into ScopeTree #148658

TODO: this needs an edition guide PR to update the chapter on block tail temporary scopes

This implements a revised version of the temporary lifetime extension semantics I suggested in #145838 (comment), with the goal of making temporary lifetimes and drop order more consistent between extending and non-extending blocks. As a consequence, this undoes the breaking change introduced by #145838 (but in exchange has a much larger surface area).

The change this PR hopes to enforce is a general rule: any expression's temporaries should have the same relative drop order regardless of whether the expression is in an extending context or not: let _ = $expr; and drop($expr); should have the same drop order. To achieve that, this PR applies lifetime extension rules bottom-up, starting from borrow expressions and super let, rather than top-down from let statements and consts/statics. For example:

// This `temp()` is now extended past the block tail in all contexts.
{ &temp() }

now extends the lifetime of temp() to outlive the block tail in Rust 2024 regardless of whether the block is an extending expression in a let statement initializer (in which context it was already extended to outlive the block before this PR). The scoping rules for tails of extending blocks remain the same: extending subexpressions' temporary scopes are extended based on the source of the lifetime extension (e.g. to match the scope of a parent let statement's bindings). For blocks not extended by any other source, extending borrows in the tail expression now share a temporary scope with the result of the block. This can in turn extend nested blocks within blocks' tail expressions:

// This `temp()` is extended past the outer block tail.
// It is now dropped after the reference to it at the `;`.
f({{ &temp() }});

// This context-sensitivity is consistent with `let`:
// This `temp()` was already extended.
// It is still dropped after `x` at the end of its scope.
let x = {{ &temp() }};

Since this uses the same rules as let, it only applies to extending sub-expressions.

// This `temp()` is still never extended in any context.
// In Rust 2024, it is dropped at the end of the block tail.
{ identity(&temp()) }

This also applies to if expressions' blocks and to match arms in all editions, since lifetime extension applies to both of them as well and they both drop their non-extended temporaries. This is where breakage from #145838 was observed:

if cond { &temp() } else { &temp() }

now extends temp() to have the same temporary scope as the result of the if expression.

As a further consequence, this makes super let in if expressions' blocks more consistent with block expressions:

if cond() {
    super let x = temp();
    &temp
} else {
    super let x = temp();
    &temp
}

previously only worked in extending contexts (since the super lets would be extended), and now it works everywhere.

Reference PR: rust-lang/reference#2051

@rustbot label +T-lang

[dianne]

@rustbot label -stable-nominated

I'm not intending to stable-nominate this, at least. Someone else can, but I don't expect it's needed or that it would be accepted.

[traviscross]

Does this only affect code in Rust 2024, or would you expect any visible difference in earlier editions?

[dianne]

It should only be visible in Rust 2024. The only extending expressions that introduce temporary drop scopes are Rust 2024 block tail expressions. Edit: this is also visible on earlier editions through if expressions' blocks.

Suppose we have a macro extending!, for which $expr is extending if extending!($expr) is extending. Under this PR, in a non-extending context, { extending!(&temp()) } would give temp() the same temporary scope as the result of the block. Prior to Rust 2021, they're already in the same scope, due to extending! being unable to introduce temporary scopes.

Or to generalize this, the aim of this PR is that in a non-extending context, extending!(&temp()) should give temp() the same temporary scope as the expansion, similar to how let x = extending!(&temp()); gives temp() the same scope as x. This already holds in Rust 2021 and prior.

If new expressions are added to Rust that are both extending and temporary scopes, I'd want this behavior to apply to them as well.

[traviscross]

Since this would effectively reduce the scope of the Rust 2024 tail expression temporary scope change, we'd also want to be sure to reflect that in the behavior of the tail-expr-drop-order lint.

[dianne]

I haven't done extensive testing, but see this test diff for that lint: lint-tail-expr-drop-order-borrowck.rs. I'm applying the lifetime extension rules on all editions, and lifetime extension prevents the temporary scope from being registered as potentially forwards-incompatible (even though the extended scopes are technically the same as the old scopes in old editions). Though I think I've convinced myself at this point that lifetime extension doesn't need to be applied to block tails of non-extending old-edition blocks¹, so potentially the lint change could be implemented in some other way instead.

Footnotes

1. I was worried about mixed-edition code, but I don't think it's an issue anymore. ↩

[bors]

☔ The latest upstream changes (presumably #146666) made this pull request unmergeable. Please resolve the merge conflicts.

[dianne]

I've made some revisions. This should now properly handle if expressions' blocks, meaning it affects all editions (since if blocks are both terminating in all editions and extending when the if expression is extending). Of note, I didn't notice at the time, but I think #145838 affected all editions as well (including the real-world breakage), due to if blocks working like that.

I think the implementation will likely need optimization and cleanup, but it might take a bit of refactoring to get it to a good place, so I'd like to get a vibe check on the design first, if there's room for it in a lang team meeting.

@rustbot label +I-lang-nominated

[rustbot]

r? @nnethercote

rustbot has assigned @nnethercote.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[rust-timer]

Finished benchmarking commit (9f93af2): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.2%, 0.4%]	2
Regressions ❌ (secondary)	0.6%	[0.6%, 0.6%]	3
Improvements ✅ (primary)	-0.4%	[-0.9%, -0.2%]	5
Improvements ✅ (secondary)	-0.1%	[-0.3%, -0.1%]	9
All ❌✅ (primary)	-0.2%	[-0.9%, 0.4%]	7

Max RSS (memory usage)

Results (primary -1.8%, secondary 1.2%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	1.8%	[1.8%, 1.8%]	1
Regressions ❌ (secondary)	3.2%	[1.7%, 5.5%]	10
Improvements ✅ (primary)	-3.0%	[-3.6%, -2.1%]	3
Improvements ✅ (secondary)	-2.6%	[-3.6%, -1.9%]	5
All ❌✅ (primary)	-1.8%	[-3.6%, 1.8%]	4

Cycles

Results (primary 3.4%, secondary 2.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	3.4%	[3.4%, 3.4%]	1
Regressions ❌ (secondary)	5.2%	[1.8%, 10.7%]	9
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-3.8%	[-5.0%, -2.6%]	5
All ❌✅ (primary)	3.4%	[3.4%, 3.4%]	1

Binary size

Results (primary 0.0%, secondary 0.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.1%	[0.0%, 0.3%]	3
Regressions ❌ (secondary)	0.0%	[0.0%, 0.0%]	2
Improvements ✅ (primary)	-0.1%	[-0.1%, -0.0%]	2
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.0%	[-0.1%, 0.3%]	5

Bootstrap: 474.243s -> 475.297s (0.22%)
Artifact size: 390.98 MiB -> 390.72 MiB (-0.07%)

[dianne]

I'd have to profile this and maybe also split out the refactors to get a better idea of what's going on with perf, but my guesses are:

• The refactors probably introduced some perf improvements. In particular, getting rid of RvalueCandidate cut out some unnecessary work.
• Since I don't see the cranelift-codegen (debug) and cargo (opt) regressions there anymore, it's possible that Do not lifetime-extend array/slice indices #147083 did its job and the regressions in codegen were indeed from imprecise array/slice index lifetimes. Or maybe they were balanced out by improvements; e.g. cargo (opt) is showing up as improved. We'd probably have to profile the refactors separately to know for sure.
• It's likely the final commit regressed region_scope_tree and/or thir_body because I didn't want to be too clever with hash map usage before profiling; I expect there's a bunch of unnecessary lookups/insertions. There being a couple check regressions supports this. I know how to improve it if that's indeed the cause, though.

I'll work on getting this PR split up and on that idea to hopefully improve the final commit's perf.

[dianne]

@bors try @rust-timer queue

After the perf run I'll clean up the commit history and start splitting out the refactors into separate PRs.

[rust-bors]

☀️ Try build successful (CI)
Build commit: c2e32f1 (c2e32f1c9652b13ed99608599c1e855462f421f3, parent: c90bcb9571b7aab0d8beaa2ce8a998ffaf079d38)

[rust-timer]

Finished benchmarking commit (c2e32f1): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.4%	[0.0%, 0.6%]	4
Improvements ✅ (primary)	-0.3%	[-0.9%, -0.1%]	6
Improvements ✅ (secondary)	-0.2%	[-0.4%, -0.0%]	7
All ❌✅ (primary)	-0.3%	[-0.9%, -0.1%]	6

Max RSS (memory usage)

Results (primary 0.5%, secondary -2.7%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.9%	[1.0%, 4.8%]	2
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-1.9%	[-3.2%, -0.7%]	2
Improvements ✅ (secondary)	-2.7%	[-3.5%, -2.2%]	5
All ❌✅ (primary)	0.5%	[-3.2%, 4.8%]	4

Cycles

Results (secondary 4.6%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	4.6%	[4.6%, 4.6%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

Results (primary 0.0%, secondary 0.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.1%	[0.0%, 0.3%]	3
Regressions ❌ (secondary)	0.0%	[0.0%, 0.0%]	1
Improvements ✅ (primary)	-0.1%	[-0.1%, -0.0%]	2
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.0%	[-0.1%, 0.3%]	5

Bootstrap: 475.185s -> 478.589s (0.72%)
Artifact size: 390.81 MiB -> 390.83 MiB (0.00%)

[dianne]

That might be enough of an improvement over the last run to justify the tiny bit of added complexity? I'm not totally sure what the right tradeoffs are there, but for now I'm inclined to go with that latest version since it fixes the instruction count regressions in realistic code (and in particular should make the incremental cache smaller). I've simplified the commit history to make that version of things a bit more natural and added/improved some comments.

The deep_vec regressions are still present, unfortunately. It looks like it spends a lot of time in thir_body, so likely it's sensitive to the cleanups there. Hopefully splitting them out will give a better idea of what's actually going on.

[dianne]

Looks like flakiness? I'll retry CI. By coincidence, the error message is a great reminder that the edition guide's chapter on Rust 2024 block tail temporary scopes will need updating too. I'll make a note in the PR text so it doesn't slip by.

Edit: Just noticed another PR failed due to an npm install error too. Looks like it's #t-infra > CI keeps failing because of npm error.

[craterbot]

🚧 Experiment pr-146098-6 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more