Implement RFC 3943: MIR move elimination by Amanieu · Pull Request #157943 · rust-lang/rust

Amanieu · 2026-06-15T22:42:55Z

This PR implements rust-lang/rfcs#3943: a MIR optimization pass that eliminates unnecessary copies. Since the new pass relies on the new MIR semantics from RFC 3943, it is gated behind the the -Z mir-move-elimination flag. Enabling this flag also disables the DestinationPropagation pass, which is completely superseded by this one.

There are 3 main parts to this optimization:

PreciseLiveness is an analysis that calculates, at a sub-statement granularity, the points in a function where a local requires storage to be allocated. This is more fine-grained than MaybeStorageLive, and takes borrows into account.
TailCopyToMove is a small pre-processing pass which turns copies into the return place just before a return into moves. This is necessary to allow the source of the copy to be unified with the return place.
MoveElimination is the main pass which optimizes away moves through place unification.

The RFC text and the top-level comments for the various passes have more details on the internals of the optimization.

r? @dianqk

cc @rust-lang/wg-mir-opt

rustbot · 2026-06-15T22:42:59Z

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

This PR changes MIR

cc @oli-obk, @RalfJung, @JakobDegen, @vakaras

This PR changes rustc_public

cc @oli-obk, @celinval, @ouz-a, @makai410

rustbot · 2026-06-15T22:43:01Z

dianqk is currently at their maximum review capacity.
They may take a while to respond.

cjgillot

Can part of the code, in particular the liveness analysis be shared with the existing DestinationPropagation? Or is this meant to supersede it completely?

View changes since this review

cjgillot · 2026-06-21T17:11:21Z

+    tcx: TyCtxt<'tcx>,
+    body: &mir::Body<'tcx>,
+    pass_name: Option<&'static str>,
+) -> (Vec<(Local, Location)>, IndexVec<BasicBlock, DenseBitSet<Local>>) {


BitMatrix<BasicBlock, Local>?

In PreciseLiveness::apply_block_start_effect we need to have a DenseBitSet<Local> to call .intersect on, which BitMatrix doesn't expose.

cjgillot · 2026-06-21T17:40:19Z

+        // Notably this kills any dead results produced by a predecessor's terminator.
+        state.intersect(&live_on_entry[block]);
+
+        for local in state.iter() {
+            builder.gen_(local, points.entry_point(block), SplitPointEffect::Early);
+        }
+
+        for (statement_index, statement) in block_data.statements.iter().enumerate() {
+            let location = Location { block, statement_index };
+            let point = points.point_from_location(location);
+
+            // StorageDead always kills a local, even if it has been borrowed.
+            if let mir::StatementKind::StorageDead(local) = statement.kind {
+                builder.kill(local, point, SplitPointEffect::Late);
+                continue;
+            }
+
+            // Kill moved operands if the whole local was moved.
+            VisitPlacesWith(|place: Place<'tcx>, ctxt| {
+                if ctxt == PlaceContext::NonMutatingUse(NonMutatingUseContext::Move) {
+                    if let Some(local) = place.as_local() {
+                        builder.kill(local, point, SplitPointEffect::Early);
+                    }
+                }
+            })
+            .visit_statement(statement, location);
+
+            // Kill any locals which are no longer used after this statement.
+            for &(local, _) in kill_point_map[point] {
+                builder.kill(local, point, SplitPointEffect::Early);
+            }
+
+            // Gen destination places.
+            VisitPlacesWith(|place: Place<'tcx>, ctxt| match DefUse::for_place(place, ctxt) {
+                DefUse::Def | DefUse::PartialWrite => {
+                    builder.gen_(place.local, point, SplitPointEffect::Late)
+                }
+                DefUse::Use | DefUse::NonUse => {}
+            })
+            .visit_statement(statement, location);
+
+            // Kill any dead destination places: they will only appear at
+            // the late point of the statement they are generated in, which is
+            // sufficient for determining overlap.
+            for &(local, _) in kill_point_map[point] {
+                builder.kill(local, point, SplitPointEffect::Late);
+            }
+        }
+
+        let location = Location { block, statement_index: block_data.statements.len() };
+        let point = points.point_from_location(location);
+        let terminator = block_data.terminator();
+
+        // Kill moved operands if the whole local was moved. Also kill dropped
+        // places if the entire local was dropped.
+        VisitPlacesWith(|place: Place<'tcx>, ctxt| {
+            if let PlaceContext::NonMutatingUse(NonMutatingUseContext::Move)
+            | PlaceContext::MutatingUse(MutatingUseContext::Drop) = ctxt
+            {
+                if let Some(local) = place.as_local() {
+                    builder.kill(local, point, SplitPointEffect::Early);
+                }
+            }
+        })
+        .visit_terminator(terminator, location);
+
+        // Kill any locals which are no longer used after this terminator.
+        for &(local, _) in kill_point_map[point] {
+            builder.kill(local, point, SplitPointEffect::Early);
+        }
+
+        // Gen destination places.
+        VisitPlacesWith(|place: Place<'tcx>, ctxt| match DefUse::for_place(place, ctxt) {
+            DefUse::Def | DefUse::PartialWrite => {
+                builder.gen_(place.local, point, SplitPointEffect::Late)
+            }
+            DefUse::Use | DefUse::NonUse => {}
+        })
+        .visit_terminator(terminator, location);
+
+        // Move arguments to a call are treated specially: the place that they
+        // represent is passed directly to the callee, which means that they are
+        // not allowed to alias any other move operand or the destination place.
+        // This is represented here by extending their live range to the late
+        // part, making it overlap with that of the destination place.
+        //
+        // Notably, this *doesn't* apply to TailCall.
+        if let mir::TerminatorKind::Call {
+            func: _,
+            args,
+            destination: _,
+            target: _,
+            unwind: _,
+            call_source: _,
+            fn_span: _,
+        } = &terminator.kind
+        {
+            for arg in args {
+                if let mir::Operand::Move(place) = arg.node {
+                    builder.gen_(place.local, point, SplitPointEffect::Late);
+                    builder.kill(place.local, point, SplitPointEffect::Late);
+                }
+            }
+        }


This duplicates a lot of code in impl Analysis for PreciseLiveness. Any way to deduplicate?

Not easily: the two are sufficiently different that I'm not sure the code would be clearer if we tried to deduplicate. The PreciseLiveness analysis is simpler since it only needs to determine the liveness at the end of the statement. On the other hand matrix construction needs to determine the liveness both at the mid point of the instruction and after it, and record both in the matrix.

cjgillot · 2026-06-21T17:52:23Z

+            trace!("cannot unify {a:?} and {b:?} involving a rust-call tuple argument");
+            return None;
+        }
+


Can we remove a common projection tail from a and b before this match? Allowing to merge _1.foo.bar with _2.blah.foo.bar?

We could, but that's not really useful in practice if you consider the kind of patterns that trigger this optimization. Specifically, in order for the lifetimes to not overlap the RHS of an assignment needs to end its lifetime at the assignment. This only happens for a move of a whole local (no projections) or for non-borrowed locals where this is its last use.

cjgillot · 2026-06-21T17:59:37Z

+    fn visit_aggregate_assign(
+        &mut self,
+        dest: Place<'tcx>,
+        project_field: impl Fn(TyCtxt<'tcx>, Place<'tcx>, FieldIdx, Ty<'tcx>) -> Place<'tcx>,


Should the callback return a &[PlaceElem<'tcx>] and call project_deeper here?

Returning a slice is awkward because there's no good lifetime to give it.

cjgillot · 2026-06-21T19:16:53Z

+            // Under the local lifetime semantics from RFC 3943, `StorageLive` does not allocate,
+            // and `StorageDead` has no effect if the local was already freed by a move. These
+            // markers therefore do not affect whether a copy can be treated as a final use.
+            StatementKind::StorageLive(_) | StatementKind::StorageDead(_) | StatementKind::Nop => {}


Why can't StorageDead(l) perform a used_after.remove(l)?

This is a backwards analysis where used_after tracks any local that is used later on the current path. We only ever add to used_after as we walk backwards through the function, and never remove from it.

Amanieu · 2026-06-22T13:59:33Z

Can part of the code, in particular the liveness analysis be shared with the existing DestinationPropagation? Or is this meant to supersede it completely?

This optimization is meant to supersede DestinationPropagation entirely, which is why -Z mir-move-elimination disables DestinationPropagation. Also PreciseLiveness is based on the new MIR semantics in RFC 3943 which hasn't been accepted, which technically makes this MIR optimization unsound at the moment.

Separately, I think there may be value in having the coroutine pass re-use PreciseLiveness to determine the set of locals live across yields.

rustbot · 2026-06-23T00:08:05Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

dianqk · 2026-06-27T14:53:00Z

+        VisitPlacesWith(|place: Place<'tcx>, ctxt| {
+            if let PlaceContext::NonMutatingUse(NonMutatingUseContext::Move) = ctxt {
+                if let Some(local) = place.as_local() {
+                    builder.kill(local, point, SplitPointEffect::Early);


Question: Should we just treat the call arguments as live and ignore gen and kill for them? I don't see how that helps with move elimination, though I know it makes the analysis a bit less precise.

View changes since the review

Are you talking about argument locals (_1, _2) that come function arguments? We do want to track lifetimes for those because we can merge them with other locals if their lifetimes don't overlap.

Yes. Here is an example: f98897f which skips gen&kill for arguments. I don't see tests that are broken. I cannot imagine that a case can use the move fact. Or can we only kill in the late phase?

We still need to kill moved arguments after the call. It's just that they are killed at the late point instead of the early point to model the fact that they are not allowed to overlap with the return place.

I think it's fine to skip kills, which is an over-approximation, still has the fact. IIUC, the current code is:

// Early: Kill(_1) _2 = Call(move _1) -> BB1, BB2 // Late: Gen(_1, _2), Kill(_1)

I mean change this to:

// Early: - _2 = Call(move _1) -> BB1, BB2 // Late: Gen(_2)

Or can we change to:

// Early: - _2 = Call(move _1) -> BB1, BB2 // Late: Gen(_2), Kill(_1)

or

// Early: - _2 = Call(move _1) -> BB1, BB2 // Late: Gen(_2) BB1: Kill(_1) BB2: Kill(_1)

?

I understand what you mean now. You're right, the code looks better the way you suggest in f98897f. Since Call is a terminator, the kill_all at the end will end the lifetimes of any moved call operands at the late point anyways.

Actually that still doesn't work because the kill point loop immediately afterwards may end the lifetime of a call argument early, which is incorrect if it is a move argument. So in the end my preference is for keeping the code as it is today.

saethlin · 2026-07-03T04:55:46Z

I threw rustlantis at this PR and it found that this program:

#![feature(custom_mir, core_intrinsics, lazy_get)]

extern crate core;

use core::intrinsics::mir::*;

fn dump_var<T>(val: T) { }

#[custom_mir(dialect = "runtime", phase = "initial")]
pub fn fn1() {
    mir!{
        let _1: *const bool;
        let _2: bool;
        let _3: bool;
        {
            _1 = core::ptr::addr_of!(_3);
            *_1 = true;
            _2 = _3;
            Call(RET = dump_var(Move(_2)), ReturnTo(bb1), UnwindUnreachable())
        }
        bb1 = {
            Call(RET = dump_var(_3), ReturnTo(bb2), UnwindUnreachable())
        }
        bb2 = {
            Return()
        }
    }
}

pub fn main() {
    fn1();
}

gets this diff applied:

-// MIR for `fn1` before MoveElimination
+// MIR for `fn1` after MoveElimination
 
 fn fn1() -> () {
     let mut _0: ();
@@ -7,14 +7,14 @@ fn fn1() -> () {
     let mut _3: bool;
 
     bb0: {
-        _1 = &raw const _3;
+        _1 = &raw const _2;
         (*_1) = const true;
-        _2 = copy _3;
+        nop;
         _0 = dump_var::<bool>(move _2) -> [return: bb1, unwind unreachable];
     }
 
     bb1: {
-        _0 = dump_var::<bool>(copy _3) -> [return: bb2, unwind unreachable];
+        _0 = dump_var::<bool>(copy _2) -> [return: bb2, unwind unreachable];
     }
 
     bb2: {

It seems like the current implementation introduces use of a place after it is moved out of?

Amanieu · 2026-07-03T05:34:18Z

That's strange, it should have refused to merge _2 and _3 because their live ranges overlap. I'll look into it.

tmiasko · 2026-07-03T05:39:36Z

_3 is not allocated when executing _1 = core::ptr::addr_of!(_3);, so this program has undefined behavior.

RalfJung · 2026-07-04T17:21:49Z

The original program has no storage statement which means all locals are implicitly live at the beginning.

Amanieu · 2026-07-04T17:23:27Z

Not with the new MIR semantics. The local is only allocated when it is first written to.

RalfJung · 2026-07-04T17:29:01Z

Ah... then Rustlantis will need to be patched.

This flag also has the effect of disabling DestinationPropagation, which is already covered by move elimination.

Amanieu · 2026-08-29T12:45:14Z

All feedback has been addressed:

StorageLive is now inserted in cleanup blocks where necessary, using an analysis to avoid doing so when unnecessary.
StorageLive is now always inserted after StorageDead at the same point, eliminating false overlaps.
StorageDead for all locals is now inserted before Return terminators.

@bors try @rust-timer queue

Implement RFC 3943: MIR move elimination

rust-bors · 2026-08-29T13:45:23Z

☀️ Try build successful (CI)
Build commit: 205c95f (205c95fbc538d0641e0d6c0fc11c4a4f21f59df4)
Base parent: 6bb812b (6bb812b714cce8e08864b7b569303f25ab08eaad)

rust-timer · 2026-08-29T14:27:43Z

Finished benchmarking commit (205c95f): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	1.1%	[0.2%, 7.0%]	36
Regressions ❌ (secondary)	9.8%	[0.2%, 699.8%]	92
Improvements ✅ (primary)	-1.0%	[-5.5%, -0.1%]	192
Improvements ✅ (secondary)	-0.9%	[-10.5%, -0.1%]	215
All ❌✅ (primary)	-0.7%	[-5.5%, 7.0%]	228

Max RSS (memory usage)

Results (primary -0.9%, secondary -1.7%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.5%	[1.0%, 4.0%]	2
Regressions ❌ (secondary)	3.6%	[0.8%, 5.7%]	6
Improvements ✅ (primary)	-1.7%	[-2.9%, -0.5%]	9
Improvements ✅ (secondary)	-2.7%	[-4.3%, -1.3%]	32
All ❌✅ (primary)	-0.9%	[-2.9%, 4.0%]	11

Cycles

Results (primary -2.2%, secondary 2.6%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	3.7%	[1.9%, 7.1%]	3
Regressions ❌ (secondary)	35.1%	[1.8%, 275.7%]	10
Improvements ✅ (primary)	-2.7%	[-4.5%, -1.4%]	40
Improvements ✅ (secondary)	-3.1%	[-8.4%, -1.9%]	57
All ❌✅ (primary)	-2.2%	[-4.5%, 7.1%]	43

Binary size

Results (primary 0.6%, secondary 1.7%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	1.9%	[0.1%, 5.8%]	78
Regressions ❌ (secondary)	2.0%	[0.1%, 5.8%]	153
Improvements ✅ (primary)	-1.5%	[-8.4%, -0.0%]	43
Improvements ✅ (secondary)	-1.4%	[-8.4%, -0.0%]	14
All ❌✅ (primary)	0.6%	[-8.4%, 5.8%]	121

Bootstrap: 475.357s -> 469.357s (-1.26%)
Artifact size: 402.87 MiB -> 399.72 MiB (-0.78%)

Amanieu · 2026-08-31T06:56:08Z

I opened #162048 for the Miri/rustc_const_eval adaptations needed for the new MIR semantics. It builds on top of this PR.

rust-bors · 2026-09-05T23:29:14Z

☔ The latest upstream changes (presumably #160524) made this pull request unmergeable. Please resolve the merge conflicts by rebasing.

tmiasko · 2026-09-17T10:39:00Z

Could you split this into smaller pull request? For example into TailCopyToMove, PreciseLiveness, basic MoveElimination, apply_alias_fixup, and reconstruct_storage.

Amanieu · 2026-09-17T16:17:43Z

Could you split this into smaller pull request? For example into TailCopyToMove, PreciseLiveness, basic MoveElimination, apply_alias_fixup, and reconstruct_storage.

I wouldn't split apply_alias_fixup, it is required for soundness.

For the rest, I can make 4 PRs like this:

PreciseLiveness
TailCopyToMove
MoveElimination (including alias fixup) -- depends on PRs 1 & 2.
Storage reconstruction -- depends on PR 3

Let me know if that is OK, and who I should assign as a reviewer.

tmiasko · 2026-09-17T18:26:58Z

Sounds good. You can assign me as a reviewer. I would suggest TailCopyToMove as a first part.

dianqk · 2026-09-18T00:05:53Z

Sounds good too. I should be able to make time for the smaller PRs.

Edited: I should be available during the first week of October.

Amanieu · 2026-09-19T06:52:45Z

I've been working on the Miri support and it uncovered several issues with the semantics. So I'll be focusing on fixing those first.

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jun 15, 2026

rustbot assigned dianqk Jun 15, 2026

This comment has been minimized.

Sign in to view

Amanieu mentioned this pull request Jun 15, 2026

[WIP] MIR move elimination pass #156046

Closed

Amanieu force-pushed the move-elimination branch from f12157e to ca2c2ac Compare June 15, 2026 22:47

cjgillot self-assigned this Jun 16, 2026

cjgillot reviewed Jun 21, 2026

View reviewed changes

tmiasko reviewed Jun 22, 2026

View reviewed changes

Comment thread compiler/rustc_mir_dataflow/src/impls/precise_liveness.rs Outdated

Amanieu force-pushed the move-elimination branch from 6724dd7 to 974d4c0 Compare June 23, 2026 00:08

Amanieu force-pushed the move-elimination branch from 974d4c0 to 34776e8 Compare June 23, 2026 00:16

This comment has been minimized.

Sign in to view

Amanieu force-pushed the move-elimination branch from 34776e8 to 454a459 Compare June 23, 2026 09:58

dianqk reviewed Jun 27, 2026

View reviewed changes

tmiasko reviewed Jun 28, 2026

View reviewed changes

Comment thread compiler/rustc_mir_transform/src/tail_copy_to_move.rs

Comment thread compiler/rustc_mir_transform/src/move_elimination.rs Outdated

Comment thread compiler/rustc_mir_transform/src/tail_copy_to_move.rs

tmiasko reviewed Jun 28, 2026

View reviewed changes

Comment thread compiler/rustc_mir_transform/src/move_elimination.rs Outdated

tmiasko reviewed Jun 29, 2026

View reviewed changes

Comment thread compiler/rustc_mir_transform/src/move_elimination.rs Outdated

tmiasko reviewed Jun 30, 2026

View reviewed changes

Comment thread compiler/rustc_mir_transform/src/move_elimination.rs

cjgillot mentioned this pull request Jul 1, 2026

Fix coroutine MIR saved local remapping #158655

Merged

nxsaken mentioned this pull request Jul 2, 2026

MIR move elimination rust-lang/goals#396

Open

Amanieu added 4 commits August 24, 2026 20:08

Add a MIR pass which turns copies into moves just before a return

f7d7777

Gate move elimination behind -Z mir-move-elimination

34906fe

This flag also has the effect of disabling DestinationPropagation, which is already covered by move elimination.

Address review feedback

de7cec8

Address more review feedback

b04dd7b

Amanieu force-pushed the move-elimination branch from d70d9f0 to b04dd7b Compare August 27, 2026 00:27

Amanieu added 4 commits August 27, 2026 12:34

Add InitializedBeforeUse analysis to control placement of StorageLive

813aabe

Bind MIR test variables through debug declarations

2b12dfa

End live ranges before a Return terminator

8f0bbeb

Emit StorageLive after StorageDead to avoid false overlaps

03564de

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Aug 29, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Aug 29, 2026

Auto merge of #157943 - Amanieu:move-elimination, r=<try>

205c95f

Implement RFC 3943: MIR move elimination

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Aug 29, 2026

Amanieu mentioned this pull request Aug 29, 2026

Miscompilation of unsigned division rust-lang/rustc_codegen_gcc#967

Closed

Amanieu force-pushed the move-elimination branch from 842859f to 03564de Compare August 31, 2026 06:51

Amanieu mentioned this pull request Aug 31, 2026

Add Miri support for move elimination semantics #162048

Draft

Uh oh!

Conversation

Amanieu commented Jun 15, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Jun 15, 2026

Uh oh!

rustbot commented Jun 15, 2026

Uh oh!

This comment has been minimized.

cjgillot left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Amanieu commented Jun 22, 2026

Uh oh!

Uh oh!

rustbot commented Jun 23, 2026

Uh oh!

This comment has been minimized.

dianqk Jun 27, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

saethlin commented Jul 3, 2026

Uh oh!

Amanieu commented Jul 3, 2026

Uh oh!

tmiasko commented Jul 3, 2026

Uh oh!

RalfJung commented Jul 4, 2026

Uh oh!

Amanieu commented Jul 4, 2026

Uh oh!

RalfJung commented Jul 4, 2026

Uh oh!

Amanieu commented Aug 29, 2026

Amanieu commented Jun 15, 2026 •

edited by rustbot

Loading

cjgillot left a comment •

edited by rustbot

Loading

dianqk Jun 27, 2026 •

edited by rustbot

Loading

dianqk commented Sep 18, 2026 •

edited

Loading