Skip to content

[LV] Use wide lane masks as the canonical form when tail-folding & interleaving #209484

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kmclaughlin-arm merged 14 commits into from
Aug 12, 2026
Merged

[LV] Use wide lane masks as the canonical form when tail-folding & interleaving #209484

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
337b77e
[LV] Use wide lane masks as the canonical form when tail-folding & in…
kmclaughlin-arm Jul 8, 2026
0733140
- Simplify ExtractSubvectorForPart recipe for removing unnecessary ex…
kmclaughlin-arm Jul 14, 2026
536da07
- Rebase & update vplan-printing-alias-mask.ll
kmclaughlin-arm Jul 15, 2026
23b783f
- Add ActiveLaneMaskForControlFlow
kmclaughlin-arm Jul 17, 2026
a588ea0
- Rename ActiveLaneMaskForControlFlow -> WideActiveLaneMask
kmclaughlin-arm Jul 20, 2026
5d3e353
- Removed Multiplier from existing ActiveLaneMask VPInstruction
kmclaughlin-arm Jul 22, 2026
380e4a6
- Add a test with const trip count
kmclaughlin-arm Jul 27, 2026
38af60a
- Swap checks for ActiveLaneMask with WideActiveLaneMask in VPRecipes
kmclaughlin-arm Aug 5, 2026
6a3a19d
- Address new feedback:
kmclaughlin-arm Aug 7, 2026
d535d6f
- Remove unused variable 'VF'
kmclaughlin-arm Aug 7, 2026
c51aea1
- Add test for simplifyBranchConditionForVFAndUF with known trip count
kmclaughlin-arm Aug 10, 2026
d4a5095
- Remove match(Mask, WideActiveLaneMask) from verifyLastActiveLaneRecipe
kmclaughlin-arm Aug 11, 2026
764042d
- Move new test into new sve-wide-mask-outside-use.ll file
kmclaughlin-arm Aug 11, 2026
8ea1f3b
- Simplify scalable_wide_lane_mask_outside_load_use test
kmclaughlin-arm Aug 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 4 additions & 20 deletions llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,11 +255,6 @@ static cl::opt<TailFoldingStyle> ForceTailFoldingStyle(
"Use predicated EVL instructions for tail folding. If EVL "
"is unsupported, fallback to data-without-lane-mask.")));

cl::opt<bool> llvm::EnableWideActiveLaneMask(
"enable-wide-lane-mask", cl::init(false), cl::Hidden,
cl::desc("Enable use of wide lane masks when used for control flow in "
"tail-folded loops"));

static cl::opt<bool> EnableInterleavedMemAccesses(
"enable-interleaved-mem-accesses", cl::init(false), cl::Hidden,
cl::desc("Enable vectorization on interleaved memory accesses in a loop"));
Expand Down Expand Up @@ -1219,15 +1214,6 @@ class LoopVectorizationCostModel {
return PartialAliasMaskingStatus == AliasMaskingStatus::Enabled;
}

/// Returns true if the use of wide lane masks is requested and the loop is
/// using tail-folding with a lane mask for control flow.
bool useWideActiveLaneMask() const {
if (!EnableWideActiveLaneMask)
return false;

return getTailFoldingStyle() == TailFoldingStyle::DataAndControlFlow;
}

/// Returns true if the instructions in this block requires predication
/// for any reason, e.g. because tail folding now requires a predicate
/// or because the block in the original loop was predicated.
Expand Down Expand Up @@ -3651,12 +3637,10 @@ LoopVectorizationPlanner::selectInterleaveCount(VPlan &Plan, ElementCount VF,
// 3. We don't interleave if we think that we will spill registers to memory
// due to the increased register pressure.

// Only interleave tail-folded loops if wide lane masks are requested, as the
// overhead of multiple instructions to calculate the predicate is likely
// not beneficial. If an epilogue is not allowed for any other reason,
// do not interleave.
if (!CM.isEpilogueAllowed() &&
!(CM.preferTailFoldedLoop() && CM.useWideActiveLaneMask()))

@fhahn fhahn Jul 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this change behavior for other tail folding styles that do not use ALM?

@david-arm david-arm Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this just reverting back to the original code before useWideActiveLaneMask was introduced?

@kmclaughlin-arm kmclaughlin-arm Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was intended to revert selectInterleaveCount back to the original code before #163387, however CM.preferTailFoldedLoop() should also have been removed. I've removed this in the latest commit, which prevents any tail-folded loop from being interleaved without forcing as it was originally.

// Do not interleave tail-folded loops, as the overhead of multiple
// instructions to calculate the predicate is likely not beneficial.
// If an epilogue is not allowed for any other reason, do not interleave.
if (!CM.isEpilogueAllowed())
return 1;

if (any_of(Plan.getVectorLoopRegion()->getEntryBasicBlock()->phis(),
Expand Down
11 changes: 10 additions & 1 deletion llvm/lib/Transforms/Vectorize/VPlan.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1242,8 +1242,17 @@ class LLVM_ABI_FOR_TEST VPInstruction : public VPRecipeWithIRFlags,
// Creates a mask where each lane is active (true) whilst the current
// counter (first operand + index) is less than the second operand. i.e.
// mask[i] = icmpt ult (op0 + i), op1
// The size of the mask returned is VF * Multiplier (UF, third op).
// ActiveLaneMask is used for tail-folding, with the exception of the

@fhahn fhahn Aug 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: It is also used for early-exit loops with stores

// DataAndControlFlow style. The size of the mask returned is VF.
// When unrolled, ActiveLaneMask is duplicated.

@david-arm david-arm Aug 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this still true given that WideActiveLaneMask is now the canonical form for unrolled, tail-folded loops? If WideActiveLaneMask isn't used as the canonical form for all loops, it would be good to explain in which scenarios unrolling using ActiveLaneMask occurs.

@kmclaughlin-arm kmclaughlin-arm Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WideActiveLaneMask is only the canonical form for unrolled, tail-folded loops if the style is DataAndControlFlow. For other styles, we still use ActiveLaneMask and will duplicate the mask (strict-fadd-interleave-only.ll does test this scenario, although the intrinsic is replaced with icmp because the VF is scalar). I've tried to rewrite the comment a bit to make this clearer though.

ActiveLaneMask,
// As above, but takes an additional operand (Multiplier). The size of
// the mask returned is VF * Multiplier (UF, op2).
// WideActiveLaneMask is used for control flow and is unrolled by widening,
// with one extract vector created per unroll part.
WideActiveLaneMask,

@fhahn fhahn Jul 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please document to make clear what the difference to regular ActiveeLaneMask is

@kmclaughlin-arm kmclaughlin-arm Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a new comment for WideActiveLaneMask which describes the unrolling behaviour (also updated the ActiveLaneMask description to remove Multiplier).

// Extracts each unrolled part of a (VF * UF) widened vector/mask.
ExtractVectorForPart,
ExplicitVectorLength,
// Represents the incoming loop-invariant alias-mask. All memory accesses
// in the loop must stay within the active lanes.
Expand Down
55 changes: 24 additions & 31 deletions llvm/lib/Transforms/Vectorize/VPlanLowering.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ void VPlanTransforms::replaceWideCanonicalIVWithWideIV(

// Add a VPActiveLaneMaskPHIRecipe and related recipes to \p Plan and replace
// the loop terminator with a branch-on-cond recipe with the negated
// active-lane-mask as operand. Note that this turns the loop into an
// wide-active-lane-mask as operand. Note that this turns the loop into an
// uncountable one. Only the existing terminator is replaced, all other existing
// recipes/users remain unchanged, except for poison-generating flags being
// dropped from the canonical IV increment. Return the created
Expand All @@ -128,15 +128,18 @@ void VPlanTransforms::replaceWideCanonicalIVWithWideIV(
//
// vector.ph:
// %EntryInc = canonical-iv-increment-for-part CanonicalIVStart
// %EntryALM = active-lane-mask %EntryInc, TC
// %EntryALM = wide-active-lane-mask %EntryInc, TC
// %EntryALMPart = extract-vector-for-part %EntryALM, ir<0>
//
// vector.body:
// ...
// %P = active-lane-mask-phi [ %EntryALM, %vector.ph ], [ %ALM, %vector.body ]
// %P = active-lane-mask-phi [ %EntryALMPart, %vector.ph ],
// [ %ALMPart, %vector.body ]
// ...
// %InLoopInc = canonical-iv-increment-for-part CanonicalIVIncrement
// %ALM = active-lane-mask %InLoopInc, TC
// %Negated = Not %ALM
// %ALM = wide-active-lane-mask %InLoopInc, TC
// %ALMPart = extract-vector-for-part %ALM, ir<0>
// %Negated = Not %ALMPart
// branch-on-cond %Negated
//
static VPActiveLaneMaskPHIRecipe *
Expand All @@ -148,29 +151,22 @@ addVPLaneMaskPhiAndUpdateExitBranch(VPlan &Plan) {
// TODO: Check if dropping the flags is needed.
TopRegion->clearCanonicalIVNUW(CanonicalIVIncrement);
DebugLoc DL = CanonicalIVIncrement->getDebugLoc();
// We can't use StartV directly in the ActiveLaneMask VPInstruction, since
// we have to take unrolling into account. Each part needs to start at
// Part * VF
auto *VecPreheader = Plan.getVectorPreheader();
VPBuilder Builder(VecPreheader);

// Create the ActiveLaneMask instruction using the correct start values.
VPValue *TC = Plan.getTripCount();
VPValue *VF = &Plan.getVF();

auto *EntryIncrement =
Builder.createOverflowingOp(VPInstruction::CanonicalIVIncrementForPart,
{StartV, VF}, {}, DL, "index.part.next");

// Create the active lane mask instruction in the VPlan preheader.
// Create the wide active lane mask instruction in the VPlan preheader.
VPValue *ALMMultiplier =
Plan.getConstantInt(TopRegion->getCanonicalIVType(), 1);
auto *EntryALM = Builder.createNaryOp(VPInstruction::ActiveLaneMask,
{EntryIncrement, TC, ALMMultiplier}, DL,
auto *EntryALM = Builder.createNaryOp(VPInstruction::WideActiveLaneMask,

@david-arm david-arm Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're always doing this for all targets regardless of the interleave count do we still need VPInstruction::ActiveLaneMask? Also, I think there is currently work being done by @artagnon to enable simplification of active lane masks with constant operands in the InstSimplifyFolder when detecting these recipes. That's probably fine, but any simplifications will need to now look at both recipes.

@kmclaughlin-arm kmclaughlin-arm Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ActiveLaneMask is needed as it will still be used for other styles of tail-folding (where the mask is not used for control-flow).

I'm happy to take a look at InstSimplify after this; I didn't know that there was work being done for ActiveLaneMask, but it sounds like it could apply to WideActiveLaneMask too.

{StartV, TC, ALMMultiplier}, DL,
"active.lane.mask.entry");
EntryALM = Builder.createNaryOp(VPInstruction::ExtractVectorForPart,
Comment thread
kmclaughlin-arm marked this conversation as resolved.
{EntryALM, Plan.getConstantInt(64, 0)}, DL,
"extract.entry.alm.part");

// Now create the ActiveLaneMaskPhi recipe in the main loop using the
// preheader ActiveLaneMask instruction.
// preheader WideActiveLaneMask instruction.
auto *LaneMaskPhi =
new VPActiveLaneMaskPHIRecipe(EntryALM, DebugLoc::getUnknown());
auto *HeaderVPBB = TopRegion->getEntryBasicBlock();
Expand All @@ -180,12 +176,12 @@ addVPLaneMaskPhiAndUpdateExitBranch(VPlan &Plan) {
// original terminator.
VPRecipeBase *OriginalTerminator = EB->getTerminator();
Builder.setInsertPoint(OriginalTerminator);
auto *InLoopIncrement = Builder.createOverflowingOp(
VPInstruction::CanonicalIVIncrementForPart,
{CanonicalIVIncrement, &Plan.getVF()}, {}, DL);
auto *ALM = Builder.createNaryOp(VPInstruction::ActiveLaneMask,
{InLoopIncrement, TC, ALMMultiplier}, DL,
"active.lane.mask.next");
auto *ALM = Builder.createNaryOp(VPInstruction::WideActiveLaneMask,
{CanonicalIVIncrement, TC, ALMMultiplier},
DL, "active.lane.mask.next");
ALM = Builder.createNaryOp(VPInstruction::ExtractVectorForPart,

@david-arm david-arm Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory if we're extracting part 0 of WideActiveLaneMask{0, 9, 1} where VF=4, IC=2, then there's nothing stopping vplan from simplifying this to ActiveLaneMask{0, 9} in future. Is it a requirement for this form to persist until vplan execution? If so, it might be a bit fragile and perhaps worth adding a comment here saying this code should not be simplified. Either that or ensure that we can still use the normal ActiveLaneMask for control flow.

@kmclaughlin-arm kmclaughlin-arm Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no requirement for this to persist in vplan, although later optimisations which are looking for the pattern of a single wide lane mask plus extracts would be affected. I don't think replacing an extract in this way would be valid however, when the wide lane mask has multiple uses because we are interleaving?
I can add a comment if you think it's worth making this clear though.

{ALM, Plan.getConstantInt(64, 0)}, DL,
"extract.next.alm.part");
LaneMaskPhi->addBackedgeValue(ALM);

// Replace the original terminator with BranchOnCond. We have to invert the
Expand Down Expand Up @@ -215,12 +211,9 @@ void VPlanTransforms::materializeHeaderMask(
VPIRFlags::WrapFlagsTy(/*HasNUW=*/true, /*HasNSW=*/false)));
VPValue *Mask;
if (UseActiveLaneMask) {
VPValue *ALMMultiplier =
Plan.getConstantInt(LoopRegion->getCanonicalIVType(), 1);
Mask = Builder.createNaryOp(
VPInstruction::ActiveLaneMask,
{WideCanonicalIV, Plan.getTripCount(), ALMMultiplier}, nullptr,
"active.lane.mask");
Mask = Builder.createNaryOp(VPInstruction::ActiveLaneMask,
{WideCanonicalIV, Plan.getTripCount()}, nullptr,
"active.lane.mask");
} else {
Mask = Builder.createICmp(CmpInst::ICMP_ULE, WideCanonicalIV,
Plan.getOrCreateBackedgeTakenCount());
Expand Down
13 changes: 10 additions & 3 deletions llvm/lib/Transforms/Vectorize/VPlanPatternMatch.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -447,16 +447,23 @@ m_ExtractLastLaneOfLastPart(const Op0_t &Op0) {
return m_ExtractLastLane(m_ExtractLastPart(Op0));
}

template <typename Op0_t, typename Op1_t>
inline VPInstruction_match<VPInstruction::ExtractVectorForPart, Op0_t, Op1_t>
m_ExtractVectorForPart(const Op0_t &Op0, const Op1_t &Op1) {
return m_VPInstruction<VPInstruction::ExtractVectorForPart>(Op0, Op1);
}

template <typename Op0_t>
inline VPInstruction_match<VPInstruction::ExtractPenultimateElement, Op0_t>
m_ExtractPenultimateElement(const Op0_t &Op0) {
return m_VPInstruction<VPInstruction::ExtractPenultimateElement>(Op0);
}

template <typename Op0_t, typename Op1_t, typename Op2_t>
inline VPInstruction_match<VPInstruction::ActiveLaneMask, Op0_t, Op1_t, Op2_t>
m_ActiveLaneMask(const Op0_t &Op0, const Op1_t &Op1, const Op2_t &Op2) {
return m_VPInstruction<VPInstruction::ActiveLaneMask>(Op0, Op1, Op2);
inline VPInstruction_match<VPInstruction::WideActiveLaneMask, Op0_t, Op1_t,
Op2_t>
m_WideActiveLaneMask(const Op0_t &Op0, const Op1_t &Op1, const Op2_t &Op2) {
return m_VPInstruction<VPInstruction::WideActiveLaneMask>(Op0, Op1, Op2);
}

inline VPInstruction_match<VPInstruction::AnyOf> m_AnyOf() {
Expand Down
46 changes: 39 additions & 7 deletions llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -505,6 +505,7 @@ Type *llvm::computeScalarTypeForInstruction(unsigned Opcode,
AssertOperandType(1, Op0Ty);
return IntegerType::get(Ctx, 1);
case VPInstruction::ActiveLaneMask:
case VPInstruction::WideActiveLaneMask:
assert(Op0Ty->isIntegerTy() && "expected integer operand");
AssertOperandType(1, Op0Ty);
return IntegerType::get(Ctx, 1);
Expand Down Expand Up @@ -656,6 +657,7 @@ unsigned VPInstruction::getNumOperandsForOpcode() const {
case Instruction::FCmp:
case Instruction::ExtractElement:
case Instruction::Store:
case VPInstruction::ActiveLaneMask:
case VPInstruction::BranchOnCount:
case VPInstruction::BranchOnTwoConds:
case VPInstruction::FirstOrderRecurrenceSplice:
Expand All @@ -666,10 +668,11 @@ unsigned VPInstruction::getNumOperandsForOpcode() const {
case VPInstruction::WideIVStep:
case VPInstruction::CalculateTripCountMinusVF:
case VPInstruction::ResumeForEpilogue:
case VPInstruction::ExtractVectorForPart:
return 2;
case Instruction::InsertElement:
case Instruction::Select:
case VPInstruction::ActiveLaneMask:
case VPInstruction::WideActiveLaneMask:
case VPInstruction::ReductionStartVector:
return 3;
case Instruction::Call:
Expand Down Expand Up @@ -793,20 +796,25 @@ Value *VPInstruction::generate(VPTransformState &State) {
return Builder.CreateSelectFMF(Cond, Op1, Op2, getFastMathFlagsOrNone(),
Name);
}
case VPInstruction::ActiveLaneMask: {
case VPInstruction::ActiveLaneMask:
case VPInstruction::WideActiveLaneMask: {
// Get first lane of vector induction variable.
Value *VIVElem0 = State.get(getOperand(0), VPLane(0));
// Get the original loop tripcount.
Value *ScalarTC = State.get(getOperand(1), VPLane(0));

uint64_t Multiplier =
getOpcode() == VPInstruction::WideActiveLaneMask
? cast<VPConstantInt>(getOperand(2))->getZExtValue()
: 1;

// If this part of the active lane mask is scalar, generate the CMP directly
// to avoid unnecessary extracts.
if (State.VF.isScalar())
if (State.VF.isScalar() && Multiplier == 1)
return Builder.CreateCmp(CmpInst::Predicate::ICMP_ULT, VIVElem0, ScalarTC,
Name);

ElementCount EC = State.VF.multiplyCoefficientBy(
cast<VPConstantInt>(getOperand(2))->getZExtValue());
ElementCount EC = State.VF.multiplyCoefficientBy(Multiplier);
auto *PredTy = VectorType::get(Builder.getInt1Ty(), EC);
return Builder.CreateIntrinsic(Intrinsic::get_active_lane_mask,
{PredTy, ScalarTC->getType()},
Expand Down Expand Up @@ -1112,6 +1120,17 @@ Value *VPInstruction::generate(VPTransformState &State) {

return Result;
}
case VPInstruction::ExtractVectorForPart: {
Value *Src = State.get(getOperand(0));
Type *DstTy = VectorType::get(getScalarType(), State.VF);
uint64_t Part = cast<VPConstantInt>(getOperand(1))->getZExtValue();

if (Src->getType() == DstTy)
return Src;

return Builder.CreateExtractVector(
DstTy, Src, Builder.getInt64(State.VF.getKnownMinValue() * Part), Name);
}
default:
llvm_unreachable("Unsupported opcode for instruction");
}
Expand Down Expand Up @@ -1419,9 +1438,13 @@ InstructionCost VPInstruction::computeCost(ElementCount VF,
TargetTransformInfo::SK_Splice, cast<VectorType>(VectorTy),
cast<VectorType>(VectorTy), {}, Ctx.CostKind, -1);
}
case VPInstruction::ActiveLaneMask: {
case VPInstruction::ActiveLaneMask:
case VPInstruction::WideActiveLaneMask: {
Type *ArgTy = getOperand(0)->getScalarType();
unsigned Multiplier = cast<VPConstantInt>(getOperand(2))->getZExtValue();
uint64_t Multiplier =
getOpcode() == VPInstruction::WideActiveLaneMask
? cast<VPConstantInt>(getOperand(2))->getZExtValue()
: 1;
Type *RetTy = toVectorTy(Type::getInt1Ty(Ctx.LLVMCtx), VF * Multiplier);
IntrinsicCostAttributes Attrs(Intrinsic::get_active_lane_mask, RetTy,
{ArgTy, ArgTy});
Expand Down Expand Up @@ -1631,12 +1654,14 @@ bool VPInstruction::opcodeMayReadOrWriteFromMemory() const {
case VPInstruction::ExtractLastPart:
case VPInstruction::ExtractPenultimateElement:
case VPInstruction::ActiveLaneMask:
case VPInstruction::WideActiveLaneMask:
case VPInstruction::IncomingAliasMask:
case VPInstruction::ExitingIVValue:
case VPInstruction::ExplicitVectorLength:
case VPInstruction::FirstActiveLane:
case VPInstruction::LastActiveLane:
case VPInstruction::ExtractLastActive:
case VPInstruction::ExtractVectorForPart:
case VPInstruction::FirstOrderRecurrenceSplice:
case VPInstruction::LogicalAnd:
case VPInstruction::LogicalOr:
Expand Down Expand Up @@ -1687,6 +1712,7 @@ bool VPInstruction::usesFirstLaneOnly(const VPValue *Op) const {
return vputils::onlyFirstLaneUsed(this);
case Instruction::Load:
case VPInstruction::ActiveLaneMask:
case VPInstruction::WideActiveLaneMask:
case VPInstruction::ExplicitVectorLength:
case VPInstruction::CalculateTripCountMinusVF:
case VPInstruction::CanonicalIVIncrementForPart:
Expand Down Expand Up @@ -1759,6 +1785,9 @@ void VPInstruction::printRecipe(raw_ostream &O, const Twine &Indent,
case VPInstruction::ActiveLaneMask:
O << "active lane mask";
break;
case VPInstruction::WideActiveLaneMask:
O << "wide active lane mask";
break;
case VPInstruction::IncomingAliasMask:
O << "incoming-alias-mask";
break;
Expand Down Expand Up @@ -1810,6 +1839,9 @@ void VPInstruction::printRecipe(raw_ostream &O, const Twine &Indent,
case VPInstruction::ExtractPenultimateElement:
O << "extract-penultimate-element";
break;
case VPInstruction::ExtractVectorForPart:
O << "extract-vector-for-part";
break;
case VPInstruction::ComputeReductionResult:
O << "compute-reduction-result";
break;
Expand Down
Loading
Loading