[SLP]Fix narrow-tree gate width and extract miscount by alexey-bataev · Pull Request #216797 · llvm/llvm-project

alexey-bataev · 2026-08-17T18:01:34Z

Measure the narrowness of in-loop trees by the actual vectorization
width instead of the feeder-load width, and skip vector-typed scalars
in the instruction count check to match the cost model.

Fixes #216715

Created using spr 1.3.7

llvmorg-github-actions · 2026-08-17T18:02:19Z

@llvm/pr-subscribers-llvm-transforms

@llvm/pr-subscribers-vectorizers

Author: Alexey Bataev (alexey-bataev)

Changes

Measure the narrowness of in-loop trees by the actual vectorization
width instead of the feeder-load width, and skip vector-typed scalars
in the instruction count check to match the cost model.

Fixes #216715

Full diff: https://github.com/llvm/llvm-project/pull/216797.diff

3 Files Affected:

(modified) llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp (+15-2)
(modified) llvm/test/Transforms/SLPVectorizer/AArch64/abs-mul-buildvector-in-loop.ll (+15-14)
(modified) llvm/test/Transforms/SLPVectorizer/AArch64/long-non-power-of-2.ll (+4-2)

diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 166554b922572..527d2e49e4f90 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -13776,6 +13776,12 @@ uint64_t BoUpSLP::getNumVectorInsts(bool HasTreeLoop) {
       continue;
     if (EU.User && EphValues.count(EU.User))
       continue;
+    // Vector-typed scalars are consumed as whole vectors, no extract
+    // instruction is emitted for them.
+    if (isVectorizedTy(EU.Scalar->getType()) &&
+        (!SLPReVec ||
+         (EU.E.hasState() && EU.E.getOpcode() == Instruction::InsertElement)))
+      continue;
     if (ExternalUsesAsOriginalScalar.contains(EU.Scalar))
       continue;
     if (!CountedExtracts.insert(EU.Scalar).second)
@@ -19292,8 +19298,15 @@ BoUpSLP::calculateTreeCostAndTrimNonProfitable(ArrayRef<Value *> VectorizedVals,
     return Cost;
   // The narrow non-profitable tree in loop? Skip, may cause regressions.
   constexpr unsigned PartLimit = 2;
-  const unsigned Sz =
-      getVectorElementSize(VectorizableTree.front()->Scalars.front());
+  // Measure the narrowness by the width the tree is vectorized at.
+  const TreeEntry *RootTE = VectorizableTree.front().get();
+  unsigned Sz = getVectorElementSize(RootTE->Scalars.front());
+  if (auto It = MinBWs.find(RootTE); It != MinBWs.end())
+    Sz = It->second.first;
+  else
+    Sz = std::max<unsigned>(
+        Sz, DL->getTypeSizeInBits(
+                getValueType(RootTE->Scalars.front())->getScalarType()));
   const unsigned MinVF = getMinVF(Sz);
   if (Cost >= -SLPCostThreshold &&
       VectorizableTree.front()->Scalars.size() * PartLimit <= MinVF &&
diff --git a/llvm/test/Transforms/SLPVectorizer/AArch64/abs-mul-buildvector-in-loop.ll b/llvm/test/Transforms/SLPVectorizer/AArch64/abs-mul-buildvector-in-loop.ll
index ef80d9e6bcfd9..1d1616698fc17 100644
--- a/llvm/test/Transforms/SLPVectorizer/AArch64/abs-mul-buildvector-in-loop.ll
+++ b/llvm/test/Transforms/SLPVectorizer/AArch64/abs-mul-buildvector-in-loop.ll
@@ -17,18 +17,12 @@ define void @test(ptr noalias %src, ptr noalias %descriptor, i32 %N) {
 ; CHECK-NEXT:    [[I_033:%.*]] = phi i32 [ [[INC:%.*]], %[[FOR_BODY]] ], [ 0, %[[ENTRY]] ]
 ; CHECK-NEXT:    [[ARRAYIDX]] = getelementptr inbounds nuw i8, ptr [[SRC_ADDR_035]], i64 1
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i8, ptr [[ARRAYIDX]], align 1
-; CHECK-NEXT:    [[CONV:%.*]] = zext i8 [[TMP0]] to i32
 ; CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds i8, ptr [[SRC_ADDR_035]], i64 -1
 ; CHECK-NEXT:    [[TMP1:%.*]] = load i8, ptr [[ARRAYIDX1]], align 1
-; CHECK-NEXT:    [[CONV2:%.*]] = zext i8 [[TMP1]] to i32
-; CHECK-NEXT:    [[SUB:%.*]] = sub nsw i32 [[CONV]], [[CONV2]]
 ; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds nuw i8, ptr [[SRC_ADDR_035]], i64 17
 ; CHECK-NEXT:    [[TMP2:%.*]] = load i8, ptr [[ARRAYIDX3]], align 1
-; CHECK-NEXT:    [[CONV4:%.*]] = zext i8 [[TMP2]] to i32
 ; CHECK-NEXT:    [[ARRAYIDX5:%.*]] = getelementptr inbounds i8, ptr [[SRC_ADDR_035]], i64 -17
 ; CHECK-NEXT:    [[TMP3:%.*]] = load i8, ptr [[ARRAYIDX5]], align 1
-; CHECK-NEXT:    [[CONV6:%.*]] = zext i8 [[TMP3]] to i32
-; CHECK-NEXT:    [[SUB7:%.*]] = sub nsw i32 [[CONV4]], [[CONV6]]
 ; CHECK-NEXT:    [[TMP4:%.*]] = load i8, ptr [[P_034]], align 1
 ; CHECK-NEXT:    [[CONV9:%.*]] = zext i8 [[TMP4]] to i32
 ; CHECK-NEXT:    [[INCDEC_PTR10]] = getelementptr inbounds nuw i8, ptr [[P_034]], i64 2
@@ -36,15 +30,22 @@ define void @test(ptr noalias %src, ptr noalias %descriptor, i32 %N) {
 ; CHECK-NEXT:    [[TMP5:%.*]] = load i8, ptr [[BIN]], align 1
 ; CHECK-NEXT:    [[IDX_EXT:%.*]] = zext i8 [[TMP5]] to i64
 ; CHECK-NEXT:    [[ADD_PTR:%.*]] = getelementptr inbounds nuw [4 x i8], ptr [[DESCRIPTOR]], i64 [[IDX_EXT]]
-; CHECK-NEXT:    [[MUL:%.*]] = mul nsw i32 [[SUB]], [[CONV9]]
-; CHECK-NEXT:    [[MUL12:%.*]] = mul nsw i32 [[SUB7]], [[CONV9]]
-; CHECK-NEXT:    [[TMP6:%.*]] = tail call i32 @llvm.abs.i32(i32 [[MUL]], i1 true)
-; CHECK-NEXT:    [[TMP7:%.*]] = tail call i32 @llvm.abs.i32(i32 [[MUL12]], i1 true)
+; CHECK-NEXT:    [[CONV4:%.*]] = zext i8 [[TMP2]] to i32
+; CHECK-NEXT:    [[CONV:%.*]] = zext i8 [[TMP0]] to i32
+; CHECK-NEXT:    [[CONV6:%.*]] = zext i8 [[TMP3]] to i32
+; CHECK-NEXT:    [[CONV2:%.*]] = zext i8 [[TMP1]] to i32
+; CHECK-NEXT:    [[SUB7:%.*]] = sub nsw i32 [[CONV4]], [[CONV6]]
+; CHECK-NEXT:    [[SUB:%.*]] = sub nsw i32 [[CONV]], [[CONV2]]
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x i32> poison, i32 [[SUB]], i64 0
+; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x i32> [[TMP6]], i32 [[SUB7]], i64 1
+; CHECK-NEXT:    [[TMP15:%.*]] = insertelement <2 x i32> poison, i32 [[CONV9]], i64 0
+; CHECK-NEXT:    [[TMP9:%.*]] = shufflevector <2 x i32> [[TMP15]], <2 x i32> poison, <2 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP10:%.*]] = mul nsw <2 x i32> [[TMP7]], [[TMP9]]
+; CHECK-NEXT:    [[TMP11:%.*]] = call <2 x i32> @llvm.abs.v2i32(<2 x i32> [[TMP10]], i1 true)
 ; CHECK-NEXT:    [[TMP8:%.*]] = load <4 x i32>, ptr [[ADD_PTR]], align 4
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <4 x i32> poison, i32 [[MUL]], i64 0
-; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <4 x i32> [[TMP9]], i32 [[MUL12]], i64 1
-; CHECK-NEXT:    [[TMP11:%.*]] = insertelement <4 x i32> [[TMP10]], i32 [[TMP6]], i64 2
-; CHECK-NEXT:    [[TMP12:%.*]] = insertelement <4 x i32> [[TMP11]], i32 [[TMP7]], i64 3
+; CHECK-NEXT:    [[TMP16:%.*]] = shufflevector <2 x i32> [[TMP10]], <2 x i32> poison, <4 x i32> <i32 0, i32 1, i32 poison, i32 poison>
+; CHECK-NEXT:    [[TMP14:%.*]] = shufflevector <2 x i32> [[TMP11]], <2 x i32> poison, <4 x i32> <i32 0, i32 1, i32 poison, i32 poison>
+; CHECK-NEXT:    [[TMP12:%.*]] = shufflevector <4 x i32> [[TMP16]], <4 x i32> [[TMP14]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
 ; CHECK-NEXT:    [[TMP13:%.*]] = add nsw <4 x i32> [[TMP8]], [[TMP12]]
 ; CHECK-NEXT:    store <4 x i32> [[TMP13]], ptr [[ADD_PTR]], align 4
 ; CHECK-NEXT:    [[INC]] = add nuw nsw i32 [[I_033]], 1
diff --git a/llvm/test/Transforms/SLPVectorizer/AArch64/long-non-power-of-2.ll b/llvm/test/Transforms/SLPVectorizer/AArch64/long-non-power-of-2.ll
index e1ca81edd2b39..c6489c2085000 100644
--- a/llvm/test/Transforms/SLPVectorizer/AArch64/long-non-power-of-2.ll
+++ b/llvm/test/Transforms/SLPVectorizer/AArch64/long-non-power-of-2.ll
@@ -7,6 +7,8 @@ define i1 @test(ptr %arg, ptr %arg1, i64 %arg2, ptr %arg3) {
 ; CHECK-NEXT:  [[BB:.*:]]
 ; CHECK-NEXT:    [[GETELEMENTPTR:%.*]] = getelementptr i8, ptr [[ARG1]], i64 [[ARG2]]
 ; CHECK-NEXT:    [[GETELEMENTPTR4:%.*]] = getelementptr i8, ptr null, i64 0
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x ptr> <ptr null, ptr poison>, ptr [[ARG3]], i64 1
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr i8, <2 x ptr> [[TMP2]], <2 x i64> <i64 -32, i64 -432>
 ; CHECK-NEXT:    [[GETELEMENTPTR5:%.*]] = getelementptr i8, ptr null, i64 -32
 ; CHECK-NEXT:    [[TMP32:%.*]] = getelementptr i8, ptr [[ARG3]], i64 -440
 ; CHECK-NEXT:    [[GETELEMENTPTR7:%.*]] = getelementptr i8, ptr [[ARG1]], i64 0
@@ -28,8 +30,8 @@ define i1 @test(ptr %arg, ptr %arg1, i64 %arg2, ptr %arg3) {
 ; CHECK-NEXT:    [[AND25:%.*]] = and i1 false, [[ICMP24]]
 ; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <4 x ptr> <ptr poison, ptr poison, ptr null, ptr null>, ptr [[ARG]], i64 1
 ; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <4 x ptr> [[TMP0]], ptr [[GETELEMENTPTR7]], i64 0
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <4 x ptr> <ptr poison, ptr poison, ptr null, ptr null>, ptr [[GETELEMENTPTR5]], i64 0
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <4 x ptr> [[TMP2]], ptr [[GETELEMENTPTR8]], i64 1
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <2 x ptr> [[TMP1]], <2 x ptr> poison, <4 x i32> <i32 0, i32 1, i32 poison, i32 poison>
+; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <4 x ptr> <ptr poison, ptr poison, ptr null, ptr null>, <4 x ptr> [[TMP4]], <4 x i32> <i32 4, i32 5, i32 2, i32 3>
 ; CHECK-NEXT:    [[TMP20:%.*]] = icmp ult <4 x ptr> [[TMP19]], [[TMP7]]
 ; CHECK-NEXT:    [[ICMP58:%.*]] = icmp ult ptr [[GETELEMENTPTR]], null
 ; CHECK-NEXT:    [[ICMP62:%.*]] = icmp ult ptr [[GETELEMENTPTR]], null

bababuck · 2026-08-17T20:34:31Z

-      getVectorElementSize(VectorizableTree.front()->Scalars.front());
+  // Measure the narrowness by the width the tree is vectorized at.
+  const TreeEntry *RootTE = VectorizableTree.front().get();
+  unsigned Sz = getVectorElementSize(RootTE->Scalars.front());


Suggested change

unsigned Sz = getVectorElementSize(RootTE->Scalars.front());

unsigned Sz;

Then else max won't work, better to put it in else

Created using spr 1.3.7

bababuck

LGTM

hazzlim · 2026-08-17T23:23:57Z

-  const unsigned Sz =
-      getVectorElementSize(VectorizableTree.front()->Scalars.front());
+  // Measure the narrowness by the width the tree is vectorized at.
+  const TreeEntry *RootTE = VectorizableTree.front().get();


Suggested change

const TreeEntry *RootTE = VectorizableTree.front().get();

const TreeEntry *RootTE = &getRootNode();

Created using spr 1.3.7

Measure the narrowness of in-loop trees by the actual vectorization width instead of the feeder-load width, and skip vector-typed scalars in the instruction count check to match the cost model. Fixes #216715 Reviewers: efriedma-quic, bababuck, RKSimon, kartcq Pull Request: llvm/llvm-project#216797

kartcq · 2026-08-19T03:57:26Z

Thanks for fixing this promptly.
Could we backport this fix to the branch 23.x as well?
@alexey-bataev @bababuck

alexey-bataev · 2026-08-19T11:23:51Z

/cherry-pick 3763110

llvmbot · 2026-08-19T11:32:00Z

Failed to cherry-pick: 3763110

https://github.com/llvm/llvm-project/actions/runs/32247374462

Please manually backport the fix and push it to your github fork. Once this is done, please create a pull request

kartcq · 2026-08-20T10:08:31Z

I have did a manual cherry-pick of this to branch 23.x - #217569
Kindly help review @alexey-bataev @bababuck
Thanks

Measure the narrowness of in-loop trees by the actual vectorization width instead of the feeder-load width, and skip vector-typed scalars in the instruction count check to match the cost model. Fixes llvm#216715 Reviewers: efriedma-quic, bababuck, RKSimon, kartcq Pull Request: llvm#216797

[𝘀𝗽𝗿] initial version

b9dc5aa

Created using spr 1.3.7

llvmorg-github-actions Bot added vectorizers llvm:transforms labels Aug 17, 2026

alexey-bataev requested review from RKSimon, bababuck and kartcq August 17, 2026 18:01

bababuck reviewed Aug 17, 2026

View reviewed changes

Comment thread llvm/test/Transforms/SLPVectorizer/AArch64/long-non-power-of-2.ll

bababuck reviewed Aug 17, 2026

View reviewed changes

Rebase, address comment

a89ce98

Created using spr 1.3.7

bababuck approved these changes Aug 17, 2026

View reviewed changes

hazzlim reviewed Aug 17, 2026

View reviewed changes

kartcq requested a review from efriedma-quic August 18, 2026 04:06

Rebase, address comment

eaf75a7

Created using spr 1.3.7

alexey-bataev merged commit 3763110 into main Aug 18, 2026
8 of 12 checks passed

alexey-bataev deleted the users/alexey-bataev/spr/slpfix-narrow-tree-gate-width-and-extract-miscount-1 branch August 18, 2026 11:50

alexey-bataev added this to the LLVM 23.x Release milestone Aug 19, 2026

github-project-automation Bot added this to LLVM Release Status Aug 19, 2026

github-project-automation Bot moved this to Needs Triage in LLVM Release Status Aug 19, 2026

github-project-automation Bot moved this from Needs Triage to Done in LLVM Release Status Aug 19, 2026

llvmbot added the release:cherry-pick-failed label Aug 19, 2026

This was referenced Aug 20, 2026

Release/23.x #217568

Closed

release/23.x: [SLP]Fix narrow-tree gate width and extract miscount #217569

Open

Navigation Menu

Uh oh!

[SLP]Fix narrow-tree gate width and extract miscount - #216797

[SLP]Fix narrow-tree gate width and extract miscount#216797
alexey-bataev merged 3 commits into
mainfrom
users/alexey-bataev/spr/slpfix-narrow-tree-gate-width-and-extract-miscount-1

alexey-bataev commented Aug 17, 2026

Uh oh!

llvmorg-github-actions Bot commented Aug 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

bababuck Aug 17, 2026

Uh oh!

alexey-bataev Aug 17, 2026 •

edited

Loading

Uh oh!

bababuck left a comment

Uh oh!

hazzlim Aug 17, 2026

Uh oh!

Uh oh!

kartcq commented Aug 19, 2026

Uh oh!

alexey-bataev commented Aug 19, 2026

Uh oh!

llvmbot commented Aug 19, 2026

Uh oh!

kartcq commented Aug 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	unsigned Sz = getVectorElementSize(RootTE->Scalars.front());
	unsigned Sz;

	const TreeEntry *RootTE = VectorizableTree.front().get();
	const TreeEntry *RootTE = &getRootNode();

Uh oh!

Conversation

alexey-bataev commented Aug 17, 2026

Uh oh!

llvmorg-github-actions Bot commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

bababuck Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

alexey-bataev Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bababuck left a comment

Choose a reason for hiding this comment

Uh oh!

hazzlim Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kartcq commented Aug 19, 2026

Uh oh!

alexey-bataev commented Aug 19, 2026

Uh oh!

llvmbot commented Aug 19, 2026

Uh oh!

kartcq commented Aug 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

llvmorg-github-actions Bot commented Aug 17, 2026 •

edited

Loading

alexey-bataev Aug 17, 2026 •

edited

Loading