[TTI] Improve histogram intrinsic costing - #216337
Open
huntergr-arm wants to merge 6 commits into
Open
Conversation
huntergr-arm
requested review from
Andarwinux,
davemgreen,
david-arm and
sdesmalen-arm
August 14, 2026 14:57
llvmorg-github-actions
Bot
added
backend:AArch64
backend:X86
vectorizers
llvm:analysis
|
@llvm/pr-subscribers-backend-x86 @llvm/pr-subscribers-backend-aarch64 Author: Graham Hunter (huntergr-arm) ChangesThe current cost model underestimates histogram intrinsics. Since we'd like to turn histogram autovectorization on by default, we need to improve that. AArch64 currently doesn't take the gather or scatter into account, so that needs to be added. This is step one for that improvement, since we don't supply enough information to determine whether 32b indices can be used instead of full pointers. I'm planning on moving the costing to use getMemIntrinsicInstrCost in a later patch to enable that. X86 with AVX512 will vectorize in some cases, but falls back to the default scalarization leading to bad performance: #158292 (comment) This uses the default cost model, which didn't cost extracts from different lanes or consider the second extract+cmp+branch needed for masking, so we need to add that as well. Patch is 76.94 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/216337.diff 9 Files Affected:
diff --git a/llvm/include/llvm/CodeGen/BasicTTIImpl.h b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
index 6c7bb56563ab6..feee0989247ad 100644
--- a/llvm/include/llvm/CodeGen/BasicTTIImpl.h
+++ b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
@@ -2605,10 +2605,13 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
if (!PtrsTy)
return InstructionCost::getInvalid();
+ FixedVectorType *MaskTy = cast<FixedVectorType>(ICA.getArgTypes()[2]);
+ Type *MaskEltTy = MaskTy->getScalarType();
+
Align Alignment = thisT()->DL.getABITypeAlign(EltTy);
InstructionCost Cost = 0;
- Cost += thisT()->getVectorInstrCost(Instruction::ExtractElement, PtrsTy,
- CostKind, 1, nullptr, nullptr);
+
+ // Cost the main load->update->store sequence, for one element.
Cost += thisT()->getMemoryOpCost(Instruction::Load, EltTy, Alignment, 0,
CostKind);
switch (IID) {
@@ -2636,7 +2639,26 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
}
Cost += thisT()->getMemoryOpCost(Instruction::Store, EltTy, Alignment, 0,
CostKind);
+
+ // Add the cost of a compare + branch for the mask; for a type-only cost
+ // we cannot check whether the mask is all-true.
+ Cost += thisT()->getCmpSelInstrCost(
+ Instruction::ICmp, MaskEltTy, MaskEltTy, CmpInst::ICMP_EQ, CostKind);
+ Cost += thisT()->getCFInstrCost(Instruction::CondBr, CostKind);
+
+ // Multiply to find the cost for all elements.
Cost *= PtrsTy->getNumElements();
+
+ // Add in the cost of the extracts; the lanes may have different costs.
+ for (unsigned Lane = 0; Lane < PtrsTy->getNumElements(); ++Lane) {
+ // Pointer extract.
+ Cost += thisT()->getVectorInstrCost(Instruction::ExtractElement, PtrsTy,
+ CostKind, Lane, nullptr, nullptr);
+ // Mask extract.
+ Cost += thisT()->getVectorInstrCost(Instruction::ExtractElement, MaskTy,
+ CostKind, Lane, nullptr, nullptr);
+ }
+
return Cost;
}
case Intrinsic::get_active_lane_mask: {
diff --git a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
index b452666213738..824288cce51de 100644
--- a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
+++ b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
@@ -69,11 +69,6 @@ static cl::opt<bool> EnableOrLikeSelectOpt("enable-aarch64-or-like-select",
static cl::opt<bool> EnableLSRCostOpt("enable-aarch64-lsr-cost-opt",
cl::init(true), cl::Hidden);
-// A complete guess as to a reasonable cost.
-static cl::opt<unsigned>
- BaseHistCntCost("aarch64-base-histcnt-cost", cl::init(8), cl::Hidden,
- cl::desc("The cost of a histcnt instruction"));
-
static cl::opt<unsigned> DMBLookaheadThreshold(
"dmb-lookahead-threshold", cl::init(10), cl::Hidden,
cl::desc("The number of instructions to search for a redundant dmb"));
@@ -564,8 +559,10 @@ static bool isUnpackedVectorVT(EVT VecVT) {
VecVT.getSizeInBits().getKnownMinValue() < AArch64::SVEBitsPerBlock;
}
-static InstructionCost getHistogramCost(const AArch64Subtarget *ST,
- const IntrinsicCostAttributes &ICA) {
+InstructionCost
+AArch64TTIImpl::getHistogramCost(const AArch64Subtarget *ST,
+ const IntrinsicCostAttributes &ICA,
+ TTI::TargetCostKind CostKind) const {
// We need to know at least the number of elements in the vector of buckets
// and the size of each element to update.
if (ICA.getArgTypes().size() < 2)
@@ -575,9 +572,18 @@ static InstructionCost getHistogramCost(const AArch64Subtarget *ST,
if (!ST->hasSVE2())
return InstructionCost::getInvalid();
- Type *BucketPtrsTy = ICA.getArgTypes()[0]; // Type of vector of pointers
- Type *EltTy = ICA.getArgTypes()[1]; // Type of bucket elements
- unsigned TotalHistCnts = 1;
+ auto *BucketPtrsTy = cast<VectorType>(ICA.getArgTypes()[0]);
+ Type *EltTy = ICA.getArgTypes()[1];
+
+ InstructionCost Cost = 0;
+ /// Get gather/scatter costs.
+ /// TODO: Find a way to get more info about the pointers, so we can determine
+ /// whether we can use 32b indices or require full 64b pointers.
+ Type *DataTy = VectorType::get(EltTy, BucketPtrsTy->getElementCount());
+ MemIntrinsicCostAttributes GMICA(Intrinsic::masked_gather, DataTy, Align(1));
+ Cost += getGatherScatterOpCost(GMICA, CostKind);
+ MemIntrinsicCostAttributes SMICA(Intrinsic::masked_scatter, DataTy, Align(1));
+ Cost += getGatherScatterOpCost(SMICA, CostKind);
unsigned EltSize = EltTy->getScalarSizeInBits();
// Only allow (up to 64b) integers or pointers
@@ -594,13 +600,14 @@ static InstructionCost getHistogramCost(const AArch64Subtarget *ST,
// HistCnt only supports 32b and 64b element types
unsigned LegalEltSize = EltSize <= 32 ? 32 : 64;
- if (EC == 2 || (LegalEltSize == 32 && EC == 4))
- return InstructionCost(BaseHistCntCost);
-
+ // HistCnt is the same cost as a vector integer add on 128b implementations,
+ // but will likely increase on wider vector types. Multiply by vscale as
+ // a quick estimate.
+ unsigned VScale = ST->getVScaleForTuning();
unsigned NaturalVectorWidth = AArch64::SVEBitsPerBlock / LegalEltSize;
- TotalHistCnts = EC / NaturalVectorWidth;
+ unsigned TotalHistCnts = EC / NaturalVectorWidth;
- return InstructionCost(BaseHistCntCost * TotalHistCnts);
+ return Cost + VScale * TotalHistCnts;
}
return InstructionCost::getInvalid();
@@ -620,7 +627,7 @@ AArch64TTIImpl::getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
switch (ICA.getID()) {
case Intrinsic::experimental_vector_histogram_add: {
- InstructionCost HistCost = getHistogramCost(ST, ICA);
+ InstructionCost HistCost = getHistogramCost(ST, ICA, CostKind);
// If the cost isn't valid, we may still be able to scalarize
if (HistCost.isValid())
return HistCost;
diff --git a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h
index cd71b1179bd45..faa51a042fbbd 100644
--- a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h
+++ b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h
@@ -133,6 +133,12 @@ class AArch64TTIImpl final : public BasicTTIImplBase<AArch64TTIImpl> {
return 31;
}
+ /// Gets the cost for a histogram operation, consisting of at least one
+ /// set of gather + histcnt + scatter instructions.
+ InstructionCost getHistogramCost(const AArch64Subtarget *ST,
+ const IntrinsicCostAttributes &ICA,
+ TTI::TargetCostKind CostKind) const;
+
InstructionCost
getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
TTI::TargetCostKind CostKind) const override;
diff --git a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
index 53ab77fbd28b1..4d41356dc427b 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
@@ -2479,11 +2479,9 @@ void VPHistogramRecipe::execute(VPTransformState &State) {
InstructionCost VPHistogramRecipe::computeCost(ElementCount VF,
VPCostContext &Ctx) const {
- // FIXME: Take the gather and scatter into account as well. For now we're
- // generating the same cost as the fallback path, but we'll likely
- // need to create a new TTI method for determining the cost, including
+ // FIXME: Improve the TTI method for determining the cost, including
// whether we can use base + vec-of-smaller-indices or just
- // vec-of-pointers.
+ // vec-of-pointers for the gather and scatter.
assert(VF.isVector() && "Invalid VF for histogram cost");
Type *AddressTy = getOperand(0)->getScalarType();
VPValue *IncAmt = getOperand(1);
diff --git a/llvm/test/Analysis/CostModel/AArch64/histograms.ll b/llvm/test/Analysis/CostModel/AArch64/histograms.ll
index b3057dea58886..05938c13c7899 100644
--- a/llvm/test/Analysis/CostModel/AArch64/histograms.ll
+++ b/llvm/test/Analysis/CostModel/AArch64/histograms.ll
@@ -10,37 +10,37 @@ define void @histograms() {
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.add.nxv4p0.i32(<vscale x 4 x ptr> poison, i32 1, <vscale x 4 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.add.nxv8p0.i16(<vscale x 8 x ptr> poison, i16 1, <vscale x 8 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.add.nxv16p0.i8(<vscale x 16 x ptr> poison, i8 1, <vscale x 16 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.experimental.vector.histogram.add.v2p0.i64(<2 x ptr> poison, i64 1, <2 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 20 for instruction: call void @llvm.experimental.vector.histogram.add.v4p0.i32(<4 x ptr> poison, i32 1, <4 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 40 for instruction: call void @llvm.experimental.vector.histogram.add.v8p0.i16(<8 x ptr> poison, i16 1, <8 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 80 for instruction: call void @llvm.experimental.vector.histogram.add.v16p0.i8(<16 x ptr> poison, i8 1, <16 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 16 for instruction: call void @llvm.experimental.vector.histogram.add.v2p0.i64(<2 x ptr> poison, i64 1, <2 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 32 for instruction: call void @llvm.experimental.vector.histogram.add.v4p0.i32(<4 x ptr> poison, i32 1, <4 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 64 for instruction: call void @llvm.experimental.vector.histogram.add.v8p0.i16(<8 x ptr> poison, i16 1, <8 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 128 for instruction: call void @llvm.experimental.vector.histogram.add.v16p0.i8(<16 x ptr> poison, i8 1, <16 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.nxv1p0.i64(<vscale x 1 x ptr> poison, i64 1, <vscale x 1 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.nxv2p0.i64(<vscale x 2 x ptr> poison, i64 1, <vscale x 2 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.nxv4p0.i32(<vscale x 4 x ptr> poison, i32 1, <vscale x 4 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.nxv8p0.i16(<vscale x 8 x ptr> poison, i16 1, <vscale x 8 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.nxv16p0.i8(<vscale x 16 x ptr> poison, i8 1, <vscale x 16 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 12 for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.v2p0.i64(<2 x ptr> poison, i64 1, <2 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 24 for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.v4p0.i32(<4 x ptr> poison, i32 1, <4 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 64 for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.v8p0.i16(<8 x ptr> poison, i16 1, <8 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 128 for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.v16p0.i8(<16 x ptr> poison, i8 1, <16 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.v2p0.i64(<2 x ptr> poison, i64 1, <2 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 36 for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.v4p0.i32(<4 x ptr> poison, i32 1, <4 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 88 for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.v8p0.i16(<8 x ptr> poison, i16 1, <8 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 176 for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.v16p0.i8(<16 x ptr> poison, i8 1, <16 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.umax.nxv1p0.i64(<vscale x 1 x ptr> poison, i64 1, <vscale x 1 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.umax.nxv2p0.i64(<vscale x 2 x ptr> poison, i64 1, <vscale x 2 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.umax.nxv4p0.i32(<vscale x 4 x ptr> poison, i32 1, <vscale x 4 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.umax.nxv8p0.i16(<vscale x 8 x ptr> poison, i16 1, <vscale x 8 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.umax.nxv16p0.i8(<vscale x 16 x ptr> poison, i8 1, <vscale x 16 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 12 for instruction: call void @llvm.experimental.vector.histogram.umax.v2p0.i64(<2 x ptr> poison, i64 1, <2 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 24 for instruction: call void @llvm.experimental.vector.histogram.umax.v4p0.i32(<4 x ptr> poison, i32 1, <4 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 48 for instruction: call void @llvm.experimental.vector.histogram.umax.v8p0.i16(<8 x ptr> poison, i16 1, <8 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 96 for instruction: call void @llvm.experimental.vector.histogram.umax.v16p0.i8(<16 x ptr> poison, i8 1, <16 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.experimental.vector.histogram.umax.v2p0.i64(<2 x ptr> poison, i64 1, <2 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 36 for instruction: call void @llvm.experimental.vector.histogram.umax.v4p0.i32(<4 x ptr> poison, i32 1, <4 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 72 for instruction: call void @llvm.experimental.vector.histogram.umax.v8p0.i16(<8 x ptr> poison, i16 1, <8 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 144 for instruction: call void @llvm.experimental.vector.histogram.umax.v16p0.i8(<16 x ptr> poison, i8 1, <16 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.umin.nxv1p0.i64(<vscale x 1 x ptr> poison, i64 1, <vscale x 1 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.umin.nxv2p0.i64(<vscale x 2 x ptr> poison, i64 1, <vscale x 2 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.umin.nxv4p0.i32(<vscale x 4 x ptr> poison, i32 1, <vscale x 4 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.umin.nxv8p0.i16(<vscale x 8 x ptr> poison, i16 1, <vscale x 8 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.umin.nxv16p0.i8(<vscale x 16 x ptr> poison, i8 1, <vscale x 16 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 12 for instruction: call void @llvm.experimental.vector.histogram.umin.v2p0.i64(<2 x ptr> poison, i64 1, <2 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 24 for instruction: call void @llvm.experimental.vector.histogram.umin.v4p0.i32(<4 x ptr> poison, i32 1, <4 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 48 for instruction: call void @llvm.experimental.vector.histogram.umin.v8p0.i16(<8 x ptr> poison, i16 1, <8 x i1> poison)
-; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 96 for instruction: call void @llvm.experimental.vector.histogram.umin.v16p0.i8(<16 x ptr> poison, i8 1, <16 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 18 for instruction: call void @llvm.experimental.vector.histogram.umin.v2p0.i64(<2 x ptr> poison, i64 1, <2 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 36 for instruction: call void @llvm.experimental.vector.histogram.umin.v4p0.i32(<4 x ptr> poison, i32 1, <4 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 72 for instruction: call void @llvm.experimental.vector.histogram.umin.v8p0.i16(<8 x ptr> poison, i16 1, <8 x i1> poison)
+; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 144 for instruction: call void @llvm.experimental.vector.histogram.umin.v16p0.i8(<16 x ptr> poison, i8 1, <16 x i1> poison)
; CHECK-NEON-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; CHECK-SVE-LABEL: 'histograms'
@@ -49,76 +49,76 @@ define void @histograms() {
; CHECK-SVE-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.add.nxv4p0.i32(<vscale x 4 x ptr> poison, i32 1, <vscale x 4 x i1> poison)
; CHECK-SVE-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.add.nxv8p0.i16(<vscale x 8 x ptr> poison, i16 1, <vscale x 8 x i1> poison)
; CHECK-SVE-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.add.nxv16p0.i8(<vscale x 16 x ptr> poison, i8 1, <vscale x 16 x i1> poison)
-; CHECK-SVE-NEXT: Cost Model: Found an estimated cost of 10 for instruction: call void @llvm.experimental.vector.histogram.add.v2p0.i64(<2 x ptr> poison, i64 1, <2 x i1> poison)
-; CHECK-SVE-NEXT: Cost Model: Found an estimated cost of 20 for instruction: call void @llvm.experimental.vector.histogram.add.v4p0.i32(<4 x ptr> poison, i32 1, <4 x i1> poison)
-; CHECK-SVE-NEXT: Cost Model: Found an estimated cost of 40 for instruction: call void @llvm.experimental.vector.histogram.add.v8p0.i16(<8 x ptr> poison, i16 1, <8 x i1> poison)
-; CHECK-SVE-NEXT: Cost Model: Found an estimated cost of 80 for instruction: call void @llvm.experimental.vector.histogram.add.v16p0.i8(<16 x ptr> poison, i8 1, <16 x i1> poison)
+; CHECK-SVE-NEXT: Cost Model: Found an estimated cost of 16 for instruction: call void @llvm.experimental.vector.histogram.add.v2p0.i64(<2 x ptr> poison, i64 1, <2 x i1> poison)
+; CHECK-SVE-NEXT: Cost Model: Found an estimated cost of 32 for instruction: call void @llvm.experimental.vector.histogram.add.v4p0.i32(<4 x ptr> poison, i32 1, <4 x i1> poison)
+; CHECK-SVE-NEXT: Cost Model: Found an estimated cost of 64 for instruction: call void @llvm.experimental.vector.histogram.add.v8p0.i16(<8 x ptr> poison, i16 1, <8 x i1> poison)
+; CHECK-SVE-NEXT: Cost Model: Found an estimated cost of 128 for instruction: call void @llvm.experimental.vector.histogram.add.v16p0.i8(<16 x ptr> poison, i8 1, <16 x i1> poison)
; CHECK-SVE-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.nxv1p0.i64(<vscale x 1 x ptr> poison, i64 1, <vscale x 1 x i1> poison)
; CHECK-SVE-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.nxv2p0.i64(<vscale x 2 x ptr> poison, i64 1, <vscale x 2 x i1> poison)
; CHECK-SVE-NEXT: Cost Model: Invalid cost for instruction: call void @llvm.experimental.vector.histogram.uadd.sat.nxv4p0.i32(<vsca...
[truncated]
|
Member
|
LGTM, but better to use -mcpu=x86-64-v2/3/4 for x86. |
RKSimon
reviewed
Aug 15, 2026
| ; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py UTC_ARGS: --version 6 | ||
| ; RUN: opt < %s -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=all -mcpu=x86-64-v2 | FileCheck %s --check-prefix=X8664-V2 | ||
| ; RUN: opt < %s -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=all -mcpu=x86-64-v3 | FileCheck %s --check-prefix=X8664-V3 | ||
| ; RUN: opt < %s -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=all -mcpu=x86-64-v4 | FileCheck %s --check-prefix=X8664-V4 |
Contributor
There was a problem hiding this comment.
; RUN: opt < %s -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=all -mcpu=x86-64 | FileCheck %s --check-prefixes=SSE,SSE2
; RUN: opt < %s -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=all -mcpu=x86-64-v2 | FileCheck %s --check-prefixes=SSE,SSE42
; RUN: opt < %s -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=all -mcpu=x86-64-v3 | FileCheck %s --check-prefixes=AVX,AVX2
; RUN: opt < %s -passes="print<cost-model>" 2>&1 -disable-output -cost-kind=all -mcpu=x86-64-v4 | FileCheck %s --check-prefixes=AVX,AVX512
RKSimon
self-requested a review
August 17, 2026 12:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The current cost model underestimates histogram intrinsics. Since we'd like to turn histogram autovectorization on by default, we need to improve that.
AArch64 currently doesn't take the gather or scatter into account, so that needs to be added. This is step one for that improvement, since we don't supply enough information to determine whether 32b indices can be used instead of full pointers. I'm planning on moving the costing to use getMemIntrinsicInstrCost in a later patch to enable that.
X86 with AVX512 will vectorize in some cases, but falls back to the default scalarization leading to bad performance: #158292 (comment)
This uses the default cost model, which didn't cost extracts from different lanes or consider the second extract+cmp+branch needed for masking, so we need to add that as well.