This is an archive of the discontinued LLVM Phabricator instance.

Differential D126680

[VPlan] Replace BranchOnCount with BranchOnCond if TC <= UF * VF.
ClosedPublic

Authored by fhahn on May 30 2022, 2:41 PM.

Details

Reviewers
Ayal
gilr
rengolin
david-arm
Commits
rGeaf48dd9b079: [VPlan] Replace BranchOnCount with BranchOnCond if TC <= UF * VF.
Summary

Try to simplify BranchOnCount to BranchOnCond true if TC <= UF * VF.

This is an alternative to D121899 which simplifies the VPlan directly
instead of doing so late in code-gen.

The potential benefit of doing this in VPlan is that this may help
cost-modeling in the future. The reason this is done in prepareToExecute
at the moment is that a single plan may be used for multiple VFs/UFs.

There are further simplifications that can be applied as follow ups:

  1. Replace inductions with constants
  2. Replace vector region with regular block.

Fixes #55354.

Depends on D126679.

Diff Detail

Event Timeline

fhahn created this revision.May 30 2022, 2:41 PM
Herald added a project: Restricted Project. · View Herald TranscriptMay 30 2022, 2:41 PM
Herald added subscribers: tschuett, psnobl, rogfer01 and 2 others. · View Herald Transcript
fhahn requested review of this revision.May 30 2022, 2:41 PM
Herald added a project: Restricted Project. · View Herald TranscriptMay 30 2022, 2:41 PM
Herald added a subscriber: vkmr. · View Herald Transcript
Harbormaster completed remote builds in B166972: Diff 432989.May 30 2022, 2:42 PM
fhahn mentioned this in D121899: [LoopVectorize] Optimise away the icmp when tail-folding for some low trip counts.May 30 2022, 2:43 PM
Ayal accepted this revision.Jun 2 2022, 3:07 AM

This looks good to me, thanks!

Added some nits and potential follow-ups.

No need to add a test for #55354, as it is already covered by existing tests, right?

llvm/lib/Transforms/Vectorize/VPlan.cpp
770

Independent of this patch?

905

CanonicalIVStartValue is assumed to be non negative, right?

This is as close to a VPlan2VPlan transformation as we can get at the moment in general. The TC <= VF case (using lower-bound of 1 for UF) could be optimized as an early VPlan2VPlan transformation, considering VPlan's VF range. But UF is currently set after finalizing a VPlan, except if set manually and/or optimizing for size.

907

nit: can fold condition e.g. into && isa<ConstantInt>...

909

getFixedValue()?

912

(Could alternatively set the two operands of Term to the same value for trivial comparison, given that BranchOnCount actually means ReiterateWhileNotEqual. Currently BranchOnCond can also represent loop branches, and using it is clearer. May be good to have a single representation for loop branches, independent of this patch.)

This revision is now accepted and ready to land.Jun 2 2022, 3:07 AM
fhahn updated this revision to Diff 434399.Jun 6 2022, 1:33 AM
fhahn marked 4 inline comments as done.

Address latest comments, thanks!

llvm/lib/Transforms/Vectorize/VPlan.cpp
905

CanonicalIVStartValue is assumed to be non negative, right?

Ah yes, it is always non-negative. When preparing to execute the plan for the main vector loop it will be set to nullptr, for the epilogue vector loop it will be set to a positive start value.

But I think it only makes sense for now to simplify for the main vector loop, so I'll added a check.

This is as close to a VPlan2VPlan transformation as we can get at the moment in general. The TC <= VF case (using lower-bound of 1 for UF) could be optimized as an early VPlan2VPlan transformation, considering VPlan's VF range. But UF is currently set after finalizing a VPlan, except if set manually and/or optimizing for size.

Yes, I think it would be good to move this to a VPlan2VPlan transform as follow-up, but we need to think a bit more on how to model constraints on the UF I think.

907

Reduced the nesting level, thanks!

909

Unfortunately I think getKnownMinValue is needed to handle scalable VFs here. there are some test cases that would crash with getFixedValue.

912

May be good to have a single representation for loop branches, independent of this patch.

Sounds good as follow-up!

This revision was landed with ongoing or failed builds.Jun 6 2022, 1:39 AM
Closed by commit rGeaf48dd9b079: [VPlan] Replace BranchOnCount with BranchOnCond if TC <= UF * VF. (authored by fhahn). · Explain Why
This revision was automatically updated to reflect the committed changes.
fhahn added a commit: rGeaf48dd9b079: [VPlan] Replace BranchOnCount with BranchOnCond if TC <= UF * VF..
Harbormaster completed remote builds in B167993: Diff 434399.Jun 6 2022, 2:20 AM
david-arm added a comment.Jun 7 2022, 1:09 AM

Thanks a lot for this @fhahn! I think this means I can now abandon D121899, right?

fhahn added a comment.Jun 7 2022, 2:15 PM
In D126680#3562666, @david-arm wrote:

Thanks a lot for this @fhahn! I think this means I can now abandon D121899, right?

Yes I think so ,thanks!

aeubanks added a subscriber: aeubanks.Jun 30 2022, 1:41 PM

this seems to be causing a miscompile, see https://github.com/llvm/llvm-project/issues/56319

fhahn added a comment.Jul 1 2022, 5:52 AM
In D126680#3623434, @aeubanks wrote:

this seems to be causing a miscompile, see https://github.com/llvm/llvm-project/issues/56319

Thanks, the issue is that in some cases with epilogue vectorization the main vector loop is effectively dead because we need to execute at least one iteration of the scalar loop, e.g. due to interleave groups. If the same plan is re-used for both the main vector loop and the epilogue vector loop, then the re-used plan has been optimized in a way that doesn't apply to the epilogue loop.

I pushed 0dddf04caba5, which disables the optimization when doing epilogue vectorization. I'll also work on a better fix, which is to have 2 separate plans for the different loops, which avoids the problem by construction.

Revision Contents

PathSize
llvm/
lib/
Transforms/
Vectorize/
23 lines
test/
Transforms/
LoopVectorize/
AArch64/
3 lines
X86/
3 lines
5 lines
3 lines
20 lines

Diff 434400

llvm/lib/Transforms/Vectorize/VPlan.cpp

Show First 20 Lines • Show All 761 Lines • ▼ Show 20 Lines case VPInstruction::CanonicalIVIncrementNUW: {
} }
State.set(this, Next, Part); State.set(this, Next, Part);
break; break;
} }
case VPInstruction::BranchOnCond: { case VPInstruction::BranchOnCond: {
if (Part != 0) if (Part != 0)
break; break;
AyalUnsubmitted
Not Done
ReplyInline Actions

Independent of this patch?

Ayal: Independent of this patch?
Value *Cond = State.get(getOperand(0), VPIteration(Part, 0)); Value *Cond = State.get(getOperand(0), VPIteration(Part, 0));
VPRegionBlock *ParentRegion = getParent()->getParent(); VPRegionBlock *ParentRegion = getParent()->getParent();
VPBasicBlock *Header = ParentRegion->getEntryBasicBlock(); VPBasicBlock *Header = ParentRegion->getEntryBasicBlock();
// Replace the temporary unreachable terminator with a new conditional // Replace the temporary unreachable terminator with a new conditional
// branch, hooking it up to backward destination for exiting blocks now and // branch, hooking it up to backward destination for exiting blocks now and
// to forward destination(s) later when they are created. // to forward destination(s) later when they are created.
BranchInst *CondBr = BranchInst *CondBr =
▲ Show 20 Lines • Show All 115 Lines • ▼ Show 20 Lines assert((Opcode == Instruction::FAdd || Opcode == Instruction::FMul ||
Opcode == Instruction::FCmp) && Opcode == Instruction::FCmp) &&
"this op can't take fast-math flags"); "this op can't take fast-math flags");
FMF = FMFNew; FMF = FMFNew;
} }
void VPlan::prepareToExecute(Value *TripCountV, Value *VectorTripCountV, void VPlan::prepareToExecute(Value *TripCountV, Value *VectorTripCountV,
Value *CanonicalIVStartValue, Value *CanonicalIVStartValue,
VPTransformState &State) { VPTransformState &State) {
VPBasicBlock *ExitingVPBB = getVectorLoopRegion()->getExitingBasicBlock();
auto *Term = dyn_cast<VPInstruction>(&ExitingVPBB->back());
// Try to simplify BranchOnCount to 'BranchOnCond true' if TC <= VF * UF when
AyalUnsubmitted
Done
ReplyInline Actions

CanonicalIVStartValue is assumed to be non negative, right?

This is as close to a VPlan2VPlan transformation as we can get at the moment in general. The TC <= VF case (using lower-bound of 1 for UF) could be optimized as an early VPlan2VPlan transformation, considering VPlan's VF range. But UF is currently set after finalizing a VPlan, except if set manually and/or optimizing for size.

Ayal: CanonicalIVStartValue is assumed to be non negative, right? This is as close to a VPlan2VPlan…
fhahnAuthorUnsubmitted
Done
ReplyInline Actions

CanonicalIVStartValue is assumed to be non negative, right?

Ah yes, it is always non-negative. When preparing to execute the plan for the main vector loop it will be set to nullptr, for the epilogue vector loop it will be set to a positive start value.

But I think it only makes sense for now to simplify for the main vector loop, so I'll added a check.

This is as close to a VPlan2VPlan transformation as we can get at the moment in general. The TC <= VF case (using lower-bound of 1 for UF) could be optimized as an early VPlan2VPlan transformation, considering VPlan's VF range. But UF is currently set after finalizing a VPlan, except if set manually and/or optimizing for size.

Yes, I think it would be good to move this to a VPlan2VPlan transform as follow-up, but we need to think a bit more on how to model constraints on the UF I think.

fhahn: > CanonicalIVStartValue is assumed to be non negative, right? Ah yes, it is always non…
// preparing to execute the plan for the main vector loop.
if (!CanonicalIVStartValue && Term &&
AyalUnsubmitted
Done
ReplyInline Actions

nit: can fold condition e.g. into && isa<ConstantInt>...

Ayal: nit: can fold condition e.g. into `&& isa<ConstantInt>`...
fhahnAuthorUnsubmitted
Done
ReplyInline Actions

Reduced the nesting level, thanks!

fhahn: Reduced the nesting level, thanks!
Term->getOpcode() == VPInstruction::BranchOnCount &&
isa<ConstantInt>(TripCountV)) {
AyalUnsubmitted
Done
ReplyInline Actions

getFixedValue()?

Ayal: getFixedValue()?
fhahnAuthorUnsubmitted
Done
ReplyInline Actions

Unfortunately I think getKnownMinValue is needed to handle scalable VFs here. there are some test cases that would crash with getFixedValue.

fhahn: Unfortunately I think `getKnownMinValue` is needed to handle scalable VFs here. there are some…
ConstantInt *C = cast<ConstantInt>(TripCountV);
uint64_t TCVal = C->getZExtValue();
if (TCVal && TCVal <= State.VF.getKnownMinValue() * State.UF) {
AyalUnsubmitted
Done
ReplyInline Actions

(Could alternatively set the two operands of Term to the same value for trivial comparison, given that BranchOnCount actually means ReiterateWhileNotEqual. Currently BranchOnCond can also represent loop branches, and using it is clearer. May be good to have a single representation for loop branches, independent of this patch.)

Ayal: (Could alternatively set the two operands of Term to the same value for trivial comparison…
fhahnAuthorUnsubmitted
Done
ReplyInline Actions

May be good to have a single representation for loop branches, independent of this patch.

Sounds good as follow-up!

fhahn: > May be good to have a single representation for loop branches, independent of this patch.
auto *BOC =
new VPInstruction(VPInstruction::BranchOnCond,
{getOrAddExternalDef(State.Builder.getTrue())});
Term->eraseFromParent();
ExitingVPBB->appendRecipe(BOC);
// TODO: Further simplifications are possible
// 1. Replace inductions with constants.
// 2. Replace vector loop region with VPBasicBlock.
}
}
// Check if the trip count is needed, and if so build it. // Check if the trip count is needed, and if so build it.
if (TripCount && TripCount->getNumUsers()) { if (TripCount && TripCount->getNumUsers()) {
for (unsigned Part = 0, UF = State.UF; Part < UF; ++Part) for (unsigned Part = 0, UF = State.UF; Part < UF; ++Part)
State.set(TripCount, TripCountV, Part); State.set(TripCount, TripCountV, Part);
} }
// Check if the backedge taken count is needed, and if so build it. // Check if the backedge taken count is needed, and if so build it.
if (BackedgeTakenCount && BackedgeTakenCount->getNumUsers()) { if (BackedgeTakenCount && BackedgeTakenCount->getNumUsers()) {
▲ Show 20 Lines • Show All 883 Lines • Show Last 20 Lines

llvm/test/Transforms/LoopVectorize/AArch64/sve-low-trip-count.ll

Show First 20 Lines • Show All 41 Lines • ▼ Show 20 Lines
; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %vector.ph ], [ [[INDEX_NEXT:%.*]], %vector.body ] ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %vector.ph ], [ [[INDEX_NEXT:%.*]], %vector.body ]
; CHECK: [[ACTIVE_LANE_MASK:%.*]] = call <vscale x 16 x i1> @llvm.get.active.lane.mask.nxv16i1.i64(i64 {{%.*}}, i64 5) ; CHECK: [[ACTIVE_LANE_MASK:%.*]] = call <vscale x 16 x i1> @llvm.get.active.lane.mask.nxv16i1.i64(i64 {{%.*}}, i64 5)
; CHECK: {{%.*}} = call <vscale x 16 x i8> @llvm.masked.load.nxv16i8.p0nxv16i8(<vscale x 16 x i8>* {{%.*}}, i32 1, <vscale x 16 x i1> [[ACTIVE_LANE_MASK]], <vscale x 16 x i8> poison) ; CHECK: {{%.*}} = call <vscale x 16 x i8> @llvm.masked.load.nxv16i8.p0nxv16i8(<vscale x 16 x i8>* {{%.*}}, i32 1, <vscale x 16 x i1> [[ACTIVE_LANE_MASK]], <vscale x 16 x i8> poison)
; CHECK: {{%.*}} = call <vscale x 16 x i8> @llvm.masked.load.nxv16i8.p0nxv16i8(<vscale x 16 x i8>* {{%.*}}, i32 1, <vscale x 16 x i1> [[ACTIVE_LANE_MASK]], <vscale x 16 x i8> poison) ; CHECK: {{%.*}} = call <vscale x 16 x i8> @llvm.masked.load.nxv16i8.p0nxv16i8(<vscale x 16 x i8>* {{%.*}}, i32 1, <vscale x 16 x i1> [[ACTIVE_LANE_MASK]], <vscale x 16 x i8> poison)
; CHECK: call void @llvm.masked.store.nxv16i8.p0nxv16i8(<vscale x 16 x i8> {{%.*}}, <vscale x 16 x i8>* {{%.*}}, i32 1, <vscale x 16 x i1> [[ACTIVE_LANE_MASK]]) ; CHECK: call void @llvm.masked.store.nxv16i8.p0nxv16i8(<vscale x 16 x i8> {{%.*}}, <vscale x 16 x i8>* {{%.*}}, i32 1, <vscale x 16 x i1> [[ACTIVE_LANE_MASK]])
; CHECK: [[VSCALE:%.*]] = call i64 @llvm.vscale.i64() ; CHECK: [[VSCALE:%.*]] = call i64 @llvm.vscale.i64()
; CHECK-NEXT: [[VF:%.*]] = mul i64 [[VSCALE]], 16 ; CHECK-NEXT: [[VF:%.*]] = mul i64 [[VSCALE]], 16
; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], [[VF]] ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], [[VF]]
; CHECK-NEXT: [[COND:%.*]] = icmp eq i64 [[INDEX_NEXT]], {{%.*}} ; CHECK-NEXT: br i1 true, label %middle.block, label %vector.body
; CHECK-NEXT: br i1 [[COND]], label %middle.block, label %vector.body
; ;
entry: entry:
br label %for.body br label %for.body
for.body: ; preds = %entry, %for.body for.body: ; preds = %entry, %for.body
%i.08 = phi i64 [ 0, %entry ], [ %inc, %for.body ] %i.08 = phi i64 [ 0, %entry ], [ %inc, %for.body ]
%arrayidx = getelementptr inbounds i8, i8* %src, i64 %i.08 %arrayidx = getelementptr inbounds i8, i8* %src, i64 %i.08
%0 = load i8, i8* %arrayidx, align 1 %0 = load i8, i8* %arrayidx, align 1
Show All 14 Lines

llvm/test/Transforms/LoopVectorize/X86/constant-fold.ll

Show All 20 Lines
; CHECK-NEXT: [[OFFSET_IDX:%.*]] = trunc i32 [[INDEX]] to i16 ; CHECK-NEXT: [[OFFSET_IDX:%.*]] = trunc i32 [[INDEX]] to i16
; CHECK-NEXT: [[TMP0:%.*]] = add i16 [[OFFSET_IDX]], 0 ; CHECK-NEXT: [[TMP0:%.*]] = add i16 [[OFFSET_IDX]], 0
; CHECK-NEXT: [[TMP1:%.*]] = sext i16 [[TMP0]] to i64 ; CHECK-NEXT: [[TMP1:%.*]] = sext i16 [[TMP0]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = getelementptr [2 x i16*], [2 x i16*]* @b, i16 0, i64 [[TMP1]] ; CHECK-NEXT: [[TMP2:%.*]] = getelementptr [2 x i16*], [2 x i16*]* @b, i16 0, i64 [[TMP1]]
; CHECK-NEXT: [[TMP3:%.*]] = getelementptr i16*, i16** [[TMP2]], i32 0 ; CHECK-NEXT: [[TMP3:%.*]] = getelementptr i16*, i16** [[TMP2]], i32 0
; CHECK-NEXT: [[TMP4:%.*]] = bitcast i16** [[TMP3]] to <2 x i16*>* ; CHECK-NEXT: [[TMP4:%.*]] = bitcast i16** [[TMP3]] to <2 x i16*>*
; CHECK-NEXT: store <2 x i16*> <i16* getelementptr inbounds ([1 x %rec8], [1 x %rec8]* @a, i32 0, i32 0, i32 0), i16* getelementptr inbounds ([1 x %rec8], [1 x %rec8]* @a, i32 0, i32 0, i32 0)>, <2 x i16*>* [[TMP4]], align 8 ; CHECK-NEXT: store <2 x i16*> <i16* getelementptr inbounds ([1 x %rec8], [1 x %rec8]* @a, i32 0, i32 0, i32 0), i16* getelementptr inbounds ([1 x %rec8], [1 x %rec8]* @a, i32 0, i32 0, i32 0)>, <2 x i16*>* [[TMP4]], align 8
; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 2 ; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 2
; CHECK-NEXT: [[TMP5:%.*]] = icmp eq i32 [[INDEX_NEXT]], 2 ; CHECK-NEXT: br i1 true, label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP0:!llvm.loop !.*]]
; CHECK-NEXT: br i1 [[TMP5]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP0:!llvm.loop !.*]]
; CHECK: middle.block: ; CHECK: middle.block:
; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i32 2, 2 ; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i32 2, 2
; CHECK-NEXT: br i1 [[CMP_N]], label [[BB3:%.*]], label [[SCALAR_PH]] ; CHECK-NEXT: br i1 [[CMP_N]], label [[BB3:%.*]], label [[SCALAR_PH]]
; CHECK: scalar.ph: ; CHECK: scalar.ph:
; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i16 [ 2, [[MIDDLE_BLOCK]] ], [ 0, [[BB1:%.*]] ] ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i16 [ 2, [[MIDDLE_BLOCK]] ], [ 0, [[BB1:%.*]] ]
; CHECK-NEXT: br label [[BB2:%.*]] ; CHECK-NEXT: br label [[BB2:%.*]]
; CHECK: bb2: ; CHECK: bb2:
; CHECK-NEXT: [[C_1_0:%.*]] = phi i16 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[_TMP9:%.*]], [[BB2]] ] ; CHECK-NEXT: [[C_1_0:%.*]] = phi i16 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[_TMP9:%.*]], [[BB2]] ]
Show All 31 Lines

llvm/test/Transforms/LoopVectorize/X86/outer_loop_test1_no_explicit_vect_width.ll

Show First 20 Lines • Show All 65 Lines • ▼ Show 20 Lines
; AVX: %[[AAddr2:.*]] = getelementptr inbounds [8 x [8 x i32]], [8 x [8 x i32]]* @arr, i64 0, <8 x i64> %[[InnerPhi]], <8 x i64> %[[VecInd]] ; AVX: %[[AAddr2:.*]] = getelementptr inbounds [8 x [8 x i32]], [8 x [8 x i32]]* @arr, i64 0, <8 x i64> %[[InnerPhi]], <8 x i64> %[[VecInd]]
; AVX: call void @llvm.masked.scatter.v8i32.v8p0i32(<8 x i32> %[[StoreVal]], <8 x i32*> %[[AAddr2]], i32 4, <8 x i1> <i1 true, i1 true, i1 true ; AVX: call void @llvm.masked.scatter.v8i32.v8p0i32(<8 x i32> %[[StoreVal]], <8 x i32*> %[[AAddr2]], i32 4, <8 x i1> <i1 true, i1 true, i1 true
; AVX: %[[InnerPhiNext]] = add nuw nsw <8 x i64> %[[InnerPhi]], <i64 1, i64 1, i64 1, i64 1, i64 1, i64 1, i64 1, i64 1> ; AVX: %[[InnerPhiNext]] = add nuw nsw <8 x i64> %[[InnerPhi]], <i64 1, i64 1, i64 1, i64 1, i64 1, i64 1, i64 1, i64 1>
; AVX: %[[VecCond:.*]] = icmp eq <8 x i64> %[[InnerPhiNext]], <i64 8, i64 8, i64 8, i64 8, i64 8, i64 8, i64 8, i64 8> ; AVX: %[[VecCond:.*]] = icmp eq <8 x i64> %[[InnerPhiNext]], <i64 8, i64 8, i64 8, i64 8, i64 8, i64 8, i64 8, i64 8>
; AVX: %[[InnerCond:.*]] = extractelement <8 x i1> %[[VecCond]], i32 0 ; AVX: %[[InnerCond:.*]] = extractelement <8 x i1> %[[VecCond]], i32 0
; AVX: br i1 %[[InnerCond]], label %[[ForInc]], label %[[InnerLoop]] ; AVX: br i1 %[[InnerCond]], label %[[ForInc]], label %[[InnerLoop]]
; AVX: [[ForInc]]: ; AVX: [[ForInc]]:
; AVX: %[[IndNext]] = add nuw i64 %[[Ind]], 8
; AVX: %[[VecIndNext]] = add <8 x i64> %[[VecInd]], <i64 8, i64 8, i64 8, i64 8, i64 8, i64 8, i64 8, i64 8> ; AVX: %[[VecIndNext]] = add <8 x i64> %[[VecInd]], <i64 8, i64 8, i64 8, i64 8, i64 8, i64 8, i64 8, i64 8>
; AVX: %[[Cmp:.*]] = icmp eq i64 %[[IndNext]], 8 ; AVX: %[[IndNext]] = add nuw i64 %[[Ind]], 8
; AVX: br i1 %[[Cmp]], label %middle.block, label %vector.body ; AVX: br i1 true, label %middle.block, label %vector.body
@arr2 = external global [8 x i32], align 16 @arr2 = external global [8 x i32], align 16
@arr = external global [8 x [8 x i32]], align 16 @arr = external global [8 x [8 x i32]], align 16
; Function Attrs: norecurse nounwind uwtable ; Function Attrs: norecurse nounwind uwtable
define void @foo(i32 %n) { define void @foo(i32 %n) {
entry: entry:
br label %for.body br label %for.body
Show All 29 Lines

llvm/test/Transforms/LoopVectorize/X86/pr34438.ll

Show All 24 Lines
; CHECK-NEXT: [[TMP4:%.*]] = getelementptr inbounds float, float* [[A:%.*]], i64 [[TMP0]] ; CHECK-NEXT: [[TMP4:%.*]] = getelementptr inbounds float, float* [[A:%.*]], i64 [[TMP0]]
; CHECK-NEXT: [[TMP5:%.*]] = getelementptr inbounds float, float* [[TMP4]], i32 0 ; CHECK-NEXT: [[TMP5:%.*]] = getelementptr inbounds float, float* [[TMP4]], i32 0
; CHECK-NEXT: [[TMP6:%.*]] = bitcast float* [[TMP5]] to <8 x float>* ; CHECK-NEXT: [[TMP6:%.*]] = bitcast float* [[TMP5]] to <8 x float>*
; CHECK-NEXT: [[WIDE_LOAD1:%.*]] = load <8 x float>, <8 x float>* [[TMP6]], align 4, !llvm.access.group !0 ; CHECK-NEXT: [[WIDE_LOAD1:%.*]] = load <8 x float>, <8 x float>* [[TMP6]], align 4, !llvm.access.group !0
; CHECK-NEXT: [[TMP7:%.*]] = fadd fast <8 x float> [[WIDE_LOAD]], [[WIDE_LOAD1]] ; CHECK-NEXT: [[TMP7:%.*]] = fadd fast <8 x float> [[WIDE_LOAD]], [[WIDE_LOAD1]]
; CHECK-NEXT: [[TMP8:%.*]] = bitcast float* [[TMP5]] to <8 x float>* ; CHECK-NEXT: [[TMP8:%.*]] = bitcast float* [[TMP5]] to <8 x float>*
; CHECK-NEXT: store <8 x float> [[TMP7]], <8 x float>* [[TMP8]], align 4, !llvm.access.group !0 ; CHECK-NEXT: store <8 x float> [[TMP7]], <8 x float>* [[TMP8]], align 4, !llvm.access.group !0
; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 8 ; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 8
; CHECK-NEXT: [[TMP9:%.*]] = icmp eq i64 [[INDEX_NEXT]], 8 ; CHECK-NEXT: br i1 true, label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP1:!llvm.loop !.*]]
; CHECK-NEXT: br i1 [[TMP9]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], [[LOOP1:!llvm.loop !.*]]
; CHECK: middle.block: ; CHECK: middle.block:
; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 8, 8 ; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 8, 8
; CHECK-NEXT: br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]] ; CHECK-NEXT: br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]]
; CHECK: scalar.ph: ; CHECK: scalar.ph:
; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 8, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ] ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 8, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
; CHECK-NEXT: br label [[FOR_BODY:%.*]] ; CHECK-NEXT: br label [[FOR_BODY:%.*]]
; CHECK: for.body: ; CHECK: for.body:
; CHECK-NEXT: [[INDVARS_IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ] ; CHECK-NEXT: [[INDVARS_IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
Show All 34 Lines

llvm/test/Transforms/LoopVectorize/X86/pr42674.ll

; NOTE: Assertions have been autogenerated by utils/update_test_checks.py ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
; RUN: opt %s -loop-vectorize -instcombine -simplifycfg -simplifycfg-require-and-preserve-domtree=1 -mtriple=x86_64-unknown-linux-gnu -mattr=avx512vl,avx512dq,avx512bw -S | FileCheck %s ; RUN: opt %s -loop-vectorize -instcombine -simplifycfg -simplifycfg-require-and-preserve-domtree=1 -mtriple=x86_64-unknown-linux-gnu -mattr=avx512vl,avx512dq,avx512bw -S | FileCheck %s
@bytes = global [128 x i8] zeroinitializer, align 16 @bytes = global [128 x i8] zeroinitializer, align 16
; Make sure we end up with vector code for this loop. We used to try to create ; Make sure we end up with vector code for this loop. We used to try to create
; a VF=64,UF=4 loop, but the scalar trip count is only 128 so ; a VF=64,UF=4 loop, but the scalar trip count is only 128 so
; the vector loop was dead code leaving only a scalar remainder. ; the vector loop was dead code leaving only a scalar remainder.
define zeroext i8 @sum() { define zeroext i8 @sum() {
; CHECK-LABEL: @sum( ; CHECK-LABEL: @sum(
; CHECK-NEXT: iter.check: ; CHECK-NEXT: iter.check:
; CHECK-NEXT: br label [[VECTOR_BODY:%.*]] ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds [128 x i8], [128 x i8]* @bytes, i64 0, i64 0
; CHECK: vector.body:
; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <64 x i8> [ zeroinitializer, [[ENTRY]] ], [ [[TMP4:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[VEC_PHI1:%.*]] = phi <64 x i8> [ zeroinitializer, [[ENTRY]] ], [ [[TMP5:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds [128 x i8], [128 x i8]* @bytes, i64 0, i64 [[INDEX]]
; CHECK-NEXT: [[TMP1:%.*]] = bitcast i8* [[TMP0]] to <64 x i8>* ; CHECK-NEXT: [[TMP1:%.*]] = bitcast i8* [[TMP0]] to <64 x i8>*
; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <64 x i8>, <64 x i8>* [[TMP1]], align 16 ; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <64 x i8>, <64 x i8>* [[TMP1]], align 16
; CHECK-NEXT: [[TMP2:%.*]] = getelementptr inbounds i8, i8* [[TMP0]], i64 64 ; CHECK-NEXT: [[TMP2:%.*]] = getelementptr inbounds i8, i8* [[TMP0]], i64 64
; CHECK-NEXT: [[TMP3:%.*]] = bitcast i8* [[TMP2]] to <64 x i8>* ; CHECK-NEXT: [[TMP3:%.*]] = bitcast i8* [[TMP2]] to <64 x i8>*
; CHECK-NEXT: [[WIDE_LOAD2:%.*]] = load <64 x i8>, <64 x i8>* [[TMP3]], align 16 ; CHECK-NEXT: [[WIDE_LOAD2:%.*]] = load <64 x i8>, <64 x i8>* [[TMP3]], align 16
; CHECK-NEXT: [[TMP4]] = add <64 x i8> [[WIDE_LOAD]], [[VEC_PHI]] ; CHECK-NEXT: [[TMP4:%.*]] = add <64 x i8> [[WIDE_LOAD]], zeroinitializer
; CHECK-NEXT: [[TMP5]] = add <64 x i8> [[WIDE_LOAD2]], [[VEC_PHI1]] ; CHECK-NEXT: [[TMP5:%.*]] = add <64 x i8> [[WIDE_LOAD2]], zeroinitializer
; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 128 ; CHECK-NEXT: [[INDEX_NEXT:%.*]] = add nuw i64 0, 128
; CHECK-NEXT: [[TMP6:%.*]] = icmp eq i64 [[INDEX]], 0
; CHECK-NEXT: br i1 [[TMP6]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !0
; CHECK: middle.block:
; CHECK-NEXT: [[BIN_RDX:%.*]] = add <64 x i8> [[TMP5]], [[TMP4]] ; CHECK-NEXT: [[BIN_RDX:%.*]] = add <64 x i8> [[TMP5]], [[TMP4]]
; CHECK-NEXT: [[TMP7:%.*]] = call i8 @llvm.vector.reduce.add.v64i8(<64 x i8> [[BIN_RDX]]) ; CHECK-NEXT: [[TMP6:%.*]] = call i8 @llvm.vector.reduce.add.v64i8(<64 x i8> [[BIN_RDX]])
; CHECK-NEXT: ret i8 [[TMP7]] ; CHECK-NEXT: ret i8 [[TMP6]]
; ;
entry: entry:
br label %for.body br label %for.body
for.body: ; preds = %for.body, %entry for.body: ; preds = %for.body, %entry
%indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ] %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
%r.010 = phi i8 [ 0, %entry ], [ %add, %for.body ] %r.010 = phi i8 [ 0, %entry ], [ %add, %for.body ]
%arrayidx = getelementptr inbounds [128 x i8], [128 x i8]* @bytes, i64 0, i64 %indvars.iv %arrayidx = getelementptr inbounds [128 x i8], [128 x i8]* @bytes, i64 0, i64 %indvars.iv
Show All 10 Lines