llvm::LoopVectorizationPlanner Class Reference

Planner drives the vectorization process after having passed Legality checks. More...

#include "Transforms/Vectorize/LoopVectorizationPlanner.h"

Public Member Functions
	LoopVectorizationPlanner (Loop *L, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo &TTI, LoopVectorizationLegality *Legal, LoopVectorizationCostModel &CM, InterleavedAccessInfo &IAI, PredicatedScalarEvolution &PSE, const LoopVectorizeHints &Hints, OptimizationRemarkEmitter *ORE)
void	plan (ElementCount UserVF, unsigned UserIC)
	Build VPlans for the specified UserVF and UserIC if they are non-zero or all applicable candidate VFs otherwise.
VectorizationFactor	planInVPlanNativePath (ElementCount UserVF)
	Use the VPlan-native path to plan how to best vectorize, return the best VF and its cost.
VPlan &	getPlanFor (ElementCount VF) const
	Return the VPlan for VF.
VectorizationFactor	computeBestVF ()
	Compute and return the most profitable vectorization factor.
unsigned	selectInterleaveCount (VPlan &Plan, ElementCount VF, InstructionCost LoopCost)
DenseMap< const SCEV *, Value * >	executePlan (ElementCount VF, unsigned UF, VPlan &BestPlan, InnerLoopVectorizer &LB, DominatorTree *DT, bool VectorizingEpilogue)
	Generate the IR code for the vectorized loop captured in VPlan BestPlan according to the best selected VF and UF.
void	printPlans (raw_ostream &O)
bool	hasPlanWithVF (ElementCount VF) const
	Look through the existing plans and return true if we have one with vectorization factor VF.
VectorizationFactor	selectEpilogueVectorizationFactor (const ElementCount MaxVF, unsigned IC)
void	emitInvalidCostRemarks (OptimizationRemarkEmitter *ORE)
	Emit remarks for recipes with invalid costs in the available VPlans.
void	addMinimumIterationCheck (VPlan &Plan, ElementCount VF, unsigned UF, ElementCount MinProfitableTripCount) const
	Create a check to Plan to see if the vector loop should be executed based on its trip count.
void	updateLoopMetadataAndProfileInfo (Loop *VectorLoop, VPBasicBlock *HeaderVPBB, const VPlan &Plan, bool VectorizingEpilogue, MDNode *OrigLoopID, std::optional< unsigned > OrigAverageTripCount, unsigned OrigLoopInvocationWeight, unsigned EstimatedVFxUF, bool DisableRuntimeUnroll)
	Update loop metadata and profile info for both the scalar remainder loop and VectorLoop, if it exists.

Static Public Member Functions
static bool	getDecisionAndClampRange (const std::function< bool(ElementCount)> &Predicate, VFRange &Range)
	Test a Predicate on a Range of VF's.

Protected Member Functions
void	buildVPlans (ElementCount MinVF, ElementCount MaxVF)
	Build VPlans for power-of-2 VF's between MinVF and MaxVF inclusive, according to the information gathered by Legal when it checked if it is legal to vectorize the loop.

Detailed Description

Planner drives the vectorization process after having passed Legality checks.

Definition at line 434 of file LoopVectorizationPlanner.h.

Constructor & Destructor Documentation

LoopVectorizationPlanner()

llvm::LoopVectorizationPlanner::LoopVectorizationPlanner ( Loop * L, LoopInfo * LI, DominatorTree * DT, const TargetLibraryInfo * TLI, const TargetTransformInfo & TTI, LoopVectorizationLegality * Legal, LoopVectorizationCostModel & CM, InterleavedAccessInfo & IAI, PredicatedScalarEvolution & PSE, const LoopVectorizeHints & Hints, OptimizationRemarkEmitter * ORE )

inline

Definition at line 490 of file LoopVectorizationPlanner.h.

Member Function Documentation

addMinimumIterationCheck()

void LoopVectorizationPlanner::addMinimumIterationCheck	(	VPlan &	Plan,
		ElementCount	VF,
		unsigned	UF,
		ElementCount	MinProfitableTripCount ) const

Create a check to Plan to see if the vector loop should be executed based on its trip count.

Definition at line 9199 of file LoopVectorize.cpp.

References llvm::VPlanTransforms::addMinimumIterationCheck(), llvm::DataAndControlFlowWithoutRuntimeCheck, llvm::hasBranchWeightMD(), isIndvarOverflowCheckKnownFalse(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isVector(), and MinItersBypassWeights.

Referenced by llvm::LoopVectorizePass::processLoop(), and processLoopInVPlanNativePath().

buildVPlans()

void LoopVectorizationPlanner::buildVPlans ( ElementCount MinVF, ElementCount MaxVF )

protected

Build VPlans for power-of-2 VF's between MinVF and MaxVF inclusive, according to the information gathered by Legal when it checked if it is legal to vectorize the loop.

Build VPlans for the full range of feasible VF's = {MinVF, 2 * MinVF, 4 * MinVF, ..., MaxVF} by repeatedly building a VPlan for a sub-range of VF's starting at a given VF and extending it as much as possible.

Each vectorization decision can potentially shorten this sub-range during buildVPlan().

Definition at line 1595 of file VPlan.cpp.

References llvm::VFRange::End, llvm::VPRegionBlock::getExiting(), llvm::VPlan::getVectorLoopRegion(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownLT(), llvm::VPlanTransforms::optimize(), and llvm::VPBlockBase::setName().

Referenced by planInVPlanNativePath().

computeBestVF()

VectorizationFactor LoopVectorizationPlanner::computeBestVF

(

)

Compute and return the most profitable vectorization factor.

Also collect all profitable VFs in ProfitableVFs.

Definition at line 6993 of file LoopVectorize.cpp.

References llvm::any_of(), assert(), llvm::calculateRegisterUsageForPlan(), llvm::VectorizationFactor::Cost, llvm::dbgs(), llvm::VectorizationFactor::Disabled(), llvm::LoopVectorizeHints::FK_Enabled, ForceTargetNumVectorRegs, llvm::ElementCount::getFixed(), llvm::InstructionCost::getMax(), getPlanFor(), llvm::VPlan::hasEarlyExit(), hasPlanWithVF(), hasReplicatorRegion(), I, llvm::ElementCount::isScalar(), LLVM_DEBUG, P, planContainsAdditionalSimplifications(), llvm::VectorizationFactor::ScalarCost, llvm::ArrayRef< T >::size(), llvm::size(), llvm::TargetTransformInfo::TCK_CodeSize, llvm::TargetTransformInfo::TCK_Latency, llvm::TargetTransformInfo::TCK_RecipThroughput, llvm::TargetTransformInfo::TCK_SizeAndLatency, llvm::VPlan::vectorFactors(), llvm::VectorizationFactor::Width, and willGenerateVectors().

Referenced by llvm::LoopVectorizePass::processLoop().

emitInvalidCostRemarks()

void LoopVectorizationPlanner::emitInvalidCostRemarks

(

OptimizationRemarkEmitter *

ORE

)

Emit remarks for recipes with invalid costs in the available VPlans.

Definition at line 3912 of file LoopVectorize.cpp.

References A(), llvm::ArrayRef(), assert(), B(), llvm::VPBlockUtils::blocksOnly(), llvm::TypeSwitch< T, ResultT >::Case(), llvm::cast(), llvm::dyn_cast(), llvm::SmallVectorImpl< T >::emplace_back(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::Value::getName(), llvm::Instruction::getOpcodeName(), I, Int, llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownLT(), llvm::reportVectorizationInfo(), llvm::sort(), llvm::CallingConv::Tail, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::try_emplace(), and llvm::vp_depth_first_deep().

Referenced by llvm::LoopVectorizePass::processLoop().

executePlan()

DenseMap< const SCEV *, Value * > LoopVectorizationPlanner::executePlan	(	ElementCount	VF,
		unsigned	UF,
		VPlan &	BestPlan,
		InnerLoopVectorizer &	LB,
		DominatorTree *	DT,
		bool	VectorizingEpilogue )

Generate the IR code for the vectorized loop captured in VPlan BestPlan according to the best selected VF and UF.

TODO: VectorizingEpilogue indicates if the executed VPlan is for the epilogue vector loop. It should be removed once the re-use issue has been fixed.

Returns a mapping of SCEVs to their expanded IR values. Note that this is a temporary workaround needed due to the current epilogue handling.

Definition at line 7182 of file LoopVectorize.cpp.

References llvm::InnerLoopVectorizer::AC, llvm::VPlanTransforms::addBranchWeightToMiddleTerminator(), assert(), llvm::InnerLoopVectorizer::Builder, llvm::VPlanTransforms::canonicalizeEVLLoops(), llvm::cast(), llvm::VPlanTransforms::convertToConcreteRecipes(), llvm::InnerLoopVectorizer::createVectorizedLoopSkeleton(), llvm::VPlanTransforms::cse(), DEBUG_TYPE, llvm::VPlanTransforms::dissolveLoopRegions(), estimateElementCount(), llvm::VPlan::execute(), llvm::VPlanTransforms::expandSCEVs(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), llvm::ScalarEvolution::forgetBlockAndLoopDispositions(), llvm::ScalarEvolution::forgetLcssaPhiWithNewPredecessor(), llvm::ScalarEvolution::forgetLoop(), llvm::VPlan::getEntry(), llvm::VPlan::getExitBlocks(), llvm::vputils::getFirstLoopHeader(), llvm::VPValue::getLiveInIRValue(), llvm::getLoopEstimatedTripCount(), llvm::VPlan::getScalarPreheader(), llvm::VPBlockBase::getSingleSuccessor(), llvm::InnerLoopVectorizer::getTripCount(), llvm::VPlan::getTripCount(), llvm::VPlan::getVectorPreheader(), llvm::hasBranchWeightMD(), llvm::VPlan::hasEarlyExit(), llvm::VPlan::hasUF(), llvm::VPlan::hasVF(), llvm::ElementCount::isScalar(), llvm::ElementCount::isVector(), llvm::VPlanTransforms::materializeBackedgeTakenCount(), llvm::VPlanTransforms::materializeBroadcasts(), llvm::VPlanTransforms::materializeBuildVectors(), llvm::VPlanTransforms::materializeVectorTripCount(), llvm::VPlanTransforms::materializeVFAndVFxUF(), llvm::VPlanTransforms::narrowInterleaveGroups(), llvm::VPlanTransforms::optimizeForVFAndUF(), llvm::InnerLoopVectorizer::printDebugTracesAtEnd(), llvm::InnerLoopVectorizer::printDebugTracesAtStart(), llvm::VPlanTransforms::removeBranchOnConst(), llvm::VPlanTransforms::removeDeadRecipes(), replaceVPBBWithIRVPBB(), llvm::VPlanTransforms::replicateByVF(), llvm::TargetTransformInfo::RGK_FixedWidthVector, llvm::InnerLoopVectorizer::RTChecks, llvm::VPlanTransforms::runPass(), llvm::InnerLoopVectorizer::setTripCount(), llvm::VPlanTransforms::simplifyRecipes(), llvm::VPlanTransforms::unrollByUF(), updateLoopMetadataAndProfileInfo(), and llvm::verifyVPlanIsValid().

Referenced by llvm::LoopVectorizePass::processLoop(), and processLoopInVPlanNativePath().

getDecisionAndClampRange()

bool LoopVectorizationPlanner::getDecisionAndClampRange ( const std::function< bool(ElementCount)> & Predicate, VFRange & Range )

static

Test a Predicate on a Range of VF's.

Return the value of applying Predicate on Range.Start, possibly decreasing Range.End such that the returned value holds for the entire Range.

Definition at line 1576 of file VPlan.cpp.

References assert(), and Range.

Referenced by addExitUsersForFirstOrderRecurrences(), llvm::VPRecipeBuilder::handleReplication(), llvm::VPRecipeBuilder::tryToCreateWidenRecipe(), tryToMatchAndCreateExtendedReduction(), and tryToMatchAndCreateMulAccumulateReduction().

getPlanFor()

VPlan & LoopVectorizationPlanner::getPlanFor

(

ElementCount

VF

)

const

Return the VPlan for VF.

At the moment, there is always a single VPlan for each VF.

Definition at line 1611 of file VPlan.cpp.

References assert(), llvm::count_if(), and llvm_unreachable.

Referenced by computeBestVF(), llvm::LoopVectorizePass::processLoop(), processLoopInVPlanNativePath(), and selectEpilogueVectorizationFactor().

hasPlanWithVF()

bool llvm::LoopVectorizationPlanner::hasPlanWithVF ( ElementCount VF ) const

inline

Look through the existing plans and return true if we have one with vectorization factor VF.

Definition at line 545 of file LoopVectorizationPlanner.h.

References llvm::any_of().

Referenced by computeBestVF(), llvm::LoopVectorizePass::processLoop(), and selectEpilogueVectorizationFactor().

plan()

void LoopVectorizationPlanner::plan	(	ElementCount	UserVF,
		unsigned	UserIC )

Build VPlans for the specified UserVF and UserIC if they are non-zero or all applicable candidate VFs otherwise.

If vectorization and interleaving should be avoided up-front, no plans are generated.

Definition at line 6623 of file LoopVectorize.cpp.

References assert(), llvm::dbgs(), llvm::FixedScalableVFPair::FixedVF, llvm::ElementCount::getFixed(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::ElementCount::getScalable(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownLE(), llvm::isPowerOf2_32(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), LLVM_DEBUG, printPlans(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::reportVectorizationInfo(), llvm::FixedScalableVFPair::ScalableVF, and useMaskedInterleavedAccesses().

Referenced by llvm::LoopVectorizePass::processLoop().

planInVPlanNativePath()

VectorizationFactor LoopVectorizationPlanner::planInVPlanNativePath

(

ElementCount

UserVF

)

Use the VPlan-native path to plan how to best vectorize, return the best VF and its cost.

Definition at line 6569 of file LoopVectorize.cpp.

References assert(), buildVPlans(), llvm::dbgs(), determineVPlanVF(), llvm::VectorizationFactor::Disabled(), llvm::EnableVPlanNativePath, ForceTargetSupportsScalableVectors, llvm::ElementCount::getFixed(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::isPowerOf2_32(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isZero(), LLVM_DEBUG, llvm::reportVectorizationFailure(), and VPlanBuildStressTest.

Referenced by processLoopInVPlanNativePath().

printPlans()

void LoopVectorizationPlanner::printPlans

(

raw_ostream &

O

)

Definition at line 1740 of file VPlan.cpp.

References llvm::VPlan::print(), llvm::VPlan::printDOT(), and PrintVPlansInDotFormat.

Referenced by plan().

selectEpilogueVectorizationFactor()

VectorizationFactor LoopVectorizationPlanner::selectEpilogueVectorizationFactor	(	const ElementCount	MaxVF,
		unsigned	IC )

Returns

The most profitable vectorization factor and the cost of that VF for vectorizing the epilogue. Returns VectorizationFactor::Disabled if epilogue vectorization is not supported for the loop.

Definition at line 4336 of file LoopVectorize.cpp.

References assert(), llvm::dbgs(), llvm::VectorizationFactor::Disabled(), EnableEpilogueVectorization, EpilogueVectorizationForceVF, estimateElementCount(), llvm::ScalarEvolution::getConstant(), llvm::ScalarEvolution::getElementCount(), llvm::ElementCount::getFixed(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getFixedValue(), getPlanFor(), llvm::vputils::getSCEVExprForVPValue(), llvm::ScalarEvolution::getUnsignedRangeMax(), llvm::ScalarEvolution::getURemExpr(), llvm::APInt::getZExtValue(), hasPlanWithVF(), llvm::CmpInst::ICMP_UGT, llvm::CmpInst::ICMP_ULT, llvm::isa(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isFixed(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownGE(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownGT(), llvm::ScalarEvolution::isKnownPredicate(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::SCEV::isZero(), and LLVM_DEBUG.

Referenced by llvm::LoopVectorizePass::processLoop().

selectInterleaveCount()

unsigned LoopVectorizationPlanner::selectInterleaveCount	(	VPlan &	Plan,
		ElementCount	VF,
		InstructionCost	LoopCost )

Returns

The desired interleave count. If interleave count has been specified by metadata it will be returned. Otherwise, the interleave count is computed and returned. VF and LoopCost are the selected vectorization factor and the cost of the selected VF.

Definition at line 4522 of file LoopVectorize.cpp.

References llvm::any_of(), assert(), llvm::bit_floor(), llvm::VPBlockUtils::blocksOnly(), llvm::calculateRegisterUsageForPlan(), llvm::dbgs(), llvm::dyn_cast(), EnableIndVarRegisterHeur, EnableLoadStoreRuntimeInterleave, estimateElementCount(), F, ForceTargetMaxScalarInterleaveFactor, ForceTargetMaxVectorInterleaveFactor, ForceTargetNumScalarRegs, ForceTargetNumVectorRegs, llvm::VPRegionBlock::getEntry(), llvm::VPBlockBase::getEntryBasicBlock(), getSmallBestKnownTC(), getSmallConstantTripCount(), llvm::InstructionCost::getValue(), llvm::VPlan::getVectorLoopRegion(), llvm::VPlan::hasEarlyExit(), llvm::isa(), llvm::IsaPred, llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isNonZero(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::InstructionCost::isValid(), llvm::ElementCount::isVector(), LLVM_DEBUG, MaxNestedScalarReductionIC, llvm::VPBasicBlock::phis(), SmallLoopCost, and llvm::vp_depth_first_deep().

Referenced by llvm::LoopVectorizePass::processLoop().

updateLoopMetadataAndProfileInfo()

void LoopVectorizationPlanner::updateLoopMetadataAndProfileInfo	(	Loop *	VectorLoop,
		VPBasicBlock *	HeaderVPBB,
		const VPlan &	Plan,
		bool	VectorizingEpilogue,
		MDNode *	OrigLoopID,
		std::optional< unsigned >	OrigAverageTripCount,
		unsigned	OrigLoopInvocationWeight,
		unsigned	EstimatedVFxUF,
		bool	DisableRuntimeUnroll )

Update loop metadata and profile info for both the scalar remainder loop and VectorLoop, if it exists.

Keeps all loop hints from the original loop on the vector loop and replaces vectorizer-specific metadata. The loop ID of the original loop OrigLoopID must be passed, together with the average trip count and invocation weight of the original loop (OrigAverageTripCount and OrigLoopInvocationWeight respectively). They cannot be retrieved after the plan has been executed, as the original loop may have been removed.

Definition at line 1662 of file VPlan.cpp.

References addRuntimeUnrollDisableMetaData(), llvm::VPlan::getScalarPreheader(), llvm::VPBlockBase::hasPredecessors(), LLVMLoopVectorizeFollowupAll, LLVMLoopVectorizeFollowupEpilogue, LLVMLoopVectorizeFollowupVectorized, llvm::makeFollowupLoopID(), llvm::setLoopEstimatedTripCount(), llvm::Loop::setLoopID(), and llvm::TargetTransformInfo::UnrollingPreferences::UnrollVectorizedLoop.

Referenced by executePlan().

---

The documentation for this class was generated from the following files:

• lib/Transforms/Vectorize/LoopVectorizationPlanner.h
• lib/Transforms/Vectorize/LoopVectorize.cpp
• lib/Transforms/Vectorize/VPlan.cpp