LLVM 20.0.0git
Public Member Functions | Static Public Member Functions | Protected Member Functions | List of all members
llvm::LoopVectorizationPlanner Class Reference

Planner drives the vectorization process after having passed Legality checks. More...

#include "Transforms/Vectorize/LoopVectorizationPlanner.h"

Public Member Functions

 LoopVectorizationPlanner (Loop *L, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo &TTI, LoopVectorizationLegality *Legal, LoopVectorizationCostModel &CM, InterleavedAccessInfo &IAI, PredicatedScalarEvolution &PSE, const LoopVectorizeHints &Hints, OptimizationRemarkEmitter *ORE)
 
void plan (ElementCount UserVF, unsigned UserIC)
 Build VPlans for the specified UserVF and UserIC if they are non-zero or all applicable candidate VFs otherwise.
 
VectorizationFactor planInVPlanNativePath (ElementCount UserVF)
 Use the VPlan-native path to plan how to best vectorize, return the best VF and its cost.
 
VPlangetPlanFor (ElementCount VF) const
 Return the VPlan for VF.
 
VectorizationFactor computeBestVF ()
 Compute and return the most profitable vectorization factor.
 
DenseMap< const SCEV *, Value * > executePlan (ElementCount VF, unsigned UF, VPlan &BestPlan, InnerLoopVectorizer &LB, DominatorTree *DT, bool VectorizingEpilogue, const DenseMap< const SCEV *, Value * > *ExpandedSCEVs=nullptr)
 Generate the IR code for the vectorized loop captured in VPlan BestPlan according to the best selected VF and UF.
 
void printPlans (raw_ostream &O)
 
bool hasPlanWithVF (ElementCount VF) const
 Look through the existing plans and return true if we have one with vectorization factor VF.
 
VectorizationFactor selectEpilogueVectorizationFactor (const ElementCount MaxVF, unsigned IC)
 
void emitInvalidCostRemarks (OptimizationRemarkEmitter *ORE)
 Emit remarks for recipes with invalid costs in the available VPlans.
 

Static Public Member Functions

static bool getDecisionAndClampRange (const std::function< bool(ElementCount)> &Predicate, VFRange &Range)
 Test a Predicate on a Range of VF's.
 

Protected Member Functions

void buildVPlans (ElementCount MinVF, ElementCount MaxVF)
 Build VPlans for power-of-2 VF's between MinVF and MaxVF inclusive, according to the information gathered by Legal when it checked if it is legal to vectorize the loop.
 

Detailed Description

Planner drives the vectorization process after having passed Legality checks.

Definition at line 353 of file LoopVectorizationPlanner.h.

Constructor & Destructor Documentation

◆ LoopVectorizationPlanner()

llvm::LoopVectorizationPlanner::LoopVectorizationPlanner ( Loop L,
LoopInfo LI,
DominatorTree DT,
const TargetLibraryInfo TLI,
const TargetTransformInfo TTI,
LoopVectorizationLegality Legal,
LoopVectorizationCostModel CM,
InterleavedAccessInfo IAI,
PredicatedScalarEvolution PSE,
const LoopVectorizeHints Hints,
OptimizationRemarkEmitter ORE 
)
inline

Definition at line 409 of file LoopVectorizationPlanner.h.

References llvm::IRSimilarity::Legal.

Member Function Documentation

◆ buildVPlans()

void LoopVectorizationPlanner::buildVPlans ( ElementCount  MinVF,
ElementCount  MaxVF 
)
protected

Build VPlans for power-of-2 VF's between MinVF and MaxVF inclusive, according to the information gathered by Legal when it checked if it is legal to vectorize the loop.

Build VPlans for the full range of feasible VF's = {MinVF, 2 * MinVF, 4 * MinVF, ..., MaxVF} by repeatedly building a VPlan for a sub-range of VF's starting at a given VF and extending it as much as possible.

Each vectorization decision can potentially shorten this sub-range during buildVPlan().

Definition at line 1622 of file VPlan.cpp.

References llvm::VFRange::End, llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownLT(), llvm::VPlanTransforms::optimize(), llvm::VPTransformState::Plan, and llvm::VPTransformState::VF.

Referenced by planInVPlanNativePath().

◆ computeBestVF()

VectorizationFactor LoopVectorizationPlanner::computeBestVF ( )

◆ emitInvalidCostRemarks()

void LoopVectorizationPlanner::emitInvalidCostRemarks ( OptimizationRemarkEmitter ORE)

◆ executePlan()

DenseMap< const SCEV *, Value * > LoopVectorizationPlanner::executePlan ( ElementCount  VF,
unsigned  UF,
VPlan BestPlan,
InnerLoopVectorizer LB,
DominatorTree DT,
bool  VectorizingEpilogue,
const DenseMap< const SCEV *, Value * > *  ExpandedSCEVs = nullptr 
)

Generate the IR code for the vectorized loop captured in VPlan BestPlan according to the best selected VF and UF.

TODO: VectorizingEpilogue indicates if the executed VPlan is for the epilogue vector loop. It should be removed once the re-use issue has been fixed. ExpandedSCEVs is passed during execution of the plan for epilogue loop to re-use expansion results generated during main plan execution.

Returns a mapping of SCEVs to their expanded IR values. Note that this is a temporary workaround needed due to the current epilogue handling.

Definition at line 7787 of file LoopVectorize.cpp.

References _, addRuntimeUnrollDisableMetaData(), assert(), llvm::InnerLoopVectorizer::Builder, llvm::VPTransformState::Builder, llvm::VPTransformState::CFG, llvm::VPlanTransforms::convertToConcreteRecipes(), llvm::InnerLoopVectorizer::createVectorizedLoopSkeleton(), llvm::SmallVectorBase< Size_T >::empty(), llvm::VPBasicBlock::empty(), llvm::VPlan::execute(), llvm::VPBasicBlock::execute(), llvm::VPTransformState::ExpandedSCEVs, fixReductionScalarResumeWhenVectorizingEpilog(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), llvm::VPTransformState::get(), llvm::InnerLoopVectorizer::getAdditionalBypassBlock(), llvm::RuntimePointerChecking::getChecks(), llvm::BasicBlock::getContext(), llvm::RuntimePointerChecking::getDiffChecks(), llvm::VPlan::getEntry(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::InnerLoopVectorizer::getInductionAdditionalBypassValue(), llvm::LoopVectorizationLegality::getInductionVars(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::LoopVectorizationLegality::getLAI(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::Loop::getLoopID(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::VPlan::getMiddleBlock(), llvm::InnerLoopVectorizer::getOrCreateVectorTripCount(), llvm::LoopAccessInfo::getRuntimePointerChecking(), llvm::PredicatedScalarEvolution::getSE(), llvm::BasicBlock::getTerminator(), llvm::InnerLoopVectorizer::getTripCount(), llvm::VPlan::getTripCount(), llvm::VPlan::getUF(), llvm::TargetTransformInfo::getUnrollingPreferences(), llvm::VPlan::getVectorLoopRegion(), llvm::LoopVectorizationLegality::getWidestInductionType(), llvm::hasBranchWeightMD(), llvm::VPlan::hasUF(), llvm::LoopVectorizationLegality::hasUncountableEarlyExit(), llvm::VPlan::hasVF(), llvm::InnerLoopVectorizer::Legal, LLVMLoopVectorizeFollowupAll, LLVMLoopVectorizeFollowupVectorized, llvm::InnerLoopVectorizer::LoopVectorPreHeader, llvm::VPTransformState::LVer, llvm::makeFollowupLoopID(), llvm::VPlanTransforms::optimizeForVFAndUF(), llvm::LoopVersioning::prepareNoAliasMetadata(), llvm::VPlan::prepareToExecute(), llvm::VPTransformState::CFGState::PrevBB, llvm::InnerLoopVectorizer::printDebugTracesAtEnd(), llvm::InnerLoopVectorizer::printDebugTracesAtStart(), llvm::VPlanTransforms::removeDeadRecipes(), llvm::LoopVectorizeHints::setAlreadyVectorized(), llvm::setBranchWeights(), llvm::IRBuilderBase::SetInsertPoint(), llvm::InnerLoopVectorizer::setTripCount(), llvm::VPlanTransforms::unrollByUF(), llvm::TargetTransformInfo::UnrollingPreferences::UnrollVectorizedLoop, llvm::DominatorTreeBase< NodeT, IsPostDom >::verify(), llvm::VPTransformState::VF, and llvm::VPTransformState::CFGState::VPBB2IRBB.

Referenced by llvm::LoopVectorizePass::processLoop(), and processLoopInVPlanNativePath().

◆ getDecisionAndClampRange()

bool LoopVectorizationPlanner::getDecisionAndClampRange ( const std::function< bool(ElementCount)> &  Predicate,
VFRange Range 
)
static

Test a Predicate on a Range of VF's.

Return the value of applying Predicate on Range.Start, possibly decreasing Range.End such that the returned value holds for the entire Range.

Definition at line 1603 of file VPlan.cpp.

References assert(), and Range.

Referenced by llvm::VPRecipeBuilder::handleReplication(), and llvm::VPRecipeBuilder::tryToCreateWidenRecipe().

◆ getPlanFor()

VPlan & LoopVectorizationPlanner::getPlanFor ( ElementCount  VF) const

◆ hasPlanWithVF()

bool llvm::LoopVectorizationPlanner::hasPlanWithVF ( ElementCount  VF) const
inline

Look through the existing plans and return true if we have one with vectorization factor VF.

Definition at line 459 of file LoopVectorizationPlanner.h.

References llvm::any_of().

Referenced by computeBestVF(), llvm::LoopVectorizePass::processLoop(), and selectEpilogueVectorizationFactor().

◆ plan()

void LoopVectorizationPlanner::plan ( ElementCount  UserVF,
unsigned  UserIC 
)

Build VPlans for the specified UserVF and UserIC if they are non-zero or all applicable candidate VFs otherwise.

If vectorization and interleaving should be avoided up-front, no plans are generated.

Definition at line 7270 of file LoopVectorize.cpp.

References assert(), llvm::LoopVectorizationCostModel::blockNeedsPredicationForAnyReason(), llvm::LoopVectorizationCostModel::collectElementTypesForWidening(), llvm::LoopVectorizationCostModel::collectInLoopReductions(), llvm::LoopVectorizationCostModel::collectInstsToScalarize(), llvm::LoopVectorizationCostModel::collectUniformsAndScalars(), llvm::LoopVectorizationCostModel::collectValuesToIgnore(), llvm::LoopVectorizationCostModel::computeMaxVF(), llvm::dbgs(), llvm::FixedScalableVFPair::FixedVF, llvm::LoopVectorizationCostModel::foldTailByMasking(), llvm::ElementCount::getFixed(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::ElementCount::getScalable(), llvm::LoopVectorizationCostModel::InterleaveInfo, llvm::LoopVectorizationCostModel::invalidateCostModelingDecisions(), llvm::InterleavedAccessInfo::invalidateGroups(), llvm::LoopBase< BlockT, LoopT >::isInnermost(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownLE(), llvm::isPowerOf2_32(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isVector(), LLVM_DEBUG, llvm::LoopVectorizationLegality::prepareToFoldTailByMasking(), printPlans(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::reportVectorizationInfo(), llvm::FixedScalableVFPair::ScalableVF, llvm::LoopVectorizationCostModel::selectUserVectorizationFactor(), and useMaskedInterleavedAccesses().

Referenced by llvm::LoopVectorizePass::processLoop().

◆ planInVPlanNativePath()

VectorizationFactor LoopVectorizationPlanner::planInVPlanNativePath ( ElementCount  UserVF)

◆ printPlans()

void LoopVectorizationPlanner::printPlans ( raw_ostream O)

◆ selectEpilogueVectorizationFactor()

VectorizationFactor LoopVectorizationPlanner::selectEpilogueVectorizationFactor ( const ElementCount  MaxVF,
unsigned  IC 
)
Returns
The most profitable vectorization factor and the cost of that VF for vectorizing the epilogue. Returns VectorizationFactor::Disabled if epilogue vectorization is not supported for the loop.

Definition at line 4836 of file LoopVectorize.cpp.

References assert(), llvm::dbgs(), llvm::VectorizationFactor::Disabled(), EnableEpilogueVectorization, EpilogueVectorizationForceVF, llvm::ScalarEvolution::getConstant(), getEstimatedRuntimeVF(), llvm::ElementCount::getFixed(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::BasicBlock::getParent(), getPlanFor(), llvm::vputils::getSCEVExprForVPValue(), llvm::PredicatedScalarEvolution::getSE(), llvm::VPlan::getTripCount(), llvm::ScalarEvolution::getUnsignedRangeMax(), llvm::ScalarEvolution::getURemExpr(), llvm::LoopVectorizationLegality::getWidestInductionType(), llvm::APInt::getZExtValue(), llvm::Function::hasMinSize(), llvm::Function::hasOptSize(), hasPlanWithVF(), llvm::CmpInst::ICMP_UGT, llvm::CmpInst::ICMP_ULT, llvm::LoopVectorizationCostModel::isEpilogueVectorizationProfitable(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownGE(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownGT(), llvm::ScalarEvolution::isKnownPredicate(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::LoopVectorizationCostModel::isScalarEpilogueAllowed(), and LLVM_DEBUG.

Referenced by llvm::LoopVectorizePass::processLoop().


The documentation for this class was generated from the following files: