LLVM 19.0.0git
|
Planner drives the vectorization process after having passed Legality checks. More...
#include "Transforms/Vectorize/LoopVectorizationPlanner.h"
Public Member Functions | |
LoopVectorizationPlanner (Loop *L, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo &TTI, LoopVectorizationLegality *Legal, LoopVectorizationCostModel &CM, InterleavedAccessInfo &IAI, PredicatedScalarEvolution &PSE, const LoopVectorizeHints &Hints, OptimizationRemarkEmitter *ORE) | |
std::optional< VectorizationFactor > | plan (ElementCount UserVF, unsigned UserIC) |
Plan how to best vectorize, return the best VF and its cost, or std::nullopt if vectorization and interleaving should be avoided up front. | |
VectorizationFactor | planInVPlanNativePath (ElementCount UserVF) |
Use the VPlan-native path to plan how to best vectorize, return the best VF and its cost. | |
VPlan & | getBestPlanFor (ElementCount VF) const |
Return the best VPlan for VF . | |
std::pair< DenseMap< const SCEV *, Value * >, DenseMap< const RecurrenceDescriptor *, Value * > > | executePlan (ElementCount VF, unsigned UF, VPlan &BestPlan, InnerLoopVectorizer &LB, DominatorTree *DT, bool IsEpilogueVectorization, const DenseMap< const SCEV *, Value * > *ExpandedSCEVs=nullptr) |
Generate the IR code for the vectorized loop captured in VPlan BestPlan according to the best selected VF and UF . | |
void | printPlans (raw_ostream &O) |
bool | hasPlanWithVF (ElementCount VF) const |
Look through the existing plans and return true if we have one with vectorization factor VF . | |
VectorizationFactor | selectEpilogueVectorizationFactor (const ElementCount MaxVF, unsigned IC) |
Static Public Member Functions | |
static bool | getDecisionAndClampRange (const std::function< bool(ElementCount)> &Predicate, VFRange &Range) |
Test a Predicate on a Range of VF's. | |
Protected Member Functions | |
void | buildVPlans (ElementCount MinVF, ElementCount MaxVF) |
Build VPlans for power-of-2 VF's between MinVF and MaxVF inclusive, according to the information gathered by Legal when it checked if it is legal to vectorize the loop. | |
Planner drives the vectorization process after having passed Legality checks.
Definition at line 298 of file LoopVectorizationPlanner.h.
|
inline |
Definition at line 338 of file LoopVectorizationPlanner.h.
References llvm::IRSimilarity::Legal.
|
protected |
Build VPlans for power-of-2 VF's between MinVF
and MaxVF
inclusive, according to the information gathered by Legal when it checked if it is legal to vectorize the loop.
Build VPlans for the full range of feasible VF's = {MinVF
, 2 * MinVF
, 4 * MinVF
, ..., MaxVF}
by repeatedly building a VPlan for a sub-range of VF's starting at a given VF and extending it as much as possible.
Each vectorization decision can potentially shorten this sub-range during buildVPlan().
Definition at line 7891 of file LoopVectorize.cpp.
References llvm::VFRange::End, llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownLT(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().
Referenced by planInVPlanNativePath().
std::pair< DenseMap< const SCEV *, Value * >, DenseMap< const RecurrenceDescriptor *, Value * > > LoopVectorizationPlanner::executePlan | ( | ElementCount | VF, |
unsigned | UF, | ||
VPlan & | BestPlan, | ||
InnerLoopVectorizer & | LB, | ||
DominatorTree * | DT, | ||
bool | IsEpilogueVectorization, | ||
const DenseMap< const SCEV *, Value * > * | ExpandedSCEVs = nullptr |
||
) |
Generate the IR code for the vectorized loop captured in VPlan BestPlan
according to the best selected VF
and UF
.
TODO: IsEpilogueVectorization
is needed to avoid issues due to epilogue vectorization re-using plans for both the main and epilogue vector loops. It should be removed once the re-use issue has been fixed. ExpandedSCEVs
is passed during execution of the plan for epilogue loop to re-use expansion results generated during main plan execution.
Returns a mapping of SCEVs to their expanded IR values and a mapping for the reduction resume values. Note that this is a temporary workaround needed due to the current epilogue handling.
Definition at line 7439 of file LoopVectorize.cpp.
References AddRuntimeUnrollDisableMetaData(), assert(), llvm::InnerLoopVectorizer::Builder, llvm::VPTransformState::Builder, llvm::VPTransformState::CFG, createAndCollectMergePhiForReduction(), llvm::InnerLoopVectorizer::createVectorizedLoopSkeleton(), llvm::dbgs(), llvm::VPlan::dump(), llvm::SmallVectorBase< Size_T >::empty(), llvm::VPBasicBlock::empty(), llvm::VPlan::execute(), llvm::VPBasicBlock::execute(), llvm::VPTransformState::ExpandedSCEVs, llvm::InnerLoopVectorizer::fixVectorizedLoop(), llvm::VPTransformState::get(), llvm::RuntimePointerChecking::getChecks(), llvm::BasicBlock::getContext(), llvm::RuntimePointerChecking::getDiffChecks(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::LoopVectorizationLegality::getLAI(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::Loop::getLoopID(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::InnerLoopVectorizer::getOrCreateVectorTripCount(), llvm::VPlan::getPreheader(), llvm::LoopAccessInfo::getRuntimePointerChecking(), llvm::PredicatedScalarEvolution::getSE(), llvm::VPBlockBase::getSingleSuccessor(), llvm::BasicBlock::getTerminator(), llvm::InnerLoopVectorizer::getTripCount(), llvm::VPlan::getTripCount(), llvm::TargetTransformInfo::getUnrollingPreferences(), llvm::VPlan::getVectorLoopRegion(), llvm::VPlan::hasUF(), llvm::VPlan::hasVF(), llvm::InnerLoopVectorizer::Legal, LLVM_DEBUG, LLVMLoopVectorizeFollowupAll, LLVMLoopVectorizeFollowupVectorized, llvm::VPTransformState::LVer, llvm::makeFollowupLoopID(), llvm::VPlanTransforms::optimizeForVFAndUF(), llvm::LoopVersioning::prepareNoAliasMetadata(), llvm::VPlan::prepareToExecute(), llvm::VPTransformState::CFGState::PrevBB, llvm::InnerLoopVectorizer::printDebugTracesAtEnd(), llvm::InnerLoopVectorizer::printDebugTracesAtStart(), llvm::LoopVectorizeHints::setAlreadyVectorized(), llvm::IRBuilderBase::SetInsertPoint(), llvm::VPlan::setName(), llvm::InnerLoopVectorizer::setTripCount(), llvm::TargetTransformInfo::UnrollingPreferences::UnrollVectorizedLoop, and llvm::VPTransformState::CFGState::VPBB2IRBB.
Referenced by llvm::LoopVectorizePass::processLoop(), and processLoopInVPlanNativePath().
VPlan & LoopVectorizationPlanner::getBestPlanFor | ( | ElementCount | VF | ) | const |
Return the best VPlan for VF
.
Definition at line 7333 of file LoopVectorize.cpp.
References assert(), llvm::count_if(), and llvm_unreachable.
Referenced by llvm::LoopVectorizePass::processLoop(), and processLoopInVPlanNativePath().
|
static |
Test a Predicate
on a Range
of VF's.
Return the value of applying Predicate
on Range.Start, possibly decreasing Range.End such that the returned value holds for the entire Range
.
Definition at line 7872 of file LoopVectorize.cpp.
References assert().
Referenced by llvm::VPRecipeBuilder::handleReplication(), and llvm::VPRecipeBuilder::tryToCreateWidenRecipe().
|
inline |
Look through the existing plans and return true if we have one with vectorization factor VF
.
Definition at line 383 of file LoopVectorizationPlanner.h.
References llvm::any_of().
Referenced by plan(), and selectEpilogueVectorizationFactor().
std::optional< VectorizationFactor > LoopVectorizationPlanner::plan | ( | ElementCount | UserVF, |
unsigned | UserIC | ||
) |
Plan how to best vectorize, return the best VF and its cost, or std::nullopt if vectorization and interleaving should be avoided up front.
Definition at line 7247 of file LoopVectorize.cpp.
References assert(), llvm::LoopVectorizationCostModel::blockNeedsPredicationForAnyReason(), llvm::LoopVectorizationCostModel::collectElementTypesForWidening(), llvm::LoopVectorizationCostModel::collectInLoopReductions(), llvm::LoopVectorizationCostModel::collectInstsToScalarize(), llvm::LoopVectorizationCostModel::collectUniformsAndScalars(), llvm::LoopVectorizationCostModel::collectValuesToIgnore(), llvm::LoopVectorizationCostModel::computeMaxVF(), llvm::dbgs(), llvm::VectorizationFactor::Disabled(), llvm::FixedScalableVFPair::FixedVF, llvm::ElementCount::getFixed(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::ElementCount::getScalable(), hasPlanWithVF(), llvm::FixedScalableVFPair::hasVector(), llvm::SmallSet< T, N, C >::insert(), llvm::LoopVectorizationCostModel::InterleaveInfo, llvm::LoopVectorizationCostModel::invalidateCostModelingDecisions(), llvm::InterleavedAccessInfo::invalidateGroups(), llvm::LoopBase< BlockT, LoopT >::isInnermost(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownLE(), llvm::isPowerOf2_32(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::ElementCount::isVector(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isZero(), LLVM_DEBUG, printPlans(), llvm::reportVectorizationInfo(), llvm::FixedScalableVFPair::ScalableVF, llvm::VectorizationFactor::ScalarCost, llvm::LoopVectorizationCostModel::selectUserVectorizationFactor(), useMaskedInterleavedAccesses(), and llvm::VectorizationFactor::Width.
Referenced by llvm::LoopVectorizePass::processLoop().
VectorizationFactor LoopVectorizationPlanner::planInVPlanNativePath | ( | ElementCount | UserVF | ) |
Use the VPlan-native path to plan how to best vectorize, return the best VF and its cost.
Definition at line 7195 of file LoopVectorize.cpp.
References assert(), buildVPlans(), llvm::dbgs(), determineVPlanVF(), llvm::VectorizationFactor::Disabled(), llvm::EnableVPlanNativePath, ForceTargetSupportsScalableVectors, llvm::ElementCount::getFixed(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::LoopBase< BlockT, LoopT >::isInnermost(), llvm::isPowerOf2_32(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isZero(), LLVM_DEBUG, llvm::reportVectorizationFailure(), llvm::TargetTransformInfo::supportsScalableVectors(), and VPlanBuildStressTest.
Referenced by processLoopInVPlanNativePath().
void LoopVectorizationPlanner::printPlans | ( | raw_ostream & | O | ) |
Definition at line 7566 of file LoopVectorize.cpp.
References PrintVPlansInDotFormat.
Referenced by plan().
VectorizationFactor LoopVectorizationPlanner::selectEpilogueVectorizationFactor | ( | const ElementCount | MaxVF, |
unsigned | IC | ||
) |
Definition at line 5067 of file LoopVectorize.cpp.
References llvm::createTripCountSCEV(), llvm::dbgs(), llvm::VectorizationFactor::Disabled(), EnableEpilogueVectorization, EpilogueVectorizationForceVF, llvm::ScalarEvolution::getConstant(), llvm::ElementCount::getFixed(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::BasicBlock::getParent(), llvm::PredicatedScalarEvolution::getSE(), llvm::ScalarEvolution::getURemExpr(), getVScaleForTuning(), llvm::LoopVectorizationLegality::getWidestInductionType(), llvm::Function::hasMinSize(), llvm::Function::hasOptSize(), hasPlanWithVF(), llvm::CmpInst::ICMP_UGT, llvm::LoopVectorizationCostModel::isEpilogueVectorizationProfitable(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownGE(), llvm::ScalarEvolution::isKnownPredicate(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::LoopVectorizationCostModel::isScalarEpilogueAllowed(), and LLVM_DEBUG.
Referenced by llvm::LoopVectorizePass::processLoop().