LLVM 20.0.0git
|
VPlan models a candidate for vectorization, encoding various decisions take to produce efficient output IR, including which branches, basic-blocks and output IR instructions to generate, and their cost. More...
#include "Transforms/Vectorize/VPlan.h"
Public Member Functions | |
VPlan (VPBasicBlock *Preheader, VPValue *TC, VPBasicBlock *Entry) | |
Construct a VPlan with original preheader Preheader , trip count TC and Entry to the plan. | |
VPlan (VPBasicBlock *Preheader, VPBasicBlock *Entry) | |
Construct a VPlan with original preheader Preheader and Entry to the plan. | |
~VPlan () | |
void | prepareToExecute (Value *TripCount, Value *VectorTripCount, Value *CanonicalIVStartValue, VPTransformState &State) |
Prepare the plan for execution, setting up the required live-in values. | |
void | execute (VPTransformState *State) |
Generate the IR code for this VPlan. | |
InstructionCost | cost (ElementCount VF, VPCostContext &Ctx) |
Return the cost of this plan. | |
VPBasicBlock * | getEntry () |
const VPBasicBlock * | getEntry () const |
VPValue * | getTripCount () const |
The trip count of the original loop. | |
void | resetTripCount (VPValue *NewTripCount) |
Resets the trip count for the VPlan. | |
VPValue * | getOrCreateBackedgeTakenCount () |
The backedge taken count of the original loop. | |
VPValue & | getVectorTripCount () |
The vector trip count. | |
VPValue & | getVFxUF () |
Returns VF * UF of the vector loop region. | |
void | addVF (ElementCount VF) |
void | setVF (ElementCount VF) |
bool | hasVF (ElementCount VF) |
bool | hasScalableVF () |
iterator_range< SmallSetVector< ElementCount, 2 >::iterator > | vectorFactors () const |
Returns an iterator range over all VFs of the plan. | |
bool | hasScalarVFOnly () const |
bool | hasUF (unsigned UF) const |
void | setUF (unsigned UF) |
std::string | getName () const |
Return a string with the name of the plan and the applicable VFs and UFs. | |
void | setName (const Twine &newName) |
VPValue * | getOrAddLiveIn (Value *V) |
Gets the live-in VPValue for V or adds a new live-in (if none exists yet) for V . | |
VPValue * | getLiveIn (Value *V) const |
Return the live-in VPValue for V , if there is one or nullptr otherwise. | |
void | printLiveIns (raw_ostream &O) const |
Print the live-ins of this VPlan to O . | |
void | print (raw_ostream &O) const |
Print this VPlan to O . | |
void | printDOT (raw_ostream &O) const |
Print this VPlan in DOT format to O . | |
LLVM_DUMP_METHOD void | dump () const |
Dump the plan to stderr (for debugging). | |
VPRegionBlock * | getVectorLoopRegion () |
Returns the VPRegionBlock of the vector loop. | |
const VPRegionBlock * | getVectorLoopRegion () const |
VPCanonicalIVPHIRecipe * | getCanonicalIV () |
Returns the canonical induction recipe of the vector loop. | |
void | addLiveOut (PHINode *PN, VPValue *V) |
const MapVector< PHINode *, VPLiveOut * > & | getLiveOuts () const |
VPValue * | getSCEVExpansion (const SCEV *S) const |
void | addSCEVExpansion (const SCEV *S, VPValue *V) |
VPBasicBlock * | getPreheader () |
const VPBasicBlock * | getPreheader () const |
VPlan * | duplicate () |
Clone the current VPlan, update all VPValues of the new VPlan and cloned recipes to refer to the clones, and return it. | |
Static Public Member Functions | |
static VPlanPtr | createInitialVPlan (const SCEV *TripCount, ScalarEvolution &PSE, bool RequiresScalarEpilogueCheck, bool TailFolded, Loop *TheLoop) |
Create initial VPlan, having an "entry" VPBasicBlock (wrapping original scalar pre-header ) which contains SCEV expansions that need to happen before the CFG is modified; a VPBasicBlock for the vector pre-header, followed by a region for the vector loop, followed by the middle VPBasicBlock. | |
Friends | |
class | VPlanPrinter |
class | VPSlotTracker |
VPlan models a candidate for vectorization, encoding various decisions take to produce efficient output IR, including which branches, basic-blocks and output IR instructions to generate, and their cost.
VPlan holds a Hierarchical-CFG of VPBasicBlocks and VPRegionBlocks rooted at an Entry VPBasicBlock.
|
inline |
|
inline |
Construct a VPlan with original preheader Preheader
and Entry
to the plan.
At the moment, Preheader
and Entry
need to be disconnected, as the bypass blocks between them are not yet modeled in VPlan.
Definition at line 3343 of file VPlan.h.
References assert(), llvm::VPBlockBase::getNumPredecessors(), llvm::VPBlockBase::getNumSuccessors(), and llvm::VPBlockBase::setPlan().
VPlan::~VPlan | ( | ) |
Definition at line 842 of file VPlan.cpp.
References llvm::Block, llvm::VPBlockBase::deleteCFG(), and llvm::vp_depth_first_shallow().
Definition at line 1187 of file VPlan.cpp.
References assert().
Referenced by addLiveOutsForFirstOrderRecurrences(), and addUsersInExitBlock().
Definition at line 3503 of file VPlan.h.
References assert(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains().
Referenced by llvm::vputils::getOrCreateVPValueForSCEVExpr().
|
inline |
Definition at line 3406 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::insert().
InstructionCost VPlan::cost | ( | ElementCount | VF, |
VPCostContext & | Ctx | ||
) |
Return the cost of this plan.
Definition at line 1091 of file VPlan.cpp.
References llvm::VPTransformState::VF.
|
static |
Create initial VPlan, having an "entry" VPBasicBlock (wrapping original scalar pre-header ) which contains SCEV expansions that need to happen before the CFG is modified; a VPBasicBlock for the vector pre-header, followed by a region for the vector loop, followed by the middle VPBasicBlock.
If a check is needed to guard executing the scalar epilogue loop, it will be added to the middle block, together with VPBasicBlocks for the scalar preheader and exit blocks.
Definition at line 863 of file VPlan.cpp.
References llvm::VPInstruction::BranchOnCond, llvm::VPTransformState::Builder, llvm::VPBlockUtils::connectBlocks(), llvm::Type::getContext(), llvm::Type::getInt1Ty(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), getOrAddLiveIn(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::BasicBlock::getTerminator(), getTripCount(), llvm::ConstantInt::getTrue(), llvm::SCEV::getType(), llvm::LoopBase< BlockT, LoopT >::getUniqueExitBlock(), getVectorTripCount(), llvm::CmpInst::ICMP_EQ, llvm::VPBlockUtils::insertBlockAfter(), and llvm::VPTransformState::Plan.
LLVM_DUMP_METHOD void VPlan::dump | ( | ) | const |
Dump the plan to stderr (for debugging).
Definition at line 1184 of file VPlan.cpp.
References llvm::dbgs(), and llvm::print().
Referenced by llvm::LoopVectorizationPlanner::executePlan().
VPlan * VPlan::duplicate | ( | ) |
Clone the current VPlan, update all VPValues of the new VPlan and cloned recipes to refer to the clones, and return it.
Definition at line 1230 of file VPlan.cpp.
References _, assert(), llvm::VPBasicBlock::clone(), cloneFrom(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains(), Name, and remapOperands().
Referenced by llvm::LoopVectorizePass::processLoop().
void VPlan::execute | ( | VPTransformState * | State | ) |
Generate the IR code for this VPlan.
Generate the code inside the preheader and body of the vectorized loop.
Assumes a single pre-header basic-block was created for this. Introduce additional basic-blocks as needed, and fill them all.
Definition at line 981 of file VPlan.cpp.
References llvm::GenericDomTreeUpdater< DerivedT, DomTreeT, PostDomTreeT >::applyUpdates(), assert(), llvm::Block, llvm::VPTransformState::Builder, llvm::VPTransformState::CFG, llvm::DominatorTreeBase< BasicBlock, false >::Delete, llvm::VPTransformState::CFGState::DTU, llvm::Instruction::eraseFromParent(), llvm::VPTransformState::CFGState::ExitBB, llvm::GenericDomTreeUpdater< DerivedT, DomTreeT, PostDomTreeT >::flush(), GEP, llvm::VPTransformState::get(), llvm::BasicBlock::getContext(), llvm::GenericDomTreeUpdater< DerivedT, DomTreeT, PostDomTreeT >::getDomTree(), llvm::VPBlockBase::getExitingBasicBlock(), llvm::ilist_node_with_parent< NodeTy, ParentTy, Options >::getPrevNode(), llvm::BasicBlock::getSingleSuccessor(), llvm::VPBlockBase::getSuccessors(), llvm::BasicBlock::getTerminator(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::Instruction::moveBefore(), llvm::VPTransformState::CFGState::PrevBB, llvm::VPTransformState::CFGState::PrevVPBB, replaceVPBBWithIRVPBB(), llvm::IRBuilderBase::SetInsertPoint(), llvm::VPTransformState::UF, llvm::DominatorTreeBase< NodeT, IsPostDom >::verify(), llvm::VPTransformState::VF, llvm::vp_depth_first_shallow(), and llvm::VPTransformState::CFGState::VPBB2IRBB.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
inline |
Returns the canonical induction recipe of the vector loop.
Definition at line 3484 of file VPlan.h.
References llvm::VPBasicBlock::begin(), llvm::VPBasicBlock::empty(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPBlockBase::getSingleSuccessor(), and getVectorLoopRegion().
Referenced by llvm::VPlanTransforms::addActiveLaneMask(), addLiveOutsForFirstOrderRecurrences(), addVPLaneMaskPhiAndUpdateExitBranch(), collectAllHeaderMasks(), llvm::VPRecipeBuilder::createHeaderMask(), createScalarIVSteps(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPlanTransforms::optimizeForVFAndUF(), removeRedundantCanonicalIVs(), simplifyRecipes(), llvm::VPlanTransforms::truncateToMinimalBitwidths(), llvm::VPlanTransforms::tryAddExplicitVectorLength(), and willGenerateVectors().
|
inline |
Definition at line 3376 of file VPlan.h.
Referenced by addReplicateRegions(), llvm::VPlanPrinter::dump(), llvm::InnerLoopVectorizer::fixNonInductionPHIs(), llvm::DomTreeNodeTraits< VPBlockBase >::getEntryNode(), getVectorLoopRegion(), mergeBlocksIntoPredecessors(), mergeReplicateRegionsIntoSuccessors(), removeDeadRecipes(), removeRedundantExpandSCEVRecipes(), llvm::VPBlockBase::setPlan(), simplifyRecipes(), sinkScalarOperands(), and llvm::VPlanTransforms::truncateToMinimalBitwidths().
|
inline |
Return the live-in VPValue for V
, if there is one or nullptr otherwise.
Definition at line 3459 of file VPlan.h.
References llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::lookup().
Definition at line 3495 of file VPlan.h.
Referenced by llvm::InnerLoopVectorizer::fixVectorizedLoop().
std::string VPlan::getName | ( | ) | const |
Return a string with the name of the plan and the applicable VFs and UFs.
Definition at line 1154 of file VPlan.cpp.
References llvm::drop_begin(), Name, llvm::VPTransformState::UF, and llvm::VPTransformState::VF.
Referenced by llvm::VPlanPrinter::dump().
Gets the live-in VPValue for V
or adds a new live-in (if none exists yet) for V
.
Definition at line 3442 of file VPlan.h.
References assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), llvm::VPValue::isLiveIn(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().
Referenced by addCanonicalIVRecipes(), addLiveOutsForFirstOrderRecurrences(), addUsersInExitBlock(), createInitialVPlan(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::VPRecipeBuilder::getVPValueOrAddLiveIn(), legalizeAndOptimizeInductions(), llvm::VPRecipeBuilder::mapToVPValues(), llvm::VPlanTransforms::optimizeForVFAndUF(), and llvm::LoopVectorizePass::processLoop().
|
inline |
The backedge taken count of the original loop.
Definition at line 3394 of file VPlan.h.
Referenced by llvm::VPRecipeBuilder::createHeaderMask(), and llvm::vputils::isHeaderMask().
|
inline |
Definition at line 3509 of file VPlan.h.
Referenced by llvm::VPlanPrinter::dump(), llvm::LoopVectorizationPlanner::executePlan(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::LoopVectorizePass::processLoop(), and llvm::VPBlockBase::setPlan().
|
inline |
Definition at line 3499 of file VPlan.h.
References llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::lookup().
Referenced by llvm::InnerLoopVectorizer::fixupIVUsers(), and llvm::vputils::getOrCreateVPValueForSCEVExpr().
|
inline |
The trip count of the original loop.
Definition at line 3380 of file VPlan.h.
References assert().
Referenced by llvm::VPlanTransforms::addActiveLaneMask(), addVPLaneMaskPhiAndUpdateExitBranch(), createInitialVPlan(), llvm::LoopVectorizationPlanner::executePlan(), llvm::vputils::isHeaderMask(), llvm::LoopVectorizePass::processLoop(), and llvm::VPlanTransforms::tryAddExplicitVectorLength().
|
inline |
Returns the VPRegionBlock of the vector loop.
Definition at line 3476 of file VPlan.h.
References getEntry().
Referenced by addCanonicalIVRecipes(), addLiveOutsForFirstOrderRecurrences(), addUsersInExitBlock(), addVPLaneMaskPhiAndUpdateExitBranch(), llvm::VPlanTransforms::adjustFixedOrderRecurrences(), llvm::VPlanTransforms::clearReductionWrapFlags(), collectAllHeaderMasks(), llvm::collectEphemeralRecipesForVPlan(), collectUsersInExitBlock(), llvm::VPRecipeBuilder::createHeaderMask(), createScalarIVSteps(), llvm::LoopVectorizationPlanner::executePlan(), llvm::VPLiveOut::fixPhi(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), llvm::VPTransformState::get(), getCanonicalIV(), legalizeAndOptimizeInductions(), llvm::VPlanTransforms::optimizeForVFAndUF(), llvm::LoopVectorizePass::processLoop(), removeRedundantCanonicalIVs(), removeRedundantInductionCasts(), llvm::VPlanTransforms::truncateToMinimalBitwidths(), llvm::VPlanTransforms::tryAddExplicitVectorLength(), llvm::VPInterleavedAccessInfo::VPInterleavedAccessInfo(), and willGenerateVectors().
|
inline |
Definition at line 3479 of file VPlan.h.
References getEntry().
|
inline |
The vector trip count.
Definition at line 3401 of file VPlan.h.
Referenced by addCanonicalIVRecipes(), and createInitialVPlan().
|
inline |
Returns VF * UF of the vector loop region.
Definition at line 3404 of file VPlan.h.
Referenced by addCanonicalIVRecipes().
|
inline |
Definition at line 3415 of file VPlan.h.
References llvm::any_of(), and llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable().
Referenced by legalizeAndOptimizeInductions().
|
inline |
Definition at line 3425 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::size().
Referenced by sinkScalarOperands().
Definition at line 3427 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::contains(), and llvm::SetVector< T, Vector, Set, N >::empty().
Referenced by llvm::LoopVectorizationPlanner::executePlan(), llvm::VPlanTransforms::optimizeForVFAndUF(), and setUF().
|
inline |
Definition at line 3414 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::count().
Referenced by llvm::LoopVectorizationPlanner::executePlan(), llvm::LoopVectorizationPlanner::getPlanFor(), legalizeAndOptimizeInductions(), llvm::VPlanTransforms::optimizeForVFAndUF(), and setVF().
void VPlan::prepareToExecute | ( | Value * | TripCount, |
Value * | VectorTripCount, | ||
Value * | CanonicalIVStartValue, | ||
VPTransformState & | State | ||
) |
Prepare the plan for execution, setting up the required live-in values.
Definition at line 921 of file VPlan.cpp.
References llvm::all_of(), assert(), llvm::VPTransformState::Builder, llvm::VPTransformState::CFG, llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), IV, llvm::VPTransformState::CFGState::PrevBB, llvm::VPTransformState::UF, and llvm::VPTransformState::VF.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
LLVM_DUMP_METHOD void VPlan::print | ( | raw_ostream & | O | ) | const |
Print this VPlan to O
.
Definition at line 1128 of file VPlan.cpp.
References llvm::Block, getName(), and llvm::vp_depth_first_shallow().
Referenced by llvm::operator<<(), and llvm::LoopVectorizationPlanner::printPlans().
LLVM_DUMP_METHOD void VPlan::printDOT | ( | raw_ostream & | O | ) | const |
Print this VPlan in DOT format to O
.
Definition at line 1178 of file VPlan.cpp.
References llvm::Metadata::dump(), and Printer.
Referenced by llvm::LoopVectorizationPlanner::printPlans().
void VPlan::printLiveIns | ( | raw_ostream & | O | ) | const |
Print the live-ins of this VPlan to O
.
Definition at line 1098 of file VPlan.cpp.
Referenced by llvm::VPlanPrinter::dump().
|
inline |
Resets the trip count for the VPlan.
The caller must make sure all uses of the original trip count have been replaced.
Definition at line 3387 of file VPlan.h.
References assert(), and llvm::VPValue::getNumUsers().
Referenced by llvm::LoopVectorizePass::processLoop().
Definition at line 3438 of file VPlan.h.
References Name, and llvm::Twine::str().
Referenced by llvm::VPlanHCFGBuilder::buildHierarchicalCFG(), and llvm::LoopVectorizationPlanner::executePlan().
|
inline |
Definition at line 3429 of file VPlan.h.
References assert(), llvm::SetVector< T, Vector, Set, N >::clear(), hasUF(), and llvm::SetVector< T, Vector, Set, N >::insert().
Referenced by llvm::VPlanTransforms::optimizeForVFAndUF(), and llvm::VPlanTransforms::tryAddExplicitVectorLength().
|
inline |
Definition at line 3408 of file VPlan.h.
References assert(), llvm::SetVector< T, Vector, Set, N >::clear(), hasVF(), and llvm::SetVector< T, Vector, Set, N >::insert().
Referenced by llvm::VPlanTransforms::optimizeForVFAndUF().
|
inline |
Returns an iterator range over all VFs of the plan.
Definition at line 3421 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::begin(), and llvm::SetVector< T, Vector, Set, N >::end().
Referenced by llvm::LoopVectorizationPlanner::computeBestVF().
|
friend |
|
friend |