LLVM 19.0.0git
|
VPlan models a candidate for vectorization, encoding various decisions take to produce efficient output IR, including which branches, basic-blocks and output IR instructions to generate, and their cost. More...
#include "Transforms/Vectorize/VPlan.h"
Public Member Functions | |
VPlan (VPBasicBlock *Preheader, VPValue *TC, VPBasicBlock *Entry) | |
Construct a VPlan with original preheader Preheader , trip count TC and Entry to the plan. | |
VPlan (VPBasicBlock *Preheader, VPBasicBlock *Entry) | |
Construct a VPlan with original preheader Preheader and Entry to the plan. | |
~VPlan () | |
void | prepareToExecute (Value *TripCount, Value *VectorTripCount, Value *CanonicalIVStartValue, VPTransformState &State) |
Prepare the plan for execution, setting up the required live-in values. | |
void | execute (VPTransformState *State) |
Generate the IR code for this VPlan. | |
VPBasicBlock * | getEntry () |
const VPBasicBlock * | getEntry () const |
VPValue * | getTripCount () const |
The trip count of the original loop. | |
void | resetTripCount (VPValue *NewTripCount) |
Resets the trip count for the VPlan. | |
VPValue * | getOrCreateBackedgeTakenCount () |
The backedge taken count of the original loop. | |
VPValue & | getVectorTripCount () |
The vector trip count. | |
VPValue & | getVFxUF () |
Returns VF * UF of the vector loop region. | |
void | disableValue2VPValue () |
Mark the plan to indicate that using Value2VPValue is not safe any longer, because it may be stale. | |
void | addVF (ElementCount VF) |
void | setVF (ElementCount VF) |
bool | hasVF (ElementCount VF) |
bool | hasScalableVF () |
bool | hasScalarVFOnly () const |
bool | hasUF (unsigned UF) const |
void | setUF (unsigned UF) |
std::string | getName () const |
Return a string with the name of the plan and the applicable VFs and UFs. | |
void | setName (const Twine &newName) |
void | addVPValue (Value *V, VPValue *VPV) |
VPValue * | getVPValue (Value *V) |
Returns the VPValue for V . | |
VPValue * | getVPValueOrAddLiveIn (Value *V) |
Gets the VPValue for V or adds a new live-in (if none exists yet) for V . | |
void | printLiveIns (raw_ostream &O) const |
Print the live-ins of this VPlan to O . | |
void | print (raw_ostream &O) const |
Print this VPlan to O . | |
void | printDOT (raw_ostream &O) const |
Print this VPlan in DOT format to O . | |
LLVM_DUMP_METHOD void | dump () const |
Dump the plan to stderr (for debugging). | |
iterator_range< mapped_iterator< Use *, std::function< VPValue *(Value *)> > > | mapToVPValues (User::op_range Operands) |
Returns a range mapping the values the range Operands to their corresponding VPValues. | |
VPRegionBlock * | getVectorLoopRegion () |
Returns the VPRegionBlock of the vector loop. | |
const VPRegionBlock * | getVectorLoopRegion () const |
VPCanonicalIVPHIRecipe * | getCanonicalIV () |
Returns the canonical induction recipe of the vector loop. | |
void | addLiveOut (PHINode *PN, VPValue *V) |
void | removeLiveOut (PHINode *PN) |
const MapVector< PHINode *, VPLiveOut * > & | getLiveOuts () const |
VPValue * | getSCEVExpansion (const SCEV *S) const |
void | addSCEVExpansion (const SCEV *S, VPValue *V) |
VPBasicBlock * | getPreheader () |
const VPBasicBlock * | getPreheader () const |
VPlan * | duplicate () |
Clone the current VPlan, update all VPValues of the new VPlan and cloned recipes to refer to the clones, and return it. | |
Static Public Member Functions | |
static VPlanPtr | createInitialVPlan (const SCEV *TripCount, ScalarEvolution &PSE) |
Create initial VPlan skeleton, having an "entry" VPBasicBlock (wrapping original scalar pre-header) which contains SCEV expansions that need to happen before the CFG is modified; a VPBasicBlock for the vector pre-header, followed by a region for the vector loop, followed by the middle VPBasicBlock. | |
Friends | |
class | VPlanPrinter |
class | VPSlotTracker |
VPlan models a candidate for vectorization, encoding various decisions take to produce efficient output IR, including which branches, basic-blocks and output IR instructions to generate, and their cost.
VPlan holds a Hierarchical-CFG of VPBasicBlocks and VPRegionBlocks rooted at an Entry VPBasicBlock.
|
inline |
|
inline |
Construct a VPlan with original preheader Preheader
and Entry
to the plan.
At the moment, Preheader
and Entry
need to be disconnected, as the bypass blocks between them are not yet modeled in VPlan.
Definition at line 2899 of file VPlan.h.
References assert(), llvm::VPBlockBase::getNumPredecessors(), llvm::VPBlockBase::getNumSuccessors(), and llvm::VPBlockBase::setPlan().
VPlan::~VPlan | ( | ) |
Definition at line 757 of file VPlan.cpp.
References llvm::Block, llvm::VPBlockBase::deleteCFG(), and llvm::vp_depth_first_shallow().
Definition at line 1003 of file VPlan.cpp.
References assert().
Referenced by addUsersInExitBlock().
Definition at line 3074 of file VPlan.h.
References assert(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains().
Referenced by llvm::vputils::getOrCreateVPValueForSCEVExpr().
|
inline |
Definition at line 2959 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::insert().
Definition at line 2987 of file VPlan.h.
References assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), and llvm::VPValue::isLiveIn().
Referenced by getVPValueOrAddLiveIn().
|
static |
Create initial VPlan skeleton, having an "entry" VPBasicBlock (wrapping original scalar pre-header) which contains SCEV expansions that need to happen before the CFG is modified; a VPBasicBlock for the vector pre-header, followed by a region for the vector loop, followed by the middle VPBasicBlock.
Definition at line 778 of file VPlan.cpp.
References llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::VPBlockUtils::insertBlockAfter(), and llvm::VPTransformState::Plan.
|
inline |
LLVM_DUMP_METHOD void VPlan::dump | ( | ) | const |
Dump the plan to stderr (for debugging).
Definition at line 1000 of file VPlan.cpp.
References llvm::dbgs(), and llvm::print().
Referenced by llvm::LoopVectorizationPlanner::executePlan().
VPlan * VPlan::duplicate | ( | ) |
Clone the current VPlan, update all VPValues of the new VPlan and cloned recipes to refer to the clones, and return it.
Definition at line 1084 of file VPlan.cpp.
References _, assert(), llvm::VPBasicBlock::clone(), cloneSESE(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains(), Name, and remapOperands().
Referenced by llvm::LoopVectorizePass::processLoop().
void VPlan::execute | ( | VPTransformState * | State | ) |
Generate the IR code for this VPlan.
Generate the code inside the preheader and body of the vectorized loop.
Assumes a single pre-header basic-block was created for this. Introduce additional basic-blocks as needed, and fill them all.
Definition at line 834 of file VPlan.cpp.
References llvm::DominatorTreeBase< NodeT, IsPostDom >::addNewBlock(), llvm::Block, llvm::VPTransformState::Builder, llvm::VPTransformState::CFG, llvm::VPTransformState::DT, llvm::EnableVPlanNativePath, llvm::VPTransformState::CFGState::ExitBB, GEP, llvm::VPTransformState::get(), llvm::VPBlockBase::getExitingBasicBlock(), llvm::ilist_node_with_parent< NodeTy, ParentTy, Options >::getPrevNode(), llvm::BasicBlock::getSingleSuccessor(), llvm::BasicBlock::getTerminator(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::Instruction::moveBefore(), llvm::VPTransformState::CFGState::PrevBB, llvm::VPTransformState::CFGState::PrevVPBB, llvm::IRBuilderBase::SetInsertPoint(), llvm::VPTransformState::UF, llvm::VPTransformState::VF, llvm::vp_depth_first_shallow(), and llvm::VPTransformState::CFGState::VPBB2IRBB.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
inline |
Returns the canonical induction recipe of the vector loop.
Definition at line 3050 of file VPlan.h.
References llvm::VPBasicBlock::begin(), llvm::VPBasicBlock::empty(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPBlockBase::getSingleSuccessor(), and getVectorLoopRegion().
Referenced by llvm::VPlanTransforms::addActiveLaneMask(), addVPLaneMaskPhiAndUpdateExitBranch(), llvm::VPRecipeBuilder::createHeaderMask(), createScalarIVSteps(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPlanTransforms::optimizeForVFAndUF(), removeRedundantCanonicalIVs(), simplifyRecipes(), and llvm::VPlanTransforms::truncateToMinimalBitwidths().
|
inline |
Definition at line 2925 of file VPlan.h.
Referenced by addReplicateRegions(), llvm::VPlanPrinter::dump(), llvm::InnerLoopVectorizer::fixNonInductionPHIs(), llvm::DomTreeNodeTraits< VPBlockBase >::getEntryNode(), getVectorLoopRegion(), mergeBlocksIntoPredecessors(), mergeReplicateRegionsIntoSuccessors(), removeDeadRecipes(), removeRedundantExpandSCEVRecipes(), llvm::VPBlockBase::setPlan(), simplifyRecipes(), sinkScalarOperands(), llvm::VPlanTransforms::truncateToMinimalBitwidths(), and llvm::verifyVPlanIsValid().
|
inline |
Definition at line 3066 of file VPlan.h.
Referenced by llvm::InnerLoopVectorizer::fixVectorizedLoop(), and llvm::verifyVPlanIsValid().
std::string VPlan::getName | ( | ) | const |
Return a string with the name of the plan and the applicable VFs and UFs.
Definition at line 970 of file VPlan.cpp.
References llvm::drop_begin(), Name, llvm::VPTransformState::UF, and llvm::VPTransformState::VF.
Referenced by llvm::VPlanPrinter::dump().
|
inline |
The backedge taken count of the original loop.
Definition at line 2943 of file VPlan.h.
Referenced by llvm::VPlanTransforms::addActiveLaneMask(), and llvm::VPRecipeBuilder::createHeaderMask().
|
inline |
Definition at line 3080 of file VPlan.h.
Referenced by llvm::VPlanPrinter::dump(), llvm::LoopVectorizationPlanner::executePlan(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::LoopVectorizePass::processLoop(), and llvm::VPBlockBase::setPlan().
|
inline |
Definition at line 3070 of file VPlan.h.
References llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::lookup().
Referenced by llvm::InnerLoopVectorizer::fixupIVUsers(), and llvm::vputils::getOrCreateVPValueForSCEVExpr().
|
inline |
The trip count of the original loop.
Definition at line 2929 of file VPlan.h.
References assert().
Referenced by llvm::VPlanTransforms::addActiveLaneMask(), addVPLaneMaskPhiAndUpdateExitBranch(), llvm::LoopVectorizationPlanner::executePlan(), and llvm::LoopVectorizePass::processLoop().
|
inline |
Returns the VPRegionBlock of the vector loop.
Definition at line 3042 of file VPlan.h.
References getEntry().
Referenced by addCanonicalIVRecipes(), addVPLaneMaskPhiAndUpdateExitBranch(), llvm::VPlanTransforms::adjustFixedOrderRecurrences(), llvm::VPlanTransforms::clearReductionWrapFlags(), llvm::VPRecipeBuilder::createHeaderMask(), createScalarIVSteps(), llvm::LoopVectorizationPlanner::executePlan(), llvm::VPLiveOut::fixPhi(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), llvm::VPTransformState::get(), getCanonicalIV(), llvm::VPlanTransforms::optimizeForVFAndUF(), optimizeInductions(), llvm::LoopVectorizePass::processLoop(), removeRedundantCanonicalIVs(), removeRedundantInductionCasts(), llvm::VPlanTransforms::truncateToMinimalBitwidths(), llvm::verifyVPlanIsValid(), and llvm::VPInterleavedAccessInfo::VPInterleavedAccessInfo().
|
inline |
Definition at line 3045 of file VPlan.h.
References getEntry().
|
inline |
The vector trip count.
Definition at line 2950 of file VPlan.h.
Referenced by addCanonicalIVRecipes().
|
inline |
Returns VF * UF of the vector loop region.
Definition at line 2953 of file VPlan.h.
Referenced by addCanonicalIVRecipes().
Returns the VPValue for V
.
Definition at line 2996 of file VPlan.h.
References assert(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count().
Referenced by getVPValueOrAddLiveIn().
Gets the VPValue for V
or adds a new live-in (if none exists yet) for V
.
Definition at line 3006 of file VPlan.h.
References addVPValue(), assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), getVPValue(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().
Referenced by addCanonicalIVRecipes(), addUsersInExitBlock(), llvm::VPRecipeBuilder::createEdgeMask(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), mapToVPValues(), llvm::VPlanTransforms::optimizeForVFAndUF(), and llvm::LoopVectorizePass::processLoop().
|
inline |
Definition at line 2968 of file VPlan.h.
References llvm::any_of(), and llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable().
|
inline |
Definition at line 2972 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::size().
Referenced by sinkScalarOperands().
Definition at line 2974 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::contains(), and llvm::SetVector< T, Vector, Set, N >::empty().
Referenced by llvm::LoopVectorizationPlanner::executePlan(), llvm::VPlanTransforms::optimizeForVFAndUF(), and setUF().
|
inline |
Definition at line 2967 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::count().
Referenced by llvm::LoopVectorizationPlanner::executePlan(), llvm::VPlanTransforms::optimizeForVFAndUF(), optimizeInductions(), and setVF().
|
inline |
Returns a range mapping the values the range Operands
to their corresponding VPValues.
Definition at line 3034 of file VPlan.h.
References getVPValueOrAddLiveIn(), llvm::map_range(), and Operands.
Referenced by llvm::VPRecipeBuilder::handleReplication().
void VPlan::prepareToExecute | ( | Value * | TripCount, |
Value * | VectorTripCount, | ||
Value * | CanonicalIVStartValue, | ||
VPTransformState & | State | ||
) |
Prepare the plan for execution, setting up the required live-in values.
Definition at line 792 of file VPlan.cpp.
References llvm::all_of(), assert(), llvm::VPTransformState::Builder, llvm::VPTransformState::CFG, llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), IV, llvm::VPTransformState::CFGState::PrevBB, llvm::VPTransformState::UF, and llvm::VPTransformState::VF.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
LLVM_DUMP_METHOD void VPlan::print | ( | raw_ostream & | O | ) | const |
Print this VPlan to O
.
Definition at line 944 of file VPlan.cpp.
References llvm::Block, getName(), and llvm::vp_depth_first_shallow().
Referenced by llvm::operator<<().
LLVM_DUMP_METHOD void VPlan::printDOT | ( | raw_ostream & | O | ) | const |
void VPlan::printLiveIns | ( | raw_ostream & | O | ) | const |
Print the live-ins of this VPlan to O
.
Definition at line 914 of file VPlan.cpp.
Referenced by llvm::VPlanPrinter::dump().
|
inline |
Definition at line 3061 of file VPlan.h.
References llvm::MapVector< KeyT, ValueT, MapType, VectorType >::erase().
Referenced by llvm::InnerLoopVectorizer::fixFixedOrderRecurrence(), and llvm::InnerLoopVectorizer::fixupIVUsers().
|
inline |
Resets the trip count for the VPlan.
The caller must make sure all uses of the original trip count have been replaced.
Definition at line 2936 of file VPlan.h.
References assert(), and llvm::VPValue::getNumUsers().
Referenced by llvm::LoopVectorizePass::processLoop().
Definition at line 2985 of file VPlan.h.
References Name, and llvm::Twine::str().
Referenced by llvm::VPlanHCFGBuilder::buildHierarchicalCFG(), and llvm::LoopVectorizationPlanner::executePlan().
|
inline |
Definition at line 2976 of file VPlan.h.
References assert(), llvm::SetVector< T, Vector, Set, N >::clear(), hasUF(), and llvm::SetVector< T, Vector, Set, N >::insert().
Referenced by llvm::VPlanTransforms::optimizeForVFAndUF().
|
inline |
Definition at line 2961 of file VPlan.h.
References assert(), llvm::SetVector< T, Vector, Set, N >::clear(), hasVF(), and llvm::SetVector< T, Vector, Set, N >::insert().
Referenced by llvm::VPlanTransforms::optimizeForVFAndUF().
|
friend |
|
friend |