LLVM 17.0.0git
|
VPlan models a candidate for vectorization, encoding various decisions take to produce efficient output IR, including which branches, basic-blocks and output IR instructions to generate, and their cost. More...
#include "Transforms/Vectorize/VPlan.h"
Public Member Functions | |
VPlan (VPBlockBase *Entry=nullptr) | |
~VPlan () | |
void | prepareToExecute (Value *TripCount, Value *VectorTripCount, Value *CanonicalIVStartValue, VPTransformState &State, bool IsEpilogueVectorization) |
Prepare the plan for execution, setting up the required live-in values. | |
void | execute (VPTransformState *State) |
Generate the IR code for this VPlan. | |
VPBlockBase * | getEntry () |
const VPBlockBase * | getEntry () const |
VPBlockBase * | setEntry (VPBlockBase *Block) |
VPValue * | getOrCreateTripCount () |
The trip count of the original loop. | |
VPValue * | getOrCreateBackedgeTakenCount () |
The backedge taken count of the original loop. | |
VPValue & | getVectorTripCount () |
The vector trip count. | |
void | disableValue2VPValue () |
Mark the plan to indicate that using Value2VPValue is not safe any longer, because it may be stale. | |
void | addVF (ElementCount VF) |
void | setVF (ElementCount VF) |
bool | hasVF (ElementCount VF) |
bool | hasScalarVFOnly () const |
bool | hasUF (unsigned UF) const |
void | setUF (unsigned UF) |
std::string | getName () const |
Return a string with the name of the plan and the applicable VFs and UFs. | |
void | setName (const Twine &newName) |
VPValue * | getOrAddExternalDef (Value *V) |
Get the existing or add a new external definition for V . | |
void | addVPValue (Value *V) |
void | addVPValue (Value *V, VPValue *VPV) |
VPValue * | getVPValue (Value *V, bool OverrideAllowed=false) |
Returns the VPValue for V . | |
VPValue * | getOrAddVPValue (Value *V, bool OverrideAllowed=false) |
Gets the VPValue or adds a new one (if none exists yet) for V . | |
void | removeVPValueFor (Value *V) |
void | print (raw_ostream &O) const |
Print this VPlan to O . | |
void | printDOT (raw_ostream &O) const |
Print this VPlan in DOT format to O . | |
LLVM_DUMP_METHOD void | dump () const |
Dump the plan to stderr (for debugging). | |
iterator_range< mapped_iterator< Use *, std::function< VPValue *(Value *)> > > | mapToVPValues (User::op_range Operands) |
Returns a range mapping the values the range Operands to their corresponding VPValues. | |
VPRegionBlock * | getVectorLoopRegion () |
Returns the VPRegionBlock of the vector loop. | |
const VPRegionBlock * | getVectorLoopRegion () const |
VPCanonicalIVPHIRecipe * | getCanonicalIV () |
Returns the canonical induction recipe of the vector loop. | |
VPActiveLaneMaskPHIRecipe * | getActiveLaneMaskPhi () |
Find and return the VPActiveLaneMaskPHIRecipe from the header - there be only one at most. | |
void | addLiveOut (PHINode *PN, VPValue *V) |
void | clearLiveOuts () |
void | removeLiveOut (PHINode *PN) |
const MapVector< PHINode *, VPLiveOut * > & | getLiveOuts () const |
Friends | |
class | VPlanPrinter |
class | VPSlotTracker |
VPlan models a candidate for vectorization, encoding various decisions take to produce efficient output IR, including which branches, basic-blocks and output IR instructions to generate, and their cost.
VPlan holds a Hierarchical-CFG of VPBasicBlocks and VPRegionBlocks rooted at an Entry VPBlock.
|
inline |
VPlan::~VPlan | ( | ) |
Definition at line 582 of file VPlan.cpp.
References llvm::Block, llvm::VPBlockBase::deleteCFG(), P, and llvm::vp_depth_first_shallow().
|
inline |
Definition at line 2270 of file VPlan.h.
References llvm::SetVector< T, Vector, Set >::insert().
|
inline |
Definition at line 2303 of file VPlan.h.
References assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().
Referenced by getOrAddVPValue().
Definition at line 2313 of file VPlan.h.
References assert(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count().
|
inline |
Definition at line 2393 of file VPlan.h.
Referenced by llvm::InnerLoopVectorizer::fixVectorizedLoop().
|
inline |
LLVM_DUMP_METHOD void VPlan::dump | ( | ) | const |
Dump the plan to stderr (for debugging).
Definition at line 824 of file VPlan.cpp.
References llvm::dbgs(), and llvm::print().
void VPlan::execute | ( | VPTransformState * | State | ) |
Generate the IR code for this VPlan.
Generate the code inside the preheader and body of the vectorized loop.
Assumes a single pre-header basic-block was created for this. Introduce additional basic-blocks as needed, and fill them all.
Definition at line 665 of file VPlan.cpp.
References llvm::DominatorTreeBase< NodeT, IsPostDom >::addNewBlock(), llvm::Block, llvm::VPTransformState::Builder, llvm::VPTransformState::CFG, llvm::VPTransformState::DT, llvm::EnableVPlanNativePath, llvm::VPTransformState::CFGState::ExitBB, GEP, llvm::VPTransformState::get(), llvm::VPBlockBase::getExitingBasicBlock(), llvm::PHINode::getIncomingValue(), llvm::ilist_node_with_parent< NodeTy, ParentTy, Options >::getPrevNode(), llvm::BasicBlock::getSingleSuccessor(), llvm::BasicBlock::getTerminator(), llvm::Instruction::moveBefore(), llvm::VPTransformState::CFGState::PrevBB, llvm::VPTransformState::CFGState::PrevVPBB, llvm::PHINode::setIncomingBlock(), llvm::IRBuilderBase::SetInsertPoint(), llvm::VPTransformState::UF, llvm::VPTransformState::VF, llvm::vp_depth_first_shallow(), llvm::VPTransformState::CFGState::VPBB2IRBB, and llvm::VPTransformState::VPValue2Value.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
VPActiveLaneMaskPHIRecipe * VPlan::getActiveLaneMaskPhi | ( | ) |
Find and return the VPActiveLaneMaskPHIRecipe from the header - there be only one at most.
If there isn't one, then return nullptr.
Definition at line 602 of file VPlan.cpp.
Referenced by llvm::VPRecipeBuilder::createBlockInMask().
|
inline |
Returns the canonical induction recipe of the vector loop.
Definition at line 2378 of file VPlan.h.
References llvm::VPBasicBlock::begin(), llvm::VPBasicBlock::empty(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPBlockBase::getSingleSuccessor(), and getVectorLoopRegion().
Referenced by llvm::VPRecipeBuilder::createBlockInMask(), llvm::VPlanTransforms::optimizeForVFAndUF(), llvm::VPlanTransforms::optimizeInductions(), and llvm::VPlanTransforms::removeRedundantCanonicalIVs().
|
inline |
Definition at line 2240 of file VPlan.h.
Referenced by addCanonicalIVRecipes(), addReplicateRegions(), llvm::InnerLoopVectorizer::collectPoisonGeneratingRecipes(), llvm::VPlanPrinter::dump(), llvm::InnerLoopVectorizer::fixNonInductionPHIs(), llvm::DomTreeNodeTraits< VPBlockBase >::getEntryNode(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), getVectorLoopRegion(), llvm::VPlanTransforms::mergeBlocksIntoPredecessors(), mergeReplicateRegionsIntoSuccessors(), llvm::VPlanTransforms::removeDeadRecipes(), llvm::VPlanTransforms::removeRedundantExpandSCEVRecipes(), llvm::VPBlockBase::setPlan(), sinkScalarOperands(), and llvm::VPlanVerifier::verifyPlanIsValid().
|
inline |
Definition at line 2404 of file VPlan.h.
Referenced by llvm::InnerLoopVectorizer::fixVectorizedLoop(), and llvm::VPlanVerifier::verifyPlanIsValid().
std::string VPlan::getName | ( | ) | const |
Return a string with the name of the plan and the applicable VFs and UFs.
Definition at line 794 of file VPlan.cpp.
References llvm::drop_begin(), Name, llvm::VPTransformState::UF, and llvm::VPTransformState::VF.
Referenced by llvm::VPlanPrinter::dump().
Get the existing or add a new external definition for V
.
Definition at line 2296 of file VPlan.h.
References I, and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::insert().
Referenced by llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::VPlanTransforms::optimizeForVFAndUF(), and llvm::LoopVectorizePass::processLoop().
Gets the VPValue or adds a new one (if none exists yet) for V
.
OverrideAllowed
can be used to disable checking whether it is safe to query VPValues using IR Values.
Definition at line 2333 of file VPlan.h.
References addVPValue(), assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), and getVPValue().
Referenced by addCanonicalIVRecipes(), addUsersInExitBlock(), llvm::VPRecipeBuilder::createEdgeMask(), and mapToVPValues().
|
inline |
The backedge taken count of the original loop.
Definition at line 2257 of file VPlan.h.
Referenced by llvm::VPRecipeBuilder::createBlockInMask().
|
inline |
The trip count of the original loop.
Definition at line 2250 of file VPlan.h.
Referenced by addCanonicalIVRecipes(), and llvm::VPRecipeBuilder::createBlockInMask().
|
inline |
Returns the VPRegionBlock of the vector loop.
Definition at line 2370 of file VPlan.h.
References getEntry().
Referenced by addCanonicalIVRecipes(), llvm::VPlanTransforms::adjustFixedOrderRecurrences(), llvm::VPlanHCFGBuilder::buildHierarchicalCFG(), llvm::VPRecipeBuilder::createBlockInMask(), llvm::LoopVectorizationPlanner::executePlan(), llvm::InnerLoopVectorizer::fixCrossIterationPHIs(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), getCanonicalIV(), llvm::VPlanTransforms::optimizeForVFAndUF(), llvm::VPlanTransforms::optimizeInductions(), llvm::LoopVectorizePass::processLoop(), llvm::VPlanTransforms::removeRedundantCanonicalIVs(), llvm::VPlanTransforms::removeRedundantInductionCasts(), llvm::VPlanVerifier::verifyPlanIsValid(), and llvm::VPInterleavedAccessInfo::VPInterleavedAccessInfo().
|
inline |
Definition at line 2373 of file VPlan.h.
References getEntry().
|
inline |
The vector trip count.
Definition at line 2264 of file VPlan.h.
Referenced by addCanonicalIVRecipes().
Returns the VPValue for V
.
OverrideAllowed
can be used to disable checking whether it is safe to query VPValues using IR Values.
Definition at line 2322 of file VPlan.h.
References assert(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count().
Referenced by getOrAddVPValue(), and llvm::InnerLoopVectorizer::truncateToMinimalBitwidths().
|
inline |
Definition at line 2280 of file VPlan.h.
References llvm::SetVector< T, Vector, Set >::size().
Referenced by sinkScalarOperands().
Definition at line 2282 of file VPlan.h.
References llvm::SetVector< T, Vector, Set >::contains(), and llvm::SetVector< T, Vector, Set >::empty().
Referenced by llvm::LoopVectorizationPlanner::executePlan(), llvm::VPlanTransforms::optimizeForVFAndUF(), and setUF().
|
inline |
Definition at line 2278 of file VPlan.h.
References llvm::SetVector< T, Vector, Set >::count().
Referenced by llvm::LoopVectorizationPlanner::executePlan(), llvm::VPlanTransforms::optimizeForVFAndUF(), llvm::VPlanTransforms::optimizeInductions(), and setVF().
|
inline |
Returns a range mapping the values the range Operands
to their corresponding VPValues.
Definition at line 2362 of file VPlan.h.
References getOrAddVPValue(), llvm::map_range(), and Operands.
Referenced by llvm::VPRecipeBuilder::handleReplication().
void VPlan::prepareToExecute | ( | Value * | TripCount, |
Value * | VectorTripCount, | ||
Value * | CanonicalIVStartValue, | ||
VPTransformState & | State, | ||
bool | IsEpilogueVectorization | ||
) |
Prepare the plan for execution, setting up the required live-in values.
Definition at line 611 of file VPlan.cpp.
References llvm::all_of(), assert(), llvm::VPTransformState::Builder, llvm::VPTransformState::CFG, llvm::IRBuilderBase::CreateSub(), llvm::IRBuilderBase::CreateVectorSplat(), llvm::ConstantInt::get(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::ElementCount::isScalar(), IV, llvm::VPTransformState::CFGState::PrevBB, llvm::VPTransformState::set(), llvm::VPTransformState::UF, and llvm::VPTransformState::VF.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
LLVM_DUMP_METHOD void VPlan::print | ( | raw_ostream & | O | ) | const |
Print this VPlan to O
.
Definition at line 746 of file VPlan.cpp.
References llvm::Block, getName(), and llvm::vp_depth_first_shallow().
Referenced by llvm::operator<<().
LLVM_DUMP_METHOD void VPlan::printDOT | ( | raw_ostream & | O | ) | const |
|
inline |
Definition at line 2399 of file VPlan.h.
References llvm::MapVector< KeyT, ValueT, MapType, VectorType >::erase().
Referenced by llvm::InnerLoopVectorizer::fixFixedOrderRecurrence(), llvm::InnerLoopVectorizer::fixReduction(), and llvm::InnerLoopVectorizer::fixupIVUsers().
|
inline |
Definition at line 2342 of file VPlan.h.
References assert(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::erase().
|
inline |
Definition at line 2243 of file VPlan.h.
References llvm::Block.
Referenced by llvm::VPlanHCFGBuilder::buildHierarchicalCFG().
Definition at line 2293 of file VPlan.h.
References Name, and llvm::Twine::str().
Referenced by llvm::VPlanHCFGBuilder::buildHierarchicalCFG().
|
inline |
Definition at line 2284 of file VPlan.h.
References assert(), llvm::SetVector< T, Vector, Set >::clear(), hasUF(), and llvm::SetVector< T, Vector, Set >::insert().
Referenced by llvm::VPlanTransforms::optimizeForVFAndUF().
|
inline |
Definition at line 2272 of file VPlan.h.
References assert(), llvm::SetVector< T, Vector, Set >::clear(), hasVF(), and llvm::SetVector< T, Vector, Set >::insert().
Referenced by llvm::VPlanTransforms::optimizeForVFAndUF().
|
friend |
|
friend |