LLVM 20.0.0git
|
Helper class to create VPRecipies from IR instructions. More...
#include "Transforms/Vectorize/VPRecipeBuilder.h"
Public Member Functions | |
VPRecipeBuilder (VPlan &Plan, Loop *OrigLoop, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, LoopVectorizationLegality *Legal, LoopVectorizationCostModel &CM, PredicatedScalarEvolution &PSE, VPBuilder &Builder) | |
std::optional< std::pair< PartialReductionChain, unsigned > > | getScaledReductionForInstr (const Instruction *ExitInst) |
void | collectScaledReductions (VFRange &Range) |
Find all possible partial reductions in the loop and track all of those that are valid so recipes can be formed later. | |
VPRecipeBase * | tryToCreateWidenRecipe (Instruction *Instr, ArrayRef< VPValue * > Operands, VFRange &Range, VPBasicBlock *VPBB) |
Create and return a widened recipe for I if one can be created within the given VF Range . | |
VPRecipeBase * | tryToCreatePartialReduction (Instruction *Reduction, ArrayRef< VPValue * > Operands) |
Create and return a partial reduction recipe for a reduction instruction along with binary operation and reduction phi operands. | |
void | setRecipe (Instruction *I, VPRecipeBase *R) |
Set the recipe created for given ingredient. | |
void | createHeaderMask () |
Create the mask for the vector loop header block. | |
void | createBlockInMask (BasicBlock *BB) |
A helper function that computes the predicate of the block BB, assuming that the header block of the loop is set to True or the loop mask when tail folding. | |
VPValue * | getBlockInMask (BasicBlock *BB) const |
Returns the entry mask for the block BB . | |
void | createSwitchEdgeMasks (SwitchInst *SI) |
Create an edge mask for every destination of cases and/or default. | |
VPValue * | createEdgeMask (BasicBlock *Src, BasicBlock *Dst) |
A helper function that computes the predicate of the edge between SRC and DST. | |
VPValue * | getEdgeMask (BasicBlock *Src, BasicBlock *Dst) const |
A helper that returns the previously computed predicate of the edge between SRC and DST. | |
VPRecipeBase * | getRecipe (Instruction *I) |
Return the recipe created for given ingredient. | |
VPReplicateRecipe * | handleReplication (Instruction *I, VFRange &Range) |
Build a VPReplicationRecipe for I . | |
void | fixHeaderPhis () |
Add the incoming values from the backedge to reduction & first-order recurrence cross-iteration phis. | |
iterator_range< mapped_iterator< Use *, std::function< VPValue *(Value *)> > > | mapToVPValues (User::op_range Operands) |
Returns a range mapping the values of the range Operands to their corresponding VPValues. | |
VPValue * | getVPValueOrAddLiveIn (Value *V) |
Helper class to create VPRecipies from IR instructions.
Definition at line 47 of file VPRecipeBuilder.h.
|
inline |
Definition at line 152 of file VPRecipeBuilder.h.
References llvm::IRSimilarity::Legal.
void VPRecipeBuilder::collectScaledReductions | ( | VFRange & | Range | ) |
Find all possible partial reductions in the loop and track all of those that are valid so recipes can be formed later.
Definition at line 8774 of file LoopVectorize.cpp.
References _, llvm::all_of(), llvm::PartialReductionChain::ExtendA, llvm::PartialReductionChain::ExtendB, llvm::LoopVectorizationLegality::getReductionVars(), llvm::SmallSet< T, N, C >::insert(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), Range, and llvm::PartialReductionChain::Reduction.
void VPRecipeBuilder::createBlockInMask | ( | BasicBlock * | BB | ) |
A helper function that computes the predicate of the block BB, assuming that the header block of the loop is set to True or the loop mask when tail folding.
Definition at line 8307 of file LoopVectorize.cpp.
References assert(), llvm::LoopBase< BlockT, LoopT >::contains(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), createEdgeMask(), llvm::VPBuilder::createOr(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::pred_begin(), and llvm::pred_end().
VPValue * VPRecipeBuilder::createEdgeMask | ( | BasicBlock * | Src, |
BasicBlock * | Dst | ||
) |
A helper function that computes the predicate of the edge between SRC and DST.
Definition at line 8213 of file LoopVectorize.cpp.
References assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains(), llvm::VPBuilder::createLogicalAnd(), llvm::VPBuilder::createNot(), createSwitchEdgeMasks(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::find(), getBlockInMask(), llvm::BranchInst::getCondition(), llvm::Instruction::getDebugLoc(), llvm::BranchInst::getSuccessor(), llvm::LoopVectorizationLegality::getUncountableEarlyExitingBlock(), getVPValueOrAddLiveIn(), llvm::is_contained(), llvm::BranchInst::isConditional(), llvm::LoopBase< BlockT, LoopT >::isLoopExiting(), and llvm::predecessors().
Referenced by createBlockInMask().
void VPRecipeBuilder::createHeaderMask | ( | ) |
Create the mask for the vector loop header block.
Definition at line 8272 of file LoopVectorize.cpp.
References llvm::VPBuilder::createICmp(), llvm::LoopVectorizationCostModel::foldTailByMasking(), llvm::VPlan::getCanonicalIV(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPBasicBlock::getFirstNonPhi(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::VPlan::getOrCreateBackedgeTakenCount(), llvm::VPlan::getVectorLoopRegion(), llvm::CmpInst::ICMP_ULE, llvm::VPBasicBlock::insert(), IV, and llvm::VPBuilder::setInsertPoint().
void VPRecipeBuilder::createSwitchEdgeMasks | ( | SwitchInst * | SI | ) |
Create an edge mask for every destination of cases and/or default.
Definition at line 8157 of file LoopVectorize.cpp.
References llvm::all_of(), assert(), llvm::CallingConv::C, Cond, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains(), llvm::VPBuilder::createICmp(), llvm::VPBuilder::createLogicalAnd(), llvm::VPBuilder::createNot(), llvm::VPBuilder::createOr(), getBlockInMask(), llvm::LoopBase< BlockT, LoopT >::getHeader(), getVPValueOrAddLiveIn(), llvm::CmpInst::ICMP_EQ, llvm::LoopBase< BlockT, LoopT >::isLoopExiting(), and llvm::successors().
Referenced by createEdgeMask().
void VPRecipeBuilder::fixHeaderPhis | ( | ) |
Add the incoming values from the backedge to reduction & first-order recurrence cross-iteration phis.
Definition at line 8700 of file LoopVectorize.cpp.
References llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), getRecipe(), and llvm::VPDef::getVPSingleValue().
VPValue * VPRecipeBuilder::getBlockInMask | ( | BasicBlock * | BB | ) | const |
Returns the entry mask for the block BB
.
Definition at line 8299 of file LoopVectorize.cpp.
References assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::find().
Referenced by createEdgeMask(), createSwitchEdgeMasks(), handleReplication(), and llvm::VPlanTransforms::handleUncountableEarlyExit().
VPValue * VPRecipeBuilder::getEdgeMask | ( | BasicBlock * | Src, |
BasicBlock * | Dst | ||
) | const |
A helper that returns the previously computed predicate of the edge between SRC and DST.
Definition at line 8261 of file LoopVectorize.cpp.
References assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::find(), llvm::is_contained(), and llvm::predecessors().
|
inline |
Return the recipe created for given ingredient.
Definition at line 213 of file VPRecipeBuilder.h.
References assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), and I.
Referenced by addScalarResumePhis(), llvm::VPlanTransforms::createInterleaveGroups(), and fixHeaderPhis().
|
inline |
Definition at line 161 of file VPRecipeBuilder.h.
References llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::find().
Referenced by tryToCreateWidenRecipe().
Definition at line 235 of file VPRecipeBuilder.h.
References llvm::VPlan::getOrAddLiveIn(), I, and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::lookup().
Referenced by collectUsersInExitBlocks(), createEdgeMask(), and createSwitchEdgeMasks().
VPReplicateRecipe * VPRecipeBuilder::handleReplication | ( | Instruction * | I, |
VFRange & | Range | ||
) |
Build a VPReplicationRecipe for I
.
If it is predicated, add the mask as last operand. Range.End may be decreased to ensure same recipe behavior from Range.Start
to Range.End
.
Definition at line 8710 of file LoopVectorize.cpp.
References assert(), llvm::dbgs(), getBlockInMask(), llvm::LoopVectorizationPlanner::getDecisionAndClampRange(), getIntrinsicID(), I, llvm::LoopVectorizationCostModel::isPredicatedInst(), llvm::LoopVectorizationCostModel::isUniformAfterVectorization(), LLVM_DEBUG, mapToVPValues(), and Range.
iterator_range< mapped_iterator< Use *, std::function< VPValue *(Value *)> > > VPRecipeBuilder::mapToVPValues | ( | User::op_range | Operands | ) |
Returns a range mapping the values of the range Operands
to their corresponding VPValues.
Definition at line 8150 of file LoopVectorize.cpp.
References llvm::map_range(), and Operands.
Referenced by handleReplication().
|
inline |
Set the recipe created for given ingredient.
Definition at line 184 of file VPRecipeBuilder.h.
References assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains(), and I.
VPRecipeBase * VPRecipeBuilder::tryToCreatePartialReduction | ( | Instruction * | Reduction, |
ArrayRef< VPValue * > | Operands | ||
) |
Create and return a partial reduction recipe for a reduction instruction along with binary operation and reduction phi operands.
Definition at line 8958 of file LoopVectorize.cpp.
References assert(), llvm::VPValue::getDefiningRecipe(), Operands, Reduction, and std::swap().
Referenced by tryToCreateWidenRecipe().
VPRecipeBase * VPRecipeBuilder::tryToCreateWidenRecipe | ( | Instruction * | Instr, |
ArrayRef< VPValue * > | Operands, | ||
VFRange & | Range, | ||
VPBasicBlock * | VPBB | ||
) |
Create and return a widened recipe for I
if one can be created within the given VF Range
.
Definition at line 8872 of file LoopVectorize.cpp.
References assert(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::find(), GEP, llvm::LoopVectorizationPlanner::getDecisionAndClampRange(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::LoopVectorizationLegality::getHistogramInfo(), llvm::RecurrenceDescriptor::getLoopExitInstr(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::Instruction::getOpcode(), llvm::RecurrenceDescriptor::getRecurrenceStartValue(), llvm::LoopVectorizationLegality::getReductionVars(), getScaledReductionForInstr(), llvm::Value::getType(), llvm::LoopVectorizationLegality::isFixedOrderRecurrence(), llvm::LoopVectorizationCostModel::isInLoopReduction(), llvm::LoopVectorizationLegality::isReductionVariable(), llvm::ElementCount::isScalar(), llvm::make_range(), Operands, Range, tryToCreatePartialReduction(), and llvm::LoopVectorizationCostModel::useOrderedReductions().