LLVM 19.0.0git
|
Public Member Functions | |
EpilogueVectorizerEpilogueLoop (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, EpilogueLoopVectorizationInfo &EPI, LoopVectorizationLegality *LVL, llvm::LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &Checks) | |
std::pair< BasicBlock *, Value * > | createEpilogueVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs) final |
Implements the interface for creating a vectorized skeleton using the epilogue loop strategy (ie the second pass of vplan execution). | |
Public Member Functions inherited from llvm::InnerLoopAndEpilogueVectorizer | |
InnerLoopAndEpilogueVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, EpilogueLoopVectorizationInfo &EPI, LoopVectorizationLegality *LVL, llvm::LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &Checks) | |
std::pair< BasicBlock *, Value * > | createVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs) final |
Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder. | |
virtual std::pair< BasicBlock *, Value * > | createEpilogueVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs)=0 |
The interface for creating a vectorized skeleton using one of two different strategies, each corresponding to one execution of the vplan as described above. | |
Public Member Functions inherited from llvm::InnerLoopVectorizer | |
InnerLoopVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, ElementCount VecWidth, ElementCount MinProfitableTripCount, unsigned UnrollFactor, LoopVectorizationLegality *LVL, LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &RTChecks) | |
virtual | ~InnerLoopVectorizer ()=default |
virtual std::pair< BasicBlock *, Value * > | createVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs) |
Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder. | |
void | fixVectorizedLoop (VPTransformState &State, VPlan &Plan) |
Fix the vectorized code, taking care of header phi's, live-outs, and more. | |
bool | areSafetyChecksAdded () |
void | scalarizeInstruction (const Instruction *Instr, VPReplicateRecipe *RepRecipe, const VPIteration &Instance, VPTransformState &State) |
A helper function to scalarize a single Instruction in the innermost loop. | |
void | vectorizeInterleaveGroup (const InterleaveGroup< Instruction > *Group, ArrayRef< VPValue * > VPDefs, VPTransformState &State, VPValue *Addr, ArrayRef< VPValue * > StoredValues, VPValue *BlockInMask, bool NeedsMaskForGaps) |
Try to vectorize interleaved access group Group with the base address given in Addr , optionally masking the vector operations if BlockInMask is non-null. | |
void | fixNonInductionPHIs (VPlan &Plan, VPTransformState &State) |
Fix the non-induction PHIs in Plan . | |
PHINode * | createInductionResumeValue (PHINode *OrigPhi, const InductionDescriptor &ID, Value *Step, ArrayRef< BasicBlock * > BypassBlocks, std::pair< BasicBlock *, Value * > AdditionalBypass={nullptr, nullptr}) |
Create a new phi node for the induction variable OrigPhi to resume iteration count in the scalar epilogue, from where the vectorized loop left off. | |
Value * | getTripCount () const |
Returns the original loop trip count. | |
void | setTripCount (Value *TC) |
Used to set the trip count after ILV's construction and after the preheader block has been executed. | |
Protected Member Functions | |
BasicBlock * | emitMinimumVectorEpilogueIterCountCheck (BasicBlock *Bypass, BasicBlock *Insert) |
Emits an iteration count bypass check after the main vector loop has finished to see if there are any iterations left to execute by either the vector epilogue or the scalar epilogue. | |
void | printDebugTracesAtStart () override |
Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested. | |
void | printDebugTracesAtEnd () override |
Protected Member Functions inherited from llvm::InnerLoopVectorizer | |
void | fixupIVUsers (PHINode *OrigPhi, const InductionDescriptor &II, Value *VectorTripCount, Value *EndValue, BasicBlock *MiddleBlock, BasicBlock *VectorHeader, VPlan &Plan, VPTransformState &State) |
Set up the values of the IVs correctly when exiting the vector loop. | |
void | fixFixedOrderRecurrence (VPFirstOrderRecurrencePHIRecipe *PhiR, VPTransformState &State) |
Create the exit value of first order recurrences in the middle block and update their users. | |
void | sinkScalarOperands (Instruction *PredInst) |
Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it. | |
Value * | getOrCreateVectorTripCount (BasicBlock *InsertBlock) |
Returns (and creates if needed) the trip count of the widened loop. | |
Value * | createBitOrPointerCast (Value *V, VectorType *DstVTy, const DataLayout &DL) |
Returns a bitcasted value to the requested vector type. | |
void | emitIterationCountCheck (BasicBlock *Bypass) |
Emit a bypass check to see if the vector trip count is zero, including if it overflows. | |
BasicBlock * | emitSCEVChecks (BasicBlock *Bypass) |
Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct. | |
BasicBlock * | emitMemRuntimeChecks (BasicBlock *Bypass) |
Emit bypass checks to check any memory assumptions we may have made. | |
void | createVectorLoopSkeleton (StringRef Prefix) |
Emit basic blocks (prefixed with Prefix ) for the iteration check, vector loop preheader, middle block and scalar preheader. | |
void | createInductionResumeValues (const SCEV2ValueTy &ExpandedSCEVs, std::pair< BasicBlock *, Value * > AdditionalBypass={nullptr, nullptr}) |
Create new phi nodes for the induction variables to resume iteration count in the scalar epilogue, from where the vectorized loop left off. | |
BasicBlock * | completeLoopSkeleton () |
Complete the loop skeleton by adding debug MDs, creating appropriate conditional branches in the middle block, preparing the builder and running the verifier. | |
virtual void | printDebugTracesAtStart () |
Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested. | |
virtual void | printDebugTracesAtEnd () |
Additional Inherited Members | |
Public Attributes inherited from llvm::InnerLoopAndEpilogueVectorizer | |
EpilogueLoopVectorizationInfo & | EPI |
Holds and updates state information required to vectorize the main loop and its epilogue in two separate passes. | |
Protected Types inherited from llvm::InnerLoopVectorizer | |
using | PhiVector = SmallVector< PHINode *, 4 > |
A small list of PHINodes. | |
using | ScalarParts = SmallVector< SmallVector< Value *, 4 >, 2 > |
A type for scalarized values in the new loop. | |
Protected Attributes inherited from llvm::InnerLoopVectorizer | |
Loop * | OrigLoop |
The original loop. | |
PredicatedScalarEvolution & | PSE |
A wrapper around ScalarEvolution used to add runtime SCEV checks. | |
LoopInfo * | LI |
Loop Info. | |
DominatorTree * | DT |
Dominator Tree. | |
const TargetLibraryInfo * | TLI |
Target Library Info. | |
const TargetTransformInfo * | TTI |
Target Transform Info. | |
AssumptionCache * | AC |
Assumption Cache. | |
OptimizationRemarkEmitter * | ORE |
Interface to emit optimization remarks. | |
ElementCount | VF |
The vectorization SIMD factor to use. | |
ElementCount | MinProfitableTripCount |
unsigned | UF |
The vectorization unroll factor to use. | |
IRBuilder | Builder |
The builder that we use. | |
BasicBlock * | LoopVectorPreHeader |
The vector-loop preheader. | |
BasicBlock * | LoopScalarPreHeader |
The scalar-loop preheader. | |
BasicBlock * | LoopMiddleBlock |
Middle Block between the vector and the scalar. | |
BasicBlock * | LoopExitBlock |
The unique ExitBlock of the scalar loop if one exists. | |
BasicBlock * | LoopScalarBody |
The scalar loop body. | |
SmallVector< BasicBlock *, 4 > | LoopBypassBlocks |
A list of all bypass blocks. The first block is the entry of the loop. | |
SmallVector< Instruction *, 4 > | PredicatedInstructions |
Store instructions that were predicated. | |
Value * | TripCount = nullptr |
Trip count of the original loop. | |
Value * | VectorTripCount = nullptr |
Trip count of the widened loop (TripCount - TripCount % (VF*UF)) | |
LoopVectorizationLegality * | Legal |
The legality analysis. | |
LoopVectorizationCostModel * | Cost |
The profitablity analysis. | |
bool | AddedSafetyChecks = false |
DenseMap< PHINode *, Value * > | IVEndValues |
BlockFrequencyInfo * | BFI |
BFI and PSI are used to check for profile guided size optimizations. | |
ProfileSummaryInfo * | PSI |
bool | OptForSizeBasedOnProfile |
GeneratedRTChecks & | RTChecks |
Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable. | |
SmallMapVector< const RecurrenceDescriptor *, PHINode *, 4 > | ReductionResumeValues |
Definition at line 879 of file LoopVectorize.cpp.
|
inline |
Definition at line 881 of file LoopVectorize.cpp.
References llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::InnerLoopVectorizer::TripCount, and llvm::EpilogueLoopVectorizationInfo::TripCount.
|
finalvirtual |
Implements the interface for creating a vectorized skeleton using the epilogue loop strategy (ie the second pass of vplan execution).
This function is partially responsible for generating the control flow depicted in https://llvm.org/docs/Vectorizers.html#epilogue-vectorization.
Implements llvm::InnerLoopAndEpilogueVectorizer.
Definition at line 7766 of file LoopVectorize.cpp.
References llvm::PHINode::addIncoming(), assert(), llvm::DominatorTreeBase< NodeT, IsPostDom >::changeImmediateDominator(), llvm::InnerLoopVectorizer::completeLoopSkeleton(), llvm::PHINode::Create(), llvm::InnerLoopVectorizer::createInductionResumeValues(), llvm::InnerLoopVectorizer::createVectorLoopSkeleton(), llvm::InnerLoopVectorizer::DT, emitMinimumVectorEpilogueIterCountCheck(), llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::EpilogueLoopVectorizationInfo::EpilogueIterationCountCheck, llvm::EpilogueLoopVectorizationInfo::EpilogueVF, llvm::BasicBlock::getFirstNonPHI(), llvm::BasicBlock::getFirstNonPHIIt(), llvm::BasicBlock::getSinglePredecessor(), llvm::BasicBlock::getTerminator(), llvm::Instruction::insertBefore(), llvm::ElementCount::isVector(), llvm::IRSimilarity::Legal, llvm::InnerLoopVectorizer::LI, llvm::InnerLoopVectorizer::LoopBypassBlocks, llvm::InnerLoopVectorizer::LoopExitBlock, llvm::InnerLoopVectorizer::LoopScalarPreHeader, llvm::InnerLoopVectorizer::LoopVectorPreHeader, llvm::EpilogueLoopVectorizationInfo::MainLoopIterationCountCheck, llvm::EpilogueLoopVectorizationInfo::MemSafetyCheck, llvm::none_of(), llvm::BasicBlock::phis(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::User::replaceUsesOfWith(), llvm::EpilogueLoopVectorizationInfo::SCEVSafetyCheck, llvm::Value::setName(), llvm::SplitBlock(), and llvm::EpilogueLoopVectorizationInfo::VectorTripCount.
|
protected |
Emits an iteration count bypass check after the main vector loop has finished to see if there are any iterations left to execute by either the vector epilogue or the scalar epilogue.
Definition at line 7873 of file LoopVectorize.cpp.
References assert(), llvm::InnerLoopVectorizer::Builder, llvm::BranchInst::Create(), llvm::IRBuilderBase::CreateICmp(), llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), llvm::DominatorTree::dominates(), llvm::InnerLoopVectorizer::DT, llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::EpilogueLoopVectorizationInfo::EpilogueUF, llvm::EpilogueLoopVectorizationInfo::EpilogueVF, llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::hasBranchWeightMD(), llvm::CmpInst::ICMP_ULE, llvm::CmpInst::ICMP_ULT, llvm::ElementCount::isVector(), llvm::InnerLoopVectorizer::LoopBypassBlocks, llvm::InnerLoopVectorizer::LoopVectorPreHeader, llvm::InnerLoopVectorizer::OrigLoop, P, llvm::ReplaceInstWithInst(), llvm::setBranchWeights(), llvm::EpilogueLoopVectorizationInfo::TripCount, llvm::InnerLoopVectorizer::UF, llvm::EpilogueLoopVectorizationInfo::VectorTripCount, and llvm::InnerLoopVectorizer::VF.
Referenced by createEpilogueVectorizedLoopSkeleton().
|
overrideprotectedvirtual |
Reimplemented from llvm::InnerLoopVectorizer.
Definition at line 7927 of file LoopVectorize.cpp.
References llvm::dbgs(), DEBUG_WITH_TYPE, llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::BasicBlock::getParent(), llvm::InnerLoopVectorizer::OrigLoop, and VerboseDebug.
|
overrideprotectedvirtual |
Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.
Reimplemented from llvm::InnerLoopVectorizer.
Definition at line 7919 of file LoopVectorize.cpp.
References llvm::dbgs(), llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::EpilogueLoopVectorizationInfo::EpilogueUF, llvm::EpilogueLoopVectorizationInfo::EpilogueVF, and LLVM_DEBUG.