LLVM 19.0.0git
Public Member Functions | Protected Member Functions | List of all members
llvm::EpilogueVectorizerMainLoop Class Reference

A specialized derived class of inner loop vectorizer that performs vectorization of main loops in the process of vectorizing loops and their epilogues. More...

Inheritance diagram for llvm::EpilogueVectorizerMainLoop:
Inheritance graph
[legend]

Public Member Functions

 EpilogueVectorizerMainLoop (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, EpilogueLoopVectorizationInfo &EPI, LoopVectorizationLegality *LVL, llvm::LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &Check)
 
std::pair< BasicBlock *, Value * > createEpilogueVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs) final
 Implements the interface for creating a vectorized skeleton using the main loop strategy (ie the first pass of vplan execution).
 
- Public Member Functions inherited from llvm::InnerLoopAndEpilogueVectorizer
 InnerLoopAndEpilogueVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, EpilogueLoopVectorizationInfo &EPI, LoopVectorizationLegality *LVL, llvm::LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &Checks)
 
std::pair< BasicBlock *, Value * > createVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs) final
 Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder.
 
virtual std::pair< BasicBlock *, Value * > createEpilogueVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs)=0
 The interface for creating a vectorized skeleton using one of two different strategies, each corresponding to one execution of the vplan as described above.
 
- Public Member Functions inherited from llvm::InnerLoopVectorizer
 InnerLoopVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, ElementCount VecWidth, ElementCount MinProfitableTripCount, unsigned UnrollFactor, LoopVectorizationLegality *LVL, LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &RTChecks)
 
virtual ~InnerLoopVectorizer ()=default
 
virtual std::pair< BasicBlock *, Value * > createVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs)
 Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder.
 
void fixVectorizedLoop (VPTransformState &State, VPlan &Plan)
 Fix the vectorized code, taking care of header phi's, live-outs, and more.
 
bool areSafetyChecksAdded ()
 
void scalarizeInstruction (const Instruction *Instr, VPReplicateRecipe *RepRecipe, const VPIteration &Instance, VPTransformState &State)
 A helper function to scalarize a single Instruction in the innermost loop.
 
void vectorizeInterleaveGroup (const InterleaveGroup< Instruction > *Group, ArrayRef< VPValue * > VPDefs, VPTransformState &State, VPValue *Addr, ArrayRef< VPValue * > StoredValues, VPValue *BlockInMask, bool NeedsMaskForGaps)
 Try to vectorize interleaved access group Group with the base address given in Addr, optionally masking the vector operations if BlockInMask is non-null.
 
void fixNonInductionPHIs (VPlan &Plan, VPTransformState &State)
 Fix the non-induction PHIs in Plan.
 
bool useOrderedReductions (const RecurrenceDescriptor &RdxDesc)
 Returns true if the reordering of FP operations is not allowed, but we are able to vectorize with strict in-order reductions for the given RdxDesc.
 
PHINodecreateInductionResumeValue (PHINode *OrigPhi, const InductionDescriptor &ID, Value *Step, ArrayRef< BasicBlock * > BypassBlocks, std::pair< BasicBlock *, Value * > AdditionalBypass={nullptr, nullptr})
 Create a new phi node for the induction variable OrigPhi to resume iteration count in the scalar epilogue, from where the vectorized loop left off.
 
ValuegetTripCount () const
 Returns the original loop trip count.
 
void setTripCount (Value *TC)
 Used to set the trip count after ILV's construction and after the preheader block has been executed.
 

Protected Member Functions

BasicBlockemitIterationCountCheck (BasicBlock *Bypass, bool ForEpilogue)
 Emits an iteration count bypass check once for the main loop (when ForEpilogue is false) and once for the epilogue loop (when ForEpilogue is true).
 
void printDebugTracesAtStart () override
 Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.
 
void printDebugTracesAtEnd () override
 
- Protected Member Functions inherited from llvm::InnerLoopVectorizer
void fixupIVUsers (PHINode *OrigPhi, const InductionDescriptor &II, Value *VectorTripCount, Value *EndValue, BasicBlock *MiddleBlock, BasicBlock *VectorHeader, VPlan &Plan, VPTransformState &State)
 Set up the values of the IVs correctly when exiting the vector loop.
 
void fixFixedOrderRecurrence (VPFirstOrderRecurrencePHIRecipe *PhiR, VPTransformState &State)
 Create the exit value of first order recurrences in the middle block and update their users.
 
void fixReduction (VPReductionPHIRecipe *Phi, VPTransformState &State)
 Create code for the loop exit value of the reduction.
 
void sinkScalarOperands (Instruction *PredInst)
 Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it.
 
ValuegetOrCreateVectorTripCount (BasicBlock *InsertBlock)
 Returns (and creates if needed) the trip count of the widened loop.
 
ValuecreateBitOrPointerCast (Value *V, VectorType *DstVTy, const DataLayout &DL)
 Returns a bitcasted value to the requested vector type.
 
void emitIterationCountCheck (BasicBlock *Bypass)
 Emit a bypass check to see if the vector trip count is zero, including if it overflows.
 
BasicBlockemitSCEVChecks (BasicBlock *Bypass)
 Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct.
 
BasicBlockemitMemRuntimeChecks (BasicBlock *Bypass)
 Emit bypass checks to check any memory assumptions we may have made.
 
void createVectorLoopSkeleton (StringRef Prefix)
 Emit basic blocks (prefixed with Prefix) for the iteration check, vector loop preheader, middle block and scalar preheader.
 
void createInductionResumeValues (const SCEV2ValueTy &ExpandedSCEVs, std::pair< BasicBlock *, Value * > AdditionalBypass={nullptr, nullptr})
 Create new phi nodes for the induction variables to resume iteration count in the scalar epilogue, from where the vectorized loop left off.
 
BasicBlockcompleteLoopSkeleton ()
 Complete the loop skeleton by adding debug MDs, creating appropriate conditional branches in the middle block, preparing the builder and running the verifier.
 
virtual void printDebugTracesAtStart ()
 Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.
 
virtual void printDebugTracesAtEnd ()
 

Additional Inherited Members

- Public Types inherited from llvm::InnerLoopVectorizer
using VectorParts = SmallVector< Value *, 2 >
 A type for vectorized values in the new loop.
 
- Public Attributes inherited from llvm::InnerLoopAndEpilogueVectorizer
EpilogueLoopVectorizationInfoEPI
 Holds and updates state information required to vectorize the main loop and its epilogue in two separate passes.
 
- Protected Types inherited from llvm::InnerLoopVectorizer
using PhiVector = SmallVector< PHINode *, 4 >
 A small list of PHINodes.
 
using ScalarParts = SmallVector< SmallVector< Value *, 4 >, 2 >
 A type for scalarized values in the new loop.
 
- Protected Attributes inherited from llvm::InnerLoopVectorizer
LoopOrigLoop
 The original loop.
 
PredicatedScalarEvolutionPSE
 A wrapper around ScalarEvolution used to add runtime SCEV checks.
 
LoopInfoLI
 Loop Info.
 
DominatorTreeDT
 Dominator Tree.
 
const TargetLibraryInfoTLI
 Target Library Info.
 
const TargetTransformInfoTTI
 Target Transform Info.
 
AssumptionCacheAC
 Assumption Cache.
 
OptimizationRemarkEmitterORE
 Interface to emit optimization remarks.
 
ElementCount VF
 The vectorization SIMD factor to use.
 
ElementCount MinProfitableTripCount
 
unsigned UF
 The vectorization unroll factor to use.
 
IRBuilder Builder
 The builder that we use.
 
BasicBlockLoopVectorPreHeader
 The vector-loop preheader.
 
BasicBlockLoopScalarPreHeader
 The scalar-loop preheader.
 
BasicBlockLoopMiddleBlock
 Middle Block between the vector and the scalar.
 
BasicBlockLoopExitBlock
 The unique ExitBlock of the scalar loop if one exists.
 
BasicBlockLoopScalarBody
 The scalar loop body.
 
SmallVector< BasicBlock *, 4 > LoopBypassBlocks
 A list of all bypass blocks. The first block is the entry of the loop.
 
SmallVector< Instruction *, 4 > PredicatedInstructions
 Store instructions that were predicated.
 
ValueTripCount = nullptr
 Trip count of the original loop.
 
ValueVectorTripCount = nullptr
 Trip count of the widened loop (TripCount - TripCount % (VF*UF))
 
LoopVectorizationLegalityLegal
 The legality analysis.
 
LoopVectorizationCostModelCost
 The profitablity analysis.
 
bool AddedSafetyChecks = false
 
DenseMap< PHINode *, Value * > IVEndValues
 
BlockFrequencyInfoBFI
 BFI and PSI are used to check for profile guided size optimizations.
 
ProfileSummaryInfoPSI
 
bool OptForSizeBasedOnProfile
 
GeneratedRTChecks & RTChecks
 Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable.
 
SmallMapVector< const RecurrenceDescriptor *, PHINode *, 4 > ReductionResumeValues
 

Detailed Description

A specialized derived class of inner loop vectorizer that performs vectorization of main loops in the process of vectorizing loops and their epilogues.

Definition at line 865 of file LoopVectorize.cpp.

Constructor & Destructor Documentation

◆ EpilogueVectorizerMainLoop()

llvm::EpilogueVectorizerMainLoop::EpilogueVectorizerMainLoop ( Loop OrigLoop,
PredicatedScalarEvolution PSE,
LoopInfo LI,
DominatorTree DT,
const TargetLibraryInfo TLI,
const TargetTransformInfo TTI,
AssumptionCache AC,
OptimizationRemarkEmitter ORE,
EpilogueLoopVectorizationInfo EPI,
LoopVectorizationLegality LVL,
llvm::LoopVectorizationCostModel CM,
BlockFrequencyInfo BFI,
ProfileSummaryInfo PSI,
GeneratedRTChecks &  Check 
)
inline

Definition at line 867 of file LoopVectorize.cpp.

References Check.

Member Function Documentation

◆ createEpilogueVectorizedLoopSkeleton()

std::pair< BasicBlock *, Value * > EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton ( const SCEV2ValueTy &  ExpandedSCEVs)
finalvirtual

◆ emitIterationCountCheck()

BasicBlock * EpilogueVectorizerMainLoop::emitIterationCountCheck ( BasicBlock Bypass,
bool  ForEpilogue 
)
protected

Emits an iteration count bypass check once for the main loop (when ForEpilogue is false) and once for the epilogue loop (when ForEpilogue is true).

Definition at line 7639 of file LoopVectorize.cpp.

References assert(), llvm::InnerLoopVectorizer::Builder, llvm::DominatorTreeBase< NodeT, IsPostDom >::changeImmediateDominator(), llvm::BranchInst::Create(), llvm::IRBuilderBase::CreateICmp(), llvm::createStepForVF(), llvm::InnerLoopVectorizer::DT, llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::EpilogueLoopVectorizationInfo::EpilogueUF, llvm::EpilogueLoopVectorizationInfo::EpilogueVF, llvm::DomTreeNodeBase< NodeT >::getIDom(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::DominatorTreeBase< NodeT, IsPostDom >::getNode(), llvm::BasicBlock::getTerminator(), llvm::InnerLoopVectorizer::getTripCount(), llvm::Value::getType(), llvm::hasBranchWeightMD(), llvm::CmpInst::ICMP_ULE, llvm::CmpInst::ICMP_ULT, llvm::ElementCount::isVector(), llvm::InnerLoopVectorizer::LI, llvm::InnerLoopVectorizer::LoopBypassBlocks, llvm::InnerLoopVectorizer::LoopExitBlock, llvm::InnerLoopVectorizer::LoopVectorPreHeader, MinItersBypassWeights, llvm::InnerLoopVectorizer::OrigLoop, P, llvm::DominatorTreeBase< NodeT, IsPostDom >::properlyDominates(), llvm::ReplaceInstWithInst(), llvm::setBranchWeights(), llvm::Value::setName(), llvm::SplitBlock(), llvm::EpilogueLoopVectorizationInfo::TripCount, llvm::InnerLoopVectorizer::UF, and llvm::InnerLoopVectorizer::VF.

Referenced by createEpilogueVectorizedLoopSkeleton().

◆ printDebugTracesAtEnd()

void EpilogueVectorizerMainLoop::printDebugTracesAtEnd ( )
overrideprotectedvirtual

◆ printDebugTracesAtStart()

void EpilogueVectorizerMainLoop::printDebugTracesAtStart ( )
overrideprotectedvirtual

Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.

Reimplemented from llvm::InnerLoopVectorizer.

Definition at line 7621 of file LoopVectorize.cpp.

References llvm::dbgs(), llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::EpilogueLoopVectorizationInfo::EpilogueUF, llvm::EpilogueLoopVectorizationInfo::EpilogueVF, LLVM_DEBUG, llvm::EpilogueLoopVectorizationInfo::MainLoopUF, and llvm::EpilogueLoopVectorizationInfo::MainLoopVF.


The documentation for this class was generated from the following file: