LLVM  14.0.0git
Public Member Functions | Protected Member Functions | List of all members
llvm::EpilogueVectorizerEpilogueLoop Class Reference
Inheritance diagram for llvm::EpilogueVectorizerEpilogueLoop:
Inheritance graph
[legend]
Collaboration diagram for llvm::EpilogueVectorizerEpilogueLoop:
Collaboration graph
[legend]

Public Member Functions

 EpilogueVectorizerEpilogueLoop (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, EpilogueLoopVectorizationInfo &EPI, LoopVectorizationLegality *LVL, llvm::LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &Checks)
 
BasicBlockcreateEpilogueVectorizedLoopSkeleton () final override
 Implements the interface for creating a vectorized skeleton using the epilogue loop strategy (ie the second pass of vplan execution). More...
 
- Public Member Functions inherited from llvm::InnerLoopAndEpilogueVectorizer
 InnerLoopAndEpilogueVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, EpilogueLoopVectorizationInfo &EPI, LoopVectorizationLegality *LVL, llvm::LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &Checks)
 
BasicBlockcreateVectorizedLoopSkeleton () final override
 Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder. More...
 
- Public Member Functions inherited from llvm::InnerLoopVectorizer
 InnerLoopVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, ElementCount VecWidth, unsigned UnrollFactor, LoopVectorizationLegality *LVL, LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &RTChecks)
 
virtual ~InnerLoopVectorizer ()=default
 
void widenInstruction (Instruction &I, VPValue *Def, VPUser &Operands, VPTransformState &State)
 Widen a single instruction within the innermost loop. More...
 
void widenCallInstruction (CallInst &I, VPValue *Def, VPUser &ArgOperands, VPTransformState &State)
 Widen a single call instruction within the innermost loop. More...
 
void widenSelectInstruction (SelectInst &I, VPValue *VPDef, VPUser &Operands, bool InvariantCond, VPTransformState &State)
 Widen a single select instruction within the innermost loop. More...
 
void fixVectorizedLoop (VPTransformState &State)
 Fix the vectorized code, taking care of header phi's, live-outs, and more. More...
 
bool areSafetyChecksAdded ()
 
void widenGEP (GetElementPtrInst *GEP, VPValue *VPDef, VPUser &Indices, unsigned UF, ElementCount VF, bool IsPtrLoopInvariant, SmallBitVector &IsIndexLoopInvariant, VPTransformState &State)
 Vectorize a single GetElementPtrInst based on information gathered and decisions taken during planning. More...
 
void widenPHIInstruction (Instruction *PN, VPWidenPHIRecipe *PhiR, VPTransformState &State)
 Vectorize a single first-order recurrence or pointer induction PHINode in a block. More...
 
void scalarizeInstruction (Instruction *Instr, VPValue *Def, VPUser &Operands, const VPIteration &Instance, bool IfPredicateInstr, VPTransformState &State)
 A helper function to scalarize a single Instruction in the innermost loop. More...
 
void widenIntOrFpInduction (PHINode *IV, Value *Start, TruncInst *Trunc, VPValue *Def, VPValue *CastDef, VPTransformState &State)
 Widen an integer or floating-point induction variable IV. More...
 
void packScalarIntoVectorValue (VPValue *Def, const VPIteration &Instance, VPTransformState &State)
 Construct the vector value of a scalarized value V one lane at a time. More...
 
void vectorizeInterleaveGroup (const InterleaveGroup< Instruction > *Group, ArrayRef< VPValue * > VPDefs, VPTransformState &State, VPValue *Addr, ArrayRef< VPValue * > StoredValues, VPValue *BlockInMask=nullptr)
 Try to vectorize interleaved access group Group with the base address given in Addr, optionally masking the vector operations if BlockInMask is non-null. More...
 
void vectorizeMemoryInstruction (Instruction *Instr, VPTransformState &State, VPValue *Def, VPValue *Addr, VPValue *StoredValue, VPValue *BlockInMask, bool ConsecutiveStride, bool Reverse)
 Vectorize Load and Store instructions with the base address given in Addr, optionally masking the vector operations if BlockInMask is non-null. More...
 
void setDebugLocFromInst (const Value *V, Optional< IRBuilder<> * > CustomBuilder=None)
 Set the debug location in the builder Ptr using the debug location in V. More...
 
void fixNonInductionPHIs (VPTransformState &State)
 Fix the non-induction PHIs in the OrigPHIsToFix vector. More...
 
bool useOrderedReductions (RecurrenceDescriptor &RdxDesc)
 Returns true if the reordering of FP operations is not allowed, but we are able to vectorize with strict in-order reductions for the given RdxDesc. More...
 
virtual ValuegetBroadcastInstrs (Value *V)
 Create a broadcast instruction. More...
 

Protected Member Functions

BasicBlockemitMinimumVectorEpilogueIterCountCheck (Loop *L, BasicBlock *Bypass, BasicBlock *Insert)
 Emits an iteration count bypass check after the main vector loop has finished to see if there are any iterations left to execute by either the vector epilogue or the scalar epilogue. More...
 
void printDebugTracesAtStart () override
 Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested. More...
 
void printDebugTracesAtEnd () override
 
- Protected Member Functions inherited from llvm::InnerLoopVectorizer
void fixupIVUsers (PHINode *OrigPhi, const InductionDescriptor &II, Value *CountRoundDown, Value *EndValue, BasicBlock *MiddleBlock)
 Set up the values of the IVs correctly when exiting the vector loop. More...
 
PHINodecreateInductionVariable (Loop *L, Value *Start, Value *End, Value *Step, Instruction *DL)
 Create a new induction variable inside L. More...
 
void fixCrossIterationPHIs (VPTransformState &State)
 Handle all cross-iteration phis in the header. More...
 
void fixFirstOrderRecurrence (VPWidenPHIRecipe *PhiR, VPTransformState &State)
 Create the exit value of first order recurrences in the middle block and update their users. More...
 
void fixReduction (VPReductionPHIRecipe *Phi, VPTransformState &State)
 Create code for the loop exit value of the reduction. More...
 
void clearReductionWrapFlags (const RecurrenceDescriptor &RdxDesc, VPTransformState &State)
 Clear NSW/NUW flags from reduction instructions if necessary. More...
 
void fixLCSSAPHIs (VPTransformState &State)
 Fixup the LCSSA phi nodes in the unique exit block. More...
 
void sinkScalarOperands (Instruction *PredInst)
 Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it. More...
 
void truncateToMinimalBitwidths (VPTransformState &State)
 Shrinks vector element sizes to the smallest bitwidth they can be legally represented as. More...
 
virtual ValuegetStepVector (Value *Val, int StartIdx, Value *Step, Instruction::BinaryOps Opcode=Instruction::BinaryOpsEnd)
 This function adds (StartIdx * Step, (StartIdx + 1) * Step, (StartIdx + 2) * Step, ...) to each vector element of Val. More...
 
void buildScalarSteps (Value *ScalarIV, Value *Step, Instruction *EntryVal, const InductionDescriptor &ID, VPValue *Def, VPValue *CastDef, VPTransformState &State)
 Compute scalar induction steps. More...
 
void createVectorIntOrFpInductionPHI (const InductionDescriptor &II, Value *Step, Value *Start, Instruction *EntryVal, VPValue *Def, VPValue *CastDef, VPTransformState &State)
 Create a vector induction phi node based on an existing scalar one. More...
 
bool shouldScalarizeInstruction (Instruction *I) const
 Returns true if an instruction I should be scalarized instead of vectorized for the chosen vectorization factor. More...
 
bool needsScalarInduction (Instruction *IV) const
 Returns true if we should generate a scalar version of IV. More...
 
void recordVectorLoopValueForInductionCast (const InductionDescriptor &ID, const Instruction *EntryVal, Value *VectorLoopValue, VPValue *CastDef, VPTransformState &State, unsigned Part, unsigned Lane=UINT_MAX)
 If there is a cast involved in the induction variable ID, which should be ignored in the vectorized loop body, this function records the VectorLoopValue of the respective Phi also as the VectorLoopValue of the cast. More...
 
virtual ValuereverseVector (Value *Vec)
 Generate a shuffle sequence that will reverse the vector Vec. More...
 
ValuegetOrCreateTripCount (Loop *NewLoop)
 Returns (and creates if needed) the original loop trip count. More...
 
ValuegetOrCreateVectorTripCount (Loop *NewLoop)
 Returns (and creates if needed) the trip count of the widened loop. More...
 
ValuecreateBitOrPointerCast (Value *V, VectorType *DstVTy, const DataLayout &DL)
 Returns a bitcasted value to the requested vector type. More...
 
void emitMinimumIterationCountCheck (Loop *L, BasicBlock *Bypass)
 Emit a bypass check to see if the vector trip count is zero, including if it overflows. More...
 
BasicBlockemitSCEVChecks (Loop *L, BasicBlock *Bypass)
 Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct. More...
 
BasicBlockemitMemRuntimeChecks (Loop *L, BasicBlock *Bypass)
 Emit bypass checks to check any memory assumptions we may have made. More...
 
ValueemitTransformedIndex (IRBuilder<> &B, Value *Index, ScalarEvolution *SE, const DataLayout &DL, const InductionDescriptor &ID) const
 Compute the transformed value of Index at offset StartValue using step StepValue. More...
 
LoopcreateVectorLoopSkeleton (StringRef Prefix)
 Emit basic blocks (prefixed with Prefix) for the iteration check, vector loop preheader, middle block and scalar preheader. More...
 
void createInductionResumeValues (Loop *L, Value *VectorTripCount, std::pair< BasicBlock *, Value * > AdditionalBypass={nullptr, nullptr})
 Create new phi nodes for the induction variables to resume iteration count in the scalar epilogue, from where the vectorized loop left off (given by VectorTripCount). More...
 
BasicBlockcompleteLoopSkeleton (Loop *L, MDNode *OrigLoopID)
 Complete the loop skeleton by adding debug MDs, creating appropriate conditional branches in the middle block, preparing the builder and running the verifier. More...
 
void addNewMetadata (Instruction *To, const Instruction *Orig)
 Add additional metadata to To that was not present on Orig. More...
 
void addMetadata (Instruction *To, Instruction *From)
 Add metadata from one instruction to another. More...
 
void addMetadata (ArrayRef< Value * > To, Instruction *From)
 Similar to the previous function but it adds the metadata to a vector of instructions. More...
 

Additional Inherited Members

- Public Types inherited from llvm::InnerLoopVectorizer
using VectorParts = SmallVector< Value *, 2 >
 A type for vectorized values in the new loop. More...
 
- Public Attributes inherited from llvm::InnerLoopAndEpilogueVectorizer
EpilogueLoopVectorizationInfoEPI
 Holds and updates state information required to vectorize the main loop and its epilogue in two separate passes. More...
 
- Protected Types inherited from llvm::InnerLoopVectorizer
using PhiVector = SmallVector< PHINode *, 4 >
 A small list of PHINodes. More...
 
using ScalarParts = SmallVector< SmallVector< Value *, 4 >, 2 >
 A type for scalarized values in the new loop. More...
 
- Protected Attributes inherited from llvm::InnerLoopVectorizer
LoopOrigLoop
 The original loop. More...
 
PredicatedScalarEvolutionPSE
 A wrapper around ScalarEvolution used to add runtime SCEV checks. More...
 
LoopInfoLI
 Loop Info. More...
 
DominatorTreeDT
 Dominator Tree. More...
 
AAResultsAA
 Alias Analysis. More...
 
const TargetLibraryInfoTLI
 Target Library Info. More...
 
const TargetTransformInfoTTI
 Target Transform Info. More...
 
AssumptionCacheAC
 Assumption Cache. More...
 
OptimizationRemarkEmitterORE
 Interface to emit optimization remarks. More...
 
std::unique_ptr< LoopVersioningLVer
 LoopVersioning. More...
 
ElementCount VF
 The vectorization SIMD factor to use. More...
 
unsigned UF
 The vectorization unroll factor to use. More...
 
IRBuilder Builder
 The builder that we use. More...
 
BasicBlockLoopVectorPreHeader
 The vector-loop preheader. More...
 
BasicBlockLoopScalarPreHeader
 The scalar-loop preheader. More...
 
BasicBlockLoopMiddleBlock
 Middle Block between the vector and the scalar. More...
 
BasicBlockLoopExitBlock
 The unique ExitBlock of the scalar loop if one exists. More...
 
BasicBlockLoopVectorBody
 The vector loop body. More...
 
BasicBlockLoopScalarBody
 The scalar loop body. More...
 
SmallVector< BasicBlock *, 4 > LoopBypassBlocks
 A list of all bypass blocks. The first block is the entry of the loop. More...
 
PHINodeInduction = nullptr
 The new Induction variable which was added to the new block. More...
 
PHINodeOldInduction = nullptr
 The induction variable of the old basic block. More...
 
SmallVector< Instruction *, 4 > PredicatedInstructions
 Store instructions that were predicated. More...
 
ValueTripCount = nullptr
 Trip count of the original loop. More...
 
ValueVectorTripCount = nullptr
 Trip count of the widened loop (TripCount - TripCount % (VF*UF)) More...
 
LoopVectorizationLegalityLegal
 The legality analysis. More...
 
LoopVectorizationCostModelCost
 The profitablity analysis. More...
 
bool AddedSafetyChecks = false
 
DenseMap< PHINode *, Value * > IVEndValues
 
SmallVector< PHINode *, 8 > OrigPHIsToFix
 
BlockFrequencyInfoBFI
 BFI and PSI are used to check for profile guided size optimizations. More...
 
ProfileSummaryInfoPSI
 
bool OptForSizeBasedOnProfile
 
GeneratedRTChecksRTChecks
 Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable. More...
 

Detailed Description

Definition at line 995 of file LoopVectorize.cpp.

Constructor & Destructor Documentation

◆ EpilogueVectorizerEpilogueLoop()

llvm::EpilogueVectorizerEpilogueLoop::EpilogueVectorizerEpilogueLoop ( Loop OrigLoop,
PredicatedScalarEvolution PSE,
LoopInfo LI,
DominatorTree DT,
const TargetLibraryInfo TLI,
const TargetTransformInfo TTI,
AssumptionCache AC,
OptimizationRemarkEmitter ORE,
EpilogueLoopVectorizationInfo EPI,
LoopVectorizationLegality LVL,
llvm::LoopVectorizationCostModel CM,
BlockFrequencyInfo BFI,
ProfileSummaryInfo PSI,
GeneratedRTChecks Checks 
)
inline

Definition at line 997 of file LoopVectorize.cpp.

Member Function Documentation

◆ createEpilogueVectorizedLoopSkeleton()

BasicBlock * EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton ( )
finaloverridevirtual

Implements the interface for creating a vectorized skeleton using the epilogue loop strategy (ie the second pass of vplan execution).

This function is partially responsible for generating the control flow depicted in https://llvm.org/docs/Vectorizers.html#epilogue-vectorization.

Implements llvm::InnerLoopAndEpilogueVectorizer.

Definition at line 8492 of file LoopVectorize.cpp.

References llvm::PHINode::addIncoming(), AddRuntimeUnrollDisableMetaData(), assert(), llvm::DominatorTreeBase< NodeT, IsPostDom >::changeImmediateDominator(), llvm::InnerLoopVectorizer::completeLoopSkeleton(), llvm::InnerLoopVectorizer::Cost, llvm::PHINode::Create(), llvm::InnerLoopVectorizer::createInductionResumeValues(), llvm::InnerLoopVectorizer::createInductionVariable(), llvm::InnerLoopVectorizer::createVectorLoopSkeleton(), llvm::InnerLoopVectorizer::DT, llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::EpilogueLoopVectorizationInfo::EpilogueIterationCountCheck, llvm::EpilogueLoopVectorizationInfo::EpilogueVF, llvm::ConstantInt::get(), getDebugLocFromInstOrOperands(), llvm::BasicBlock::getFirstNonPHI(), llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::Loop::getLoopID(), llvm::InnerLoopVectorizer::getOrCreateVectorTripCount(), llvm::LoopVectorizationLegality::getPrimaryInduction(), llvm::BasicBlock::getSinglePredecessor(), llvm::BasicBlock::getTerminator(), llvm::LoopVectorizationLegality::getWidestInductionType(), llvm::InnerLoopVectorizer::Induction, llvm::InnerLoopVectorizer::Legal, llvm::InnerLoopVectorizer::LI, llvm::InnerLoopVectorizer::LoopBypassBlocks, llvm::InnerLoopVectorizer::LoopExitBlock, llvm::InnerLoopVectorizer::LoopScalarPreHeader, llvm::InnerLoopVectorizer::LoopVectorPreHeader, llvm::EpilogueLoopVectorizationInfo::MainLoopIterationCountCheck, llvm::EpilogueLoopVectorizationInfo::MemSafetyCheck, llvm::InnerLoopVectorizer::OldInduction, llvm::InnerLoopVectorizer::OrigLoop, llvm::User::replaceUsesOfWith(), llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), llvm::EpilogueLoopVectorizationInfo::SCEVSafetyCheck, llvm::Value::setName(), llvm::SplitBlock(), llvm::InnerLoopVectorizer::UF, llvm::EpilogueLoopVectorizationInfo::VectorTripCount, and llvm::InnerLoopVectorizer::VF.

◆ emitMinimumVectorEpilogueIterCountCheck()

BasicBlock * EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck ( Loop L,
BasicBlock Bypass,
BasicBlock Insert 
)
protected

◆ printDebugTracesAtEnd()

void EpilogueVectorizerEpilogueLoop::printDebugTracesAtEnd ( )
overrideprotectedvirtual

◆ printDebugTracesAtStart()

void EpilogueVectorizerEpilogueLoop::printDebugTracesAtStart ( )
overrideprotectedvirtual

Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.

Reimplemented from llvm::InnerLoopVectorizer.

Definition at line 8612 of file LoopVectorize.cpp.

References llvm::dbgs(), llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::EpilogueLoopVectorizationInfo::EpilogueUF, llvm::EpilogueLoopVectorizationInfo::EpilogueVF, and LLVM_DEBUG.


The documentation for this class was generated from the following file: