An extension of the inner loop vectorizer that creates a skeleton for a vectorized loop that has its epilogue (residual) also vectorized. More...
Public Member Functions | |
| InnerLoopAndEpilogueVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, EpilogueLoopVectorizationInfo &EPI, LoopVectorizationLegality *LVL, llvm::LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &Checks, VPlan &Plan) | |
| BasicBlock * | createVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs) final |
| Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder. | |
| virtual BasicBlock * | createEpilogueVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs)=0 |
| The interface for creating a vectorized skeleton using one of two different strategies, each corresponding to one execution of the vplan as described above. | |
 Public Member Functions inherited from llvm::InnerLoopVectorizer | |
| InnerLoopVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, ElementCount VecWidth, ElementCount MinProfitableTripCount, unsigned UnrollFactor, LoopVectorizationLegality *LVL, LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &RTChecks, VPlan &Plan) | |
| virtual | ~InnerLoopVectorizer ()=default |
| virtual BasicBlock * | createVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs) |
| Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder. | |
| void | fixVectorizedLoop (VPTransformState &State) |
| Fix the vectorized code, taking care of header phi's, and more. | |
| bool | areSafetyChecksAdded () |
| void | scalarizeInstruction (const Instruction *Instr, VPReplicateRecipe *RepRecipe, const VPLane &Lane, VPTransformState &State) |
| A helper function to scalarize a single Instruction in the innermost loop. | |
| void | fixNonInductionPHIs (VPTransformState &State) |
Fix the non-induction PHIs in Plan. | |
| void | createInductionResumeVPValue (VPIRInstruction *InductionPhiIRI, const InductionDescriptor &ID, Value *Step, ArrayRef< BasicBlock * > BypassBlocks, VPBuilder &ScalarPHBuilder, Value *MainVectorTripCount=nullptr) |
Create a ResumePHI VPInstruction for the induction InductionPhiIRI to resume iteration count in the scalar epilogue from where the vectorized loop left off, and add it to the scalar preheader of VPlan. | |
| Value * | getTripCount () const |
| Returns the original loop trip count. | |
| void | setTripCount (Value *TC) |
| Used to set the trip count after ILV's construction and after the preheader block has been executed. | |
| Value * | getInductionAdditionalBypassValue (PHINode *OrigPhi) const |
| induction header phi. | |
| BasicBlock * | getAdditionalBypassBlock () const |
| Return the additional bypass block which targets the scalar loop by skipping the epilogue loop after completing the main loop. | |
Public Attributes | |
| EpilogueLoopVectorizationInfo & | EPI |
| Holds and updates state information required to vectorize the main loop and its epilogue in two separate passes. | |
Additional Inherited Members | |
| Protected Member Functions inherited from llvm::InnerLoopVectorizer | |
| virtual void | fixupIVUsers (PHINode *OrigPhi, const InductionDescriptor &II, Value *VectorTripCount, BasicBlock *MiddleBlock, VPTransformState &State) |
| Set up the values of the IVs correctly when exiting the vector loop. | |
| void | sinkScalarOperands (Instruction *PredInst) |
| Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it. | |
| Value * | getOrCreateVectorTripCount (BasicBlock *InsertBlock) |
| Returns (and creates if needed) the trip count of the widened loop. | |
| void | emitIterationCountCheck (BasicBlock *Bypass) |
| Emit a bypass check to see if the vector trip count is zero, including if it overflows. | |
| BasicBlock * | emitSCEVChecks (BasicBlock *Bypass) |
| Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct. | |
| BasicBlock * | emitMemRuntimeChecks (BasicBlock *Bypass) |
| Emit bypass checks to check any memory assumptions we may have made. | |
| void | createVectorLoopSkeleton (StringRef Prefix) |
Emit basic blocks (prefixed with Prefix) for the iteration check, vector loop preheader, middle block and scalar preheader. | |
| void | createInductionResumeVPValues (const SCEV2ValueTy &ExpandedSCEVs, Value *MainVectorTripCount=nullptr, SmallPtrSetImpl< PHINode * > *IVSubset=nullptr) |
| Create new phi nodes for the induction variables to resume iteration count in the scalar epilogue, from where the vectorized loop left off. | |
| virtual void | printDebugTracesAtStart () |
| Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested. | |
| virtual void | printDebugTracesAtEnd () |
| Protected Attributes inherited from llvm::InnerLoopVectorizer | |
| Loop * | OrigLoop |
| The original loop. | |
| PredicatedScalarEvolution & | PSE |
| A wrapper around ScalarEvolution used to add runtime SCEV checks. | |
| LoopInfo * | LI |
| Loop Info. | |
| DominatorTree * | DT |
| Dominator Tree. | |
| const TargetLibraryInfo * | TLI |
| Target Library Info. | |
| const TargetTransformInfo * | TTI |
| Target Transform Info. | |
| AssumptionCache * | AC |
| Assumption Cache. | |
| OptimizationRemarkEmitter * | ORE |
| Interface to emit optimization remarks. | |
| ElementCount | VF |
| The vectorization SIMD factor to use. | |
| ElementCount | MinProfitableTripCount |
| unsigned | UF |
| The vectorization unroll factor to use. | |
| IRBuilder | Builder |
| The builder that we use. | |
| BasicBlock * | LoopVectorPreHeader |
| The vector-loop preheader. | |
| BasicBlock * | LoopScalarPreHeader |
| The scalar-loop preheader. | |
| BasicBlock * | LoopMiddleBlock |
| Middle Block between the vector and the scalar. | |
| SmallVector< BasicBlock *, 4 > | LoopBypassBlocks |
| A list of all bypass blocks. The first block is the entry of the loop. | |
| SmallVector< Instruction *, 4 > | PredicatedInstructions |
| Store instructions that were predicated. | |
| Value * | TripCount = nullptr |
| Trip count of the original loop. | |
| Value * | VectorTripCount = nullptr |
| Trip count of the widened loop (TripCount - TripCount % (VF*UF)) | |
| LoopVectorizationLegality * | Legal |
| The legality analysis. | |
| LoopVectorizationCostModel * | Cost |
| The profitablity analysis. | |
| bool | AddedSafetyChecks = false |
| BlockFrequencyInfo * | BFI |
| BFI and PSI are used to check for profile guided size optimizations. | |
| ProfileSummaryInfo * | PSI |
| bool | OptForSizeBasedOnProfile |
| GeneratedRTChecks & | RTChecks |
| Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable. | |
| DenseMap< PHINode *, Value * > | Induction2AdditionalBypassValue |
| Mapping of induction phis to their additional bypass values. | |
| BasicBlock * | AdditionalBypassBlock = nullptr |
| The additional bypass block which conditionally skips over the epilogue loop after executing the main loop. | |
| VPlan & | Plan |
Detailed Description
An extension of the inner loop vectorizer that creates a skeleton for a vectorized loop that has its epilogue (residual) also vectorized.
The idea is to run the vplan on a given loop twice, firstly to setup the skeleton and vectorize the main loop, and secondly to complete the skeleton from the first step and vectorize the epilogue. This is achieved by deriving two concrete strategy classes from this base class and invoking them in succession from the loop vectorizer planner.
Definition at line 737 of file LoopVectorize.cpp.
Constructor & Destructor Documentation
◆ InnerLoopAndEpilogueVectorizer()
|
inline |
Definition at line 739 of file LoopVectorize.cpp.
Member Function Documentation
◆ createEpilogueVectorizedLoopSkeleton()
|
pure virtual |
The interface for creating a vectorized skeleton using one of two different strategies, each corresponding to one execution of the vplan as described above.
Implemented in llvm::EpilogueVectorizerMainLoop, and llvm::EpilogueVectorizerEpilogueLoop.
Referenced by createVectorizedLoopSkeleton().
◆ createVectorizedLoopSkeleton()
|
inlinefinalvirtual |
Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder.
Control flow is generated around the vectorized (and scalar epilogue) loops consisting of various checks and bypasses. Return the pre-header block of the new loop. In the case of epilogue vectorization, this function is overriden to handle the more complex control flow around the loops. ExpandedSCEVs is used to look up SCEV expansions for expressions needed during skeleton creation.
Reimplemented from llvm::InnerLoopVectorizer.
Definition at line 755 of file LoopVectorize.cpp.
References createEpilogueVectorizedLoopSkeleton().
Member Data Documentation
◆ EPI
| EpilogueLoopVectorizationInfo& llvm::InnerLoopAndEpilogueVectorizer::EPI |
Holds and updates state information required to vectorize the main loop and its epilogue in two separate passes.
This setup helps us avoid regenerating and recomputing runtime safety checks. It also helps us to shorten the iteration-count-check path length for the cases where the iteration count of the loop is so small that the main vector loop is completely skipped.
Definition at line 771 of file LoopVectorize.cpp.
Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), llvm::EpilogueVectorizerEpilogueLoop::EpilogueVectorizerEpilogueLoop(), llvm::EpilogueVectorizerMainLoop::printDebugTracesAtStart(), and llvm::EpilogueVectorizerEpilogueLoop::printDebugTracesAtStart().
The documentation for this class was generated from the following file:
- lib/Transforms/Vectorize/LoopVectorize.cpp
