LLVM 20.0.0git
|
InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF). More...
Public Member Functions | |
InnerLoopVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, ElementCount VecWidth, ElementCount MinProfitableTripCount, unsigned UnrollFactor, LoopVectorizationLegality *LVL, LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &RTChecks, VPlan &Plan) | |
virtual | ~InnerLoopVectorizer ()=default |
virtual BasicBlock * | createVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs) |
Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder. | |
void | fixVectorizedLoop (VPTransformState &State) |
Fix the vectorized code, taking care of header phi's, and more. | |
bool | areSafetyChecksAdded () |
void | scalarizeInstruction (const Instruction *Instr, VPReplicateRecipe *RepRecipe, const VPLane &Lane, VPTransformState &State) |
A helper function to scalarize a single Instruction in the innermost loop. | |
void | fixNonInductionPHIs (VPTransformState &State) |
Fix the non-induction PHIs in Plan . | |
Value * | getTripCount () const |
Returns the original loop trip count. | |
void | setTripCount (Value *TC) |
Used to set the trip count after ILV's construction and after the preheader block has been executed. | |
Value * | getInductionAdditionalBypassValue (PHINode *OrigPhi) const |
induction header phi. | |
BasicBlock * | getAdditionalBypassBlock () const |
Return the additional bypass block which targets the scalar loop by skipping the epilogue loop after completing the main loop. | |
Protected Member Functions | |
virtual void | fixupIVUsers (PHINode *OrigPhi, const InductionDescriptor &II, Value *VectorTripCount, BasicBlock *MiddleBlock, VPTransformState &State) |
Set up the values of the IVs correctly when exiting the vector loop. | |
void | sinkScalarOperands (Instruction *PredInst) |
Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it. | |
Value * | getOrCreateVectorTripCount (BasicBlock *InsertBlock) |
Returns (and creates if needed) the trip count of the widened loop. | |
void | emitIterationCountCheck (BasicBlock *Bypass) |
Emit a bypass check to see if the vector trip count is zero, including if it overflows. | |
BasicBlock * | emitSCEVChecks (BasicBlock *Bypass) |
Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct. | |
BasicBlock * | emitMemRuntimeChecks (BasicBlock *Bypass) |
Emit bypass checks to check any memory assumptions we may have made. | |
void | createVectorLoopSkeleton (StringRef Prefix) |
Emit basic blocks (prefixed with Prefix ) for the iteration check, vector loop preheader, middle block and scalar preheader. | |
void | createInductionAdditionalBypassValues (const SCEV2ValueTy &ExpandedSCEVs, Value *MainVectorTripCount) |
Create and record the values for induction variables to resume coming from the additional bypass block. | |
virtual void | printDebugTracesAtStart () |
Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested. | |
virtual void | printDebugTracesAtEnd () |
void | introduceCheckBlockInVPlan (BasicBlock *CheckIRBB) |
Introduces a new VPIRBasicBlock for CheckIRBB to Plan between the vector preheader and its predecessor, also connecting the new block to the scalar preheader. | |
Protected Attributes | |
Loop * | OrigLoop |
The original loop. | |
PredicatedScalarEvolution & | PSE |
A wrapper around ScalarEvolution used to add runtime SCEV checks. | |
LoopInfo * | LI |
Loop Info. | |
DominatorTree * | DT |
Dominator Tree. | |
const TargetLibraryInfo * | TLI |
Target Library Info. | |
const TargetTransformInfo * | TTI |
Target Transform Info. | |
AssumptionCache * | AC |
Assumption Cache. | |
OptimizationRemarkEmitter * | ORE |
Interface to emit optimization remarks. | |
ElementCount | VF |
The vectorization SIMD factor to use. | |
ElementCount | MinProfitableTripCount |
unsigned | UF |
The vectorization unroll factor to use. | |
IRBuilder | Builder |
The builder that we use. | |
BasicBlock * | LoopVectorPreHeader |
The vector-loop preheader. | |
BasicBlock * | LoopScalarPreHeader |
The scalar-loop preheader. | |
BasicBlock * | LoopMiddleBlock |
Middle Block between the vector and the scalar. | |
SmallVector< BasicBlock *, 4 > | LoopBypassBlocks |
A list of all bypass blocks. The first block is the entry of the loop. | |
SmallVector< Instruction *, 4 > | PredicatedInstructions |
Store instructions that were predicated. | |
Value * | TripCount = nullptr |
Trip count of the original loop. | |
Value * | VectorTripCount = nullptr |
Trip count of the widened loop (TripCount - TripCount % (VF*UF)) | |
LoopVectorizationLegality * | Legal |
The legality analysis. | |
LoopVectorizationCostModel * | Cost |
The profitablity analysis. | |
bool | AddedSafetyChecks = false |
BlockFrequencyInfo * | BFI |
BFI and PSI are used to check for profile guided size optimizations. | |
ProfileSummaryInfo * | PSI |
bool | OptForSizeBasedOnProfile |
GeneratedRTChecks & | RTChecks |
Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable. | |
DenseMap< PHINode *, Value * > | Induction2AdditionalBypassValue |
Mapping of induction phis to their additional bypass values. | |
BasicBlock * | AdditionalBypassBlock = nullptr |
The additional bypass block which conditionally skips over the epilogue loop after executing the main loop. | |
VPlan & | Plan |
VPBlockBase * | VectorPHVPB |
The vector preheader block of Plan , used as target for check blocks introduced during skeleton creation. | |
Friends | |
class | LoopVectorizationPlanner |
InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF).
This class performs the widening of scalars into vectors, or multiple scalars. This class also implements the following features:
Definition at line 466 of file LoopVectorize.cpp.
|
inline |
Definition at line 468 of file LoopVectorize.cpp.
References BFI, llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::IRPass, OptForSizeBasedOnProfile, OrigLoop, PSI, and llvm::shouldOptimizeForSize().
|
virtualdefault |
|
inline |
Definition at line 507 of file LoopVectorize.cpp.
References AddedSafetyChecks.
Referenced by llvm::LoopVectorizePass::processLoop().
|
protected |
Create and record the values for induction variables to resume coming from the additional bypass block.
Definition at line 2682 of file LoopVectorize.cpp.
References assert(), emitTransformedIndex(), getAdditionalBypassBlock(), getExpandedStep(), llvm::LoopVectorizationLegality::getInductionVars(), llvm::LoopVectorizationLegality::getPrimaryInduction(), II, Induction2AdditionalBypassValue, Legal, llvm::IRBuilderBase::setFastMathFlags(), and llvm::Value::setName().
Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton().
|
virtual |
Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder.
Control flow is generated around the vectorized (and scalar epilogue) loops consisting of various checks and bypasses. Return the pre-header block of the new loop. In the case of epilogue vectorization, this function is overriden to handle the more complex control flow around the loops. ExpandedSCEVs
is used to look up SCEV expansions for expressions needed during skeleton creation.
Reimplemented in llvm::InnerLoopAndEpilogueVectorizer.
Definition at line 2718 of file LoopVectorize.cpp.
References createVectorLoopSkeleton(), emitIterationCountCheck(), emitMemRuntimeChecks(), emitSCEVChecks(), LoopScalarPreHeader, and LoopVectorPreHeader.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
protected |
Emit basic blocks (prefixed with Prefix
) for the iteration check, vector loop preheader, middle block and scalar preheader.
Definition at line 2628 of file LoopVectorize.cpp.
References assert(), DT, llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::VPlan::getMiddleBlock(), llvm::VPlan::getScalarPreheader(), llvm::BasicBlock::getTerminator(), llvm::LoopBase< BlockT, LoopT >::getUniqueLatchExitBlock(), llvm::ElementCount::isVector(), LI, LoopMiddleBlock, LoopScalarPreHeader, LoopVectorPreHeader, OrigLoop, Plan, replaceVPBBWithIRVPBB(), llvm::SplitBlock(), and VF.
Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().
|
protected |
Emit a bypass check to see if the vector trip count is zero, including if it overflows.
Definition at line 2472 of file LoopVectorize.cpp.
References llvm::ScalarEvolution::applyLoopGuards(), assert(), Builder, llvm::BranchInst::Create(), llvm::IRBuilderBase::CreateBinaryIntrinsic(), llvm::IRBuilderBase::CreateICmp(), llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), llvm::DataAndControlFlowWithoutRuntimeCheck, DT, llvm::IRBuilderBase::getFalse(), llvm::DomTreeNodeBase< NodeT >::getIDom(), llvm::CmpInst::getInversePredicate(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::DominatorTreeBase< NodeT, IsPostDom >::getNode(), llvm::ScalarEvolution::getSCEV(), llvm::PredicatedScalarEvolution::getSE(), llvm::BasicBlock::getTerminator(), getTripCount(), llvm::IRBuilderBase::getTrue(), llvm::Value::getType(), llvm::hasBranchWeightMD(), llvm::CmpInst::ICMP_ULE, llvm::CmpInst::ICMP_ULT, introduceCheckBlockInVPlan(), isIndvarOverflowCheckKnownFalse(), llvm::ScalarEvolution::isKnownPredicate(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isVector(), LHS, LI, LoopBypassBlocks, LoopVectorPreHeader, MinItersBypassWeights, MinProfitableTripCount, llvm::None, OrigLoop, P, llvm::DominatorTreeBase< NodeT, IsPostDom >::properlyDominates(), PSE, llvm::ReplaceInstWithInst(), llvm::setBranchWeights(), llvm::SplitBlock(), UF, and VF.
Referenced by createVectorizedLoopSkeleton().
|
protected |
Emit bypass checks to check any memory assumptions we may have made.
Returns the block containing the checks or nullptr if no checks have been added.
Definition at line 2576 of file LoopVectorize.cpp.
References AddedSafetyChecks, assert(), DEBUG_TYPE, llvm::OptimizationRemarkEmitter::emit(), llvm::EnableVPlanNativePath, llvm::LoopVectorizeHints::FK_Enabled, llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::BasicBlock::getParent(), llvm::Loop::getStartLoc(), llvm::Function::hasOptSize(), introduceCheckBlockInVPlan(), LoopBypassBlocks, LoopVectorPreHeader, OptForSizeBasedOnProfile, ORE, OrigLoop, and RTChecks.
Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().
|
protected |
Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct.
Returns the block containing the checks or nullptr if no checks have been added.
Definition at line 2557 of file LoopVectorize.cpp.
References AddedSafetyChecks, assert(), llvm::LoopVectorizeHints::FK_Enabled, llvm::BasicBlock::getParent(), llvm::Function::hasOptSize(), introduceCheckBlockInVPlan(), LoopBypassBlocks, LoopVectorPreHeader, OptForSizeBasedOnProfile, and RTChecks.
Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().
void InnerLoopVectorizer::fixNonInductionPHIs | ( | VPTransformState & | State | ) |
Fix the non-induction PHIs in Plan
.
Definition at line 3119 of file LoopVectorize.cpp.
References llvm::PHINode::addIncoming(), Builder, llvm::VPTransformState::CFG, llvm::VPTransformState::get(), llvm::VPlan::getEntry(), llvm::VPWidenPHIRecipe::getIncomingBlock(), llvm::VPWidenPHIRecipe::getIncomingValue(), llvm::VPUser::getNumOperands(), Idx, P, Plan, llvm::IRBuilderBase::SetInsertPoint(), llvm::vp_depth_first_deep(), and llvm::VPTransformState::CFGState::VPBB2IRBB.
Referenced by fixVectorizedLoop().
|
protectedvirtual |
Set up the values of the IVs correctly when exiting the vector loop.
Reimplemented in llvm::EpilogueVectorizerMainLoop.
Definition at line 2782 of file LoopVectorize.cpp.
References llvm::all_of(), assert(), B, llvm::LoopBase< BlockT, LoopT >::contains(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::empty(), llvm::VPTransformState::get(), llvm::PHINode::getIncomingValueForBlock(), llvm::VPValue::getLiveInIRValue(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::VPlan::getSCEVExpansion(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), I, II, llvm::Type::isFloatingPointTy(), llvm::Type::isIntegerTy(), llvm::VPValue::isLiveIn(), llvm::Type::isPointerTy(), OrigLoop, P, PHI, Plan, PostInc, llvm::predecessors(), llvm::Value::setName(), and llvm::Value::users().
Referenced by fixVectorizedLoop().
void InnerLoopVectorizer::fixVectorizedLoop | ( | VPTransformState & | State | ) |
Fix the vectorized code, taking care of header phi's, and more.
Definition at line 2979 of file LoopVectorize.cpp.
References llvm::VPTransformState::CFG, cse(), llvm::EnableVPlanNativePath, fixNonInductionPHIs(), fixupIVUsers(), llvm::ScalarEvolution::forgetBlockAndLoopDispositions(), llvm::ScalarEvolution::forgetLcssaPhiWithNewPredecessor(), llvm::ScalarEvolution::forgetLoop(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::LoopBase< BlockT, LoopT >::getExitBlocks(), llvm::LoopVectorizationLegality::getInductionVars(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), getOrCreateVectorTripCount(), llvm::PredicatedScalarEvolution::getSE(), llvm::VPlan::getVectorLoopRegion(), llvm::ElementCount::isVector(), Legal, LI, LoopMiddleBlock, OrigLoop, llvm::VPTransformState::Plan, PredicatedInstructions, PSE, llvm::setProfileInfoAfterUnrolling(), sinkScalarOperands(), UF, VF, and llvm::VPTransformState::CFGState::VPBB2IRBB.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
inline |
Return the additional bypass block which targets the scalar loop by skipping the epilogue loop after completing the main loop.
Definition at line 537 of file LoopVectorize.cpp.
References AdditionalBypassBlock, and assert().
Referenced by createInductionAdditionalBypassValues(), and llvm::LoopVectorizationPlanner::executePlan().
|
inline |
induction header phi.
Definition at line 531 of file LoopVectorize.cpp.
References Induction2AdditionalBypassValue.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
protected |
Returns (and creates if needed) the trip count of the widened loop.
Definition at line 2408 of file LoopVectorize.cpp.
References assert(), Builder, llvm::IRBuilderBase::CreateAdd(), llvm::IRBuilderBase::CreateICmpEQ(), llvm::IRBuilderBase::CreateSelect(), llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), llvm::IRBuilderBase::CreateURem(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::BasicBlock::getTerminator(), getTripCount(), llvm::Value::getType(), llvm::isPowerOf2_32(), llvm::ElementCount::isVector(), UF, VectorTripCount, and VF.
Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::LoopVectorizationPlanner::executePlan(), and fixVectorizedLoop().
|
inline |
Returns the original loop trip count.
Definition at line 522 of file LoopVectorize.cpp.
References TripCount.
Referenced by emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::LoopVectorizationPlanner::executePlan(), getOrCreateVectorTripCount(), and llvm::LoopVectorizePass::processLoop().
|
protected |
Introduces a new VPIRBasicBlock for CheckIRBB
to Plan between the vector preheader and its predecessor, also connecting the new block to the scalar preheader.
Definition at line 2457 of file LoopVectorize.cpp.
References assert(), llvm::VPBlockUtils::connectBlocks(), llvm::VPlan::createVPIRBasicBlock(), llvm::VPBlockBase::getNumSuccessors(), llvm::VPlan::getScalarPreheader(), llvm::VPBlockBase::getSinglePredecessor(), llvm::VPBlockBase::getSuccessors(), llvm::VPBlockUtils::insertOnEdge(), Plan, llvm::VPBlockBase::swapSuccessors(), and VectorPHVPB.
Referenced by emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), and emitSCEVChecks().
|
inlineprotectedvirtual |
Reimplemented in llvm::EpilogueVectorizerMainLoop, and llvm::EpilogueVectorizerEpilogueLoop.
Definition at line 584 of file LoopVectorize.cpp.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
inlineprotectedvirtual |
Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.
Reimplemented in llvm::EpilogueVectorizerMainLoop, and llvm::EpilogueVectorizerEpilogueLoop.
Definition at line 583 of file LoopVectorize.cpp.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
void InnerLoopVectorizer::scalarizeInstruction | ( | const Instruction * | Instr, |
VPReplicateRecipe * | RepRecipe, | ||
const VPLane & | Lane, | ||
VPTransformState & | State | ||
) |
A helper function to scalarize a single Instruction in the innermost loop.
Generates a sequence of scalar instances for each lane between MinLane
and MaxLane
, times each part between MinPart
and MaxPart
, inclusive. Uses the VPValue operands from RepRecipe
instead of Instr's
operands.
Definition at line 2349 of file LoopVectorize.cpp.
References AC, llvm::VPTransformState::addNewMetadata(), llvm::all_of(), assert(), llvm::VPTransformState::Builder, DL, llvm::enumerate(), llvm::VPTransformState::get(), llvm::VPLane::getFirstLane(), llvm::VPBlockBase::getParent(), llvm::VPRecipeBase::getParent(), llvm::Value::getType(), I, II, llvm::VPTypeAnalysis::inferScalarType(), llvm::IRBuilderBase::Insert(), llvm::VPRegionBlock::isReplicator(), llvm::vputils::isUniformAfterVectorization(), llvm::VPUser::operands(), PredicatedInstructions, llvm::AssumptionCache::registerAssumption(), llvm::VPTransformState::set(), llvm::VPTransformState::setDebugLocFrom(), llvm::VPRecipeWithIRFlags::setFlags(), llvm::Value::setName(), llvm::User::setOperand(), and llvm::VPTransformState::TypeAnalysis.
Referenced by llvm::VPReplicateRecipe::execute().
|
inline |
Used to set the trip count after ILV's construction and after the preheader block has been executed.
Note that this always holds the trip count of the original loop for both main loop and epilogue vectorization.
Definition at line 527 of file LoopVectorize.cpp.
References TripCount.
Referenced by llvm::LoopVectorizationPlanner::executePlan(), and llvm::LoopVectorizePass::processLoop().
|
protected |
Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it.
Definition at line 3043 of file LoopVectorize.cpp.
References llvm::all_of(), llvm::SmallVectorTemplateCommon< T, typename >::begin(), llvm::SmallVectorImpl< T >::clear(), llvm::SetVector< T, Vector, Set, N >::empty(), llvm::SmallVectorTemplateCommon< T, typename >::end(), llvm::PHINode::getIncomingValueNumForOperand(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::ilist_detail::node_parent_access< NodeTy, ParentTy >::getParent(), I, llvm::SetVector< T, Vector, Set, N >::insert(), LI, llvm::User::op_begin(), llvm::User::op_end(), llvm::SetVector< T, Vector, Set, N >::pop_back_val(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().
Referenced by fixVectorizedLoop().
|
friend |
Definition at line 544 of file LoopVectorize.cpp.
|
protected |
Assumption Cache.
Definition at line 612 of file LoopVectorize.cpp.
Referenced by scalarizeInstruction().
|
protected |
Definition at line 660 of file LoopVectorize.cpp.
Referenced by areSafetyChecksAdded(), emitMemRuntimeChecks(), and emitSCEVChecks().
|
protected |
The additional bypass block which conditionally skips over the epilogue loop after executing the main loop.
Needed to resume inductions and reductions during epilogue vectorization.
Definition at line 682 of file LoopVectorize.cpp.
Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), and getAdditionalBypassBlock().
|
protected |
BFI and PSI are used to check for profile guided size optimizations.
Definition at line 663 of file LoopVectorize.cpp.
Referenced by InnerLoopVectorizer().
|
protected |
The builder that we use.
Definition at line 628 of file LoopVectorize.cpp.
Referenced by emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), llvm::LoopVectorizationPlanner::executePlan(), fixNonInductionPHIs(), and getOrCreateVectorTripCount().
|
protected |
The profitablity analysis.
Definition at line 657 of file LoopVectorize.cpp.
|
protected |
Dominator Tree.
Definition at line 603 of file LoopVectorize.cpp.
Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), and llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck().
Mapping of induction phis to their additional bypass values.
They need to be added as operands to phi nodes in the scalar loop preheader after the epilogue skeleton has been created.
Definition at line 677 of file LoopVectorize.cpp.
Referenced by createInductionAdditionalBypassValues(), and getInductionAdditionalBypassValue().
|
protected |
The legality analysis.
Definition at line 654 of file LoopVectorize.cpp.
Referenced by createInductionAdditionalBypassValues(), llvm::LoopVectorizationPlanner::executePlan(), and fixVectorizedLoop().
|
protected |
Loop Info.
Definition at line 600 of file LoopVectorize.cpp.
Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), fixVectorizedLoop(), and sinkScalarOperands().
|
protected |
A list of all bypass blocks. The first block is the entry of the loop.
Definition at line 642 of file LoopVectorize.cpp.
Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), and emitSCEVChecks().
|
protected |
Middle Block between the vector and the scalar.
Definition at line 639 of file LoopVectorize.cpp.
Referenced by createVectorLoopSkeleton(), and fixVectorizedLoop().
|
protected |
The scalar-loop preheader.
Definition at line 636 of file LoopVectorize.cpp.
Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createVectorizedLoopSkeleton(), and createVectorLoopSkeleton().
|
protected |
The vector-loop preheader.
Definition at line 633 of file LoopVectorize.cpp.
Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createVectorizedLoopSkeleton(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), emitSCEVChecks(), and llvm::LoopVectorizationPlanner::executePlan().
|
protected |
Definition at line 621 of file LoopVectorize.cpp.
Referenced by emitIterationCountCheck().
|
protected |
Definition at line 668 of file LoopVectorize.cpp.
Referenced by emitMemRuntimeChecks(), emitSCEVChecks(), and InnerLoopVectorizer().
|
protected |
Interface to emit optimization remarks.
Definition at line 615 of file LoopVectorize.cpp.
Referenced by emitMemRuntimeChecks().
|
protected |
The original loop.
Definition at line 592 of file LoopVectorize.cpp.
Referenced by createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), fixupIVUsers(), fixVectorizedLoop(), InnerLoopVectorizer(), llvm::EpilogueVectorizerMainLoop::printDebugTracesAtEnd(), and llvm::EpilogueVectorizerEpilogueLoop::printDebugTracesAtEnd().
|
protected |
Definition at line 684 of file LoopVectorize.cpp.
Referenced by createVectorLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), fixNonInductionPHIs(), fixupIVUsers(), and introduceCheckBlockInVPlan().
|
protected |
Store instructions that were predicated.
Definition at line 645 of file LoopVectorize.cpp.
Referenced by fixVectorizedLoop(), and scalarizeInstruction().
|
protected |
A wrapper around ScalarEvolution used to add runtime SCEV checks.
Applies dynamic knowledge to simplify SCEV expressions and converts them to a more usable form.
Definition at line 597 of file LoopVectorize.cpp.
Referenced by emitIterationCountCheck(), and fixVectorizedLoop().
|
protected |
Definition at line 664 of file LoopVectorize.cpp.
Referenced by InnerLoopVectorizer().
|
protected |
Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable.
Definition at line 672 of file LoopVectorize.cpp.
Referenced by emitMemRuntimeChecks(), and emitSCEVChecks().
|
protected |
Target Library Info.
Definition at line 606 of file LoopVectorize.cpp.
|
protected |
Trip count of the original loop.
Definition at line 648 of file LoopVectorize.cpp.
Referenced by llvm::EpilogueVectorizerEpilogueLoop::EpilogueVectorizerEpilogueLoop(), getTripCount(), and setTripCount().
|
protected |
Target Transform Info.
Definition at line 609 of file LoopVectorize.cpp.
|
protected |
The vectorization unroll factor to use.
Each scalar is vectorized to this many different vector instructions.
Definition at line 625 of file LoopVectorize.cpp.
Referenced by emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), fixVectorizedLoop(), and getOrCreateVectorTripCount().
|
protected |
The vector preheader block of Plan
, used as target for check blocks introduced during skeleton creation.
Definition at line 688 of file LoopVectorize.cpp.
Referenced by introduceCheckBlockInVPlan().
|
protected |
Trip count of the widened loop (TripCount - TripCount % (VF*UF))
Definition at line 651 of file LoopVectorize.cpp.
Referenced by getOrCreateVectorTripCount().
|
protected |
The vectorization SIMD factor to use.
Each vector will have this many vector elements.
Definition at line 619 of file LoopVectorize.cpp.
Referenced by createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), fixVectorizedLoop(), and getOrCreateVectorTripCount().