LLVM 20.0.0git
Public Member Functions | Protected Member Functions | Protected Attributes | Friends | List of all members
llvm::InnerLoopVectorizer Class Reference

InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF). More...

Inheritance diagram for llvm::InnerLoopVectorizer:
Inheritance graph
[legend]

Public Member Functions

 InnerLoopVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, ElementCount VecWidth, ElementCount MinProfitableTripCount, unsigned UnrollFactor, LoopVectorizationLegality *LVL, LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &RTChecks, VPlan &Plan)
 
virtual ~InnerLoopVectorizer ()=default
 
virtual BasicBlockcreateVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs)
 Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder.
 
void fixVectorizedLoop (VPTransformState &State)
 Fix the vectorized code, taking care of header phi's, and more.
 
bool areSafetyChecksAdded ()
 
void scalarizeInstruction (const Instruction *Instr, VPReplicateRecipe *RepRecipe, const VPLane &Lane, VPTransformState &State)
 A helper function to scalarize a single Instruction in the innermost loop.
 
void fixNonInductionPHIs (VPTransformState &State)
 Fix the non-induction PHIs in Plan.
 
ValuegetTripCount () const
 Returns the original loop trip count.
 
void setTripCount (Value *TC)
 Used to set the trip count after ILV's construction and after the preheader block has been executed.
 
ValuegetInductionAdditionalBypassValue (PHINode *OrigPhi) const
 induction header phi.
 
BasicBlockgetAdditionalBypassBlock () const
 Return the additional bypass block which targets the scalar loop by skipping the epilogue loop after completing the main loop.
 

Protected Member Functions

virtual void fixupIVUsers (PHINode *OrigPhi, const InductionDescriptor &II, Value *VectorTripCount, BasicBlock *MiddleBlock, VPTransformState &State)
 Set up the values of the IVs correctly when exiting the vector loop.
 
void sinkScalarOperands (Instruction *PredInst)
 Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it.
 
ValuegetOrCreateVectorTripCount (BasicBlock *InsertBlock)
 Returns (and creates if needed) the trip count of the widened loop.
 
void emitIterationCountCheck (BasicBlock *Bypass)
 Emit a bypass check to see if the vector trip count is zero, including if it overflows.
 
BasicBlockemitSCEVChecks (BasicBlock *Bypass)
 Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct.
 
BasicBlockemitMemRuntimeChecks (BasicBlock *Bypass)
 Emit bypass checks to check any memory assumptions we may have made.
 
void createVectorLoopSkeleton (StringRef Prefix)
 Emit basic blocks (prefixed with Prefix) for the iteration check, vector loop preheader, middle block and scalar preheader.
 
void createInductionAdditionalBypassValues (const SCEV2ValueTy &ExpandedSCEVs, Value *MainVectorTripCount)
 Create and record the values for induction variables to resume coming from the additional bypass block.
 
virtual void printDebugTracesAtStart ()
 Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.
 
virtual void printDebugTracesAtEnd ()
 
void introduceCheckBlockInVPlan (BasicBlock *CheckIRBB)
 Introduces a new VPIRBasicBlock for CheckIRBB to Plan between the vector preheader and its predecessor, also connecting the new block to the scalar preheader.
 

Protected Attributes

LoopOrigLoop
 The original loop.
 
PredicatedScalarEvolutionPSE
 A wrapper around ScalarEvolution used to add runtime SCEV checks.
 
LoopInfoLI
 Loop Info.
 
DominatorTreeDT
 Dominator Tree.
 
const TargetLibraryInfoTLI
 Target Library Info.
 
const TargetTransformInfoTTI
 Target Transform Info.
 
AssumptionCacheAC
 Assumption Cache.
 
OptimizationRemarkEmitterORE
 Interface to emit optimization remarks.
 
ElementCount VF
 The vectorization SIMD factor to use.
 
ElementCount MinProfitableTripCount
 
unsigned UF
 The vectorization unroll factor to use.
 
IRBuilder Builder
 The builder that we use.
 
BasicBlockLoopVectorPreHeader
 The vector-loop preheader.
 
BasicBlockLoopScalarPreHeader
 The scalar-loop preheader.
 
BasicBlockLoopMiddleBlock
 Middle Block between the vector and the scalar.
 
SmallVector< BasicBlock *, 4 > LoopBypassBlocks
 A list of all bypass blocks. The first block is the entry of the loop.
 
SmallVector< Instruction *, 4 > PredicatedInstructions
 Store instructions that were predicated.
 
ValueTripCount = nullptr
 Trip count of the original loop.
 
ValueVectorTripCount = nullptr
 Trip count of the widened loop (TripCount - TripCount % (VF*UF))
 
LoopVectorizationLegalityLegal
 The legality analysis.
 
LoopVectorizationCostModelCost
 The profitablity analysis.
 
bool AddedSafetyChecks = false
 
BlockFrequencyInfoBFI
 BFI and PSI are used to check for profile guided size optimizations.
 
ProfileSummaryInfoPSI
 
bool OptForSizeBasedOnProfile
 
GeneratedRTChecks & RTChecks
 Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable.
 
DenseMap< PHINode *, Value * > Induction2AdditionalBypassValue
 Mapping of induction phis to their additional bypass values.
 
BasicBlockAdditionalBypassBlock = nullptr
 The additional bypass block which conditionally skips over the epilogue loop after executing the main loop.
 
VPlanPlan
 
VPBlockBaseVectorPHVPB
 The vector preheader block of Plan, used as target for check blocks introduced during skeleton creation.
 

Friends

class LoopVectorizationPlanner
 

Detailed Description

InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF).

This class performs the widening of scalars into vectors, or multiple scalars. This class also implements the following features:

Definition at line 466 of file LoopVectorize.cpp.

Constructor & Destructor Documentation

◆ InnerLoopVectorizer()

llvm::InnerLoopVectorizer::InnerLoopVectorizer ( Loop OrigLoop,
PredicatedScalarEvolution PSE,
LoopInfo LI,
DominatorTree DT,
const TargetLibraryInfo TLI,
const TargetTransformInfo TTI,
AssumptionCache AC,
OptimizationRemarkEmitter ORE,
ElementCount  VecWidth,
ElementCount  MinProfitableTripCount,
unsigned  UnrollFactor,
LoopVectorizationLegality LVL,
LoopVectorizationCostModel CM,
BlockFrequencyInfo BFI,
ProfileSummaryInfo PSI,
GeneratedRTChecks &  RTChecks,
VPlan Plan 
)
inline

◆ ~InnerLoopVectorizer()

virtual llvm::InnerLoopVectorizer::~InnerLoopVectorizer ( )
virtualdefault

Member Function Documentation

◆ areSafetyChecksAdded()

bool llvm::InnerLoopVectorizer::areSafetyChecksAdded ( )
inline

Definition at line 507 of file LoopVectorize.cpp.

References AddedSafetyChecks.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ createInductionAdditionalBypassValues()

void InnerLoopVectorizer::createInductionAdditionalBypassValues ( const SCEV2ValueTy &  ExpandedSCEVs,
Value MainVectorTripCount 
)
protected

◆ createVectorizedLoopSkeleton()

BasicBlock * InnerLoopVectorizer::createVectorizedLoopSkeleton ( const SCEV2ValueTy &  ExpandedSCEVs)
virtual

Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder.

Control flow is generated around the vectorized (and scalar epilogue) loops consisting of various checks and bypasses. Return the pre-header block of the new loop. In the case of epilogue vectorization, this function is overriden to handle the more complex control flow around the loops. ExpandedSCEVs is used to look up SCEV expansions for expressions needed during skeleton creation.

Reimplemented in llvm::InnerLoopAndEpilogueVectorizer.

Definition at line 2718 of file LoopVectorize.cpp.

References createVectorLoopSkeleton(), emitIterationCountCheck(), emitMemRuntimeChecks(), emitSCEVChecks(), LoopScalarPreHeader, and LoopVectorPreHeader.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ createVectorLoopSkeleton()

void InnerLoopVectorizer::createVectorLoopSkeleton ( StringRef  Prefix)
protected

◆ emitIterationCountCheck()

void InnerLoopVectorizer::emitIterationCountCheck ( BasicBlock Bypass)
protected

Emit a bypass check to see if the vector trip count is zero, including if it overflows.

Definition at line 2472 of file LoopVectorize.cpp.

References llvm::ScalarEvolution::applyLoopGuards(), assert(), Builder, llvm::BranchInst::Create(), llvm::IRBuilderBase::CreateBinaryIntrinsic(), llvm::IRBuilderBase::CreateICmp(), llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), llvm::DataAndControlFlowWithoutRuntimeCheck, DT, llvm::IRBuilderBase::getFalse(), llvm::DomTreeNodeBase< NodeT >::getIDom(), llvm::CmpInst::getInversePredicate(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::DominatorTreeBase< NodeT, IsPostDom >::getNode(), llvm::ScalarEvolution::getSCEV(), llvm::PredicatedScalarEvolution::getSE(), llvm::BasicBlock::getTerminator(), getTripCount(), llvm::IRBuilderBase::getTrue(), llvm::Value::getType(), llvm::hasBranchWeightMD(), llvm::CmpInst::ICMP_ULE, llvm::CmpInst::ICMP_ULT, introduceCheckBlockInVPlan(), isIndvarOverflowCheckKnownFalse(), llvm::ScalarEvolution::isKnownPredicate(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isVector(), LHS, LI, LoopBypassBlocks, LoopVectorPreHeader, MinItersBypassWeights, MinProfitableTripCount, llvm::None, OrigLoop, P, llvm::DominatorTreeBase< NodeT, IsPostDom >::properlyDominates(), PSE, llvm::ReplaceInstWithInst(), llvm::setBranchWeights(), llvm::SplitBlock(), UF, and VF.

Referenced by createVectorizedLoopSkeleton().

◆ emitMemRuntimeChecks()

BasicBlock * InnerLoopVectorizer::emitMemRuntimeChecks ( BasicBlock Bypass)
protected

◆ emitSCEVChecks()

BasicBlock * InnerLoopVectorizer::emitSCEVChecks ( BasicBlock Bypass)
protected

Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct.

Returns the block containing the checks or nullptr if no checks have been added.

Definition at line 2557 of file LoopVectorize.cpp.

References AddedSafetyChecks, assert(), llvm::LoopVectorizeHints::FK_Enabled, llvm::BasicBlock::getParent(), llvm::Function::hasOptSize(), introduceCheckBlockInVPlan(), LoopBypassBlocks, LoopVectorPreHeader, OptForSizeBasedOnProfile, and RTChecks.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().

◆ fixNonInductionPHIs()

void InnerLoopVectorizer::fixNonInductionPHIs ( VPTransformState State)

◆ fixupIVUsers()

void InnerLoopVectorizer::fixupIVUsers ( PHINode OrigPhi,
const InductionDescriptor II,
Value VectorTripCount,
BasicBlock MiddleBlock,
VPTransformState State 
)
protectedvirtual

◆ fixVectorizedLoop()

void InnerLoopVectorizer::fixVectorizedLoop ( VPTransformState State)

◆ getAdditionalBypassBlock()

BasicBlock * llvm::InnerLoopVectorizer::getAdditionalBypassBlock ( ) const
inline

Return the additional bypass block which targets the scalar loop by skipping the epilogue loop after completing the main loop.

Definition at line 537 of file LoopVectorize.cpp.

References AdditionalBypassBlock, and assert().

Referenced by createInductionAdditionalBypassValues(), and llvm::LoopVectorizationPlanner::executePlan().

◆ getInductionAdditionalBypassValue()

Value * llvm::InnerLoopVectorizer::getInductionAdditionalBypassValue ( PHINode OrigPhi) const
inline

induction header phi.

Definition at line 531 of file LoopVectorize.cpp.

References Induction2AdditionalBypassValue.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ getOrCreateVectorTripCount()

Value * InnerLoopVectorizer::getOrCreateVectorTripCount ( BasicBlock InsertBlock)
protected

◆ getTripCount()

Value * llvm::InnerLoopVectorizer::getTripCount ( ) const
inline

◆ introduceCheckBlockInVPlan()

void InnerLoopVectorizer::introduceCheckBlockInVPlan ( BasicBlock CheckIRBB)
protected

◆ printDebugTracesAtEnd()

virtual void llvm::InnerLoopVectorizer::printDebugTracesAtEnd ( )
inlineprotectedvirtual

◆ printDebugTracesAtStart()

virtual void llvm::InnerLoopVectorizer::printDebugTracesAtStart ( )
inlineprotectedvirtual

Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.

Reimplemented in llvm::EpilogueVectorizerMainLoop, and llvm::EpilogueVectorizerEpilogueLoop.

Definition at line 583 of file LoopVectorize.cpp.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ scalarizeInstruction()

void InnerLoopVectorizer::scalarizeInstruction ( const Instruction Instr,
VPReplicateRecipe RepRecipe,
const VPLane Lane,
VPTransformState State 
)

◆ setTripCount()

void llvm::InnerLoopVectorizer::setTripCount ( Value TC)
inline

Used to set the trip count after ILV's construction and after the preheader block has been executed.

Note that this always holds the trip count of the original loop for both main loop and epilogue vectorization.

Definition at line 527 of file LoopVectorize.cpp.

References TripCount.

Referenced by llvm::LoopVectorizationPlanner::executePlan(), and llvm::LoopVectorizePass::processLoop().

◆ sinkScalarOperands()

void InnerLoopVectorizer::sinkScalarOperands ( Instruction PredInst)
protected

Friends And Related Function Documentation

◆ LoopVectorizationPlanner

friend class LoopVectorizationPlanner
friend

Definition at line 544 of file LoopVectorize.cpp.

Member Data Documentation

◆ AC

AssumptionCache* llvm::InnerLoopVectorizer::AC
protected

Assumption Cache.

Definition at line 612 of file LoopVectorize.cpp.

Referenced by scalarizeInstruction().

◆ AddedSafetyChecks

bool llvm::InnerLoopVectorizer::AddedSafetyChecks = false
protected

Definition at line 660 of file LoopVectorize.cpp.

Referenced by areSafetyChecksAdded(), emitMemRuntimeChecks(), and emitSCEVChecks().

◆ AdditionalBypassBlock

BasicBlock* llvm::InnerLoopVectorizer::AdditionalBypassBlock = nullptr
protected

The additional bypass block which conditionally skips over the epilogue loop after executing the main loop.

Needed to resume inductions and reductions during epilogue vectorization.

Definition at line 682 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), and getAdditionalBypassBlock().

◆ BFI

BlockFrequencyInfo* llvm::InnerLoopVectorizer::BFI
protected

BFI and PSI are used to check for profile guided size optimizations.

Definition at line 663 of file LoopVectorize.cpp.

Referenced by InnerLoopVectorizer().

◆ Builder

IRBuilder llvm::InnerLoopVectorizer::Builder
protected

◆ Cost

LoopVectorizationCostModel* llvm::InnerLoopVectorizer::Cost
protected

The profitablity analysis.

Definition at line 657 of file LoopVectorize.cpp.

◆ DT

DominatorTree* llvm::InnerLoopVectorizer::DT
protected

◆ Induction2AdditionalBypassValue

DenseMap<PHINode *, Value *> llvm::InnerLoopVectorizer::Induction2AdditionalBypassValue
protected

Mapping of induction phis to their additional bypass values.

They need to be added as operands to phi nodes in the scalar loop preheader after the epilogue skeleton has been created.

Definition at line 677 of file LoopVectorize.cpp.

Referenced by createInductionAdditionalBypassValues(), and getInductionAdditionalBypassValue().

◆ Legal

LoopVectorizationLegality* llvm::InnerLoopVectorizer::Legal
protected

◆ LI

LoopInfo* llvm::InnerLoopVectorizer::LI
protected

◆ LoopBypassBlocks

SmallVector<BasicBlock *, 4> llvm::InnerLoopVectorizer::LoopBypassBlocks
protected

◆ LoopMiddleBlock

BasicBlock* llvm::InnerLoopVectorizer::LoopMiddleBlock
protected

Middle Block between the vector and the scalar.

Definition at line 639 of file LoopVectorize.cpp.

Referenced by createVectorLoopSkeleton(), and fixVectorizedLoop().

◆ LoopScalarPreHeader

BasicBlock* llvm::InnerLoopVectorizer::LoopScalarPreHeader
protected

◆ LoopVectorPreHeader

BasicBlock* llvm::InnerLoopVectorizer::LoopVectorPreHeader
protected

◆ MinProfitableTripCount

ElementCount llvm::InnerLoopVectorizer::MinProfitableTripCount
protected

Definition at line 621 of file LoopVectorize.cpp.

Referenced by emitIterationCountCheck().

◆ OptForSizeBasedOnProfile

bool llvm::InnerLoopVectorizer::OptForSizeBasedOnProfile
protected

Definition at line 668 of file LoopVectorize.cpp.

Referenced by emitMemRuntimeChecks(), emitSCEVChecks(), and InnerLoopVectorizer().

◆ ORE

OptimizationRemarkEmitter* llvm::InnerLoopVectorizer::ORE
protected

Interface to emit optimization remarks.

Definition at line 615 of file LoopVectorize.cpp.

Referenced by emitMemRuntimeChecks().

◆ OrigLoop

Loop* llvm::InnerLoopVectorizer::OrigLoop
protected

◆ Plan

VPlan& llvm::InnerLoopVectorizer::Plan
protected

◆ PredicatedInstructions

SmallVector<Instruction *, 4> llvm::InnerLoopVectorizer::PredicatedInstructions
protected

Store instructions that were predicated.

Definition at line 645 of file LoopVectorize.cpp.

Referenced by fixVectorizedLoop(), and scalarizeInstruction().

◆ PSE

PredicatedScalarEvolution& llvm::InnerLoopVectorizer::PSE
protected

A wrapper around ScalarEvolution used to add runtime SCEV checks.

Applies dynamic knowledge to simplify SCEV expressions and converts them to a more usable form.

Definition at line 597 of file LoopVectorize.cpp.

Referenced by emitIterationCountCheck(), and fixVectorizedLoop().

◆ PSI

ProfileSummaryInfo* llvm::InnerLoopVectorizer::PSI
protected

Definition at line 664 of file LoopVectorize.cpp.

Referenced by InnerLoopVectorizer().

◆ RTChecks

GeneratedRTChecks& llvm::InnerLoopVectorizer::RTChecks
protected

Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable.

Definition at line 672 of file LoopVectorize.cpp.

Referenced by emitMemRuntimeChecks(), and emitSCEVChecks().

◆ TLI

const TargetLibraryInfo* llvm::InnerLoopVectorizer::TLI
protected

Target Library Info.

Definition at line 606 of file LoopVectorize.cpp.

◆ TripCount

Value* llvm::InnerLoopVectorizer::TripCount = nullptr
protected

Trip count of the original loop.

Definition at line 648 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::EpilogueVectorizerEpilogueLoop(), getTripCount(), and setTripCount().

◆ TTI

const TargetTransformInfo* llvm::InnerLoopVectorizer::TTI
protected

Target Transform Info.

Definition at line 609 of file LoopVectorize.cpp.

◆ UF

unsigned llvm::InnerLoopVectorizer::UF
protected

The vectorization unroll factor to use.

Each scalar is vectorized to this many different vector instructions.

Definition at line 625 of file LoopVectorize.cpp.

Referenced by emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), fixVectorizedLoop(), and getOrCreateVectorTripCount().

◆ VectorPHVPB

VPBlockBase* llvm::InnerLoopVectorizer::VectorPHVPB
protected

The vector preheader block of Plan, used as target for check blocks introduced during skeleton creation.

Definition at line 688 of file LoopVectorize.cpp.

Referenced by introduceCheckBlockInVPlan().

◆ VectorTripCount

Value* llvm::InnerLoopVectorizer::VectorTripCount = nullptr
protected

Trip count of the widened loop (TripCount - TripCount % (VF*UF))

Definition at line 651 of file LoopVectorize.cpp.

Referenced by getOrCreateVectorTripCount().

◆ VF

ElementCount llvm::InnerLoopVectorizer::VF
protected

The documentation for this class was generated from the following file: