A specialized derived class of inner loop vectorizer that performs vectorization of main loops in the process of vectorizing loops and their epilogues. More...

Inheritance diagram for llvm::EpilogueVectorizerMainLoop:

[legend]

Public Member Functions
	EpilogueVectorizerMainLoop (Loop OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo LI, DominatorTree DT, const TargetTransformInfo TTI, AssumptionCache AC, EpilogueLoopVectorizationInfo &EPI, LoopVectorizationCostModel CM, BlockFrequencyInfo BFI, ProfileSummaryInfo PSI, GeneratedRTChecks &Check, VPlan &Plan)
BasicBlock *	createVectorizedLoopSkeleton () final
	Implements the interface for creating a vectorized skeleton using the main loop strategy (i.e., the first pass of VPlan execution).
Public Member Functions inherited from llvm::InnerLoopAndEpilogueVectorizer
	InnerLoopAndEpilogueVectorizer (Loop OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo LI, DominatorTree DT, const TargetTransformInfo TTI, AssumptionCache AC, EpilogueLoopVectorizationInfo &EPI, LoopVectorizationCostModel CM, BlockFrequencyInfo BFI, ProfileSummaryInfo PSI, GeneratedRTChecks &Checks, VPlan &Plan, ElementCount VecWidth, ElementCount MinProfitableTripCount, unsigned UnrollFactor)
Public Member Functions inherited from llvm::InnerLoopVectorizer
	InnerLoopVectorizer (Loop OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo LI, DominatorTree DT, const TargetTransformInfo TTI, AssumptionCache AC, ElementCount VecWidth, unsigned UnrollFactor, LoopVectorizationCostModel CM, BlockFrequencyInfo BFI, ProfileSummaryInfo PSI, GeneratedRTChecks &RTChecks, VPlan &Plan)
virtual	~InnerLoopVectorizer ()=default
void	fixVectorizedLoop (VPTransformState &State)
	Fix the vectorized code, taking care of header phi's, and more.
void	fixNonInductionPHIs (VPTransformState &State)
	Fix the non-induction PHIs in `Plan`.
Value *	getTripCount () const
	Returns the original loop trip count.
void	setTripCount (Value *TC)
	Used to set the trip count after ILV's construction and after the preheader block has been executed.

Protected Member Functions
void	introduceCheckBlockInVPlan (BasicBlock *CheckIRBB)
	Introduces a new VPIRBasicBlock for `CheckIRBB` to Plan between the vector preheader and its predecessor, also connecting the new block to the scalar preheader.
Value *	createIterationCountCheck (BasicBlock *VectorPH, ElementCount VF, unsigned UF) const
BasicBlock *	emitIterationCountCheck (BasicBlock VectorPH, BasicBlock Bypass, bool ForEpilogue)
	Emits an iteration count bypass check once for the main loop (when `ForEpilogue` is false) and once for the epilogue loop (when `ForEpilogue` is true).
void	printDebugTracesAtStart () override
	Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.
void	printDebugTracesAtEnd () override
Protected Member Functions inherited from llvm::InnerLoopVectorizer
BasicBlock *	createScalarPreheader (StringRef Prefix)
	Create and return a new IR basic block for the scalar preheader whose name is prefixed with `Prefix`.

Additional Inherited Members
Public Attributes inherited from llvm::InnerLoopAndEpilogueVectorizer
EpilogueLoopVectorizationInfo &	EPI
	Holds and updates state information required to vectorize the main loop and its epilogue in two separate passes.
Protected Attributes inherited from llvm::InnerLoopAndEpilogueVectorizer
ElementCount	MinProfitableTripCount
Protected Attributes inherited from llvm::InnerLoopVectorizer
Loop *	OrigLoop
	The original loop.
PredicatedScalarEvolution &	PSE
	A wrapper around ScalarEvolution used to add runtime SCEV checks.
LoopInfo *	LI
	Loop Info.
DominatorTree *	DT
	Dominator Tree.
const TargetTransformInfo *	TTI
	Target Transform Info.
AssumptionCache *	AC
	Assumption Cache.
ElementCount	VF
	The vectorization SIMD factor to use.
unsigned	UF
	The vectorization unroll factor to use.
IRBuilder	Builder
	The builder that we use.
Value *	TripCount = nullptr
	Trip count of the original loop.
LoopVectorizationCostModel *	Cost
	The profitablity analysis.
BlockFrequencyInfo *	BFI
	BFI and PSI are used to check for profile guided size optimizations.
ProfileSummaryInfo *	PSI
GeneratedRTChecks &	RTChecks
	Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable.
VPlan &	Plan
VPBasicBlock *	VectorPHVPBB
	The vector preheader block of `Plan`, used as target for check blocks introduced during skeleton creation.

Detailed Description

A specialized derived class of inner loop vectorizer that performs vectorization of main loops in the process of vectorizing loops and their epilogues.

Definition at line 651 of file LoopVectorize.cpp.

Constructor & Destructor Documentation

◆ EpilogueVectorizerMainLoop()

llvm::EpilogueVectorizerMainLoop::EpilogueVectorizerMainLoop	(	Loop *	OrigLoop,
		PredicatedScalarEvolution &	PSE,
		LoopInfo *	LI,
		DominatorTree *	DT,
		const TargetTransformInfo *	TTI,
		AssumptionCache *	AC,
		EpilogueLoopVectorizationInfo &	EPI,
		LoopVectorizationCostModel *	CM,
		BlockFrequencyInfo *	BFI,
		ProfileSummaryInfo *	PSI,
		GeneratedRTChecks &	Check,
		VPlan &	Plan )

inline

Definition at line 653 of file LoopVectorize.cpp.

References llvm::InnerLoopVectorizer::AC, llvm::InnerLoopVectorizer::BFI, llvm::InnerLoopVectorizer::DT, llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::InnerLoopAndEpilogueVectorizer::InnerLoopAndEpilogueVectorizer(), llvm::InnerLoopVectorizer::LI, llvm::InnerLoopVectorizer::OrigLoop, llvm::InnerLoopVectorizer::Plan, llvm::InnerLoopVectorizer::PSE, and llvm::InnerLoopVectorizer::PSI.

Member Function Documentation

◆ createIterationCountCheck()

Value * EpilogueVectorizerMainLoop::createIterationCountCheck	(	BasicBlock *	VectorPH,
		ElementCount	VF,
		unsigned	UF ) const

protected

Definition at line 2274 of file LoopVectorize.cpp.

References llvm::ScalarEvolution::applyLoopGuards(), llvm::InnerLoopVectorizer::Builder, llvm::cast(), llvm::Count, llvm::createStepForVF(), llvm::DataAndControlFlowWithoutRuntimeCheck, llvm::BasicBlock::getContext(), llvm::BasicBlock::getDataLayout(), llvm::CmpInst::getInversePredicate(), getMask(), llvm::ScalarEvolution::getSCEV(), llvm::BasicBlock::getTerminator(), llvm::InnerLoopVectorizer::getTripCount(), llvm::CmpInst::ICMP_ULE, llvm::CmpInst::ICMP_ULT, isIndvarOverflowCheckKnownFalse(), llvm::ScalarEvolution::isKnownPredicate(), llvm::InnerLoopAndEpilogueVectorizer::MinProfitableTripCount, llvm::None, llvm::InnerLoopVectorizer::OrigLoop, P, llvm::InnerLoopVectorizer::PSE, llvm::InnerLoopVectorizer::UF, and llvm::InnerLoopVectorizer::VF.

Referenced by emitIterationCountCheck().

◆ createVectorizedLoopSkeleton()

BasicBlock * EpilogueVectorizerMainLoop::createVectorizedLoopSkeleton ( )

finalvirtual

Implements the interface for creating a vectorized skeleton using the main loop strategy (i.e., the first pass of VPlan execution).

This function is partially responsible for generating the control flow depicted in https://llvm.org/docs/Vectorizers.html#epilogue-vectorization.

Reimplemented from llvm::InnerLoopVectorizer.

Definition at line 7277 of file LoopVectorize.cpp.

References llvm::cast(), llvm::InnerLoopVectorizer::createScalarPreheader(), emitIterationCountCheck(), llvm::InnerLoopAndEpilogueVectorizer::EPI, and llvm::BasicBlock::getSinglePredecessor().

◆ emitIterationCountCheck()

BasicBlock * EpilogueVectorizerMainLoop::emitIterationCountCheck	(	BasicBlock *	VectorPH,
		BasicBlock *	Bypass,
		bool	ForEpilogue )

protected

Emits an iteration count bypass check once for the main loop (when ForEpilogue is false) and once for the epilogue loop (when ForEpilogue is true).

Definition at line 7319 of file LoopVectorize.cpp.

References assert(), llvm::Count, llvm::BranchInst::Create(), createIterationCountCheck(), llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::ElementCount::getFixed(), llvm::BasicBlock::getTerminator(), llvm::InnerLoopVectorizer::getTripCount(), llvm::hasBranchWeightMD(), introduceCheckBlockInVPlan(), llvm::InnerLoopVectorizer::LI, MinItersBypassWeights, llvm::InnerLoopAndEpilogueVectorizer::MinProfitableTripCount, llvm::InnerLoopVectorizer::OrigLoop, llvm::ReplaceInstWithInst(), replaceVPBBWithIRVPBB(), llvm::setBranchWeights(), llvm::Value::setName(), llvm::SplitBlock(), and llvm::InnerLoopVectorizer::VectorPHVPBB.

Referenced by createVectorizedLoopSkeleton().

◆ introduceCheckBlockInVPlan()

void EpilogueVectorizerMainLoop::introduceCheckBlockInVPlan ( BasicBlock * CheckIRBB )

protected

Introduces a new VPIRBasicBlock for CheckIRBB to Plan between the vector preheader and its predecessor, also connecting the new block to the scalar preheader.

Definition at line 2249 of file LoopVectorize.cpp.

References assert(), llvm::cast(), llvm::VPBlockUtils::connectBlocks(), llvm::VPBlockBase::getNumPredecessors(), llvm::VPBlockBase::getNumSuccessors(), llvm::VPBlockBase::getSuccessors(), llvm::VPBlockUtils::insertOnEdge(), llvm::isa(), llvm::InnerLoopVectorizer::Plan, llvm::VPBlockBase::swapSuccessors(), and llvm::InnerLoopVectorizer::VectorPHVPBB.