A specialized derived class of inner loop vectorizer that performs vectorization of main loops in the process of vectorizing loops and their epilogues. More...

Inheritance diagram for llvm::EpilogueVectorizerMainLoop:

[legend]

Public Member Functions
	EpilogueVectorizerMainLoop (Loop OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo LI, DominatorTree DT, const TargetLibraryInfo TLI, const TargetTransformInfo TTI, AssumptionCache AC, OptimizationRemarkEmitter ORE, EpilogueLoopVectorizationInfo &EPI, LoopVectorizationLegality LVL, llvm::LoopVectorizationCostModel CM, BlockFrequencyInfo BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &Check, VPlan &Plan)

BasicBlock *	createEpilogueVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs) final
	Implements the interface for creating a vectorized skeleton using the main loop strategy (ie the first pass of vplan execution).

Public Member Functions inherited from llvm::InnerLoopAndEpilogueVectorizer
	InnerLoopAndEpilogueVectorizer (Loop OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo LI, DominatorTree DT, const TargetLibraryInfo TLI, const TargetTransformInfo TTI, AssumptionCache AC, OptimizationRemarkEmitter ORE, EpilogueLoopVectorizationInfo &EPI, LoopVectorizationLegality LVL, llvm::LoopVectorizationCostModel CM, BlockFrequencyInfo BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &Checks, VPlan &Plan)

BasicBlock *	createVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs) final
	Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder.

virtual BasicBlock *	createEpilogueVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs)=0
	The interface for creating a vectorized skeleton using one of two different strategies, each corresponding to one execution of the vplan as described above.

Public Member Functions inherited from llvm::InnerLoopVectorizer
	InnerLoopVectorizer (Loop OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo LI, DominatorTree DT, const TargetLibraryInfo TLI, const TargetTransformInfo TTI, AssumptionCache AC, OptimizationRemarkEmitter ORE, ElementCount VecWidth, ElementCount MinProfitableTripCount, unsigned UnrollFactor, LoopVectorizationLegality LVL, LoopVectorizationCostModel CM, BlockFrequencyInfo BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &RTChecks, VPlan &Plan)

virtual	~InnerLoopVectorizer ()=default

virtual BasicBlock *	createVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs)
	Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder.

void	fixVectorizedLoop (VPTransformState &State)
	Fix the vectorized code, taking care of header phi's, and more.

bool	areSafetyChecksAdded ()

void	scalarizeInstruction (const Instruction Instr, VPReplicateRecipe RepRecipe, const VPLane &Lane, VPTransformState &State)
	A helper function to scalarize a single Instruction in the innermost loop.

void	fixNonInductionPHIs (VPTransformState &State)
	Fix the non-induction PHIs in `Plan`.

void	createInductionResumeVPValue (VPIRInstruction InductionPhiIRI, const InductionDescriptor &ID, Value Step, ArrayRef< BasicBlock * > BypassBlocks, VPBuilder &ScalarPHBuilder, Value *MainVectorTripCount=nullptr)
	Create a ResumePHI VPInstruction for the induction `InductionPhiIRI` to resume iteration count in the scalar epilogue from where the vectorized loop left off, and add it to the scalar preheader of VPlan.

Value *	getTripCount () const
	Returns the original loop trip count.

void	setTripCount (Value *TC)
	Used to set the trip count after ILV's construction and after the preheader block has been executed.

Value *	getInductionAdditionalBypassValue (PHINode *OrigPhi) const
	induction header phi.

BasicBlock *	getAdditionalBypassBlock () const
	Return the additional bypass block which targets the scalar loop by skipping the epilogue loop after completing the main loop.

Protected Member Functions
BasicBlock *	emitIterationCountCheck (BasicBlock *Bypass, bool ForEpilogue)
	Emits an iteration count bypass check once for the main loop (when `ForEpilogue` is false) and once for the epilogue loop (when `ForEpilogue` is true).

void	printDebugTracesAtStart () override
	Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.

void	printDebugTracesAtEnd () override

void	fixupIVUsers (PHINode OrigPhi, const InductionDescriptor &II, Value VectorTripCount, BasicBlock *MiddleBlock, VPTransformState &State) override
	Set up the values of the IVs correctly when exiting the vector loop.

Protected Member Functions inherited from llvm::InnerLoopVectorizer
virtual void	fixupIVUsers (PHINode OrigPhi, const InductionDescriptor &II, Value VectorTripCount, BasicBlock *MiddleBlock, VPTransformState &State)
	Set up the values of the IVs correctly when exiting the vector loop.

void	sinkScalarOperands (Instruction *PredInst)
	Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it.

Value *	getOrCreateVectorTripCount (BasicBlock *InsertBlock)
	Returns (and creates if needed) the trip count of the widened loop.

void	emitIterationCountCheck (BasicBlock *Bypass)
	Emit a bypass check to see if the vector trip count is zero, including if it overflows.

BasicBlock *	emitSCEVChecks (BasicBlock *Bypass)
	Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct.

BasicBlock *	emitMemRuntimeChecks (BasicBlock *Bypass)
	Emit bypass checks to check any memory assumptions we may have made.

void	createVectorLoopSkeleton (StringRef Prefix)
	Emit basic blocks (prefixed with `Prefix`) for the iteration check, vector loop preheader, middle block and scalar preheader.

void	createInductionResumeVPValues (const SCEV2ValueTy &ExpandedSCEVs, Value MainVectorTripCount=nullptr, SmallPtrSetImpl< PHINode > *IVSubset=nullptr)
	Create new phi nodes for the induction variables to resume iteration count in the scalar epilogue, from where the vectorized loop left off.

virtual void	printDebugTracesAtStart ()
	Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.

virtual void	printDebugTracesAtEnd ()

Additional Inherited Members
Public Attributes inherited from llvm::InnerLoopAndEpilogueVectorizer
EpilogueLoopVectorizationInfo &	EPI
	Holds and updates state information required to vectorize the main loop and its epilogue in two separate passes.

Protected Attributes inherited from llvm::InnerLoopVectorizer
Loop *	OrigLoop
	The original loop.

PredicatedScalarEvolution &	PSE
	A wrapper around ScalarEvolution used to add runtime SCEV checks.

LoopInfo *	LI
	Loop Info.

DominatorTree *	DT
	Dominator Tree.

const TargetLibraryInfo *	TLI
	Target Library Info.

const TargetTransformInfo *	TTI
	Target Transform Info.

AssumptionCache *	AC
	Assumption Cache.

OptimizationRemarkEmitter *	ORE
	Interface to emit optimization remarks.

ElementCount	VF
	The vectorization SIMD factor to use.

ElementCount	MinProfitableTripCount

unsigned	UF
	The vectorization unroll factor to use.

IRBuilder	Builder
	The builder that we use.

BasicBlock *	LoopVectorPreHeader
	The vector-loop preheader.

BasicBlock *	LoopScalarPreHeader
	The scalar-loop preheader.

BasicBlock *	LoopMiddleBlock
	Middle Block between the vector and the scalar.

SmallVector< BasicBlock *, 4 >	LoopBypassBlocks
	A list of all bypass blocks. The first block is the entry of the loop.

SmallVector< Instruction *, 4 >	PredicatedInstructions
	Store instructions that were predicated.

Value *	TripCount = nullptr
	Trip count of the original loop.

Value *	VectorTripCount = nullptr
	Trip count of the widened loop (TripCount - TripCount % (VF*UF))

LoopVectorizationLegality *	Legal
	The legality analysis.

LoopVectorizationCostModel *	Cost
	The profitablity analysis.

bool	AddedSafetyChecks = false

BlockFrequencyInfo *	BFI
	BFI and PSI are used to check for profile guided size optimizations.

ProfileSummaryInfo *	PSI

bool	OptForSizeBasedOnProfile

GeneratedRTChecks &	RTChecks
	Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable.

DenseMap< PHINode , Value >	Induction2AdditionalBypassValue
	Mapping of induction phis to their additional bypass values.

BasicBlock *	AdditionalBypassBlock = nullptr
	The additional bypass block which conditionally skips over the epilogue loop after executing the main loop.

VPlan &	Plan

Detailed Description

A specialized derived class of inner loop vectorizer that performs vectorization of main loops in the process of vectorizing loops and their epilogues.

Definition at line 777 of file LoopVectorize.cpp.

Constructor & Destructor Documentation

◆ EpilogueVectorizerMainLoop()

llvm::EpilogueVectorizerMainLoop::EpilogueVectorizerMainLoop	(	Loop *	OrigLoop,
		PredicatedScalarEvolution &	PSE,
		LoopInfo *	LI,
		DominatorTree *	DT,
		const TargetLibraryInfo *	TLI,
		const TargetTransformInfo *	TTI,
		AssumptionCache *	AC,
		OptimizationRemarkEmitter *	ORE,
		EpilogueLoopVectorizationInfo &	EPI,
		LoopVectorizationLegality *	LVL,
		llvm::LoopVectorizationCostModel *	CM,
		BlockFrequencyInfo *	BFI,
		ProfileSummaryInfo *	PSI,
		GeneratedRTChecks &	Check,
		VPlan &	Plan
	)

inline

Definition at line 779 of file LoopVectorize.cpp.

References Check.

Member Function Documentation

◆ createEpilogueVectorizedLoopSkeleton()

BasicBlock * EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton ( const SCEV2ValueTy & ExpandedSCEVs )

finalvirtual

Implements the interface for creating a vectorized skeleton using the main loop strategy (ie the first pass of vplan execution).

This function is partially responsible for generating the control flow depicted in https://llvm.org/docs/Vectorizers.html#epilogue-vectorization.

Implements llvm::InnerLoopAndEpilogueVectorizer.

Definition at line 7940 of file LoopVectorize.cpp.

References llvm::InnerLoopVectorizer::createInductionResumeVPValues(), llvm::InnerLoopVectorizer::createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::InnerLoopVectorizer::emitMemRuntimeChecks(), llvm::InnerLoopVectorizer::emitSCEVChecks(), llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::EpilogueLoopVectorizationInfo::EpilogueIterationCountCheck, llvm::EpilogueLoopVectorizationInfo::EpiloguePlan, llvm::VPBlockBase::getEntryBasicBlock(), llvm::InnerLoopVectorizer::getOrCreateVectorTripCount(), llvm::VPlan::getVectorLoopRegion(), H, llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::InnerLoopVectorizer::LoopScalarPreHeader, llvm::InnerLoopVectorizer::LoopVectorPreHeader, llvm::EpilogueLoopVectorizationInfo::MainLoopIterationCountCheck, llvm::EpilogueLoopVectorizationInfo::MemSafetyCheck, llvm::VPBasicBlock::phis(), llvm::EpilogueLoopVectorizationInfo::SCEVSafetyCheck, llvm::Value::setName(), and llvm::EpilogueLoopVectorizationInfo::VectorTripCount.

◆ emitIterationCountCheck()

BasicBlock * EpilogueVectorizerMainLoop::emitIterationCountCheck	(	BasicBlock *	Bypass,
		bool	ForEpilogue
	)

protected

Emits an iteration count bypass check once for the main loop (when ForEpilogue is false) and once for the epilogue loop (when ForEpilogue is true).

Definition at line 8003 of file LoopVectorize.cpp.

Referenced by createEpilogueVectorizedLoopSkeleton().

◆ fixupIVUsers()

void llvm::EpilogueVectorizerMainLoop::fixupIVUsers	(	PHINode *	OrigPhi,
		const InductionDescriptor &	II,
		Value *	VectorTripCount,
		BasicBlock *	MiddleBlock,
		VPTransformState &	State
	)