InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF). More...

Inheritance diagram for llvm::InnerLoopVectorizer:

[legend]

Public Member Functions
	InnerLoopVectorizer (Loop OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo LI, DominatorTree DT, const TargetLibraryInfo TLI, const TargetTransformInfo TTI, AssumptionCache AC, OptimizationRemarkEmitter ORE, ElementCount VecWidth, ElementCount MinProfitableTripCount, unsigned UnrollFactor, LoopVectorizationLegality LVL, LoopVectorizationCostModel CM, BlockFrequencyInfo BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &RTChecks, VPlan &Plan)

virtual	~InnerLoopVectorizer ()=default

virtual BasicBlock *	createVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs)
	Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder.

void	fixVectorizedLoop (VPTransformState &State)
	Fix the vectorized code, taking care of header phi's, and more.

bool	areSafetyChecksAdded ()

void	scalarizeInstruction (const Instruction Instr, VPReplicateRecipe RepRecipe, const VPLane &Lane, VPTransformState &State)
	A helper function to scalarize a single Instruction in the innermost loop.

void	fixNonInductionPHIs (VPTransformState &State)
	Fix the non-induction PHIs in `Plan`.

void	createInductionResumeVPValue (VPIRInstruction InductionPhiIRI, const InductionDescriptor &ID, Value Step, ArrayRef< BasicBlock * > BypassBlocks, VPBuilder &ScalarPHBuilder, Value *MainVectorTripCount=nullptr)
	Create a ResumePHI VPInstruction for the induction `InductionPhiIRI` to resume iteration count in the scalar epilogue from where the vectorized loop left off, and add it to the scalar preheader of VPlan.

Value *	getTripCount () const
	Returns the original loop trip count.

void	setTripCount (Value *TC)
	Used to set the trip count after ILV's construction and after the preheader block has been executed.

Value *	getInductionAdditionalBypassValue (PHINode *OrigPhi) const
	induction header phi.

BasicBlock *	getAdditionalBypassBlock () const
	Return the additional bypass block which targets the scalar loop by skipping the epilogue loop after completing the main loop.

Protected Member Functions
virtual void	fixupIVUsers (PHINode OrigPhi, const InductionDescriptor &II, Value VectorTripCount, BasicBlock *MiddleBlock, VPTransformState &State)
	Set up the values of the IVs correctly when exiting the vector loop.

void	sinkScalarOperands (Instruction *PredInst)
	Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it.

Value *	getOrCreateVectorTripCount (BasicBlock *InsertBlock)
	Returns (and creates if needed) the trip count of the widened loop.

void	emitIterationCountCheck (BasicBlock *Bypass)
	Emit a bypass check to see if the vector trip count is zero, including if it overflows.

BasicBlock *	emitSCEVChecks (BasicBlock *Bypass)
	Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct.

BasicBlock *	emitMemRuntimeChecks (BasicBlock *Bypass)
	Emit bypass checks to check any memory assumptions we may have made.

void	createVectorLoopSkeleton (StringRef Prefix)
	Emit basic blocks (prefixed with `Prefix`) for the iteration check, vector loop preheader, middle block and scalar preheader.

void	createInductionResumeVPValues (const SCEV2ValueTy &ExpandedSCEVs, Value MainVectorTripCount=nullptr, SmallPtrSetImpl< PHINode > *IVSubset=nullptr)
	Create new phi nodes for the induction variables to resume iteration count in the scalar epilogue, from where the vectorized loop left off.

virtual void	printDebugTracesAtStart ()
	Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.

virtual void	printDebugTracesAtEnd ()

Protected Attributes
Loop *	OrigLoop
	The original loop.

PredicatedScalarEvolution &	PSE
	A wrapper around ScalarEvolution used to add runtime SCEV checks.

LoopInfo *	LI
	Loop Info.

DominatorTree *	DT
	Dominator Tree.

const TargetLibraryInfo *	TLI
	Target Library Info.

const TargetTransformInfo *	TTI
	Target Transform Info.

AssumptionCache *	AC
	Assumption Cache.

OptimizationRemarkEmitter *	ORE
	Interface to emit optimization remarks.

ElementCount	VF
	The vectorization SIMD factor to use.

ElementCount	MinProfitableTripCount

unsigned	UF
	The vectorization unroll factor to use.

IRBuilder	Builder
	The builder that we use.

BasicBlock *	LoopVectorPreHeader
	The vector-loop preheader.

BasicBlock *	LoopScalarPreHeader
	The scalar-loop preheader.

BasicBlock *	LoopMiddleBlock
	Middle Block between the vector and the scalar.

SmallVector< BasicBlock *, 4 >	LoopBypassBlocks
	A list of all bypass blocks. The first block is the entry of the loop.

SmallVector< Instruction *, 4 >	PredicatedInstructions
	Store instructions that were predicated.

Value *	TripCount = nullptr
	Trip count of the original loop.

Value *	VectorTripCount = nullptr
	Trip count of the widened loop (TripCount - TripCount % (VF*UF))

LoopVectorizationLegality *	Legal
	The legality analysis.

LoopVectorizationCostModel *	Cost
	The profitablity analysis.

bool	AddedSafetyChecks = false

BlockFrequencyInfo *	BFI
	BFI and PSI are used to check for profile guided size optimizations.

ProfileSummaryInfo *	PSI

bool	OptForSizeBasedOnProfile

GeneratedRTChecks &	RTChecks
	Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable.

DenseMap< PHINode , Value >	Induction2AdditionalBypassValue
	Mapping of induction phis to their additional bypass values.

BasicBlock *	AdditionalBypassBlock = nullptr
	The additional bypass block which conditionally skips over the epilogue loop after executing the main loop.

VPlan &	Plan

Friends
class	LoopVectorizationPlanner

Detailed Description

InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF).

This class performs the widening of scalars into vectors, or multiple scalars. This class also implements the following features:

It inserts an epilogue loop for handling loops that don't have iteration counts that are known to be a multiple of the vectorization factor.
It handles the code generation for reduction variables.
Scalarization (implementation using scalars) of un-vectorizable instructions. InnerLoopVectorizer does not perform any vectorization-legality checks, and relies on the caller to check for the different legality aspects. The InnerLoopVectorizer relies on the LoopVectorizationLegality class to provide information about the induction and reduction variables that were found to a given vectorization factor.

Definition at line 466 of file LoopVectorize.cpp.

Constructor & Destructor Documentation

◆ InnerLoopVectorizer()

llvm::InnerLoopVectorizer::InnerLoopVectorizer	(	Loop *	OrigLoop,
		PredicatedScalarEvolution &	PSE,
		LoopInfo *	LI,
		DominatorTree *	DT,
		const TargetLibraryInfo *	TLI,
		const TargetTransformInfo *	TTI,
		AssumptionCache *	AC,
		OptimizationRemarkEmitter *	ORE,
		ElementCount	VecWidth,
		ElementCount	MinProfitableTripCount,
		unsigned	UnrollFactor,
		LoopVectorizationLegality *	LVL,
		LoopVectorizationCostModel *	CM,
		BlockFrequencyInfo *	BFI,
		ProfileSummaryInfo *	PSI,
		GeneratedRTChecks &	RTChecks,
		VPlan &	Plan
	)

inline

Definition at line 468 of file LoopVectorize.cpp.

References BFI, llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::IRPass, OptForSizeBasedOnProfile, OrigLoop, PSI, and llvm::shouldOptimizeForSize().

◆ ~InnerLoopVectorizer()

virtual llvm::InnerLoopVectorizer::~InnerLoopVectorizer ( )

virtualdefault

Member Function Documentation

◆ areSafetyChecksAdded()

bool llvm::InnerLoopVectorizer::areSafetyChecksAdded ( )

inline

Definition at line 506 of file LoopVectorize.cpp.

References AddedSafetyChecks.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ createInductionResumeVPValue()

void InnerLoopVectorizer::createInductionResumeVPValue	(	VPIRInstruction *	InductionPhiIRI,
		const InductionDescriptor &	ID,
		Value *	Step,
		ArrayRef< BasicBlock * >	BypassBlocks,
		VPBuilder &	ScalarPHBuilder,
		Value *	MainVectorTripCount = `nullptr`
	)

Create a ResumePHI VPInstruction for the induction InductionPhiIRI to resume iteration count in the scalar epilogue from where the vectorized loop left off, and add it to the scalar preheader of VPlan.

Also creates the induction resume value, and the value for the bypass block, if needed. Step is the SCEV-expanded induction step to use. In cases where the loop skeleton is more complicated (i.e., epilogue vectorization) and the resume values can come from an additional bypass block, MainVectorTripCount provides the trip count of the main vector loop, used to compute the resume value reaching the scalar loop preheader directly from this additional bypass block.

Definition at line 2644 of file LoopVectorize.cpp.

References llvm::VPUser::addOperand(), assert(), B, llvm::VPBuilder::createNaryOp(), emitTransformedIndex(), getAdditionalBypassBlock(), llvm::VPIRInstruction::getInstruction(), llvm::VPUser::getNumOperands(), llvm::VPlan::getOrAddLiveIn(), getOrCreateVectorTripCount(), llvm::LoopVectorizationLegality::getPrimaryInduction(), llvm::BasicBlock::getTerminator(), II, Induction2AdditionalBypassValue, Legal, LoopVectorPreHeader, Plan, llvm::VPInstruction::ResumePhi, llvm::Value::setName(), and VectorTripCount.

Referenced by createInductionResumeVPValues().

◆ createInductionResumeVPValues()

void InnerLoopVectorizer::createInductionResumeVPValues	(	const SCEV2ValueTy &	ExpandedSCEVs,
		Value *	MainVectorTripCount = `nullptr`,
		SmallPtrSetImpl< PHINode * > *	IVSubset = `nullptr`
	)

protected

Create new phi nodes for the induction variables to resume iteration count in the scalar epilogue, from where the vectorized loop left off.

In cases where the loop skeleton is more complicated (i.e. epilogue vectorization), MainVectorTripCount provides the trip count of the main loop, used to compute these resume values. If IVSubset is provided, it contains the phi nodes for which resume values are needed, because they will generate wide induction phis in the epilogue loop.

Definition at line 2736 of file LoopVectorize.cpp.

References llvm::VPBasicBlock::begin(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::contains(), llvm::SmallPtrSetImpl< PtrType >::contains(), createInductionResumeVPValue(), llvm::VPBuilder::createNaryOp(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::find(), getExpandedStep(), llvm::LoopVectorizationLegality::getInductionVars(), llvm::VPlan::getOrAddLiveIn(), llvm::VPlan::getScalarHeader(), llvm::VPlan::getScalarPreheader(), llvm::Value::getType(), llvm::VPlan::getVectorTripCount(), II, Legal, LoopBypassBlocks, Plan, llvm::VPInstruction::ResumePhi, and VectorTripCount.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().

◆ createVectorizedLoopSkeleton()

BasicBlock * InnerLoopVectorizer::createVectorizedLoopSkeleton ( const SCEV2ValueTy & ExpandedSCEVs )

virtual

Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder.

Control flow is generated around the vectorized (and scalar epilogue) loops consisting of various checks and bypasses. Return the pre-header block of the new loop. In the case of epilogue vectorization, this function is overriden to handle the more complex control flow around the loops. ExpandedSCEVs is used to look up SCEV expansions for expressions needed during skeleton creation.

Reimplemented in llvm::InnerLoopAndEpilogueVectorizer.

Definition at line 2778 of file LoopVectorize.cpp.

References createInductionResumeVPValues(), createVectorLoopSkeleton(), emitIterationCountCheck(), emitMemRuntimeChecks(), emitSCEVChecks(), LoopScalarPreHeader, and LoopVectorPreHeader.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ createVectorLoopSkeleton()

void InnerLoopVectorizer::createVectorLoopSkeleton ( StringRef Prefix )

protected

Emit basic blocks (prefixed with Prefix) for the iteration check, vector loop preheader, middle block and scalar preheader.

Definition at line 2629 of file LoopVectorize.cpp.

References assert(), DT, llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::BasicBlock::getTerminator(), llvm::LoopBase< BlockT, LoopT >::getUniqueLatchExitBlock(), llvm::ElementCount::isVector(), LI, LoopMiddleBlock, LoopScalarPreHeader, LoopVectorPreHeader, OrigLoop, llvm::SplitBlock(), and VF.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().

◆ emitIterationCountCheck()

void InnerLoopVectorizer::emitIterationCountCheck ( BasicBlock * Bypass )

protected

Emit a bypass check to see if the vector trip count is zero, including if it overflows.

Definition at line 2488 of file LoopVectorize.cpp.

References llvm::ScalarEvolution::applyLoopGuards(), assert(), Builder, llvm::BranchInst::Create(), llvm::IRBuilderBase::CreateBinaryIntrinsic(), llvm::IRBuilderBase::CreateICmp(), llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), llvm::DataAndControlFlowWithoutRuntimeCheck, DT, llvm::IRBuilderBase::getFalse(), llvm::DomTreeNodeBase< NodeT >::getIDom(), llvm::CmpInst::getInversePredicate(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::DominatorTreeBase< NodeT, IsPostDom >::getNode(), llvm::ScalarEvolution::getSCEV(), llvm::PredicatedScalarEvolution::getSE(), llvm::BasicBlock::getTerminator(), getTripCount(), llvm::IRBuilderBase::getTrue(), llvm::Value::getType(), llvm::hasBranchWeightMD(), llvm::CmpInst::ICMP_ULE, llvm::CmpInst::ICMP_ULT, introduceCheckBlockInVPlan(), isIndvarOverflowCheckKnownFalse(), llvm::ScalarEvolution::isKnownPredicate(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isVector(), LHS, LI, LoopBypassBlocks, LoopVectorPreHeader, MinItersBypassWeights, MinProfitableTripCount, llvm::None, OrigLoop, P, Plan, llvm::DominatorTreeBase< NodeT, IsPostDom >::properlyDominates(), PSE, llvm::ReplaceInstWithInst(), llvm::setBranchWeights(), llvm::SplitBlock(), UF, and VF.

Referenced by createVectorizedLoopSkeleton().

◆ emitMemRuntimeChecks()

BasicBlock * InnerLoopVectorizer::emitMemRuntimeChecks ( BasicBlock * Bypass )

protected

Emit bypass checks to check any memory assumptions we may have made.

Returns the block containing the checks or nullptr if no checks have been added.

Definition at line 2592 of file LoopVectorize.cpp.

References AddedSafetyChecks, assert(), DEBUG_TYPE, llvm::OptimizationRemarkEmitter::emit(), llvm::EnableVPlanNativePath, llvm::LoopVectorizeHints::FK_Enabled, llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::BasicBlock::getParent(), llvm::Loop::getStartLoc(), llvm::Function::hasOptSize(), introduceCheckBlockInVPlan(), LoopBypassBlocks, LoopVectorPreHeader, OptForSizeBasedOnProfile, ORE, OrigLoop, Plan, and RTChecks.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().

◆ emitSCEVChecks()

BasicBlock * InnerLoopVectorizer::emitSCEVChecks ( BasicBlock * Bypass )

protected

Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct.

Returns the block containing the checks or nullptr if no checks have been added.

Definition at line 2573 of file LoopVectorize.cpp.

References AddedSafetyChecks, assert(), llvm::LoopVectorizeHints::FK_Enabled, llvm::BasicBlock::getParent(), llvm::Function::hasOptSize(), introduceCheckBlockInVPlan(), LoopBypassBlocks, LoopVectorPreHeader, OptForSizeBasedOnProfile, Plan, and RTChecks.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().

◆ fixNonInductionPHIs()

void InnerLoopVectorizer::fixNonInductionPHIs ( VPTransformState & State )

Fix the non-induction PHIs in Plan.

Definition at line 3197 of file LoopVectorize.cpp.

References llvm::PHINode::addIncoming(), Builder, llvm::VPTransformState::CFG, llvm::VPTransformState::get(), llvm::VPlan::getEntry(), llvm::VPWidenPHIRecipe::getIncomingBlock(), llvm::VPWidenPHIRecipe::getIncomingValue(), llvm::VPUser::getNumOperands(), Idx, P, Plan, llvm::IRBuilderBase::SetInsertPoint(), llvm::vp_depth_first_deep(), and llvm::VPTransformState::CFGState::VPBB2IRBB.

Referenced by fixVectorizedLoop().

◆ fixupIVUsers()

void InnerLoopVectorizer::fixupIVUsers	(	PHINode *	OrigPhi,
		const InductionDescriptor &	II,
		Value *	VectorTripCount,
		BasicBlock *	MiddleBlock,
		VPTransformState &	State
	)

protectedvirtual

Set up the values of the IVs correctly when exiting the vector loop.

Reimplemented in llvm::EpilogueVectorizerMainLoop.

Definition at line 2845 of file LoopVectorize.cpp.

References llvm::all_of(), assert(), B, llvm::LoopBase< BlockT, LoopT >::contains(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::empty(), llvm::VPTransformState::get(), llvm::PHINode::getIncomingValueForBlock(), llvm::VPValue::getLiveInIRValue(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::VPlan::getSCEVExpansion(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), I, II, llvm::Type::isFloatingPointTy(), llvm::Type::isIntegerTy(), llvm::VPValue::isLiveIn(), llvm::Type::isPointerTy(), OrigLoop, P, PHI, Plan, PostInc, llvm::predecessors(), llvm::Value::setName(), and llvm::Value::users().

Referenced by fixVectorizedLoop().

◆ fixVectorizedLoop()

void InnerLoopVectorizer::fixVectorizedLoop ( VPTransformState & State )

Fix the vectorized code, taking care of header phi's, and more.

Definition at line 3042 of file LoopVectorize.cpp.

References llvm::LoopBase< BlockT, LoopT >::addBasicBlockToLoop(), llvm::VPTransformState::CFG, cse(), llvm::EnableVPlanNativePath, fixNonInductionPHIs(), fixupIVUsers(), llvm::ScalarEvolution::forgetBlockAndLoopDispositions(), llvm::ScalarEvolution::forgetLcssaPhiWithNewPredecessor(), llvm::ScalarEvolution::forgetLoop(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::LoopBase< BlockT, LoopT >::getExitBlocks(), llvm::LoopVectorizationLegality::getInductionVars(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::VPlan::getMiddleBlock(), getOrCreateVectorTripCount(), llvm::LoopBase< BlockT, LoopT >::getParentLoop(), llvm::PredicatedScalarEvolution::getSE(), llvm::VPBlockBase::getSinglePredecessor(), llvm::VPlan::getVectorLoopRegion(), llvm::LoopVectorizationLegality::hasUncountableEarlyExit(), llvm::ElementCount::isVector(), Legal, LI, LoopMiddleBlock, OrigLoop, llvm::VPTransformState::Plan, PredicatedInstructions, PSE, llvm::setProfileInfoAfterUnrolling(), sinkScalarOperands(), UF, VF, and llvm::VPTransformState::CFGState::VPBB2IRBB.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ getAdditionalBypassBlock()

BasicBlock * llvm::InnerLoopVectorizer::getAdditionalBypassBlock ( ) const

inline

Return the additional bypass block which targets the scalar loop by skipping the epilogue loop after completing the main loop.

Definition at line 552 of file LoopVectorize.cpp.

References AdditionalBypassBlock, and assert().

Referenced by createInductionResumeVPValue(), and llvm::LoopVectorizationPlanner::executePlan().

◆ getInductionAdditionalBypassValue()

Value * llvm::InnerLoopVectorizer::getInductionAdditionalBypassValue ( PHINode * OrigPhi ) const

inline

induction header phi.

Definition at line 546 of file LoopVectorize.cpp.

References Induction2AdditionalBypassValue.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ getOrCreateVectorTripCount()

Value * InnerLoopVectorizer::getOrCreateVectorTripCount ( BasicBlock * InsertBlock )

protected

Returns (and creates if needed) the trip count of the widened loop.

Definition at line 2420 of file LoopVectorize.cpp.

References assert(), Builder, llvm::IRBuilderBase::CreateAdd(), llvm::IRBuilderBase::CreateICmpEQ(), llvm::IRBuilderBase::CreateSelect(), llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), llvm::IRBuilderBase::CreateURem(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::BasicBlock::getTerminator(), getTripCount(), llvm::Value::getType(), llvm::isPowerOf2_32(), llvm::ElementCount::isVector(), UF, VectorTripCount, and VF.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), createInductionResumeVPValue(), llvm::LoopVectorizationPlanner::executePlan(), and fixVectorizedLoop().

◆ getTripCount()

Value * llvm::InnerLoopVectorizer::getTripCount ( ) const

inline

Returns the original loop trip count.

Definition at line 537 of file LoopVectorize.cpp.

References TripCount.

Referenced by emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::LoopVectorizationPlanner::executePlan(), getOrCreateVectorTripCount(), and llvm::LoopVectorizePass::processLoop().

◆ printDebugTracesAtEnd()

virtual void llvm::InnerLoopVectorizer::printDebugTracesAtEnd ( )

inlineprotectedvirtual

Reimplemented in llvm::EpilogueVectorizerMainLoop, and llvm::EpilogueVectorizerEpilogueLoop.

Definition at line 606 of file LoopVectorize.cpp.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ printDebugTracesAtStart()

virtual void llvm::InnerLoopVectorizer::printDebugTracesAtStart ( )

inlineprotectedvirtual

Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.

Reimplemented in llvm::EpilogueVectorizerMainLoop, and llvm::EpilogueVectorizerEpilogueLoop.

Definition at line 605 of file LoopVectorize.cpp.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ scalarizeInstruction()

void InnerLoopVectorizer::scalarizeInstruction	(	const Instruction *	Instr,
		VPReplicateRecipe *	RepRecipe,
		const VPLane &	Lane,
		VPTransformState &	State
	)

A helper function to scalarize a single Instruction in the innermost loop.

Generates a sequence of scalar instances for each lane between MinLane and MaxLane, times each part between MinPart and MaxPart, inclusive. Uses the VPValue operands from RepRecipe instead of Instr's operands.

Definition at line 2361 of file LoopVectorize.cpp.

References AC, llvm::VPTransformState::addNewMetadata(), llvm::all_of(), assert(), llvm::VPTransformState::Builder, DL, llvm::enumerate(), llvm::VPTransformState::get(), llvm::VPLane::getFirstLane(), llvm::VPBlockBase::getParent(), llvm::VPRecipeBase::getParent(), llvm::Value::getType(), I, II, llvm::VPTypeAnalysis::inferScalarType(), llvm::IRBuilderBase::Insert(), llvm::VPRegionBlock::isReplicator(), llvm::vputils::isUniformAfterVectorization(), llvm::VPUser::operands(), PredicatedInstructions, llvm::AssumptionCache::registerAssumption(), llvm::VPTransformState::set(), llvm::VPTransformState::setDebugLocFrom(), llvm::VPRecipeWithIRFlags::setFlags(), llvm::Value::setName(), llvm::User::setOperand(), and llvm::VPTransformState::TypeAnalysis.

Referenced by llvm::VPReplicateRecipe::execute().

◆ setTripCount()

void llvm::InnerLoopVectorizer::setTripCount ( Value * TC )

inline

Used to set the trip count after ILV's construction and after the preheader block has been executed.

Note that this always holds the trip count of the original loop for both main loop and epilogue vectorization.

Definition at line 542 of file LoopVectorize.cpp.

References TripCount.

Referenced by llvm::LoopVectorizationPlanner::executePlan(), and llvm::LoopVectorizePass::processLoop().

◆ sinkScalarOperands()

void InnerLoopVectorizer::sinkScalarOperands ( Instruction * PredInst )

protected

Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it.

Definition at line 3121 of file LoopVectorize.cpp.

References llvm::all_of(), llvm::SmallVectorTemplateCommon< T, typename >::begin(), llvm::SmallVectorImpl< T >::clear(), llvm::SetVector< T, Vector, Set, N >::empty(), llvm::SmallVectorTemplateCommon< T, typename >::end(), llvm::PHINode::getIncomingValueNumForOperand(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::ilist_detail::node_parent_access< NodeTy, ParentTy >::getParent(), I, llvm::SetVector< T, Vector, Set, N >::insert(), LI, llvm::User::op_begin(), llvm::User::op_end(), llvm::SetVector< T, Vector, Set, N >::pop_back_val(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().

Referenced by fixVectorizedLoop().

Friends And Related Function Documentation

◆ LoopVectorizationPlanner

friend class LoopVectorizationPlanner

friend

Definition at line 559 of file LoopVectorize.cpp.

Member Data Documentation

◆ AC

AssumptionCache* llvm::InnerLoopVectorizer::AC

protected

Assumption Cache.

Definition at line 629 of file LoopVectorize.cpp.

Referenced by scalarizeInstruction().

◆ AddedSafetyChecks

bool llvm::InnerLoopVectorizer::AddedSafetyChecks = false

protected

Definition at line 677 of file LoopVectorize.cpp.

Referenced by areSafetyChecksAdded(), emitMemRuntimeChecks(), and emitSCEVChecks().

◆ AdditionalBypassBlock

BasicBlock* llvm::InnerLoopVectorizer::AdditionalBypassBlock = nullptr

protected

The additional bypass block which conditionally skips over the epilogue loop after executing the main loop.

Needed to resume inductions and reductions during epilogue vectorization.

Definition at line 699 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), and getAdditionalBypassBlock().

◆ BFI

BlockFrequencyInfo* llvm::InnerLoopVectorizer::BFI

protected

BFI and PSI are used to check for profile guided size optimizations.

Definition at line 680 of file LoopVectorize.cpp.

Referenced by InnerLoopVectorizer().

◆ Builder

IRBuilder llvm::InnerLoopVectorizer::Builder

protected

The builder that we use.

Definition at line 645 of file LoopVectorize.cpp.

Referenced by emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), llvm::LoopVectorizationPlanner::executePlan(), fixNonInductionPHIs(), and getOrCreateVectorTripCount().

◆ Cost

LoopVectorizationCostModel* llvm::InnerLoopVectorizer::Cost

protected

The profitablity analysis.

Definition at line 674 of file LoopVectorize.cpp.

◆ DT

DominatorTree* llvm::InnerLoopVectorizer::DT

protected

Dominator Tree.

Definition at line 620 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), and llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck().

◆ Induction2AdditionalBypassValue

DenseMap<PHINode *, Value *> llvm::InnerLoopVectorizer::Induction2AdditionalBypassValue

protected

Mapping of induction phis to their additional bypass values.

They need to be added as operands to phi nodes in the scalar loop preheader after the epilogue skeleton has been created.

Definition at line 694 of file LoopVectorize.cpp.

Referenced by createInductionResumeVPValue(), and getInductionAdditionalBypassValue().

◆ Legal

LoopVectorizationLegality* llvm::InnerLoopVectorizer::Legal

protected

The legality analysis.

Definition at line 671 of file LoopVectorize.cpp.

Referenced by createInductionResumeVPValue(), createInductionResumeVPValues(), llvm::LoopVectorizationPlanner::executePlan(), and fixVectorizedLoop().

◆ LI

LoopInfo* llvm::InnerLoopVectorizer::LI

protected

Loop Info.

Definition at line 617 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), fixVectorizedLoop(), and sinkScalarOperands().

◆ LoopBypassBlocks

SmallVector<BasicBlock *, 4> llvm::InnerLoopVectorizer::LoopBypassBlocks

protected

A list of all bypass blocks. The first block is the entry of the loop.

Definition at line 659 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createInductionResumeVPValues(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), and emitSCEVChecks().

◆ LoopMiddleBlock

BasicBlock* llvm::InnerLoopVectorizer::LoopMiddleBlock

protected

Middle Block between the vector and the scalar.

Definition at line 656 of file LoopVectorize.cpp.

Referenced by createVectorLoopSkeleton(), and fixVectorizedLoop().

◆ LoopScalarPreHeader

BasicBlock* llvm::InnerLoopVectorizer::LoopScalarPreHeader

protected

The scalar-loop preheader.

Definition at line 653 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createVectorizedLoopSkeleton(), and createVectorLoopSkeleton().

◆ LoopVectorPreHeader

BasicBlock* llvm::InnerLoopVectorizer::LoopVectorPreHeader

protected

The vector-loop preheader.

Definition at line 650 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createInductionResumeVPValue(), createVectorizedLoopSkeleton(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), emitSCEVChecks(), and llvm::LoopVectorizationPlanner::executePlan().

◆ MinProfitableTripCount

ElementCount llvm::InnerLoopVectorizer::MinProfitableTripCount

protected

Definition at line 638 of file LoopVectorize.cpp.

Referenced by emitIterationCountCheck().

◆ OptForSizeBasedOnProfile

bool llvm::InnerLoopVectorizer::OptForSizeBasedOnProfile

protected

Definition at line 685 of file LoopVectorize.cpp.

Referenced by emitMemRuntimeChecks(), emitSCEVChecks(), and InnerLoopVectorizer().

◆ ORE

OptimizationRemarkEmitter* llvm::InnerLoopVectorizer::ORE

protected

Interface to emit optimization remarks.

Definition at line 632 of file LoopVectorize.cpp.

Referenced by emitMemRuntimeChecks().

◆ OrigLoop

Loop* llvm::InnerLoopVectorizer::OrigLoop

protected

The original loop.

Definition at line 609 of file LoopVectorize.cpp.

Referenced by createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), fixupIVUsers(), fixVectorizedLoop(), InnerLoopVectorizer(), llvm::EpilogueVectorizerMainLoop::printDebugTracesAtEnd(), and llvm::EpilogueVectorizerEpilogueLoop::printDebugTracesAtEnd().

◆ Plan

VPlan& llvm::InnerLoopVectorizer::Plan

protected

Definition at line 701 of file LoopVectorize.cpp.

Referenced by createInductionResumeVPValue(), createInductionResumeVPValues(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), emitSCEVChecks(), fixNonInductionPHIs(), and fixupIVUsers().

◆ PredicatedInstructions

SmallVector<Instruction *, 4> llvm::InnerLoopVectorizer::PredicatedInstructions

protected

Store instructions that were predicated.

Definition at line 662 of file LoopVectorize.cpp.

Referenced by fixVectorizedLoop(), and scalarizeInstruction().

◆ PSE

PredicatedScalarEvolution& llvm::InnerLoopVectorizer::PSE

protected

A wrapper around ScalarEvolution used to add runtime SCEV checks.

Applies dynamic knowledge to simplify SCEV expressions and converts them to a more usable form.

Definition at line 614 of file LoopVectorize.cpp.

Referenced by emitIterationCountCheck(), and fixVectorizedLoop().

◆ PSI

ProfileSummaryInfo* llvm::InnerLoopVectorizer::PSI

protected

Definition at line 681 of file LoopVectorize.cpp.

Referenced by InnerLoopVectorizer().

◆ RTChecks

GeneratedRTChecks& llvm::InnerLoopVectorizer::RTChecks

protected

Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable.

Definition at line 689 of file LoopVectorize.cpp.

Referenced by emitMemRuntimeChecks(), and emitSCEVChecks().

◆ TLI

const TargetLibraryInfo* llvm::InnerLoopVectorizer::TLI

protected

Target Library Info.

Definition at line 623 of file LoopVectorize.cpp.

◆ TripCount

Value* llvm::InnerLoopVectorizer::TripCount = nullptr

protected

Trip count of the original loop.

Definition at line 665 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::EpilogueVectorizerEpilogueLoop(), getTripCount(), and setTripCount().

◆ TTI

const TargetTransformInfo* llvm::InnerLoopVectorizer::TTI

protected

Target Transform Info.

Definition at line 626 of file LoopVectorize.cpp.

◆ UF

unsigned llvm::InnerLoopVectorizer::UF

protected

The vectorization unroll factor to use.

Each scalar is vectorized to this many different vector instructions.

Definition at line 642 of file LoopVectorize.cpp.

Referenced by emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), fixVectorizedLoop(), and getOrCreateVectorTripCount().

◆ VectorTripCount

Value* llvm::InnerLoopVectorizer::VectorTripCount = nullptr

protected

Trip count of the widened loop (TripCount - TripCount % (VF*UF))

Definition at line 668 of file LoopVectorize.cpp.

Referenced by createInductionResumeVPValue(), createInductionResumeVPValues(), and getOrCreateVectorTripCount().

◆ VF

ElementCount llvm::InnerLoopVectorizer::VF

protected

The vectorization SIMD factor to use.

Each vector will have this many vector elements.

Definition at line 636 of file LoopVectorize.cpp.

Referenced by createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), fixVectorizedLoop(), and getOrCreateVectorTripCount().

The documentation for this class was generated from the following file:

lib/Transforms/Vectorize/LoopVectorize.cpp

Public Member Functions

Protected Member Functions

Protected Attributes

Friends

Detailed Description

Constructor & Destructor Documentation

◆ InnerLoopVectorizer()

◆ ~InnerLoopVectorizer()

Member Function Documentation

◆ areSafetyChecksAdded()

◆ createInductionResumeVPValue()

◆ createInductionResumeVPValues()

◆ createVectorizedLoopSkeleton()

◆ createVectorLoopSkeleton()

◆ emitIterationCountCheck()

◆ emitMemRuntimeChecks()

◆ emitSCEVChecks()

◆ fixNonInductionPHIs()

◆ fixupIVUsers()

◆ fixVectorizedLoop()

◆ getAdditionalBypassBlock()

◆ getInductionAdditionalBypassValue()

◆ getOrCreateVectorTripCount()

◆ getTripCount()

◆ printDebugTracesAtEnd()

◆ printDebugTracesAtStart()

◆ scalarizeInstruction()

◆ setTripCount()

◆ sinkScalarOperands()

Friends And Related Function Documentation

◆ LoopVectorizationPlanner

Member Data Documentation

◆ AC

◆ AddedSafetyChecks

◆ AdditionalBypassBlock

◆ BFI

◆ Builder

◆ Cost

◆ DT

◆ Induction2AdditionalBypassValue

◆ Legal

◆ LI

◆ LoopBypassBlocks

◆ LoopMiddleBlock

◆ LoopScalarPreHeader

◆ LoopVectorPreHeader

◆ MinProfitableTripCount

◆ OptForSizeBasedOnProfile

◆ ORE

◆ OrigLoop

◆ Plan

◆ PredicatedInstructions

◆ PSE

◆ PSI

◆ RTChecks

◆ TLI

◆ TripCount

◆ TTI

◆ UF

◆ VectorTripCount

◆ VF