VPlan models a candidate for vectorization, encoding various decisions take to produce efficient output IR, including which branches, basic-blocks and output IR instructions to generate, and their cost. More...

#include "Transforms/Vectorize/VPlan.h"

Public Member Functions
	VPlan (VPBasicBlock Preheader, VPValue TC, VPBasicBlock *Entry)
	Construct a VPlan with original preheader `Preheader`, trip count `TC` and `Entry` to the plan.

	VPlan (VPBasicBlock Preheader, VPBasicBlock Entry)
	Construct a VPlan with original preheader `Preheader` and `Entry` to the plan.

	~VPlan ()

void	prepareToExecute (Value TripCount, Value VectorTripCount, Value *CanonicalIVStartValue, VPTransformState &State)
	Prepare the plan for execution, setting up the required live-in values.

void	execute (VPTransformState *State)
	Generate the IR code for this VPlan.

InstructionCost	cost (ElementCount VF, VPCostContext &Ctx)
	Return the cost of this plan.

VPBasicBlock *	getEntry ()

const VPBasicBlock *	getEntry () const

VPValue *	getTripCount () const
	The trip count of the original loop.

void	resetTripCount (VPValue *NewTripCount)
	Resets the trip count for the VPlan.

VPValue *	getOrCreateBackedgeTakenCount ()
	The backedge taken count of the original loop.

VPValue &	getVectorTripCount ()
	The vector trip count.

VPValue &	getVFxUF ()
	Returns VF * UF of the vector loop region.

void	addVF (ElementCount VF)

void	setVF (ElementCount VF)

bool	hasVF (ElementCount VF)

bool	hasScalableVF ()

iterator_range< SmallSetVector< ElementCount, 2 >::iterator >	vectorFactors () const
	Returns an iterator range over all VFs of the plan.

bool	hasScalarVFOnly () const

bool	hasUF (unsigned UF) const

void	setUF (unsigned UF)

std::string	getName () const
	Return a string with the name of the plan and the applicable VFs and UFs.

void	setName (const Twine &newName)

VPValue *	getOrAddLiveIn (Value *V)
	Gets the live-in VPValue for `V` or adds a new live-in (if none exists yet) for `V`.

VPValue *	getLiveIn (Value *V) const
	Return the live-in VPValue for `V`, if there is one or nullptr otherwise.

void	printLiveIns (raw_ostream &O) const
	Print the live-ins of this VPlan to `O`.

void	print (raw_ostream &O) const
	Print this VPlan to `O`.

void	printDOT (raw_ostream &O) const
	Print this VPlan in DOT format to `O`.

LLVM_DUMP_METHOD void	dump () const
	Dump the plan to stderr (for debugging).

VPRegionBlock *	getVectorLoopRegion ()
	Returns the VPRegionBlock of the vector loop.

const VPRegionBlock *	getVectorLoopRegion () const

VPCanonicalIVPHIRecipe *	getCanonicalIV ()
	Returns the canonical induction recipe of the vector loop.

void	addLiveOut (PHINode PN, VPValue V)

const MapVector< PHINode , VPLiveOut > &	getLiveOuts () const

VPValue *	getSCEVExpansion (const SCEV *S) const

void	addSCEVExpansion (const SCEV S, VPValue V)

VPBasicBlock *	getPreheader ()

const VPBasicBlock *	getPreheader () const

VPlan *	duplicate ()
	Clone the current VPlan, update all VPValues of the new VPlan and cloned recipes to refer to the clones, and return it.

Static Public Member Functions
static VPlanPtr	createInitialVPlan (const SCEV TripCount, ScalarEvolution &PSE, bool RequiresScalarEpilogueCheck, bool TailFolded, Loop TheLoop)
	Create initial VPlan, having an "entry" VPBasicBlock (wrapping original scalar pre-header ) which contains SCEV expansions that need to happen before the CFG is modified; a VPBasicBlock for the vector pre-header, followed by a region for the vector loop, followed by the middle VPBasicBlock.

Friends
class	VPlanPrinter

class	VPSlotTracker

Detailed Description

VPlan models a candidate for vectorization, encoding various decisions take to produce efficient output IR, including which branches, basic-blocks and output IR instructions to generate, and their cost.

VPlan holds a Hierarchical-CFG of VPBasicBlocks and VPRegionBlocks rooted at an Entry VPBasicBlock.

Definition at line 3268 of file VPlan.h.

Constructor & Destructor Documentation

◆ VPlan() [1/2]

llvm::VPlan::VPlan	(	VPBasicBlock *	Preheader,
		VPValue *	TC,
		VPBasicBlock *	Entry
	)

inline

Construct a VPlan with original preheader Preheader, trip count TC and Entry to the plan.

At the moment, Preheader and Entry need to be disconnected, as the bypass blocks between them are not yet modeled in VPlan.

Definition at line 3328 of file VPlan.h.

◆ VPlan() [2/2]

llvm::VPlan::VPlan	(	VPBasicBlock *	Preheader,
		VPBasicBlock *	Entry
	)

inline

Construct a VPlan with original preheader Preheader and Entry to the plan.

At the moment, Preheader and Entry need to be disconnected, as the bypass blocks between them are not yet modeled in VPlan.

Definition at line 3337 of file VPlan.h.

References assert(), llvm::VPBlockBase::getNumPredecessors(), llvm::VPBlockBase::getNumSuccessors(), and llvm::VPBlockBase::setPlan().

◆ ~VPlan()

VPlan::~VPlan ( )

Definition at line 842 of file VPlan.cpp.

References llvm::Block, llvm::VPBlockBase::deleteCFG(), and llvm::vp_depth_first_shallow().

Member Function Documentation

◆ addLiveOut()

void VPlan::addLiveOut	(	PHINode *	PN,
		VPValue *	V
	)

Definition at line 1187 of file VPlan.cpp.

References assert().

Referenced by addLiveOutsForFirstOrderRecurrences(), and addUsersInExitBlock().

◆ addSCEVExpansion()

void llvm::VPlan::addSCEVExpansion	(	const SCEV *	S,
		VPValue *	V
	)

inline

Definition at line 3497 of file VPlan.h.

References assert(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains().

Referenced by llvm::vputils::getOrCreateVPValueForSCEVExpr().

◆ addVF()

void llvm::VPlan::addVF ( ElementCount VF )

inline

Definition at line 3400 of file VPlan.h.

References llvm::SetVector< T, Vector, Set, N >::insert().

◆ cost()

InstructionCost VPlan::cost	(	ElementCount	VF,
		VPCostContext &	Ctx
	)

Return the cost of this plan.

Definition at line 1091 of file VPlan.cpp.

References llvm::VPTransformState::VF.

◆ createInitialVPlan()

VPlanPtr VPlan::createInitialVPlan	(	const SCEV *	TripCount,
		ScalarEvolution &	PSE,
		bool	RequiresScalarEpilogueCheck,
		bool	TailFolded,
		Loop *	TheLoop
	)

static

Create initial VPlan, having an "entry" VPBasicBlock (wrapping original scalar pre-header ) which contains SCEV expansions that need to happen before the CFG is modified; a VPBasicBlock for the vector pre-header, followed by a region for the vector loop, followed by the middle VPBasicBlock.

If a check is needed to guard executing the scalar epilogue loop, it will be added to the middle block, together with VPBasicBlocks for the scalar preheader and exit blocks.

Definition at line 863 of file VPlan.cpp.

References llvm::VPInstruction::BranchOnCond, llvm::VPTransformState::Builder, llvm::VPBlockUtils::connectBlocks(), llvm::Type::getContext(), llvm::Type::getInt1Ty(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), getOrAddLiveIn(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::BasicBlock::getTerminator(), getTripCount(), llvm::ConstantInt::getTrue(), llvm::SCEV::getType(), llvm::LoopBase< BlockT, LoopT >::getUniqueExitBlock(), getVectorTripCount(), llvm::CmpInst::ICMP_EQ, llvm::VPBlockUtils::insertBlockAfter(), and llvm::VPTransformState::Plan.

◆ dump()

LLVM_DUMP_METHOD void VPlan::dump ( ) const

Dump the plan to stderr (for debugging).

Definition at line 1184 of file VPlan.cpp.

References llvm::dbgs(), and llvm::print().

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ duplicate()

VPlan * VPlan::duplicate ( )

Clone the current VPlan, update all VPValues of the new VPlan and cloned recipes to refer to the clones, and return it.

Definition at line 1230 of file VPlan.cpp.

References _, assert(), llvm::VPBasicBlock::clone(), cloneFrom(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains(), Name, and remapOperands().

Referenced by llvm::LoopVectorizePass::processLoop().

◆ execute()

void VPlan::execute ( VPTransformState * State )

Generate the IR code for this VPlan.

Generate the code inside the preheader and body of the vectorized loop.

Assumes a single pre-header basic-block was created for this. Introduce additional basic-blocks as needed, and fill them all.

Definition at line 981 of file VPlan.cpp.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ getCanonicalIV()

VPCanonicalIVPHIRecipe * llvm::VPlan::getCanonicalIV ( )

inline

Returns the canonical induction recipe of the vector loop.

Definition at line 3478 of file VPlan.h.

References llvm::VPBasicBlock::begin(), llvm::VPBasicBlock::empty(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPBlockBase::getSingleSuccessor(), and getVectorLoopRegion().

Referenced by llvm::VPlanTransforms::addActiveLaneMask(), addLiveOutsForFirstOrderRecurrences(), addVPLaneMaskPhiAndUpdateExitBranch(), llvm::VPlanTransforms::adjustFixedOrderRecurrences(), collectAllHeaderMasks(), llvm::VPRecipeBuilder::createHeaderMask(), createScalarIVSteps(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPlanTransforms::optimizeForVFAndUF(), removeRedundantCanonicalIVs(), simplifyRecipes(), llvm::VPlanTransforms::truncateToMinimalBitwidths(), llvm::VPlanTransforms::tryAddExplicitVectorLength(), and willGenerateVectors().

◆ getEntry() [1/2]

VPBasicBlock * llvm::VPlan::getEntry ( )

inline

Definition at line 3370 of file VPlan.h.

Referenced by addReplicateRegions(), llvm::VPlanPrinter::dump(), llvm::InnerLoopVectorizer::fixNonInductionPHIs(), llvm::DomTreeNodeTraits< VPBlockBase >::getEntryNode(), getVectorLoopRegion(), mergeBlocksIntoPredecessors(), mergeReplicateRegionsIntoSuccessors(), removeDeadRecipes(), removeRedundantExpandSCEVRecipes(), llvm::VPBlockBase::setPlan(), simplifyRecipes(), sinkScalarOperands(), and llvm::VPlanTransforms::truncateToMinimalBitwidths().

◆ getEntry() [2/2]

const VPBasicBlock * llvm::VPlan::getEntry ( ) const

inline

Definition at line 3371 of file VPlan.h.

◆ getLiveIn()

VPValue * llvm::VPlan::getLiveIn ( Value * V ) const

inline

Return the live-in VPValue for V, if there is one or nullptr otherwise.

Definition at line 3453 of file VPlan.h.

References llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::lookup().

◆ getLiveOuts()

const MapVector< PHINode *, VPLiveOut * > & llvm::VPlan::getLiveOuts ( ) const

inline

Definition at line 3489 of file VPlan.h.

Referenced by llvm::InnerLoopVectorizer::fixVectorizedLoop().

◆ getName()

std::string VPlan::getName ( ) const

Return a string with the name of the plan and the applicable VFs and UFs.

Definition at line 1154 of file VPlan.cpp.

References llvm::drop_begin(), Name, llvm::VPTransformState::UF, and llvm::VPTransformState::VF.

Referenced by llvm::VPlanPrinter::dump().

◆ getOrAddLiveIn()

VPValue * llvm::VPlan::getOrAddLiveIn ( Value * V )

inline

Gets the live-in VPValue for V or adds a new live-in (if none exists yet) for V.

Definition at line 3436 of file VPlan.h.

References assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), llvm::VPValue::isLiveIn(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().

Referenced by addCanonicalIVRecipes(), addLiveOutsForFirstOrderRecurrences(), createInitialVPlan(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::VPRecipeBuilder::getVPValueOrAddLiveIn(), legalizeAndOptimizeInductions(), llvm::VPRecipeBuilder::mapToVPValues(), llvm::VPlanTransforms::optimizeForVFAndUF(), and llvm::LoopVectorizePass::processLoop().

◆ getOrCreateBackedgeTakenCount()

VPValue * llvm::VPlan::getOrCreateBackedgeTakenCount ( )

inline

The backedge taken count of the original loop.

Definition at line 3388 of file VPlan.h.

Referenced by llvm::VPRecipeBuilder::createHeaderMask(), and llvm::vputils::isHeaderMask().

◆ getPreheader() [1/2]

VPBasicBlock * llvm::VPlan::getPreheader ( )

inline

Returns: The block corresponding to the original preheader.

Definition at line 3503 of file VPlan.h.

Referenced by llvm::VPlanPrinter::dump(), llvm::LoopVectorizationPlanner::executePlan(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::LoopVectorizePass::processLoop(), and llvm::VPBlockBase::setPlan().

◆ getPreheader() [2/2]

const VPBasicBlock * llvm::VPlan::getPreheader ( ) const

inline

Definition at line 3504 of file VPlan.h.

◆ getSCEVExpansion()

VPValue * llvm::VPlan::getSCEVExpansion ( const SCEV * S ) const

inline

Definition at line 3493 of file VPlan.h.

References llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::lookup().

Referenced by llvm::InnerLoopVectorizer::fixupIVUsers(), and llvm::vputils::getOrCreateVPValueForSCEVExpr().

◆ getTripCount()

VPValue * llvm::VPlan::getTripCount ( ) const

inline

The trip count of the original loop.

Definition at line 3374 of file VPlan.h.

References assert().

Referenced by llvm::VPlanTransforms::addActiveLaneMask(), addVPLaneMaskPhiAndUpdateExitBranch(), createInitialVPlan(), llvm::LoopVectorizationPlanner::executePlan(), llvm::vputils::isHeaderMask(), llvm::LoopVectorizePass::processLoop(), and llvm::VPlanTransforms::tryAddExplicitVectorLength().

◆ getVectorLoopRegion() [1/2]

VPRegionBlock * llvm::VPlan::getVectorLoopRegion ( )

inline

Returns the VPRegionBlock of the vector loop.

Definition at line 3470 of file VPlan.h.

References getEntry().

Referenced by addCanonicalIVRecipes(), addLiveOutsForFirstOrderRecurrences(), addUsersInExitBlock(), addVPLaneMaskPhiAndUpdateExitBranch(), llvm::VPlanTransforms::adjustFixedOrderRecurrences(), llvm::VPlanTransforms::clearReductionWrapFlags(), collectAllHeaderMasks(), llvm::collectEphemeralRecipesForVPlan(), llvm::VPRecipeBuilder::createHeaderMask(), createScalarIVSteps(), llvm::LoopVectorizationPlanner::executePlan(), llvm::VPLiveOut::fixPhi(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), llvm::VPTransformState::get(), getCanonicalIV(), legalizeAndOptimizeInductions(), llvm::VPlanTransforms::optimizeForVFAndUF(), llvm::LoopVectorizePass::processLoop(), removeRedundantCanonicalIVs(), removeRedundantInductionCasts(), llvm::VPlanTransforms::truncateToMinimalBitwidths(), llvm::VPlanTransforms::tryAddExplicitVectorLength(), llvm::VPInterleavedAccessInfo::VPInterleavedAccessInfo(), and willGenerateVectors().

◆ getVectorLoopRegion() [2/2]

const VPRegionBlock * llvm::VPlan::getVectorLoopRegion ( ) const

inline

Definition at line 3473 of file VPlan.h.

References getEntry().

◆ getVectorTripCount()

VPValue & llvm::VPlan::getVectorTripCount ( )

inline

The vector trip count.

Definition at line 3395 of file VPlan.h.

Referenced by addCanonicalIVRecipes(), and createInitialVPlan().

◆ getVFxUF()

VPValue & llvm::VPlan::getVFxUF ( )

inline

Returns VF * UF of the vector loop region.

Definition at line 3398 of file VPlan.h.

Referenced by addCanonicalIVRecipes().

◆ hasScalableVF()

bool llvm::VPlan::hasScalableVF ( )

inline

Definition at line 3409 of file VPlan.h.

References llvm::any_of(), and llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable().

Referenced by legalizeAndOptimizeInductions().

◆ hasScalarVFOnly()

bool llvm::VPlan::hasScalarVFOnly ( ) const

inline

Definition at line 3419 of file VPlan.h.

References llvm::SetVector< T, Vector, Set, N >::size().

Referenced by sinkScalarOperands().

◆ hasUF()

bool llvm::VPlan::hasUF ( unsigned UF ) const

inline

Definition at line 3421 of file VPlan.h.

References llvm::SetVector< T, Vector, Set, N >::contains(), and llvm::SetVector< T, Vector, Set, N >::empty().

Referenced by llvm::LoopVectorizationPlanner::executePlan(), llvm::VPlanTransforms::optimizeForVFAndUF(), and setUF().

◆ hasVF()

bool llvm::VPlan::hasVF ( ElementCount VF )

inline

Definition at line 3408 of file VPlan.h.

References llvm::SetVector< T, Vector, Set, N >::count().

Referenced by llvm::LoopVectorizationPlanner::executePlan(), llvm::LoopVectorizationPlanner::getPlanFor(), legalizeAndOptimizeInductions(), llvm::VPlanTransforms::optimizeForVFAndUF(), and setVF().

◆ prepareToExecute()

void VPlan::prepareToExecute	(	Value *	TripCount,
		Value *	VectorTripCount,
		Value *	CanonicalIVStartValue,
		VPTransformState &	State
	)

Prepare the plan for execution, setting up the required live-in values.

Definition at line 921 of file VPlan.cpp.

References llvm::all_of(), assert(), llvm::VPTransformState::Builder, llvm::VPTransformState::CFG, llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), IV, llvm::VPTransformState::CFGState::PrevBB, llvm::VPTransformState::UF, and llvm::VPTransformState::VF.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ print()

LLVM_DUMP_METHOD void VPlan::print ( raw_ostream & O ) const

Print this VPlan to O.

Definition at line 1128 of file VPlan.cpp.

References llvm::Block, getName(), and llvm::vp_depth_first_shallow().

Referenced by llvm::operator<<(), and llvm::LoopVectorizationPlanner::printPlans().

◆ printDOT()

LLVM_DUMP_METHOD void VPlan::printDOT ( raw_ostream & O ) const

Print this VPlan in DOT format to O.

Definition at line 1178 of file VPlan.cpp.

References llvm::Metadata::dump(), and Printer.

Referenced by llvm::LoopVectorizationPlanner::printPlans().

◆ printLiveIns()

void VPlan::printLiveIns ( raw_ostream & O ) const

Print the live-ins of this VPlan to O.

Definition at line 1098 of file VPlan.cpp.

Referenced by llvm::VPlanPrinter::dump().

◆ resetTripCount()

void llvm::VPlan::resetTripCount ( VPValue * NewTripCount )

inline

Resets the trip count for the VPlan.

The caller must make sure all uses of the original trip count have been replaced.

Definition at line 3381 of file VPlan.h.

References assert(), and llvm::VPValue::getNumUsers().

Referenced by llvm::LoopVectorizePass::processLoop().

◆ setName()

void llvm::VPlan::setName ( const Twine & newName )

inline

Definition at line 3432 of file VPlan.h.

References Name, and llvm::Twine::str().

Referenced by llvm::VPlanHCFGBuilder::buildHierarchicalCFG(), and llvm::LoopVectorizationPlanner::executePlan().

◆ setUF()

void llvm::VPlan::setUF ( unsigned UF )

inline

Definition at line 3423 of file VPlan.h.

References assert(), llvm::SetVector< T, Vector, Set, N >::clear(), hasUF(), and llvm::SetVector< T, Vector, Set, N >::insert().

Referenced by llvm::VPlanTransforms::optimizeForVFAndUF(), and llvm::VPlanTransforms::tryAddExplicitVectorLength().

◆ setVF()

void llvm::VPlan::setVF ( ElementCount VF )

inline

Definition at line 3402 of file VPlan.h.

References assert(), llvm::SetVector< T, Vector, Set, N >::clear(), hasVF(), and llvm::SetVector< T, Vector, Set, N >::insert().

Referenced by llvm::VPlanTransforms::optimizeForVFAndUF().

◆ vectorFactors()

iterator_range< SmallSetVector< ElementCount, 2 >::iterator > llvm::VPlan::vectorFactors ( ) const

inline

Returns an iterator range over all VFs of the plan.

Definition at line 3415 of file VPlan.h.

References llvm::SetVector< T, Vector, Set, N >::begin(), and llvm::SetVector< T, Vector, Set, N >::end().

Referenced by llvm::LoopVectorizationPlanner::computeBestVF().

Friends And Related Function Documentation

◆ VPlanPrinter

friend class VPlanPrinter

friend

Definition at line 3269 of file VPlan.h.

◆ VPSlotTracker

friend class VPSlotTracker

friend

Definition at line 3270 of file VPlan.h.

The documentation for this class was generated from the following files:

lib/Transforms/Vectorize/VPlan.h
lib/Transforms/Vectorize/VPlan.cpp

Public Member Functions

Static Public Member Functions

Friends

Detailed Description

Constructor & Destructor Documentation

◆ VPlan() [1/2]

◆ VPlan() [2/2]

◆ ~VPlan()

Member Function Documentation

◆ addLiveOut()

◆ addSCEVExpansion()

◆ addVF()

◆ cost()

◆ createInitialVPlan()

◆ dump()

◆ duplicate()

◆ execute()

◆ getCanonicalIV()

◆ getEntry() [1/2]

◆ getEntry() [2/2]

◆ getLiveIn()

◆ getLiveOuts()

◆ getName()

◆ getOrAddLiveIn()

◆ getOrCreateBackedgeTakenCount()

◆ getPreheader() [1/2]

◆ getPreheader() [2/2]

◆ getSCEVExpansion()

◆ getTripCount()

◆ getVectorLoopRegion() [1/2]

◆ getVectorLoopRegion() [2/2]

◆ getVectorTripCount()

◆ getVFxUF()

◆ hasScalableVF()

◆ hasScalarVFOnly()

◆ hasUF()

◆ hasVF()

◆ prepareToExecute()

◆ print()

◆ printDOT()

◆ printLiveIns()

◆ resetTripCount()

◆ setName()

◆ setUF()

◆ setVF()

◆ vectorFactors()

Friends And Related Function Documentation

◆ VPlanPrinter

◆ VPSlotTracker