VPlan models a candidate for vectorization, encoding various decisions take to produce efficient output IR, including which branches, basic-blocks and output IR instructions to generate, and their cost. More...

#include "Transforms/Vectorize/VPlan.h"

Public Member Functions
	VPlan (VPBasicBlock Preheader, VPValue TC, VPBasicBlock *Entry)
	Construct a VPlan with original preheader `Preheader`, trip count `TC` and `Entry` to the plan.

	VPlan (VPBasicBlock Preheader, VPBasicBlock Entry)
	Construct a VPlan with original preheader `Preheader` and `Entry` to the plan.

	~VPlan ()

void	prepareToExecute (Value TripCount, Value VectorTripCount, Value *CanonicalIVStartValue, VPTransformState &State, bool IsEpilogueVectorization)
	Prepare the plan for execution, setting up the required live-in values.

void	execute (VPTransformState *State)
	Generate the IR code for this VPlan.

VPBasicBlock *	getEntry ()

const VPBasicBlock *	getEntry () const

VPValue *	getTripCount () const
	The trip count of the original loop.

VPValue *	getOrCreateBackedgeTakenCount ()
	The backedge taken count of the original loop.

VPValue &	getVectorTripCount ()
	The vector trip count.

void	disableValue2VPValue ()
	Mark the plan to indicate that using Value2VPValue is not safe any longer, because it may be stale.

void	addVF (ElementCount VF)

void	setVF (ElementCount VF)

bool	hasVF (ElementCount VF)

bool	hasScalarVFOnly () const

bool	hasUF (unsigned UF) const

void	setUF (unsigned UF)

std::string	getName () const
	Return a string with the name of the plan and the applicable VFs and UFs.

void	setName (const Twine &newName)

void	addVPValue (Value V, VPValue VPV)

VPValue *	getVPValue (Value *V, bool OverrideAllowed=false)
	Returns the VPValue for `V`.

VPValue *	getVPValueOrAddLiveIn (Value *V)
	Gets the VPValue for `V` or adds a new live-in (if none exists yet) for `V`.

void	print (raw_ostream &O) const
	Print this VPlan to `O`.

void	printDOT (raw_ostream &O) const
	Print this VPlan in DOT format to `O`.

LLVM_DUMP_METHOD void	dump () const
	Dump the plan to stderr (for debugging).

iterator_range< mapped_iterator< Use , std::function< VPValue (Value *)> > >	mapToVPValues (User::op_range Operands)
	Returns a range mapping the values the range `Operands` to their corresponding VPValues.

VPRegionBlock *	getVectorLoopRegion ()
	Returns the VPRegionBlock of the vector loop.

const VPRegionBlock *	getVectorLoopRegion () const

VPCanonicalIVPHIRecipe *	getCanonicalIV ()
	Returns the canonical induction recipe of the vector loop.

VPActiveLaneMaskPHIRecipe *	getActiveLaneMaskPhi ()
	Find and return the VPActiveLaneMaskPHIRecipe from the header - there be only one at most.

void	addLiveOut (PHINode PN, VPValue V)

void	removeLiveOut (PHINode *PN)

const MapVector< PHINode , VPLiveOut > &	getLiveOuts () const

VPValue *	getSCEVExpansion (const SCEV *S) const

void	addSCEVExpansion (const SCEV S, VPValue V)

VPBasicBlock *	getPreheader ()

const VPBasicBlock *	getPreheader () const

Static Public Member Functions
static VPlanPtr	createInitialVPlan (const SCEV *TripCount, ScalarEvolution &PSE)
	Create an initial VPlan with preheader and entry blocks.

Friends
class	VPlanPrinter

class	VPSlotTracker

Detailed Description

VPlan models a candidate for vectorization, encoding various decisions take to produce efficient output IR, including which branches, basic-blocks and output IR instructions to generate, and their cost.

VPlan holds a Hierarchical-CFG of VPBasicBlocks and VPRegionBlocks rooted at an Entry VPBasicBlock.

Definition at line 2472 of file VPlan.h.

Constructor & Destructor Documentation

◆ VPlan() [1/2]

llvm::VPlan::VPlan	(	VPBasicBlock *	Preheader,
		VPValue *	TC,
		VPBasicBlock *	Entry
	)

inline

Construct a VPlan with original preheader Preheader, trip count TC and Entry to the plan.

At the moment, Preheader and Entry need to be disconnected, as the bypass blocks between them are not yet modeled in VPlan.

Definition at line 2531 of file VPlan.h.

◆ VPlan() [2/2]

llvm::VPlan::VPlan	(	VPBasicBlock *	Preheader,
		VPBasicBlock *	Entry
	)

inline

Construct a VPlan with original preheader Preheader and Entry to the plan.

At the moment, Preheader and Entry need to be disconnected, as the bypass blocks between them are not yet modeled in VPlan.

Definition at line 2540 of file VPlan.h.

References assert(), llvm::VPBlockBase::getNumPredecessors(), llvm::VPBlockBase::getNumSuccessors(), and llvm::VPBlockBase::setPlan().

◆ ~VPlan()

VPlan::~VPlan ( )

Definition at line 691 of file VPlan.cpp.

References llvm::Block, llvm::VPBlockBase::deleteCFG(), and llvm::vp_depth_first_shallow().

Member Function Documentation

◆ addLiveOut()

void VPlan::addLiveOut	(	PHINode *	PN,
		VPValue *	V
	)

Definition at line 929 of file VPlan.cpp.

References assert().

Referenced by addUsersInExitBlock().

◆ addSCEVExpansion()

void llvm::VPlan::addSCEVExpansion	(	const SCEV *	S,
		VPValue *	V
	)

inline

Definition at line 2702 of file VPlan.h.

References assert(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains().

Referenced by llvm::vputils::getOrCreateVPValueForSCEVExpr().

◆ addVF()

void llvm::VPlan::addVF ( ElementCount VF )

inline

Definition at line 2587 of file VPlan.h.

References llvm::SetVector< T, Vector, Set, N >::insert().

◆ addVPValue()

void llvm::VPlan::addVPValue	(	Value *	V,
		VPValue *	VPV
	)

inline

Definition at line 2612 of file VPlan.h.

References assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), and llvm::VPValue::isLiveIn().

Referenced by getVPValueOrAddLiveIn().

◆ createInitialVPlan()

VPlanPtr VPlan::createInitialVPlan	(	const SCEV *	TripCount,
		ScalarEvolution &	PSE
	)

static

Create an initial VPlan with preheader and entry blocks.

Creates a VPExpandSCEVRecipe for TripCount and uses it as plan's trip count.

Definition at line 712 of file VPlan.cpp.

References llvm::vputils::getOrCreateVPValueForSCEVExpr(), and llvm::VPTransformState::Plan.

◆ disableValue2VPValue()

void llvm::VPlan::disableValue2VPValue ( )

inline

Mark the plan to indicate that using Value2VPValue is not safe any longer, because it may be stale.

Definition at line 2585 of file VPlan.h.

◆ dump()

LLVM_DUMP_METHOD void VPlan::dump ( ) const

Dump the plan to stderr (for debugging).

Definition at line 926 of file VPlan.cpp.

References llvm::dbgs(), and llvm::print().

◆ execute()

void VPlan::execute ( VPTransformState * State )

Generate the IR code for this VPlan.

Generate the code inside the preheader and body of the vectorized loop.

Assumes a single pre-header basic-block was created for this. Introduce additional basic-blocks as needed, and fill them all.

Definition at line 772 of file VPlan.cpp.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ getActiveLaneMaskPhi()

VPActiveLaneMaskPHIRecipe * VPlan::getActiveLaneMaskPhi ( )

Find and return the VPActiveLaneMaskPHIRecipe from the header - there be only one at most.

If there isn't one, then return nullptr.

Definition at line 721 of file VPlan.cpp.

Referenced by llvm::VPRecipeBuilder::createHeaderMask().

◆ getCanonicalIV()

VPCanonicalIVPHIRecipe * llvm::VPlan::getCanonicalIV ( )

inline

Returns the canonical induction recipe of the vector loop.

Definition at line 2674 of file VPlan.h.

References llvm::VPBasicBlock::begin(), llvm::VPBasicBlock::empty(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPBlockBase::getSingleSuccessor(), and getVectorLoopRegion().

Referenced by llvm::VPRecipeBuilder::createHeaderMask(), createScalarIVSteps(), llvm::VPWidenPointerInductionRecipe::execute(), and llvm::VPlanTransforms::optimizeForVFAndUF().

◆ getEntry() [1/2]

VPBasicBlock * llvm::VPlan::getEntry ( )

inline

Definition at line 2564 of file VPlan.h.

Referenced by addReplicateRegions(), llvm::VPlanPrinter::dump(), llvm::InnerLoopVectorizer::fixNonInductionPHIs(), llvm::DomTreeNodeTraits< VPBlockBase >::getEntryNode(), getVectorLoopRegion(), mergeReplicateRegionsIntoSuccessors(), llvm::VPBlockBase::setPlan(), simplifyRecipes(), sinkScalarOperands(), and llvm::VPlanVerifier::verifyPlanIsValid().

◆ getEntry() [2/2]

const VPBasicBlock * llvm::VPlan::getEntry ( ) const

inline

Definition at line 2565 of file VPlan.h.

◆ getLiveOuts()

const MapVector< PHINode *, VPLiveOut * > & llvm::VPlan::getLiveOuts ( ) const

inline

Definition at line 2694 of file VPlan.h.

Referenced by llvm::InnerLoopVectorizer::fixVectorizedLoop(), and llvm::VPlanVerifier::verifyPlanIsValid().

◆ getName()

std::string VPlan::getName ( ) const

Return a string with the name of the plan and the applicable VFs and UFs.

Definition at line 896 of file VPlan.cpp.

References llvm::drop_begin(), Name, llvm::VPTransformState::UF, and llvm::VPTransformState::VF.

Referenced by llvm::VPlanPrinter::dump().

◆ getOrCreateBackedgeTakenCount()

VPValue * llvm::VPlan::getOrCreateBackedgeTakenCount ( )

inline

The backedge taken count of the original loop.

Definition at line 2574 of file VPlan.h.

Referenced by llvm::VPRecipeBuilder::createHeaderMask().

◆ getPreheader() [1/2]

VPBasicBlock * llvm::VPlan::getPreheader ( )

inline

Returns: The block corresponding to the original preheader.

Definition at line 2708 of file VPlan.h.

Referenced by llvm::VPlanPrinter::dump(), llvm::LoopVectorizationPlanner::executePlan(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::LoopVectorizePass::processLoop(), and llvm::VPBlockBase::setPlan().

◆ getPreheader() [2/2]

const VPBasicBlock * llvm::VPlan::getPreheader ( ) const

inline

Definition at line 2709 of file VPlan.h.

◆ getSCEVExpansion()

VPValue * llvm::VPlan::getSCEVExpansion ( const SCEV * S ) const

inline

Definition at line 2698 of file VPlan.h.

References llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::lookup().

Referenced by llvm::InnerLoopVectorizer::fixupIVUsers(), and llvm::vputils::getOrCreateVPValueForSCEVExpr().

◆ getTripCount()

VPValue * llvm::VPlan::getTripCount ( ) const

inline

The trip count of the original loop.

Definition at line 2568 of file VPlan.h.

References assert().

Referenced by addCanonicalIVRecipes(), llvm::VPRecipeBuilder::createHeaderMask(), and llvm::LoopVectorizationPlanner::executePlan().

◆ getVectorLoopRegion() [1/2]

VPRegionBlock * llvm::VPlan::getVectorLoopRegion ( )

inline

◆ getVectorLoopRegion() [2/2]

const VPRegionBlock * llvm::VPlan::getVectorLoopRegion ( ) const

inline

Definition at line 2669 of file VPlan.h.

References getEntry().

◆ getVectorTripCount()

VPValue & llvm::VPlan::getVectorTripCount ( )

inline

The vector trip count.

Definition at line 2581 of file VPlan.h.

Referenced by addCanonicalIVRecipes().

◆ getVPValue()

VPValue * llvm::VPlan::getVPValue	(	Value *	V,
		bool	OverrideAllowed = `false`
	)

inline

Returns the VPValue for V.

OverrideAllowed can be used to disable /// checking whether it is safe to query VPValues using IR Values.

Definition at line 2622 of file VPlan.h.

References assert(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count().

Referenced by getVPValueOrAddLiveIn(), and llvm::InnerLoopVectorizer::truncateToMinimalBitwidths().

◆ getVPValueOrAddLiveIn()

VPValue * llvm::VPlan::getVPValueOrAddLiveIn ( Value * V )

inline

Gets the VPValue for V or adds a new live-in (if none exists yet) for V.

Definition at line 2633 of file VPlan.h.

References addVPValue(), assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), getVPValue(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().

Referenced by addCanonicalIVRecipes(), addUsersInExitBlock(), llvm::VPRecipeBuilder::createEdgeMask(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), mapToVPValues(), llvm::VPlanTransforms::optimizeForVFAndUF(), and llvm::LoopVectorizePass::processLoop().

◆ hasScalarVFOnly()

bool llvm::VPlan::hasScalarVFOnly ( ) const

inline

Definition at line 2597 of file VPlan.h.

References llvm::SetVector< T, Vector, Set, N >::size().

Referenced by sinkScalarOperands().

◆ hasUF()

bool llvm::VPlan::hasUF ( unsigned UF ) const

inline

Definition at line 2599 of file VPlan.h.

References llvm::SetVector< T, Vector, Set, N >::contains(), and llvm::SetVector< T, Vector, Set, N >::empty().

Referenced by llvm::LoopVectorizationPlanner::executePlan(), llvm::VPlanTransforms::optimizeForVFAndUF(), and setUF().

◆ hasVF()

bool llvm::VPlan::hasVF ( ElementCount VF )

inline

Definition at line 2595 of file VPlan.h.

References llvm::SetVector< T, Vector, Set, N >::count().

Referenced by llvm::LoopVectorizationPlanner::executePlan(), llvm::VPlanTransforms::optimizeForVFAndUF(), and setVF().

◆ mapToVPValues()

iterator_range< mapped_iterator< Use *, std::function< VPValue *(Value *)> > > llvm::VPlan::mapToVPValues ( User::op_range Operands )

inline

Returns a range mapping the values the range Operands to their corresponding VPValues.

Definition at line 2658 of file VPlan.h.

References getVPValueOrAddLiveIn(), llvm::map_range(), and Operands.

Referenced by llvm::VPRecipeBuilder::handleReplication().

◆ prepareToExecute()

void VPlan::prepareToExecute	(	Value *	TripCount,
		Value *	VectorTripCount,
		Value *	CanonicalIVStartValue,
		VPTransformState &	State,
		bool	IsEpilogueVectorization
	)

Prepare the plan for execution, setting up the required live-in values.

Definition at line 730 of file VPlan.cpp.

References llvm::all_of(), assert(), llvm::VPTransformState::Builder, llvm::VPTransformState::CFG, llvm::IRBuilderBase::CreateSub(), llvm::IRBuilderBase::CreateVectorSplat(), llvm::ConstantInt::get(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::ElementCount::isScalar(), IV, llvm::VPTransformState::CFGState::PrevBB, llvm::VPTransformState::set(), llvm::VPTransformState::UF, and llvm::VPTransformState::VF.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ print()

LLVM_DUMP_METHOD void VPlan::print ( raw_ostream & O ) const

Print this VPlan to O.

Definition at line 853 of file VPlan.cpp.

References llvm::Block, getName(), and llvm::vp_depth_first_shallow().

Referenced by llvm::operator<<().

◆ printDOT()

LLVM_DUMP_METHOD void VPlan::printDOT ( raw_ostream & O ) const

Print this VPlan in DOT format to O.

Definition at line 920 of file VPlan.cpp.

References llvm::Metadata::dump(), and Printer.

◆ removeLiveOut()

void llvm::VPlan::removeLiveOut ( PHINode * PN )

inline

Definition at line 2689 of file VPlan.h.

References llvm::MapVector< KeyT, ValueT, MapType, VectorType >::erase().

Referenced by llvm::InnerLoopVectorizer::fixFixedOrderRecurrence(), llvm::InnerLoopVectorizer::fixReduction(), and llvm::InnerLoopVectorizer::fixupIVUsers().

◆ setName()

void llvm::VPlan::setName ( const Twine & newName )

inline

Definition at line 2610 of file VPlan.h.

References Name, and llvm::Twine::str().

Referenced by llvm::VPlanHCFGBuilder::buildHierarchicalCFG().

◆ setUF()

void llvm::VPlan::setUF ( unsigned UF )

inline

Definition at line 2601 of file VPlan.h.

References assert(), llvm::SetVector< T, Vector, Set, N >::clear(), hasUF(), and llvm::SetVector< T, Vector, Set, N >::insert().

Referenced by llvm::VPlanTransforms::optimizeForVFAndUF().

◆ setVF()

void llvm::VPlan::setVF ( ElementCount VF )

inline

Definition at line 2589 of file VPlan.h.

References assert(), llvm::SetVector< T, Vector, Set, N >::clear(), hasVF(), and llvm::SetVector< T, Vector, Set, N >::insert().

Referenced by llvm::VPlanTransforms::optimizeForVFAndUF().

Friends And Related Function Documentation

◆ VPlanPrinter

friend class VPlanPrinter

friend

Definition at line 2473 of file VPlan.h.

◆ VPSlotTracker

friend class VPSlotTracker

friend

Definition at line 2474 of file VPlan.h.

The documentation for this class was generated from the following files:

lib/Transforms/Vectorize/VPlan.h
lib/Transforms/Vectorize/VPlan.cpp

Public Member Functions

Static Public Member Functions

Friends

Detailed Description

Constructor & Destructor Documentation

◆ VPlan() [1/2]

◆ VPlan() [2/2]

◆ ~VPlan()

Member Function Documentation

◆ addLiveOut()

◆ addSCEVExpansion()

◆ addVF()

◆ addVPValue()

◆ createInitialVPlan()

◆ disableValue2VPValue()

◆ dump()

◆ execute()

◆ getActiveLaneMaskPhi()

◆ getCanonicalIV()

◆ getEntry() [1/2]

◆ getEntry() [2/2]

◆ getLiveOuts()

◆ getName()

◆ getOrCreateBackedgeTakenCount()

◆ getPreheader() [1/2]

◆ getPreheader() [2/2]

◆ getSCEVExpansion()

◆ getTripCount()

◆ getVectorLoopRegion() [1/2]

◆ getVectorLoopRegion() [2/2]

◆ getVectorTripCount()

◆ getVPValue()

◆ getVPValueOrAddLiveIn()

◆ hasScalarVFOnly()

◆ hasUF()

◆ hasVF()

◆ mapToVPValues()

◆ prepareToExecute()

◆ print()

◆ printDOT()

◆ removeLiveOut()

◆ setName()

◆ setUF()

◆ setVF()

Friends And Related Function Documentation

◆ VPlanPrinter

◆ VPSlotTracker