Holds state needed to make cost decisions before computing costs per-VF, including the maximum VFs. More...

#include "Transforms/Vectorize/LoopVectorizationPlanner.h"

Public Member Functions
	VFSelectionContext (const TargetTransformInfo &TTI, const LoopVectorizationLegality Legal, const Loop TheLoop, const Function &F, PredicatedScalarEvolution &PSE, OptimizationRemarkEmitter ORE, const LoopVectorizeHints Hints, bool OptForSize)
std::optional< unsigned >	getVScaleForTuning () const
bool	shouldConsiderRegPressureForVF (ElementCount VF) const
bool	supportsScalableVectors () const
void	collectElementTypesForWidening (const SmallPtrSetImpl< const Value * > *ValuesToIgnore=nullptr)
	Collect element types in the loop that need widening.
std::pair< unsigned, unsigned >	getSmallestAndWidestTypes () const
FixedScalableVFPair	computeFeasibleMaxVF (unsigned MaxTripCount, ElementCount UserVF, unsigned UserIC, bool FoldTailByMasking, bool RequiresScalarEpilogue)
std::optional< unsigned >	getMaxSafeElements () const
	Return maximum safe number of elements to be processed per vector iteration, which do not prevent store-load forwarding and are safe with regard to the memory dependencies.
bool	useOrderedReductions (const RecurrenceDescriptor &RdxDesc) const
	Returns true if we should use strict in-order reductions for the given RdxDesc.
bool	isLegalMaskedStore (Type DataType, Value Ptr, Align Alignment, unsigned AddressSpace) const
	Returns true if the target machine supports masked store operation for the given `DataType` and kind of access to `Ptr`.
bool	isLegalMaskedLoad (Type DataType, Value Ptr, Align Alignment, unsigned AddressSpace) const
	Returns true if the target machine supports masked load operation for the given `DataType` and kind of access to `Ptr`.
bool	isLegalGatherOrScatter (Value *V, ElementCount VF) const
	Returns true if the target machine can represent `V` as a masked gather or scatter operation.
void	collectInLoopReductions ()
	Split reductions into those that happen in the loop, and those that happen outside.
bool	isInLoopReduction (PHINode *Phi) const
	Returns true if the Phi is part of an inloop reduction.
const SmallPtrSetImpl< PHINode * > &	getInLoopReductions () const
	Returns the set of in-loop reduction PHIs.
Instruction *	getInLoopReductionImmediateChain (Instruction *I) const
	Returns the immediate chain operand of in-loop reduction operation `I`, or nullptr if `I` is not an in-loop reduction operation.
bool	runtimeChecksRequired ()
	Check whether vectorization would require runtime checks.

Public Attributes
const TTI::TargetCostKind	CostKind
	The kind of cost that we are calculating.
const bool	OptForSize
	Whether this loop should be optimized for size based on function attribute or profile information.

Detailed Description

Holds state needed to make cost decisions before computing costs per-VF, including the maximum VFs.

Definition at line 517 of file LoopVectorizationPlanner.h.

Constructor & Destructor Documentation

◆ VFSelectionContext()

llvm::VFSelectionContext::VFSelectionContext	(	const TargetTransformInfo &	TTI,
		const LoopVectorizationLegality *	Legal,
		const Loop *	TheLoop,
		const Function &	F,
		PredicatedScalarEvolution &	PSE,
		OptimizationRemarkEmitter *	ORE,
		const LoopVectorizeHints *	Hints,
		bool	OptForSize )

inline

Definition at line 597 of file LoopVectorizationPlanner.h.

References CostKind, and OptForSize.

Member Function Documentation

◆ collectElementTypesForWidening()

void VFSelectionContext::collectElementTypesForWidening ( const SmallPtrSetImpl< const Value * > * ValuesToIgnore = nullptr )

Collect element types in the loop that need widening.

Definition at line 464 of file LoopVectorizationPlanner.cpp.

References assert(), llvm::SmallPtrSetImpl< PtrType >::contains(), llvm::dyn_cast(), llvm::RecurrenceDescriptor::getRecurrenceKind(), llvm::RecurrenceDescriptor::getRecurrenceType(), I, llvm::isa(), llvm::PreferInLoopReductions, T, and useOrderedReductions().

Referenced by processLoopInVPlanNativePath().

◆ collectInLoopReductions()

void VFSelectionContext::collectInLoopReductions ( )

Split reductions into those that happen in the loop, and those that happen outside.

In-loop reductions are collected into InLoopReductions. InLoopReductionImmediateChains is filled with each in-loop reduction operation and its immediate chain operand for use during cost modelling.

Definition at line 566 of file LoopVectorizationPlanner.cpp.

References llvm::dbgs(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::RecurrenceDescriptor::getRecurrenceKind(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::RecurrenceDescriptor::getReductionOpChain(), llvm::RecurrenceDescriptor::hasUsesOutsideReductionChain(), I, llvm::RecurrenceDescriptor::isAnyOfRecurrenceKind(), llvm::RecurrenceDescriptor::isFindIVRecurrenceKind(), llvm::RecurrenceDescriptor::isFindLastRecurrenceKind(), LLVM_DEBUG, llvm::PreferInLoopReductions, and useOrderedReductions().

◆ computeFeasibleMaxVF()

FixedScalableVFPair VFSelectionContext::computeFeasibleMaxVF	(	unsigned	MaxTripCount,
		ElementCount	UserVF,
		unsigned	UserIC,
		bool	FoldTailByMasking,
		bool	RequiresScalarEpilogue )

Returns: An upper bound for the vectorization factors for both fixed and scalable vectorization, where the minimum-known number of elements is a power-of-2 larger than zero. If scalable vectorization is disabled or unsupported, then the scalable part will be equal to ElementCount::getScalable(0). Also sets MaxSafeElements.

Definition at line 322 of file LoopVectorizationPlanner.cpp.

References assert(), llvm::bit_floor(), llvm::dbgs(), DEBUG_TYPE, llvm::ElementCount::getFixed(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::ElementCount::getScalable(), getSmallestAndWidestTypes(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownGT(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownLE(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), LLVM_DEBUG, and supportsScalableVectors().

◆ getInLoopReductionImmediateChain()

Instruction * llvm::VFSelectionContext::getInLoopReductionImmediateChain ( Instruction * I ) const

inline

Returns the immediate chain operand of in-loop reduction operation I, or nullptr if I is not an in-loop reduction operation.

Definition at line 685 of file LoopVectorizationPlanner.h.

References I.

◆ getInLoopReductions()

const SmallPtrSetImpl< PHINode * > & llvm::VFSelectionContext::getInLoopReductions ( ) const

inline

Returns the set of in-loop reduction PHIs.

Definition at line 679 of file LoopVectorizationPlanner.h.

◆ getMaxSafeElements()

std::optional< unsigned > llvm::VFSelectionContext::getMaxSafeElements ( ) const

inline

Return maximum safe number of elements to be processed per vector iteration, which do not prevent store-load forwarding and are safe with regard to the memory dependencies.

Required for EVL-based VPlans to correctly calculate AVL (application vector length) as min(remaining AVL, MaxSafeElements). Set by computeFeasibleMaxVF. TODO: need to consider adjusting cost model to use this value as a vectorization factor for EVL-based vectorization.

Definition at line 645 of file LoopVectorizationPlanner.h.

◆ getSmallestAndWidestTypes()

std::pair< unsigned, unsigned > VFSelectionContext::getSmallestAndWidestTypes ( ) const

Returns: The size (in bits) of the smallest and widest types in the code that need to be vectorized. We ignore values that remain scalar such as 64 bit loop indices.

Definition at line 435 of file LoopVectorizationPlanner.cpp.

References _, DL, and T.

Referenced by computeFeasibleMaxVF(), and determineVPlanVF().