Merges shuffle masks and emits final shuffle instruction, if required. More...

Inheritance diagram for slpvectorizer::BoUpSLP::ShuffleInstructionBuilder:

Public Member Functions
	ShuffleInstructionBuilder (Type *ScalarTy, IRBuilderBase &Builder, BoUpSLP &R)
Value *	adjustExtracts (const TreeEntry *E, MutableArrayRef< int > Mask, ArrayRef< std::optional< TTI::ShuffleKind > > ShuffleKinds, unsigned NumParts, bool &UseVecBaseAsInput)
	Adjusts extractelements after reusing them.
std::optional< Value * >	needToDelay (const TreeEntry E, ArrayRef< SmallVector< const TreeEntry > > Deps) const
	Checks if the specified entry `E` needs to be delayed because of its dependency nodes.
void	resetForSameNode ()
	Reset the builder to handle perfect diamond match.
void	add (const TreeEntry &E1, const TreeEntry &E2, ArrayRef< int > Mask)
	Adds 2 input vectors (in form of tree entries) and the mask for their shuffling.
void	add (const TreeEntry &E1, ArrayRef< int > Mask)
	Adds single input vector (in form of tree entry) and the mask for its shuffling.
void	add (Value V1, Value V2, ArrayRef< int > Mask)
	Adds 2 input vectors and the mask for their shuffling.
void	add (Value *V1, ArrayRef< int > Mask, bool=false, TTI::VectorInstrContext=TTI::VectorInstrContext::None)
	Adds another one input vector and the mask for the shuffling.
void	addOrdered (Value *V1, ArrayRef< unsigned > Order)
	Adds another one input vector and the mask for the shuffling.
Value *	gather (ArrayRef< Value * > VL, unsigned MaskVF=0, Value *Root=nullptr, TTI::VectorInstrContext=TTI::VectorInstrContext::None)
Value *	createExtractedFieldsVector (Value *Src, unsigned FieldWidth, ArrayRef< int > Mask, const TreeEntry &)
	Emits the gather of zero-extended sub-fields of width `FieldWidth` of the same wider integer scalar `Src`, permuted by `Mask`, as a bitcast to the field vector plus a permutation and an extension to the lane type.
Value *	createFreeze (Value *V)
Value *	finalize (ArrayRef< int > ExtMask, ArrayRef< std::pair< const TreeEntry , unsigned > > SubVectors, ArrayRef< int > SubVectorsMask, unsigned VF=0, function_ref< void(Value &, SmallVectorImpl< int > &, function_ref< Value (Value , Value *, ArrayRef< int >)>)> Action={})
	Finalize emission of the shuffles.
	~ShuffleInstructionBuilder ()

Additional Inherited Members
Protected Member Functions inherited from llvm::slpvectorizer::BaseShuffleAnalysis
	BaseShuffleAnalysis (Type *ScalarTy)
unsigned	getVF (Value *V) const
	V is expected to be a vectorized value.
Static Protected Member Functions inherited from llvm::slpvectorizer::BaseShuffleAnalysis
static bool	isIdentityMask (ArrayRef< int > Mask, const FixedVectorType *VecTy, bool IsStrict)
	Checks if the mask is an identity mask.
static void	combineMasks (unsigned LocalVF, SmallVectorImpl< int > &Mask, ArrayRef< int > ExtMask)
	Tries to combine 2 different masks into single one.
static bool	peekThroughShuffles (Value *&V, SmallVectorImpl< int > &Mask, bool SinglePermute)
	Looks through shuffles trying to reduce final number of shuffles in the code.
template<typename T, typename ShuffleBuilderTy, typename... Args>
static T	createShuffle (Value V1, Value V2, ArrayRef< int > Mask, ShuffleBuilderTy &Builder, Type *ScalarTy, bool ReVec, Args... Arguments)
	Smart shuffle instruction emission, walks through shuffles trees and tries to find the best matching vector for the actual shuffle instruction.
static void	transformMaskAfterShuffle (MutableArrayRef< int > CommonMask, ArrayRef< int > Mask)
	Transforms mask `CommonMask` per given `Mask` to make proper set after shuffle emission.
Protected Attributes inherited from llvm::slpvectorizer::BaseShuffleAnalysis
Type *	ScalarTy = nullptr

Detailed Description

Merges shuffle masks and emits final shuffle instruction, if required.

It supports shuffling of 2 input vectors. It implements lazy shuffles emission, when the actual shuffle instruction is generated only if this is actually required. Otherwise, the shuffle instruction emission is delayed till the end of the process, to reduce the number of emitted instructions and further analysis/transformations. The class also will look through the previously emitted shuffle instructions and properly mark indices in mask as undef. For example, given the code

%s1 = shufflevector <2 x ty> %0, poison, <1, 0>

%s2 = shufflevector <2 x ty> %1, poison, <1, 0>

and if need to emit shuffle of s1 and s2 with mask <1, 0, 3, 2>, it will look through s1 and s2 and emit

%res = shufflevector <2 x ty> %0, %1, <0, 1, 2, 3>

instead. If 2 operands are of different size, the smallest one will be resized and the mask recalculated properly. For example, given the code

%s1 = shufflevector <2 x ty> %0, poison, <1, 0, 1, 0>

%s2 = shufflevector <2 x ty> %1, poison, <1, 0, 1, 0>

and if need to emit shuffle of s1 and s2 with mask <1, 0, 5, 4>, it will look through s1 and s2 and emit

%res = shufflevector <2 x ty> %0, %1, <0, 1, 2, 3>

instead.

Definition at line 22148 of file SLPVectorizer.cpp.

Constructor & Destructor Documentation

◆ ShuffleInstructionBuilder()

slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::ShuffleInstructionBuilder	(	Type *	ScalarTy,
		IRBuilderBase &	Builder,
		BoUpSLP &	R )

inline

Definition at line 22285 of file SLPVectorizer.cpp.

References llvm::slpvectorizer::BaseShuffleAnalysis::BaseShuffleAnalysis(), slpvectorizer::BoUpSLP::BoUpSLP(), and llvm::slpvectorizer::BaseShuffleAnalysis::ScalarTy.

◆ ~ShuffleInstructionBuilder()

slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::~ShuffleInstructionBuilder ( )

inline

Definition at line 22701 of file SLPVectorizer.cpp.

References assert().

Member Function Documentation

◆ add() [1/4]

void slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::add	(	const TreeEntry &	E1,
		ArrayRef< int >	Mask )

inline

Adds single input vector (in form of tree entry) and the mask for its shuffling.

Definition at line 22466 of file SLPVectorizer.cpp.

References add(), and llvm::V1.

◆ add() [2/4]

void slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::add	(	const TreeEntry &	E1,
		const TreeEntry &	E2,
		ArrayRef< int >	Mask )

inline

Adds 2 input vectors (in form of tree entries) and the mask for their shuffling.

Definition at line 22459 of file SLPVectorizer.cpp.

References add(), and llvm::V1.

Referenced by add(), add(), and addOrdered().

◆ add() [3/4]

void slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::add	(	Value *	V1,
		ArrayRef< int >	Mask,
		bool	= false,
		TTI::VectorInstrContext	= TTI::VectorInstrContext::None )

inline

Adds another one input vector and the mask for the shuffling.

Definition at line 22505 of file SLPVectorizer.cpp.

References assert(), llvm::cast(), llvm::find(), llvm::slpvectorizer::BaseShuffleAnalysis::getVF(), llvm::isa(), llvm::TargetTransformInfo::None, llvm::PoisonMaskElem, llvm::slpvectorizer::BaseShuffleAnalysis::transformMaskAfterShuffle(), and llvm::V1.

◆ add() [4/4]

void slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::add	(	Value *	V1,
		Value *	V2,
		ArrayRef< int >	Mask )

inline

Adds 2 input vectors and the mask for their shuffling.

Definition at line 22471 of file SLPVectorizer.cpp.

References assert(), llvm::cast(), llvm::Value::getType(), llvm::slpvectorizer::BaseShuffleAnalysis::getVF(), llvm::isa(), llvm::PoisonMaskElem, llvm::slpvectorizer::BaseShuffleAnalysis::transformMaskAfterShuffle(), and llvm::V1.

◆ addOrdered()

void slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::addOrdered	(	Value *	V1,
		ArrayRef< unsigned >	Order )

inline

Adds another one input vector and the mask for the shuffling.

Definition at line 22559 of file SLPVectorizer.cpp.

References add(), llvm::slpvectorizer::inversePermutation(), and llvm::V1.

◆ adjustExtracts()

Value * slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::adjustExtracts	(	const TreeEntry *	E,
		MutableArrayRef< int >	Mask,
		ArrayRef< std::optional< TTI::ShuffleKind > >	ShuffleKinds,
		unsigned	NumParts,
		bool &	UseVecBaseAsInput )

inline

Adjusts extractelements after reusing them.

Definition at line 22289 of file SLPVectorizer.cpp.

References llvm::any_of(), assert(), llvm::cast(), llvm::count(), llvm::count_if(), llvm::ArrayRef< T >::empty(), llvm::ArrayRef< T >::front(), I, llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::isa(), llvm::PoisonMaskElem, and llvm::slpvectorizer::reorderScalars().

◆ createExtractedFieldsVector()

Value * slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::createExtractedFieldsVector	(	Value *	Src,
		unsigned	FieldWidth,
		ArrayRef< int >	Mask,
		const TreeEntry &	)

inline

Emits the gather of zero-extended sub-fields of width FieldWidth of the same wider integer scalar Src, permuted by Mask, as a bitcast to the field vector plus a permutation and an extension to the lane type.

Definition at line 22575 of file SLPVectorizer.cpp.

References assert(), llvm::dyn_cast(), llvm::IntegerType::get(), llvm::slpvectorizer::getWidenedType(), and I.

◆ createFreeze()

Value * slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::createFreeze ( Value * V )

inline

Definition at line 22592 of file SLPVectorizer.cpp.

◆ finalize()

Value * slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::finalize	(	ArrayRef< int >	ExtMask,
		ArrayRef< std::pair< const TreeEntry *, unsigned > >	SubVectors,
		ArrayRef< int >	SubVectorsMask,
		unsigned	VF = 0,
		function_ref< void(Value &, SmallVectorImpl< int > &, function_ref< Value (Value , Value , ArrayRef< int >)>)>	Action = {} )

inline

Finalize emission of the shuffles.

Parameters

Action the action (if any) to be performed before final applying of the ExtMask mask.

Definition at line 22596 of file SLPVectorizer.cpp.

◆ gather()

Value * slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::gather	(	ArrayRef< Value * >	VL,
		unsigned	MaskVF = 0,
		Value *	Root = nullptr,
		TTI::VectorInstrContext	= TTI::VectorInstrContext::None )

inline

Definition at line 22564 of file SLPVectorizer.cpp.

References llvm::TargetTransformInfo::None, llvm::slpvectorizer::BaseShuffleAnalysis::ScalarTy, and llvm::V1.

◆ needToDelay()

std::optional< Value * > slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::needToDelay	(	const TreeEntry *	E,
		ArrayRef< SmallVector< const TreeEntry * > >	Deps ) const

inline

Checks if the specified entry E needs to be delayed because of its dependency nodes.

Definition at line 22435 of file SLPVectorizer.cpp.

References llvm::all_of(), llvm::PoisonValue::get(), llvm::PointerType::getUnqual(), llvm::slpvectorizer::getWidenedType(), and llvm::slpvectorizer::BaseShuffleAnalysis::ScalarTy.

◆ resetForSameNode()

void slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::resetForSameNode ( )

inline

Reset the builder to handle perfect diamond match.

Definition at line 22452 of file SLPVectorizer.cpp.

The documentation for this class was generated from the following file:

lib/Transforms/Vectorize/SLPVectorizer.cpp

Public Member Functions

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ ShuffleInstructionBuilder()

◆ ~ShuffleInstructionBuilder()

Member Function Documentation

◆ add() [1/4]

◆ add() [2/4]

◆ add() [3/4]

◆ add() [4/4]

◆ addOrdered()

◆ adjustExtracts()

◆ createExtractedFieldsVector()

◆ createFreeze()

◆ finalize()

◆ gather()

◆ needToDelay()

◆ resetForSameNode()