#include "Transforms/Vectorize/VPlanTransforms.h"

Static Public Member Functions
static void	VPInstructionsToVPRecipes (VPlanPtr &Plan, function_ref< const InductionDescriptor (PHINode )> GetIntOrFpInductionDescriptor, ScalarEvolution &SE, const TargetLibraryInfo &TLI)
	Replaces the VPInstructions in `Plan` with corresponding widen recipes.

static bool	adjustFixedOrderRecurrences (VPlan &Plan, VPBuilder &Builder)
	Sink users of fixed-order recurrences after the recipe defining their previous value.

static void	clearReductionWrapFlags (VPlan &Plan)
	Clear NSW/NUW flags from reduction instructions if necessary.

static void	optimizeForVFAndUF (VPlan &Plan, ElementCount BestVF, unsigned BestUF, PredicatedScalarEvolution &PSE)
	Optimize `Plan` based on `BestVF` and `BestUF`.

static void	optimize (VPlan &Plan, ScalarEvolution &SE)
	Apply VPlan-to-VPlan optimizations to `Plan`, including induction recipe optimizations, dead recipe removal, replicate region optimizations and block merging.

static void	createAndOptimizeReplicateRegions (VPlan &Plan)
	Wrap predicated VPReplicateRecipes with a mask operand in an if-then region block and remove the mask operand.

static void	addActiveLaneMask (VPlan &Plan, bool UseActiveLaneMaskForControlFlow, bool DataAndControlFlowWithoutRuntimeCheck)
	Replace (ICMP_ULE, wide canonical IV, backedge-taken-count) checks with an (active-lane-mask recipe, wide canonical IV, trip-count).

Detailed Description

Definition at line 31 of file VPlanTransforms.h.

Member Function Documentation

◆ addActiveLaneMask()

void VPlanTransforms::addActiveLaneMask	(	VPlan &	Plan,
		bool	UseActiveLaneMaskForControlFlow,
		bool	DataAndControlFlowWithoutRuntimeCheck
	)

static

Replace (ICMP_ULE, wide canonical IV, backedge-taken-count) checks with an (active-lane-mask recipe, wide canonical IV, trip-count).

If UseActiveLaneMaskForControlFlow is true, introduce an VPActiveLaneMaskPHIRecipe. If DataAndControlFlowWithoutRuntimeCheck is true, no minimum-iteration runtime check will be created (during skeleton creation) and instead it is handled using active-lane-mask. DataAndControlFlowWithoutRuntimeCheck implies UseActiveLaneMaskForControlFlow.

Definition at line 964 of file VPlanTransforms.cpp.

References llvm::VPInstruction::ActiveLaneMask, addVPLaneMaskPhiAndUpdateExitBranch(), assert(), llvm::DataAndControlFlowWithoutRuntimeCheck, llvm::find_if(), llvm::VPlan::getCanonicalIV(), llvm::VPlan::getOrCreateBackedgeTakenCount(), llvm::VPlan::getTripCount(), llvm::VPDef::getVPSingleValue(), llvm::CmpInst::ICMP_ULE, llvm::VPRecipeBase::insertAfter(), and llvm::VPValue::users().

◆ adjustFixedOrderRecurrences()

bool VPlanTransforms::adjustFixedOrderRecurrences	(	VPlan &	Plan,
		VPBuilder &	Builder
	)

static

Sink users of fixed-order recurrences after the recipe defining their previous value.

Then introduce FirstOrderRecurrenceSplice VPInstructions to combine the value from the recurrence phis and previous values. The current implementation assumes all users can be sunk after the previous value, which is enforced by earlier legality checks.

Returns: true if all users of fixed-order recurrences could be re-arranged as needed or false if it is not possible. In the latter case, Plan is not valid.

Definition at line 725 of file VPlanTransforms.cpp.

References assert(), Builder, llvm::VPInstruction::FirstOrderRecurrenceSplice, llvm::VPRegionBlock::getEntry(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPBasicBlock::getFirstNonPhi(), llvm::ilist_node_impl< OptionsT >::getIterator(), llvm::VPRecipeBase::getParent(), llvm::VPlan::getVectorLoopRegion(), llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::VPBasicBlock::phis(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::DominatorTreeBase< NodeT, IsPostDom >::recalculate(), and sinkRecurrenceUsersAfterPrevious().

◆ clearReductionWrapFlags()

void VPlanTransforms::clearReductionWrapFlags ( VPlan & Plan )

static

Clear NSW/NUW flags from reduction instructions if necessary.

Definition at line 771 of file VPlanTransforms.cpp.

References llvm::Add, llvm::VPValue::getDefiningRecipe(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::RecurrenceDescriptor::getRecurrenceKind(), llvm::VPlan::getVectorLoopRegion(), I, llvm::SetVector< T, Vector, Set, N >::insert(), llvm::Mul, llvm::VPBasicBlock::phis(), llvm::SetVector< T, Vector, Set, N >::size(), and llvm::VPValue::users().

◆ createAndOptimizeReplicateRegions()

void VPlanTransforms::createAndOptimizeReplicateRegions ( VPlan & Plan )

static

Wrap predicated VPReplicateRecipes with a mask operand in an if-then region block and remove the mask operand.

Optimize the created regions by iteratively sinking scalar operands into the region, followed by merging regions until no improvements are remaining.

Definition at line 372 of file VPlanTransforms.cpp.

References addReplicateRegions(), mergeReplicateRegionsIntoSuccessors(), and sinkScalarOperands().

Referenced by optimize().

◆ optimize()

void VPlanTransforms::optimize	(	VPlan &	Plan,
		ScalarEvolution &	SE
	)

static

Apply VPlan-to-VPlan optimizations to Plan, including induction recipe optimizations, dead recipe removal, replicate region optimizations and block merging.

Definition at line 846 of file VPlanTransforms.cpp.

References createAndOptimizeReplicateRegions(), and simplifyRecipes().

◆ optimizeForVFAndUF()

void VPlanTransforms::optimizeForVFAndUF	(	VPlan &	Plan,
		ElementCount	BestVF,
		unsigned	BestUF,
		PredicatedScalarEvolution &	PSE
	)

static

Optimize Plan based on BestVF and BestUF.

This may restrict the resulting plan to BestVF and BestUF.

Definition at line 584 of file VPlanTransforms.cpp.

References llvm::VPBasicBlock::appendRecipe(), assert(), llvm::VPBasicBlock::back(), llvm::VPInstruction::BranchOnCond, llvm::VPInstruction::BranchOnCount, llvm::CallingConv::C, canSimplifyBranchOnCond(), llvm::createTripCountSCEV(), llvm::VPlan::getCanonicalIV(), llvm::ScalarEvolution::getConstant(), llvm::ScalarEvolution::getContext(), llvm::VPBlockBase::getExitingBasicBlock(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::VPValue::getLiveInIRValue(), llvm::PredicatedScalarEvolution::getSE(), llvm::VPHeaderPHIRecipe::getStartValue(), llvm::ConstantInt::getTrue(), llvm::SCEV::getType(), llvm::Value::getType(), llvm::VPlan::getVectorLoopRegion(), llvm::VPlan::getVPValueOrAddLiveIn(), llvm::VPlan::hasUF(), llvm::VPlan::hasVF(), llvm::CmpInst::ICMP_ULE, llvm::ScalarEvolution::isKnownPredicate(), llvm::SCEV::isZero(), llvm::VPlan::setUF(), and llvm::VPlan::setVF().

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ VPInstructionsToVPRecipes()

void VPlanTransforms::VPInstructionsToVPRecipes	(	VPlanPtr &	Plan,
		function_ref< const InductionDescriptor (PHINode )>	GetIntOrFpInductionDescriptor,
		ScalarEvolution &	SE,
		const TargetLibraryInfo &	TLI
	)

static

Replaces the VPInstructions in Plan with corresponding widen recipes.

Definition at line 30 of file VPlanTransforms.cpp.

References assert(), llvm::drop_end(), GEP, llvm::VPDef::getNumDefinedValues(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::VPValue::getUnderlyingValue(), llvm::getVectorIntrinsicIDForCall(), llvm::VPDef::getVPSingleValue(), llvm::VPRecipeBase::insertBefore(), llvm::make_early_inc_range(), llvm::make_range(), llvm::User::operands(), and llvm::VPValue::replaceAllUsesWith().

The documentation for this struct was generated from the following files:

lib/Transforms/Vectorize/VPlanTransforms.h
lib/Transforms/Vectorize/VPlanTransforms.cpp

Static Public Member Functions

Detailed Description

Member Function Documentation

◆ addActiveLaneMask()

◆ adjustFixedOrderRecurrences()

◆ clearReductionWrapFlags()

◆ createAndOptimizeReplicateRegions()

◆ optimize()

◆ optimizeForVFAndUF()

◆ VPInstructionsToVPRecipes()