LLVM 20.0.0git
Static Public Member Functions | List of all members
llvm::VPlanTransforms Struct Reference

#include "Transforms/Vectorize/VPlanTransforms.h"

Static Public Member Functions

static void VPInstructionsToVPRecipes (VPlanPtr &Plan, function_ref< const InductionDescriptor *(PHINode *)> GetIntOrFpInductionDescriptor, ScalarEvolution &SE, const TargetLibraryInfo &TLI)
 Replaces the VPInstructions in Plan with corresponding widen recipes.
 
static bool adjustFixedOrderRecurrences (VPlan &Plan, VPBuilder &Builder)
 Sink users of fixed-order recurrences after the recipe defining their previous value.
 
static void clearReductionWrapFlags (VPlan &Plan)
 Clear NSW/NUW flags from reduction instructions if necessary.
 
static void optimizeForVFAndUF (VPlan &Plan, ElementCount BestVF, unsigned BestUF, PredicatedScalarEvolution &PSE)
 Optimize Plan based on BestVF and BestUF.
 
static void optimize (VPlan &Plan, ScalarEvolution &SE)
 Apply VPlan-to-VPlan optimizations to Plan, including induction recipe optimizations, dead recipe removal, replicate region optimizations and block merging.
 
static void createAndOptimizeReplicateRegions (VPlan &Plan)
 Wrap predicated VPReplicateRecipes with a mask operand in an if-then region block and remove the mask operand.
 
static void addActiveLaneMask (VPlan &Plan, bool UseActiveLaneMaskForControlFlow, bool DataAndControlFlowWithoutRuntimeCheck)
 Replace (ICMP_ULE, wide canonical IV, backedge-taken-count) checks with an (active-lane-mask recipe, wide canonical IV, trip-count).
 
static void truncateToMinimalBitwidths (VPlan &Plan, const MapVector< Instruction *, uint64_t > &MinBWs, LLVMContext &Ctx)
 Insert truncates and extends for any truncated recipe.
 
static void dropPoisonGeneratingRecipes (VPlan &Plan, function_ref< bool(BasicBlock *)> BlockNeedsPredication)
 Drop poison flags from recipes that may generate a poison value that is used after vectorization, even when their operands are not poison.
 
static bool tryAddExplicitVectorLength (VPlan &Plan)
 Add a VPEVLBasedIVPHIRecipe and related recipes to Plan and replaces all uses except the canonical IV increment of VPCanonicalIVPHIRecipe with a VPEVLBasedIVPHIRecipe.
 

Detailed Description

Definition at line 29 of file VPlanTransforms.h.

Member Function Documentation

◆ addActiveLaneMask()

void VPlanTransforms::addActiveLaneMask ( VPlan Plan,
bool  UseActiveLaneMaskForControlFlow,
bool  DataAndControlFlowWithoutRuntimeCheck 
)
static

Replace (ICMP_ULE, wide canonical IV, backedge-taken-count) checks with an (active-lane-mask recipe, wide canonical IV, trip-count).

If UseActiveLaneMaskForControlFlow is true, introduce an VPActiveLaneMaskPHIRecipe. If DataAndControlFlowWithoutRuntimeCheck is true, no minimum-iteration runtime check will be created (during skeleton creation) and instead it is handled using active-lane-mask. DataAndControlFlowWithoutRuntimeCheck implies UseActiveLaneMaskForControlFlow.

Definition at line 1281 of file VPlanTransforms.cpp.

References llvm::VPInstruction::ActiveLaneMask, addVPLaneMaskPhiAndUpdateExitBranch(), assert(), B, collectAllHeaderMasks(), llvm::DataAndControlFlowWithoutRuntimeCheck, llvm::find_if(), llvm::VPlan::getCanonicalIV(), llvm::VPBuilder::getToInsertAfter(), llvm::VPlan::getTripCount(), and llvm::VPValue::users().

◆ adjustFixedOrderRecurrences()

bool VPlanTransforms::adjustFixedOrderRecurrences ( VPlan Plan,
VPBuilder Builder 
)
static

Sink users of fixed-order recurrences after the recipe defining their previous value.

Then introduce FirstOrderRecurrenceSplice VPInstructions to combine the value from the recurrence phis and previous values. The current implementation assumes all users can be sunk after the previous value, which is enforced by earlier legality checks.

Returns
true if all users of fixed-order recurrences could be re-arranged as needed or false if it is not possible. In the latter case, Plan is not valid.

Definition at line 818 of file VPlanTransforms.cpp.

References assert(), llvm::VPBuilder::createNaryOp(), llvm::VPInstruction::FirstOrderRecurrenceSplice, llvm::VPRegionBlock::getEntry(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPBasicBlock::getFirstNonPhi(), llvm::ilist_node_impl< OptionsT >::getIterator(), llvm::VPRecipeBase::getParent(), llvm::VPlan::getVectorLoopRegion(), llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::VPBasicBlock::phis(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::DominatorTreeBase< NodeT, IsPostDom >::recalculate(), llvm::VPBuilder::setInsertPoint(), and sinkRecurrenceUsersAfterPrevious().

◆ clearReductionWrapFlags()

void VPlanTransforms::clearReductionWrapFlags ( VPlan Plan)
static

◆ createAndOptimizeReplicateRegions()

void VPlanTransforms::createAndOptimizeReplicateRegions ( VPlan Plan)
static

Wrap predicated VPReplicateRecipes with a mask operand in an if-then region block and remove the mask operand.

Optimize the created regions by iteratively sinking scalar operands into the region, followed by merging regions until no improvements are remaining.

Definition at line 402 of file VPlanTransforms.cpp.

References addReplicateRegions(), mergeBlocksIntoPredecessors(), mergeReplicateRegionsIntoSuccessors(), and sinkScalarOperands().

Referenced by optimize().

◆ dropPoisonGeneratingRecipes()

void VPlanTransforms::dropPoisonGeneratingRecipes ( VPlan Plan,
function_ref< bool(BasicBlock *)>  BlockNeedsPredication 
)
static

Drop poison flags from recipes that may generate a poison value that is used after vectorization, even when their operands are not poison.

Those recipes meet the following conditions:

  • Contribute to the address computation of a recipe generating a widen memory load/store (VPWidenMemoryInstructionRecipe or VPInterleaveRecipe).
  • Such a widen memory load/store has at least one underlying Instruction that is in a basic block that needs predication and after vectorization the generated instruction won't be predicated. Uses BlockNeedsPredication to check if a block needs predicating. TODO: Replace BlockNeedsPredication callback with retrieving info from VPlan directly.

Definition at line 1434 of file VPlanTransforms.cpp.

References llvm::SmallVectorBase< Size_T >::empty(), for(), if(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().

◆ optimize()

void VPlanTransforms::optimize ( VPlan Plan,
ScalarEvolution SE 
)
static

Apply VPlan-to-VPlan optimizations to Plan, including induction recipe optimizations, dead recipe removal, replicate region optimizations and block merging.

Definition at line 1118 of file VPlanTransforms.cpp.

References createAndOptimizeReplicateRegions(), llvm::ScalarEvolution::getContext(), legalizeAndOptimizeInductions(), mergeBlocksIntoPredecessors(), removeDeadRecipes(), removeRedundantCanonicalIVs(), removeRedundantExpandSCEVRecipes(), removeRedundantInductionCasts(), and simplifyRecipes().

Referenced by llvm::LoopVectorizationPlanner::buildVPlans().

◆ optimizeForVFAndUF()

void VPlanTransforms::optimizeForVFAndUF ( VPlan Plan,
ElementCount  BestVF,
unsigned  BestUF,
PredicatedScalarEvolution PSE 
)
static

◆ truncateToMinimalBitwidths()

void VPlanTransforms::truncateToMinimalBitwidths ( VPlan Plan,
const MapVector< Instruction *, uint64_t > &  MinBWs,
LLVMContext Ctx 
)
static

◆ tryAddExplicitVectorLength()

bool VPlanTransforms::tryAddExplicitVectorLength ( VPlan Plan)
static

Add a VPEVLBasedIVPHIRecipe and related recipes to Plan and replaces all uses except the canonical IV increment of VPCanonicalIVPHIRecipe with a VPEVLBasedIVPHIRecipe.

VPCanonicalIVPHIRecipe is only used to control the loop after this transformation.

Returns
true if the transformation succeeds, or false if it doesn't.

VPCanonicalIVPHIRecipe is used only for loop iterations counting after this transformation.

The function uses the following definitions: StartV is the canonical induction start value.

The function adds the following recipes:

vector.ph: ...

vector.body: ... EVLPhi = EXPLICIT-VECTOR-LENGTH-BASED-IV-PHI [ StartV, vector.ph ], [ NextEVLIV, vector.body ] VPEVL = EXPLICIT-VECTOR-LENGTH EVLPhi, original TC ... NextEVLIV = add IVSize (cast i32 VPEVVL to IVSize), EVLPhi ...

Definition at line 1336 of file VPlanTransforms.cpp.

References llvm::any_of(), assert(), collectAllHeaderMasks(), collectUsersRecursively(), llvm::VPInstruction::ExplicitVectorLength, llvm::VPlan::getCanonicalIV(), llvm::VPRecipeBase::getDebugLoc(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPDef::getNumDefinedValues(), llvm::VPlan::getTripCount(), llvm::VPlan::getVectorLoopRegion(), llvm::VPDef::getVPSingleValue(), llvm::VPRecipeBase::insertBefore(), llvm_unreachable, recursivelyDeleteDeadRecipes(), llvm::VPValue::replaceAllUsesWith(), and llvm::VPlan::setUF().

◆ VPInstructionsToVPRecipes()

void VPlanTransforms::VPInstructionsToVPRecipes ( VPlanPtr Plan,
function_ref< const InductionDescriptor *(PHINode *)>  GetIntOrFpInductionDescriptor,
ScalarEvolution SE,
const TargetLibraryInfo TLI 
)
static

The documentation for this struct was generated from the following files: