LLVM 20.0.0git
|
#include "Transforms/Vectorize/VPlanTransforms.h"
Static Public Member Functions | |
static void | VPInstructionsToVPRecipes (VPlanPtr &Plan, function_ref< const InductionDescriptor *(PHINode *)> GetIntOrFpInductionDescriptor, ScalarEvolution &SE, const TargetLibraryInfo &TLI) |
Replaces the VPInstructions in Plan with corresponding widen recipes. | |
static bool | adjustFixedOrderRecurrences (VPlan &Plan, VPBuilder &Builder) |
Sink users of fixed-order recurrences after the recipe defining their previous value. | |
static void | clearReductionWrapFlags (VPlan &Plan) |
Clear NSW/NUW flags from reduction instructions if necessary. | |
static void | optimizeForVFAndUF (VPlan &Plan, ElementCount BestVF, unsigned BestUF, PredicatedScalarEvolution &PSE) |
Optimize Plan based on BestVF and BestUF . | |
static void | optimize (VPlan &Plan, ScalarEvolution &SE) |
Apply VPlan-to-VPlan optimizations to Plan , including induction recipe optimizations, dead recipe removal, replicate region optimizations and block merging. | |
static void | createAndOptimizeReplicateRegions (VPlan &Plan) |
Wrap predicated VPReplicateRecipes with a mask operand in an if-then region block and remove the mask operand. | |
static void | addActiveLaneMask (VPlan &Plan, bool UseActiveLaneMaskForControlFlow, bool DataAndControlFlowWithoutRuntimeCheck) |
Replace (ICMP_ULE, wide canonical IV, backedge-taken-count) checks with an (active-lane-mask recipe, wide canonical IV, trip-count). | |
static void | truncateToMinimalBitwidths (VPlan &Plan, const MapVector< Instruction *, uint64_t > &MinBWs, LLVMContext &Ctx) |
Insert truncates and extends for any truncated recipe. | |
static void | dropPoisonGeneratingRecipes (VPlan &Plan, function_ref< bool(BasicBlock *)> BlockNeedsPredication) |
Drop poison flags from recipes that may generate a poison value that is used after vectorization, even when their operands are not poison. | |
static bool | tryAddExplicitVectorLength (VPlan &Plan) |
Add a VPEVLBasedIVPHIRecipe and related recipes to Plan and replaces all uses except the canonical IV increment of VPCanonicalIVPHIRecipe with a VPEVLBasedIVPHIRecipe. | |
Definition at line 29 of file VPlanTransforms.h.
|
static |
Replace (ICMP_ULE, wide canonical IV, backedge-taken-count) checks with an (active-lane-mask recipe, wide canonical IV, trip-count).
If UseActiveLaneMaskForControlFlow
is true, introduce an VPActiveLaneMaskPHIRecipe. If DataAndControlFlowWithoutRuntimeCheck
is true, no minimum-iteration runtime check will be created (during skeleton creation) and instead it is handled using active-lane-mask. DataAndControlFlowWithoutRuntimeCheck
implies UseActiveLaneMaskForControlFlow
.
Definition at line 1375 of file VPlanTransforms.cpp.
References llvm::VPInstruction::ActiveLaneMask, addVPLaneMaskPhiAndUpdateExitBranch(), assert(), B, collectAllHeaderMasks(), llvm::DataAndControlFlowWithoutRuntimeCheck, llvm::find_if(), llvm::VPlan::getCanonicalIV(), llvm::VPBuilder::getToInsertAfter(), llvm::VPlan::getTripCount(), and llvm::VPValue::users().
Sink users of fixed-order recurrences after the recipe defining their previous value.
Then introduce FirstOrderRecurrenceSplice VPInstructions to combine the value from the recurrence phis and previous values. The current implementation assumes all users can be sunk after the previous value, which is enforced by earlier legality checks.
Plan
is not valid. Definition at line 818 of file VPlanTransforms.cpp.
References assert(), llvm::VPBuilder::createNaryOp(), llvm::VPInstruction::ExtractFromEnd, llvm::VPInstruction::FirstOrderRecurrenceSplice, llvm::VPlan::getCanonicalIV(), llvm::VPRegionBlock::getEntry(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPBasicBlock::getFirstNonPhi(), llvm::ilist_node_impl< OptionsT >::getIterator(), llvm::VPRecipeBase::getParent(), llvm::VPCanonicalIVPHIRecipe::getScalarType(), llvm::VPBlockBase::getSingleSuccessor(), llvm::VPBasicBlock::getTerminator(), llvm::VPlan::getVectorLoopRegion(), llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::VPBasicBlock::phis(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::DominatorTreeBase< NodeT, IsPostDom >::recalculate(), llvm::VPBuilder::setInsertPoint(), and sinkRecurrenceUsersAfterPrevious().
|
static |
Clear NSW/NUW flags from reduction instructions if necessary.
Definition at line 971 of file VPlanTransforms.cpp.
References llvm::Add, collectUsersRecursively(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::RecurrenceDescriptor::getRecurrenceKind(), llvm::VPlan::getVectorLoopRegion(), llvm::Mul, and llvm::VPBasicBlock::phis().
|
static |
Wrap predicated VPReplicateRecipes with a mask operand in an if-then region block and remove the mask operand.
Optimize the created regions by iteratively sinking scalar operands into the region, followed by merging regions until no improvements are remaining.
Definition at line 402 of file VPlanTransforms.cpp.
References addReplicateRegions(), mergeBlocksIntoPredecessors(), mergeReplicateRegionsIntoSuccessors(), and sinkScalarOperands().
Referenced by optimize().
|
static |
Drop poison flags from recipes that may generate a poison value that is used after vectorization, even when their operands are not poison.
Those recipes meet the following conditions:
BlockNeedsPredication
to check if a block needs predicating. TODO: Replace BlockNeedsPredication callback with retrieving info from VPlan directly. Definition at line 1528 of file VPlanTransforms.cpp.
References llvm::SmallVectorBase< Size_T >::empty(), for(), if(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().
|
static |
Apply VPlan-to-VPlan optimizations to Plan
, including induction recipe optimizations, dead recipe removal, replicate region optimizations and block merging.
Definition at line 1212 of file VPlanTransforms.cpp.
References createAndOptimizeReplicateRegions(), llvm::ScalarEvolution::getContext(), legalizeAndOptimizeInductions(), mergeBlocksIntoPredecessors(), removeDeadRecipes(), removeRedundantCanonicalIVs(), removeRedundantExpandSCEVRecipes(), removeRedundantInductionCasts(), and simplifyRecipes().
Referenced by llvm::LoopVectorizationPlanner::buildVPlans().
|
static |
Optimize Plan
based on BestVF
and BestUF
.
This may restrict the resulting plan to BestVF
and BestUF
.
Definition at line 672 of file VPlanTransforms.cpp.
References llvm::VPBasicBlock::appendRecipe(), assert(), llvm::VPBasicBlock::back(), llvm::VPInstruction::BranchOnCond, llvm::CallingConv::C, llvm::createTripCountSCEV(), llvm::VPlan::getCanonicalIV(), llvm::ScalarEvolution::getContext(), llvm::ScalarEvolution::getElementCount(), llvm::VPBlockBase::getExitingBasicBlock(), llvm::VPValue::getLiveInIRValue(), llvm::VPlan::getOrAddLiveIn(), llvm::PredicatedScalarEvolution::getSE(), llvm::VPHeaderPHIRecipe::getStartValue(), llvm::ConstantInt::getTrue(), llvm::SCEV::getType(), llvm::Value::getType(), llvm::VPlan::getVectorLoopRegion(), llvm::VPlan::hasUF(), llvm::VPlan::hasVF(), llvm::CmpInst::ICMP_ULE, llvm::ScalarEvolution::isKnownPredicate(), llvm::SCEV::isZero(), llvm::PatternMatch::m_Not(), llvm::PatternMatch::match(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::multiplyCoefficientBy(), recursivelyDeleteDeadRecipes(), llvm::VPlan::setUF(), and llvm::VPlan::setVF().
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
static |
Insert truncates and extends for any truncated recipe.
Redundant casts will be folded later.
Definition at line 1080 of file VPlanTransforms.cpp.
References llvm::all_of(), llvm::VPBasicBlock::appendRecipe(), assert(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::contains(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains(), llvm::IntegerType::get(), llvm::VPlan::getCanonicalIV(), llvm::VPlan::getEntry(), llvm::Type::getScalarSizeInBits(), llvm::VPCanonicalIVPHIRecipe::getScalarType(), llvm::VPValue::getUnderlyingValue(), llvm::VPlan::getVectorLoopRegion(), Idx, llvm::VPTypeAnalysis::inferScalarType(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::insert(), llvm::VPRecipeBase::insertBefore(), llvm::isa(), llvm::Type::isIntegerTy(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::lookup(), llvm::make_early_inc_range(), llvm::PatternMatch::match(), llvm::VPValue::replaceAllUsesWith(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::size(), and llvm::vp_depth_first_deep().
Add a VPEVLBasedIVPHIRecipe and related recipes to Plan
and replaces all uses except the canonical IV increment of VPCanonicalIVPHIRecipe with a VPEVLBasedIVPHIRecipe.
VPCanonicalIVPHIRecipe is only used to control the loop after this transformation.
VPCanonicalIVPHIRecipe is used only for loop iterations counting after this transformation.
The function uses the following definitions: StartV is the canonical induction start value.
The function adds the following recipes:
vector.ph: ...
vector.body: ... EVLPhi = EXPLICIT-VECTOR-LENGTH-BASED-IV-PHI [ StartV, vector.ph ], [ NextEVLIV, vector.body ] VPEVL = EXPLICIT-VECTOR-LENGTH EVLPhi, original TC ... NextEVLIV = add IVSize (cast i32 VPEVVL to IVSize), EVLPhi ...
Definition at line 1430 of file VPlanTransforms.cpp.
References llvm::any_of(), assert(), collectAllHeaderMasks(), collectUsersRecursively(), llvm::VPInstruction::ExplicitVectorLength, llvm::VPlan::getCanonicalIV(), llvm::VPRecipeBase::getDebugLoc(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPDef::getNumDefinedValues(), llvm::VPlan::getTripCount(), llvm::VPlan::getVectorLoopRegion(), llvm::VPDef::getVPSingleValue(), llvm::VPRecipeBase::insertBefore(), llvm_unreachable, recursivelyDeleteDeadRecipes(), llvm::VPValue::replaceAllUsesWith(), and llvm::VPlan::setUF().
|
static |
Replaces the VPInstructions in Plan
with corresponding widen recipes.
Definition at line 30 of file VPlanTransforms.cpp.
References assert(), GEP, llvm::VPDef::getNumDefinedValues(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::VPValue::getUnderlyingValue(), llvm::getVectorIntrinsicIDForCall(), llvm::VPDef::getVPSingleValue(), II, llvm::VPRecipeBase::insertBefore(), llvm::make_early_inc_range(), llvm::make_range(), llvm::User::operands(), and llvm::VPValue::replaceAllUsesWith().