Functions | |
| bool | onlyFirstLaneUsed (const VPValue *Def) |
Returns true if only the first lane of Def is used. | |
| bool | onlyFirstPartUsed (const VPValue *Def) |
Returns true if only the first part of Def is used. | |
| bool | onlyScalarValuesUsed (const VPValue *Def) |
Returns true if only scalar values of Def are used by all users. | |
| VPValue * | getOrCreateVPValueForSCEVExpr (VPlan &Plan, const SCEV *Expr) |
Get or create a VPValue that corresponds to the expansion of Expr. | |
| const SCEV * | getSCEVExprForVPValue (const VPValue *V, PredicatedScalarEvolution &PSE, const Loop *L=nullptr) |
Return the SCEV expression for V. | |
| bool | isAddressSCEVForCost (const SCEV *Addr, ScalarEvolution &SE, const Loop *L) |
Returns true if Addr is an address SCEV that can be passed to TTI::getAddressComputationCost, i.e. | |
| bool | isSingleScalar (const VPValue *VPV) |
Returns true if VPV is a single scalar, either because it produces the same value for all lanes or only has its first lane used. | |
| bool | isHeaderMask (const VPValue *V, const VPlan &Plan) |
Return true if V is a header mask in Plan. | |
| bool | isUniformAcrossVFsAndUFs (VPValue *V) |
Checks if V is uniform across all VF lanes and UF parts. | |
| VPBasicBlock * | getFirstLoopHeader (VPlan &Plan, VPDominatorTree &VPDT) |
| Returns the header block of the first, top-level loop, or null if none exist. | |
| unsigned | getVFScaleFactor (VPRecipeBase *R) |
| Get the VF scaling factor applied to the recipe's output, if the recipe has one. | |
| LLVM_ABI_FOR_TEST std::optional< VPValue * > | getRecipesForUncountableExit (VPlan &Plan, SmallVectorImpl< VPRecipeBase * > &Recipes, SmallVectorImpl< VPRecipeBase * > &GEPs) |
| Returns the VPValue representing the uncountable exit comparison used by AnyOf if the recipes it depends on can be traced back to live-ins and the addresses (in GEP/PtrAdd form) of any (non-masked) load used in generating the values for the comparison. | |
| std::optional< MemoryLocation > | getMemoryLocation (const VPRecipeBase &R) |
Return a MemoryLocation for R with noalias metadata populated from R, if the recipe is supported and std::nullopt otherwise. | |
| VPIRFlags | getFlagsFromIndDesc (const InductionDescriptor &ID) |
Extracts and returns NoWrap and FastMath flags from the induction binop in ID. | |
| template<typename PredT> | |
| VPRecipeBase * | findRecipe (VPValue *Start, PredT Pred) |
Search Start's users for a recipe satisfying Pred, looking through recipes with definitions. | |
| template<typename MatchT> | |
| static VPRecipeBase * | findUserOf (VPValue *V, const MatchT &P) |
If V is used by a recipe matching pattern P, return it. | |
| template<unsigned Opcode> | |
| static VPInstruction * | findUserOf (VPValue *V) |
If V is used by a VPInstruction with Opcode, return it. | |
| VPInstruction * | findComputeReductionResult (VPReductionPHIRecipe *PhiR) |
Find the ComputeReductionResult recipe for PhiR, looking through selects inserted for predicated reductions or tail folding. | |
| VPSingleDefRecipe * | findHeaderMask (VPlan &Plan) |
| Collect the header mask with the pattern: (ICMP_ULE, WideCanonicalIV, backedge-taken-count) TODO: Introduce explicit recipe for header-mask instead of searching the header-mask pattern manually. | |
Function Documentation
◆ findComputeReductionResult()
| VPInstruction * llvm::vputils::findComputeReductionResult | ( | VPReductionPHIRecipe * | PhiR | ) |
Find the ComputeReductionResult recipe for PhiR, looking through selects inserted for predicated reductions or tail folding.
Definition at line 655 of file VPlanUtils.cpp.
References llvm::cast(), findUserOf(), llvm::VPHeaderPHIRecipe::getBackedgeValue(), llvm::PatternMatch::m_Select(), and llvm::VPlanPatternMatch::m_VPValue().
Referenced by llvm::VPlanTransforms::createPartialReductions(), and llvm::VPlanTransforms::handleMaxMinNumReductions().
◆ findHeaderMask()
| VPSingleDefRecipe * llvm::vputils::findHeaderMask | ( | VPlan & | Plan | ) |
Collect the header mask with the pattern: (ICMP_ULE, WideCanonicalIV, backedge-taken-count) TODO: Introduce explicit recipe for header-mask instead of searching the header-mask pattern manually.
Definition at line 571 of file VPlanUtils.cpp.
References assert(), llvm::cast(), llvm::count_if(), llvm::dyn_cast(), llvm::iterator_range< IteratorT >::end(), llvm::find_if(), llvm::VPRegionBlock::getCanonicalIV(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPlan::getVectorLoopRegion(), llvm::IsaPred, isHeaderMask(), llvm::VPBasicBlock::phis(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), and llvm::VPValue::users().
Referenced by llvm::VPlanTransforms::addActiveLaneMask(), fixupVFUsersForEVL(), llvm::VPlanTransforms::handleFindLastReductions(), llvm::VPlanTransforms::optimizeFindIVReductions(), and reassociateHeaderMask().
◆ findRecipe()
|
inline |
Search Start's users for a recipe satisfying Pred, looking through recipes with definitions.
Definition at line 111 of file VPlanUtils.h.
References llvm::cast(), llvm::VPValue::getDefiningRecipe(), I, llvm::SetVector< T, Vector, Set, N >::insert(), llvm::SetVector< T, Vector, Set, N >::size(), and llvm::VPValue::users().
Referenced by findFindIVSelect(), fixReductionScalarResumeWhenVectorizingEpilog(), llvm::VPlanTransforms::optimizeFindIVReductions(), and preparePlanForEpilogueVectorLoop().
◆ findUserOf() [1/2]
|
static |
If V is used by a VPInstruction with Opcode, return it.
Otherwise return nullptr.
Definition at line 139 of file VPlanUtils.h.
References llvm::cast_or_null(), findUserOf(), and llvm::VPlanPatternMatch::m_VPInstruction().
◆ findUserOf() [2/2]
|
static |
If V is used by a recipe matching pattern P, return it.
Otherwise return nullptr;
Definition at line 132 of file VPlanUtils.h.
References llvm::cast(), llvm::find_if(), llvm::PatternMatch::match_fn(), and P.
Referenced by findComputeReductionResult(), findUserOf(), llvm::VPlanTransforms::handleFindLastReductions(), and llvm::VPlanTransforms::handleMultiUseReductions().
◆ getFirstLoopHeader()
| VPBasicBlock * llvm::vputils::getFirstLoopHeader | ( | VPlan & | Plan, |
| VPDominatorTree & | VPDT ) |
Returns the header block of the first, top-level loop, or null if none exist.
Definition at line 451 of file VPlanUtils.cpp.
References llvm::cast(), llvm::find_if(), llvm::VPlan::getEntry(), I, llvm::VPBlockUtils::isHeader(), and llvm::vp_depth_first_shallow().
Referenced by llvm::VPlan::execute(), llvm::LoopVectorizationPlanner::executePlan(), and llvm::InnerLoopVectorizer::fixVectorizedLoop().
◆ getFlagsFromIndDesc()
|
inline |
Extracts and returns NoWrap and FastMath flags from the induction binop in ID.
Definition at line 94 of file VPlanUtils.h.
References assert(), llvm::dyn_cast_if_present(), llvm::InductionDescriptor::IK_FpInduction, and llvm::InductionDescriptor::IK_IntInduction.
Referenced by createWidenInductionRecipe().
◆ getMemoryLocation()
| std::optional< MemoryLocation > llvm::vputils::getMemoryLocation | ( | const VPRecipeBase & | R | ) |
Return a MemoryLocation for R with noalias metadata populated from R, if the recipe is supported and std::nullopt otherwise.
The pointer of the location is conservatively set to nullptr.
Definition at line 640 of file VPlanUtils.cpp.
References llvm::dyn_cast().
Referenced by canSinkStoreWithNoAliasCheck(), llvm::VPlanTransforms::hoistInvariantLoads(), and llvm::VPlanTransforms::hoistPredicatedLoads().
◆ getOrCreateVPValueForSCEVExpr()
Get or create a VPValue that corresponds to the expansion of Expr.
If Expr is a SCEVConstant or SCEVUnknown, return a VPValue wrapping the live-in value. Otherwise return a VPExpandSCEVRecipe to expand Expr. If Plan's pre-header already contains a recipe expanding Expr, return it. If not, create a new one.
Definition at line 38 of file VPlanUtils.cpp.
References llvm::VPBasicBlock::appendRecipe(), llvm::dyn_cast(), llvm::VPlan::getEntry(), llvm::VPlan::getOrAddLiveIn(), and llvm::isa().
Referenced by addInitialSkeleton(), createWidenInductionRecipe(), and llvm::VPlanTransforms::replaceSymbolicStrides().
◆ getRecipesForUncountableExit()
| std::optional< VPValue * > llvm::vputils::getRecipesForUncountableExit | ( | VPlan & | Plan, |
| SmallVectorImpl< VPRecipeBase * > & | Recipes, | ||
| SmallVectorImpl< VPRecipeBase * > & | GEPs ) |
Returns the VPValue representing the uncountable exit comparison used by AnyOf if the recipes it depends on can be traced back to live-ins and the addresses (in GEP/PtrAdd form) of any (non-masked) load used in generating the values for the comparison.
The recipes are stored in Recipes, and recipes forming an address for a load are also added to GEPs.
Definition at line 476 of file VPlanUtils.cpp.
References llvm::dyn_cast(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), GEP, llvm::VPlan::getVectorLoopRegion(), llvm::VPlanPatternMatch::m_AnyOf(), llvm::VPlanPatternMatch::m_BranchOnTwoConds(), llvm::VPlanPatternMatch::m_GetElementPtr(), llvm::PatternMatch::m_ICmp(), llvm::VPlanPatternMatch::m_LiveIn(), llvm::VPlanPatternMatch::m_VPValue(), llvm::SCEVPatternMatch::match(), llvm::SmallVectorImpl< T >::pop_back_val(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().
◆ getSCEVExprForVPValue()
| const SCEV * llvm::vputils::getSCEVExprForVPValue | ( | const VPValue * | V, |
| PredicatedScalarEvolution & | PSE, | ||
| const Loop * | L = nullptr ) |
Return the SCEV expression for V.
Returns SCEVCouldNotCompute if no SCEV expression could be constructed.
Definition at line 140 of file VPlanUtils.cpp.
References AbstractManglingParser< Derived, Alloc >::Ops, llvm::any_of(), llvm::ArrayRef(), llvm::TypeSwitch< T, ResultT >::Case(), llvm::Default, llvm::dyn_cast(), llvm::SCEV::FlagAnyWrap, llvm::SCEV::FlagNSW, llvm::ScalarEvolution::getAddExpr(), llvm::ScalarEvolution::getAddRecExpr(), llvm::ScalarEvolution::getConstant(), llvm::ScalarEvolution::getCouldNotCompute(), llvm::ScalarEvolution::getGEPExpr(), llvm::ScalarEvolution::getMinusOne(), llvm::ScalarEvolution::getMinusSCEV(), llvm::ScalarEvolution::getMulExpr(), llvm::ScalarEvolution::getOne(), llvm::PredicatedScalarEvolution::getPredicatedSCEV(), llvm::ScalarEvolution::getSCEV(), getSCEVExprForVPValue(), llvm::PredicatedScalarEvolution::getSE(), llvm::ScalarEvolution::getSignExtendExpr(), llvm::ScalarEvolution::getSMaxExpr(), llvm::ScalarEvolution::getSMinExpr(), llvm::ScalarEvolution::getTruncateExpr(), llvm::ScalarEvolution::getTruncateOrSignExtend(), llvm::SCEV::getType(), llvm::Value::getType(), llvm::ScalarEvolution::getUDivExpr(), llvm::ScalarEvolution::getUMaxExpr(), llvm::ScalarEvolution::getUMinExpr(), llvm::ScalarEvolution::getURemExpr(), llvm::ScalarEvolution::getZeroExtendExpr(), llvm::VPTypeAnalysis::inferScalarType(), llvm::isa(), llvm::IsaPred, llvm::ScalarEvolution::isSCEVable(), IV, llvm::PatternMatch::m_Add(), llvm::PatternMatch::m_APInt(), llvm::VPlanPatternMatch::m_Binary(), llvm::VPlanPatternMatch::m_c_BinaryAnd(), llvm::VPlanPatternMatch::m_GetElementPtr(), llvm::PatternMatch::m_Intrinsic(), llvm::PatternMatch::m_Mul(), llvm::MIPatternMatch::m_Not(), llvm::PatternMatch::m_SExt(), llvm::PatternMatch::m_Sub(), llvm::PatternMatch::m_Trunc(), llvm::VPlanPatternMatch::m_VPValue(), llvm::PatternMatch::m_ZExt(), llvm::SCEVPatternMatch::match(), poisonGuaranteesUB(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().
Referenced by llvm::VPlanTransforms::addMinimumIterationCheck(), collectComplementaryPredicatedMemOps(), createWidenInductionRecipe(), getAddressAccessSCEV(), getOptimizableIVOf(), getSCEVExprForVPValue(), isConditionTrueViaVFAndUF(), llvm::VPlanTransforms::optimizeFindIVReductions(), llvm::LoopVectorizationPlanner::selectEpilogueVectorizationFactor(), simplifyBranchConditionForVFAndUF(), simplifyKnownEVL(), and simplifyLiveInsWithSCEV().
◆ getVFScaleFactor()
| unsigned llvm::vputils::getVFScaleFactor | ( | VPRecipeBase * | R | ) |
Get the VF scaling factor applied to the recipe's output, if the recipe has one.
Definition at line 459 of file VPlanUtils.cpp.
References assert(), llvm::cast(), llvm::dyn_cast(), getOpcode(), llvm::isa(), and llvm::VPInstruction::ReductionStartVector.
Referenced by llvm::calculateRegisterUsageForPlan().
◆ isAddressSCEVForCost()
| bool llvm::vputils::isAddressSCEVForCost | ( | const SCEV * | Addr, |
| ScalarEvolution & | SE, | ||
| const Loop * | L ) |
Returns true if Addr is an address SCEV that can be passed to TTI::getAddressComputationCost, i.e.
the address SCEV is loop invariant, an affine AddRec (i.e. induction ), or an add expression of such operands or a sign-extended AddRec.
Definition at line 322 of file VPlanUtils.cpp.
References llvm::all_of(), llvm::dyn_cast(), llvm::ScalarEvolution::isLoopInvariant(), llvm::SCEVPatternMatch::m_SCEV(), llvm::SCEVPatternMatch::m_scev_AffineAddRec(), and llvm::SCEVPatternMatch::match().
Referenced by getAddressAccessSCEV(), and getAddressAccessSCEV().
◆ isHeaderMask()
Return true if V is a header mask in Plan.
Definition at line 53 of file VPlanUtils.cpp.
References A(), assert(), B(), llvm::cast(), llvm::VPlan::getBackedgeTakenCount(), llvm::VPRegionBlock::getCanonicalIV(), llvm::VPlan::getTripCount(), llvm::VPlan::getVectorLoopRegion(), llvm::VPlan::getVF(), llvm::VPlan::hasScalarVFOnly(), llvm::isa(), llvm::VPlanPatternMatch::m_ActiveLaneMask(), llvm::PatternMatch::m_ICmp(), llvm::PatternMatch::m_One(), llvm::VPlanPatternMatch::m_ScalarIVSteps(), llvm::PatternMatch::m_Specific(), llvm::VPlanPatternMatch::m_VPValue(), and llvm::SCEVPatternMatch::match().
Referenced by findHeaderMask().
◆ isSingleScalar()
Returns true if VPV is a single scalar, either because it produces the same value for all lanes or only has its first lane used.
Definition at line 360 of file VPlanUtils.cpp.
References llvm::all_of(), llvm::dyn_cast(), llvm::VPValue::getDefiningRecipe(), llvm::isa(), llvm::VPRegionBlock::isReplicator(), isSingleScalar(), llvm::VPUser::operands(), and preservesUniformity().
Referenced by cloneForLane(), llvm::VPWidenMemoryRecipe::computeCost(), llvm::VPIRPhi::execute(), llvm::VPWidenRecipe::execute(), llvm::VPTransformState::get(), llvm::VPTransformState::get(), isSingleScalar(), legalizeAndOptimizeInductions(), llvm::VPlanTransforms::materializePacksAndUnpacks(), narrowToSingleScalarRecipes(), planContainsAdditionalSimplifications(), scalarizeInstruction(), simplifyRecipe(), and llvm::VPWidenGEPRecipe::usesFirstLaneOnly().
◆ isUniformAcrossVFsAndUFs()
Checks if V is uniform across all VF lanes and UF parts.
It is considered as such if it is either loop invariant (defined outside the vector region) or its operand is known to be uniform across all VFs and UFs (e.g. VPDerivedIV or VPCanonicalIVPHI).
Definition at line 397 of file VPlanUtils.cpp.
References llvm::all_of(), llvm::TypeSwitch< T, ResultT >::Case(), llvm::Default, llvm::VPBasicBlock::getEnclosingLoopRegion(), llvm::VPlan::getEntry(), llvm::VPInstruction::getOpcode(), llvm::VPUser::getOperand(), llvm::VPBlockBase::getPlan(), llvm::VPlan::getVectorPreheader(), llvm::isa(), llvm::VPRecipeBase::isScalarCast(), isUniformAcrossVFsAndUFs(), llvm::VPlanPatternMatch::m_VPInstruction(), llvm::SCEVPatternMatch::match(), llvm::VPUser::operands(), and preservesUniformity().
Referenced by isKnownMonotonic(), isUniformAcrossVFsAndUFs(), and narrowToSingleScalarRecipes().
◆ onlyFirstLaneUsed()
Returns true if only the first lane of Def is used.
Definition at line 23 of file VPlanUtils.cpp.
References llvm::all_of().
Referenced by llvm::VPBlendRecipe::computeCost(), llvm::VPInstruction::computeCost(), llvm::VPInstruction::doesGeneratePerAllLanes(), llvm::VPInstruction::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPTransformState::get(), legalizeAndOptimizeInductions(), llvm::VPlanTransforms::materializePacksAndUnpacks(), llvm::VPlanTransforms::narrowInterleaveGroups(), llvm::VPWidenPointerInductionRecipe::onlyScalarsGenerated(), planContainsAdditionalSimplifications(), removeRedundantCanonicalIVs(), simplifyRecipe(), and llvm::VPInstruction::usesFirstLaneOnly().
◆ onlyFirstPartUsed()
Returns true if only the first part of Def is used.
Definition at line 28 of file VPlanUtils.cpp.
References llvm::all_of().
Referenced by llvm::VPInstruction::usesFirstPartOnly().
◆ onlyScalarValuesUsed()
Returns true if only scalar values of Def are used by all users.
Definition at line 33 of file VPlanUtils.cpp.
References llvm::all_of().
Referenced by llvm::calculateRegisterUsageForPlan(), llvm::VPlanTransforms::materializeBroadcasts(), llvm::VPlanTransforms::materializeFactors(), llvm::VPWidenPointerInductionRecipe::onlyScalarsGenerated(), removeRedundantCanonicalIVs(), and llvm::VPReplicateRecipe::shouldPack().
Generated on for LLVM by
1.14.0