LLVM 18.0.0git
|
LoopVectorizationCostModel - estimates the expected speedups due to vectorization. More...
Classes | |
struct | CallWideningDecision |
struct | RegisterUsage |
A struct that represents some properties of the register usage of a loop. More... | |
Public Types | |
enum | InstWidening { CM_Unknown , CM_Widen , CM_Widen_Reverse , CM_Interleave , CM_GatherScatter , CM_Scalarize , CM_VectorCall , CM_IntrinsicCall } |
Decision that was taken during cost calculation for memory instruction. More... | |
using | VectorizationCostTy = std::pair< InstructionCost, bool > |
The vectorization cost is a combination of the cost itself and a boolean indicating whether any of the contributing operations will actually operate on vector values after type legalization in the backend. | |
Public Member Functions | |
LoopVectorizationCostModel (ScalarEpilogueLowering SEL, Loop *L, PredicatedScalarEvolution &PSE, LoopInfo *LI, LoopVectorizationLegality *Legal, const TargetTransformInfo &TTI, const TargetLibraryInfo *TLI, DemandedBits *DB, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, const Function *F, const LoopVectorizeHints *Hints, InterleavedAccessInfo &IAI) | |
FixedScalableVFPair | computeMaxVF (ElementCount UserVF, unsigned UserIC) |
bool | runtimeChecksRequired () |
bool | selectUserVectorizationFactor (ElementCount UserVF) |
Setup cost-based decisions for user vectorization factor. | |
std::pair< unsigned, unsigned > | getSmallestAndWidestTypes () |
unsigned | selectInterleaveCount (ElementCount VF, InstructionCost LoopCost) |
void | setCostBasedWideningDecision (ElementCount VF) |
Memory access instruction may be vectorized in more than one way. | |
void | setVectorizedCallDecision (ElementCount VF) |
A call may be vectorized in different ways depending on whether we have vectorized variants available and whether the target supports masking. | |
SmallVector< RegisterUsage, 8 > | calculateRegisterUsage (ArrayRef< ElementCount > VFs) |
void | collectValuesToIgnore () |
Collect values we want to ignore in the cost model. | |
void | collectElementTypesForWidening () |
Collect all element types in the loop for which widening is needed. | |
void | collectInLoopReductions () |
Split reductions into those that happen in the loop, and those that happen outside. | |
bool | useOrderedReductions (const RecurrenceDescriptor &RdxDesc) const |
Returns true if we should use strict in-order reductions for the given RdxDesc. | |
const MapVector< Instruction *, uint64_t > & | getMinimalBitwidths () const |
bool | isProfitableToScalarize (Instruction *I, ElementCount VF) const |
bool | isUniformAfterVectorization (Instruction *I, ElementCount VF) const |
Returns true if I is known to be uniform after vectorization. | |
bool | isScalarAfterVectorization (Instruction *I, ElementCount VF) const |
Returns true if I is known to be scalar after vectorization. | |
bool | canTruncateToMinimalBitwidth (Instruction *I, ElementCount VF) const |
void | setWideningDecision (Instruction *I, ElementCount VF, InstWidening W, InstructionCost Cost) |
Save vectorization decision W and Cost taken by the cost model for instruction I and vector width VF . | |
void | setWideningDecision (const InterleaveGroup< Instruction > *Grp, ElementCount VF, InstWidening W, InstructionCost Cost) |
Save vectorization decision W and Cost taken by the cost model for interleaving group Grp and vector width VF . | |
InstWidening | getWideningDecision (Instruction *I, ElementCount VF) const |
Return the cost model decision for the given instruction I and vector width VF . | |
InstructionCost | getWideningCost (Instruction *I, ElementCount VF) |
Return the vectorization cost for the given instruction I and vector width VF . | |
void | setCallWideningDecision (CallInst *CI, ElementCount VF, InstWidening Kind, Function *Variant, Intrinsic::ID IID, std::optional< unsigned > MaskPos, InstructionCost Cost) |
CallWideningDecision | getCallWideningDecision (CallInst *CI, ElementCount VF) const |
bool | isOptimizableIVTruncate (Instruction *I, ElementCount VF) |
Return True if instruction I is an optimizable truncate whose operand is an induction variable. | |
void | collectInstsToScalarize (ElementCount VF) |
Collects the instructions to scalarize for each predicated instruction in the loop. | |
void | collectUniformsAndScalars (ElementCount VF) |
Collect Uniform and Scalar values for the given VF . | |
bool | isLegalMaskedStore (Type *DataType, Value *Ptr, Align Alignment) const |
Returns true if the target machine supports masked store operation for the given DataType and kind of access to Ptr . | |
bool | isLegalMaskedLoad (Type *DataType, Value *Ptr, Align Alignment) const |
Returns true if the target machine supports masked load operation for the given DataType and kind of access to Ptr . | |
bool | isLegalGatherOrScatter (Value *V, ElementCount VF) |
Returns true if the target machine can represent V as a masked gather or scatter operation. | |
bool | canVectorizeReductions (ElementCount VF) const |
Returns true if the target machine supports all of the reduction variables found for the given VF. | |
bool | isDivRemScalarWithPredication (InstructionCost ScalarCost, InstructionCost SafeDivisorCost) const |
Given costs for both strategies, return true if the scalar predication lowering should be used for div/rem. | |
bool | isScalarWithPredication (Instruction *I, ElementCount VF) const |
Returns true if I is an instruction which requires predication and for which our chosen predication strategy is scalarization (i.e. | |
bool | isPredicatedInst (Instruction *I) const |
Returns true if I is an instruction that needs to be predicated at runtime. | |
std::pair< InstructionCost, InstructionCost > | getDivRemSpeculationCost (Instruction *I, ElementCount VF) const |
Return the costs for our two available strategies for lowering a div/rem operation which requires speculating at least one lane. | |
bool | memoryInstructionCanBeWidened (Instruction *I, ElementCount VF) |
Returns true if I is a memory instruction with consecutive memory access that can be widened. | |
bool | interleavedAccessCanBeWidened (Instruction *I, ElementCount VF) |
Returns true if I is a memory instruction in an interleaved-group of memory accesses that can be vectorized with wide vector loads/stores and shuffles. | |
bool | isAccessInterleaved (Instruction *Instr) |
Check if Instr belongs to any interleaved access group. | |
const InterleaveGroup< Instruction > * | getInterleavedAccessGroup (Instruction *Instr) |
Get the interleaved access group that Instr belongs to. | |
bool | requiresScalarEpilogue (bool IsVectorizing) const |
Returns true if we're required to use a scalar epilogue for at least the final iteration of the original loop. | |
bool | requiresScalarEpilogue (VFRange Range) const |
Returns true if we're required to use a scalar epilogue for at least the final iteration of the original loop for all VFs in Range . | |
bool | isScalarEpilogueAllowed () const |
Returns true if a scalar epilogue is not allowed due to optsize or a loop hint annotation. | |
TailFoldingStyle | getTailFoldingStyle (bool IVUpdateMayOverflow=true) const |
Returns the TailFoldingStyle that is best for the current loop. | |
bool | foldTailByMasking () const |
Returns true if all loop blocks should be masked to fold tail loop. | |
bool | blockNeedsPredicationForAnyReason (BasicBlock *BB) const |
Returns true if the instructions in this block requires predication for any reason, e.g. | |
bool | isInLoopReduction (PHINode *Phi) const |
Returns true if the Phi is part of an inloop reduction. | |
InstructionCost | getVectorIntrinsicCost (CallInst *CI, ElementCount VF) const |
Estimate cost of an intrinsic call instruction CI if it were vectorized with factor VF. | |
InstructionCost | getVectorCallCost (CallInst *CI, ElementCount VF) const |
Estimate cost of a call instruction CI if it were vectorized with factor VF. | |
void | invalidateCostModelingDecisions () |
Invalidates decisions already taken by the cost model. | |
VectorizationCostTy | expectedCost (ElementCount VF, SmallVectorImpl< InstructionVFPair > *Invalid=nullptr) |
Returns the expected execution cost. | |
bool | hasPredStores () const |
bool | isEpilogueVectorizationProfitable (const ElementCount VF) const |
Returns true if epilogue vectorization is considered profitable, and false otherwise. | |
Public Attributes | |
Loop * | TheLoop |
The loop that we evaluate. | |
PredicatedScalarEvolution & | PSE |
Predicated scalar evolution analysis. | |
LoopInfo * | LI |
Loop Info analysis. | |
LoopVectorizationLegality * | Legal |
Vectorization legality. | |
const TargetTransformInfo & | TTI |
Vector target information. | |
const TargetLibraryInfo * | TLI |
Target Library Info. | |
DemandedBits * | DB |
Demanded bits analysis. | |
AssumptionCache * | AC |
Assumption cache. | |
OptimizationRemarkEmitter * | ORE |
Interface to emit optimization remarks. | |
const Function * | TheFunction |
const LoopVectorizeHints * | Hints |
Loop Vectorize Hint. | |
InterleavedAccessInfo & | InterleaveInfo |
The interleave access information contains groups of interleaved accesses with the same stride and close to each other. | |
SmallPtrSet< const Value *, 16 > | ValuesToIgnore |
Values to ignore in the cost model. | |
SmallPtrSet< const Value *, 16 > | VecValuesToIgnore |
Values to ignore in the cost model when VF > 1. | |
SmallPtrSet< Type *, 16 > | ElementTypesInLoop |
All element types found in the loop. | |
LoopVectorizationCostModel - estimates the expected speedups due to vectorization.
In many cases vectorization is not profitable. This can happen because of a number of reasons. In this class we mainly attempt to predict the expected speedup/slowdowns due to the supported instruction set. We use the TargetTransformInfo to query the different backends for the cost of different operations.
Definition at line 1211 of file LoopVectorize.cpp.
using llvm::LoopVectorizationCostModel::VectorizationCostTy = std::pair<InstructionCost, bool> |
The vectorization cost is a combination of the cost itself and a boolean indicating whether any of the contributing operations will actually operate on vector values after type legalization in the backend.
If this latter value is false, then all operations will be scalarized (i.e. no vectorization has actually taken place).
Definition at line 1686 of file LoopVectorize.cpp.
Decision that was taken during cost calculation for memory instruction.
Enumerator | |
---|---|
CM_Unknown | |
CM_Widen | |
CM_Widen_Reverse | |
CM_Interleave | |
CM_GatherScatter | |
CM_Scalarize | |
CM_VectorCall | |
CM_IntrinsicCall |
Definition at line 1375 of file LoopVectorize.cpp.
|
inline |
Definition at line 1213 of file LoopVectorize.cpp.
|
inline |
Returns true if the instructions in this block requires predication for any reason, e.g.
because tail folding now requires a predicate or because the block in the original loop was predicated.
Definition at line 1654 of file LoopVectorize.cpp.
References llvm::LoopVectorizationLegality::blockNeedsPredication(), foldTailByMasking(), and Legal.
Referenced by collectInstsToScalarize(), interleavedAccessCanBeWidened(), isPredicatedInst(), and llvm::LoopVectorizationPlanner::plan().
SmallVector< LoopVectorizationCostModel::RegisterUsage, 8 > LoopVectorizationCostModel::calculateRegisterUsage | ( | ArrayRef< ElementCount > | VFs | ) |
Definition at line 5724 of file LoopVectorize.cpp.
References llvm::all_of(), llvm::LoopBlocksDFS::beginRPO(), collectInLoopReductions(), collectUniformsAndScalars(), llvm::LoopBase< BlockT, LoopT >::contains(), llvm::SmallPtrSetImpl< PtrType >::count(), llvm::dbgs(), llvm::LoopBlocksDFS::endRPO(), llvm::SmallPtrSetImpl< PtrType >::erase(), llvm::VectorType::get(), llvm::ElementCount::getFixed(), llvm::TargetTransformInfo::getRegisterClassForType(), llvm::TargetTransformInfo::getRegisterClassName(), llvm::TargetTransformInfo::getRegUsageForType(), I, llvm::SetVector< T, Vector, Set, N >::insert(), llvm::SmallPtrSetImpl< PtrType >::insert(), isScalarAfterVectorization(), llvm::Type::isTokenTy(), llvm::VectorType::isValidElementType(), llvm::ElementCount::isVector(), LI, llvm::List, LLVM_DEBUG, llvm::LoopVectorizationCostModel::RegisterUsage::LoopInvariantRegs, llvm::make_range(), llvm::LoopVectorizationCostModel::RegisterUsage::MaxLocalUsers, llvm::LoopBlocksDFS::perform(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::RegUsage, llvm::ArrayRef< T >::size(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::size(), llvm::SmallPtrSetImplBase::size(), llvm::SmallVectorBase< Size_T >::size(), TheLoop, ToRemove, TTI, ValuesToIgnore, and VecValuesToIgnore.
Referenced by selectInterleaveCount().
|
inline |
I
can be truncated to a smaller bitwidth for vectorization factor VF
. Definition at line 1368 of file LoopVectorize.cpp.
References I, isProfitableToScalarize(), isScalarAfterVectorization(), and llvm::ElementCount::isVector().
|
inline |
Returns true if the target machine supports all of the reduction variables found for the given VF.
Definition at line 1539 of file LoopVectorize.cpp.
References llvm::all_of(), llvm::LoopVectorizationLegality::getReductionVars(), Legal, and Reduction.
void LoopVectorizationCostModel::collectElementTypesForWidening | ( | ) |
Collect all element types in the loop for which widening is needed.
Definition at line 5433 of file LoopVectorize.cpp.
References assert(), llvm::LoopBase< BlockT, LoopT >::blocks(), ElementTypesInLoop, llvm::MapVector< KeyT, ValueT, MapType, VectorType >::find(), llvm::RecurrenceDescriptor::getOpcode(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::LoopVectorizationLegality::getReductionVars(), I, llvm::LoopVectorizationLegality::isReductionVariable(), Legal, llvm::TargetTransformInfo::preferInLoopReduction(), PreferInLoopReductions, TheLoop, useOrderedReductions(), and ValuesToIgnore.
Referenced by llvm::LoopVectorizationPlanner::plan(), and processLoopInVPlanNativePath().
void LoopVectorizationCostModel::collectInLoopReductions | ( | ) |
Split reductions into those that happen in the loop, and those that happen outside.
In loop reductions are collected into InLoopReductions.
Definition at line 7310 of file LoopVectorize.cpp.
References llvm::dbgs(), llvm::SmallVectorBase< Size_T >::empty(), llvm::RecurrenceDescriptor::getOpcode(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::RecurrenceDescriptor::getReductionOpChain(), llvm::LoopVectorizationLegality::getReductionVars(), I, llvm::SmallPtrSetImpl< PtrType >::insert(), Legal, LLVM_DEBUG, Opcode, llvm::TargetTransformInfo::preferInLoopReduction(), PreferInLoopReductions, Reduction, TheLoop, and useOrderedReductions().
Referenced by calculateRegisterUsage(), and llvm::LoopVectorizationPlanner::plan().
void LoopVectorizationCostModel::collectInstsToScalarize | ( | ElementCount | VF | ) |
Collects the instructions to scalarize for each predicated instruction in the loop.
Definition at line 5944 of file LoopVectorize.cpp.
References llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::begin(), blockNeedsPredicationForAnyReason(), llvm::LoopBase< BlockT, LoopT >::blocks(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::clear(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), I, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::insert(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), isScalarWithPredication(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isZero(), and TheLoop.
Referenced by llvm::LoopVectorizationPlanner::plan(), and selectUserVectorizationFactor().
|
inline |
Collect Uniform and Scalar values for the given VF
.
The sets depend on CM decision for Load/Store instructions that may be vectorized as interleave, gather-scatter or scalarized. Also make a decision on what to do about call instructions in the loop at that VF – scalarize, call a known vector routine, or call a vector intrinsic.
Definition at line 1498 of file LoopVectorize.cpp.
References llvm::ElementCount::isScalar(), setCostBasedWideningDecision(), and setVectorizedCallDecision().
Referenced by calculateRegisterUsage(), llvm::LoopVectorizationPlanner::plan(), and selectUserVectorizationFactor().
void LoopVectorizationCostModel::collectValuesToIgnore | ( | ) |
Collect values we want to ignore in the cost model.
Definition at line 7280 of file LoopVectorize.cpp.
References AC, llvm::SmallVectorTemplateCommon< T, typename >::begin(), llvm::SmallPtrSetImpl< PtrType >::begin(), llvm::LoopBase< BlockT, LoopT >::blocks(), llvm::CodeMetrics::collectEphemeralValues(), llvm::SmallVectorTemplateCommon< T, typename >::end(), llvm::SmallPtrSetImpl< PtrType >::end(), llvm::RecurrenceDescriptor::getCastInsts(), llvm::InductionDescriptor::getCastInsts(), llvm::LoopVectorizationLegality::getInductionVars(), llvm::LoopVectorizationLegality::getReductionVars(), I, llvm::LoopVectorizationLegality::isInvariantAddressOfReduction(), Legal, Reduction, TheLoop, ValuesToIgnore, and VecValuesToIgnore.
Referenced by llvm::LoopVectorizationPlanner::plan().
FixedScalableVFPair LoopVectorizationCostModel::computeMaxVF | ( | ElementCount | UserVF, |
unsigned | UserIC | ||
) |
Definition at line 4767 of file LoopVectorize.cpp.
References llvm::ScalarEvolution::applyLoopGuards(), assert(), llvm::CM_ScalarEpilogueAllowed, llvm::CM_ScalarEpilogueNotAllowedLowTripLoop, llvm::CM_ScalarEpilogueNotAllowedOptSize, llvm::CM_ScalarEpilogueNotAllowedUsePredicate, llvm::CM_ScalarEpilogueNotNeededUsePredicate, llvm::dbgs(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::empty(), llvm::FixedScalableVFPair::FixedVF, llvm::ScalarEvolution::getAddExpr(), llvm::PredicatedScalarEvolution::getBackedgeTakenCount(), llvm::ScalarEvolution::getConstant(), llvm::LoopBase< BlockT, LoopT >::getExitingBlock(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getFixedValue(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), getMaxVScale(), llvm::FixedScalableVFPair::getNone(), llvm::ScalarEvolution::getOne(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::PredicatedScalarEvolution::getSE(), llvm::ScalarEvolution::getSmallConstantMaxTripCount(), llvm::ScalarEvolution::getSmallConstantTripCount(), llvm::SCEV::getType(), llvm::ScalarEvolution::getURemExpr(), llvm::TargetTransformInfo::hasBranchDivergence(), InterleaveInfo, llvm::InterleavedAccessInfo::invalidateGroupsRequiringScalarEpilogue(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isNonZero(), llvm::isPowerOf2_32(), llvm::TargetTransformInfo::isVScaleKnownToBeAPowerOfTwo(), llvm::SCEV::isZero(), Legal, LLVM_DEBUG, llvm::RuntimePointerChecking::Need, ORE, llvm::LoopVectorizationLegality::prepareToFoldTailByMasking(), PSE, llvm::reportVectorizationFailure(), runtimeChecksRequired(), llvm::FixedScalableVFPair::ScalableVF, TheFunction, TheLoop, and useMaskedInterleavedAccesses().
Referenced by llvm::LoopVectorizationPlanner::plan().
LoopVectorizationCostModel::VectorizationCostTy LoopVectorizationCostModel::expectedCost | ( | ElementCount | VF, |
SmallVectorImpl< InstructionVFPair > * | Invalid = nullptr |
||
) |
Returns the expected execution cost.
The unit of the cost does not matter because we use the 'cost' units to compare different vector widths. The cost that is returned is not normalized by the factor width. If Invalid
is not nullptr, this function will add a pair(Instruction*, ElementCount) to Invalid
for each instruction that has an Invalid cost for the given VF.
Definition at line 6096 of file LoopVectorize.cpp.
References llvm::LoopVectorizationLegality::blockNeedsPredication(), llvm::LoopBase< BlockT, LoopT >::blocks(), llvm::CallingConv::C, llvm::dbgs(), ForceTargetInstructionCost, getReciprocalPredBlockProb(), I, llvm::Invalid, llvm::ElementCount::isScalar(), llvm::ElementCount::isVector(), Legal, LLVM_DEBUG, TheLoop, ValuesToIgnore, and VecValuesToIgnore.
Referenced by selectInterleaveCount(), and selectUserVectorizationFactor().
|
inline |
Returns true if all loop blocks should be masked to fold tail loop.
Definition at line 1647 of file LoopVectorize.cpp.
References getTailFoldingStyle(), and llvm::None.
Referenced by blockNeedsPredicationForAnyReason(), llvm::VPRecipeBuilder::createHeaderMask(), and setCostBasedWideningDecision().
|
inline |
Definition at line 1456 of file LoopVectorize.cpp.
References assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::at(), and llvm::ElementCount::isScalar().
std::pair< InstructionCost, InstructionCost > LoopVectorizationCostModel::getDivRemSpeculationCost | ( | Instruction * | I, |
ElementCount | VF | ||
) | const |
Return the costs for our two available strategies for lowering a div/rem operation which requires speculating at least one lane.
First result is for scalarization (will be invalid for scalable vectors); second is for the safe-divisor strategy.
Definition at line 4180 of file LoopVectorize.cpp.
References assert(), llvm::CmpInst::BAD_ICMP_PREDICATE, CostKind, llvm::TargetTransformInfo::getArithmeticInstrCost(), llvm::TargetTransformInfo::getCFInstrCost(), llvm::TargetTransformInfo::getCmpSelInstrCost(), llvm::Type::getInt1Ty(), llvm::InstructionCost::getInvalid(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::TargetTransformInfo::getOperandInfo(), getReciprocalPredBlockProb(), I, llvm::LoopVectorizationLegality::isInvariant(), llvm::isSafeToSpeculativelyExecute(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::TargetTransformInfo::OperandValueInfo::Kind, Legal, llvm::TargetTransformInfo::OK_AnyValue, llvm::TargetTransformInfo::OK_UniformValue, Operands, llvm::TargetTransformInfo::TCK_RecipThroughput, and llvm::ToVectorTy().
Referenced by isScalarWithPredication().
|
inline |
Get the interleaved access group that Instr
belongs to.
Definition at line 1597 of file LoopVectorize.cpp.
References llvm::InterleavedAccessInfo::getInterleaveGroup(), and InterleaveInfo.
Referenced by interleavedAccessCanBeWidened(), and setCostBasedWideningDecision().
|
inline |
Definition at line 1307 of file LoopVectorize.cpp.
Definition at line 5402 of file LoopVectorize.cpp.
References DL, ElementTypesInLoop, llvm::MapVector< KeyT, ValueT, MapType, VectorType >::empty(), llvm::Module::getDataLayout(), llvm::RecurrenceDescriptor::getMinWidthCastToRecurrenceTypeInBits(), llvm::GlobalValue::getParent(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::Type::getScalarSizeInBits(), Legal, and TheFunction.
Referenced by determineVPlanVF().
|
inline |
Returns the TailFoldingStyle that is best for the current loop.
Definition at line 1636 of file LoopVectorize.cpp.
References ForceTailFoldingStyle, llvm::TargetTransformInfo::getPreferredTailFoldingStyle(), and llvm::None.
Referenced by foldTailByMasking().
InstructionCost LoopVectorizationCostModel::getVectorCallCost | ( | CallInst * | CI, |
ElementCount | VF | ||
) | const |
Estimate cost of a call instruction CI if it were vectorized with factor VF.
Return the cost of the instruction, including scalarization overhead if it's needed.
Definition at line 3359 of file LoopVectorize.cpp.
References llvm::CallBase::args(), CostKind, llvm::CallBase::getCalledFunction(), llvm::TargetTransformInfo::getCallInstrCost(), llvm::Value::getType(), getVectorIntrinsicCost(), llvm::getVectorIntrinsicIDForCall(), llvm::RecurrenceDescriptor::isFMulAddIntrinsic(), llvm::ElementCount::isScalar(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), RetTy, llvm::TargetTransformInfo::TCK_RecipThroughput, and TLI.
InstructionCost LoopVectorizationCostModel::getVectorIntrinsicCost | ( | CallInst * | CI, |
ElementCount | VF | ||
) | const |
Estimate cost of an intrinsic call instruction CI if it were vectorized with factor VF.
Return the cost of the instruction, including scalarization overhead if it's needed.
Definition at line 3394 of file LoopVectorize.cpp.
References llvm::CallBase::args(), Arguments, assert(), llvm::CallBase::getCalledFunction(), llvm::Function::getFunctionType(), llvm::TargetTransformInfo::getIntrinsicInstrCost(), llvm::Value::getType(), llvm::getVectorIntrinsicIDForCall(), MaybeVectorizeType(), llvm::FunctionType::param_begin(), llvm::FunctionType::param_end(), RetTy, llvm::TargetTransformInfo::TCK_RecipThroughput, and TLI.
Referenced by getVectorCallCost(), and setVectorizedCallDecision().
|
inline |
Return the vectorization cost for the given instruction I
and vector width VF
.
Definition at line 1431 of file LoopVectorize.cpp.
References assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains(), I, and llvm::ElementCount::isVector().
|
inline |
Return the cost model decision for the given instruction I
and vector width VF
.
Return CM_Unknown if this instruction did not pass through the cost modeling.
Definition at line 1415 of file LoopVectorize.cpp.
References assert(), CM_GatherScatter, CM_Unknown, llvm::EnableVPlanNativePath, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::find(), I, and llvm::ElementCount::isVector().
Referenced by interleavedAccessCanBeWidened(), and setCostBasedWideningDecision().
|
inline |
Definition at line 1698 of file LoopVectorize.cpp.
bool LoopVectorizationCostModel::interleavedAccessCanBeWidened | ( | Instruction * | I, |
ElementCount | VF | ||
) |
Returns true if I
is a memory instruction in an interleaved-group of memory accesses that can be vectorized with wide vector loads/stores and shuffles.
Definition at line 4245 of file LoopVectorize.cpp.
References assert(), blockNeedsPredicationForAnyReason(), CM_Unknown, DL, getInterleavedAccessGroup(), llvm::getLoadStoreAlignment(), llvm::getLoadStoreType(), getWideningDecision(), hasIrregularType(), I, isAccessInterleaved(), llvm::TargetTransformInfo::isLegalMaskedLoad(), llvm::TargetTransformInfo::isLegalMaskedStore(), llvm::LoopVectorizationLegality::isMaskRequired(), isScalarEpilogueAllowed(), Legal, and useMaskedInterleavedAccesses().
Referenced by setCostBasedWideningDecision().
|
inline |
Invalidates decisions already taken by the cost model.
Definition at line 1674 of file LoopVectorize.cpp.
References llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::clear().
Referenced by llvm::LoopVectorizationPlanner::plan().
|
inline |
Check if Instr
belongs to any interleaved access group.
Definition at line 1591 of file LoopVectorize.cpp.
References InterleaveInfo, and llvm::InterleavedAccessInfo::isInterleaved().
Referenced by interleavedAccessCanBeWidened(), and setCostBasedWideningDecision().
|
inline |
Given costs for both strategies, return true if the scalar predication lowering should be used for div/rem.
This incorporates an override option so it is not simply a cost comparison.
Definition at line 1549 of file LoopVectorize.cpp.
References llvm::cl::BOU_FALSE, llvm::cl::BOU_TRUE, llvm::cl::BOU_UNSET, ForceSafeDivisor, and llvm_unreachable.
Referenced by isScalarWithPredication().
bool LoopVectorizationCostModel::isEpilogueVectorizationProfitable | ( | const ElementCount | VF | ) | const |
Returns true if epilogue vectorization is considered profitable, and false otherwise.
VF
is the vectorization factor chosen for the original loop.
Definition at line 5279 of file LoopVectorize.cpp.
References EpilogueVectorizationMinVF, llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::TargetTransformInfo::getMaxInterleaveFactor(), getVScaleForTuning(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::TargetTransformInfo::preferEpilogueVectorization(), and TheLoop.
Referenced by llvm::LoopVectorizationPlanner::selectEpilogueVectorizationFactor().
Returns true if the Phi is part of an inloop reduction.
Definition at line 1659 of file LoopVectorize.cpp.
Referenced by llvm::VPRecipeBuilder::tryToCreateWidenRecipe().
|
inline |
Returns true if the target machine can represent V
as a masked gather or scatter operation.
Definition at line 1524 of file LoopVectorize.cpp.
References llvm::VectorType::get(), llvm::getLoadStoreAlignment(), llvm::getLoadStoreType(), llvm::TargetTransformInfo::isLegalMaskedGather(), llvm::TargetTransformInfo::isLegalMaskedScatter(), llvm::ElementCount::isVector(), and LI.
Referenced by setCostBasedWideningDecision().
|
inline |
Returns true if the target machine supports masked load operation for the given DataType
and kind of access to Ptr
.
Definition at line 1517 of file LoopVectorize.cpp.
References llvm::LoopVectorizationLegality::isConsecutivePtr(), llvm::TargetTransformInfo::isLegalMaskedLoad(), Legal, and Ptr.
Referenced by isScalarWithPredication().
|
inline |
Returns true if the target machine supports masked store operation for the given DataType
and kind of access to Ptr
.
Definition at line 1510 of file LoopVectorize.cpp.
References llvm::LoopVectorizationLegality::isConsecutivePtr(), llvm::TargetTransformInfo::isLegalMaskedStore(), Legal, and Ptr.
Referenced by isScalarWithPredication().
|
inline |
Return True if instruction I
is an optimizable truncate whose operand is an induction variable.
Such a truncate will be removed by adding a new induction variable with the destination type.
Definition at line 1465 of file LoopVectorize.cpp.
References llvm::LoopVectorizationLegality::getPrimaryInduction(), I, llvm::LoopVectorizationLegality::isInductionPhi(), llvm::TargetTransformInfo::isTruncateFree(), Legal, and llvm::ToVectorTy().
bool LoopVectorizationCostModel::isPredicatedInst | ( | Instruction * | I | ) | const |
Returns true if I
is an instruction that needs to be predicated at runtime.
The result is independent of the predication mechanism. Superset of instructions that return true for isScalarWithPredication.
Definition at line 4137 of file LoopVectorize.cpp.
References llvm::LoopVectorizationLegality::blockNeedsPredication(), blockNeedsPredicationForAnyReason(), llvm::getLoadStorePointerOperand(), I, llvm::LoopVectorizationLegality::isInvariant(), llvm::Loop::isLoopInvariant(), llvm::LoopVectorizationLegality::isMaskRequired(), llvm::isSafeToSpeculativelyExecute(), Legal, and TheLoop.
Referenced by llvm::VPRecipeBuilder::handleReplication(), and isScalarWithPredication().
|
inline |
I
for vectorization factor VF
. Definition at line 1313 of file LoopVectorize.cpp.
References assert(), llvm::EnableVPlanNativePath, I, and llvm::ElementCount::isVector().
Referenced by canTruncateToMinimalBitwidth().
|
inline |
Returns true if I
is known to be scalar after vectorization.
Definition at line 1351 of file LoopVectorize.cpp.
References assert(), llvm::EnableVPlanNativePath, I, and llvm::ElementCount::isScalar().
Referenced by calculateRegisterUsage(), and canTruncateToMinimalBitwidth().
|
inline |
Returns true if a scalar epilogue is not allowed due to optsize or a loop hint annotation.
Definition at line 1630 of file LoopVectorize.cpp.
References llvm::CM_ScalarEpilogueAllowed.
Referenced by interleavedAccessCanBeWidened(), requiresScalarEpilogue(), llvm::LoopVectorizationPlanner::selectEpilogueVectorizationFactor(), and selectInterleaveCount().
bool LoopVectorizationCostModel::isScalarWithPredication | ( | Instruction * | I, |
ElementCount | VF | ||
) | const |
Returns true if I
is an instruction which requires predication and for which our chosen predication strategy is scalarization (i.e.
we don't have an alternate strategy such as masking available). VF
is the vectorization factor that will be used to vectorize I
.
Definition at line 4096 of file LoopVectorize.cpp.
References CM_Scalarize, llvm::VectorType::get(), getDivRemSpeculationCost(), llvm::getLoadStoreAlignment(), llvm::getLoadStorePointerOperand(), llvm::getLoadStoreType(), I, isDivRemScalarWithPredication(), llvm::TargetTransformInfo::isLegalMaskedGather(), isLegalMaskedLoad(), llvm::TargetTransformInfo::isLegalMaskedScatter(), isLegalMaskedStore(), isPredicatedInst(), llvm::ElementCount::isScalar(), llvm::ElementCount::isVector(), and Ptr.
Referenced by collectInstsToScalarize(), memoryInstructionCanBeWidened(), and setCostBasedWideningDecision().
|
inline |
Returns true if I
is known to be uniform after vectorization.
Definition at line 1329 of file LoopVectorize.cpp.
References assert(), llvm::EnableVPlanNativePath, I, and llvm::ElementCount::isScalar().
Referenced by llvm::VPRecipeBuilder::handleReplication().
bool LoopVectorizationCostModel::memoryInstructionCanBeWidened | ( | Instruction * | I, |
ElementCount | VF | ||
) |
Returns true if I
is a memory instruction with consecutive memory access that can be widened.
Definition at line 4314 of file LoopVectorize.cpp.
References assert(), DL, llvm::getLoadStorePointerOperand(), llvm::getLoadStoreType(), hasIrregularType(), I, llvm::LoopVectorizationLegality::isConsecutivePtr(), isScalarWithPredication(), Legal, and Ptr.
Referenced by setCostBasedWideningDecision().
Returns true if we're required to use a scalar epilogue for at least the final iteration of the original loop.
Definition at line 1603 of file LoopVectorize.cpp.
References llvm::LoopBase< BlockT, LoopT >::getExitingBlock(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), InterleaveInfo, isScalarEpilogueAllowed(), llvm::InterleavedAccessInfo::requiresScalarEpilogue(), and TheLoop.
Referenced by requiresScalarEpilogue().
Returns true if we're required to use a scalar epilogue for at least the final iteration of the original loop for all VFs in Range
.
A scalar epilogue must either be required for all VFs in Range
or for none.
Definition at line 1617 of file LoopVectorize.cpp.
References llvm::all_of(), assert(), llvm::none_of(), and requiresScalarEpilogue().
bool LoopVectorizationCostModel::runtimeChecksRequired | ( | ) |
Definition at line 4567 of file LoopVectorize.cpp.
References llvm::dbgs(), llvm::LoopVectorizationLegality::getLAI(), llvm::PredicatedScalarEvolution::getPredicate(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::LoopAccessInfo::getSymbolicStrides(), llvm::SCEVPredicate::isAlwaysTrue(), Legal, LLVM_DEBUG, llvm::RuntimePointerChecking::Need, ORE, PSE, llvm::reportVectorizationFailure(), and TheLoop.
Referenced by computeMaxVF().
unsigned LoopVectorizationCostModel::selectInterleaveCount | ( | ElementCount | VF, |
InstructionCost | LoopCost | ||
) |
Definition at line 5477 of file LoopVectorize.cpp.
References llvm::any_of(), assert(), llvm::bit_floor(), llvm::LoopBase< BlockT, LoopT >::blocks(), calculateRegisterUsage(), llvm::dbgs(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::empty(), llvm::TargetTransformInfo::enableAggressiveInterleaving(), EnableIndVarRegisterHeur, EnableLoadStoreRuntimeInterleave, expectedCost(), F, ForceTargetMaxScalarInterleaveFactor, ForceTargetMaxVectorInterleaveFactor, ForceTargetNumScalarRegs, ForceTargetNumVectorRegs, llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::LoopBase< BlockT, LoopT >::getLoopDepth(), llvm::TargetTransformInfo::getMaxInterleaveFactor(), llvm::TargetTransformInfo::getNumberOfRegisters(), llvm::LoopVectorizationLegality::getNumLoads(), llvm::LoopVectorizationLegality::getNumStores(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::TargetTransformInfo::getRegisterClassName(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::PredicatedScalarEvolution::getSE(), getSmallBestKnownTC(), llvm::InstructionCost::getValue(), InterleaveSmallLoopScalarReduction, llvm::LoopVectorizationLegality::isSafeForAnyVectorWidth(), llvm::ElementCount::isScalar(), isScalarEpilogueAllowed(), llvm::InstructionCost::isValid(), llvm::ElementCount::isVector(), Legal, LLVM_DEBUG, MaxNestedScalarReductionIC, llvm::RuntimePointerChecking::Need, PSE, Reduction, SmallLoopCost, TheLoop, and TinyTripCountInterleaveThreshold.
Referenced by llvm::LoopVectorizePass::processLoop().
|
inline |
Setup cost-based decisions for user vectorization factor.
Definition at line 1237 of file LoopVectorize.cpp.
References collectInstsToScalarize(), collectUniformsAndScalars(), and expectedCost().
Referenced by llvm::LoopVectorizationPlanner::plan().
|
inline |
Definition at line 1447 of file LoopVectorize.cpp.
References assert(), and llvm::ElementCount::isScalar().
Referenced by setVectorizedCallDecision().
void LoopVectorizationCostModel::setCostBasedWideningDecision | ( | ElementCount | VF | ) |
Memory access instruction may be vectorized in more than one way.
Form of instruction after vectorization depends on cost. This function takes cost-based decisions for Load/Store instructions and collects them in a map. This decisions map is used for building the lists of loop-uniform and loop-scalar instructions. The calculated cost is saved with widening decision in order to avoid redundant calculations.
Definition at line 6623 of file LoopVectorize.cpp.
References llvm::append_range(), assert(), llvm::LoopBase< BlockT, LoopT >::blocks(), CM_GatherScatter, CM_Interleave, CM_Scalarize, CM_Unknown, CM_Widen, CM_Widen_Reverse, llvm::LoopBase< BlockT, LoopT >::contains(), llvm::SmallVectorBase< Size_T >::empty(), foldTailByMasking(), llvm::ElementCount::getFixed(), getInterleavedAccessGroup(), llvm::InstructionCost::getInvalid(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::getLoadStorePointerOperand(), llvm::getLoadStoreType(), getWideningDecision(), I, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::insert(), llvm::SmallPtrSetImpl< PtrType >::insert(), interleavedAccessCanBeWidened(), isAccessInterleaved(), llvm::LoopVectorizationLegality::isConsecutivePtr(), isLegalGatherOrScatter(), llvm::Loop::isLoopInvariant(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), isScalarWithPredication(), llvm::LoopVectorizationLegality::isUniformMemOp(), Legal, memoryInstructionCanBeWidened(), llvm::SmallVectorImpl< T >::pop_back_val(), llvm::TargetTransformInfo::prefersVectorizedAddressing(), Ptr, llvm::SmallVectorTemplateBase< T, bool >::push_back(), setWideningDecision(), and TheLoop.
Referenced by collectUniformsAndScalars().
void LoopVectorizationCostModel::setVectorizedCallDecision | ( | ElementCount | VF | ) |
A call may be vectorized in different ways depending on whether we have vectorized variants available and whether the target supports masking.
This function analyzes all calls in the function at the supplied VF, makes a decision based on the costs of available options, and stores that decision in a map for use in planning and plan execution.
Definition at line 6809 of file LoopVectorize.cpp.
References llvm::CallBase::args(), assert(), llvm::LoopBase< BlockT, LoopT >::blocks(), CM_IntrinsicCall, CM_Scalarize, CM_VectorCall, CostKind, llvm::VectorType::get(), llvm::CallBase::getArgOperand(), llvm::CallBase::getCalledFunction(), llvm::TargetTransformInfo::getCallInstrCost(), llvm::Type::getContext(), llvm::Module::getFunction(), llvm::Function::getFunctionType(), llvm::Type::getInt1Ty(), llvm::InstructionCost::getInvalid(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::VFDatabase::getMappings(), llvm::Instruction::getModule(), llvm::VFInfo::getParamIndexForOptionalMask(), llvm::PredicatedScalarEvolution::getSCEV(), llvm::PredicatedScalarEvolution::getSE(), llvm::TargetTransformInfo::getShuffleCost(), llvm::Value::getType(), getVectorIntrinsicCost(), llvm::getVectorIntrinsicIDForCall(), llvm::GlobalPredicate, I, Info, llvm::RecurrenceDescriptor::isFMulAddIntrinsic(), llvm::ScalarEvolution::isLoopInvariant(), llvm::LoopVectorizationLegality::isMaskRequired(), llvm::CallBase::isNoBuiltin(), llvm::ElementCount::isScalar(), Legal, llvm::Intrinsic::not_intrinsic, llvm::OMP_Uniform, PSE, llvm::SmallVectorTemplateBase< T, bool >::push_back(), RetTy, setCallWideningDecision(), llvm::TargetTransformInfo::SK_Broadcast, llvm::TargetTransformInfo::TCK_RecipThroughput, TheLoop, TLI, llvm::ToVectorTy(), and llvm::Vector.
Referenced by collectUniformsAndScalars().
|
inline |
Save vectorization decision W
and Cost
taken by the cost model for interleaving group Grp
and vector width VF
.
Broadcast this decicion to all instructions inside the group. But the cost will be assigned to one instruction only.
Definition at line 1396 of file LoopVectorize.cpp.
References assert(), llvm::InterleaveGroup< InstTy >::getFactor(), llvm::InterleaveGroup< InstTy >::getInsertPos(), llvm::InterleaveGroup< InstTy >::getMember(), I, and llvm::ElementCount::isVector().
|
inline |
Save vectorization decision W
and Cost
taken by the cost model for instruction I
and vector width VF
.
Definition at line 1388 of file LoopVectorize.cpp.
References assert(), I, and llvm::ElementCount::isVector().
Referenced by setCostBasedWideningDecision().
|
inline |
Returns true if we should use strict in-order reductions for the given RdxDesc.
This is true if the -enable-strict-reductions flag is passed, the IsOrdered flag of RdxDesc is set and we do not allow reordering of FP operations.
Definition at line 1300 of file LoopVectorize.cpp.
References llvm::LoopVectorizeHints::allowReordering(), Hints, and llvm::RecurrenceDescriptor::isOrdered().
Referenced by collectElementTypesForWidening(), collectInLoopReductions(), and llvm::VPRecipeBuilder::tryToCreateWidenRecipe().
AssumptionCache* llvm::LoopVectorizationCostModel::AC |
Assumption cache.
Definition at line 1917 of file LoopVectorize.cpp.
Referenced by collectValuesToIgnore().
DemandedBits* llvm::LoopVectorizationCostModel::DB |
Demanded bits analysis.
Definition at line 1914 of file LoopVectorize.cpp.
SmallPtrSet<Type *, 16> llvm::LoopVectorizationCostModel::ElementTypesInLoop |
All element types found in the loop.
Definition at line 1938 of file LoopVectorize.cpp.
Referenced by collectElementTypesForWidening(), and getSmallestAndWidestTypes().
const LoopVectorizeHints* llvm::LoopVectorizationCostModel::Hints |
Loop Vectorize Hint.
Definition at line 1925 of file LoopVectorize.cpp.
Referenced by useOrderedReductions().
InterleavedAccessInfo& llvm::LoopVectorizationCostModel::InterleaveInfo |
The interleave access information contains groups of interleaved accesses with the same stride and close to each other.
Definition at line 1929 of file LoopVectorize.cpp.
Referenced by computeMaxVF(), getInterleavedAccessGroup(), isAccessInterleaved(), llvm::LoopVectorizationPlanner::plan(), and requiresScalarEpilogue().
LoopVectorizationLegality* llvm::LoopVectorizationCostModel::Legal |
Vectorization legality.
Definition at line 1905 of file LoopVectorize.cpp.
Referenced by blockNeedsPredicationForAnyReason(), canVectorizeReductions(), collectElementTypesForWidening(), collectInLoopReductions(), collectValuesToIgnore(), computeMaxVF(), expectedCost(), getDivRemSpeculationCost(), getSmallestAndWidestTypes(), interleavedAccessCanBeWidened(), isLegalMaskedLoad(), isLegalMaskedStore(), isOptimizableIVTruncate(), isPredicatedInst(), memoryInstructionCanBeWidened(), runtimeChecksRequired(), selectInterleaveCount(), setCostBasedWideningDecision(), and setVectorizedCallDecision().
LoopInfo* llvm::LoopVectorizationCostModel::LI |
Loop Info analysis.
Definition at line 1902 of file LoopVectorize.cpp.
Referenced by calculateRegisterUsage(), and isLegalGatherOrScatter().
OptimizationRemarkEmitter* llvm::LoopVectorizationCostModel::ORE |
Interface to emit optimization remarks.
Definition at line 1920 of file LoopVectorize.cpp.
Referenced by computeMaxVF(), and runtimeChecksRequired().
PredicatedScalarEvolution& llvm::LoopVectorizationCostModel::PSE |
Predicated scalar evolution analysis.
Definition at line 1899 of file LoopVectorize.cpp.
Referenced by computeMaxVF(), runtimeChecksRequired(), selectInterleaveCount(), and setVectorizedCallDecision().
Definition at line 1922 of file LoopVectorize.cpp.
Referenced by computeMaxVF(), and getSmallestAndWidestTypes().
Loop* llvm::LoopVectorizationCostModel::TheLoop |
The loop that we evaluate.
Definition at line 1896 of file LoopVectorize.cpp.
Referenced by calculateRegisterUsage(), collectElementTypesForWidening(), collectInLoopReductions(), collectInstsToScalarize(), collectValuesToIgnore(), computeMaxVF(), expectedCost(), isEpilogueVectorizationProfitable(), isPredicatedInst(), requiresScalarEpilogue(), runtimeChecksRequired(), selectInterleaveCount(), setCostBasedWideningDecision(), and setVectorizedCallDecision().
const TargetLibraryInfo* llvm::LoopVectorizationCostModel::TLI |
Target Library Info.
Definition at line 1911 of file LoopVectorize.cpp.
Referenced by getVectorCallCost(), getVectorIntrinsicCost(), and setVectorizedCallDecision().
const TargetTransformInfo& llvm::LoopVectorizationCostModel::TTI |
Vector target information.
Definition at line 1908 of file LoopVectorize.cpp.
Referenced by calculateRegisterUsage().
SmallPtrSet<const Value *, 16> llvm::LoopVectorizationCostModel::ValuesToIgnore |
Values to ignore in the cost model.
Definition at line 1932 of file LoopVectorize.cpp.
Referenced by calculateRegisterUsage(), collectElementTypesForWidening(), collectValuesToIgnore(), and expectedCost().
SmallPtrSet<const Value *, 16> llvm::LoopVectorizationCostModel::VecValuesToIgnore |
Values to ignore in the cost model when VF > 1.
Definition at line 1935 of file LoopVectorize.cpp.
Referenced by calculateRegisterUsage(), collectValuesToIgnore(), and expectedCost().