LLVM  6.0.0svn
Public Types | Public Member Functions | Protected Types | Protected Member Functions | Protected Attributes | Friends | List of all members
llvm::InnerLoopVectorizer Class Reference

InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF). More...

Inheritance diagram for llvm::InnerLoopVectorizer:
Inheritance graph
[legend]
Collaboration diagram for llvm::InnerLoopVectorizer:
Collaboration graph
[legend]

Public Types

using VectorParts = SmallVector< Value *, 2 >
 A type for vectorized values in the new loop. More...
 

Public Member Functions

 InnerLoopVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, unsigned VecWidth, unsigned UnrollFactor, LoopVectorizationLegality *LVL, LoopVectorizationCostModel *CM)
 
virtual ~InnerLoopVectorizer ()=default
 
BasicBlockcreateVectorizedLoopSkeleton ()
 Create a new empty loop. More...
 
void widenInstruction (Instruction &I)
 Widen a single instruction within the innermost loop. More...
 
void fixVectorizedLoop ()
 Fix the vectorized code, taking care of header phi's, live-outs, and more. More...
 
bool areSafetyChecksAdded ()
 
VectorParts createBlockInMask (BasicBlock *BB)
 A helper function that computes the predicate of the block BB, assuming that the header block of the loop is set to True. More...
 
void widenPHIInstruction (Instruction *PN, unsigned UF, unsigned VF)
 Vectorize a single PHINode in a block. More...
 
void scalarizeInstruction (Instruction *Instr, const VPIteration &Instance, bool IfPredicateInstr)
 A helper function to scalarize a single Instruction in the innermost loop. More...
 
void widenIntOrFpInduction (PHINode *IV, TruncInst *Trunc=nullptr)
 Widen an integer or floating-point induction variable IV. More...
 
ValuegetOrCreateVectorValue (Value *V, unsigned Part)
 getOrCreateVectorValue and getOrCreateScalarValue coordinate to generate a vector or scalar value on-demand if one is not yet available. More...
 
ValuegetOrCreateScalarValue (Value *V, const VPIteration &Instance)
 Return a value in the new loop corresponding to V from the original loop at unroll and vector indices Instance. More...
 
void packScalarIntoVectorValue (Value *V, const VPIteration &Instance)
 Construct the vector value of a scalarized value V one lane at a time. More...
 
void vectorizeInterleaveGroup (Instruction *Instr)
 Try to vectorize the interleaved access group that Instr belongs to. More...
 

Protected Types

using PhiVector = SmallVector< PHINode *, 4 >
 A small list of PHINodes. More...
 
using ScalarParts = SmallVector< SmallVector< Value *, 4 >, 2 >
 A type for scalarized values in the new loop. More...
 
using EdgeMaskCacheTy = DenseMap< std::pair< BasicBlock *, BasicBlock * >, VectorParts >
 
using BlockMaskCacheTy = DenseMap< BasicBlock *, VectorParts >
 

Protected Member Functions

void fixupIVUsers (PHINode *OrigPhi, const InductionDescriptor &II, Value *CountRoundDown, Value *EndValue, BasicBlock *MiddleBlock)
 Set up the values of the IVs correctly when exiting the vector loop. More...
 
PHINodecreateInductionVariable (Loop *L, Value *Start, Value *End, Value *Step, Instruction *DL)
 Create a new induction variable inside L. More...
 
void fixCrossIterationPHIs ()
 Handle all cross-iteration phis in the header. More...
 
void fixFirstOrderRecurrence (PHINode *Phi)
 Fix a first-order recurrence. More...
 
void fixReduction (PHINode *Phi)
 Fix a reduction cross-iteration phi. More...
 
void fixLCSSAPHIs ()
 The Loop exit block may have single value PHI nodes with some incoming value. More...
 
void sinkScalarOperands (Instruction *PredInst)
 Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it. More...
 
void truncateToMinimalBitwidths ()
 Shrinks vector element sizes to the smallest bitwidth they can be legally represented as. More...
 
VectorParts createEdgeMask (BasicBlock *Src, BasicBlock *Dst)
 A helper function that computes the predicate of the edge between SRC and DST. More...
 
void updateAnalysis ()
 Insert the new loop to the loop hierarchy and pass manager and update the analysis passes. More...
 
virtual void vectorizeMemoryInstruction (Instruction *Instr)
 Vectorize Load and Store instructions,. More...
 
virtual ValuegetBroadcastInstrs (Value *V)
 Create a broadcast instruction. More...
 
virtual ValuegetStepVector (Value *Val, int StartIdx, Value *Step, Instruction::BinaryOps Opcode=Instruction::BinaryOpsEnd)
 This function adds (StartIdx, StartIdx + Step, StartIdx + 2*Step, ...) to each vector element of Val. More...
 
void buildScalarSteps (Value *ScalarIV, Value *Step, Value *EntryVal, const InductionDescriptor &ID)
 Compute scalar induction steps. More...
 
void createVectorIntOrFpInductionPHI (const InductionDescriptor &II, Value *Step, Instruction *EntryVal)
 Create a vector induction phi node based on an existing scalar one. More...
 
bool shouldScalarizeInstruction (Instruction *I) const
 Returns true if an instruction I should be scalarized instead of vectorized for the chosen vectorization factor. More...
 
bool needsScalarInduction (Instruction *IV) const
 Returns true if we should generate a scalar version of IV. More...
 
virtual ValuereverseVector (Value *Vec)
 Generate a shuffle sequence that will reverse the vector Vec. More...
 
ValuegetOrCreateTripCount (Loop *NewLoop)
 Returns (and creates if needed) the original loop trip count. More...
 
ValuegetOrCreateVectorTripCount (Loop *NewLoop)
 Returns (and creates if needed) the trip count of the widened loop. More...
 
ValuecreateBitOrPointerCast (Value *V, VectorType *DstVTy, const DataLayout &DL)
 Returns a bitcasted value to the requested vector type. More...
 
void emitMinimumIterationCountCheck (Loop *L, BasicBlock *Bypass)
 Emit a bypass check to see if the vector trip count is zero, including if it overflows. More...
 
void emitSCEVChecks (Loop *L, BasicBlock *Bypass)
 Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct. More...
 
void emitMemRuntimeChecks (Loop *L, BasicBlock *Bypass)
 Emit bypass checks to check any memory assumptions we may have made. More...
 
void addNewMetadata (Instruction *To, const Instruction *Orig)
 Add additional metadata to To that was not present on Orig. More...
 
void addMetadata (Instruction *To, Instruction *From)
 Add metadata from one instruction to another. More...
 
void addMetadata (ArrayRef< Value *> To, Instruction *From)
 Similar to the previous function but it adds the metadata to a vector of instructions. More...
 
void setDebugLocFromInst (IRBuilder<> &B, const Value *Ptr)
 Set the debug location in the builder using the debug location in the instruction. More...
 

Protected Attributes

LoopOrigLoop
 The original loop. More...
 
PredicatedScalarEvolutionPSE
 A wrapper around ScalarEvolution used to add runtime SCEV checks. More...
 
LoopInfoLI
 Loop Info. More...
 
DominatorTreeDT
 Dominator Tree. More...
 
AliasAnalysisAA
 Alias Analysis. More...
 
const TargetLibraryInfoTLI
 Target Library Info. More...
 
const TargetTransformInfoTTI
 Target Transform Info. More...
 
AssumptionCacheAC
 Assumption Cache. More...
 
OptimizationRemarkEmitterORE
 Interface to emit optimization remarks. More...
 
std::unique_ptr< LoopVersioningLVer
 LoopVersioning. More...
 
unsigned VF
 The vectorization SIMD factor to use. More...
 
unsigned UF
 The vectorization unroll factor to use. More...
 
IRBuilder Builder
 The builder that we use. More...
 
BasicBlockLoopVectorPreHeader
 The vector-loop preheader. More...
 
BasicBlockLoopScalarPreHeader
 The scalar-loop preheader. More...
 
BasicBlockLoopMiddleBlock
 Middle Block between the vector and the scalar. More...
 
BasicBlockLoopExitBlock
 The ExitBlock of the scalar loop. More...
 
BasicBlockLoopVectorBody
 The vector loop body. More...
 
BasicBlockLoopScalarBody
 The scalar loop body. More...
 
SmallVector< BasicBlock *, 4 > LoopBypassBlocks
 A list of all bypass blocks. The first block is the entry of the loop. More...
 
PHINodeInduction = nullptr
 The new Induction variable which was added to the new block. More...
 
PHINodeOldInduction = nullptr
 The induction variable of the old basic block. More...
 
VectorizerValueMap VectorLoopValueMap
 Maps values from the original loop to their corresponding values in the vectorized loop. More...
 
SmallVector< Instruction *, 4 > PredicatedInstructions
 Store instructions that were predicated. More...
 
EdgeMaskCacheTy EdgeMaskCache
 
BlockMaskCacheTy BlockMaskCache
 
ValueTripCount = nullptr
 Trip count of the original loop. More...
 
ValueVectorTripCount = nullptr
 Trip count of the widened loop (TripCount - TripCount % (VF*UF)) More...
 
LoopVectorizationLegality * Legal
 The legality analysis. More...
 
LoopVectorizationCostModel * Cost
 The profitablity analysis. More...
 
bool AddedSafetyChecks = false
 
DenseMap< PHINode *, Value * > IVEndValues
 

Friends

class LoopVectorizationPlanner
 

Detailed Description

InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF).

This class performs the widening of scalars into vectors, or multiple scalars. This class also implements the following features:

Definition at line 417 of file LoopVectorize.cpp.

Member Typedef Documentation

◆ BlockMaskCacheTy

Definition at line 523 of file LoopVectorize.cpp.

◆ EdgeMaskCacheTy

Definition at line 522 of file LoopVectorize.cpp.

◆ PhiVector

A small list of PHINodes.

Definition at line 511 of file LoopVectorize.cpp.

◆ ScalarParts

A type for scalarized values in the new loop.

Each value from the original loop, when scalarized, is represented by UF x VF scalar values in the new unrolled loop, where UF is the unroll factor and VF is the vectorization factor.

Definition at line 517 of file LoopVectorize.cpp.

◆ VectorParts

A type for vectorized values in the new loop.

Each value from the original loop, when vectorized, is represented by UF vector values in the new unrolled loop, where UF is the unroll factor.

Definition at line 448 of file LoopVectorize.cpp.

Constructor & Destructor Documentation

◆ InnerLoopVectorizer()

llvm::InnerLoopVectorizer::InnerLoopVectorizer ( Loop OrigLoop,
PredicatedScalarEvolution PSE,
LoopInfo LI,
DominatorTree DT,
const TargetLibraryInfo TLI,
const TargetTransformInfo TTI,
AssumptionCache AC,
OptimizationRemarkEmitter ORE,
unsigned  VecWidth,
unsigned  UnrollFactor,
LoopVectorizationLegality *  LVL,
LoopVectorizationCostModel *  CM 
)
inline

◆ ~InnerLoopVectorizer()

virtual llvm::InnerLoopVectorizer::~InnerLoopVectorizer ( )
virtualdefault

Referenced by InnerLoopVectorizer().

Member Function Documentation

◆ addMetadata() [1/2]

void InnerLoopVectorizer::addMetadata ( Instruction To,
Instruction From 
)
protected

Add metadata from one instruction to another.

This includes both the original MDs from From and additional ones (

See also
addNewMetadata). Use this for newly created instructions in the vector loop.

Definition at line 839 of file LoopVectorize.cpp.

References addNewMetadata(), and llvm::propagateMetadata().

Referenced by addMetadata(), createVectorIntOrFpInductionPHI(), vectorizeInterleaveGroup(), vectorizeMemoryInstruction(), widenInstruction(), and widenIntOrFpInduction().

◆ addMetadata() [2/2]

void InnerLoopVectorizer::addMetadata ( ArrayRef< Value *>  To,
Instruction From 
)
protected

Similar to the previous function but it adds the metadata to a vector of instructions.

Definition at line 845 of file LoopVectorize.cpp.

References llvm::abs(), addMetadata(), llvm::AMDGPU::HSAMD::Kernel::Arg::Key::Align, llvm::OptimizationRemarkAnalysis::AlwaysPrint, Arg, llvm::AMDGPU::HSAMD::Kernel::Key::Args, assert(), B, llvm::LoopAccessInfo::blockNeedsPredication(), C, Context, llvm::DenseMapBase< DenseMap< KeyT, ValueT, KeyInfoT, BucketT >, KeyT, ValueT, KeyInfoT, BucketT >::count(), llvm::dbgs(), DEBUG, DT, llvm::dyn_cast(), llvm::OptimizationRemarkEmitter::emit(), llvm::ArrayRef< T >::empty(), llvm::StringRef::endswith(), llvm::DenseMapBase< DenseMap< KeyT, ValueT, KeyInfoT, BucketT >, KeyT, ValueT, KeyInfoT, BucketT >::erase(), F(), llvm::ConstantInt::get(), llvm::ConstantAsMetadata::get(), llvm::MDString::get(), llvm::MDNode::get(), getAlignment(), llvm::LoopAccessInfo::getDepChecker(), llvm::MemoryDepChecker::getDependences(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::Type::getInt32Ty(), llvm::MDNode::getNumOperands(), llvm::MDNode::getOperand(), llvm::Loop::getStartLoc(), llvm::MDString::getString(), llvm::ScaledNumbers::getWidth(), llvm::ConstantInt::getZExtValue(), H, I, llvm::SmallSet< T, N, C >::insert(), llvm::DenseMapBase< DenseMap< KeyT, ValueT, KeyInfoT, BucketT >, KeyT, ValueT, KeyInfoT, BucketT >::insert(), llvm::VectorizerParams::isInterleaveForced(), llvm::isPowerOf2_32(), llvm::HexagonMCInstrInfo::isPredicated(), Kind, LI, llvm_unreachable, llvm::DenseMapBase< DenseMap< KeyT, ValueT, KeyInfoT, BucketT >, KeyT, ValueT, KeyInfoT, BucketT >::lookup(), LV_NAME, MaxInterleaveFactor, MaxInterleaveGroupFactor, llvm::VectorizerParams::MaxVectorWidth, llvm::StringRef::npos, ORE, llvm::cl::Prefix, PSE, llvm::SmallVectorTemplateBase< T, isPodLike< T >::value >::push_back(), llvm::MDNode::replaceOperandWith(), llvm::cl::Sink, llvm::SmallVectorTemplateCommon< T >::size(), llvm::AMDGPU::HSAMD::Kernel::Arg::Key::Size, llvm::StringRef::startswith(), llvm::Twine::str(), llvm::StringRef::substr(), llvm::VectorizerParams::VectorizationFactor, and llvm::VectorizerParams::VectorizationInterleave.

◆ addNewMetadata()

void InnerLoopVectorizer::addNewMetadata ( Instruction To,
const Instruction Orig 
)
protected

Add additional metadata to To that was not present on Orig.

Currently this is used to add the noalias annotations based on the inserted memchecks. Use this for instructions that are cloned into the vector loop.

Definition at line 831 of file LoopVectorize.cpp.

References LVer.

Referenced by addMetadata(), and scalarizeInstruction().

◆ areSafetyChecksAdded()

bool llvm::InnerLoopVectorizer::areSafetyChecksAdded ( )
inline

Definition at line 443 of file LoopVectorize.cpp.

References AddedSafetyChecks.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ buildScalarSteps()

void InnerLoopVectorizer::buildScalarSteps ( Value ScalarIV,
Value Step,
Value EntryVal,
const InductionDescriptor ID 
)
protected

Compute scalar induction steps.

ScalarIV is the scalar induction variable on which to base the steps, Step is the size of the step, and EntryVal is the value from the original loop that maps to the steps. Note that EntryVal doesn't have to be an induction variable (e.g., it can be a truncate instruction).

Definition at line 2743 of file LoopVectorize.cpp.

References llvm::MCID::Add, addFastMathFlag(), assert(), Builder, Cost, llvm::IRBuilder< T, Inserter >::CreateBinOp(), llvm::InductionDescriptor::getInductionOpcode(), llvm::getPtrStride(), llvm::Type::getScalarType(), getSignedIntOrFpConstant(), llvm::Value::getType(), llvm::Type::isIntegerTy(), PSE, llvm::VectorizerValueMap::setScalarValue(), UF, VectorLoopValueMap, and VF.

Referenced by widenIntOrFpInduction().

◆ createBitOrPointerCast()

Value * InnerLoopVectorizer::createBitOrPointerCast ( Value V,
VectorType DstVTy,
const DataLayout DL 
)
protected

◆ createBlockInMask()

InnerLoopVectorizer::VectorParts InnerLoopVectorizer::createBlockInMask ( BasicBlock BB)

◆ createEdgeMask()

InnerLoopVectorizer::VectorParts InnerLoopVectorizer::createEdgeMask ( BasicBlock Src,
BasicBlock Dst 
)
protected

◆ createInductionVariable()

PHINode * InnerLoopVectorizer::createInductionVariable ( Loop L,
Value Start,
Value End,
Value Step,
Instruction DL 
)
protected

◆ createVectorIntOrFpInductionPHI()

void InnerLoopVectorizer::createVectorIntOrFpInductionPHI ( const InductionDescriptor II,
Value Step,
Instruction EntryVal 
)
protected

◆ createVectorizedLoopSkeleton()

BasicBlock * InnerLoopVectorizer::createVectorizedLoopSkeleton ( )

Create a new empty loop.

Unlink the old loop and connect the new one. Return the pre-header block of the new loop.

Definition at line 3492 of file LoopVectorize.cpp.

References llvm::LoopBase< BlockT, LoopT >::addBasicBlockToLoop(), llvm::LoopBase< BlockT, LoopT >::addChildLoop(), llvm::PHINode::addIncoming(), llvm::LoopInfoBase< BlockT, LoopT >::addTopLevelLoop(), llvm::LoopInfoBase< BlockT, LoopT >::AllocateLoop(), assert(), B, Builder, llvm::CmpInst::Create(), llvm::PHINode::Create(), llvm::BranchInst::Create(), createInductionVariable(), emitMemRuntimeChecks(), emitMinimumIterationCountCheck(), emitSCEVChecks(), llvm::ConstantInt::get(), llvm::PHINode::getBasicBlockIndex(), llvm::CastInst::getCastOpcode(), llvm::Module::getDataLayout(), getDebugLocFromInstOrOperands(), llvm::LoopBase< BlockT, LoopT >::getExitBlock(), llvm::BasicBlock::getFirstInsertionPt(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::Loop::getLoopID(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::BasicBlock::getModule(), getOrCreateTripCount(), getOrCreateVectorTripCount(), llvm::LoopBase< BlockT, LoopT >::getParentLoop(), llvm::PredicatedScalarEvolution::getSE(), llvm::InductionDescriptor::getStartValue(), llvm::InductionDescriptor::getStep(), llvm::BasicBlock::getTerminator(), llvm::SCEV::getType(), llvm::Value::getType(), llvm::CmpInst::ICMP_EQ, Induction, IVEndValues, Legal, LI, List, LoopBypassBlocks, LoopExitBlock, LoopMiddleBlock, LoopScalarBody, LoopScalarPreHeader, LoopVectorBody, LoopVectorPreHeader, OldInduction, ORE, OrigLoop, PSE, llvm::ReplaceInstWithInst(), llvm::PHINode::setIncomingValue(), llvm::IRBuilderBase::SetInsertPoint(), llvm::Loop::setLoopID(), llvm::Value::setName(), llvm::BasicBlock::splitBasicBlock(), llvm::InductionDescriptor::transform(), UF, and VF.

Referenced by llvm::LoopVectorizationPlanner::executePlan(), and InnerLoopVectorizer().

◆ emitMemRuntimeChecks()

void InnerLoopVectorizer::emitMemRuntimeChecks ( Loop L,
BasicBlock Bypass 
)
protected

◆ emitMinimumIterationCountCheck()

void InnerLoopVectorizer::emitMinimumIterationCountCheck ( Loop L,
BasicBlock Bypass 
)
protected

◆ emitSCEVChecks()

void InnerLoopVectorizer::emitSCEVChecks ( Loop L,
BasicBlock Bypass 
)
protected

◆ fixCrossIterationPHIs()

void InnerLoopVectorizer::fixCrossIterationPHIs ( )
protected

Handle all cross-iteration phis in the header.

Definition at line 4060 of file LoopVectorize.cpp.

References llvm::dyn_cast(), fixFirstOrderRecurrence(), fixReduction(), llvm::LoopBase< BlockT, LoopT >::getHeader(), I, Legal, and OrigLoop.

Referenced by fixVectorizedLoop().

◆ fixFirstOrderRecurrence()

void InnerLoopVectorizer::fixFirstOrderRecurrence ( PHINode Phi)
protected

◆ fixLCSSAPHIs()

void InnerLoopVectorizer::fixLCSSAPHIs ( )
protected

The Loop exit block may have single value PHI nodes with some incoming value.

While vectorizing we only handled real values that were defined inside the loop and we should have one value for each predecessor of its parent basic block. See PR14725.

Definition at line 4430 of file LoopVectorize.cpp.

References assert(), llvm::dyn_cast(), llvm::Loop::isLoopInvariant(), LoopExitBlock, LoopMiddleBlock, and OrigLoop.

Referenced by fixVectorizedLoop().

◆ fixReduction()

void InnerLoopVectorizer::fixReduction ( PHINode Phi)
protected

Fix a reduction cross-iteration phi.

This is the second phase of vectorizing this phi node.

Definition at line 4251 of file LoopVectorize.cpp.

References addFastMathFlag(), llvm::PHINode::addIncoming(), assert(), llvm::BasicBlock::begin(), Builder, llvm::PHINode::Create(), llvm::IRBuilder< T, Inserter >::CreateBinOp(), llvm::IRBuilder< T, Inserter >::CreateInsertElement(), llvm::RecurrenceDescriptor::createMinMaxOp(), llvm::IRBuilder< T, Inserter >::CreateSExt(), llvm::createTargetReduction(), llvm::IRBuilder< T, Inserter >::CreateTrunc(), llvm::IRBuilder< T, Inserter >::CreateVectorSplat(), llvm::IRBuilder< T, Inserter >::CreateZExt(), llvm::dyn_cast(), E, llvm::BasicBlock::end(), llvm::VectorType::get(), llvm::PHINode::getBasicBlockIndex(), llvm::BasicBlock::getFirstInsertionPt(), llvm::PHINode::getIncomingValue(), llvm::PHINode::getIncomingValueForBlock(), llvm::IRBuilderBase::getInt32(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::PHINode::getNumIncomingValues(), getOrCreateVectorValue(), llvm::RecurrenceDescriptor::getRecurrenceBinOp(), llvm::RecurrenceDescriptor::getRecurrenceIdentity(), llvm::RecurrenceDescriptor::getRecurrenceKind(), llvm::Type::getScalarType(), llvm::ConstantVector::getSplat(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::VectorizerValueMap::getVectorValue(), I, Legal, LI, LoopBypassBlocks, LoopExitBlock, LoopMiddleBlock, LoopScalarPreHeader, LoopVectorBody, LoopVectorPreHeader, OrigLoop, llvm::VectorizerValueMap::resetVectorValue(), llvm::RecurrenceDescriptor::RK_FloatMinMax, llvm::RecurrenceDescriptor::RK_IntegerMinMax, setDebugLocFromInst(), llvm::PHINode::setIncomingValue(), llvm::IRBuilderBase::SetInsertPoint(), TTI, UF, VectorLoopValueMap, and VF.

Referenced by fixCrossIterationPHIs().

◆ fixupIVUsers()

void InnerLoopVectorizer::fixupIVUsers ( PHINode OrigPhi,
const InductionDescriptor II,
Value CountRoundDown,
Value EndValue,
BasicBlock MiddleBlock 
)
protected

◆ fixVectorizedLoop()

void InnerLoopVectorizer::fixVectorizedLoop ( )

◆ getBroadcastInstrs()

Value * InnerLoopVectorizer::getBroadcastInstrs ( Value V)
protectedvirtual

Create a broadcast instruction.

This method generates a broadcast instruction (shuffle) for loop invariant values and for the induction value. If this is the induction variable then we extend it to N, N+1, ... this is needed because each iteration in the loop corresponds to a SIMD element.

Definition at line 2490 of file LoopVectorize.cpp.

References Builder, llvm::IRBuilder< T, Inserter >::CreateVectorSplat(), llvm::dyn_cast(), llvm::Instruction::getParent(), llvm::BasicBlock::getTerminator(), llvm::Loop::isLoopInvariant(), LoopVectorBody, LoopVectorPreHeader, OrigLoop, llvm::IRBuilderBase::SetInsertPoint(), and VF.

Referenced by getOrCreateVectorValue(), llvm::InnerLoopUnroller::InnerLoopUnroller(), and widenIntOrFpInduction().

◆ getOrCreateScalarValue()

Value * InnerLoopVectorizer::getOrCreateScalarValue ( Value V,
const VPIteration Instance 
)

Return a value in the new loop corresponding to V from the original loop at unroll and vector indices Instance.

If the value has been vectorized but not scalarized, the necessary extractelement instruction will be generated.

Definition at line 2870 of file LoopVectorize.cpp.

References assert(), Builder, Cost, llvm::IRBuilder< T, Inserter >::CreateExtractElement(), llvm::IRBuilderBase::getInt32(), getOrCreateVectorValue(), llvm::VectorizerValueMap::getScalarValue(), llvm::VectorizerValueMap::hasScalarValue(), llvm::Loop::isLoopInvariant(), llvm::VPIteration::Lane, OrigLoop, llvm::VPIteration::Part, VectorLoopValueMap, and VF.

Referenced by scalarizeInstruction(), vectorizeInterleaveGroup(), vectorizeMemoryInstruction(), and widenInstruction().

◆ getOrCreateTripCount()

Value * InnerLoopVectorizer::getOrCreateTripCount ( Loop NewLoop)
protected

◆ getOrCreateVectorTripCount()

Value * InnerLoopVectorizer::getOrCreateVectorTripCount ( Loop NewLoop)
protected

◆ getOrCreateVectorValue()

Value * InnerLoopVectorizer::getOrCreateVectorValue ( Value V,
unsigned  Part 
)

getOrCreateVectorValue and getOrCreateScalarValue coordinate to generate a vector or scalar value on-demand if one is not yet available.

When vectorizing a loop, we visit the definition of an instruction before its uses. When visiting the definition, we either vectorize or scalarize the instruction, creating an entry for it in the corresponding map. (In some cases, such as induction variables, we will create both vector and scalar entries.) Then, as we encounter uses of the definition, we derive values for each scalar or vector use unless such a value is already available. For example, if we scalarize a definition and one of its uses is vector, we build the required vector on-demand with an insertelement sequence when visiting the use. Otherwise, if the use is scalar, we can use the existing scalar definition.

Return a value in the new loop corresponding to V from the original loop at unroll index Part. If the value has already been vectorized, the corresponding vector entry in VectorLoopValueMap is returned. If, however, the value has a scalar entry in VectorLoopValueMap, we construct a new vector value on-demand by inserting the scalar values into a vector with an insertelement sequence. If the value has been neither vectorized nor scalarized, it must be loop invariant, so we simply broadcast the value into a vector.

Definition at line 2797 of file LoopVectorize.cpp.

References assert(), B, Builder, Cost, llvm::ConstantInt::get(), llvm::VectorType::get(), llvm::UndefValue::get(), getBroadcastInstrs(), llvm::VectorizerValueMap::getScalarValue(), llvm::Value::getType(), llvm::VectorizerValueMap::getVectorValue(), llvm::VectorizerValueMap::hasAnyScalarValue(), llvm::VectorizerValueMap::hasVectorValue(), Induction, llvm::Type::isVectorTy(), llvm::Type::isVoidTy(), Legal, packScalarIntoVectorValue(), llvm::IRBuilderBase::restoreIP(), llvm::IRBuilderBase::saveIP(), llvm::IRBuilderBase::SetInsertPoint(), llvm::VectorizerValueMap::setVectorValue(), llvm::RegState::Undef, VectorLoopValueMap, and VF.

Referenced by createEdgeMask(), fixFirstOrderRecurrence(), fixReduction(), getOrCreateScalarValue(), truncateToMinimalBitwidths(), vectorizeInterleaveGroup(), vectorizeMemoryInstruction(), widenInstruction(), and widenPHIInstruction().

◆ getStepVector()

Value * InnerLoopVectorizer::getStepVector ( Value Val,
int  StartIdx,
Value Step,
Instruction::BinaryOps  Opcode = Instruction::BinaryOpsEnd 
)
protectedvirtual

◆ needsScalarInduction()

bool InnerLoopVectorizer::needsScalarInduction ( Instruction IV) const
protected

Returns true if we should generate a scalar version of IV.

Definition at line 2582 of file LoopVectorize.cpp.

References llvm::any_of(), llvm::LoopBase< BlockT, LoopT >::contains(), OrigLoop, shouldScalarizeInstruction(), and llvm::Value::users().

Referenced by widenIntOrFpInduction().

◆ packScalarIntoVectorValue()

void InnerLoopVectorizer::packScalarIntoVectorValue ( Value V,
const VPIteration Instance 
)

◆ reverseVector()

Value * InnerLoopVectorizer::reverseVector ( Value Vec)
protectedvirtual

◆ scalarizeInstruction()

void InnerLoopVectorizer::scalarizeInstruction ( Instruction Instr,
const VPIteration Instance,
bool  IfPredicateInstr 
)

◆ setDebugLocFromInst()

void InnerLoopVectorizer::setDebugLocFromInst ( IRBuilder<> &  B,
const Value Ptr 
)
protected

◆ shouldScalarizeInstruction()

bool InnerLoopVectorizer::shouldScalarizeInstruction ( Instruction I) const
protected

Returns true if an instruction I should be scalarized instead of vectorized for the chosen vectorization factor.

Definition at line 2577 of file LoopVectorize.cpp.

References Cost, and VF.

Referenced by needsScalarInduction(), and widenIntOrFpInduction().

◆ sinkScalarOperands()

void InnerLoopVectorizer::sinkScalarOperands ( Instruction PredInst)
protected

◆ truncateToMinimalBitwidths()

void InnerLoopVectorizer::truncateToMinimalBitwidths ( )
protected

◆ updateAnalysis()

void InnerLoopVectorizer::updateAnalysis ( )
protected

◆ vectorizeInterleaveGroup()

void InnerLoopVectorizer::vectorizeInterleaveGroup ( Instruction Instr)

◆ vectorizeMemoryInstruction()

void InnerLoopVectorizer::vectorizeMemoryInstruction ( Instruction Instr)
protectedvirtual

◆ widenInstruction()

void InnerLoopVectorizer::widenInstruction ( Instruction I)

Widen a single instruction within the innermost loop.

Vectorize casts.

Definition at line 4695 of file LoopVectorize.cpp.

References llvm::MCID::Add, addMetadata(), Arg, llvm::AMDGPU::HSAMD::Kernel::Key::Args, assert(), B, Builder, C, llvm::MCID::Call, llvm::Instruction::copyFastMathFlags(), llvm::Function::Create(), llvm::IRBuilder< T, Inserter >::CreateBinOp(), llvm::IRBuilder< T, Inserter >::CreateCall(), llvm::IRBuilder< T, Inserter >::CreateCast(), llvm::IRBuilder< T, Inserter >::CreateFCmp(), llvm::IRBuilder< T, Inserter >::CreateGEP(), llvm::IRBuilder< T, Inserter >::CreateICmp(), llvm::IRBuilder< T, Inserter >::CreateInBoundsGEP(), llvm::IRBuilder< T, Inserter >::CreateSelect(), llvm::IRBuilder< T, Inserter >::CreateVectorSplat(), llvm::dbgs(), DEBUG, llvm::dyn_cast(), llvm::StringRef::empty(), llvm::GlobalValue::ExternalLinkage, GEP, llvm::FunctionType::get(), llvm::VectorType::get(), llvm::Intrinsic::getDeclaration(), llvm::Module::getFunction(), llvm::Instruction::getOpcode(), llvm::User::getOperand(), getOrCreateScalarValue(), getOrCreateVectorValue(), llvm::Instruction::getParent(), llvm::BasicBlock::getParent(), llvm::GlobalValue::getParent(), llvm::PredicatedScalarEvolution::getSCEV(), llvm::PredicatedScalarEvolution::getSE(), getVectorCallCost(), getVectorIntrinsicCost(), llvm::getVectorIntrinsicIDForCall(), llvm::TargetLibraryInfo::getVectorizedFunction(), llvm::Loop::hasLoopInvariantOperands(), llvm::hasVectorInstrinsicScalarOpd(), I, llvm::IRBuilder< T, Inserter >::Insert(), llvm::Loop::isLoopInvariant(), llvm::ScalarEvolution::isLoopInvariant(), llvm_unreachable, llvm::SPII::Load, llvm::make_range(), OrigLoop, PSE, llvm::SmallVectorTemplateBase< T, isPodLike< T >::value >::push_back(), llvm::SmallVectorTemplateBase< T, isPodLike >::push_back(), llvm::MCID::Select, setDebugLocFromInst(), llvm::IRBuilderBase::setFastMathFlags(), llvm::VectorizerValueMap::setVectorValue(), llvm::SPII::Store, TLI, ToVectorTy(), TTI, UF, vectorizeMemoryInstruction(), VectorLoopValueMap, and VF.

Referenced by AddRuntimeUnrollDisableMetaData(), and InnerLoopVectorizer().

◆ widenIntOrFpInduction()

void InnerLoopVectorizer::widenIntOrFpInduction ( PHINode IV,
TruncInst Trunc = nullptr 
)

◆ widenPHIInstruction()

void InnerLoopVectorizer::widenPHIInstruction ( Instruction PN,
unsigned  UF,
unsigned  VF 
)

Friends And Related Function Documentation

◆ LoopVectorizationPlanner

friend class LoopVectorizationPlanner
friend

Definition at line 508 of file LoopVectorize.cpp.

Member Data Documentation

◆ AA

AliasAnalysis* llvm::InnerLoopVectorizer::AA
protected

◆ AC

AssumptionCache* llvm::InnerLoopVectorizer::AC
protected

◆ AddedSafetyChecks

bool llvm::InnerLoopVectorizer::AddedSafetyChecks = false
protected

Definition at line 755 of file LoopVectorize.cpp.

Referenced by areSafetyChecksAdded(), emitMemRuntimeChecks(), and emitSCEVChecks().

◆ BlockMaskCache

BlockMaskCacheTy llvm::InnerLoopVectorizer::BlockMaskCache
protected

Definition at line 740 of file LoopVectorize.cpp.

Referenced by createBlockInMask().

◆ Builder

IRBuilder llvm::InnerLoopVectorizer::Builder
protected

◆ Cost

LoopVectorizationCostModel* llvm::InnerLoopVectorizer::Cost
protected

◆ DT

DominatorTree* llvm::InnerLoopVectorizer::DT
protected

◆ EdgeMaskCache

EdgeMaskCacheTy llvm::InnerLoopVectorizer::EdgeMaskCache
protected

Definition at line 739 of file LoopVectorize.cpp.

Referenced by createEdgeMask().

◆ Induction

PHINode* llvm::InnerLoopVectorizer::Induction = nullptr
protected

◆ IVEndValues

DenseMap<PHINode *, Value *> llvm::InnerLoopVectorizer::IVEndValues
protected

Definition at line 759 of file LoopVectorize.cpp.

Referenced by createVectorizedLoopSkeleton(), and fixVectorizedLoop().

◆ Legal

LoopVectorizationLegality* llvm::InnerLoopVectorizer::Legal
protected

◆ LI

LoopInfo* llvm::InnerLoopVectorizer::LI
protected

◆ LoopBypassBlocks

SmallVector<BasicBlock *, 4> llvm::InnerLoopVectorizer::LoopBypassBlocks
protected

A list of all bypass blocks. The first block is the entry of the loop.

Definition at line 722 of file LoopVectorize.cpp.

Referenced by createVectorizedLoopSkeleton(), emitMemRuntimeChecks(), emitMinimumIterationCountCheck(), emitSCEVChecks(), fixReduction(), and updateAnalysis().

◆ LoopExitBlock

BasicBlock* llvm::InnerLoopVectorizer::LoopExitBlock
protected

The ExitBlock of the scalar loop.

Definition at line 713 of file LoopVectorize.cpp.

Referenced by createVectorizedLoopSkeleton(), fixFirstOrderRecurrence(), fixLCSSAPHIs(), fixReduction(), and updateAnalysis().

◆ LoopMiddleBlock

BasicBlock* llvm::InnerLoopVectorizer::LoopMiddleBlock
protected

Middle Block between the vector and the scalar.

Definition at line 710 of file LoopVectorize.cpp.

Referenced by createVectorizedLoopSkeleton(), fixFirstOrderRecurrence(), fixLCSSAPHIs(), fixReduction(), fixVectorizedLoop(), and updateAnalysis().

◆ LoopScalarBody

BasicBlock* llvm::InnerLoopVectorizer::LoopScalarBody
protected

The scalar loop body.

Definition at line 719 of file LoopVectorize.cpp.

Referenced by createVectorizedLoopSkeleton(), and updateAnalysis().

◆ LoopScalarPreHeader

BasicBlock* llvm::InnerLoopVectorizer::LoopScalarPreHeader
protected

The scalar-loop preheader.

Definition at line 707 of file LoopVectorize.cpp.

Referenced by createVectorizedLoopSkeleton(), fixFirstOrderRecurrence(), fixReduction(), and updateAnalysis().

◆ LoopVectorBody

BasicBlock* llvm::InnerLoopVectorizer::LoopVectorBody
protected

◆ LoopVectorPreHeader

BasicBlock* llvm::InnerLoopVectorizer::LoopVectorPreHeader
protected

◆ LVer

std::unique_ptr<LoopVersioning> llvm::InnerLoopVectorizer::LVer
protected

LoopVersioning.

It's only set up (non-null) if memchecks were used.

This is currently only used to add no-alias metadata based on the memchecks. The actually versioning is performed manually.

Definition at line 688 of file LoopVectorize.cpp.

Referenced by addNewMetadata(), and emitMemRuntimeChecks().

◆ OldInduction

PHINode* llvm::InnerLoopVectorizer::OldInduction = nullptr
protected

The induction variable of the old basic block.

Definition at line 728 of file LoopVectorize.cpp.

Referenced by createInductionVariable(), createVectorizedLoopSkeleton(), and widenIntOrFpInduction().

◆ ORE

OptimizationRemarkEmitter* llvm::InnerLoopVectorizer::ORE
protected

◆ OrigLoop

Loop* llvm::InnerLoopVectorizer::OrigLoop
protected

◆ PredicatedInstructions

SmallVector<Instruction *, 4> llvm::InnerLoopVectorizer::PredicatedInstructions
protected

Store instructions that were predicated.

Definition at line 737 of file LoopVectorize.cpp.

Referenced by fixVectorizedLoop(), and scalarizeInstruction().

◆ PSE

PredicatedScalarEvolution& llvm::InnerLoopVectorizer::PSE
protected

◆ TLI

const TargetLibraryInfo* llvm::InnerLoopVectorizer::TLI
protected

◆ TripCount

Value* llvm::InnerLoopVectorizer::TripCount = nullptr
protected

Trip count of the original loop.

Definition at line 743 of file LoopVectorize.cpp.

Referenced by getOrCreateTripCount().

◆ TTI

const TargetTransformInfo* llvm::InnerLoopVectorizer::TTI
protected

◆ UF

unsigned llvm::InnerLoopVectorizer::UF
protected

◆ VectorLoopValueMap

VectorizerValueMap llvm::InnerLoopVectorizer::VectorLoopValueMap
protected

Maps values from the original loop to their corresponding values in the vectorized loop.

A key value can map to either vector values, scalar values or both kinds of values, depending on whether the key was vectorized and scalarized.

Definition at line 734 of file LoopVectorize.cpp.

Referenced by buildScalarSteps(), createVectorIntOrFpInductionPHI(), llvm::LoopVectorizationPlanner::executePlan(), fixFirstOrderRecurrence(), fixReduction(), getOrCreateScalarValue(), getOrCreateVectorValue(), packScalarIntoVectorValue(), scalarizeInstruction(), truncateToMinimalBitwidths(), vectorizeInterleaveGroup(), vectorizeMemoryInstruction(), widenInstruction(), widenIntOrFpInduction(), and widenPHIInstruction().

◆ VectorTripCount

Value* llvm::InnerLoopVectorizer::VectorTripCount = nullptr
protected

Trip count of the widened loop (TripCount - TripCount % (VF*UF))

Definition at line 746 of file LoopVectorize.cpp.

Referenced by getOrCreateVectorTripCount().

◆ VF

unsigned llvm::InnerLoopVectorizer::VF
protected

The documentation for this class was generated from the following file: