LLVM  6.0.0svn
Public Types | Public Member Functions | Protected Types | Protected Member Functions | Protected Attributes | Friends | List of all members
llvm::InnerLoopVectorizer Class Reference

InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF). More...

Inheritance diagram for llvm::InnerLoopVectorizer:
Inheritance graph
[legend]
Collaboration diagram for llvm::InnerLoopVectorizer:
Collaboration graph
[legend]

Public Types

using VectorParts = SmallVector< Value *, 2 >
 A type for vectorized values in the new loop. More...
 

Public Member Functions

 InnerLoopVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, unsigned VecWidth, unsigned UnrollFactor, LoopVectorizationLegality *LVL, LoopVectorizationCostModel *CM)
 
virtual ~InnerLoopVectorizer ()=default
 
BasicBlockcreateVectorizedLoopSkeleton ()
 Create a new empty loop. More...
 
void widenInstruction (Instruction &I)
 Widen a single instruction within the innermost loop. More...
 
void fixVectorizedLoop ()
 Fix the vectorized code, taking care of header phi's, live-outs, and more. More...
 
bool areSafetyChecksAdded ()
 
void widenPHIInstruction (Instruction *PN, unsigned UF, unsigned VF)
 Vectorize a single PHINode in a block. More...
 
void scalarizeInstruction (Instruction *Instr, const VPIteration &Instance, bool IfPredicateInstr)
 A helper function to scalarize a single Instruction in the innermost loop. More...
 
void widenIntOrFpInduction (PHINode *IV, TruncInst *Trunc=nullptr)
 Widen an integer or floating-point induction variable IV. More...
 
ValuegetOrCreateVectorValue (Value *V, unsigned Part)
 getOrCreateVectorValue and getOrCreateScalarValue coordinate to generate a vector or scalar value on-demand if one is not yet available. More...
 
ValuegetOrCreateScalarValue (Value *V, const VPIteration &Instance)
 Return a value in the new loop corresponding to V from the original loop at unroll and vector indices Instance. More...
 
void packScalarIntoVectorValue (Value *V, const VPIteration &Instance)
 Construct the vector value of a scalarized value V one lane at a time. More...
 
void vectorizeInterleaveGroup (Instruction *Instr)
 Try to vectorize the interleaved access group that Instr belongs to. More...
 
void vectorizeMemoryInstruction (Instruction *Instr, VectorParts *BlockInMask=nullptr)
 Vectorize Load and Store instructions, optionally masking the vector operations if BlockInMask is non-null. More...
 
void setDebugLocFromInst (IRBuilder<> &B, const Value *Ptr)
 Set the debug location in the builder using the debug location in the instruction. More...
 

Protected Types

using PhiVector = SmallVector< PHINode *, 4 >
 A small list of PHINodes. More...
 
using ScalarParts = SmallVector< SmallVector< Value *, 4 >, 2 >
 A type for scalarized values in the new loop. More...
 

Protected Member Functions

void fixupIVUsers (PHINode *OrigPhi, const InductionDescriptor &II, Value *CountRoundDown, Value *EndValue, BasicBlock *MiddleBlock)
 Set up the values of the IVs correctly when exiting the vector loop. More...
 
PHINodecreateInductionVariable (Loop *L, Value *Start, Value *End, Value *Step, Instruction *DL)
 Create a new induction variable inside L. More...
 
void fixCrossIterationPHIs ()
 Handle all cross-iteration phis in the header. More...
 
void fixFirstOrderRecurrence (PHINode *Phi)
 Fix a first-order recurrence. More...
 
void fixReduction (PHINode *Phi)
 Fix a reduction cross-iteration phi. More...
 
void fixLCSSAPHIs ()
 The Loop exit block may have single value PHI nodes with some incoming value. More...
 
void sinkScalarOperands (Instruction *PredInst)
 Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it. More...
 
void truncateToMinimalBitwidths ()
 Shrinks vector element sizes to the smallest bitwidth they can be legally represented as. More...
 
void updateAnalysis ()
 Insert the new loop to the loop hierarchy and pass manager and update the analysis passes. More...
 
virtual ValuegetBroadcastInstrs (Value *V)
 Create a broadcast instruction. More...
 
virtual ValuegetStepVector (Value *Val, int StartIdx, Value *Step, Instruction::BinaryOps Opcode=Instruction::BinaryOpsEnd)
 This function adds (StartIdx, StartIdx + Step, StartIdx + 2*Step, ...) to each vector element of Val. More...
 
void buildScalarSteps (Value *ScalarIV, Value *Step, Value *EntryVal, const InductionDescriptor &ID)
 Compute scalar induction steps. More...
 
void createVectorIntOrFpInductionPHI (const InductionDescriptor &II, Value *Step, Instruction *EntryVal)
 Create a vector induction phi node based on an existing scalar one. More...
 
bool shouldScalarizeInstruction (Instruction *I) const
 Returns true if an instruction I should be scalarized instead of vectorized for the chosen vectorization factor. More...
 
bool needsScalarInduction (Instruction *IV) const
 Returns true if we should generate a scalar version of IV. More...
 
void recordVectorLoopValueForInductionCast (const InductionDescriptor &ID, Value *VectorLoopValue, unsigned Part, unsigned Lane=UINT_MAX)
 If there is a cast involved in the induction variable ID, which should be ignored in the vectorized loop body, this function records the VectorLoopValue of the respective Phi also as the VectorLoopValue of the cast. More...
 
virtual ValuereverseVector (Value *Vec)
 Generate a shuffle sequence that will reverse the vector Vec. More...
 
ValuegetOrCreateTripCount (Loop *NewLoop)
 Returns (and creates if needed) the original loop trip count. More...
 
ValuegetOrCreateVectorTripCount (Loop *NewLoop)
 Returns (and creates if needed) the trip count of the widened loop. More...
 
ValuecreateBitOrPointerCast (Value *V, VectorType *DstVTy, const DataLayout &DL)
 Returns a bitcasted value to the requested vector type. More...
 
void emitMinimumIterationCountCheck (Loop *L, BasicBlock *Bypass)
 Emit a bypass check to see if the vector trip count is zero, including if it overflows. More...
 
void emitSCEVChecks (Loop *L, BasicBlock *Bypass)
 Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct. More...
 
void emitMemRuntimeChecks (Loop *L, BasicBlock *Bypass)
 Emit bypass checks to check any memory assumptions we may have made. More...
 
void addNewMetadata (Instruction *To, const Instruction *Orig)
 Add additional metadata to To that was not present on Orig. More...
 
void addMetadata (Instruction *To, Instruction *From)
 Add metadata from one instruction to another. More...
 
void addMetadata (ArrayRef< Value *> To, Instruction *From)
 Similar to the previous function but it adds the metadata to a vector of instructions. More...
 

Protected Attributes

LoopOrigLoop
 The original loop. More...
 
PredicatedScalarEvolutionPSE
 A wrapper around ScalarEvolution used to add runtime SCEV checks. More...
 
LoopInfoLI
 Loop Info. More...
 
DominatorTreeDT
 Dominator Tree. More...
 
AliasAnalysisAA
 Alias Analysis. More...
 
const TargetLibraryInfoTLI
 Target Library Info. More...
 
const TargetTransformInfoTTI
 Target Transform Info. More...
 
AssumptionCacheAC
 Assumption Cache. More...
 
OptimizationRemarkEmitterORE
 Interface to emit optimization remarks. More...
 
std::unique_ptr< LoopVersioningLVer
 LoopVersioning. More...
 
unsigned VF
 The vectorization SIMD factor to use. More...
 
unsigned UF
 The vectorization unroll factor to use. More...
 
IRBuilder Builder
 The builder that we use. More...
 
BasicBlockLoopVectorPreHeader
 The vector-loop preheader. More...
 
BasicBlockLoopScalarPreHeader
 The scalar-loop preheader. More...
 
BasicBlockLoopMiddleBlock
 Middle Block between the vector and the scalar. More...
 
BasicBlockLoopExitBlock
 The ExitBlock of the scalar loop. More...
 
BasicBlockLoopVectorBody
 The vector loop body. More...
 
BasicBlockLoopScalarBody
 The scalar loop body. More...
 
SmallVector< BasicBlock *, 4 > LoopBypassBlocks
 A list of all bypass blocks. The first block is the entry of the loop. More...
 
PHINodeInduction = nullptr
 The new Induction variable which was added to the new block. More...
 
PHINodeOldInduction = nullptr
 The induction variable of the old basic block. More...
 
VectorizerValueMap VectorLoopValueMap
 Maps values from the original loop to their corresponding values in the vectorized loop. More...
 
SmallVector< Instruction *, 4 > PredicatedInstructions
 Store instructions that were predicated. More...
 
ValueTripCount = nullptr
 Trip count of the original loop. More...
 
ValueVectorTripCount = nullptr
 Trip count of the widened loop (TripCount - TripCount % (VF*UF)) More...
 
LoopVectorizationLegality * Legal
 The legality analysis. More...
 
LoopVectorizationCostModel * Cost
 The profitablity analysis. More...
 
bool AddedSafetyChecks = false
 
DenseMap< PHINode *, Value * > IVEndValues
 

Friends

class LoopVectorizationPlanner
 

Detailed Description

InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF).

This class performs the widening of scalars into vectors, or multiple scalars. This class also implements the following features:

Definition at line 421 of file LoopVectorize.cpp.

Member Typedef Documentation

◆ PhiVector

A small list of PHINodes.

Definition at line 519 of file LoopVectorize.cpp.

◆ ScalarParts

A type for scalarized values in the new loop.

Each value from the original loop, when scalarized, is represented by UF x VF scalar values in the new unrolled loop, where UF is the unroll factor and VF is the vectorization factor.

Definition at line 525 of file LoopVectorize.cpp.

◆ VectorParts

A type for vectorized values in the new loop.

Each value from the original loop, when vectorized, is represented by UF vector values in the new unrolled loop, where UF is the unroll factor.

Definition at line 452 of file LoopVectorize.cpp.

Constructor & Destructor Documentation

◆ InnerLoopVectorizer()

llvm::InnerLoopVectorizer::InnerLoopVectorizer ( Loop OrigLoop,
PredicatedScalarEvolution PSE,
LoopInfo LI,
DominatorTree DT,
const TargetLibraryInfo TLI,
const TargetTransformInfo TTI,
AssumptionCache AC,
OptimizationRemarkEmitter ORE,
unsigned  VecWidth,
unsigned  UnrollFactor,
LoopVectorizationLegality *  LVL,
LoopVectorizationCostModel *  CM 
)
inline

◆ ~InnerLoopVectorizer()

virtual llvm::InnerLoopVectorizer::~InnerLoopVectorizer ( )
virtualdefault

Referenced by InnerLoopVectorizer().

Member Function Documentation

◆ addMetadata() [1/2]

void InnerLoopVectorizer::addMetadata ( Instruction To,
Instruction From 
)
protected

Add metadata from one instruction to another.

This includes both the original MDs from From and additional ones (

See also
addNewMetadata). Use this for newly created instructions in the vector loop.

Definition at line 842 of file LoopVectorize.cpp.

References addNewMetadata(), and llvm::propagateMetadata().

Referenced by addMetadata(), createVectorIntOrFpInductionPHI(), vectorizeMemoryInstruction(), widenInstruction(), and widenIntOrFpInduction().

◆ addMetadata() [2/2]

void InnerLoopVectorizer::addMetadata ( ArrayRef< Value *>  To,
Instruction From 
)
protected

Similar to the previous function but it adds the metadata to a vector of instructions.

Definition at line 848 of file LoopVectorize.cpp.

References llvm::abs(), addMetadata(), llvm::AMDGPU::HSAMD::Kernel::Arg::Key::Align, llvm::OptimizationRemarkAnalysis::AlwaysPrint, Arg, llvm::AMDGPU::HSAMD::Kernel::Key::Args, assert(), B, llvm::LoopAccessInfo::blockNeedsPredication(), C, Context, llvm::DenseMapBase< DenseMap< KeyT, ValueT, KeyInfoT, BucketT >, KeyT, ValueT, KeyInfoT, BucketT >::count(), llvm::dbgs(), DEBUG, DT, llvm::dyn_cast(), llvm::OptimizationRemarkEmitter::emit(), llvm::ArrayRef< T >::empty(), llvm::StringRef::endswith(), llvm::DenseMapBase< DenseMap< KeyT, ValueT, KeyInfoT, BucketT >, KeyT, ValueT, KeyInfoT, BucketT >::erase(), F(), llvm::ConstantInt::get(), llvm::ConstantAsMetadata::get(), llvm::MDString::get(), llvm::MDNode::get(), getAlignment(), llvm::LoopAccessInfo::getDepChecker(), llvm::MemoryDepChecker::getDependences(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::Type::getInt32Ty(), llvm::MDNode::getNumOperands(), llvm::MDNode::getOperand(), llvm::Loop::getStartLoc(), llvm::MDString::getString(), llvm::ScaledNumbers::getWidth(), llvm::ConstantInt::getZExtValue(), H, I, llvm::SmallSet< T, N, C >::insert(), llvm::DenseMapBase< DenseMap< KeyT, ValueT, KeyInfoT, BucketT >, KeyT, ValueT, KeyInfoT, BucketT >::insert(), llvm::VectorizerParams::isInterleaveForced(), llvm::isPowerOf2_32(), llvm::HexagonMCInstrInfo::isPredicated(), Kind, LI, llvm_unreachable, llvm::DenseMapBase< DenseMap< KeyT, ValueT, KeyInfoT, BucketT >, KeyT, ValueT, KeyInfoT, BucketT >::lookup(), LV_NAME, MaxInterleaveFactor, MaxInterleaveGroupFactor, llvm::VectorizerParams::MaxVectorWidth, llvm::StringRef::npos, ORE, llvm::cl::Prefix, llvm::propagateMetadata(), PSE, llvm::SmallVectorTemplateBase< T, isPodLike< T >::value >::push_back(), llvm::MDNode::replaceOperandWith(), llvm::cl::Sink, llvm::SmallVectorTemplateCommon< T >::size(), llvm::AMDGPU::HSAMD::Kernel::Arg::Key::Size, llvm::StringRef::startswith(), llvm::Twine::str(), llvm::StringRef::substr(), llvm::transform(), llvm::VectorizerParams::VectorizationFactor, and llvm::VectorizerParams::VectorizationInterleave.

◆ addNewMetadata()

void InnerLoopVectorizer::addNewMetadata ( Instruction To,
const Instruction Orig 
)
protected

Add additional metadata to To that was not present on Orig.

Currently this is used to add the noalias annotations based on the inserted memchecks. Use this for instructions that are cloned into the vector loop.

Definition at line 834 of file LoopVectorize.cpp.

References LVer.

Referenced by addMetadata(), and scalarizeInstruction().

◆ areSafetyChecksAdded()

bool llvm::InnerLoopVectorizer::areSafetyChecksAdded ( )
inline

Definition at line 447 of file LoopVectorize.cpp.

References AddedSafetyChecks.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ buildScalarSteps()

void InnerLoopVectorizer::buildScalarSteps ( Value ScalarIV,
Value Step,
Value EntryVal,
const InductionDescriptor ID 
)
protected

Compute scalar induction steps.

ScalarIV is the scalar induction variable on which to base the steps, Step is the size of the step, and EntryVal is the value from the original loop that maps to the steps. Note that EntryVal doesn't have to be an induction variable (e.g., it can be a truncate instruction).

Definition at line 2835 of file LoopVectorize.cpp.

References llvm::MCID::Add, addFastMathFlag(), assert(), Builder, Cost, llvm::IRBuilder< T, Inserter >::CreateBinOp(), llvm::InductionDescriptor::getInductionOpcode(), llvm::getPtrStride(), llvm::Type::getScalarType(), getSignedIntOrFpConstant(), llvm::Value::getType(), llvm::Type::isIntegerTy(), PSE, recordVectorLoopValueForInductionCast(), llvm::VectorizerValueMap::setScalarValue(), UF, VectorLoopValueMap, and VF.

Referenced by widenIntOrFpInduction().

◆ createBitOrPointerCast()

Value * InnerLoopVectorizer::createBitOrPointerCast ( Value V,
VectorType DstVTy,
const DataLayout DL 
)
protected

◆ createInductionVariable()

PHINode * InnerLoopVectorizer::createInductionVariable ( Loop L,
Value Start,
Value End,
Value Step,
Instruction DL 
)
protected

◆ createVectorIntOrFpInductionPHI()

void InnerLoopVectorizer::createVectorIntOrFpInductionPHI ( const InductionDescriptor II,
Value Step,
Instruction EntryVal 
)
protected

◆ createVectorizedLoopSkeleton()

BasicBlock * InnerLoopVectorizer::createVectorizedLoopSkeleton ( )

Create a new empty loop.

Unlink the old loop and connect the new one. Return the pre-header block of the new loop.

Definition at line 3591 of file LoopVectorize.cpp.

References llvm::LoopBase< BlockT, LoopT >::addBasicBlockToLoop(), llvm::LoopBase< BlockT, LoopT >::addChildLoop(), llvm::PHINode::addIncoming(), llvm::LoopInfoBase< BlockT, LoopT >::addTopLevelLoop(), llvm::LoopInfoBase< BlockT, LoopT >::AllocateLoop(), assert(), B, Builder, llvm::CmpInst::Create(), llvm::PHINode::Create(), llvm::BranchInst::Create(), createInductionVariable(), emitMemRuntimeChecks(), emitMinimumIterationCountCheck(), emitSCEVChecks(), llvm::ConstantInt::get(), llvm::PHINode::getBasicBlockIndex(), llvm::CastInst::getCastOpcode(), llvm::Module::getDataLayout(), getDebugLocFromInstOrOperands(), llvm::LoopBase< BlockT, LoopT >::getExitBlock(), llvm::BasicBlock::getFirstInsertionPt(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::Loop::getLoopID(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::BasicBlock::getModule(), getOrCreateTripCount(), getOrCreateVectorTripCount(), llvm::LoopBase< BlockT, LoopT >::getParentLoop(), llvm::PredicatedScalarEvolution::getSE(), llvm::InductionDescriptor::getStartValue(), llvm::InductionDescriptor::getStep(), llvm::BasicBlock::getTerminator(), llvm::SCEV::getType(), llvm::Value::getType(), llvm::CmpInst::ICMP_EQ, Induction, IVEndValues, Legal, LI, List, LoopBypassBlocks, LoopExitBlock, LoopMiddleBlock, LoopScalarBody, LoopScalarPreHeader, LoopVectorBody, LoopVectorPreHeader, OldInduction, ORE, OrigLoop, PSE, llvm::ReplaceInstWithInst(), llvm::PHINode::setIncomingValue(), llvm::IRBuilderBase::SetInsertPoint(), llvm::Loop::setLoopID(), llvm::Value::setName(), llvm::BasicBlock::splitBasicBlock(), llvm::InductionDescriptor::transform(), UF, and VF.

Referenced by llvm::LoopVectorizationPlanner::executePlan(), and InnerLoopVectorizer().

◆ emitMemRuntimeChecks()

void InnerLoopVectorizer::emitMemRuntimeChecks ( Loop L,
BasicBlock Bypass 
)
protected

◆ emitMinimumIterationCountCheck()

void InnerLoopVectorizer::emitMinimumIterationCountCheck ( Loop L,
BasicBlock Bypass 
)
protected

◆ emitSCEVChecks()

void InnerLoopVectorizer::emitSCEVChecks ( Loop L,
BasicBlock Bypass 
)
protected

◆ fixCrossIterationPHIs()

void InnerLoopVectorizer::fixCrossIterationPHIs ( )
protected

Handle all cross-iteration phis in the header.

Definition at line 4159 of file LoopVectorize.cpp.

References llvm::dyn_cast(), fixFirstOrderRecurrence(), fixReduction(), llvm::LoopBase< BlockT, LoopT >::getHeader(), I, Legal, and OrigLoop.

Referenced by fixVectorizedLoop().

◆ fixFirstOrderRecurrence()

void InnerLoopVectorizer::fixFirstOrderRecurrence ( PHINode Phi)
protected

◆ fixLCSSAPHIs()

void InnerLoopVectorizer::fixLCSSAPHIs ( )
protected

The Loop exit block may have single value PHI nodes with some incoming value.

While vectorizing we only handled real values that were defined inside the loop and we should have one value for each predecessor of its parent basic block. See PR14725.

Definition at line 4529 of file LoopVectorize.cpp.

References assert(), llvm::dyn_cast(), llvm::Loop::isLoopInvariant(), LoopExitBlock, LoopMiddleBlock, and OrigLoop.

Referenced by fixVectorizedLoop().

◆ fixReduction()

void InnerLoopVectorizer::fixReduction ( PHINode Phi)
protected

Fix a reduction cross-iteration phi.

This is the second phase of vectorizing this phi node.

Definition at line 4350 of file LoopVectorize.cpp.

References addFastMathFlag(), llvm::PHINode::addIncoming(), assert(), llvm::BasicBlock::begin(), Builder, llvm::PHINode::Create(), llvm::IRBuilder< T, Inserter >::CreateBinOp(), llvm::IRBuilder< T, Inserter >::CreateInsertElement(), llvm::RecurrenceDescriptor::createMinMaxOp(), llvm::IRBuilder< T, Inserter >::CreateSExt(), llvm::createTargetReduction(), llvm::IRBuilder< T, Inserter >::CreateTrunc(), llvm::IRBuilder< T, Inserter >::CreateVectorSplat(), llvm::IRBuilder< T, Inserter >::CreateZExt(), llvm::dyn_cast(), E, llvm::BasicBlock::end(), llvm::VectorType::get(), llvm::PHINode::getBasicBlockIndex(), llvm::BasicBlock::getFirstInsertionPt(), llvm::PHINode::getIncomingValue(), llvm::PHINode::getIncomingValueForBlock(), llvm::IRBuilderBase::getInt32(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::PHINode::getNumIncomingValues(), getOrCreateVectorValue(), llvm::RecurrenceDescriptor::getRecurrenceBinOp(), llvm::RecurrenceDescriptor::getRecurrenceIdentity(), llvm::RecurrenceDescriptor::getRecurrenceKind(), llvm::Type::getScalarType(), llvm::ConstantVector::getSplat(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::VectorizerValueMap::getVectorValue(), I, Legal, LI, LoopBypassBlocks, LoopExitBlock, LoopMiddleBlock, LoopScalarPreHeader, LoopVectorBody, LoopVectorPreHeader, OrigLoop, llvm::VectorizerValueMap::resetVectorValue(), llvm::RecurrenceDescriptor::RK_FloatMinMax, llvm::RecurrenceDescriptor::RK_IntegerMinMax, setDebugLocFromInst(), llvm::PHINode::setIncomingValue(), llvm::IRBuilderBase::SetInsertPoint(), TTI, UF, VectorLoopValueMap, and VF.

Referenced by fixCrossIterationPHIs().

◆ fixupIVUsers()

void InnerLoopVectorizer::fixupIVUsers ( PHINode OrigPhi,
const InductionDescriptor II,
Value CountRoundDown,
Value EndValue,
BasicBlock MiddleBlock 
)
protected

◆ fixVectorizedLoop()

void InnerLoopVectorizer::fixVectorizedLoop ( )

◆ getBroadcastInstrs()

Value * InnerLoopVectorizer::getBroadcastInstrs ( Value V)
protectedvirtual

Create a broadcast instruction.

This method generates a broadcast instruction (shuffle) for loop invariant values and for the induction value. If this is the induction variable then we extend it to N, N+1, ... this is needed because each iteration in the loop corresponds to a SIMD element.

Definition at line 2564 of file LoopVectorize.cpp.

References Builder, llvm::IRBuilder< T, Inserter >::CreateVectorSplat(), llvm::dyn_cast(), llvm::Instruction::getParent(), llvm::BasicBlock::getTerminator(), llvm::Loop::isLoopInvariant(), LoopVectorBody, LoopVectorPreHeader, OrigLoop, llvm::IRBuilderBase::SetInsertPoint(), and VF.

Referenced by getOrCreateVectorValue(), llvm::InnerLoopUnroller::InnerLoopUnroller(), and widenIntOrFpInduction().

◆ getOrCreateScalarValue()

Value * InnerLoopVectorizer::getOrCreateScalarValue ( Value V,
const VPIteration Instance 
)

Return a value in the new loop corresponding to V from the original loop at unroll and vector indices Instance.

If the value has been vectorized but not scalarized, the necessary extractelement instruction will be generated.

Definition at line 2963 of file LoopVectorize.cpp.

References assert(), Builder, Cost, llvm::IRBuilder< T, Inserter >::CreateExtractElement(), llvm::IRBuilderBase::getInt32(), getOrCreateVectorValue(), llvm::VectorizerValueMap::getScalarValue(), llvm::VectorizerValueMap::hasScalarValue(), llvm::Loop::isLoopInvariant(), llvm::VPIteration::Lane, OrigLoop, llvm::VPIteration::Part, VectorLoopValueMap, and VF.

Referenced by scalarizeInstruction(), vectorizeInterleaveGroup(), vectorizeMemoryInstruction(), and widenInstruction().

◆ getOrCreateTripCount()

Value * InnerLoopVectorizer::getOrCreateTripCount ( Loop NewLoop)
protected

◆ getOrCreateVectorTripCount()

Value * InnerLoopVectorizer::getOrCreateVectorTripCount ( Loop NewLoop)
protected

◆ getOrCreateVectorValue()

Value * InnerLoopVectorizer::getOrCreateVectorValue ( Value V,
unsigned  Part 
)

getOrCreateVectorValue and getOrCreateScalarValue coordinate to generate a vector or scalar value on-demand if one is not yet available.

When vectorizing a loop, we visit the definition of an instruction before its uses. When visiting the definition, we either vectorize or scalarize the instruction, creating an entry for it in the corresponding map. (In some cases, such as induction variables, we will create both vector and scalar entries.) Then, as we encounter uses of the definition, we derive values for each scalar or vector use unless such a value is already available. For example, if we scalarize a definition and one of its uses is vector, we build the required vector on-demand with an insertelement sequence when visiting the use. Otherwise, if the use is scalar, we can use the existing scalar definition.

Return a value in the new loop corresponding to V from the original loop at unroll index Part. If the value has already been vectorized, the corresponding vector entry in VectorLoopValueMap is returned. If, however, the value has a scalar entry in VectorLoopValueMap, we construct a new vector value on-demand by inserting the scalar values into a vector with an insertelement sequence. If the value has been neither vectorized nor scalarized, it must be loop invariant, so we simply broadcast the value into a vector.

Definition at line 2890 of file LoopVectorize.cpp.

References assert(), B, Builder, Cost, llvm::ConstantInt::get(), llvm::VectorType::get(), llvm::UndefValue::get(), getBroadcastInstrs(), llvm::VectorizerValueMap::getScalarValue(), llvm::Value::getType(), llvm::VectorizerValueMap::getVectorValue(), llvm::VectorizerValueMap::hasAnyScalarValue(), llvm::VectorizerValueMap::hasVectorValue(), Induction, llvm::Type::isVectorTy(), llvm::Type::isVoidTy(), Legal, packScalarIntoVectorValue(), llvm::IRBuilderBase::restoreIP(), llvm::IRBuilderBase::saveIP(), llvm::IRBuilderBase::SetInsertPoint(), llvm::VectorizerValueMap::setVectorValue(), llvm::RegState::Undef, VectorLoopValueMap, and VF.

Referenced by AddRuntimeUnrollDisableMetaData(), fixFirstOrderRecurrence(), fixReduction(), getOrCreateScalarValue(), truncateToMinimalBitwidths(), vectorizeInterleaveGroup(), vectorizeMemoryInstruction(), and widenInstruction().

◆ getStepVector()

Value * InnerLoopVectorizer::getStepVector ( Value Val,
int  StartIdx,
Value Step,
Instruction::BinaryOps  Opcode = Instruction::BinaryOpsEnd 
)
protectedvirtual

◆ needsScalarInduction()

bool InnerLoopVectorizer::needsScalarInduction ( Instruction IV) const
protected

Returns true if we should generate a scalar version of IV.

Definition at line 2657 of file LoopVectorize.cpp.

References llvm::any_of(), llvm::LoopBase< BlockT, LoopT >::contains(), OrigLoop, shouldScalarizeInstruction(), and llvm::Value::users().

Referenced by widenIntOrFpInduction().

◆ packScalarIntoVectorValue()

void InnerLoopVectorizer::packScalarIntoVectorValue ( Value V,
const VPIteration Instance 
)

◆ recordVectorLoopValueForInductionCast()

void InnerLoopVectorizer::recordVectorLoopValueForInductionCast ( const InductionDescriptor ID,
Value VectorLoopValue,
unsigned  Part,
unsigned  Lane = UINT_MAX 
)
protected

If there is a cast involved in the induction variable ID, which should be ignored in the vectorized loop body, this function records the VectorLoopValue of the respective Phi also as the VectorLoopValue of the cast.

We had already proved that the casted Phi is equal to the uncasted Phi in the vectorized loop (under a runtime guard), and therefore there is no need to vectorize the cast - the same value can be used in the vector loop for both the Phi and the cast. If VectorLoopValue is a scalarized value, Lane is also specified, Otherwise, VectorLoopValue is a widened/vectorized value.

Definition at line 2667 of file LoopVectorize.cpp.

References llvm::SmallVectorTemplateCommon< T >::begin(), llvm::SmallVectorBase::empty(), llvm::InductionDescriptor::getCastInsts(), llvm::VectorizerValueMap::setScalarValue(), llvm::VectorizerValueMap::setVectorValue(), and VectorLoopValueMap.

Referenced by buildScalarSteps(), createVectorIntOrFpInductionPHI(), and widenIntOrFpInduction().

◆ reverseVector()

Value * InnerLoopVectorizer::reverseVector ( Value Vec)
protectedvirtual

◆ scalarizeInstruction()

void InnerLoopVectorizer::scalarizeInstruction ( Instruction Instr,
const VPIteration Instance,
bool  IfPredicateInstr 
)

◆ setDebugLocFromInst()

void InnerLoopVectorizer::setDebugLocFromInst ( IRBuilder<> &  B,
const Value Ptr 
)

◆ shouldScalarizeInstruction()

bool InnerLoopVectorizer::shouldScalarizeInstruction ( Instruction I) const
protected

Returns true if an instruction I should be scalarized instead of vectorized for the chosen vectorization factor.

Definition at line 2652 of file LoopVectorize.cpp.

References Cost, and VF.

Referenced by needsScalarInduction(), and widenIntOrFpInduction().

◆ sinkScalarOperands()

void InnerLoopVectorizer::sinkScalarOperands ( Instruction PredInst)
protected

◆ truncateToMinimalBitwidths()

void InnerLoopVectorizer::truncateToMinimalBitwidths ( )
protected

◆ updateAnalysis()

void InnerLoopVectorizer::updateAnalysis ( )
protected

◆ vectorizeInterleaveGroup()

void InnerLoopVectorizer::vectorizeInterleaveGroup ( Instruction Instr)

◆ vectorizeMemoryInstruction()

void InnerLoopVectorizer::vectorizeMemoryInstruction ( Instruction Instr,
VectorParts BlockInMask = nullptr 
)

◆ widenInstruction()

void InnerLoopVectorizer::widenInstruction ( Instruction I)

Widen a single instruction within the innermost loop.

Vectorize casts.

Definition at line 4689 of file LoopVectorize.cpp.

References llvm::MCID::Add, addMetadata(), Arg, llvm::AMDGPU::HSAMD::Kernel::Key::Args, assert(), B, Builder, C, llvm::MCID::Call, llvm::Instruction::copyFastMathFlags(), llvm::Function::Create(), llvm::IRBuilder< T, Inserter >::CreateBinOp(), llvm::IRBuilder< T, Inserter >::CreateCall(), llvm::IRBuilder< T, Inserter >::CreateCast(), llvm::IRBuilder< T, Inserter >::CreateFCmp(), llvm::IRBuilder< T, Inserter >::CreateGEP(), llvm::IRBuilder< T, Inserter >::CreateICmp(), llvm::IRBuilder< T, Inserter >::CreateInBoundsGEP(), llvm::IRBuilder< T, Inserter >::CreateSelect(), llvm::IRBuilder< T, Inserter >::CreateVectorSplat(), llvm::dbgs(), DEBUG, llvm::dyn_cast(), llvm::StringRef::empty(), llvm::GlobalValue::ExternalLinkage, GEP, llvm::FunctionType::get(), llvm::VectorType::get(), llvm::Intrinsic::getDeclaration(), llvm::Module::getFunction(), llvm::Instruction::getOpcode(), llvm::User::getOperand(), getOrCreateScalarValue(), getOrCreateVectorValue(), llvm::Instruction::getParent(), llvm::BasicBlock::getParent(), llvm::GlobalValue::getParent(), llvm::PredicatedScalarEvolution::getSCEV(), llvm::PredicatedScalarEvolution::getSE(), getVectorCallCost(), getVectorIntrinsicCost(), llvm::getVectorIntrinsicIDForCall(), llvm::TargetLibraryInfo::getVectorizedFunction(), llvm::Loop::hasLoopInvariantOperands(), llvm::hasVectorInstrinsicScalarOpd(), I, llvm::IRBuilder< T, Inserter >::Insert(), llvm::Loop::isLoopInvariant(), llvm::ScalarEvolution::isLoopInvariant(), llvm_unreachable, llvm::make_range(), OrigLoop, PSE, llvm::SmallVectorTemplateBase< T, isPodLike< T >::value >::push_back(), llvm::SmallVectorTemplateBase< T, isPodLike >::push_back(), llvm::MCID::Select, setDebugLocFromInst(), llvm::IRBuilderBase::setFastMathFlags(), llvm::VectorizerValueMap::setVectorValue(), TLI, ToVectorTy(), TTI, UF, VectorLoopValueMap, and VF.

Referenced by AddRuntimeUnrollDisableMetaData(), and InnerLoopVectorizer().

◆ widenIntOrFpInduction()

void InnerLoopVectorizer::widenIntOrFpInduction ( PHINode IV,
TruncInst Trunc = nullptr 
)

◆ widenPHIInstruction()

void InnerLoopVectorizer::widenPHIInstruction ( Instruction PN,
unsigned  UF,
unsigned  VF 
)

Friends And Related Function Documentation

◆ LoopVectorizationPlanner

friend class LoopVectorizationPlanner
friend

Definition at line 516 of file LoopVectorize.cpp.

Member Data Documentation

◆ AA

AliasAnalysis* llvm::InnerLoopVectorizer::AA
protected

◆ AC

AssumptionCache* llvm::InnerLoopVectorizer::AC
protected

◆ AddedSafetyChecks

bool llvm::InnerLoopVectorizer::AddedSafetyChecks = false
protected

Definition at line 757 of file LoopVectorize.cpp.

Referenced by areSafetyChecksAdded(), emitMemRuntimeChecks(), and emitSCEVChecks().

◆ Builder

IRBuilder llvm::InnerLoopVectorizer::Builder
protected

◆ Cost

LoopVectorizationCostModel* llvm::InnerLoopVectorizer::Cost
protected

◆ DT

DominatorTree* llvm::InnerLoopVectorizer::DT
protected

◆ Induction

PHINode* llvm::InnerLoopVectorizer::Induction = nullptr
protected

◆ IVEndValues

DenseMap<PHINode *, Value *> llvm::InnerLoopVectorizer::IVEndValues
protected

Definition at line 761 of file LoopVectorize.cpp.

Referenced by createVectorizedLoopSkeleton(), and fixVectorizedLoop().

◆ Legal

LoopVectorizationLegality* llvm::InnerLoopVectorizer::Legal
protected

◆ LI

LoopInfo* llvm::InnerLoopVectorizer::LI
protected

◆ LoopBypassBlocks

SmallVector<BasicBlock *, 4> llvm::InnerLoopVectorizer::LoopBypassBlocks
protected

A list of all bypass blocks. The first block is the entry of the loop.

Definition at line 727 of file LoopVectorize.cpp.

Referenced by createVectorizedLoopSkeleton(), emitMemRuntimeChecks(), emitMinimumIterationCountCheck(), emitSCEVChecks(), fixReduction(), and updateAnalysis().

◆ LoopExitBlock

BasicBlock* llvm::InnerLoopVectorizer::LoopExitBlock
protected

The ExitBlock of the scalar loop.

Definition at line 718 of file LoopVectorize.cpp.

Referenced by createVectorizedLoopSkeleton(), fixFirstOrderRecurrence(), fixLCSSAPHIs(), fixReduction(), and updateAnalysis().

◆ LoopMiddleBlock

BasicBlock* llvm::InnerLoopVectorizer::LoopMiddleBlock
protected

Middle Block between the vector and the scalar.

Definition at line 715 of file LoopVectorize.cpp.

Referenced by createVectorizedLoopSkeleton(), fixFirstOrderRecurrence(), fixLCSSAPHIs(), fixReduction(), fixVectorizedLoop(), and updateAnalysis().

◆ LoopScalarBody

BasicBlock* llvm::InnerLoopVectorizer::LoopScalarBody
protected

The scalar loop body.

Definition at line 724 of file LoopVectorize.cpp.

Referenced by createVectorizedLoopSkeleton(), and updateAnalysis().

◆ LoopScalarPreHeader

BasicBlock* llvm::InnerLoopVectorizer::LoopScalarPreHeader
protected

The scalar-loop preheader.

Definition at line 712 of file LoopVectorize.cpp.

Referenced by createVectorizedLoopSkeleton(), fixFirstOrderRecurrence(), fixReduction(), and updateAnalysis().

◆ LoopVectorBody

BasicBlock* llvm::InnerLoopVectorizer::LoopVectorBody
protected

◆ LoopVectorPreHeader

BasicBlock* llvm::InnerLoopVectorizer::LoopVectorPreHeader
protected

◆ LVer

std::unique_ptr<LoopVersioning> llvm::InnerLoopVectorizer::LVer
protected

LoopVersioning.

It's only set up (non-null) if memchecks were used.

This is currently only used to add no-alias metadata based on the memchecks. The actually versioning is performed manually.

Definition at line 693 of file LoopVectorize.cpp.

Referenced by addNewMetadata(), and emitMemRuntimeChecks().

◆ OldInduction

PHINode* llvm::InnerLoopVectorizer::OldInduction = nullptr
protected

The induction variable of the old basic block.

Definition at line 733 of file LoopVectorize.cpp.

Referenced by createInductionVariable(), createVectorizedLoopSkeleton(), and widenIntOrFpInduction().

◆ ORE

OptimizationRemarkEmitter* llvm::InnerLoopVectorizer::ORE
protected

◆ OrigLoop

Loop* llvm::InnerLoopVectorizer::OrigLoop
protected

◆ PredicatedInstructions

SmallVector<Instruction *, 4> llvm::InnerLoopVectorizer::PredicatedInstructions
protected

Store instructions that were predicated.

Definition at line 742 of file LoopVectorize.cpp.

Referenced by fixVectorizedLoop(), and scalarizeInstruction().

◆ PSE

PredicatedScalarEvolution& llvm::InnerLoopVectorizer::PSE
protected

◆ TLI

const TargetLibraryInfo* llvm::InnerLoopVectorizer::TLI
protected

◆ TripCount

Value* llvm::InnerLoopVectorizer::TripCount = nullptr
protected

Trip count of the original loop.

Definition at line 745 of file LoopVectorize.cpp.

Referenced by getOrCreateTripCount().

◆ TTI

const TargetTransformInfo* llvm::InnerLoopVectorizer::TTI
protected

◆ UF

unsigned llvm::InnerLoopVectorizer::UF
protected

◆ VectorLoopValueMap

VectorizerValueMap llvm::InnerLoopVectorizer::VectorLoopValueMap
protected

Maps values from the original loop to their corresponding values in the vectorized loop.

A key value can map to either vector values, scalar values or both kinds of values, depending on whether the key was vectorized and scalarized.

Definition at line 739 of file LoopVectorize.cpp.

Referenced by buildScalarSteps(), createVectorIntOrFpInductionPHI(), llvm::LoopVectorizationPlanner::executePlan(), fixFirstOrderRecurrence(), fixReduction(), getOrCreateScalarValue(), getOrCreateVectorValue(), packScalarIntoVectorValue(), recordVectorLoopValueForInductionCast(), scalarizeInstruction(), truncateToMinimalBitwidths(), vectorizeInterleaveGroup(), vectorizeMemoryInstruction(), widenInstruction(), widenIntOrFpInduction(), and widenPHIInstruction().

◆ VectorTripCount

Value* llvm::InnerLoopVectorizer::VectorTripCount = nullptr
protected

Trip count of the widened loop (TripCount - TripCount % (VF*UF))

Definition at line 748 of file LoopVectorize.cpp.

Referenced by getOrCreateVectorTripCount().

◆ VF

unsigned llvm::InnerLoopVectorizer::VF
protected

The documentation for this class was generated from the following file: