LLVM 17.0.0git
|
VPTransformState holds information passed down when "executing" a VPlan, needed for generating the output IR. More...
#include "Transforms/Vectorize/VPlan.h"
Classes | |
struct | CFGState |
Hold state information used when constructing the CFG of the output IR, traversing the VPBasicBlocks and generating corresponding IR BasicBlocks. More... | |
struct | DataState |
Public Member Functions | |
VPTransformState (ElementCount VF, unsigned UF, LoopInfo *LI, DominatorTree *DT, IRBuilderBase &Builder, InnerLoopVectorizer *ILV, VPlan *Plan) | |
Value * | get (VPValue *Def, unsigned Part) |
Get the generated Value for a given VPValue and a given Part. | |
Value * | get (VPValue *Def, const VPIteration &Instance) |
Get the generated Value for a given VPValue and given Part and Lane. | |
bool | hasVectorValue (VPValue *Def, unsigned Part) |
bool | hasAnyVectorValue (VPValue *Def) const |
bool | hasScalarValue (VPValue *Def, VPIteration Instance) |
void | set (VPValue *Def, Value *V, unsigned Part) |
Set the generated Value for a given VPValue and a given Part. | |
void | reset (VPValue *Def, Value *V, unsigned Part) |
Reset an existing vector value for Def and a given Part . | |
void | set (VPValue *Def, Value *V, const VPIteration &Instance) |
Set the generated scalar V for Def and the given Instance . | |
void | reset (VPValue *Def, Value *V, const VPIteration &Instance) |
Reset an existing scalar value for Def and a given Instance . | |
void | addNewMetadata (Instruction *To, const Instruction *Orig) |
Add additional metadata to To that was not present on Orig . | |
void | addMetadata (Instruction *To, Instruction *From) |
Add metadata from one instruction to another. | |
void | addMetadata (ArrayRef< Value * > To, Instruction *From) |
Similar to the previous function but it adds the metadata to a vector of instructions. | |
void | setDebugLocFromInst (const Value *V) |
Set the debug location in the builder using the debug location in V . | |
Public Attributes | |
ElementCount | VF |
The chosen Vectorization and Unroll Factors of the loop being vectorized. | |
unsigned | UF |
std::optional< VPIteration > | Instance |
Hold the indices to generate specific scalar instructions. | |
struct llvm::VPTransformState::DataState | Data |
struct llvm::VPTransformState::CFGState | CFG |
LoopInfo * | LI |
Hold a pointer to LoopInfo to register new basic blocks in the loop. | |
DominatorTree * | DT |
Hold a pointer to Dominator Tree to register new basic blocks in the loop. | |
IRBuilderBase & | Builder |
Hold a reference to the IRBuilder used to generate output IR code. | |
VPValue2ValueTy | VPValue2Value |
Value * | CanonicalIV = nullptr |
Hold the canonical scalar IV of the vector loop (start=0, step=VF*UF). | |
InnerLoopVectorizer * | ILV |
Hold a pointer to InnerLoopVectorizer to reuse its IR generation methods. | |
VPlan * | Plan |
Pointer to the VPlan code is generated for. | |
SmallPtrSet< VPRecipeBase *, 16 > | MayGeneratePoisonRecipes |
Holds recipes that may generate a poison value that is used after vectorization, even when their operands are not poison. | |
Loop * | CurrentVectorLoop = nullptr |
The loop object for the current parent region, or nullptr. | |
std::unique_ptr< LoopVersioning > | LVer |
LoopVersioning. | |
VPTransformState holds information passed down when "executing" a VPlan, needed for generating the output IR.
|
inline |
void VPTransformState::addMetadata | ( | ArrayRef< Value * > | To, |
Instruction * | From | ||
) |
Similar to the previous function but it adds the metadata to a vector of instructions.
Definition at line 250 of file VPlan.cpp.
References addMetadata(), From, and I.
void VPTransformState::addMetadata | ( | Instruction * | To, |
Instruction * | From | ||
) |
Add metadata from one instruction to another.
This includes both the original MDs from From
and additional ones (
Definition at line 245 of file VPlan.cpp.
References addNewMetadata(), From, and llvm::propagateMetadata().
Referenced by addMetadata(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), and llvm::VPWidenMemoryInstructionRecipe::execute().
void VPTransformState::addNewMetadata | ( | Instruction * | To, |
const Instruction * | Orig | ||
) |
Add additional metadata to To
that was not present on Orig
.
Currently this is used to add the noalias annotations based on the inserted memchecks. Use this for instructions that are cloned into the vector loop.
Definition at line 237 of file VPlan.cpp.
References LVer.
Referenced by addMetadata(), and llvm::InnerLoopVectorizer::scalarizeInstruction().
Value * VPTransformState::get | ( | VPValue * | Def, |
const VPIteration & | Instance | ||
) |
Get the generated Value for a given VPValue and given Part and Lane.
Definition at line 211 of file VPlan.cpp.
References assert(), Builder, llvm::IRBuilderBase::CreateExtractElement(), Data, hasScalarValue(), hasVectorValue(), Instance, llvm::VPTransformState::DataState::PerPartOutput, llvm::VPTransformState::DataState::PerPartScalars, and VF.
Get the generated Value for a given VPValue and a given Part.
Note that as some Defs are still created by ILV and managed in its ValueMap, this method will delegate the call to ILV in such cases in order to provide callers a consistent API.
Definition at line 9887 of file LoopVectorize.cpp.
References assert(), B, llvm::InnerLoopVectorizer::Builder, llvm::Data, llvm::get(), llvm::VectorType::get(), llvm::PoisonValue::get(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::vputils::isUniformAfterVectorization(), llvm::IRBuilderBase::restoreIP(), llvm::IRBuilderBase::saveIP(), llvm::IRBuilderBase::SetInsertPoint(), and llvm::InnerLoopVectorizer::VF.
Referenced by llvm::InnerLoopVectorizer::clearReductionWrapFlags(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPWidenPHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPWidenMemoryInstructionRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPlan::execute(), llvm::InnerLoopVectorizer::fixFixedOrderRecurrence(), llvm::InnerLoopVectorizer::fixNonInductionPHIs(), llvm::VPLiveOut::fixPhi(), llvm::InnerLoopVectorizer::fixReduction(), llvm::InnerLoopVectorizer::packScalarIntoVectorValue(), llvm::InnerLoopVectorizer::scalarizeInstruction(), llvm::InnerLoopVectorizer::truncateToMinimalBitwidths(), and llvm::InnerLoopVectorizer::vectorizeInterleaveGroup().
Definition at line 250 of file VPlan.h.
References Data, and llvm::VPTransformState::DataState::PerPartOutput.
Referenced by llvm::InnerLoopVectorizer::truncateToMinimalBitwidths().
|
inline |
Definition at line 254 of file VPlan.h.
References Data, I, Instance, llvm::VPTransformState::DataState::PerPartScalars, and VF.
Referenced by llvm::VPPredInstPHIRecipe::execute(), and get().
Definition at line 244 of file VPlan.h.
References Data, I, and llvm::VPTransformState::DataState::PerPartOutput.
Referenced by llvm::VPPredInstPHIRecipe::execute(), and get().
|
inline |
Reset an existing vector value for Def
and a given Part
.
Definition at line 273 of file VPlan.h.
References assert(), Data, and llvm::VPTransformState::DataState::PerPartOutput.
Referenced by llvm::VPPredInstPHIRecipe::execute(), llvm::InnerLoopVectorizer::fixReduction(), and llvm::InnerLoopVectorizer::truncateToMinimalBitwidths().
|
inline |
Set the generated Value for a given VPValue and a given Part.
Definition at line 265 of file VPlan.h.
References Data, llvm::VPTransformState::DataState::PerPartOutput, and UF.
Referenced by buildScalarSteps(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPWidenPHIRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPWidenMemoryInstructionRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPCanonicalIVPHIRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::InnerLoopVectorizer::packScalarIntoVectorValue(), llvm::VPlan::prepareToExecute(), llvm::InnerLoopVectorizer::scalarizeInstruction(), and llvm::InnerLoopVectorizer::vectorizeInterleaveGroup().
Set the debug location in the builder using the debug location in V
.
Definition at line 257 of file VPlan.cpp.
References Builder, llvm::DILocation::cloneByMultiplyingDuplicationFactor(), llvm::dbgs(), llvm::EnableFSDiscriminator, llvm::Instruction::getDebugLoc(), llvm::Instruction::getFunction(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), LLVM_DEBUG, llvm::IRBuilderBase::SetCurrentDebugLocation(), llvm::Function::shouldEmitDebugInfoForProfiling(), UF, and VF.
Referenced by llvm::VPWidenRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPWidenMemoryInstructionRecipe::execute(), llvm::InnerLoopVectorizer::fixReduction(), llvm::InnerLoopVectorizer::scalarizeInstruction(), and llvm::InnerLoopVectorizer::vectorizeInterleaveGroup().
IRBuilderBase& llvm::VPTransformState::Builder |
Hold a reference to the IRBuilder used to generate output IR code.
Definition at line 360 of file VPlan.h.
Referenced by buildScalarSteps(), llvm::VPInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPWidenPHIRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPWidenMemoryInstructionRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPlan::execute(), llvm::VPBasicBlock::execute(), llvm::VPLiveOut::fixPhi(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), get(), llvm::InnerLoopVectorizer::packScalarIntoVectorValue(), llvm::VPlan::prepareToExecute(), llvm::InnerLoopVectorizer::scalarizeInstruction(), and setDebugLocFromInst().
Value* llvm::VPTransformState::CanonicalIV = nullptr |
struct llvm::VPTransformState::CFGState llvm::VPTransformState::CFG |
Referenced by llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPCanonicalIVPHIRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPlan::execute(), llvm::VPBasicBlock::execute(), llvm::VPRegionBlock::execute(), llvm::InnerLoopVectorizer::fixNonInductionPHIs(), llvm::InnerLoopVectorizer::fixReduction(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), and llvm::VPlan::prepareToExecute().
Loop* llvm::VPTransformState::CurrentVectorLoop = nullptr |
The loop object for the current parent region, or nullptr.
Definition at line 378 of file VPlan.h.
Referenced by llvm::VPReductionPHIRecipe::execute(), llvm::VPBasicBlock::execute(), and llvm::VPRegionBlock::execute().
struct llvm::VPTransformState::DataState llvm::VPTransformState::Data |
Referenced by get(), hasAnyVectorValue(), hasScalarValue(), hasVectorValue(), reset(), and set().
DominatorTree* llvm::VPTransformState::DT |
Hold a pointer to Dominator Tree to register new basic blocks in the loop.
Definition at line 357 of file VPlan.h.
Referenced by llvm::VPlan::execute().
InnerLoopVectorizer* llvm::VPTransformState::ILV |
Hold a pointer to InnerLoopVectorizer to reuse its IR generation methods.
Definition at line 368 of file VPlan.h.
Referenced by llvm::VPInterleaveRecipe::execute(), llvm::VPReductionRecipe::execute(), and llvm::VPReplicateRecipe::execute().
std::optional<VPIteration> llvm::VPTransformState::Instance |
Hold the indices to generate specific scalar instructions.
Null indicates that all instances are to be generated, using either scalar or vector instructions.
Definition at line 220 of file VPlan.h.
Referenced by buildScalarSteps(), llvm::VPInstruction::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::VPBasicBlock::execute(), llvm::VPRegionBlock::execute(), get(), hasScalarValue(), reset(), and set().
LoopInfo* llvm::VPTransformState::LI |
Hold a pointer to LoopInfo to register new basic blocks in the loop.
Definition at line 354 of file VPlan.h.
Referenced by llvm::VPBasicBlock::execute(), and llvm::VPRegionBlock::execute().
std::unique_ptr<LoopVersioning> llvm::VPTransformState::LVer |
It's only set up (non-null) if memchecks were used.
This is currently only used to add no-alias metadata based on the memchecks. The actually versioning is performed manually.
Definition at line 385 of file VPlan.h.
Referenced by addNewMetadata().
SmallPtrSet<VPRecipeBase *, 16> llvm::VPTransformState::MayGeneratePoisonRecipes |
Holds recipes that may generate a poison value that is used after vectorization, even when their operands are not poison.
Definition at line 375 of file VPlan.h.
Referenced by llvm::VPWidenRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), and llvm::InnerLoopVectorizer::scalarizeInstruction().
VPlan* llvm::VPTransformState::Plan |
Pointer to the VPlan code is generated for.
Definition at line 371 of file VPlan.h.
Referenced by llvm::InnerLoopVectorizer::collectPoisonGeneratingRecipes(), llvm::VPlanPrinter::dump(), llvm::InnerLoopVectorizer::fixCrossIterationPHIs(), llvm::InnerLoopVectorizer::fixFixedOrderRecurrence(), llvm::InnerLoopVectorizer::fixReduction(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::InnerLoopVectorizer::truncateToMinimalBitwidths(), and llvm::VPInterleavedAccessInfo::VPInterleavedAccessInfo().
unsigned llvm::VPTransformState::UF |
Definition at line 215 of file VPlan.h.
Referenced by buildScalarSteps(), llvm::VPInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPWidenMemoryInstructionRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPCanonicalIVPHIRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPlan::execute(), llvm::VPRegionBlock::execute(), llvm::VPLiveOut::fixPhi(), llvm::VPlan::getName(), llvm::VPlan::prepareToExecute(), set(), and setDebugLocFromInst().
ElementCount llvm::VPTransformState::VF |
The chosen Vectorization and Unroll Factors of the loop being vectorized.
Definition at line 214 of file VPlan.h.
Referenced by buildScalarSteps(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPWidenMemoryInstructionRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPlan::execute(), llvm::VPRegionBlock::execute(), llvm::VPLiveOut::fixPhi(), get(), llvm::VPlan::getName(), hasScalarValue(), llvm::VPlan::prepareToExecute(), reset(), set(), and setDebugLocFromInst().
VPValue2ValueTy llvm::VPTransformState::VPValue2Value |
Definition at line 362 of file VPlan.h.
Referenced by llvm::VPlan::execute().