LLVM 20.0.0git
|
VPTransformState holds information passed down when "executing" a VPlan, needed for generating the output IR. More...
#include "Transforms/Vectorize/VPlan.h"
Classes | |
struct | CFGState |
Hold state information used when constructing the CFG of the output IR, traversing the VPBasicBlocks and generating corresponding IR BasicBlocks. More... | |
struct | DataState |
Public Member Functions | |
VPTransformState (const TargetTransformInfo *TTI, ElementCount VF, unsigned UF, LoopInfo *LI, DominatorTree *DT, IRBuilderBase &Builder, InnerLoopVectorizer *ILV, VPlan *Plan, Type *CanonicalIVTy) | |
Value * | get (VPValue *Def, bool IsScalar=false) |
Get the generated vector Value for a given VPValue Def if IsScalar is false, otherwise return the generated scalar. | |
Value * | get (VPValue *Def, const VPLane &Lane) |
Get the generated Value for a given VPValue and given Part and Lane. | |
bool | hasVectorValue (VPValue *Def) |
bool | hasScalarValue (VPValue *Def, VPLane Lane) |
void | set (VPValue *Def, Value *V, bool IsScalar=false) |
Set the generated vector Value for a given VPValue, if IsScalar is false. | |
void | reset (VPValue *Def, Value *V) |
Reset an existing vector value for Def and a given Part . | |
void | set (VPValue *Def, Value *V, const VPLane &Lane) |
Set the generated scalar V for Def and the given Lane . | |
void | reset (VPValue *Def, Value *V, const VPLane &Lane) |
Reset an existing scalar value for Def and a given Lane . | |
void | addNewMetadata (Instruction *To, const Instruction *Orig) |
Add additional metadata to To that was not present on Orig . | |
void | addMetadata (Value *To, Instruction *From) |
Add metadata from one instruction to another. | |
void | setDebugLocFrom (DebugLoc DL) |
Set the debug location in the builder using the debug location DL . | |
void | packScalarIntoVectorValue (VPValue *Def, const VPLane &Lane) |
Construct the vector value of a scalarized value V one lane at a time. | |
Public Attributes | |
const TargetTransformInfo * | TTI |
Target Transform Info. | |
ElementCount | VF |
The chosen Vectorization Factor of the loop being vectorized. | |
std::optional< VPLane > | Lane |
Hold the index to generate specific scalar instructions. | |
struct llvm::VPTransformState::DataState | Data |
struct llvm::VPTransformState::CFGState | CFG |
LoopInfo * | LI |
Hold a pointer to LoopInfo to register new basic blocks in the loop. | |
IRBuilderBase & | Builder |
Hold a reference to the IRBuilder used to generate output IR code. | |
InnerLoopVectorizer * | ILV |
Hold a pointer to InnerLoopVectorizer to reuse its IR generation methods. | |
VPlan * | Plan |
Pointer to the VPlan code is generated for. | |
Loop * | CurrentVectorLoop = nullptr |
The loop object for the current parent region, or nullptr. | |
LoopVersioning * | LVer = nullptr |
LoopVersioning. | |
DenseMap< const SCEV *, Value * > | ExpandedSCEVs |
Map SCEVs to their expanded values. | |
VPTypeAnalysis | TypeAnalysis |
VPlan-based type analysis. | |
VPTransformState holds information passed down when "executing" a VPlan, needed for generating the output IR.
VPTransformState::VPTransformState | ( | const TargetTransformInfo * | TTI, |
ElementCount | VF, | ||
unsigned | UF, | ||
LoopInfo * | LI, | ||
DominatorTree * | DT, | ||
IRBuilderBase & | Builder, | ||
InnerLoopVectorizer * | ILV, | ||
VPlan * | Plan, | ||
Type * | CanonicalIVTy | ||
) |
void VPTransformState::addMetadata | ( | Value * | To, |
Instruction * | From | ||
) |
Add metadata from one instruction to another.
This includes both the original MDs from From
and additional ones (
Definition at line 365 of file VPlan.cpp.
References addNewMetadata(), From, and llvm::propagateMetadata().
Referenced by llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPWidenIntrinsicRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenLoadEVLRecipe::execute(), llvm::VPWidenStoreRecipe::execute(), llvm::VPWidenStoreEVLRecipe::execute(), and llvm::VPWidenEVLRecipe::execute().
void VPTransformState::addNewMetadata | ( | Instruction * | To, |
const Instruction * | Orig | ||
) |
Add additional metadata to To
that was not present on Orig
.
Currently this is used to add the noalias annotations based on the inserted memchecks. Use this for instructions that are cloned into the vector loop.
Definition at line 357 of file VPlan.cpp.
References llvm::LoopVersioning::annotateInstWithNoAlias(), and LVer.
Referenced by addMetadata(), and llvm::InnerLoopVectorizer::scalarizeInstruction().
Get the generated vector Value for a given VPValue Def
if IsScalar
is false, otherwise return the generated scalar.
\See set.
Definition at line 253 of file VPlan.cpp.
References assert(), B, Builder, CFG, llvm::IRBuilderBase::CreateVectorSplat(), Data, llvm::VectorType::get(), llvm::PoisonValue::get(), get(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::BasicBlock::getTerminator(), llvm::VPlan::getVectorPreheader(), hasScalarValue(), hasVectorValue(), llvm::isa(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::vputils::isUniformAfterVectorization(), Lane, llvm::vputils::onlyFirstLaneUsed(), packScalarIntoVectorValue(), Plan, llvm::IRBuilderBase::restoreIP(), llvm::IRBuilderBase::saveIP(), set(), llvm::IRBuilderBase::SetInsertPoint(), VF, llvm::VPTransformState::CFGState::VPBB2IRBB, llvm::VPTransformState::DataState::VPV2Scalars, and llvm::VPTransformState::DataState::VPV2Vector.
Referenced by llvm::VPIRInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPWidenIntrinsicRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPHistogramRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPReverseVectorPointerRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPScalarPHIRecipe::execute(), llvm::VPWidenPHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReductionEVLRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenLoadEVLRecipe::execute(), llvm::VPWidenStoreRecipe::execute(), llvm::VPWidenStoreEVLRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPWidenEVLRecipe::execute(), llvm::VPlan::execute(), llvm::LoopVectorizationPlanner::executePlan(), llvm::InnerLoopVectorizer::fixNonInductionPHIs(), fixReductionScalarResumeWhenVectorizingEpilog(), llvm::InnerLoopVectorizer::fixupIVUsers(), get(), packScalarIntoVectorValue(), and llvm::InnerLoopVectorizer::scalarizeInstruction().
Get the generated Value for a given VPValue and given Part and Lane.
Definition at line 228 of file VPlan.cpp.
References assert(), Builder, llvm::IRBuilderBase::CreateExtractElement(), Data, llvm::VPLane::getFirstLane(), hasScalarValue(), hasVectorValue(), llvm::vputils::isUniformAfterVectorization(), Lane, VF, llvm::VPTransformState::DataState::VPV2Scalars, and llvm::VPTransformState::DataState::VPV2Vector.
Definition at line 268 of file VPlan.h.
References Data, I, Lane, VF, and llvm::VPTransformState::DataState::VPV2Scalars.
Referenced by llvm::VPPredInstPHIRecipe::execute(), and get().
Definition at line 266 of file VPlan.h.
References Data, and llvm::VPTransformState::DataState::VPV2Vector.
Referenced by llvm::VPPredInstPHIRecipe::execute(), and get().
Construct the vector value of a scalarized value V
one lane at a time.
Definition at line 398 of file VPlan.cpp.
References Builder, llvm::IRBuilderBase::CreateInsertElement(), get(), Lane, set(), and VF.
Referenced by llvm::VPReplicateRecipe::execute(), and get().
Reset an existing vector value for Def
and a given Part
.
Definition at line 289 of file VPlan.h.
References assert(), Data, and llvm::VPTransformState::DataState::VPV2Vector.
Referenced by llvm::VPPredInstPHIRecipe::execute().
Set the generated vector Value for a given VPValue, if IsScalar
is false.
If IsScalar
is true, set the scalar in lane 0.
Definition at line 278 of file VPlan.h.
References assert(), Data, llvm::ElementCount::isScalar(), set(), VF, and llvm::VPTransformState::DataState::VPV2Vector.
Referenced by llvm::VPInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPScalarCastRecipe::execute(), llvm::VPWidenIntrinsicRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPReverseVectorPointerRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPScalarPHIRecipe::execute(), llvm::VPWidenPHIRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReductionEVLRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenLoadEVLRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPWidenEVLRecipe::execute(), get(), packScalarIntoVectorValue(), llvm::InnerLoopVectorizer::scalarizeInstruction(), and set().
void VPTransformState::setDebugLocFrom | ( | DebugLoc | DL | ) |
Set the debug location in the builder using the debug location DL
.
Definition at line 376 of file VPlan.cpp.
References Builder, llvm::DILocation::cloneByMultiplyingDuplicationFactor(), llvm::dbgs(), DL, llvm::EnableFSDiscriminator, llvm::IRBuilderBase::GetInsertBlock(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::BasicBlock::getParent(), llvm::VPlan::getUF(), LLVM_DEBUG, Plan, llvm::IRBuilderBase::SetCurrentDebugLocation(), llvm::Function::shouldEmitDebugInfoForProfiling(), and VF.
Referenced by llvm::VPInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPWidenIntrinsicRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPHistogramRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPReverseVectorPointerRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenLoadEVLRecipe::execute(), llvm::VPWidenStoreRecipe::execute(), llvm::VPWidenStoreEVLRecipe::execute(), llvm::VPWidenEVLRecipe::execute(), and llvm::InnerLoopVectorizer::scalarizeInstruction().
IRBuilderBase& llvm::VPTransformState::Builder |
Hold a reference to the IRBuilder used to generate output IR code.
Definition at line 368 of file VPlan.h.
Referenced by llvm::VPlan::createInitialVPlan(), llvm::VPInstruction::execute(), llvm::VPIRInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPWidenIntrinsicRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPHistogramRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPReverseVectorPointerRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPScalarPHIRecipe::execute(), llvm::VPWidenPHIRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReductionEVLRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenLoadEVLRecipe::execute(), llvm::VPWidenStoreRecipe::execute(), llvm::VPWidenStoreEVLRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPWidenEVLRecipe::execute(), llvm::VPlan::execute(), llvm::VPBasicBlock::execute(), llvm::VPIRBasicBlock::execute(), llvm::LoopVectorizationPlanner::executePlan(), get(), packScalarIntoVectorValue(), llvm::VPlan::prepareToExecute(), llvm::InnerLoopVectorizer::scalarizeInstruction(), and setDebugLocFrom().
struct llvm::VPTransformState::CFGState llvm::VPTransformState::CFG |
Referenced by llvm::VPBasicBlock::connectToPredecessors(), llvm::VPIRInstruction::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPScalarPHIRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPlan::execute(), llvm::VPBasicBlock::execute(), llvm::VPIRBasicBlock::execute(), llvm::VPRegionBlock::execute(), llvm::LoopVectorizationPlanner::executePlan(), llvm::VPBasicBlock::executeRecipes(), llvm::InnerLoopVectorizer::fixNonInductionPHIs(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), get(), and llvm::VPlan::prepareToExecute().
Loop* llvm::VPTransformState::CurrentVectorLoop = nullptr |
The loop object for the current parent region, or nullptr.
Definition at line 377 of file VPlan.h.
Referenced by llvm::VPReductionPHIRecipe::execute(), llvm::VPBasicBlock::execute(), and llvm::VPRegionBlock::execute().
struct llvm::VPTransformState::DataState llvm::VPTransformState::Data |
Referenced by get(), hasScalarValue(), hasVectorValue(), reset(), and set().
Map SCEVs to their expanded values.
Populated when executing VPExpandSCEVRecipes.
Definition at line 388 of file VPlan.h.
Referenced by llvm::VPExpandSCEVRecipe::execute(), and llvm::LoopVectorizationPlanner::executePlan().
InnerLoopVectorizer* llvm::VPTransformState::ILV |
Hold a pointer to InnerLoopVectorizer to reuse its IR generation methods.
Definition at line 371 of file VPlan.h.
Referenced by llvm::VPReplicateRecipe::execute().
std::optional<VPLane> llvm::VPTransformState::Lane |
Hold the index to generate specific scalar instructions.
Null indicates that all instances are to be generated, using either scalar or vector instructions.
Definition at line 249 of file VPlan.h.
Referenced by llvm::VPInstruction::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReductionEVLRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPBasicBlock::execute(), llvm::VPRegionBlock::execute(), get(), hasScalarValue(), packScalarIntoVectorValue(), reset(), and set().
LoopInfo* llvm::VPTransformState::LI |
Hold a pointer to LoopInfo to register new basic blocks in the loop.
Definition at line 365 of file VPlan.h.
Referenced by llvm::VPBasicBlock::execute(), and llvm::VPRegionBlock::execute().
LoopVersioning* llvm::VPTransformState::LVer = nullptr |
It's only set up (non-null) if memchecks were used.
This is currently only used to add no-alias metadata based on the memchecks. The actually versioning is performed manually.
Definition at line 384 of file VPlan.h.
Referenced by addNewMetadata(), and llvm::LoopVectorizationPlanner::executePlan().
VPlan* llvm::VPTransformState::Plan |
Pointer to the VPlan code is generated for.
Definition at line 374 of file VPlan.h.
Referenced by llvm::LoopVectorizationPlanner::buildVPlans(), llvm::VPlan::createInitialVPlan(), llvm::VPlanPrinter::dump(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), get(), llvm::LoopVectorizationPlanner::getPlanFor(), llvm::LoopVectorizationPlanner::printPlans(), setDebugLocFrom(), and llvm::VPInterleavedAccessInfo::VPInterleavedAccessInfo().
const TargetTransformInfo* llvm::VPTransformState::TTI |
Target Transform Info.
Definition at line 241 of file VPlan.h.
Referenced by llvm::VPWidenIntrinsicRecipe::execute().
VPTypeAnalysis llvm::VPTransformState::TypeAnalysis |
VPlan-based type analysis.
Definition at line 391 of file VPlan.h.
Referenced by llvm::VPWidenRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), and llvm::InnerLoopVectorizer::scalarizeInstruction().
ElementCount llvm::VPTransformState::VF |
The chosen Vectorization Factor of the loop being vectorized.
Definition at line 244 of file VPlan.h.
Referenced by llvm::LoopVectorizationPlanner::buildVPlans(), llvm::VPlan::cost(), llvm::VPBasicBlock::cost(), llvm::VPRegionBlock::cost(), llvm::VPlan::duplicate(), llvm::VPInstruction::execute(), llvm::VPIRInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPWidenIntrinsicRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPReverseVectorPointerRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReductionEVLRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenLoadEVLRecipe::execute(), llvm::VPWidenStoreEVLRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPWidenEVLRecipe::execute(), llvm::VPlan::execute(), llvm::VPRegionBlock::execute(), llvm::LoopVectorizationPlanner::executePlan(), get(), llvm::VPlan::getName(), llvm::LoopVectorizationPlanner::getPlanFor(), hasScalarValue(), packScalarIntoVectorValue(), llvm::VPlan::prepareToExecute(), llvm::VPlan::printLiveIns(), reset(), set(), and setDebugLocFrom().