LLVM 19.0.0git
Classes | Public Member Functions | Public Attributes | List of all members
llvm::VPTransformState Struct Reference

VPTransformState holds information passed down when "executing" a VPlan, needed for generating the output IR. More...

#include "Transforms/Vectorize/VPlan.h"

Classes

struct  CFGState
 Hold state information used when constructing the CFG of the output IR, traversing the VPBasicBlocks and generating corresponding IR BasicBlocks. More...
 
struct  DataState
 

Public Member Functions

 VPTransformState (ElementCount VF, unsigned UF, LoopInfo *LI, DominatorTree *DT, IRBuilderBase &Builder, InnerLoopVectorizer *ILV, VPlan *Plan, LLVMContext &Ctx)
 
Value * get (VPValue *Def, unsigned Part, bool IsScalar=false)
 Get the generated vector Value for a given VPValue Def and a given Part if IsScalar is false, otherwise return the generated scalar for Part.
 
Value * get (VPValue *Def, const VPIteration &Instance)
 Get the generated Value for a given VPValue and given Part and Lane.
 
bool hasVectorValue (VPValue *Def, unsigned Part)
 
bool hasScalarValue (VPValue *Def, VPIteration Instance)
 
void set (VPValue *Def, Value *V, unsigned Part, bool IsScalar=false)
 Set the generated vector Value for a given VPValue and a given Part, if IsScalar is false.
 
void reset (VPValue *Def, Value *V, unsigned Part)
 Reset an existing vector value for Def and a given Part.
 
void set (VPValue *Def, Value *V, const VPIteration &Instance)
 Set the generated scalar V for Def and the given Instance.
 
void reset (VPValue *Def, Value *V, const VPIteration &Instance)
 Reset an existing scalar value for Def and a given Instance.
 
void addNewMetadata (Instruction *To, const Instruction *Orig)
 Add additional metadata to To that was not present on Orig.
 
void addMetadata (Value *To, Instruction *From)
 Add metadata from one instruction to another.
 
void setDebugLocFrom (DebugLoc DL)
 Set the debug location in the builder using the debug location DL.
 
void packScalarIntoVectorValue (VPValue *Def, const VPIteration &Instance)
 Construct the vector value of a scalarized value V one lane at a time.
 

Public Attributes

ElementCount VF
 The chosen Vectorization and Unroll Factors of the loop being vectorized.
 
unsigned UF
 
VPValue * EVL = nullptr
 If EVL (Explicit Vector Length) is not nullptr, then EVL must be a valid value set during plan transformation, possibly a default value = whole vector register length.
 
std::optional< VPIteration > Instance
 Hold the indices to generate specific scalar instructions.
 
struct llvm::VPTransformState::DataState Data
 
struct llvm::VPTransformState::CFGState CFG
 
LoopInfo * LI
 Hold a pointer to LoopInfo to register new basic blocks in the loop.
 
DominatorTree * DT
 Hold a pointer to Dominator Tree to register new basic blocks in the loop.
 
IRBuilderBase & Builder
 Hold a reference to the IRBuilder used to generate output IR code.
 
InnerLoopVectorizer * ILV
 Hold a pointer to InnerLoopVectorizer to reuse its IR generation methods.
 
VPlan * Plan
 Pointer to the VPlan code is generated for.
 
Loop * CurrentVectorLoop = nullptr
 The loop object for the current parent region, or nullptr.
 
LoopVersioning * LVer = nullptr
 LoopVersioning.
 
DenseMap< const SCEV *, Value * > ExpandedSCEVs
 Map SCEVs to their expanded values.
 
VPTypeAnalysis TypeAnalysis
 VPlan-based type analysis.
 

Detailed Description

VPTransformState holds information passed down when "executing" a VPlan, needed for generating the output IR.

Definition at line 236 of file VPlan.h.

Constructor & Destructor Documentation

◆ VPTransformState()

VPTransformState::VPTransformState ( ElementCount  VF,
unsigned  UF,
LoopInfo *  LI,
DominatorTree *  DT,
IRBuilderBase &  Builder,
InnerLoopVectorizer *  ILV,
VPlan *  Plan,
LLVMContext &  Ctx 
)

Definition at line 217 of file VPlan.cpp.

Member Function Documentation

◆ addMetadata()

void VPTransformState::addMetadata ( Value *  To,
Instruction *  From 
)

Add metadata from one instruction to another.

This includes both the original MDs from From and additional ones (

See also
addNewMetadata). Use this for newly created instructions in the vector loop.

Definition at line 361 of file VPlan.cpp.

References addNewMetadata(), From, and llvm::propagateMetadata().

Referenced by llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), and llvm::VPWidenStoreRecipe::execute().

◆ addNewMetadata()

void VPTransformState::addNewMetadata ( Instruction *  To,
const Instruction *  Orig 
)

Add additional metadata to To that was not present on Orig.

Currently this is used to add the noalias annotations based on the inserted memchecks. Use this for instructions that are cloned into the vector loop.

Definition at line 353 of file VPlan.cpp.

References llvm::LoopVersioning::annotateInstWithNoAlias(), and LVer.

Referenced by addMetadata(), and llvm::InnerLoopVectorizer::scalarizeInstruction().

◆ get() [1/2]

Value * VPTransformState::get ( VPValue *  Def,
const VPIteration &  Instance 
)

Get the generated Value for a given VPValue and given Part and Lane.

Definition at line 225 of file VPlan.cpp.

References assert(), Builder, llvm::IRBuilderBase::CreateExtractElement(), Data, hasScalarValue(), hasVectorValue(), Instance, llvm::VPTransformState::DataState::PerPartOutput, llvm::VPTransformState::DataState::PerPartScalars, and VF.

◆ get() [2/2]

Value * VPTransformState::get ( VPValue *  Def,
unsigned  Part,
bool  IsScalar = false 
)

Get the generated vector Value for a given VPValue Def and a given Part if IsScalar is false, otherwise return the generated scalar for Part.

\See set.

Definition at line 247 of file VPlan.cpp.

References assert(), B, Builder, CFG, llvm::IRBuilderBase::CreateVectorSplat(), Data, llvm::VectorType::get(), llvm::PoisonValue::get(), get(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::VPBlockBase::getSinglePredecessor(), llvm::BasicBlock::getTerminator(), llvm::VPlan::getVectorLoopRegion(), hasScalarValue(), hasVectorValue(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::vputils::isUniformAfterVectorization(), packScalarIntoVectorValue(), llvm::VPTransformState::DataState::PerPartOutput, llvm::VPTransformState::DataState::PerPartScalars, Plan, llvm::IRBuilderBase::restoreIP(), llvm::IRBuilderBase::saveIP(), set(), llvm::IRBuilderBase::SetInsertPoint(), VF, and llvm::VPTransformState::CFGState::VPBB2IRBB.

Referenced by createAndCollectMergePhiForReduction(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPScalarCastRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPWidenPHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenStoreRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPEVLBasedIVPHIRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPlan::execute(), llvm::LoopVectorizationPlanner::executePlan(), llvm::InnerLoopVectorizer::fixFixedOrderRecurrence(), llvm::InnerLoopVectorizer::fixNonInductionPHIs(), llvm::VPLiveOut::fixPhi(), llvm::InnerLoopVectorizer::fixupIVUsers(), get(), packScalarIntoVectorValue(), llvm::InnerLoopVectorizer::scalarizeInstruction(), and llvm::InnerLoopVectorizer::vectorizeInterleaveGroup().

◆ hasScalarValue()

bool llvm::VPTransformState::hasScalarValue ( VPValue *  Def,
VPIteration  Instance 
)
inline

Definition at line 285 of file VPlan.h.

References Data, I, Instance, llvm::VPTransformState::DataState::PerPartScalars, and VF.

Referenced by llvm::VPPredInstPHIRecipe::execute(), and get().

◆ hasVectorValue()

bool llvm::VPTransformState::hasVectorValue ( VPValue *  Def,
unsigned  Part 
)
inline

Definition at line 279 of file VPlan.h.

References Data, I, and llvm::VPTransformState::DataState::PerPartOutput.

Referenced by llvm::VPPredInstPHIRecipe::execute(), and get().

◆ packScalarIntoVectorValue()

void VPTransformState::packScalarIntoVectorValue ( VPValue *  Def,
const VPIteration &  Instance 
)

Construct the vector value of a scalarized value V one lane at a time.

Definition at line 393 of file VPlan.cpp.

References Builder, llvm::IRBuilderBase::CreateInsertElement(), get(), Instance, set(), and VF.

Referenced by llvm::VPReplicateRecipe::execute(), and get().

◆ reset() [1/2]

void llvm::VPTransformState::reset ( VPValue *  Def,
Value *  V,
const VPIteration &  Instance 
)
inline

Reset an existing scalar value for Def and a given Instance.

Definition at line 334 of file VPlan.h.

References assert(), Data, Instance, llvm::VPTransformState::DataState::PerPartScalars, and VF.

◆ reset() [2/2]

void llvm::VPTransformState::reset ( VPValue *  Def,
Value *  V,
unsigned  Part 
)
inline

Reset an existing vector value for Def and a given Part.

Definition at line 312 of file VPlan.h.

References assert(), Data, and llvm::VPTransformState::DataState::PerPartOutput.

Referenced by llvm::VPPredInstPHIRecipe::execute().

◆ set() [1/2]

void llvm::VPTransformState::set ( VPValue *  Def,
Value *  V,
const VPIteration &  Instance 
)
inline

Set the generated scalar V for Def and the given Instance.

Definition at line 320 of file VPlan.h.

References assert(), Data, Instance, llvm::VPTransformState::DataState::PerPartScalars, and VF.

◆ set() [2/2]

void llvm::VPTransformState::set ( VPValue *  Def,
Value *  V,
unsigned  Part,
bool  IsScalar = false 
)
inline

Set the generated vector Value for a given VPValue and a given Part, if IsScalar is false.

If IsScalar is true, set the scalar in (Part, 0).

Definition at line 297 of file VPlan.h.

References assert(), Data, llvm::ElementCount::isScalar(), llvm::VPTransformState::DataState::PerPartOutput, set(), UF, and VF.

Referenced by llvm::VPInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPScalarCastRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPWidenPHIRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPCanonicalIVPHIRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPEVLBasedIVPHIRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), get(), packScalarIntoVectorValue(), llvm::InnerLoopVectorizer::scalarizeInstruction(), set(), and llvm::InnerLoopVectorizer::vectorizeInterleaveGroup().

◆ setDebugLocFrom()

void VPTransformState::setDebugLocFrom ( DebugLoc  DL)

Set the debug location in the builder using the debug location DL.

Definition at line 372 of file VPlan.cpp.

References Builder, llvm::DILocation::cloneByMultiplyingDuplicationFactor(), llvm::dbgs(), DL, llvm::EnableFSDiscriminator, llvm::IRBuilderBase::GetInsertBlock(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::BasicBlock::getParent(), LLVM_DEBUG, llvm::IRBuilderBase::SetCurrentDebugLocation(), llvm::Function::shouldEmitDebugInfoForProfiling(), UF, and VF.

Referenced by llvm::VPInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenStoreRecipe::execute(), llvm::InnerLoopVectorizer::scalarizeInstruction(), and llvm::InnerLoopVectorizer::vectorizeInterleaveGroup().

Member Data Documentation

◆ Builder

IRBuilderBase& llvm::VPTransformState::Builder

Hold a reference to the IRBuilder used to generate output IR code.

Definition at line 398 of file VPlan.h.

Referenced by llvm::VPInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPWidenPHIRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenStoreRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPEVLBasedIVPHIRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPlan::execute(), llvm::VPBasicBlock::execute(), llvm::LoopVectorizationPlanner::executePlan(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), get(), packScalarIntoVectorValue(), llvm::VPlan::prepareToExecute(), llvm::InnerLoopVectorizer::scalarizeInstruction(), and setDebugLocFrom().

◆ CFG

struct llvm::VPTransformState::CFGState llvm::VPTransformState::CFG

Referenced by llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPCanonicalIVPHIRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPEVLBasedIVPHIRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPlan::execute(), llvm::VPBasicBlock::execute(), llvm::VPRegionBlock::execute(), llvm::LoopVectorizationPlanner::executePlan(), llvm::InnerLoopVectorizer::fixNonInductionPHIs(), llvm::VPLiveOut::fixPhi(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), get(), and llvm::VPlan::prepareToExecute().

◆ CurrentVectorLoop

Loop* llvm::VPTransformState::CurrentVectorLoop = nullptr

The loop object for the current parent region, or nullptr.

Definition at line 407 of file VPlan.h.

Referenced by llvm::VPReductionPHIRecipe::execute(), llvm::VPBasicBlock::execute(), and llvm::VPRegionBlock::execute().

◆ Data

struct llvm::VPTransformState::DataState llvm::VPTransformState::Data

Referenced by get(), hasScalarValue(), hasVectorValue(), reset(), and set().

◆ DT

DominatorTree* llvm::VPTransformState::DT

Hold a pointer to Dominator Tree to register new basic blocks in the loop.

Definition at line 395 of file VPlan.h.

Referenced by llvm::VPlan::execute().

◆ EVL

VPValue* llvm::VPTransformState::EVL = nullptr

If EVL (Explicit Vector Length) is not nullptr, then EVL must be a valid value set during plan transformation, possibly a default value = whole vector register length.

EVL is created only if TTI prefers predicated vectorization, thus if EVL is not nullptr it also implies preference for predicated vectorization. TODO: this is a temporarily solution, the EVL must be explicitly used by the recipes and must be removed here.

Definition at line 252 of file VPlan.h.

Referenced by llvm::VPWidenLoadRecipe::execute(), and llvm::VPWidenStoreRecipe::execute().

◆ ExpandedSCEVs

DenseMap<const SCEV *, Value *> llvm::VPTransformState::ExpandedSCEVs

Map SCEVs to their expanded values.

Populated when executing VPExpandSCEVRecipes.

Definition at line 418 of file VPlan.h.

Referenced by llvm::VPExpandSCEVRecipe::execute(), and llvm::LoopVectorizationPlanner::executePlan().

◆ ILV

InnerLoopVectorizer* llvm::VPTransformState::ILV

Hold a pointer to InnerLoopVectorizer to reuse its IR generation methods.

Definition at line 401 of file VPlan.h.

Referenced by llvm::VPInterleaveRecipe::execute(), and llvm::VPReplicateRecipe::execute().

◆ Instance

std::optional<VPIteration> llvm::VPTransformState::Instance

Hold the indices to generate specific scalar instructions.

Null indicates that all instances are to be generated, using either scalar or vector instructions.

Definition at line 257 of file VPlan.h.

Referenced by llvm::VPInstruction::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPBasicBlock::execute(), llvm::VPRegionBlock::execute(), get(), hasScalarValue(), packScalarIntoVectorValue(), reset(), and set().

◆ LI

LoopInfo* llvm::VPTransformState::LI

Hold a pointer to LoopInfo to register new basic blocks in the loop.

Definition at line 392 of file VPlan.h.

Referenced by llvm::VPBasicBlock::execute(), and llvm::VPRegionBlock::execute().

◆ LVer

LoopVersioning* llvm::VPTransformState::LVer = nullptr

LoopVersioning.

It's only set up (non-null) if memchecks were used.

This is currently only used to add no-alias metadata based on the memchecks. The actually versioning is performed manually.

Definition at line 414 of file VPlan.h.

Referenced by addNewMetadata(), and llvm::LoopVectorizationPlanner::executePlan().

◆ Plan

VPlan* llvm::VPTransformState::Plan

Pointer to the VPlan code is generated for.

Definition at line 404 of file VPlan.h.

Referenced by llvm::VPlan::createInitialVPlan(), llvm::VPlanPrinter::dump(), llvm::InnerLoopVectorizer::fixFixedOrderRecurrence(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), get(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), and llvm::VPInterleavedAccessInfo::VPInterleavedAccessInfo().

◆ TypeAnalysis

VPTypeAnalysis llvm::VPTransformState::TypeAnalysis

VPlan-based type analysis.

Definition at line 421 of file VPlan.h.

Referenced by llvm::VPWidenRecipe::execute(), and llvm::InnerLoopVectorizer::scalarizeInstruction().

◆ UF

unsigned llvm::VPTransformState::UF

Definition at line 243 of file VPlan.h.

Referenced by createAndCollectMergePhiForReduction(), llvm::VPInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPScalarCastRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenStoreRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPCanonicalIVPHIRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPEVLBasedIVPHIRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPlan::execute(), llvm::VPRegionBlock::execute(), llvm::VPLiveOut::fixPhi(), llvm::VPlan::getName(), llvm::VPlan::prepareToExecute(), set(), and setDebugLocFrom().

◆ VF

ElementCount llvm::VPTransformState::VF

The chosen Vectorization and Unroll Factors of the loop being vectorized.

Definition at line 242 of file VPlan.h.

Referenced by llvm::VPInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPlan::execute(), llvm::VPRegionBlock::execute(), llvm::VPLiveOut::fixPhi(), get(), llvm::VPlan::getName(), hasScalarValue(), packScalarIntoVectorValue(), llvm::VPlan::prepareToExecute(), reset(), set(), and setDebugLocFrom().

The documentation for this struct was generated from the following files:
Generated on Fri Apr 19 2024 01:26:56 for LLVM by doxygen 1.9.6