LLVM  14.0.0git
Classes | Public Member Functions | Static Public Member Functions | List of all members
llvm::RecurrenceDescriptor Class Reference

The RecurrenceDescriptor is used to identify recurrences variables in a loop. More...

#include "llvm/Analysis/IVDescriptors.h"

Classes

class  InstDesc
 This POD struct holds information about a potential recurrence operation. More...
 

Public Member Functions

 RecurrenceDescriptor ()=default
 
 RecurrenceDescriptor (Value *Start, Instruction *Exit, RecurKind K, FastMathFlags FMF, Instruction *ExactFP, Type *RT, bool Signed, bool Ordered, SmallPtrSetImpl< Instruction * > &CI)
 
ValuegetRecurrenceIdentity (RecurKind K, Type *Tp, FastMathFlags FMF)
 Returns identity corresponding to the RecurrenceKind. More...
 
RecurKind getRecurrenceKind () const
 
unsigned getOpcode () const
 
FastMathFlags getFastMathFlags () const
 
TrackingVH< ValuegetRecurrenceStartValue () const
 
InstructiongetLoopExitInstr () const
 
bool hasExactFPMath () const
 Returns true if the recurrence has floating-point math that requires precise (ordered) operations. More...
 
InstructiongetExactFPMathInst () const
 Returns 1st non-reassociative FP instruction in the PHI node's use-chain. More...
 
TypegetRecurrenceType () const
 Returns the type of the recurrence. More...
 
const SmallPtrSet< Instruction *, 8 > & getCastInsts () const
 Returns a reference to the instructions used for type-promoting the recurrence. More...
 
bool isSigned () const
 Returns true if all source operands of the recurrence are SExtInsts. More...
 
bool isOrdered () const
 Expose an ordered FP reduction to the instance users. More...
 
SmallVector< Instruction *, 4 > getReductionOpChain (PHINode *Phi, Loop *L) const
 Attempts to find a chain of operations from Phi to LoopExitInst that can be treated as a set of reductions instructions for in-loop reductions. More...
 

Static Public Member Functions

static InstDesc isRecurrenceInstr (Loop *L, PHINode *Phi, Instruction *I, RecurKind Kind, InstDesc &Prev, FastMathFlags FuncFMF)
 Returns a struct describing if the instruction 'I' can be a recurrence variable of type 'Kind' for a Loop L and reduction PHI Phi. More...
 
static bool hasMultipleUsesOf (Instruction *I, SmallPtrSetImpl< Instruction * > &Insts, unsigned MaxNumUses)
 Returns true if instruction I has multiple uses in Insts. More...
 
static bool areAllUsesIn (Instruction *I, SmallPtrSetImpl< Instruction * > &Set)
 Returns true if all uses of the instruction I is within the Set. More...
 
static InstDesc isMinMaxPattern (Instruction *I, RecurKind Kind, const InstDesc &Prev)
 Returns a struct describing if the instruction is a llvm. More...
 
static InstDesc isSelectCmpPattern (Loop *Loop, PHINode *OrigPhi, Instruction *I, InstDesc &Prev)
 Returns a struct describing whether the instruction is either a Select(ICmp(A, B), X, Y), or Select(FCmp(A, B), X, Y) where one of (X, Y) is a loop invariant integer and the other is a PHI value. More...
 
static InstDesc isConditionalRdxPattern (RecurKind Kind, Instruction *I)
 Returns a struct describing if the instruction is a Select(FCmp(X, Y), (Z = X op PHINode), PHINode) instruction pattern. More...
 
static unsigned getOpcode (RecurKind Kind)
 Returns the opcode corresponding to the RecurrenceKind. More...
 
static bool AddReductionVar (PHINode *Phi, RecurKind Kind, Loop *TheLoop, FastMathFlags FuncFMF, RecurrenceDescriptor &RedDes, DemandedBits *DB=nullptr, AssumptionCache *AC=nullptr, DominatorTree *DT=nullptr)
 Returns true if Phi is a reduction of type Kind and adds it to the RecurrenceDescriptor. More...
 
static bool isReductionPHI (PHINode *Phi, Loop *TheLoop, RecurrenceDescriptor &RedDes, DemandedBits *DB=nullptr, AssumptionCache *AC=nullptr, DominatorTree *DT=nullptr)
 Returns true if Phi is a reduction in TheLoop. More...
 
static bool isFirstOrderRecurrence (PHINode *Phi, Loop *TheLoop, MapVector< Instruction *, Instruction * > &SinkAfter, DominatorTree *DT)
 Returns true if Phi is a first-order recurrence. More...
 
static bool isIntegerRecurrenceKind (RecurKind Kind)
 Returns true if the recurrence kind is an integer kind. More...
 
static bool isFloatingPointRecurrenceKind (RecurKind Kind)
 Returns true if the recurrence kind is a floating point kind. More...
 
static bool isArithmeticRecurrenceKind (RecurKind Kind)
 Returns true if the recurrence kind is an arithmetic kind. More...
 
static bool isIntMinMaxRecurrenceKind (RecurKind Kind)
 Returns true if the recurrence kind is an integer min/max kind. More...
 
static bool isFPMinMaxRecurrenceKind (RecurKind Kind)
 Returns true if the recurrence kind is a floating-point min/max kind. More...
 
static bool isMinMaxRecurrenceKind (RecurKind Kind)
 Returns true if the recurrence kind is any min/max kind. More...
 
static bool isSelectCmpRecurrenceKind (RecurKind Kind)
 Returns true if the recurrence kind is of the form select(cmp(),x,y) where one of (x,y) is loop invariant. More...
 
static bool isFMulAddIntrinsic (Instruction *I)
 Returns true if the instruction is a call to the llvm.fmuladd intrinsic. More...
 

Detailed Description

The RecurrenceDescriptor is used to identify recurrences variables in a loop.

Reduction is a special case of recurrence that has uses of the recurrence variable outside the loop. The method isReductionPHI identifies reductions that are basic recurrences.

Basic recurrences are defined as the summation, product, OR, AND, XOR, min, or max of a set of terms. For example: for(i=0; i<n; i++) { total += array[i]; } is a summation of array elements. Basic recurrences are a special case of chains of recurrences (CR). See ScalarEvolution for CR references. This struct holds information about recurrence variables.

Definition at line 73 of file IVDescriptors.h.

Constructor & Destructor Documentation

◆ RecurrenceDescriptor() [1/2]

llvm::RecurrenceDescriptor::RecurrenceDescriptor ( )
default

◆ RecurrenceDescriptor() [2/2]

llvm::RecurrenceDescriptor::RecurrenceDescriptor ( Value Start,
Instruction Exit,
RecurKind  K,
FastMathFlags  FMF,
Instruction ExactFP,
Type RT,
bool  Signed,
bool  Ordered,
SmallPtrSetImpl< Instruction * > &  CI 
)
inline

Definition at line 77 of file IVDescriptors.h.

Member Function Documentation

◆ AddReductionVar()

bool RecurrenceDescriptor::AddReductionVar ( PHINode Phi,
RecurKind  Kind,
Loop TheLoop,
FastMathFlags  FuncFMF,
RecurrenceDescriptor RedDes,
DemandedBits DB = nullptr,
AssumptionCache AC = nullptr,
DominatorTree DT = nullptr 
)
static

Returns true if Phi is a reduction of type Kind and adds it to the RecurrenceDescriptor.

If either DB is non-null or AC and DT are non-null, the minimal bit width needed to compute the reduction will be computed.

Definition at line 228 of file IVDescriptors.cpp.

References llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::PHINode::getIncomingValueForBlock(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::PHINode::getNumIncomingValues(), llvm::Instruction::getParent(), and llvm::Value::getType().

◆ areAllUsesIn()

bool RecurrenceDescriptor::areAllUsesIn ( Instruction I,
SmallPtrSetImpl< Instruction * > &  Set 
)
static

Returns true if all uses of the instruction I is within the Set.

Definition at line 44 of file IVDescriptors.cpp.

References llvm::SmallPtrSetImpl< PtrType >::count(), and I.

◆ getCastInsts()

const SmallPtrSet<Instruction *, 8>& llvm::RecurrenceDescriptor::getCastInsts ( ) const
inline

Returns a reference to the instructions used for type-promoting the recurrence.

Definition at line 252 of file IVDescriptors.h.

Referenced by llvm::LoopVectorizationCostModel::collectValuesToIgnore().

◆ getExactFPMathInst()

Instruction* llvm::RecurrenceDescriptor::getExactFPMathInst ( ) const
inline

Returns 1st non-reassociative FP instruction in the PHI node's use-chain.

Definition at line 213 of file IVDescriptors.h.

◆ getFastMathFlags()

FastMathFlags llvm::RecurrenceDescriptor::getFastMathFlags ( ) const
inline

◆ getLoopExitInstr()

Instruction* llvm::RecurrenceDescriptor::getLoopExitInstr ( ) const
inline

◆ getOpcode() [1/2]

unsigned llvm::RecurrenceDescriptor::getOpcode ( ) const
inline

Definition at line 200 of file IVDescriptors.h.

References getOpcode(), and getRecurrenceKind().

Referenced by llvm::InnerLoopVectorizer::fixReduction(), and getOpcode().

◆ getOpcode() [2/2]

unsigned RecurrenceDescriptor::getOpcode ( RecurKind  Kind)
static

◆ getRecurrenceIdentity()

Value * RecurrenceDescriptor::getRecurrenceIdentity ( RecurKind  K,
Type Tp,
FastMathFlags  FMF 
)

◆ getRecurrenceKind()

RecurKind llvm::RecurrenceDescriptor::getRecurrenceKind ( ) const
inline

◆ getRecurrenceStartValue()

TrackingVH<Value> llvm::RecurrenceDescriptor::getRecurrenceStartValue ( ) const
inline

◆ getRecurrenceType()

Type* llvm::RecurrenceDescriptor::getRecurrenceType ( ) const
inline

Returns the type of the recurrence.

This type can be narrower than the actual type of the Phi if the recurrence has been type-promoted.

Definition at line 248 of file IVDescriptors.h.

Referenced by llvm::LoopVectorizationCostModel::collectElementTypesForWidening(), llvm::LoopVectorizationCostModel::collectInLoopReductions(), llvm::InnerLoopVectorizer::fixReduction(), llvm::RISCVTTIImpl::isLegalToVectorizeReduction(), and llvm::AArch64TTIImpl::isLegalToVectorizeReduction().

◆ getReductionOpChain()

SmallVector< Instruction *, 4 > RecurrenceDescriptor::getReductionOpChain ( PHINode Phi,
Loop L 
) const

Attempts to find a chain of operations from Phi to LoopExitInst that can be treated as a set of reductions instructions for in-loop reductions.

Definition at line 1019 of file IVDescriptors.cpp.

References getOpcode(), llvm::Value::hasNUses(), llvm::SelectPatternResult::isMinOrMax(), and llvm::matchSelectPattern().

Referenced by llvm::LoopVectorizationCostModel::collectInLoopReductions().

◆ hasExactFPMath()

bool llvm::RecurrenceDescriptor::hasExactFPMath ( ) const
inline

Returns true if the recurrence has floating-point math that requires precise (ordered) operations.

Definition at line 210 of file IVDescriptors.h.

Referenced by llvm::LoopVectorizationLegality::canVectorizeFPMath().

◆ hasMultipleUsesOf()

bool RecurrenceDescriptor::hasMultipleUsesOf ( Instruction I,
SmallPtrSetImpl< Instruction * > &  Insts,
unsigned  MaxNumUses 
)
static

Returns true if instruction I has multiple uses in Insts.

Definition at line 734 of file IVDescriptors.cpp.

References llvm::SmallPtrSetImpl< PtrType >::count(), and I.

◆ isArithmeticRecurrenceKind()

bool RecurrenceDescriptor::isArithmeticRecurrenceKind ( RecurKind  Kind)
static

Returns true if the recurrence kind is an arithmetic kind.

Definition at line 76 of file IVDescriptors.cpp.

References llvm::Add, llvm::FAdd, llvm::FMul, llvm::FMulAdd, and llvm::Mul.

◆ isConditionalRdxPattern()

RecurrenceDescriptor::InstDesc RecurrenceDescriptor::isConditionalRdxPattern ( RecurKind  Kind,
Instruction I 
)
static

Returns a struct describing if the instruction is a Select(FCmp(X, Y), (Z = X op PHINode), PHINode) instruction pattern.

Returns true if the select instruction has users in the compare-and-add reduction pattern below.

The select instruction argument is the last one in the sequence.

sum.1 = phi ... ... cmp = fcmp pred %0, CFP add = fadd %0, sum.1 sum.2 = select cmp, add, sum.1

Definition at line 647 of file IVDescriptors.cpp.

References llvm::FAdd, llvm::tgtok::FalseVal, llvm::FMul, llvm::Value::hasOneUse(), I, llvm::Instruction::isBinaryOp(), llvm::Instruction::isFast(), llvm::PatternMatch::m_FAdd(), llvm::PatternMatch::m_FMul(), llvm::PatternMatch::m_FSub(), llvm::PatternMatch::m_Value(), llvm::PatternMatch::match(), SI, and llvm::tgtok::TrueVal.

◆ isFirstOrderRecurrence()

bool RecurrenceDescriptor::isFirstOrderRecurrence ( PHINode Phi,
Loop TheLoop,
MapVector< Instruction *, Instruction * > &  SinkAfter,
DominatorTree DT 
)
static

Returns true if Phi is a first-order recurrence.

A first-order recurrence is a non-reduction recurrence relation in which the value of the recurrence in the current loop iteration equals a value defined in the previous iteration. SinkAfter includes pairs of instructions where the first will be rescheduled to appear after the second if/when the loop is vectorized. It may be augmented with additional pairs if needed in order to handle Phi as a first-order recurrence.

Definition at line 833 of file IVDescriptors.cpp.

References B, llvm::LoopBase< BlockT, LoopT >::contains(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::count(), llvm::DominatorTree::dominates(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::end(), llvm::BasicBlock::end(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::find(), llvm::PHINode::getBasicBlockIndex(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::PHINode::getIncomingValueForBlock(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::PHINode::getNumIncomingValues(), llvm::Instruction::getParent(), I, llvm::SmallVectorImpl< T >::insert(), llvm::SmallVectorImpl< T >::pop_back_val(), set, and llvm::Value::users().

◆ isFloatingPointRecurrenceKind()

bool RecurrenceDescriptor::isFloatingPointRecurrenceKind ( RecurKind  Kind)
static

Returns true if the recurrence kind is a floating point kind.

Definition at line 72 of file IVDescriptors.cpp.

References llvm::None.

◆ isFMulAddIntrinsic()

static bool llvm::RecurrenceDescriptor::isFMulAddIntrinsic ( Instruction I)
inlinestatic

Returns true if the instruction is a call to the llvm.fmuladd intrinsic.

Definition at line 266 of file IVDescriptors.h.

References I.

Referenced by checkOrderedReduction().

◆ isFPMinMaxRecurrenceKind()

static bool llvm::RecurrenceDescriptor::isFPMinMaxRecurrenceKind ( RecurKind  Kind)
inlinestatic

Returns true if the recurrence kind is a floating-point min/max kind.

Definition at line 231 of file IVDescriptors.h.

References llvm::FMax, and llvm::FMin.

Referenced by isMinMaxRecurrenceKind().

◆ isIntegerRecurrenceKind()

bool RecurrenceDescriptor::isIntegerRecurrenceKind ( RecurKind  Kind)
static

Returns true if the recurrence kind is an integer kind.

Definition at line 52 of file IVDescriptors.cpp.

References llvm::Add, llvm::And, llvm::Mul, llvm::Or, llvm::SelectFCmp, llvm::SelectICmp, llvm::SMax, llvm::SMin, llvm::UMax, llvm::UMin, and llvm::Xor.

◆ isIntMinMaxRecurrenceKind()

static bool llvm::RecurrenceDescriptor::isIntMinMaxRecurrenceKind ( RecurKind  Kind)
inlinestatic

Returns true if the recurrence kind is an integer min/max kind.

Definition at line 225 of file IVDescriptors.h.

References llvm::SMax, llvm::SMin, llvm::UMax, and llvm::UMin.

Referenced by isMinMaxRecurrenceKind().

◆ isMinMaxPattern()

RecurrenceDescriptor::InstDesc RecurrenceDescriptor::isMinMaxPattern ( Instruction I,
RecurKind  Kind,
const InstDesc Prev 
)
static

Returns a struct describing if the instruction is a llvm.

(s/u)(min/max), llvm.minnum/maxnum or a Select(ICmp(X, Y), X, Y) pair of instructions corresponding to a min(X, Y) or max(X, Y), matching the recurrence kind Kind. Prev specifies the description of an already processed select instruction, so its corresponding cmp can be matched to it.

Definition at line 591 of file IVDescriptors.cpp.

References assert(), llvm::FMax, llvm::FMin, llvm::RecurrenceDescriptor::InstDesc::getRecKind(), I, llvm::PatternMatch::m_Cmp(), llvm::PatternMatch::m_OneUse(), llvm::PatternMatch::m_OrdFMax(), llvm::PatternMatch::m_OrdFMin(), llvm::PatternMatch::m_Select(), llvm::PatternMatch::m_SMax(), llvm::PatternMatch::m_SMin(), llvm::PatternMatch::m_UMax(), llvm::PatternMatch::m_UMin(), llvm::PatternMatch::m_UnordFMax(), llvm::PatternMatch::m_UnordFMin(), llvm::PatternMatch::m_Value(), llvm::PatternMatch::match(), llvm::MCID::Select, llvm::SMax, llvm::SMin, llvm::UMax, and llvm::UMin.

◆ isMinMaxRecurrenceKind()

static bool llvm::RecurrenceDescriptor::isMinMaxRecurrenceKind ( RecurKind  Kind)
inlinestatic

Returns true if the recurrence kind is any min/max kind.

Definition at line 236 of file IVDescriptors.h.

References isFPMinMaxRecurrenceKind(), and isIntMinMaxRecurrenceKind().

Referenced by llvm::VPReductionPHIRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::getOrderedReduction(), and llvm::getShuffleReduction().

◆ isOrdered()

bool llvm::RecurrenceDescriptor::isOrdered ( ) const
inline

Expose an ordered FP reduction to the instance users.

Definition at line 258 of file IVDescriptors.h.

Referenced by llvm::LoopVectorizationLegality::canVectorizeFPMath(), and llvm::LoopVectorizationCostModel::useOrderedReductions().

◆ isRecurrenceInstr()

RecurrenceDescriptor::InstDesc RecurrenceDescriptor::isRecurrenceInstr ( Loop L,
PHINode Phi,
Instruction I,
RecurKind  Kind,
InstDesc Prev,
FastMathFlags  FuncFMF 
)
static

Returns a struct describing if the instruction 'I' can be a recurrence variable of type 'Kind' for a Loop L and reduction PHI Phi.

If the recurrence is a min/max pattern of select(icmp()) this function advances the instruction pointer 'I' from the compare instruction to the select instruction and stores this pointer in 'PatternLastInst' member of the returned struct.

Definition at line 684 of file IVDescriptors.cpp.

References llvm::Add, llvm::MCID::Add, llvm::And, assert(), llvm::MCID::Call, llvm::FAdd, llvm::FMul, llvm::FMulAdd, llvm::RecurrenceDescriptor::InstDesc::getExactFPMathInst(), llvm::RecurrenceDescriptor::InstDesc::getRecKind(), I, LLVM_FALLTHROUGH, llvm::Mul, llvm::FastMathFlags::noNaNs(), llvm::None, llvm::FastMathFlags::noSignedZeros(), llvm::Or, llvm::MCID::Select, and llvm::Xor.

◆ isReductionPHI()

bool RecurrenceDescriptor::isReductionPHI ( PHINode Phi,
Loop TheLoop,
RecurrenceDescriptor RedDes,
DemandedBits DB = nullptr,
AssumptionCache AC = nullptr,
DominatorTree DT = nullptr 
)
static

Returns true if Phi is a reduction in TheLoop.

The RecurrenceDescriptor is returned in RedDes. If either DB is non-null or AC and DT are non-null, the minimal bit width needed to compute the reduction will be computed.

Definition at line 748 of file IVDescriptors.cpp.

References llvm::Add, llvm::And, llvm::dbgs(), F, llvm::FAdd, llvm::FMax, llvm::FMin, llvm::FMul, llvm::FMulAdd, llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::BasicBlock::getParent(), LLVM_DEBUG, llvm::Mul, llvm::Or, llvm::SelectFCmp, llvm::SelectICmp, llvm::FastMathFlags::setNoNaNs(), llvm::FastMathFlags::setNoSignedZeros(), llvm::SMax, llvm::SMin, llvm::UMax, llvm::UMin, and llvm::Xor.

Referenced by findInnerReductionPhi().

◆ isSelectCmpPattern()

RecurrenceDescriptor::InstDesc RecurrenceDescriptor::isSelectCmpPattern ( Loop Loop,
PHINode OrigPhi,
Instruction I,
InstDesc Prev 
)
static

Returns a struct describing whether the instruction is either a Select(ICmp(A, B), X, Y), or Select(FCmp(A, B), X, Y) where one of (X, Y) is a loop invariant integer and the other is a PHI value.

Prev specifies the description of an already processed select instruction, so its corresponding cmp can be matched to it.

Definition at line 555 of file IVDescriptors.cpp.

References llvm::RecurrenceDescriptor::InstDesc::getRecKind(), I, llvm::Loop::isLoopInvariant(), llvm::PatternMatch::m_Cmp(), llvm::PatternMatch::m_OneUse(), llvm::PatternMatch::m_Select(), llvm::PatternMatch::m_Value(), llvm::PatternMatch::match(), llvm::MCID::Select, llvm::SelectFCmp, llvm::SelectICmp, and SI.

◆ isSelectCmpRecurrenceKind()

static bool llvm::RecurrenceDescriptor::isSelectCmpRecurrenceKind ( RecurKind  Kind)
inlinestatic

Returns true if the recurrence kind is of the form select(cmp(),x,y) where one of (x,y) is loop invariant.

Definition at line 242 of file IVDescriptors.h.

References llvm::SelectFCmp, and llvm::SelectICmp.

Referenced by llvm::createSelectCmpTargetReduction(), llvm::createTargetReduction(), llvm::VPReductionPHIRecipe::execute(), and llvm::InnerLoopVectorizer::fixReduction().

◆ isSigned()

bool llvm::RecurrenceDescriptor::isSigned ( ) const
inline

Returns true if all source operands of the recurrence are SExtInsts.

Definition at line 255 of file IVDescriptors.h.

Referenced by llvm::InnerLoopVectorizer::fixReduction().


The documentation for this class was generated from the following files: