#include "Target/AMDGPU/GCNHazardRecognizer.h"

Inheritance diagram for llvm::GCNHazardRecognizer:

Public Types
enum class	OperatingMode { PreRA , PostRA , HazardRecognizerMode }
	Operating mode for the hazard recognizer. More...
typedef function_ref< bool(const MachineInstr &)>	IsHazardFn
typedef function_ref< bool(const MachineInstr &, int WaitStates)>	IsExpiredFn
typedef function_ref< unsigned int(const MachineInstr &)>	GetNumWaitStatesFn
Public Types inherited from llvm::ScheduleHazardRecognizer
enum	HazardType { NoHazard , Hazard , NoopHazard }

Public Member Functions
	GCNHazardRecognizer (const MachineFunction &MF, OperatingMode Mode, MachineLoopInfo *MLI=nullptr)
	Construct with explicit operating mode.
	GCNHazardRecognizer (const MachineFunction &MF, MachineLoopInfo *MLI=nullptr)
	Legacy constructor - defaults to PostRA mode.
	~GCNHazardRecognizer ()
OperatingMode	getOperatingMode () const
	Returns the current operating mode.
bool	isPreRA () const
	Returns true if running in pre-RA scheduling mode.
bool	isPostRA () const
	Returns true if running in post-RA scheduling mode.
bool	isSchedulerMode () const
	Returns true if running as a scheduler (pre-RA or post-RA).
bool	hasPhysRegs () const
	Returns true if instruction operands are physical registers, so that hazards defined by register dependences can be detected.
bool	isHazardRecognizerMode () const
	Returns true if running as the standalone hazard recognizer pass.
bool	inCoExecWindow () const
	Returns true if currently inside a WMMA co-execution window.
std::optional< unsigned >	getCurrentCoExecStage () const
	Returns the current stage within the co-execution window, or nullopt.
const AMDGPU::CoExecInfo &	getActiveCoExecInfo () const
	Returns the active co-execution info (slot masks, preferences).
bool	atIssueLimit () const override
	atIssueLimit - Return true if no more instructions may be issued in this cycle.
void	EmitInstruction (SUnit *SU) override
	EmitInstruction - This callback is invoked when an instruction is emitted, to advance the hazard state.
void	EmitInstruction (MachineInstr *MI) override
	This overload will be used when the hazard recognizer is being used by a non-scheduling pass, which does not use SUnits.
HazardType	getHazardType (SUnit *SU, int Stalls) override
	getHazardType - Return the hazard type of emitting this node.
unsigned	getHazardWaitStates (MachineInstr *MI) const
	Returns the number of wait states until all hazards for `MI` are resolved.
void	EmitNoop () override
	EmitNoop - This callback is invoked when a noop was added to the instruction stream.
unsigned	PreEmitNoops (MachineInstr *) override
	This overload will be used when the hazard recognizer is being used by a non-scheduling pass, which does not use SUnits.
unsigned	PreEmitNoopsCommon (MachineInstr *) const
void	AdvanceCycle () override
	AdvanceCycle - This callback is invoked whenever the next top-down instruction to be scheduled cannot issue in the current cycle, either because of latency or resource conflicts.
void	RecedeCycle () override
	RecedeCycle - This callback is invoked whenever the next bottom-up instruction to be scheduled cannot issue in the current cycle, either because of latency or resource conflicts.
bool	ShouldPreferAnother (SUnit *SU) const override
	ShouldPreferAnother - This callback may be invoked if getHazardType returns NoHazard.
void	Reset () override
	Reset - This callback is invoked when a new block of instructions is about to be schedule.
Public Member Functions inherited from llvm::ScheduleHazardRecognizer
	ScheduleHazardRecognizer ()=default
virtual	~ScheduleHazardRecognizer ()
unsigned	getMaxLookAhead () const
bool	isEnabled () const
virtual unsigned	PreEmitNoops (SUnit *)
	PreEmitNoops - This callback is invoked prior to emitting an instruction.
virtual void	EmitNoops (unsigned Quantity)
	EmitNoops - This callback is invoked when noops were added to the instruction stream.

Static Public Member Functions
static AMDGPU::CoExecMaskT	getCoExecMaskForMI (const MachineInstr &MI, const SIInstrInfo &TII)
	Get the CoExecMask for a given instruction.

Additional Inherited Members
Protected Attributes inherited from llvm::ScheduleHazardRecognizer
unsigned	MaxLookAhead = 0
	MaxLookAhead - Indicate the number of cycles in the scoreboard state.

Detailed Description

Definition at line 36 of file GCNHazardRecognizer.h.

Member Typedef Documentation

◆ GetNumWaitStatesFn

typedef function_ref<unsigned int(const MachineInstr &)> llvm::GCNHazardRecognizer::GetNumWaitStatesFn

Definition at line 40 of file GCNHazardRecognizer.h.

◆ IsExpiredFn

typedef function_ref<bool(const MachineInstr &, int WaitStates)> llvm::GCNHazardRecognizer::IsExpiredFn

Definition at line 39 of file GCNHazardRecognizer.h.

◆ IsHazardFn

typedef function_ref<bool(const MachineInstr &)> llvm::GCNHazardRecognizer::IsHazardFn

Definition at line 38 of file GCNHazardRecognizer.h.

Member Enumeration Documentation

◆ OperatingMode

enum class llvm::GCNHazardRecognizer::OperatingMode

strong

Operating mode for the hazard recognizer.

Two independent properties follow from it.

The scheduler modes (PreRA, PostRA) report a hazard so that the scheduler can pick some other instruction, and to do that they track pipeline state across the cycles they are told about. HazardRecognizerMode inserts s_nop or v_nop to ensure correctness of the generated code.

Operands are physical registers in every mode but PreRA, so hazards defined by register dependences are checked only when hasPhysRegs() holds.

Enumerator
PreRA
PostRA
HazardRecognizerMode

Definition at line 53 of file GCNHazardRecognizer.h.

Constructor & Destructor Documentation

◆ GCNHazardRecognizer() [1/2]

GCNHazardRecognizer::GCNHazardRecognizer	(	const MachineFunction &	MF,
		GCNHazardRecognizer::OperatingMode	Mode,
		MachineLoopInfo *	MLI = nullptr )

Construct with explicit operating mode.

Definition at line 77 of file GCNHazardRecognizer.cpp.

References llvm::dbgs(), isPreRA(), LLVM_DEBUG, llvm::ScheduleHazardRecognizer::MaxLookAhead, and shouldRunLdsBranchVmemWARHazardFixup().

Referenced by GCNHazardRecognizer().

◆ GCNHazardRecognizer() [2/2]

GCNHazardRecognizer::GCNHazardRecognizer	(	const MachineFunction &	MF,
		MachineLoopInfo *	MLI = nullptr )

Legacy constructor - defaults to PostRA mode.

Definition at line 92 of file GCNHazardRecognizer.cpp.

References GCNHazardRecognizer(), and PostRA.

◆ ~GCNHazardRecognizer()

GCNHazardRecognizer::~GCNHazardRecognizer ( )

Definition at line 96 of file GCNHazardRecognizer.cpp.

References llvm::dbgs(), LLVM_DEBUG, and llvm::AMDGPU::MaxCoExecStages.

Member Function Documentation

◆ AdvanceCycle()

void GCNHazardRecognizer::AdvanceCycle ( )

overridevirtual

AdvanceCycle - This callback is invoked whenever the next top-down instruction to be scheduled cannot issue in the current cycle, either because of latency or resource conflicts.

This should increment the internal state of the hazard recognizer so that previously "Hazard" instructions will now not be hazards.

Reimplemented from llvm::ScheduleHazardRecognizer.

Definition at line 808 of file GCNHazardRecognizer.cpp.

References llvm::ScheduleHazardRecognizer::getMaxLookAhead(), isSchedulerMode(), llvm::SIInstrInfo::isSWMMAC(), llvm::SIInstrInfo::isVALU(), and llvm::SIInstrInfo::isWMMA().

◆ atIssueLimit()

bool llvm::GCNHazardRecognizer::atIssueLimit ( ) const

inlineoverridevirtual

atIssueLimit - Return true if no more instructions may be issued in this cycle.

FIXME: remove this once MachineScheduler is the only client.

Reimplemented from llvm::ScheduleHazardRecognizer.

Definition at line 318 of file GCNHazardRecognizer.h.

◆ EmitInstruction() [1/2]

void GCNHazardRecognizer::EmitInstruction ( MachineInstr * )

overridevirtual

This overload will be used when the hazard recognizer is being used by a non-scheduling pass, which does not use SUnits.

Reimplemented from llvm::ScheduleHazardRecognizer.

Definition at line 448 of file GCNHazardRecognizer.cpp.

References isSchedulerMode(), and MI.

◆ EmitInstruction() [2/2]

void GCNHazardRecognizer::EmitInstruction ( SUnit * )

overridevirtual

EmitInstruction - This callback is invoked when an instruction is emitted, to advance the hazard state.

Reimplemented from llvm::ScheduleHazardRecognizer.

Definition at line 444 of file GCNHazardRecognizer.cpp.

References EmitInstruction(), and llvm::SUnit::getInstr().

Referenced by EmitInstruction().

◆ EmitNoop()

void GCNHazardRecognizer::EmitNoop ( )

overridevirtual

EmitNoop - This callback is invoked when a noop was added to the instruction stream.

Reimplemented from llvm::ScheduleHazardRecognizer.

Definition at line 804 of file GCNHazardRecognizer.cpp.

◆ getActiveCoExecInfo()

const AMDGPU::CoExecInfo & llvm::GCNHazardRecognizer::getActiveCoExecInfo ( ) const

inline

Returns the active co-execution info (slot masks, preferences).

Definition at line 310 of file GCNHazardRecognizer.h.

◆ getCoExecMaskForMI()

AMDGPU::CoExecMaskT GCNHazardRecognizer::getCoExecMaskForMI	(	const MachineInstr &	MI,
		const SIInstrInfo &	TII )

static

Get the CoExecMask for a given instruction.

Definition at line 296 of file GCNHazardRecognizer.cpp.

References llvm::AMDGPU::classifyFlavor(), llvm::AMDGPU::getCoExecMask(), and MI.

◆ getCurrentCoExecStage()

std::optional< unsigned > llvm::GCNHazardRecognizer::getCurrentCoExecStage ( ) const

inline

Returns the current stage within the co-execution window, or nullopt.

Definition at line 305 of file GCNHazardRecognizer.h.

◆ getHazardType()

ScheduleHazardRecognizer::HazardType GCNHazardRecognizer::getHazardType	(	SUnit *	,
		int	Stalls )

overridevirtual

getHazardType - Return the hazard type of emitting this node.

There are three possible results. Either:

NoHazard: it is legal to issue this instruction on this cycle.
Hazard: issuing this instruction would stall the machine. If some other instruction is available, issue it first.
NoopHazard: issuing this instruction would break the program. If some other instruction can be issued, do so, otherwise issue a noop.

Reimplemented from llvm::ScheduleHazardRecognizer.

Definition at line 549 of file GCNHazardRecognizer.cpp.

References llvm::SUnit::getInstr(), hasPhysRegs(), llvm::ScheduleHazardRecognizer::Hazard, isDivFMas(), llvm::SIInstrInfo::isDPP(), llvm::SIInstrInfo::isDS(), llvm::SIInstrInfo::isEXP(), isHazardRecognizerMode(), isLdsDma(), llvm::SIInstrInfo::isMAI(), isRFE(), isRWLane(), isSchedulerMode(), isSendMsgTraceDataOrGDS(), isSGetReg(), isSMovRel(), llvm::SIInstrInfo::isSMRD(), isSSetReg(), llvm::SIInstrInfo::isVALU(), llvm::SIInstrInfo::isVMEM(), MI, llvm::ScheduleHazardRecognizer::NoHazard, and llvm::ScheduleHazardRecognizer::NoopHazard.

◆ getHazardWaitStates()

unsigned GCNHazardRecognizer::getHazardWaitStates ( MachineInstr * MI ) const

Returns the number of wait states until all hazards for MI are resolved.

This is useful for scheduling heuristics that want cycle-accurate hazard information rather than just a boolean. Unlike PreEmitNoops, this does not modify state or fix hazards.

Definition at line 715 of file GCNHazardRecognizer.cpp.

References hasPhysRegs(), isSchedulerMode(), MI, and PreEmitNoopsCommon().

◆ getOperatingMode()

OperatingMode llvm::GCNHazardRecognizer::getOperatingMode ( ) const

inline

Returns the current operating mode.

Definition at line 277 of file GCNHazardRecognizer.h.

◆ hasPhysRegs()

bool llvm::GCNHazardRecognizer::hasPhysRegs ( ) const

inline

Returns true if instruction operands are physical registers, so that hazards defined by register dependences can be detected.

Definition at line 290 of file GCNHazardRecognizer.h.

References isPreRA().

Referenced by getHazardType(), and getHazardWaitStates().

◆ inCoExecWindow()

bool llvm::GCNHazardRecognizer::inCoExecWindow ( ) const

inline

Returns true if currently inside a WMMA co-execution window.

Definition at line 302 of file GCNHazardRecognizer.h.

◆ isHazardRecognizerMode()

bool llvm::GCNHazardRecognizer::isHazardRecognizerMode ( ) const

inline

Returns true if running as the standalone hazard recognizer pass.

Definition at line 293 of file GCNHazardRecognizer.h.

References HazardRecognizerMode.

Referenced by getHazardType(), PreEmitNoops(), and RecedeCycle().

◆ isPostRA()

bool llvm::GCNHazardRecognizer::isPostRA ( ) const

inline

Returns true if running in post-RA scheduling mode.

Definition at line 283 of file GCNHazardRecognizer.h.

References PostRA.

Referenced by isSchedulerMode().

◆ isPreRA()

bool llvm::GCNHazardRecognizer::isPreRA ( ) const

inline

Returns true if running in pre-RA scheduling mode.

Definition at line 280 of file GCNHazardRecognizer.h.

References PreRA.

Referenced by GCNHazardRecognizer(), hasPhysRegs(), and isSchedulerMode().

◆ isSchedulerMode()

bool llvm::GCNHazardRecognizer::isSchedulerMode ( ) const

inline

Returns true if running as a scheduler (pre-RA or post-RA).

Definition at line 286 of file GCNHazardRecognizer.h.

References isPostRA(), and isPreRA().

Referenced by AdvanceCycle(), EmitInstruction(), getHazardType(), getHazardWaitStates(), and Reset().

◆ PreEmitNoops()

unsigned GCNHazardRecognizer::PreEmitNoops ( MachineInstr * )

overridevirtual

This overload will be used when the hazard recognizer is being used by a non-scheduling pass, which does not use SUnits.

Reimplemented from llvm::ScheduleHazardRecognizer.

Definition at line 706 of file GCNHazardRecognizer.cpp.

References assert(), isHazardRecognizerMode(), MI, NopPadding, and PreEmitNoopsCommon().

◆ PreEmitNoopsCommon()

unsigned GCNHazardRecognizer::PreEmitNoopsCommon ( MachineInstr * MI ) const

Definition at line 732 of file GCNHazardRecognizer.cpp.

References isDivFMas(), llvm::SIInstrInfo::isDPP(), llvm::SIInstrInfo::isDS(), llvm::SIInstrInfo::isEXP(), isLdsDma(), llvm::SIInstrInfo::isMAI(), isPermlane(), isRFE(), isRWLane(), isSendMsgTraceDataOrGDS(), isSGetReg(), isSMovRel(), llvm::SIInstrInfo::isSMRD(), isSSetReg(), llvm::SIInstrInfo::isVALU(), llvm::SIInstrInfo::isVMEM(), and MI.

Referenced by getHazardWaitStates(), and PreEmitNoops().

◆ RecedeCycle()

void GCNHazardRecognizer::RecedeCycle ( )

overridevirtual

RecedeCycle - This callback is invoked whenever the next bottom-up instruction to be scheduled cannot issue in the current cycle, either because of latency or resource conflicts.

Reimplemented from llvm::ScheduleHazardRecognizer.

Definition at line 871 of file GCNHazardRecognizer.cpp.

References assert(), and isHazardRecognizerMode().

◆ Reset()

void GCNHazardRecognizer::Reset ( )

overridevirtual

Reset - This callback is invoked when a new block of instructions is about to be schedule.

The hazard state should be set to an initialized state.

Reimplemented from llvm::ScheduleHazardRecognizer.

Definition at line 110 of file GCNHazardRecognizer.cpp.

References isSchedulerMode().

◆ ShouldPreferAnother()

bool GCNHazardRecognizer::ShouldPreferAnother ( SUnit * ) const

overridevirtual

ShouldPreferAnother - This callback may be invoked if getHazardType returns NoHazard.

If, even though there is no hazard, it would be better to schedule another available instruction, this callback should return true.

Reimplemented from llvm::ScheduleHazardRecognizer.

Definition at line 3844 of file GCNHazardRecognizer.cpp.

References llvm::SUnit::getInstr(), llvm::SUnit::isInstr(), llvm::SIInstrInfo::isMFMA(), and MI.

The documentation for this class was generated from the following files:

lib/Target/AMDGPU/GCNHazardRecognizer.h
lib/Target/AMDGPU/GCNHazardRecognizer.cpp

Public Types

Public Member Functions

Static Public Member Functions

Additional Inherited Members

Detailed Description

Member Typedef Documentation

◆ GetNumWaitStatesFn

◆ IsExpiredFn

◆ IsHazardFn

Member Enumeration Documentation

◆ OperatingMode

Constructor & Destructor Documentation

◆ GCNHazardRecognizer() [1/2]

◆ GCNHazardRecognizer() [2/2]

◆ ~GCNHazardRecognizer()

Member Function Documentation

◆ AdvanceCycle()

◆ atIssueLimit()

◆ EmitInstruction() [1/2]

◆ EmitInstruction() [2/2]

◆ EmitNoop()

◆ getActiveCoExecInfo()

◆ getCoExecMaskForMI()

◆ getCurrentCoExecStage()

◆ getHazardType()

◆ getHazardWaitStates()

◆ getOperatingMode()

◆ hasPhysRegs()

◆ inCoExecWindow()

◆ isHazardRecognizerMode()

◆ isPostRA()

◆ isPreRA()

◆ isSchedulerMode()

◆ PreEmitNoops()

◆ PreEmitNoopsCommon()

◆ RecedeCycle()

◆ Reset()

◆ ShouldPreferAnother()