LLVM 24.0.0git
AMDGPURegisterBankInfo.cpp
Go to the documentation of this file.
1//===- AMDGPURegisterBankInfo.cpp -------------------------------*- C++ -*-==//
2//
3// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
4// See https://llvm.org/LICENSE.txt for license information.
5// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
6//
7//===----------------------------------------------------------------------===//
8/// \file
9/// This file implements the targeting of the RegisterBankInfo class for
10/// AMDGPU.
11///
12/// \par
13///
14/// AMDGPU has unique register bank constraints that require special high level
15/// strategies to deal with. There are two main true physical register banks
16/// VGPR (vector), and SGPR (scalar). Additionally the VCC register bank is a
17/// sort of pseudo-register bank needed to represent SGPRs used in a vector
18/// boolean context. There is also the AGPR bank, which is a special purpose
19/// physical register bank present on some subtargets.
20///
21/// Copying from VGPR to SGPR is generally illegal, unless the value is known to
22/// be uniform. It is generally not valid to legalize operands by inserting
23/// copies as on other targets. Operations which require uniform, SGPR operands
24/// generally require scalarization by repeatedly executing the instruction,
25/// activating each set of lanes using a unique set of input values. This is
26/// referred to as a waterfall loop.
27///
28/// \par Booleans
29///
30/// Booleans (s1 values) requires special consideration. A vector compare result
31/// is naturally a bitmask with one bit per lane, in a 32 or 64-bit
32/// register. These are represented with the VCC bank. During selection, we need
33/// to be able to unambiguously go back from a register class to a register
34/// bank. To distinguish whether an SGPR should use the SGPR or VCC register
35/// bank, we need to know the use context type. An SGPR s1 value always means a
36/// VCC bank value, otherwise it will be the SGPR bank. A scalar compare sets
37/// SCC, which is a 1-bit unaddressable register. This will need to be copied to
38/// a 32-bit virtual register. Taken together, this means we need to adjust the
39/// type of boolean operations to be regbank legal. All SALU booleans need to be
40/// widened to 32-bits, and all VALU booleans need to be s1 values.
41///
42/// A noteworthy exception to the s1-means-vcc rule is for legalization artifact
43/// casts. G_TRUNC s1 results, and G_SEXT/G_ZEXT/G_ANYEXT sources are never vcc
44/// bank. A non-boolean source (such as a truncate from a 1-bit load from
45/// memory) will require a copy to the VCC bank which will require clearing the
46/// high bits and inserting a compare.
47///
48/// \par Constant bus restriction
49///
50/// VALU instructions have a limitation known as the constant bus
51/// restriction. Most VALU instructions can use SGPR operands, but may read at
52/// most 1 SGPR or constant literal value (this to 2 in gfx10 for most
53/// instructions). This is one unique SGPR, so the same SGPR may be used for
54/// multiple operands. From a register bank perspective, any combination of
55/// operands should be legal as an SGPR, but this is contextually dependent on
56/// the SGPR operands all being the same register. There is therefore optimal to
57/// choose the SGPR with the most uses to minimize the number of copies.
58///
59/// We avoid trying to solve this problem in RegBankSelect. Any VALU G_*
60/// operation should have its source operands all mapped to VGPRs (except for
61/// VCC), inserting copies from any SGPR operands. This the most trivial legal
62/// mapping. Anything beyond the simplest 1:1 instruction selection would be too
63/// complicated to solve here. Every optimization pattern or instruction
64/// selected to multiple outputs would have to enforce this rule, and there
65/// would be additional complexity in tracking this rule for every G_*
66/// operation. By forcing all inputs to VGPRs, it also simplifies the task of
67/// picking the optimal operand combination from a post-isel optimization pass.
68///
69//===----------------------------------------------------------------------===//
70
72
74#include "AMDGPUInstrInfo.h"
75#include "AMDGPULaneMaskUtils.h"
76#include "GCNSubtarget.h"
78#include "SIRegisterInfo.h"
84#include "llvm/IR/IntrinsicsAMDGPU.h"
85
86#define GET_TARGET_REGBANK_IMPL
87#include "AMDGPUGenRegisterBank.inc"
88
89// This file will be TableGen'ed at some point.
90#include "AMDGPUGenRegisterBankInfo.def"
91
92using namespace llvm;
93using namespace MIPatternMatch;
94
95namespace {
96
97// Observer to apply a register bank to new registers created by LegalizerHelper.
98class ApplyRegBankMapping final : public GISelChangeObserver {
99private:
101 const AMDGPURegisterBankInfo &RBI;
103 const RegisterBank *NewBank;
105
106public:
107 ApplyRegBankMapping(MachineIRBuilder &B, const AMDGPURegisterBankInfo &RBI_,
108 MachineRegisterInfo &MRI_, const RegisterBank *RB)
109 : B(B), RBI(RBI_), MRI(MRI_), NewBank(RB) {
110 assert(!B.isObservingChanges());
111 B.setChangeObserver(*this);
112 }
113
114 ~ApplyRegBankMapping() override {
115 for (MachineInstr *MI : NewInsts)
116 applyBank(*MI);
117
118 B.stopObservingChanges();
119 }
120
121 /// Set any registers that don't have a set register class or bank to SALU.
122 void applyBank(MachineInstr &MI) {
123 const unsigned Opc = MI.getOpcode();
124 if (Opc == AMDGPU::G_ANYEXT || Opc == AMDGPU::G_ZEXT ||
125 Opc == AMDGPU::G_SEXT) {
126 // LegalizerHelper wants to use the basic legalization artifacts when
127 // widening etc. We don't handle selection with vcc in artifact sources,
128 // so we need to use a select instead to handle these properly.
129 Register DstReg = MI.getOperand(0).getReg();
130 Register SrcReg = MI.getOperand(1).getReg();
131 const RegisterBank *SrcBank = RBI.getRegBank(SrcReg, MRI, *RBI.TRI);
132 if (SrcBank == &AMDGPU::VCCRegBank) {
133 const LLT S32 = LLT::scalar(32);
134 assert(MRI.getType(SrcReg) == LLT::scalar(1));
135 assert(MRI.getType(DstReg) == S32);
136 assert(NewBank == &AMDGPU::VGPRRegBank);
137
138 // Replace the extension with a select, which really uses the boolean
139 // source.
140 B.setInsertPt(*MI.getParent(), MI);
141
142 auto True = B.buildConstant(S32, Opc == AMDGPU::G_SEXT ? -1 : 1);
143 auto False = B.buildConstant(S32, 0);
144 B.buildSelect(DstReg, SrcReg, True, False);
145 MRI.setRegBank(True.getReg(0), *NewBank);
146 MRI.setRegBank(False.getReg(0), *NewBank);
147 MI.eraseFromParent();
148 }
149
150 assert(!MRI.getRegClassOrRegBank(DstReg));
151 MRI.setRegBank(DstReg, *NewBank);
152 return;
153 }
154
155#ifndef NDEBUG
156 if (Opc == AMDGPU::G_TRUNC) {
157 Register DstReg = MI.getOperand(0).getReg();
158 const RegisterBank *DstBank = RBI.getRegBank(DstReg, MRI, *RBI.TRI);
159 assert(DstBank != &AMDGPU::VCCRegBank);
160 }
161#endif
162
163 for (MachineOperand &Op : MI.operands()) {
164 if (!Op.isReg())
165 continue;
166
167 // We may see physical registers if building a real MI
168 Register Reg = Op.getReg();
169 if (Reg.isPhysical() || MRI.getRegClassOrRegBank(Reg))
170 continue;
171
172 const RegisterBank *RB = NewBank;
173 if (MRI.getType(Reg) == LLT::scalar(1)) {
174 assert(NewBank == &AMDGPU::VGPRRegBank &&
175 "s1 operands should only be used for vector bools");
176 assert((MI.getOpcode() != AMDGPU::G_TRUNC &&
177 MI.getOpcode() != AMDGPU::G_ANYEXT) &&
178 "not expecting legalization artifacts here");
179 RB = &AMDGPU::VCCRegBank;
180 }
181
182 MRI.setRegBank(Reg, *RB);
183 }
184 }
185
186 void erasingInstr(MachineInstr &MI) override {}
187
188 void createdInstr(MachineInstr &MI) override {
189 // At this point, the instruction was just inserted and has no operands.
190 NewInsts.push_back(&MI);
191 }
192
193 void changingInstr(MachineInstr &MI) override {}
194 void changedInstr(MachineInstr &MI) override {
195 // FIXME: In principle we should probably add the instruction to NewInsts,
196 // but the way the LegalizerHelper uses the observer, we will always see the
197 // registers we need to set the regbank on also referenced in a new
198 // instruction.
199 }
200};
201
202} // anonymous namespace
203
205 : Subtarget(ST), TRI(Subtarget.getRegisterInfo()),
206 TII(Subtarget.getInstrInfo()) {
207
208 // HACK: Until this is fully tablegen'd.
209 static llvm::once_flag InitializeRegisterBankFlag;
210
211 static auto InitializeRegisterBankOnce = [this]() {
212 assert(&getRegBank(AMDGPU::SGPRRegBankID) == &AMDGPU::SGPRRegBank &&
213 &getRegBank(AMDGPU::VGPRRegBankID) == &AMDGPU::VGPRRegBank &&
214 &getRegBank(AMDGPU::AGPRRegBankID) == &AMDGPU::AGPRRegBank);
215 (void)this;
216 };
217
218 llvm::call_once(InitializeRegisterBankFlag, InitializeRegisterBankOnce);
219}
220
221static bool isVectorRegisterBank(const RegisterBank &Bank) {
222 unsigned BankID = Bank.getID();
223 return BankID == AMDGPU::VGPRRegBankID || BankID == AMDGPU::AGPRRegBankID;
224}
225
227 return RB != &AMDGPU::SGPRRegBank;
228}
229
231 const RegisterBank &Src,
232 TypeSize Size) const {
233 // TODO: Should there be a UniformVGPRRegBank which can use readfirstlane?
234 if (Dst.getID() == AMDGPU::SGPRRegBankID &&
235 (isVectorRegisterBank(Src) || Src.getID() == AMDGPU::VCCRegBankID)) {
236 return std::numeric_limits<unsigned>::max();
237 }
238
239 // Bool values are tricky, because the meaning is based on context. The SCC
240 // and VCC banks are for the natural scalar and vector conditions produced by
241 // a compare.
242 //
243 // Legalization doesn't know about the necessary context, so an s1 use may
244 // have been a truncate from an arbitrary value, in which case a copy (lowered
245 // as a compare with 0) needs to be inserted.
246 if (Size == 1 &&
247 (Dst.getID() == AMDGPU::SGPRRegBankID) &&
248 (isVectorRegisterBank(Src) ||
249 Src.getID() == AMDGPU::SGPRRegBankID ||
250 Src.getID() == AMDGPU::VCCRegBankID))
251 return std::numeric_limits<unsigned>::max();
252
253 // There is no direct copy between AGPRs.
254 if (Dst.getID() == AMDGPU::AGPRRegBankID &&
255 Src.getID() == AMDGPU::AGPRRegBankID)
256 return 4;
257
258 return RegisterBankInfo::copyCost(Dst, Src, Size);
259}
260
262 const ValueMapping &ValMapping,
263 const RegisterBank *CurBank) const {
264 // Check if this is a breakdown for G_LOAD to move the pointer from SGPR to
265 // VGPR.
266 // FIXME: Is there a better way to do this?
267 if (ValMapping.NumBreakDowns >= 2 || ValMapping.BreakDown[0].Length >= 64)
268 return 10; // This is expensive.
269
270 assert(ValMapping.NumBreakDowns == 2 &&
271 ValMapping.BreakDown[0].Length == 32 &&
272 ValMapping.BreakDown[0].StartIdx == 0 &&
273 ValMapping.BreakDown[1].Length == 32 &&
274 ValMapping.BreakDown[1].StartIdx == 32 &&
275 ValMapping.BreakDown[0].RegBank == ValMapping.BreakDown[1].RegBank);
276
277 // 32-bit extract of a 64-bit value is just access of a subregister, so free.
278 // TODO: Cost of 0 hits assert, though it's not clear it's what we really
279 // want.
280
281 // TODO: 32-bit insert to a 64-bit SGPR may incur a non-free copy due to SGPR
282 // alignment restrictions, but this probably isn't important.
283 return 1;
284}
285
286const RegisterBank &
288 LLT Ty) const {
289 // We promote real scalar booleans to SReg_32. Any SGPR using s1 is really a
290 // VCC-like use.
291 if (TRI->isSGPRClass(&RC)) {
292 // FIXME: This probably came from a copy from a physical register, which
293 // should be inferable from the copied to-type. We don't have many boolean
294 // physical register constraints so just assume a normal SGPR for now.
295 if (!Ty.isValid())
296 return AMDGPU::SGPRRegBank;
297
298 return Ty == LLT::scalar(1) ? AMDGPU::VCCRegBank : AMDGPU::SGPRRegBank;
299 }
300
301 return TRI->isAGPRClass(&RC) ? AMDGPU::AGPRRegBank : AMDGPU::VGPRRegBank;
302}
303
304template <unsigned NumOps>
307 const MachineInstr &MI, const MachineRegisterInfo &MRI,
308 const std::array<unsigned, NumOps> RegSrcOpIdx,
310
311 InstructionMappings AltMappings;
312
314
315 unsigned Sizes[NumOps];
316 for (unsigned I = 0; I < NumOps; ++I) {
317 Register Reg = MI.getOperand(RegSrcOpIdx[I]).getReg();
318 Sizes[I] = TRI->getRegSizeInBits(Reg, MRI);
319 }
320
321 for (unsigned I = 0, E = MI.getNumExplicitDefs(); I != E; ++I) {
322 unsigned SizeI = TRI->getRegSizeInBits(MI.getOperand(I).getReg(), MRI);
323 Operands[I] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, SizeI);
324 }
325
326 // getInstrMapping's default mapping uses ID 1, so start at 2.
327 unsigned MappingID = 2;
328 for (const auto &Entry : Table) {
329 for (unsigned I = 0; I < NumOps; ++I) {
330 int OpIdx = RegSrcOpIdx[I];
331 Operands[OpIdx] = AMDGPU::getValueMapping(Entry.RegBanks[I], Sizes[I]);
332 }
333
334 AltMappings.push_back(&getInstructionMapping(MappingID++, Entry.Cost,
336 Operands.size()));
337 }
338
339 return AltMappings;
340}
341
344 const MachineInstr &MI, const MachineRegisterInfo &MRI) const {
346 case Intrinsic::amdgcn_readlane: {
347 static const OpRegBankEntry<3> Table[2] = {
348 // Perfectly legal.
349 { { AMDGPU::SGPRRegBankID, AMDGPU::VGPRRegBankID, AMDGPU::SGPRRegBankID }, 1 },
350
351 // Need a readfirstlane for the index.
352 { { AMDGPU::SGPRRegBankID, AMDGPU::VGPRRegBankID, AMDGPU::VGPRRegBankID }, 2 }
353 };
354
355 const std::array<unsigned, 3> RegSrcOpIdx = { { 0, 2, 3 } };
356 return addMappingFromTable<3>(MI, MRI, RegSrcOpIdx, Table);
357 }
358 case Intrinsic::amdgcn_writelane: {
359 static const OpRegBankEntry<4> Table[4] = {
360 // Perfectly legal.
361 { { AMDGPU::VGPRRegBankID, AMDGPU::SGPRRegBankID, AMDGPU::SGPRRegBankID, AMDGPU::VGPRRegBankID }, 1 },
362
363 // Need readfirstlane of first op
364 { { AMDGPU::VGPRRegBankID, AMDGPU::VGPRRegBankID, AMDGPU::SGPRRegBankID, AMDGPU::VGPRRegBankID }, 2 },
365
366 // Need readfirstlane of second op
367 { { AMDGPU::VGPRRegBankID, AMDGPU::SGPRRegBankID, AMDGPU::VGPRRegBankID, AMDGPU::VGPRRegBankID }, 2 },
368
369 // Need readfirstlane of both ops
370 { { AMDGPU::VGPRRegBankID, AMDGPU::VGPRRegBankID, AMDGPU::VGPRRegBankID, AMDGPU::VGPRRegBankID }, 3 }
371 };
372
373 // rsrc, voffset, offset
374 const std::array<unsigned, 4> RegSrcOpIdx = { { 0, 2, 3, 4 } };
375 return addMappingFromTable<4>(MI, MRI, RegSrcOpIdx, Table);
376 }
377 default:
379 }
380}
381
384 const MachineInstr &MI, const MachineRegisterInfo &MRI) const {
385
387 case Intrinsic::amdgcn_s_buffer_load:
388 case Intrinsic::amdgcn_ptr_s_buffer_load: {
389 static const OpRegBankEntry<2> Table[4] = {
390 // Perfectly legal.
391 { { AMDGPU::SGPRRegBankID, AMDGPU::SGPRRegBankID }, 1 },
392
393 // Only need 1 register in loop
394 { { AMDGPU::SGPRRegBankID, AMDGPU::VGPRRegBankID }, 300 },
395
396 // Have to waterfall the resource.
397 { { AMDGPU::VGPRRegBankID, AMDGPU::SGPRRegBankID }, 1000 },
398
399 // Have to waterfall the resource, and the offset.
400 { { AMDGPU::VGPRRegBankID, AMDGPU::VGPRRegBankID }, 1500 }
401 };
402
403 // rsrc, offset
404 const std::array<unsigned, 2> RegSrcOpIdx = { { 2, 3 } };
405 return addMappingFromTable<2>(MI, MRI, RegSrcOpIdx, Table);
406 }
407 case Intrinsic::amdgcn_ds_ordered_add:
408 case Intrinsic::amdgcn_ds_ordered_swap: {
409 // VGPR = M0, VGPR
410 static const OpRegBankEntry<3> Table[2] = {
411 // Perfectly legal.
412 { { AMDGPU::VGPRRegBankID, AMDGPU::SGPRRegBankID, AMDGPU::VGPRRegBankID }, 1 },
413
414 // Need a readfirstlane for m0
415 { { AMDGPU::VGPRRegBankID, AMDGPU::VGPRRegBankID, AMDGPU::VGPRRegBankID }, 2 }
416 };
417
418 const std::array<unsigned, 3> RegSrcOpIdx = { { 0, 2, 3 } };
419 return addMappingFromTable<3>(MI, MRI, RegSrcOpIdx, Table);
420 }
421 case Intrinsic::amdgcn_s_sendmsg:
422 case Intrinsic::amdgcn_s_sendmsghalt: {
423 // FIXME: Should have no register for immediate
424 static const OpRegBankEntry<1> Table[2] = {
425 // Perfectly legal.
426 { { AMDGPU::SGPRRegBankID }, 1 },
427
428 // Need readlane
429 { { AMDGPU::VGPRRegBankID }, 3 }
430 };
431
432 const std::array<unsigned, 1> RegSrcOpIdx = { { 2 } };
433 return addMappingFromTable<1>(MI, MRI, RegSrcOpIdx, Table);
434 }
435 default:
437 }
438}
439
440// FIXME: Returns uniform if there's no source value information. This is
441// probably wrong.
443 if (!MI.hasOneMemOperand())
444 return false;
445
446 const MachineMemOperand *MMO = *MI.memoperands_begin();
447 const unsigned AS = MMO->getAddrSpace();
448 const bool IsConst = AS == AMDGPUAS::CONSTANT_ADDRESS ||
450 const unsigned MemSize = 8 * MMO->getSize().getValue();
451
452 // Require 4-byte alignment.
453 return (MMO->getAlign() >= Align(4) ||
454 (Subtarget.hasScalarSubwordLoads() &&
455 ((MemSize == 16 && MMO->getAlign() >= Align(2)) ||
456 (MemSize == 8 && MMO->getAlign() >= Align(1))))) &&
457 // Can't do a scalar atomic load.
458 !MMO->isAtomic() &&
459 // Don't use scalar loads for volatile accesses to non-constant address
460 // spaces.
461 (IsConst || !MMO->isVolatile()) &&
462 // Memory must be known constant, or not written before this load.
463 (IsConst || MMO->isInvariant() || (MMO->getFlags() & MONoClobber)) &&
465}
466
469 const MachineInstr &MI) const {
470
471 const MachineFunction &MF = *MI.getMF();
472 const MachineRegisterInfo &MRI = MF.getRegInfo();
473
474
475 InstructionMappings AltMappings;
476 switch (MI.getOpcode()) {
477 case TargetOpcode::G_CONSTANT:
478 case TargetOpcode::G_IMPLICIT_DEF: {
479 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
480 if (Size == 1) {
481 static const OpRegBankEntry<1> Table[3] = {
482 { { AMDGPU::VGPRRegBankID }, 1 },
483 { { AMDGPU::SGPRRegBankID }, 1 },
484 { { AMDGPU::VCCRegBankID }, 1 }
485 };
486
487 return addMappingFromTable<1>(MI, MRI, {{ 0 }}, Table);
488 }
489
490 [[fallthrough]];
491 }
492 case TargetOpcode::G_FCONSTANT:
493 case TargetOpcode::G_FRAME_INDEX:
494 case TargetOpcode::G_GLOBAL_VALUE: {
495 static const OpRegBankEntry<1> Table[2] = {
496 { { AMDGPU::VGPRRegBankID }, 1 },
497 { { AMDGPU::SGPRRegBankID }, 1 }
498 };
499
500 return addMappingFromTable<1>(MI, MRI, {{ 0 }}, Table);
501 }
502 case TargetOpcode::G_AND:
503 case TargetOpcode::G_OR:
504 case TargetOpcode::G_XOR: {
505 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
506
507 if (Size == 1) {
508 // s_{and|or|xor}_b32 set scc when the result of the 32-bit op is not 0.
509 const InstructionMapping &SCCMapping = getInstructionMapping(
510 1, 1, getOperandsMapping(
511 {AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 32),
512 AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 32),
513 AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 32)}),
514 3); // Num Operands
515 AltMappings.push_back(&SCCMapping);
516
517 const InstructionMapping &VCCMapping0 = getInstructionMapping(
518 2, 1, getOperandsMapping(
519 {AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, Size),
520 AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, Size),
521 AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, Size)}),
522 3); // Num Operands
523 AltMappings.push_back(&VCCMapping0);
524 return AltMappings;
525 }
526
527 if (Size != 64)
528 break;
529
530 const InstructionMapping &SSMapping = getInstructionMapping(
531 1, 1, getOperandsMapping(
532 {AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size),
533 AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size),
534 AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size)}),
535 3); // Num Operands
536 AltMappings.push_back(&SSMapping);
537
538 const InstructionMapping &VVMapping = getInstructionMapping(
539 2, 2, getOperandsMapping(
540 {AMDGPU::getValueMappingSGPR64Only(AMDGPU::VGPRRegBankID, Size),
541 AMDGPU::getValueMappingSGPR64Only(AMDGPU::VGPRRegBankID, Size),
542 AMDGPU::getValueMappingSGPR64Only(AMDGPU::VGPRRegBankID, Size)}),
543 3); // Num Operands
544 AltMappings.push_back(&VVMapping);
545 break;
546 }
547 case TargetOpcode::G_LOAD:
548 case TargetOpcode::G_ZEXTLOAD:
549 case TargetOpcode::G_SEXTLOAD: {
550 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
551 LLT PtrTy = MRI.getType(MI.getOperand(1).getReg());
552 unsigned PtrSize = PtrTy.getSizeInBits();
553 unsigned AS = PtrTy.getAddressSpace();
554
558 const InstructionMapping &SSMapping = getInstructionMapping(
559 1, 1, getOperandsMapping(
560 {AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size),
561 AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, PtrSize)}),
562 2); // Num Operands
563 AltMappings.push_back(&SSMapping);
564 }
565
566 const InstructionMapping &VVMapping = getInstructionMapping(
567 2, 1,
569 {AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size),
570 AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, PtrSize)}),
571 2); // Num Operands
572 AltMappings.push_back(&VVMapping);
573
574 // It may be possible to have a vgpr = load sgpr mapping here, because
575 // the mubuf instructions support this kind of load, but probably for only
576 // gfx7 and older. However, the addressing mode matching in the instruction
577 // selector should be able to do a better job of detecting and selecting
578 // these kinds of loads from the vgpr = load vgpr mapping.
579
580 return AltMappings;
581
582 }
583 case TargetOpcode::G_SELECT: {
584 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
585 const InstructionMapping &SSMapping = getInstructionMapping(1, 1,
586 getOperandsMapping({AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size),
587 AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 1),
588 AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size),
589 AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size)}),
590 4); // Num Operands
591 AltMappings.push_back(&SSMapping);
592
593 const InstructionMapping &VVMapping = getInstructionMapping(2, 1,
594 getOperandsMapping({AMDGPU::getValueMappingSGPR64Only(AMDGPU::VGPRRegBankID, Size),
595 AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, 1),
596 AMDGPU::getValueMappingSGPR64Only(AMDGPU::VGPRRegBankID, Size),
597 AMDGPU::getValueMappingSGPR64Only(AMDGPU::VGPRRegBankID, Size)}),
598 4); // Num Operands
599 AltMappings.push_back(&VVMapping);
600
601 return AltMappings;
602 }
603 case TargetOpcode::G_UADDE:
604 case TargetOpcode::G_USUBE:
605 case TargetOpcode::G_SADDE:
606 case TargetOpcode::G_SSUBE: {
607 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
608 const InstructionMapping &SSMapping = getInstructionMapping(1, 1,
610 {AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size),
611 AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 1),
612 AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size),
613 AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size),
614 AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 1)}),
615 5); // Num Operands
616 AltMappings.push_back(&SSMapping);
617
618 const InstructionMapping &VVMapping = getInstructionMapping(2, 1,
619 getOperandsMapping({AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size),
620 AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, 1),
621 AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size),
622 AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size),
623 AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, 1)}),
624 5); // Num Operands
625 AltMappings.push_back(&VVMapping);
626 return AltMappings;
627 }
628 case AMDGPU::G_BRCOND: {
629 assert(MRI.getType(MI.getOperand(0).getReg()).getSizeInBits() == 1);
630
631 // TODO: Change type to 32 for scalar
633 1, 1, getOperandsMapping(
634 {AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 1), nullptr}),
635 2); // Num Operands
636 AltMappings.push_back(&SMapping);
637
639 1, 1, getOperandsMapping(
640 {AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, 1), nullptr }),
641 2); // Num Operands
642 AltMappings.push_back(&VMapping);
643 return AltMappings;
644 }
645 case AMDGPU::G_INTRINSIC:
646 case AMDGPU::G_INTRINSIC_CONVERGENT:
648 case AMDGPU::G_INTRINSIC_W_SIDE_EFFECTS:
649 case AMDGPU::G_INTRINSIC_CONVERGENT_W_SIDE_EFFECTS:
651 default:
652 break;
653 }
655}
656
660 LLT HalfTy,
661 Register Reg) const {
662 assert(HalfTy.getSizeInBits() == 32);
663 MachineRegisterInfo *MRI = B.getMRI();
664 Register LoLHS = MRI->createGenericVirtualRegister(HalfTy);
665 Register HiLHS = MRI->createGenericVirtualRegister(HalfTy);
666 const RegisterBank *Bank = getRegBank(Reg, *MRI, *TRI);
667 MRI->setRegBank(LoLHS, *Bank);
668 MRI->setRegBank(HiLHS, *Bank);
669
670 Regs.push_back(LoLHS);
671 Regs.push_back(HiLHS);
672
673 B.buildInstr(AMDGPU::G_UNMERGE_VALUES)
674 .addDef(LoLHS)
675 .addDef(HiLHS)
676 .addUse(Reg);
677}
678
679/// Replace the current type each register in \p Regs has with \p NewTy
681 LLT NewTy) {
682 for (Register Reg : Regs) {
683 assert(MRI.getType(Reg).getSizeInBits() == NewTy.getSizeInBits());
684 MRI.setType(Reg, NewTy);
685 }
686}
687
689 if (Ty.isVector()) {
690 assert(Ty.getElementCount().isKnownMultipleOf(2));
691 return LLT::scalarOrVector(Ty.getElementCount().divideCoefficientBy(2),
692 Ty.getElementType());
693 }
694
695 assert(Ty.getScalarSizeInBits() % 2 == 0);
696 return LLT::scalar(Ty.getScalarSizeInBits() / 2);
697}
698
699// Build one or more V_READFIRSTLANE_B32 instructions to move the given vector
700// source value into a scalar register.
703 Register Src) const {
704 LLT Ty = MRI.getType(Src);
705 const RegisterBank *Bank = getRegBank(Src, MRI, *TRI);
706
707 if (Bank == &AMDGPU::SGPRRegBank)
708 return Src;
709
710 unsigned Bits = Ty.getSizeInBits();
711 assert(Bits % 32 == 0);
712
713 if (Bank != &AMDGPU::VGPRRegBank) {
714 // We need to copy from AGPR to VGPR
715 Src = B.buildCopy(Ty, Src).getReg(0);
716 MRI.setRegBank(Src, AMDGPU::VGPRRegBank);
717 }
718
719 LLT S32 = LLT::scalar(32);
720 unsigned NumParts = Bits / 32;
723
724 if (Bits == 32) {
725 SrcParts.push_back(Src);
726 } else {
727 auto Unmerge = B.buildUnmerge(S32, Src);
728 for (unsigned i = 0; i < NumParts; ++i)
729 SrcParts.push_back(Unmerge.getReg(i));
730 }
731
732 for (unsigned i = 0; i < NumParts; ++i) {
733 Register SrcPart = SrcParts[i];
734 Register DstPart = MRI.createVirtualRegister(&AMDGPU::SReg_32_XM0RegClass);
735 MRI.setType(DstPart, NumParts == 1 ? Ty : S32);
736
737 const TargetRegisterClass *Constrained =
738 constrainGenericRegister(SrcPart, AMDGPU::VGPR_32RegClass, MRI);
739 (void)Constrained;
740 assert(Constrained && "Failed to constrain readfirstlane src reg");
741
742 B.buildInstr(AMDGPU::V_READFIRSTLANE_B32, {DstPart}, {SrcPart});
743
744 DstParts.push_back(DstPart);
745 }
746
747 if (Bits == 32)
748 return DstParts[0];
749
750 Register Dst = B.buildMergeLikeInstr(Ty, DstParts).getReg(0);
751 MRI.setRegBank(Dst, AMDGPU::SGPRRegBank);
752 return Dst;
753}
754
755/// Legalize instruction \p MI where operands in \p OpIndices must be SGPRs. If
756/// any of the required SGPR operands are VGPRs, perform a waterfall loop to
757/// execute the instruction for each unique combination of values in all lanes
758/// in the wave. The block will be split such that rest of the instructions are
759/// moved to a new block.
760///
761/// Essentially performs this loop:
762//
763/// Save Execution Mask
764/// For (Lane : Wavefront) {
765/// Enable Lane, Disable all other lanes
766/// SGPR = read SGPR value for current lane from VGPR
767/// VGPRResult[Lane] = use_op SGPR
768/// }
769/// Restore Execution Mask
770///
771/// There is additional complexity to try for compare values to identify the
772/// unique values used.
775 SmallSet<Register, 4> &SGPROperandRegs) const {
776 // Track use registers which have already been expanded with a readfirstlane
777 // sequence. This may have multiple uses if moving a sequence.
778 DenseMap<Register, Register> WaterfalledRegMap;
779
780 MachineBasicBlock &MBB = B.getMBB();
781 MachineFunction *MF = &B.getMF();
782
783 const TargetRegisterClass *WaveRC = TRI->getWaveMaskRegClass();
784 const AMDGPU::LaneMaskConstants &LMC =
786
787#ifndef NDEBUG
788 const int OrigRangeSize = std::distance(Range.begin(), Range.end());
789#endif
790
791 MachineRegisterInfo &MRI = *B.getMRI();
792 Register SaveExecReg = MRI.createVirtualRegister(WaveRC);
793 Register InitSaveExecReg = MRI.createVirtualRegister(WaveRC);
794
795 // Don't bother using generic instructions/registers for the exec mask.
796 B.buildInstr(TargetOpcode::IMPLICIT_DEF)
797 .addDef(InitSaveExecReg);
798
799 Register PhiExec = MRI.createVirtualRegister(WaveRC);
800 Register NewExec = MRI.createVirtualRegister(WaveRC);
801
802 // To insert the loop we need to split the block. Move everything before this
803 // point to a new block, and insert a new empty block before this instruction.
806 MachineBasicBlock *RemainderBB = MF->CreateMachineBasicBlock();
807 MachineBasicBlock *RestoreExecBB = MF->CreateMachineBasicBlock();
809 ++MBBI;
810 MF->insert(MBBI, LoopBB);
811 MF->insert(MBBI, BodyBB);
812 MF->insert(MBBI, RestoreExecBB);
813 MF->insert(MBBI, RemainderBB);
814
815 LoopBB->addSuccessor(BodyBB);
816 BodyBB->addSuccessor(RestoreExecBB);
817 BodyBB->addSuccessor(LoopBB);
818
819 // Move the rest of the block into a new block.
821 RemainderBB->splice(RemainderBB->begin(), &MBB, Range.end(), MBB.end());
822
823 MBB.addSuccessor(LoopBB);
824 RestoreExecBB->addSuccessor(RemainderBB);
825
826 B.setInsertPt(*LoopBB, LoopBB->end());
827
828 B.buildInstr(TargetOpcode::PHI)
829 .addDef(PhiExec)
830 .addReg(InitSaveExecReg)
831 .addMBB(&MBB)
832 .addReg(NewExec)
833 .addMBB(BodyBB);
834
835 const DebugLoc &DL = B.getDL();
836
837 MachineInstr &FirstInst = *Range.begin();
838
839 // Move the instruction into the loop body. Note we moved everything after
840 // Range.end() already into a new block, so Range.end() is no longer valid.
841 BodyBB->splice(BodyBB->end(), &MBB, Range.begin(), MBB.end());
842
843 // Figure out the iterator range after splicing the instructions.
844 MachineBasicBlock::iterator NewBegin = FirstInst.getIterator();
845 auto NewEnd = BodyBB->end();
846
847 B.setMBB(*LoopBB);
848
849 LLT S1 = LLT::scalar(1);
850 Register CondReg;
851
852 assert(std::distance(NewBegin, NewEnd) == OrigRangeSize);
853
854 for (MachineInstr &MI : make_range(NewBegin, NewEnd)) {
855 for (MachineOperand &Op : MI.all_uses()) {
856 Register OldReg = Op.getReg();
857 if (!SGPROperandRegs.count(OldReg))
858 continue;
859
860 // See if we already processed this register in another instruction in the
861 // sequence.
862 auto OldVal = WaterfalledRegMap.find(OldReg);
863 if (OldVal != WaterfalledRegMap.end()) {
864 Op.setReg(OldVal->second);
865 continue;
866 }
867
868 Register OpReg = Op.getReg();
869 LLT OpTy = MRI.getType(OpReg);
870
871 const RegisterBank *OpBank = getRegBank(OpReg, MRI, *TRI);
872 if (OpBank != &AMDGPU::VGPRRegBank) {
873 // Insert copy from AGPR to VGPR before the loop.
874 B.setMBB(MBB);
875 OpReg = B.buildCopy(OpTy, OpReg).getReg(0);
876 MRI.setRegBank(OpReg, AMDGPU::VGPRRegBank);
877 B.setMBB(*LoopBB);
878 }
879
880 Register CurrentLaneReg = buildReadFirstLane(B, MRI, OpReg);
881
882 // Build the comparison(s).
883 unsigned OpSize = OpTy.getSizeInBits();
884 bool Is64 = OpSize % 64 == 0;
885 unsigned PartSize = Is64 ? 64 : 32;
886 LLT PartTy = LLT::scalar(PartSize);
887 unsigned NumParts = OpSize / PartSize;
889 SmallVector<Register, 8> CurrentLaneParts;
890
891 if (NumParts == 1) {
892 OpParts.push_back(OpReg);
893 CurrentLaneParts.push_back(CurrentLaneReg);
894 } else {
895 auto UnmergeOp = B.buildUnmerge(PartTy, OpReg);
896 auto UnmergeCurrentLane = B.buildUnmerge(PartTy, CurrentLaneReg);
897 for (unsigned i = 0; i < NumParts; ++i) {
898 OpParts.push_back(UnmergeOp.getReg(i));
899 CurrentLaneParts.push_back(UnmergeCurrentLane.getReg(i));
900 MRI.setRegBank(OpParts[i], AMDGPU::VGPRRegBank);
901 MRI.setRegBank(CurrentLaneParts[i], AMDGPU::SGPRRegBank);
902 }
903 }
904
905 for (unsigned i = 0; i < NumParts; ++i) {
906 auto CmpReg = B.buildICmp(CmpInst::ICMP_EQ, S1, CurrentLaneParts[i],
907 OpParts[i]).getReg(0);
908 MRI.setRegBank(CmpReg, AMDGPU::VCCRegBank);
909
910 if (!CondReg) {
911 CondReg = CmpReg;
912 } else {
913 CondReg = B.buildAnd(S1, CondReg, CmpReg).getReg(0);
914 MRI.setRegBank(CondReg, AMDGPU::VCCRegBank);
915 }
916 }
917
918 Op.setReg(CurrentLaneReg);
919
920 // Make sure we don't re-process this register again.
921 WaterfalledRegMap.insert(std::pair(OldReg, Op.getReg()));
922 }
923 }
924
925 // The ballot becomes a no-op during instruction selection.
926 CondReg = B.buildIntrinsic(Intrinsic::amdgcn_ballot,
927 {LLT::scalar(Subtarget.isWave32() ? 32 : 64)})
928 .addReg(CondReg)
929 .getReg(0);
930 MRI.setRegClass(CondReg, WaveRC);
931
932 // Update EXEC, save the original EXEC value to VCC.
933 B.buildInstr(LMC.AndSaveExecOpc)
934 .addDef(NewExec)
935 .addReg(CondReg)
936 .setOperandDead(3);
937
938 MRI.setSimpleHint(NewExec, CondReg);
939
940 B.setInsertPt(*BodyBB, BodyBB->end());
941
942 // Update EXEC, switch all done bits to 0 and all todo bits to 1.
943 B.buildInstr(LMC.XorTermOpc)
944 .addDef(LMC.ExecReg)
945 .addReg(LMC.ExecReg)
946 .addReg(NewExec)
947 .setOperandDead(3);
948
949 // XXX - s_xor_b64 sets scc to 1 if the result is nonzero, so can we use
950 // s_cbranch_scc0?
951
952 // Loop back to V_READFIRSTLANE_B32 if there are still variants to cover.
953 B.buildInstr(AMDGPU::SI_WATERFALL_LOOP).addMBB(LoopBB);
954
955 // Save the EXEC mask before the loop.
956 BuildMI(MBB, MBB.end(), DL, TII->get(LMC.MovOpc), SaveExecReg)
957 .addReg(LMC.ExecReg);
958
959 // Restore the EXEC mask after the loop.
960 B.setMBB(*RestoreExecBB);
961 B.buildInstr(LMC.MovTermOpc).addDef(LMC.ExecReg).addReg(SaveExecReg);
962
963 // Set the insert point after the original instruction, so any new
964 // instructions will be in the remainder.
965 B.setInsertPt(*RemainderBB, RemainderBB->begin());
966
967 return true;
968}
969
970// Return any unique registers used by \p MI at \p OpIndices that need to be
971// handled in a waterfall loop. Returns these registers in \p
972// SGPROperandRegs. Returns true if there are any operands to handle and a
973// waterfall loop is necessary.
975 SmallSet<Register, 4> &SGPROperandRegs, MachineInstr &MI,
976 MachineRegisterInfo &MRI, ArrayRef<unsigned> OpIndices) const {
977 for (unsigned Op : OpIndices) {
978 assert(MI.getOperand(Op).isUse());
979 Register Reg = MI.getOperand(Op).getReg();
980 const RegisterBank *OpBank = getRegBank(Reg, MRI, *TRI);
981 if (OpBank->getID() != AMDGPU::SGPRRegBankID)
982 SGPROperandRegs.insert(Reg);
983 }
984
985 // No operands need to be replaced, so no need to loop.
986 return !SGPROperandRegs.empty();
987}
988
991 // Use a set to avoid extra readfirstlanes in the case where multiple operands
992 // are the same register.
993 SmallSet<Register, 4> SGPROperandRegs;
994
995 if (!collectWaterfallOperands(SGPROperandRegs, MI, *B.getMRI(), OpIndices))
996 return false;
997
998 MachineBasicBlock::iterator I = MI.getIterator();
999 return executeInWaterfallLoop(B, make_range(I, std::next(I)),
1000 SGPROperandRegs);
1001}
1002
1003// Legalize an operand that must be an SGPR by inserting a readfirstlane.
1005 MachineIRBuilder &B, MachineInstr &MI, unsigned OpIdx) const {
1006 Register Reg = MI.getOperand(OpIdx).getReg();
1007 MachineRegisterInfo &MRI = *B.getMRI();
1008 const RegisterBank *Bank = getRegBank(Reg, MRI, *TRI);
1009 if (Bank == &AMDGPU::SGPRRegBank)
1010 return;
1011
1012 Reg = buildReadFirstLane(B, MRI, Reg);
1013 MI.getOperand(OpIdx).setReg(Reg);
1014}
1015
1016/// Split \p Ty into 2 pieces. The first will have \p FirstSize bits, and the
1017/// rest will be in the remainder.
1018static std::pair<LLT, LLT> splitUnequalType(LLT Ty, unsigned FirstSize) {
1019 unsigned TotalSize = Ty.getSizeInBits();
1020 if (!Ty.isVector())
1021 return {LLT::scalar(FirstSize), LLT::scalar(TotalSize - FirstSize)};
1022
1023 LLT EltTy = Ty.getElementType();
1024 unsigned EltSize = EltTy.getSizeInBits();
1025 assert(FirstSize % EltSize == 0);
1026
1027 unsigned FirstPartNumElts = FirstSize / EltSize;
1028 unsigned RemainderElts = (TotalSize - FirstSize) / EltSize;
1029
1030 return {LLT::scalarOrVector(ElementCount::getFixed(FirstPartNumElts), EltTy),
1031 LLT::scalarOrVector(ElementCount::getFixed(RemainderElts), EltTy)};
1032}
1033
1035 if (!Ty.isVector())
1036 return LLT::scalar(128);
1037
1038 LLT EltTy = Ty.getElementType();
1039 assert(128 % EltTy.getSizeInBits() == 0);
1040 return LLT::fixed_vector(128 / EltTy.getSizeInBits(), EltTy);
1041}
1042
1046 MachineInstr &MI) const {
1047 MachineRegisterInfo &MRI = *B.getMRI();
1048 Register DstReg = MI.getOperand(0).getReg();
1049 const LLT LoadTy = MRI.getType(DstReg);
1050 unsigned LoadSize = LoadTy.getSizeInBits();
1051 MachineMemOperand *MMO = *MI.memoperands_begin();
1052 const unsigned MaxNonSmrdLoadSize = 128;
1053
1054 const RegisterBank *DstBank =
1055 OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
1056 if (DstBank == &AMDGPU::SGPRRegBank) {
1057 // There are some special cases that we need to look at for 32 bit and 96
1058 // bit SGPR loads otherwise we have nothing to do.
1059 if (LoadSize != 32 && (LoadSize != 96 || Subtarget.hasScalarDwordx3Loads()))
1060 return false;
1061
1062 const unsigned MemSize = 8 * MMO->getSize().getValue();
1063 // Scalar loads of size 8 or 16 bit with proper alignment may be widened to
1064 // 32 bit. Check to see if we need to widen the memory access, 8 or 16 bit
1065 // scalar loads should have a load size of 32 but memory access size of less
1066 // than 32.
1067 if (LoadSize == 32 &&
1068 (MemSize == 32 || LoadTy.isVector() || !isScalarLoadLegal(MI)))
1069 return false;
1070
1071 if (LoadSize == 32 &&
1072 ((MemSize == 8 && MMO->getAlign() >= Align(1)) ||
1073 (MemSize == 16 && MMO->getAlign() >= Align(2))) &&
1075 Subtarget.getGeneration() >= AMDGPUSubtarget::GFX12)
1076 return false;
1077
1078 Register PtrReg = MI.getOperand(1).getReg();
1079
1080 ApplyRegBankMapping ApplyBank(B, *this, MRI, DstBank);
1081
1082 if (LoadSize == 32) {
1083 // This is an extending load from a sub-dword size. Widen the memory
1084 // access size to 4 bytes and clear the extra high bits appropriately
1085 const LLT S32 = LLT::scalar(32);
1086 if (MI.getOpcode() == AMDGPU::G_SEXTLOAD) {
1087 // Must extend the sign bit into higher bits for a G_SEXTLOAD
1088 auto WideLoad = B.buildLoadFromOffset(S32, PtrReg, *MMO, 0);
1089 B.buildSExtInReg(MI.getOperand(0), WideLoad, MemSize);
1090 } else if (MI.getOpcode() == AMDGPU::G_ZEXTLOAD) {
1091 // Must extend zero into higher bits with an AND for a G_ZEXTLOAD
1092 auto WideLoad = B.buildLoadFromOffset(S32, PtrReg, *MMO, 0);
1093 B.buildZExtInReg(MI.getOperand(0), WideLoad, MemSize);
1094 } else
1095 // We do not need to touch the higher bits for regular loads.
1096 B.buildLoadFromOffset(MI.getOperand(0), PtrReg, *MMO, 0);
1097 } else {
1098 // 96-bit loads are only available for vector loads. We need to split this
1099 // into a 64-bit part, and 32 (unless we can widen to a 128-bit load).
1100 if (MMO->getAlign() < Align(16)) {
1101 LegalizerHelper Helper(B.getMF(), ApplyBank, B);
1102 LLT Part64, Part32;
1103 std::tie(Part64, Part32) = splitUnequalType(LoadTy, 64);
1104 if (Helper.reduceLoadStoreWidth(cast<GAnyLoad>(MI), 0, Part64) !=
1106 return false;
1107 return true;
1108 }
1109 LLT WiderTy = widen96To128(LoadTy);
1110 auto WideLoad = B.buildLoadFromOffset(WiderTy, PtrReg, *MMO, 0);
1111 if (WiderTy.isScalar()) {
1112 B.buildTrunc(MI.getOperand(0), WideLoad);
1113 } else {
1114 B.buildDeleteTrailingVectorElements(MI.getOperand(0).getReg(),
1115 WideLoad);
1116 }
1117 }
1118
1119 MI.eraseFromParent();
1120 return true;
1121 }
1122
1123 // 128-bit loads are supported for all instruction types.
1124 if (LoadSize <= MaxNonSmrdLoadSize)
1125 return false;
1126
1127 SmallVector<Register, 1> SrcRegs(OpdMapper.getVRegs(1));
1128
1129 if (SrcRegs.empty())
1130 SrcRegs.push_back(MI.getOperand(1).getReg());
1131
1132 // RegBankSelect only emits scalar types, so we need to reset the pointer
1133 // operand to a pointer type.
1134 Register BasePtrReg = SrcRegs[0];
1135 LLT PtrTy = MRI.getType(MI.getOperand(1).getReg());
1136 MRI.setType(BasePtrReg, PtrTy);
1137
1138 // The following are the loads not splitted enough during legalization
1139 // because it was not clear they are smem-load or vmem-load
1142 assert(LoadSize % MaxNonSmrdLoadSize == 0);
1143 unsigned NumSplitParts = LoadTy.getSizeInBits() / MaxNonSmrdLoadSize;
1144 const LLT LoadSplitTy = LoadTy.divide(NumSplitParts);
1145 ApplyRegBankMapping O(B, *this, MRI, &AMDGPU::VGPRRegBank);
1146 LegalizerHelper Helper(B.getMF(), O, B);
1147 if (LoadTy.isVector()) {
1148 if (Helper.fewerElementsVector(MI, 0, LoadSplitTy) !=
1150 return false;
1151 } else {
1152 if (Helper.narrowScalar(MI, 0, LoadSplitTy) != LegalizerHelper::Legalized)
1153 return false;
1154 }
1155 }
1156
1157 MRI.setRegBank(DstReg, AMDGPU::VGPRRegBank);
1158 return true;
1159}
1160
1164 MachineInstr &MI) const {
1165 MachineRegisterInfo &MRI = *B.getMRI();
1166 const MachineFunction &MF = B.getMF();
1167 const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
1168 const auto &TFI = *ST.getFrameLowering();
1169
1170 // Guard in case the stack growth direction ever changes with scratch
1171 // instructions.
1172 assert(TFI.getStackGrowthDirection() == TargetFrameLowering::StackGrowsUp &&
1173 "Stack grows upwards for AMDGPU");
1174
1175 Register Dst = MI.getOperand(0).getReg();
1176 Register AllocSize = MI.getOperand(1).getReg();
1177 Align Alignment = assumeAligned(MI.getOperand(2).getImm());
1178
1179 // When using flat-scratch, the stack offset is unscaled.
1180 const bool HasFlatScratch = ST.hasFlatScratchEnabled();
1181 const unsigned WavefrontSizeLog2 = ST.getWavefrontSizeLog2();
1182
1183 const RegisterBank *SizeBank = getRegBank(AllocSize, MRI, *TRI);
1184
1185 if (SizeBank != &AMDGPU::SGPRRegBank) {
1186 auto WaveReduction =
1187 B.buildIntrinsic(Intrinsic::amdgcn_wave_reduce_umax, {LLT::scalar(32)})
1188 .addUse(AllocSize)
1189 .addImm(0);
1190 AllocSize = WaveReduction.getReg(0);
1191 }
1192
1193 LLT PtrTy = MRI.getType(Dst);
1195
1197 Register SPReg = Info->getStackPtrOffsetReg();
1198 ApplyRegBankMapping ApplyBank(B, *this, MRI, &AMDGPU::SGPRRegBank);
1199
1200 Register ScaledSize = AllocSize;
1201 if (!HasFlatScratch) {
1202 auto WaveSize = B.buildConstant(LLT::scalar(32), WavefrontSizeLog2);
1203 ScaledSize = B.buildShl(IntPtrTy, AllocSize, WaveSize).getReg(0);
1204 }
1205
1206 auto OldSP = B.buildCopy(PtrTy, SPReg);
1207 if (Alignment > TFI.getStackAlign()) {
1208 const uint64_t ScaledAlignment =
1209 HasFlatScratch ? Alignment.value()
1210 : (Alignment.value() << WavefrontSizeLog2);
1211 const uint64_t StackAlignMask = ScaledAlignment - 1;
1212 auto Tmp1 = B.buildPtrAdd(PtrTy, OldSP,
1213 B.buildConstant(LLT::scalar(32), StackAlignMask));
1214 B.buildMaskLowPtrBits(Dst, Tmp1,
1215 (HasFlatScratch
1216 ? Log2(Alignment)
1217 : Log2(Alignment) + WavefrontSizeLog2));
1218 } else {
1219 B.buildCopy(Dst, OldSP);
1220 }
1221 auto PtrAdd = B.buildPtrAdd(PtrTy, Dst, ScaledSize);
1222 B.buildCopy(SPReg, PtrAdd);
1223 MI.eraseFromParent();
1224 return true;
1225}
1226
1230 int RsrcIdx) const {
1231 const int NumDefs = MI.getNumExplicitDefs();
1232
1233 // The reported argument index is relative to the IR intrinsic call arguments,
1234 // so we need to shift by the number of defs and the intrinsic ID.
1235 RsrcIdx += NumDefs + 1;
1236
1237 // Insert copies to VGPR arguments.
1238 applyDefaultMapping(OpdMapper);
1239
1240 // Fixup any SGPR arguments.
1241 SmallVector<unsigned, 4> SGPRIndexes;
1242 for (int I = NumDefs, NumOps = MI.getNumOperands(); I != NumOps; ++I) {
1243 if (!MI.getOperand(I).isReg())
1244 continue;
1245
1246 // If this intrinsic has a sampler, it immediately follows rsrc.
1247 if (I == RsrcIdx || I == RsrcIdx + 1)
1248 SGPRIndexes.push_back(I);
1249 }
1250
1251 executeInWaterfallLoop(B, MI, SGPRIndexes);
1252 return true;
1253}
1254
1255// Analyze a combined offset from an llvm.amdgcn.s.buffer intrinsic and store
1256// the three offsets (voffset, soffset and instoffset)
1258 MachineIRBuilder &B, Register CombinedOffset, Register &VOffsetReg,
1259 Register &SOffsetReg, int64_t &InstOffsetVal, Align Alignment) const {
1260 const LLT S32 = LLT::scalar(32);
1261 MachineRegisterInfo *MRI = B.getMRI();
1262
1263 if (std::optional<int64_t> Imm =
1264 getIConstantVRegSExtVal(CombinedOffset, *MRI)) {
1265 uint32_t SOffset, ImmOffset;
1266 if (TII->splitMUBUFOffset(*Imm, SOffset, ImmOffset, Alignment)) {
1267 VOffsetReg = B.buildConstant(S32, 0).getReg(0);
1268 SOffsetReg = B.buildConstant(S32, SOffset).getReg(0);
1269 InstOffsetVal = ImmOffset;
1270
1271 B.getMRI()->setRegBank(VOffsetReg, AMDGPU::VGPRRegBank);
1272 B.getMRI()->setRegBank(SOffsetReg, AMDGPU::SGPRRegBank);
1273 return SOffset + ImmOffset;
1274 }
1275 }
1276
1277 const bool CheckNUW = Subtarget.hasGFX1250Insts();
1278 Register Base;
1279 unsigned Offset;
1280
1281 std::tie(Base, Offset) =
1282 AMDGPU::getBaseWithConstantOffset(*MRI, CombinedOffset,
1283 /*KnownBits=*/nullptr,
1284 /*CheckNUW=*/CheckNUW);
1285
1286 uint32_t SOffset, ImmOffset;
1287 if (static_cast<int32_t>(Offset) > 0 &&
1288 TII->splitMUBUFOffset(Offset, SOffset, ImmOffset, Alignment)) {
1289 if (getRegBank(Base, *MRI, *TRI) == &AMDGPU::VGPRRegBank) {
1290 VOffsetReg = Base;
1291 SOffsetReg = B.buildConstant(S32, SOffset).getReg(0);
1292 B.getMRI()->setRegBank(SOffsetReg, AMDGPU::SGPRRegBank);
1293 InstOffsetVal = ImmOffset;
1294 return 0; // XXX - Why is this 0?
1295 }
1296
1297 // If we have SGPR base, we can use it for soffset.
1298 if (SOffset == 0) {
1299 VOffsetReg = B.buildConstant(S32, 0).getReg(0);
1300 B.getMRI()->setRegBank(VOffsetReg, AMDGPU::VGPRRegBank);
1301 SOffsetReg = Base;
1302 InstOffsetVal = ImmOffset;
1303 return 0; // XXX - Why is this 0?
1304 }
1305 }
1306
1307 // Handle the variable sgpr + vgpr case.
1308 MachineInstr *Add = getOpcodeDef(AMDGPU::G_ADD, CombinedOffset, *MRI);
1309 if (Add && static_cast<int32_t>(Offset) >= 0 &&
1310 (!CheckNUW || Add->getFlag(MachineInstr::NoUWrap))) {
1311 Register Src0 = getSrcRegIgnoringCopies(Add->getOperand(1).getReg(), *MRI);
1312 Register Src1 = getSrcRegIgnoringCopies(Add->getOperand(2).getReg(), *MRI);
1313
1314 const RegisterBank *Src0Bank = getRegBank(Src0, *MRI, *TRI);
1315 const RegisterBank *Src1Bank = getRegBank(Src1, *MRI, *TRI);
1316
1317 if (Src0Bank == &AMDGPU::VGPRRegBank && Src1Bank == &AMDGPU::SGPRRegBank) {
1318 VOffsetReg = Src0;
1319 SOffsetReg = Src1;
1320 return 0;
1321 }
1322
1323 if (Src0Bank == &AMDGPU::SGPRRegBank && Src1Bank == &AMDGPU::VGPRRegBank) {
1324 VOffsetReg = Src1;
1325 SOffsetReg = Src0;
1326 return 0;
1327 }
1328 }
1329
1330 // Ensure we have a VGPR for the combined offset. This could be an issue if we
1331 // have an SGPR offset and a VGPR resource.
1332 if (getRegBank(CombinedOffset, *MRI, *TRI) == &AMDGPU::VGPRRegBank) {
1333 VOffsetReg = CombinedOffset;
1334 } else {
1335 VOffsetReg = B.buildCopy(S32, CombinedOffset).getReg(0);
1336 B.getMRI()->setRegBank(VOffsetReg, AMDGPU::VGPRRegBank);
1337 }
1338
1339 SOffsetReg = B.buildConstant(S32, 0).getReg(0);
1340 B.getMRI()->setRegBank(SOffsetReg, AMDGPU::SGPRRegBank);
1341 return 0;
1342}
1343
1345 switch (Opc) {
1346 case AMDGPU::G_AMDGPU_S_BUFFER_LOAD:
1347 return AMDGPU::G_AMDGPU_BUFFER_LOAD;
1348 case AMDGPU::G_AMDGPU_S_BUFFER_LOAD_UBYTE:
1349 return AMDGPU::G_AMDGPU_BUFFER_LOAD_UBYTE;
1350 case AMDGPU::G_AMDGPU_S_BUFFER_LOAD_SBYTE:
1351 return AMDGPU::G_AMDGPU_BUFFER_LOAD_SBYTE;
1352 case AMDGPU::G_AMDGPU_S_BUFFER_LOAD_USHORT:
1353 return AMDGPU::G_AMDGPU_BUFFER_LOAD_USHORT;
1354 case AMDGPU::G_AMDGPU_S_BUFFER_LOAD_SSHORT:
1355 return AMDGPU::G_AMDGPU_BUFFER_LOAD_SSHORT;
1356 default:
1357 break;
1358 }
1359 llvm_unreachable("Unexpected s_buffer_load opcode");
1360}
1361
1363 MachineIRBuilder &B, const OperandsMapper &OpdMapper) const {
1364 MachineInstr &MI = OpdMapper.getMI();
1365 MachineRegisterInfo &MRI = OpdMapper.getMRI();
1366
1367 const LLT S32 = LLT::scalar(32);
1368 Register Dst = MI.getOperand(0).getReg();
1369 LLT Ty = MRI.getType(Dst);
1370
1371 const RegisterBank *RSrcBank =
1372 OpdMapper.getInstrMapping().getOperandMapping(1).BreakDown[0].RegBank;
1373 const RegisterBank *OffsetBank =
1374 OpdMapper.getInstrMapping().getOperandMapping(2).BreakDown[0].RegBank;
1375 if (RSrcBank == &AMDGPU::SGPRRegBank &&
1376 OffsetBank == &AMDGPU::SGPRRegBank)
1377 return true; // Legal mapping
1378
1379 // FIXME: 96-bit case was widened during legalize. We need to narrow it back
1380 // here but don't have an MMO.
1381
1382 unsigned LoadSize = Ty.getSizeInBits();
1383 int NumLoads = 1;
1384 if (LoadSize == 256 || LoadSize == 512) {
1385 NumLoads = LoadSize / 128;
1386 Ty = Ty.divide(NumLoads);
1387 }
1388
1389 // Use the alignment to ensure that the required offsets will fit into the
1390 // immediate offsets.
1391 const Align Alignment = NumLoads > 1 ? Align(16 * NumLoads) : Align(1);
1392
1393 MachineFunction &MF = B.getMF();
1394
1395 Register SOffset;
1396 Register VOffset;
1397 int64_t ImmOffset = 0;
1398
1399 unsigned MMOOffset = setBufferOffsets(B, MI.getOperand(2).getReg(), VOffset,
1400 SOffset, ImmOffset, Alignment);
1401
1402 // TODO: 96-bit loads were widened to 128-bit results. Shrink the result if we
1403 // can, but we need to track an MMO for that.
1404 const unsigned MemSize = (Ty.getSizeInBits() + 7) / 8;
1405 const Align MemAlign(4); // FIXME: ABI type alignment?
1410 MemSize, MemAlign);
1411 if (MMOOffset != 0)
1412 BaseMMO = MF.getMachineMemOperand(BaseMMO, MMOOffset, MemSize);
1413
1414 // If only the offset is divergent, emit a MUBUF buffer load instead. We can
1415 // assume that the buffer is unswizzled.
1416
1417 Register RSrc = MI.getOperand(1).getReg();
1418 Register VIndex = B.buildConstant(S32, 0).getReg(0);
1419 B.getMRI()->setRegBank(VIndex, AMDGPU::VGPRRegBank);
1420 unsigned CachePolicy = MI.getOperand(3).getImm();
1421
1422 SmallVector<Register, 4> LoadParts(NumLoads);
1423
1424 MachineBasicBlock::iterator MII = MI.getIterator();
1425 MachineInstrSpan Span(MII, &B.getMBB());
1426
1427 for (int i = 0; i < NumLoads; ++i) {
1428 if (NumLoads == 1) {
1429 LoadParts[i] = Dst;
1430 } else {
1431 LoadParts[i] = MRI.createGenericVirtualRegister(Ty);
1432 MRI.setRegBank(LoadParts[i], AMDGPU::VGPRRegBank);
1433 }
1434
1435 if (i != 0)
1436 BaseMMO = MF.getMachineMemOperand(BaseMMO, 16, MemSize);
1437
1438 B.buildInstr(getSBufferLoadCorrespondingBufferLoadOpcode(MI.getOpcode()))
1439 .addDef(LoadParts[i]) // vdata
1440 .addUse(RSrc) // rsrc
1441 .addUse(VIndex) // vindex
1442 .addUse(VOffset) // voffset
1443 .addUse(SOffset) // soffset
1444 .addImm(ImmOffset + 16 * i) // offset(imm)
1445 .addImm(CachePolicy) // cachepolicy, swizzled buffer(imm)
1446 .addImm(0) // idxen(imm)
1447 .addMemOperand(BaseMMO);
1448 }
1449
1450 // TODO: If only the resource is a VGPR, it may be better to execute the
1451 // scalar load in the waterfall loop if the resource is expected to frequently
1452 // be dynamically uniform.
1453 if (RSrcBank != &AMDGPU::SGPRRegBank) {
1454 // Remove the original instruction to avoid potentially confusing the
1455 // waterfall loop logic.
1456 B.setInstr(*Span.begin());
1457 MI.eraseFromParent();
1458
1459 SmallSet<Register, 4> OpsToWaterfall;
1460
1461 OpsToWaterfall.insert(RSrc);
1462 executeInWaterfallLoop(B, make_range(Span.begin(), Span.end()),
1463 OpsToWaterfall);
1464 }
1465
1466 if (NumLoads != 1) {
1467 if (Ty.isVector())
1468 B.buildConcatVectors(Dst, LoadParts);
1469 else
1470 B.buildMergeLikeInstr(Dst, LoadParts);
1471 }
1472
1473 // We removed the instruction earlier with a waterfall loop.
1474 if (RSrcBank == &AMDGPU::SGPRRegBank)
1475 MI.eraseFromParent();
1476
1477 return true;
1478}
1479
1481 const OperandsMapper &OpdMapper,
1482 bool Signed) const {
1483 MachineInstr &MI = OpdMapper.getMI();
1484 MachineRegisterInfo &MRI = OpdMapper.getMRI();
1485
1486 // Insert basic copies
1487 applyDefaultMapping(OpdMapper);
1488
1489 Register DstReg = MI.getOperand(0).getReg();
1490 LLT Ty = MRI.getType(DstReg);
1491
1492 const LLT S32 = LLT::scalar(32);
1493
1494 unsigned FirstOpnd = isa<GIntrinsic>(MI) ? 2 : 1;
1495 Register SrcReg = MI.getOperand(FirstOpnd).getReg();
1496 Register OffsetReg = MI.getOperand(FirstOpnd + 1).getReg();
1497 Register WidthReg = MI.getOperand(FirstOpnd + 2).getReg();
1498
1499 const RegisterBank *DstBank =
1500 OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
1501 if (DstBank == &AMDGPU::VGPRRegBank) {
1502 if (Ty == S32)
1503 return true;
1504
1505 // There is no 64-bit vgpr bitfield extract instructions so the operation
1506 // is expanded to a sequence of instructions that implement the operation.
1507 ApplyRegBankMapping ApplyBank(B, *this, MRI, &AMDGPU::VGPRRegBank);
1508
1509 const LLT S64 = LLT::scalar(64);
1510 // Shift the source operand so that extracted bits start at bit 0.
1511 auto ShiftOffset = Signed ? B.buildAShr(S64, SrcReg, OffsetReg)
1512 : B.buildLShr(S64, SrcReg, OffsetReg);
1513 auto UnmergeSOffset = B.buildUnmerge({S32, S32}, ShiftOffset);
1514
1515 // A 64-bit bitfield extract uses the 32-bit bitfield extract instructions
1516 // if the width is a constant.
1517 if (auto ConstWidth = getIConstantVRegValWithLookThrough(WidthReg, MRI)) {
1518 // Use the 32-bit bitfield extract instruction if the width is a constant.
1519 // Depending on the width size, use either the low or high 32-bits.
1520 auto Zero = B.buildConstant(S32, 0);
1521 auto WidthImm = ConstWidth->Value.getZExtValue();
1522 if (WidthImm <= 32) {
1523 // Use bitfield extract on the lower 32-bit source, and then sign-extend
1524 // or clear the upper 32-bits.
1525 auto Extract =
1526 Signed ? B.buildSbfx(S32, UnmergeSOffset.getReg(0), Zero, WidthReg)
1527 : B.buildUbfx(S32, UnmergeSOffset.getReg(0), Zero, WidthReg);
1528 auto Extend =
1529 Signed ? B.buildAShr(S32, Extract, B.buildConstant(S32, 31)) : Zero;
1530 B.buildMergeLikeInstr(DstReg, {Extract, Extend});
1531 } else {
1532 // Use bitfield extract on upper 32-bit source, and combine with lower
1533 // 32-bit source.
1534 auto UpperWidth = B.buildConstant(S32, WidthImm - 32);
1535 auto Extract =
1536 Signed
1537 ? B.buildSbfx(S32, UnmergeSOffset.getReg(1), Zero, UpperWidth)
1538 : B.buildUbfx(S32, UnmergeSOffset.getReg(1), Zero, UpperWidth);
1539 B.buildMergeLikeInstr(DstReg, {UnmergeSOffset.getReg(0), Extract});
1540 }
1541 MI.eraseFromParent();
1542 return true;
1543 }
1544
1545 // Expand to Src >> Offset << (64 - Width) >> (64 - Width) using 64-bit
1546 // operations.
1547 auto ExtShift = B.buildSub(S32, B.buildConstant(S32, 64), WidthReg);
1548 auto SignBit = B.buildShl(S64, ShiftOffset, ExtShift);
1549 if (Signed)
1550 B.buildAShr(S64, SignBit, ExtShift);
1551 else
1552 B.buildLShr(S64, SignBit, ExtShift);
1553 MI.eraseFromParent();
1554 return true;
1555 }
1556
1557 // The scalar form packs the offset and width in a single operand.
1558
1559 ApplyRegBankMapping ApplyBank(B, *this, MRI, &AMDGPU::SGPRRegBank);
1560
1561 // Ensure the high bits are clear to insert the offset.
1562 auto OffsetMask = B.buildConstant(S32, maskTrailingOnes<unsigned>(6));
1563 auto ClampOffset = B.buildAnd(S32, OffsetReg, OffsetMask);
1564
1565 // Zeros out the low bits, so don't bother clamping the input value.
1566 auto ShiftWidth = B.buildShl(S32, WidthReg, B.buildConstant(S32, 16));
1567
1568 // Transformation function, pack the offset and width of a BFE into
1569 // the format expected by the S_BFE_I32 / S_BFE_U32. In the second
1570 // source, bits [5:0] contain the offset and bits [22:16] the width.
1571 auto MergedInputs = B.buildOr(S32, ClampOffset, ShiftWidth);
1572
1573 // TODO: It might be worth using a pseudo here to avoid scc clobber and
1574 // register class constraints.
1575 unsigned Opc = Ty == S32 ? (Signed ? AMDGPU::S_BFE_I32 : AMDGPU::S_BFE_U32) :
1576 (Signed ? AMDGPU::S_BFE_I64 : AMDGPU::S_BFE_U64);
1577
1578 auto MIB = B.buildInstr(Opc, {DstReg}, {SrcReg, MergedInputs});
1579 constrainSelectedInstRegOperands(*MIB, *TII, *TRI, *this);
1580
1581 MI.eraseFromParent();
1582 return true;
1583}
1584
1586 MachineIRBuilder &B, const OperandsMapper &OpdMapper) const {
1587 MachineInstr &MI = OpdMapper.getMI();
1588 MachineRegisterInfo &MRI = OpdMapper.getMRI();
1589
1590 // Insert basic copies.
1591 applyDefaultMapping(OpdMapper);
1592
1593 Register Dst0 = MI.getOperand(0).getReg();
1594 Register Dst1 = MI.getOperand(1).getReg();
1595 Register Src0 = MI.getOperand(2).getReg();
1596 Register Src1 = MI.getOperand(3).getReg();
1597 Register Src2 = MI.getOperand(4).getReg();
1598
1599 if (MRI.getRegBankOrNull(Src0) == &AMDGPU::VGPRRegBank)
1600 return true;
1601
1602 bool IsUnsigned = MI.getOpcode() == AMDGPU::G_AMDGPU_MAD_U64_U32;
1603 LLT S1 = LLT::scalar(1);
1604 LLT S32 = LLT::scalar(32);
1605
1606 bool DstOnValu = MRI.getRegBankOrNull(Src2) == &AMDGPU::VGPRRegBank;
1607 bool Accumulate = true;
1608
1609 if (!DstOnValu) {
1610 if (mi_match(Src2, MRI, m_ZeroInt()))
1611 Accumulate = false;
1612 }
1613
1614 // Keep the multiplication on the SALU.
1615 Register DstHi;
1616 Register DstLo = B.buildMul(S32, Src0, Src1).getReg(0);
1617 bool MulHiInVgpr = false;
1618
1619 MRI.setRegBank(DstLo, AMDGPU::SGPRRegBank);
1620
1621 if (Subtarget.hasSMulHi()) {
1622 DstHi = IsUnsigned ? B.buildUMulH(S32, Src0, Src1).getReg(0)
1623 : B.buildSMulH(S32, Src0, Src1).getReg(0);
1624 MRI.setRegBank(DstHi, AMDGPU::SGPRRegBank);
1625 } else {
1626 Register VSrc0 = B.buildCopy(S32, Src0).getReg(0);
1627 Register VSrc1 = B.buildCopy(S32, Src1).getReg(0);
1628
1629 MRI.setRegBank(VSrc0, AMDGPU::VGPRRegBank);
1630 MRI.setRegBank(VSrc1, AMDGPU::VGPRRegBank);
1631
1632 DstHi = IsUnsigned ? B.buildUMulH(S32, VSrc0, VSrc1).getReg(0)
1633 : B.buildSMulH(S32, VSrc0, VSrc1).getReg(0);
1634 MRI.setRegBank(DstHi, AMDGPU::VGPRRegBank);
1635
1636 if (!DstOnValu) {
1637 DstHi = buildReadFirstLane(B, MRI, DstHi);
1638 } else {
1639 MulHiInVgpr = true;
1640 }
1641 }
1642
1643 // Accumulate and produce the "carry-out" bit.
1644 //
1645 // The "carry-out" is defined as bit 64 of the result when computed as a
1646 // big integer. For unsigned multiply-add, this matches the usual definition
1647 // of carry-out. For signed multiply-add, bit 64 is the sign bit of the
1648 // result, which is determined as:
1649 // sign(Src0 * Src1) + sign(Src2) + carry-out from unsigned 64-bit add
1650 LLT CarryType = DstOnValu ? S1 : S32;
1651 const RegisterBank &CarryBank =
1652 DstOnValu ? AMDGPU::VCCRegBank : AMDGPU::SGPRRegBank;
1653 const RegisterBank &DstBank =
1654 DstOnValu ? AMDGPU::VGPRRegBank : AMDGPU::SGPRRegBank;
1655 Register Carry;
1656 Register Zero;
1657
1658 if (!IsUnsigned) {
1659 Zero = B.buildConstant(S32, 0).getReg(0);
1660 MRI.setRegBank(Zero,
1661 MulHiInVgpr ? AMDGPU::VGPRRegBank : AMDGPU::SGPRRegBank);
1662
1663 Carry = B.buildICmp(CmpInst::ICMP_SLT, MulHiInVgpr ? S1 : S32, DstHi, Zero)
1664 .getReg(0);
1665 MRI.setRegBank(Carry, MulHiInVgpr ? AMDGPU::VCCRegBank
1666 : AMDGPU::SGPRRegBank);
1667
1668 if (DstOnValu && !MulHiInVgpr) {
1669 Carry = B.buildTrunc(S1, Carry).getReg(0);
1670 MRI.setRegBank(Carry, AMDGPU::VCCRegBank);
1671 }
1672 }
1673
1674 if (Accumulate) {
1675 if (DstOnValu) {
1676 DstLo = B.buildCopy(S32, DstLo).getReg(0);
1677 DstHi = B.buildCopy(S32, DstHi).getReg(0);
1678 MRI.setRegBank(DstLo, AMDGPU::VGPRRegBank);
1679 MRI.setRegBank(DstHi, AMDGPU::VGPRRegBank);
1680 }
1681
1682 auto Unmerge = B.buildUnmerge(S32, Src2);
1683 Register Src2Lo = Unmerge.getReg(0);
1684 Register Src2Hi = Unmerge.getReg(1);
1685 MRI.setRegBank(Src2Lo, DstBank);
1686 MRI.setRegBank(Src2Hi, DstBank);
1687
1688 if (!IsUnsigned) {
1689 auto Src2Sign = B.buildICmp(CmpInst::ICMP_SLT, CarryType, Src2Hi, Zero);
1690 MRI.setRegBank(Src2Sign.getReg(0), CarryBank);
1691
1692 Carry = B.buildXor(CarryType, Carry, Src2Sign).getReg(0);
1693 MRI.setRegBank(Carry, CarryBank);
1694 }
1695
1696 auto AddLo = B.buildUAddo(S32, CarryType, DstLo, Src2Lo);
1697 DstLo = AddLo.getReg(0);
1698 Register CarryLo = AddLo.getReg(1);
1699 MRI.setRegBank(DstLo, DstBank);
1700 MRI.setRegBank(CarryLo, CarryBank);
1701
1702 auto AddHi = B.buildUAdde(S32, CarryType, DstHi, Src2Hi, CarryLo);
1703 DstHi = AddHi.getReg(0);
1704 MRI.setRegBank(DstHi, DstBank);
1705
1706 Register CarryHi = AddHi.getReg(1);
1707 MRI.setRegBank(CarryHi, CarryBank);
1708
1709 if (IsUnsigned) {
1710 Carry = CarryHi;
1711 } else {
1712 Carry = B.buildXor(CarryType, Carry, CarryHi).getReg(0);
1713 MRI.setRegBank(Carry, CarryBank);
1714 }
1715 } else {
1716 if (IsUnsigned) {
1717 Carry = B.buildConstant(CarryType, 0).getReg(0);
1718 MRI.setRegBank(Carry, CarryBank);
1719 }
1720 }
1721
1722 B.buildMergeLikeInstr(Dst0, {DstLo, DstHi});
1723
1724 if (DstOnValu) {
1725 B.buildCopy(Dst1, Carry);
1726 } else {
1727 B.buildTrunc(Dst1, Carry);
1728 }
1729
1730 MI.eraseFromParent();
1731 return true;
1732}
1733
1734// Return a suitable opcode for extending the operands of Opc when widening.
1735static unsigned getExtendOp(unsigned Opc) {
1736 switch (Opc) {
1737 case TargetOpcode::G_ASHR:
1738 case TargetOpcode::G_SMIN:
1739 case TargetOpcode::G_SMAX:
1740 return TargetOpcode::G_SEXT;
1741 case TargetOpcode::G_LSHR:
1742 case TargetOpcode::G_UMIN:
1743 case TargetOpcode::G_UMAX:
1744 return TargetOpcode::G_ZEXT;
1745 default:
1746 return TargetOpcode::G_ANYEXT;
1747 }
1748}
1749
1750// Emit a legalized extension from <2 x s16> to 2 32-bit components, avoiding
1751// any illegal vector extend or unmerge operations.
1752static std::pair<Register, Register>
1753unpackV2S16ToS32(MachineIRBuilder &B, Register Src, unsigned ExtOpcode) {
1754 const LLT S32 = LLT::scalar(32);
1755 auto Bitcast = B.buildBitcast(S32, Src);
1756
1757 if (ExtOpcode == TargetOpcode::G_SEXT) {
1758 auto ExtLo = B.buildSExtInReg(S32, Bitcast, 16);
1759 auto ShiftHi = B.buildAShr(S32, Bitcast, B.buildConstant(S32, 16));
1760 return std::pair(ExtLo.getReg(0), ShiftHi.getReg(0));
1761 }
1762
1763 auto ShiftHi = B.buildLShr(S32, Bitcast, B.buildConstant(S32, 16));
1764 if (ExtOpcode == TargetOpcode::G_ZEXT) {
1765 auto ExtLo = B.buildAnd(S32, Bitcast, B.buildConstant(S32, 0xffff));
1766 return std::pair(ExtLo.getReg(0), ShiftHi.getReg(0));
1767 }
1768
1769 assert(ExtOpcode == TargetOpcode::G_ANYEXT);
1770 return std::pair(Bitcast.getReg(0), ShiftHi.getReg(0));
1771}
1772
1773// For cases where only a single copy is inserted for matching register banks.
1774// Replace the register in the instruction operand
1776 const AMDGPURegisterBankInfo::OperandsMapper &OpdMapper, unsigned OpIdx) {
1777 SmallVector<unsigned, 1> SrcReg(OpdMapper.getVRegs(OpIdx));
1778 if (!SrcReg.empty()) {
1779 assert(SrcReg.size() == 1);
1780 OpdMapper.getMI().getOperand(OpIdx).setReg(SrcReg[0]);
1781 return true;
1782 }
1783
1784 return false;
1785}
1786
1787/// Handle register layout difference for f16 images for some subtargets.
1790 Register Reg) const {
1791 if (!Subtarget.hasUnpackedD16VMem())
1792 return Reg;
1793
1794 const LLT S16 = LLT::scalar(16);
1795 LLT StoreVT = MRI.getType(Reg);
1796 if (!StoreVT.isVector() || StoreVT.getElementType() != S16)
1797 return Reg;
1798
1799 auto Unmerge = B.buildUnmerge(S16, Reg);
1800
1801
1802 SmallVector<Register, 4> WideRegs;
1803 for (int I = 0, E = Unmerge->getNumOperands() - 1; I != E; ++I)
1804 WideRegs.push_back(Unmerge.getReg(I));
1805
1806 const LLT S32 = LLT::scalar(32);
1807 int NumElts = StoreVT.getNumElements();
1808
1809 return B.buildMergeLikeInstr(LLT::fixed_vector(NumElts, S32), WideRegs)
1810 .getReg(0);
1811}
1812
1814 Register SrcReg) const {
1815 MachineRegisterInfo &MRI = *B.getMRI();
1816 LLT SrcTy = MRI.getType(SrcReg);
1817 if (SrcTy.getSizeInBits() == 32) {
1818 // Use a v_mov_b32 here to make the exec dependency explicit.
1819 B.buildInstr(AMDGPU::V_MOV_B32_e32)
1820 .addDef(DstReg)
1821 .addUse(SrcReg);
1822 return constrainGenericRegister(DstReg, AMDGPU::VGPR_32RegClass, MRI) &&
1823 constrainGenericRegister(SrcReg, AMDGPU::SReg_32RegClass, MRI);
1824 }
1825
1826 Register TmpReg0 = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass);
1827 Register TmpReg1 = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass);
1828
1829 B.buildInstr(AMDGPU::V_MOV_B32_e32)
1830 .addDef(TmpReg0)
1831 .addUse(SrcReg, {}, AMDGPU::sub0);
1832 B.buildInstr(AMDGPU::V_MOV_B32_e32)
1833 .addDef(TmpReg1)
1834 .addUse(SrcReg, {}, AMDGPU::sub1);
1835 B.buildInstr(AMDGPU::REG_SEQUENCE)
1836 .addDef(DstReg)
1837 .addUse(TmpReg0)
1838 .addImm(AMDGPU::sub0)
1839 .addUse(TmpReg1)
1840 .addImm(AMDGPU::sub1);
1841
1842 return constrainGenericRegister(SrcReg, AMDGPU::SReg_64RegClass, MRI) &&
1843 constrainGenericRegister(DstReg, AMDGPU::VReg_64RegClass, MRI);
1844}
1845
1846/// Utility function for pushing dynamic vector indexes with a constant offset
1847/// into waterfall loops.
1849 MachineInstr &IdxUseInstr,
1850 unsigned OpIdx,
1851 unsigned ConstOffset) {
1852 MachineRegisterInfo &MRI = *B.getMRI();
1853 const LLT S32 = LLT::scalar(32);
1854 Register WaterfallIdx = IdxUseInstr.getOperand(OpIdx).getReg();
1855 B.setInsertPt(*IdxUseInstr.getParent(), IdxUseInstr.getIterator());
1856
1857 auto MaterializedOffset = B.buildConstant(S32, ConstOffset);
1858
1859 auto Add = B.buildAdd(S32, WaterfallIdx, MaterializedOffset);
1860 MRI.setRegBank(MaterializedOffset.getReg(0), AMDGPU::SGPRRegBank);
1861 MRI.setRegBank(Add.getReg(0), AMDGPU::SGPRRegBank);
1862 IdxUseInstr.getOperand(OpIdx).setReg(Add.getReg(0));
1863}
1864
1865/// Implement extending a 32-bit value to a 64-bit value. \p Lo32Reg is the
1866/// original 32-bit source value (to be inserted in the low part of the combined
1867/// 64-bit result), and \p Hi32Reg is the high half of the combined 64-bit
1868/// value.
1870 Register Hi32Reg, Register Lo32Reg,
1871 unsigned ExtOpc,
1872 const RegisterBank &RegBank,
1873 bool IsBooleanSrc = false) {
1874 if (ExtOpc == AMDGPU::G_ZEXT) {
1875 B.buildConstant(Hi32Reg, 0);
1876 } else if (ExtOpc == AMDGPU::G_SEXT) {
1877 if (IsBooleanSrc) {
1878 // If we know the original source was an s1, the high half is the same as
1879 // the low.
1880 B.buildCopy(Hi32Reg, Lo32Reg);
1881 } else {
1882 // Replicate sign bit from 32-bit extended part.
1883 auto ShiftAmt = B.buildConstant(LLT::scalar(32), 31);
1884 B.getMRI()->setRegBank(ShiftAmt.getReg(0), RegBank);
1885 B.buildAShr(Hi32Reg, Lo32Reg, ShiftAmt);
1886 }
1887 } else {
1888 assert(ExtOpc == AMDGPU::G_ANYEXT && "not an integer extension");
1889 B.buildUndef(Hi32Reg);
1890 }
1891}
1892
1893bool AMDGPURegisterBankInfo::foldExtractEltToCmpSelect(
1895 const OperandsMapper &OpdMapper) const {
1896 MachineRegisterInfo &MRI = *B.getMRI();
1897
1898 Register VecReg = MI.getOperand(1).getReg();
1899 Register Idx = MI.getOperand(2).getReg();
1900
1901 const RegisterBank &IdxBank =
1902 *OpdMapper.getInstrMapping().getOperandMapping(2).BreakDown[0].RegBank;
1903
1904 bool IsDivergentIdx = IdxBank != AMDGPU::SGPRRegBank;
1905
1906 LLT VecTy = MRI.getType(VecReg);
1907 unsigned EltSize = VecTy.getScalarSizeInBits();
1908 unsigned NumElem = VecTy.getNumElements();
1909
1910 if (!SITargetLowering::shouldExpandVectorDynExt(EltSize, NumElem,
1911 IsDivergentIdx, &Subtarget))
1912 return false;
1913
1914 LLT S32 = LLT::scalar(32);
1915
1916 const RegisterBank &DstBank =
1917 *OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
1918 const RegisterBank &SrcBank =
1919 *OpdMapper.getInstrMapping().getOperandMapping(1).BreakDown[0].RegBank;
1920
1921 const RegisterBank &CCBank =
1922 (DstBank == AMDGPU::SGPRRegBank &&
1923 SrcBank == AMDGPU::SGPRRegBank &&
1924 IdxBank == AMDGPU::SGPRRegBank) ? AMDGPU::SGPRRegBank
1925 : AMDGPU::VCCRegBank;
1926 LLT CCTy = (CCBank == AMDGPU::SGPRRegBank) ? S32 : LLT::scalar(1);
1927
1928 if (CCBank == AMDGPU::VCCRegBank && IdxBank == AMDGPU::SGPRRegBank) {
1929 Idx = B.buildCopy(S32, Idx)->getOperand(0).getReg();
1930 MRI.setRegBank(Idx, AMDGPU::VGPRRegBank);
1931 }
1932
1933 LLT EltTy = VecTy.getScalarType();
1934 SmallVector<Register, 2> DstRegs(OpdMapper.getVRegs(0));
1935 unsigned NumLanes = DstRegs.size();
1936 if (!NumLanes)
1937 NumLanes = 1;
1938 else
1939 EltTy = MRI.getType(DstRegs[0]);
1940
1941 auto UnmergeToEltTy = B.buildUnmerge(EltTy, VecReg);
1942 SmallVector<Register, 2> Res(NumLanes);
1943 for (unsigned L = 0; L < NumLanes; ++L)
1944 Res[L] = UnmergeToEltTy.getReg(L);
1945
1946 for (unsigned I = 1; I < NumElem; ++I) {
1947 auto IC = B.buildConstant(S32, I);
1948 MRI.setRegBank(IC->getOperand(0).getReg(), AMDGPU::SGPRRegBank);
1949 auto Cmp = B.buildICmp(CmpInst::ICMP_EQ, CCTy, Idx, IC);
1950 MRI.setRegBank(Cmp->getOperand(0).getReg(), CCBank);
1951
1952 for (unsigned L = 0; L < NumLanes; ++L) {
1953 auto S = B.buildSelect(EltTy, Cmp,
1954 UnmergeToEltTy.getReg(I * NumLanes + L), Res[L]);
1955
1956 for (unsigned N : { 0, 2, 3 })
1957 MRI.setRegBank(S->getOperand(N).getReg(), DstBank);
1958
1959 Res[L] = S->getOperand(0).getReg();
1960 }
1961 }
1962
1963 for (unsigned L = 0; L < NumLanes; ++L) {
1964 Register DstReg = (NumLanes == 1) ? MI.getOperand(0).getReg() : DstRegs[L];
1965 B.buildCopy(DstReg, Res[L]);
1966 MRI.setRegBank(DstReg, DstBank);
1967 }
1968
1969 MRI.setRegBank(MI.getOperand(0).getReg(), DstBank);
1970 MI.eraseFromParent();
1971
1972 return true;
1973}
1974
1975// Insert a cross regbank copy for a register if it already has a bank that
1976// differs from the one we want to set.
1979 const RegisterBank &Bank) {
1980 const RegisterBank *CurrBank = MRI.getRegBankOrNull(Reg);
1981 if (CurrBank && *CurrBank != Bank) {
1982 Register Copy = B.buildCopy(MRI.getType(Reg), Reg).getReg(0);
1983 MRI.setRegBank(Copy, Bank);
1984 return Copy;
1985 }
1986
1987 MRI.setRegBank(Reg, Bank);
1988 return Reg;
1989}
1990
1991bool AMDGPURegisterBankInfo::foldInsertEltToCmpSelect(
1993 const OperandsMapper &OpdMapper) const {
1994
1995 MachineRegisterInfo &MRI = *B.getMRI();
1996 Register VecReg = MI.getOperand(1).getReg();
1997 Register Idx = MI.getOperand(3).getReg();
1998
1999 const RegisterBank &IdxBank =
2000 *OpdMapper.getInstrMapping().getOperandMapping(3).BreakDown[0].RegBank;
2001
2002 bool IsDivergentIdx = IdxBank != AMDGPU::SGPRRegBank;
2003
2004 LLT VecTy = MRI.getType(VecReg);
2005 unsigned EltSize = VecTy.getScalarSizeInBits();
2006 unsigned NumElem = VecTy.getNumElements();
2007
2008 if (!SITargetLowering::shouldExpandVectorDynExt(EltSize, NumElem,
2009 IsDivergentIdx, &Subtarget))
2010 return false;
2011
2012 LLT S32 = LLT::scalar(32);
2013
2014 const RegisterBank &DstBank =
2015 *OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
2016 const RegisterBank &SrcBank =
2017 *OpdMapper.getInstrMapping().getOperandMapping(1).BreakDown[0].RegBank;
2018 const RegisterBank &InsBank =
2019 *OpdMapper.getInstrMapping().getOperandMapping(2).BreakDown[0].RegBank;
2020
2021 const RegisterBank &CCBank =
2022 (DstBank == AMDGPU::SGPRRegBank &&
2023 SrcBank == AMDGPU::SGPRRegBank &&
2024 InsBank == AMDGPU::SGPRRegBank &&
2025 IdxBank == AMDGPU::SGPRRegBank) ? AMDGPU::SGPRRegBank
2026 : AMDGPU::VCCRegBank;
2027 LLT CCTy = (CCBank == AMDGPU::SGPRRegBank) ? S32 : LLT::scalar(1);
2028
2029 if (CCBank == AMDGPU::VCCRegBank && IdxBank == AMDGPU::SGPRRegBank) {
2030 Idx = B.buildCopy(S32, Idx)->getOperand(0).getReg();
2031 MRI.setRegBank(Idx, AMDGPU::VGPRRegBank);
2032 }
2033
2034 LLT EltTy = VecTy.getScalarType();
2035 SmallVector<Register, 2> InsRegs(OpdMapper.getVRegs(2));
2036 unsigned NumLanes = InsRegs.size();
2037 if (!NumLanes) {
2038 NumLanes = 1;
2039 InsRegs.push_back(MI.getOperand(2).getReg());
2040 } else {
2041 EltTy = MRI.getType(InsRegs[0]);
2042 }
2043
2044 auto UnmergeToEltTy = B.buildUnmerge(EltTy, VecReg);
2045 SmallVector<Register, 16> Ops(NumElem * NumLanes);
2046
2047 for (unsigned I = 0; I < NumElem; ++I) {
2048 auto IC = B.buildConstant(S32, I);
2049 MRI.setRegBank(IC->getOperand(0).getReg(), AMDGPU::SGPRRegBank);
2050 auto Cmp = B.buildICmp(CmpInst::ICMP_EQ, CCTy, Idx, IC);
2051 MRI.setRegBank(Cmp->getOperand(0).getReg(), CCBank);
2052
2053 for (unsigned L = 0; L < NumLanes; ++L) {
2054 Register Op0 = constrainRegToBank(MRI, B, InsRegs[L], DstBank);
2055 Register Op1 = UnmergeToEltTy.getReg(I * NumLanes + L);
2056 Op1 = constrainRegToBank(MRI, B, Op1, DstBank);
2057
2058 Register Select = B.buildSelect(EltTy, Cmp, Op0, Op1).getReg(0);
2059 MRI.setRegBank(Select, DstBank);
2060
2061 Ops[I * NumLanes + L] = Select;
2062 }
2063 }
2064
2065 LLT MergeTy = LLT::fixed_vector(Ops.size(), EltTy);
2066 if (MergeTy == MRI.getType(MI.getOperand(0).getReg())) {
2067 B.buildBuildVector(MI.getOperand(0), Ops);
2068 } else {
2069 auto Vec = B.buildBuildVector(MergeTy, Ops);
2070 MRI.setRegBank(Vec->getOperand(0).getReg(), DstBank);
2071 B.buildBitcast(MI.getOperand(0).getReg(), Vec);
2072 }
2073
2074 MRI.setRegBank(MI.getOperand(0).getReg(), DstBank);
2075 MI.eraseFromParent();
2076
2077 return true;
2078}
2079
2080// Break s_mul_u64 into 32-bit vector operations.
2082 MachineIRBuilder &B, const OperandsMapper &OpdMapper) const {
2083 SmallVector<Register, 2> DefRegs(OpdMapper.getVRegs(0));
2084 SmallVector<Register, 2> Src0Regs(OpdMapper.getVRegs(1));
2085 SmallVector<Register, 2> Src1Regs(OpdMapper.getVRegs(2));
2086
2087 // All inputs are SGPRs, nothing special to do.
2088 if (DefRegs.empty()) {
2089 assert(Src0Regs.empty() && Src1Regs.empty());
2090 applyDefaultMapping(OpdMapper);
2091 return;
2092 }
2093
2094 assert(DefRegs.size() == 2);
2095 assert(Src0Regs.size() == Src1Regs.size() &&
2096 (Src0Regs.empty() || Src0Regs.size() == 2));
2097
2098 MachineRegisterInfo &MRI = OpdMapper.getMRI();
2099 MachineInstr &MI = OpdMapper.getMI();
2100 Register DstReg = MI.getOperand(0).getReg();
2101 LLT HalfTy = LLT::scalar(32);
2102
2103 // Depending on where the source registers came from, the generic code may
2104 // have decided to split the inputs already or not. If not, we still need to
2105 // extract the values.
2106
2107 if (Src0Regs.empty())
2108 split64BitValueForMapping(B, Src0Regs, HalfTy, MI.getOperand(1).getReg());
2109 else
2110 setRegsToType(MRI, Src0Regs, HalfTy);
2111
2112 if (Src1Regs.empty())
2113 split64BitValueForMapping(B, Src1Regs, HalfTy, MI.getOperand(2).getReg());
2114 else
2115 setRegsToType(MRI, Src1Regs, HalfTy);
2116
2117 setRegsToType(MRI, DefRegs, HalfTy);
2118
2119 // The multiplication is done as follows:
2120 //
2121 // Op1H Op1L
2122 // * Op0H Op0L
2123 // --------------------
2124 // Op1H*Op0L Op1L*Op0L
2125 // + Op1H*Op0H Op1L*Op0H
2126 // -----------------------------------------
2127 // (Op1H*Op0L + Op1L*Op0H + carry) Op1L*Op0L
2128 //
2129 // We drop Op1H*Op0H because the result of the multiplication is a 64-bit
2130 // value and that would overflow.
2131 // The low 32-bit value is Op1L*Op0L.
2132 // The high 32-bit value is Op1H*Op0L + Op1L*Op0H + carry (from
2133 // Op1L*Op0L).
2134
2135 ApplyRegBankMapping ApplyBank(B, *this, MRI, &AMDGPU::VGPRRegBank);
2136
2137 Register Hi = B.buildUMulH(HalfTy, Src0Regs[0], Src1Regs[0]).getReg(0);
2138 Register MulLoHi = B.buildMul(HalfTy, Src0Regs[0], Src1Regs[1]).getReg(0);
2139 Register Add = B.buildAdd(HalfTy, Hi, MulLoHi).getReg(0);
2140 Register MulHiLo = B.buildMul(HalfTy, Src0Regs[1], Src1Regs[0]).getReg(0);
2141 B.buildAdd(DefRegs[1], Add, MulHiLo);
2142 B.buildMul(DefRegs[0], Src0Regs[0], Src1Regs[0]);
2143
2144 MRI.setRegBank(DstReg, AMDGPU::VGPRRegBank);
2145 MI.eraseFromParent();
2146}
2147
2149 MachineIRBuilder &B, const OperandsMapper &OpdMapper) const {
2150 MachineInstr &MI = OpdMapper.getMI();
2151 B.setInstrAndDebugLoc(MI);
2152 unsigned Opc = MI.getOpcode();
2153 MachineRegisterInfo &MRI = OpdMapper.getMRI();
2154 switch (Opc) {
2155 case AMDGPU::G_CONSTANT:
2156 case AMDGPU::G_IMPLICIT_DEF: {
2157 Register DstReg = MI.getOperand(0).getReg();
2158 LLT DstTy = MRI.getType(DstReg);
2159 if (DstTy != LLT::scalar(1))
2160 break;
2161
2162 const RegisterBank *DstBank =
2163 OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
2164 if (DstBank == &AMDGPU::VCCRegBank)
2165 break;
2166 SmallVector<Register, 1> DefRegs(OpdMapper.getVRegs(0));
2167 if (DefRegs.empty())
2168 DefRegs.push_back(DstReg);
2169
2170 B.setInsertPt(*MI.getParent(), ++MI.getIterator());
2171
2172 Register NewDstReg = MRI.createGenericVirtualRegister(LLT::scalar(32));
2173 LLVMContext &Ctx = B.getMF().getFunction().getContext();
2174
2175 MI.getOperand(0).setReg(NewDstReg);
2176 if (Opc != AMDGPU::G_IMPLICIT_DEF) {
2177 uint64_t ConstVal = MI.getOperand(1).getCImm()->getZExtValue();
2178 MI.getOperand(1).setCImm(
2179 ConstantInt::get(IntegerType::getInt32Ty(Ctx), ConstVal));
2180 }
2181
2182 MRI.setRegBank(NewDstReg, *DstBank);
2183 B.buildTrunc(DefRegs[0], NewDstReg);
2184 return;
2185 }
2186 case AMDGPU::G_PHI: {
2187 Register DstReg = MI.getOperand(0).getReg();
2188 LLT DstTy = MRI.getType(DstReg);
2189 if (DstTy != LLT::scalar(1))
2190 break;
2191
2192 const LLT S32 = LLT::scalar(32);
2193 const RegisterBank *DstBank =
2194 OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
2195 if (DstBank == &AMDGPU::VCCRegBank) {
2196 applyDefaultMapping(OpdMapper);
2197 // The standard handling only considers the result register bank for
2198 // phis. For VCC, blindly inserting a copy when the phi is lowered will
2199 // produce an invalid copy. We can only copy with some kind of compare to
2200 // get a vector boolean result. Insert a register bank copy that will be
2201 // correctly lowered to a compare.
2202 for (unsigned I = 1, E = MI.getNumOperands(); I != E; I += 2) {
2203 Register SrcReg = MI.getOperand(I).getReg();
2204 const RegisterBank *SrcBank = getRegBank(SrcReg, MRI, *TRI);
2205
2206 if (SrcBank != &AMDGPU::VCCRegBank) {
2207 MachineBasicBlock *SrcMBB = MI.getOperand(I + 1).getMBB();
2208 B.setInsertPt(*SrcMBB, SrcMBB->getFirstTerminator());
2209
2210 auto Copy = B.buildCopy(LLT::scalar(1), SrcReg);
2211 MRI.setRegBank(Copy.getReg(0), AMDGPU::VCCRegBank);
2212 MI.getOperand(I).setReg(Copy.getReg(0));
2213 }
2214 }
2215
2216 return;
2217 }
2218
2219 // Phi handling is strange and only considers the bank of the destination.
2220 substituteSimpleCopyRegs(OpdMapper, 0);
2221
2222 // Promote SGPR/VGPR booleans to s32
2223 ApplyRegBankMapping ApplyBank(B, *this, MRI, DstBank);
2224 B.setInsertPt(B.getMBB(), MI);
2225 LegalizerHelper Helper(B.getMF(), ApplyBank, B);
2226
2227 if (Helper.widenScalar(MI, 0, S32) != LegalizerHelper::Legalized)
2228 llvm_unreachable("widen scalar should have succeeded");
2229
2230 return;
2231 }
2232 case AMDGPU::G_FCMP:
2233 if (!Subtarget.hasSALUFloatInsts())
2234 break;
2235 [[fallthrough]];
2236 case AMDGPU::G_ICMP:
2237 case AMDGPU::G_UADDO:
2238 case AMDGPU::G_USUBO:
2239 case AMDGPU::G_UADDE:
2240 case AMDGPU::G_SADDE:
2241 case AMDGPU::G_USUBE:
2242 case AMDGPU::G_SSUBE: {
2243 unsigned BoolDstOp =
2244 (Opc == AMDGPU::G_ICMP || Opc == AMDGPU::G_FCMP) ? 0 : 1;
2245 Register DstReg = MI.getOperand(BoolDstOp).getReg();
2246
2247 const RegisterBank *DstBank =
2248 OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
2249 if (DstBank != &AMDGPU::SGPRRegBank)
2250 break;
2251
2252 const bool HasCarryIn = MI.getNumOperands() == 5;
2253
2254 // If this is a scalar compare, promote the result to s32, as the selection
2255 // will end up using a copy to a 32-bit vreg.
2256 const LLT S32 = LLT::scalar(32);
2257 Register NewDstReg = MRI.createGenericVirtualRegister(S32);
2258 MRI.setRegBank(NewDstReg, AMDGPU::SGPRRegBank);
2259 MI.getOperand(BoolDstOp).setReg(NewDstReg);
2260
2261 if (HasCarryIn) {
2262 Register NewSrcReg = MRI.createGenericVirtualRegister(S32);
2263 MRI.setRegBank(NewSrcReg, AMDGPU::SGPRRegBank);
2264 B.buildZExt(NewSrcReg, MI.getOperand(4).getReg());
2265 MI.getOperand(4).setReg(NewSrcReg);
2266 }
2267
2268 MachineBasicBlock *MBB = MI.getParent();
2269 B.setInsertPt(*MBB, std::next(MI.getIterator()));
2270
2271 // If we had a constrained VCC result register, a copy was inserted to VCC
2272 // from SGPR.
2273 SmallVector<Register, 1> DefRegs(OpdMapper.getVRegs(0));
2274 if (DefRegs.empty())
2275 DefRegs.push_back(DstReg);
2276 B.buildTrunc(DefRegs[0], NewDstReg);
2277 return;
2278 }
2279 case AMDGPU::G_SELECT: {
2280 Register DstReg = MI.getOperand(0).getReg();
2281 LLT DstTy = MRI.getType(DstReg);
2282
2283 SmallVector<Register, 1> CondRegs(OpdMapper.getVRegs(1));
2284 if (CondRegs.empty())
2285 CondRegs.push_back(MI.getOperand(1).getReg());
2286 else {
2287 assert(CondRegs.size() == 1);
2288 }
2289
2290 const RegisterBank *CondBank = getRegBank(CondRegs[0], MRI, *TRI);
2291 if (CondBank == &AMDGPU::SGPRRegBank) {
2292 const LLT S32 = LLT::scalar(32);
2293 Register NewCondReg = MRI.createGenericVirtualRegister(S32);
2294 MRI.setRegBank(NewCondReg, AMDGPU::SGPRRegBank);
2295
2296 MI.getOperand(1).setReg(NewCondReg);
2297 B.buildZExt(NewCondReg, CondRegs[0]);
2298 }
2299
2300 if (DstTy.getSizeInBits() != 64)
2301 break;
2302
2303 LLT HalfTy = getHalfSizedType(DstTy);
2304
2305 SmallVector<Register, 2> DefRegs(OpdMapper.getVRegs(0));
2306 SmallVector<Register, 2> Src1Regs(OpdMapper.getVRegs(2));
2307 SmallVector<Register, 2> Src2Regs(OpdMapper.getVRegs(3));
2308
2309 // All inputs are SGPRs, nothing special to do.
2310 if (DefRegs.empty()) {
2311 assert(Src1Regs.empty() && Src2Regs.empty());
2312 break;
2313 }
2314
2315 if (Src1Regs.empty())
2316 split64BitValueForMapping(B, Src1Regs, HalfTy, MI.getOperand(2).getReg());
2317 else {
2318 setRegsToType(MRI, Src1Regs, HalfTy);
2319 }
2320
2321 if (Src2Regs.empty())
2322 split64BitValueForMapping(B, Src2Regs, HalfTy, MI.getOperand(3).getReg());
2323 else
2324 setRegsToType(MRI, Src2Regs, HalfTy);
2325
2326 setRegsToType(MRI, DefRegs, HalfTy);
2327
2328 auto Flags = MI.getFlags();
2329 B.buildSelect(DefRegs[0], CondRegs[0], Src1Regs[0], Src2Regs[0], Flags);
2330 B.buildSelect(DefRegs[1], CondRegs[0], Src1Regs[1], Src2Regs[1], Flags);
2331
2332 MRI.setRegBank(DstReg, AMDGPU::VGPRRegBank);
2333 MI.eraseFromParent();
2334 return;
2335 }
2336 case AMDGPU::G_BRCOND: {
2337 Register CondReg = MI.getOperand(0).getReg();
2338 // FIXME: Should use legalizer helper, but should change bool ext type.
2339 const RegisterBank *CondBank =
2340 OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
2341
2342 if (CondBank == &AMDGPU::SGPRRegBank) {
2343 const LLT S32 = LLT::scalar(32);
2344 Register NewCondReg = MRI.createGenericVirtualRegister(S32);
2345 MRI.setRegBank(NewCondReg, AMDGPU::SGPRRegBank);
2346
2347 MI.getOperand(0).setReg(NewCondReg);
2348 B.buildZExt(NewCondReg, CondReg);
2349 return;
2350 }
2351
2352 break;
2353 }
2354 case AMDGPU::G_AND:
2355 case AMDGPU::G_OR:
2356 case AMDGPU::G_XOR: {
2357 // 64-bit and is only available on the SALU, so split into 2 32-bit ops if
2358 // there is a VGPR input.
2359 Register DstReg = MI.getOperand(0).getReg();
2360 LLT DstTy = MRI.getType(DstReg);
2361
2362 const RegisterBank *DstBank =
2363 OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
2364
2365 if (DstTy.getSizeInBits() == 1) {
2366 if (DstBank == &AMDGPU::VCCRegBank)
2367 break;
2368
2369 MachineFunction *MF = MI.getMF();
2370 ApplyRegBankMapping ApplyBank(B, *this, MRI, DstBank);
2371 LegalizerHelper Helper(*MF, ApplyBank, B);
2372
2373 if (Helper.widenScalar(MI, 0, LLT::scalar(32)) !=
2375 llvm_unreachable("widen scalar should have succeeded");
2376 return;
2377 }
2378
2379 if (DstTy.getSizeInBits() == 16 && DstBank == &AMDGPU::SGPRRegBank) {
2380 const LLT S32 = LLT::scalar(32);
2381 MachineBasicBlock *MBB = MI.getParent();
2382 MachineFunction *MF = MBB->getParent();
2383 ApplyRegBankMapping ApplySALU(B, *this, MRI, &AMDGPU::SGPRRegBank);
2384 LegalizerHelper Helper(*MF, ApplySALU, B);
2385 // Widen to S32, but handle `G_XOR x, -1` differently. Legalizer widening
2386 // will use a G_ANYEXT to extend the -1 which prevents matching G_XOR -1
2387 // as "not".
2388 if (MI.getOpcode() == AMDGPU::G_XOR &&
2389 mi_match(MI.getOperand(2).getReg(), MRI, m_SpecificICstOrSplat(-1))) {
2390 Helper.widenScalarSrc(MI, S32, 1, AMDGPU::G_ANYEXT);
2391 Helper.widenScalarSrc(MI, S32, 2, AMDGPU::G_SEXT);
2392 Helper.widenScalarDst(MI, S32);
2393 } else {
2394 if (Helper.widenScalar(MI, 0, S32) != LegalizerHelper::Legalized)
2395 llvm_unreachable("widen scalar should have succeeded");
2396 }
2397 return;
2398 }
2399
2400 if (DstTy.getSizeInBits() != 64)
2401 break;
2402
2403 LLT HalfTy = getHalfSizedType(DstTy);
2404 SmallVector<Register, 2> DefRegs(OpdMapper.getVRegs(0));
2405 SmallVector<Register, 2> Src0Regs(OpdMapper.getVRegs(1));
2406 SmallVector<Register, 2> Src1Regs(OpdMapper.getVRegs(2));
2407
2408 // All inputs are SGPRs, nothing special to do.
2409 if (DefRegs.empty()) {
2410 assert(Src0Regs.empty() && Src1Regs.empty());
2411 break;
2412 }
2413
2414 assert(DefRegs.size() == 2);
2415 assert(Src0Regs.size() == Src1Regs.size() &&
2416 (Src0Regs.empty() || Src0Regs.size() == 2));
2417
2418 // Depending on where the source registers came from, the generic code may
2419 // have decided to split the inputs already or not. If not, we still need to
2420 // extract the values.
2421
2422 if (Src0Regs.empty())
2423 split64BitValueForMapping(B, Src0Regs, HalfTy, MI.getOperand(1).getReg());
2424 else
2425 setRegsToType(MRI, Src0Regs, HalfTy);
2426
2427 if (Src1Regs.empty())
2428 split64BitValueForMapping(B, Src1Regs, HalfTy, MI.getOperand(2).getReg());
2429 else
2430 setRegsToType(MRI, Src1Regs, HalfTy);
2431
2432 setRegsToType(MRI, DefRegs, HalfTy);
2433
2434 auto Flags = MI.getFlags();
2435 B.buildInstr(Opc, {DefRegs[0]}, {Src0Regs[0], Src1Regs[0]}, Flags);
2436 B.buildInstr(Opc, {DefRegs[1]}, {Src0Regs[1], Src1Regs[1]}, Flags);
2437
2438 MRI.setRegBank(DstReg, AMDGPU::VGPRRegBank);
2439 MI.eraseFromParent();
2440 return;
2441 }
2442 case AMDGPU::G_ABS: {
2443 Register SrcReg = MI.getOperand(1).getReg();
2444 const RegisterBank *SrcBank = MRI.getRegBankOrNull(SrcReg);
2445
2446 // There is no VALU abs instruction so we need to replace it with a sub and
2447 // max combination.
2448 if (SrcBank && SrcBank == &AMDGPU::VGPRRegBank) {
2449 MachineFunction *MF = MI.getMF();
2450 ApplyRegBankMapping Apply(B, *this, MRI, &AMDGPU::VGPRRegBank);
2451 LegalizerHelper Helper(*MF, Apply, B);
2452
2454 llvm_unreachable("lowerAbsToMaxNeg should have succeeded");
2455 return;
2456 }
2457 [[fallthrough]];
2458 }
2459 case AMDGPU::G_ADD:
2460 case AMDGPU::G_SUB:
2461 case AMDGPU::G_MUL:
2462 case AMDGPU::G_SHL:
2463 case AMDGPU::G_LSHR:
2464 case AMDGPU::G_ASHR:
2465 case AMDGPU::G_SMIN:
2466 case AMDGPU::G_SMAX:
2467 case AMDGPU::G_UMIN:
2468 case AMDGPU::G_UMAX: {
2469 Register DstReg = MI.getOperand(0).getReg();
2470 LLT DstTy = MRI.getType(DstReg);
2471
2472 // Special case for s_mul_u64. There is not a vector equivalent of
2473 // s_mul_u64. Hence, we have to break down s_mul_u64 into 32-bit vector
2474 // multiplications.
2475 if (!Subtarget.useVMulU64Inst() && Opc == AMDGPU::G_MUL &&
2476 DstTy.getSizeInBits() == 64) {
2477 applyMappingSMULU64(B, OpdMapper);
2478 return;
2479 }
2480
2481 // 16-bit operations are VALU only, but can be promoted to 32-bit SALU.
2482 // Packed 16-bit operations need to be scalarized and promoted.
2483 if (DstTy != LLT::scalar(16) && DstTy != LLT::fixed_vector(2, 16))
2484 break;
2485
2486 const RegisterBank *DstBank =
2487 OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
2488 if (DstBank == &AMDGPU::VGPRRegBank)
2489 break;
2490
2491 const LLT S32 = LLT::scalar(32);
2492 MachineBasicBlock *MBB = MI.getParent();
2493 MachineFunction *MF = MBB->getParent();
2494 ApplyRegBankMapping ApplySALU(B, *this, MRI, &AMDGPU::SGPRRegBank);
2495
2496 if (DstTy.isVector() && Opc == AMDGPU::G_ABS) {
2497 Register WideSrcLo, WideSrcHi;
2498
2499 std::tie(WideSrcLo, WideSrcHi) =
2500 unpackV2S16ToS32(B, MI.getOperand(1).getReg(), TargetOpcode::G_SEXT);
2501 auto Lo = B.buildInstr(AMDGPU::G_ABS, {S32}, {WideSrcLo});
2502 auto Hi = B.buildInstr(AMDGPU::G_ABS, {S32}, {WideSrcHi});
2503 B.buildBuildVectorTrunc(DstReg, {Lo.getReg(0), Hi.getReg(0)});
2504 MI.eraseFromParent();
2505 return;
2506 }
2507
2508 if (DstTy.isVector()) {
2509 Register WideSrc0Lo, WideSrc0Hi;
2510 Register WideSrc1Lo, WideSrc1Hi;
2511
2512 unsigned ExtendOp = getExtendOp(MI.getOpcode());
2513 std::tie(WideSrc0Lo, WideSrc0Hi)
2514 = unpackV2S16ToS32(B, MI.getOperand(1).getReg(), ExtendOp);
2515 std::tie(WideSrc1Lo, WideSrc1Hi)
2516 = unpackV2S16ToS32(B, MI.getOperand(2).getReg(), ExtendOp);
2517 auto Lo = B.buildInstr(MI.getOpcode(), {S32}, {WideSrc0Lo, WideSrc1Lo});
2518 auto Hi = B.buildInstr(MI.getOpcode(), {S32}, {WideSrc0Hi, WideSrc1Hi});
2519 B.buildBuildVectorTrunc(DstReg, {Lo.getReg(0), Hi.getReg(0)});
2520 MI.eraseFromParent();
2521 } else {
2522 LegalizerHelper Helper(*MF, ApplySALU, B);
2523
2524 if (Helper.widenScalar(MI, 0, S32) != LegalizerHelper::Legalized)
2525 llvm_unreachable("widen scalar should have succeeded");
2526
2527 // FIXME: s16 shift amounts should be legal.
2528 if (Opc == AMDGPU::G_SHL || Opc == AMDGPU::G_LSHR ||
2529 Opc == AMDGPU::G_ASHR) {
2530 B.setInsertPt(*MBB, MI.getIterator());
2531 if (Helper.widenScalar(MI, 1, S32) != LegalizerHelper::Legalized)
2532 llvm_unreachable("widen scalar should have succeeded");
2533 }
2534 }
2535
2536 return;
2537 }
2538 case AMDGPU::G_AMDGPU_S_MUL_I64_I32:
2539 case AMDGPU::G_AMDGPU_S_MUL_U64_U32: {
2540 // This is a special case for s_mul_u64. We use
2541 // G_AMDGPU_S_MUL_I64_I32 opcode to represent an s_mul_u64 operation
2542 // where the 33 higher bits are sign-extended and
2543 // G_AMDGPU_S_MUL_U64_U32 opcode to represent an s_mul_u64 operation
2544 // where the 32 higher bits are zero-extended. In case scalar registers are
2545 // selected, both opcodes are lowered as s_mul_u64. If the vector registers
2546 // are selected, then G_AMDGPU_S_MUL_I64_I32 and
2547 // G_AMDGPU_S_MUL_U64_U32 are lowered with a vector mad instruction.
2548
2549 // Insert basic copies.
2550 applyDefaultMapping(OpdMapper);
2551
2552 Register DstReg = MI.getOperand(0).getReg();
2553 Register SrcReg0 = MI.getOperand(1).getReg();
2554 Register SrcReg1 = MI.getOperand(2).getReg();
2555 const LLT S32 = LLT::scalar(32);
2556 const LLT S64 = LLT::scalar(64);
2557 assert(MRI.getType(DstReg) == S64 && "This is a special case for s_mul_u64 "
2558 "that handles only 64-bit operands.");
2559 const RegisterBank *DstBank =
2560 OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
2561
2562 // Replace G_AMDGPU_S_MUL_I64_I32 and G_AMDGPU_S_MUL_U64_U32
2563 // with s_mul_u64 operation.
2564 if (DstBank == &AMDGPU::SGPRRegBank) {
2565 MI.setDesc(TII->get(AMDGPU::S_MUL_U64));
2566 MRI.setRegClass(DstReg, &AMDGPU::SGPR_64RegClass);
2567 MRI.setRegClass(SrcReg0, &AMDGPU::SGPR_64RegClass);
2568 MRI.setRegClass(SrcReg1, &AMDGPU::SGPR_64RegClass);
2569 return;
2570 }
2571
2572 // Replace G_AMDGPU_S_MUL_I64_I32 and G_AMDGPU_S_MUL_U64_U32
2573 // with a vector mad.
2574 assert(MRI.getRegBankOrNull(DstReg) == &AMDGPU::VGPRRegBank &&
2575 "The destination operand should be in vector registers.");
2576
2577 // Extract the lower subregister from the first operand.
2578 Register Op0L = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass);
2579 MRI.setRegClass(Op0L, &AMDGPU::VGPR_32RegClass);
2580 MRI.setType(Op0L, S32);
2581 B.buildTrunc(Op0L, SrcReg0);
2582
2583 // Extract the lower subregister from the second operand.
2584 Register Op1L = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass);
2585 MRI.setRegClass(Op1L, &AMDGPU::VGPR_32RegClass);
2586 MRI.setType(Op1L, S32);
2587 B.buildTrunc(Op1L, SrcReg1);
2588
2589 unsigned NewOpc = Opc == AMDGPU::G_AMDGPU_S_MUL_U64_U32
2590 ? AMDGPU::G_AMDGPU_MAD_U64_U32
2591 : AMDGPU::G_AMDGPU_MAD_I64_I32;
2592
2594 Register Zero64 = B.buildConstant(S64, 0).getReg(0);
2595 MRI.setRegClass(Zero64, &AMDGPU::VReg_64RegClass);
2596 Register CarryOut = MRI.createVirtualRegister(&AMDGPU::VReg_64RegClass);
2597 MRI.setRegClass(CarryOut, &AMDGPU::VReg_64RegClass);
2598 B.buildInstr(NewOpc, {DstReg, CarryOut}, {Op0L, Op1L, Zero64});
2599 MI.eraseFromParent();
2600 return;
2601 }
2602 case AMDGPU::G_SEXT_INREG: {
2603 SmallVector<Register, 2> SrcRegs(OpdMapper.getVRegs(1));
2604 if (SrcRegs.empty())
2605 break; // Nothing to repair
2606
2607 const LLT S32 = LLT::scalar(32);
2608 ApplyRegBankMapping O(B, *this, MRI, &AMDGPU::VGPRRegBank);
2609
2610 // Don't use LegalizerHelper's narrowScalar. It produces unwanted G_SEXTs
2611 // we would need to further expand, and doesn't let us directly set the
2612 // result registers.
2613 SmallVector<Register, 2> DstRegs(OpdMapper.getVRegs(0));
2614
2615 int Amt = MI.getOperand(2).getImm();
2616 if (Amt <= 32) {
2617 // Downstream users have expectations for the high bit behavior, so freeze
2618 // incoming undefined bits.
2619 if (Amt == 32) {
2620 // The low bits are unchanged.
2621 B.buildFreeze(DstRegs[0], SrcRegs[0]);
2622 } else {
2623 auto Freeze = B.buildFreeze(S32, SrcRegs[0]);
2624 // Extend in the low bits and propagate the sign bit to the high half.
2625 B.buildSExtInReg(DstRegs[0], Freeze, Amt);
2626 }
2627
2628 B.buildAShr(DstRegs[1], DstRegs[0], B.buildConstant(S32, 31));
2629 } else {
2630 // The low bits are unchanged, and extend in the high bits.
2631 // No freeze required
2632 B.buildCopy(DstRegs[0], SrcRegs[0]);
2633 B.buildSExtInReg(DstRegs[1], DstRegs[0], Amt - 32);
2634 }
2635
2636 Register DstReg = MI.getOperand(0).getReg();
2637 MRI.setRegBank(DstReg, AMDGPU::VGPRRegBank);
2638 MI.eraseFromParent();
2639 return;
2640 }
2641 case AMDGPU::G_CTPOP:
2642 case AMDGPU::G_BITREVERSE: {
2643 const RegisterBank *DstBank =
2644 OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
2645 if (DstBank == &AMDGPU::SGPRRegBank)
2646 break;
2647
2648 Register SrcReg = MI.getOperand(1).getReg();
2649 const LLT S32 = LLT::scalar(32);
2650 LLT Ty = MRI.getType(SrcReg);
2651 if (Ty == S32)
2652 break;
2653
2654 ApplyRegBankMapping ApplyVALU(B, *this, MRI, &AMDGPU::VGPRRegBank);
2655
2656 MachineFunction &MF = B.getMF();
2657 LegalizerHelper Helper(MF, ApplyVALU, B);
2658
2659 if (Helper.narrowScalar(MI, 1, S32) != LegalizerHelper::Legalized)
2660 llvm_unreachable("narrowScalar should have succeeded");
2661 return;
2662 }
2663 case AMDGPU::G_AMDGPU_FFBH_U32:
2664 case AMDGPU::G_AMDGPU_FFBL_B32:
2665 case AMDGPU::G_CTLZ_ZERO_POISON:
2666 case AMDGPU::G_CTTZ_ZERO_POISON: {
2667 const RegisterBank *DstBank =
2668 OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
2669 if (DstBank == &AMDGPU::SGPRRegBank)
2670 break;
2671
2672 Register SrcReg = MI.getOperand(1).getReg();
2673 const LLT S32 = LLT::scalar(32);
2674 LLT Ty = MRI.getType(SrcReg);
2675 if (Ty == S32)
2676 break;
2677
2678 // We can narrow this more efficiently than Helper can by using ffbh/ffbl
2679 // which return -1 when the input is zero:
2680 // (ctlz_zero_poison hi:lo) -> (umin (ffbh hi), (add (ffbh lo), 32))
2681 // (cttz_zero_poison hi:lo) -> (umin (add (ffbl hi), 32), (ffbl lo))
2682 // (ffbh hi:lo) -> (umin (ffbh hi), (uaddsat (ffbh lo), 32))
2683 // (ffbl hi:lo) -> (umin (uaddsat (ffbh hi), 32), (ffbh lo))
2684 ApplyRegBankMapping ApplyVALU(B, *this, MRI, &AMDGPU::VGPRRegBank);
2685 SmallVector<Register, 2> SrcRegs(OpdMapper.getVRegs(1));
2686 unsigned NewOpc = Opc == AMDGPU::G_CTLZ_ZERO_POISON
2687 ? (unsigned)AMDGPU::G_AMDGPU_FFBH_U32
2688 : Opc == AMDGPU::G_CTTZ_ZERO_POISON
2689 ? (unsigned)AMDGPU::G_AMDGPU_FFBL_B32
2690 : Opc;
2691 unsigned Idx = NewOpc == AMDGPU::G_AMDGPU_FFBH_U32;
2692 auto X = B.buildInstr(NewOpc, {S32}, {SrcRegs[Idx]});
2693 auto Y = B.buildInstr(NewOpc, {S32}, {SrcRegs[Idx ^ 1]});
2694 unsigned AddOpc =
2695 Opc == AMDGPU::G_CTLZ_ZERO_POISON || Opc == AMDGPU::G_CTTZ_ZERO_POISON
2696 ? AMDGPU::G_ADD
2697 : AMDGPU::G_UADDSAT;
2698 Y = B.buildInstr(AddOpc, {S32}, {Y, B.buildConstant(S32, 32)});
2699 Register DstReg = MI.getOperand(0).getReg();
2700 B.buildUMin(DstReg, X, Y);
2701 MI.eraseFromParent();
2702 return;
2703 }
2704 case AMDGPU::G_SEXT:
2705 case AMDGPU::G_ZEXT:
2706 case AMDGPU::G_ANYEXT: {
2707 Register SrcReg = MI.getOperand(1).getReg();
2708 LLT SrcTy = MRI.getType(SrcReg);
2709 const bool Signed = Opc == AMDGPU::G_SEXT;
2710
2711 assert(OpdMapper.getVRegs(1).empty());
2712
2713 const RegisterBank *SrcBank =
2714 OpdMapper.getInstrMapping().getOperandMapping(1).BreakDown[0].RegBank;
2715
2716 Register DstReg = MI.getOperand(0).getReg();
2717 LLT DstTy = MRI.getType(DstReg);
2718 if (DstTy.isScalar() &&
2719 SrcBank != &AMDGPU::SGPRRegBank &&
2720 SrcBank != &AMDGPU::VCCRegBank &&
2721 // FIXME: Should handle any type that round to s64 when irregular
2722 // breakdowns supported.
2723 DstTy.getSizeInBits() == 64 &&
2724 SrcTy.getSizeInBits() <= 32) {
2725 SmallVector<Register, 2> DefRegs(OpdMapper.getVRegs(0));
2726
2727 // Extend to 32-bit, and then extend the low half.
2728 if (Signed) {
2729 // TODO: Should really be buildSExtOrCopy
2730 B.buildSExtOrTrunc(DefRegs[0], SrcReg);
2731 } else if (Opc == AMDGPU::G_ZEXT) {
2732 B.buildZExtOrTrunc(DefRegs[0], SrcReg);
2733 } else {
2734 B.buildAnyExtOrTrunc(DefRegs[0], SrcReg);
2735 }
2736
2737 extendLow32IntoHigh32(B, DefRegs[1], DefRegs[0], Opc, *SrcBank);
2738 MRI.setRegBank(DstReg, *SrcBank);
2739 MI.eraseFromParent();
2740 return;
2741 }
2742
2743 if (SrcTy != LLT::scalar(1))
2744 return;
2745
2746 // It is not legal to have a legalization artifact with a VCC source. Rather
2747 // than introducing a copy, insert the select we would have to select the
2748 // copy to.
2749 if (SrcBank == &AMDGPU::VCCRegBank) {
2750 SmallVector<Register, 2> DefRegs(OpdMapper.getVRegs(0));
2751
2752 const RegisterBank *DstBank = &AMDGPU::VGPRRegBank;
2753
2754 unsigned DstSize = DstTy.getSizeInBits();
2755 // 64-bit select is SGPR only
2756 const bool UseSel64 = DstSize > 32 &&
2757 SrcBank->getID() == AMDGPU::SGPRRegBankID;
2758
2759 // TODO: Should s16 select be legal?
2760 LLT SelType = UseSel64 ? LLT::scalar(64) : LLT::scalar(32);
2761 auto True = B.buildConstant(SelType, Signed ? -1 : 1);
2762 auto False = B.buildConstant(SelType, 0);
2763
2764 MRI.setRegBank(True.getReg(0), *DstBank);
2765 MRI.setRegBank(False.getReg(0), *DstBank);
2766 MRI.setRegBank(DstReg, *DstBank);
2767
2768 if (DstSize > 32) {
2769 B.buildSelect(DefRegs[0], SrcReg, True, False);
2770 extendLow32IntoHigh32(B, DefRegs[1], DefRegs[0], Opc, *SrcBank, true);
2771 } else if (DstSize < 32) {
2772 auto Sel = B.buildSelect(SelType, SrcReg, True, False);
2773 MRI.setRegBank(Sel.getReg(0), *DstBank);
2774 B.buildTrunc(DstReg, Sel);
2775 } else {
2776 B.buildSelect(DstReg, SrcReg, True, False);
2777 }
2778
2779 MI.eraseFromParent();
2780 return;
2781 }
2782
2783 break;
2784 }
2785 case AMDGPU::G_EXTRACT_VECTOR_ELT: {
2786 SmallVector<Register, 2> DstRegs(OpdMapper.getVRegs(0));
2787
2788 assert(OpdMapper.getVRegs(1).empty() && OpdMapper.getVRegs(2).empty());
2789
2790 Register DstReg = MI.getOperand(0).getReg();
2791 Register SrcReg = MI.getOperand(1).getReg();
2792
2793 const LLT S32 = LLT::scalar(32);
2794 LLT DstTy = MRI.getType(DstReg);
2795 LLT SrcTy = MRI.getType(SrcReg);
2796
2797 if (foldExtractEltToCmpSelect(B, MI, OpdMapper))
2798 return;
2799
2800 const ValueMapping &DstMapping
2801 = OpdMapper.getInstrMapping().getOperandMapping(0);
2802 const RegisterBank *DstBank = DstMapping.BreakDown[0].RegBank;
2803 const RegisterBank *SrcBank =
2804 OpdMapper.getInstrMapping().getOperandMapping(1).BreakDown[0].RegBank;
2805 const RegisterBank *IdxBank =
2806 OpdMapper.getInstrMapping().getOperandMapping(2).BreakDown[0].RegBank;
2807
2808 Register BaseIdxReg;
2809 unsigned ConstOffset;
2810 std::tie(BaseIdxReg, ConstOffset) =
2811 AMDGPU::getBaseWithConstantOffset(MRI, MI.getOperand(2).getReg());
2812
2813 // See if the index is an add of a constant which will be foldable by moving
2814 // the base register of the index later if this is going to be executed in a
2815 // waterfall loop. This is essentially to reassociate the add of a constant
2816 // with the readfirstlane.
2817 bool ShouldMoveIndexIntoLoop = IdxBank != &AMDGPU::SGPRRegBank &&
2818 ConstOffset > 0 &&
2819 ConstOffset < SrcTy.getNumElements();
2820
2821 // Move the base register. We'll re-insert the add later.
2822 if (ShouldMoveIndexIntoLoop)
2823 MI.getOperand(2).setReg(BaseIdxReg);
2824
2825 // If this is a VGPR result only because the index was a VGPR result, the
2826 // actual indexing will be done on the SGPR source vector, which will
2827 // produce a scalar result. We need to copy to the VGPR result inside the
2828 // waterfall loop.
2829 const bool NeedCopyToVGPR = DstBank == &AMDGPU::VGPRRegBank &&
2830 SrcBank == &AMDGPU::SGPRRegBank;
2831 if (DstRegs.empty()) {
2832 applyDefaultMapping(OpdMapper);
2833
2835
2836 if (NeedCopyToVGPR) {
2837 // We don't want a phi for this temporary reg.
2838 Register TmpReg = MRI.createGenericVirtualRegister(DstTy);
2839 MRI.setRegBank(TmpReg, AMDGPU::SGPRRegBank);
2840 MI.getOperand(0).setReg(TmpReg);
2841 B.setInsertPt(*MI.getParent(), ++MI.getIterator());
2842
2843 // Use a v_mov_b32 here to make the exec dependency explicit.
2844 buildVCopy(B, DstReg, TmpReg);
2845 }
2846
2847 // Re-insert the constant offset add inside the waterfall loop.
2848 if (ShouldMoveIndexIntoLoop)
2849 reinsertVectorIndexAdd(B, MI, 2, ConstOffset);
2850
2851 return;
2852 }
2853
2854 assert(DstTy.getSizeInBits() == 64);
2855
2856 LLT Vec32 = LLT::fixed_vector(2 * SrcTy.getNumElements(), 32);
2857
2858 auto CastSrc = B.buildBitcast(Vec32, SrcReg);
2859 auto One = B.buildConstant(S32, 1);
2860
2861 MachineBasicBlock::iterator MII = MI.getIterator();
2862
2863 // Split the vector index into 32-bit pieces. Prepare to move all of the
2864 // new instructions into a waterfall loop if necessary.
2865 //
2866 // Don't put the bitcast or constant in the loop.
2867 MachineInstrSpan Span(MII, &B.getMBB());
2868
2869 // Compute 32-bit element indices, (2 * OrigIdx, 2 * OrigIdx + 1).
2870 auto IdxLo = B.buildShl(S32, BaseIdxReg, One);
2871 auto IdxHi = B.buildAdd(S32, IdxLo, One);
2872
2873 auto Extract0 = B.buildExtractVectorElement(DstRegs[0], CastSrc, IdxLo);
2874 auto Extract1 = B.buildExtractVectorElement(DstRegs[1], CastSrc, IdxHi);
2875
2876 MRI.setRegBank(DstReg, *DstBank);
2877 MRI.setRegBank(CastSrc.getReg(0), *SrcBank);
2878 MRI.setRegBank(One.getReg(0), AMDGPU::SGPRRegBank);
2879 MRI.setRegBank(IdxLo.getReg(0), AMDGPU::SGPRRegBank);
2880 MRI.setRegBank(IdxHi.getReg(0), AMDGPU::SGPRRegBank);
2881
2882 SmallSet<Register, 4> OpsToWaterfall;
2883 if (!collectWaterfallOperands(OpsToWaterfall, MI, MRI, { 2 })) {
2884 MI.eraseFromParent();
2885 return;
2886 }
2887
2888 // Remove the original instruction to avoid potentially confusing the
2889 // waterfall loop logic.
2890 B.setInstr(*Span.begin());
2891 MI.eraseFromParent();
2892 executeInWaterfallLoop(B, make_range(Span.begin(), Span.end()),
2893 OpsToWaterfall);
2894
2895 if (NeedCopyToVGPR) {
2896 MachineBasicBlock *LoopBB = Extract1->getParent();
2897 Register TmpReg0 = MRI.createGenericVirtualRegister(S32);
2898 Register TmpReg1 = MRI.createGenericVirtualRegister(S32);
2899 MRI.setRegBank(TmpReg0, AMDGPU::SGPRRegBank);
2900 MRI.setRegBank(TmpReg1, AMDGPU::SGPRRegBank);
2901
2902 Extract0->getOperand(0).setReg(TmpReg0);
2903 Extract1->getOperand(0).setReg(TmpReg1);
2904
2905 B.setInsertPt(*LoopBB, ++Extract1->getIterator());
2906
2907 buildVCopy(B, DstRegs[0], TmpReg0);
2908 buildVCopy(B, DstRegs[1], TmpReg1);
2909 }
2910
2911 if (ShouldMoveIndexIntoLoop)
2912 reinsertVectorIndexAdd(B, *IdxLo, 1, ConstOffset);
2913
2914 return;
2915 }
2916 case AMDGPU::G_INSERT_VECTOR_ELT: {
2917 SmallVector<Register, 2> InsRegs(OpdMapper.getVRegs(2));
2918
2919 Register DstReg = MI.getOperand(0).getReg();
2920 LLT VecTy = MRI.getType(DstReg);
2921
2922 assert(OpdMapper.getVRegs(0).empty());
2923 assert(OpdMapper.getVRegs(3).empty());
2924
2925 if (substituteSimpleCopyRegs(OpdMapper, 1))
2926 MRI.setType(MI.getOperand(1).getReg(), VecTy);
2927
2928 if (foldInsertEltToCmpSelect(B, MI, OpdMapper))
2929 return;
2930
2931 const RegisterBank *IdxBank =
2932 OpdMapper.getInstrMapping().getOperandMapping(3).BreakDown[0].RegBank;
2933
2934 Register SrcReg = MI.getOperand(1).getReg();
2935 Register InsReg = MI.getOperand(2).getReg();
2936 LLT InsTy = MRI.getType(InsReg);
2937 (void)InsTy;
2938
2939 Register BaseIdxReg;
2940 unsigned ConstOffset;
2941 std::tie(BaseIdxReg, ConstOffset) =
2942 AMDGPU::getBaseWithConstantOffset(MRI, MI.getOperand(3).getReg());
2943
2944 // See if the index is an add of a constant which will be foldable by moving
2945 // the base register of the index later if this is going to be executed in a
2946 // waterfall loop. This is essentially to reassociate the add of a constant
2947 // with the readfirstlane.
2948 bool ShouldMoveIndexIntoLoop = IdxBank != &AMDGPU::SGPRRegBank &&
2949 ConstOffset > 0 &&
2950 ConstOffset < VecTy.getNumElements();
2951
2952 // Move the base register. We'll re-insert the add later.
2953 if (ShouldMoveIndexIntoLoop)
2954 MI.getOperand(3).setReg(BaseIdxReg);
2955
2956
2957 if (InsRegs.empty()) {
2959
2960 // Re-insert the constant offset add inside the waterfall loop.
2961 if (ShouldMoveIndexIntoLoop) {
2962 reinsertVectorIndexAdd(B, MI, 3, ConstOffset);
2963 }
2964
2965 return;
2966 }
2967
2968 assert(InsTy.getSizeInBits() == 64);
2969
2970 const LLT S32 = LLT::scalar(32);
2971 LLT Vec32 = LLT::fixed_vector(2 * VecTy.getNumElements(), 32);
2972
2973 auto CastSrc = B.buildBitcast(Vec32, SrcReg);
2974 auto One = B.buildConstant(S32, 1);
2975
2976 // Split the vector index into 32-bit pieces. Prepare to move all of the
2977 // new instructions into a waterfall loop if necessary.
2978 //
2979 // Don't put the bitcast or constant in the loop.
2981
2982 // Compute 32-bit element indices, (2 * OrigIdx, 2 * OrigIdx + 1).
2983 auto IdxLo = B.buildShl(S32, BaseIdxReg, One);
2984 auto IdxHi = B.buildAdd(S32, IdxLo, One);
2985
2986 auto InsLo = B.buildInsertVectorElement(Vec32, CastSrc, InsRegs[0], IdxLo);
2987 auto InsHi = B.buildInsertVectorElement(Vec32, InsLo, InsRegs[1], IdxHi);
2988
2989 const RegisterBank *DstBank =
2990 OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
2991 const RegisterBank *SrcBank =
2992 OpdMapper.getInstrMapping().getOperandMapping(1).BreakDown[0].RegBank;
2993 const RegisterBank *InsSrcBank =
2994 OpdMapper.getInstrMapping().getOperandMapping(2).BreakDown[0].RegBank;
2995
2996 MRI.setRegBank(InsReg, *InsSrcBank);
2997 MRI.setRegBank(CastSrc.getReg(0), *SrcBank);
2998 MRI.setRegBank(InsLo.getReg(0), *DstBank);
2999 MRI.setRegBank(InsHi.getReg(0), *DstBank);
3000 MRI.setRegBank(One.getReg(0), AMDGPU::SGPRRegBank);
3001 MRI.setRegBank(IdxLo.getReg(0), AMDGPU::SGPRRegBank);
3002 MRI.setRegBank(IdxHi.getReg(0), AMDGPU::SGPRRegBank);
3003
3004
3005 SmallSet<Register, 4> OpsToWaterfall;
3006 if (!collectWaterfallOperands(OpsToWaterfall, MI, MRI, { 3 })) {
3007 B.setInsertPt(B.getMBB(), MI);
3008 B.buildBitcast(DstReg, InsHi);
3009 MI.eraseFromParent();
3010 return;
3011 }
3012
3013 B.setInstr(*Span.begin());
3014 MI.eraseFromParent();
3015
3016 // Figure out the point after the waterfall loop before mangling the control
3017 // flow.
3018 executeInWaterfallLoop(B, make_range(Span.begin(), Span.end()),
3019 OpsToWaterfall);
3020
3021 // The insertion point is now right after the original instruction.
3022 //
3023 // Keep the bitcast to the original vector type out of the loop. Doing this
3024 // saved an extra phi we don't need inside the loop.
3025 B.buildBitcast(DstReg, InsHi);
3026
3027 // Re-insert the constant offset add inside the waterfall loop.
3028 if (ShouldMoveIndexIntoLoop)
3029 reinsertVectorIndexAdd(B, *IdxLo, 1, ConstOffset);
3030
3031 return;
3032 }
3033 case AMDGPU::G_AMDGPU_BUFFER_LOAD:
3034 case AMDGPU::G_AMDGPU_BUFFER_LOAD_USHORT:
3035 case AMDGPU::G_AMDGPU_BUFFER_LOAD_SSHORT:
3036 case AMDGPU::G_AMDGPU_BUFFER_LOAD_UBYTE:
3037 case AMDGPU::G_AMDGPU_BUFFER_LOAD_SBYTE:
3038 case AMDGPU::G_AMDGPU_BUFFER_LOAD_TFE:
3039 case AMDGPU::G_AMDGPU_BUFFER_LOAD_USHORT_TFE:
3040 case AMDGPU::G_AMDGPU_BUFFER_LOAD_SSHORT_TFE:
3041 case AMDGPU::G_AMDGPU_BUFFER_LOAD_UBYTE_TFE:
3042 case AMDGPU::G_AMDGPU_BUFFER_LOAD_SBYTE_TFE:
3043 case AMDGPU::G_AMDGPU_BUFFER_LOAD_FORMAT:
3044 case AMDGPU::G_AMDGPU_BUFFER_LOAD_FORMAT_TFE:
3045 case AMDGPU::G_AMDGPU_BUFFER_LOAD_FORMAT_D16:
3046 case AMDGPU::G_AMDGPU_BUFFER_LOAD_FORMAT_D16_TFE:
3047 case AMDGPU::G_AMDGPU_TBUFFER_LOAD_FORMAT:
3048 case AMDGPU::G_AMDGPU_TBUFFER_LOAD_FORMAT_D16:
3049 case AMDGPU::G_AMDGPU_BUFFER_STORE:
3050 case AMDGPU::G_AMDGPU_BUFFER_STORE_BYTE:
3051 case AMDGPU::G_AMDGPU_BUFFER_STORE_SHORT:
3052 case AMDGPU::G_AMDGPU_BUFFER_STORE_FORMAT:
3053 case AMDGPU::G_AMDGPU_BUFFER_STORE_FORMAT_D16:
3054 case AMDGPU::G_AMDGPU_TBUFFER_STORE_FORMAT:
3055 case AMDGPU::G_AMDGPU_TBUFFER_STORE_FORMAT_D16: {
3056 applyDefaultMapping(OpdMapper);
3057 executeInWaterfallLoop(B, MI, {1, 4});
3058 return;
3059 }
3060 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_SWAP:
3061 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_ADD:
3062 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_SUB:
3063 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_SMIN:
3064 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_UMIN:
3065 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_SMAX:
3066 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_UMAX:
3067 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_AND:
3068 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_OR:
3069 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_XOR:
3070 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_INC:
3071 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_DEC:
3072 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_SUB_CLAMP_U32:
3073 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_COND_SUB_U32:
3074 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_FADD:
3075 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_FMIN:
3076 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_FMAX: {
3077 applyDefaultMapping(OpdMapper);
3078 executeInWaterfallLoop(B, MI, {2, 5});
3079 return;
3080 }
3081 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_CMPSWAP: {
3082 applyDefaultMapping(OpdMapper);
3083 executeInWaterfallLoop(B, MI, {3, 6});
3084 return;
3085 }
3086 case AMDGPU::G_AMDGPU_S_BUFFER_LOAD:
3087 case AMDGPU::G_AMDGPU_S_BUFFER_LOAD_UBYTE:
3088 case AMDGPU::G_AMDGPU_S_BUFFER_LOAD_SBYTE:
3089 case AMDGPU::G_AMDGPU_S_BUFFER_LOAD_USHORT:
3090 case AMDGPU::G_AMDGPU_S_BUFFER_LOAD_SSHORT: {
3091 applyMappingSBufferLoad(B, OpdMapper);
3092 return;
3093 }
3094 case AMDGPU::G_AMDGPU_S_BUFFER_PREFETCH:
3097 return;
3098 case AMDGPU::G_INTRINSIC:
3099 case AMDGPU::G_INTRINSIC_CONVERGENT: {
3100 switch (cast<GIntrinsic>(MI).getIntrinsicID()) {
3101 case Intrinsic::amdgcn_readlane: {
3102 substituteSimpleCopyRegs(OpdMapper, 2);
3103
3104 assert(OpdMapper.getVRegs(0).empty());
3105 assert(OpdMapper.getVRegs(3).empty());
3106
3107 // Make sure the index is an SGPR. It doesn't make sense to run this in a
3108 // waterfall loop, so assume it's a uniform value.
3109 constrainOpWithReadfirstlane(B, MI, 3); // Index
3110 return;
3111 }
3112 case Intrinsic::amdgcn_writelane: {
3113 assert(OpdMapper.getVRegs(0).empty());
3114 assert(OpdMapper.getVRegs(2).empty());
3115 assert(OpdMapper.getVRegs(3).empty());
3116
3117 substituteSimpleCopyRegs(OpdMapper, 4); // VGPR input val
3118 constrainOpWithReadfirstlane(B, MI, 2); // Source value
3119 constrainOpWithReadfirstlane(B, MI, 3); // Index
3120 return;
3121 }
3122 case Intrinsic::amdgcn_interp_p1:
3123 case Intrinsic::amdgcn_interp_p2:
3124 case Intrinsic::amdgcn_interp_mov:
3125 case Intrinsic::amdgcn_interp_p1_f16:
3126 case Intrinsic::amdgcn_interp_p2_f16:
3127 case Intrinsic::amdgcn_lds_param_load: {
3128 applyDefaultMapping(OpdMapper);
3129
3130 // Readlane for m0 value, which is always the last operand.
3131 // FIXME: Should this be a waterfall loop instead?
3132 constrainOpWithReadfirstlane(B, MI, MI.getNumOperands() - 1); // Index
3133 return;
3134 }
3135 case Intrinsic::amdgcn_interp_inreg_p10:
3136 case Intrinsic::amdgcn_interp_inreg_p2:
3137 case Intrinsic::amdgcn_interp_inreg_p10_f16:
3138 case Intrinsic::amdgcn_interp_inreg_p2_f16:
3139 case Intrinsic::amdgcn_interp_p10_rtz_f16:
3140 case Intrinsic::amdgcn_interp_p2_rtz_f16:
3141 case Intrinsic::amdgcn_permlane16_swap:
3142 case Intrinsic::amdgcn_permlane32_swap:
3143 applyDefaultMapping(OpdMapper);
3144 return;
3145 case Intrinsic::amdgcn_permlane16:
3146 case Intrinsic::amdgcn_permlanex16: {
3147 // Doing a waterfall loop over these wouldn't make any sense.
3148 substituteSimpleCopyRegs(OpdMapper, 2);
3149 substituteSimpleCopyRegs(OpdMapper, 3);
3152 return;
3153 }
3154 case Intrinsic::amdgcn_permlane_bcast:
3155 case Intrinsic::amdgcn_permlane_up:
3156 case Intrinsic::amdgcn_permlane_down:
3157 case Intrinsic::amdgcn_permlane_xor:
3158 // Doing a waterfall loop over these wouldn't make any sense.
3161 return;
3162 case Intrinsic::amdgcn_permlane_idx_gen: {
3164 return;
3165 }
3166 case Intrinsic::amdgcn_sbfe:
3167 applyMappingBFE(B, OpdMapper, true);
3168 return;
3169 case Intrinsic::amdgcn_ubfe:
3170 applyMappingBFE(B, OpdMapper, false);
3171 return;
3172 case Intrinsic::amdgcn_inverse_ballot:
3173 case Intrinsic::amdgcn_s_bitreplicate:
3174 case Intrinsic::amdgcn_s_quadmask:
3175 case Intrinsic::amdgcn_s_wqm:
3176 applyDefaultMapping(OpdMapper);
3177 constrainOpWithReadfirstlane(B, MI, 2); // Mask
3178 return;
3179 case Intrinsic::amdgcn_ballot:
3180 // Use default handling and insert copy to vcc source.
3181 break;
3182 }
3183 break;
3184 }
3185 case AMDGPU::G_AMDGPU_INTRIN_IMAGE_LOAD:
3186 case AMDGPU::G_AMDGPU_INTRIN_IMAGE_LOAD_D16:
3187 case AMDGPU::G_AMDGPU_INTRIN_IMAGE_LOAD_NORET:
3188 case AMDGPU::G_AMDGPU_INTRIN_IMAGE_STORE:
3189 case AMDGPU::G_AMDGPU_INTRIN_IMAGE_STORE_D16: {
3190 const AMDGPU::RsrcIntrinsic *RSrcIntrin =
3192 assert(RSrcIntrin && RSrcIntrin->IsImage);
3193 // Non-images can have complications from operands that allow both SGPR
3194 // and VGPR. For now it's too complicated to figure out the final opcode
3195 // to derive the register bank from the MCInstrDesc.
3196 applyMappingImage(B, MI, OpdMapper, RSrcIntrin->RsrcArg);
3197 return;
3198 }
3199 case AMDGPU::G_AMDGPU_BVH_INTERSECT_RAY:
3200 case AMDGPU::G_AMDGPU_BVH8_INTERSECT_RAY:
3201 case AMDGPU::G_AMDGPU_BVH_DUAL_INTERSECT_RAY: {
3202 bool IsDualOrBVH8 =
3203 MI.getOpcode() == AMDGPU::G_AMDGPU_BVH_DUAL_INTERSECT_RAY ||
3204 MI.getOpcode() == AMDGPU::G_AMDGPU_BVH8_INTERSECT_RAY;
3205 unsigned NumMods = IsDualOrBVH8 ? 0 : 1; // Has A16 modifier
3206 unsigned LastRegOpIdx = MI.getNumExplicitOperands() - 1 - NumMods;
3207 applyDefaultMapping(OpdMapper);
3208 executeInWaterfallLoop(B, MI, {LastRegOpIdx});
3209 return;
3210 }
3211 case AMDGPU::G_INTRINSIC_W_SIDE_EFFECTS:
3212 case AMDGPU::G_INTRINSIC_CONVERGENT_W_SIDE_EFFECTS: {
3213 auto IntrID = cast<GIntrinsic>(MI).getIntrinsicID();
3214 switch (IntrID) {
3215 case Intrinsic::amdgcn_ds_ordered_add:
3216 case Intrinsic::amdgcn_ds_ordered_swap: {
3217 // This is only allowed to execute with 1 lane, so readfirstlane is safe.
3218 assert(OpdMapper.getVRegs(0).empty());
3219 substituteSimpleCopyRegs(OpdMapper, 3);
3221 return;
3222 }
3223 case Intrinsic::amdgcn_ds_gws_init:
3224 case Intrinsic::amdgcn_ds_gws_barrier:
3225 case Intrinsic::amdgcn_ds_gws_sema_br: {
3226 // Only the first lane is executes, so readfirstlane is safe.
3227 substituteSimpleCopyRegs(OpdMapper, 1);
3229 return;
3230 }
3231 case Intrinsic::amdgcn_ds_gws_sema_v:
3232 case Intrinsic::amdgcn_ds_gws_sema_p:
3233 case Intrinsic::amdgcn_ds_gws_sema_release_all: {
3234 // Only the first lane is executes, so readfirstlane is safe.
3236 return;
3237 }
3238 case Intrinsic::amdgcn_ds_append:
3239 case Intrinsic::amdgcn_ds_consume: {
3241 return;
3242 }
3243 case Intrinsic::amdgcn_s_alloc_vgpr:
3245 return;
3246 case Intrinsic::amdgcn_s_sendmsg:
3247 case Intrinsic::amdgcn_s_sendmsghalt: {
3248 // FIXME: Should this use a waterfall loop?
3250 return;
3251 }
3252 case Intrinsic::amdgcn_s_setreg: {
3254 return;
3255 }
3256 case Intrinsic::amdgcn_s_ttracedata:
3258 return;
3259 case Intrinsic::amdgcn_raw_buffer_load_lds:
3260 case Intrinsic::amdgcn_raw_buffer_load_async_lds:
3261 case Intrinsic::amdgcn_raw_ptr_buffer_load_lds:
3262 case Intrinsic::amdgcn_raw_ptr_buffer_load_async_lds: {
3263 applyDefaultMapping(OpdMapper);
3264 constrainOpWithReadfirstlane(B, MI, 1); // rsrc
3266 constrainOpWithReadfirstlane(B, MI, 5); // soffset
3267 return;
3268 }
3269 case Intrinsic::amdgcn_struct_buffer_load_lds:
3270 case Intrinsic::amdgcn_struct_buffer_load_async_lds:
3271 case Intrinsic::amdgcn_struct_ptr_buffer_load_lds:
3272 case Intrinsic::amdgcn_struct_ptr_buffer_load_async_lds: {
3273 applyDefaultMapping(OpdMapper);
3274 constrainOpWithReadfirstlane(B, MI, 1); // rsrc
3276 constrainOpWithReadfirstlane(B, MI, 6); // soffset
3277 return;
3278 }
3279 case Intrinsic::amdgcn_cluster_load_async_to_lds_b8:
3280 case Intrinsic::amdgcn_cluster_load_async_to_lds_b32:
3281 case Intrinsic::amdgcn_cluster_load_async_to_lds_b64:
3282 case Intrinsic::amdgcn_cluster_load_async_to_lds_b128: {
3283 applyDefaultMapping(OpdMapper);
3285 return;
3286 }
3287 case Intrinsic::amdgcn_load_to_lds:
3288 case Intrinsic::amdgcn_load_async_to_lds:
3289 case Intrinsic::amdgcn_global_load_lds:
3290 case Intrinsic::amdgcn_global_load_async_lds: {
3291 applyDefaultMapping(OpdMapper);
3293 return;
3294 }
3295 case Intrinsic::amdgcn_lds_direct_load: {
3296 applyDefaultMapping(OpdMapper);
3297 // Readlane for m0 value, which is always the last operand.
3298 constrainOpWithReadfirstlane(B, MI, MI.getNumOperands() - 1); // Index
3299 return;
3300 }
3301 case Intrinsic::amdgcn_exp_row:
3302 applyDefaultMapping(OpdMapper);
3304 return;
3305 case Intrinsic::amdgcn_cluster_load_b32:
3306 case Intrinsic::amdgcn_cluster_load_b64:
3307 case Intrinsic::amdgcn_cluster_load_b128: {
3308 applyDefaultMapping(OpdMapper);
3310 return;
3311 }
3312 case Intrinsic::amdgcn_s_sleep_var:
3313 assert(OpdMapper.getVRegs(1).empty());
3315 return;
3316 case Intrinsic::amdgcn_s_barrier_join:
3317 case Intrinsic::amdgcn_s_wakeup_barrier:
3319 return;
3320 case Intrinsic::amdgcn_s_barrier_init:
3321 case Intrinsic::amdgcn_s_barrier_signal_var:
3324 return;
3325 case Intrinsic::amdgcn_s_get_barrier_state:
3326 case Intrinsic::amdgcn_s_get_named_barrier_state: {
3328 return;
3329 }
3330 case Intrinsic::amdgcn_s_prefetch_data:
3331 case Intrinsic::amdgcn_s_prefetch_inst: {
3332 Register PtrReg = MI.getOperand(1).getReg();
3333 unsigned AS = MRI.getType(PtrReg).getAddressSpace();
3337 } else
3338 MI.eraseFromParent();
3339 return;
3340 }
3341 case Intrinsic::amdgcn_tensor_load_to_lds:
3342 case Intrinsic::amdgcn_tensor_store_from_lds: {
3348 return;
3349 }
3350 default: {
3351 if (const AMDGPU::RsrcIntrinsic *RSrcIntrin =
3353 // Non-images can have complications from operands that allow both SGPR
3354 // and VGPR. For now it's too complicated to figure out the final opcode
3355 // to derive the register bank from the MCInstrDesc.
3356 if (RSrcIntrin->IsImage) {
3357 applyMappingImage(B, MI, OpdMapper, RSrcIntrin->RsrcArg);
3358 return;
3359 }
3360 }
3361
3362 break;
3363 }
3364 }
3365 break;
3366 }
3367 case AMDGPU::G_SI_CALL: {
3368 // Use a set to avoid extra readfirstlanes in the case where multiple
3369 // operands are the same register.
3370 SmallSet<Register, 4> SGPROperandRegs;
3371
3372 if (!collectWaterfallOperands(SGPROperandRegs, MI, MRI, {1}))
3373 break;
3374
3375 // Move all copies to physical SGPRs that are used by the call instruction
3376 // into the loop block. Start searching for these copies until the
3377 // ADJCALLSTACKUP.
3378 unsigned FrameSetupOpcode = AMDGPU::ADJCALLSTACKUP;
3379 unsigned FrameDestroyOpcode = AMDGPU::ADJCALLSTACKDOWN;
3380
3381 // Move all non-copies before the copies, so that a complete range can be
3382 // moved into the waterfall loop.
3383 SmallVector<MachineInstr *, 4> NonCopyInstrs;
3384 // Count of NonCopyInstrs found until the current LastCopy.
3385 unsigned NonCopyInstrsLen = 0;
3387 MachineBasicBlock::iterator LastCopy = Start;
3388 MachineBasicBlock *MBB = MI.getParent();
3389 const SIMachineFunctionInfo *Info =
3390 MBB->getParent()->getInfo<SIMachineFunctionInfo>();
3391 while (Start->getOpcode() != FrameSetupOpcode) {
3392 --Start;
3393 bool IsCopy = false;
3394 if (Start->getOpcode() == AMDGPU::COPY) {
3395 auto &Dst = Start->getOperand(0);
3396 if (Dst.isReg()) {
3397 Register Reg = Dst.getReg();
3398 if (Reg.isPhysical() && MI.readsRegister(Reg, TRI)) {
3399 IsCopy = true;
3400 } else {
3401 // Also move the copy from the scratch rsrc descriptor into the loop
3402 // to allow it to be optimized away.
3403 auto &Src = Start->getOperand(1);
3404 if (Src.isReg()) {
3405 Reg = Src.getReg();
3406 IsCopy = Info->getScratchRSrcReg() == Reg;
3407 }
3408 }
3409 }
3410 }
3411
3412 if (IsCopy) {
3413 LastCopy = Start;
3414 NonCopyInstrsLen = NonCopyInstrs.size();
3415 } else {
3416 NonCopyInstrs.push_back(&*Start);
3417 }
3418 }
3419 NonCopyInstrs.resize(NonCopyInstrsLen);
3420
3421 for (auto *NonCopy : reverse(NonCopyInstrs)) {
3422 MBB->splice(LastCopy, MBB, NonCopy->getIterator());
3423 }
3424 Start = LastCopy;
3425
3426 // Do the same for copies after the loop
3427 NonCopyInstrs.clear();
3428 NonCopyInstrsLen = 0;
3430 LastCopy = End;
3431 while (End->getOpcode() != FrameDestroyOpcode) {
3432 ++End;
3433 bool IsCopy = false;
3434 if (End->getOpcode() == AMDGPU::COPY) {
3435 auto &Src = End->getOperand(1);
3436 if (Src.isReg()) {
3437 Register Reg = Src.getReg();
3438 IsCopy = Reg.isPhysical() && MI.modifiesRegister(Reg, TRI);
3439 }
3440 }
3441
3442 if (IsCopy) {
3443 LastCopy = End;
3444 NonCopyInstrsLen = NonCopyInstrs.size();
3445 } else {
3446 NonCopyInstrs.push_back(&*End);
3447 }
3448 }
3449 NonCopyInstrs.resize(NonCopyInstrsLen);
3450
3451 End = LastCopy;
3452 ++LastCopy;
3453 for (auto *NonCopy : reverse(NonCopyInstrs)) {
3454 MBB->splice(LastCopy, MBB, NonCopy->getIterator());
3455 }
3456
3457 ++End;
3458 B.setInsertPt(B.getMBB(), Start);
3459 executeInWaterfallLoop(B, make_range(Start, End), SGPROperandRegs);
3460 break;
3461 }
3462 case AMDGPU::G_AMDGPU_FLAT_LOAD_MONITOR:
3463 case AMDGPU::G_AMDGPU_GLOBAL_LOAD_MONITOR:
3464 case AMDGPU::G_LOAD:
3465 case AMDGPU::G_ZEXTLOAD:
3466 case AMDGPU::G_SEXTLOAD: {
3467 if (applyMappingLoad(B, OpdMapper, MI))
3468 return;
3469 break;
3470 }
3471 case AMDGPU::G_DYN_STACKALLOC:
3472 applyMappingDynStackAlloc(B, OpdMapper, MI);
3473 return;
3474 case AMDGPU::G_STACKRESTORE: {
3475 applyDefaultMapping(OpdMapper);
3477 return;
3478 }
3479 case AMDGPU::G_SBFX:
3480 applyMappingBFE(B, OpdMapper, /*Signed*/ true);
3481 return;
3482 case AMDGPU::G_UBFX:
3483 applyMappingBFE(B, OpdMapper, /*Signed*/ false);
3484 return;
3485 case AMDGPU::G_AMDGPU_MAD_U64_U32:
3486 case AMDGPU::G_AMDGPU_MAD_I64_I32:
3487 applyMappingMAD_64_32(B, OpdMapper);
3488 return;
3489 case AMDGPU::G_PREFETCH: {
3490 if (!Subtarget.hasSafeSmemPrefetch() && !Subtarget.hasVmemPrefInsts()) {
3491 MI.eraseFromParent();
3492 return;
3493 }
3494 Register PtrReg = MI.getOperand(0).getReg();
3495 unsigned PtrBank = getRegBankID(PtrReg, MRI, AMDGPU::SGPRRegBankID);
3496 if (PtrBank == AMDGPU::VGPRRegBankID &&
3497 (!Subtarget.hasVmemPrefInsts() || !MI.getOperand(3).getImm())) {
3498 // Cannot do I$ prefetch with divergent pointer.
3499 MI.eraseFromParent();
3500 return;
3501 }
3502 unsigned AS = MRI.getType(PtrReg).getAddressSpace();
3505 (!Subtarget.hasSafeSmemPrefetch() &&
3507 !MI.getOperand(3).getImm() /* I$ prefetch */))) {
3508 MI.eraseFromParent();
3509 return;
3510 }
3511 applyDefaultMapping(OpdMapper);
3512 return;
3513 }
3514 default:
3515 break;
3516 }
3517
3518 return applyDefaultMapping(OpdMapper);
3519}
3520
3521// vgpr, sgpr -> vgpr
3522// vgpr, agpr -> vgpr
3523// agpr, agpr -> agpr
3524// agpr, sgpr -> vgpr
3525static unsigned regBankUnion(unsigned RB0, unsigned RB1) {
3526 if (RB0 == AMDGPU::InvalidRegBankID)
3527 return RB1;
3528 if (RB1 == AMDGPU::InvalidRegBankID)
3529 return RB0;
3530
3531 if (RB0 == AMDGPU::SGPRRegBankID && RB1 == AMDGPU::SGPRRegBankID)
3532 return AMDGPU::SGPRRegBankID;
3533
3534 if (RB0 == AMDGPU::AGPRRegBankID && RB1 == AMDGPU::AGPRRegBankID)
3535 return AMDGPU::AGPRRegBankID;
3536
3537 return AMDGPU::VGPRRegBankID;
3538}
3539
3540static unsigned regBankBoolUnion(unsigned RB0, unsigned RB1) {
3541 if (RB0 == AMDGPU::InvalidRegBankID)
3542 return RB1;
3543 if (RB1 == AMDGPU::InvalidRegBankID)
3544 return RB0;
3545
3546 // vcc, vcc -> vcc
3547 // vcc, sgpr -> vcc
3548 // vcc, vgpr -> vcc
3549 if (RB0 == AMDGPU::VCCRegBankID || RB1 == AMDGPU::VCCRegBankID)
3550 return AMDGPU::VCCRegBankID;
3551
3552 // vcc, vgpr -> vgpr
3553 return regBankUnion(RB0, RB1);
3554}
3555
3557 const MachineInstr &MI) const {
3558 unsigned RegBank = AMDGPU::InvalidRegBankID;
3559
3560 for (const MachineOperand &MO : MI.operands()) {
3561 if (!MO.isReg())
3562 continue;
3563 Register Reg = MO.getReg();
3564 if (const RegisterBank *Bank = getRegBank(Reg, MRI, *TRI)) {
3565 RegBank = regBankUnion(RegBank, Bank->getID());
3566 if (RegBank == AMDGPU::VGPRRegBankID)
3567 break;
3568 }
3569 }
3570
3571 return RegBank;
3572}
3573
3575 const MachineFunction &MF = *MI.getMF();
3576 const MachineRegisterInfo &MRI = MF.getRegInfo();
3577 for (const MachineOperand &MO : MI.operands()) {
3578 if (!MO.isReg())
3579 continue;
3580 Register Reg = MO.getReg();
3581 if (const RegisterBank *Bank = getRegBank(Reg, MRI, *TRI)) {
3582 if (Bank->getID() != AMDGPU::SGPRRegBankID)
3583 return false;
3584 }
3585 }
3586 return true;
3587}
3588
3591 const MachineFunction &MF = *MI.getMF();
3592 const MachineRegisterInfo &MRI = MF.getRegInfo();
3593 SmallVector<const ValueMapping*, 8> OpdsMapping(MI.getNumOperands());
3594
3595 for (unsigned i = 0, e = MI.getNumOperands(); i != e; ++i) {
3596 const MachineOperand &SrcOp = MI.getOperand(i);
3597 if (!SrcOp.isReg())
3598 continue;
3599
3600 unsigned Size = TRI->getRegSizeInBits(SrcOp.getReg(), MRI);
3601 OpdsMapping[i] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size);
3602 }
3603 return getInstructionMapping(1, 1, getOperandsMapping(OpdsMapping),
3604 MI.getNumOperands());
3605}
3606
3609 const MachineFunction &MF = *MI.getMF();
3610 const MachineRegisterInfo &MRI = MF.getRegInfo();
3611 SmallVector<const ValueMapping*, 8> OpdsMapping(MI.getNumOperands());
3612
3613 // Even though we technically could use SGPRs, this would require knowledge of
3614 // the constant bus restriction. Force all sources to VGPR (except for VCC).
3615 //
3616 // TODO: Unary ops are trivially OK, so accept SGPRs?
3617 for (unsigned i = 0, e = MI.getNumOperands(); i != e; ++i) {
3618 const MachineOperand &Src = MI.getOperand(i);
3619 if (!Src.isReg())
3620 continue;
3621
3622 unsigned Size = TRI->getRegSizeInBits(Src.getReg(), MRI);
3623 unsigned BankID = Size == 1 ? AMDGPU::VCCRegBankID : AMDGPU::VGPRRegBankID;
3624 OpdsMapping[i] = AMDGPU::getValueMapping(BankID, Size);
3625 }
3626
3627 return getInstructionMapping(1, 1, getOperandsMapping(OpdsMapping),
3628 MI.getNumOperands());
3629}
3630
3633 const MachineFunction &MF = *MI.getMF();
3634 const MachineRegisterInfo &MRI = MF.getRegInfo();
3635 SmallVector<const ValueMapping*, 8> OpdsMapping(MI.getNumOperands());
3636
3637 for (unsigned I = 0, E = MI.getNumOperands(); I != E; ++I) {
3638 const MachineOperand &Op = MI.getOperand(I);
3639 if (!Op.isReg())
3640 continue;
3641
3642 unsigned Size = TRI->getRegSizeInBits(Op.getReg(), MRI);
3643 OpdsMapping[I] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
3644 }
3645
3646 return getInstructionMapping(1, 1, getOperandsMapping(OpdsMapping),
3647 MI.getNumOperands());
3648}
3649
3652 const MachineInstr &MI,
3653 int RsrcIdx) const {
3654 // The reported argument index is relative to the IR intrinsic call arguments,
3655 // so we need to shift by the number of defs and the intrinsic ID.
3656 RsrcIdx += MI.getNumExplicitDefs() + 1;
3657
3658 const int NumOps = MI.getNumOperands();
3660
3661 // TODO: Should packed/unpacked D16 difference be reported here as part of
3662 // the value mapping?
3663 for (int I = 0; I != NumOps; ++I) {
3664 if (!MI.getOperand(I).isReg())
3665 continue;
3666
3667 Register OpReg = MI.getOperand(I).getReg();
3668 // We replace some dead address operands with $noreg
3669 if (!OpReg)
3670 continue;
3671
3672 unsigned Size = TRI->getRegSizeInBits(OpReg, MRI);
3673
3674 // FIXME: Probably need a new intrinsic register bank searchable table to
3675 // handle arbitrary intrinsics easily.
3676 //
3677 // If this has a sampler, it immediately follows rsrc.
3678 const bool MustBeSGPR = I == RsrcIdx || I == RsrcIdx + 1;
3679
3680 if (MustBeSGPR) {
3681 // If this must be an SGPR, so we must report whatever it is as legal.
3682 unsigned NewBank = getRegBankID(OpReg, MRI, AMDGPU::SGPRRegBankID);
3683 OpdsMapping[I] = AMDGPU::getValueMapping(NewBank, Size);
3684 } else {
3685 // Some operands must be VGPR, and these are easy to copy to.
3686 OpdsMapping[I] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
3687 }
3688 }
3689
3690 return getInstructionMapping(1, 1, getOperandsMapping(OpdsMapping), NumOps);
3691}
3692
3693/// Return the mapping for a pointer argument.
3696 Register PtrReg) const {
3697 LLT PtrTy = MRI.getType(PtrReg);
3698 unsigned Size = PtrTy.getSizeInBits();
3699 if (Subtarget.useFlatForGlobal() ||
3701 return AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
3702
3703 // If we're using MUBUF instructions for global memory, an SGPR base register
3704 // is possible. Otherwise this needs to be a VGPR.
3705 const RegisterBank *PtrBank = getRegBank(PtrReg, MRI, *TRI);
3706 return AMDGPU::getValueMapping(PtrBank->getID(), Size);
3707}
3708
3711
3712 const MachineFunction &MF = *MI.getMF();
3713 const MachineRegisterInfo &MRI = MF.getRegInfo();
3715 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
3716 Register PtrReg = MI.getOperand(1).getReg();
3717 LLT PtrTy = MRI.getType(PtrReg);
3718 unsigned AS = PtrTy.getAddressSpace();
3719 unsigned PtrSize = PtrTy.getSizeInBits();
3720
3721 const ValueMapping *ValMapping;
3722 const ValueMapping *PtrMapping;
3723
3724 const RegisterBank *PtrBank = getRegBank(PtrReg, MRI, *TRI);
3725
3726 if (PtrBank == &AMDGPU::SGPRRegBank && AMDGPU::isFlatGlobalAddrSpace(AS)) {
3727 if (isScalarLoadLegal(MI)) {
3728 // We have a uniform instruction so we want to use an SMRD load
3729 ValMapping = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size);
3730 PtrMapping = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, PtrSize);
3731 } else {
3732 ValMapping = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
3733
3734 // If we're using MUBUF instructions for global memory, an SGPR base
3735 // register is possible. Otherwise this needs to be a VGPR.
3736 unsigned PtrBankID = Subtarget.useFlatForGlobal() ?
3737 AMDGPU::VGPRRegBankID : AMDGPU::SGPRRegBankID;
3738
3739 PtrMapping = AMDGPU::getValueMapping(PtrBankID, PtrSize);
3740 }
3741 } else {
3742 ValMapping = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
3743 PtrMapping = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, PtrSize);
3744 }
3745
3746 OpdsMapping[0] = ValMapping;
3747 OpdsMapping[1] = PtrMapping;
3749 1, 1, getOperandsMapping(OpdsMapping), MI.getNumOperands());
3750 return Mapping;
3751
3752 // FIXME: Do we want to add a mapping for FLAT load, or should we just
3753 // handle that during instruction selection?
3754}
3755
3756unsigned
3758 const MachineRegisterInfo &MRI,
3759 unsigned Default) const {
3760 const RegisterBank *Bank = getRegBank(Reg, MRI, *TRI);
3761 return Bank ? Bank->getID() : Default;
3762}
3763
3766 const MachineRegisterInfo &MRI,
3767 const TargetRegisterInfo &TRI) const {
3768 // Lie and claim anything is legal, even though this needs to be an SGPR
3769 // applyMapping will have to deal with it as a waterfall loop.
3770 unsigned Bank = getRegBankID(Reg, MRI, AMDGPU::SGPRRegBankID);
3771 unsigned Size = TRI.getRegSizeInBits(Reg, MRI);
3772 return AMDGPU::getValueMapping(Bank, Size);
3773}
3774
3777 const MachineRegisterInfo &MRI,
3778 const TargetRegisterInfo &TRI) const {
3779 unsigned Size = TRI.getRegSizeInBits(Reg, MRI);
3780 return AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
3781}
3782
3785 const MachineRegisterInfo &MRI,
3786 const TargetRegisterInfo &TRI) const {
3787 unsigned Size = TRI.getRegSizeInBits(Reg, MRI);
3788 return AMDGPU::getValueMapping(AMDGPU::AGPRRegBankID, Size);
3789}
3790
3791///
3792/// This function must return a legal mapping, because
3793/// AMDGPURegisterBankInfo::getInstrAlternativeMappings() is not called
3794/// in RegBankSelect::Mode::Fast. Any mapping that would cause a
3795/// VGPR to SGPR generated is illegal.
3796///
3797// Operands that must be SGPRs must accept potentially divergent VGPRs as
3798// legal. These will be dealt with in applyMappingImpl.
3799//
3802 const MachineFunction &MF = *MI.getMF();
3803 const MachineRegisterInfo &MRI = MF.getRegInfo();
3804
3805 if (MI.isCopy() || MI.getOpcode() == AMDGPU::G_FREEZE) {
3806 Register DstReg = MI.getOperand(0).getReg();
3807 Register SrcReg = MI.getOperand(1).getReg();
3808
3809 // The default logic bothers to analyze impossible alternative mappings. We
3810 // want the most straightforward mapping, so just directly handle this.
3811 const RegisterBank *DstBank = getRegBank(DstReg, MRI, *TRI);
3812 const RegisterBank *SrcBank = getRegBank(SrcReg, MRI, *TRI);
3813
3814 // For COPY between a physical reg and an s1, there is no type associated so
3815 // we need to take the virtual register's type as a hint on how to interpret
3816 // s1 values.
3817 unsigned Size;
3818 if (!SrcReg.isVirtual() && !DstBank &&
3819 MRI.getType(DstReg) == LLT::scalar(1)) {
3820 DstBank = &AMDGPU::VCCRegBank;
3821 Size = 1;
3822 } else if (!DstReg.isVirtual() && MRI.getType(SrcReg) == LLT::scalar(1)) {
3823 DstBank = &AMDGPU::VCCRegBank;
3824 Size = 1;
3825 } else {
3826 Size = TRI->getRegSizeInBits(DstReg, MRI);
3827 }
3828
3829 if (!DstBank)
3830 DstBank = SrcBank;
3831 else if (!SrcBank)
3832 SrcBank = DstBank;
3833
3834 if (MI.getOpcode() != AMDGPU::G_FREEZE &&
3835 cannotCopy(*DstBank, *SrcBank, TypeSize::getFixed(Size)))
3837
3838 const ValueMapping &ValMap = getValueMapping(0, Size, *DstBank);
3839 unsigned OpdsMappingSize = MI.isCopy() ? 1 : 2;
3840 SmallVector<const ValueMapping *, 1> OpdsMapping(OpdsMappingSize);
3841 OpdsMapping[0] = &ValMap;
3842 if (MI.getOpcode() == AMDGPU::G_FREEZE)
3843 OpdsMapping[1] = &ValMap;
3844
3845 return getInstructionMapping(
3846 1, /*Cost*/ 1,
3847 /*OperandsMapping*/ getOperandsMapping(OpdsMapping), OpdsMappingSize);
3848 }
3849
3850 if (MI.isRegSequence()) {
3851 // If any input is a VGPR, the result must be a VGPR. The default handling
3852 // assumes any copy between banks is legal.
3853 unsigned BankID = AMDGPU::SGPRRegBankID;
3854
3855 for (unsigned I = 1, E = MI.getNumOperands(); I != E; I += 2) {
3856 auto OpBank = getRegBankID(MI.getOperand(I).getReg(), MRI);
3857 // It doesn't make sense to use vcc or scc banks here, so just ignore
3858 // them.
3859 if (OpBank != AMDGPU::SGPRRegBankID) {
3860 BankID = AMDGPU::VGPRRegBankID;
3861 break;
3862 }
3863 }
3864 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
3865
3866 const ValueMapping &ValMap = getValueMapping(0, Size, getRegBank(BankID));
3867 return getInstructionMapping(
3868 1, /*Cost*/ 1,
3869 /*OperandsMapping*/ getOperandsMapping({&ValMap}), 1);
3870 }
3871
3872 // The default handling is broken and doesn't handle illegal SGPR->VGPR copies
3873 // properly.
3874 //
3875 // TODO: There are additional exec masking dependencies to analyze.
3876 if (auto *PHI = dyn_cast<GPhi>(&MI)) {
3877 unsigned ResultBank = AMDGPU::InvalidRegBankID;
3878 Register DstReg = PHI->getReg(0);
3879
3880 // Sometimes the result may have already been assigned a bank.
3881 if (const RegisterBank *DstBank = getRegBank(DstReg, MRI, *TRI))
3882 ResultBank = DstBank->getID();
3883
3884 for (unsigned I = 0; I < PHI->getNumIncomingValues(); ++I) {
3885 Register Reg = PHI->getIncomingValue(I);
3886 const RegisterBank *Bank = getRegBank(Reg, MRI, *TRI);
3887
3888 // FIXME: Assuming VGPR for any undetermined inputs.
3889 if (!Bank || Bank->getID() == AMDGPU::VGPRRegBankID) {
3890 ResultBank = AMDGPU::VGPRRegBankID;
3891 break;
3892 }
3893
3894 // FIXME: Need to promote SGPR case to s32
3895 unsigned OpBank = Bank->getID();
3896 ResultBank = regBankBoolUnion(ResultBank, OpBank);
3897 }
3898
3899 assert(ResultBank != AMDGPU::InvalidRegBankID);
3900
3901 unsigned Size = MRI.getType(DstReg).getSizeInBits();
3902
3903 const ValueMapping &ValMap =
3904 getValueMapping(0, Size, getRegBank(ResultBank));
3905 return getInstructionMapping(
3906 1, /*Cost*/ 1,
3907 /*OperandsMapping*/ getOperandsMapping({&ValMap}), 1);
3908 }
3909
3911 if (Mapping.isValid())
3912 return Mapping;
3913
3914 SmallVector<const ValueMapping*, 8> OpdsMapping(MI.getNumOperands());
3915
3916 switch (MI.getOpcode()) {
3917 default:
3919
3920 case AMDGPU::G_AND:
3921 case AMDGPU::G_OR:
3922 case AMDGPU::G_XOR:
3923 case AMDGPU::G_MUL: {
3924 unsigned Size = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
3925 if (Size == 1) {
3926 const RegisterBank *DstBank
3927 = getRegBank(MI.getOperand(0).getReg(), MRI, *TRI);
3928
3929 unsigned TargetBankID = AMDGPU::InvalidRegBankID;
3930 unsigned BankLHS = AMDGPU::InvalidRegBankID;
3931 unsigned BankRHS = AMDGPU::InvalidRegBankID;
3932 if (DstBank) {
3933 TargetBankID = DstBank->getID();
3934 if (DstBank == &AMDGPU::VCCRegBank) {
3935 TargetBankID = AMDGPU::VCCRegBankID;
3936 BankLHS = AMDGPU::VCCRegBankID;
3937 BankRHS = AMDGPU::VCCRegBankID;
3938 } else {
3939 BankLHS = getRegBankID(MI.getOperand(1).getReg(), MRI,
3940 AMDGPU::SGPRRegBankID);
3941 BankRHS = getRegBankID(MI.getOperand(2).getReg(), MRI,
3942 AMDGPU::SGPRRegBankID);
3943 }
3944 } else {
3945 BankLHS = getRegBankID(MI.getOperand(1).getReg(), MRI,
3946 AMDGPU::VCCRegBankID);
3947 BankRHS = getRegBankID(MI.getOperand(2).getReg(), MRI,
3948 AMDGPU::VCCRegBankID);
3949
3950 // Both inputs should be true booleans to produce a boolean result.
3951 if (BankLHS == AMDGPU::VGPRRegBankID || BankRHS == AMDGPU::VGPRRegBankID) {
3952 TargetBankID = AMDGPU::VGPRRegBankID;
3953 } else if (BankLHS == AMDGPU::VCCRegBankID || BankRHS == AMDGPU::VCCRegBankID) {
3954 TargetBankID = AMDGPU::VCCRegBankID;
3955 BankLHS = AMDGPU::VCCRegBankID;
3956 BankRHS = AMDGPU::VCCRegBankID;
3957 } else if (BankLHS == AMDGPU::SGPRRegBankID && BankRHS == AMDGPU::SGPRRegBankID) {
3958 TargetBankID = AMDGPU::SGPRRegBankID;
3959 }
3960 }
3961
3962 OpdsMapping[0] = AMDGPU::getValueMapping(TargetBankID, Size);
3963 OpdsMapping[1] = AMDGPU::getValueMapping(BankLHS, Size);
3964 OpdsMapping[2] = AMDGPU::getValueMapping(BankRHS, Size);
3965 break;
3966 }
3967
3968 if (Size == 64) {
3969
3970 if (isSALUMapping(MI)) {
3971 OpdsMapping[0] = getValueMappingSGPR64Only(AMDGPU::SGPRRegBankID, Size);
3972 OpdsMapping[1] = OpdsMapping[2] = OpdsMapping[0];
3973 } else {
3974 if (MI.getOpcode() == AMDGPU::G_MUL && Subtarget.useVMulU64Inst())
3975 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
3976 else
3977 OpdsMapping[0] =
3978 getValueMappingSGPR64Only(AMDGPU::VGPRRegBankID, Size);
3979 unsigned Bank1 = getRegBankID(MI.getOperand(1).getReg(), MRI /*, DefaultBankID*/);
3980 OpdsMapping[1] = AMDGPU::getValueMapping(Bank1, Size);
3981
3982 unsigned Bank2 = getRegBankID(MI.getOperand(2).getReg(), MRI /*, DefaultBankID*/);
3983 OpdsMapping[2] = AMDGPU::getValueMapping(Bank2, Size);
3984 }
3985
3986 break;
3987 }
3988
3989 [[fallthrough]];
3990 }
3991 case AMDGPU::G_PTR_ADD:
3992 case AMDGPU::G_PTRMASK:
3993 case AMDGPU::G_ADD:
3994 case AMDGPU::G_SUB:
3995 case AMDGPU::G_SHL:
3996 case AMDGPU::G_LSHR:
3997 case AMDGPU::G_ASHR:
3998 case AMDGPU::G_UADDO:
3999 case AMDGPU::G_USUBO:
4000 case AMDGPU::G_UADDE:
4001 case AMDGPU::G_SADDE:
4002 case AMDGPU::G_USUBE:
4003 case AMDGPU::G_SSUBE:
4004 case AMDGPU::G_ABS:
4005 case AMDGPU::G_SHUFFLE_VECTOR:
4006 case AMDGPU::G_SBFX:
4007 case AMDGPU::G_UBFX:
4008 case AMDGPU::G_AMDGPU_S_MUL_I64_I32:
4009 case AMDGPU::G_AMDGPU_S_MUL_U64_U32:
4010 if (isSALUMapping(MI)) {
4011 LLT Ty = MRI.getType(MI.getOperand(0).getReg());
4012 unsigned Size = Ty.getSizeInBits();
4013 // Packed add and sub are VALU only.
4014 if (Subtarget.hasAnyPackedU64Ops() && Ty.isVector() && Size == 128)
4015 return getDefaultMappingVOP(MI);
4016 return getDefaultMappingSOP(MI);
4017 }
4018 return getDefaultMappingVOP(MI);
4019 case AMDGPU::G_SMIN:
4020 case AMDGPU::G_SMAX:
4021 case AMDGPU::G_UMIN:
4022 case AMDGPU::G_UMAX:
4023 if (isSALUMapping(MI)) {
4024 // There are no scalar 64-bit min and max, use vector instruction instead.
4025 if (MRI.getType(MI.getOperand(0).getReg()).getSizeInBits() == 64 &&
4026 Subtarget.useMinMaxI64Insts())
4027 return getDefaultMappingVOP(MI);
4028 return getDefaultMappingSOP(MI);
4029 }
4030 return getDefaultMappingVOP(MI);
4031 case AMDGPU::G_FADD:
4032 case AMDGPU::G_FSUB:
4033 case AMDGPU::G_FMUL:
4034 case AMDGPU::G_FMA:
4035 case AMDGPU::G_FFLOOR:
4036 case AMDGPU::G_FCEIL:
4037 case AMDGPU::G_INTRINSIC_ROUNDEVEN:
4038 case AMDGPU::G_INTRINSIC_TRUNC:
4039 case AMDGPU::G_STRICT_FADD:
4040 case AMDGPU::G_STRICT_FSUB:
4041 case AMDGPU::G_STRICT_FMUL:
4042 case AMDGPU::G_STRICT_FMA: {
4043 LLT Ty = MRI.getType(MI.getOperand(0).getReg());
4044 unsigned Size = Ty.getSizeInBits();
4045 if (Subtarget.hasSALUFloatInsts() && Ty.isScalar() &&
4046 (Size == 32 || Size == 16) && isSALUMapping(MI))
4047 return getDefaultMappingSOP(MI);
4048 return getDefaultMappingVOP(MI);
4049 }
4050 case AMDGPU::G_FMINNUM:
4051 case AMDGPU::G_FMAXNUM:
4052 case AMDGPU::G_FMINIMUMNUM:
4053 case AMDGPU::G_FMAXIMUMNUM: {
4054 LLT Ty = MRI.getType(MI.getOperand(0).getReg());
4055 unsigned Size = Ty.getSizeInBits();
4056 // This case is for GFX11.7, which has the following features:
4057 // 1. `hasIEEEMinimumMaximumInsts()` returns true
4058 // 2. `hasSALUMinimumMaximumInsts()` returns false
4059 // This indicates that GFX11.7 is using:
4060 // 1. `s_max/min_f32/f16` for scalar
4061 // 2. `v_max/min_num_f32/f16` for vector
4062 // By testing on real card, `s_max/min_f32/f16` has the same behavior
4063 // as `v_max/min_num_f32/f16`. So for GFX11.7, even if
4064 // `hasSALUMinimumMaximumInsts()` returns false, we can still choose SGPR
4066 bool ScalarNumIsSafe =
4067 !MFI->getMode().IEEE || Subtarget.hasIEEEMinimumMaximumInsts();
4068 if (Subtarget.hasSALUFloatInsts() && Ty.isScalar() &&
4069 (Size == 32 || Size == 16) && isSALUMapping(MI) && ScalarNumIsSafe)
4070 return getDefaultMappingSOP(MI);
4071 return getDefaultMappingVOP(MI);
4072 }
4073 case AMDGPU::G_FMINIMUM:
4074 case AMDGPU::G_FMAXIMUM: {
4075 LLT Ty = MRI.getType(MI.getOperand(0).getReg());
4076 unsigned Size = Ty.getSizeInBits();
4077 if (Subtarget.hasSALUMinimumMaximumInsts() && Ty.isScalar() &&
4078 (Size == 32 || Size == 16) && isSALUMapping(MI))
4079 return getDefaultMappingSOP(MI);
4080 return getDefaultMappingVOP(MI);
4081 }
4082 case AMDGPU::G_FPTOSI:
4083 case AMDGPU::G_FPTOUI:
4084 case AMDGPU::G_FPTOSI_SAT:
4085 case AMDGPU::G_FPTOUI_SAT:
4086 case AMDGPU::G_SITOFP:
4087 case AMDGPU::G_UITOFP: {
4088 unsigned SizeDst = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4089 unsigned SizeSrc = MRI.getType(MI.getOperand(1).getReg()).getSizeInBits();
4090 if (Subtarget.hasSALUFloatInsts() && SizeDst == 32 && SizeSrc == 32 &&
4092 return getDefaultMappingSOP(MI);
4093 return getDefaultMappingVOP(MI);
4094 }
4095 case AMDGPU::G_FPTRUNC:
4096 case AMDGPU::G_FPEXT: {
4097 unsigned SizeDst = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4098 unsigned SizeSrc = MRI.getType(MI.getOperand(1).getReg()).getSizeInBits();
4099 if (Subtarget.hasSALUFloatInsts() && SizeDst != 64 && SizeSrc != 64 &&
4101 return getDefaultMappingSOP(MI);
4102 return getDefaultMappingVOP(MI);
4103 }
4104 case AMDGPU::G_FSQRT:
4105 case AMDGPU::G_FEXP2:
4106 case AMDGPU::G_FLOG2: {
4107 unsigned Size = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4108 if (Subtarget.hasPseudoScalarTrans() && (Size == 16 || Size == 32) &&
4110 return getDefaultMappingSOP(MI);
4111 return getDefaultMappingVOP(MI);
4112 }
4113 case AMDGPU::G_SADDSAT: // FIXME: Could lower sat ops for SALU
4114 case AMDGPU::G_SSUBSAT:
4115 case AMDGPU::G_UADDSAT:
4116 case AMDGPU::G_USUBSAT:
4117 case AMDGPU::G_FMAD:
4118 case AMDGPU::G_FLDEXP:
4119 case AMDGPU::G_FMINNUM_IEEE:
4120 case AMDGPU::G_FMAXNUM_IEEE:
4121 case AMDGPU::G_FCANONICALIZE:
4122 case AMDGPU::G_STRICT_FLDEXP:
4123 case AMDGPU::G_BSWAP: // TODO: Somehow expand for scalar?
4124 case AMDGPU::G_FSHR: // TODO: Expand for scalar
4125 case AMDGPU::G_AMDGPU_FMIN_LEGACY:
4126 case AMDGPU::G_AMDGPU_FMAX_LEGACY:
4127 case AMDGPU::G_AMDGPU_RCP_IFLAG:
4128 case AMDGPU::G_AMDGPU_CVT_F32_UBYTE0:
4129 case AMDGPU::G_AMDGPU_CVT_F32_UBYTE1:
4130 case AMDGPU::G_AMDGPU_CVT_F32_UBYTE2:
4131 case AMDGPU::G_AMDGPU_CVT_F32_UBYTE3:
4132 case AMDGPU::G_AMDGPU_CVT_PK_I16_I32:
4133 case AMDGPU::G_AMDGPU_SMED3:
4134 case AMDGPU::G_AMDGPU_FMED3:
4135 return getDefaultMappingVOP(MI);
4136 case AMDGPU::G_UMULH:
4137 case AMDGPU::G_SMULH: {
4138 if (Subtarget.hasScalarMulHiInsts() && isSALUMapping(MI))
4139 return getDefaultMappingSOP(MI);
4140 return getDefaultMappingVOP(MI);
4141 }
4142 case AMDGPU::G_AMDGPU_MAD_U64_U32:
4143 case AMDGPU::G_AMDGPU_MAD_I64_I32: {
4144 // Three possible mappings:
4145 //
4146 // - Default SOP
4147 // - Default VOP
4148 // - Scalar multiply: src0 and src1 are SGPRs, the rest is VOP.
4149 //
4150 // This allows instruction selection to keep the multiplication part of the
4151 // instruction on the SALU.
4152 bool AllSalu = true;
4153 bool MulSalu = true;
4154 for (unsigned i = 0; i < 5; ++i) {
4155 Register Reg = MI.getOperand(i).getReg();
4156 if (const RegisterBank *Bank = getRegBank(Reg, MRI, *TRI)) {
4157 if (Bank->getID() != AMDGPU::SGPRRegBankID) {
4158 AllSalu = false;
4159 if (i == 2 || i == 3) {
4160 MulSalu = false;
4161 break;
4162 }
4163 }
4164 }
4165 }
4166
4167 if (AllSalu)
4168 return getDefaultMappingSOP(MI);
4169
4170 // If the multiply-add is full-rate in VALU, use that even if the
4171 // multiplication part is scalar. Accumulating separately on the VALU would
4172 // take two instructions.
4173 if (!MulSalu || Subtarget.hasFullRate64Ops())
4174 return getDefaultMappingVOP(MI);
4175
4176 // Keep the multiplication on the SALU, then accumulate on the VALU.
4177 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 64);
4178 OpdsMapping[1] = AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, 1);
4179 OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 32);
4180 OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 32);
4181 OpdsMapping[4] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 64);
4182 break;
4183 }
4184 case AMDGPU::G_IMPLICIT_DEF: {
4185 unsigned Size = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4186 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size);
4187 break;
4188 }
4189 case AMDGPU::G_FCONSTANT:
4190 case AMDGPU::G_CONSTANT:
4191 case AMDGPU::G_GLOBAL_VALUE:
4192 case AMDGPU::G_FRAME_INDEX:
4193 case AMDGPU::G_BLOCK_ADDR:
4194 case AMDGPU::G_READSTEADYCOUNTER:
4195 case AMDGPU::G_READCYCLECOUNTER: {
4196 unsigned Size = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4197 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size);
4198 break;
4199 }
4200 case AMDGPU::G_DYN_STACKALLOC: {
4201 // Result is always uniform, and a wave reduction is needed for the source.
4202 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 32);
4203 unsigned SrcBankID = getRegBankID(MI.getOperand(1).getReg(), MRI);
4204 OpdsMapping[1] = AMDGPU::getValueMapping(SrcBankID, 32);
4205 break;
4206 }
4207 case AMDGPU::G_AMDGPU_WAVE_ADDRESS: {
4208 // This case is weird because we expect a physical register in the source,
4209 // but need to set a bank anyway.
4210 //
4211 // TODO: We could select the result to SGPR or VGPR
4212 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 32);
4213 OpdsMapping[1] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 32);
4214 break;
4215 }
4216 case AMDGPU::G_INSERT: {
4217 unsigned BankID = getMappingType(MRI, MI);
4218 unsigned DstSize = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
4219 unsigned SrcSize = TRI->getRegSizeInBits(MI.getOperand(1).getReg(), MRI);
4220 unsigned EltSize = TRI->getRegSizeInBits(MI.getOperand(2).getReg(), MRI);
4221 OpdsMapping[0] = AMDGPU::getValueMapping(BankID, DstSize);
4222 OpdsMapping[1] = AMDGPU::getValueMapping(BankID, SrcSize);
4223 OpdsMapping[2] = AMDGPU::getValueMapping(BankID, EltSize);
4224 OpdsMapping[3] = nullptr;
4225 break;
4226 }
4227 case AMDGPU::G_EXTRACT: {
4228 unsigned BankID = getRegBankID(MI.getOperand(1).getReg(), MRI);
4229 unsigned DstSize = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
4230 unsigned SrcSize = TRI->getRegSizeInBits(MI.getOperand(1).getReg(), MRI);
4231 OpdsMapping[0] = AMDGPU::getValueMapping(BankID, DstSize);
4232 OpdsMapping[1] = AMDGPU::getValueMapping(BankID, SrcSize);
4233 OpdsMapping[2] = nullptr;
4234 break;
4235 }
4236 case AMDGPU::G_BUILD_VECTOR:
4237 case AMDGPU::G_BUILD_VECTOR_TRUNC: {
4238 LLT DstTy = MRI.getType(MI.getOperand(0).getReg());
4239 if (DstTy == LLT::fixed_vector(2, 16)) {
4240 unsigned DstSize = DstTy.getSizeInBits();
4241 unsigned SrcSize = MRI.getType(MI.getOperand(1).getReg()).getSizeInBits();
4242 unsigned Src0BankID = getRegBankID(MI.getOperand(1).getReg(), MRI);
4243 unsigned Src1BankID = getRegBankID(MI.getOperand(2).getReg(), MRI);
4244 unsigned DstBankID = regBankUnion(Src0BankID, Src1BankID);
4245
4246 OpdsMapping[0] = AMDGPU::getValueMapping(DstBankID, DstSize);
4247 OpdsMapping[1] = AMDGPU::getValueMapping(Src0BankID, SrcSize);
4248 OpdsMapping[2] = AMDGPU::getValueMapping(Src1BankID, SrcSize);
4249 break;
4250 }
4251
4252 [[fallthrough]];
4253 }
4254 case AMDGPU::G_MERGE_VALUES:
4255 case AMDGPU::G_CONCAT_VECTORS: {
4256 unsigned Bank = getMappingType(MRI, MI);
4257 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4258 unsigned SrcSize = MRI.getType(MI.getOperand(1).getReg()).getSizeInBits();
4259
4260 OpdsMapping[0] = AMDGPU::getValueMapping(Bank, DstSize);
4261 // Op1 and Dst should use the same register bank.
4262 for (unsigned i = 1, e = MI.getNumOperands(); i != e; ++i)
4263 OpdsMapping[i] = AMDGPU::getValueMapping(Bank, SrcSize);
4264 break;
4265 }
4266 case AMDGPU::G_BITREVERSE:
4267 case AMDGPU::G_BITCAST:
4268 case AMDGPU::G_INTTOPTR:
4269 case AMDGPU::G_PTRTOINT:
4270 case AMDGPU::G_FABS:
4271 case AMDGPU::G_FNEG: {
4272 unsigned Size = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4273 unsigned BankID = getRegBankID(MI.getOperand(1).getReg(), MRI);
4274 OpdsMapping[0] = OpdsMapping[1] = AMDGPU::getValueMapping(BankID, Size);
4275 break;
4276 }
4277 case AMDGPU::G_AMDGPU_FFBH_U32:
4278 case AMDGPU::G_AMDGPU_FFBL_B32:
4279 case AMDGPU::G_CTLZ_ZERO_POISON:
4280 case AMDGPU::G_CTTZ_ZERO_POISON: {
4281 unsigned Size = MRI.getType(MI.getOperand(1).getReg()).getSizeInBits();
4282 unsigned BankID = getRegBankID(MI.getOperand(1).getReg(), MRI);
4283 OpdsMapping[0] = AMDGPU::getValueMapping(BankID, 32);
4284 OpdsMapping[1] = AMDGPU::getValueMappingSGPR64Only(BankID, Size);
4285 break;
4286 }
4287 case AMDGPU::G_CTPOP: {
4288 unsigned Size = MRI.getType(MI.getOperand(1).getReg()).getSizeInBits();
4289 unsigned BankID = getRegBankID(MI.getOperand(1).getReg(), MRI);
4290 OpdsMapping[0] = AMDGPU::getValueMapping(BankID, 32);
4291
4292 // This should really be getValueMappingSGPR64Only, but allowing the generic
4293 // code to handle the register split just makes using LegalizerHelper more
4294 // difficult.
4295 OpdsMapping[1] = AMDGPU::getValueMapping(BankID, Size);
4296 break;
4297 }
4298 case AMDGPU::G_TRUNC: {
4299 Register Dst = MI.getOperand(0).getReg();
4300 Register Src = MI.getOperand(1).getReg();
4301 unsigned Bank = getRegBankID(Src, MRI);
4302 unsigned DstSize = TRI->getRegSizeInBits(Dst, MRI);
4303 unsigned SrcSize = TRI->getRegSizeInBits(Src, MRI);
4304 OpdsMapping[0] = AMDGPU::getValueMapping(Bank, DstSize);
4305 OpdsMapping[1] = AMDGPU::getValueMapping(Bank, SrcSize);
4306 break;
4307 }
4308 case AMDGPU::G_ZEXT:
4309 case AMDGPU::G_SEXT:
4310 case AMDGPU::G_ANYEXT:
4311 case AMDGPU::G_SEXT_INREG: {
4312 Register Dst = MI.getOperand(0).getReg();
4313 Register Src = MI.getOperand(1).getReg();
4314 unsigned DstSize = TRI->getRegSizeInBits(Dst, MRI);
4315 unsigned SrcSize = TRI->getRegSizeInBits(Src, MRI);
4316
4317 unsigned DstBank;
4318 const RegisterBank *SrcBank = getRegBank(Src, MRI, *TRI);
4319 assert(SrcBank);
4320 switch (SrcBank->getID()) {
4321 case AMDGPU::SGPRRegBankID:
4322 DstBank = AMDGPU::SGPRRegBankID;
4323 break;
4324 default:
4325 DstBank = AMDGPU::VGPRRegBankID;
4326 break;
4327 }
4328
4329 // Scalar extend can use 64-bit BFE, but VGPRs require extending to
4330 // 32-bits, and then to 64.
4331 OpdsMapping[0] = AMDGPU::getValueMappingSGPR64Only(DstBank, DstSize);
4332 OpdsMapping[1] = AMDGPU::getValueMappingSGPR64Only(SrcBank->getID(),
4333 SrcSize);
4334 break;
4335 }
4336 case AMDGPU::G_IS_FPCLASS: {
4337 Register SrcReg = MI.getOperand(1).getReg();
4338 unsigned SrcSize = MRI.getType(SrcReg).getSizeInBits();
4339 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4340 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, DstSize);
4341 OpdsMapping[1] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, SrcSize);
4342 break;
4343 }
4344 case AMDGPU::G_STORE: {
4345 assert(MI.getOperand(0).isReg());
4346 unsigned Size = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4347
4348 // FIXME: We need to specify a different reg bank once scalar stores are
4349 // supported.
4350 const ValueMapping *ValMapping =
4351 AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
4352 OpdsMapping[0] = ValMapping;
4353 OpdsMapping[1] = getValueMappingForPtr(MRI, MI.getOperand(1).getReg());
4354 break;
4355 }
4356 case AMDGPU::G_ICMP:
4357 case AMDGPU::G_FCMP: {
4358 unsigned Size = MRI.getType(MI.getOperand(2).getReg()).getSizeInBits();
4359
4360 // See if the result register has already been constrained to vcc, which may
4361 // happen due to control flow intrinsic lowering.
4362 unsigned DstBank = getRegBankID(MI.getOperand(0).getReg(), MRI,
4363 AMDGPU::SGPRRegBankID);
4364 unsigned Op2Bank = getRegBankID(MI.getOperand(2).getReg(), MRI);
4365 unsigned Op3Bank = getRegBankID(MI.getOperand(3).getReg(), MRI);
4366
4367 auto canUseSCCICMP = [&]() {
4368 auto Pred =
4369 static_cast<CmpInst::Predicate>(MI.getOperand(1).getPredicate());
4370 return Size == 32 ||
4371 (Size == 64 &&
4372 (Pred == CmpInst::ICMP_EQ || Pred == CmpInst::ICMP_NE) &&
4373 Subtarget.hasScalarCompareEq64());
4374 };
4375 auto canUseSCCFCMP = [&]() {
4376 return Subtarget.hasSALUFloatInsts() && (Size == 32 || Size == 16);
4377 };
4378
4379 bool isICMP = MI.getOpcode() == AMDGPU::G_ICMP;
4380 bool CanUseSCC = DstBank == AMDGPU::SGPRRegBankID &&
4381 Op2Bank == AMDGPU::SGPRRegBankID &&
4382 Op3Bank == AMDGPU::SGPRRegBankID &&
4383 (isICMP ? canUseSCCICMP() : canUseSCCFCMP());
4384
4385 DstBank = CanUseSCC ? AMDGPU::SGPRRegBankID : AMDGPU::VCCRegBankID;
4386 unsigned SrcBank = CanUseSCC ? AMDGPU::SGPRRegBankID : AMDGPU::VGPRRegBankID;
4387
4388 // TODO: Use 32-bit for scalar output size.
4389 // SCC results will need to be copied to a 32-bit SGPR virtual register.
4390 const unsigned ResultSize = 1;
4391
4392 OpdsMapping[0] = AMDGPU::getValueMapping(DstBank, ResultSize);
4393 OpdsMapping[1] = nullptr; // Predicate Operand.
4394 OpdsMapping[2] = AMDGPU::getValueMapping(SrcBank, Size);
4395 OpdsMapping[3] = AMDGPU::getValueMapping(SrcBank, Size);
4396 break;
4397 }
4398 case AMDGPU::G_EXTRACT_VECTOR_ELT: {
4399 // VGPR index can be used for waterfall when indexing a SGPR vector.
4400 unsigned SrcBankID = getRegBankID(MI.getOperand(1).getReg(), MRI);
4401 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4402 unsigned SrcSize = MRI.getType(MI.getOperand(1).getReg()).getSizeInBits();
4403 unsigned IdxSize = MRI.getType(MI.getOperand(2).getReg()).getSizeInBits();
4404 unsigned IdxBank = getRegBankID(MI.getOperand(2).getReg(), MRI);
4405 unsigned OutputBankID = regBankUnion(SrcBankID, IdxBank);
4406
4407 OpdsMapping[0] = AMDGPU::getValueMappingSGPR64Only(OutputBankID, DstSize);
4408 OpdsMapping[1] = AMDGPU::getValueMapping(SrcBankID, SrcSize);
4409
4410 // The index can be either if the source vector is VGPR.
4411 OpdsMapping[2] = AMDGPU::getValueMapping(IdxBank, IdxSize);
4412 break;
4413 }
4414 case AMDGPU::G_INSERT_VECTOR_ELT: {
4415 unsigned OutputBankID = isSALUMapping(MI) ?
4416 AMDGPU::SGPRRegBankID : AMDGPU::VGPRRegBankID;
4417
4418 unsigned VecSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4419 unsigned InsertSize = MRI.getType(MI.getOperand(2).getReg()).getSizeInBits();
4420 unsigned IdxSize = MRI.getType(MI.getOperand(3).getReg()).getSizeInBits();
4421 unsigned InsertEltBankID = getRegBankID(MI.getOperand(2).getReg(), MRI);
4422 unsigned IdxBankID = getRegBankID(MI.getOperand(3).getReg(), MRI);
4423
4424 OpdsMapping[0] = AMDGPU::getValueMapping(OutputBankID, VecSize);
4425 OpdsMapping[1] = AMDGPU::getValueMapping(OutputBankID, VecSize);
4426
4427 // This is a weird case, because we need to break down the mapping based on
4428 // the register bank of a different operand.
4429 if (InsertSize == 64 && OutputBankID == AMDGPU::VGPRRegBankID) {
4430 OpdsMapping[2] = AMDGPU::getValueMappingSplit64(InsertEltBankID,
4431 InsertSize);
4432 } else {
4433 assert(InsertSize == 32 || InsertSize == 64);
4434 OpdsMapping[2] = AMDGPU::getValueMapping(InsertEltBankID, InsertSize);
4435 }
4436
4437 // The index can be either if the source vector is VGPR.
4438 OpdsMapping[3] = AMDGPU::getValueMapping(IdxBankID, IdxSize);
4439 break;
4440 }
4441 case AMDGPU::G_UNMERGE_VALUES: {
4442 unsigned Bank = getMappingType(MRI, MI);
4443
4444 // Op1 and Dst should use the same register bank.
4445 // FIXME: Shouldn't this be the default? Why do we need to handle this?
4446 for (unsigned i = 0, e = MI.getNumOperands(); i != e; ++i) {
4447 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(i).getReg(), MRI);
4448 OpdsMapping[i] = AMDGPU::getValueMapping(Bank, Size);
4449 }
4450 break;
4451 }
4452 case AMDGPU::G_AMDGPU_BUFFER_LOAD:
4453 case AMDGPU::G_AMDGPU_BUFFER_LOAD_UBYTE:
4454 case AMDGPU::G_AMDGPU_BUFFER_LOAD_SBYTE:
4455 case AMDGPU::G_AMDGPU_BUFFER_LOAD_USHORT:
4456 case AMDGPU::G_AMDGPU_BUFFER_LOAD_SSHORT:
4457 case AMDGPU::G_AMDGPU_BUFFER_LOAD_TFE:
4458 case AMDGPU::G_AMDGPU_BUFFER_LOAD_UBYTE_TFE:
4459 case AMDGPU::G_AMDGPU_BUFFER_LOAD_SBYTE_TFE:
4460 case AMDGPU::G_AMDGPU_BUFFER_LOAD_USHORT_TFE:
4461 case AMDGPU::G_AMDGPU_BUFFER_LOAD_SSHORT_TFE:
4462 case AMDGPU::G_AMDGPU_BUFFER_LOAD_FORMAT:
4463 case AMDGPU::G_AMDGPU_BUFFER_LOAD_FORMAT_TFE:
4464 case AMDGPU::G_AMDGPU_BUFFER_LOAD_FORMAT_D16:
4465 case AMDGPU::G_AMDGPU_BUFFER_LOAD_FORMAT_D16_TFE:
4466 case AMDGPU::G_AMDGPU_TBUFFER_LOAD_FORMAT:
4467 case AMDGPU::G_AMDGPU_TBUFFER_LOAD_FORMAT_D16:
4468 case AMDGPU::G_AMDGPU_TBUFFER_STORE_FORMAT:
4469 case AMDGPU::G_AMDGPU_TBUFFER_STORE_FORMAT_D16:
4470 case AMDGPU::G_AMDGPU_BUFFER_STORE:
4471 case AMDGPU::G_AMDGPU_BUFFER_STORE_BYTE:
4472 case AMDGPU::G_AMDGPU_BUFFER_STORE_SHORT:
4473 case AMDGPU::G_AMDGPU_BUFFER_STORE_FORMAT:
4474 case AMDGPU::G_AMDGPU_BUFFER_STORE_FORMAT_D16: {
4475 OpdsMapping[0] = getVGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
4476
4477 // rsrc
4478 OpdsMapping[1] = getSGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
4479
4480 // vindex
4481 OpdsMapping[2] = getVGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
4482
4483 // voffset
4484 OpdsMapping[3] = getVGPROpMapping(MI.getOperand(3).getReg(), MRI, *TRI);
4485
4486 // soffset
4487 OpdsMapping[4] = getSGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI);
4488
4489 // Any remaining operands are immediates and were correctly null
4490 // initialized.
4491 break;
4492 }
4493 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_SWAP:
4494 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_ADD:
4495 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_SUB:
4496 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_SMIN:
4497 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_UMIN:
4498 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_SMAX:
4499 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_UMAX:
4500 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_AND:
4501 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_OR:
4502 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_XOR:
4503 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_INC:
4504 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_DEC:
4505 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_SUB_CLAMP_U32:
4506 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_COND_SUB_U32:
4507 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_FADD:
4508 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_FMIN:
4509 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_FMAX: {
4510 // vdata_out
4511 OpdsMapping[0] = getVGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
4512
4513 // vdata_in
4514 OpdsMapping[1] = getVGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
4515
4516 // rsrc
4517 OpdsMapping[2] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
4518
4519 // vindex
4520 OpdsMapping[3] = getVGPROpMapping(MI.getOperand(3).getReg(), MRI, *TRI);
4521
4522 // voffset
4523 OpdsMapping[4] = getVGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI);
4524
4525 // soffset
4526 OpdsMapping[5] = getSGPROpMapping(MI.getOperand(5).getReg(), MRI, *TRI);
4527
4528 // Any remaining operands are immediates and were correctly null
4529 // initialized.
4530 break;
4531 }
4532 case AMDGPU::G_AMDGPU_BUFFER_ATOMIC_CMPSWAP: {
4533 // vdata_out
4534 OpdsMapping[0] = getVGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
4535
4536 // vdata_in
4537 OpdsMapping[1] = getVGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
4538
4539 // cmp
4540 OpdsMapping[2] = getVGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
4541
4542 // rsrc
4543 OpdsMapping[3] = getSGPROpMapping(MI.getOperand(3).getReg(), MRI, *TRI);
4544
4545 // vindex
4546 OpdsMapping[4] = getVGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI);
4547
4548 // voffset
4549 OpdsMapping[5] = getVGPROpMapping(MI.getOperand(5).getReg(), MRI, *TRI);
4550
4551 // soffset
4552 OpdsMapping[6] = getSGPROpMapping(MI.getOperand(6).getReg(), MRI, *TRI);
4553
4554 // Any remaining operands are immediates and were correctly null
4555 // initialized.
4556 break;
4557 }
4558 case AMDGPU::G_AMDGPU_S_BUFFER_LOAD:
4559 case AMDGPU::G_AMDGPU_S_BUFFER_LOAD_UBYTE:
4560 case AMDGPU::G_AMDGPU_S_BUFFER_LOAD_SBYTE:
4561 case AMDGPU::G_AMDGPU_S_BUFFER_LOAD_USHORT:
4562 case AMDGPU::G_AMDGPU_S_BUFFER_LOAD_SSHORT: {
4563 // Lie and claim everything is legal, even though some need to be
4564 // SGPRs. applyMapping will have to deal with it as a waterfall loop.
4565 OpdsMapping[1] = getSGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
4566 OpdsMapping[2] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
4567
4568 // We need to convert this to a MUBUF if either the resource of offset is
4569 // VGPR.
4570 unsigned RSrcBank = OpdsMapping[1]->BreakDown[0].RegBank->getID();
4571 unsigned OffsetBank = OpdsMapping[2]->BreakDown[0].RegBank->getID();
4572 unsigned ResultBank = regBankUnion(RSrcBank, OffsetBank);
4573
4574 unsigned Size0 = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4575 OpdsMapping[0] = AMDGPU::getValueMapping(ResultBank, Size0);
4576 break;
4577 }
4578 case AMDGPU::G_AMDGPU_S_BUFFER_PREFETCH:
4579 OpdsMapping[0] = getSGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
4580 OpdsMapping[2] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
4581 break;
4582 case AMDGPU::G_AMDGPU_SPONENTRY: {
4583 unsigned Size = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4584 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size);
4585 break;
4586 }
4587 case AMDGPU::G_INTRINSIC:
4588 case AMDGPU::G_INTRINSIC_CONVERGENT: {
4589 switch (cast<GIntrinsic>(MI).getIntrinsicID()) {
4590 default:
4592 case Intrinsic::amdgcn_div_fmas: {
4593 // The unscaled IEEE-input case can select S_FMAC_F32.
4594 if (Subtarget.hasSALUFloatInsts() &&
4595 MRI.getType(MI.getOperand(0).getReg()) == LLT::scalar(32) &&
4597 return getDefaultMappingSOP(MI);
4598 return getDefaultMappingVOP(MI);
4599 }
4600 case Intrinsic::amdgcn_div_fixup:
4601 case Intrinsic::amdgcn_trig_preop:
4602 case Intrinsic::amdgcn_sin:
4603 case Intrinsic::amdgcn_cos:
4604 case Intrinsic::amdgcn_log_clamp:
4605 case Intrinsic::amdgcn_rcp_legacy:
4606 case Intrinsic::amdgcn_rsq_legacy:
4607 case Intrinsic::amdgcn_rsq_clamp:
4608 case Intrinsic::amdgcn_tanh:
4609 case Intrinsic::amdgcn_fmul_legacy:
4610 case Intrinsic::amdgcn_fma_legacy:
4611 case Intrinsic::amdgcn_frexp_mant:
4612 case Intrinsic::amdgcn_frexp_exp:
4613 case Intrinsic::amdgcn_fract:
4614 case Intrinsic::amdgcn_cvt_pknorm_i16:
4615 case Intrinsic::amdgcn_cvt_pknorm_u16:
4616 case Intrinsic::amdgcn_cvt_pk_i16:
4617 case Intrinsic::amdgcn_cvt_pk_u16:
4618 case Intrinsic::amdgcn_cvt_sr_pk_f16_f32:
4619 case Intrinsic::amdgcn_cvt_sr_pk_bf16_f32:
4620 case Intrinsic::amdgcn_cvt_pk_f16_fp8:
4621 case Intrinsic::amdgcn_cvt_pk_f16_bf8:
4622 case Intrinsic::amdgcn_cvt_pk_fp8_f16:
4623 case Intrinsic::amdgcn_cvt_pk_bf8_f16:
4624 case Intrinsic::amdgcn_cvt_sr_fp8_f16:
4625 case Intrinsic::amdgcn_cvt_sr_bf8_f16:
4626 case Intrinsic::amdgcn_cvt_scale_pk8_f16_fp8:
4627 case Intrinsic::amdgcn_cvt_scale_pk8_bf16_fp8:
4628 case Intrinsic::amdgcn_cvt_scale_pk8_f16_bf8:
4629 case Intrinsic::amdgcn_cvt_scale_pk8_bf16_bf8:
4630 case Intrinsic::amdgcn_cvt_scale_pk8_f16_fp4:
4631 case Intrinsic::amdgcn_cvt_scale_pk8_bf16_fp4:
4632 case Intrinsic::amdgcn_cvt_scale_pk8_f32_fp8:
4633 case Intrinsic::amdgcn_cvt_scale_pk8_f32_bf8:
4634 case Intrinsic::amdgcn_cvt_scale_pk8_f32_fp4:
4635 case Intrinsic::amdgcn_cvt_scale_pk16_f16_fp6:
4636 case Intrinsic::amdgcn_cvt_scale_pk16_bf16_fp6:
4637 case Intrinsic::amdgcn_cvt_scale_pk16_f16_bf6:
4638 case Intrinsic::amdgcn_cvt_scale_pk16_bf16_bf6:
4639 case Intrinsic::amdgcn_cvt_scale_pk16_f32_fp6:
4640 case Intrinsic::amdgcn_cvt_scale_pk16_f32_bf6:
4641 case Intrinsic::amdgcn_cvt_scale_pk32_f16_fp6:
4642 case Intrinsic::amdgcn_cvt_scale_pk32_bf16_fp6:
4643 case Intrinsic::amdgcn_cvt_scale_pk32_f16_bf6:
4644 case Intrinsic::amdgcn_cvt_scale_pk32_bf16_bf6:
4645 case Intrinsic::amdgcn_cvt_scale_pk32_f32_fp6:
4646 case Intrinsic::amdgcn_cvt_scale_pk32_f32_bf6:
4647 case Intrinsic::amdgcn_cvt_scalef32_pk8_fp8_bf16:
4648 case Intrinsic::amdgcn_cvt_scalef32_pk8_bf8_bf16:
4649 case Intrinsic::amdgcn_cvt_scalef32_pk8_fp8_f16:
4650 case Intrinsic::amdgcn_cvt_scalef32_pk8_bf8_f16:
4651 case Intrinsic::amdgcn_cvt_scalef32_pk8_fp8_f32:
4652 case Intrinsic::amdgcn_cvt_scalef32_pk8_bf8_f32:
4653 case Intrinsic::amdgcn_cvt_scalef32_pk8_fp4_f32:
4654 case Intrinsic::amdgcn_cvt_scalef32_pk8_fp4_f16:
4655 case Intrinsic::amdgcn_cvt_scalef32_pk8_fp4_bf16:
4656 case Intrinsic::amdgcn_cvt_scalef32_pk16_fp6_f32:
4657 case Intrinsic::amdgcn_cvt_scalef32_pk16_bf6_f32:
4658 case Intrinsic::amdgcn_cvt_scalef32_pk16_fp6_f16:
4659 case Intrinsic::amdgcn_cvt_scalef32_pk16_bf6_f16:
4660 case Intrinsic::amdgcn_cvt_scalef32_pk16_fp6_bf16:
4661 case Intrinsic::amdgcn_cvt_scalef32_pk16_bf6_bf16:
4662 case Intrinsic::amdgcn_cvt_scalef32_sr_pk8_fp8_bf16:
4663 case Intrinsic::amdgcn_cvt_scalef32_sr_pk8_bf8_bf16:
4664 case Intrinsic::amdgcn_cvt_scalef32_sr_pk8_fp8_f16:
4665 case Intrinsic::amdgcn_cvt_scalef32_sr_pk8_bf8_f16:
4666 case Intrinsic::amdgcn_cvt_scalef32_sr_pk8_fp8_f32:
4667 case Intrinsic::amdgcn_cvt_scalef32_sr_pk8_bf8_f32:
4668 case Intrinsic::amdgcn_cvt_scalef32_sr_pk8_fp4_f32:
4669 case Intrinsic::amdgcn_cvt_scalef32_sr_pk8_fp4_f16:
4670 case Intrinsic::amdgcn_cvt_scalef32_sr_pk8_fp4_bf16:
4671 case Intrinsic::amdgcn_cvt_scalef32_sr_pk16_fp6_f32:
4672 case Intrinsic::amdgcn_cvt_scalef32_sr_pk16_bf6_f32:
4673 case Intrinsic::amdgcn_cvt_scalef32_sr_pk16_fp6_f16:
4674 case Intrinsic::amdgcn_cvt_scalef32_sr_pk16_bf6_f16:
4675 case Intrinsic::amdgcn_cvt_scalef32_sr_pk16_fp6_bf16:
4676 case Intrinsic::amdgcn_cvt_scalef32_sr_pk16_bf6_bf16:
4677 case Intrinsic::amdgcn_sat_pk4_i4_i8:
4678 case Intrinsic::amdgcn_sat_pk4_u4_u8:
4679 case Intrinsic::amdgcn_fmed3:
4680 case Intrinsic::amdgcn_cubeid:
4681 case Intrinsic::amdgcn_cubema:
4682 case Intrinsic::amdgcn_cubesc:
4683 case Intrinsic::amdgcn_cubetc:
4684 case Intrinsic::amdgcn_sffbh:
4685 case Intrinsic::amdgcn_fmad_ftz:
4686 case Intrinsic::amdgcn_mbcnt_lo:
4687 case Intrinsic::amdgcn_mbcnt_hi:
4688 case Intrinsic::amdgcn_mul_u24:
4689 case Intrinsic::amdgcn_mul_i24:
4690 case Intrinsic::amdgcn_mulhi_u24:
4691 case Intrinsic::amdgcn_mulhi_i24:
4692 case Intrinsic::amdgcn_lerp:
4693 case Intrinsic::amdgcn_sad_u8:
4694 case Intrinsic::amdgcn_msad_u8:
4695 case Intrinsic::amdgcn_sad_hi_u8:
4696 case Intrinsic::amdgcn_sad_u16:
4697 case Intrinsic::amdgcn_qsad_pk_u16_u8:
4698 case Intrinsic::amdgcn_mqsad_pk_u16_u8:
4699 case Intrinsic::amdgcn_mqsad_u32_u8:
4700 case Intrinsic::amdgcn_cvt_pk_u8_f32:
4701 case Intrinsic::amdgcn_alignbyte:
4702 case Intrinsic::amdgcn_perm:
4703 case Intrinsic::amdgcn_prng_b32:
4704 case Intrinsic::amdgcn_exclusive_scan_sum_i32:
4705 case Intrinsic::amdgcn_exclusive_scan_sum_u32:
4706 case Intrinsic::amdgcn_exclusive_scan_xor_b32:
4707 case Intrinsic::amdgcn_exclusive_scan_or_b32:
4708 case Intrinsic::amdgcn_exclusive_scan_and_b32:
4709 case Intrinsic::amdgcn_exclusive_scan_min_i16:
4710 case Intrinsic::amdgcn_exclusive_scan_min_u16:
4711 case Intrinsic::amdgcn_exclusive_scan_min_i32:
4712 case Intrinsic::amdgcn_exclusive_scan_min_u32:
4713 case Intrinsic::amdgcn_exclusive_scan_max_i16:
4714 case Intrinsic::amdgcn_exclusive_scan_max_u16:
4715 case Intrinsic::amdgcn_exclusive_scan_max_i32:
4716 case Intrinsic::amdgcn_exclusive_scan_max_u32:
4717 case Intrinsic::amdgcn_wave_match_b32:
4718 case Intrinsic::amdgcn_fdot2:
4719 case Intrinsic::amdgcn_sdot2:
4720 case Intrinsic::amdgcn_udot2:
4721 case Intrinsic::amdgcn_sdot4:
4722 case Intrinsic::amdgcn_udot4:
4723 case Intrinsic::amdgcn_sdot8:
4724 case Intrinsic::amdgcn_udot8:
4725 case Intrinsic::amdgcn_fdot2_bf16_bf16:
4726 case Intrinsic::amdgcn_fdot2_f16_f16:
4727 case Intrinsic::amdgcn_fdot2_f32_bf16:
4728 case Intrinsic::amdgcn_fdot2c_f32_bf16:
4729 case Intrinsic::amdgcn_sudot4:
4730 case Intrinsic::amdgcn_sudot8:
4731 case Intrinsic::amdgcn_dot4_f32_fp8_bf8:
4732 case Intrinsic::amdgcn_dot4_f32_bf8_fp8:
4733 case Intrinsic::amdgcn_dot4_f32_fp8_fp8:
4734 case Intrinsic::amdgcn_dot4_f32_bf8_bf8:
4735 case Intrinsic::amdgcn_cvt_f32_fp8:
4736 case Intrinsic::amdgcn_cvt_f32_fp8_e5m3:
4737 case Intrinsic::amdgcn_cvt_f32_bf8:
4738 case Intrinsic::amdgcn_cvt_off_f32_i4:
4739 case Intrinsic::amdgcn_cvt_pk_f32_fp8:
4740 case Intrinsic::amdgcn_cvt_pk_f32_bf8:
4741 case Intrinsic::amdgcn_cvt_pk_fp8_f32:
4742 case Intrinsic::amdgcn_cvt_pk_fp8_f32_e5m3:
4743 case Intrinsic::amdgcn_cvt_pk_bf8_f32:
4744 case Intrinsic::amdgcn_cvt_sr_fp8_f32:
4745 case Intrinsic::amdgcn_cvt_sr_fp8_f32_e5m3:
4746 case Intrinsic::amdgcn_cvt_sr_bf8_f32:
4747 case Intrinsic::amdgcn_cvt_sr_bf16_f32:
4748 case Intrinsic::amdgcn_cvt_sr_f16_f32:
4749 case Intrinsic::amdgcn_cvt_f16_fp8:
4750 case Intrinsic::amdgcn_cvt_f16_bf8:
4751 case Intrinsic::amdgcn_cvt_scalef32_pk32_fp6_f16:
4752 case Intrinsic::amdgcn_cvt_scalef32_pk32_bf6_f16:
4753 case Intrinsic::amdgcn_cvt_scalef32_pk32_fp6_bf16:
4754 case Intrinsic::amdgcn_cvt_scalef32_pk32_bf6_bf16:
4755 case Intrinsic::amdgcn_cvt_scalef32_pk32_fp6_f32:
4756 case Intrinsic::amdgcn_cvt_scalef32_pk32_bf6_f32:
4757 case Intrinsic::amdgcn_cvt_scalef32_f16_fp8:
4758 case Intrinsic::amdgcn_cvt_scalef32_f16_bf8:
4759 case Intrinsic::amdgcn_cvt_scalef32_f32_fp8:
4760 case Intrinsic::amdgcn_cvt_scalef32_f32_bf8:
4761 case Intrinsic::amdgcn_cvt_scalef32_pk_fp8_f32:
4762 case Intrinsic::amdgcn_cvt_scalef32_pk_bf8_f32:
4763 case Intrinsic::amdgcn_cvt_scalef32_pk_f32_fp8:
4764 case Intrinsic::amdgcn_cvt_scalef32_pk_f32_bf8:
4765 case Intrinsic::amdgcn_cvt_scalef32_pk_fp8_f16:
4766 case Intrinsic::amdgcn_cvt_scalef32_pk_fp8_bf16:
4767 case Intrinsic::amdgcn_cvt_scalef32_pk_bf8_f16:
4768 case Intrinsic::amdgcn_cvt_scalef32_pk_bf8_bf16:
4769 case Intrinsic::amdgcn_cvt_scalef32_pk_f32_fp4:
4770 case Intrinsic::amdgcn_cvt_scalef32_pk_fp4_f32:
4771 case Intrinsic::amdgcn_cvt_scalef32_pk_f16_fp4:
4772 case Intrinsic::amdgcn_cvt_scalef32_pk_bf16_fp4:
4773 case Intrinsic::amdgcn_cvt_scalef32_pk32_f32_fp6:
4774 case Intrinsic::amdgcn_cvt_scalef32_pk32_f32_bf6:
4775 case Intrinsic::amdgcn_cvt_scalef32_pk32_f16_bf6:
4776 case Intrinsic::amdgcn_cvt_scalef32_pk32_bf16_bf6:
4777 case Intrinsic::amdgcn_cvt_scalef32_pk32_f16_fp6:
4778 case Intrinsic::amdgcn_cvt_scalef32_pk32_bf16_fp6:
4779 case Intrinsic::amdgcn_cvt_scalef32_pk_f16_bf8:
4780 case Intrinsic::amdgcn_cvt_scalef32_pk_bf16_bf8:
4781 case Intrinsic::amdgcn_cvt_scalef32_pk_f16_fp8:
4782 case Intrinsic::amdgcn_cvt_scalef32_pk_bf16_fp8:
4783 case Intrinsic::amdgcn_cvt_scalef32_pk_fp4_f16:
4784 case Intrinsic::amdgcn_cvt_scalef32_pk_fp4_bf16:
4785 case Intrinsic::amdgcn_cvt_scalef32_sr_pk_fp4_f16:
4786 case Intrinsic::amdgcn_cvt_scalef32_sr_pk_fp4_bf16:
4787 case Intrinsic::amdgcn_cvt_scalef32_sr_pk_fp4_f32:
4788 case Intrinsic::amdgcn_cvt_scalef32_sr_pk32_bf6_bf16:
4789 case Intrinsic::amdgcn_cvt_scalef32_sr_pk32_bf6_f16:
4790 case Intrinsic::amdgcn_cvt_scalef32_sr_pk32_bf6_f32:
4791 case Intrinsic::amdgcn_cvt_scalef32_sr_pk32_fp6_bf16:
4792 case Intrinsic::amdgcn_cvt_scalef32_sr_pk32_fp6_f16:
4793 case Intrinsic::amdgcn_cvt_scalef32_sr_pk32_fp6_f32:
4794 case Intrinsic::amdgcn_cvt_scalef32_sr_bf8_bf16:
4795 case Intrinsic::amdgcn_cvt_scalef32_sr_bf8_f16:
4796 case Intrinsic::amdgcn_cvt_scalef32_sr_bf8_f32:
4797 case Intrinsic::amdgcn_cvt_scalef32_sr_fp8_bf16:
4798 case Intrinsic::amdgcn_cvt_scalef32_sr_fp8_f16:
4799 case Intrinsic::amdgcn_cvt_scalef32_sr_fp8_f32:
4800 case Intrinsic::amdgcn_ashr_pk_i8_i32:
4801 case Intrinsic::amdgcn_ashr_pk_u8_i32:
4802 case Intrinsic::amdgcn_cvt_scalef32_2xpk16_fp6_f32:
4803 case Intrinsic::amdgcn_cvt_scalef32_2xpk16_bf6_f32:
4804 case Intrinsic::amdgcn_wmma_bf16_16x16x16_bf16:
4805 case Intrinsic::amdgcn_wmma_f16_16x16x16_f16:
4806 case Intrinsic::amdgcn_wmma_bf16_16x16x16_bf16_tied:
4807 case Intrinsic::amdgcn_wmma_f16_16x16x16_f16_tied:
4808 case Intrinsic::amdgcn_wmma_f32_16x16x16_bf16:
4809 case Intrinsic::amdgcn_wmma_f32_16x16x16_f16:
4810 case Intrinsic::amdgcn_wmma_i32_16x16x16_iu4:
4811 case Intrinsic::amdgcn_wmma_i32_16x16x16_iu8:
4812 case Intrinsic::amdgcn_wmma_f32_16x16x16_fp8_fp8:
4813 case Intrinsic::amdgcn_wmma_f32_16x16x16_fp8_bf8:
4814 case Intrinsic::amdgcn_wmma_f32_16x16x16_bf8_fp8:
4815 case Intrinsic::amdgcn_wmma_f32_16x16x16_bf8_bf8:
4816 case Intrinsic::amdgcn_wmma_i32_16x16x32_iu4:
4817 case Intrinsic::amdgcn_swmmac_f32_16x16x32_f16:
4818 case Intrinsic::amdgcn_swmmac_f32_16x16x32_bf16:
4819 case Intrinsic::amdgcn_swmmac_f16_16x16x32_f16:
4820 case Intrinsic::amdgcn_swmmac_bf16_16x16x32_bf16:
4821 case Intrinsic::amdgcn_swmmac_i32_16x16x32_iu8:
4822 case Intrinsic::amdgcn_swmmac_i32_16x16x32_iu4:
4823 case Intrinsic::amdgcn_swmmac_i32_16x16x64_iu4:
4824 case Intrinsic::amdgcn_swmmac_f32_16x16x32_fp8_fp8:
4825 case Intrinsic::amdgcn_swmmac_f32_16x16x32_fp8_bf8:
4826 case Intrinsic::amdgcn_swmmac_f32_16x16x32_bf8_fp8:
4827 case Intrinsic::amdgcn_swmmac_f32_16x16x32_bf8_bf8:
4828 case Intrinsic::amdgcn_wmma_f64_16x16x4_f64:
4829 case Intrinsic::amdgcn_wmma_f32_16x16x4_f32:
4830 case Intrinsic::amdgcn_wmma_f32_16x16x32_bf16:
4831 case Intrinsic::amdgcn_wmma_f32_16x16x32_f16:
4832 case Intrinsic::amdgcn_wmma_f16_16x16x32_f16:
4833 case Intrinsic::amdgcn_wmma_bf16_16x16x32_bf16:
4834 case Intrinsic::amdgcn_wmma_bf16f32_16x16x32_bf16:
4835 case Intrinsic::amdgcn_wmma_f32_16x16x64_fp8_fp8:
4836 case Intrinsic::amdgcn_wmma_f32_16x16x64_fp8_bf8:
4837 case Intrinsic::amdgcn_wmma_f32_16x16x64_bf8_fp8:
4838 case Intrinsic::amdgcn_wmma_f32_16x16x64_bf8_bf8:
4839 case Intrinsic::amdgcn_wmma_f16_16x16x64_fp8_fp8:
4840 case Intrinsic::amdgcn_wmma_f16_16x16x64_fp8_bf8:
4841 case Intrinsic::amdgcn_wmma_f16_16x16x64_bf8_fp8:
4842 case Intrinsic::amdgcn_wmma_f16_16x16x64_bf8_bf8:
4843 case Intrinsic::amdgcn_wmma_f16_16x16x128_fp8_fp8:
4844 case Intrinsic::amdgcn_wmma_f16_16x16x128_fp8_bf8:
4845 case Intrinsic::amdgcn_wmma_f16_16x16x128_bf8_fp8:
4846 case Intrinsic::amdgcn_wmma_f16_16x16x128_bf8_bf8:
4847 case Intrinsic::amdgcn_wmma_f32_16x16x128_fp8_fp8:
4848 case Intrinsic::amdgcn_wmma_f32_16x16x128_fp8_bf8:
4849 case Intrinsic::amdgcn_wmma_f32_16x16x128_bf8_fp8:
4850 case Intrinsic::amdgcn_wmma_f32_16x16x128_bf8_bf8:
4851 case Intrinsic::amdgcn_wmma_i32_16x16x64_iu8:
4852 case Intrinsic::amdgcn_wmma_f32_16x16x128_f8f6f4:
4853 case Intrinsic::amdgcn_wmma_scale_f32_16x16x128_f8f6f4:
4854 case Intrinsic::amdgcn_wmma_scale16_f32_16x16x128_f8f6f4:
4855 case Intrinsic::amdgcn_wmma_f32_32x16x128_f4:
4856 case Intrinsic::amdgcn_wmma_scale_f32_32x16x128_f4:
4857 case Intrinsic::amdgcn_wmma_scale16_f32_32x16x128_f4:
4858 case Intrinsic::amdgcn_swmmac_f16_16x16x64_f16:
4859 case Intrinsic::amdgcn_swmmac_bf16_16x16x64_bf16:
4860 case Intrinsic::amdgcn_swmmac_f32_16x16x64_bf16:
4861 case Intrinsic::amdgcn_swmmac_bf16f32_16x16x64_bf16:
4862 case Intrinsic::amdgcn_swmmac_f32_16x16x64_f16:
4863 case Intrinsic::amdgcn_swmmac_f32_16x16x128_fp8_fp8:
4864 case Intrinsic::amdgcn_swmmac_f32_16x16x128_fp8_bf8:
4865 case Intrinsic::amdgcn_swmmac_f32_16x16x128_bf8_fp8:
4866 case Intrinsic::amdgcn_swmmac_f32_16x16x128_bf8_bf8:
4867 case Intrinsic::amdgcn_swmmac_f16_16x16x128_fp8_fp8:
4868 case Intrinsic::amdgcn_swmmac_f16_16x16x128_fp8_bf8:
4869 case Intrinsic::amdgcn_swmmac_f16_16x16x128_bf8_fp8:
4870 case Intrinsic::amdgcn_swmmac_f16_16x16x128_bf8_bf8:
4871 case Intrinsic::amdgcn_swmmac_i32_16x16x128_iu8:
4872 case Intrinsic::amdgcn_perm_pk16_b4_u4:
4873 case Intrinsic::amdgcn_perm_pk16_b6_u4:
4874 case Intrinsic::amdgcn_perm_pk16_b8_u4:
4875 case Intrinsic::amdgcn_add_max_i32:
4876 case Intrinsic::amdgcn_add_max_u32:
4877 case Intrinsic::amdgcn_add_min_i32:
4878 case Intrinsic::amdgcn_add_min_u32:
4879 case Intrinsic::amdgcn_pk_add_max_i16:
4880 case Intrinsic::amdgcn_pk_add_max_u16:
4881 case Intrinsic::amdgcn_pk_add_min_i16:
4882 case Intrinsic::amdgcn_pk_add_min_u16:
4883 return getDefaultMappingVOP(MI);
4884 case Intrinsic::amdgcn_log:
4885 case Intrinsic::amdgcn_exp2:
4886 case Intrinsic::amdgcn_rcp:
4887 case Intrinsic::amdgcn_rsq:
4888 case Intrinsic::amdgcn_sqrt: {
4889 LLT Ty = MRI.getType(MI.getOperand(0).getReg());
4890 unsigned Size = Ty.getSizeInBits();
4891 // There is no pseudo scalar transcendental instruction for bf16.
4892 if (Subtarget.hasPseudoScalarTrans() && !Ty.isBFloat16() &&
4893 (Size == 16 || Size == 32) && isSALUMapping(MI))
4894 return getDefaultMappingSOP(MI);
4895 return getDefaultMappingVOP(MI);
4896 }
4897 case Intrinsic::amdgcn_sbfe:
4898 case Intrinsic::amdgcn_ubfe:
4899 if (isSALUMapping(MI))
4900 return getDefaultMappingSOP(MI);
4901 return getDefaultMappingVOP(MI);
4902 case Intrinsic::amdgcn_ds_swizzle:
4903 case Intrinsic::amdgcn_ds_permute:
4904 case Intrinsic::amdgcn_ds_bpermute:
4905 case Intrinsic::amdgcn_update_dpp:
4906 case Intrinsic::amdgcn_mov_dpp8:
4907 case Intrinsic::amdgcn_mov_dpp:
4908 case Intrinsic::amdgcn_strict_wwm:
4909 case Intrinsic::amdgcn_wwm:
4910 case Intrinsic::amdgcn_strict_wqm:
4911 case Intrinsic::amdgcn_wqm:
4912 case Intrinsic::amdgcn_softwqm:
4913 case Intrinsic::amdgcn_set_inactive:
4914 case Intrinsic::amdgcn_set_inactive_chain_arg:
4915 case Intrinsic::amdgcn_permlane64:
4916 case Intrinsic::amdgcn_ds_bpermute_fi_b32:
4918 case Intrinsic::amdgcn_cvt_pkrtz:
4919 if (Subtarget.hasSALUFloatInsts() && isSALUMapping(MI))
4920 return getDefaultMappingSOP(MI);
4921 return getDefaultMappingVOP(MI);
4922 case Intrinsic::amdgcn_kernarg_segment_ptr:
4923 case Intrinsic::amdgcn_s_getpc:
4924 case Intrinsic::amdgcn_groupstaticsize:
4925 case Intrinsic::amdgcn_reloc_constant:
4926 case Intrinsic::returnaddress: {
4927 unsigned Size = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4928 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size);
4929 break;
4930 }
4931 case Intrinsic::amdgcn_wqm_vote: {
4932 unsigned Size = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4933 OpdsMapping[0] = OpdsMapping[2]
4934 = AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, Size);
4935 break;
4936 }
4937 case Intrinsic::amdgcn_ps_live: {
4938 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, 1);
4939 break;
4940 }
4941 case Intrinsic::amdgcn_div_scale: {
4942 unsigned Dst0Size = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4943 unsigned Dst1Size = MRI.getType(MI.getOperand(1).getReg()).getSizeInBits();
4944 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Dst0Size);
4945 OpdsMapping[1] = AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, Dst1Size);
4946
4947 unsigned SrcSize = MRI.getType(MI.getOperand(3).getReg()).getSizeInBits();
4948 OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, SrcSize);
4949 OpdsMapping[4] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, SrcSize);
4950 break;
4951 }
4952 case Intrinsic::amdgcn_class: {
4953 Register Src0Reg = MI.getOperand(2).getReg();
4954 Register Src1Reg = MI.getOperand(3).getReg();
4955 unsigned Src0Size = MRI.getType(Src0Reg).getSizeInBits();
4956 unsigned Src1Size = MRI.getType(Src1Reg).getSizeInBits();
4957 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4958 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, DstSize);
4959 OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Src0Size);
4960 OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Src1Size);
4961 break;
4962 }
4963 case Intrinsic::amdgcn_readlane: {
4964 // This must be an SGPR, but accept a VGPR.
4965 Register IdxReg = MI.getOperand(3).getReg();
4966 unsigned IdxSize = MRI.getType(IdxReg).getSizeInBits();
4967 unsigned IdxBank = getRegBankID(IdxReg, MRI, AMDGPU::SGPRRegBankID);
4968 OpdsMapping[3] = AMDGPU::getValueMapping(IdxBank, IdxSize);
4969 [[fallthrough]];
4970 }
4971 case Intrinsic::amdgcn_readfirstlane: {
4972 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4973 unsigned SrcSize = MRI.getType(MI.getOperand(2).getReg()).getSizeInBits();
4974 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, DstSize);
4975 OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, SrcSize);
4976 break;
4977 }
4978 case Intrinsic::amdgcn_writelane: {
4979 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
4980 Register SrcReg = MI.getOperand(2).getReg();
4981 unsigned SrcSize = MRI.getType(SrcReg).getSizeInBits();
4982 unsigned SrcBank = getRegBankID(SrcReg, MRI, AMDGPU::SGPRRegBankID);
4983 Register IdxReg = MI.getOperand(3).getReg();
4984 unsigned IdxSize = MRI.getType(IdxReg).getSizeInBits();
4985 unsigned IdxBank = getRegBankID(IdxReg, MRI, AMDGPU::SGPRRegBankID);
4986 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, DstSize);
4987
4988 // These 2 must be SGPRs, but accept VGPRs. Readfirstlane will be inserted
4989 // to legalize.
4990 OpdsMapping[2] = AMDGPU::getValueMapping(SrcBank, SrcSize);
4991 OpdsMapping[3] = AMDGPU::getValueMapping(IdxBank, IdxSize);
4992 OpdsMapping[4] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, SrcSize);
4993 break;
4994 }
4995 case Intrinsic::amdgcn_if_break: {
4996 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
4997 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size);
4998 OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, 1);
4999 OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size);
5000 break;
5001 }
5002 case Intrinsic::amdgcn_permlane16:
5003 case Intrinsic::amdgcn_permlanex16: {
5004 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
5005 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5006 OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5007 OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5008 OpdsMapping[4] = getSGPROpMapping(MI.getOperand(3).getReg(), MRI, *TRI);
5009 OpdsMapping[5] = getSGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI);
5010 break;
5011 }
5012 case Intrinsic::amdgcn_permlane_bcast:
5013 case Intrinsic::amdgcn_permlane_up:
5014 case Intrinsic::amdgcn_permlane_down:
5015 case Intrinsic::amdgcn_permlane_xor: {
5016 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
5017 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5018 OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5019 OpdsMapping[3] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5020 OpdsMapping[4] = getSGPROpMapping(MI.getOperand(3).getReg(), MRI, *TRI);
5021 break;
5022 }
5023 case Intrinsic::amdgcn_permlane_idx_gen: {
5024 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
5025 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5026 OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5027 OpdsMapping[3] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5028 break;
5029 }
5030 case Intrinsic::amdgcn_permlane16_var:
5031 case Intrinsic::amdgcn_permlanex16_var: {
5032 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
5033 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5034 OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5035 OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5036 OpdsMapping[4] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5037 break;
5038 }
5039 case Intrinsic::amdgcn_mfma_f32_4x4x1f32:
5040 case Intrinsic::amdgcn_mfma_f32_4x4x4f16:
5041 case Intrinsic::amdgcn_mfma_i32_4x4x4i8:
5042 case Intrinsic::amdgcn_mfma_f32_4x4x2bf16:
5043 case Intrinsic::amdgcn_mfma_f32_16x16x1f32:
5044 case Intrinsic::amdgcn_mfma_f32_16x16x4f32:
5045 case Intrinsic::amdgcn_mfma_f32_16x16x4f16:
5046 case Intrinsic::amdgcn_mfma_f32_16x16x16f16:
5047 case Intrinsic::amdgcn_mfma_i32_16x16x4i8:
5048 case Intrinsic::amdgcn_mfma_i32_16x16x16i8:
5049 case Intrinsic::amdgcn_mfma_f32_16x16x2bf16:
5050 case Intrinsic::amdgcn_mfma_f32_16x16x8bf16:
5051 case Intrinsic::amdgcn_mfma_f32_32x32x1f32:
5052 case Intrinsic::amdgcn_mfma_f32_32x32x2f32:
5053 case Intrinsic::amdgcn_mfma_f32_32x32x4f16:
5054 case Intrinsic::amdgcn_mfma_f32_32x32x8f16:
5055 case Intrinsic::amdgcn_mfma_i32_32x32x4i8:
5056 case Intrinsic::amdgcn_mfma_i32_32x32x8i8:
5057 case Intrinsic::amdgcn_mfma_f32_32x32x2bf16:
5058 case Intrinsic::amdgcn_mfma_f32_32x32x4bf16:
5059 case Intrinsic::amdgcn_mfma_f32_32x32x4bf16_1k:
5060 case Intrinsic::amdgcn_mfma_f32_16x16x4bf16_1k:
5061 case Intrinsic::amdgcn_mfma_f32_4x4x4bf16_1k:
5062 case Intrinsic::amdgcn_mfma_f32_32x32x8bf16_1k:
5063 case Intrinsic::amdgcn_mfma_f32_16x16x16bf16_1k:
5064 case Intrinsic::amdgcn_mfma_f64_16x16x4f64:
5065 case Intrinsic::amdgcn_mfma_f64_4x4x4f64:
5066 case Intrinsic::amdgcn_mfma_i32_16x16x32_i8:
5067 case Intrinsic::amdgcn_mfma_i32_32x32x16_i8:
5068 case Intrinsic::amdgcn_mfma_f32_16x16x8_xf32:
5069 case Intrinsic::amdgcn_mfma_f32_32x32x4_xf32:
5070 case Intrinsic::amdgcn_mfma_f32_16x16x32_bf8_bf8:
5071 case Intrinsic::amdgcn_mfma_f32_16x16x32_bf8_fp8:
5072 case Intrinsic::amdgcn_mfma_f32_16x16x32_fp8_bf8:
5073 case Intrinsic::amdgcn_mfma_f32_16x16x32_fp8_fp8:
5074 case Intrinsic::amdgcn_mfma_f32_32x32x16_bf8_bf8:
5075 case Intrinsic::amdgcn_mfma_f32_32x32x16_bf8_fp8:
5076 case Intrinsic::amdgcn_mfma_f32_32x32x16_fp8_bf8:
5077 case Intrinsic::amdgcn_mfma_f32_32x32x16_fp8_fp8:
5078 case Intrinsic::amdgcn_mfma_f32_16x16x32_f16:
5079 case Intrinsic::amdgcn_mfma_f32_32x32x16_f16:
5080 case Intrinsic::amdgcn_mfma_i32_16x16x64_i8:
5081 case Intrinsic::amdgcn_mfma_i32_32x32x32_i8:
5082 case Intrinsic::amdgcn_mfma_f32_16x16x32_bf16: {
5083 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
5084 unsigned MinNumRegsRequired = DstSize / 32;
5085
5086 // Default for MAI intrinsics.
5087 // srcC can also be an immediate which can be folded later.
5088 // FIXME: Should we eventually add an alternative mapping with AGPR src
5089 // for srcA/srcB?
5090 //
5091 // vdst, srcA, srcB, srcC
5093
5094 bool UseAGPRForm = !Subtarget.hasGFX90AInsts() ||
5095 Info->selectAGPRFormMFMA(MinNumRegsRequired);
5096
5097 OpdsMapping[0] =
5098 UseAGPRForm ? getAGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI)
5099 : getVGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
5100 OpdsMapping[2] = getVGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5101 OpdsMapping[3] = getVGPROpMapping(MI.getOperand(3).getReg(), MRI, *TRI);
5102 OpdsMapping[4] =
5103 UseAGPRForm ? getAGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI)
5104 : getVGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI);
5105 break;
5106 }
5107 case Intrinsic::amdgcn_mfma_scale_f32_16x16x128_f8f6f4:
5108 case Intrinsic::amdgcn_mfma_scale_f32_32x32x64_f8f6f4: {
5109 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
5110 unsigned MinNumRegsRequired = DstSize / 32;
5111
5113 bool UseAGPRForm = Info->selectAGPRFormMFMA(MinNumRegsRequired);
5114
5115 OpdsMapping[0] =
5116 UseAGPRForm ? getAGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI)
5117 : getVGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
5118
5119 OpdsMapping[2] = getVGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5120 OpdsMapping[3] = getVGPROpMapping(MI.getOperand(3).getReg(), MRI, *TRI);
5121 OpdsMapping[4] =
5122 UseAGPRForm ? getAGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI)
5123 : getVGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI);
5124
5125 OpdsMapping[8] = getVGPROpMapping(MI.getOperand(8).getReg(), MRI, *TRI);
5126 OpdsMapping[10] = getVGPROpMapping(MI.getOperand(10).getReg(), MRI, *TRI);
5127 break;
5128 }
5129 case Intrinsic::amdgcn_smfmac_f32_16x16x32_f16:
5130 case Intrinsic::amdgcn_smfmac_f32_32x32x16_f16:
5131 case Intrinsic::amdgcn_smfmac_f32_16x16x32_bf16:
5132 case Intrinsic::amdgcn_smfmac_f32_32x32x16_bf16:
5133 case Intrinsic::amdgcn_smfmac_i32_16x16x64_i8:
5134 case Intrinsic::amdgcn_smfmac_i32_32x32x32_i8:
5135 case Intrinsic::amdgcn_smfmac_f32_16x16x64_bf8_bf8:
5136 case Intrinsic::amdgcn_smfmac_f32_16x16x64_bf8_fp8:
5137 case Intrinsic::amdgcn_smfmac_f32_16x16x64_fp8_bf8:
5138 case Intrinsic::amdgcn_smfmac_f32_16x16x64_fp8_fp8:
5139 case Intrinsic::amdgcn_smfmac_f32_32x32x32_bf8_bf8:
5140 case Intrinsic::amdgcn_smfmac_f32_32x32x32_bf8_fp8:
5141 case Intrinsic::amdgcn_smfmac_f32_32x32x32_fp8_bf8:
5142 case Intrinsic::amdgcn_smfmac_f32_32x32x32_fp8_fp8:
5143 case Intrinsic::amdgcn_smfmac_f32_16x16x64_f16:
5144 case Intrinsic::amdgcn_smfmac_f32_32x32x32_f16:
5145 case Intrinsic::amdgcn_smfmac_f32_16x16x64_bf16:
5146 case Intrinsic::amdgcn_smfmac_f32_32x32x32_bf16:
5147 case Intrinsic::amdgcn_smfmac_i32_16x16x128_i8:
5148 case Intrinsic::amdgcn_smfmac_i32_32x32x64_i8:
5149 case Intrinsic::amdgcn_smfmac_f32_16x16x128_bf8_bf8:
5150 case Intrinsic::amdgcn_smfmac_f32_16x16x128_bf8_fp8:
5151 case Intrinsic::amdgcn_smfmac_f32_16x16x128_fp8_bf8:
5152 case Intrinsic::amdgcn_smfmac_f32_16x16x128_fp8_fp8:
5153 case Intrinsic::amdgcn_smfmac_f32_32x32x64_bf8_bf8:
5154 case Intrinsic::amdgcn_smfmac_f32_32x32x64_bf8_fp8:
5155 case Intrinsic::amdgcn_smfmac_f32_32x32x64_fp8_bf8:
5156 case Intrinsic::amdgcn_smfmac_f32_32x32x64_fp8_fp8: {
5157 Register DstReg = MI.getOperand(0).getReg();
5158 unsigned DstSize = MRI.getType(DstReg).getSizeInBits();
5159 unsigned MinNumRegsRequired = DstSize / 32;
5161 bool UseAGPRForm = Info->selectAGPRFormMFMA(MinNumRegsRequired);
5162
5163 // vdst, srcA, srcB, srcC, idx
5164 OpdsMapping[0] = UseAGPRForm ? getAGPROpMapping(DstReg, MRI, *TRI)
5165 : getVGPROpMapping(DstReg, MRI, *TRI);
5166
5167 OpdsMapping[2] = getVGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5168 OpdsMapping[3] = getVGPROpMapping(MI.getOperand(3).getReg(), MRI, *TRI);
5169 OpdsMapping[4] =
5170 UseAGPRForm ? getAGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI)
5171 : getVGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI);
5172 OpdsMapping[5] = getVGPROpMapping(MI.getOperand(5).getReg(), MRI, *TRI);
5173 break;
5174 }
5175 case Intrinsic::amdgcn_interp_p1:
5176 case Intrinsic::amdgcn_interp_p2:
5177 case Intrinsic::amdgcn_interp_mov:
5178 case Intrinsic::amdgcn_interp_p1_f16:
5179 case Intrinsic::amdgcn_interp_p2_f16:
5180 case Intrinsic::amdgcn_lds_param_load: {
5181 const int M0Idx = MI.getNumOperands() - 1;
5182 Register M0Reg = MI.getOperand(M0Idx).getReg();
5183 unsigned M0Bank = getRegBankID(M0Reg, MRI, AMDGPU::SGPRRegBankID);
5184 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
5185
5186 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, DstSize);
5187 for (int I = 2; I != M0Idx && MI.getOperand(I).isReg(); ++I)
5188 OpdsMapping[I] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5189
5190 // Must be SGPR, but we must take whatever the original bank is and fix it
5191 // later.
5192 OpdsMapping[M0Idx] = AMDGPU::getValueMapping(M0Bank, 32);
5193 break;
5194 }
5195 case Intrinsic::amdgcn_interp_inreg_p10:
5196 case Intrinsic::amdgcn_interp_inreg_p2:
5197 case Intrinsic::amdgcn_interp_inreg_p10_f16:
5198 case Intrinsic::amdgcn_interp_inreg_p2_f16:
5199 case Intrinsic::amdgcn_interp_p10_rtz_f16:
5200 case Intrinsic::amdgcn_interp_p2_rtz_f16: {
5201 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
5202 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, DstSize);
5203 OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5204 OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5205 OpdsMapping[4] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5206 break;
5207 }
5208 case Intrinsic::amdgcn_permlane16_swap:
5209 case Intrinsic::amdgcn_permlane32_swap: {
5210 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
5211 OpdsMapping[0] = OpdsMapping[1] = OpdsMapping[3] = OpdsMapping[4] =
5212 AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, DstSize);
5213 break;
5214 }
5215 case Intrinsic::amdgcn_ballot: {
5216 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
5217 unsigned SrcSize = MRI.getType(MI.getOperand(2).getReg()).getSizeInBits();
5218 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, DstSize);
5219 OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, SrcSize);
5220 break;
5221 }
5222 case Intrinsic::amdgcn_inverse_ballot: {
5223 // This must be an SGPR, but accept a VGPR.
5224 Register MaskReg = MI.getOperand(2).getReg();
5225 unsigned MaskSize = MRI.getType(MaskReg).getSizeInBits();
5226 unsigned MaskBank = getRegBankID(MaskReg, MRI, AMDGPU::SGPRRegBankID);
5227 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, 1);
5228 OpdsMapping[2] = AMDGPU::getValueMapping(MaskBank, MaskSize);
5229 break;
5230 }
5231 case Intrinsic::amdgcn_bitop3: {
5232 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
5233 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5234 OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5235 OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5236 OpdsMapping[4] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5237 break;
5238 }
5239 case Intrinsic::amdgcn_s_quadmask:
5240 case Intrinsic::amdgcn_s_wqm: {
5241 Register MaskReg = MI.getOperand(2).getReg();
5242 unsigned MaskSize = MRI.getType(MaskReg).getSizeInBits();
5243 unsigned MaskBank = getRegBankID(MaskReg, MRI, AMDGPU::SGPRRegBankID);
5244 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, MaskSize);
5245 OpdsMapping[2] = AMDGPU::getValueMapping(MaskBank, MaskSize);
5246 break;
5247 }
5248 case Intrinsic::amdgcn_wave_reduce_add:
5249 case Intrinsic::amdgcn_wave_reduce_fadd:
5250 case Intrinsic::amdgcn_wave_reduce_sub:
5251 case Intrinsic::amdgcn_wave_reduce_fsub:
5252 case Intrinsic::amdgcn_wave_reduce_min:
5253 case Intrinsic::amdgcn_wave_reduce_umin:
5254 case Intrinsic::amdgcn_wave_reduce_fmin:
5255 case Intrinsic::amdgcn_wave_reduce_max:
5256 case Intrinsic::amdgcn_wave_reduce_umax:
5257 case Intrinsic::amdgcn_wave_reduce_fmax:
5258 case Intrinsic::amdgcn_wave_reduce_and:
5259 case Intrinsic::amdgcn_wave_reduce_or:
5260 case Intrinsic::amdgcn_wave_reduce_xor: {
5261 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
5262 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, DstSize);
5263 unsigned OpSize = MRI.getType(MI.getOperand(2).getReg()).getSizeInBits();
5264 auto regBankID =
5265 isSALUMapping(MI) ? AMDGPU::SGPRRegBankID : AMDGPU::VGPRRegBankID;
5266 OpdsMapping[2] = AMDGPU::getValueMapping(regBankID, OpSize);
5267 break;
5268 }
5269 case Intrinsic::amdgcn_s_bitreplicate: {
5270 Register MaskReg = MI.getOperand(2).getReg();
5271 unsigned MaskBank = getRegBankID(MaskReg, MRI, AMDGPU::SGPRRegBankID);
5272 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 64);
5273 OpdsMapping[2] = AMDGPU::getValueMapping(MaskBank, 32);
5274 break;
5275 }
5276 case Intrinsic::amdgcn_wave_shuffle: {
5277 unsigned OpSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
5278 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, OpSize);
5279 OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, OpSize);
5280 OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, OpSize);
5281 break;
5282 }
5283 }
5284 break;
5285 }
5286 case AMDGPU::G_AMDGPU_INTRIN_IMAGE_LOAD:
5287 case AMDGPU::G_AMDGPU_INTRIN_IMAGE_LOAD_D16:
5288 case AMDGPU::G_AMDGPU_INTRIN_IMAGE_LOAD_NORET:
5289 case AMDGPU::G_AMDGPU_INTRIN_IMAGE_STORE:
5290 case AMDGPU::G_AMDGPU_INTRIN_IMAGE_STORE_D16: {
5291 auto IntrID = AMDGPU::getIntrinsicID(MI);
5292 const AMDGPU::RsrcIntrinsic *RSrcIntrin = AMDGPU::lookupRsrcIntrinsic(IntrID);
5293 assert(RSrcIntrin && "missing RsrcIntrinsic for image intrinsic");
5294 // Non-images can have complications from operands that allow both SGPR
5295 // and VGPR. For now it's too complicated to figure out the final opcode
5296 // to derive the register bank from the MCInstrDesc.
5297 assert(RSrcIntrin->IsImage);
5298 return getImageMapping(MRI, MI, RSrcIntrin->RsrcArg);
5299 }
5300 case AMDGPU::G_AMDGPU_BVH_INTERSECT_RAY:
5301 case AMDGPU::G_AMDGPU_BVH8_INTERSECT_RAY:
5302 case AMDGPU::G_AMDGPU_BVH_DUAL_INTERSECT_RAY: {
5303 bool IsDualOrBVH8 =
5304 MI.getOpcode() == AMDGPU::G_AMDGPU_BVH_DUAL_INTERSECT_RAY ||
5305 MI.getOpcode() == AMDGPU::G_AMDGPU_BVH8_INTERSECT_RAY;
5306 unsigned NumMods = IsDualOrBVH8 ? 0 : 1; // Has A16 modifier
5307 unsigned LastRegOpIdx = MI.getNumExplicitOperands() - 1 - NumMods;
5308 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
5309 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, DstSize);
5310 if (IsDualOrBVH8) {
5311 OpdsMapping[1] = AMDGPU::getValueMapping(
5312 AMDGPU::VGPRRegBankID,
5313 MRI.getType(MI.getOperand(1).getReg()).getSizeInBits());
5314 OpdsMapping[2] = AMDGPU::getValueMapping(
5315 AMDGPU::VGPRRegBankID,
5316 MRI.getType(MI.getOperand(2).getReg()).getSizeInBits());
5317 }
5318 OpdsMapping[LastRegOpIdx] =
5319 getSGPROpMapping(MI.getOperand(LastRegOpIdx).getReg(), MRI, *TRI);
5320 if (LastRegOpIdx == 3) {
5321 // Sequential form: all operands combined into VGPR256/VGPR512
5322 unsigned Size = MRI.getType(MI.getOperand(2).getReg()).getSizeInBits();
5323 if (Size > 256)
5324 Size = 512;
5325 OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5326 } else {
5327 // NSA form
5328 unsigned FirstSrcOpIdx = IsDualOrBVH8 ? 4 : 2;
5329 for (unsigned I = FirstSrcOpIdx; I < LastRegOpIdx; ++I) {
5330 unsigned Size = MRI.getType(MI.getOperand(I).getReg()).getSizeInBits();
5331 OpdsMapping[I] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5332 }
5333 }
5334 break;
5335 }
5336 case AMDGPU::G_INTRINSIC_W_SIDE_EFFECTS:
5337 case AMDGPU::G_INTRINSIC_CONVERGENT_W_SIDE_EFFECTS: {
5338 auto IntrID = cast<GIntrinsic>(MI).getIntrinsicID();
5339 switch (IntrID) {
5340 case Intrinsic::amdgcn_s_getreg:
5341 case Intrinsic::amdgcn_s_memtime:
5342 case Intrinsic::amdgcn_s_memrealtime:
5343 case Intrinsic::amdgcn_s_get_waveid_in_workgroup:
5344 case Intrinsic::amdgcn_s_sendmsg_rtn: {
5345 unsigned Size = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
5346 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size);
5347 break;
5348 }
5349 case Intrinsic::amdgcn_global_atomic_fmin_num:
5350 case Intrinsic::amdgcn_global_atomic_fmax_num:
5351 case Intrinsic::amdgcn_flat_atomic_fmin_num:
5352 case Intrinsic::amdgcn_flat_atomic_fmax_num:
5353 case Intrinsic::amdgcn_global_atomic_ordered_add_b64:
5354 case Intrinsic::amdgcn_global_load_tr_b64:
5355 case Intrinsic::amdgcn_global_load_tr_b128:
5356 case Intrinsic::amdgcn_global_load_tr4_b64:
5357 case Intrinsic::amdgcn_global_load_tr6_b96:
5358 case Intrinsic::amdgcn_ds_load_tr8_b64:
5359 case Intrinsic::amdgcn_ds_load_tr16_b128:
5360 case Intrinsic::amdgcn_ds_load_tr4_b64:
5361 case Intrinsic::amdgcn_ds_load_tr6_b96:
5362 case Intrinsic::amdgcn_ds_read_tr4_b64:
5363 case Intrinsic::amdgcn_ds_read_tr6_b96:
5364 case Intrinsic::amdgcn_ds_read_tr8_b64:
5365 case Intrinsic::amdgcn_ds_read_tr16_b64:
5366 case Intrinsic::amdgcn_ds_atomic_async_barrier_arrive_b64:
5367 case Intrinsic::amdgcn_ds_atomic_barrier_arrive_rtn_b64:
5369 case Intrinsic::amdgcn_ds_ordered_add:
5370 case Intrinsic::amdgcn_ds_ordered_swap: {
5371 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
5372 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, DstSize);
5373 unsigned M0Bank = getRegBankID(MI.getOperand(2).getReg(), MRI,
5374 AMDGPU::SGPRRegBankID);
5375 OpdsMapping[2] = AMDGPU::getValueMapping(M0Bank, 32);
5376 OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5377 break;
5378 }
5379 case Intrinsic::amdgcn_ds_append:
5380 case Intrinsic::amdgcn_ds_consume: {
5381 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
5382 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, DstSize);
5383 OpdsMapping[2] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5384 break;
5385 }
5386 case Intrinsic::amdgcn_exp_compr:
5387 OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5388 OpdsMapping[4] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5389 break;
5390 case Intrinsic::amdgcn_exp:
5391 // FIXME: Could we support packed types here?
5392 OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5393 OpdsMapping[4] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5394 OpdsMapping[5] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5395 OpdsMapping[6] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5396 break;
5397 case Intrinsic::amdgcn_exp_row:
5398 OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5399 OpdsMapping[4] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5400 OpdsMapping[5] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5401 OpdsMapping[6] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5402 OpdsMapping[8] = getSGPROpMapping(MI.getOperand(8).getReg(), MRI, *TRI);
5403 break;
5404 case Intrinsic::amdgcn_s_alloc_vgpr:
5405 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 1);
5406 OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 32);
5407 break;
5408 case Intrinsic::amdgcn_s_sendmsg:
5409 case Intrinsic::amdgcn_s_sendmsghalt: {
5410 // This must be an SGPR, but accept a VGPR.
5411 unsigned Bank = getRegBankID(MI.getOperand(2).getReg(), MRI,
5412 AMDGPU::SGPRRegBankID);
5413 OpdsMapping[2] = AMDGPU::getValueMapping(Bank, 32);
5414 break;
5415 }
5416 case Intrinsic::amdgcn_s_setreg: {
5417 // This must be an SGPR, but accept a VGPR.
5418 unsigned Bank = getRegBankID(MI.getOperand(2).getReg(), MRI,
5419 AMDGPU::SGPRRegBankID);
5420 OpdsMapping[2] = AMDGPU::getValueMapping(Bank, 32);
5421 break;
5422 }
5423 case Intrinsic::amdgcn_s_ttracedata: {
5424 // This must be an SGPR, but accept a VGPR.
5425 unsigned Bank =
5426 getRegBankID(MI.getOperand(1).getReg(), MRI, AMDGPU::SGPRRegBankID);
5427 OpdsMapping[1] = AMDGPU::getValueMapping(Bank, 32);
5428 break;
5429 }
5430 case Intrinsic::amdgcn_end_cf: {
5431 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(1).getReg(), MRI);
5432 OpdsMapping[1] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size);
5433 break;
5434 }
5435 case Intrinsic::amdgcn_else: {
5436 unsigned WaveSize = TRI->getRegSizeInBits(MI.getOperand(1).getReg(), MRI);
5437 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, 1);
5438 OpdsMapping[1] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, WaveSize);
5439 OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, WaveSize);
5440 break;
5441 }
5442 case Intrinsic::amdgcn_init_whole_wave:
5443 case Intrinsic::amdgcn_live_mask: {
5444 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, 1);
5445 break;
5446 }
5447 case Intrinsic::amdgcn_wqm_demote:
5448 case Intrinsic::amdgcn_kill: {
5449 OpdsMapping[1] = AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, 1);
5450 break;
5451 }
5452 case Intrinsic::amdgcn_ptr_s_buffer_load: {
5453 OpdsMapping[0] = getSGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
5454 OpdsMapping[2] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5455 OpdsMapping[3] = getSGPROpMapping(MI.getOperand(3).getReg(), MRI, *TRI);
5456 break;
5457 }
5458 case Intrinsic::amdgcn_raw_buffer_load:
5459 case Intrinsic::amdgcn_raw_ptr_buffer_load:
5460 case Intrinsic::amdgcn_raw_atomic_buffer_load:
5461 case Intrinsic::amdgcn_raw_ptr_atomic_buffer_load:
5462 case Intrinsic::amdgcn_raw_tbuffer_load:
5463 case Intrinsic::amdgcn_raw_ptr_tbuffer_load: {
5464 // FIXME: Should make intrinsic ID the last operand of the instruction,
5465 // then this would be the same as store
5466 OpdsMapping[0] = getVGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
5467 OpdsMapping[2] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5468 OpdsMapping[3] = getVGPROpMapping(MI.getOperand(3).getReg(), MRI, *TRI);
5469 OpdsMapping[4] = getSGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI);
5470 break;
5471 }
5472 case Intrinsic::amdgcn_raw_buffer_load_lds:
5473 case Intrinsic::amdgcn_raw_buffer_load_async_lds:
5474 case Intrinsic::amdgcn_raw_ptr_buffer_load_lds:
5475 case Intrinsic::amdgcn_raw_ptr_buffer_load_async_lds: {
5476 OpdsMapping[1] = getSGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
5477 OpdsMapping[2] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5478 OpdsMapping[4] = getVGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI);
5479 OpdsMapping[5] = getSGPROpMapping(MI.getOperand(5).getReg(), MRI, *TRI);
5480 break;
5481 }
5482 case Intrinsic::amdgcn_raw_buffer_store:
5483 case Intrinsic::amdgcn_raw_ptr_buffer_store:
5484 case Intrinsic::amdgcn_raw_buffer_store_format:
5485 case Intrinsic::amdgcn_raw_ptr_buffer_store_format:
5486 case Intrinsic::amdgcn_raw_tbuffer_store:
5487 case Intrinsic::amdgcn_raw_ptr_tbuffer_store: {
5488 OpdsMapping[1] = getVGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
5489 OpdsMapping[2] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5490 OpdsMapping[3] = getVGPROpMapping(MI.getOperand(3).getReg(), MRI, *TRI);
5491 OpdsMapping[4] = getSGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI);
5492 break;
5493 }
5494 case Intrinsic::amdgcn_struct_buffer_load:
5495 case Intrinsic::amdgcn_struct_ptr_buffer_load:
5496 case Intrinsic::amdgcn_struct_tbuffer_load:
5497 case Intrinsic::amdgcn_struct_ptr_tbuffer_load:
5498 case Intrinsic::amdgcn_struct_atomic_buffer_load:
5499 case Intrinsic::amdgcn_struct_ptr_atomic_buffer_load: {
5500 OpdsMapping[0] = getVGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
5501 OpdsMapping[2] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5502 OpdsMapping[3] = getVGPROpMapping(MI.getOperand(3).getReg(), MRI, *TRI);
5503 OpdsMapping[4] = getVGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI);
5504 OpdsMapping[5] = getSGPROpMapping(MI.getOperand(5).getReg(), MRI, *TRI);
5505 break;
5506 }
5507 case Intrinsic::amdgcn_struct_buffer_load_lds:
5508 case Intrinsic::amdgcn_struct_buffer_load_async_lds:
5509 case Intrinsic::amdgcn_struct_ptr_buffer_load_lds:
5510 case Intrinsic::amdgcn_struct_ptr_buffer_load_async_lds: {
5511 OpdsMapping[1] = getSGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
5512 OpdsMapping[2] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5513 OpdsMapping[4] = getVGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI);
5514 OpdsMapping[5] = getVGPROpMapping(MI.getOperand(5).getReg(), MRI, *TRI);
5515 OpdsMapping[6] = getSGPROpMapping(MI.getOperand(6).getReg(), MRI, *TRI);
5516 break;
5517 }
5518 case Intrinsic::amdgcn_struct_buffer_store:
5519 case Intrinsic::amdgcn_struct_ptr_buffer_store:
5520 case Intrinsic::amdgcn_struct_tbuffer_store:
5521 case Intrinsic::amdgcn_struct_ptr_tbuffer_store: {
5522 OpdsMapping[1] = getVGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
5523 OpdsMapping[2] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5524 OpdsMapping[3] = getVGPROpMapping(MI.getOperand(3).getReg(), MRI, *TRI);
5525 OpdsMapping[4] = getVGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI);
5526 OpdsMapping[5] = getSGPROpMapping(MI.getOperand(5).getReg(), MRI, *TRI);
5527 break;
5528 }
5529 case Intrinsic::amdgcn_init_exec_from_input: {
5530 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(1).getReg(), MRI);
5531 OpdsMapping[1] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size);
5532 break;
5533 }
5534 case Intrinsic::amdgcn_ds_gws_init:
5535 case Intrinsic::amdgcn_ds_gws_barrier:
5536 case Intrinsic::amdgcn_ds_gws_sema_br: {
5537 OpdsMapping[1] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5538
5539 // This must be an SGPR, but accept a VGPR.
5540 unsigned Bank = getRegBankID(MI.getOperand(2).getReg(), MRI,
5541 AMDGPU::SGPRRegBankID);
5542 OpdsMapping[2] = AMDGPU::getValueMapping(Bank, 32);
5543 break;
5544 }
5545 case Intrinsic::amdgcn_ds_gws_sema_v:
5546 case Intrinsic::amdgcn_ds_gws_sema_p:
5547 case Intrinsic::amdgcn_ds_gws_sema_release_all: {
5548 // This must be an SGPR, but accept a VGPR.
5549 unsigned Bank = getRegBankID(MI.getOperand(1).getReg(), MRI,
5550 AMDGPU::SGPRRegBankID);
5551 OpdsMapping[1] = AMDGPU::getValueMapping(Bank, 32);
5552 break;
5553 }
5554 case Intrinsic::amdgcn_cluster_load_b32:
5555 case Intrinsic::amdgcn_cluster_load_b64:
5556 case Intrinsic::amdgcn_cluster_load_b128: {
5557 OpdsMapping[0] = getVGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
5558 OpdsMapping[2] = getVGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5559 unsigned M0Bank =
5560 getRegBankID(MI.getOperand(4).getReg(), MRI, AMDGPU::SGPRRegBankID);
5561 OpdsMapping[4] = AMDGPU::getValueMapping(M0Bank, 32);
5562 break;
5563 }
5564 case Intrinsic::amdgcn_cluster_load_async_to_lds_b8:
5565 case Intrinsic::amdgcn_cluster_load_async_to_lds_b32:
5566 case Intrinsic::amdgcn_cluster_load_async_to_lds_b64:
5567 case Intrinsic::amdgcn_cluster_load_async_to_lds_b128: {
5568 OpdsMapping[1] = getVGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
5569 // LDS address goes into $vdst (VGPR).
5570 OpdsMapping[2] = getVGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5571 unsigned M0Bank =
5572 getRegBankID(MI.getOperand(5).getReg(), MRI, AMDGPU::SGPRRegBankID);
5573 OpdsMapping[5] = AMDGPU::getValueMapping(M0Bank, 32);
5574 break;
5575 }
5576 case Intrinsic::amdgcn_global_store_async_from_lds_b8:
5577 case Intrinsic::amdgcn_global_store_async_from_lds_b32:
5578 case Intrinsic::amdgcn_global_store_async_from_lds_b64:
5579 case Intrinsic::amdgcn_global_store_async_from_lds_b128:
5580 case Intrinsic::amdgcn_global_load_async_to_lds_b8:
5581 case Intrinsic::amdgcn_global_load_async_to_lds_b32:
5582 case Intrinsic::amdgcn_global_load_async_to_lds_b64:
5583 case Intrinsic::amdgcn_global_load_async_to_lds_b128: {
5584 OpdsMapping[1] = getVGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
5585 // LDS address goes into $vdst/$vdata (VGPR).
5586 OpdsMapping[2] = getVGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5587 break;
5588 }
5589 case Intrinsic::amdgcn_load_to_lds:
5590 case Intrinsic::amdgcn_load_async_to_lds:
5591 case Intrinsic::amdgcn_global_load_lds:
5592 case Intrinsic::amdgcn_global_load_async_lds: {
5593 OpdsMapping[1] = getVGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
5594 // LDS address goes into M0 (SGPR).
5595 OpdsMapping[2] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5596 break;
5597 }
5598 case Intrinsic::amdgcn_lds_direct_load: {
5599 const int M0Idx = MI.getNumOperands() - 1;
5600 Register M0Reg = MI.getOperand(M0Idx).getReg();
5601 unsigned M0Bank = getRegBankID(M0Reg, MRI, AMDGPU::SGPRRegBankID);
5602 unsigned DstSize = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
5603
5604 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, DstSize);
5605 for (int I = 2; I != M0Idx && MI.getOperand(I).isReg(); ++I)
5606 OpdsMapping[I] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, 32);
5607
5608 // Must be SGPR, but we must take whatever the original bank is and fix it
5609 // later.
5610 OpdsMapping[M0Idx] = AMDGPU::getValueMapping(M0Bank, 32);
5611 break;
5612 }
5613 case Intrinsic::amdgcn_ds_add_gs_reg_rtn:
5614 case Intrinsic::amdgcn_ds_sub_gs_reg_rtn:
5615 OpdsMapping[0] = getVGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
5616 OpdsMapping[2] = getVGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5617 break;
5618 case Intrinsic::amdgcn_ds_bvh_stack_rtn:
5619 case Intrinsic::amdgcn_ds_bvh_stack_push4_pop1_rtn:
5620 case Intrinsic::amdgcn_ds_bvh_stack_push8_pop1_rtn:
5621 case Intrinsic::amdgcn_ds_bvh_stack_push8_pop2_rtn: {
5622 OpdsMapping[0] =
5623 getVGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI); // %vdst
5624 OpdsMapping[1] =
5625 getVGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI); // %addr
5626 OpdsMapping[3] =
5627 getVGPROpMapping(MI.getOperand(3).getReg(), MRI, *TRI); // %addr
5628 OpdsMapping[4] =
5629 getVGPROpMapping(MI.getOperand(4).getReg(), MRI, *TRI); // %data0
5630 OpdsMapping[5] =
5631 getVGPROpMapping(MI.getOperand(5).getReg(), MRI, *TRI); // %data1
5632 break;
5633 }
5634 case Intrinsic::amdgcn_s_sleep_var:
5635 OpdsMapping[1] = getSGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
5636 break;
5637 case Intrinsic::amdgcn_s_barrier_join:
5638 case Intrinsic::amdgcn_s_wakeup_barrier:
5639 OpdsMapping[1] = getSGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
5640 break;
5641 case Intrinsic::amdgcn_s_barrier_init:
5642 case Intrinsic::amdgcn_s_barrier_signal_var:
5643 OpdsMapping[1] = getSGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
5644 OpdsMapping[2] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5645 break;
5646 case Intrinsic::amdgcn_s_barrier_signal_isfirst: {
5647 const unsigned ResultSize = 1;
5648 OpdsMapping[0] =
5649 AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, ResultSize);
5650 break;
5651 }
5652 case Intrinsic::amdgcn_s_get_barrier_state:
5653 case Intrinsic::amdgcn_s_get_named_barrier_state: {
5654 OpdsMapping[0] = getSGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
5655 OpdsMapping[2] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5656 break;
5657 }
5658 case Intrinsic::amdgcn_pops_exiting_wave_id:
5659 return getDefaultMappingSOP(MI);
5660 case Intrinsic::amdgcn_tensor_load_to_lds:
5661 case Intrinsic::amdgcn_tensor_store_from_lds: {
5662 // Lie and claim everything is legal, even all operands need to be
5663 // SGPRs. applyMapping will have to deal with it with readfirstlane.
5664 for (unsigned I = 1; I < MI.getNumOperands(); ++I) {
5665 if (MI.getOperand(I).isReg()) {
5666 Register Reg = MI.getOperand(I).getReg();
5667 auto OpBank = getRegBankID(Reg, MRI);
5668 unsigned Size = TRI->getRegSizeInBits(Reg, MRI);
5669 OpdsMapping[I] = AMDGPU::getValueMapping(OpBank, Size);
5670 }
5671 }
5672 break;
5673 }
5674 case Intrinsic::amdgcn_s_prefetch_data:
5675 case Intrinsic::amdgcn_s_prefetch_inst: {
5676 OpdsMapping[1] = getSGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
5677 OpdsMapping[2] = getSGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5678 break;
5679 }
5680 case Intrinsic::amdgcn_flat_prefetch:
5681 case Intrinsic::amdgcn_global_prefetch:
5682 return getDefaultMappingVOP(MI);
5683 default:
5685 }
5686 break;
5687 }
5688 case AMDGPU::G_SELECT: {
5689 unsigned Size = MRI.getType(MI.getOperand(0).getReg()).getSizeInBits();
5690 unsigned Op2Bank = getRegBankID(MI.getOperand(2).getReg(), MRI,
5691 AMDGPU::SGPRRegBankID);
5692 unsigned Op3Bank = getRegBankID(MI.getOperand(3).getReg(), MRI,
5693 AMDGPU::SGPRRegBankID);
5694 bool SGPRSrcs = Op2Bank == AMDGPU::SGPRRegBankID &&
5695 Op3Bank == AMDGPU::SGPRRegBankID;
5696
5697 unsigned CondBankDefault = SGPRSrcs ?
5698 AMDGPU::SGPRRegBankID : AMDGPU::VCCRegBankID;
5699 unsigned CondBank = getRegBankID(MI.getOperand(1).getReg(), MRI,
5700 CondBankDefault);
5701 if (CondBank == AMDGPU::SGPRRegBankID)
5702 CondBank = SGPRSrcs ? AMDGPU::SGPRRegBankID : AMDGPU::VCCRegBankID;
5703 else if (CondBank == AMDGPU::VGPRRegBankID)
5704 CondBank = AMDGPU::VCCRegBankID;
5705
5706 unsigned Bank = SGPRSrcs && CondBank == AMDGPU::SGPRRegBankID ?
5707 AMDGPU::SGPRRegBankID : AMDGPU::VGPRRegBankID;
5708
5709 assert(CondBank == AMDGPU::VCCRegBankID || CondBank == AMDGPU::SGPRRegBankID);
5710
5711 // TODO: Should report 32-bit for scalar condition type.
5712 if (Size == 64) {
5713 OpdsMapping[0] = AMDGPU::getValueMappingSGPR64Only(Bank, Size);
5714 OpdsMapping[1] = AMDGPU::getValueMapping(CondBank, 1);
5715 OpdsMapping[2] = AMDGPU::getValueMappingSGPR64Only(Bank, Size);
5716 OpdsMapping[3] = AMDGPU::getValueMappingSGPR64Only(Bank, Size);
5717 } else {
5718 OpdsMapping[0] = AMDGPU::getValueMapping(Bank, Size);
5719 OpdsMapping[1] = AMDGPU::getValueMapping(CondBank, 1);
5720 OpdsMapping[2] = AMDGPU::getValueMapping(Bank, Size);
5721 OpdsMapping[3] = AMDGPU::getValueMapping(Bank, Size);
5722 }
5723
5724 break;
5725 }
5726
5727 case AMDGPU::G_SI_CALL: {
5728 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, 64);
5729 // Lie and claim everything is legal, even though some need to be
5730 // SGPRs. applyMapping will have to deal with it as a waterfall loop.
5731 OpdsMapping[1] = getSGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
5732
5733 // Allow anything for implicit arguments
5734 for (unsigned I = 4; I < MI.getNumOperands(); ++I) {
5735 if (MI.getOperand(I).isReg()) {
5736 Register Reg = MI.getOperand(I).getReg();
5737 auto OpBank = getRegBankID(Reg, MRI);
5738 unsigned Size = TRI->getRegSizeInBits(Reg, MRI);
5739 OpdsMapping[I] = AMDGPU::getValueMapping(OpBank, Size);
5740 }
5741 }
5742 break;
5743 }
5744 case AMDGPU::G_LOAD:
5745 case AMDGPU::G_ZEXTLOAD:
5746 case AMDGPU::G_SEXTLOAD:
5747 return getInstrMappingForLoad(MI);
5748
5749 case AMDGPU::G_ATOMICRMW_XCHG:
5750 case AMDGPU::G_ATOMICRMW_ADD:
5751 case AMDGPU::G_ATOMICRMW_SUB:
5752 case AMDGPU::G_ATOMICRMW_AND:
5753 case AMDGPU::G_ATOMICRMW_OR:
5754 case AMDGPU::G_ATOMICRMW_XOR:
5755 case AMDGPU::G_ATOMICRMW_MAX:
5756 case AMDGPU::G_ATOMICRMW_MIN:
5757 case AMDGPU::G_ATOMICRMW_UMAX:
5758 case AMDGPU::G_ATOMICRMW_UMIN:
5759 case AMDGPU::G_ATOMICRMW_FADD:
5760 case AMDGPU::G_ATOMICRMW_FMIN:
5761 case AMDGPU::G_ATOMICRMW_FMAX:
5762 case AMDGPU::G_ATOMICRMW_UINC_WRAP:
5763 case AMDGPU::G_ATOMICRMW_UDEC_WRAP:
5764 case AMDGPU::G_ATOMICRMW_USUB_COND:
5765 case AMDGPU::G_ATOMICRMW_USUB_SAT:
5766 case AMDGPU::G_AMDGPU_ATOMIC_CMPXCHG: {
5767 OpdsMapping[0] = getVGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
5768 OpdsMapping[1] = getValueMappingForPtr(MRI, MI.getOperand(1).getReg());
5769 OpdsMapping[2] = getVGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5770 break;
5771 }
5772 case AMDGPU::G_ATOMIC_CMPXCHG: {
5773 OpdsMapping[0] = getVGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
5774 OpdsMapping[1] = getValueMappingForPtr(MRI, MI.getOperand(1).getReg());
5775 OpdsMapping[2] = getVGPROpMapping(MI.getOperand(2).getReg(), MRI, *TRI);
5776 OpdsMapping[3] = getVGPROpMapping(MI.getOperand(3).getReg(), MRI, *TRI);
5777 break;
5778 }
5779 case AMDGPU::G_BRCOND: {
5780 unsigned Bank = getRegBankID(MI.getOperand(0).getReg(), MRI,
5781 AMDGPU::SGPRRegBankID);
5782 assert(MRI.getType(MI.getOperand(0).getReg()).getSizeInBits() == 1);
5783 if (Bank != AMDGPU::SGPRRegBankID)
5784 Bank = AMDGPU::VCCRegBankID;
5785
5786 OpdsMapping[0] = AMDGPU::getValueMapping(Bank, 1);
5787 break;
5788 }
5789 case AMDGPU::G_INTRINSIC_FPTRUNC_ROUND:
5790 return getDefaultMappingVOP(MI);
5791 case AMDGPU::G_PREFETCH:
5792 OpdsMapping[0] = getSGPROpMapping(MI.getOperand(0).getReg(), MRI, *TRI);
5793 break;
5794 case AMDGPU::G_AMDGPU_WHOLE_WAVE_FUNC_SETUP:
5795 case AMDGPU::G_AMDGPU_WHOLE_WAVE_FUNC_RETURN:
5796 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VCCRegBankID, 1);
5797 break;
5798 case AMDGPU::G_AMDGPU_FLAT_LOAD_MONITOR:
5799 case AMDGPU::G_AMDGPU_GLOBAL_LOAD_MONITOR: {
5800 unsigned Size = TRI->getRegSizeInBits(MI.getOperand(0).getReg(), MRI);
5801 unsigned PtrSize = TRI->getRegSizeInBits(MI.getOperand(1).getReg(), MRI);
5802 OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, Size);
5803 OpdsMapping[1] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, PtrSize);
5804 break;
5805 }
5806 }
5807
5808 return getInstructionMapping(/*ID*/1, /*Cost*/1,
5809 getOperandsMapping(OpdsMapping),
5810 MI.getNumOperands());
5811}
unsigned NumParts
static unsigned getIntrinsicID(const SDNode *N)
assert(UImm &&(UImm !=~static_cast< T >(0)) &&"Invalid immediate!")
uint64_t Imm
Contains the definition of a TargetInstrInfo class that is common to all AMD GPUs.
constexpr LLT S16
constexpr LLT S1
constexpr LLT S32
constexpr LLT S64
AMDGPU Register Bank Select
static bool substituteSimpleCopyRegs(const AMDGPURegisterBankInfo::OperandsMapper &OpdMapper, unsigned OpIdx)
static unsigned regBankBoolUnion(unsigned RB0, unsigned RB1)
static Register constrainRegToBank(MachineRegisterInfo &MRI, MachineIRBuilder &B, Register &Reg, const RegisterBank &Bank)
static std::pair< Register, Register > unpackV2S16ToS32(MachineIRBuilder &B, Register Src, unsigned ExtOpcode)
static void extendLow32IntoHigh32(MachineIRBuilder &B, Register Hi32Reg, Register Lo32Reg, unsigned ExtOpc, const RegisterBank &RegBank, bool IsBooleanSrc=false)
Implement extending a 32-bit value to a 64-bit value.
static unsigned getExtendOp(unsigned Opc)
static bool isVectorRegisterBank(const RegisterBank &Bank)
static unsigned regBankUnion(unsigned RB0, unsigned RB1)
static std::pair< LLT, LLT > splitUnequalType(LLT Ty, unsigned FirstSize)
Split Ty into 2 pieces.
static void setRegsToType(MachineRegisterInfo &MRI, ArrayRef< Register > Regs, LLT NewTy)
Replace the current type each register in Regs has with NewTy.
static void reinsertVectorIndexAdd(MachineIRBuilder &B, MachineInstr &IdxUseInstr, unsigned OpIdx, unsigned ConstOffset)
Utility function for pushing dynamic vector indexes with a constant offset into waterfall loops.
static LLT widen96To128(LLT Ty)
static LLT getHalfSizedType(LLT Ty)
static unsigned getSBufferLoadCorrespondingBufferLoadOpcode(unsigned Opc)
This file declares the targeting of the RegisterBankInfo class for AMDGPU.
Rewrite undef for PHI
MachineBasicBlock & MBB
MachineBasicBlock MachineBasicBlock::iterator DebugLoc DL
MachineBasicBlock MachineBasicBlock::iterator MBBI
#define X(NUM, ENUM, NAME)
Definition ELF.h:665
static GCRegistry::Add< OcamlGC > B("ocaml", "ocaml 3.10-compatible GC")
AMD GCN specific subclass of TargetSubtarget.
Declares convenience wrapper classes for interpreting MachineInstr instances as specific generic oper...
IRTranslator LLVM IR MI
const size_t AbstractManglingParser< Derived, Alloc >::NumOps
const AbstractManglingParser< Derived, Alloc >::OperatorInfo AbstractManglingParser< Derived, Alloc >::Ops[]
#define I(x, y, z)
Definition MD5.cpp:57
Contains matchers for matching SSA Machine Instructions.
This file declares the MachineIRBuilder class.
Register Reg
Promote Memory to Register
Definition Mem2Reg.cpp:110
static bool isReg(const MCInst &MI, unsigned OpNo)
ConstantRange Range(APInt(BitWidth, Low), APInt(BitWidth, High))
static constexpr MCPhysReg SPReg
MachineRegisterInfo & MRI
SI Fold Operands
Interface definition for SIRegisterInfo.
static TableGen::Emitter::Opt Y("gen-skeleton-entry", EmitSkeleton, "Generate example skeleton entry")
bool applyMappingDynStackAlloc(MachineIRBuilder &B, const OperandsMapper &OpdMapper, MachineInstr &MI) const
bool collectWaterfallOperands(SmallSet< Register, 4 > &SGPROperandRegs, MachineInstr &MI, MachineRegisterInfo &MRI, ArrayRef< unsigned > OpIndices) const
const InstructionMapping & getImageMapping(const MachineRegisterInfo &MRI, const MachineInstr &MI, int RsrcIdx) const
InstructionMappings addMappingFromTable(const MachineInstr &MI, const MachineRegisterInfo &MRI, const std::array< unsigned, NumOps > RegSrcOpIdx, ArrayRef< OpRegBankEntry< NumOps > > Table) const
unsigned copyCost(const RegisterBank &A, const RegisterBank &B, TypeSize Size) const override
Get the cost of a copy from B to A, or put differently, get the cost of A = COPY B.
RegisterBankInfo::InstructionMappings getInstrAlternativeMappingsIntrinsicWSideEffects(const MachineInstr &MI, const MachineRegisterInfo &MRI) const
bool buildVCopy(MachineIRBuilder &B, Register DstReg, Register SrcReg) const
bool executeInWaterfallLoop(MachineIRBuilder &B, iterator_range< MachineBasicBlock::iterator > Range, SmallSet< Register, 4 > &SGPROperandRegs) const
Legalize instruction MI where operands in OpIndices must be SGPRs.
const RegisterBank & getRegBankFromRegClass(const TargetRegisterClass &RC, LLT) const override
Get a register bank that covers RC.
AMDGPURegisterBankInfo(const GCNSubtarget &STI)
bool applyMappingMAD_64_32(MachineIRBuilder &B, const OperandsMapper &OpdMapper) const
unsigned getRegBankID(Register Reg, const MachineRegisterInfo &MRI, unsigned Default=AMDGPU::VGPRRegBankID) const
Register handleD16VData(MachineIRBuilder &B, MachineRegisterInfo &MRI, Register Reg) const
Handle register layout difference for f16 images for some subtargets.
const RegisterBankInfo::InstructionMapping & getInstrMappingForLoad(const MachineInstr &MI) const
void applyMappingImpl(MachineIRBuilder &Builder, const OperandsMapper &OpdMapper) const override
See RegisterBankInfo::applyMapping.
bool applyMappingBFE(MachineIRBuilder &B, const OperandsMapper &OpdMapper, bool Signed) const
bool applyMappingImage(MachineIRBuilder &B, MachineInstr &MI, const OperandsMapper &OpdMapper, int RSrcIdx) const
const ValueMapping * getVGPROpMapping(Register Reg, const MachineRegisterInfo &MRI, const TargetRegisterInfo &TRI) const
bool isScalarLoadLegal(const MachineInstr &MI) const
unsigned setBufferOffsets(MachineIRBuilder &B, Register CombinedOffset, Register &VOffsetReg, Register &SOffsetReg, int64_t &InstOffsetVal, Align Alignment) const
const ValueMapping * getSGPROpMapping(Register Reg, const MachineRegisterInfo &MRI, const TargetRegisterInfo &TRI) const
bool applyMappingLoad(MachineIRBuilder &B, const OperandsMapper &OpdMapper, MachineInstr &MI) const
void split64BitValueForMapping(MachineIRBuilder &B, SmallVector< Register, 2 > &Regs, LLT HalfTy, Register Reg) const
Split 64-bit value Reg into two 32-bit halves and populate them into Regs.
const ValueMapping * getValueMappingForPtr(const MachineRegisterInfo &MRI, Register Ptr) const
Return the mapping for a pointer argument.
unsigned getMappingType(const MachineRegisterInfo &MRI, const MachineInstr &MI) const
RegisterBankInfo::InstructionMappings getInstrAlternativeMappingsIntrinsic(const MachineInstr &MI, const MachineRegisterInfo &MRI) const
bool isDivergentRegBank(const RegisterBank *RB) const override
Returns true if the register bank is considered divergent.
void constrainOpWithReadfirstlane(MachineIRBuilder &B, MachineInstr &MI, unsigned OpIdx) const
InstructionMappings getInstrAlternativeMappings(const MachineInstr &MI) const override
Get the alternative mappings for MI.
const InstructionMapping & getDefaultMappingSOP(const MachineInstr &MI) const
const InstructionMapping & getDefaultMappingAllVGPR(const MachineInstr &MI) const
const InstructionMapping & getInstrMapping(const MachineInstr &MI) const override
This function must return a legal mapping, because AMDGPURegisterBankInfo::getInstrAlternativeMapping...
unsigned getBreakDownCost(const ValueMapping &ValMapping, const RegisterBank *CurBank=nullptr) const override
Get the cost of using ValMapping to decompose a register.
const ValueMapping * getAGPROpMapping(Register Reg, const MachineRegisterInfo &MRI, const TargetRegisterInfo &TRI) const
const InstructionMapping & getDefaultMappingVOP(const MachineInstr &MI) const
bool isSALUMapping(const MachineInstr &MI) const
Register buildReadFirstLane(MachineIRBuilder &B, MachineRegisterInfo &MRI, Register Src) const
bool applyMappingSBufferLoad(MachineIRBuilder &B, const OperandsMapper &OpdMapper) const
void applyMappingSMULU64(MachineIRBuilder &B, const OperandsMapper &OpdMapper) const
static const LaneMaskConstants & get(const GCNSubtarget &ST)
Represent a constant reference to an array (0 or more elements consecutively in memory),...
Definition ArrayRef.h:40
Predicate
This enumeration lists the possible predicates for CmpInst subclasses.
Definition InstrTypes.h:740
@ ICMP_SLT
signed less than
Definition InstrTypes.h:769
@ ICMP_NE
not equal
Definition InstrTypes.h:762
A debug info location.
Definition DebugLoc.h:126
iterator find(const_arg_type_t< KeyT > Val)
Definition DenseMap.h:766
iterator end()
Definition DenseMap.h:686
std::pair< iterator, bool > insert(const std::pair< KeyT, ValueT > &KV)
Definition DenseMap.h:827
static constexpr ElementCount getFixed(ScalarTy MinVal)
Definition TypeSize.h:306
Abstract class that contains various methods for clients to notify about changes.
constexpr unsigned getScalarSizeInBits() const
constexpr bool isScalar() const
LLT getScalarType() const
static constexpr LLT scalar(unsigned SizeInBits)
Get a low-level scalar or aggregate "bag of bits".
constexpr uint16_t getNumElements() const
Returns the number of elements in a vector LLT.
constexpr bool isVector() const
constexpr TypeSize getSizeInBits() const
Returns the total size of the type. Must only be called on sized types.
LLT divide(int Factor) const
Return a type that is Factor times smaller.
constexpr unsigned getAddressSpace() const
static constexpr LLT fixed_vector(unsigned NumElements, unsigned ScalarSizeInBits)
Get a low-level fixed-width vector of some number of elements and element width.
LLT getElementType() const
Returns the vector's element type. Only valid for vector types.
static constexpr LLT scalarOrVector(ElementCount EC, LLT ScalarTy)
This is an important class for using LLVM in a threaded context.
Definition LLVMContext.h:76
LLVM_ABI void widenScalarSrc(MachineInstr &MI, LLT WideTy, unsigned OpIdx, unsigned ExtOpcode)
Legalize a single operand OpIdx of the machine instruction MI as a Use by extending the operand's typ...
LLVM_ABI LegalizeResult lowerAbsToMaxNeg(MachineInstr &MI)
LLVM_ABI LegalizeResult narrowScalar(MachineInstr &MI, unsigned TypeIdx, LLT NarrowTy)
Legalize an instruction by reducing the width of the underlying scalar type.
LLVM_ABI LegalizeResult reduceLoadStoreWidth(GLoadStore &MI, unsigned TypeIdx, LLT NarrowTy)
@ Legalized
Instruction has been legalized and the MachineFunction changed.
LLVM_ABI LegalizeResult fewerElementsVector(MachineInstr &MI, unsigned TypeIdx, LLT NarrowTy)
Legalize a vector instruction by splitting into multiple components, each acting on the same scalar t...
LLVM_ABI LegalizeResult widenScalar(MachineInstr &MI, unsigned TypeIdx, LLT WideTy)
Legalize an instruction by performing the operation on a wider scalar type (for example a 16-bit addi...
LLVM_ABI void widenScalarDst(MachineInstr &MI, LLT WideTy, unsigned OpIdx=0, unsigned TruncOpcode=TargetOpcode::G_TRUNC)
Legalize a single operand OpIdx of the machine instruction MI as a Def by extending the operand's typ...
TypeSize getValue() const
LLVM_ABI void transferSuccessorsAndUpdatePHIs(MachineBasicBlock *FromMBB)
Transfers all the successors, as in transferSuccessors, and update PHI operands in the successor bloc...
LLVM_ABI iterator getFirstTerminator()
Returns an iterator to the first terminator instruction of this basic block.
LLVM_ABI void addSuccessor(MachineBasicBlock *Succ, BranchProbability Prob=BranchProbability::getUnknown())
Add Succ as a successor of this MachineBasicBlock.
const MachineFunction * getParent() const
Return the MachineFunction containing this basic block.
void splice(iterator Where, MachineBasicBlock *Other, iterator From)
Take an instruction from MBB 'Other' at the position From, and insert it into this MBB right before '...
MachineInstrBundleIterator< MachineInstr > iterator
const TargetSubtargetInfo & getSubtarget() const
getSubtarget - Return the subtarget for which this machine code is being compiled.
MachineRegisterInfo & getRegInfo()
getRegInfo - Return information about the registers currently in use.
BasicBlockListType::iterator iterator
Ty * getInfo()
getInfo - Keep track of various per-function pieces of information for backends that would like to do...
MachineMemOperand * getMachineMemOperand(MachinePointerInfo PtrInfo, MachineMemOperand::Flags F, LLT MemTy, Align BaseAlignment, const MMOMetadata &Metadata=MMOMetadata(), SyncScope::ID SSID=SyncScope::System, AtomicOrdering Ordering=AtomicOrdering::NotAtomic, AtomicOrdering FailureOrdering=AtomicOrdering::NotAtomic)
getMachineMemOperand - Allocate a new MachineMemOperand.
MachineBasicBlock * CreateMachineBasicBlock(const BasicBlock *BB=nullptr, std::optional< UniqueBBID > BBID=std::nullopt)
CreateMachineInstr - Allocate a new MachineInstr.
void insert(iterator MBBI, MachineBasicBlock *MBB)
Helper class to build MachineInstr.
const MachineInstrBuilder & addReg(Register RegNo, RegState Flags={}, unsigned SubReg=0) const
Add a new virtual register operand.
MachineInstrSpan provides an interface to get an iteration range containing the instruction it was in...
MachineBasicBlock::iterator begin()
MachineBasicBlock::iterator end()
Representation of each machine instruction.
const MachineBasicBlock * getParent() const
const MachineOperand & getOperand(unsigned i) const
A description of a memory reference used in the backend.
LocationSize getSize() const
Return the size in bytes of the memory reference.
unsigned getAddrSpace() const
bool isAtomic() const
Returns true if this operation has an atomic ordering requirement of unordered or higher,...
@ MODereferenceable
The memory access is dereferenceable (i.e., doesn't trap).
@ MOLoad
The memory access reads data.
@ MOInvariant
The memory access always returns the same value (or traps).
Flags getFlags() const
Return the raw flags of the source value,.
LLVM_ABI Align getAlign() const
Return the minimum known alignment in bytes of the actual memory reference.
MachineOperand class - Representation of each machine instruction operand.
LLVM_ABI void setReg(Register Reg)
Change the register this operand corresponds to.
Register getReg() const
getReg - Returns the register number.
MachineRegisterInfo - Keep track of information for virtual and physical registers,...
LLT getType(Register Reg) const
Get the low-level type of Reg or LLT{} if Reg is not a generic (target independent) virtual register.
LLVM_ABI void setRegBank(Register Reg, const RegisterBank &RegBank)
Set the register bank to RegBank for Reg.
Helper class that represents how the value of an instruction may be mapped and what is the related co...
bool isValid() const
Check whether this object is valid.
Helper class used to get/create the virtual registers that will be used to replace the MachineOperand...
const InstructionMapping & getInstrMapping() const
The final mapping of the instruction.
MachineRegisterInfo & getMRI() const
The MachineRegisterInfo we used to realize the mapping.
LLVM_ABI iterator_range< SmallVectorImpl< Register >::const_iterator > getVRegs(unsigned OpIdx, bool ForDebug=false) const
Get all the virtual registers required to map the OpIdx-th operand of the instruction.
virtual InstructionMappings getInstrAlternativeMappings(const MachineInstr &MI) const
Get the alternative mappings for MI.
static const TargetRegisterClass * constrainGenericRegister(Register Reg, const TargetRegisterClass &RC, MachineRegisterInfo &MRI)
Constrain the (possibly generic) virtual register Reg to RC.
const InstructionMapping & getInstructionMapping(unsigned ID, unsigned Cost, const ValueMapping *OperandsMapping, unsigned NumOperands) const
Method to get a uniquely generated InstructionMapping.
static void applyDefaultMapping(const OperandsMapper &OpdMapper)
Helper method to apply something that is like the default mapping.
const ValueMapping & getValueMapping(unsigned StartIdx, unsigned Length, const RegisterBank &RegBank) const
The most common ValueMapping consists of a single PartialMapping.
const InstructionMapping & getInvalidInstructionMapping() const
Method to get a uniquely generated invalid InstructionMapping.
const RegisterBank & getRegBank(unsigned ID)
Get the register bank identified by ID.
const unsigned * Sizes
Hold the sizes of the register banks for all HwModes.
bool cannotCopy(const RegisterBank &Dst, const RegisterBank &Src, TypeSize Size) const
const ValueMapping * getOperandsMapping(Iterator Begin, Iterator End) const
Get the uniquely generated array of ValueMapping for the elements of between Begin and End.
SmallVector< const InstructionMapping *, 4 > InstructionMappings
Convenient type to represent the alternatives for mapping an instruction.
virtual unsigned copyCost(const RegisterBank &A, const RegisterBank &B, TypeSize Size) const
Get the cost of a copy from B to A, or put differently, get the cost of A = COPY B.
const InstructionMapping & getInstrMappingImpl(const MachineInstr &MI) const
Try to get the mapping of MI.
This class implements the register bank concept.
unsigned getID() const
Get the identifier of this register bank.
Wrapper class representing virtual and physical registers.
Definition Register.h:20
constexpr bool isVirtual() const
Return true if the specified register number is in the virtual register namespace.
Definition Register.h:79
This class keeps track of the SPI_SP_INPUT_ADDR config register, which tells the hardware which inter...
SIModeRegisterDefaults getMode() const
bool selectAGPRFormMFMA(unsigned NumRegs) const
Return true if an MFMA that requires at least NumRegs should select to the AGPR form,...
static bool shouldExpandVectorDynExt(unsigned EltSize, unsigned NumElem, bool IsDivergentIdx, const GCNSubtarget *Subtarget)
Check if EXTRACT_VECTOR_ELT/INSERT_VECTOR_ELT (<n x e>, var-idx) should be expanded into a set of cmp...
SmallSet - This maintains a set of unique values, optimizing for the case when the set is small (less...
Definition SmallSet.h:134
size_type count(const T &V) const
count - Return 1 if the element is in the set, 0 otherwise.
Definition SmallSet.h:176
bool empty() const
Definition SmallSet.h:169
std::pair< const_iterator, bool > insert(const T &V)
insert - Insert an element into the set if it isn't already there.
Definition SmallSet.h:184
void resize(size_type N)
void push_back(const T &Elt)
This is a 'vector' (really, a variable-sized array), optimized for the case when the array is small.
Register getReg() const
TargetRegisterInfo base class - We assume that the target defines a static array of TargetRegisterDes...
static constexpr TypeSize getFixed(ScalarTy ExactSize)
Definition TypeSize.h:352
static LLVM_ABI IntegerType * getInt32Ty(LLVMContext &C)
Definition Type.cpp:294
self_iterator getIterator()
Definition ilist_node.h:123
A range adaptor for a pair of iterators.
#define llvm_unreachable(msg)
Marks that the current location is not supposed to be reachable.
@ CONSTANT_ADDRESS_32BIT
Address space for 32-bit constant memory.
@ REGION_ADDRESS
Address space for region memory. (GDS)
@ LOCAL_ADDRESS
Address space for local memory.
@ CONSTANT_ADDRESS
Address space for constant memory (VTX2).
@ PRIVATE_ADDRESS
Address space for private memory.
@ BUFFER_RESOURCE
Address space for 128-bit buffer resources.
bool isFlatGlobalAddrSpace(unsigned AS)
bool isUniformMMO(const MachineMemOperand *MMO)
bool isUnscaledIEEEDivFmas(const MachineInstr &MI, const MachineRegisterInfo &MRI)
Return whether div.fmas can use ordinary FMA without changing denormals.
bool isExtendedGlobalAddrSpace(unsigned AS)
Intrinsic::ID getIntrinsicID(const MachineInstr &I)
Return the intrinsic ID for opcodes with the G_AMDGPU_INTRIN_ prefix.
std::pair< Register, unsigned > getBaseWithConstantOffset(MachineRegisterInfo &MRI, Register Reg, GISelValueTracking *ValueTracking=nullptr, bool CheckNUW=false)
Returns base register and constant offset.
const RsrcIntrinsic * lookupRsrcIntrinsic(unsigned Intr)
SpecificConstantMatch m_ZeroInt()
Convenience matchers for specific integer values.
bool mi_match(Reg R, const MachineRegisterInfo &MRI, Pattern &&P)
SpecificConstantOrSplatMatch m_SpecificICstOrSplat(const APInt &RequestedValue)
Matches a RequestedValue constant or a constant splat of RequestedValue.
This is an optimization pass for GlobalISel generic memory operations.
@ Offset
Definition DWP.cpp:577
LLVM_ABI MachineInstr * getOpcodeDef(unsigned Opcode, Register Reg, const MachineRegisterInfo &MRI)
See if Reg is defined by an single def instruction that is Opcode.
Definition Utils.cpp:679
MachineInstrBuilder BuildMI(MachineFunction &MF, const MIMetadata &MIMD, const MCInstrDesc &MCID)
Builder interface. Specify how to create the initial instruction itself.
decltype(auto) dyn_cast(const From &Val)
dyn_cast<X> - Return the argument parameter cast to the specified type.
Definition Casting.h:643
LLVM_ABI void constrainSelectedInstRegOperands(MachineInstr &I, const TargetInstrInfo &TII, const TargetRegisterInfo &TRI, const RegisterBankInfo &RBI)
Mutate the newly-selected instruction I to constrain its (possibly generic) virtual register operands...
Definition Utils.cpp:163
iterator_range< T > make_range(T x, T y)
Convenience function for iterating over sub-ranges.
LLVM_ABI std::optional< int64_t > getIConstantVRegSExtVal(Register VReg, const MachineRegisterInfo &MRI)
If VReg is defined by a G_CONSTANT fits in int64_t returns it.
Definition Utils.cpp:321
static const MachineMemOperand::Flags MONoClobber
Mark the MMO of a uniform load if there are no potentially clobbering stores on any path from the sta...
Definition SIInstrInfo.h:49
auto reverse(ContainerTy &&C)
Definition STLExtras.h:408
class LLVM_GSL_OWNER SmallVector
Forward declaration of SmallVector so that calculateSmallVectorDefaultInlinedElements can reference s...
bool isa(const From &Val)
isa<X> - Return true if the parameter to the template is an instance of one of the template type argu...
Definition Casting.h:547
@ Add
Sum of integers.
IntPtrTy
Definition InstrProf.h:82
DWARFExpression::Operation Op
void call_once(once_flag &flag, Function &&F, Args &&... ArgList)
Execute the function specified as a parameter once.
Definition Threading.h:86
decltype(auto) cast(const From &Val)
cast<X> - Return the argument parameter cast to the specified type.
Definition Casting.h:559
LLVM_ABI std::optional< ValueAndVReg > getIConstantVRegValWithLookThrough(Register VReg, const MachineRegisterInfo &MRI, bool LookThroughInstrs=true)
If VReg is defined by a statically evaluable chain of instructions rooted on a G_CONSTANT returns its...
Definition Utils.cpp:440
Align assumeAligned(uint64_t Value)
Treats the value 0 as a 1, so Align is always at least 1.
Definition Alignment.h:105
unsigned Log2(Align A)
Returns the log2 of the alignment.
Definition Alignment.h:202
LLVM_ABI Register getSrcRegIgnoringCopies(Register Reg, const MachineRegisterInfo &MRI)
Find the source register for Reg, folding away any trivial copies.
Definition Utils.cpp:527
constexpr T maskTrailingOnes(unsigned N)
Create a bitmask with the N right-most bits set to 1, and all other bits set to 0.
Definition MathExtras.h:78
@ End
Run the machine verifier once, at the end of the pipeline.
Definition CodeGen.h:320
MCRegisterClass TargetRegisterClass
Definition FastISel.h:58
#define N
This struct is a compact representation of a valid (non-zero power of two) alignment.
Definition Alignment.h:39
This class contains a discriminated union of information about pointers in memory operands,...
unsigned StartIdx
Number of bits at which this partial mapping starts in the original value.
const RegisterBank * RegBank
Register bank where the partial value lives.
unsigned Length
Length of this mapping in bits.
Helper struct that represents how a value is mapped through different register banks.
unsigned NumBreakDowns
Number of partial mapping to break down this value.
const PartialMapping * BreakDown
How the value is broken down between the different register banks.
bool IEEE
Floating point opcodes that support exception flag gathering quiet and propagate signaling NaN inputs...
The llvm::once_flag structure.
Definition Threading.h:67