LLVM 24.0.0git
AMDGPUSwLowerLDS.cpp
Go to the documentation of this file.
1//===-- AMDGPUSwLowerLDS.cpp -----------------------------------------===//
2//
3// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
4// See https://llvm.org/LICENSE.txt for license information.
5// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
6//
7//===----------------------------------------------------------------------===//
8//
9// This pass lowers the local data store, LDS, uses in kernel and non-kernel
10// functions in module to use dynamically allocated global memory.
11// Packed LDS Layout is emulated in the global memory.
12// The lowered memory instructions from LDS to global memory are then
13// instrumented for address sanitizer, to catch addressing errors.
14// This pass only work when address sanitizer has been enabled and has
15// instrumented the IR. It identifies that IR has been instrumented using
16// "nosanitize_address" module flag.
17//
18// Replacement of Kernel LDS accesses:
19// For a kernel, LDS access can be static or dynamic which are direct
20// (accessed within kernel) and indirect (accessed through non-kernels).
21// All these LDS accesses corresponding to kernel will be packed together,
22// where all static LDS accesses will be allocated first and then dynamic
23// LDS follows. The total size with alignment is calculated. A new LDS global
24// will be created for the kernel called "SW LDS" and it will have the
25// attribute "amdgpu-lds-size" attached with value of the size calculated.
26// All the LDS accesses in the module will be replaced by GEP with offset
27// into the "Sw LDS".
28// A new "llvm.amdgcn.<kernel>.dynlds" is created per kernel accessing
29// the dynamic LDS. This will be marked used by kernel and will have
30// MD_absolue_symbol metadata set to total static LDS size, Since dynamic
31// LDS allocation starts after all static LDS allocation.
32//
33// A device global memory equal to the total LDS size will be allocated.
34// At the prologue of the kernel, a single work-item from the
35// work-group, does a "malloc" and stores the pointer of the
36// allocation in "SW LDS".
37//
38// To store the offsets corresponding to all LDS accesses, another global
39// variable is created which will be called "SW LDS metadata" in this pass.
40// - SW LDS Global:
41// It is LDS global of ptr type with name
42// "llvm.amdgcn.sw.lds.<kernel-name>".
43// - Metadata Global:
44// It is of struct type, with n members. n equals the number of LDS
45// globals accessed by the kernel(direct and indirect). Each member of
46// struct is another struct of type {i32, i32, i32}. First member
47// corresponds to offset, second member corresponds to size of LDS global
48// being replaced and third represents the total aligned size. It will
49// have name "llvm.amdgcn.sw.lds.<kernel-name>.md". This global will have
50// an initializer with static LDS related offsets and sizes initialized.
51// But for dynamic LDS related entries, offsets will be initialized to
52// previous static LDS allocation end offset. Sizes for them will be zero
53// initially. These dynamic LDS offset and size values will be updated
54// within the kernel, since kernel can read the dynamic LDS size
55// allocation done at runtime with query to "hidden_dynamic_lds_size"
56// hidden kernel argument.
57//
58// At the epilogue of kernel, allocated memory would be made free by the same
59// single work-item.
60//
61// Replacement of non-kernel LDS accesses:
62// Multiple kernels can access the same non-kernel function.
63// All the kernels accessing LDS through non-kernels, or reaching
64// non-kernels with LDS instructions, are sorted and assigned a kernel-id.
65// All the LDS globals accessed by non-kernels are sorted. This
66// information is used to build two tables:
67// - Base table:
68// Base table will have single row, with elements of the row
69// placed as per kernel ID. Each element in the row corresponds
70// to ptr of "SW LDS" variable created for that kernel.
71// - Offset table:
72// Offset table will have multiple rows and columns.
73// Rows are assumed to be from 0 to (n-1). n is total number
74// of kernels accessing the LDS through non-kernels.
75// Each row will have m elements. m is the total number of
76// unique LDS globals accessed by all non-kernels.
77// Each element in the row correspond to the ptr of
78// the replacement of LDS global done by that particular kernel.
79// A LDS variable in non-kernel will be replaced based on the information
80// from base and offset tables. Based on kernel-id query, ptr of "SW
81// LDS" for that corresponding kernel is obtained from base table.
82// The Offset into the base "SW LDS" is obtained from
83// corresponding element in offset table. With this information, replacement
84// value is obtained.
85//===----------------------------------------------------------------------===//
86
87#include "AMDGPU.h"
89#include "AMDGPUMemoryUtils.h"
91#include "llvm/ADT/StringRef.h"
94#include "llvm/IR/Constants.h"
95#include "llvm/IR/DIBuilder.h"
96#include "llvm/IR/DebugInfo.h"
98#include "llvm/IR/IRBuilder.h"
100#include "llvm/IR/Instructions.h"
101#include "llvm/IR/MDBuilder.h"
103#include "llvm/Pass.h"
107
108#include <algorithm>
109
110#define DEBUG_TYPE "amdgpu-sw-lower-lds"
111#define COV5_HIDDEN_DYN_LDS_SIZE_ARG 15
112
113using namespace llvm;
114using namespace AMDGPU;
115
116namespace {
117
118using DomTreeCallback = function_ref<DominatorTree *(Function &F)>;
119
120struct LDSAccessTypeInfo {
121 SetVector<GlobalVariable *> StaticLDSGlobals;
122 SetVector<GlobalVariable *> DynamicLDSGlobals;
123};
124
125// Struct to hold all the Metadata required for a kernel
126// to replace a LDS global uses with corresponding offset
127// in to device global memory.
128struct KernelLDSParameters {
129 GlobalVariable *SwLDS = nullptr;
130 GlobalVariable *SwDynLDS = nullptr;
131 GlobalVariable *SwLDSMetadata = nullptr;
132 LDSAccessTypeInfo DirectAccess;
133 LDSAccessTypeInfo IndirectAccess;
135 LDSToReplacementIndicesMap;
136 uint32_t MallocSize = 0;
137 uint32_t LDSSize = 0;
138 SmallVector<std::pair<uint32_t, uint32_t>, 64> RedzoneOffsetAndSizeVector;
139};
140
141// Struct to store information for creation of offset table
142// for all the non-kernel LDS accesses.
143struct NonKernelLDSParameters {
144 GlobalVariable *LDSBaseTable = nullptr;
145 GlobalVariable *LDSOffsetTable = nullptr;
146 SetVector<Function *> OrderedKernels;
147 SetVector<GlobalVariable *> OrdereLDSGlobals;
148};
149
150struct AsanInstrumentInfo {
151 int Scale = 0;
152 uint32_t Offset = 0;
153 SetVector<Instruction *> Instructions;
154};
155
156struct FunctionsAndLDSAccess {
157 MapVector<Function *, KernelLDSParameters> KernelToLDSParametersMap;
158 SetVector<Function *> KernelsWithIndirectLDSAccess;
159 SetVector<Function *> NonKernelsWithLDSInstructions;
160 SetVector<GlobalVariable *> AllNonKernelLDSAccess;
161 FunctionVariableMap NonKernelToLDSAccessMap;
162};
163
164class AMDGPUSwLowerLDS {
165public:
166 AMDGPUSwLowerLDS(Module &Mod, DomTreeCallback Callback)
167 : M(Mod), IRB(M), DTCallback(Callback) {}
168 bool run();
169 void getUsesOfLDSByNonKernels();
170 void getNonKernelsWithLDSInstructions(const CallGraph &CG);
172 getOrderedIndirectLDSAccessingKernels(SetVector<Function *> &Kernels);
174 getOrderedNonKernelAllLDSGlobals(SetVector<GlobalVariable *> &Variables);
175 void buildSwLDSGlobal(Function *Func);
176 void buildSwDynLDSGlobal(Function *Func);
177 void populateSwMetadataGlobal(Function *Func);
178 void populateSwLDSAttributeAndMetadata(Function *Func);
179 void populateLDSToReplacementIndicesMap(Function *Func);
180 void getLDSMemoryInstructions(Function *Func,
181 SetVector<Instruction *> &LDSInstructions);
182 void replaceKernelLDSAccesses(Function *Func);
183 Value *getTranslatedGlobalMemoryPtrOfLDS(Value *LoadMallocPtr, Value *LDSPtr);
184 Value *getTranslatedLDSPtrOfFlat(Value *LoadMallocPtr, Value *FlatPtr,
185 Type *LDSPtrTy);
186 void translateLDSMemoryOperationsToGlobalMemory(
187 Function *Func, Value *LoadMallocPtr,
188 SetVector<Instruction *> &LDSInstructions);
189 void poisonRedzones(Function *Func, Value *MallocPtr);
190 void lowerKernelLDSAccesses(Function *Func, DomTreeUpdater &DTU);
191 void buildNonKernelLDSOffsetTable(NonKernelLDSParameters &NKLDSParams);
192 void buildNonKernelLDSBaseTable(NonKernelLDSParameters &NKLDSParams);
193 Constant *
194 getAddressesOfVariablesInKernel(Function *Func,
195 SetVector<GlobalVariable *> &Variables);
196 void lowerNonKernelLDSAccesses(Function *Func,
197 SetVector<GlobalVariable *> &LDSGlobals,
198 NonKernelLDSParameters &NKLDSParams);
199 void
200 updateMallocSizeForDynamicLDS(Function *Func, Value **CurrMallocSize,
201 Value *HiddenDynLDSSize,
202 SetVector<GlobalVariable *> &DynamicLDSGlobals);
203 void initAsanInfo();
204
205private:
206 Module &M;
207 IRBuilder<> IRB;
208 DomTreeCallback DTCallback;
209 FunctionsAndLDSAccess FuncLDSAccessInfo;
210 AsanInstrumentInfo AsanInfo;
211};
212
213template <typename T> SetVector<T> sortByName(std::vector<T> &&V) {
214 // Sort the vector of globals or Functions based on their name.
215 // Returns a SetVector of globals/Functions.
216 sort(V, [](const auto *L, const auto *R) {
217 return L->getName() < R->getName();
218 });
220}
221
222SetVector<GlobalVariable *> AMDGPUSwLowerLDS::getOrderedNonKernelAllLDSGlobals(
223 SetVector<GlobalVariable *> &Variables) {
224 // Sort all the non-kernel LDS accesses based on their name.
225 return sortByName(
226 std::vector<GlobalVariable *>(Variables.begin(), Variables.end()));
227}
228
229SetVector<Function *> AMDGPUSwLowerLDS::getOrderedIndirectLDSAccessingKernels(
230 SetVector<Function *> &Kernels) {
231 // Sort the non-kernels accessing LDS based on their name.
232 // Also assign a kernel ID metadata based on the sorted order.
233 LLVMContext &Ctx = M.getContext();
234 if (Kernels.size() > UINT32_MAX) {
235 report_fatal_error("Unimplemented SW LDS lowering for > 2**32 kernels");
236 }
237 SetVector<Function *> OrderedKernels =
238 sortByName(std::vector<Function *>(Kernels.begin(), Kernels.end()));
239 for (size_t i = 0; i < Kernels.size(); i++) {
240 Metadata *AttrMDArgs[1] = {
242 };
243 Function *Func = OrderedKernels[i];
244 Func->setMetadata("llvm.amdgcn.lds.kernel.id",
245 MDNode::get(Ctx, AttrMDArgs));
246 }
247 return OrderedKernels;
248}
249
250// LDS memory operations and casts between flat and LDS.
251static bool isLDSMemoryInstruction(const Instruction &Inst) {
252 if (auto *LI = dyn_cast<LoadInst>(&Inst))
253 return LI->getPointerAddressSpace() == AMDGPUAS::LOCAL_ADDRESS;
254 if (auto *SI = dyn_cast<StoreInst>(&Inst))
255 return SI->getPointerAddressSpace() == AMDGPUAS::LOCAL_ADDRESS;
256 if (auto *RMW = dyn_cast<AtomicRMWInst>(&Inst))
257 return RMW->getPointerAddressSpace() == AMDGPUAS::LOCAL_ADDRESS;
258 if (auto *XCHG = dyn_cast<AtomicCmpXchgInst>(&Inst))
259 return XCHG->getPointerAddressSpace() == AMDGPUAS::LOCAL_ADDRESS;
260 if (auto *ASC = dyn_cast<AddrSpaceCastInst>(&Inst)) {
261 unsigned SrcAS = ASC->getSrcAddressSpace();
262 unsigned DstAS = ASC->getDestAddressSpace();
263 return (SrcAS == AMDGPUAS::LOCAL_ADDRESS &&
264 DstAS == AMDGPUAS::FLAT_ADDRESS) ||
265 (SrcAS == AMDGPUAS::FLAT_ADDRESS &&
266 DstAS == AMDGPUAS::LOCAL_ADDRESS);
267 }
268 if (auto *MI = dyn_cast<AnyMemIntrinsic>(&Inst)) {
269 if (MI->getDestAddressSpace() == AMDGPUAS::LOCAL_ADDRESS)
270 return true;
271 auto *MTI = dyn_cast<AnyMemTransferInst>(MI);
272 return MTI && MTI->getSourceAddressSpace() == AMDGPUAS::LOCAL_ADDRESS;
273 }
274 return false;
275}
276
277void AMDGPUSwLowerLDS::getNonKernelsWithLDSInstructions(const CallGraph &CG) {
278 // Non-kernels with LDS instructions find the malloc buffer through the base
279 // table, so every kernel reaching them needs a kernel ID, even one without
280 // LDS.
281 DenseMap<Function *, bool> HasLDSInstructionsCache;
282 auto HasLDSInstructions = [&](Function *F) {
283 auto [It, Inserted] = HasLDSInstructionsCache.try_emplace(F, false);
284 if (Inserted) {
285 It->second = any_of(instructions(*F), isLDSMemoryInstruction);
286 if (It->second)
287 FuncLDSAccessInfo.NonKernelsWithLDSInstructions.insert(F);
288 }
289 return It->second;
290 };
291
292 // Indirect calls may reach any address taken non-kernel.
293 SmallVector<Function *> AddressTakenFuncs;
294 for (Function &F : M)
295 if (!F.isDeclaration() && !isKernel(F) &&
296 F.hasAddressTaken(nullptr, /*IgnoreCallbackUses=*/false,
297 /*IgnoreAssumeLikeCalls=*/false,
298 /*IgnoreLLVMUsed=*/true))
299 AddressTakenFuncs.push_back(&F);
300
301 for (Function &Kernel : M) {
302 if (Kernel.isDeclaration() || !isKernel(Kernel))
303 continue;
304
305 bool ReachesLDSInstructions = false;
306 bool SeenUnknownCall = false;
307 SmallVector<Function *> WorkList = {&Kernel};
309 auto Visit = [&](Function *Callee) {
310 if (Callee->isDeclaration() || isKernel(*Callee) ||
311 !Visited.insert(Callee).second)
312 return;
313 ReachesLDSInstructions |= HasLDSInstructions(Callee);
314 WorkList.push_back(Callee);
315 };
316
317 while (!WorkList.empty()) {
318 Function *F = WorkList.pop_back_val();
319 for (const CallGraphNode::CallRecord &CallRecord : *CG[F]) {
320 if (!CallRecord.second)
321 continue;
322 if (Function *Callee = CallRecord.second->getFunction()) {
323 Visit(Callee);
324 continue;
325 }
326 if (SeenUnknownCall)
327 continue;
328 SeenUnknownCall = true;
329 for (Function *PotentialCallee : AddressTakenFuncs)
330 Visit(PotentialCallee);
331 }
332 }
333
334 if (ReachesLDSInstructions)
335 FuncLDSAccessInfo.KernelsWithIndirectLDSAccess.insert(&Kernel);
336 }
337}
338
339void AMDGPUSwLowerLDS::getUsesOfLDSByNonKernels() {
340 for (GlobalVariable *GV : FuncLDSAccessInfo.AllNonKernelLDSAccess) {
342 continue;
343
344 for (User *V : GV->users()) {
345 if (auto *I = dyn_cast<Instruction>(V)) {
346 Function *F = I->getFunction();
347 if (!isKernel(*F) && !F->isDeclaration())
348 FuncLDSAccessInfo.NonKernelToLDSAccessMap[F].insert(GV);
349 }
350 }
351 }
352}
353
354static void recordLDSAbsoluteAddress(Module &M, GlobalVariable *GV,
355 uint32_t Address) {
356 // Write the specified address into metadata where it can be retrieved by
357 // the assembler. Format is a half open range, [Address Address+1)
358 LLVMContext &Ctx = M.getContext();
359 auto *IntTy = M.getDataLayout().getIntPtrType(Ctx, AMDGPUAS::LOCAL_ADDRESS);
360 MDBuilder MDB(Ctx);
361 MDNode *MetadataNode = MDB.createRange(ConstantInt::get(IntTy, Address),
362 ConstantInt::get(IntTy, Address + 1));
363 GV->setMetadata(LLVMContext::MD_absolute_symbol, MetadataNode);
364}
365
366static void addLDSSizeAttribute(Function *Func, uint32_t Offset,
367 bool IsDynLDS) {
368 if (Offset != 0) {
369 std::string Buffer;
370 raw_string_ostream SS{Buffer};
371 SS << Offset;
372 if (IsDynLDS)
373 SS << "," << Offset;
374 Func->addFnAttr("amdgpu-lds-size", Buffer);
375 }
376}
377
378static void markUsedByKernel(Function *Func, GlobalVariable *SGV) {
379 BasicBlock *Entry = &Func->getEntryBlock();
380 IRBuilder<> Builder(Entry->getFirstNonPHIIt());
381
382 Function *Decl = Intrinsic::getOrInsertDeclaration(Func->getParent(),
383 Intrinsic::donothing, {});
384
385 Value *UseInstance[] = {SGV};
386
387 Builder.CreateCall(Decl, {},
388 {OperandBundleDefT<Value *>("ExplicitUse", UseInstance)});
389}
390
391void AMDGPUSwLowerLDS::buildSwLDSGlobal(Function *Func) {
392 // Create new LDS global required for each kernel to store
393 // device global memory pointer.
394 auto &LDSParams = FuncLDSAccessInfo.KernelToLDSParametersMap[Func];
395 // Create new global pointer variable
396 LDSParams.SwLDS = new GlobalVariable(
397 M, IRB.getPtrTy(), false, GlobalValue::InternalLinkage,
398 PoisonValue::get(IRB.getPtrTy()), "llvm.amdgcn.sw.lds." + Func->getName(),
401 MD.NoAddress = true;
402 LDSParams.SwLDS->setSanitizerMetadata(MD);
403}
404
405void AMDGPUSwLowerLDS::buildSwDynLDSGlobal(Function *Func) {
406 // Create new Dyn LDS global if kernel accesses dyn LDS.
407 auto &LDSParams = FuncLDSAccessInfo.KernelToLDSParametersMap[Func];
408 if (LDSParams.DirectAccess.DynamicLDSGlobals.empty() &&
409 LDSParams.IndirectAccess.DynamicLDSGlobals.empty())
410 return;
411 // Create new global pointer variable
412 auto *emptyCharArray = ArrayType::get(IRB.getInt8Ty(), 0);
413 LDSParams.SwDynLDS = new GlobalVariable(
414 M, emptyCharArray, false, GlobalValue::ExternalLinkage, nullptr,
415 "llvm.amdgcn." + Func->getName() + ".dynlds", nullptr,
417 markUsedByKernel(Func, LDSParams.SwDynLDS);
419 MD.NoAddress = true;
420 LDSParams.SwDynLDS->setSanitizerMetadata(MD);
421}
422
423void AMDGPUSwLowerLDS::populateSwLDSAttributeAndMetadata(Function *Func) {
424 auto &LDSParams = FuncLDSAccessInfo.KernelToLDSParametersMap[Func];
425 bool IsDynLDSUsed = LDSParams.SwDynLDS;
426 uint32_t Offset = LDSParams.LDSSize;
427 recordLDSAbsoluteAddress(M, LDSParams.SwLDS, 0);
428 addLDSSizeAttribute(Func, Offset, IsDynLDSUsed);
429 if (LDSParams.SwDynLDS)
430 recordLDSAbsoluteAddress(M, LDSParams.SwDynLDS, Offset);
431}
432
433void AMDGPUSwLowerLDS::populateSwMetadataGlobal(Function *Func) {
434 // Create new metadata global for every kernel and initialize the
435 // start offsets and sizes corresponding to each LDS accesses.
436 auto &LDSParams = FuncLDSAccessInfo.KernelToLDSParametersMap[Func];
437 auto &Ctx = M.getContext();
438 auto &DL = M.getDataLayout();
439 std::vector<Type *> Items;
440 Type *Int32Ty = IRB.getInt32Ty();
441 std::vector<Constant *> Initializers;
442 Align MaxAlignment(1);
443 auto UpdateMaxAlignment = [&MaxAlignment, &DL](GlobalVariable *GV) {
444 Align GVAlign = AMDGPU::getAlign(DL, GV);
445 MaxAlignment = std::max(MaxAlignment, GVAlign);
446 };
447
448 for (GlobalVariable *GV : LDSParams.DirectAccess.StaticLDSGlobals)
449 UpdateMaxAlignment(GV);
450
451 for (GlobalVariable *GV : LDSParams.DirectAccess.DynamicLDSGlobals)
452 UpdateMaxAlignment(GV);
453
454 for (GlobalVariable *GV : LDSParams.IndirectAccess.StaticLDSGlobals)
455 UpdateMaxAlignment(GV);
456
457 for (GlobalVariable *GV : LDSParams.IndirectAccess.DynamicLDSGlobals)
458 UpdateMaxAlignment(GV);
459
460 //{StartOffset, AlignedSizeInBytes}
461 SmallString<128> MDItemStr;
462 raw_svector_ostream MDItemOS(MDItemStr);
463 MDItemOS << "llvm.amdgcn.sw.lds." << Func->getName() << ".md.item";
464
465 StructType *LDSItemTy =
466 StructType::create(Ctx, {Int32Ty, Int32Ty, Int32Ty}, MDItemOS.str());
467 uint32_t &MallocSize = LDSParams.MallocSize;
468 SetVector<GlobalVariable *> UniqueLDSGlobals;
469 int AsanScale = AsanInfo.Scale;
470 auto buildInitializerForSwLDSMD =
471 [&](SetVector<GlobalVariable *> &LDSGlobals) {
472 for (auto &GV : LDSGlobals) {
473 if (is_contained(UniqueLDSGlobals, GV))
474 continue;
475 UniqueLDSGlobals.insert(GV);
476
477 Type *Ty = GV->getValueType();
478 const uint64_t SizeInBytes = DL.getTypeAllocSize(Ty);
479 Items.push_back(LDSItemTy);
480 Constant *ItemStartOffset = ConstantInt::get(Int32Ty, MallocSize);
481 Constant *SizeInBytesConst = ConstantInt::get(Int32Ty, SizeInBytes);
482 // Get redzone size corresponding a size.
483 const uint64_t RightRedzoneSize =
484 AMDGPU::getRedzoneSizeForGlobal(AsanScale, SizeInBytes);
485 // Update MallocSize with current size and redzone size.
486 MallocSize += SizeInBytes;
487 if (!AMDGPU::isDynamicLDS(*GV))
488 LDSParams.RedzoneOffsetAndSizeVector.emplace_back(MallocSize,
489 RightRedzoneSize);
490 MallocSize += RightRedzoneSize;
491 // Align current size plus redzone.
492 uint64_t AlignedSize =
493 alignTo(SizeInBytes + RightRedzoneSize, MaxAlignment);
494 Constant *AlignedSizeInBytesConst =
495 ConstantInt::get(Int32Ty, AlignedSize);
496 // Align MallocSize
497 MallocSize = alignTo(MallocSize, MaxAlignment);
498 Constant *InitItem =
499 ConstantStruct::get(LDSItemTy, {ItemStartOffset, SizeInBytesConst,
500 AlignedSizeInBytesConst});
501 Initializers.push_back(InitItem);
502 }
503 };
504 SetVector<GlobalVariable *> SwLDSVector;
505 SwLDSVector.insert(LDSParams.SwLDS);
506 buildInitializerForSwLDSMD(SwLDSVector);
507 buildInitializerForSwLDSMD(LDSParams.DirectAccess.StaticLDSGlobals);
508 buildInitializerForSwLDSMD(LDSParams.IndirectAccess.StaticLDSGlobals);
509 buildInitializerForSwLDSMD(LDSParams.DirectAccess.DynamicLDSGlobals);
510 buildInitializerForSwLDSMD(LDSParams.IndirectAccess.DynamicLDSGlobals);
511
512 // Update the LDS size used by the kernel.
513 Type *Ty = LDSParams.SwLDS->getValueType();
514 const uint64_t SizeInBytes = DL.getTypeAllocSize(Ty);
515 uint64_t AlignedSize = alignTo(SizeInBytes, MaxAlignment);
516 LDSParams.LDSSize = AlignedSize;
517 SmallString<128> MDTypeStr;
518 raw_svector_ostream MDTypeOS(MDTypeStr);
519 MDTypeOS << "llvm.amdgcn.sw.lds." << Func->getName() << ".md.type";
520 StructType *MetadataStructType =
521 StructType::create(Ctx, Items, MDTypeOS.str());
522 SmallString<128> MDStr;
523 raw_svector_ostream MDOS(MDStr);
524 MDOS << "llvm.amdgcn.sw.lds." << Func->getName() << ".md";
525 LDSParams.SwLDSMetadata = new GlobalVariable(
526 M, MetadataStructType, false, GlobalValue::InternalLinkage,
527 PoisonValue::get(MetadataStructType), MDOS.str(), nullptr,
529 Constant *data = ConstantStruct::get(MetadataStructType, Initializers);
530 LDSParams.SwLDSMetadata->setInitializer(data);
531 assert(LDSParams.SwLDS);
532 // Set the alignment to MaxAlignment for SwLDS.
533 LDSParams.SwLDS->setAlignment(MaxAlignment);
534 if (LDSParams.SwDynLDS)
535 LDSParams.SwDynLDS->setAlignment(MaxAlignment);
537 MD.NoAddress = true;
538 LDSParams.SwLDSMetadata->setSanitizerMetadata(MD);
539}
540
541void AMDGPUSwLowerLDS::populateLDSToReplacementIndicesMap(Function *Func) {
542 // Fill the corresponding LDS replacement indices for each LDS access
543 // related to this kernel.
544 auto &LDSParams = FuncLDSAccessInfo.KernelToLDSParametersMap[Func];
545 SetVector<GlobalVariable *> UniqueLDSGlobals;
546 auto PopulateIndices = [&](SetVector<GlobalVariable *> &LDSGlobals,
547 uint32_t &Idx) {
548 for (auto &GV : LDSGlobals) {
549 if (is_contained(UniqueLDSGlobals, GV))
550 continue;
551 UniqueLDSGlobals.insert(GV);
552 LDSParams.LDSToReplacementIndicesMap[GV] = {0, Idx, 0};
553 ++Idx;
554 }
555 };
556 uint32_t Idx = 0;
557 SetVector<GlobalVariable *> SwLDSVector;
558 SwLDSVector.insert(LDSParams.SwLDS);
559 PopulateIndices(SwLDSVector, Idx);
560 PopulateIndices(LDSParams.DirectAccess.StaticLDSGlobals, Idx);
561 PopulateIndices(LDSParams.IndirectAccess.StaticLDSGlobals, Idx);
562 PopulateIndices(LDSParams.DirectAccess.DynamicLDSGlobals, Idx);
563 PopulateIndices(LDSParams.IndirectAccess.DynamicLDSGlobals, Idx);
564}
565
566static void replacesUsesOfGlobalInFunction(Function *Func, GlobalVariable *GV,
567 Value *Replacement) {
568 // Replace all uses of LDS global in this Function with a Replacement.
569 auto ReplaceUsesLambda = [Func](const Use &U) -> bool {
570 auto *V = U.getUser();
571 if (auto *Inst = dyn_cast<Instruction>(V)) {
572 auto *Func1 = Inst->getFunction();
573 if (Func == Func1)
574 return true;
575 }
576 return false;
577 };
578 GV->replaceUsesWithIf(Replacement, ReplaceUsesLambda);
579}
580
581void AMDGPUSwLowerLDS::replaceKernelLDSAccesses(Function *Func) {
582 auto &LDSParams = FuncLDSAccessInfo.KernelToLDSParametersMap[Func];
583 GlobalVariable *SwLDS = LDSParams.SwLDS;
584 assert(SwLDS);
585 GlobalVariable *SwLDSMetadata = LDSParams.SwLDSMetadata;
586 assert(SwLDSMetadata);
587 StructType *SwLDSMetadataStructType =
588 cast<StructType>(SwLDSMetadata->getValueType());
589 Type *Int32Ty = IRB.getInt32Ty();
590 auto &IndirectAccess = LDSParams.IndirectAccess;
591 auto &DirectAccess = LDSParams.DirectAccess;
592 // Replace all uses of LDS global in this Function with a Replacement.
593 SetVector<GlobalVariable *> UniqueLDSGlobals;
594 auto ReplaceLDSGlobalUses = [&](SetVector<GlobalVariable *> &LDSGlobals) {
595 for (auto &GV : LDSGlobals) {
596 // Do not generate instructions if LDS access is in non-kernel
597 // i.e indirect-access.
598 if ((IndirectAccess.StaticLDSGlobals.contains(GV) ||
599 IndirectAccess.DynamicLDSGlobals.contains(GV)) &&
600 (!DirectAccess.StaticLDSGlobals.contains(GV) &&
601 !DirectAccess.DynamicLDSGlobals.contains(GV)))
602 continue;
603 if (is_contained(UniqueLDSGlobals, GV))
604 continue;
605 UniqueLDSGlobals.insert(GV);
606 auto &Indices = LDSParams.LDSToReplacementIndicesMap[GV];
607 assert(Indices.size() == 3);
608 Constant *GEPIdx[] = {ConstantInt::get(Int32Ty, Indices[0]),
609 ConstantInt::get(Int32Ty, Indices[1]),
610 ConstantInt::get(Int32Ty, Indices[2])};
612 Func->getDataLayout(), SwLDSMetadataStructType, SwLDSMetadata, GEPIdx,
614 Value *Offset = IRB.CreateLoad(Int32Ty, GEP);
615 Value *BasePlusOffset =
616 IRB.CreateInBoundsGEP(IRB.getInt8Ty(), SwLDS, {Offset});
617 LLVM_DEBUG(GV->printAsOperand(dbgs() << "Sw LDS Lowering, Replacing LDS ",
618 false));
619 replacesUsesOfGlobalInFunction(Func, GV, BasePlusOffset);
620 }
621 };
622 ReplaceLDSGlobalUses(DirectAccess.StaticLDSGlobals);
623 ReplaceLDSGlobalUses(IndirectAccess.StaticLDSGlobals);
624 ReplaceLDSGlobalUses(DirectAccess.DynamicLDSGlobals);
625 ReplaceLDSGlobalUses(IndirectAccess.DynamicLDSGlobals);
626}
627
628void AMDGPUSwLowerLDS::updateMallocSizeForDynamicLDS(
629 Function *Func, Value **CurrMallocSize, Value *HiddenDynLDSSize,
630 SetVector<GlobalVariable *> &DynamicLDSGlobals) {
631 auto &LDSParams = FuncLDSAccessInfo.KernelToLDSParametersMap[Func];
632 Type *Int32Ty = IRB.getInt32Ty();
633
634 GlobalVariable *SwLDS = LDSParams.SwLDS;
635 GlobalVariable *SwLDSMetadata = LDSParams.SwLDSMetadata;
636 assert(SwLDS && SwLDSMetadata);
637 StructType *MetadataStructType =
638 cast<StructType>(SwLDSMetadata->getValueType());
639 unsigned MaxAlignment = SwLDS->getAlign().valueOrOne().value();
640 Value *MaxAlignValue = IRB.getInt32(MaxAlignment);
641 Value *MaxAlignValueMinusOne = IRB.getInt32(MaxAlignment - 1);
642
643 for (GlobalVariable *DynGV : DynamicLDSGlobals) {
644 auto &Indices = LDSParams.LDSToReplacementIndicesMap[DynGV];
645 // Update the Offset metadata.
646 Constant *Index0 = ConstantInt::get(Int32Ty, 0);
647 Constant *Index1 = ConstantInt::get(Int32Ty, Indices[1]);
648
649 Constant *Index2Offset = ConstantInt::get(Int32Ty, 0);
650 auto *GEPForOffset = IRB.CreateInBoundsGEP(
651 MetadataStructType, SwLDSMetadata, {Index0, Index1, Index2Offset});
652
653 IRB.CreateStore(*CurrMallocSize, GEPForOffset);
654 // Update the size and Aligned Size metadata.
655 Constant *Index2Size = ConstantInt::get(Int32Ty, 1);
656 auto *GEPForSize = IRB.CreateInBoundsGEP(MetadataStructType, SwLDSMetadata,
657 {Index0, Index1, Index2Size});
658
659 Value *CurrDynLDSSize = IRB.CreateLoad(Int32Ty, HiddenDynLDSSize);
660 IRB.CreateStore(CurrDynLDSSize, GEPForSize);
661 Constant *Index2AlignedSize = ConstantInt::get(Int32Ty, 2);
662 auto *GEPForAlignedSize = IRB.CreateInBoundsGEP(
663 MetadataStructType, SwLDSMetadata, {Index0, Index1, Index2AlignedSize});
664
665 Value *AlignedDynLDSSize =
666 IRB.CreateAdd(CurrDynLDSSize, MaxAlignValueMinusOne);
667 AlignedDynLDSSize = IRB.CreateUDiv(AlignedDynLDSSize, MaxAlignValue);
668 AlignedDynLDSSize = IRB.CreateMul(AlignedDynLDSSize, MaxAlignValue);
669 IRB.CreateStore(AlignedDynLDSSize, GEPForAlignedSize);
670
671 // Update the Current Malloc Size
672 *CurrMallocSize = IRB.CreateAdd(*CurrMallocSize, AlignedDynLDSSize);
673 }
674}
675
676static DebugLoc getOrCreateDebugLoc(const Instruction *InsertBefore,
677 DISubprogram *SP) {
678 assert(InsertBefore);
679 if (InsertBefore->getDebugLoc())
680 return InsertBefore->getDebugLoc();
681 if (SP)
682 return DILocation::get(SP->getContext(), SP->getLine(), 1, SP);
683 return DebugLoc();
684}
685
686void AMDGPUSwLowerLDS::getLDSMemoryInstructions(
687 Function *Func, SetVector<Instruction *> &LDSInstructions) {
688 for (Instruction &Inst : instructions(Func))
689 if (isLDSMemoryInstruction(Inst))
690 LDSInstructions.insert(&Inst);
691}
692
693Value *AMDGPUSwLowerLDS::getTranslatedGlobalMemoryPtrOfLDS(Value *LoadMallocPtr,
694 Value *LDSPtr) {
695 assert(LDSPtr && "Invalid LDS pointer operand");
696 Type *LDSPtrType = LDSPtr->getType();
697 LLVMContext &Ctx = M.getContext();
698 const DataLayout &DL = M.getDataLayout();
699 Type *IntTy = DL.getIntPtrType(Ctx, AMDGPUAS::LOCAL_ADDRESS);
700 if (auto *VecPtrTy = dyn_cast<VectorType>(LDSPtrType)) {
701 // Handle vector of pointers
702 ElementCount NumElements = VecPtrTy->getElementCount();
703 IntTy = VectorType::get(IntTy, NumElements);
704 }
705 Value *GepIndex = IRB.CreatePtrToInt(LDSPtr, IntTy);
706 return IRB.CreateInBoundsGEP(IRB.getInt8Ty(), LoadMallocPtr, {GepIndex});
707}
708
709// A flat pointer to lowered LDS holds a global address inside the malloc
710// buffer, but LDS values in this pass are offsets into that buffer.
711// Rebase it as inttoptr(trunc(ptrtoaddr(flat) - ptrtoaddr(buf))).
712Value *AMDGPUSwLowerLDS::getTranslatedLDSPtrOfFlat(Value *LoadMallocPtr,
713 Value *FlatPtr,
714 Type *LDSPtrTy) {
715 const DataLayout &DL = M.getDataLayout();
716 Value *FlatAddr = IRB.CreatePtrToAddr(FlatPtr);
717 Value *Base = IRB.CreatePtrToAddr(LoadMallocPtr);
718 if (auto *VecTy = dyn_cast<VectorType>(FlatAddr->getType()))
719 Base = IRB.CreateVectorSplat(VecTy->getElementCount(), Base);
720 Value *Diff = IRB.CreateSub(FlatAddr, Base);
721 Value *Offset = IRB.CreateTrunc(Diff, DL.getAddressType(LDSPtrTy));
722 Value *LDSPtr = IRB.CreateIntToPtr(Offset, LDSPtrTy);
723 // Flat null is 0 but LDS null is -1, which the rebased offset cannot keep.
725 Constant::getNullValue(FlatPtr->getType()), LDSPtrTy);
726 return IRB.CreateSelect(IRB.CreateIsNull(FlatPtr), LDSNull, LDSPtr);
727}
728
729void AMDGPUSwLowerLDS::translateLDSMemoryOperationsToGlobalMemory(
730 Function *Func, Value *LoadMallocPtr,
731 SetVector<Instruction *> &LDSInstructions) {
732 LLVM_DEBUG(dbgs() << "Translating LDS memory operations to global memory : "
733 << Func->getName());
734 for (Instruction *Inst : LDSInstructions) {
735 IRB.SetInsertPoint(Inst);
736 if (LoadInst *LI = dyn_cast<LoadInst>(Inst)) {
737 Value *LIOperand = LI->getPointerOperand();
738 Value *Replacement =
739 getTranslatedGlobalMemoryPtrOfLDS(LoadMallocPtr, LIOperand);
740 LoadInst *NewLI =
741 IRB.CreateLoad(LI->getType(), Replacement, LI->getProperties());
742 AsanInfo.Instructions.insert(NewLI);
743 LI->replaceAllUsesWith(NewLI);
744 LI->eraseFromParent();
745 } else if (StoreInst *SI = dyn_cast<StoreInst>(Inst)) {
746 Value *SIOperand = SI->getPointerOperand();
747 Value *Replacement =
748 getTranslatedGlobalMemoryPtrOfLDS(LoadMallocPtr, SIOperand);
749 StoreInst *NewSI = IRB.CreateStore(SI->getValueOperand(), Replacement,
750 SI->getProperties());
751 AsanInfo.Instructions.insert(NewSI);
752 SI->replaceAllUsesWith(NewSI);
753 SI->eraseFromParent();
754 } else if (AtomicRMWInst *RMW = dyn_cast<AtomicRMWInst>(Inst)) {
755 Value *RMWPtrOperand = RMW->getPointerOperand();
756 Value *RMWValOperand = RMW->getValOperand();
757 Value *Replacement =
758 getTranslatedGlobalMemoryPtrOfLDS(LoadMallocPtr, RMWPtrOperand);
759 AtomicRMWInst *NewRMW = IRB.CreateAtomicRMW(
760 RMW->getOperation(), Replacement, RMWValOperand, RMW->getAlign(),
761 RMW->getOrdering(), RMW->getSyncScopeID());
762 NewRMW->setVolatile(RMW->isVolatile());
763 AsanInfo.Instructions.insert(NewRMW);
764 RMW->replaceAllUsesWith(NewRMW);
765 RMW->eraseFromParent();
766 } else if (AtomicCmpXchgInst *XCHG = dyn_cast<AtomicCmpXchgInst>(Inst)) {
767 Value *XCHGPtrOperand = XCHG->getPointerOperand();
768 Value *Replacement =
769 getTranslatedGlobalMemoryPtrOfLDS(LoadMallocPtr, XCHGPtrOperand);
771 Replacement, XCHG->getCompareOperand(), XCHG->getNewValOperand(),
772 XCHG->getAlign(), XCHG->getSuccessOrdering(),
773 XCHG->getFailureOrdering(), XCHG->getSyncScopeID());
774 NewXCHG->setVolatile(XCHG->isVolatile());
775 AsanInfo.Instructions.insert(NewXCHG);
776 XCHG->replaceAllUsesWith(NewXCHG);
777 XCHG->eraseFromParent();
778 } else if (AnyMemIntrinsic *MI = dyn_cast<AnyMemIntrinsic>(Inst)) {
779 Value *NewDest = MI->getRawDest();
780 if (MI->getDestAddressSpace() == AMDGPUAS::LOCAL_ADDRESS)
781 NewDest = getTranslatedGlobalMemoryPtrOfLDS(LoadMallocPtr, NewDest);
782 CallInst *NewMI = nullptr;
784 if (MI->isAtomic()) {
786 NewDest, MSI->getValue(), MSI->getLength(),
787 MSI->getDestAlign().valueOrOne(), MSI->getElementSizeInBytes());
788 } else {
789 NewMI = IRB.CreateMemSet(NewDest, MSI->getValue(), MSI->getLength(),
790 MSI->getDestAlign(),
791 cast<MemSetInst>(MI)->isVolatile());
792 }
794 Value *NewSrc = MTI->getRawSource();
795 if (MTI->getSourceAddressSpace() == AMDGPUAS::LOCAL_ADDRESS)
796 NewSrc = getTranslatedGlobalMemoryPtrOfLDS(LoadMallocPtr, NewSrc);
797 if (MI->isAtomic()) {
798 if (MI->getIntrinsicID() ==
799 Intrinsic::memmove_element_unordered_atomic) {
801 NewDest, MTI->getDestAlign().valueOrOne(), NewSrc,
802 MTI->getSourceAlign().valueOrOne(), MTI->getLength(),
803 MTI->getElementSizeInBytes());
804 } else {
806 NewDest, MTI->getDestAlign().valueOrOne(), NewSrc,
807 MTI->getSourceAlign().valueOrOne(), MTI->getLength(),
808 MTI->getElementSizeInBytes());
809 }
810 } else {
811 NewMI = IRB.CreateMemTransferInst(
812 MI->getIntrinsicID(), NewDest, MTI->getDestAlign(), NewSrc,
813 MTI->getSourceAlign(), MTI->getLength(),
814 cast<MemTransferInst>(MI)->isVolatile());
815 }
816 } else
817 reportFatalUsageError("Unimplemented LDS lowering memory intrinsic");
818 AsanInfo.Instructions.insert(NewMI);
819 MI->replaceAllUsesWith(NewMI);
820 MI->eraseFromParent();
821 } else if (AddrSpaceCastInst *ASC = dyn_cast<AddrSpaceCastInst>(Inst)) {
822 Value *AIOperand = ASC->getPointerOperand();
823 Value *NewAI = nullptr;
824 if (ASC->getDestAddressSpace() == AMDGPUAS::LOCAL_ADDRESS) {
825 NewAI =
826 getTranslatedLDSPtrOfFlat(LoadMallocPtr, AIOperand, ASC->getType());
827 } else {
828 Value *Replacement =
829 getTranslatedGlobalMemoryPtrOfLDS(LoadMallocPtr, AIOperand);
830 // Note: No need to add the instruction to AsanInfo instructions to be
831 // instrumented list. FLAT_ADDRESS ptr would have been already
832 // instrumented by asan pass prior to this pass.
833 NewAI = IRB.CreateAddrSpaceCast(Replacement, ASC->getType());
834 }
835 ASC->replaceAllUsesWith(NewAI);
836 ASC->eraseFromParent();
837 } else
838 report_fatal_error("Unimplemented LDS lowering instruction");
839 }
840}
841
842void AMDGPUSwLowerLDS::poisonRedzones(Function *Func, Value *MallocPtr) {
843 auto &LDSParams = FuncLDSAccessInfo.KernelToLDSParametersMap[Func];
844 Type *Int64Ty = IRB.getInt64Ty();
845 Type *VoidTy = IRB.getVoidTy();
846 FunctionCallee AsanPoisonRegion = M.getOrInsertFunction(
847 "__asan_poison_region",
848 FunctionType::get(VoidTy, {Int64Ty, Int64Ty}, false));
849
850 auto RedzonesVec = LDSParams.RedzoneOffsetAndSizeVector;
851 size_t VecSize = RedzonesVec.size();
852 for (unsigned i = 0; i < VecSize; i++) {
853 auto &RedzonePair = RedzonesVec[i];
854 uint64_t RedzoneOffset = RedzonePair.first;
855 uint64_t RedzoneSize = RedzonePair.second;
856 Value *RedzoneAddrOffset = IRB.CreateInBoundsGEP(
857 IRB.getInt8Ty(), MallocPtr, {IRB.getInt64(RedzoneOffset)});
858 Value *RedzoneAddress = IRB.CreatePtrToInt(RedzoneAddrOffset, Int64Ty);
859 IRB.CreateCall(AsanPoisonRegion,
860 {RedzoneAddress, IRB.getInt64(RedzoneSize)});
861 }
862}
863
864void AMDGPUSwLowerLDS::lowerKernelLDSAccesses(Function *Func,
865 DomTreeUpdater &DTU) {
866 LLVM_DEBUG(dbgs() << "Sw Lowering Kernel LDS for : " << Func->getName());
867 auto &LDSParams = FuncLDSAccessInfo.KernelToLDSParametersMap[Func];
868 auto &Ctx = M.getContext();
869 auto *PrevEntryBlock = &Func->getEntryBlock();
870 SetVector<Instruction *> LDSInstructions;
871 getLDSMemoryInstructions(Func, LDSInstructions);
872 const DataLayout &DL = M.getDataLayout();
873
874 // Create malloc block.
875 auto *MallocBlock = BasicBlock::Create(Ctx, "Malloc", Func, PrevEntryBlock);
876
877 // Create WIdBlock block which has instructions related to selection of
878 // {0,0,0} indiex work item in the work group.
879 auto *WIdBlock = BasicBlock::Create(Ctx, "WId", Func, MallocBlock);
880
881 // Move constant-size allocas from the original entry block to the new entry
882 // block (WIdBlock) so they remain static allocas. Splice the leading cluster
883 // in bulk, then move any stragglers that are interleaved with other
884 // instructions.
885 auto SplitIt = PrevEntryBlock->getFirstNonPHIOrAlloca();
886 WIdBlock->splice(WIdBlock->end(), PrevEntryBlock, PrevEntryBlock->begin(),
887 SplitIt);
888 for (Instruction &I : make_early_inc_range(*PrevEntryBlock))
889 if (auto *AI = dyn_cast<AllocaInst>(&I))
890 if (isa<ConstantInt>(AI->getArraySize()))
891 AI->moveBefore(*WIdBlock, WIdBlock->end());
892
893 IRB.SetInsertPoint(WIdBlock->end());
894 DebugLoc FirstDL =
895 getOrCreateDebugLoc(&*PrevEntryBlock->begin(), Func->getSubprogram());
896 IRB.SetCurrentDebugLocation(FirstDL);
897 Value *WIdx = IRB.CreateIntrinsic(Intrinsic::amdgcn_workitem_id_x, {});
898 Value *WIdy = IRB.CreateIntrinsic(Intrinsic::amdgcn_workitem_id_y, {});
899 Value *WIdz = IRB.CreateIntrinsic(Intrinsic::amdgcn_workitem_id_z, {});
900 Value *XYOr = IRB.CreateOr(WIdx, WIdy);
901 Value *XYZOr = IRB.CreateOr(XYOr, WIdz);
902 Value *WIdzCond = IRB.CreateICmpEQ(XYZOr, IRB.getInt32(0));
903
904 // All work items will branch to PrevEntryBlock except {0,0,0} index
905 // work item which will branch to malloc block.
906 IRB.CreateCondBr(WIdzCond, MallocBlock, PrevEntryBlock);
907
908 // Malloc block
909 IRB.SetInsertPoint(MallocBlock->begin());
910
911 // If Dynamic LDS globals are accessed by the kernel,
912 // Get the size of dyn lds from hidden dyn_lds_size kernel arg.
913 // Update the corresponding metadata global entries for this dyn lds global.
914 GlobalVariable *SwLDS = LDSParams.SwLDS;
915 GlobalVariable *SwLDSMetadata = LDSParams.SwLDSMetadata;
916 assert(SwLDS && SwLDSMetadata);
917 StructType *MetadataStructType =
918 cast<StructType>(SwLDSMetadata->getValueType());
919 Type *Int32Ty = IRB.getInt32Ty();
920 Type *Int64Ty = IRB.getInt64Ty();
921
922 SetVector<GlobalVariable *> UniqueLDSGlobals;
923 auto GetUniqueLDSGlobals = [&](SetVector<GlobalVariable *> &LDSGlobals) {
924 for (auto &GV : LDSGlobals) {
925 if (is_contained(UniqueLDSGlobals, GV))
926 continue;
927 UniqueLDSGlobals.insert(GV);
928 }
929 };
930
931 GetUniqueLDSGlobals(LDSParams.DirectAccess.StaticLDSGlobals);
932 GetUniqueLDSGlobals(LDSParams.IndirectAccess.StaticLDSGlobals);
933 // The metadata global always has an item for the SwLDS pointer itself, so
934 // there is at least one static item and the last one ends the static region.
935 unsigned LastStaticLDSIdx = UniqueLDSGlobals.size();
936 UniqueLDSGlobals.clear();
937
938 auto *GEPForEndStaticLDSOffset =
939 IRB.CreateInBoundsGEP(MetadataStructType, SwLDSMetadata,
940 {ConstantInt::get(Int32Ty, 0),
941 ConstantInt::get(Int32Ty, LastStaticLDSIdx),
942 ConstantInt::get(Int32Ty, 0)});
943
944 auto *GEPForEndStaticLDSSize =
945 IRB.CreateInBoundsGEP(MetadataStructType, SwLDSMetadata,
946 {ConstantInt::get(Int32Ty, 0),
947 ConstantInt::get(Int32Ty, LastStaticLDSIdx),
948 ConstantInt::get(Int32Ty, 2)});
949
950 Value *EndStaticLDSOffset = IRB.CreateLoad(Int32Ty, GEPForEndStaticLDSOffset);
951 Value *EndStaticLDSSize = IRB.CreateLoad(Int32Ty, GEPForEndStaticLDSSize);
952 Value *CurrMallocSize = IRB.CreateAdd(EndStaticLDSOffset, EndStaticLDSSize);
953
954 if (LDSParams.SwDynLDS) {
957 "Dynamic LDS size query is only supported for CO V5 and later.");
958 // Get size from hidden dyn_lds_size argument of kernel
960 IRB.CreateIntrinsic(Intrinsic::amdgcn_implicitarg_ptr, {});
961 Value *HiddenDynLDSSize = IRB.CreateInBoundsGEP(
962 ImplicitArg->getType(), ImplicitArg,
963 {ConstantInt::get(Int64Ty, COV5_HIDDEN_DYN_LDS_SIZE_ARG)});
964 UniqueLDSGlobals.clear();
965 GetUniqueLDSGlobals(LDSParams.DirectAccess.DynamicLDSGlobals);
966 GetUniqueLDSGlobals(LDSParams.IndirectAccess.DynamicLDSGlobals);
967 updateMallocSizeForDynamicLDS(Func, &CurrMallocSize, HiddenDynLDSSize,
968 UniqueLDSGlobals);
969 }
970
971 CurrMallocSize = IRB.CreateZExt(CurrMallocSize, Int64Ty);
972
973 // Create a call to malloc function which does device global memory allocation
974 // with size equals to all LDS global accesses size in this kernel.
975 Value *ReturnAddress = IRB.CreateIntrinsic(
976 Intrinsic::returnaddress, IRB.getPtrTy(DL.getProgramAddressSpace()),
977 {IRB.getInt32(0)});
978 FunctionCallee MallocFunc = M.getOrInsertFunction(
979 StringRef("__asan_malloc_impl"),
980 FunctionType::get(Int64Ty, {Int64Ty, Int64Ty}, false));
981 Value *RAPtrToInt = IRB.CreatePtrToInt(ReturnAddress, Int64Ty);
982 Value *MallocCall = IRB.CreateCall(MallocFunc, {CurrMallocSize, RAPtrToInt});
983
984 Value *MallocPtr =
986
987 // Create store of malloc to new global
988 IRB.CreateStore(MallocPtr, SwLDS);
989
990 // Create calls to __asan_poison_region to poison redzones.
991 poisonRedzones(Func, MallocPtr);
992
993 // Create branch to PrevEntryBlock
994 IRB.CreateBr(PrevEntryBlock);
995
996 // Create wave-group barrier at the starting of Previous entry block
997 Type *Int1Ty = IRB.getInt1Ty();
998 IRB.SetInsertPoint(PrevEntryBlock->begin());
999 auto *XYZCondPhi = IRB.CreatePHI(Int1Ty, 2, "xyzCond");
1000 XYZCondPhi->addIncoming(IRB.getInt1(0), WIdBlock);
1001 XYZCondPhi->addIncoming(IRB.getInt1(1), MallocBlock);
1002
1003 IRB.CreateIntrinsic(Intrinsic::amdgcn_s_barrier, {});
1004
1005 // Load malloc pointer from Sw LDS.
1006 Value *LoadMallocPtr =
1008
1009 // Replace All uses of LDS globals with new LDS pointers.
1010 replaceKernelLDSAccesses(Func);
1011
1012 // Replace Memory Operations on LDS with corresponding
1013 // global memory pointers.
1014 translateLDSMemoryOperationsToGlobalMemory(Func, LoadMallocPtr,
1015 LDSInstructions);
1016
1017 auto *CondFreeBlock = BasicBlock::Create(Ctx, "CondFree", Func);
1018 auto *FreeBlock = BasicBlock::Create(Ctx, "Free", Func);
1019 auto *EndBlock = BasicBlock::Create(Ctx, "End", Func);
1020 for (BasicBlock &BB : *Func) {
1021 if (!BB.empty()) {
1022 if (ReturnInst *RI = dyn_cast<ReturnInst>(&BB.back())) {
1023 RI->eraseFromParent();
1024 IRB.SetInsertPoint(BB.end());
1025 IRB.CreateBr(CondFreeBlock);
1026 }
1027 }
1028 }
1029
1030 // Cond Free Block
1031 IRB.SetInsertPoint(CondFreeBlock->begin());
1032 IRB.CreateIntrinsic(Intrinsic::amdgcn_s_barrier, {});
1033 IRB.CreateCondBr(XYZCondPhi, FreeBlock, EndBlock);
1034
1035 // Free Block
1036 IRB.SetInsertPoint(FreeBlock->begin());
1037
1038 // Free the previously allocate device global memory.
1039 FunctionCallee AsanFreeFunc = M.getOrInsertFunction(
1040 StringRef("__asan_free_impl"),
1041 FunctionType::get(IRB.getVoidTy(), {Int64Ty, Int64Ty}, false));
1042 Value *ReturnAddr = IRB.CreateIntrinsic(
1043 Intrinsic::returnaddress, IRB.getPtrTy(DL.getProgramAddressSpace()),
1044 IRB.getInt32(0));
1045 Value *RAPToInt = IRB.CreatePtrToInt(ReturnAddr, Int64Ty);
1046 Value *MallocPtrToInt = IRB.CreatePtrToInt(LoadMallocPtr, Int64Ty);
1047 IRB.CreateCall(AsanFreeFunc, {MallocPtrToInt, RAPToInt});
1048
1049 IRB.CreateBr(EndBlock);
1050
1051 // End Block
1052 IRB.SetInsertPoint(EndBlock->begin());
1053 IRB.CreateRetVoid();
1054 // Update the DomTree with corresponding links to basic blocks.
1055 DTU.applyUpdates({{DominatorTree::Insert, WIdBlock, MallocBlock},
1056 {DominatorTree::Insert, MallocBlock, PrevEntryBlock},
1057 {DominatorTree::Insert, CondFreeBlock, FreeBlock},
1058 {DominatorTree::Insert, FreeBlock, EndBlock}});
1059}
1060
1061Constant *AMDGPUSwLowerLDS::getAddressesOfVariablesInKernel(
1062 Function *Func, SetVector<GlobalVariable *> &Variables) {
1063 Type *Int32Ty = IRB.getInt32Ty();
1064 ArrayType *KernelOffsetsType =
1066
1067 // Kernels with no direct or indirect LDS uses never read their offset row.
1068 auto KernelIt = FuncLDSAccessInfo.KernelToLDSParametersMap.find(Func);
1069 if (KernelIt == FuncLDSAccessInfo.KernelToLDSParametersMap.end() ||
1070 !KernelIt->second.SwLDSMetadata)
1071 return PoisonValue::get(KernelOffsetsType);
1072
1073 auto &LDSParams = KernelIt->second;
1074 GlobalVariable *SwLDSMetadata = LDSParams.SwLDSMetadata;
1075 auto *SwLDSMetadataStructType =
1076 cast<StructType>(SwLDSMetadata->getValueType());
1077
1078 SmallVector<Constant *> Elements;
1079 for (auto *GV : Variables) {
1080 auto It = LDSParams.LDSToReplacementIndicesMap.find(GV);
1081 if (It == LDSParams.LDSToReplacementIndicesMap.end()) {
1082 Elements.push_back(
1084 continue;
1085 }
1086 auto &Indices = It->second;
1087 Constant *GEPIdx[] = {ConstantInt::get(Int32Ty, Indices[0]),
1088 ConstantInt::get(Int32Ty, Indices[1]),
1089 ConstantInt::get(Int32Ty, Indices[2])};
1091 Func->getDataLayout(), SwLDSMetadataStructType, SwLDSMetadata, GEPIdx,
1093 Elements.push_back(GEP);
1094 }
1095 return ConstantArray::get(KernelOffsetsType, Elements);
1096}
1097
1098void AMDGPUSwLowerLDS::buildNonKernelLDSBaseTable(
1099 NonKernelLDSParameters &NKLDSParams) {
1100 // Base table will have single row, with elements of the row
1101 // placed as per kernel ID. Each element in the row corresponds
1102 // to addresss of "SW LDS" global of the kernel.
1103 auto &Kernels = NKLDSParams.OrderedKernels;
1104 if (Kernels.empty())
1105 return;
1106 const size_t NumberKernels = Kernels.size();
1108 ArrayType *AllKernelsOffsetsType = ArrayType::get(LDSPtrTy, NumberKernels);
1109 std::vector<Constant *> OverallConstantExprElts(NumberKernels);
1110 for (size_t i = 0; i < NumberKernels; i++) {
1111 // Null for kernels without LDS. Lowered functions still load this entry
1112 // on entry, but only use it on LDS paths these kernels never take.
1113 Constant *SwLDS = ConstantPointerNull::get(LDSPtrTy);
1114 auto It = FuncLDSAccessInfo.KernelToLDSParametersMap.find(Kernels[i]);
1115 if (It != FuncLDSAccessInfo.KernelToLDSParametersMap.end() &&
1116 It->second.SwLDS)
1117 SwLDS = It->second.SwLDS;
1118 OverallConstantExprElts[i] = SwLDS;
1119 }
1120 Constant *init =
1121 ConstantArray::get(AllKernelsOffsetsType, OverallConstantExprElts);
1122 NKLDSParams.LDSBaseTable = new GlobalVariable(
1123 M, AllKernelsOffsetsType, true, GlobalValue::InternalLinkage, init,
1124 "llvm.amdgcn.sw.lds.base.table", nullptr, GlobalValue::NotThreadLocal,
1127 MD.NoAddress = true;
1128 NKLDSParams.LDSBaseTable->setSanitizerMetadata(MD);
1129}
1130
1131void AMDGPUSwLowerLDS::buildNonKernelLDSOffsetTable(
1132 NonKernelLDSParameters &NKLDSParams) {
1133 // Offset table will have multiple rows and columns.
1134 // Rows are assumed to be from 0 to (n-1). n is total number
1135 // of kernels accessing the LDS through non-kernels.
1136 // Each row will have m elements. m is the total number of
1137 // unique LDS globals accessed by non-kernels.
1138 // Each element in the row correspond to the address of
1139 // the replacement of LDS global done by that particular kernel.
1140 auto &Variables = NKLDSParams.OrdereLDSGlobals;
1141 auto &Kernels = NKLDSParams.OrderedKernels;
1142 if (Variables.empty() || Kernels.empty())
1143 return;
1144 const size_t NumberVariables = Variables.size();
1145 const size_t NumberKernels = Kernels.size();
1146
1147 ArrayType *KernelOffsetsType =
1148 ArrayType::get(IRB.getPtrTy(AMDGPUAS::GLOBAL_ADDRESS), NumberVariables);
1149
1150 ArrayType *AllKernelsOffsetsType =
1151 ArrayType::get(KernelOffsetsType, NumberKernels);
1152 std::vector<Constant *> overallConstantExprElts(NumberKernels);
1153 for (size_t i = 0; i < NumberKernels; i++) {
1154 Function *Func = Kernels[i];
1155 overallConstantExprElts[i] =
1156 getAddressesOfVariablesInKernel(Func, Variables);
1157 }
1158 Constant *Init =
1159 ConstantArray::get(AllKernelsOffsetsType, overallConstantExprElts);
1160 NKLDSParams.LDSOffsetTable = new GlobalVariable(
1161 M, AllKernelsOffsetsType, true, GlobalValue::InternalLinkage, Init,
1162 "llvm.amdgcn.sw.lds.offset.table", nullptr, GlobalValue::NotThreadLocal,
1165 MD.NoAddress = true;
1166 NKLDSParams.LDSOffsetTable->setSanitizerMetadata(MD);
1167}
1168
1169void AMDGPUSwLowerLDS::lowerNonKernelLDSAccesses(
1170 Function *Func, SetVector<GlobalVariable *> &LDSGlobals,
1171 NonKernelLDSParameters &NKLDSParams) {
1172 // Replace LDS access in non-kernel with replacement queried from
1173 // Base table and offset from offset table.
1174 LLVM_DEBUG(dbgs() << "Sw LDS lowering, lower non-kernel access for : "
1175 << Func->getName());
1176 auto InsertAt = Func->getEntryBlock().getFirstNonPHIOrAlloca();
1177 IRB.SetInsertPoint(InsertAt);
1178
1179 // Get LDS memory instructions.
1180 SetVector<Instruction *> LDSInstructions;
1181 getLDSMemoryInstructions(Func, LDSInstructions);
1182
1183 auto *KernelId = IRB.CreateIntrinsic(Intrinsic::amdgcn_lds_kernel_id, {});
1184 GlobalVariable *LDSBaseTable = NKLDSParams.LDSBaseTable;
1185 GlobalVariable *LDSOffsetTable = NKLDSParams.LDSOffsetTable;
1186 auto &OrdereLDSGlobals = NKLDSParams.OrdereLDSGlobals;
1187 Value *BaseGEP = IRB.CreateInBoundsGEP(
1188 LDSBaseTable->getValueType(), LDSBaseTable, {IRB.getInt32(0), KernelId});
1189 Value *BaseLoad =
1190 IRB.CreateLoad(IRB.getPtrTy(AMDGPUAS::LOCAL_ADDRESS), BaseGEP);
1191 Value *LoadMallocPtr =
1192 IRB.CreateLoad(IRB.getPtrTy(AMDGPUAS::GLOBAL_ADDRESS), BaseLoad);
1193
1194 for (GlobalVariable *GV : LDSGlobals) {
1195 const auto *GVIt = llvm::find(OrdereLDSGlobals, GV);
1196 assert(GVIt != OrdereLDSGlobals.end());
1197 uint32_t GVOffset = std::distance(OrdereLDSGlobals.begin(), GVIt);
1198
1199 Value *OffsetGEP = IRB.CreateInBoundsGEP(
1200 LDSOffsetTable->getValueType(), LDSOffsetTable,
1201 {IRB.getInt32(0), KernelId, IRB.getInt32(GVOffset)});
1202 Value *OffsetLoad =
1203 IRB.CreateLoad(IRB.getPtrTy(AMDGPUAS::GLOBAL_ADDRESS), OffsetGEP);
1204 Value *Offset = IRB.CreateLoad(IRB.getInt32Ty(), OffsetLoad);
1205 Value *BasePlusOffset =
1206 IRB.CreateInBoundsGEP(IRB.getInt8Ty(), BaseLoad, {Offset});
1207 LLVM_DEBUG(dbgs() << "Sw LDS Lowering, Replace non-kernel LDS for "
1208 << GV->getName());
1209 replacesUsesOfGlobalInFunction(Func, GV, BasePlusOffset);
1210 }
1211 translateLDSMemoryOperationsToGlobalMemory(Func, LoadMallocPtr,
1212 LDSInstructions);
1213}
1214
1215static void reorderStaticDynamicIndirectLDSSet(KernelLDSParameters &LDSParams) {
1216 // Sort Static, dynamic LDS globals which are either
1217 // direct or indirect access on basis of name.
1218 auto &DirectAccess = LDSParams.DirectAccess;
1219 auto &IndirectAccess = LDSParams.IndirectAccess;
1220 LDSParams.DirectAccess.StaticLDSGlobals = sortByName(
1221 std::vector<GlobalVariable *>(DirectAccess.StaticLDSGlobals.begin(),
1222 DirectAccess.StaticLDSGlobals.end()));
1223 LDSParams.DirectAccess.DynamicLDSGlobals = sortByName(
1224 std::vector<GlobalVariable *>(DirectAccess.DynamicLDSGlobals.begin(),
1225 DirectAccess.DynamicLDSGlobals.end()));
1226 LDSParams.IndirectAccess.StaticLDSGlobals = sortByName(
1227 std::vector<GlobalVariable *>(IndirectAccess.StaticLDSGlobals.begin(),
1228 IndirectAccess.StaticLDSGlobals.end()));
1229 LDSParams.IndirectAccess.DynamicLDSGlobals = sortByName(
1230 std::vector<GlobalVariable *>(IndirectAccess.DynamicLDSGlobals.begin(),
1231 IndirectAccess.DynamicLDSGlobals.end()));
1232}
1233
1234void AMDGPUSwLowerLDS::initAsanInfo() {
1235 // Get Shadow mapping scale and offset.
1236 unsigned LongSize =
1237 M.getDataLayout().getPointerSizeInBits(AMDGPUAS::GLOBAL_ADDRESS);
1239 int Scale;
1240 bool OrShadowOffset;
1241 llvm::getAddressSanitizerParams(M.getContext(), M.getTargetTriple(), LongSize,
1242 false, &Offset, &Scale, &OrShadowOffset);
1243 AsanInfo.Scale = Scale;
1244 AsanInfo.Offset = Offset;
1245}
1246
1247static bool hasFnWithSanitizeAddressAttr(FunctionVariableMap &LDSAccesses) {
1248 for (auto &K : LDSAccesses) {
1249 Function *F = K.first;
1250 if (!F)
1251 continue;
1252 if (F->hasFnAttribute(Attribute::SanitizeAddress))
1253 return true;
1254 }
1255 return false;
1256}
1257
1258bool AMDGPUSwLowerLDS::run() {
1259 bool Changed = false;
1260
1261 CallGraph CG = CallGraph(M);
1262
1263 Changed |=
1265
1266 // Get all the direct and indirect access of LDS for all the kernels.
1268
1269 // Flag to decide whether to lower all the LDS accesses
1270 // based on sanitize_address attribute.
1271 bool LowerAllLDS = hasFnWithSanitizeAddressAttr(LDSUsesInfo.DirectAccess) ||
1272 hasFnWithSanitizeAddressAttr(LDSUsesInfo.IndirectAccess);
1273
1274 if (!LowerAllLDS)
1275 return Changed;
1276
1277 // Utility to group LDS access into direct, indirect, static and dynamic.
1278 auto PopulateKernelStaticDynamicLDS = [&](FunctionVariableMap &LDSAccesses,
1279 bool DirectAccess) {
1280 for (auto &K : LDSAccesses) {
1281 Function *F = K.first;
1282 if (!F || K.second.empty())
1283 continue;
1284
1285 assert(isKernel(*F));
1286
1287 // Only inserts if key isn't already in the map.
1288 FuncLDSAccessInfo.KernelToLDSParametersMap.insert(
1289 {F, KernelLDSParameters()});
1290
1291 auto &LDSParams = FuncLDSAccessInfo.KernelToLDSParametersMap[F];
1292 if (!DirectAccess)
1293 FuncLDSAccessInfo.KernelsWithIndirectLDSAccess.insert(F);
1294 for (GlobalVariable *GV : K.second) {
1295 if (!DirectAccess) {
1296 if (AMDGPU::isDynamicLDS(*GV))
1297 LDSParams.IndirectAccess.DynamicLDSGlobals.insert(GV);
1298 else
1299 LDSParams.IndirectAccess.StaticLDSGlobals.insert(GV);
1300 FuncLDSAccessInfo.AllNonKernelLDSAccess.insert(GV);
1301 } else {
1302 if (AMDGPU::isDynamicLDS(*GV))
1303 LDSParams.DirectAccess.DynamicLDSGlobals.insert(GV);
1304 else
1305 LDSParams.DirectAccess.StaticLDSGlobals.insert(GV);
1306 }
1307 }
1308 }
1309 };
1310
1311 PopulateKernelStaticDynamicLDS(LDSUsesInfo.DirectAccess, true);
1312 PopulateKernelStaticDynamicLDS(LDSUsesInfo.IndirectAccess, false);
1313
1314 // Get address sanitizer scale.
1315 initAsanInfo();
1316
1317 for (auto &K : FuncLDSAccessInfo.KernelToLDSParametersMap) {
1318 Function *Func = K.first;
1319 auto &LDSParams = FuncLDSAccessInfo.KernelToLDSParametersMap[Func];
1320 if (LDSParams.DirectAccess.StaticLDSGlobals.empty() &&
1321 LDSParams.DirectAccess.DynamicLDSGlobals.empty() &&
1322 LDSParams.IndirectAccess.StaticLDSGlobals.empty() &&
1323 LDSParams.IndirectAccess.DynamicLDSGlobals.empty())
1324 continue;
1325
1327 CG, Func,
1328 {"amdgpu-no-workitem-id-x", "amdgpu-no-workitem-id-y",
1329 "amdgpu-no-workitem-id-z", "amdgpu-no-heap-ptr"});
1330 reorderStaticDynamicIndirectLDSSet(LDSParams);
1331 buildSwLDSGlobal(Func);
1332 buildSwDynLDSGlobal(Func);
1333 populateSwMetadataGlobal(Func);
1334 populateSwLDSAttributeAndMetadata(Func);
1335 populateLDSToReplacementIndicesMap(Func);
1336 DomTreeUpdater DTU(DTCallback(*Func), DomTreeUpdater::UpdateStrategy::Lazy);
1337 lowerKernelLDSAccesses(Func, DTU);
1338 Changed = true;
1339 }
1340
1341 // Get the Uses of LDS from non-kernels.
1342 getUsesOfLDSByNonKernels();
1343
1344 // Get non-kernels with LDS instructions and the kernels reaching them.
1345 getNonKernelsWithLDSInstructions(CG);
1346
1347 // TODO: removeFnAttrFromReachable does not visit callees of indirect call
1348 // targets.
1349 for (Function *Func : FuncLDSAccessInfo.KernelsWithIndirectLDSAccess)
1350 removeFnAttrFromReachable(CG, Func, {"amdgpu-no-lds-kernel-id"});
1351
1352 // Lower LDS accesses in non-kernels.
1353 if (!FuncLDSAccessInfo.NonKernelToLDSAccessMap.empty() ||
1354 !FuncLDSAccessInfo.NonKernelsWithLDSInstructions.empty()) {
1355 NonKernelLDSParameters NKLDSParams;
1356 NKLDSParams.OrderedKernels = getOrderedIndirectLDSAccessingKernels(
1357 FuncLDSAccessInfo.KernelsWithIndirectLDSAccess);
1358 NKLDSParams.OrdereLDSGlobals = getOrderedNonKernelAllLDSGlobals(
1359 FuncLDSAccessInfo.AllNonKernelLDSAccess);
1360 buildNonKernelLDSBaseTable(NKLDSParams);
1361 buildNonKernelLDSOffsetTable(NKLDSParams);
1362 for (auto &K : FuncLDSAccessInfo.NonKernelToLDSAccessMap) {
1363 Function *Func = K.first;
1364 DenseSet<GlobalVariable *> &LDSGlobals = K.second;
1365 SetVector<GlobalVariable *> OrderedLDSGlobals = sortByName(
1366 std::vector<GlobalVariable *>(LDSGlobals.begin(), LDSGlobals.end()));
1367 lowerNonKernelLDSAccesses(Func, OrderedLDSGlobals, NKLDSParams);
1368 }
1369 for (Function *Func : FuncLDSAccessInfo.NonKernelsWithLDSInstructions) {
1370 if (FuncLDSAccessInfo.NonKernelToLDSAccessMap.contains(Func))
1371 continue;
1372 SetVector<GlobalVariable *> NoLDSGlobals;
1373 lowerNonKernelLDSAccesses(Func, NoLDSGlobals, NKLDSParams);
1374 }
1375 Changed = true;
1376 }
1377
1378 if (!Changed)
1379 return Changed;
1380
1381 for (auto &GV : make_early_inc_range(M.globals())) {
1383 // probably want to remove from used lists
1385 if (GV.use_empty())
1386 GV.eraseFromParent();
1387 }
1388 }
1389
1390 if (M.getContext()
1391 .getLibraryOptions()
1392 .get<AMDGPUOptions>()
1393 .asan_instrument_lds) {
1394 SmallVector<InterestingMemoryOperand, 16> OperandsToInstrument;
1395 for (Instruction *Inst : AsanInfo.Instructions) {
1396 SmallVector<InterestingMemoryOperand, 1> InterestingOperands;
1397 getInterestingMemoryOperands(M, Inst, InterestingOperands);
1398 llvm::append_range(OperandsToInstrument, InterestingOperands);
1399 }
1400 for (auto &Operand : OperandsToInstrument) {
1401 Value *Addr = Operand.getPtr();
1402 instrumentAddress(M, IRB, Operand.getInsn(), Operand.getInsn(), Addr,
1403 Operand.Alignment.valueOrOne(), Operand.TypeStoreSize,
1404 Operand.IsWrite, nullptr, false, false, AsanInfo.Scale,
1405 AsanInfo.Offset);
1406 Changed = true;
1407 }
1408 }
1409
1410 return Changed;
1411}
1412
1413class AMDGPUSwLowerLDSLegacy : public ModulePass {
1414public:
1415 static char ID;
1416 AMDGPUSwLowerLDSLegacy() : ModulePass(ID) {}
1417 bool runOnModule(Module &M) override;
1418 void getAnalysisUsage(AnalysisUsage &AU) const override {
1420 }
1421};
1422} // namespace
1423
1424char AMDGPUSwLowerLDSLegacy::ID = 0;
1425char &llvm::AMDGPUSwLowerLDSLegacyPassID = AMDGPUSwLowerLDSLegacy::ID;
1426
1427INITIALIZE_PASS_BEGIN(AMDGPUSwLowerLDSLegacy, "amdgpu-sw-lower-lds",
1428 "AMDGPU Software lowering of LDS", false, false)
1430INITIALIZE_PASS_END(AMDGPUSwLowerLDSLegacy, "amdgpu-sw-lower-lds",
1431 "AMDGPU Software lowering of LDS", false, false)
1432
1433bool AMDGPUSwLowerLDSLegacy::runOnModule(Module &M) {
1434 // AddressSanitizer pass adds "nosanitize_address" module flag if it has
1435 // instrumented the IR. Return early if the flag is not present.
1436 if (!M.getModuleFlag("nosanitize_address"))
1437 return false;
1438 DominatorTreeWrapperPass *const DTW =
1439 getAnalysisIfAvailable<DominatorTreeWrapperPass>();
1440 auto DTCallback = [&DTW](Function &F) -> DominatorTree * {
1441 return DTW ? &DTW->getDomTree() : nullptr;
1442 };
1443
1444 AMDGPUSwLowerLDS SwLowerLDSImpl(M, DTCallback);
1445 bool IsChanged = SwLowerLDSImpl.run();
1446 return IsChanged;
1447}
1448
1450 return new AMDGPUSwLowerLDSLegacy();
1451}
1452
1455 // AddressSanitizer pass adds "nosanitize_address" module flag if it has
1456 // instrumented the IR. Return early if the flag is not present.
1457 if (!M.getModuleFlag("nosanitize_address"))
1458 return PreservedAnalyses::all();
1459 auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
1460 auto DTCallback = [&FAM](Function &F) -> DominatorTree * {
1461 return &FAM.getResult<DominatorTreeAnalysis>(F);
1462 };
1463 AMDGPUSwLowerLDS SwLowerLDSImpl(M, DTCallback);
1464 bool IsChanged = SwLowerLDSImpl.run();
1465 if (!IsChanged)
1466 return PreservedAnalyses::all();
1467
1470 return PA;
1471}
assert(UImm &&(UImm !=~static_cast< T >(0)) &&"Invalid immediate!")
uint64_t uint64_t
MachineBasicBlock MachineBasicBlock::iterator DebugLoc DL
Expand Atomic instructions
This file provides interfaces used to build and manipulate a call graph, which is a very useful tool ...
This file contains the declarations for the subclasses of Constant, which represent the different fla...
Hexagon Common GEP
IRTranslator LLVM IR MI
#define F(x, y, z)
Definition MD5.cpp:54
#define I(x, y, z)
Definition MD5.cpp:57
if(auto Err=PB.parsePassPipeline(MPM, Passes)) return wrap(std MPM run * Mod
FunctionAnalysisManager FAM
#define INITIALIZE_PASS_DEPENDENCY(depName)
Definition PassSupport.h:42
#define INITIALIZE_PASS_END(passName, arg, name, cfg, analysis)
Definition PassSupport.h:44
#define INITIALIZE_PASS_BEGIN(passName, arg, name, cfg, analysis)
Definition PassSupport.h:39
static Split data
This file contains some functions that are useful when dealing with strings.
#define LLVM_DEBUG(...)
Definition Debug.h:119
static DebugLoc getOrCreateDebugLoc(const Instruction *InsertBefore, DISubprogram *SP)
This class represents a conversion between pointers from one address space to another.
PassT::Result & getResult(IRUnitT &IR, ExtraArgTs... ExtraArgs)
Get the result of an analysis pass for a given IR unit.
Represent the analysis usage information of a pass.
AnalysisUsage & addPreserved()
Add the specified Pass class to the set of analyses preserved by this pass.
This class represents any memset intrinsic.
static LLVM_ABI ArrayType * get(Type *ElementType, uint64_t NumElements)
This static method is the primary way to construct an ArrayType.
An instruction that atomically checks whether a specified value is in a memory location,...
void setVolatile(bool V)
Specify whether this is a volatile cmpxchg.
an instruction that atomically reads a memory location, combines it with another value,...
void setVolatile(bool V)
Specify whether this is a volatile RMW or not.
LLVM Basic Block Representation.
Definition BasicBlock.h:62
static BasicBlock * Create(LLVMContext &Context, const Twine &Name="", Function *Parent=nullptr, BasicBlock *InsertBefore=nullptr)
Creates a new BasicBlock.
Definition BasicBlock.h:206
std::pair< std::optional< WeakTrackingVH >, CallGraphNode * > CallRecord
A pair of the calling instruction (a call or invoke) and the call graph node being called.
Definition CallGraph.h:174
The basic data container for the call graph of a Module of IR.
Definition CallGraph.h:72
This class represents a function call, abstracting a target machine's calling convention.
static LLVM_ABI Constant * get(ArrayType *T, ArrayRef< Constant * > V)
static ConstantAsMetadata * get(Constant *C)
Definition Metadata.h:549
static LLVM_SUPPRESS_DEPRECATED_DECLARATIONS_PUSH Constant * getGetElementPtr(Type *Ty, Constant *C, ArrayRef< Constant * > IdxList, GEPNoWrapFlags NW=GEPNoWrapFlags::none(), std::optional< ConstantRange > InRange=std::nullopt, Type *OnlyIfReducedTy=nullptr)
Getelementptr form.
Definition Constants.h:1477
static LLVM_ABI Constant * getAddrSpaceCast(Constant *C, Type *Ty, bool OnlyIfReduced=false)
static LLVM_ABI ConstantPointerNull * get(PointerType *T)
Static factory methods - Return objects of the specified value.
static LLVM_ABI Constant * get(StructType *T, ArrayRef< Constant * > V)
This is an important base class in LLVM.
Definition Constant.h:44
LLVM_ABI void removeDeadConstantUsers() const
If there are any dead constant users dangling off of this constant, remove them.
static LLVM_ABI Constant * getNullValue(Type *Ty)
Constructor to create a '0' constant of arbitrary type.
Subprogram description. Uses SubclassData1.
A parsed version of the target data layout string in and methods for querying it.
Definition DataLayout.h:64
A debug info location.
Definition DebugLoc.h:126
std::pair< iterator, bool > try_emplace(KeyT &&Key, Ts &&...Args)
Definition DenseMap.h:856
Implements a dense probed hash-table based set.
Definition DenseSet.h:283
Analysis pass which computes a DominatorTree.
Definition Dominators.h:241
Legacy analysis pass which computes a DominatorTree.
Definition Dominators.h:277
Concrete subclass of DominatorTreeBase that is used to compute a normal dominator tree.
Definition Dominators.h:122
A handy container for a FunctionType+Callee-pointer pair, which can be passed around as a single enti...
static LLVM_ABI FunctionType * get(Type *Result, ArrayRef< Type * > Params, bool isVarArg)
This static method is the primary way of constructing a FunctionType.
static GEPNoWrapFlags inBounds()
void applyUpdates(ArrayRef< UpdateT > Updates)
Submit updates to all available trees.
LLVM_ABI void setMetadata(unsigned KindID, MDNode *Node)
Set a particular kind of metadata attachment.
LLVM_ABI bool isDeclaration() const
Return true if the primary definition of this global value is outside of the current translation unit...
Definition Globals.cpp:408
LLVM_ABI void setSanitizerMetadata(SanitizerMetadata Meta)
Definition Globals.cpp:324
@ InternalLinkage
Rename collisions when linking (static functions).
Definition GlobalValue.h:60
@ ExternalLinkage
Externally visible function.
Definition GlobalValue.h:53
Type * getValueType() const
MaybeAlign getAlign() const
Returns the alignment of the given variable.
LLVM_ABI void eraseFromParent()
eraseFromParent - This method unlinks 'this' from the containing module and deletes it.
Definition Globals.cpp:609
ConstantInt * getInt1(bool V)
Get a constant value representing either true or false.
Definition IRBuilder.h:446
Value * CreateAddrSpaceCast(Value *V, Type *DestTy, const Twine &Name="", bool IsNonNull=false)
Definition IRBuilder.h:2324
AtomicCmpXchgInst * CreateAtomicCmpXchg(Value *Ptr, Value *Cmp, Value *New, MaybeAlign Align, AtomicOrdering SuccessOrdering, AtomicOrdering FailureOrdering, SyncScope::ID SSID=SyncScope::System)
Definition IRBuilder.h:1974
IntegerType * getInt1Ty()
Fetch the type representing a single bit.
Definition IRBuilder.h:513
CondBrInst * CreateCondBr(Value *Cond, BasicBlock *True, BasicBlock *False, MDNode *BranchWeights=nullptr, MDNode *Unpredictable=nullptr)
Create a conditional 'br Cond, TrueDest, FalseDest' instruction.
Definition IRBuilder.h:1218
Value * CreatePtrToAddr(Value *V, const Twine &Name="")
Definition IRBuilder.h:2305
LLVM_ABI Value * CreateVectorSplat(unsigned NumElts, Value *V, const Twine &Name="")
Return a vector value that contains.
LLVM_ABI Value * CreateSelect(Value *C, Value *True, Value *False, const Twine &Name="", Instruction *MDFrom=nullptr)
Value * CreateIntToPtr(Value *V, Type *DestTy, const Twine &Name="")
Definition IRBuilder.h:2314
void SetCurrentDebugLocation(const DebugLoc &L)
Set location information used by debugging information.
Definition IRBuilder.h:229
IntegerType * getInt32Ty()
Fetch the type representing a 32-bit integer.
Definition IRBuilder.h:528
IntegerType * getInt64Ty()
Fetch the type representing a 64-bit integer.
Definition IRBuilder.h:533
Value * CreateInBoundsGEP(Type *Ty, Value *Ptr, ArrayRef< Value * > IdxList, const Twine &Name="")
Definition IRBuilder.h:2021
LLVM_ABI CallInst * CreateElementUnorderedAtomicMemMove(Value *Dst, Align DstAlign, Value *Src, Align SrcAlign, Value *Size, uint32_t ElementSize, const AAMDNodes &AAInfo=AAMDNodes())
Create and insert an element unordered-atomic memmove between the specified pointers.
Value * CreateUDiv(Value *LHS, Value *RHS, const Twine &Name="", bool isExact=false)
Definition IRBuilder.h:1475
ConstantInt * getInt64(uint64_t C)
Get a constant 64-bit value.
Definition IRBuilder.h:476
UncondBrInst * CreateBr(BasicBlock *Dest)
Create an unconditional 'br label X' instruction.
Definition IRBuilder.h:1212
ConstantInt * getInt32(uint32_t C)
Get a constant 32-bit value.
Definition IRBuilder.h:471
PHINode * CreatePHI(Type *Ty, unsigned NumReservedValues, const Twine &Name="")
Definition IRBuilder.h:2626
Value * CreateICmpEQ(Value *LHS, Value *RHS, const Twine &Name="")
Definition IRBuilder.h:2458
Value * CreateSub(Value *LHS, Value *RHS, const Twine &Name="", bool HasNUW=false, bool HasNSW=false)
Definition IRBuilder.h:1441
LoadInst * CreateLoad(Type *Ty, Value *Ptr, const char *Name)
Provided to resolve 'CreateLoad(Ty, Ptr, "...")' correctly, instead of converting the string to 'bool...
Definition IRBuilder.h:1913
CallInst * CreateElementUnorderedAtomicMemSet(Value *Ptr, Value *Val, uint64_t Size, Align Alignment, uint32_t ElementSize, const AAMDNodes &AAInfo=AAMDNodes())
Create and insert an element unordered-atomic memset of the region of memory starting at the given po...
Definition IRBuilder.h:623
CallInst * CreateMemSet(Value *Ptr, Value *Val, uint64_t Size, MaybeAlign Align, bool isVolatile=false, const AAMDNodes &AAInfo=AAMDNodes())
Create and insert a memset to the specified pointer and the specified value.
Definition IRBuilder.h:602
Value * CreateZExt(Value *V, Type *DestTy, const Twine &Name="", bool IsNonNeg=false)
Definition IRBuilder.h:2197
ReturnInst * CreateRetVoid()
Create a 'ret void' instruction.
Definition IRBuilder.h:1189
LLVM_ABI Value * CreateIntrinsic(Intrinsic::ID ID, ArrayRef< Type * > OverloadTypes, ArrayRef< Value * > Args, FMFSource FMFSource={}, const Twine &Name="", ArrayRef< OperandBundleDef > OpBundles={}, function_ref< void(CallInst *)> SetFn=[](CallInst *) {})
Variant to create a possibly constant-folded intrinsic.
StoreInst * CreateStore(Value *Val, Value *Ptr, bool isVolatile=false)
Definition IRBuilder.h:1931
Value * CreateAdd(Value *LHS, Value *RHS, const Twine &Name="", bool HasNUW=false, bool HasNSW=false)
Definition IRBuilder.h:1424
Value * CreatePtrToInt(Value *V, Type *DestTy, const Twine &Name="")
Definition IRBuilder.h:2309
CallInst * CreateCall(FunctionType *FTy, Value *Callee, ArrayRef< Value * > Args={}, const Twine &Name="", MDNode *FPMathTag=nullptr)
Definition IRBuilder.h:2640
Value * CreateTrunc(Value *V, Type *DestTy, const Twine &Name="", bool IsNUW=false, bool IsNSW=false)
Definition IRBuilder.h:2183
PointerType * getPtrTy(unsigned AddrSpace=0)
Fetch the type representing a pointer.
Definition IRBuilder.h:571
Value * CreateIsNull(Value *Arg, const Twine &Name="")
Return a boolean value testing if Arg == 0.
Definition IRBuilder.h:2837
void SetInsertPoint(BasicBlock *TheBB)
This specifies that created instructions should be appended to the end of the specified block.
Definition IRBuilder.h:197
Type * getVoidTy()
Fetch the type representing void.
Definition IRBuilder.h:566
LLVM_ABI CallInst * CreateElementUnorderedAtomicMemCpy(Value *Dst, Align DstAlign, Value *Src, Align SrcAlign, Value *Size, uint32_t ElementSize, const AAMDNodes &AAInfo=AAMDNodes())
Create and insert an element unordered-atomic memcpy between the specified pointers.
LLVM_ABI CallInst * CreateMemTransferInst(Intrinsic::ID IntrID, Value *Dst, MaybeAlign DstAlign, Value *Src, MaybeAlign SrcAlign, Value *Size, bool isVolatile=false, const AAMDNodes &AAInfo=AAMDNodes())
Value * CreateOr(Value *LHS, Value *RHS, const Twine &Name="", bool IsDisjoint=false)
Definition IRBuilder.h:1594
IntegerType * getInt8Ty()
Fetch the type representing an 8-bit integer.
Definition IRBuilder.h:518
Value * CreateMul(Value *LHS, Value *RHS, const Twine &Name="", bool HasNUW=false, bool HasNSW=false)
Definition IRBuilder.h:1458
AtomicRMWInst * CreateAtomicRMW(AtomicRMWInst::BinOp Op, Value *Ptr, Value *Val, MaybeAlign Align, AtomicOrdering Ordering, SyncScope::ID SSID=SyncScope::System, bool Elementwise=false)
Definition IRBuilder.h:1987
This provides a uniform API for creating instructions and inserting them into a basic block: either a...
Definition IRBuilder.h:2990
const DebugLoc & getDebugLoc() const
Return the debug location for this node as a DebugLoc.
LLVM_ABI const Function * getFunction() const
Return the function this instruction belongs to.
This is an important class for using LLVM in a threaded context.
Definition LLVMContext.h:76
An instruction for reading from memory.
LLVM_ABI MDNode * createRange(const APInt &Lo, const APInt &Hi)
Return metadata describing the range [Lo, Hi).
Definition MDBuilder.cpp:92
Metadata node.
Definition Metadata.h:1087
static MDTuple * get(LLVMContext &Context, ArrayRef< Metadata * > MDs)
Definition Metadata.h:1588
This class implements a map that also provides access to all stored values in a deterministic order.
Definition MapVector.h:39
bool empty() const
Definition MapVector.h:80
bool contains(const KeyT &Key) const
Definition MapVector.h:149
std::pair< iterator, bool > insert(const std::pair< KeyT, ValueT > &KV)
Definition MapVector.h:127
Root of the metadata hierarchy.
Definition Metadata.h:65
ModulePass class - This class is used to implement unstructured interprocedural optimizations and ana...
Definition Pass.h:253
A Module instance is used to store all the information related to an LLVM module.
Definition Module.h:68
A container for an operand bundle being viewed as a set of values rather than a set of uses.
void addIncoming(Value *V, BasicBlock *BB)
Add an incoming value to the end of the PHI list.
static LLVM_ABI PoisonValue * get(Type *T)
Static factory methods - Return an 'poison' object of the specified type.
A set of analyses that are preserved following a run of a transformation pass.
Definition Analysis.h:112
static PreservedAnalyses all()
Construct a special preserved set that preserves all passes.
Definition Analysis.h:118
PreservedAnalyses & preserve()
Mark an analysis as preserved.
Definition Analysis.h:133
Return a value (possibly void), from a function.
A vector that has set insertion semantics.
Definition SetVector.h:57
size_type size() const
Determine the number of elements in the SetVector.
Definition SetVector.h:103
iterator end()
Get an iterator to the end of the SetVector.
Definition SetVector.h:118
void clear()
Completely clear the SetVector.
Definition SetVector.h:273
bool empty() const
Determine if the SetVector is empty or not.
Definition SetVector.h:100
iterator begin()
Get an iterator to the beginning of the SetVector.
Definition SetVector.h:112
bool insert(const value_type &X)
Insert a new element into the SetVector.
Definition SetVector.h:157
std::pair< iterator, bool > insert(PtrType Ptr)
Inserts Ptr if and only if there is no element in the container equal to Ptr.
SmallPtrSet - This class implements a set which is optimized for holding SmallSize or less elements.
SmallString - A SmallString is just a SmallVector with methods and accessors that make it work better...
Definition SmallString.h:28
void push_back(const T &Elt)
This is a 'vector' (really, a variable-sized array), optimized for the case when the array is small.
An instruction for storing to memory.
Represent a constant reference to a string, i.e.
Definition StringRef.h:56
Class to represent struct types.
static LLVM_ABI StructType * create(LLVMContext &Context, StringRef Name)
This creates an identified struct.
Definition Type.cpp:657
Target-Independent Code Generator Pass Configuration Options.
The instances of the Type class are immutable: once they are created, they are never changed.
Definition Type.h:46
A Use represents the edge between a Value definition and its users.
Definition Use.h:35
LLVM Value Representation.
Definition Value.h:75
Type * getType() const
All values are typed, get the type of this value.
Definition Value.h:257
LLVM_ABI void replaceAllUsesWith(Value *V)
Change all uses of this to point to a new Value.
Definition Value.cpp:548
iterator_range< user_iterator > users()
Definition Value.h:428
LLVM_ABI void printAsOperand(raw_ostream &O, bool PrintType=true, const Module *M=nullptr) const
Print the name of this Value out to the specified raw_ostream.
bool use_empty() const
Definition Value.h:348
LLVM_ABI bool replaceUsesWithIf(Value *New, llvm::function_ref< bool(Use &U)> ShouldReplace)
Go through the uses list for this definition and make each use point to "V" if the callback ShouldRep...
Definition Value.cpp:556
LLVM_ABI StringRef getName() const
Return a constant reference to the value's name.
Definition Value.cpp:314
static LLVM_ABI VectorType * get(Type *ElementType, ElementCount EC)
This static method is the primary way to construct an VectorType.
An efficient, type-erasing, non-owning reference to a callable.
A raw_ostream that writes to an std::string.
A raw_ostream that writes to an SmallVector or SmallString.
StringRef str() const
Return a StringRef for the vector contents.
@ LOCAL_ADDRESS
Address space for local memory.
@ FLAT_ADDRESS
Address space for flat memory.
@ GLOBAL_ADDRESS
Address space for global memory (RAT0, VTX0).
GVUsesInfoTy getTransitiveUsesOfLDSForLowering(const CallGraph &CG, Module &M)
Collects all uses of LDS Global Variables in M using getUsesOfGVByFunction, with isLDSVariableToLower...
void getInterestingMemoryOperands(Module &M, Instruction *I, SmallVectorImpl< InterestingMemoryOperand > &Interesting)
Get all the memory operands from the instruction that needs to be instrumented.
bool isDynamicLDS(const GlobalVariable &GV)
unsigned getAMDHSACodeObjectVersion(const Module &M)
void removeFnAttrFromReachable(CallGraph &CG, Function *KernelRoot, ArrayRef< StringRef > FnAttrs)
Strip FnAttr attribute from any functions where we may have introduced its use.
bool eliminateGVConstantExprUsesFromAllInstructions(Module &M, function_ref< bool(const GlobalVariable &)> Filter)
Iterates over all GlobalVariables in M, and whenever Filter returns true, replace all constant users ...
LLVM_READNONE constexpr bool isKernel(CallingConv::ID CC)
MapVector< Function *, DenseSet< GlobalVariable * > > FunctionVariableMap
bool isLDSVariableToLower(const GlobalVariable &GV)
Align getAlign(const DataLayout &DL, const GlobalVariable *GV)
void instrumentAddress(Module &M, IRBuilder<> &IRB, Instruction *OrigIns, Instruction *InsertBefore, Value *Addr, Align Alignment, TypeSize TypeStoreSize, bool IsWrite, Value *SizeArgument, bool UseCalls, bool Recover, int AsanScale, int AsanOffset)
Instrument the memory operand Addr.
uint64_t getRedzoneSizeForGlobal(int AsanScale, uint64_t SizeInBytes)
Given SizeInBytes of the Value to be instrunmented, Returns the redzone size corresponding to it.
LLVM_ABI Function * getOrInsertDeclaration(Module *M, ID id, ArrayRef< Type * > OverloadTys={})
Look up the Function declaration of the intrinsic id in the Module M.
This is an optimization pass for GlobalISel generic memory operations.
ModulePass * createAMDGPUSwLowerLDSLegacyPass()
@ Offset
Definition DWP.cpp:577
auto find(R &&Range, const T &Val)
Provide wrappers to std::find which take ranges instead of having to pass begin/end explicitly.
Definition STLExtras.h:1781
decltype(auto) dyn_cast(const From &Val)
dyn_cast<X> - Return the argument parameter cast to the specified type.
Definition Casting.h:643
constexpr from_range_t from_range
void append_range(Container &C, Range &&R)
Wrapper function to append range R to container C.
Definition STLExtras.h:2229
iterator_range< early_inc_iterator_impl< detail::IterOfRange< RangeT > > > make_early_inc_range(RangeT &&Range)
Make a range that does early increment to allow mutation of the underlying range without disrupting i...
Definition STLExtras.h:649
InnerAnalysisManagerProxy< FunctionAnalysisManager, Module > FunctionAnalysisManagerModuleProxy
Provide the FunctionAnalysisManager to Module proxy.
bool any_of(R &&range, UnaryPredicate P)
Provide wrappers to std::any_of which take ranges instead of having to pass begin/end explicitly.
Definition STLExtras.h:1762
char & AMDGPUSwLowerLDSLegacyPassID
void sort(IteratorTy Start, IteratorTy End)
Definition STLExtras.h:1652
LLVM_ABI raw_ostream & dbgs()
dbgs() - This returns a reference to a raw_ostream for debugging messages.
Definition Debug.cpp:209
LLVM_ABI void report_fatal_error(Error Err, bool gen_crash_diag=true)
Definition Error.cpp:163
constexpr uint64_t alignTo(uint64_t Size, Align A)
Returns a multiple of A needed to store Size bytes.
Definition Alignment.h:149
bool isa(const From &Val)
isa<X> - Return true if the parameter to the template is an instance of one of the template type argu...
Definition Casting.h:547
LLVM_ABI void getAddressSanitizerParams(const LLVMContext &Ctx, const Triple &TargetTriple, int LongSize, bool IsKasan, uint64_t *ShadowBase, int *MappingScale, bool *OrShadowOffset)
decltype(auto) cast(const From &Val)
cast<X> - Return the argument parameter cast to the specified type.
Definition Casting.h:559
bool is_contained(R &&Range, const E &Element)
Returns true if Element is found in Range.
Definition STLExtras.h:1963
AnalysisManager< Module > ModuleAnalysisManager
Convenience typedef for the Module analysis manager.
Definition MIRParser.h:39
LLVM_ABI void reportFatalUsageError(Error Err)
Report a fatal error that does not indicate a bug in LLVM.
Definition Error.cpp:177
PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM)
FunctionVariableMap DirectAccess
FunctionVariableMap IndirectAccess
This struct is a compact representation of a valid (non-zero power of two) alignment.
Definition Alignment.h:39
constexpr uint64_t value() const
This is a hole in the type system and should not be abused.
Definition Alignment.h:82
Align valueOrOne() const
For convenience, returns a valid alignment or 1 if undefined.
Definition Alignment.h:135