LLVM 24.0.0git
AArch64FrameLowering.cpp
Go to the documentation of this file.
1//===- AArch64FrameLowering.cpp - AArch64 Frame Lowering -------*- C++ -*-====//
2//
3// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
4// See https://llvm.org/LICENSE.txt for license information.
5// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
6//
7//===----------------------------------------------------------------------===//
8//
9// This file contains the AArch64 implementation of TargetFrameLowering class.
10//
11// On AArch64, stack frames are structured as follows:
12//
13// The stack grows downward.
14//
15// All of the individual frame areas on the frame below are optional, i.e. it's
16// possible to create a function so that the particular area isn't present
17// in the frame.
18//
19// At function entry, the "frame" looks as follows:
20//
21// | | Higher address
22// |-----------------------------------|
23// | |
24// | arguments passed on the stack |
25// | |
26// |-----------------------------------| <- sp
27// | | Lower address
28//
29//
30// After the prologue has run, the frame has the following general structure.
31// Note that this doesn't depict the case where a red-zone is used. Also,
32// technically the last frame area (VLAs) doesn't get created until in the
33// main function body, after the prologue is run. However, it's depicted here
34// for completeness.
35//
36// | | Higher address
37// |-----------------------------------|
38// | |
39// | arguments passed on the stack |
40// | |
41// |-----------------------------------|
42// | |
43// | (Win64 only) varargs from reg |
44// | |
45// |-----------------------------------|
46// | |
47// | (Win64 only) callee-saved SVE reg |
48// | |
49// |-----------------------------------|
50// | |
51// | callee-saved gpr registers | <--.
52// | | | On Darwin platforms these
53// |- - - - - - - - - - - - - - - - - -| | callee saves are swapped,
54// | prev_lr | | (frame record first)
55// | prev_fp | <--'
56// | async context if needed |
57// | (a.k.a. "frame record") |
58// |-----------------------------------| <- fp(=x29)
59// Default SVE stack layout Split SVE objects
60// (aarch64-split-sve-objects=false) (aarch64-split-sve-objects=true)
61// |-----------------------------------| |-----------------------------------|
62// | <hazard padding> | | callee-saved PPR registers |
63// |-----------------------------------| |-----------------------------------|
64// | | | PPR stack objects |
65// | callee-saved fp/simd/SVE regs | |-----------------------------------|
66// | | | <hazard padding> |
67// |-----------------------------------| |-----------------------------------|
68// | | | callee-saved ZPR/FPR registers |
69// | SVE stack objects | |-----------------------------------|
70// | | | ZPR stack objects |
71// |-----------------------------------| |-----------------------------------|
72// ^ NB: FPR CSRs are promoted to ZPRs
73// |-----------------------------------|
74// |.empty.space.to.make.part.below....|
75// |.aligned.in.case.it.needs.more.than| (size of this area is unknown at
76// |.the.standard.16-byte.alignment....| compile time; if present)
77// |-----------------------------------|
78// | local variables of fixed size |
79// | including spill slots |
80// | <FPR> |
81// | <hazard padding> |
82// | <GPR> |
83// |-----------------------------------| <- bp(not defined by ABI,
84// |.variable-sized.local.variables....| LLVM chooses X19)
85// |.(VLAs)............................| (size of this area is unknown at
86// |...................................| compile time)
87// |-----------------------------------| <- sp
88// | | Lower address
89//
90//
91// To access the data in a frame, at-compile time, a constant offset must be
92// computable from one of the pointers (fp, bp, sp) to access it. The size
93// of the areas with a dotted background cannot be computed at compile-time
94// if they are present, making it required to have all three of fp, bp and
95// sp to be set up to be able to access all contents in the frame areas,
96// assuming all of the frame areas are non-empty.
97//
98// For most functions, some of the frame areas are empty. For those functions,
99// it may not be necessary to set up fp or bp:
100// * A base pointer is definitely needed when there are both VLAs and local
101// variables with more-than-default alignment requirements.
102// * A frame pointer is definitely needed when there are local variables with
103// more-than-default alignment requirements.
104//
105// For Darwin platforms the frame-record (fp, lr) is stored at the top of the
106// callee-saved area, since the unwind encoding does not allow for encoding
107// this dynamically and existing tools depend on this layout. For other
108// platforms, the frame-record is stored at the bottom of the (gpr) callee-saved
109// area to allow SVE stack objects (allocated directly below the callee-saves,
110// if available) to be accessed directly from the framepointer.
111// The SVE spill/fill instructions have VL-scaled addressing modes such
112// as:
113// ldr z8, [fp, #-7 mul vl]
114// For SVE the size of the vector length (VL) is not known at compile-time, so
115// '#-7 mul vl' is an offset that can only be evaluated at runtime. With this
116// layout, we don't need to add an unscaled offset to the framepointer before
117// accessing the SVE object in the frame.
118//
119// In some cases when a base pointer is not strictly needed, it is generated
120// anyway when offsets from the frame pointer to access local variables become
121// so large that the offset can't be encoded in the immediate fields of loads
122// or stores.
123//
124// Outgoing function arguments must be at the bottom of the stack frame when
125// calling another function. If we do not have variable-sized stack objects, we
126// can allocate a "reserved call frame" area at the bottom of the local
127// variable area, large enough for all outgoing calls. If we do have VLAs, then
128// the stack pointer must be decremented and incremented around each call to
129// make space for the arguments below the VLAs.
130//
131// FIXME: also explain the redzone concept.
132//
133// About stack hazards: Under some SME contexts, a coprocessor with its own
134// separate cache can used for FP operations. This can create hazards if the CPU
135// and the SME unit try to access the same area of memory, including if the
136// access is to an area of the stack. To try to alleviate this we attempt to
137// introduce extra padding into the stack frame between FP and GPR accesses,
138// controlled by the aarch64-stack-hazard-size option. Without changing the
139// layout of the stack frame in the diagram above, a stack object of size
140// aarch64-stack-hazard-size is added between GPR and FPR CSRs. Another is added
141// to the stack objects section, and stack objects are sorted so that FPR >
142// Hazard padding slot > GPRs (where possible). Unfortunately some things are
143// not handled well (VLA area, arguments on the stack, objects with both GPR and
144// FPR accesses), but if those are controlled by the user then the entire stack
145// frame becomes GPR at the start/end with FPR in the middle, surrounded by
146// Hazard padding.
147//
148// An example of the prologue:
149//
150// .globl __foo
151// .align 2
152// __foo:
153// Ltmp0:
154// .cfi_startproc
155// .cfi_personality 155, ___gxx_personality_v0
156// Leh_func_begin:
157// .cfi_lsda 16, Lexception33
158//
159// stp xa,bx, [sp, -#offset]!
160// ...
161// stp x28, x27, [sp, #offset-32]
162// stp fp, lr, [sp, #offset-16]
163// add fp, sp, #offset - 16
164// sub sp, sp, #1360
165//
166// The Stack:
167// +-------------------------------------------+
168// 10000 | ........ | ........ | ........ | ........ |
169// 10004 | ........ | ........ | ........ | ........ |
170// +-------------------------------------------+
171// 10008 | ........ | ........ | ........ | ........ |
172// 1000c | ........ | ........ | ........ | ........ |
173// +===========================================+
174// 10010 | X28 Register |
175// 10014 | X28 Register |
176// +-------------------------------------------+
177// 10018 | X27 Register |
178// 1001c | X27 Register |
179// +===========================================+
180// 10020 | Frame Pointer |
181// 10024 | Frame Pointer |
182// +-------------------------------------------+
183// 10028 | Link Register |
184// 1002c | Link Register |
185// +===========================================+
186// 10030 | ........ | ........ | ........ | ........ |
187// 10034 | ........ | ........ | ........ | ........ |
188// +-------------------------------------------+
189// 10038 | ........ | ........ | ........ | ........ |
190// 1003c | ........ | ........ | ........ | ........ |
191// +-------------------------------------------+
192//
193// [sp] = 10030 :: >>initial value<<
194// sp = 10020 :: stp fp, lr, [sp, #-16]!
195// fp = sp == 10020 :: mov fp, sp
196// [sp] == 10020 :: stp x28, x27, [sp, #-16]!
197// sp == 10010 :: >>final value<<
198//
199// The frame pointer (w29) points to address 10020. If we use an offset of
200// '16' from 'w29', we get the CFI offsets of -8 for w30, -16 for w29, -24
201// for w27, and -32 for w28:
202//
203// Ltmp1:
204// .cfi_def_cfa w29, 16
205// Ltmp2:
206// .cfi_offset w30, -8
207// Ltmp3:
208// .cfi_offset w29, -16
209// Ltmp4:
210// .cfi_offset w27, -24
211// Ltmp5:
212// .cfi_offset w28, -32
213//
214//===----------------------------------------------------------------------===//
215
216#include "AArch64FrameLowering.h"
217#include "AArch64InstrInfo.h"
220#include "AArch64RegisterInfo.h"
221#include "AArch64SMEAttributes.h"
222#include "AArch64Subtarget.h"
225#include "llvm/ADT/ScopeExit.h"
226#include "llvm/ADT/SmallVector.h"
244#include "llvm/IR/Attributes.h"
245#include "llvm/IR/CallingConv.h"
246#include "llvm/IR/DataLayout.h"
247#include "llvm/IR/DebugLoc.h"
248#include "llvm/IR/Function.h"
249#include "llvm/IR/Module.h"
250#include "llvm/MC/MCAsmInfo.h"
251#include "llvm/MC/MCDwarf.h"
252#include "llvm/Support/Debug.h"
258#include <cassert>
259#include <cstdint>
260#include <iterator>
261#include <optional>
262#include <vector>
263
264using namespace llvm;
265
266#define DEBUG_TYPE "frame-info"
267
268int64_t
270 MachineBasicBlock &MBB) const {
271 MachineBasicBlock::iterator MBBI = MBB.getLastNonDebugInstr();
273 bool IsTailCallReturn = (MBB.end() != MBBI)
275 : false;
276
277 int64_t ArgumentPopSize = 0;
278 if (IsTailCallReturn) {
279 MachineOperand &StackAdjust = MBBI->getOperand(1);
280
281 // For a tail-call in a callee-pops-arguments environment, some or all of
282 // the stack may actually be in use for the call's arguments, this is
283 // calculated during LowerCall and consumed here...
284 ArgumentPopSize = StackAdjust.getImm();
285 } else {
286 // ... otherwise the amount to pop is *all* of the argument space,
287 // conveniently stored in the MachineFunctionInfo by
288 // LowerFormalArguments. This will, of course, be zero for the C calling
289 // convention.
290 ArgumentPopSize = AFI->getArgumentStackToRestore();
291 }
292
293 return ArgumentPopSize;
294}
295
298 if (!MF.needsWinCFI() ||
300 return false;
301
302 // Funclets don't tail call, and are not what the unwind info is about.
303 MachineBasicBlock::iterator LastI = MBB.getLastNonDebugInstr();
304 if (LastI != MBB.end() && (LastI->getOpcode() == AArch64::CATCHRET ||
305 LastI->getOpcode() == AArch64::CLEANUPRET))
306 return false;
307
308 return getArgumentStackToRestore(MF, MBB) != 0;
309}
310
312 MachineFunction &MF);
313
314enum class AssignObjectOffsets { No, Yes };
315/// Process all the SVE stack objects and the SVE stack size and offsets for
316/// each object. If AssignOffsets is "Yes", the offsets get assigned (and SVE
317/// stack sizes set). Returns the size of the SVE stack.
319 AssignObjectOffsets AssignOffsets);
320
321static unsigned getStackHazardSize(const MachineFunction &MF) {
322 return MF.getSubtarget<AArch64Subtarget>().getStreamingHazardSize();
323}
324
330
333 // With split SVE objects, the hazard padding is added to the PPR region,
334 // which places it between the [GPR, PPR] area and the [ZPR, FPR] area. This
335 // avoids hazards between both GPRs and FPRs and ZPRs and PPRs.
338 : 0,
339 AFI->getStackSizePPR());
340}
341
342// Conservatively, returns true if the function is likely to have SVE vectors
343// on the stack. This function is safe to be called before callee-saves or
344// object offsets have been determined.
346 const MachineFunction &MF) {
347 auto *AFI = MF.getInfo<AArch64FunctionInfo>();
348 if (AFI->isSVECC())
349 return true;
350
351 if (AFI->hasCalculatedStackSizeSVE())
352 return bool(AFL.getSVEStackSize(MF));
353
354 const MachineFrameInfo &MFI = MF.getFrameInfo();
355 for (int FI = MFI.getObjectIndexBegin(); FI < MFI.getObjectIndexEnd(); FI++) {
356 if (MFI.hasScalableStackID(FI))
357 return true;
358 }
359
360 return false;
361}
362
363static bool isTargetWindows(const MachineFunction &MF) {
364 // Note: UEFI follows the Windows ARM64 ABI and uses Windows CFI. The value
365 // returned here must align with the predicate used for returning the list of
366 // callee saved regs in AArch64RegisterInfo::getCalleeSavedRegs(), so that we
367 // use invalidateWindowsRegisterPairing() where appropriate.
368 return MF.getSubtarget<AArch64Subtarget>().isTargetWindowsOrUEFI();
369}
370
372 const MachineFunction &MF) const {
373 auto *AFI = MF.getInfo<AArch64FunctionInfo>();
374 return isTargetWindows(MF) && AFI->getSVECalleeSavedStackSize();
375}
376
377/// Returns true if a homogeneous prolog or epilog code can be emitted
378/// for the size optimization. If possible, a frame helper call is injected.
379/// When Exit block is given, this check is for epilog.
380bool AArch64FrameLowering::homogeneousPrologEpilog(
381 MachineFunction &MF, MachineBasicBlock *Exit) const {
382 if (!MF.getFunction().hasMinSize())
383 return false;
384 const AArch64Options &CLOpts =
385 MF.getSubtarget<AArch64Subtarget>().getCLOpts();
386 if (!CLOpts.homogeneous_prolog_epilog)
387 return false;
388 if (CLOpts.redzone)
389 return false;
390
391 // TODO: Window is supported yet.
392 if (isTargetWindows(MF))
393 return false;
394
395 // TODO: SVE is not supported yet.
396 if (isLikelyToHaveSVEStack(*this, MF))
397 return false;
398
399 // Bail on stack adjustment needed on return for simplicity.
400 const MachineFrameInfo &MFI = MF.getFrameInfo();
401 const TargetRegisterInfo *RegInfo = MF.getSubtarget().getRegisterInfo();
402 if (MFI.hasVarSizedObjects() || RegInfo->hasStackRealignment(MF))
403 return false;
404 if (Exit && getArgumentStackToRestore(MF, *Exit))
405 return false;
406
407 auto *AFI = MF.getInfo<AArch64FunctionInfo>();
408 if (AFI->hasSwiftAsyncContext() || AFI->hasStreamingModeChanges())
409 return false;
410
411 // If there are an odd number of GPRs before LR and FP in the CSRs list,
412 // they will not be paired into one RegGroupInfo, which is incompatible with
413 // the assumption made by the homogeneous prolog epilog pass.
414 const MCPhysReg *CSRegs = MF.getRegInfo().getCalleeSavedRegs();
415 unsigned NumGPRs = 0;
416 for (unsigned I = 0; CSRegs[I]; ++I) {
417 Register Reg = CSRegs[I];
418 if (Reg == AArch64::LR) {
419 assert(CSRegs[I + 1] == AArch64::FP);
420 if (NumGPRs % 2 != 0)
421 return false;
422 break;
423 }
424 if (AArch64::GPR64RegClass.contains(Reg))
425 ++NumGPRs;
426 }
427
428 return true;
429}
430
431/// Returns true if CSRs should be paired.
432bool AArch64FrameLowering::producePairRegisters(MachineFunction &MF) const {
433 return produceCompactUnwindFrame(*this, MF) || homogeneousPrologEpilog(MF);
434}
435
436/// This is the biggest offset to the stack pointer we can encode in aarch64
437/// instructions (without using a separate calculation and a temp register).
438/// Note that the exception here are vector stores/loads which cannot encode any
439/// displacements (see estimateRSStackSizeLimit(), isAArch64FrameOffsetLegal()).
440static const unsigned DefaultSafeSPDisplacement = 255;
441
442/// Look at each instruction that references stack frames and return the stack
443/// size limit beyond which some of these instructions will require a scratch
444/// register during their expansion later.
446 // FIXME: For now, just conservatively guesstimate based on unscaled indexing
447 // range. We'll end up allocating an unnecessary spill slot a lot, but
448 // realistically that's not a big deal at this stage of the game.
449 for (MachineBasicBlock &MBB : MF) {
450 for (MachineInstr &MI : MBB) {
451 if (MI.isDebugInstr() || MI.isPseudo() ||
452 MI.getOpcode() == AArch64::ADDXri ||
453 MI.getOpcode() == AArch64::ADDSXri)
454 continue;
455
456 for (const MachineOperand &MO : MI.operands()) {
457 if (!MO.isFI())
458 continue;
459
461 if (isAArch64FrameOffsetLegal(MI, Offset, nullptr, nullptr, nullptr) ==
463 return 0;
464 }
465 }
466 }
468}
469
474
475unsigned
476AArch64FrameLowering::getFixedObjectSize(const MachineFunction &MF,
477 const AArch64FunctionInfo *AFI,
478 bool IsWin64, bool IsFunclet) const {
479 assert(AFI->getTailCallReservedStack() % 16 == 0 &&
480 "Tail call reserved stack must be aligned to 16 bytes");
481 if (!IsWin64) {
482 return AFI->getTailCallReservedStack();
483 } else if (IsFunclet) {
484 // A funclet has its own frame; the tail call reserve belongs to the parent
485 // function's incoming argument area.
486 return 0;
487 } else {
488 unsigned FixedObjectSize = AFI->getTailCallReservedStack();
489
490 // Var args are stored here in the primary function.
491 FixedObjectSize += AFI->getVarArgsGPRSize();
492
493 if (MF.hasEHFunclets()) {
494 // Catch objects are stored here in the primary function.
495 const MachineFrameInfo &MFI = MF.getFrameInfo();
496 const WinEHFuncInfo &EHInfo = *MF.getWinEHFuncInfo();
497 SmallSetVector<int, 8> CatchObjFrameIndices;
498 for (const WinEHTryBlockMapEntry &TBME : EHInfo.TryBlockMap) {
499 for (const WinEHHandlerType &H : TBME.HandlerArray) {
500 int FrameIndex = H.CatchObj.FrameIndex;
501 if ((FrameIndex != INT_MAX) &&
502 CatchObjFrameIndices.insert(FrameIndex)) {
503 FixedObjectSize = alignTo(FixedObjectSize,
504 MFI.getObjectAlign(FrameIndex).value()) +
505 MFI.getObjectSize(FrameIndex);
506 }
507 }
508 }
509 // To support EH funclets we allocate an UnwindHelp object
510 FixedObjectSize += 8;
511 }
512 return alignTo(FixedObjectSize, 16);
513 }
514}
515
517 if (!MF.getSubtarget<AArch64Subtarget>().getCLOpts().redzone)
518 return false;
519
520 // Don't use the red zone if the function explicitly asks us not to.
521 // This is typically used for kernel code.
523 const unsigned RedZoneSize =
524 Subtarget.getTargetLowering()->getRedZoneSize(MF.getFunction());
525 if (!RedZoneSize)
526 return false;
527
528 const MachineFrameInfo &MFI = MF.getFrameInfo();
530 uint64_t NumBytes = AFI->getLocalStackSize();
531
532 // If neither NEON or SVE are available, a COPY from one Q-reg to
533 // another requires a spill -> reload sequence. We can do that
534 // using a pre-decrementing store/post-decrementing load, but
535 // if we do so, we can't use the Red Zone.
536 bool LowerQRegCopyThroughMem = Subtarget.hasFPARMv8() &&
537 !Subtarget.isNeonAvailable() &&
538 !Subtarget.hasSVE();
539
540 return !(MFI.hasCalls() || hasFP(MF) || NumBytes > RedZoneSize ||
541 AFI->hasSVEStackSize() || LowerQRegCopyThroughMem);
542}
543
544/// hasFPImpl - Return true if the specified function should have a dedicated
545/// frame pointer register.
547 const MachineFrameInfo &MFI = MF.getFrameInfo();
548 const TargetRegisterInfo *RegInfo = MF.getSubtarget().getRegisterInfo();
550
551 // Win64 EH requires a frame pointer if funclets are present, as the locals
552 // are accessed off the frame pointer in both the parent function and the
553 // funclets.
554 if (MF.hasEHFunclets())
555 return true;
556
557 // Unwinding out of a callee that pops stack arguments we reserved in our
558 // prologue leaves SP higher than our unwind info expects, so on Windows we
559 // must be found via the frame pointer instead.
561 return true;
562
563 // When the stack guard is mixed with the frame pointer, a dedicated FP is
564 // required so the guard value remains stable in the presence of dynamic
565 // stack allocations (e.g. _alloca on MSVCRT).
566 if (MFI.hasStackProtectorIndex()) {
567 const auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
568 if (Subtarget.getTargetLowering()->useStackGuardMixFP())
569 return true;
570 }
571
572 // Retain behavior of always omitting the FP for leaf functions when possible.
574 return true;
575 if (MFI.hasVarSizedObjects() || MFI.isFrameAddressTaken() ||
576 MFI.hasStackMap() || MFI.hasPatchPoint() ||
577 RegInfo->hasStackRealignment(MF))
578 return true;
579
580 // If we:
581 //
582 // 1. Have streaming mode changes
583 // OR:
584 // 2. Have a streaming body with SVE stack objects
585 //
586 // Then the value of VG restored when unwinding to this function may not match
587 // the value of VG used to set up the stack.
588 //
589 // This is a problem as the CFA can be described with an expression of the
590 // form: CFA = SP + NumBytes + VG * NumScalableBytes.
591 //
592 // If the value of VG used in that expression does not match the value used to
593 // set up the stack, an incorrect address for the CFA will be computed, and
594 // unwinding will fail.
595 //
596 // We work around this issue by ensuring the frame-pointer can describe the
597 // CFA in either of these cases.
598 if (AFI.needsDwarfUnwindInfo(MF) &&
601 return true;
602 // With large callframes around we may need to use FP to access the scavenging
603 // emergency spillslot.
604 //
605 // Unfortunately some calls to hasFP() like machine verifier ->
606 // getReservedReg() -> hasFP in the middle of global isel are too early
607 // to know the max call frame size. Hopefully conservatively returning "true"
608 // in those cases is fine.
609 // DefaultSafeSPDisplacement is fine as we only emergency spill GP regs.
610 if (!MFI.isMaxCallFrameSizeComputed() ||
612 return true;
613
614 return false;
615}
616
617/// Should the Frame Pointer be reserved for the current function?
619 const Triple &TT = MF.getFunction().getParent()->getTargetTriple();
620
621 // These OSes require the frame chain is valid, even if the current frame does
622 // not use a frame pointer.
623 if (TT.isOSDarwin() || TT.isOSWindowsOrUEFI())
624 return true;
625
626 // If the function has a frame pointer, it is reserved.
627 if (hasFP(MF))
628 return true;
629
630 // Frontend has requested to preserve the frame pointer.
631 if (MF.framePointerIsReserved())
632 return true;
633
634 return false;
635}
636
637/// hasReservedCallFrame - Under normal circumstances, when a frame pointer is
638/// not required, we reserve argument space for call sites in the function
639/// immediately on entry to the current function. This eliminates the need for
640/// add/sub sp brackets around call sites. Returns true if the call frame is
641/// included as part of the stack frame.
643 const MachineFunction &MF) const {
644 // The stack probing code for the dynamically allocated outgoing arguments
645 // area assumes that the stack is probed at the top - either by the prologue
646 // code, which issues a probe if `hasVarSizedObjects` return true, or by the
647 // most recent variable-sized object allocation. Changing the condition here
648 // may need to be followed up by changes to the probe issuing logic.
649 return !MF.getFrameInfo().hasVarSizedObjects();
650}
651
655
657 const AArch64InstrInfo *TII = Subtarget.getInstrInfo();
658 const AArch64TargetLowering *TLI = Subtarget.getTargetLowering();
659 [[maybe_unused]] MachineFrameInfo &MFI = MF.getFrameInfo();
660 DebugLoc DL = I->getDebugLoc();
661 unsigned Opc = I->getOpcode();
662 bool IsDestroy = Opc == TII->getCallFrameDestroyOpcode();
663 uint64_t CalleePopAmount = IsDestroy ? I->getOperand(1).getImm() : 0;
664
665 if (!hasReservedCallFrame(MF)) {
666 int64_t Amount = I->getOperand(0).getImm();
667 Amount = alignTo(Amount, getStackAlign());
668 if (!IsDestroy)
669 Amount = -Amount;
670
671 // N.b. if CalleePopAmount is valid but zero (i.e. callee would pop, but it
672 // doesn't have to pop anything), then the first operand will be zero too so
673 // this adjustment is a no-op.
674 if (CalleePopAmount == 0) {
675 // FIXME: in-function stack adjustment for calls is limited to 24-bits
676 // because there's no guaranteed temporary register available.
677 //
678 // ADD/SUB (immediate) has only LSL #0 and LSL #12 available.
679 // 1) For offset <= 12-bit, we use LSL #0
680 // 2) For 12-bit <= offset <= 24-bit, we use two instructions. One uses
681 // LSL #0, and the other uses LSL #12.
682 //
683 // Most call frames will be allocated at the start of a function so
684 // this is OK, but it is a limitation that needs dealing with.
685 assert(Amount > -0xffffff && Amount < 0xffffff && "call frame too large");
686
687 if (TLI->hasInlineStackProbe(MF) &&
689 // When stack probing is enabled, the decrement of SP may need to be
690 // probed. We only need to do this if the call site needs 1024 bytes of
691 // space or more, because a region smaller than that is allowed to be
692 // unprobed at an ABI boundary. We rely on the fact that SP has been
693 // probed exactly at this point, either by the prologue or most recent
694 // dynamic allocation.
696 "non-reserved call frame without var sized objects?");
697 Register ScratchReg =
698 MF.getRegInfo().createVirtualRegister(&AArch64::GPR64RegClass);
699 inlineStackProbeFixed(I, ScratchReg, -Amount, StackOffset::get(0, 0));
700 } else {
701 emitFrameOffset(MBB, I, DL, AArch64::SP, AArch64::SP,
702 StackOffset::getFixed(Amount), TII);
703 }
704 }
705 } else if (CalleePopAmount != 0) {
706 // If the calling convention demands that the callee pops arguments from the
707 // stack, we want to add it back if we have a reserved call frame.
708 assert(CalleePopAmount < 0xffffff && "call frame too large");
709 emitFrameOffset(MBB, I, DL, AArch64::SP, AArch64::SP,
710 StackOffset::getFixed(-(int64_t)CalleePopAmount), TII);
711 }
712 return MBB.erase(I);
713}
714
716 MachineBasicBlock &MBB) const {
717
718 MachineFunction &MF = *MBB.getParent();
719 const auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
720 const auto &TRI = *Subtarget.getRegisterInfo();
721 const auto &MFI = *MF.getInfo<AArch64FunctionInfo>();
722
723 CFIInstBuilder CFIBuilder(MBB, MBB.begin(), MachineInstr::NoFlags);
724
725 // Reset the CFA to `SP + 0`.
726 CFIBuilder.buildDefCFA(AArch64::SP, 0);
727
728 // Flip the RA sign state.
729 if (MFI.shouldSignReturnAddress(MF)) {
730 if (MFI.branchProtectionPAuthLR()) {
731 CFIBuilder.buildNegateRAStateWithPC();
732 } else if (!MF.getTarget().getTargetTriple().isOSBinFormatMachO()) {
733 CFIBuilder.buildNegateRAState();
734 }
735 }
736
737 // Shadow call stack uses X18, reset it.
738 if (MFI.needsShadowCallStackPrologueEpilogue(MF))
739 CFIBuilder.buildSameValue(AArch64::X18);
740
741 // Emit .cfi_same_value for callee-saved registers.
742 const std::vector<CalleeSavedInfo> &CSI =
744 for (const auto &Info : CSI) {
745 MCRegister Reg = Info.getReg();
746 if (!TRI.regNeedsCFI(Reg, Reg))
747 continue;
748 CFIBuilder.buildSameValue(Reg);
749 }
750}
751
753 switch (Reg.id()) {
754 default:
755 // The called routine is expected to preserve r19-r28
756 // r29 and r30 are used as frame pointer and link register resp.
757 return 0;
758
759 // GPRs
760#define CASE(n) \
761 case AArch64::W##n: \
762 case AArch64::X##n: \
763 return AArch64::X##n
764 CASE(0);
765 CASE(1);
766 CASE(2);
767 CASE(3);
768 CASE(4);
769 CASE(5);
770 CASE(6);
771 CASE(7);
772 CASE(8);
773 CASE(9);
774 CASE(10);
775 CASE(11);
776 CASE(12);
777 CASE(13);
778 CASE(14);
779 CASE(15);
780 CASE(16);
781 CASE(17);
782 CASE(18);
783#undef CASE
784
785 // FPRs
786#define CASE(n) \
787 case AArch64::B##n: \
788 case AArch64::H##n: \
789 case AArch64::S##n: \
790 case AArch64::D##n: \
791 case AArch64::Q##n: \
792 return HasSVE ? AArch64::Z##n : AArch64::Q##n
793 CASE(0);
794 CASE(1);
795 CASE(2);
796 CASE(3);
797 CASE(4);
798 CASE(5);
799 CASE(6);
800 CASE(7);
801 CASE(8);
802 CASE(9);
803 CASE(10);
804 CASE(11);
805 CASE(12);
806 CASE(13);
807 CASE(14);
808 CASE(15);
809 CASE(16);
810 CASE(17);
811 CASE(18);
812 CASE(19);
813 CASE(20);
814 CASE(21);
815 CASE(22);
816 CASE(23);
817 CASE(24);
818 CASE(25);
819 CASE(26);
820 CASE(27);
821 CASE(28);
822 CASE(29);
823 CASE(30);
824 CASE(31);
825#undef CASE
826 }
827}
828
829void AArch64FrameLowering::emitZeroCallUsedRegs(BitVector RegsToZero,
831 RegScavenger *) const {
832 // Insertion point.
834
835 // Fake a debug loc.
836 DebugLoc DL;
837 if (MBBI != MBB.end())
838 DL = MBBI->getDebugLoc();
839
840 const MachineFunction &MF = *MBB.getParent();
841 const AArch64Subtarget &STI = MF.getSubtarget<AArch64Subtarget>();
842 const AArch64RegisterInfo &TRI = *STI.getRegisterInfo();
843
844 BitVector GPRsToZero(TRI.getNumRegs());
845 BitVector FPRsToZero(TRI.getNumRegs());
846 bool HasSVE = STI.isSVEorStreamingSVEAvailable();
847 // Without an FP unit (e.g. -mgeneral-regs-only) the FP/vector registers can't
848 // hold a value and there is no instruction to clear them, so leave them out.
849 bool HasFPR = STI.hasFPARMv8();
850 for (MCRegister Reg : RegsToZero.set_bits()) {
851 if (TRI.isGeneralPurposeRegister(MF, Reg)) {
852 // For GPRs, we only care to clear out the 64-bit register.
853 if (MCRegister XReg = getRegisterOrZero(Reg, HasSVE))
854 GPRsToZero.set(XReg);
855 } else if (HasFPR && AArch64InstrInfo::isFpOrNEON(Reg)) {
856 // For FPRs,
857 if (MCRegister XReg = getRegisterOrZero(Reg, HasSVE))
858 FPRsToZero.set(XReg);
859 }
860 }
861
862 const AArch64InstrInfo &TII = *STI.getInstrInfo();
863
864 // Zero out GPRs.
865 for (MCRegister Reg : GPRsToZero.set_bits())
866 TII.buildClearRegister(Reg, MBB, MBBI, DL);
867
868 // Zero out FP/vector registers.
869 for (MCRegister Reg : FPRsToZero.set_bits())
870 TII.buildClearRegister(Reg, MBB, MBBI, DL);
871
872 if (HasSVE) {
873 for (MCRegister PReg :
874 {AArch64::P0, AArch64::P1, AArch64::P2, AArch64::P3, AArch64::P4,
875 AArch64::P5, AArch64::P6, AArch64::P7, AArch64::P8, AArch64::P9,
876 AArch64::P10, AArch64::P11, AArch64::P12, AArch64::P13, AArch64::P14,
877 AArch64::P15}) {
878 if (RegsToZero[PReg])
879 BuildMI(MBB, MBBI, DL, TII.get(AArch64::PFALSE), PReg);
880 }
881 }
882}
883
884bool AArch64FrameLowering::windowsRequiresStackProbe(
885 const MachineFunction &MF, uint64_t StackSizeInBytes) const {
886 const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
887 const AArch64FunctionInfo &MFI = *MF.getInfo<AArch64FunctionInfo>();
888 // TODO: When implementing stack protectors, take that into account
889 // for the probe threshold.
890 return Subtarget.isTargetWindowsOrUEFI() && MFI.hasStackProbing() &&
891 StackSizeInBytes >= uint64_t(MFI.getStackProbeSize());
892}
893
895 const MachineBasicBlock &MBB) {
896 const MachineFunction *MF = MBB.getParent();
897 LiveRegs.addLiveIns(MBB);
898 // Mark callee saved registers as used so we will not choose them.
899 const MCPhysReg *CSRegs = MF->getRegInfo().getCalleeSavedRegs();
900 for (unsigned i = 0; CSRegs[i]; ++i)
901 LiveRegs.addReg(CSRegs[i]);
902}
903
905AArch64FrameLowering::findScratchNonCalleeSaveRegister(MachineBasicBlock *MBB,
906 bool HasCall) const {
908
909 // If MBB is an entry block, use X9 as the scratch register
910 // preserve_none functions may be using X9 to pass arguments,
911 // so prefer to pick an available register below.
912 if (&MF->front() == MBB &&
914 return AArch64::X9;
915
916 const AArch64Subtarget &Subtarget = MF->getSubtarget<AArch64Subtarget>();
917 const AArch64RegisterInfo &TRI = *Subtarget.getRegisterInfo();
918 LivePhysRegs LiveRegs(TRI);
919 getLiveRegsForEntryMBB(LiveRegs, *MBB);
920 if (HasCall) {
921 LiveRegs.addReg(AArch64::X16);
922 LiveRegs.addReg(AArch64::X17);
923 LiveRegs.addReg(AArch64::X18);
924 }
925
926 // Prefer X9 since it was historically used for the prologue scratch reg.
927 const MachineRegisterInfo &MRI = MF->getRegInfo();
928 if (LiveRegs.available(MRI, AArch64::X9))
929 return AArch64::X9;
930
931 for (unsigned Reg : AArch64::GPR64RegClass) {
932 if (LiveRegs.available(MRI, Reg))
933 return Reg;
934 }
935 return Register();
936}
937
939 const MachineBasicBlock &MBB) const {
940 const MachineFunction *MF = MBB.getParent();
942 return false;
943
944 MachineBasicBlock *TmpMBB = const_cast<MachineBasicBlock *>(&MBB);
946 const AArch64RegisterInfo *RegInfo = Subtarget.getRegisterInfo();
947 const AArch64TargetLowering *TLI = Subtarget.getTargetLowering();
949
950 if (AFI->hasSwiftAsyncContext()) {
951 const AArch64RegisterInfo &TRI = *Subtarget.getRegisterInfo();
952 const MachineRegisterInfo &MRI = MF->getRegInfo();
955 // The StoreSwiftAsyncContext clobbers X16 and X17. Make sure they are
956 // available.
957 if (!LiveRegs.available(MRI, AArch64::X16) ||
958 !LiveRegs.available(MRI, AArch64::X17))
959 return false;
960 }
961
962 // Certain stack probing sequences might clobber flags, then we can't use
963 // the block as a prologue if the flags register is a live-in.
965 MBB.isLiveIn(AArch64::NZCV))
966 return false;
967
968 if (RegInfo->hasStackRealignment(*MF) || TLI->hasInlineStackProbe(*MF))
969 if (!findScratchNonCalleeSaveRegister(TmpMBB).isValid())
970 return false;
971
972 // May need a scratch register (for return value) if require making a special
973 // call
974 if (requiresSaveVG(*MF) ||
975 windowsRequiresStackProbe(*MF, std::numeric_limits<uint64_t>::max()))
976 if (!findScratchNonCalleeSaveRegister(TmpMBB, true).isValid())
977 return false;
978
979 return true;
980}
981
983 const MachineFunction &MF) const {
984 const Function &F = MF.getFunction();
986
987 // The range-insertion pass currently runs only on Windows, not UEFI.
988 if (!Subtarget.isTargetWindows())
989 return false;
990
991 // Range insertion relies on SEH pseudos, which are emitted only when the
992 // function needs an unwind table entry.
993 if (!F.needsUnwindTableEntry())
994 return false;
995
996 // Splitting ranges does not yet preserve language-specific handler data.
997 if (F.hasPersonalityFn())
998 return false;
999
1000 // Funclets have independent frames; range insertion assumes a single frame.
1001 if (MF.hasEHFunclets())
1002 return false;
1003
1004 // Naked functions do not have a compiler-generated prologue to shrink-wrap.
1005 if (F.hasFnAttribute(Attribute::Naked))
1006 return false;
1007
1008 // Signed return addresses need authentication-state handling across ranges
1009 // that is not yet implemented.
1012 return false;
1013
1014 return true;
1015}
1016
1017bool AArch64FrameLowering::shouldSignReturnAddressEverywhere(
1018 const MachineFunction &MF) const {
1019 // FIXME: With WinCFI, extra care should be taken to place SEH_PACSignLR
1020 // and SEH_EpilogEnd instructions in the correct order.
1021 if (MF.usesWindowsUnwindInfo())
1022 return false;
1025}
1026
1027// Given a load or a store instruction, generate an appropriate unwinding SEH
1028// code on Windows.
1030AArch64FrameLowering::insertSEH(MachineBasicBlock::iterator MBBI,
1031 const AArch64InstrInfo &TII,
1032 MachineInstr::MIFlag Flag) const {
1033 unsigned Opc = MBBI->getOpcode();
1034 MachineBasicBlock *MBB = MBBI->getParent();
1035 MachineFunction &MF = *MBB->getParent();
1036 DebugLoc DL = MBBI->getDebugLoc();
1037 unsigned ImmIdx = MBBI->getNumOperands() - 1;
1038 int Imm = MBBI->getOperand(ImmIdx).getImm();
1041 const AArch64RegisterInfo *RegInfo = Subtarget.getRegisterInfo();
1042
1043 switch (Opc) {
1044 default:
1045 report_fatal_error("No SEH Opcode for this instruction");
1046 case AArch64::STR_ZXI:
1047 case AArch64::LDR_ZXI: {
1048 unsigned Reg0 = RegInfo->getSEHRegNum(MBBI->getOperand(0).getReg());
1049 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveZReg))
1050 .addImm(Reg0)
1051 .addImm(Imm)
1052 .setMIFlag(Flag);
1053 break;
1054 }
1055 case AArch64::STR_PXI:
1056 case AArch64::LDR_PXI: {
1057 unsigned Reg0 = RegInfo->getSEHRegNum(MBBI->getOperand(0).getReg());
1058 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SavePReg))
1059 .addImm(Reg0)
1060 .addImm(Imm)
1061 .setMIFlag(Flag);
1062 break;
1063 }
1064 case AArch64::LDPDpost:
1065 Imm = -Imm;
1066 [[fallthrough]];
1067 case AArch64::STPDpre: {
1068 unsigned Reg0 = RegInfo->getSEHRegNum(MBBI->getOperand(1).getReg());
1069 unsigned Reg1 = RegInfo->getSEHRegNum(MBBI->getOperand(2).getReg());
1070 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveFRegP_X))
1071 .addImm(Reg0)
1072 .addImm(Reg1)
1073 .addImm(Imm * 8)
1074 .setMIFlag(Flag);
1075 break;
1076 }
1077 case AArch64::LDPXpost:
1078 Imm = -Imm;
1079 [[fallthrough]];
1080 case AArch64::STPXpre: {
1081 Register Reg0 = MBBI->getOperand(1).getReg();
1082 Register Reg1 = MBBI->getOperand(2).getReg();
1083 if (Reg0 == AArch64::FP && Reg1 == AArch64::LR)
1084 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveFPLR_X))
1085 .addImm(Imm * 8)
1086 .setMIFlag(Flag);
1087 else
1088 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveRegP_X))
1089 .addImm(RegInfo->getSEHRegNum(Reg0))
1090 .addImm(RegInfo->getSEHRegNum(Reg1))
1091 .addImm(Imm * 8)
1092 .setMIFlag(Flag);
1093 break;
1094 }
1095 case AArch64::LDRDpost:
1096 Imm = -Imm;
1097 [[fallthrough]];
1098 case AArch64::STRDpre: {
1099 unsigned Reg = RegInfo->getSEHRegNum(MBBI->getOperand(1).getReg());
1100 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveFReg_X))
1101 .addImm(Reg)
1102 .addImm(Imm)
1103 .setMIFlag(Flag);
1104 break;
1105 }
1106 case AArch64::LDRXpost:
1107 Imm = -Imm;
1108 [[fallthrough]];
1109 case AArch64::STRXpre: {
1110 unsigned Reg = RegInfo->getSEHRegNum(MBBI->getOperand(1).getReg());
1111 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveReg_X))
1112 .addImm(Reg)
1113 .addImm(Imm)
1114 .setMIFlag(Flag);
1115 break;
1116 }
1117 case AArch64::STPDi:
1118 case AArch64::LDPDi: {
1119 unsigned Reg0 = RegInfo->getSEHRegNum(MBBI->getOperand(0).getReg());
1120 unsigned Reg1 = RegInfo->getSEHRegNum(MBBI->getOperand(1).getReg());
1121 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveFRegP))
1122 .addImm(Reg0)
1123 .addImm(Reg1)
1124 .addImm(Imm * 8)
1125 .setMIFlag(Flag);
1126 break;
1127 }
1128 case AArch64::STPXi:
1129 case AArch64::LDPXi: {
1130 Register Reg0 = MBBI->getOperand(0).getReg();
1131 Register Reg1 = MBBI->getOperand(1).getReg();
1132
1133 int SEHReg0 = RegInfo->getSEHRegNum(Reg0);
1134 int SEHReg1 = RegInfo->getSEHRegNum(Reg1);
1135
1136 if (Reg0 == AArch64::FP && Reg1 == AArch64::LR)
1137 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveFPLR))
1138 .addImm(Imm * 8)
1139 .setMIFlag(Flag);
1140 else if (SEHReg0 >= 19 && SEHReg1 >= 19)
1141 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveRegP))
1142 .addImm(SEHReg0)
1143 .addImm(SEHReg1)
1144 .addImm(Imm * 8)
1145 .setMIFlag(Flag);
1146 else
1147 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveAnyRegIP))
1148 .addImm(SEHReg0)
1149 .addImm(SEHReg1)
1150 .addImm(Imm * 8)
1151 .setMIFlag(Flag);
1152 break;
1153 }
1154 case AArch64::STRXui:
1155 case AArch64::LDRXui: {
1156 int Reg = RegInfo->getSEHRegNum(MBBI->getOperand(0).getReg());
1157 if (Reg >= 19)
1158 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveReg))
1159 .addImm(Reg)
1160 .addImm(Imm * 8)
1161 .setMIFlag(Flag);
1162 else
1163 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveAnyRegI))
1164 .addImm(Reg)
1165 .addImm(Imm * 8)
1166 .setMIFlag(Flag);
1167 break;
1168 }
1169 case AArch64::STRDui:
1170 case AArch64::LDRDui: {
1171 unsigned Reg = RegInfo->getSEHRegNum(MBBI->getOperand(0).getReg());
1172 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveFReg))
1173 .addImm(Reg)
1174 .addImm(Imm * 8)
1175 .setMIFlag(Flag);
1176 break;
1177 }
1178 case AArch64::STRQui:
1179 case AArch64::LDRQui: {
1180 unsigned Reg = RegInfo->getSEHRegNum(MBBI->getOperand(0).getReg());
1181 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveAnyRegQ))
1182 .addImm(Reg)
1183 .addImm(Imm * 16)
1184 .setMIFlag(Flag);
1185 break;
1186 }
1187 case AArch64::STPQi:
1188 case AArch64::LDPQi: {
1189 unsigned Reg0 = RegInfo->getSEHRegNum(MBBI->getOperand(0).getReg());
1190 unsigned Reg1 = RegInfo->getSEHRegNum(MBBI->getOperand(1).getReg());
1191 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveAnyRegQP))
1192 .addImm(Reg0)
1193 .addImm(Reg1)
1194 .addImm(Imm * 16)
1195 .setMIFlag(Flag);
1196 break;
1197 }
1198 case AArch64::LDRQpost:
1199 Imm = -Imm;
1200 [[fallthrough]];
1201 case AArch64::STRQpre: {
1202 unsigned Reg = RegInfo->getSEHRegNum(MBBI->getOperand(1).getReg());
1203 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveAnyRegQX))
1204 .addImm(Reg)
1205 .addImm(Imm)
1206 .setMIFlag(Flag);
1207 break;
1208 }
1209 case AArch64::LDPQpost:
1210 Imm = -Imm;
1211 [[fallthrough]];
1212 case AArch64::STPQpre: {
1213 unsigned Reg0 = RegInfo->getSEHRegNum(MBBI->getOperand(1).getReg());
1214 unsigned Reg1 = RegInfo->getSEHRegNum(MBBI->getOperand(2).getReg());
1215 MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveAnyRegQPX))
1216 .addImm(Reg0)
1217 .addImm(Reg1)
1218 .addImm(Imm * 16)
1219 .setMIFlag(Flag);
1220 break;
1221 }
1222 }
1223 auto I = MBB->insertAfter(MBBI, MIB);
1224 return I;
1225}
1226
1229 if (!AFI->needsDwarfUnwindInfo(MF) || !AFI->hasStreamingModeChanges())
1230 return false;
1231 // For Darwin platforms we don't save VG for non-SVE functions, even if SME
1232 // is enabled with streaming mode changes.
1233 auto &ST = MF.getSubtarget<AArch64Subtarget>();
1234 if (ST.isTargetDarwin())
1235 return ST.hasSVE();
1236 return true;
1237}
1238
1240 MachineFunction &MF) const {
1242 const AArch64InstrInfo *TII = Subtarget.getInstrInfo();
1243
1244 auto EmitSignRA = [&](MachineBasicBlock &MBB) {
1245 DebugLoc DL; // Set debug location to unknown.
1247
1248 BuildMI(MBB, MBBI, DL, TII->get(AArch64::PAUTH_PROLOGUE))
1250 };
1251
1252 auto EmitAuthRA = [&](MachineBasicBlock &MBB) {
1253 DebugLoc DL;
1254 MachineBasicBlock::iterator MBBI = MBB.getFirstTerminator();
1255 if (MBBI != MBB.end())
1256 DL = MBBI->getDebugLoc();
1257
1258 TII->createPauthEpilogueInstr(MBB, DL);
1259 };
1260
1261 // This should be in sync with PEIImpl::calculateSaveRestoreBlocks.
1262 EmitSignRA(MF.front());
1263 for (MachineBasicBlock &MBB : MF) {
1264 if (MBB.isEHFuncletEntry())
1265 EmitSignRA(MBB);
1266 if (MBB.isReturnBlock())
1267 EmitAuthRA(MBB);
1268 }
1269}
1270
1272 MachineBasicBlock &MBB) const {
1273 AArch64PrologueEmitter PrologueEmitter(MF, MBB, *this);
1274 PrologueEmitter.emitPrologue();
1275}
1276
1278 MachineBasicBlock &MBB) const {
1279 AArch64EpilogueEmitter EpilogueEmitter(MF, MBB, *this);
1280 EpilogueEmitter.emitEpilogue();
1281}
1282
1285 MF.getInfo<AArch64FunctionInfo>()->needsDwarfUnwindInfo(MF);
1286}
1287
1289 return enableCFIFixup(MF) &&
1290 MF.getInfo<AArch64FunctionInfo>()->needsAsyncDwarfUnwindInfo(MF);
1291}
1292
1293/// getFrameIndexReference - Provide a base+offset reference to an FI slot for
1294/// debug info. It's the same as what we use for resolving the code-gen
1295/// references for now. FIXME: This can go wrong when references are
1296/// SP-relative and simple call frames aren't used.
1299 Register &FrameReg) const {
1301 MF, FI, FrameReg,
1302 /*PreferFP=*/
1303 MF.getFunction().hasFnAttribute(Attribute::SanitizeHWAddress) ||
1304 MF.getFunction().hasFnAttribute(Attribute::SanitizeMemTag),
1305 /*ForSimm=*/false);
1306}
1307
1310 int FI) const {
1311 // This function serves to provide a comparable offset from a single reference
1312 // point (the value of SP at function entry) that can be used for analysis,
1313 // e.g. the stack-frame-layout analysis pass. It is not guaranteed to be
1314 // correct for all objects in the presence of VLA-area objects or dynamic
1315 // stack re-alignment.
1316
1317 const auto &MFI = MF.getFrameInfo();
1318
1319 int64_t ObjectOffset = MFI.getObjectOffset(FI);
1320 StackOffset ZPRStackSize = getZPRStackSize(MF);
1321 StackOffset PPRStackSize = getPPRStackSize(MF);
1322 StackOffset SVEStackSize = ZPRStackSize + PPRStackSize;
1323
1324 // For VLA-area objects, just emit an offset at the end of the stack frame.
1325 // Whilst not quite correct, these objects do live at the end of the frame and
1326 // so it is more useful for analysis for the offset to reflect this.
1327 if (MFI.isVariableSizedObjectIndex(FI)) {
1328 return StackOffset::getFixed(-((int64_t)MFI.getStackSize())) - SVEStackSize;
1329 }
1330
1331 // This is correct in the absence of any SVE stack objects.
1332 if (!SVEStackSize)
1333 return StackOffset::getFixed(ObjectOffset - getOffsetOfLocalArea());
1334
1335 const auto *AFI = MF.getInfo<AArch64FunctionInfo>();
1336 bool FPAfterSVECalleeSaves = hasSVECalleeSavesAboveFrameRecord(MF);
1337 if (MFI.hasScalableStackID(FI)) {
1338 if (FPAfterSVECalleeSaves &&
1339 -ObjectOffset <= (int64_t)AFI->getSVECalleeSavedStackSize()) {
1340 assert(!AFI->hasSplitSVEObjects() &&
1341 "split-sve-objects not supported with FPAfterSVECalleeSaves");
1342 return StackOffset::getScalable(ObjectOffset);
1343 }
1344 StackOffset AccessOffset{};
1345 // The scalable vectors are below (lower address) the scalable predicates
1346 // with split SVE objects, so we must subtract the size of the predicates.
1347 if (AFI->hasSplitSVEObjects() &&
1348 MFI.getStackID(FI) == TargetStackID::ScalableVector)
1349 AccessOffset = -PPRStackSize;
1350 return AccessOffset +
1351 StackOffset::get(-((int64_t)AFI->getCalleeSavedStackSize()),
1352 ObjectOffset);
1353 }
1354
1355 bool IsFixed = MFI.isFixedObjectIndex(FI);
1356 bool IsCSR =
1357 !IsFixed && ObjectOffset >= -((int)AFI->getCalleeSavedStackSize(MFI));
1358
1359 StackOffset ScalableOffset = {};
1360 if (!IsFixed && !IsCSR) {
1361 ScalableOffset = -SVEStackSize;
1362 } else if (FPAfterSVECalleeSaves && IsCSR) {
1363 ScalableOffset =
1365 }
1366
1367 return StackOffset::getFixed(ObjectOffset) + ScalableOffset;
1368}
1369
1375
1376StackOffset AArch64FrameLowering::getFPOffset(const MachineFunction &MF,
1377 int64_t ObjectOffset) const {
1378 const auto *AFI = MF.getInfo<AArch64FunctionInfo>();
1379 const auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
1380 const Function &F = MF.getFunction();
1381 bool IsWin64 = Subtarget.isCallingConvWin64(F.getCallingConv(), F.isVarArg());
1382 unsigned FixedObject =
1383 getFixedObjectSize(MF, AFI, IsWin64, /*IsFunclet=*/false);
1384 int64_t CalleeSaveSize = AFI->getCalleeSavedStackSize(MF.getFrameInfo());
1385 int64_t FPAdjust =
1386 CalleeSaveSize - AFI->getCalleeSaveBaseToFrameRecordOffset();
1387 return StackOffset::getFixed(ObjectOffset + FixedObject + FPAdjust);
1388}
1389
1390StackOffset AArch64FrameLowering::getStackOffset(const MachineFunction &MF,
1391 int64_t ObjectOffset) const {
1392 const auto &MFI = MF.getFrameInfo();
1393 return StackOffset::getFixed(ObjectOffset + (int64_t)MFI.getStackSize());
1394}
1395
1396// TODO: This function currently does not work for scalable vectors.
1398 int FI) const {
1399 const AArch64RegisterInfo *RegInfo =
1400 MF.getSubtarget<AArch64Subtarget>().getRegisterInfo();
1401 int ObjectOffset = MF.getFrameInfo().getObjectOffset(FI);
1402 return RegInfo->getLocalAddressRegister(MF) == AArch64::FP
1403 ? getFPOffset(MF, ObjectOffset).getFixed()
1404 : getStackOffset(MF, ObjectOffset).getFixed();
1405}
1406
1408 const MachineFunction &MF, int FI, Register &FrameReg, bool PreferFP,
1409 bool ForSimm) const {
1410 const auto &MFI = MF.getFrameInfo();
1411 int64_t ObjectOffset = MFI.getObjectOffset(FI);
1412 bool isFixed = MFI.isFixedObjectIndex(FI);
1413 auto StackID = static_cast<TargetStackID::Value>(MFI.getStackID(FI));
1414 return resolveFrameOffsetReference(MF, ObjectOffset, isFixed, StackID,
1415 FrameReg, PreferFP, ForSimm);
1416}
1417
1419 const MachineFunction &MF, int64_t ObjectOffset, bool isFixed,
1420 TargetStackID::Value StackID, Register &FrameReg, bool PreferFP,
1421 bool ForSimm) const {
1422 const auto &MFI = MF.getFrameInfo();
1423 const auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
1424 const AArch64RegisterInfo *RegInfo = Subtarget.getRegisterInfo();
1425 const auto *AFI = MF.getInfo<AArch64FunctionInfo>();
1426
1427 int64_t FPOffset = getFPOffset(MF, ObjectOffset).getFixed();
1428 int64_t Offset = getStackOffset(MF, ObjectOffset).getFixed();
1429
1430 // The fixed object area sits above the callee-saved area and can grow
1431 // large enough to displace it; account for that displacement here so CSR
1432 // objects aren't misclassified as locals and addressed via the base
1433 // pointer.
1434 bool IsWin64 = Subtarget.isCallingConvWin64(MF.getFunction().getCallingConv(),
1435 MF.getFunction().isVarArg());
1436 const int64_t FixedObjectSize =
1437 getFixedObjectSize(MF, AFI, IsWin64, /*IsFunclet*/ false);
1438 bool isCSR =
1439 !isFixed && ObjectOffset >= -((int64_t)AFI->getCalleeSavedStackSize(MFI) +
1440 FixedObjectSize);
1441 bool isSVE = MFI.isScalableStackID(StackID);
1442
1443 StackOffset ZPRStackSize = getZPRStackSize(MF);
1444 StackOffset PPRStackSize = getPPRStackSize(MF);
1445 StackOffset SVEStackSize = ZPRStackSize + PPRStackSize;
1446
1447 // Use frame pointer to reference fixed objects. Use it for locals if
1448 // there are VLAs or a dynamically realigned SP (and thus the SP isn't
1449 // reliable as a base). Make sure useFPForScavengingIndex() does the
1450 // right thing for the emergency spill slot.
1451 bool UseFP = false;
1452 if (AFI->hasStackFrame() && !isSVE) {
1453 // We shouldn't prefer using the FP to access fixed-sized stack objects when
1454 // there are scalable (SVE) objects in between the FP and the fixed-sized
1455 // objects.
1456 PreferFP &= !SVEStackSize;
1457
1458 // Note: Keeping the following as multiple 'if' statements rather than
1459 // merging to a single expression for readability.
1460 //
1461 // Argument access should always use the FP.
1462 if (isFixed) {
1463 UseFP = hasFP(MF);
1464 } else if (isCSR && RegInfo->hasStackRealignment(MF)) {
1465 // References to the CSR area must use FP if we're re-aligning the stack
1466 // since the dynamically-sized alignment padding is between the SP/BP and
1467 // the CSR area.
1468 assert(hasFP(MF) && "Re-aligned stack must have frame pointer");
1469 UseFP = true;
1470 } else if (hasFP(MF) && !RegInfo->hasStackRealignment(MF)) {
1471 // If the FPOffset is negative and we're producing a signed immediate, we
1472 // have to keep in mind that the available offset range for negative
1473 // offsets is smaller than for positive ones. If an offset is available
1474 // via the FP and the SP, use whichever is closest.
1475 bool FPOffsetFits = !ForSimm || FPOffset >= -256;
1476 PreferFP |= Offset > -FPOffset && !SVEStackSize;
1477
1478 if (FPOffset >= 0) {
1479 // If the FPOffset is positive, that'll always be best, as the SP/BP
1480 // will be even further away.
1481 UseFP = true;
1482 } else if (MFI.hasVarSizedObjects()) {
1483 // If we have variable sized objects, we can use either FP or BP, as the
1484 // SP offset is unknown. We can use the base pointer if we have one and
1485 // FP is not preferred. If not, we're stuck with using FP.
1486 bool CanUseBP = RegInfo->hasBasePointer(MF);
1487 if (FPOffsetFits && CanUseBP) // Both are ok. Pick the best.
1488 UseFP = PreferFP;
1489 else if (!CanUseBP) // Can't use BP. Forced to use FP.
1490 UseFP = true;
1491 // else we can use BP and FP, but the offset from FP won't fit.
1492 // That will make us scavenge registers which we can probably avoid by
1493 // using BP. If it won't fit for BP either, we'll scavenge anyway.
1494 } else if (MF.hasEHFunclets() && !RegInfo->hasBasePointer(MF)) {
1495 // Funclets access the locals contained in the parent's stack frame
1496 // via the frame pointer, so we have to use the FP in the parent
1497 // function.
1498 (void) Subtarget;
1499 assert(Subtarget.isCallingConvWin64(MF.getFunction().getCallingConv(),
1500 MF.getFunction().isVarArg()) &&
1501 "Funclets should only be present on Win64");
1502 UseFP = true;
1503 } else {
1504 // We have the choice between FP and (SP or BP).
1505 if (FPOffsetFits && PreferFP) // If FP is the best fit, use it.
1506 UseFP = true;
1507 }
1508 }
1509 }
1510
1511 assert(
1512 ((isFixed || isCSR) || !RegInfo->hasStackRealignment(MF) || !UseFP) &&
1513 "In the presence of dynamic stack pointer realignment, "
1514 "non-argument/CSR objects cannot be accessed through the frame pointer");
1515
1516 bool FPAfterSVECalleeSaves = hasSVECalleeSavesAboveFrameRecord(MF);
1517
1518 if (isSVE) {
1519 StackOffset FPOffset = StackOffset::get(
1520 -AFI->getCalleeSaveBaseToFrameRecordOffset(), ObjectOffset);
1521 StackOffset SPOffset =
1522 SVEStackSize +
1523 StackOffset::get(MFI.getStackSize() - AFI->getCalleeSavedStackSize(),
1524 ObjectOffset);
1525
1526 // With split SVE objects the ObjectOffset is relative to the split area
1527 // (i.e. the PPR area or ZPR area respectively).
1528 if (AFI->hasSplitSVEObjects() && StackID == TargetStackID::ScalableVector) {
1529 // If we're accessing an SVE vector with split SVE objects...
1530 // - From the FP we need to move down past the PPR area:
1531 FPOffset -= PPRStackSize;
1532 // - From the SP we only need to move up to the ZPR area:
1533 SPOffset -= PPRStackSize;
1534 // Note: `SPOffset = SVEStackSize + ...`, so `-= PPRStackSize` results in
1535 // `SPOffset = ZPRStackSize + ...`.
1536 }
1537
1538 if (FPAfterSVECalleeSaves) {
1540 if (-ObjectOffset <= (int64_t)AFI->getSVECalleeSavedStackSize()) {
1543 }
1544 }
1545
1546 // Always use the FP for SVE spills if available and beneficial.
1547 if (hasFP(MF) && (SPOffset.getFixed() ||
1548 FPOffset.getScalable() < SPOffset.getScalable() ||
1549 RegInfo->hasStackRealignment(MF))) {
1550 FrameReg = RegInfo->getFrameRegister(MF);
1551 return FPOffset;
1552 }
1553 FrameReg = RegInfo->hasBasePointer(MF) ? RegInfo->getBaseRegister()
1554 : MCRegister(AArch64::SP);
1555
1556 return SPOffset;
1557 }
1558
1559 StackOffset SVEAreaOffset = {};
1560 if (FPAfterSVECalleeSaves) {
1561 // In this stack layout, the FP is in between the callee saves and other
1562 // SVE allocations.
1563 StackOffset SVECalleeSavedStack =
1565 if (UseFP) {
1566 if (isFixed)
1567 SVEAreaOffset = SVECalleeSavedStack;
1568 else if (!isCSR)
1569 SVEAreaOffset = SVECalleeSavedStack - SVEStackSize;
1570 } else {
1571 if (isFixed)
1572 SVEAreaOffset = SVEStackSize;
1573 else if (isCSR)
1574 SVEAreaOffset = SVEStackSize - SVECalleeSavedStack;
1575 }
1576 } else {
1577 if (UseFP && !(isFixed || isCSR))
1578 SVEAreaOffset = -SVEStackSize;
1579 if (!UseFP && (isFixed || isCSR))
1580 SVEAreaOffset = SVEStackSize;
1581 }
1582
1583 if (UseFP) {
1584 FrameReg = RegInfo->getFrameRegister(MF);
1585 return StackOffset::getFixed(FPOffset) + SVEAreaOffset;
1586 }
1587
1588 // Use the base pointer if we have one.
1589 if (RegInfo->hasBasePointer(MF))
1590 FrameReg = RegInfo->getBaseRegister();
1591 else {
1592 assert(!MFI.hasVarSizedObjects() &&
1593 "Can't use SP when we have var sized objects.");
1594 FrameReg = AArch64::SP;
1595 // If we're using the red zone for this function, the SP won't actually
1596 // be adjusted, so the offsets will be negative. They're also all
1597 // within range of the signed 9-bit immediate instructions.
1598 if (canUseRedZone(MF))
1599 Offset -= AFI->getLocalStackSize();
1600 }
1601
1602 return StackOffset::getFixed(Offset) + SVEAreaOffset;
1603}
1604
1606 // Do not set a kill flag on values that are also marked as live-in. This
1607 // happens with the @llvm-returnaddress intrinsic and with arguments passed in
1608 // callee saved registers.
1609 // Omitting the kill flags is conservatively correct even if the live-in
1610 // is not used after all.
1611 bool IsLiveIn = MF.getRegInfo().isLiveIn(Reg);
1612 return getKillRegState(!IsLiveIn);
1613}
1614
1616 MachineFunction &MF) {
1620 return Subtarget.isTargetMachO() &&
1621 !(Subtarget.getTargetLowering()->supportSwiftError() &&
1622 Attrs.hasAttrSomewhere(Attribute::SwiftError)) &&
1624 !AFL.requiresSaveVG(MF) && !AFI->isSVECC();
1625}
1626
1627static bool invalidateWindowsRegisterPairing(bool SpillExtendedVolatile,
1628 int PairOffset, unsigned Reg1,
1629 unsigned Reg2, bool NeedsWinCFI,
1630 const TargetRegisterInfo *TRI) {
1631 // If we are generating register pairs for a Windows function that requires
1632 // EH support, then pair consecutive registers only. There are no unwind
1633 // opcodes for saves/restores of non-consecutive register pairs.
1634 // The unwind opcodes are save_regp, save_regp_x, save_fregp, save_frepg_x,
1635 // save_lrpair.
1636 // https://docs.microsoft.com/en-us/cpp/build/arm64-exception-handling
1637
1638 if (Reg2 == AArch64::FP)
1639 return true;
1640 if (!NeedsWinCFI)
1641 return false;
1642
1643 // ARM64EC introduced `save_any_regp`, which expects 16-byte alignment.
1644 // This is handled by only allowing paired spills for registers spilled at
1645 // even positions (which should be 16-byte aligned, as other GPRs/FPRs are
1646 // 8-bytes). We carve out an exception for {FP,LR}, which does not require
1647 // 16-byte alignment in the uop representation.
1648 if (TRI->getEncodingValue(Reg2) == TRI->getEncodingValue(Reg1) + 1)
1649 return SpillExtendedVolatile
1650 ? !((Reg1 == AArch64::FP && Reg2 == AArch64::LR) ||
1651 (PairOffset % 16) == 0)
1652 : false;
1653
1654 // If pairing a GPR with LR, the pair can be described by the save_lrpair
1655 // opcode. The save_lrpair opcode requires the first register to be odd.
1656 if (Reg1 >= AArch64::X19 && Reg1 <= AArch64::X27 &&
1657 (Reg1 - AArch64::X19) % 2 == 0 && Reg2 == AArch64::LR)
1658 return false;
1659 return true;
1660}
1661
1662/// Returns true if Reg1 and Reg2 cannot be paired using a ldp/stp instruction.
1663/// WindowsCFI requires that only consecutive registers can be paired.
1664/// LR and FP need to be allocated together when the frame needs to save
1665/// the frame-record. This means any other register pairing with LR is invalid.
1666static bool invalidateRegisterPairing(bool SpillExtendedVolatile,
1667 int PairOffset, unsigned Reg1,
1668 unsigned Reg2, bool UsesWinAAPCS,
1669 bool NeedsWinCFI, bool NeedsFrameRecord,
1670 const TargetRegisterInfo *TRI) {
1671 if (UsesWinAAPCS)
1672 return invalidateWindowsRegisterPairing(SpillExtendedVolatile, PairOffset,
1673 Reg1, Reg2, NeedsWinCFI, TRI);
1674
1675 // If we need to store the frame record, don't pair any register
1676 // with LR other than FP.
1677 if (NeedsFrameRecord)
1678 return Reg2 == AArch64::LR;
1679
1680 return false;
1681}
1682
1683// Returns true if Offset (in bytes) is aligned to the instruction's scale and
1684// the scaled immediate is within the instruction's valid range.
1685static bool isValidMemOpOffset(const AArch64InstrInfo *TII, unsigned Opcode,
1686 int Offset) {
1687 int64_t MinOff, MaxOff;
1688 TypeSize ScaleValue(0U, false), Width(0U, false);
1689 if (!TII->getMemOpInfo(Opcode, ScaleValue, Width, MinOff, MaxOff))
1690 return false;
1691
1692 if (Offset % ScaleValue.getKnownMinValue() != 0)
1693 return false;
1694
1695 Offset /= ScaleValue.getKnownMinValue();
1696 return Offset >= MinOff && Offset <= MaxOff;
1697}
1698
1699namespace {
1700
1701struct RegGroupInfo {
1702 Register Reg1;
1703 Register Reg2;
1704 int FrameIdx;
1705 int Offset;
1706 enum RegType { GPR, FPR64, FPR128, PPR, ZPR, VG } Type;
1707 const TargetRegisterClass *RC;
1708
1709 RegGroupInfo() = default;
1710
1711 bool isPaired() const { return getNumRegs() == 2; }
1712
1713 bool isQuad() const { return getNumRegs() == 4; }
1714
1715 bool isGrouped() const { return isPaired() || isQuad(); }
1716
1717 unsigned getNumRegs() const {
1718 if (!Reg2.isValid())
1719 return 1;
1720 if (Type != ZPR)
1721 return 2;
1722 // ZPR groups are contiguous with Reg2 holding the lower register.
1723 unsigned NumRegs = (Reg1 - Reg2) + 1;
1724 assert(NumRegs == 2 || NumRegs == 4);
1725 return NumRegs;
1726 }
1727
1728 bool isScalable() const { return Type == PPR || Type == ZPR; }
1729
1730 SmallVector<Register, 4> getRegisters() const {
1731 if (isPaired())
1732 return {Reg1, Reg2};
1733 if (isQuad()) {
1734 assert(Reg1 - 3 == Reg2 && "Quad not contiguous");
1735 return {Reg1, Reg1 - 1, Reg1 - 2, Reg1 - 3};
1736 }
1737 return {Reg1};
1738 }
1739
1740 auto getFrameIndexes() const {
1741 return seq(FrameIdx, FrameIdx + int(getNumRegs()));
1742 }
1743
1744 Register getAsMultiVectorRegister() const {
1745 assert(Type == ZPR && isGrouped() && "Expected ZPR register group");
1746 return (isQuad() ? AArch64::Z0_Z1_Z2_Z3 : AArch64::Z0_Z1) +
1747 (Reg2 - AArch64::Z0);
1748 }
1749
1750 void dump(const TargetRegisterInfo *TRI) const {
1751 dbgs() << "(";
1752 interleaveComma(getRegisters(), dbgs(),
1753 [&](Register Reg) { dbgs() << printReg(Reg, TRI); });
1754 dbgs() << ") -> fi#(";
1755 interleaveComma(getFrameIndexes(), dbgs(),
1756 [&](int FrameIdx) { dbgs() << FrameIdx; });
1757 dbgs() << ")\n";
1758 }
1759};
1760
1761} // end anonymous namespace
1762
1764 for (unsigned PReg = AArch64::P8; PReg <= AArch64::P15; ++PReg) {
1765 if (SavedRegs.test(PReg)) {
1766 unsigned PNReg = PReg - AArch64::P0 + AArch64::PN0;
1767 return MCRegister(PNReg);
1768 }
1769 }
1770 return MCRegister();
1771}
1772
1773// The multivector LD/ST are available only for SME or SVE2p1 targets
1775 MachineFunction &MF) {
1776 if (Subtarget.getCLOpts().disable_multivector_spill_fill)
1777 return false;
1778
1779 SMEAttrs FuncAttrs = MF.getInfo<AArch64FunctionInfo>()->getSMEFnAttrs();
1780 bool IsLocallyStreaming =
1781 FuncAttrs.hasStreamingBody() && !FuncAttrs.hasStreamingInterface();
1782
1783 // Only when in streaming mode SME2 instructions can be safely used.
1784 // It is not safe to use SME2 instructions when in streaming compatible or
1785 // locally streaming mode.
1786 return Subtarget.hasSVE2p1() ||
1787 (Subtarget.hasSME2() &&
1788 (!IsLocallyStreaming && Subtarget.isStreaming()));
1789}
1790
1792 MachineFunction &MF,
1794 const TargetRegisterInfo *TRI,
1796 bool NeedsFrameRecord) {
1797
1798 if (CSI.empty())
1799 return;
1800
1801 const AArch64InstrInfo *TII =
1802 MF.getSubtarget<AArch64Subtarget>().getInstrInfo();
1803 bool IsWindows = isTargetWindows(MF);
1804 bool NeedsWinCFI = MF.needsWinCFI();
1806 unsigned StackHazardSize = getStackHazardSize(MF);
1807 MachineFrameInfo &MFI = MF.getFrameInfo();
1809 unsigned Count = CSI.size();
1810 (void)CC;
1811 // MachO's compact unwind format relies on all registers being stored in
1812 // pairs.
1813 assert((!produceCompactUnwindFrame(AFL, MF) ||
1816 (Count & 1) == 0) &&
1817 "Odd number of callee-saved regs to spill!");
1818 int ByteOffset = AFI->getCalleeSavedStackSize();
1819 int StackFillDir = -1;
1820 int RegInc = 1;
1821 unsigned FirstReg = 0;
1822 if (IsWindows) {
1823 // For WinCFI, fill the stack from the bottom up.
1824 ByteOffset = 0;
1825 StackFillDir = 1;
1826 // As the CSI array is reversed to match PrologEpilogInserter, iterate
1827 // backwards, to pair up registers starting from lower numbered registers.
1828 RegInc = -1;
1829 FirstReg = Count - 1;
1830 }
1831
1832 bool FPAfterSVECalleeSaves = AFL.hasSVECalleeSavesAboveFrameRecord(MF);
1833 // Windows AAPCS has x9-x15 as volatile registers, x16-x17 as intra-procedural
1834 // scratch, x18 as platform reserved. However, clang has extended calling
1835 // convensions such as preserve_most and preserve_all which treat these as
1836 // CSR. As such, the ARM64 unwind uOPs bias registers by 19. We use ARM64EC
1837 // uOPs which have separate restrictions. We need to check for that.
1838 //
1839 // NOTE: we currently do not account for the D registers as LLVM does not
1840 // support non-ABI compliant D register spills.
1841 bool SpillExtendedVolatile =
1842 IsWindows && llvm::any_of(CSI, [](const CalleeSavedInfo &CSI) {
1843 const auto &Reg = CSI.getReg();
1844 return Reg >= AArch64::X0 && Reg <= AArch64::X18;
1845 });
1846
1847 int ZPRByteOffset = 0;
1848 int PPRByteOffset = 0;
1849 bool SplitPPRs = AFI->hasSplitSVEObjects();
1850 if (SplitPPRs) {
1851 ZPRByteOffset = AFI->getZPRCalleeSavedStackSize();
1852 PPRByteOffset = AFI->getPPRCalleeSavedStackSize();
1853 } else if (!FPAfterSVECalleeSaves) {
1854 ZPRByteOffset =
1856 // Unused: Everything goes in ZPR space.
1857 PPRByteOffset = 0;
1858 }
1859
1860 bool NeedGapToAlignStack = AFI->hasCalleeSaveStackFreeSpace();
1861 Register LastReg = 0;
1862 bool HasCSHazardPadding = AFI->hasStackHazardSlotIndex() && !SplitPPRs;
1863
1864 auto AlignOffset = [StackFillDir](int Offset, int Align) {
1865 if (StackFillDir < 0)
1866 return alignDown(Offset, Align);
1867 return alignTo(Offset, Align);
1868 };
1869
1870 // When iterating backwards, the loop condition relies on unsigned wraparound.
1871 for (unsigned i = FirstReg; i < Count; i += RegInc) {
1872 RegGroupInfo RGI;
1873 RGI.Reg1 = CSI[i].getReg();
1874
1875 if (AArch64::GPR64RegClass.contains(RGI.Reg1)) {
1876 RGI.Type = RegGroupInfo::GPR;
1877 RGI.RC = &AArch64::GPR64RegClass;
1878 } else if (AArch64::FPR64RegClass.contains(RGI.Reg1)) {
1879 RGI.Type = RegGroupInfo::FPR64;
1880 RGI.RC = &AArch64::FPR64RegClass;
1881 } else if (AArch64::FPR128RegClass.contains(RGI.Reg1)) {
1882 RGI.Type = RegGroupInfo::FPR128;
1883 RGI.RC = &AArch64::FPR128RegClass;
1884 } else if (AArch64::ZPRRegClass.contains(RGI.Reg1)) {
1885 RGI.Type = RegGroupInfo::ZPR;
1886 RGI.RC = &AArch64::ZPRRegClass;
1887 } else if (AArch64::PPRRegClass.contains(RGI.Reg1)) {
1888 RGI.Type = RegGroupInfo::PPR;
1889 RGI.RC = &AArch64::PPRRegClass;
1890 } else if (RGI.Reg1 == AArch64::VG) {
1891 RGI.Type = RegGroupInfo::VG;
1892 RGI.RC = &AArch64::FIXED_REGSRegClass;
1893 } else {
1894 llvm_unreachable("Unsupported register class.");
1895 }
1896
1897 int &ScalableByteOffset = RGI.Type == RegGroupInfo::PPR && SplitPPRs
1898 ? PPRByteOffset
1899 : ZPRByteOffset;
1900
1901 // Add the stack hazard size as we transition from GPR->FPR CSRs.
1902 if (HasCSHazardPadding &&
1903 (!LastReg || !AArch64InstrInfo::isFpOrNEON(LastReg)) &&
1905 ByteOffset += StackFillDir * StackHazardSize;
1906 LastReg = RGI.Reg1;
1907
1908 int Scale = TRI->getSpillSize(*RGI.RC);
1909 // Add the next reg to the pair if it is in the same register class.
1910 if (unsigned(i + RegInc) < Count && !HasCSHazardPadding) {
1911 MCRegister NextReg = CSI[i + RegInc].getReg();
1912 int Aligned = AlignOffset(ByteOffset, Scale);
1913 int PairOffset = IsWindows ? Aligned : Aligned + StackFillDir * 2 * Scale;
1914 bool PairFitsImmRange =
1915 PairOffset / Scale >= -64 && PairOffset / Scale <= 63;
1916 switch (RGI.Type) {
1917 case RegGroupInfo::GPR:
1918 case RegGroupInfo::FPR64:
1919 case RegGroupInfo::FPR128: {
1920 const TargetRegisterClass &SpillRC =
1921 RGI.Type == RegGroupInfo::GPR ? AArch64::GPR64RegClass
1922 : RGI.Type == RegGroupInfo::FPR64 ? AArch64::FPR64RegClass
1923 : AArch64::FPR128RegClass;
1924
1925 if (SpillRC.contains(NextReg) && PairFitsImmRange &&
1926 !invalidateRegisterPairing(SpillExtendedVolatile, PairOffset,
1927 RGI.Reg1, NextReg, IsWindows,
1928 NeedsWinCFI, NeedsFrameRecord, TRI))
1929 RGI.Reg2 = NextReg;
1930 break;
1931 }
1932 case RegGroupInfo::PPR:
1933 break;
1934 case RegGroupInfo::ZPR: {
1935 // Windows support is possible but the order requirement is reversed.
1936 // Also, WinCFI has no support for group ZPR loads/stores yet.
1937 if (isTargetWindows(MF) || AFI->getPredicateRegForFillSpill() == 0)
1938 break;
1939
1940 // We expect to see pairs and quads in descending order (e.g. [z7, z6,
1941 // z5, z4]). We ensure this in `orderZPRCalleeSavesForGroups()`. This
1942 // is required as (for Linux) StackFillDir is negative (so we start at
1943 // higher addresses) and we need to ensure the lower register in the
1944 // pair has the lower address to store the registers in the correct
1945 // order.
1946 auto GetZPRTupleLowestReg =
1947 [&](unsigned NumRegs) -> std::optional<Register> {
1948 // Check Reg1 is the last register in a tuple.
1949 if (RGI.Reg1 % NumRegs != NumRegs - 1)
1950 return std::nullopt;
1951
1952 // Check there's enough registers for the tuple.
1953 if (i + RegInc * NumRegs > Count)
1954 return std::nullopt;
1955
1956 for (unsigned Idx = 1; Idx < NumRegs; ++Idx) {
1957 Register Reg = CSI[i + RegInc * Idx].getReg();
1958 Register ExpectedReg = RGI.Reg1 - Idx;
1959 if (Reg != ExpectedReg)
1960 return std::nullopt;
1961 }
1962
1963 return RGI.Reg1 - NumRegs + 1;
1964 };
1965
1966 for (unsigned TupleSize : {4, 2}) {
1967 if (auto LowReg = GetZPRTupleLowestReg(TupleSize)) {
1968 // Note: ST1B has the same offset constraints.
1969 unsigned Opcode =
1970 TupleSize == 4 ? AArch64::LD1B_4Z_IMM : AArch64::LD1B_2Z_IMM;
1971 const int Offset =
1972 (ScalableByteOffset + StackFillDir * TupleSize * Scale);
1973 if (isValidMemOpOffset(TII, Opcode, Offset)) {
1974 RGI.Reg2 = *LowReg;
1975 break;
1976 }
1977 }
1978 }
1979
1980 break;
1981 }
1982 case RegGroupInfo::VG:
1983 break;
1984 }
1985 }
1986
1987 // GPRs and FPRs are saved in pairs of 64-bit regs. We expect the CSI
1988 // list to come in sorted by frame index so that we can issue the store
1989 // pair instructions directly. Assert if we see anything otherwise.
1990 //
1991 // The order of the registers in the list is controlled by
1992 // getCalleeSavedRegs(), so they will always be in-order, as well.
1993 [[maybe_unused]] bool RegsOrdered =
1994 all_of(seq(RGI.getNumRegs() - 1), [&](int RegOffset) {
1995 return CSI[i + RegInc * RegOffset].getFrameIdx() + RegInc ==
1996 CSI[i + RegInc * RegOffset + RegInc].getFrameIdx();
1997 });
1998 assert(RegsOrdered && "Out of order callee saved regs!");
1999
2000 assert((!RGI.isQuad() || RGI.Type == RegGroupInfo::ZPR) &&
2001 "Currently only ZPR's support four-register spills");
2002
2003 assert((!RGI.isPaired() || !NeedsFrameRecord || RGI.Reg2 != AArch64::FP ||
2004 RGI.Reg1 == AArch64::LR) &&
2005 "FrameRecord must be allocated together with LR");
2006
2007 // Windows AAPCS has FP and LR reversed.
2008 assert((!RGI.isPaired() || !NeedsFrameRecord || RGI.Reg1 != AArch64::FP ||
2009 RGI.Reg2 == AArch64::LR) &&
2010 "FrameRecord must be allocated together with LR");
2011
2012 // MachO's compact unwind format relies on all registers being stored in
2013 // adjacent register pairs.
2014 assert((!produceCompactUnwindFrame(AFL, MF) ||
2017 (RGI.isPaired() &&
2018 ((RGI.Reg1 == AArch64::LR && RGI.Reg2 == AArch64::FP) ||
2019 RGI.Reg1 + 1 == RGI.Reg2))) &&
2020 "Callee-save registers not saved as adjacent register pair!");
2021
2022 RGI.FrameIdx = CSI[i].getFrameIdx();
2023 if (IsWindows)
2024 // RGI.FrameIdx must be the lower index of the group
2025 RGI.FrameIdx = CSI[i + RegInc * (RGI.getNumRegs() - 1)].getFrameIdx();
2026
2027 // Realign the scalable offset if necessary. This is relevant when spilling
2028 // predicates on Windows.
2029 if (RGI.isScalable() && ScalableByteOffset % Scale != 0)
2030 ScalableByteOffset = AlignOffset(ScalableByteOffset, Scale);
2031
2032 // Realign the fixed offset if necessary. This is relevant when spilling Q
2033 // registers after spilling an odd amount of X registers.
2034 if (!RGI.isScalable() && ByteOffset % Scale != 0)
2035 ByteOffset = AlignOffset(ByteOffset, Scale);
2036
2037 int OffsetPre = RGI.isScalable() ? ScalableByteOffset : ByteOffset;
2038 assert(OffsetPre % Scale == 0);
2039
2040 if (RGI.isScalable())
2041 ScalableByteOffset += StackFillDir * RGI.getNumRegs() * Scale;
2042 else
2043 ByteOffset += StackFillDir * RGI.getNumRegs() * Scale;
2044
2045 // Swift's async context is directly before FP, so allocate an extra
2046 // 8 bytes for it.
2047 if (NeedsFrameRecord && AFI->hasSwiftAsyncContext() &&
2048 ((!IsWindows && RGI.Reg2 == AArch64::FP) ||
2049 (IsWindows && RGI.Reg2 == AArch64::LR)))
2050 ByteOffset += StackFillDir * 8;
2051
2052 // Round up size of non-pair to pair size if we need to pad the
2053 // callee-save area to ensure 16-byte alignment.
2054 if (NeedGapToAlignStack && !IsWindows && !RGI.isScalable() &&
2055 RGI.Type != RegGroupInfo::FPR128 && !RGI.isPaired() &&
2056 ByteOffset % 16 != 0) {
2057 ByteOffset += 8 * StackFillDir;
2058 assert(MFI.getObjectAlign(RGI.FrameIdx) <= Align(16));
2059 // A stack frame with a gap looks like this, bottom up:
2060 // d9, d8. x21, gap, x20, x19.
2061 // Set extra alignment on the x21 object to create the gap above it.
2062 MFI.setObjectAlignment(RGI.FrameIdx, Align(16));
2063 NeedGapToAlignStack = false;
2064 }
2065
2066 int OffsetPost = RGI.isScalable() ? ScalableByteOffset : ByteOffset;
2067 assert(OffsetPost % Scale == 0);
2068 // If filling top down (default), we want the offset after incrementing it.
2069 // If filling bottom up (WinCFI) we need the original offset.
2070 int Offset = IsWindows ? OffsetPre : OffsetPost;
2071
2072 // The FP, LR pair goes 8 bytes into our expanded 24-byte slot so that the
2073 // Swift context can directly precede FP.
2074 if (NeedsFrameRecord && AFI->hasSwiftAsyncContext() &&
2075 ((!IsWindows && RGI.Reg2 == AArch64::FP) ||
2076 (IsWindows && RGI.Reg2 == AArch64::LR)))
2077 Offset += 8;
2078
2079 assert(
2080 (!RGI.isGrouped() || RGI.Type != RegGroupInfo::ZPR ||
2082 TII, (RGI.isQuad() ? AArch64::LD1B_4Z_IMM : AArch64::LD1B_2Z_IMM),
2083 Offset)) &&
2084 "Offset invalid for LD1B/ST1B pair/quad variant immediate");
2085
2086 RGI.Offset = Offset / Scale;
2087 assert((!RGI.isGrouped() || RGI.Type == RegGroupInfo::ZPR ||
2088 (!RGI.isScalable() && RGI.Offset >= -64 && RGI.Offset <= 63) ||
2089 (RGI.isScalable() && RGI.Offset >= -256 && RGI.Offset <= 255)) &&
2090 "Offset out of bounds for LDP/STP immediate");
2091
2092 auto isFrameRecord = [&] {
2093 if (RGI.isPaired())
2094 return IsWindows ? RGI.Reg1 == AArch64::FP && RGI.Reg2 == AArch64::LR
2095 : RGI.Reg1 == AArch64::LR && RGI.Reg2 == AArch64::FP;
2096 // Otherwise, look for the frame record as two unpaired registers. This is
2097 // needed for -aarch64-stack-hazard-size=<val>, which disables register
2098 // pairing (as the padding may be too large for the LDP/STP offset). Note:
2099 // On Windows, this check works out as current reg == FP, next reg == LR,
2100 // and on other platforms current reg == FP, previous reg == LR. This
2101 // works out as the correct pre-increment or post-increment offsets
2102 // respectively.
2103 return i > 0 && RGI.Reg1 == AArch64::FP &&
2104 CSI[i - 1].getReg() == AArch64::LR;
2105 };
2106
2107 // Save the offset to frame record so that the FP register can point to the
2108 // innermost frame record (spilled FP and LR registers).
2109 if (NeedsFrameRecord && isFrameRecord())
2111
2112 RegGroups.push_back(RGI);
2113 i += RegInc * (RGI.getNumRegs() - 1);
2114 }
2115 if (IsWindows) {
2116 // If we need an alignment gap in the stack, align the topmost stack
2117 // object. A stack frame with a gap looks like this, bottom up:
2118 // x19, d8. d9, gap.
2119 // Set extra alignment on the topmost stack object (the first element in
2120 // CSI, which goes top down), to create the gap above it.
2121 if (AFI->hasCalleeSaveStackFreeSpace())
2122 MFI.setObjectAlignment(CSI[0].getFrameIdx(), Align(16));
2123 // We iterated bottom up over the registers; flip RegGroups back to top
2124 // down order.
2125 std::reverse(RegGroups.begin(), RegGroups.end());
2126 }
2127}
2128
2132 MachineFunction &MF = *MBB.getParent();
2134 auto &TLI = *Subtarget.getTargetLowering();
2135 const AArch64InstrInfo &TII = *Subtarget.getInstrInfo();
2136 bool NeedsWinCFI = MF.needsWinCFI();
2137 DebugLoc DL;
2139
2140 computeCalleeSaveRegisterGroups(*this, MF, CSI, TRI, RegGroups, hasFP(MF));
2141
2143 // Refresh the reserved regs in case there are any potential changes since the
2144 // last freeze.
2145 MRI.freezeReservedRegs();
2146
2147 if (homogeneousPrologEpilog(MF)) {
2148 auto MIB = BuildMI(MBB, MI, DL, TII.get(AArch64::HOM_Prolog))
2150
2151 for (auto &RGI : RegGroups) {
2152 MIB.addReg(RGI.Reg1);
2153 MIB.addReg(RGI.Reg2);
2154
2155 // Update register live in.
2156 if (!MRI.isReserved(RGI.Reg1))
2157 MBB.addLiveIn(RGI.Reg1);
2158 if (RGI.isPaired() && !MRI.isReserved(RGI.Reg2))
2159 MBB.addLiveIn(RGI.Reg2);
2160 }
2161 return true;
2162 }
2163 bool PTrueCreated = false;
2164 for (const RegGroupInfo &RGI : llvm::reverse(RegGroups)) {
2165 Register Reg1 = RGI.Reg1;
2166 Register Reg2 = RGI.Reg2;
2167 unsigned StrOpc;
2168
2169 // Issue sequence of spills for cs regs. The first spill may be converted
2170 // to a pre-decrement store later by emitPrologue if the callee-save stack
2171 // area allocation can't be combined with the local stack area allocation.
2172 // For example:
2173 // stp x22, x21, [sp, #0] // addImm(+0)
2174 // stp x20, x19, [sp, #16] // addImm(+2)
2175 // stp fp, lr, [sp, #32] // addImm(+4)
2176 // Rationale: This sequence saves uop updates compared to a sequence of
2177 // pre-increment spills like stp xi,xj,[sp,#-16]!
2178 // Note: Similar rationale and sequence for restores in epilog.
2179 unsigned Size = TRI->getSpillSize(*RGI.RC);
2180 Align Alignment = TRI->getSpillAlign(*RGI.RC);
2181 switch (RGI.Type) {
2182 case RegGroupInfo::GPR:
2183 StrOpc = RGI.isPaired() ? AArch64::STPXi : AArch64::STRXui;
2184 break;
2185 case RegGroupInfo::FPR64:
2186 StrOpc = RGI.isPaired() ? AArch64::STPDi : AArch64::STRDui;
2187 break;
2188 case RegGroupInfo::FPR128:
2189 StrOpc = RGI.isPaired() ? AArch64::STPQi : AArch64::STRQui;
2190 break;
2191 case RegGroupInfo::ZPR:
2192 StrOpc = RGI.isQuad()
2193 ? AArch64::ST1B_4Z_IMM
2194 : (RGI.isPaired() ? AArch64::ST1B_2Z_IMM : AArch64::STR_ZXI);
2195 break;
2196 case RegGroupInfo::PPR:
2197 StrOpc = AArch64::STR_PXI;
2198 break;
2199 case RegGroupInfo::VG:
2200 StrOpc = AArch64::STRXui;
2201 break;
2202 }
2203
2204 Register X0Scratch;
2205 llvm::scope_exit RestoreX0([&] {
2206 if (X0Scratch.isValid())
2207 BuildMI(MBB, MI, DL, TII.get(TargetOpcode::COPY), AArch64::X0)
2208 .addReg(X0Scratch)
2210 });
2211
2212 if (Reg1 == AArch64::VG) {
2213 // Find an available register to store value of VG to.
2214 Reg1 = findScratchNonCalleeSaveRegister(&MBB, true);
2215 assert(Reg1.isValid());
2216 if (MF.getSubtarget<AArch64Subtarget>().hasSVE()) {
2217 BuildMI(MBB, MI, DL, TII.get(AArch64::CNTD_XPiI), Reg1)
2218 .addImm(31)
2219 .addImm(1)
2221 } else {
2223 if (any_of(MBB.liveins(),
2224 [&STI](const MachineBasicBlock::RegisterMaskPair &LiveIn) {
2225 return STI.getRegisterInfo()->isSuperOrSubRegisterEq(
2226 AArch64::X0, LiveIn.PhysReg);
2227 })) {
2228 X0Scratch = Reg1;
2229 BuildMI(MBB, MI, DL, TII.get(TargetOpcode::COPY), X0Scratch)
2230 .addReg(AArch64::X0)
2232 }
2233
2234 RTLIB::Libcall LC = RTLIB::SMEABI_GET_CURRENT_VG;
2235 const uint32_t *RegMask =
2236 TRI->getCallPreservedMask(MF, TLI.getLibcallCallingConv(LC));
2237 BuildMI(MBB, MI, DL, TII.get(AArch64::BL))
2238 .addExternalSymbol(TLI.getLibcallName(LC))
2239 .addRegMask(RegMask)
2240 .addReg(AArch64::X0, RegState::ImplicitDefine)
2242 Reg1 = AArch64::X0;
2243 }
2244 }
2245
2246 LLVM_DEBUG(dbgs() << "CSR spill: ");
2247 LLVM_DEBUG(RGI.dump(TRI));
2248
2249 assert((!isTargetWindows(MF) ||
2250 !(Reg1 == AArch64::LR && Reg2 == AArch64::FP)) &&
2251 "Windows unwdinding requires a consecutive (FP,LR) pair");
2252
2254 unsigned PnReg = AFI->getPredicateRegForFillSpill();
2255
2256 if (!PTrueCreated && RGI.isGrouped() && RGI.isScalable()) {
2257 [[maybe_unused]] const AArch64Subtarget &Subtarget =
2259 assert((PnReg != 0 && enableMultiVectorSpillFill(Subtarget, MF)) &&
2260 "Expects SVE2.1 or SME2 target and a predicate register");
2261 PTrueCreated = true;
2262 BuildMI(MBB, MI, DL, TII.get(AArch64::PTRUE_C_B), PnReg)
2264 }
2265
2266 SmallVector<Register, 4> Registers = RGI.getRegisters();
2268 if (!MRI.isReserved(Reg))
2269 MBB.addLiveIn(Reg);
2270
2271 MachineInstrBuilder MIB = BuildMI(MBB, MI, DL, TII.get(StrOpc));
2272 if (RGI.isGrouped() && RGI.isScalable()) {
2273 assert(!isTargetWindows(MF) &&
2274 "Scalable register groups are not supported by Windows CFI yet");
2275#ifdef EXPENSIVE_CHECKS
2276 auto IsPPR = [](const RegGroupInfo &c) {
2277 return c.Type == RegGroupInfo::PPR;
2278 };
2279 auto PPRBegin = std::find_if(RegGroups.begin(), RegGroups.end(), IsPPR);
2280 auto IsZPR = [](const RegGroupInfo &c) {
2281 return c.Type == RegGroupInfo::ZPR;
2282 };
2283 auto ZPRBegin = std::find_if(RegGroups.begin(), RegGroups.end(), IsZPR);
2284 assert(!(PPRBegin < ZPRBegin) &&
2285 "Expected callee save predicate to be handled first");
2286#endif
2287
2288 assert(RGI.Reg2 < RGI.Reg1 && "Expected reversed ZPR group");
2289 MIB.addReg(RGI.getAsMultiVectorRegister());
2290
2291 MIB.addReg(PnReg);
2292 MIB.addReg(AArch64::SP)
2293 .addImm(RGI.Offset /
2294 RGI.getNumRegs()) // [sp, #imm*size*vscale],
2295 // where size*vscale is implicit
2297 } else { // The code when the pair of ZReg is not present
2298 // Windows unwind codes require consecutive registers if registers are
2299 // paired. Make the switch here, so that the code below will save (x,x+1)
2300 // and not (x+1,x).
2301 if (isTargetWindows(MF) && RGI.isPaired())
2302 std::swap(Reg1, Reg2);
2303 if (RGI.isPaired()) {
2304 MIB.addReg(Reg2, getPrologueDeath(MF, Reg2));
2305 }
2306 MIB.addReg(Reg1, getPrologueDeath(MF, Reg1))
2307 .addReg(AArch64::SP)
2308 .addImm(RGI.Offset) // [sp, #offset*vscale],
2309 // where factor*vscale is implicit
2311 if (NeedsWinCFI)
2312 insertSEH(MIB, TII, MachineInstr::FrameSetup);
2313 }
2314
2315 // Handle the StackIDs.
2316 MachineFrameInfo &MFI = MF.getFrameInfo();
2317 for (int FI :
2318 reverse_conditionally(RGI.getFrameIndexes(), !isTargetWindows(MF))) {
2319 MIB.addMemOperand(
2321 MachineMemOperand::MOStore, Size, Alignment));
2322 if (RGI.Type == RegGroupInfo::ZPR)
2324 else if (RGI.Type == RegGroupInfo::PPR)
2326 }
2327 }
2328 return true;
2329}
2330
2334 MachineFunction &MF = *MBB.getParent();
2335 const AArch64InstrInfo &TII =
2336 *MF.getSubtarget<AArch64Subtarget>().getInstrInfo();
2337 DebugLoc DL;
2339 bool NeedsWinCFI = MF.needsWinCFI();
2340
2341 if (MBBI != MBB.end())
2342 DL = MBBI->getDebugLoc();
2343
2344 computeCalleeSaveRegisterGroups(*this, MF, CSI, TRI, RegGroups, hasFP(MF));
2345 if (homogeneousPrologEpilog(MF, &MBB)) {
2346 auto MIB = BuildMI(MBB, MBBI, DL, TII.get(AArch64::HOM_Epilog))
2348 for (auto &RGI : RegGroups) {
2349 MIB.addReg(RGI.Reg1, RegState::Define);
2350 MIB.addReg(RGI.Reg2, RegState::Define);
2351 }
2352 return true;
2353 }
2354
2355 // For performance reasons restore SVE register in increasing order
2356 auto IsPPR = [](const RegGroupInfo &c) {
2357 return c.Type == RegGroupInfo::PPR;
2358 };
2359 auto PPRBegin = llvm::find_if(RegGroups, IsPPR);
2360 auto PPREnd = std::find_if_not(PPRBegin, RegGroups.end(), IsPPR);
2361 std::reverse(PPRBegin, PPREnd);
2362 auto IsZPR = [](const RegGroupInfo &c) {
2363 return c.Type == RegGroupInfo::ZPR;
2364 };
2365 auto ZPRBegin = llvm::find_if(RegGroups, IsZPR);
2366 auto ZPREnd = std::find_if_not(ZPRBegin, RegGroups.end(), IsZPR);
2367 std::reverse(ZPRBegin, ZPREnd);
2368
2369 // If the return address is authenticated before the epilogue, it stays in LR
2370 // rather than being restored here, as that would replace it with the
2371 // unauthenticated value from the stack.
2372 bool KeepLR = authenticatesReturnAddressEarly(MF, MBB);
2373
2374 bool PTrueCreated = false;
2375 for (const RegGroupInfo &RGI : RegGroups) {
2376 Register Reg1 = RGI.Reg1;
2377 Register Reg2 = RGI.Reg2;
2378
2379 // Issue sequence of restores for cs regs. The last restore may be converted
2380 // to a post-increment load later by emitEpilogue if the callee-save stack
2381 // area allocation can't be combined with the local stack area allocation.
2382 // For example:
2383 // ldp fp, lr, [sp, #32] // addImm(+4)
2384 // ldp x20, x19, [sp, #16] // addImm(+2)
2385 // ldp x22, x21, [sp, #0] // addImm(+0)
2386 // Note: see comment in spillCalleeSavedRegisters()
2387 unsigned LdrOpc;
2388 unsigned Size = TRI->getSpillSize(*RGI.RC);
2389 Align Alignment = TRI->getSpillAlign(*RGI.RC);
2390 switch (RGI.Type) {
2391 case RegGroupInfo::GPR:
2392 LdrOpc = RGI.isPaired() ? AArch64::LDPXi : AArch64::LDRXui;
2393 break;
2394 case RegGroupInfo::FPR64:
2395 LdrOpc = RGI.isPaired() ? AArch64::LDPDi : AArch64::LDRDui;
2396 break;
2397 case RegGroupInfo::FPR128:
2398 LdrOpc = RGI.isPaired() ? AArch64::LDPQi : AArch64::LDRQui;
2399 break;
2400 case RegGroupInfo::ZPR:
2401 LdrOpc = RGI.isQuad()
2402 ? AArch64::LD1B_4Z_IMM
2403 : (RGI.isPaired() ? AArch64::LD1B_2Z_IMM : AArch64::LDR_ZXI);
2404 break;
2405 case RegGroupInfo::PPR:
2406 LdrOpc = AArch64::LDR_PXI;
2407 break;
2408 case RegGroupInfo::VG:
2409 continue;
2410 }
2411
2412 LLVM_DEBUG(dbgs() << "CSR restore: ");
2413 LLVM_DEBUG(RGI.dump(TRI));
2414
2416 unsigned PnReg = AFI->getPredicateRegForFillSpill();
2417
2418 if (!PTrueCreated && RGI.isGrouped() && RGI.isScalable()) {
2419 [[maybe_unused]] const AArch64Subtarget &Subtarget =
2421 assert((PnReg != 0 && enableMultiVectorSpillFill(Subtarget, MF)) &&
2422 "Expects SVE2.1 or SME2 target and a predicate register");
2423 PTrueCreated = true;
2424 BuildMI(MBB, MBBI, DL, TII.get(AArch64::PTRUE_C_B), PnReg)
2426 }
2427
2428 if (KeepLR && RGI.Type == RegGroupInfo::GPR &&
2429 (Reg1 == AArch64::LR || Reg2 == AArch64::LR)) {
2430 // Windows pairs LR with the register before it, which is the one at the
2431 // lower address, so restore just that register. The pair has not been
2432 // swapped yet at this point, so that register is Reg1.
2433 if (!RGI.isPaired())
2434 continue;
2435 assert(Reg2 == AArch64::LR && "Expected LR to be the second of the pair");
2437 BuildMI(MBB, MBBI, DL, TII.get(AArch64::LDRXui));
2438 MIB.addReg(Reg1, getDefRegState(true));
2439 MIB.addReg(AArch64::SP)
2440 .addImm(RGI.Offset)
2443 MachinePointerInfo::getFixedStack(MF, RGI.FrameIdx),
2445 if (NeedsWinCFI)
2446 insertSEH(MIB, TII, MachineInstr::FrameDestroy);
2447 continue;
2448 }
2449
2450 MachineInstrBuilder MIB = BuildMI(MBB, MBBI, DL, TII.get(LdrOpc));
2451 if (RGI.isGrouped() && RGI.isScalable()) {
2452 assert(!isTargetWindows(MF) &&
2453 "Scalable register groups are not supported by Windows CFI yet");
2454#ifdef EXPENSIVE_CHECKS
2455 assert(!(PPRBegin < ZPRBegin) &&
2456 "Expected callee save predicate to be handled first");
2457#endif
2458 assert(RGI.Reg2 < RGI.Reg1 && "Expected reversed ZPR group");
2459 MIB.addReg(RGI.getAsMultiVectorRegister(), getDefRegState(true));
2460 MIB.addReg(PnReg);
2461 MIB.addReg(AArch64::SP)
2462 .addImm(RGI.Offset /
2463 RGI.getNumRegs()) // [sp, #imm*size*vscale]
2464 // where size*vscale is implicit
2466 } else {
2467 // Windows unwind codes require consecutive registers if registers are
2468 // paired. Make the switch here, so that the code below will save (x,x+1)
2469 // and not (x+1,x).
2470 if (isTargetWindows(MF) && RGI.isPaired())
2471 std::swap(Reg1, Reg2);
2472 if (RGI.isPaired())
2473 MIB.addReg(Reg2, getDefRegState(true));
2474 MIB.addReg(Reg1, getDefRegState(true));
2475 MIB.addReg(AArch64::SP)
2476 .addImm(RGI.Offset) // [sp, #offset*vscale]
2477 // where factor*vscale is implicit
2479 if (NeedsWinCFI)
2480 insertSEH(MIB, TII, MachineInstr::FrameDestroy);
2481 }
2482
2483 for (int FI :
2484 reverse_conditionally(RGI.getFrameIndexes(), !isTargetWindows(MF)))
2485 MIB.addMemOperand(
2487 MachineMemOperand::MOLoad, Size, Alignment));
2488 }
2489 return true;
2490}
2491
2492// Return the FrameID for a MMO.
2493static std::optional<int> getMMOFrameID(MachineMemOperand *MMO,
2494 const MachineFrameInfo &MFI) {
2495 auto *PSV =
2497 if (PSV)
2498 return std::optional<int>(PSV->getFrameIndex());
2499
2500 if (MMO->getValue()) {
2501 if (auto *Al = dyn_cast<AllocaInst>(getUnderlyingObject(MMO->getValue()))) {
2502 for (int FI = MFI.getObjectIndexBegin(); FI < MFI.getObjectIndexEnd();
2503 FI++)
2504 if (MFI.getObjectAllocation(FI) == Al)
2505 return FI;
2506 }
2507 }
2508
2509 return std::nullopt;
2510}
2511
2512// Return the FrameID for a Load/Store instruction by looking at the first MMO.
2513static std::optional<int> getLdStFrameID(const MachineInstr &MI,
2514 const MachineFrameInfo &MFI) {
2515 if (!MI.mayLoadOrStore() || MI.getNumMemOperands() < 1)
2516 return std::nullopt;
2517
2518 return getMMOFrameID(*MI.memoperands_begin(), MFI);
2519}
2520
2521// Returns true if the LDST MachineInstr \p MI is a PPR access.
2522static bool isPPRAccess(const MachineInstr &MI) {
2523 return AArch64::PPRRegClass.contains(MI.getOperand(0).getReg());
2524}
2525
2526// Check if a Hazard slot is needed for the current function, and if so create
2527// one for it. The index is stored in AArch64FunctionInfo->StackHazardSlotIndex,
2528// which can be used to determine if any hazard padding is needed.
2529void AArch64FrameLowering::determineStackHazardSlot(
2530 MachineFunction &MF, BitVector &SavedRegs) const {
2531 unsigned StackHazardSize = getStackHazardSize(MF);
2532 auto *AFI = MF.getInfo<AArch64FunctionInfo>();
2533 if (StackHazardSize == 0 || StackHazardSize % 16 != 0 ||
2535 return;
2536
2537 // Stack hazards are only needed in streaming functions.
2538 SMEAttrs Attrs = AFI->getSMEFnAttrs();
2539 const AArch64Options &CLOpts =
2540 MF.getSubtarget<AArch64Subtarget>().getCLOpts();
2541 if (!CLOpts.stack_hazard_in_non_streaming &&
2542 Attrs.hasNonStreamingInterfaceAndBody())
2543 return;
2544
2545 MachineFrameInfo &MFI = MF.getFrameInfo();
2546
2547 // Add a hazard slot if there are any CSR FPR registers, or are any fp-only
2548 // stack objects.
2549 bool HasFPRCSRs = any_of(SavedRegs.set_bits(), [](unsigned Reg) {
2550 return AArch64::FPR64RegClass.contains(Reg) ||
2551 AArch64::FPR128RegClass.contains(Reg) ||
2552 AArch64::ZPRRegClass.contains(Reg);
2553 });
2554 bool HasPPRCSRs = any_of(SavedRegs.set_bits(), [](unsigned Reg) {
2555 return AArch64::PPRRegClass.contains(Reg);
2556 });
2557 bool HasFPRStackObjects = false;
2558 bool HasPPRStackObjects = false;
2559 if (!HasFPRCSRs || CLOpts.split_sve_objects) {
2560 enum SlotType : uint8_t {
2561 Unknown = 0,
2562 ZPRorFPR = 1 << 0,
2563 PPR = 1 << 1,
2564 GPR = 1 << 2,
2566 };
2567
2568 // Find stack slots solely used for one kind of register (ZPR, PPR, etc.),
2569 // based on the kinds of accesses used in the function.
2570 SmallVector<SlotType> SlotTypes(MFI.getObjectIndexEnd(), SlotType::Unknown);
2571 for (auto &MBB : MF) {
2572 for (auto &MI : MBB) {
2573 std::optional<int> FI = getLdStFrameID(MI, MFI);
2574 if (!FI || FI < 0 || FI > int(SlotTypes.size()))
2575 continue;
2576 if (MFI.hasScalableStackID(*FI)) {
2577 SlotTypes[*FI] |=
2578 isPPRAccess(MI) ? SlotType::PPR : SlotType::ZPRorFPR;
2579 } else {
2580 SlotTypes[*FI] |= AArch64InstrInfo::isFpOrNEON(MI)
2581 ? SlotType::ZPRorFPR
2582 : SlotType::GPR;
2583 }
2584 }
2585 }
2586
2587 for (int FI = 0; FI < int(SlotTypes.size()); ++FI) {
2588 HasFPRStackObjects |= SlotTypes[FI] == SlotType::ZPRorFPR;
2589 // For SplitSVEObjects remember that this stack slot is a predicate, this
2590 // will be needed later when determining the frame layout.
2591 if (SlotTypes[FI] == SlotType::PPR) {
2593 HasPPRStackObjects = true;
2594 }
2595 }
2596 }
2597
2598 if (HasFPRCSRs || HasFPRStackObjects) {
2599 int ID = MFI.CreateStackObject(StackHazardSize, Align(16), false);
2600 LLVM_DEBUG(dbgs() << "Created Hazard slot at " << ID << " size "
2601 << StackHazardSize << "\n");
2602 AFI->setStackHazardSlotIndex(ID);
2603 }
2604
2605 if (!AFI->hasStackHazardSlotIndex())
2606 return;
2607
2608 if (CLOpts.split_sve_objects) {
2609 CallingConv::ID CC = MF.getFunction().getCallingConv();
2611 AFI->setSplitSVEObjects(true);
2612 LLVM_DEBUG(dbgs() << "Using SplitSVEObjects for SVE CC function\n");
2613 return;
2614 }
2615
2616 // We only use SplitSVEObjects in non-SVE CC functions if there's a
2617 // possibility of a stack hazard between PPRs and ZPRs/FPRs.
2618 LLVM_DEBUG(dbgs() << "Determining if SplitSVEObjects should be used in "
2619 "non-SVE CC function...\n");
2620
2621 // If another calling convention is explicitly set FPRs can't be promoted to
2622 // ZPR callee-saves.
2624 LLVM_DEBUG(
2625 dbgs()
2626 << "Calling convention is not supported with SplitSVEObjects\n");
2627 return;
2628 }
2629
2630 if (!HasPPRCSRs && !HasPPRStackObjects) {
2631 LLVM_DEBUG(
2632 dbgs() << "Not using SplitSVEObjects as no PPRs are on the stack\n");
2633 return;
2634 }
2635
2636 if (!HasFPRCSRs && !HasFPRStackObjects) {
2637 LLVM_DEBUG(
2638 dbgs()
2639 << "Not using SplitSVEObjects as no FPRs or ZPRs are on the stack\n");
2640 return;
2641 }
2642
2643 [[maybe_unused]] const AArch64Subtarget &Subtarget =
2644 MF.getSubtarget<AArch64Subtarget>();
2646 "Expected SVE to be available for PPRs");
2647
2648 const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();
2649 // With SplitSVEObjects the CS hazard padding is placed between the
2650 // PPRs and ZPRs. If there are any FPR CS there would be a hazard between
2651 // them and the CS GRPs. Avoid this by promoting all FPR CS to ZPRs.
2652 BitVector FPRZRegs(SavedRegs.size());
2653 for (size_t Reg = 0, E = SavedRegs.size(); HasFPRCSRs && Reg < E; ++Reg) {
2654 BitVector::reference RegBit = SavedRegs[Reg];
2655 if (!RegBit)
2656 continue;
2657 unsigned SubRegIdx = 0;
2658 if (AArch64::FPR64RegClass.contains(Reg))
2659 SubRegIdx = AArch64::dsub;
2660 else if (AArch64::FPR128RegClass.contains(Reg))
2661 SubRegIdx = AArch64::zsub;
2662 else
2663 continue;
2664 // Clear the bit for the FPR save.
2665 RegBit = false;
2666 // Mark that we should save the corresponding ZPR.
2667 Register ZReg =
2668 TRI->getMatchingSuperReg(Reg, SubRegIdx, &AArch64::ZPRRegClass);
2669 FPRZRegs.set(ZReg);
2670 }
2671 SavedRegs |= FPRZRegs;
2672
2673 AFI->setSplitSVEObjects(true);
2674 LLVM_DEBUG(dbgs() << "SplitSVEObjects enabled!\n");
2675 }
2676}
2677
2679 BitVector &SavedRegs,
2680 RegScavenger *RS) const {
2682 // All calls are tail calls in GHC calling conv, and functions have no
2683 // prologue/epilogue.
2686 AFI->setSVECalleeSavedStackSize(/*ZPR=*/0, /*PPR=*/0);
2687 return;
2688 }
2689
2691
2693 const AArch64RegisterInfo *RegInfo = Subtarget.getRegisterInfo();
2694 Register UnspilledCSGPR;
2695 Register UnspilledCSGPRPaired;
2696
2697 MachineFrameInfo &MFI = MF.getFrameInfo();
2698 const MCPhysReg *CSRegs = MF.getRegInfo().getCalleeSavedRegs();
2699
2700 MCRegister BasePointerReg =
2701 RegInfo->hasBasePointer(MF) ? RegInfo->getBaseRegister() : MCRegister();
2702
2703 Register ExtraCSSpill;
2704 bool HasUnpairedGPR64 = false;
2705 bool HasPairZReg = false;
2706 BitVector UserReservedRegs = RegInfo->getUserReservedRegs(MF);
2707 BitVector ReservedRegs = RegInfo->getReservedRegs(MF);
2708
2709 // Figure out which callee-saved registers to save/restore.
2710 for (unsigned i = 0; CSRegs[i]; ++i) {
2711 const MCRegister Reg = CSRegs[i];
2712
2713 // Add the base pointer register to SavedRegs if it is callee-save.
2714 if (Reg == BasePointerReg)
2715 SavedRegs.set(Reg);
2716
2717 // Don't save manually reserved registers set through +reserve-x#i,
2718 // even for callee-saved registers, as per GCC's behavior.
2719 if (UserReservedRegs[Reg]) {
2720 SavedRegs.reset(Reg);
2721 continue;
2722 }
2723
2724 bool RegUsed = SavedRegs.test(Reg);
2725 MCRegister PairedReg;
2726 const bool RegIsGPR64 = AArch64::GPR64RegClass.contains(Reg);
2727 if (RegIsGPR64 || AArch64::FPR64RegClass.contains(Reg) ||
2728 AArch64::FPR128RegClass.contains(Reg)) {
2729 // Compensate for odd numbers of GP CSRs.
2730 // For now, all the known cases of odd number of CSRs are of GPRs.
2731 if (HasUnpairedGPR64)
2732 PairedReg = CSRegs[i % 2 == 0 ? i - 1 : i + 1];
2733 else
2734 PairedReg = CSRegs[i ^ 1];
2735 }
2736
2737 // If the function requires all the GP registers to save (SavedRegs),
2738 // and there are an odd number of GP CSRs at the same time (CSRegs),
2739 // PairedReg could be in a different register class from Reg, which would
2740 // lead to a FPR (usually D8) accidentally being marked saved.
2741 if (RegIsGPR64 && !AArch64::GPR64RegClass.contains(PairedReg)) {
2742 PairedReg = Register();
2743 HasUnpairedGPR64 = true;
2744 }
2745 assert(!PairedReg.isValid() ||
2746 AArch64::GPR64RegClass.contains(Reg, PairedReg) ||
2747 AArch64::FPR64RegClass.contains(Reg, PairedReg) ||
2748 AArch64::FPR128RegClass.contains(Reg, PairedReg));
2749
2750 if (!RegUsed) {
2751 if (AArch64::GPR64RegClass.contains(Reg) && !ReservedRegs[Reg]) {
2752 UnspilledCSGPR = Reg;
2753 UnspilledCSGPRPaired = PairedReg;
2754 }
2755 continue;
2756 }
2757
2758 // MachO's compact unwind format relies on all registers being stored in
2759 // pairs.
2760 // FIXME: the usual format is actually better if unwinding isn't needed.
2761 if (producePairRegisters(MF) && PairedReg.isValid() &&
2762 !SavedRegs.test(PairedReg)) {
2763 SavedRegs.set(PairedReg);
2764 if (AArch64::GPR64RegClass.contains(PairedReg) &&
2765 !ReservedRegs[PairedReg])
2766 ExtraCSSpill = PairedReg;
2767 }
2768 // Check if there is a pair of ZRegs, so it can select PReg for spill/fill
2769 HasPairZReg |= (AArch64::ZPRRegClass.contains(Reg, CSRegs[i ^ 1]) &&
2770 SavedRegs.test(CSRegs[i ^ 1]));
2771 }
2772
2773 if (HasPairZReg && enableMultiVectorSpillFill(Subtarget, MF)) {
2775 // Find a suitable predicate register for the multi-vector spill/fill
2776 // instructions.
2777 MCRegister PnReg = findFreePredicateReg(SavedRegs);
2778 if (PnReg.isValid())
2779 AFI->setPredicateRegForFillSpill(PnReg);
2780 // If no free callee-save has been found assign one.
2781 if (!AFI->getPredicateRegForFillSpill() &&
2782 MF.getFunction().getCallingConv() ==
2784 SavedRegs.set(AArch64::P8);
2785 AFI->setPredicateRegForFillSpill(AArch64::PN8);
2786 }
2787
2788 assert(!ReservedRegs[AFI->getPredicateRegForFillSpill()] &&
2789 "Predicate cannot be a reserved register");
2790 }
2791
2793 !Subtarget.isTargetWindowsOrUEFI()) {
2794 // For Windows calling convention on a non-windows OS, where X18 is treated
2795 // as reserved, back up X18 when entering non-windows code (marked with the
2796 // Windows calling convention) and restore when returning regardless of
2797 // whether the individual function uses it - it might call other functions
2798 // that clobber it.
2799 SavedRegs.set(AArch64::X18);
2800 }
2801
2802 // Determine if a Hazard slot should be used and where it should go.
2803 // If SplitSVEObjects is used, the hazard padding is placed between the PPRs
2804 // and ZPRs. Otherwise, it goes in the callee save area.
2805 determineStackHazardSlot(MF, SavedRegs);
2806
2807 // Calculates the callee saved stack size.
2808 unsigned CSStackSize = 0;
2809 unsigned ZPRCSStackSize = 0;
2810 unsigned PPRCSStackSize = 0;
2812 for (unsigned Reg : SavedRegs.set_bits()) {
2813 auto *RC = TRI->getMinimalPhysRegClass(MCRegister(Reg));
2814 assert(RC && "expected register class!");
2815 auto SpillSize = TRI->getSpillSize(*RC);
2816 bool IsZPR = AArch64::ZPRRegClass.contains(Reg);
2817 bool IsPPR = !IsZPR && AArch64::PPRRegClass.contains(Reg);
2818 if (IsZPR)
2819 ZPRCSStackSize += SpillSize;
2820 else if (IsPPR)
2821 PPRCSStackSize += SpillSize;
2822 else {
2823 // A register and its super-register can both appear in SavedRegs.
2824 // Only the widest register is actually spilled, so skip such
2825 // sub-registers here to avoid double-counting the overlap.
2826 bool SavedSuper = any_of(TRI->superregs(Reg), [&](MCPhysReg SuperReg) {
2827 return SavedRegs.test(SuperReg);
2828 });
2829 if (!SavedSuper)
2830 CSStackSize += SpillSize;
2831 }
2832 }
2833
2834 // Save number of saved regs, so we can easily update CSStackSize later to
2835 // account for any additional 64-bit GPR saves. Note: After this point
2836 // only 64-bit GPRs can be added to SavedRegs.
2837 unsigned NumSavedRegs = SavedRegs.count();
2838
2839 // If we have hazard padding in the CS area add that to the size.
2841 CSStackSize += getStackHazardSize(MF);
2842
2843 // Increase the callee-saved stack size if the function has streaming mode
2844 // changes, as we will need to spill the value of the VG register.
2845 if (requiresSaveVG(MF))
2846 CSStackSize += 8;
2847
2848 // If we must call __arm_get_current_vg in the prologue preserve the LR.
2849 if (requiresSaveVG(MF) && !Subtarget.hasSVE())
2850 SavedRegs.set(AArch64::LR);
2851
2852 // The frame record needs to be created by saving the appropriate registers
2853 uint64_t EstimatedStackSize = MFI.estimateStackSize(MF);
2854 if (hasFP(MF) ||
2855 windowsRequiresStackProbe(MF, EstimatedStackSize + CSStackSize + 16)) {
2856 SavedRegs.set(AArch64::FP);
2857 SavedRegs.set(AArch64::LR);
2858 }
2859
2860 LLVM_DEBUG({
2861 dbgs() << "*** determineCalleeSaves\nSaved CSRs:";
2862 for (unsigned Reg : SavedRegs.set_bits())
2863 dbgs() << ' ' << printReg(MCRegister(Reg), RegInfo);
2864 dbgs() << "\n";
2865 });
2866
2867 // If any callee-saved registers are used, the frame cannot be eliminated.
2868 auto [ZPRLocalStackSize, PPRLocalStackSize] =
2870 uint64_t SVELocals = ZPRLocalStackSize + PPRLocalStackSize;
2871 uint64_t SVEStackSize =
2872 alignTo(ZPRCSStackSize + PPRCSStackSize + SVELocals, 16);
2873 bool CanEliminateFrame = (SavedRegs.count() == 0) && !SVEStackSize;
2874
2875 // The CSR spill slots have not been allocated yet, so estimateStackSize
2876 // won't include them.
2877 unsigned EstimatedStackSizeLimit = estimateRSStackSizeLimit(MF);
2878
2879 // We may address some of the stack above the canonical frame address, either
2880 // for our own arguments or during a call. Include that in calculating whether
2881 // we have complicated addressing concerns.
2882 int64_t CalleeStackUsed = 0;
2883 for (int I = MFI.getObjectIndexBegin(); I != 0; ++I) {
2884 int64_t FixedOff = MFI.getObjectOffset(I);
2885 if (FixedOff > CalleeStackUsed)
2886 CalleeStackUsed = FixedOff;
2887 }
2888
2889 // Conservatively always assume BigStack when there are SVE spills.
2890 bool BigStack = SVEStackSize || (EstimatedStackSize + CSStackSize +
2891 CalleeStackUsed) > EstimatedStackSizeLimit;
2892 if (BigStack || !CanEliminateFrame || RegInfo->cannotEliminateFrame(MF))
2893 AFI->setHasStackFrame(true);
2894
2895 // Estimate if we might need to scavenge a register at some point in order
2896 // to materialize a stack offset. If so, either spill one additional
2897 // callee-saved register or reserve a special spill slot to facilitate
2898 // register scavenging. If we already spilled an extra callee-saved register
2899 // above to keep the number of spills even, we don't need to do anything else
2900 // here.
2901 if (BigStack) {
2902 if (!ExtraCSSpill.isValid() && UnspilledCSGPR.isValid()) {
2903 LLVM_DEBUG(dbgs() << "Spilling " << printReg(UnspilledCSGPR, RegInfo)
2904 << " to get a scratch register.\n");
2905 SavedRegs.set(UnspilledCSGPR);
2906 ExtraCSSpill = UnspilledCSGPR;
2907
2908 // MachO's compact unwind format relies on all registers being stored in
2909 // pairs, so if we need to spill one extra for BigStack, then we need to
2910 // store the pair.
2911 if (producePairRegisters(MF)) {
2912 if (!UnspilledCSGPRPaired.isValid()) {
2913 // Failed to make a pair for compact unwind format, revert spilling.
2914 if (produceCompactUnwindFrame(*this, MF)) {
2915 SavedRegs.reset(UnspilledCSGPR);
2916 ExtraCSSpill = Register();
2917 }
2918 } else
2919 SavedRegs.set(UnspilledCSGPRPaired);
2920 }
2921 }
2922
2923 // If we didn't find an extra callee-saved register to spill, create
2924 // an emergency spill slot.
2925 if (!ExtraCSSpill.isValid() ||
2926 MF.getRegInfo().isPhysRegUsed(ExtraCSSpill)) {
2928 const TargetRegisterClass &RC = AArch64::GPR64RegClass;
2929 unsigned Size = TRI->getSpillSize(RC);
2930 Align Alignment = TRI->getSpillAlign(RC);
2931 int FI = MFI.CreateSpillStackObject(Size, Alignment);
2932 RS->addScavengingFrameIndex(FI);
2933 LLVM_DEBUG(dbgs() << "No available CS registers, allocated fi#" << FI
2934 << " as the emergency spill slot.\n");
2935 }
2936 }
2937
2938 // Adding the size of additional 64bit GPR saves.
2939 CSStackSize += 8 * (SavedRegs.count() - NumSavedRegs);
2940
2941 // A Swift asynchronous context extends the frame record with a pointer
2942 // directly before FP.
2943 if (hasFP(MF) && AFI->hasSwiftAsyncContext())
2944 CSStackSize += 8;
2945
2946 uint64_t AlignedCSStackSize = alignTo(CSStackSize, 16);
2947 LLVM_DEBUG(dbgs() << "Estimated stack frame size: "
2948 << EstimatedStackSize + AlignedCSStackSize << " bytes.\n");
2949
2951 AFI->getCalleeSavedStackSize() == AlignedCSStackSize) &&
2952 "Should not invalidate callee saved info");
2953
2954 // Round up to register pair alignment to avoid additional SP adjustment
2955 // instructions.
2956 AFI->setCalleeSavedStackSize(AlignedCSStackSize);
2957 AFI->setCalleeSaveStackHasFreeSpace(AlignedCSStackSize != CSStackSize);
2958 AFI->setSVECalleeSavedStackSize(ZPRCSStackSize, alignTo(PPRCSStackSize, 16));
2959}
2960
2961// Reorder ZPRs to maximize grouping, quads and pairs. The main condition we try
2962// to satisfy is the offset and the register number.
2963// Quads require a (scaled offset % 4 == 0) && (LowestRegister % 4 == 0),
2964// e.g: (z4, z5, z6, z7) where z4 is at offset 8 would be legal quad
2965// Pairs require a (scaled offset % 2 == 0) && (LowestRegister % 2 == 0).
2966// e.g: (z4, z5) where z4 is at offset 6 would be a legal pair
2969 std::vector<CalleeSavedInfo> &CSI) {
2970 assert(!isTargetWindows(MF) &&
2971 "ZPR callee-save reordering not supported on Windows");
2972
2973 auto *AFI = MF.getInfo<AArch64FunctionInfo>();
2974 if (!AFI->getPredicateRegForFillSpill())
2975 return;
2976
2977 auto IsZPR = [](CalleeSavedInfo &CS) {
2978 return AArch64::ZPRRegClass.contains(CS.getReg());
2979 };
2980
2982 ZPRSaves = ZPRSaves.drop_until(IsZPR).take_while(IsZPR);
2983
2984 if (ZPRSaves.size() < 2)
2985 return;
2986
2987 int ZPRByteOffset = AFI->getZPRCalleeSavedStackSize();
2988 if (!AFI->hasSplitSVEObjects())
2989 ZPRByteOffset += AFI->getPPRCalleeSavedStackSize();
2990
2991 const int Scale = RegInfo->getSpillSize(AArch64::ZPRRegClass);
2992 const int LowestOffset =
2993 (ZPRByteOffset / Scale) - static_cast<int>(ZPRSaves.size());
2994
2995 int NumToAlign = alignTo(LowestOffset, 4) - LowestOffset;
2996 if (NumToAlign > int(ZPRSaves.size()))
2997 return;
2998
2999 sort(ZPRSaves,
3000 [](const auto &A, const auto &B) { return A.getReg() < B.getReg(); });
3001
3005
3006 int ZPRsNeededForAlignment = NumToAlign;
3007 for (size_t i = 0; i < ZPRSaves.size();) {
3008 int ZPRsRemaining = int(ZPRSaves.size()) - i;
3009 if (i + 3 < ZPRSaves.size() &&
3010 (ZPRSaves[i].getReg() - AArch64::Z0) % 4 == 0 &&
3011 ZPRSaves[i].getReg() + 3 == ZPRSaves[i + 3].getReg() &&
3012 ZPRsRemaining - 4 >= ZPRsNeededForAlignment) {
3013 Quads.push_back(
3014 {ZPRSaves[i], ZPRSaves[i + 1], ZPRSaves[i + 2], ZPRSaves[i + 3]});
3015 i += 4;
3016 continue;
3017 }
3018 // We only need to split a pair if there's no singles. If we have a single
3019 // ZPR then all required alignments (1, 2, or 3) can use that single and a
3020 // ZPR pair.
3021 bool MustSplitPairForSingle =
3022 ZPRsNeededForAlignment == 1 && Singles.empty() && ZPRsRemaining == 2;
3023
3024 if (i + 1 < ZPRSaves.size() &&
3025 (ZPRSaves[i].getReg() - AArch64::Z0) % 2 == 0 &&
3026 ZPRSaves[i].getReg() + 1 == ZPRSaves[i + 1].getReg() &&
3027 !MustSplitPairForSingle) {
3028 Pairs.push_back({ZPRSaves[i], ZPRSaves[i + 1]});
3029 // Don't decrement the ZPRs needed if we require one ZPR for alignment and
3030 // we've yet to find a single ZPR. We may still need to split a pair.
3031 if (ZPRsNeededForAlignment != 1 || !Singles.empty())
3032 ZPRsNeededForAlignment -= 2;
3033
3034 i += 2;
3035 continue;
3036 }
3037
3038 Singles.push_back(ZPRSaves[i++]);
3039 ZPRsNeededForAlignment--;
3040 }
3041
3042 if (Quads.empty() && Pairs.empty())
3043 return;
3044
3045 int NumUnassigned = ZPRSaves.size();
3046 auto Emit = [&](ArrayRef<CalleeSavedInfo> Saves) {
3047 // During pair/quad formation, we check for the register groups in reverse
3048 // order. This is to ensures that stack indexes remain consistent with the
3049 // instruction.
3050 for (const CalleeSavedInfo &CS : Saves)
3051 ZPRSaves[--NumUnassigned] = CS;
3052 };
3053
3054 // Align the offset to a multiple of 4 (to allow for the most quads/pairs).
3055 if (NumToAlign & 0b01)
3056 Emit(Singles.pop_back_val());
3057 if (NumToAlign & 0b10) {
3058 if (!Pairs.empty())
3059 Emit(Pairs.pop_back_val());
3060 else
3061 Emit({Singles.pop_back_val(), Singles.pop_back_val()});
3062 }
3063
3064 for (const auto &Quad : Quads)
3065 Emit(Quad);
3066
3067 for (const auto &Pair : Pairs)
3068 Emit(Pair);
3069
3070 for (const auto &Single : Singles)
3071 Emit(Single);
3072}
3073
3075 MachineFunction &MF, const TargetRegisterInfo *RegInfo,
3076 std::vector<CalleeSavedInfo> &CSI) const {
3077 bool IsWindows = isTargetWindows(MF);
3078 unsigned StackHazardSize = getStackHazardSize(MF);
3079 // To match the canonical windows frame layout, reverse the list of
3080 // callee saved registers to get them laid out by PrologEpilogInserter
3081 // in the right order. (PrologEpilogInserter allocates stack objects top
3082 // down. Windows canonical prologs store higher numbered registers at
3083 // the top, thus have the CSI array start from the highest registers.)
3084 if (IsWindows)
3085 std::reverse(CSI.begin(), CSI.end());
3086
3087 if (CSI.empty())
3088 return true; // Early exit if no callee saved registers are modified!
3089
3090 // Now that we know which registers need to be saved and restored, allocate
3091 // stack slots for them.
3092 MachineFrameInfo &MFI = MF.getFrameInfo();
3093 auto *AFI = MF.getInfo<AArch64FunctionInfo>();
3094
3095 // Insert VG into the list of CSRs, immediately before LR if saved.
3096 if (requiresSaveVG(MF)) {
3097 CalleeSavedInfo VGInfo(AArch64::VG);
3098 auto It =
3099 find_if(CSI, [](auto &Info) { return Info.getReg() == AArch64::LR; });
3100 if (It != CSI.end())
3101 CSI.insert(It, VGInfo);
3102 else
3103 CSI.push_back(VGInfo);
3104 }
3105
3107 if (!IsWindows && enableMultiVectorSpillFill(Subtarget, MF))
3108 // The Windows stack layout is not supported by this reordering function
3109 // yet.
3110 orderZPRCalleeSavesForGroups(MF, RegInfo, CSI);
3111
3112 Register LastReg = 0;
3113 int HazardSlotIndex = std::numeric_limits<int>::max();
3114 for (auto &CS : CSI) {
3115 MCRegister Reg = CS.getReg();
3116 const TargetRegisterClass *RC = RegInfo->getMinimalPhysRegClass(Reg);
3117
3118 // Create a hazard slot as we switch between GPR and FPR CSRs.
3120 (!LastReg || !AArch64InstrInfo::isFpOrNEON(LastReg)) &&
3122 assert(HazardSlotIndex == std::numeric_limits<int>::max() &&
3123 "Unexpected register order for hazard slot");
3124 HazardSlotIndex = MFI.CreateStackObject(StackHazardSize, Align(8), true);
3125 LLVM_DEBUG(dbgs() << "Created CSR Hazard at slot " << HazardSlotIndex
3126 << "\n");
3127 AFI->setStackHazardCSRSlotIndex(HazardSlotIndex);
3128 MFI.setIsCalleeSavedObjectIndex(HazardSlotIndex, true);
3129 }
3130
3131 unsigned Size = RegInfo->getSpillSize(*RC);
3132 Align Alignment(RegInfo->getSpillAlign(*RC));
3133 int FrameIdx = MFI.CreateStackObject(Size, Alignment, true);
3134 CS.setFrameIdx(FrameIdx);
3135 MFI.setIsCalleeSavedObjectIndex(FrameIdx, true);
3136
3137 // Grab 8 bytes below FP for the extended asynchronous frame info.
3138 if (hasFP(MF) && AFI->hasSwiftAsyncContext() && Reg == AArch64::FP) {
3139 FrameIdx = MFI.CreateStackObject(8, Alignment, true);
3140 AFI->setSwiftAsyncContextFrameIdx(FrameIdx);
3141 MFI.setIsCalleeSavedObjectIndex(FrameIdx, true);
3142 }
3143 LastReg = Reg;
3144 }
3145
3146 // Add hazard slot in the case where no FPR CSRs are present.
3148 HazardSlotIndex == std::numeric_limits<int>::max()) {
3149 HazardSlotIndex = MFI.CreateStackObject(StackHazardSize, Align(8), true);
3150 LLVM_DEBUG(dbgs() << "Created CSR Hazard at slot " << HazardSlotIndex
3151 << "\n");
3152 AFI->setStackHazardCSRSlotIndex(HazardSlotIndex);
3153 MFI.setIsCalleeSavedObjectIndex(HazardSlotIndex, true);
3154 }
3155
3156 return true;
3157}
3158
3160 const MachineFunction &MF) const {
3162 // If the function has streaming-mode changes, don't scavenge a
3163 // spillslot in the callee-save area, as that might require an
3164 // 'addvl' in the streaming-mode-changing call-sequence when the
3165 // function doesn't use a FP.
3166 if (AFI->hasStreamingModeChanges() && !hasFP(MF))
3167 return false;
3168 // Don't allow register salvaging with hazard slots, in case it moves objects
3169 // into the wrong place.
3170 if (AFI->hasStackHazardSlotIndex())
3171 return false;
3172 return AFI->hasCalleeSaveStackFreeSpace();
3173}
3174
3175/// returns true if there are any SVE callee saves.
3177 int &Min, int &Max) {
3178 Min = std::numeric_limits<int>::max();
3179 Max = std::numeric_limits<int>::min();
3180
3181 if (!MFI.isCalleeSavedInfoValid())
3182 return false;
3183
3184 const std::vector<CalleeSavedInfo> &CSI = MFI.getCalleeSavedInfo();
3185 for (auto &CS : CSI) {
3186 if (AArch64::ZPRRegClass.contains(CS.getReg()) ||
3187 AArch64::PPRRegClass.contains(CS.getReg())) {
3188 assert((Max == std::numeric_limits<int>::min() ||
3189 Max + 1 == CS.getFrameIdx()) &&
3190 "SVE CalleeSaves are not consecutive");
3191 Min = std::min(Min, CS.getFrameIdx());
3192 Max = std::max(Max, CS.getFrameIdx());
3193 }
3194 }
3195 return Min != std::numeric_limits<int>::max();
3196}
3197
3199 AssignObjectOffsets AssignOffsets) {
3200 MachineFrameInfo &MFI = MF.getFrameInfo();
3201 auto *AFI = MF.getInfo<AArch64FunctionInfo>();
3202
3203 SVEStackSizes SVEStack{};
3204
3205 // With SplitSVEObjects we maintain separate stack offsets for predicates
3206 // (PPRs) and SVE vectors (ZPRs). When SplitSVEObjects is disabled predicates
3207 // are included in the SVE vector area.
3208 uint64_t &ZPRStackTop = SVEStack.ZPRStackSize;
3209 uint64_t &PPRStackTop =
3210 AFI->hasSplitSVEObjects() ? SVEStack.PPRStackSize : SVEStack.ZPRStackSize;
3211
3212#ifndef NDEBUG
3213 // First process all fixed stack objects.
3214 for (int I = MFI.getObjectIndexBegin(); I != 0; ++I)
3215 assert(!MFI.hasScalableStackID(I) &&
3216 "SVE vectors should never be passed on the stack by value, only by "
3217 "reference.");
3218#endif
3219
3220 auto AllocateObject = [&](int FI) {
3222 ? ZPRStackTop
3223 : PPRStackTop;
3224
3225 // FIXME: Given that the length of SVE vectors is not necessarily a power of
3226 // two, we'd need to align every object dynamically at runtime if the
3227 // alignment is larger than 16. This is not yet supported.
3228 Align Alignment = MFI.getObjectAlign(FI);
3229 if (Alignment > Align(16))
3231 "Alignment of scalable vectors > 16 bytes is not yet supported");
3232
3233 StackTop += MFI.getObjectSize(FI);
3234 StackTop = alignTo(StackTop, Alignment);
3235
3236 assert(StackTop < (uint64_t)std::numeric_limits<int64_t>::max() &&
3237 "SVE StackTop far too large?!");
3238
3239 int64_t Offset = -int64_t(StackTop);
3240 if (AssignOffsets == AssignObjectOffsets::Yes)
3241 MFI.setObjectOffset(FI, Offset);
3242
3243 LLVM_DEBUG(dbgs() << "alloc FI(" << FI << ") at SP[" << Offset << "]\n");
3244 };
3245
3246 // Then process all callee saved slots.
3247 int MinCSFrameIndex, MaxCSFrameIndex;
3248 if (getSVECalleeSaveSlotRange(MFI, MinCSFrameIndex, MaxCSFrameIndex)) {
3249 for (int FI = MinCSFrameIndex; FI <= MaxCSFrameIndex; ++FI)
3250 AllocateObject(FI);
3251 }
3252
3253 // Ensure the CS area is 16-byte aligned.
3254 PPRStackTop = alignTo(PPRStackTop, Align(16U));
3255 ZPRStackTop = alignTo(ZPRStackTop, Align(16U));
3256
3257 // Create a buffer of SVE objects to allocate and sort it.
3258 SmallVector<int, 8> ObjectsToAllocate;
3259 // If we have a stack protector, and we've previously decided that we have SVE
3260 // objects on the stack and thus need it to go in the SVE stack area, then it
3261 // needs to go first.
3262 int StackProtectorFI = -1;
3263 if (MFI.hasStackProtectorIndex()) {
3264 StackProtectorFI = MFI.getStackProtectorIndex();
3265 if (MFI.getStackID(StackProtectorFI) == TargetStackID::ScalableVector)
3266 ObjectsToAllocate.push_back(StackProtectorFI);
3267 }
3268
3269 for (int FI = 0, E = MFI.getObjectIndexEnd(); FI != E; ++FI) {
3270 if (FI == StackProtectorFI || MFI.isDeadObjectIndex(FI) ||
3272 continue;
3273
3276 continue;
3277
3278 ObjectsToAllocate.push_back(FI);
3279 }
3280
3281 // Allocate all SVE locals and spills
3282 for (unsigned FI : ObjectsToAllocate)
3283 AllocateObject(FI);
3284
3285 PPRStackTop = alignTo(PPRStackTop, Align(16U));
3286 ZPRStackTop = alignTo(ZPRStackTop, Align(16U));
3287
3288 if (AssignOffsets == AssignObjectOffsets::Yes)
3289 AFI->setStackSizeSVE(SVEStack.ZPRStackSize, SVEStack.PPRStackSize);
3290
3291 return SVEStack;
3292}
3293
3295 MachineFunction &MF, RegScavenger *RS) const {
3297 "Upwards growing stack unsupported");
3298
3300
3301 // If this function isn't doing Win64-style C++ EH, we don't need to do
3302 // anything.
3303 if (!MF.hasEHFunclets())
3304 return;
3305
3306 MachineFrameInfo &MFI = MF.getFrameInfo();
3307 auto *AFI = MF.getInfo<AArch64FunctionInfo>();
3308
3309 // Win64 C++ EH needs to allocate space for the catch objects in the fixed
3310 // object area right next to the UnwindHelp object.
3311 WinEHFuncInfo &EHInfo = *MF.getWinEHFuncInfo();
3312 int64_t CurrentOffset =
3314 for (WinEHTryBlockMapEntry &TBME : EHInfo.TryBlockMap) {
3315 for (WinEHHandlerType &H : TBME.HandlerArray) {
3316 int FrameIndex = H.CatchObj.FrameIndex;
3317 if ((FrameIndex != INT_MAX) && MFI.getObjectOffset(FrameIndex) == 0) {
3318 CurrentOffset =
3319 alignTo(CurrentOffset, MFI.getObjectAlign(FrameIndex).value());
3320 CurrentOffset += MFI.getObjectSize(FrameIndex);
3321 MFI.setObjectOffset(FrameIndex, -CurrentOffset);
3322 }
3323 }
3324 }
3325
3326 // Create an UnwindHelp object.
3327 // The UnwindHelp object is allocated at the start of the fixed object area
3328 int64_t UnwindHelpOffset = alignTo(CurrentOffset + 8, Align(16));
3329 assert(UnwindHelpOffset == getFixedObjectSize(MF, AFI, /*IsWin64*/ true,
3330 /*IsFunclet*/ false) &&
3331 "UnwindHelpOffset must be at the start of the fixed object area");
3332 int UnwindHelpFI = MFI.CreateFixedObject(/*Size*/ 8, -UnwindHelpOffset,
3333 /*IsImmutable=*/false);
3334 EHInfo.UnwindHelpFrameIdx = UnwindHelpFI;
3335
3336 MachineBasicBlock &MBB = MF.front();
3337 auto MBBI = MBB.begin();
3338 while (MBBI != MBB.end() && MBBI->getFlag(MachineInstr::FrameSetup))
3339 ++MBBI;
3340
3341 // We need to store -2 into the UnwindHelp object at the start of the
3342 // function.
3343 DebugLoc DL;
3344 RS->enterBasicBlockEnd(MBB);
3345 RS->backward(MBBI);
3346 Register DstReg = RS->FindUnusedReg(&AArch64::GPR64commonRegClass);
3347 assert(DstReg && "There must be a free register after frame setup");
3348 const AArch64InstrInfo &TII =
3349 *MF.getSubtarget<AArch64Subtarget>().getInstrInfo();
3350 BuildMI(MBB, MBBI, DL, TII.get(AArch64::MOVi64imm), DstReg).addImm(-2);
3351 BuildMI(MBB, MBBI, DL, TII.get(AArch64::STURXi))
3352 .addReg(DstReg, getKillRegState(true))
3353 .addFrameIndex(UnwindHelpFI)
3354 .addImm(0);
3355}
3356
3357namespace {
3358struct TagStoreInstr {
3360 int64_t Offset, Size;
3361 explicit TagStoreInstr(MachineInstr *MI, int64_t Offset, int64_t Size)
3362 : MI(MI), Offset(Offset), Size(Size) {}
3363};
3364
3365class TagStoreEdit {
3366 MachineFunction *MF;
3367 MachineBasicBlock *MBB;
3368 MachineRegisterInfo *MRI;
3369 // Tag store instructions that are being replaced.
3371 // Combined memref arguments of the above instructions.
3373
3374 // Replace allocation tags in [FrameReg + FrameRegOffset, FrameReg +
3375 // FrameRegOffset + Size) with the address tag of SP.
3376 Register FrameReg;
3377 StackOffset FrameRegOffset;
3378 int64_t Size;
3379 // If not std::nullopt, move FrameReg to (FrameReg + FrameRegUpdate) at the
3380 // end.
3381 std::optional<int64_t> FrameRegUpdate;
3382 // MIFlags for any FrameReg updating instructions.
3383 unsigned FrameRegUpdateFlags;
3384
3385 // Use zeroing instruction variants.
3386 bool ZeroData;
3387 DebugLoc DL;
3388
3389 void emitUnrolled(MachineBasicBlock::iterator InsertI);
3390 void emitLoop(MachineBasicBlock::iterator InsertI);
3391
3392public:
3393 TagStoreEdit(MachineBasicBlock *MBB, bool ZeroData)
3394 : MBB(MBB), ZeroData(ZeroData) {
3395 MF = MBB->getParent();
3396 MRI = &MF->getRegInfo();
3397 }
3398 // Add an instruction to be replaced. Instructions must be added in the
3399 // ascending order of Offset, and have to be adjacent.
3400 void addInstruction(TagStoreInstr I) {
3401 assert((TagStores.empty() ||
3402 TagStores.back().Offset + TagStores.back().Size == I.Offset) &&
3403 "Non-adjacent tag store instructions.");
3404 TagStores.push_back(I);
3405 }
3406 void clear() { TagStores.clear(); }
3407 // Emit equivalent code at the given location, and erase the current set of
3408 // instructions. May skip if the replacement is not profitable. May invalidate
3409 // the input iterator and replace it with a valid one.
3410 void emitCode(MachineBasicBlock::iterator &InsertI,
3411 const AArch64FrameLowering *TFI, bool TryMergeSPUpdate);
3412};
3413
3414void TagStoreEdit::emitUnrolled(MachineBasicBlock::iterator InsertI) {
3415 const AArch64InstrInfo *TII =
3416 MF->getSubtarget<AArch64Subtarget>().getInstrInfo();
3417
3418 const int64_t kMinOffset = -256 * 16;
3419 const int64_t kMaxOffset = 255 * 16;
3420
3421 Register BaseReg = FrameReg;
3422 int64_t BaseRegOffsetBytes = FrameRegOffset.getFixed();
3423 if (BaseRegOffsetBytes < kMinOffset ||
3424 BaseRegOffsetBytes + (Size - Size % 32) > kMaxOffset ||
3425 // BaseReg can be FP, which is not necessarily aligned to 16-bytes. In
3426 // that case, BaseRegOffsetBytes will not be aligned to 16 bytes, which
3427 // is required for the offset of ST2G.
3428 BaseRegOffsetBytes % 16 != 0) {
3429 Register ScratchReg = MRI->createVirtualRegister(&AArch64::GPR64RegClass);
3430 emitFrameOffset(*MBB, InsertI, DL, ScratchReg, BaseReg,
3431 StackOffset::getFixed(BaseRegOffsetBytes), TII);
3432 BaseReg = ScratchReg;
3433 BaseRegOffsetBytes = 0;
3434 }
3435
3436 MachineInstr *LastI = nullptr;
3437 while (Size) {
3438 int64_t InstrSize = (Size > 16) ? 32 : 16;
3439 unsigned Opcode =
3440 InstrSize == 16
3441 ? (ZeroData ? AArch64::STZGi : AArch64::STGi)
3442 : (ZeroData ? AArch64::STZ2Gi : AArch64::ST2Gi);
3443 assert(BaseRegOffsetBytes % 16 == 0);
3444 MachineInstr *I = BuildMI(*MBB, InsertI, DL, TII->get(Opcode))
3445 .addReg(AArch64::SP)
3446 .addReg(BaseReg)
3447 .addImm(BaseRegOffsetBytes / 16)
3448 .setMemRefs(CombinedMemRefs);
3449 // A store to [BaseReg, #0] should go last for an opportunity to fold the
3450 // final SP adjustment in the epilogue.
3451 if (BaseRegOffsetBytes == 0)
3452 LastI = I;
3453 BaseRegOffsetBytes += InstrSize;
3454 Size -= InstrSize;
3455 }
3456
3457 if (LastI)
3458 MBB->splice(InsertI, MBB, LastI);
3459}
3460
3461void TagStoreEdit::emitLoop(MachineBasicBlock::iterator InsertI) {
3462 const AArch64InstrInfo *TII =
3463 MF->getSubtarget<AArch64Subtarget>().getInstrInfo();
3464
3465 Register BaseReg = FrameRegUpdate
3466 ? FrameReg
3467 : MRI->createVirtualRegister(&AArch64::GPR64RegClass);
3468 Register SizeReg = MRI->createVirtualRegister(&AArch64::GPR64RegClass);
3469
3470 emitFrameOffset(*MBB, InsertI, DL, BaseReg, FrameReg, FrameRegOffset, TII);
3471
3472 int64_t LoopSize = Size;
3473 // If the loop size is not a multiple of 32, split off one 16-byte store at
3474 // the end to fold BaseReg update into.
3475 if (FrameRegUpdate && *FrameRegUpdate)
3476 LoopSize -= LoopSize % 32;
3477 MachineInstr *LoopI = BuildMI(*MBB, InsertI, DL,
3478 TII->get(ZeroData ? AArch64::STZGloop_wback
3479 : AArch64::STGloop_wback))
3480 .addDef(SizeReg)
3481 .addDef(BaseReg)
3482 .addImm(LoopSize)
3483 .addReg(BaseReg)
3484 .setMemRefs(CombinedMemRefs);
3485 if (FrameRegUpdate)
3486 LoopI->setFlags(FrameRegUpdateFlags);
3487
3488 int64_t ExtraBaseRegUpdate =
3489 FrameRegUpdate ? (*FrameRegUpdate - FrameRegOffset.getFixed() - Size) : 0;
3490 LLVM_DEBUG(dbgs() << "TagStoreEdit::emitLoop: LoopSize=" << LoopSize
3491 << ", Size=" << Size
3492 << ", ExtraBaseRegUpdate=" << ExtraBaseRegUpdate
3493 << ", FrameRegUpdate=" << FrameRegUpdate
3494 << ", FrameRegOffset.getFixed()="
3495 << FrameRegOffset.getFixed() << "\n");
3496 if (LoopSize < Size) {
3497 assert(FrameRegUpdate);
3498 assert(Size - LoopSize == 16);
3499 // Tag 16 more bytes at BaseReg and update BaseReg.
3500 int64_t STGOffset = ExtraBaseRegUpdate + 16;
3501 assert(STGOffset % 16 == 0 && STGOffset >= -4096 && STGOffset <= 4080 &&
3502 "STG immediate out of range");
3503 BuildMI(*MBB, InsertI, DL,
3504 TII->get(ZeroData ? AArch64::STZGPostIndex : AArch64::STGPostIndex))
3505 .addDef(BaseReg)
3506 .addReg(BaseReg)
3507 .addReg(BaseReg)
3508 .addImm(STGOffset / 16)
3509 .setMemRefs(CombinedMemRefs)
3510 .setMIFlags(FrameRegUpdateFlags);
3511 } else if (ExtraBaseRegUpdate) {
3512 // Update BaseReg.
3513 int64_t AddSubOffset = std::abs(ExtraBaseRegUpdate);
3514 assert(AddSubOffset <= 4095 && "ADD/SUB immediate out of range");
3515 BuildMI(
3516 *MBB, InsertI, DL,
3517 TII->get(ExtraBaseRegUpdate > 0 ? AArch64::ADDXri : AArch64::SUBXri))
3518 .addDef(BaseReg)
3519 .addReg(BaseReg)
3520 .addImm(AddSubOffset)
3521 .addImm(0)
3522 .setMIFlags(FrameRegUpdateFlags);
3523 }
3524}
3525
3526// Check if *II is a register update that can be merged into STGloop that ends
3527// at (Reg + Size). RemainingOffset is the required adjustment to Reg after the
3528// end of the loop.
3529bool canMergeRegUpdate(MachineBasicBlock::iterator II, unsigned Reg,
3530 int64_t Size, int64_t *TotalOffset) {
3531 MachineInstr &MI = *II;
3532 if ((MI.getOpcode() == AArch64::ADDXri ||
3533 MI.getOpcode() == AArch64::SUBXri) &&
3534 MI.getOperand(0).getReg() == Reg && MI.getOperand(1).getReg() == Reg) {
3535 unsigned Shift = AArch64_AM::getShiftValue(MI.getOperand(3).getImm());
3536 int64_t Offset = MI.getOperand(2).getImm() << Shift;
3537 if (MI.getOpcode() == AArch64::SUBXri)
3538 Offset = -Offset;
3539 int64_t PostOffset = Offset - Size;
3540 // TagStoreEdit::emitLoop might emit either an ADD/SUB after the loop, or
3541 // an STGPostIndex which does the last 16 bytes of tag write. Which one is
3542 // chosen depends on the alignment of the loop size, but the difference
3543 // between the valid ranges for the two instructions is small, so we
3544 // conservatively assume that it could be either case here.
3545 //
3546 // Max offset of STGPostIndex, minus the 16 byte tag write folded into that
3547 // instruction.
3548 const int64_t kMaxOffset = 4080 - 16;
3549 // Max offset of SUBXri.
3550 const int64_t kMinOffset = -4095;
3551 if (PostOffset <= kMaxOffset && PostOffset >= kMinOffset &&
3552 PostOffset % 16 == 0) {
3553 *TotalOffset = Offset;
3554 return true;
3555 }
3556 }
3557 return false;
3558}
3559
3560void mergeMemRefs(const SmallVectorImpl<TagStoreInstr> &TSE,
3562 MemRefs.clear();
3563 for (auto &TS : TSE) {
3564 MachineInstr *MI = TS.MI;
3565 // An instruction without memory operands may access anything. Be
3566 // conservative and return an empty list.
3567 if (MI->memoperands_empty()) {
3568 MemRefs.clear();
3569 return;
3570 }
3571 MemRefs.append(MI->memoperands_begin(), MI->memoperands_end());
3572 }
3573}
3574
3575void TagStoreEdit::emitCode(MachineBasicBlock::iterator &InsertI,
3576 const AArch64FrameLowering *TFI,
3577 bool TryMergeSPUpdate) {
3578 if (TagStores.empty())
3579 return;
3580 TagStoreInstr &FirstTagStore = TagStores[0];
3581 TagStoreInstr &LastTagStore = TagStores[TagStores.size() - 1];
3582 Size = LastTagStore.Offset - FirstTagStore.Offset + LastTagStore.Size;
3583 DL = TagStores[0].MI->getDebugLoc();
3584
3585 Register Reg;
3586 FrameRegOffset = TFI->resolveFrameOffsetReference(
3587 *MF, FirstTagStore.Offset, false /*isFixed*/,
3588 TargetStackID::Default /*StackID*/, Reg,
3589 /*PreferFP=*/false, /*ForSimm=*/true);
3590 FrameReg = Reg;
3591 FrameRegUpdate = std::nullopt;
3592
3593 mergeMemRefs(TagStores, CombinedMemRefs);
3594
3595 LLVM_DEBUG({
3596 dbgs() << "Replacing adjacent STG instructions:\n";
3597 for (const auto &Instr : TagStores) {
3598 dbgs() << " " << *Instr.MI;
3599 }
3600 });
3601
3602 // Size threshold where a loop becomes shorter than a linear sequence of
3603 // tagging instructions.
3604 const int kSetTagLoopThreshold = 176;
3605 if (Size < kSetTagLoopThreshold) {
3606 if (TagStores.size() < 2)
3607 return;
3608 emitUnrolled(InsertI);
3609 } else {
3610 MachineInstr *UpdateInstr = nullptr;
3611 int64_t TotalOffset = 0;
3612 if (TryMergeSPUpdate) {
3613 // See if we can merge base register update into the STGloop.
3614 // This is done in AArch64LoadStoreOptimizer for "normal" stores,
3615 // but STGloop is way too unusual for that, and also it only
3616 // realistically happens in function epilogue. Also, STGloop is expanded
3617 // before that pass.
3618 auto UpdateI = skipDebugInstructionsForward(InsertI, MBB->end(),
3619 /*SkipPseudoOp=*/false);
3620 if (UpdateI != MBB->end() &&
3621 canMergeRegUpdate(UpdateI, FrameReg, FrameRegOffset.getFixed() + Size,
3622 &TotalOffset)) {
3623 // Keep skipped debug instructions before the loop so debug values see
3624 // the pre-update SP; only the tag stores move past them.
3625 UpdateInstr = &*UpdateI++;
3626 InsertI = UpdateI;
3627 LLVM_DEBUG(dbgs() << "Folding SP update into loop:\n "
3628 << *UpdateInstr);
3629 }
3630 }
3631
3632 if (!UpdateInstr && TagStores.size() < 2)
3633 return;
3634
3635 if (UpdateInstr) {
3636 FrameRegUpdate = TotalOffset;
3637 FrameRegUpdateFlags = UpdateInstr->getFlags();
3638 }
3639 emitLoop(InsertI);
3640 if (UpdateInstr)
3641 UpdateInstr->eraseFromParent();
3642 }
3643
3644 for (auto &TS : TagStores)
3645 TS.MI->eraseFromParent();
3646}
3647
3648bool isMergeableStackTaggingInstruction(MachineInstr &MI, int64_t &Offset,
3649 int64_t &Size, bool &ZeroData) {
3650 MachineFunction &MF = *MI.getParent()->getParent();
3651 const MachineFrameInfo &MFI = MF.getFrameInfo();
3652
3653 unsigned Opcode = MI.getOpcode();
3654 ZeroData = (Opcode == AArch64::STZGloop || Opcode == AArch64::STZGi ||
3655 Opcode == AArch64::STZ2Gi);
3656
3657 if (Opcode == AArch64::STGloop || Opcode == AArch64::STZGloop) {
3658 if (!MI.getOperand(0).isDead() || !MI.getOperand(1).isDead())
3659 return false;
3660 if (!MI.getOperand(2).isImm() || !MI.getOperand(3).isFI())
3661 return false;
3662 Offset = MFI.getObjectOffset(MI.getOperand(3).getIndex());
3663 Size = MI.getOperand(2).getImm();
3664 return true;
3665 }
3666
3667 if (Opcode == AArch64::STGi || Opcode == AArch64::STZGi)
3668 Size = 16;
3669 else if (Opcode == AArch64::ST2Gi || Opcode == AArch64::STZ2Gi)
3670 Size = 32;
3671 else
3672 return false;
3673
3674 if (MI.getOperand(0).getReg() != AArch64::SP || !MI.getOperand(1).isFI())
3675 return false;
3676
3677 Offset = MFI.getObjectOffset(MI.getOperand(1).getIndex()) +
3678 16 * MI.getOperand(2).getImm();
3679 return true;
3680}
3681
3682static size_t countAvailableScavengerSlots(LivePhysRegs &LiveRegs,
3684 RegScavenger *RS) {
3685 auto FreeGPRs =
3686 llvm::count_if(AArch64::GPR64RegClass, [&LiveRegs, &MRI](auto Reg) {
3687 return LiveRegs.available(MRI, Reg);
3688 });
3689
3690 size_t NumEmergencySlots = 0;
3691 if (RS)
3692 NumEmergencySlots = RS->getNumScavengingFrameIndices();
3693
3694 return FreeGPRs + NumEmergencySlots;
3695}
3696
3697// Detect a run of memory tagging instructions for adjacent stack frame slots,
3698// and replace them with a shorter instruction sequence:
3699// * replace STG + STG with ST2G
3700// * replace STGloop + STGloop with STGloop
3701// This code needs to run when stack slot offsets are already known, but before
3702// FrameIndex operands in STG instructions are eliminated.
3704 const AArch64FrameLowering *TFI,
3705 RegScavenger *RS) {
3706 bool FirstZeroData;
3707 int64_t Size, Offset;
3708 MachineInstr &MI = *II;
3711 if (&MI == &MBB->instr_back())
3712 return II;
3713 if (!isMergeableStackTaggingInstruction(MI, Offset, Size, FirstZeroData))
3714 return II;
3715
3717 Instrs.emplace_back(&MI, Offset, Size);
3718
3719 constexpr int kScanLimit = 10;
3720 int Count = 0;
3722 NextI != E && Count < kScanLimit; ++NextI) {
3723 MachineInstr &MI = *NextI;
3724 bool ZeroData;
3725 int64_t Size, Offset;
3726 // Collect instructions that update memory tags with a FrameIndex operand
3727 // and (when applicable) constant size, and whose output registers are dead
3728 // (the latter is almost always the case in practice). Since these
3729 // instructions effectively have no inputs or outputs, we are free to skip
3730 // any non-aliasing instructions in between without tracking used registers.
3731 if (isMergeableStackTaggingInstruction(MI, Offset, Size, ZeroData)) {
3732 if (ZeroData != FirstZeroData)
3733 break;
3734 Instrs.emplace_back(&MI, Offset, Size);
3735 continue;
3736 }
3737
3738 // Only count non-transient, non-tagging instructions toward the scan
3739 // limit.
3740 if (!MI.isTransient())
3741 ++Count;
3742
3743 // Just in case, stop before the epilogue code starts.
3744 if (MI.getFlag(MachineInstr::FrameSetup) ||
3746 break;
3747
3748 // Reject anything that may alias the collected instructions.
3749 if (MI.mayLoadOrStore() || MI.hasUnmodeledSideEffects() || MI.isCall())
3750 break;
3751 }
3752
3753 // New code will be inserted after the last tagging instruction we've found.
3754 MachineBasicBlock::iterator InsertI = Instrs.back().MI;
3755
3756 // All the gathered stack tag instructions are merged and placed after
3757 // last tag store in the list. The check should be made if the nzcv
3758 // flag is live at the point where we are trying to insert. Otherwise
3759 // the nzcv flag might get clobbered if any stg loops are present.
3760
3761 // FIXME : This approach of bailing out from merge is conservative in
3762 // some ways like even if stg loops are not present after merge the
3763 // insert list, this liveness check is done (which is not needed).
3765 LiveRegs.addLiveOuts(*MBB);
3766 for (auto I = MBB->rbegin();; ++I) {
3767 MachineInstr &MI = *I;
3768 if (MI == InsertI)
3769 break;
3770 LiveRegs.stepBackward(*I);
3771 }
3772 InsertI++;
3773 if (LiveRegs.contains(AArch64::NZCV))
3774 return InsertI;
3775
3776 // Emitting an MTE loop requires two physical registers (BaseReg and
3777 // SizeReg). If the function is under register pressure, the register
3778 // scavenger will crash trying to allocate them. If we don't have at least
3779 // two free slots (free registers + emergency slots), bail out and fall back
3780 // to the unrolled sequence.
3781 if (countAvailableScavengerSlots(LiveRegs, MBB->getParent()->getRegInfo(),
3782 RS) < 2) {
3783 LLVM_DEBUG(
3784 dbgs() << "Failed to merge MTE stack tagging instructions into loop "
3785 << "due to high register pressure.\n");
3786 return InsertI;
3787 }
3788
3789 llvm::stable_sort(Instrs,
3790 [](const TagStoreInstr &Left, const TagStoreInstr &Right) {
3791 return Left.Offset < Right.Offset;
3792 });
3793
3794 // Make sure that we don't have any overlapping stores.
3795 int64_t CurOffset = Instrs[0].Offset;
3796 for (auto &Instr : Instrs) {
3797 if (CurOffset > Instr.Offset)
3798 return NextI;
3799 CurOffset = Instr.Offset + Instr.Size;
3800 }
3801
3802 // Find contiguous runs of tagged memory and emit shorter instruction
3803 // sequences for them when possible.
3804 TagStoreEdit TSE(MBB, FirstZeroData);
3805 std::optional<int64_t> EndOffset;
3806 for (auto &Instr : Instrs) {
3807 if (EndOffset && *EndOffset != Instr.Offset) {
3808 // Found a gap.
3809 TSE.emitCode(InsertI, TFI, /*TryMergeSPUpdate = */ false);
3810 TSE.clear();
3811 }
3812
3813 TSE.addInstruction(Instr);
3814 EndOffset = Instr.Offset + Instr.Size;
3815 }
3816
3817 const MachineFunction *MF = MBB->getParent();
3818 // Multiple FP/SP updates in a loop cannot be described by CFI instructions.
3819 TSE.emitCode(
3820 InsertI, TFI, /*TryMergeSPUpdate = */
3822
3823 return InsertI;
3824}
3825} // namespace
3826
3828 MachineFunction &MF, RegScavenger *RS = nullptr) const {
3829 bool MergeSetTag = MF.getSubtarget<AArch64Subtarget>()
3830 .getCLOpts()
3831 .stack_tagging_merge_settag;
3832 for (auto &BB : MF)
3833 for (MachineBasicBlock::iterator II = BB.begin(); II != BB.end();) {
3834 if (MergeSetTag)
3835 II = tryMergeAdjacentSTG(II, this, RS);
3836 }
3837
3838 // By the time this method is called, most of the prologue/epilogue code is
3839 // already emitted, whether its location was affected by the shrink-wrapping
3840 // optimization or not.
3841 if (!MF.getFunction().hasFnAttribute(Attribute::Naked) &&
3842 shouldSignReturnAddressEverywhere(MF))
3844}
3845
3846/// For Win64 AArch64 EH, the offset to the Unwind object is from the SP
3847/// before the update. This is easily retrieved as it is exactly the offset
3848/// that is set in processFunctionBeforeFrameFinalized.
3850 const MachineFunction &MF, int FI, Register &FrameReg,
3851 bool IgnoreSPUpdates) const {
3852 const MachineFrameInfo &MFI = MF.getFrameInfo();
3853 if (IgnoreSPUpdates) {
3854 LLVM_DEBUG(dbgs() << "Offset from the SP for " << FI << " is "
3855 << MFI.getObjectOffset(FI) << "\n");
3856 FrameReg = AArch64::SP;
3857 return StackOffset::getFixed(MFI.getObjectOffset(FI));
3858 }
3859
3860 // Go to common code if we cannot provide sp + offset.
3861 if (MFI.hasVarSizedObjects() ||
3864 return getFrameIndexReference(MF, FI, FrameReg);
3865
3866 FrameReg = AArch64::SP;
3867 return getStackOffset(MF, MFI.getObjectOffset(FI));
3868}
3869
3870/// The parent frame offset (aka dispFrame) is only used on X86_64 to retrieve
3871/// the parent's frame pointer
3873 const MachineFunction &MF) const {
3874 return 0;
3875}
3876
3877/// Funclets only need to account for space for the callee saved registers,
3878/// as the locals are accounted for in the parent's stack frame.
3880 const MachineFunction &MF) const {
3881 // This is the size of the pushed CSRs.
3882 unsigned CSSize =
3883 MF.getInfo<AArch64FunctionInfo>()->getCalleeSavedStackSize();
3884 // This is the amount of stack a funclet needs to allocate.
3885 return alignTo(CSSize + MF.getFrameInfo().getMaxCallFrameSize(),
3886 getStackAlign());
3887}
3888
3889namespace {
3890struct FrameObject {
3891 bool IsValid = false;
3892 // Index of the object in MFI.
3893 int ObjectIndex = 0;
3894 // Group ID this object belongs to.
3895 int GroupIndex = -1;
3896 // This object should be placed first (closest to SP).
3897 bool ObjectFirst = false;
3898 // This object's group (which always contains the object with
3899 // ObjectFirst==true) should be placed first.
3900 bool GroupFirst = false;
3901
3902 // Used to distinguish between FP and GPR accesses. The values are decided so
3903 // that they sort FPR < Hazard < GPR and they can be or'd together.
3904 unsigned Accesses = 0;
3905 enum { AccessFPR = 1, AccessHazard = 2, AccessGPR = 4 };
3906};
3907
3908class GroupBuilder {
3909 SmallVector<int, 8> CurrentMembers;
3910 int NextGroupIndex = 0;
3911 std::vector<FrameObject> &Objects;
3912
3913public:
3914 GroupBuilder(std::vector<FrameObject> &Objects) : Objects(Objects) {}
3915 void AddMember(int Index) { CurrentMembers.push_back(Index); }
3916 void EndCurrentGroup() {
3917 if (CurrentMembers.size() > 1) {
3918 // Create a new group with the current member list. This might remove them
3919 // from their pre-existing groups. That's OK, dealing with overlapping
3920 // groups is too hard and unlikely to make a difference.
3921 LLVM_DEBUG(dbgs() << "group:");
3922 for (int Index : CurrentMembers) {
3923 Objects[Index].GroupIndex = NextGroupIndex;
3924 LLVM_DEBUG(dbgs() << " " << Index);
3925 }
3926 LLVM_DEBUG(dbgs() << "\n");
3927 NextGroupIndex++;
3928 }
3929 CurrentMembers.clear();
3930 }
3931};
3932
3933bool FrameObjectCompare(const FrameObject &A, const FrameObject &B) {
3934 // Objects at a lower index are closer to FP; objects at a higher index are
3935 // closer to SP.
3936 //
3937 // For consistency in our comparison, all invalid objects are placed
3938 // at the end. This also allows us to stop walking when we hit the
3939 // first invalid item after it's all sorted.
3940 //
3941 // If we want to include a stack hazard region, order FPR accesses < the
3942 // hazard object < GPRs accesses in order to create a separation between the
3943 // two. For the Accesses field 1 = FPR, 2 = Hazard Object, 4 = GPR.
3944 //
3945 // Otherwise the "first" object goes first (closest to SP), followed by the
3946 // members of the "first" group.
3947 //
3948 // The rest are sorted by the group index to keep the groups together.
3949 // Higher numbered groups are more likely to be around longer (i.e. untagged
3950 // in the function epilogue and not at some earlier point). Place them closer
3951 // to SP.
3952 //
3953 // If all else equal, sort by the object index to keep the objects in the
3954 // original order.
3955 return std::make_tuple(!A.IsValid, A.Accesses, A.ObjectFirst, A.GroupFirst,
3956 A.GroupIndex, A.ObjectIndex) <
3957 std::make_tuple(!B.IsValid, B.Accesses, B.ObjectFirst, B.GroupFirst,
3958 B.GroupIndex, B.ObjectIndex);
3959}
3960} // namespace
3961
3963 const MachineFunction &MF, SmallVectorImpl<int> &ObjectsToAllocate) const {
3965
3966 if ((!MF.getSubtarget<AArch64Subtarget>().getCLOpts().order_frame_objects &&
3967 !AFI.hasSplitSVEObjects()) ||
3968 ObjectsToAllocate.empty())
3969 return;
3970
3971 const MachineFrameInfo &MFI = MF.getFrameInfo();
3972 std::vector<FrameObject> FrameObjects(MFI.getObjectIndexEnd());
3973 for (auto &Obj : ObjectsToAllocate) {
3974 FrameObjects[Obj].IsValid = true;
3975 FrameObjects[Obj].ObjectIndex = Obj;
3976 }
3977
3978 // Identify FPR vs GPR slots for hazards, and stack slots that are tagged at
3979 // the same time.
3980 GroupBuilder GB(FrameObjects);
3981 for (auto &MBB : MF) {
3982 for (auto &MI : MBB) {
3983 if (MI.isDebugInstr())
3984 continue;
3985
3986 if (AFI.hasStackHazardSlotIndex()) {
3987 std::optional<int> FI = getLdStFrameID(MI, MFI);
3988 if (FI && *FI >= 0 && *FI < (int)FrameObjects.size()) {
3989 if (MFI.getStackID(*FI) == TargetStackID::ScalableVector ||
3991 FrameObjects[*FI].Accesses |= FrameObject::AccessFPR;
3992 else
3993 FrameObjects[*FI].Accesses |= FrameObject::AccessGPR;
3994 }
3995 }
3996
3997 int OpIndex;
3998 switch (MI.getOpcode()) {
3999 case AArch64::STGloop:
4000 case AArch64::STZGloop:
4001 OpIndex = 3;
4002 break;
4003 case AArch64::STGi:
4004 case AArch64::STZGi:
4005 case AArch64::ST2Gi:
4006 case AArch64::STZ2Gi:
4007 OpIndex = 1;
4008 break;
4009 default:
4010 OpIndex = -1;
4011 }
4012
4013 int TaggedFI = -1;
4014 if (OpIndex >= 0) {
4015 const MachineOperand &MO = MI.getOperand(OpIndex);
4016 if (MO.isFI()) {
4017 int FI = MO.getIndex();
4018 if (FI >= 0 && FI < MFI.getObjectIndexEnd() &&
4019 FrameObjects[FI].IsValid)
4020 TaggedFI = FI;
4021 }
4022 }
4023
4024 // If this is a stack tagging instruction for a slot that is not part of a
4025 // group yet, either start a new group or add it to the current one.
4026 if (TaggedFI >= 0)
4027 GB.AddMember(TaggedFI);
4028 else
4029 GB.EndCurrentGroup();
4030 }
4031 // Groups should never span multiple basic blocks.
4032 GB.EndCurrentGroup();
4033 }
4034
4035 if (AFI.hasStackHazardSlotIndex()) {
4036 FrameObjects[AFI.getStackHazardSlotIndex()].Accesses =
4037 FrameObject::AccessHazard;
4038 // If a stack object is unknown or both GPR and FPR, sort it into GPR.
4039 for (auto &Obj : FrameObjects)
4040 if (!Obj.Accesses ||
4041 Obj.Accesses == (FrameObject::AccessGPR | FrameObject::AccessFPR))
4042 Obj.Accesses = FrameObject::AccessGPR;
4043 }
4044
4045 // If the function's tagged base pointer is pinned to a stack slot, we want to
4046 // put that slot first when possible. This will likely place it at SP + 0,
4047 // and save one instruction when generating the base pointer because IRG does
4048 // not allow an immediate offset.
4049 std::optional<int> TBPI = AFI.getTaggedBasePointerIndex();
4050 if (TBPI) {
4051 FrameObjects[*TBPI].ObjectFirst = true;
4052 FrameObjects[*TBPI].GroupFirst = true;
4053 int FirstGroupIndex = FrameObjects[*TBPI].GroupIndex;
4054 if (FirstGroupIndex >= 0)
4055 for (FrameObject &Object : FrameObjects)
4056 if (Object.GroupIndex == FirstGroupIndex)
4057 Object.GroupFirst = true;
4058 }
4059
4060 llvm::stable_sort(FrameObjects, FrameObjectCompare);
4061
4062 int i = 0;
4063 for (auto &Obj : FrameObjects) {
4064 // All invalid items are sorted at the end, so it's safe to stop.
4065 if (!Obj.IsValid)
4066 break;
4067 ObjectsToAllocate[i++] = Obj.ObjectIndex;
4068 }
4069
4070 LLVM_DEBUG({
4071 dbgs() << "Final frame order:\n";
4072 for (auto &Obj : FrameObjects) {
4073 if (!Obj.IsValid)
4074 break;
4075 dbgs() << " " << Obj.ObjectIndex << ": group " << Obj.GroupIndex;
4076 if (Obj.ObjectFirst)
4077 dbgs() << ", first";
4078 if (Obj.GroupFirst)
4079 dbgs() << ", group-first";
4080 dbgs() << "\n";
4081 }
4082 });
4083}
4084
4085/// Emit a loop to decrement SP until it is equal to TargetReg, with probes at
4086/// least every ProbeSize bytes. Returns an iterator of the first instruction
4087/// after the loop. The difference between SP and TargetReg must be an exact
4088/// multiple of ProbeSize.
4090AArch64FrameLowering::inlineStackProbeLoopExactMultiple(
4091 MachineBasicBlock::iterator MBBI, int64_t ProbeSize,
4092 Register TargetReg) const {
4093 MachineBasicBlock &MBB = *MBBI->getParent();
4094 MachineFunction &MF = *MBB.getParent();
4095 const AArch64InstrInfo *TII =
4096 MF.getSubtarget<AArch64Subtarget>().getInstrInfo();
4097 DebugLoc DL = MBB.findDebugLoc(MBBI);
4098
4099 MachineFunction::iterator MBBInsertPoint = std::next(MBB.getIterator());
4100 MachineBasicBlock *LoopMBB = MF.CreateMachineBasicBlock(MBB.getBasicBlock());
4101 MF.insert(MBBInsertPoint, LoopMBB);
4102 MachineBasicBlock *ExitMBB = MF.CreateMachineBasicBlock(MBB.getBasicBlock());
4103 MF.insert(MBBInsertPoint, ExitMBB);
4104
4105 // SUB SP, SP, #ProbeSize (or equivalent if ProbeSize is not encodable
4106 // in SUB).
4107 emitFrameOffset(*LoopMBB, LoopMBB->end(), DL, AArch64::SP, AArch64::SP,
4108 StackOffset::getFixed(-ProbeSize), TII,
4110 // LDR XZR, [SP]
4111 BuildMI(*LoopMBB, LoopMBB->end(), DL, TII->get(AArch64::LDRXui))
4112 .addDef(AArch64::XZR)
4113 .addReg(AArch64::SP)
4114 .addImm(0)
4118 Align(8)))
4120 // CMP SP, TargetReg
4121 BuildMI(*LoopMBB, LoopMBB->end(), DL, TII->get(AArch64::SUBSXrx64),
4122 AArch64::XZR)
4123 .addReg(AArch64::SP)
4124 .addReg(TargetReg)
4127 // B.CC Loop
4128 BuildMI(*LoopMBB, LoopMBB->end(), DL, TII->get(AArch64::Bcc))
4130 .addMBB(LoopMBB)
4132
4133 LoopMBB->addSuccessor(ExitMBB);
4134 LoopMBB->addSuccessor(LoopMBB);
4135 // Synthesize the exit MBB.
4136 ExitMBB->splice(ExitMBB->end(), &MBB, MBBI, MBB.end());
4138 MBB.addSuccessor(LoopMBB);
4139 // Update liveins.
4140 fullyRecomputeLiveIns({ExitMBB, LoopMBB});
4141
4142 return ExitMBB->begin();
4143}
4144
4145void AArch64FrameLowering::inlineStackProbeFixed(
4146 MachineBasicBlock::iterator MBBI, Register ScratchReg, int64_t FrameSize,
4147 StackOffset CFAOffset) const {
4148 MachineBasicBlock *MBB = MBBI->getParent();
4149 MachineFunction &MF = *MBB->getParent();
4150 const AArch64InstrInfo *TII =
4151 MF.getSubtarget<AArch64Subtarget>().getInstrInfo();
4152 AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
4153 bool EmitAsyncCFI = AFI->needsAsyncDwarfUnwindInfo(MF);
4154 bool HasFP = hasFP(MF);
4155
4156 DebugLoc DL;
4157 int64_t ProbeSize = MF.getInfo<AArch64FunctionInfo>()->getStackProbeSize();
4158 int64_t NumBlocks = FrameSize / ProbeSize;
4159 int64_t ResidualSize = FrameSize % ProbeSize;
4160
4161 LLVM_DEBUG(dbgs() << "Stack probing: total " << FrameSize << " bytes, "
4162 << NumBlocks << " blocks of " << ProbeSize
4163 << " bytes, plus " << ResidualSize << " bytes\n");
4164
4165 // Decrement SP by NumBlock * ProbeSize bytes, with either unrolled or
4166 // ordinary loop.
4167 if (NumBlocks <= AArch64::StackProbeMaxLoopUnroll) {
4168 for (int i = 0; i < NumBlocks; ++i) {
4169 // SUB SP, SP, #ProbeSize (or equivalent if ProbeSize is not
4170 // encodable in a SUB).
4171 emitFrameOffset(*MBB, MBBI, DL, AArch64::SP, AArch64::SP,
4172 StackOffset::getFixed(-ProbeSize), TII,
4173 MachineInstr::FrameSetup, false, false, nullptr,
4174 EmitAsyncCFI && !HasFP, CFAOffset);
4175 CFAOffset += StackOffset::getFixed(ProbeSize);
4176 // LDR XZR, [SP]
4177 BuildMI(*MBB, MBBI, DL, TII->get(AArch64::LDRXui))
4178 .addDef(AArch64::XZR)
4179 .addReg(AArch64::SP)
4180 .addImm(0)
4184 Align(8)))
4186 }
4187 } else if (NumBlocks != 0) {
4188 // SUB ScratchReg, SP, #FrameSize (or equivalent if FrameSize is not
4189 // encodable in ADD). ScrathReg may temporarily become the CFA register.
4190 emitFrameOffset(*MBB, MBBI, DL, ScratchReg, AArch64::SP,
4191 StackOffset::getFixed(-ProbeSize * NumBlocks), TII,
4192 MachineInstr::FrameSetup, false, false, nullptr,
4193 EmitAsyncCFI && !HasFP, CFAOffset);
4194 CFAOffset += StackOffset::getFixed(ProbeSize * NumBlocks);
4195 MBBI = inlineStackProbeLoopExactMultiple(MBBI, ProbeSize, ScratchReg);
4196 MBB = MBBI->getParent();
4197 if (EmitAsyncCFI && !HasFP) {
4198 // Set the CFA register back to SP.
4199 CFIInstBuilder(*MBB, MBBI, MachineInstr::FrameSetup)
4200 .buildDefCFARegister(AArch64::SP);
4201 }
4202 }
4203
4204 if (ResidualSize != 0) {
4205 // SUB SP, SP, #ResidualSize (or equivalent if ResidualSize is not encodable
4206 // in SUB).
4207 emitFrameOffset(*MBB, MBBI, DL, AArch64::SP, AArch64::SP,
4208 StackOffset::getFixed(-ResidualSize), TII,
4209 MachineInstr::FrameSetup, false, false, nullptr,
4210 EmitAsyncCFI && !HasFP, CFAOffset);
4211 if (ResidualSize > AArch64::StackProbeMaxUnprobedStack) {
4212 // LDR XZR, [SP]
4213 BuildMI(*MBB, MBBI, DL, TII->get(AArch64::LDRXui))
4214 .addDef(AArch64::XZR)
4215 .addReg(AArch64::SP)
4216 .addImm(0)
4220 Align(8)))
4222 }
4223 }
4224}
4225
4226void AArch64FrameLowering::inlineStackProbe(MachineFunction &MF,
4227 MachineBasicBlock &MBB) const {
4228 // Get the instructions that need to be replaced. We emit at most two of
4229 // these. Remember them in order to avoid complications coming from the need
4230 // to traverse the block while potentially creating more blocks.
4231 SmallVector<MachineInstr *, 4> ToReplace;
4232 for (MachineInstr &MI : MBB)
4233 if (MI.getOpcode() == AArch64::PROBED_STACKALLOC ||
4234 MI.getOpcode() == AArch64::PROBED_STACKALLOC_VAR)
4235 ToReplace.push_back(&MI);
4236
4237 for (MachineInstr *MI : ToReplace) {
4238 if (MI->getOpcode() == AArch64::PROBED_STACKALLOC) {
4239 Register ScratchReg = MI->getOperand(0).getReg();
4240 int64_t FrameSize = MI->getOperand(1).getImm();
4241 StackOffset CFAOffset = StackOffset::get(MI->getOperand(2).getImm(),
4242 MI->getOperand(3).getImm());
4243 inlineStackProbeFixed(MI->getIterator(), ScratchReg, FrameSize,
4244 CFAOffset);
4245 } else {
4246 assert(MI->getOpcode() == AArch64::PROBED_STACKALLOC_VAR &&
4247 "Stack probe pseudo-instruction expected");
4248 const AArch64InstrInfo *TII =
4249 MI->getMF()->getSubtarget<AArch64Subtarget>().getInstrInfo();
4250 Register TargetReg = MI->getOperand(0).getReg();
4251 (void)TII->probedStackAlloc(MI->getIterator(), TargetReg, true);
4252 }
4253 MI->eraseFromParent();
4254 }
4255}
4256
4259 NotAccessed = 0, // Stack object not accessed by load/store instructions.
4260 GPR = 1 << 0, // A general purpose register.
4261 PPR = 1 << 1, // A predicate register.
4262 FPR = 1 << 2, // A floating point/Neon/SVE register.
4263 };
4264
4265 int Idx;
4267 int64_t Size;
4268 unsigned AccessTypes;
4269
4271
4272 bool operator<(const StackAccess &Rhs) const {
4273 return std::make_tuple(start(), Idx) <
4274 std::make_tuple(Rhs.start(), Rhs.Idx);
4275 }
4276
4277 bool isCPU() const {
4278 // Predicate register load and store instructions execute on the CPU.
4280 }
4281 bool isSME() const { return AccessTypes & AccessType::FPR; }
4282 bool isMixed() const { return isCPU() && isSME(); }
4283
4284 int64_t start() const { return Offset.getFixed() + Offset.getScalable(); }
4285 int64_t end() const { return start() + Size; }
4286
4287 std::string getTypeString() const {
4288 switch (AccessTypes) {
4289 case AccessType::FPR:
4290 return "FPR";
4291 case AccessType::PPR:
4292 return "PPR";
4293 case AccessType::GPR:
4294 return "GPR";
4296 return "NA";
4297 default:
4298 return "Mixed";
4299 }
4300 }
4301
4302 void print(raw_ostream &OS) const {
4303 OS << getTypeString() << " stack object at [SP"
4304 << (Offset.getFixed() < 0 ? "" : "+") << Offset.getFixed();
4305 if (Offset.getScalable())
4306 OS << (Offset.getScalable() < 0 ? "" : "+") << Offset.getScalable()
4307 << " * vscale";
4308 OS << "]";
4309 }
4310};
4311
4312static inline raw_ostream &operator<<(raw_ostream &OS, const StackAccess &SA) {
4313 SA.print(OS);
4314 return OS;
4315}
4316
4317void AArch64FrameLowering::emitRemarks(
4318 const MachineFunction &MF, MachineOptimizationRemarkEmitter *ORE) const {
4319
4320 auto *AFI = MF.getInfo<AArch64FunctionInfo>();
4322 return;
4323
4324 uint64_t HazardSize = getStackHazardSize(MF);
4325 if (!HazardSize)
4326 HazardSize = MF.getSubtarget<AArch64Subtarget>()
4327 .getCLOpts()
4328 .stack_hazard_remark_size;
4329
4330 if (HazardSize == 0)
4331 return;
4332
4333 const MachineFrameInfo &MFI = MF.getFrameInfo();
4334 // Bail if function has no stack objects.
4335 if (!MFI.hasStackObjects())
4336 return;
4337
4338 std::vector<StackAccess> StackAccesses(MFI.getNumObjects());
4339
4340 size_t NumFPLdSt = 0;
4341 size_t NumNonFPLdSt = 0;
4342
4343 // Collect stack accesses via Load/Store instructions.
4344 for (const MachineBasicBlock &MBB : MF) {
4345 for (const MachineInstr &MI : MBB) {
4346 if (!MI.mayLoadOrStore() || MI.getNumMemOperands() < 1)
4347 continue;
4348 for (MachineMemOperand *MMO : MI.memoperands()) {
4349 std::optional<int> FI = getMMOFrameID(MMO, MFI);
4350 if (FI && !MFI.isDeadObjectIndex(*FI)) {
4351 int FrameIdx = *FI;
4352
4353 size_t ArrIdx = FrameIdx + MFI.getNumFixedObjects();
4354 if (StackAccesses[ArrIdx].AccessTypes == StackAccess::NotAccessed) {
4355 StackAccesses[ArrIdx].Idx = FrameIdx;
4356 StackAccesses[ArrIdx].Offset =
4357 getFrameIndexReferenceFromSP(MF, FrameIdx);
4358 StackAccesses[ArrIdx].Size = MFI.getObjectSize(FrameIdx);
4359 }
4360
4361 unsigned RegTy = StackAccess::AccessType::GPR;
4362 if (MFI.hasScalableStackID(FrameIdx))
4365 RegTy = StackAccess::FPR;
4366
4367 StackAccesses[ArrIdx].AccessTypes |= RegTy;
4368
4369 if (RegTy == StackAccess::FPR)
4370 ++NumFPLdSt;
4371 else
4372 ++NumNonFPLdSt;
4373 }
4374 }
4375 }
4376 }
4377
4378 if (NumFPLdSt == 0 || NumNonFPLdSt == 0)
4379 return;
4380
4381 llvm::sort(StackAccesses);
4382 llvm::erase_if(StackAccesses, [](const StackAccess &S) {
4384 });
4385
4388
4389 if (StackAccesses.front().isMixed())
4390 MixedObjects.push_back(&StackAccesses.front());
4391
4392 for (auto It = StackAccesses.begin(), End = std::prev(StackAccesses.end());
4393 It != End; ++It) {
4394 const auto &First = *It;
4395 const auto &Second = *(It + 1);
4396
4397 if (Second.isMixed())
4398 MixedObjects.push_back(&Second);
4399
4400 if ((First.isSME() && Second.isCPU()) ||
4401 (First.isCPU() && Second.isSME())) {
4402 uint64_t Distance = static_cast<uint64_t>(Second.start() - First.end());
4403 if (Distance < HazardSize)
4404 HazardPairs.emplace_back(&First, &Second);
4405 }
4406 }
4407
4408 auto EmitRemark = [&](llvm::StringRef Str) {
4409 ORE->emit([&]() {
4410 auto R = MachineOptimizationRemarkAnalysis(
4411 "sme", "StackHazard", MF.getFunction().getSubprogram(), &MF.front());
4412 return R << formatv("stack hazard in '{0}': ", MF.getName()).str() << Str;
4413 });
4414 };
4415
4416 for (const auto &P : HazardPairs)
4417 EmitRemark(formatv("{0} is too close to {1}", *P.first, *P.second).str());
4418
4419 for (const auto *Obj : MixedObjects)
4420 EmitRemark(
4421 formatv("{0} accessed by both GP and FP instructions", *Obj).str());
4422}
static void getLiveRegsForEntryMBB(LivePhysRegs &LiveRegs, const MachineBasicBlock &MBB)
static const unsigned DefaultSafeSPDisplacement
This is the biggest offset to the stack pointer we can encode in aarch64 instructions (without using ...
static RegState getPrologueDeath(MachineFunction &MF, unsigned Reg)
static bool produceCompactUnwindFrame(const AArch64FrameLowering &, MachineFunction &MF)
static bool invalidateWindowsRegisterPairing(bool SpillExtendedVolatile, int PairOffset, unsigned Reg1, unsigned Reg2, bool NeedsWinCFI, const TargetRegisterInfo *TRI)
bool enableMultiVectorSpillFill(const AArch64Subtarget &Subtarget, MachineFunction &MF)
static std::optional< int > getLdStFrameID(const MachineInstr &MI, const MachineFrameInfo &MFI)
static void orderZPRCalleeSavesForGroups(MachineFunction &MF, const TargetRegisterInfo *RegInfo, std::vector< CalleeSavedInfo > &CSI)
void computeCalleeSaveRegisterGroups(const AArch64FrameLowering &AFL, MachineFunction &MF, ArrayRef< CalleeSavedInfo > CSI, const TargetRegisterInfo *TRI, SmallVectorImpl< RegGroupInfo > &RegGroups, bool NeedsFrameRecord)
static bool invalidateRegisterPairing(bool SpillExtendedVolatile, int PairOffset, unsigned Reg1, unsigned Reg2, bool UsesWinAAPCS, bool NeedsWinCFI, bool NeedsFrameRecord, const TargetRegisterInfo *TRI)
Returns true if Reg1 and Reg2 cannot be paired using a ldp/stp instruction.
static bool isLikelyToHaveSVEStack(const AArch64FrameLowering &AFL, const MachineFunction &MF)
static SVEStackSizes determineSVEStackSizes(MachineFunction &MF, AssignObjectOffsets AssignOffsets)
Process all the SVE stack objects and the SVE stack size and offsets for each object.
static bool isTargetWindows(const MachineFunction &MF)
static unsigned estimateRSStackSizeLimit(MachineFunction &MF)
Look at each instruction that references stack frames and return the stack size limit beyond which so...
static bool getSVECalleeSaveSlotRange(const MachineFrameInfo &MFI, int &Min, int &Max)
returns true if there are any SVE callee saves.
static MCRegister getRegisterOrZero(MCRegister Reg, bool HasSVE)
static unsigned getStackHazardSize(const MachineFunction &MF)
static bool isValidMemOpOffset(const AArch64InstrInfo *TII, unsigned Opcode, int Offset)
MCRegister findFreePredicateReg(BitVector &SavedRegs)
static bool isPPRAccess(const MachineInstr &MI)
static std::optional< int > getMMOFrameID(MachineMemOperand *MMO, const MachineFrameInfo &MFI)
const AArch64Subtarget & Subtarget
const TargetLowering & TLI
assert(UImm &&(UImm !=~static_cast< T >(0)) &&"Invalid immediate!")
This file contains the declaration of the AArch64PrologueEmitter and AArch64EpilogueEmitter classes,...
static const int kSetTagLoopThreshold
const AArch64InstrInfo & TII
uint64_t uint64_t
uint64_t Imm
AMDGPU Reserve WWM Registers
MachineBasicBlock & MBB
MachineBasicBlock MachineBasicBlock::iterator DebugLoc DL
MachineBasicBlock MachineBasicBlock::iterator MBBI
This file contains the simple types necessary to represent the attributes associated with functions a...
#define CASE(ATTRNAME, AANAME,...)
static GCRegistry::Add< ErlangGC > A("erlang", "erlang-compatible garbage collector")
static GCRegistry::Add< CoreCLRGC > E("coreclr", "CoreCLR-compatible GC")
static GCRegistry::Add< OcamlGC > B("ocaml", "ocaml 3.10-compatible GC")
DXIL Forward Handle Accesses
IRTranslator LLVM IR MI
Module.h This file contains the declarations for the Module class.
static std::string getTypeString(Type *T)
Definition LLParser.cpp:68
This file implements the LivePhysRegs utility for tracking liveness of physical registers.
#define F(x, y, z)
Definition MD5.cpp:54
#define I(x, y, z)
Definition MD5.cpp:57
#define H(x, y, z)
Definition MD5.cpp:56
Register Reg
Register const TargetRegisterInfo * TRI
Promote Memory to Register
Definition Mem2Reg.cpp:110
uint64_t IntrinsicInst * II
#define P(N)
RISCVCC::CondCode CC
MachineRegisterInfo & MRI
This file declares the machine register scavenger class.
static bool isValid(const char C)
Returns true if C is a valid mangled character: <0-9a-zA-Z_>.
static bool contains(SmallPtrSetImpl< ConstantExpr * > &Cache, ConstantExpr *Expr, Constant *C)
Definition Value.cpp:479
This file defines the scope_exit class, which executes user-defined cleanup logic at scope exit.
This file defines the SmallVector class.
#define LLVM_DEBUG(...)
Definition Debug.h:119
StackOffset getSVEStackSize(const MachineFunction &MF) const
Returns the size of the entire SVE stackframe (PPRs + ZPRs).
StackOffset getZPRStackSize(const MachineFunction &MF) const
Returns the size of the entire ZPR stackframe (calleesaves + spills).
void processFunctionBeforeFrameIndicesReplaced(MachineFunction &MF, RegScavenger *RS) const override
processFunctionBeforeFrameIndicesReplaced - This method is called immediately before MO_FrameIndex op...
MachineBasicBlock::iterator eliminateCallFramePseudoInstr(MachineFunction &MF, MachineBasicBlock &MBB, MachineBasicBlock::iterator I) const override
This method is called during prolog/epilog code insertion to eliminate call frame setup and destroy p...
bool canUseAsPrologue(const MachineBasicBlock &MBB) const override
Check whether or not the given MBB can be used as a prologue for the target.
bool enableStackSlotScavenging(const MachineFunction &MF) const override
Returns true if the stack slot holes in the fixed and callee-save stack area should be used when allo...
bool assignCalleeSavedSpillSlots(MachineFunction &MF, const TargetRegisterInfo *TRI, std::vector< CalleeSavedInfo > &CSI) const override
assignCalleeSavedSpillSlots - Allows target to override spill slot assignment logic.
bool spillCalleeSavedRegisters(MachineBasicBlock &MBB, MachineBasicBlock::iterator MI, ArrayRef< CalleeSavedInfo > CSI, const TargetRegisterInfo *TRI) const override
spillCalleeSavedRegisters - Issues instruction(s) to spill all callee saved registers and returns tru...
bool restoreCalleeSavedRegisters(MachineBasicBlock &MBB, MachineBasicBlock::iterator MI, MutableArrayRef< CalleeSavedInfo > CSI, const TargetRegisterInfo *TRI) const override
restoreCalleeSavedRegisters - Issues instruction(s) to restore all callee saved registers and returns...
bool enableFullCFIFixup(const MachineFunction &MF) const override
enableFullCFIFixup - Returns true if we may need to fix the unwind information such that it is accura...
StackOffset getFrameIndexReferenceFromSP(const MachineFunction &MF, int FI) const override
getFrameIndexReferenceFromSP - This method returns the offset from the stack pointer to the slot of t...
bool enableCFIFixup(const MachineFunction &MF) const override
Returns true if we may need to fix the unwind information for the function.
StackOffset getNonLocalFrameIndexReference(const MachineFunction &MF, int FI) const override
getNonLocalFrameIndexReference - This method returns the offset used to reference a frame index locat...
TargetStackID::Value getStackIDForScalableVectors() const override
Returns the StackID that scalable vectors should be associated with.
bool hasFPImpl(const MachineFunction &MF) const override
hasFPImpl - Return true if the specified function should have a dedicated frame pointer register.
void emitPrologue(MachineFunction &MF, MachineBasicBlock &MBB) const override
emitProlog/emitEpilog - These methods insert prolog and epilog code into the function.
void resetCFIToInitialState(MachineBasicBlock &MBB) const override
Emit CFI instructions that recreate the state of the unwind information upon function entry.
bool hasReservedCallFrame(const MachineFunction &MF) const override
hasReservedCallFrame - Under normal circumstances, when a frame pointer is not required,...
bool hasSVECalleeSavesAboveFrameRecord(const MachineFunction &MF) const
StackOffset resolveFrameOffsetReference(const MachineFunction &MF, int64_t ObjectOffset, bool isFixed, TargetStackID::Value StackID, Register &FrameReg, bool PreferFP, bool ForSimm) const
bool canUseRedZone(const MachineFunction &MF) const
Can this function use the red zone for local allocations.
bool isFPReserved(const MachineFunction &MF) const
Should the Frame Pointer be reserved for the current function?
void processFunctionBeforeFrameFinalized(MachineFunction &MF, RegScavenger *RS) const override
processFunctionBeforeFrameFinalized - This method is called immediately before the specified function...
int getSEHFrameIndexOffset(const MachineFunction &MF, int FI) const
unsigned getWinEHFuncletFrameSize(const MachineFunction &MF) const
Funclets only need to account for space for the callee saved registers, as the locals are accounted f...
void orderFrameObjects(const MachineFunction &MF, SmallVectorImpl< int > &ObjectsToAllocate) const override
Order the symbols in the local stack frame.
void emitEpilogue(MachineFunction &MF, MachineBasicBlock &MBB) const override
StackOffset getPPRStackSize(const MachineFunction &MF) const
Returns the size of the entire PPR stackframe (calleesaves + spills + hazard padding).
int64_t getArgumentStackToRestore(MachineFunction &MF, MachineBasicBlock &MBB) const
Returns how much of the incoming argument stack area (in bytes) we should clean up in an epilogue.
void determineCalleeSaves(MachineFunction &MF, BitVector &SavedRegs, RegScavenger *RS) const override
This method determines which of the registers reported by TargetRegisterInfo::getCalleeSavedRegs() sh...
StackOffset getFrameIndexReference(const MachineFunction &MF, int FI, Register &FrameReg) const override
getFrameIndexReference - Provide a base+offset reference to an FI slot for debug info.
bool authenticatesReturnAddressEarly(MachineFunction &MF, MachineBasicBlock &MBB) const
Whether the return address is authenticated before the epilogue starts, in the return block MBB,...
bool enableWindowsCFIShrinkWrapping(const MachineFunction &MF) const override
Returns true if Windows CFI shrink wrapping should be enabled by default.
StackOffset getFrameIndexReferencePreferSP(const MachineFunction &MF, int FI, Register &FrameReg, bool IgnoreSPUpdates) const override
For Win64 AArch64 EH, the offset to the Unwind object is from the SP before the update.
StackOffset resolveFrameIndexReference(const MachineFunction &MF, int FI, Register &FrameReg, bool PreferFP, bool ForSimm) const
unsigned getWinEHParentFrameOffset(const MachineFunction &MF) const override
The parent frame offset (aka dispFrame) is only used on X86_64 to retrieve the parent's frame pointer...
bool requiresSaveVG(const MachineFunction &MF) const
void emitPacRetPlusLeafHardening(MachineFunction &MF) const
Harden the entire function with pac-ret.
AArch64FunctionInfo - This class is derived from MachineFunctionInfo and contains private AArch64-spe...
unsigned getCalleeSavedStackSize(const MachineFrameInfo &MFI) const
void setCalleeSaveBaseToFrameRecordOffset(int Offset)
SignReturnAddress getSignReturnAddressCondition() const
void setStackSizeSVE(uint64_t ZPR, uint64_t PPR)
std::optional< int > getTaggedBasePointerIndex() const
bool needsDwarfUnwindInfo(const MachineFunction &MF) const
static bool shouldSignReturnAddress(SignReturnAddress Condition, bool IsLRSpilled)
void setSVECalleeSavedStackSize(unsigned ZPR, unsigned PPR)
bool needsAsyncDwarfUnwindInfo(const MachineFunction &MF) const
static bool isTailCallReturnInst(const MachineInstr &MI)
Returns true if MI is one of the TCRETURN* instructions.
static bool isFpOrNEON(Register Reg)
Returns whether the physical register is FP or NEON.
const AArch64RegisterInfo * getRegisterInfo() const override
const AArch64InstrInfo * getInstrInfo() const override
const AArch64Options & getCLOpts() const
bool isSVEorStreamingSVEAvailable() const
Returns true if the target has access to either the full range of SVE instructions,...
bool isTargetWindowsOrUEFI() const
UEFI follows the Windows ARM64 ABI and produces PE/COFF images, so most of the Windows specific ABI h...
Represent a constant reference to an array (0 or more elements consecutively in memory),...
Definition ArrayRef.h:40
size_t size() const
Get the array size.
Definition ArrayRef.h:141
bool empty() const
Check if the array is empty.
Definition ArrayRef.h:136
bool test(unsigned Idx) const
Returns true if bit Idx is set.
Definition BitVector.h:483
BitVector & reset()
Reset all bits in the bitvector.
Definition BitVector.h:410
size_type count() const
Returns the number of bits which are set.
Definition BitVector.h:182
BitVector & set()
Set all bits in the bitvector.
Definition BitVector.h:367
iterator_range< const_set_bits_iterator > set_bits() const
Definition BitVector.h:160
size_type size() const
Returns the number of bits in this bitvector.
Definition BitVector.h:179
Helper class for creating CFI instructions and inserting them into MIR.
The CalleeSavedInfo class tracks the information need to locate where a callee saved register is in t...
A debug info location.
Definition DebugLoc.h:126
bool hasMinSize() const
Optimize this function for minimum size (-Oz).
Definition Function.h:705
CallingConv::ID getCallingConv() const
getCallingConv()/setCallingConv(CC) - These method get and set the calling convention of this functio...
Definition Function.h:273
AttributeList getAttributes() const
Return the attribute list for this Function.
Definition Function.h:329
bool isVarArg() const
isVarArg - Return true if this function takes a variable number of arguments.
Definition Function.h:230
bool hasFnAttribute(Attribute::AttrKind Kind) const
Return true if the function has the attribute.
Definition Function.cpp:731
Module * getParent()
Get the module that this global value is contained inside of...
A set of physical registers with utility functions to track liveness when walking backward/forward th...
bool contains(MCRegister Reg) const
contains - Return true if the specified register is included in this register class.
Wrapper class representing physical registers. Should be passed by value.
Definition MCRegister.h:41
constexpr bool isValid() const
Definition MCRegister.h:84
LLVM_ABI void transferSuccessorsAndUpdatePHIs(MachineBasicBlock *FromMBB)
Transfers all the successors, as in transferSuccessors, and update PHI operands in the successor bloc...
LLVM_ABI iterator getFirstTerminator()
Returns an iterator to the first terminator instruction of this basic block.
LLVM_ABI void addSuccessor(MachineBasicBlock *Succ, BranchProbability Prob=BranchProbability::getUnknown())
Add Succ as a successor of this MachineBasicBlock.
const MachineFunction * getParent() const
Return the MachineFunction containing this basic block.
reverse_iterator rbegin()
iterator insertAfter(iterator I, MachineInstr *MI)
Insert MI into the instruction list after I.
void splice(iterator Where, MachineBasicBlock *Other, iterator From)
Take an instruction from MBB 'Other' at the position From, and insert it into this MBB right before '...
MachineInstrBundleIterator< MachineInstr > iterator
The MachineFrameInfo class represents an abstract stack frame until prolog/epilog code is inserted.
LLVM_ABI int CreateFixedObject(uint64_t Size, int64_t SPOffset, bool IsImmutable, bool isAliased=false)
Create a new object at a fixed location on the stack.
bool hasVarSizedObjects() const
This method may be called any time after instruction selection is complete to determine if the stack ...
const AllocaInst * getObjectAllocation(int ObjectIdx) const
Return the underlying Alloca of the specified stack object if it exists.
LLVM_ABI int CreateStackObject(uint64_t Size, Align Alignment, bool isSpillSlot, const AllocaInst *Alloca=nullptr, uint8_t ID=0)
Create a new statically sized stack object, returning a nonnegative identifier to represent it.
bool hasCalls() const
Return true if the current function has any function calls.
bool isFrameAddressTaken() const
This method may be called any time after instruction selection is complete to determine if there is a...
void setObjectOffset(int ObjectIdx, int64_t SPOffset)
Set the stack frame offset of the specified object.
bool isCalleeSavedObjectIndex(int ObjectIdx) const
uint64_t getMaxCallFrameSize() const
Return the maximum size of a call frame that must be allocated for an outgoing function call.
bool hasPatchPoint() const
This method may be called any time after instruction selection is complete to determine if there is a...
bool hasScalableStackID(int ObjectIdx) const
int getStackProtectorIndex() const
Return the index for the stack protector object.
LLVM_ABI uint64_t estimateStackSize(const MachineFunction &MF) const
Estimate and return the size of the stack frame.
void setStackID(int ObjectIdx, uint8_t ID)
bool isCalleeSavedInfoValid() const
Has the callee saved info been calculated yet?
Align getObjectAlign(int ObjectIdx) const
Return the alignment of the specified stack object.
int64_t getObjectSize(int ObjectIdx) const
Return the size of the specified object.
bool isMaxCallFrameSizeComputed() const
bool hasStackMap() const
This method may be called any time after instruction selection is complete to determine if there is a...
LLVM_ABI int CreateSpillStackObject(uint64_t Size, Align Alignment, TargetStackID::Value StackID=TargetStackID::Default)
Create a new statically sized stack object that represents a spill slot, returning a nonnegative iden...
const std::vector< CalleeSavedInfo > & getCalleeSavedInfo() const
Returns a reference to call saved info vector for the current function.
unsigned getNumObjects() const
Return the number of objects.
int getObjectIndexEnd() const
Return one past the maximum frame object index.
bool hasStackProtectorIndex() const
bool hasStackObjects() const
Return true if there are any stack objects in this function.
uint8_t getStackID(int ObjectIdx) const
unsigned getNumFixedObjects() const
Return the number of fixed objects.
void setIsCalleeSavedObjectIndex(int ObjectIdx, bool IsCalleeSaved)
int64_t getObjectOffset(int ObjectIdx) const
Return the assigned stack offset of the specified object from the incoming stack pointer.
int getObjectIndexBegin() const
Return the minimum frame object index.
void setObjectAlignment(int ObjectIdx, Align Alignment)
setObjectAlignment - Change the alignment of the specified stack object.
bool isDeadObjectIndex(int ObjectIdx) const
Returns true if the specified index corresponds to a dead object.
const WinEHFuncInfo * getWinEHFuncInfo() const
getWinEHFuncInfo - Return information about how the current function uses Windows exception handling.
const TargetSubtargetInfo & getSubtarget() const
getSubtarget - Return the subtarget for which this machine code is being compiled.
bool framePointerIsReserved() const
Returns true if the frame pointer must always either point to a new frame record or be un-modified in...
MachineFrameInfo & getFrameInfo()
getFrameInfo - Return the frame info object for the current function.
MachineRegisterInfo & getRegInfo()
getRegInfo - Return information about the registers currently in use.
bool needsWinCFI() const
True if this function uses SEH unwind opcodes and needs an unwind table entry.
Function & getFunction()
Return the LLVM function that this machine code represents.
bool usesWindowsUnwindInfo() const
True if this function describes its unwind information with Windows SEH unwind opcodes (....
BasicBlockListType::iterator iterator
bool disableFramePointerElim() const
Returns true if frame pointer elimination should be disabled for this function.
Ty * getInfo()
getInfo - Keep track of various per-function pieces of information for backends that would like to do...
const MachineBasicBlock & front() const
MachineMemOperand * getMachineMemOperand(MachinePointerInfo PtrInfo, MachineMemOperand::Flags F, LLT MemTy, Align BaseAlignment, const MMOMetadata &Metadata=MMOMetadata(), SyncScope::ID SSID=SyncScope::System, AtomicOrdering Ordering=AtomicOrdering::NotAtomic, AtomicOrdering FailureOrdering=AtomicOrdering::NotAtomic)
getMachineMemOperand - Allocate a new MachineMemOperand.
MachineBasicBlock * CreateMachineBasicBlock(const BasicBlock *BB=nullptr, std::optional< UniqueBBID > BBID=std::nullopt)
CreateMachineInstr - Allocate a new MachineInstr.
void insert(iterator MBBI, MachineBasicBlock *MBB)
const TargetMachine & getTarget() const
getTarget - Return the target machine this machine code is compiled with
const MachineInstrBuilder & setMemRefs(ArrayRef< MachineMemOperand * > MMOs) const
const MachineInstrBuilder & addExternalSymbol(const char *FnName, unsigned TargetFlags=0) const
const MachineInstrBuilder & addReg(Register RegNo, RegState Flags={}, unsigned SubReg=0) const
Add a new virtual register operand.
const MachineInstrBuilder & setMIFlag(MachineInstr::MIFlag Flag) const
const MachineInstrBuilder & addImm(int64_t Val) const
Add a new immediate operand.
const MachineInstrBuilder & addFrameIndex(int Idx) const
const MachineInstrBuilder & addRegMask(const uint32_t *Mask) const
const MachineInstrBuilder & addMBB(MachineBasicBlock *MBB, unsigned TargetFlags=0) const
const MachineInstrBuilder & addDef(Register RegNo, RegState Flags={}, unsigned SubReg=0) const
Add a virtual register definition operand.
const MachineInstrBuilder & setMIFlags(unsigned Flags) const
const MachineInstrBuilder & addMemOperand(MachineMemOperand *MMO) const
Representation of each machine instruction.
void setFlags(unsigned flags)
uint32_t getFlags() const
Return the MI flags bitvector.
LLVM_ABI MachineInstrBundleIterator< MachineInstr > eraseFromParent()
Unlink 'this' from the containing basic block and delete it.
A description of a memory reference used in the backend.
const PseudoSourceValue * getPseudoValue() const
@ MOVolatile
The memory access is volatile.
@ MOLoad
The memory access reads data.
@ MOStore
The memory access writes data.
const Value * getValue() const
Return the base address of the memory access.
MachineOperand class - Representation of each machine instruction operand.
int64_t getImm() const
bool isFI() const
isFI - Tests if this is a MO_FrameIndex operand.
LLVM_ABI void emit(DiagnosticInfoOptimizationBase &OptDiag)
Emit an optimization remark.
MachineRegisterInfo - Keep track of information for virtual and physical registers,...
LLVM_ABI Register createVirtualRegister(const TargetRegisterClass *RegClass, StringRef Name="")
createVirtualRegister - Create and return a new virtual register in the function with the specified r...
LLVM_ABI bool isLiveIn(Register Reg) const
LLVM_ABI const MCPhysReg * getCalleeSavedRegs() const
Returns list of callee saved registers.
LLVM_ABI bool isPhysRegUsed(MCRegister PhysReg, bool SkipRegMaskTest=false) const
Return true if the specified register is modified or read in this function.
const Triple & getTargetTriple() const
Get the target triple which is a string describing the target host.
Definition Module.h:328
Represent a mutable reference to an array (0 or more elements consecutively in memory),...
Definition ArrayRef.h:294
MutableArrayRef< T > drop_until(PredicateT Pred) const
Return a copy of *this with the first N elements not satisfying the given predicate removed.
Definition ArrayRef.h:403
Wrapper class representing virtual and physical registers.
Definition Register.h:20
constexpr bool isValid() const
Definition Register.h:112
SMEAttrs is a utility class to parse the SME ACLE attributes on functions.
bool hasStreamingInterface() const
bool hasNonStreamingInterfaceAndBody() const
bool hasStreamingBody() const
bool insert(const value_type &X)
Insert a new element into the SetVector.
Definition SetVector.h:157
This class consists of common code factored out of the SmallVector class to reduce code duplication b...
reference emplace_back(ArgTypes &&... Args)
void append(ItTy in_start, ItTy in_end)
Add the specified range to the end of the SmallVector.
void push_back(const T &Elt)
LLVM_NO_SANITIZE_CFI_UNRELATED_CAST iterator begin()
This is a 'vector' (really, a variable-sized array), optimized for the case when the array is small.
StackOffset holds a fixed and a scalable offset in bytes.
Definition TypeSize.h:31
int64_t getFixed() const
Returns the fixed component of the stack.
Definition TypeSize.h:47
int64_t getScalable() const
Returns the scalable component of the stack.
Definition TypeSize.h:50
static StackOffset get(int64_t Fixed, int64_t Scalable)
Definition TypeSize.h:42
static StackOffset getScalable(int64_t Scalable)
Definition TypeSize.h:41
static StackOffset getFixed(int64_t Fixed)
Definition TypeSize.h:40
bool hasFP(const MachineFunction &MF) const
hasFP - Return true if the specified function should have a dedicated frame pointer register.
virtual void determineCalleeSaves(MachineFunction &MF, BitVector &SavedRegs, RegScavenger *RS=nullptr) const
This method determines which of the registers reported by TargetRegisterInfo::getCalleeSavedRegs() sh...
int getOffsetOfLocalArea() const
getOffsetOfLocalArea - This method returns the offset of the local area from the stack pointer on ent...
Align getStackAlign() const
getStackAlignment - This method returns the number of bytes to which the stack pointer must be aligne...
StackDirection getStackGrowthDirection() const
getStackGrowthDirection - Return the direction the stack grows
virtual bool enableCFIFixup(const MachineFunction &MF) const
Returns true if we may need to fix the unwind information for the function.
const Triple & getTargetTriple() const
TargetRegisterInfo base class - We assume that the target defines a static array of TargetRegisterDes...
bool hasStackRealignment(const MachineFunction &MF) const
True if stack realignment is required and still possible.
virtual const TargetRegisterInfo * getRegisterInfo() const =0
Return the target's register information.
Triple - Helper class for working with autoconf configuration names.
Definition Triple.h:48
bool isOSBinFormatMachO() const
Tests whether the environment is MachO.
Definition Triple.h:878
This class implements an extremely fast bulk output stream that can only output to a stream.
Definition raw_ostream.h:53
#define llvm_unreachable(msg)
Marks that the current location is not supposed to be reachable.
static unsigned getShiftValue(unsigned Imm)
getShiftValue - Extract the shift value.
static unsigned getArithExtendImm(AArch64_AM::ShiftExtendType ET, unsigned Imm)
getArithExtendImm - Encode the extend type and shift amount for an arithmetic instruction: imm: 3-bit...
const AArch64MCOptions & getCLOpts(const MCTargetOptions &Options)
const unsigned StackProbeMaxLoopUnroll
Maximum number of iterations to unroll for a constant size probing loop.
const unsigned StackProbeMaxUnprobedStack
Maximum allowed number of unprobed bytes above SP at an ABI boundary.
constexpr char Align[]
Key for Kernel::Arg::Metadata::mAlign.
constexpr char Attrs[]
Key for Kernel::Metadata::mAttrs.
unsigned ID
LLVM IR allows to use arbitrary numbers as calling convention identifiers.
Definition CallingConv.h:24
@ AArch64_SVE_VectorCall
Used between AArch64 SVE functions.
@ PreserveMost
Used for runtime calls that preserves most registers.
Definition CallingConv.h:63
@ CXX_FAST_TLS
Used for access functions.
Definition CallingConv.h:72
@ GHC
Used by the Glasgow Haskell Compiler (GHC).
Definition CallingConv.h:50
@ PreserveAll
Used for runtime calls that preserves (almost) all registers.
Definition CallingConv.h:66
@ Fast
Attempts to make calls as fast as possible (e.g.
Definition CallingConv.h:41
@ PreserveNone
Used for runtime calls that preserves none general registers.
Definition CallingConv.h:90
@ Win64
The C convention as implemented on Windows/x86-64 and AArch64.
@ SwiftTail
This follows the Swift calling convention in how arguments are passed but guarantees tail calls will ...
Definition CallingConv.h:87
@ C
The default llvm calling convention, compatible with C.
Definition CallingConv.h:34
NodeAddr< InstrNode * > Instr
Definition RDFGraph.h:389
BaseReg
Stack frame base register. Bit 0 of FREInfo.Info.
Definition SFrame.h:77
This is an optimization pass for GlobalISel generic memory operations.
void dump(const SparseBitVector< ElementSize > &LHS, raw_ostream &out)
@ Offset
Definition DWP.cpp:577
void stable_sort(R &&Range)
Definition STLExtras.h:2132
bool all_of(R &&range, UnaryPredicate P)
Provide wrappers to std::all_of which take ranges instead of having to pass begin/end explicitly.
Definition STLExtras.h:1755
MachineInstrBuilder BuildMI(MachineFunction &MF, const MIMetadata &MIMD, const MCInstrDesc &MCID)
Builder interface. Specify how to create the initial instruction itself.
int isAArch64FrameOffsetLegal(const MachineInstr &MI, StackOffset &Offset, bool *OutUseUnscaledOp=nullptr, unsigned *OutUnscaledOp=nullptr, int64_t *EmittableOffset=nullptr)
Check if the Offset is a valid frame offset for MI.
@ Unknown
Not known to have no common set bits.
RegState
Flags to represent properties of register accesses.
@ Define
Register definition.
constexpr RegState getKillRegState(bool B)
decltype(auto) dyn_cast(const From &Val)
dyn_cast<X> - Return the argument parameter cast to the specified type.
Definition Casting.h:643
void interleaveComma(const Container &c, StreamT &os, UnaryFunctor each_fn)
Definition STLExtras.h:2329
@ AArch64FrameOffsetCannotUpdate
Offset cannot apply.
constexpr T alignDown(U Value, V Align, W Skew=0)
Returns the largest unsigned integer less than or equal to Value and is Skew mod Align.
Definition MathExtras.h:546
IterT skipDebugInstructionsForward(IterT It, IterT End, bool SkipPseudoOp=true)
Increment It until it points to a non-debug instruction or to End and return the resulting iterator.
auto dyn_cast_or_null(const Y &Val)
Definition Casting.h:753
bool any_of(R &&range, UnaryPredicate P)
Provide wrappers to std::any_of which take ranges instead of having to pass begin/end explicitly.
Definition STLExtras.h:1762
auto formatv(bool Validate, const char *Fmt, Ts &&...Vals)
auto reverse(ContainerTy &&C)
Definition STLExtras.h:408
void sort(IteratorTy Start, IteratorTy End)
Definition STLExtras.h:1652
LLVM_ABI raw_ostream & dbgs()
dbgs() - This returns a reference to a raw_ostream for debugging messages.
Definition Debug.cpp:209
void emitFrameOffset(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, const DebugLoc &DL, unsigned DestReg, unsigned SrcReg, StackOffset Offset, const TargetInstrInfo *TII, MachineInstr::MIFlag=MachineInstr::NoFlags, bool SetNZCV=false, bool NeedsWinCFI=false, bool *HasWinCFI=nullptr, bool EmitCFAOffset=false, StackOffset InitialOffset={}, unsigned FrameReg=AArch64::SP)
emitFrameOffset - Emit instructions as needed to set DestReg to SrcReg plus Offset.
LLVM_ABI void report_fatal_error(Error Err, bool gen_crash_diag=true)
Definition Error.cpp:163
constexpr uint64_t alignTo(uint64_t Size, Align A)
Returns a multiple of A needed to store Size bytes.
Definition Alignment.h:149
constexpr RegState getDefRegState(bool B)
class LLVM_GSL_OWNER SmallVector
Forward declaration of SmallVector so that calculateSmallVectorDefaultInlinedElements can reference s...
LLVM_ABI const Value * getUnderlyingObject(const Value *V, unsigned MaxLookup=MaxLookupSearchDepth, bool MustPreserveProvenance=false)
This method strips off any GEP address adjustments, pointer casts or llvm.threadlocal....
auto reverse_conditionally(ContainerTy &&C, bool ShouldReverse)
Return a range that conditionally reverses C.
Definition STLExtras.h:1439
@ First
Helpers to iterate all locations in the MemoryEffectsBase class.
Definition ModRef.h:74
uint16_t MCPhysReg
An unsigned integer type large enough to represent all physical registers, but not necessarily virtua...
Definition MCRegister.h:21
RelativeUniformCounterPtr ValuesPtrExpr VTableAddr Count
Definition InstrProf.h:145
raw_ostream & operator<<(raw_ostream &OS, const APFixedPoint &FX)
auto count_if(R &&Range, UnaryPredicate P)
Wrapper function around std::count_if to count the number of times an element satisfying a given pred...
Definition STLExtras.h:2035
auto find_if(R &&Range, UnaryPredicate P)
Provide wrappers to std::find_if which take ranges instead of having to pass begin/end explicitly.
Definition STLExtras.h:1788
constexpr auto seq(T Begin, T End)
Iterate over an integral type from Begin up to - but not including - End.
Definition Sequence.h:341
void erase_if(Container &C, UnaryPredicate P)
Provide a container algorithm similar to C++ Library Fundamentals v2's erase_if which is equivalent t...
Definition STLExtras.h:2208
bool is_contained(R &&Range, const E &Element)
Returns true if Element is found in Range.
Definition STLExtras.h:1963
@ End
Run the machine verifier once, at the end of the pipeline.
Definition CodeGen.h:320
void fullyRecomputeLiveIns(ArrayRef< MachineBasicBlock * > MBBs)
Convenience function for recomputing live-in's for a set of MBBs until the computation converges.
LLVM_ABI Printable printReg(Register Reg, const TargetRegisterInfo *TRI=nullptr, unsigned SubIdx=0, const MachineRegisterInfo *MRI=nullptr)
Prints virtual and physical registers with or without a TRI instance.
MCRegisterClass TargetRegisterClass
Definition FastISel.h:58
void swap(llvm::BitVector &LHS, llvm::BitVector &RHS)
Implement std::swap in terms of BitVector swap.
Definition BitVector.h:881
bool operator<(const StackAccess &Rhs) const
void print(raw_ostream &OS) const
int64_t start() const
std::string getTypeString() const
int64_t end() const
This struct is a compact representation of a valid (non-zero power of two) alignment.
Definition Alignment.h:39
constexpr uint64_t value() const
This is a hole in the type system and should not be abused.
Definition Alignment.h:82
Pair of physical register and lane mask.
static LLVM_ABI MachinePointerInfo getUnknownStack(MachineFunction &MF)
Stack memory without other information.
static LLVM_ABI MachinePointerInfo getFixedStack(MachineFunction &MF, int FI, int64_t Offset=0)
Return a MachinePointerInfo record that refers to the specified FrameIndex.
SmallVector< WinEHTryBlockMapEntry, 4 > TryBlockMap
SmallVector< WinEHHandlerType, 1 > HandlerArray