assigned to @smithp35

I think we discussed both -n,--nmagic and -N,--omagic options before (during LLD development), but I could not find the discussion (not surprising, it perhaps was in 2017).

I wonder if you will be OK to use -N,--omagic instead.
We have a comment in code saying that:

// --omagic is an option to create old-fashioned executables in which
// .text segments are writable. Today, the option is still in use to
// create special-purpose programs such as boot loaders. It doesn't
// make sense to create PT_GNU_RELRO for such executables.

My memories + what I see in the code today together says it should place all output sections in a single RWX segment without aligning them and we used this effect for linking FreeBSD kernel. I think the build system did not respect the program headers, so it was not an issue that LLD created all sections as RWX, I suppose it might work for you too (but my memories are weak here, and I know little about Linux kernel build system, so..)

In general, If you want to change LLD, we should know what exactly linker kernel wants to do to decide what we can do I think. When we worked on booting of a FreeBSD kernel we had to investigate how the build system uses the linker deeply to decide which options we want to support in LLD and how.

+Ed just in case (he knows a lot about FreeBSD and maybe he remembers more than me).

Fixes:

I think we discussed both -n,--nmagic and -N,--omagic options before (during
LLD development), but I could not find the discussion (not surprising, it
perhaps was in 2017).

2017 -> late 2016-early 2017 (just in case someone wants to find that discussion, I think it was the time when we booted the FreeBSD kernel the first time)

In general, If you want to change LLD, we should know what exactly linker
kernel wants to do to decide what we can do I think. When we worked on

linker kernel wants to -> Linux kernel wants to

The best summary of the discussion I could find is summarised in the implementation of -N omagic https://reviews.llvm.org/D26888

From what I understand -n is equivalent to -N but without forcing the .text section to be read write.

The LLD implementation of -N as implemented doesn't seem to have any impact on the page alignment of Sections. I've not experimented yet but I'm guessing that manually setting -z max-page-size=4 (has to be a power of 2) will come close to that.

From what I understand -n is equivalent to -N but without forcing the .text
section to be read write.

The LLD implementation of -N as implemented doesn't seem to have any impact
on the page alignment of Sections.

Yes. LLD page aligns each first section in a PT_LOAD. And since -N creates a single segment (all output sections are rwx), LLD does not do anything with alignment for them. I guess that is what kernel wants probably. I guess it does not care about sections flags and/or amount of program headers.

Nathan points out this commit that added it, which adds context: ClangBuiltLinux/linux@4050740

"This patch passes the -n option to ld, which prevents it from aligning
PT_LOAD segments to the maximum page size."

To summarize: I think/guess Linux can go with -N instead of -n.

I don't think that -N is going to work as it is implemented in LLD for this case.

From a first pass over the commit message and looking up vDSO I think (please correct me if I'm wrong!):

• The vDSO is a shared object that the kernel maps into every user-space processes address space.
• The address is passed in via the auxiliary vector.
• The vDSO is not demand paged
• GNU ld defaults to 64K max-page-size and will align loadable segments on 64k boundaries, and set the program header alignment to 64k
• A 4K page size kernel may map this vDSO at only a 4k aligned address, regardless of the program header alignment, this is fine as the vDSO isn't demand paged.
• GDB however doesn't like a 64k aligned shared object being loaded at a 4k aligned address
• The -n switch prevents page alignment of the segments but otherwise preserves the separate read-only and read-write segments of the vDSO.
• In the kernel there is a linker script (derived from) vds.lds.S that has:
  /*

• We must supply the ELF program headers explicitly to get just one
• PT_LOAD segment, and set the flags explicitly to make segments read-only.
  /
  PHDRS
  {
  text PT_LOAD FLAGS(5) FILEHDR PHDRS; / PF_R|PF_X /
  dynamic PT_DYNAMIC FLAGS(4); / PF_R /
  note PT_NOTE FLAGS(4); / PF_R */
  eh_frame_hdr PT_GNU_EH_FRAME;
  }
  The start of the SECTION has:
  SECTIONS
  {
  PROVIDE(_vdso_data = . - PAGE_SIZE);
  . = VDSO_LBASE + SIZEOF_HEADERS;
  ...

Where VDSO_LBASE is defined to 0

• When using -N we make the .text section read-write, however the PHDRS flags override this and we get a single PT_LOAD with PF_R|PF_X

So far so good. Unfortunately as we haven not implemented the do not page align program header part of -N we align 0 to the page-size (no harm done there) but we set the alignment of the segment to 65536 which is what was choking gdb according to the commit message.

In LLD the best fix would be -zmax-page-size=4 and forget about -n or -N. Unfortunately ld.bfd uses this value even if there is an input section with higher alignment requirements so this would be somewhat fragile on ld.bfd.

Given that the smallest page size on aarch64 is 4k then -zmax-page-size=4096 would work on both 4k and 64k kernels, however linux upstream may be reluctant to accept that.

A solution that would work on both linkers is to use both -N and -zmax-page-size=1
ld.bfd will use -N and ignore the max-page-size
ld.lld will accept -N, but will use max-page-size=1 as an equivalent to not page aligning the segments. As LLD will still use the highest section alignment if it is higher than the max-page-size we'll get the same segment alignment as ld.bfd.

This highlights a potential difference in behaviour with respect to -zmax-page-size. ld.bfd will use that for the segment alignment even if there is a higher output section alignment within the segment, ld.lld will use the maximum of -zmax-page-size and the highest output section alignment in the segment. In practice the -zmax-page-size is likely to be highest so we won't see the difference.

A potential improvement to our -N and a cheap way to implement -n might be to just implicitly set -zmax-page-size=1 although we'd have to be careful that there was no other use for the original page size.

So far so good. Unfortunately as we haven not implemented the do not page
align program header part of -N we align 0 to the page-size (no harm done
there) but we set the alignment of the segment to 65536 which is what was
choking gdb according to the commit message.

So sounds like -N is broken with LLD when trying to debug via gdb? Sounds like a fix there would benefit/simplify -n, while fixing -N?

In LLD the best fix would be -zmax-page-size=4 and forget about -n or -N. Unfortunately ld.bfd uses this value even if there is an input section with higher alignment requirements so this would be somewhat fragile on ld.bfd.

I will say that kbuild has a macro called cc-ldoption that can test if a '-Wl,' flag will work and fall back to another if it doesn't.

We could change '-Wl,-n' to '$(cc-ldoption, -Wl$(comma)-n, -Wl$(comma)-zmax-page-size=4)' so that ld.bfd's behavior doesn't change but ld.lld accepts the fix (because it doesn't support '-n'). We'll need to be able to reproduce the GDB failure to see if it works or not.

*** Bug llvm/llvm-bugzilla-archive#41522 has been marked as a duplicate of this bug. ***

Update. This has now become a hard error due to https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/commit/?id=691efbedc60d2a7364a90e38882fc762f06f52c4

I didn't want to implement an option whose purpose is to create an executable compatible with ancient Unixes, but if -n is still in use and it's not easy to get rid of it, it's not too terrible to implement it to lld. It doesn't seem too hard to implement.

I'll take a look at this one.

It is unfortunately not quite as simple as just setting the Config->MaxPageSize to 1 as there is both Target::MaxPageSize and Target::PageSize where PageSize is what we would normally refer to as CommonPageSize (which I thought LLD didn't support). Turns out that this is used in https://reviews.llvm.org/D33630 so there are some alignTo that use Target->PageSize instead of Config->MaxPageSize to 1. This is used in 4 places:

• writeTrapInstr() which fills the last common page
• setPhdrs() setting the memsz of the RELRO program header
• assignFileOffsets() Aligning the offset of the Section immediately after the last executable section
• getPageSize() which is used to implement COMMONPAGESIZE

I've got a choice of going through each page align calculation and skipping it if Omagic or Nmagic, or in effect implementing -z common-page-size which was partially suggested in https://reviews.llvm.org/D56205 which didn't get any traction as it only affected getPageSize(). With -z common-page-size we'd then be able to set both Config->CommonPageSize and Config->MaxPageSize to 1.

First go at https://reviews.llvm.org/D61201

committed https://reviews.llvm.org/D61688 as r360593. Resolving for now, will reopen if there are any further changes required.

Thank you so much Peter (and reviewers) for all of the work that went into implementing this feature!

mentioned in issue llvm/llvm-bugzilla-archive#41522