I built a FreeBSD installation using lld @ r288228 as the linker, and almost all userland binaries segfaulted when starting up, in rtld. Bisecting identified r288107 as the culprit. The problem is not reproducible after switching back to a rtld linked with lld built at r288102. The difference is that rtld built with post-r288107 lld has an entirely zeroed .got: % diffoscope old-lld/ld-elf.so.1 new-lld/ld-elf.so.1 --- old-lld/ld-elf.so.1 +++ new-lld/ld-elf.so.1 ├── readelf --wide --hex-dump=.got {} │ @@ -1,16 +1,16 @@ │ │ Hex dump of section '.got': │ - 0x00020410 20900000 00000000 70100200 00000000 .......p....... │ - 0x00020420 e0230200 00000000 e8230200 00000000 .#.......#...... │ - 0x00020430 10030200 00000000 d0b60000 00000000 ................ │ - 0x00020440 18240200 00000000 18230200 00000000 .$.......#...... │ - 0x00020450 68240200 00000000 6c240200 00000000 h$......l$...... │ - 0x00020460 70240200 00000000 74240200 00000000 p$......t$...... │ - 0x00020470 80100200 00000000 18060000 00000000 ................ │ - 0x00020480 00000200 00000000 7c240200 00000000 ........|$...... │ - 0x00020490 10240200 00000000 00100200 00000000 .$.............. │ - 0x000204a0 04100200 00000000 10100200 00000000 ................ │ - 0x000204b0 30160200 00000000 981b0200 00000000 0............... │ - 0x000204c0 b0100200 00000000 00240200 00000000 .........$...... │ - 0x000204d0 60240200 00000000 84240200 00000000 `$.......$...... │ - 0x000204e0 f8230200 00000000 .#...... │ + 0x00020410 00000000 00000000 00000000 00000000 ................ │ + 0x00020420 00000000 00000000 00000000 00000000 ................ │ + 0x00020430 00000000 00000000 00000000 00000000 ................ │ + 0x00020440 00000000 00000000 00000000 00000000 ................ │ + 0x00020450 00000000 00000000 00000000 00000000 ................ │ + 0x00020460 00000000 00000000 00000000 00000000 ................ │ + 0x00020470 00000000 00000000 00000000 00000000 ................ │ + 0x00020480 00000000 00000000 00000000 00000000 ................ │ + 0x00020490 00000000 00000000 00000000 00000000 ................ │ + 0x000204a0 00000000 00000000 00000000 00000000 ................ │ + 0x000204b0 00000000 00000000 00000000 00000000 ................ │ + 0x000204c0 00000000 00000000 00000000 00000000 ................ │ + 0x000204d0 00000000 00000000 00000000 00000000 ................ │ + 0x000204e0 00000000 00000000 ........ ├── readelf --wide --string-dump=.comment {} │ @@ -1,6 +1,6 @@ │ │ String dump of section '.comment': │ [ 1] FreeBSD clang version 3.9.0 (tags/RELEASE_390/final 280324) (based on LLVM 3.9.0) │ [ 53] $FreeBSD$ │ - [ 5d] Linker: LLD 4.0.0 (http://llvm.org/git/lld 01db8ccdad26c748727d9638c5df3b99c8260ddc) │ + [ 5d] Linker: LLD 4.0.0 (http://llvm.org/git/lld 326233f95ee6b9c32f19d04ad06a6c369e6acc5a) │ ├── readelf --wide --hex-dump=.gnu_debuglink {} │ @@ -1,4 +1,4 @@ │ │ Hex dump of section '.gnu_debuglink': │ 0x00000000 6c642d65 6c662e73 6f2e312e 64656275 ld-elf.so.1.debu │ - 0x00000010 67000000 a26ecfb2 g....n.. │ + 0x00000010 67000000 c1a333a6 g.....3. ╵ rtld has code to determine if it needs to relocate itself or not, which defaults to #ifndef RTLD_IS_DYNAMIC #define RTLD_IS_DYNAMIC() (&_DYNAMIC != NULL) #endif RTLD_IS_DYNAMIC is false when linked with lld >= r288107, so rtld does not apply its own relocations at startup and crashes. This is arguably a FreeBSD rtld bug (and we could address it there). I'm submitting this LLD ticket for tracking the issue as it may affect other projects.
Can you attach the cpio?
Created attachment 17696 [details] testcase
I can reproduce it. Debugging.
I have a patch in review to address part of what broke after r288107: https://reviews.freebsd.org/D8687 The code removed by that change has never been used as far as I can tell, and should probably be done differently if we want to bring the functionality back in FreeBSD anyhow. However, we'd still need to change rtld to access _DYNAMIC not via the GOT.
This reduces to just cmpq $0, _DYNAMIC@GOTPCREL(%rip) The old code was creating both a relocation with an addend and putting the value in the .got position. That is the same issue the loader had. It is not a hard fix if we really have to do it. I will also take a quick look at musl to see how it handles that.
> It is not a hard fix if we really have to do it. I will also take a quick > look at musl to see how it handles that. What musl does is to have a tiny bit of assembly in arch/x86_64/crt_arch.h that includes lea _DYNAMIC(%rip),%rsi which avoids needing relocations to find the value of _DYNAMIC. Could freebsd do the same?
(In reply to comment #5) > > The old code was creating both a relocation with an addend and putting the > value in the .got position. That is the same issue the loader had. Right, although rtld has been working for some time, while the loader hasn't. Did LLD previously store the value only in the rela addend for non-GOT relocations, but in both the addend and .got slot for GOT ones? Or it could be that there are multiple issues affecting the EFI loader. > It is not a hard fix if we really have to do it. I will also take a quick > look at musl to see how it handles that. I'd rather FreeBSD not rely on it if there's no ABI requirement that it be populated, so I'd like to incorporate something like that. But I worry about the variety of other software that has less of an incentive to work with LLD.
(In reply to comment #6) > > It is not a hard fix if we really have to do it. I will also take a quick > > look at musl to see how it handles that. > > What musl does is to have a tiny bit of assembly in arch/x86_64/crt_arch.h > that includes > > lea _DYNAMIC(%rip),%rsi > > which avoids needing relocations to find the value of _DYNAMIC. > > Could freebsd do the same? I fixed the Elf_Rel case in r288451. It should be trivial now to hack it to always write the addend if we really must.
(In reply to comment #7) > (In reply to comment #5) > > > > The old code was creating both a relocation with an addend and putting the > > value in the .got position. That is the same issue the loader had. > > Right, although rtld has been working for some time, while the loader > hasn't. Did LLD previously store the value only in the rela addend for > non-GOT relocations, but in both the addend and .got slot for GOT ones? Or > it could be that there are multiple issues affecting the EFI loader. lld was writing the addend only in got entries before I think.
> I fixed the Elf_Rel case in r288451. It should be trivial now to hack it to > always write the addend if we really must. Thanks, rtld again works. I still intend to rework it to avoid relying on this if it's not an ABI requirement.
I think this can be resolved as of r288451 now that the GOT entries are populated? The FreeBSD PR for this is https://bugs.freebsd.org/214972. I expect that I'll be able to use an approach like what musl does.
Closing for now
Current versions of LLD are no longer writing the addend into the GOT entries. I've posted a FreeBSD rtld patch to handle this at https://reviews.freebsd.org/D9180.
FreeBSD rtld no longer relies on a populated GOT as of r312288 https://svnweb.freebsd.org/changeset/base/312288
(In reply to comment #14) > FreeBSD rtld no longer relies on a populated GOT as of r312288 > https://svnweb.freebsd.org/changeset/base/312288 Thanks!