Reproducible with current LLDB master, which will be 9.0, e.g. running this docker container (with extra params to allow debugging):

$ docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --security-opt apparmor=unconfined weliveindetail/lldb-bug36209:1dbd42ab

• cd bug36209
• /llvm-debug/bin/clang --version
  clang version 9.0.0 (https://github.com/llvm/llvm-project.git 1dbd42a)
  Target: x86_64-unknown-linux-gnu
  Thread model: posix
  InstalledDir: /llvm-debug/bin
• /llvm-debug/bin/clang++ -g -S -emit-llvm -o hellow.ll hellow.cpp
• /llvm-debug/bin/lli hellow.ll
  Hello World!
• /llvm-debug/bin/lldb -b -s lldb_cmds -- /llvm-debug/bin/lli
  (lldb) target create "/llvm-debug/bin/lli"
  Current executable set to '/llvm-debug/bin/lli' (x86_64).
  (lldb) command source -s 0 'lldb_cmds'
  Executing commands in '/bug36209/lldb_cmds'.
  (lldb) settings set plugin.jit-loader.gdb.enable on
  (lldb) log enable lldb jit
  (lldb) b jitbp
  Breakpoint 1: no locations (pending).
  WARNING: Unable to resolve breakpoint to any actual locations.
  (lldb) r hellow.ll
  lldb JITLoaderGDB::SetJITBreakpoint looking for JIT register hook
  lldb JITLoaderGDB::SetJITBreakpoint looking for JIT register hook
  lldb JITLoaderGDB::SetJITBreakpoint looking for JIT register hook
  lldb JITLoaderGDB::SetJITBreakpoint setting JIT breakpoint
  intern-state JITLoaderGDB::JITDebugBreakpointHit hit JIT breakpoint
  intern-state JITLoaderGDB::ReadJITDescriptorImpl registering JIT entry at 0x3b0ffb0 (2712 bytes)
  warning: failed to set breakpoint site at 0x4 for breakpoint 1.1: error: 0 sending the breakpoint request
  1 location added to breakpoint 1
  warning: failed to set breakpoint site at 0x4 for breakpoint 1.1: error: 0 sending the breakpoint request
  Hello World!
  Process 39 exited with status = 0 (0x00000000)

Image runs Ubuntu 18.04 and LLDB built from 1dbd42ab.

https://reviews.llvm.org/D61611 did a first step by fixing the "sending the breakpoint request" error. Now LLDB again manages to implement breakpoint sites in JITed code.

The next part of the regression is that we only see disassembly, while LLDB 5.0 was successfully loading and showing the respective source lines.

In a last step we should update the documentation:
http://llvm.org/docs/DebuggingJITedCode.html

This is exciting. Thanks Stefan!

I tried to fix source-level debugging in RuntimeDyldMachO a while back but was not able to wrap my head around the LLDB side of the change (MacOS had been explicitly disabled there) in the time I had. If/when you get this running I would love to go over the protocol with you: hopefully I can write a jit-debugging plugin for JITLink.

Some more investigation showed that the DWARF has some corrupt data. All fields of type DW_FORM_strp have a 0 offset. That prevents functions and compile units from being registered correctly. Fields with other data types look fine (e.g. 32 bit unsigned for DW_AT_language).

In a debug session for the regression test I created in D61611 (https://github.com/llvm/llvm-project/blob/87aa9c9e4d41ed88/lldb/test/Shell/Breakpoint/jitbp_elf.test) I put a breakpoint in DWARFFormValue::AsCString() once the JIT descriptor was registered and managed to twist the relevant fields to their correct values:

• DWARFAttributes of the first DW_TAG_subprogram DIE:
  DW_AT_name=jitbp
  DW_AT_linkage_name=_Z5jitbpv

• DWARFUnit DIE:
  DW_AT_name=/path/to/llvm-project/lldb/test/Shell/Breakpoint/Inputs/jitbp.cpp
  DW_AT_comp_dir=/path/to/llvm-build/tools/lldb/test/Breakpoint

As a result the jitbp function was added as a base-name and Symtab parsed of the compile unit's DIEs in detail in order to figure out the source code location.

It looked promising, until ResolveAddressUsingFileSections() failed to map the jitbp's AddressRange to one of the compile unit's sections. The range is DW_AT_low/high_pc == 0/3, which looked reasonable for a section offset. I did not go into more detail here as this is all standard LLDB logic. I would be a little surprised if there is a bug here that kills us. I had expected something like an incorrect type setting similar to D61611.

So, I wonder whether it's really the debugger side that's broken here or if it might be MCJIT. Not sure it's worth the time to investigate that. Maybe I can check again with ORCv2 once we have support for listeners.

These fields are empty because the object that we get from MCJIT doesn't have relocations applied. There's multiple reasons for it on the MCJIT side:

• lli doesn't set the ProcessAllSections flag for MCJIT, so
  relocations in sections like debug_info are ignored currently.

• notifyObjectLoaded() is called before we resolve relocations,
  thus we always copy over the unrelocated in-memory object.

I can think of two ways that this broke since 4.0:

(1) Either LLDB used to resolve relocations for objects it
received via the GDB JIT Interface itself and it stopped
doing so between releases 5 and 6.

(2) Or otherwise this has always been lli's responsibility
and it broke silently because the feature wasn't used
widely anyway.

I think version (1) is more likely, so I will go find a lldb-5.0 and test debugging with top-of-tree lli.

https://reviews.llvm.org/D90769 aims to fix the relocation issue and bring back source-level debugging.

Hi Stefan,

FWIW OrcV2 now supports listeners when using RTDyldObjectLinkingLayer if you want to test this under OrcV2. (I believe there is an example in llvm/examples/OrcV2Examples).

I'm on the road the next few days, but will try to check that review soon. Please ping me if you don't hear anything by next week.

-- Lang.

Yes, I want to do that. For the LLDB test case, however, it doesn't matter so much as it currently uses the same RuntimeDyld functionality under the hood as MCJIT does.

Also (note to myself) I have to patch lli first, as it currently makes the distinction whether or not to add the GDB listener based on the triple's isOSBinFormatELF(). That disqualifies JITing ELF on macOS or Windows even though it's quite a popular workaround.

Part 2 of the fix has landed. It will ship in the upcoming release 12 of LLDB. I also updated the documentation and set up a review: https://reviews.llvm.org/D90789

Just confirmed this to work with RuntimeDyld-based LLJIT on current tip-of-tree:

(lldb) run tools/lldb/test/Breakpoint/Output/jitbp_elf.test.tmp.ll
Process 89272 launched: '/workspaces/llvm-lldb/build/bin/LLJITWithGDBRegistrationListener' (x86_64)
1 location added to breakpoint 1
Process 89272 stopped

• thread #​1, name = 'LLJITWithGDBReg', stop reason = breakpoint 1.1
  frame #​0: 0x00007ffff7fd0000 JIT(0x43fa720)`jitbp() at jitbp.cpp:1:15
  -> 1 int jitbp() { return 0; }
  2 int main() { return jitbp(); }

I guess we cannot add a test for it based on the OrcV2 example executable, so we'd have to add something like an "orc-eager" kind to lli.

@​Lang: Do you think that's worth it? Anything special I should consider in case I decided to do that?

The documentation update landed and should be in the LLVM 12 release. A orc-greedy mode was considered a good addition for LLI in https://reviews.llvm.org/D90789#2491473