Created attachment 19797 [details] Full logs from LLDB 6.0, LLDB 5.0 and LLDB 4.0 With LLDB built from branch release_60 it is no longer possible to set a breakpoint in JIT-compiled code. This worked well for release_50. LLDB 6.0 fails with error message: warning: failed to set breakpoint site at 0x4 for breakpoint 1.1: error: 2 sending the breakpoint request Repro with a default configuration of LLVM/Clang/LLDB on Linux Mint 18: $ cat hellow.cpp #include <stdio.h> void jitbp() { printf("Hello World!\n"); } int main() { jitbp(); return 0; } $ clang -g -c -S -emit-llvm hellow.cpp $ ./lli hellow.ll Hello World! $ ./lldb -- lli (lldb) target create "lli" Current executable set to 'lli' (x86_64). (lldb) b jitbp Breakpoint 1: no locations (pending). WARNING: Unable to resolve breakpoint to any actual locations. (lldb) log enable lldb jit (lldb) r hellow.ll Actual output with LLDB 6.0: lldb JITLoaderGDB::SetJITBreakpoint looking for JIT register hook lldb JITLoaderGDB::SetJITBreakpoint looking for JIT register hook Process 17664 launched: '/media/LLVM/llvm60/build-debug-clang-lld-lldb/bin/lli' (x86_64) intern-state JITLoaderGDB::SetJITBreakpoint looking for JIT register hook ... intern-state JITLoaderGDB::SetJITBreakpoint looking for JIT register hook intern-state JITLoaderGDB::SetJITBreakpoint setting JIT breakpoint intern-state JITLoaderGDB::JITDebugBreakpointHit hit JIT breakpoint intern-state JITLoaderGDB::ReadJITDescriptorImpl registering JIT entry at 0x781230 (3432 bytes) warning: failed to set breakpoint site at 0x4 for breakpoint 1.1: error: 2 sending the breakpoint request 1 location added to breakpoint 1 warning: failed to set breakpoint site at 0x4 for breakpoint 1.1: error: 2 sending the breakpoint request Hello World! intern-state JITLoaderGDB::JITDebugBreakpointHit hit JIT breakpoint intern-state JITLoaderGDB::ReadJITDescriptorImpl unregistering JIT entry at 0x781230 Process 17664 exited with status = 0 (0x00000000) (lldb) q Expected output, tested with LLDB 5.0 and LLDB 4.0: JITLoaderGDB::SetJITBreakpoint looking for JIT register hook JITLoaderGDB::SetJITBreakpoint looking for JIT register hook Process 16807 launched: '/media/LLVM/llvm50/build-debug-clang-lld-lldb/bin/lli' (x86_64) JITLoaderGDB::SetJITBreakpoint looking for JIT register hook ... JITLoaderGDB::SetJITBreakpoint looking for JIT register hook JITLoaderGDB::SetJITBreakpoint setting JIT breakpoint JITLoaderGDB::JITDebugBreakpointHit hit JIT breakpoint JITLoaderGDB::ReadJITDescriptorImpl registering JIT entry at 0x754850 (3432 bytes) 1 location added to breakpoint 1 Process 16807 stopped * thread #1, name = 'lli', stop reason = breakpoint 1.1 frame #0: 0x00007ffff7fee004 JIT(0x754850)`jitbp() at hellow.cpp:4 1 #include <stdio.h> 2 3 void jitbp() { -> 4 printf("Hello World!\n"); 5 } 6 7 int main() { (lldb) n Hello World! Process 16807 stopped * thread #1, name = 'lli', stop reason = step over frame #0: 0x00007ffff7fee01c JIT(0x754850)`jitbp() at hellow.cpp:5 2 3 void jitbp() { 4 printf("Hello World!\n"); -> 5 } 6 7 int main() { 8 jitbp(); (lldb) c Process 16807 resuming JITLoaderGDB::JITDebugBreakpointHit hit JIT breakpoint JITLoaderGDB::ReadJITDescriptorImpl unregistering JIT entry at 0x754850 Process 16807 exited with status = 0 (0x00000000) (lldb) q The same issue can be reproduced with a slightly modified version of LLDB that allows debugging JITed code on Mac OS X. Please find attached the full logs.
Reproducible with current LLDB master, which will be 9.0, e.g. running this docker container (with extra params to allow debugging): $ docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --security-opt apparmor=unconfined weliveindetail/lldb-bug36209:1dbd42ab + cd bug36209 + /llvm-debug/bin/clang --version clang version 9.0.0 (https://github.com/llvm/llvm-project.git 1dbd42ab5babff6329d4f756b5f2237a2937ddb6) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /llvm-debug/bin + /llvm-debug/bin/clang++ -g -S -emit-llvm -o hellow.ll hellow.cpp + /llvm-debug/bin/lli hellow.ll Hello World! + /llvm-debug/bin/lldb -b -s lldb_cmds -- /llvm-debug/bin/lli (lldb) target create "/llvm-debug/bin/lli" Current executable set to '/llvm-debug/bin/lli' (x86_64). (lldb) command source -s 0 'lldb_cmds' Executing commands in '/bug36209/lldb_cmds'. (lldb) settings set plugin.jit-loader.gdb.enable on (lldb) log enable lldb jit (lldb) b jitbp Breakpoint 1: no locations (pending). WARNING: Unable to resolve breakpoint to any actual locations. (lldb) r hellow.ll lldb JITLoaderGDB::SetJITBreakpoint looking for JIT register hook lldb JITLoaderGDB::SetJITBreakpoint looking for JIT register hook lldb JITLoaderGDB::SetJITBreakpoint looking for JIT register hook lldb JITLoaderGDB::SetJITBreakpoint setting JIT breakpoint intern-state JITLoaderGDB::JITDebugBreakpointHit hit JIT breakpoint intern-state JITLoaderGDB::ReadJITDescriptorImpl registering JIT entry at 0x3b0ffb0 (2712 bytes) warning: failed to set breakpoint site at 0x4 for breakpoint 1.1: error: 0 sending the breakpoint request 1 location added to breakpoint 1 warning: failed to set breakpoint site at 0x4 for breakpoint 1.1: error: 0 sending the breakpoint request Hello World! Process 39 exited with status = 0 (0x00000000) Image runs Ubuntu 18.04 and LLDB built from https://github.com/llvm/llvm-project/commit/1dbd42ab.
https://reviews.llvm.org/D61611 did a first step by fixing the "sending the breakpoint request" error. Now LLDB again manages to implement breakpoint sites in JITed code. The next part of the regression is that we only see disassembly, while LLDB 5.0 was successfully loading and showing the respective source lines. In a last step we should update the documentation: http://llvm.org/docs/DebuggingJITedCode.html
This is exciting. Thanks Stefan! I tried to fix source-level debugging in RuntimeDyldMachO a while back but was not able to wrap my head around the LLDB side of the change (MacOS had been explicitly disabled there) in the time I had. If/when you get this running I would love to go over the protocol with you: hopefully I can write a jit-debugging plugin for JITLink.
Some more investigation showed that the DWARF has some corrupt data. All fields of type DW_FORM_strp have a 0 offset. That prevents functions and compile units from being registered correctly. Fields with other data types look fine (e.g. 32 bit unsigned for DW_AT_language). In a debug session for the regression test I created in D61611 (https://github.com/llvm/llvm-project/blob/87aa9c9e4d41ed88/lldb/test/Shell/Breakpoint/jitbp_elf.test) I put a breakpoint in DWARFFormValue::AsCString() once the JIT descriptor was registered and managed to twist the relevant fields to their correct values: - DWARFAttributes of the first DW_TAG_subprogram DIE: DW_AT_name=jitbp DW_AT_linkage_name=_Z5jitbpv - DWARFUnit DIE: DW_AT_name=/path/to/llvm-project/lldb/test/Shell/Breakpoint/Inputs/jitbp.cpp DW_AT_comp_dir=/path/to/llvm-build/tools/lldb/test/Breakpoint As a result the jitbp function was added as a base-name and Symtab parsed of the compile unit's DIEs in detail in order to figure out the source code location. It looked promising, until ResolveAddressUsingFileSections() failed to map the jitbp's AddressRange to one of the compile unit's sections. The range is DW_AT_low/high_pc == 0/3, which looked reasonable for a section offset. I did not go into more detail here as this is all standard LLDB logic. I would be a little surprised if there is a bug here that kills us. I had expected something like an incorrect type setting similar to D61611. So, I wonder whether it's really the debugger side that's broken here or if it might be MCJIT. Not sure it's worth the time to investigate that. Maybe I can check again with ORCv2 once we have support for listeners.
These fields are empty because the object that we get from MCJIT doesn't have relocations applied. There's multiple reasons for it on the MCJIT side: * lli doesn't set the `ProcessAllSections` flag for MCJIT, so relocations in sections like debug_info are ignored currently. * notifyObjectLoaded() is called before we resolve relocations, thus we always copy over the unrelocated in-memory object. I can think of two ways that this broke since 4.0: (1) Either LLDB used to resolve relocations for objects it received via the GDB JIT Interface itself and it stopped doing so between releases 5 and 6. (2) Or otherwise this has always been lli's responsibility and it broke silently because the feature wasn't used widely anyway. I think version (1) is more likely, so I will go find a lldb-5.0 and test debugging with top-of-tree lli.
https://reviews.llvm.org/D90769 aims to fix the relocation issue and bring back source-level debugging.
Hi Stefan, FWIW OrcV2 now supports listeners when using RTDyldObjectLinkingLayer if you want to test this under OrcV2. (I believe there is an example in llvm/examples/OrcV2Examples). I'm on the road the next few days, but will try to check that review soon. Please ping me if you don't hear anything by next week. -- Lang.
Yes, I want to do that. For the LLDB test case, however, it doesn't matter so much as it currently uses the same RuntimeDyld functionality under the hood as MCJIT does. Also (note to myself) I have to patch lli first, as it currently makes the distinction whether or not to add the GDB listener based on the triple's isOSBinFormatELF(). That disqualifies JITing ELF on macOS or Windows even though it's quite a popular workaround.
Part 2 of the fix has landed. It will ship in the upcoming release 12 of LLDB. I also updated the documentation and set up a review: https://reviews.llvm.org/D90789
Just confirmed this to work with RuntimeDyld-based LLJIT on current tip-of-tree: (lldb) run tools/lldb/test/Breakpoint/Output/jitbp_elf.test.tmp.ll Process 89272 launched: '/workspaces/llvm-lldb/build/bin/LLJITWithGDBRegistrationListener' (x86_64) 1 location added to breakpoint 1 Process 89272 stopped * thread #1, name = 'LLJITWithGDBReg', stop reason = breakpoint 1.1 frame #0: 0x00007ffff7fd0000 JIT(0x43fa720)`jitbp() at jitbp.cpp:1:15 -> 1 int jitbp() { return 0; } 2 int main() { return jitbp(); } I guess we cannot add a test for it based on the OrcV2 example executable, so we'd have to add something like an "orc-eager" kind to lli. @Lang: Do you think that's worth it? Anything special I should consider in case I decided to do that?
The documentation update landed and should be in the LLVM 12 release. A orc-greedy mode was considered a good addition for LLI in https://reviews.llvm.org/D90789#2491473