assigned to @lhames

I am working with Lang's example for dynamic recompilation Kaleidoscope-Orc-fully_lazy_with_recompile posted in http://lists.llvm.org/pipermail/llvm-dev/2015-September/090536.html.

The file which contains the printf that triggers the error.
The file which contains the printf that triggers the error:

LLVM ERROR: Undefined temporary symbol

The symbol it fails on is .L.str

main.c-- contains the call to foo

Hi Revital,

How are you compiling foo.c? Assuming you're using clang, can you attach the command-line you're running?

• Lang.

Hi Lang, I am using the following command line:

clang -c -emit-llvm foo.c -m64

Hi Revital. Could you attach your compiled bitcode? I.e. the output of 'clang -c -emit-llvm -o foo.bc foo.c -m64', for both foo_v1.c and main.c ?

Once I have the bitcode files I can reproduce the name mangling issue that I suspect is causing this.

compiled bitcode of foo_v1.c

compiled bitcode of main.c

Hi Revital,

Ahh - I think this is actually a straightforward issue: your @.str global is private.

The JIT is using a linker under the hood, and symbols beginning with '.Lxxx' are linker-local. Such symbols are not visible outside the module that they're originally compiled in. That means that they're not visible to any optimized copy of a function in a new module.

The trick is to promote all symbols to global linkage. Anything that was 'private' or 'static' should instead become 'hidden'. This should be done on the un-optimized version of the IR before it's codegen'd. The optimized version should use global-linkage decls to refer to these symbols.

You may be able to use the makeAllSymbolsExternallyAccessible function from 'include/llvm/ExecutionEngine/Orc/IndirectionUtils.h' for this.

Could you try that out and let me know if it fixes your issue?

Hi Lang,

Thanks for your answer!

I added a call to makeAllSymbolsExternallyAccessible for the module. Now I get the following error which I'm not sure what triggers it:

Mangler.cpp:37: void getNameWithPrefixImpl(llvm::raw_ostream&, const llvm::Twine&, {anonymous}::ManglerPrefixTy, const llvm::DataLayout&, char): Assertion `!Name.empty() && "getNameWithPrefix requires non-empty name"' failed.

One more thing to mention is that str is handled in createLambdaResolver in the following findSymbol:

if (auto Symbol = findSymbol(Name))
return RuntimeDyld::SymbolInfo(Symbol.getAddress(),
Symbol.getFlags());

Here is the IR of the function:

; Function Attrs: noinline nounwind uwtable
define i32 @​foo(i32 %x, i32 %y, i32* nocapture %res) #​0 {
entry:
%puts = tail call i32 @​puts(i8* getelementptr inbounds ([10 x i8], [10 x i8]* @​str, i64 0, i64 0))
%cmp = icmp sgt i32 %x, 3
%cmp1 = icmp slt i32 %y, 77
%or.cond = and i1 %cmp, %cmp1
br i1 %or.cond, label %if.end, label %if.else

if.else: ; preds = %entry
%0 = load i32, i32* %res, align 4, !tbaa !​1
%add = add nsw i32 %0, %x
br label %if.end

if.end: ; preds = %entry, %if.else
%storemerge = phi i32 [ %add, %if.else ], [ 97, %entry ]
store i32 %storemerge, i32* %res, align 4, !tbaa !​1
ret i32 %storemerge
}

I tried to debug the error and it seems that calling RuntimeDyld::SymbolInfo(Symbol.getAddress(), Symbol.getFlags for str somehow corrupted a pointer to another function. I am saving pointers to the original functions IR so I could generate different versions from them via CloneFunctions. After calling RuntimeDyld::SymbolInfo... stmt one of these pointers is corrupted and causes the ICE. This might not be related to the original error I posted.

Hi Revital,

That sounds like a different problem. I'm going to mark this bug as resolved.

As for the problem you're seeing now: RuntimeDyld::SymbolInfo doesn't do anything interesting itself, but Symbol.getAddress() may run the materialization function for that symbol, which could cause all sorts of things to happen depending on how you set up your JIT stack.

Have you run valgrind or an msan build of your project? This sounds like it could be a memory error in one of your data structures.

• Lang.

Thanks Lang!

Lang, btw I looked at the failure with valgrind as you suggested and it seems that calling
Symbol.getAddress() triggers the call to Handle = this->emitToBaseLayer(B) in llvm/ExecutionEngine/Orc/LazyEmittingLayer.h which deletes all the functions in the module.
As a result all the pointers to the functions I saved before are now invalid.
I wonder if there is a way to avoid this deletion. (I need these pointer in order to generate additional versions of the functions later on)

Hi Revital,

Whether or not the LazyEmittingLayer deletes modules depends on how the Module is added to the layer.

If you add a module to the layer as a std::unique_ptr then the layer will take ownership, and will delete the module once it's compiled (leaving your map with dangling pointers). If you add the module to the layer as a raw Module* (or a shared_ptr) then the layer will not take ownership: You will be responsible for deleting the Module. That means that you can keep it alive until you've re-compiled all functions in it.

As a side note: Do you actually need the lazy-emitting layer for what you're doing? The name is misleading (I may rename it soon to avoid confusion) - it doesn't actually handle callbacks or lazy-compilation, it just defers emitting modules until the linker needs them. If you're using CompileOnDemand for true lazy compilation then the LazyEmittingLayer is almost certainly superfluous.

• Lang.

Hi Lang, I could not add the Module to addModuleSet in LazyEmittingLayer as Module * without getting an error. I highly appreciate it if you could tell me how that can be done.
I had several tries to apply it in order to avoid deleting the functions as discussed but It seems to me that LazyEmittingLayer is tailored to get std::unique_ptr as it also contains calls to std::move(Ms) which moves the pointer.
Thanks again!

Patch for Orc Kaleidoscope lazy_codegen example.
Hi Revital,

Are you getting a compile-time or runtime error?

'std::move' doesn't imply that the type has to be a unique_ptr. It should be interpreted as 'move this value (if possible)', which works just fine for raw pointers.

I've had no trouble making the change locally. I did gloss over a detail: As you did when using unique_ptr, you should be adding a collection of Module* values (e.g. std::vector<Module*>) not a single Module*. See the attached patch for an example.

• Lang.

Lang, Thanks very much for the example and the help!! This indeed seems to be solving the compilation errors.
Revital