assigned to @adrian-prantl

Stack backtrace

0 clang 0x00000000022f0005 llvm::sys::PrintStackTrace(_IO_FILE*) + 37
1 clang 0x00000000022f080b
2 libpthread.so.0 0x00000036b200f720
3 clang 0x0000000001c6dc62 llvm::SelectionDAG::TransferDbgValues(llvm::SDValue, llvm::SDValue) + 354
4 clang 0x0000000001bd56d8 llvm::SelectionDAG::Combine(llvm::CombineLevel, llvm::AliasAnalysis&, llvm::CodeGenOpt::Level) + 2072
5 clang 0x0000000001cafef9 llvm::SelectionDAGISel::CodeGenAndEmitDAG() + 137
6 clang 0x0000000001cafd45 llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) + 2613
7 clang 0x0000000001caeb35 llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) + 933
8 clang 0x0000000001b15bf6

Adrian? This looks like the stuff you've been touching.

Compiling this file uses over 24GiB of memory, all metadata. I'll have a look.

Impressive. Most impressive.

What happens here is that SelectionDAG is combining two SDNodes and then merging their DbgValues via DAG.TransferDbgValues(), but SelectionDAG::AddDbgValue() doesn't check whether a DbgValue is already in that SDNode's list of DbgValues and happily adds all of them. The testcase with it's plethora of debug variables from layer over layers of inlined boost::* methods drives this into a pathologic situation where — when dagcombining a chain of instructions — our SDNode's DbgValMap is growing exponentially.

The attached patch changes the DbgValMap to have set semantics, which fixes the problem and reduced the memory usage to only ~500MB.

I'm still trying to come up with a usable testcase for this.

change DbgValMap to have set semantics

Bugpoint and delta-reduced testcase
Attached a bugpoint and delta-reduced testcase that has SelectionDAG insert two distinct but equivalent SDDebugValues to the same SDNode.
Unfortunately the testcase is still counting 2375 lines.

Fixed by r221709