Reproduce with:
gcc ./bugpoint.safe.bc.cbe.c.so bugpoint.test.bc.c -o bugpoint.test.bc.exe -Wl,-R.

Nick, I wasn't able to reproduce the bug:

1. CBE-compiled version fails for me in the early and (calling ostream::operator<<).
2. If I compile both chunks separately with llc to .s and then compile and link
   with gcc it crashes in the end (ostream)
   3 If I link modules together and than compile to .s - it crashes in the early
   beginning.
3. If I link modules together and compile to .s with llc --disable-fp-elim - it
   crashes in the end...

Could you please:

1. Attach .s files
2. Tell, where it segfaults for you (just gdb + backtrace + disassembly)
3. How can I reproduce stuff from the "clean" hash2 (afaik, you're testing new
   predsimplify?)

bad assembly from llc

good asm from gcc

GDB proves uninformative, which is unsurprising as the input is asm files:

Program received signal SIGSEGV, Segmentation fault.
__gnu_cxx::hashtable<std::pair<char const* const, int>, char const*,
__gnu_cxx::hash<char const*>, std::_Select1st<std::pair<char const* const, int>

, eqstr, std::allocator >::_M_initialize_buckets () at bugpoint.test.bc.s:48
48 movl (%esi), %esi
Current language: auto; currently asm
(gdb) bt
#​0 __gnu_cxx::hashtable<std::pair<char const* const, int>, char const*,
__gnu_cxx::hash<char const*>, std::_Select1st<std::pair<char const* const, int>
, eqstr, std::allocator >::_M_initialize_buckets () at bugpoint.test.bc.s:48

To reproduce:

$ cd projects/llvm-test/SingleSource/Benchmarks/Shootout-C++
$ make Output/hash2.linked.rbc
$ opt -idom -etforest -verify -lowersetjmp -funcresolve -raiseallocs
-simplifycfg -idom -domtree -domfrontier -mem2reg -globalopt -globaldce
-ipconstprop -deadargelim -instcombine -simplifycfg -basiccg -prune-eh -inline
-argpromotion -raise -tailduplicate -instcombine -simplifycfg -idom -domtree
-domfrontier -scalarrepl -instcombine -break-crit-edges -etforest -predsimplify
-tailcallelim -simplifycfg -reassociate -idom -etforest -loops -domset -domtree
-loopsimplify -domfrontier -licm -lcssa -loop-unswitch -instcombine
-scalar-evolution -indvars -loop-unroll -instcombine -idom -domset -load-vn
-etforest -domtree -gcse -sccp -instcombine -break-crit-edges -condprop -dse
-mergereturn -postidom -postdomtree -postdomfrontier -adce -simplifycfg
-simplify-libcalls -printusedtypes -deadtypeelim -constmerge -idom -etforest
-verify Output/hash2.linked.rbc -o hash2.bc
$ llc hash2.bc -o hash2.s
$ g++ hash2.s -o hash2.exe
$ ./hash2.exe
Segmentation fault

If you want to bugpoint the error, run this:
$ bugpoint hash2.bc -run-llc -append-exit-code -rel-tolerance 0.00000001
-Xlinker=-lm -Xlinker=-lstdc++ -input=/dev/null -timeout=500 --tool-args
-enable-correct-eh-support --args --

Well. The crash at my side is the same (mov (%esi), %esi). It seems, that
something wrong was on stack, since %esi doesn't contain "good" address. Will
investigate...

Well. Seems to be something wrong with fp elimination.

If I extract failed function and compile it separately - everything is ok. Even
more, the assembler for extracted function is the same as the assembler of
functions "inside" big assembler.

So, the problem is somewhere outside of the crashed function... Looking
further... I've removed extracted function from "bad" assembler and diffed the
output. Diff was prety nice (- = "good", + = "bad"):
@@ -2959,6 +2959,7 @@
subl %eax, %edi
movl %edi, 8(%esp)
call memmove

•   subl $12, %esp
    movl %edi, %eax
    sarl $31, %eax
    shrl $30, %eax
  

@@ -3021,7 +3022,6 @@
subl %eax, %edi
movl %edi, 8(%esp)
call memmove

•   subl $12, %esp
    movl %edi, %eax
    sarl $31, %eax
    shrl $30, %eax
  

@@ -3053,6 +3053,7 @@
subl %eax, %edi
movl %edi, 8(%esp)
call memmove

•   subl $12, %esp
    movl %edi, %eax
    sarl $31, %eax
    shrl $30, %eax
  

And so on. Actually files differ only on that "subl $12, %esp" lines. Cool.

Afaik, there shouldn't be any stack-manipulation at all here.

And finally:

1. before FP elimination:
   entry (0x8754fa0, LLVM BB @​0x87569c8, ID#0):
   ADJCALLSTACKDOWN 12, %ESP, %ESP
   %EAX = MOV32rm <fi#-1>, 1, %NOREG, 0
   MOV32mr %ESP, 1, %NOREG, 4, %EAX
   %ECX = MOV32rm <fi#-2>, 1, %NOREG, 0
   %ECX = SUB32rr %ECX, %EAX
   %EAX = MOV32rr %ECX
   %EAX = SAR32ri %EAX, 31
   %EAX = SHR32ri %EAX, 30
   %ECX = ADD32rr %ECX, %EAX
   %EAX = MOV32rr %ECX
   %EAX = AND32ri8 %EAX, 4294967292
   MOV32mr %ESP, 1, %NOREG, 8, %EAX
   %ECX = SAR32ri %ECX, 2
   %ECX = NEG32r %ECX
   %ESI = LEA32r %NOREG, 4, %ECX, 0
   %ESI = ADD32rm %ESI, <fi#-3>, 1, %NOREG, 0
   MOV32mr %ESP, 1, %NOREG, 0, %ESI
   CALLpcrel32 es:memmove, %EAX<imp-def,dead>, %ECX<imp-def,dead>,
   %EDX<imp-def,dead>, %FP0<imp-def,dead>, %FP1<imp-d$
   ADJCALLSTACKUP 12, 12, %ESP, %ESP
   %EAX = MOV32rr %ESI
   RET %EAX<imp-use,kill>, %EAX

Note, there are 2 adjcall* calls for memmove call

2. After FP elimination we have:

entry (0x8754fa0, LLVM BB @​0x87569c8, ID#0):
%ESP = SUB32ri8 %ESP, 20
MOV32mr %ESP, 1, %NOREG, 16, %ESI
%EAX = MOV32rm %ESP, 1, %NOREG, 24
MOV32mr %ESP, 1, %NOREG, 4, %EAX
%ECX = MOV32rm %ESP, 1, %NOREG, 28
%ECX = SUB32rr %ECX, %EAX
%EAX = MOV32rr %ECX
%EAX = SAR32ri %EAX, 31
%EAX = SHR32ri %EAX, 30
%ECX = ADD32rr %ECX, %EAX
%EAX = MOV32rr %ECX
%EAX = AND32ri8 %EAX, 4294967292
MOV32mr %ESP, 1, %NOREG, 8, %EAX
%ECX = SAR32ri %ECX, 2
%ECX = NEG32r %ECX
%ESI = LEA32r %NOREG, 4, %ECX, 0
%ESI = ADD32rm %ESI, %ESP, 1, %NOREG, 32
MOV32mr %ESP, 1, %NOREG, 0, %ESI
CALLpcrel32 es:memmove, %EAX<imp-def,dead>, %ECX<imp-def,dead>,
%EDX<imp-def,dead>, %FP0<imp-def,dead>, %FP1<imp-d$
%ESP = SUB32ri8 %ESP, 12
%EAX = MOV32rr %ESI
%ESI = MOV32rm %ESP, 1, %NOREG, 16
%ESP = ADD32ri8 %ESP, 20
RET %EAX<imp-use,kill>, %EAX

Note, we've dropped adjcallstackdown (seems to be ok), but turned adjcallstackup
to sub esp, 12. Is it correct?

If I follow Anton's instructions and remove any %esp manipulation instruction
that follows a call memmove, the program runs fine. That's on the whole program,
not the bugpoint halves.

This was the result of "sret" patch. Codegen really thought, that memmove is
struct return function :) This was due to uninitialized stuff inside
LegalizedDAG.cpp. Fortunately, the only affected part is memmove call inside x86
backend (since memcpy & memset are custom lowered and X86ISelLowering.cpp was ok).

Sorry for the breakage.

Fixed with these patches:

http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20070129/043666.html
http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20070129/043667.html
http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20070129/043668.html