These tests all fail in what looks to be the same way: libc++ :: std/language.support/support.exception/except.nested/assign.pass.cpp libc++ :: std/language.support/support.exception/except.nested/ctor_copy.pass.cpp libc++ :: std/language.support/support.exception/except.nested/ctor_default.pass.cpp libc++ :: std/language.support/support.exception/except.nested/rethrow_if_nested.pass.cpp libc++ :: std/language.support/support.exception/except.nested/rethrow_nested.pass.cpp libc++ :: std/language.support/support.exception/propagation/make_exception_ptr.pass.cpp libc++ :: std/language.support/support.exception/propagation/rethrow_exception.pass.cpp libc++ :: std/thread/futures/futures.async/async.pass.cpp libc++ :: std/thread/futures/futures.promise/dtor.pass.cpp libc++ :: std/thread/futures/futures.promise/set_exception.pass.cpp libc++ :: std/thread/futures/futures.promise/set_exception_at_thread_exit.pass.cpp libc++ :: std/thread/futures/futures.shared_future/get.pass.cpp libc++ :: std/thread/futures/futures.task/futures.task.members/dtor.pass.cpp libc++ :: std/thread/futures/futures.task/futures.task.members/make_ready_at_thread_exit.pass.cpp libc++ :: std/thread/futures/futures.task/futures.task.members/operator.pass.cpp libc++ :: std/thread/futures/futures.unique_future/get.pass.cpp To reproduce the failure, I did: $ export MACOSX_DEPLOYMENT_TARGET="10.9" $ ./llvm/utils/release/test-release.sh -release 7.0.0 -rc 1 -triple x86_64-apple-darwin -no-compare-files -j20 Here's a sample failure: FAIL: libc++ :: std/language.support/support.exception/except.nested/assign.pass.cpp (49834 of 54980) ******************** TEST 'libc++ :: std/language.support/support.exception/except.nested/assign.pass.cpp' FAILED ******************** Compiled With: ['/Users/vsk/src/llvm.org-master/rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++', '-o', '/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.o', '-x', 'c++', '/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/std/language.support/support.exception/except.nested/assign.pass.cpp', '-c', '-v', '-arch', 'x86_64', '-mmacosx-version-min=10.13', '-D_LIBCPP_DISABLE_AVAILABILITY', '-ftemplate-depth=270', '-Werror=thread-safety', '-std=c++2a', '-include', '/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/support/nasty_macros.hpp', '-nostdinc++', '-I/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/include', '-I/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/include/c++build', '-isysroot', '/Volumes/Xcode10A230a_m18A350_i16A343_t16J338_w16R338a_b16P349_XcodeInternals_32bitSupport_FastSim_Boost_ASan_36GB/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk', '-I/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/support', '-DLIBCXX_FILESYSTEM_STATIC_TEST_ROOT="/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/std/input.output/filesystems/Inputs/static_test_env"', '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_ROOT="/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/filesystem/Output/dynamic_env"', '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_HELPER="/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python /Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/support/filesystem_dynamic_test_helper.py"', '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER', '-Wall', '-Wextra', '-Werror', '-Wuser-defined-warnings', '-Wshadow', '-Wno-unused-command-line-argument', '-Wno-attributes', '-Wno-pessimizing-move', '-Wno-c++11-extensions', '-Wno-user-defined-literals', '-Wno-noexcept-type', '-Wno-aligned-allocation-unavailable', '-Wsign-compare', '-Wunused-variable', '-Wunused-parameter', '-Wunreachable-code', '-Wno-conversion', '-Wno-unused-local-typedef', '-c', '&&', '/Users/vsk/src/llvm.org-master/rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++', '-o', '/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.exe', '/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.o', '-v', '-arch', 'x86_64', '-mmacosx-version-min=10.13', '-D_LIBCPP_DISABLE_AVAILABILITY', '-ftemplate-depth=270', '-L/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib', '-Wl,-rpath,/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib', '-nodefaultlibs', '-lc++experimental', '-lc++fs', '-lc++', '-lSystem'] Command: ['/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.exe'] Exit Code: -11 In this case, assign.pass.cpp.exe does not exist. The directory it's supposed to be in exists and is empty. If I strip out the single-quotes, perform the compile+link steps manually, and run assign.pass.cpp.exe it passes.
I'll be looking at this this morning.
Odd that even though you exported MACOSX_DEPLOYMENT_TARGET="10.9", the compile command contains '-mmacosx-version-min=10.13'. (Thanks to Louis for noticing this)
There's clearly multiple things going on here. 0) Error -11 is a segmentation fault. 1) The reason you can't find the programs is that lit removes them after they've been run. They were built successfully. 2) When you build them manually, and then run them, you are running against the system dylib, so you don't see the failure. 3) The programs are definitely crashing. (see below). 4) The programs are being built with a macosx target of 10.13, not 10.9 Here's how you can build (and run) just one test. $ 'rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++' '-o' 'rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.o' '-x' 'c++' 'rc1/llvm.src/projects/libcxx/test/std/language.support/support.exception/except.nested/rethrow_nested.pass.cpp' '-c' '-v' '-arch' 'x86_64' '-mmacosx-version-min=10.9' '-D_LIBCPP_DISABLE_AVAILABILITY' '-ftemplate-depth=270' '-Werror=thread-safety' '-std=c++17' '-include' 'rc1/llvm.src/projects/libcxx/test/support/nasty_macros.hpp' '-nostdinc++' '-Irc1/llvm.src/projects/libcxx/include' '-Irc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/include/c++build' '-isysroot' '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk' '-Irc1/llvm.src/projects/libcxx/test/support' '-DLIBCXX_FILESYSTEM_STATIC_TEST_ROOT="rc1/llvm.src/projects/libcxx/test/std/input.output/filesystems/Inputs/static_test_env"' '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_ROOT="rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/filesystem/Output/dynamic_env"' '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_HELPER="/usr/bin/python rc1/llvm.src/projects/libcxx/test/support/filesystem_dynamic_test_helper.py"' '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER' '-Wall' '-Wextra' '-Werror' '-Wuser-defined-warnings' '-Wshadow' '-Wno-unused-command-line-argument' '-Wno-attributes' '-Wno-pessimizing-move' '-Wno-c++11-extensions' '-Wno-user-defined-literals' '-Wno-noexcept-type' '-Wno-aligned-allocation-unavailable' '-Wsign-compare' '-Wunused-variable' '-Wunused-parameter' '-Wunreachable-code' '-Wno-conversion' '-Wno-unused-local-typedef' '-c' $ 'rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++' '-o' 'rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.exe' 'rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.o' '-v' '-arch' 'x86_64' '-mmacosx-version-min=10.9' '-D_LIBCPP_DISABLE_AVAILABILITY' '-ftemplate-depth=270' '-Lrc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib' '-Wl,-rpath,rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib' '-nodefaultlibs' '-lc++experimental' '-lc++fs' '-lc++' '-lSystem' $ DYLD_LIBRARY_PATH=rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/lib rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.exe Segmentation fault: 11 I suspect that some of the other bits - (like libcxxabi) are not built with the same options as libcxx and/or the test programs and that is causing the crashes.
Thanks Marshall, I can reproduce the crash when specifying the correct DYLD_LIBRARY_PATH: * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT) frame #0: 0x00000001001b00cd libunwind.dylib`_Unwind_RaiseException + 93 libunwind.dylib`_Unwind_RaiseException: -> 0x1001b00cd <+93>: movaps %xmm0, 0x10(%r14) 0x1001b00d2 <+98>: leaq -0x398(%rbp), %rbx 0x1001b00d9 <+105>: movq %rbx, %rdi 0x1001b00dc <+108>: movq %r15, %rsi Target 0: (assign.pass.cpp.exe) stopped. (lldb) bt * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT) * frame #0: 0x00000001001b00cd libunwind.dylib`_Unwind_RaiseException + 93 frame #1: 0x0000000100190e9f libc++abi.dylib`__cxa_rethrow_primary_exception + 239 frame #2: 0x00000001000afebc libc++.1.dylib`std::rethrow_exception(std::exception_ptr) + 12 frame #3: 0x0000000100000ab8 assign.pass.cpp.exe`main at assign.pass.cpp:53 frame #4: 0x00007fff57147015 libdyld.dylib`start + 1 (lldb) image list [ 0] 8D2E4DC3-765E-3C89-A451-23AA36AFB082 0x0000000100000000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.exe [ 1] 8A72DE9C-A136-3506-AA02-4BA2B82DCAF3 0x0000000100003000 /usr/lib/dyld [ 2] 053A1AC0-ED15-38D8-8B05-174BDAEC9B72 0x00000001000a1000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/lib/libc++.1.dylib [ 3] CD555F3B-FDDB-35E5-A2FB-FBBF3D62031A 0x00007fff54e60000 /usr/lib/libSystem.B.dylib [ 4] 20AF2CAA-9E97-3FB5-9BCE-A4E793429348 0x0000000100173000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/lib/libc++abi.dylib ... [ 37] 1E105381-88F9-3296-813F-A1E9B14BD93C 0x00000001001a8000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/lib/libunwind.dylib
Could the GPFLT be due to an alignment exception? The crashing instruction appears to be: 0x1001b00cd <+93>: movaps %xmm0, 0x10(%r14) I think that requires 16 byte alignment, but 0x10(%r14) isn't sufficiently aligned? (lldb) reg read xmm0 xmm0 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00} (lldb) p/x $r14+0x10 (unsigned long) $2 = 0x00000001003002e8
I'm starting to suspect this has something to do with the changes we did to disable the use of aligned allocation functions when the deployment target does not support them. Maybe there's something wrong with that change. I'll try to reproduce using `./llvm/utils/release/test-release.sh -release 7.0.0 -rc 1 -triple x86_64-apple-darwin -no-compare-files` and report when I'm able to repro.
Created attachment 20783 [details] Minimal reproduction script This has nothing to do with the bootstrapping process. I can reproduce with a system compiler and the RC1 sources.
I can now also reproduce on ToT, but one has to include libunwind in the set of built projects, which I was not doing initially (using the monorepo setup).
Created attachment 20788 [details] Updated reproduction script Okay, so with Marshall's help, I was able to narrow it down to a single revision on libunwind: r291830 (around Release 5) => SEGFAULT r276215 => SEGFAULT r276214 => no SEGFAULT In other words, r276215 introduces the failure. I also attached the script I used to reproduce. My setup was a checkout of LLVM trunk with projects/libcxx on trunk, projects/libcxxabi on trunk, and projects/libunwind on the various revisions shown above.
This has been failing for a long time, but I don't know why we only caught it recently. It is not impossible that we were picking up the system's libunwind, which may be older than that revision of libunwind.
It's possible to reproduce on Trunk with our current lit test suite. It suffices to make sure that libunwind is being built `ninja -C build unwind` to trigger the issue. Also note that configuring CMake with `-DLIBCXXABI_USE_LLVM_UNWINDER=ON` seems to solve the problem (that option defaults to OFF).
With `-DLIBCXXABI_USE_LLVM_UNWINDER=OFF`, the command line used to build `cxa_exception.cpp` is: ``` $CXX -D_LIBCPP_DISABLE_EXTERN_TEMPLATE -D_LIBCPP_ENABLE_CXX17_REMOVED_UNEXPECTED_FUNCTIONS -D_LIBCXXABI_BUILDING_LIBRARY -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Iprojects/libcxxabi/src -I/Users/ldionne/work/llvm/libcxxabi/src -I$SDKROOT/usr/include/libxml2 -Iinclude -I/Users/ldionne/work/llvm/llvm/include -I/Users/ldionne/work/llvm/libcxxabi/include -I/Users/ldionne/work/llvm/libcxx/include -fPIC -fvisibility-inlines-hidden -std=c++11 -O2 -g -DNDEBUG -isysroot $SDKROOT -mmacosx-version-min=10.XX -fPIC -nostdinc++ -fstrict-aliasing -funwind-tables -D_DEBUG -std=c++11 -c /Users/ldionne/work/llvm/libcxxabi/src/cxa_exception.cpp ``` and `unwind.h` is: `$SDKROOT/usr/include/unwind.h` With `-DLIBCXXABI_USE_LLVM_UNWINDER=ON`, the command line is: ``` $CXX -DLIBCXXABI_USE_LLVM_UNWINDER -D_LIBCPP_DISABLE_EXTERN_TEMPLATE -D_LIBCPP_ENABLE_CXX17_REMOVED_UNEXPECTED_FUNCTIONS -D_LIBCXXABI_BUILDING_LIBRARY -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Iprojects/libcxxabi/src -I/Users/ldionne/work/llvm/libcxxabi/src -I$SDKROOT/usr/include/libxml2 -Iinclude -I/Users/ldionne/work/llvm/llvm/include -I/Users/ldionne/work/llvm/libcxxabi/include -I/Users/ldionne/work/llvm/libunwind/include -I/Users/ldionne/work/llvm/libcxx/include -fPIC -fvisibility-inlines-hidden -std=c++11 -O2 -g -DNDEBUG -isysroot $SDKROOT -mmacosx-version-min=10.XX -fPIC -nostdinc++ -fstrict-aliasing -funwind-tables -D_DEBUG -std=c++11 -c /Users/ldionne/work/llvm/libcxxabi/src/cxa_exception.cpp ``` and `unwind.h` is `/Users/ldionne/work/llvm/libunwind/include/unwind.h`. Notice how `-DLIBCXXABI_USE_LLVM_UNWINDER=ON` causes `-I/Users/ldionne/work/llvm/libunwind/include` to be added to the header search paths. In other words, without `-DLIBCXXABI_USE_LLVM_UNWINDER=ON`, we're using the system's `unwind.h`, to build libc++abi, and so we're using `struct _Unwind_Exception` provided by the system's `unwind.h`, which has 8 bit alignment (because it predates r276215, where the alignment was changed). However, when we link against LLVM's libunwind.dylib (using DYLD_LIBRARY_PATH as in the reproduction), we're linking against a libunwind.dylib that thinks the alignment of `struct _Unwind_Exception` is 16 bits, because it was built with the new headers. This is where we get a mismatch: libc++abi thinks the alignment of `struct _Unwind_Exception` is 8, but libunwind.dylib think it is 16. As soon as we go back to using the system's libunwind.dylib (which also thinks alignment is 8), the problem goes away. Similarly, when we start building libc++abi.dylib with the headers that know the alignment to be 16, the problem goes away.
I think we understand the problem pretty well at this point. Marshall and I have discussed the following solutions: 1. When building libunwind, use the target system's `unwind.h` headers. This means that we'll build and ship a libunwind.dylib that's compatible with the headers present on the system. This also means that libc++abi will have to be built against those system headers, otherwise we'll run into the issue encountered here, but reversed (libc++abi will think 16 bytes alignment, but libunwind will think 8 bytes, which is technically OK but let's not go there). The downside of this approach is that we're not shipping a release that's self consistent: the unwind.h header shipped with LLVM 7 should never be used, because the libunwind.dylib will have been built with different and incompatible headers. 2. Build libc++abi AND libunwind against the LLVM 7 `unwind.h` headers. This means that we'd ship libc++.dylib/libc++abi.dylib/libunwind.dylib that are consistent with the headers provided with LLVM 7, but not with the system's headers. The downside is that a program using the system's `unwind.h` headers could NOT be linked against libc++abi.dylib/libunwind.dylib shipped with LLVM 7, because those would have been built with newer and incompatible libunwind headers. We currently lean towards option (1) for lack of a better solution.
> We currently lean towards option (1) for lack of a better solution. Is there a patch or something for this?
Not yet. I've been distracted by other work and you mentioned this was not a release blocker for the moment, so I set this aside. Please LMK if you feel like this needs to be resolved for the release and I'll try to re-prioritize.
(In reply to Louis Dionne from comment #15) > Not yet. I've been distracted by other work and you mentioned this was not a > release blocker for the moment, so I set this aside. Please LMK if you feel > like this needs to be resolved for the release and I'll try to re-prioritize. Right, let's not wait for this.
Since then, Apple has updated the version of libunwind (both the headers and the dylib) shipped in SDKs. I don't think this issue is still relevant. Vedant, please re-open if you disagree.