LLVM Bugzilla is read-only and represents the historical archive of all LLVM issues filled before November 26, 2021. Use github to submit LLVM bugs

Bug 38473 - 7.0.0 rc1: 16 libc++ tests failing on macOS/x86_64 (Exit Code: -11)
Summary: 7.0.0 rc1: 16 libc++ tests failing on macOS/x86_64 (Exit Code: -11)
Status: RESOLVED FIXED
Alias: None
Product: libc++
Classification: Unclassified
Component: All Bugs (show other bugs)
Version: 7.0
Hardware: PC All
: P normal
Assignee: Louis Dionne
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-08-07 13:21 PDT by Vedant Kumar
Modified: 2020-09-10 10:26 PDT (History)
5 users (show)

See Also:
Fixed By Commit(s):


Attachments
Minimal reproduction script (1.82 KB, text/plain)
2018-08-27 13:28 PDT, Louis Dionne
Details
Updated reproduction script (1.47 KB, text/plain)
2018-08-28 09:28 PDT, Louis Dionne
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Vedant Kumar 2018-08-07 13:21:18 PDT
These tests all fail in what looks to be the same way:

    libc++ :: std/language.support/support.exception/except.nested/assign.pass.cpp
    libc++ :: std/language.support/support.exception/except.nested/ctor_copy.pass.cpp
    libc++ :: std/language.support/support.exception/except.nested/ctor_default.pass.cpp
    libc++ :: std/language.support/support.exception/except.nested/rethrow_if_nested.pass.cpp
    libc++ :: std/language.support/support.exception/except.nested/rethrow_nested.pass.cpp
    libc++ :: std/language.support/support.exception/propagation/make_exception_ptr.pass.cpp
    libc++ :: std/language.support/support.exception/propagation/rethrow_exception.pass.cpp
    libc++ :: std/thread/futures/futures.async/async.pass.cpp
    libc++ :: std/thread/futures/futures.promise/dtor.pass.cpp
    libc++ :: std/thread/futures/futures.promise/set_exception.pass.cpp
    libc++ :: std/thread/futures/futures.promise/set_exception_at_thread_exit.pass.cpp
    libc++ :: std/thread/futures/futures.shared_future/get.pass.cpp
    libc++ :: std/thread/futures/futures.task/futures.task.members/dtor.pass.cpp
    libc++ :: std/thread/futures/futures.task/futures.task.members/make_ready_at_thread_exit.pass.cpp
    libc++ :: std/thread/futures/futures.task/futures.task.members/operator.pass.cpp
    libc++ :: std/thread/futures/futures.unique_future/get.pass.cpp

To reproduce the failure, I did:

$ export MACOSX_DEPLOYMENT_TARGET="10.9"
$ ./llvm/utils/release/test-release.sh -release 7.0.0 -rc 1 -triple x86_64-apple-darwin -no-compare-files -j20

Here's a sample failure:

FAIL: libc++ :: std/language.support/support.exception/except.nested/assign.pass.cpp (49834 of 54980)
******************** TEST 'libc++ :: std/language.support/support.exception/except.nested/assign.pass.cpp' FAILED ********************
Compiled With: ['/Users/vsk/src/llvm.org-master/rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++', '-o', '/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.o', '-x', 'c++', '/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/std/language.support/support.exception/except.nested/assign.pass.cpp', '-c', '-v', '-arch', 'x86_64', '-mmacosx-version-min=10.13', '-D_LIBCPP_DISABLE_AVAILABILITY', '-ftemplate-depth=270', '-Werror=thread-safety', '-std=c++2a', '-include', '/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/support/nasty_macros.hpp', '-nostdinc++', '-I/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/include', '-I/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/include/c++build', '-isysroot', '/Volumes/Xcode10A230a_m18A350_i16A343_t16J338_w16R338a_b16P349_XcodeInternals_32bitSupport_FastSim_Boost_ASan_36GB/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk', '-I/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/support', '-DLIBCXX_FILESYSTEM_STATIC_TEST_ROOT="/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/std/input.output/filesystems/Inputs/static_test_env"', '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_ROOT="/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/filesystem/Output/dynamic_env"', '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_HELPER="/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python /Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/support/filesystem_dynamic_test_helper.py"', '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER', '-Wall', '-Wextra', '-Werror', '-Wuser-defined-warnings', '-Wshadow', '-Wno-unused-command-line-argument', '-Wno-attributes', '-Wno-pessimizing-move', '-Wno-c++11-extensions', '-Wno-user-defined-literals', '-Wno-noexcept-type', '-Wno-aligned-allocation-unavailable', '-Wsign-compare', '-Wunused-variable', '-Wunused-parameter', '-Wunreachable-code', '-Wno-conversion', '-Wno-unused-local-typedef', '-c', '&&', '/Users/vsk/src/llvm.org-master/rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++', '-o', '/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.exe', '/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.o', '-v', '-arch', 'x86_64', '-mmacosx-version-min=10.13', '-D_LIBCPP_DISABLE_AVAILABILITY', '-ftemplate-depth=270', '-L/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib', '-Wl,-rpath,/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib', '-nodefaultlibs', '-lc++experimental', '-lc++fs', '-lc++', '-lSystem']
Command: ['/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.exe']
Exit Code: -11

In this case, assign.pass.cpp.exe does not exist. The directory it's supposed to be in exists and is empty.

If I strip out the single-quotes, perform the compile+link steps manually, and run assign.pass.cpp.exe it passes.
Comment 1 Marshall Clow (home) 2018-08-13 07:13:41 PDT
I'll be looking at this this morning.
Comment 2 Marshall Clow (home) 2018-08-14 14:27:03 PDT
Odd that even though you exported MACOSX_DEPLOYMENT_TARGET="10.9", the compile command contains '-mmacosx-version-min=10.13'.

(Thanks to Louis for noticing this)
Comment 3 Marshall Clow (home) 2018-08-14 20:24:10 PDT
There's clearly multiple things going on here.

0) Error -11 is a segmentation fault.
1) The reason you can't find the programs is that lit removes them after they've been run. They were built successfully.
2) When you build them manually, and then run them, you are running against the system dylib, so you don't see the failure.
3) The programs are definitely crashing. (see below).
4) The programs are being built with a macosx target of 10.13, not 10.9

Here's how you can build (and run) just one test.

$ 'rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++' '-o' 'rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.o' '-x' 'c++' 'rc1/llvm.src/projects/libcxx/test/std/language.support/support.exception/except.nested/rethrow_nested.pass.cpp' '-c' '-v' '-arch' 'x86_64' '-mmacosx-version-min=10.9' '-D_LIBCPP_DISABLE_AVAILABILITY' '-ftemplate-depth=270' '-Werror=thread-safety' '-std=c++17' '-include' 'rc1/llvm.src/projects/libcxx/test/support/nasty_macros.hpp' '-nostdinc++' '-Irc1/llvm.src/projects/libcxx/include' '-Irc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/include/c++build' '-isysroot' '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk' '-Irc1/llvm.src/projects/libcxx/test/support' '-DLIBCXX_FILESYSTEM_STATIC_TEST_ROOT="rc1/llvm.src/projects/libcxx/test/std/input.output/filesystems/Inputs/static_test_env"' '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_ROOT="rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/filesystem/Output/dynamic_env"' '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_HELPER="/usr/bin/python rc1/llvm.src/projects/libcxx/test/support/filesystem_dynamic_test_helper.py"' '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER' '-Wall' '-Wextra' '-Werror' '-Wuser-defined-warnings' '-Wshadow' '-Wno-unused-command-line-argument' '-Wno-attributes' '-Wno-pessimizing-move' '-Wno-c++11-extensions' '-Wno-user-defined-literals' '-Wno-noexcept-type' '-Wno-aligned-allocation-unavailable' '-Wsign-compare' '-Wunused-variable' '-Wunused-parameter' '-Wunreachable-code' '-Wno-conversion' '-Wno-unused-local-typedef' '-c'

$ 'rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++' '-o' 'rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.exe' 'rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.o' '-v' '-arch' 'x86_64' '-mmacosx-version-min=10.9' '-D_LIBCPP_DISABLE_AVAILABILITY' '-ftemplate-depth=270' '-Lrc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib' '-Wl,-rpath,rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib' '-nodefaultlibs' '-lc++experimental' '-lc++fs' '-lc++' '-lSystem'


$ DYLD_LIBRARY_PATH=rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/lib rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.exe

Segmentation fault: 11

I suspect that some of the other bits - (like libcxxabi) are not built with the same options as libcxx and/or the test programs and that is causing the crashes.
Comment 4 Vedant Kumar 2018-08-24 10:52:31 PDT
Thanks Marshall, I can reproduce the crash when specifying the correct DYLD_LIBRARY_PATH:

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001001b00cd libunwind.dylib`_Unwind_RaiseException + 93
libunwind.dylib`_Unwind_RaiseException:
->  0x1001b00cd <+93>:  movaps %xmm0, 0x10(%r14)
    0x1001b00d2 <+98>:  leaq   -0x398(%rbp), %rbx
    0x1001b00d9 <+105>: movq   %rbx, %rdi
    0x1001b00dc <+108>: movq   %r15, %rsi
Target 0: (assign.pass.cpp.exe) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x00000001001b00cd libunwind.dylib`_Unwind_RaiseException + 93
    frame #1: 0x0000000100190e9f libc++abi.dylib`__cxa_rethrow_primary_exception + 239
    frame #2: 0x00000001000afebc libc++.1.dylib`std::rethrow_exception(std::exception_ptr) + 12
    frame #3: 0x0000000100000ab8 assign.pass.cpp.exe`main at assign.pass.cpp:53
    frame #4: 0x00007fff57147015 libdyld.dylib`start + 1
(lldb) image list
[  0] 8D2E4DC3-765E-3C89-A451-23AA36AFB082 0x0000000100000000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.exe 
[  1] 8A72DE9C-A136-3506-AA02-4BA2B82DCAF3 0x0000000100003000 /usr/lib/dyld 
[  2] 053A1AC0-ED15-38D8-8B05-174BDAEC9B72 0x00000001000a1000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/lib/libc++.1.dylib 
[  3] CD555F3B-FDDB-35E5-A2FB-FBBF3D62031A 0x00007fff54e60000 /usr/lib/libSystem.B.dylib 
[  4] 20AF2CAA-9E97-3FB5-9BCE-A4E793429348 0x0000000100173000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/lib/libc++abi.dylib 
...
[ 37] 1E105381-88F9-3296-813F-A1E9B14BD93C 0x00000001001a8000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/lib/libunwind.dylib
Comment 5 Vedant Kumar 2018-08-24 11:00:31 PDT
Could the GPFLT be due to an alignment exception?

The crashing instruction appears to be:

0x1001b00cd <+93>:  movaps %xmm0, 0x10(%r14)

I think that requires 16 byte alignment, but 0x10(%r14) isn't sufficiently aligned?

(lldb) reg read xmm0
    xmm0 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
(lldb) p/x $r14+0x10
(unsigned long) $2 = 0x00000001003002e8
Comment 6 Louis Dionne 2018-08-24 12:07:57 PDT
I'm starting to suspect this has something to do with the changes we did to disable the use of aligned allocation functions when the deployment target does not support them. Maybe there's something wrong with that change. I'll try to reproduce using `./llvm/utils/release/test-release.sh -release 7.0.0 -rc 1 -triple x86_64-apple-darwin -no-compare-files` and report when I'm able to repro.
Comment 7 Louis Dionne 2018-08-27 13:28:51 PDT
Created attachment 20783 [details]
Minimal reproduction script

This has nothing to do with the bootstrapping process. I can reproduce with a system compiler and the RC1 sources.
Comment 8 Louis Dionne 2018-08-27 14:04:51 PDT
I can now also reproduce on ToT, but one has to include libunwind in the set of built projects, which I was not doing initially (using the monorepo setup).
Comment 9 Louis Dionne 2018-08-28 09:28:44 PDT
Created attachment 20788 [details]
Updated reproduction script

Okay, so with Marshall's help, I was able to narrow it down to a single revision on libunwind:

r291830 (around Release 5) => SEGFAULT
r276215                    => SEGFAULT
r276214                    => no SEGFAULT

In other words, r276215 introduces the failure. I also attached the script I used to reproduce. My setup was a checkout of LLVM trunk with projects/libcxx on trunk, projects/libcxxabi on trunk, and projects/libunwind on the various revisions shown above.
Comment 10 Louis Dionne 2018-08-28 09:30:24 PDT
This has been failing for a long time, but I don't know why we only caught it recently. It is not impossible that we were picking up the system's libunwind, which may be older than that revision of libunwind.
Comment 11 Louis Dionne 2018-08-28 13:05:28 PDT
It's possible to reproduce on Trunk with our current lit test suite. It suffices to make sure that libunwind is being built `ninja -C build unwind` to trigger the issue.

Also note that configuring CMake with `-DLIBCXXABI_USE_LLVM_UNWINDER=ON` seems to solve the problem (that option defaults to OFF).
Comment 12 Louis Dionne 2018-08-28 14:12:07 PDT
With `-DLIBCXXABI_USE_LLVM_UNWINDER=OFF`, the command line used to build `cxa_exception.cpp` is:

```
$CXX  -D_LIBCPP_DISABLE_EXTERN_TEMPLATE -D_LIBCPP_ENABLE_CXX17_REMOVED_UNEXPECTED_FUNCTIONS -D_LIBCXXABI_BUILDING_LIBRARY -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS 
-Iprojects/libcxxabi/src 
-I/Users/ldionne/work/llvm/libcxxabi/src 
-I$SDKROOT/usr/include/libxml2 
-Iinclude 
-I/Users/ldionne/work/llvm/llvm/include 
-I/Users/ldionne/work/llvm/libcxxabi/include 
-I/Users/ldionne/work/llvm/libcxx/include 
-fPIC -fvisibility-inlines-hidden -std=c++11 -O2 -g -DNDEBUG -isysroot $SDKROOT -mmacosx-version-min=10.XX -fPIC -nostdinc++ -fstrict-aliasing -funwind-tables -D_DEBUG -std=c++11 -c /Users/ldionne/work/llvm/libcxxabi/src/cxa_exception.cpp
```

and `unwind.h` is: `$SDKROOT/usr/include/unwind.h`

With `-DLIBCXXABI_USE_LLVM_UNWINDER=ON`, the command line is:

```
$CXX -DLIBCXXABI_USE_LLVM_UNWINDER -D_LIBCPP_DISABLE_EXTERN_TEMPLATE -D_LIBCPP_ENABLE_CXX17_REMOVED_UNEXPECTED_FUNCTIONS -D_LIBCXXABI_BUILDING_LIBRARY -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS 
-Iprojects/libcxxabi/src 
-I/Users/ldionne/work/llvm/libcxxabi/src 
-I$SDKROOT/usr/include/libxml2 
-Iinclude 
-I/Users/ldionne/work/llvm/llvm/include 
-I/Users/ldionne/work/llvm/libcxxabi/include 
-I/Users/ldionne/work/llvm/libunwind/include 
-I/Users/ldionne/work/llvm/libcxx/include 
-fPIC -fvisibility-inlines-hidden -std=c++11 -O2 -g -DNDEBUG -isysroot $SDKROOT -mmacosx-version-min=10.XX -fPIC   -nostdinc++ -fstrict-aliasing -funwind-tables -D_DEBUG -std=c++11 -c /Users/ldionne/work/llvm/libcxxabi/src/cxa_exception.cpp
```

and `unwind.h` is `/Users/ldionne/work/llvm/libunwind/include/unwind.h`. Notice how `-DLIBCXXABI_USE_LLVM_UNWINDER=ON` causes `-I/Users/ldionne/work/llvm/libunwind/include` to be added to the header search paths.

In other words, without `-DLIBCXXABI_USE_LLVM_UNWINDER=ON`, we're using the system's `unwind.h`, to build libc++abi, and so we're using `struct _Unwind_Exception` provided by the system's `unwind.h`, which has 8 bit alignment (because it predates r276215, where the alignment was changed). However, when we link against LLVM's libunwind.dylib (using DYLD_LIBRARY_PATH as in the reproduction), we're linking against a libunwind.dylib that thinks the alignment of `struct _Unwind_Exception` is 16 bits, because it was built with the new headers.

This is where we get a mismatch: libc++abi thinks the alignment of `struct _Unwind_Exception` is 8, but libunwind.dylib think it is 16. 

As soon as we go back to using the system's libunwind.dylib (which also thinks alignment is 8), the problem goes away. Similarly, when we start building libc++abi.dylib with the headers that know the alignment to be 16, the problem goes away.
Comment 13 Louis Dionne 2018-08-28 15:42:37 PDT
I think we understand the problem pretty well at this point. Marshall and I have discussed the following solutions:

1. When building libunwind, use the target system's `unwind.h` headers. This means that we'll build and ship a libunwind.dylib that's compatible with the headers present on the system. This also means that libc++abi will have to be built against those system headers, otherwise we'll run into the issue encountered here, but reversed (libc++abi will think 16 bytes alignment, but libunwind will think 8 bytes, which is technically OK but let's not go there). The downside of this approach is that we're not shipping a release that's self consistent: the unwind.h header shipped with LLVM 7 should never be used, because the libunwind.dylib will have been built with different and incompatible headers.

2. Build libc++abi AND libunwind against the LLVM 7 `unwind.h` headers. This means that we'd ship libc++.dylib/libc++abi.dylib/libunwind.dylib that are consistent with the headers provided with LLVM 7, but not with the system's headers. The downside is that a program using the system's `unwind.h` headers could NOT be linked against libc++abi.dylib/libunwind.dylib shipped with LLVM 7, because those would have been built with newer and incompatible libunwind headers.

We currently lean towards option (1) for lack of a better solution.
Comment 14 Hans Wennborg 2018-09-04 02:54:19 PDT
> We currently lean towards option (1) for lack of a better solution.

Is there a patch or something for this?
Comment 15 Louis Dionne 2018-09-04 09:29:09 PDT
Not yet. I've been distracted by other work and you mentioned this was not a release blocker for the moment, so I set this aside. Please LMK if you feel like this needs to be resolved for the release and I'll try to re-prioritize.
Comment 16 Hans Wennborg 2018-09-05 00:55:37 PDT
(In reply to Louis Dionne from comment #15)
> Not yet. I've been distracted by other work and you mentioned this was not a
> release blocker for the moment, so I set this aside. Please LMK if you feel
> like this needs to be resolved for the release and I'll try to re-prioritize.

Right, let's not wait for this.
Comment 17 Louis Dionne 2020-09-10 10:26:53 PDT
Since then, Apple has updated the version of libunwind (both the headers and the dylib) shipped in SDKs. I don't think this issue is still relevant. Vedant, please re-open if you disagree.