clang-scan-deps unable to find system headers in p1689 format #61006

mordante · 2023-02-26T17:40:36Z

While testing modules with CMake I ran into an issue that some system headers are not found.

I tested with Debian clang version 17.0.0 (++20230226071743+b8df886c1e2a-1~exp1~20230226071846.576)

And this code

module;
#include <vector>
export module test;
void test() {}

Compilation with Clang succeeds

/usr/bin/clang++-17 -stdlib=libc++ -std=c++20 -x c++ test.cppm -c -o /tmp/test.cppm.o

But testing with the dependency scanner

/usr/bin/clang-scan-deps-17 -format=p1689 -- /usr/bin/clang++-17 -stdlib=libc++ -std=c++20 -x c++ test.cppm -c -o /tmp/test.cppm.o

Fails with the following output

Error while scanning dependencies for test.cppm:
In file included from test.cppm:3:
In file included from /usr/bin/../include/c++/v1/vector:281:
In file included from /usr/bin/../include/c++/v1/__algorithm/copy.h:12:
In file included from /usr/bin/../include/c++/v1/__algorithm/copy_move_common.h:12:
In file included from /usr/bin/../include/c++/v1/__algorithm/iterator_operations.h:15:
In file included from /usr/bin/../include/c++/v1/__iterator/advance.h:26:
In file included from /usr/bin/../include/c++/v1/cstdlib:87:
In file included from /usr/bin/../include/c++/v1/stdlib.h:94:
In file included from /usr/include/stdlib.h:31:
/usr/bin/../include/c++/v1/stddef.h:17:15: fatal error: 'stddef.h' file not found
{
  "revision": 0,
  "rules": [],
  "version": 1
}

The line where the header is not found uses #include_next <stddef.h> to include stddef.h.

I get similar errors when I don't specify libc++ and use libstdc++.

The text was updated successfully, but these errors were encountered:

llvmbot · 2023-02-26T17:40:52Z

@llvm/issue-subscribers-clang-modules

ChuanqiXu9 · 2023-02-27T03:01:06Z

What is the path of <stddef.h> in your system? Does it live in /usr/lib/clang/17/include? If it doesn't live in the directory, then I can understand the behavior.

I can reproduce this if I make a symbolic link of the real clang in another directory. Since the clang-scan-deps won't invoke <executable> actually. The clang-scan-deps just use it to get the resource dir. In the case that we can't put builtin headers like <stddef.h> in the default resource dir, we can use -I to specify the missing resource dir explicitly.

If this works for you, I'd like to update the document.

HailToDodongo · 2023-02-27T10:35:54Z

Not OP, but is was having the same issue exclusively on Ubuntu when installing the latest snapshot package via APT.
Installing it from source on another Ubuntu or Arch machine works fine.

By default the package seems to create a broken link:
/usr/lib/clang/17/include -> ../../llvm-17/lib/clang/17.0.0/include
But only /usr/lib/llvm-17/lib/clang/17/include exists.
Changing this doesn't fix it however.

After adding -I/usr/lib/clang/17/include to the clang++ options clang-scan-deps works,
without the option it still fails.

Is this behavior expected?
Because what is strange to me is that clang itself works fine even without the setting.

mordante · 2023-02-27T17:10:16Z

I did some more testing, using the good case with --trace-includes it finds stddef.h in the directory /usr/lib/llvm-17/lib/clang/17/include. Adding
-system /usr/lib/llvm-17/lib/clang/17/include fixes clang-scan-deps.

Your suggestion to use the real path instead of a symlink works too. I think it would be nice if clang-scan-deps would resolve the symlink and use the proper directory, but since there is a solution I would consider this low priority. I will update the documentation to mention this limitation.

mordante · 2023-02-27T18:38:24Z

I've created https://reviews.llvm.org/D144894

tschuett · 2023-02-27T18:45:19Z

> clang -print-resource-dir

clang knows where stddef.h is.

mordante · 2023-02-27T18:52:07Z

> clang -print-resource-dir
clang knows where stddef.h is.

Thanks for the info, this seems to work when using the symlink too. Maybe then it's trivial to fix. @ChuanqiXu9 if you want I can take a look at it.

ChuanqiXu9 · 2023-02-28T02:04:41Z

> clang -print-resource-dir
clang knows where stddef.h is.
Thanks for the info, this seems to work when using the symlink too. Maybe then it's trivial to fix. @ChuanqiXu9 if you want I can take a look at it.

I'll be pretty appreciated if you can look at it :)

huangqinjin · 2023-03-26T16:43:00Z

I am trying to use clang-scan-deps from LLVM 16 as the scanner for gcc.

Use the same code and replace /usr/bin/clang++-17 with /usr/bin/g++:

$ /opt/llvm-16/bin/clang-scan-deps -format=p1689 -- /usr/bin/g++ -std=c++20 -x c++ test.cppm -c -o /tmp/test.cppm.o
Error while scanning dependencies for test.cppm:
In file included from test.cppm:3:
In file included from /usr/bin/../lib64/gcc/x86_64-suse-linux/13/../../../../include/c++/13/vector:80:
In file included from /usr/bin/../lib64/gcc/x86_64-suse-linux/13/../../../../include/c++/13/bits/memory_resource.h:38:
/usr/bin/../lib64/gcc/x86_64-suse-linux/13/../../../../include/c++/13/cstddef:50:10: fatal error: 'stddef.h' file not found
{
  "revision": 0,
  "rules": [],
  "version": 1
}

On my system, stddef.h is in /usr/lib64/gcc/x86_64-suse-linux/13/include.

Manually adding -isystem /usr/lib64/gcc/x86_64-suse-linux/13/include resolves the issue.

superg · 2023-04-30T16:15:44Z

Experiencing the same issue with Ubuntu 22.04 clang-17 nightly packages.
Two symlinks point to non existing paths:

/usr/lib/clang/17/include -> ../../llvm-17/lib/clang/17.0.0/include
/usr/lib/clang/17/lib -> ../../llvm-17/lib/clang/17.0.0/lib

My current workaround:
sudo ln -s /usr/lib/llvm-17/lib/clang/17 /usr/lib/llvm-17/lib/clang/17.0.0

Splines · 2023-06-02T17:11:54Z

Two symlinks point to non existing paths:

Can confirm this for Clang16:

/usr/lib/clang/16/include -> ../../llvm-16/lib/clang/16.0.5/include
/usr/lib/clang/16/lib -> ../../llvm-16/lib/clang/16.0.5/lib

although I only have these folders:

/usr/lib/llvm-16/lib/clang/16/include/
/usr/lib/llvm-16/lib/clang/16/lib/

Not working yet due to `CMAKE_CXX_COMPILER_CLANG_SCAN_DEPS-NOTFOUND`. Useful links: - https://github.com/Kitware/CMake/blob/master/Help/dev/experimental.rst - llvm/llvm-project#61006 (comment) - https://stackoverflow.com/a/61244367/9655481

namniav · 2023-08-03T11:10:53Z

I have similar issue using clang-scan-deps with Apple Clang and macOS SDK. Leaving information in case it's helpful for fixing or for folks to find this issue.

$ clang-scan-deps --format=p1689 -- \
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ \
-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk \
-std=c++20 -c test.cpp -o test.cpp.o

Error while scanning dependencies for test.cpp:
In file included from test.cpp:1:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/iostream:37:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/ios:214:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/iosfwd:95:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/wchar.h:119:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/wchar.h:89:10: fatal error: 'stdarg.h' file not found
{
  "revision": 0,
  "rules": [],
  "version": 1
}

test.cpp just includes <iostream> and does nothing else.

The command following -- works along without clang-scan-deps. By adding -MD we can also exam the location of stdarg.h which clang-scan-deps fails to find.

$ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ \
-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk \
-std=c++20 -MD -c test.cpp -o test.cpp.o

In test.cpp.d:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.1.6/include/stdarg.h

A temporary sloppy workaround, at user side of clang-scan-deps, could be adding this directory to -I flag in clang-scan-deps command, but I'm not sure whether it would introduce other problems that I did't encounter yet.

ChuanqiXu9 · 2023-08-04T09:52:02Z

I sent https://reviews.llvm.org/D157081

ChuanqiXu9 · 2023-08-11T05:58:13Z

Sadly, the above patch is incorrect strictly. To solve the issue manually, we can pass --resource-dir-recipe invoke-compiler to clang-scan-deps. What the option did actually is invoke <clang++> -print-resource-dir and recorded the result and then pass the result to the command line. It may be a little bit inefficient in the current usgae of clang-scan-deps with cmake. Since cmake will invoke clang-scan-deps a lot of times for a single file and then clang-scan-deps itself can't cache the results.

Address #61006. The actual reason for the issue is about the usage of clang-scan-deps instead of its functionalities since clang-scan-deps has already considered the problem before for clang modules. So this patch tries to document the corresponding solution and add a test case for it to address the issue.

ChuanqiXu9 · 2023-08-11T07:19:44Z

Now I feel it is primarily a usage issue and we can't do a lot inside the clang-scan-deps itself. So I update the document (https://clang.llvm.org/docs/StandardCPlusPlusModules.html#possible-issues-failed-to-find-system-headers) and I am going to close the issue itself since there may not be more things to do in our repo.

While it is indeed user-infriendly and annoying, since clang-scan-deps is mainly used by build systems, so we can hope build systems to handle this well. Then we should file issues with corresponding build systems.

ChuanqiXu9 · 2023-08-11T07:29:50Z

The issue for cmake I registered is https://gitlab.kitware.com/cmake/cmake/-/issues/25180

Automated commit created by applying diff 549726 Phabricator-ID: PHID-HMBT-3hxbcvj4law6shs3llt3 Review-ID: D157817

This patch solves two problems with the recently added test, which caused CI failure in an internal downstream system: 1. It connects subsequent `RUN` statements with `&&` into a single statement; otherwise, the variable defined in the first of them is out of scope in the subsequent ones where it is used. 2. In our hermetic testing environment, the clang++ binary under test does not have access to standard include paths (`/usr/include` etc.), so the test fails because it does not find `stddef.h`. The patch thus changes the test to use a locally created `a.h` header file. Differential Revision: https://reviews.llvm.org/D157817

topazus · 2023-10-30T01:17:39Z

After adding -I/usr/lib/clang/17/include to the clang++ options clang-scan-deps works,
without the option it still fails.

Many thanks. I also encountered the similar error, which I used clang 17 and CMake 3.28 to compile a simple hello world project with c++20 modules, it also worked for me after adding -I/usr/lib/clang/17/include to clang++ compiler option.

ScherbakovAl · 2023-12-13T10:01:26Z

Hello!
If I use /usr/lib64/ccache/clang, then I get an error "/usr/include/wchar.h:35:10: fatal error: 'stddef.h' file not found".
if I add set(CMAKE_CXX_FLAGS "-I/usr/lib/clang/17/include"), then the project will be assembled.

If I use /usr/bin/clang , then the project will be built without adding a flag "-I/usr/lib/clang/17/include", but when setting up cmake there will be the following messages:

[proc] The command: clang -v exited with code: 1
[proc] The command: clang++ -v exited with code: 1

I'm using Fedora 39.

ChuanqiXu9 · 2023-12-13T10:20:21Z

Hello! If I use /usr/lib64/ccache/clang, then I get an error "/usr/include/wchar.h:35:10: fatal error: 'stddef.h' file not found". if I add set(CMAKE_CXX_FLAGS "-I/usr/lib/clang/17/include"), then the project will be assembled.

If I use /usr/bin/clang , then the project will be built without adding a flag "-I/usr/lib/clang/17/include", but when setting up cmake there will be the following messages:

[proc] The command: clang -v exited with code: 1 [proc] The command: clang++ -v exited with code: 1

I'm using Fedora 39.

I don't get your point quite a lot. For the problem of failed to find , we recorded it here: https://clang.llvm.org/docs/StandardCPlusPlusModules.html#possible-issues-failed-to-find-system-headers. And if there are other problems, I think a new issue may be worthy.

mordante added the clang:modules C++20 modules and Clang Header Modules label Feb 26, 2023

mordante assigned ChuanqiXu9 Feb 26, 2023

EugeneZelenko added the clang-tools-extra label Aug 4, 2023

ChuanqiXu9 closed this as completed Aug 11, 2023

llvm-premerge-tests-bot pushed a commit to llvm-premerge-tests/llvm-project that referenced this issue Aug 13, 2023

[clang] Fix ClangScanDeps test for llvm#61006.

a06fb76

Automated commit created by applying diff 549726 Phabricator-ID: PHID-HMBT-3hxbcvj4law6shs3llt3 Review-ID: D157817

hmenke mentioned this issue Dec 13, 2023

llvmPackages_17.clang: clang 17 cannot locate C++ std headers NixOS/nixpkgs#273875

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

clang-scan-deps unable to find system headers in p1689 format #61006

clang-scan-deps unable to find system headers in p1689 format #61006

mordante commented Feb 26, 2023 •

edited by VoltrexKeyva

llvmbot commented Feb 26, 2023

ChuanqiXu9 commented Feb 27, 2023

HailToDodongo commented Feb 27, 2023 •

edited

mordante commented Feb 27, 2023

mordante commented Feb 27, 2023

tschuett commented Feb 27, 2023

mordante commented Feb 27, 2023

ChuanqiXu9 commented Feb 28, 2023

huangqinjin commented Mar 26, 2023

superg commented Apr 30, 2023

Splines commented Jun 2, 2023

namniav commented Aug 3, 2023 •

edited

ChuanqiXu9 commented Aug 4, 2023

ChuanqiXu9 commented Aug 11, 2023 •

edited

ChuanqiXu9 commented Aug 11, 2023

ChuanqiXu9 commented Aug 11, 2023

topazus commented Oct 30, 2023

ScherbakovAl commented Dec 13, 2023 •

edited

ChuanqiXu9 commented Dec 13, 2023

clang-scan-deps unable to find system headers in p1689 format #61006

clang-scan-deps unable to find system headers in p1689 format #61006

Comments

mordante commented Feb 26, 2023 • edited by VoltrexKeyva

llvmbot commented Feb 26, 2023

ChuanqiXu9 commented Feb 27, 2023

HailToDodongo commented Feb 27, 2023 • edited

mordante commented Feb 27, 2023

mordante commented Feb 27, 2023

tschuett commented Feb 27, 2023

mordante commented Feb 27, 2023

ChuanqiXu9 commented Feb 28, 2023

huangqinjin commented Mar 26, 2023

superg commented Apr 30, 2023

Splines commented Jun 2, 2023

namniav commented Aug 3, 2023 • edited

ChuanqiXu9 commented Aug 4, 2023

ChuanqiXu9 commented Aug 11, 2023 • edited

ChuanqiXu9 commented Aug 11, 2023

ChuanqiXu9 commented Aug 11, 2023

topazus commented Oct 30, 2023

ScherbakovAl commented Dec 13, 2023 • edited

ChuanqiXu9 commented Dec 13, 2023

mordante commented Feb 26, 2023 •

edited by VoltrexKeyva

HailToDodongo commented Feb 27, 2023 •

edited

namniav commented Aug 3, 2023 •

edited

ChuanqiXu9 commented Aug 11, 2023 •

edited

ScherbakovAl commented Dec 13, 2023 •

edited