Why can't my clangd see standard headers in nix develop shells?

Published on 2024-04-10

The problem

clangd often fails to find the standard library headers when I develop CMake-based C and C++ projects under nix develop shells. The errors from Eglot look like this:

main.cpp 1 error	clang [pp_file_not_found]: 'cstdio' file not found	(eglot-check)

The reason behind clangd’s failure to find standard library headers is that when CMake generates the compile_commands.json compilation database, it does not include the “directories implicitly searched by the compiler for header files” in its output. This is a reasonable design: clangd should know these standard library include paths by default, so it would be redundant to record the paths in the compilation database.

However, in the case of compiler toolchains from Nixpkgs, the binary executables (clang, clangd, etc.) are wrapped by the cc-wrapper.sh template. The the binary executables know nothing about the standard library include paths, and it’s instead the wrapper scripts’ responsibility to call the binaries with extra -isystem arguments to tell them about the standard library include and link paths and so on. An example can be seen in the cc-wrapper.sh linked above, where the wrapper script appends the C++ standard library include paths to another variable which is then passed when exec-ing the binary:

# Get the C++ standard library flags from the environment.
NIX_CFLAGS_COMPILE_@suffixSalt@+=" $NIX_CXXSTDLIB_COMPILE_@suffixSalt@"

# A few lines later:
extraAfter=(${hardeningCFlagsAfter[@]+"${hardeningCFlagsAfter[@]}"} $NIX_CFLAGS_COMPILE_@suffixSalt@)

# The final exec.
exec @prog@ \
     ${extraBefore+"${extraBefore[@]}"} \
     ${params+"${params[@]}"} \
     ${extraAfter+"${extraAfter[@]}"}

Going back to our missing headers problem, I thought that maybe clangd isn’t wrapped properly. However, since the merge of the PR fix clangd wrapper #120229 in 2021, the clangd binary is properly wrapped with a wrapper.

There are two clangd​s in PATH!

When I tried to inspect the wrapper scripts in the Nix store directly, I found that there are two clangd​s in PATH, with the first one being the unwrapped binary. This is undesirable.

$ whereis clangd
clangd: /nix/store/dnjh11dz3f3q34kh35akir9bwiskczd2-clang-17.0.6/bin/clangd /nix/store/1j2dsbw7fi55havdhc1ik0m2xbrsf1ds-clang-tools-17.0.6/bin/clangd

There must be something that runs before the various phases (if any) of the devShell and puts the path to unwrapped clangd into PATH. Let’s examine the builder of the devShell created by mkShellNoCC.

$ nix eval .\#devShell.x86_64-linux.builder
"/nix/store/a1s263pmsci9zykm5xcdf7x9rv26w6d5-bash-5.2p26/bin/bash"
$ nix eval .\#devShell.x86_64-linux.args
[ "-e" /nix/store/vvc1sjmsc1brf4fp6dpjv0z5fcjgspr8-source/pkgs/stdenv/generic/default-builder.sh ]

The builder is bash with the default builder script. Let’s examine the script.

if [ -e "$NIX_ATTRS_SH_FILE" ]; then . "$NIX_ATTRS_SH_FILE"; elif [ -f .attrs.sh ]; then . .attrs.sh; fi

source $stdenv/setup
genericBuild

At this point, the source $stdenv/setup setup script is rather suspicious. Running the script under the devShell with NIX_DEBUG set to a high number shows the culprit: The setup-hook of the clang wrapper adds the unwrapped clang to _PATH, a path that’s then added to PATH by $stdenv/setup itself.

$ NIX_DEBUG=7 bash $stdenv/setup
# (Omitted lines...)

# setup-hook of clang-wrapper starts running.
+ source /nix/store/xxbc7fnvar70sbq8ckks4fl8jagnw52y-clang-wrapper-17.0.6/nix-support/setup-hook
# (Omitted lines...)

# setup-hook of clang-wrapper adds the path to raw clangd here!
++ addToSearchPath _PATH /nix/store/dnjh11dz3f3q34kh35akir9bwiskczd2-clang-17.0.6/bin
# (Omitted lines...)

I have yet to find out why the setup hook of the clang wrapper does this. It’d be outside of the scope of this post, anyways.

The solution

Swap the order of packages in the call to mkShell (or mkShellNoCC) so that clang-tools comes before clangd. That is, instead of this:

# Bad - clangd cannot find stdlib headers
devShell = pkgs.mkShellNoCC {
  packages = [
    # clang comes first => unwrapped clangd comes first in PATH
    pkgs.clang
    pkgs.clang-tools
    pkgs.ninja
    pkgs.cmake
  ];
};

do this:

# Good - clangd works fine
devShell = pkgs.mkShellNoCC {
  packages = [
    # clang-tools comes first => wrapped clangd comes first in PATH
    pkgs.clang-tools
    pkgs.clang
    pkgs.ninja
    pkgs.cmake
  ];
};

Misc

hiPrio does not prioritize paths added by setup hooks

There is a recent comment on Discourse by gabyx suggesting a code snippet that not only orders clang-tools first but also uses hiPrio. I tried to use hiPrio without ordering clang-tools first to see if it helps, but the unwrapped clangd still comes first in PATH, so it is unclear to me what problem exactly the hiPrio solves in this case.

It would be nice to have hiPrio (or something similar) prioritize the paths from clang-tools over the paths added by setup hooks. If you have some insights to this, tell me about it.

Another workaround

For several years up to this post, I used to add the following workaround snippet to the bottom of the CMakeLists.txt file taken from this reply to a thread on NixOS Discourse by jacg from 2021. The workaround passes all the implicit header directories explicitly to the compilers, which makes them appear in the compilation database.

if(CMAKE_EXPORT_COMPILE_COMMANDS)
  set(CMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES ${CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES})
  set(CMAKE_C_STANDARD_INCLUDE_DIRECTORIES ${CMAKE_C_IMPLICIT_INCLUDE_DIRECTORIES})
endif()

The workaround did break some of my complex CMake projects due to CMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES interfering with some configuration checks. For simple projects the workaround mostly works, but editing CMakeLists.txt for each project isn’t exactly a pleasant experience.

Clangd is tied to clang

I’ve only recently learned that one isn’t supposed to mix-and-match clangd and clang from different versions (or even different builds). For example, when developing a project cross-compiled for Android using NDK, always use the clangd from NDK, which knows the NDK header paths. Using clangd from the clang-tools derivation in Nixpkgs results in missing std header paths, because it knows nothing about the NDK headers.

The CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES workaround sort of works even for the case of NDK development. Sort of. But the non-NDK include paths for the host are still passed by the clangd wrapper to the underlying binary, which may cause other LSP mis-diagnostics, so it’s best to avoid the the workaround.