Hermetic CUDA no longer respects TF_DOWNLOAD_CLANG

[tchatow]

The hermetic cuda toolchain with --config=cuda_clang always looks for a local clang. It is useful to be able to download a hermetic clang as well.

Even with the local cuda toolchain this option was somewhat broken since third_party/tsl/third_party/clang_toolchain/download_clang.bzl uses a Chromium clang built without Cuda support; I would patch this to download clang from https://github.com/llvm/llvm-project/releases instead.

[cheshire]

@beckerhe to me it would conceptually make most sense to fix the entire toolchain by default and download it, instead of trying to build with the system compiler.

[johnnynunez]

@beckerhe to me it would conceptually make most sense to fix the entire toolchain by default and download it, instead of trying to build with the system compiler.

I'm getting the same error always in jax:
external/xla/xla/stream_executor/cuda/cuda_status.h:22:10: fatal error: 'third_party/gpus/cuda/include/cuda.h' file not found
#include "third_party/gpus/cuda/include/cuda.h"

python3 build/build.py --enable_cuda --enable_nccl=False --cuda_compute_capabilities=sm_87 --bazel_options=--repo_env=LOCAL_CUDA_PATH="/usr/local/cuda-12.2" --bazel_options=--repo_env=LOCAL_CUDNN_PATH="/usr/lib/aarch64-linux-gnu"

what i'm doing wrong?

[medaminezghal]

@beckerhe to me it would conceptually make most sense to fix the entire toolchain by default and download it, instead of trying to build with the system compiler.

I'm getting the same error always in jax: external/xla/xla/stream_executor/cuda/cuda_status.h:22:10: fatal error: 'third_party/gpus/cuda/include/cuda.h' file not found #include "third_party/gpus/cuda/include/cuda.h"

python3 build/build.py --enable_cuda --enable_nccl=False --cuda_compute_capabilities=sm_87 --bazel_options=--repo_env=LOCAL_CUDA_PATH="/usr/local/cuda-12.2" --bazel_options=--repo_env=LOCAL_CUDNN_PATH="/usr/lib/aarch64-linux-gnu"

what i'm doing wrong?

Same problem here

[beckerhe]

Can you share the full log? I've never used LOCAL_CUDA_PATH and am not sure how it is supposed to work. @ybaturina do you see something wrong? Can you also share the xla_configure.bazelrc file that the configure script generates?

In general I would recommend using hermetic CUDA as that's the only tested way of consuming CUDA. Everything else is best-effort.

[ybaturina]

Hi, do the local folders have the structure as recommended in the docs?
https://github.com/openxla/xla/blob/main/docs/hermetic_cuda.md?plain=1#L210-L235

If not, usage of the local CUDA installation won't work.

[medaminezghal]

@ybaturina For solving problems related to CUDA headers, I just use, as you suggest, HERMETIC_CUDA_VERSION with LOCAL_CUDA_PATH.

But now, I got problems related to c++ headers: math.h not found.

[ybaturina]

hi @medaminezghal, would you attach the full log please?

[medaminezghal]

@ybaturina This is my build function for JAX:

    export HERMETIC_CUDA_VERSION="12.8.1"
    export HERMETIC_CUDNN_VERSION="9.8.0"

    export HERMETIC_CUDA_COMPUTE_CAPABILITIES=sm_50,sm_52,sm_53,sm_60,sm_61,sm_62,sm_70,sm_72,sm_75,sm_80,sm_86,sm_87,sm_89,sm_90,sm_90a,compute_90

    export JAXLIB_RELEASE=$pkgver

    cd $srcdir/jax-jax-v$pkgver
    build/build.py build \
        --bazel_options="--repo_env=HERMETIC_CUDA_COMPUTE_CAPABILITIES=${HERMETIC_CUDA_COMPUTE_CAPABILITIES}" \
        --bazel_options='--repo_env=HERMETIC_CUDA_VERSION="12.8.1"' \
        --bazel_options='--repo_env=HERMETIC_CUDNN_VERSION="9.8.0"' \
        --bazel_options='--repo_env=LOCAL_CUDA_PATH=/opt/cuda' \
        --bazel_options='--repo_env=LOCAL_CUDNN_PATH=/usr' \
        --bazel_options='--repo_env=LOCAL_NCCL_PATH=/usr' \
        --bazel_path="$srcdir/bazel-${_bazel_ver}-linux-x86_64" \
        --bazel_startup_options="--output_user_root=$srcdir/bazel"\
        --build_cuda_with_clang \
        --use_clang --clang_path='/usr/bin/clang' \
        --target_cpu_features=release \
        --verbose \
        --wheels=jaxlib,jax-cuda-plugin,jax-cuda-pjrt

This is the output of Bazel build:

ERROR: /home/medaminezghal/Desktop/python-jaxlib-cuda/src/jax-jax-v0.6.0/jaxlib/cuda/BUILD:74:13: Compiling jaxlib/gpu/make_batch_pointers.cu.cc failed: (Exit 1): clang-19 failed: error executing CppCompile command (from target //jaxlib/cuda:cuda_make_batch_pointers) 
  (cd /home/medaminezghal/Desktop/python-jaxlib-cuda/src/bazel/355f84808c611d40e5f319df2534e14a/execroot/__main__ && \
  exec env - \
    CLANG_COMPILER_PATH=/usr/bin/clang \
    CLANG_CUDA_COMPILER_PATH=/usr/bin/clang \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/cuda/bin:/opt/cuda/nsight_compute:/opt/cuda/nsight_systems/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl \
    PWD=/proc/self/cwd \
  /usr/bin/clang-19 -MD -MF bazel-out/k8-opt/bin/jaxlib/cuda/_objs/cuda_make_batch_pointers/make_batch_pointers.cu.pic.d '-frandom-seed=bazel-out/k8-opt/bin/jaxlib/cuda/_objs/cuda_make_batch_pointers/make_batch_pointers.cu.pic.o' '-DEIGEN_MAX_ALIGN_BYTES=64' -DEIGEN_ALLOW_UNALIGNED_SCALARS '-DEIGEN_USE_AVX512_GEMM_KERNELS=0' '-DJAX_GPU_CUDA=1' -iquote . -iquote bazel-out/k8-opt/bin -iquote external/local_config_cuda -iquote bazel-out/k8-opt/bin/external/local_config_cuda -iquote external/cuda_cudart -iquote bazel-out/k8-opt/bin/external/cuda_cudart -iquote external/cuda_cublas -iquote bazel-out/k8-opt/bin/external/cuda_cublas -iquote external/cuda_cccl -iquote bazel-out/k8-opt/bin/external/cuda_cccl -iquote external/cuda_nvtx -iquote bazel-out/k8-opt/bin/external/cuda_nvtx -iquote external/cuda_nvcc -iquote bazel-out/k8-opt/bin/external/cuda_nvcc -iquote external/cuda_cusolver -iquote bazel-out/k8-opt/bin/external/cuda_cusolver -iquote external/cuda_cufft -iquote bazel-out/k8-opt/bin/external/cuda_cufft -iquote external/cuda_cusparse -iquote bazel-out/k8-opt/bin/external/cuda_cusparse -iquote external/cuda_curand -iquote bazel-out/k8-opt/bin/external/cuda_curand -iquote external/cuda_cupti -iquote bazel-out/k8-opt/bin/external/cuda_cupti -iquote external/cuda_nvml -iquote bazel-out/k8-opt/bin/external/cuda_nvml -iquote external/cuda_nvjitlink -iquote bazel-out/k8-opt/bin/external/cuda_nvjitlink -iquote external/cuda_cudnn -iquote bazel-out/k8-opt/bin/external/cuda_cudnn -iquote external/xla -iquote bazel-out/k8-opt/bin/external/xla -iquote external/tsl -iquote bazel-out/k8-opt/bin/external/tsl -iquote external/eigen_archive -iquote bazel-out/k8-opt/bin/external/eigen_archive -iquote external/ml_dtypes_py -iquote bazel-out/k8-opt/bin/external/ml_dtypes_py -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -iquote external/local_config_tensorrt -iquote bazel-out/k8-opt/bin/external/local_config_tensorrt -iquote external/com_google_protobuf -iquote bazel-out/k8-opt/bin/external/com_google_protobuf -iquote external/zlib -iquote bazel-out/k8-opt/bin/external/zlib -iquote external/nvshmem -iquote bazel-out/k8-opt/bin/external/nvshmem -iquote external/nccl_archive -iquote bazel-out/k8-opt/bin/external/nccl_archive -Ibazel-out/k8-opt/bin/external/local_config_cuda/cuda/_virtual_includes/cuda_headers -Ibazel-out/k8-opt/bin/external/cuda_cudart/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cublas/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cccl/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_nvtx/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_nvcc/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cusolver/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cufft/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cusparse/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_curand/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cupti/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_nvml/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_nvjitlink/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cudnn/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers -Ibazel-out/k8-opt/bin/external/nvshmem/_virtual_includes/nvshmem_config -Ibazel-out/k8-opt/bin/external/nccl_archive/_virtual_includes/nccl_config -isystem external/local_config_cuda/cuda -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda -isystem external/cuda_cudart/include -isystem bazel-out/k8-opt/bin/external/cuda_cudart/include -isystem external/cuda_cublas/include -isystem bazel-out/k8-opt/bin/external/cuda_cublas/include -isystem external/cuda_cccl/include -isystem bazel-out/k8-opt/bin/external/cuda_cccl/include -isystem external/cuda_nvtx/include -isystem bazel-out/k8-opt/bin/external/cuda_nvtx/include -isystem external/cuda_nvcc/include -isystem bazel-out/k8-opt/bin/external/cuda_nvcc/include -isystem external/cuda_cusolver/include -isystem bazel-out/k8-opt/bin/external/cuda_cusolver/include -isystem external/cuda_cufft/include -isystem bazel-out/k8-opt/bin/external/cuda_cufft/include -isystem external/cuda_cusparse/include -isystem bazel-out/k8-opt/bin/external/cuda_cusparse/include -isystem external/cuda_curand/include -isystem bazel-out/k8-opt/bin/external/cuda_curand/include -isystem external/cuda_cupti/include -isystem bazel-out/k8-opt/bin/external/cuda_cupti/include -isystem external/cuda_nvml/include -isystem bazel-out/k8-opt/bin/external/cuda_nvml/include -isystem external/cuda_nvjitlink/include -isystem bazel-out/k8-opt/bin/external/cuda_nvjitlink/include -isystem external/cuda_cudnn/include -isystem bazel-out/k8-opt/bin/external/cuda_cudnn/include -isystem external/eigen_archive -isystem bazel-out/k8-opt/bin/external/eigen_archive -isystem external/eigen_archive/mkl_include -isystem bazel-out/k8-opt/bin/external/eigen_archive/mkl_include -isystem external/com_google_protobuf/src -isystem bazel-out/k8-opt/bin/external/com_google_protobuf/src -isystem external/zlib -isystem bazel-out/k8-opt/bin/external/zlib -fmerge-all-constants -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIC -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wno-invalid-partial-specialization -fno-omit-frame-pointer -no-canonical-prefixes -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections '--cuda-path=external/cuda_nvcc' '-fvisibility=hidden' -Wno-sign-compare -Wno-unknown-warning-option -Wno-stringop-truncation -Wno-array-parameter '-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir.' '-DNB_DOMAIN=jax' -Wno-gnu-offsetof-extensions -Qunused-arguments '-Werror=mismatched-tags' '-Wno-error=c23-extensions' -mavx -Wno-gnu-offsetof-extensions -Qunused-arguments '-Werror=mismatched-tags' '-Wno-error=c23-extensions' -mavx '-std=c++17' -x cuda '-DGOOGLE_CUDA=1' '--no-cuda-include-ptx=all' '--cuda-feature=+ptx85' '--cuda-gpu-arch=sm_50' '--cuda-gpu-arch=sm_52' '--cuda-gpu-arch=sm_53' '--cuda-gpu-arch=sm_60' '--cuda-gpu-arch=sm_61' '--cuda-gpu-arch=sm_62' '--cuda-gpu-arch=sm_70' '--cuda-gpu-arch=sm_72' '--cuda-gpu-arch=sm_75' '--cuda-gpu-arch=sm_80' '--cuda-gpu-arch=sm_86' '--cuda-gpu-arch=sm_87' '--cuda-gpu-arch=sm_89' '--cuda-gpu-arch=sm_90' '--cuda-gpu-arch=sm_90a' '--cuda-include-ptx=sm_90' '--cuda-gpu-arch=sm_90' -O3 -Xcuda-fatbinary --compress-all -c jaxlib/gpu/make_batch_pointers.cu.cc -o bazel-out/k8-opt/bin/jaxlib/cuda/_objs/cuda_make_batch_pointers/make_batch_pointers.cu.pic.o)
# Configuration: c9e5c7fd2db6d063269531cdee6aab65dbe42ecf52d9cf5a8fa23322c3f8e852
# Execution platform: @@local_execution_config_platform//:platform
clang-19: warning: CUDA version is newer than the latest partially supported version 12.5 [-Wunknown-cuda-version]
In file included from <built-in>:1:
In file included from /usr/lib/clang/19/include/__clang_cuda_runtime_wrapper.h:41:
In file included from /usr/lib/clang/19/include/cuda_wrappers/cmath:27:
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/cmath:47:15: fatal error: 'math.h' file not found
   47 | #include_next <math.h>
      |               ^~~~~~~~
1 error generated when compiling for sm_50.
Target //jaxlib/tools:build_gpu_kernels_wheel failed to build

I don't know where to find full Bazel build logs.

[ybaturina]

Do you have the header file somewhere in this path?
/usr/include/c++/<c++ version>/math.h

[medaminezghal]

Yes it exists

[medaminezghal]

@ybaturina I tried to isolate the command and use it with arguments (-I, -isystem, --gcc-toolchains, --sysroot) to check if it works but the build still fails

[ybaturina]

Would you check the content of crosstool BUILD file please? For example, on my machine it's located here:
/home/ybaturina/.cache/bazel/_bazel_ybaturina/ead9107e8e47a1c42911a02736d63d03/external/local_config_cuda/crosstool/BUILD

Specifically, I'm interested in cc-compiler-local-config. This is my target definition:

cc_toolchain_config(
    name = "cc-compiler-local-config",
    cpu = "local",
    builtin_include_directories = ["/dt9/usr/include/c++/9", "/dt9/usr/include/c++/9/x86_64-pc-linux-gnu", "/dt9/usr/include/c++/9/backward", "/usr/lib/llvm-18/lib/clang/18/include", "/dt9/usr/include/x86_64-linux-gnu", "/dt9/usr/include", "/usr/lib/llvm-18/lib/clang/18/share"],
    extra_no_canonical_prefixes_flags = [],
    host_compiler_path = "clang/bin/crosstool_wrapper_driver_is_not_gcc",
    host_compiler_prefix = "/usr/bin",
    host_compiler_warnings = [],
    host_unfiltered_compile_flags = [],
    linker_bin_path = "",
    builtin_sysroot = "/dt9",
    cuda_path = "external/cuda_nvcc",
    compiler = "clang",
)

The file math.h on my machine is located inside one of builtin_include_directories - in /dt9/usr/include/c++/9.