Latest NCCL 2.24.3 might crash XGBoost.

[trivialfis]

Workaround:

export NCCL_RAS_ENABLE=0

• xgboost/ops/pipeline/test-python-wheel-impl.sh

  Line 48 in 3a2a85d

  export NCCL_RAS_ENABLE=0

• xgboost/ops/docker_run.py

  Line 73 in 461d27c

  docker_run_cli_args.extend(["-e", "NCCL_RAS_ENABLE=0"])

[trivialfis]

This should not affect conda build.

[jakirkham]

How do things look with NCCL 2.25.1-1?

[hcho3]

@jakirkham Just tried 2.25.1-1 (as part of #11202). I get the same error. I had to set the env var NCCL_RAS_ENABLE=0.

[jakirkham]

Thanks Hyunsu! 🙏

This is with conda, pip, or both?

[hcho3]

@jakirkham The issue only arises if NCCL was installed from pip. The issue does not arise if:

1. NCCL is installed from Conda
2. XGBoost was built with CMake flags: -DUSE_DLOPEN_NCCL=OFF (don't use dlopen for NCCL)

So this issue won't arise for the Conda package of XGBoost.

[trivialfis]

Need to remove CI workarounds once the new nccl is released.