Skip to content

Fall back to folly XLOG in CTRAN_LOG on AMD (#3558) - #3558

Open
sunguobao wants to merge 1 commit into
meta-pytorch:mainfrom
sunguobao:export-D115468117
Open

Fall back to folly XLOG in CTRAN_LOG on AMD (#3558)#3558
sunguobao wants to merge 1 commit into
meta-pytorch:mainfrom
sunguobao:export-D115468117

Conversation

@sunguobao

@sunguobao sunguobao commented Aug 10, 2026

Copy link
Copy Markdown

Summary:

The standalone RCCL/conda build (comms/rcclx/develop/projects/rccl) globs all
of comms/ctran into librccl and deliberately has no spdlog -- the same
premise as D115067885. The recent defolly work re-broke that build:

  • D115082769 added comms/ctran/utils/CtranLogger.h, which includes
    comms/utils/logger/SpdlogLogger.h unconditionally.
  • Subsequent diffs swapped folly/logging/xlog.h for that header across ctran,
    including in comms/ctran/utils/AsyncError.h -- a header reached from
    src/include/comm.h via comms/ctran/CtranComm.h.

SpdlogLogger.h hard-requires SPDLOG_FMT_EXTERNAL / SPDLOG_ACTIVE_LEVEL and
the spdlog headers, so every affected TU fails:

SpdlogLogger.h:15: error: "SpdlogLogger requires SPDLOG_FMT_EXTERNAL from the build target"
SpdlogLogger.h:19: error: "SpdlogLogger requires SPDLOG_ACTIVE_LEVEL from the build target"
SpdlogLogger.h:22: fatal error: 'spdlog/sinks/dist_sink.h' file not found

Because AsyncError.h is reachable from comm.h, this is not confined to ctran
sources -- it hits RCCL core (src/transport/shm.cc and effectively every TU
that includes comm.h). Trunk is currently red for this build.

Guard, do not exclude. CtranAlgo.cc, AsyncError.h and the bootstrap sources
are core ctran; dropping them from the CMake source list leaves undefined
symbols at link. Guard instead at the CtranLogger.h facade, on the condition
D115067885 established -- defined(__HIP_PLATFORM_AMD__). Guarding the facade
rather than each consumer is what makes this hold up: the AsyncError.h include
landed after the first call sites and would otherwise have needed its own guard,
and CTRAN_LOG_FIRST_N_* arrived after that.

Fall back to folly XLOG rather than compiling the macros out. That is what these
call sites used before the spdlog migration, and it keeps ROCm logging alive
instead of silently dropping ~119 call sites. LogInit.cc already registers the
folly categories unconditionally -- kCtranCategory, kCtranHeaderCategory,
and a ROCm-only getHipCtranCategory() under #if defined(USE_ROCM) -- each
with logPrefix = "CTRAN", logFilePath from NCCL_DEBUG_FILE, logLevel
from NCCL_DEBUG, and the per-thread device-id context. So level gating, the
CTRAN prefix, debug-file output and thread context are preserved on ROCm.

Scope note, worth a reviewer's attention. __HIP_PLATFORM_AMD__ is true for
every AMD build, not just the spdlog-less CMake one. Buck's AMD config does
have spdlog, so this changes its behavior too: CTRAN_LOG(ERR, ...) there no
longer feeds getLastCommsError() (the folly path has no equivalent hook), and
CTRAN_LOG_FIRST_N gating moves from spdlog levels to folly. LogUT.cc is the
evidence -- its two spdlog-semantics cases,
TestCtranLoggerPreservesLastError and
TestCtranLogFirstNPreservesEnabledBudget, assert exactly those behaviors and
cannot hold on the folly path, so they are guarded off on AMD. That is a real
loss of AMD coverage with nothing replacing it. Keying the facade on
SPDLOG_FMT_EXTERNAL / SPDLOG_ACTIVE_LEVEL instead would confine the change
to the CMake build and leave both tests intact; the platform macro was chosen
for consistency with D115067885. Happy to switch if reviewers prefer the
narrower condition.

Also adds //folly/logging:logging to the exported_deps of the target owning
CtranLogger.h. The include sits behind __HIP_PLATFORM_AMD__, but autodeps is
not preprocessor-aware and the Buck ROCm build needs the dep to compile.

No behavior change on NVIDIA: the spdlog branch, including
CTRAN_LOG_FIRST_N_IMPL and its RateLimit.h dependency, is untouched.

Thanks to the reviewer suggestion in P2456385988, which replaced an earlier
version of this diff that made CTRAN_LOG a no-op on AMD.

Reviewed By: srinathb-meta

Differential Revision: D115468117

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 10, 2026
@meta-codesync

meta-codesync Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

@sunguobao has exported this pull request. If you are a Meta employee, you can view the originating Diff in D115468117.

Summary:

The standalone RCCL/conda build (`comms/rcclx/develop/projects/rccl`) globs all
of `comms/ctran` into `librccl` and deliberately has no spdlog -- the same
premise as D115067885. The recent defolly work re-broke that build:

- D115082769 added `comms/ctran/utils/CtranLogger.h`, which includes
  `comms/utils/logger/SpdlogLogger.h` unconditionally.
- Subsequent diffs swapped `folly/logging/xlog.h` for that header across ctran,
  including in `comms/ctran/utils/AsyncError.h` -- a header reached from
  `src/include/comm.h` via `comms/ctran/CtranComm.h`.

`SpdlogLogger.h` hard-requires `SPDLOG_FMT_EXTERNAL` / `SPDLOG_ACTIVE_LEVEL` and
the spdlog headers, so every affected TU fails:

```
SpdlogLogger.h:15: error: "SpdlogLogger requires SPDLOG_FMT_EXTERNAL from the build target"
SpdlogLogger.h:19: error: "SpdlogLogger requires SPDLOG_ACTIVE_LEVEL from the build target"
SpdlogLogger.h:22: fatal error: 'spdlog/sinks/dist_sink.h' file not found
```

Because `AsyncError.h` is reachable from `comm.h`, this is not confined to ctran
sources -- it hits RCCL core (`src/transport/shm.cc` and effectively every TU
that includes `comm.h`). Trunk is currently red for this build.

Guard, do not exclude. `CtranAlgo.cc`, `AsyncError.h` and the bootstrap sources
are core ctran; dropping them from the CMake source list leaves undefined
symbols at link. Guard instead at the `CtranLogger.h` facade, on the condition
D115067885 established -- `defined(__HIP_PLATFORM_AMD__)`. Guarding the facade
rather than each consumer is what makes this hold up: the `AsyncError.h` include
landed after the first call sites and would otherwise have needed its own guard,
and `CTRAN_LOG_FIRST_N_*` arrived after that.

Fall back to folly XLOG rather than compiling the macros out. That is what these
call sites used before the spdlog migration, and it keeps ROCm logging alive
instead of silently dropping ~119 call sites. `LogInit.cc` already registers the
folly categories unconditionally -- `kCtranCategory`, `kCtranHeaderCategory`,
and a ROCm-only `getHipCtranCategory()` under `#if defined(USE_ROCM)` -- each
with `logPrefix = "CTRAN"`, `logFilePath` from `NCCL_DEBUG_FILE`, `logLevel`
from `NCCL_DEBUG`, and the per-thread device-id context. So level gating, the
CTRAN prefix, debug-file output and thread context are preserved on ROCm.

Scope note, worth a reviewer's attention. `__HIP_PLATFORM_AMD__` is true for
*every* AMD build, not just the spdlog-less CMake one. Buck's AMD config does
have spdlog, so this changes its behavior too: `CTRAN_LOG(ERR, ...)` there no
longer feeds `getLastCommsError()` (the folly path has no equivalent hook), and
`CTRAN_LOG_FIRST_N` gating moves from spdlog levels to folly. `LogUT.cc` is the
evidence -- its two spdlog-semantics cases,
`TestCtranLoggerPreservesLastError` and
`TestCtranLogFirstNPreservesEnabledBudget`, assert exactly those behaviors and
cannot hold on the folly path, so they are guarded off on AMD. That is a real
loss of AMD coverage with nothing replacing it. Keying the facade on
`SPDLOG_FMT_EXTERNAL` / `SPDLOG_ACTIVE_LEVEL` instead would confine the change
to the CMake build and leave both tests intact; the platform macro was chosen
for consistency with D115067885. Happy to switch if reviewers prefer the
narrower condition.

Also adds `//folly/logging:logging` to the `exported_deps` of the target owning
`CtranLogger.h`. The include sits behind `__HIP_PLATFORM_AMD__`, but autodeps is
not preprocessor-aware and the Buck ROCm build needs the dep to compile.

No behavior change on NVIDIA: the spdlog branch, including
`CTRAN_LOG_FIRST_N_IMPL` and its `RateLimit.h` dependency, is untouched.

Thanks to the reviewer suggestion in P2456385988, which replaced an earlier
version of this diff that made `CTRAN_LOG` a no-op on AMD.

Reviewed By: srinathb-meta

Differential Revision: D115468117
@meta-codesync meta-codesync Bot changed the title Fall back to folly XLOG in CTRAN_LOG on AMD Fall back to folly XLOG in CTRAN_LOG on AMD (#3558) Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot. meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant