Skip to content

get_or_load_func drops the kernel name from CUDA symbol-lookup errors #3884

Description

@astorise

Summary

When a CUDA kernel symbol is missing from a loaded module, the resulting error
does not say which kernel. CudaDevice::get_or_load_func takes the name as an
argument and discards it on the failure path:

// candle-core/src/cuda_backend/device.rs
pub fn get_or_load_func(&self, fn_name: &str, mdl: &kernels::Module) -> Result<CudaFunc> {
    let ms = self.modules.read().unwrap();
    if let Some(mdl) = ms.mdls[mdl.index()].as_ref() {
        let func = mdl.load_function(fn_name).w()?;   // <- fn_name dropped
        ...
    }
    drop(ms);
    let mut ms = self.modules.write().unwrap();
    let cuda_module = self.context.load_module(mdl.ptx().into()).w()?;
    ms.mdls[mdl.index()] = Some(cuda_module.clone());
    let func = cuda_module.load_function(fn_name).w()?; // <- and here
    ...
}

.w() wraps cudarc's DriverError unchanged, so the caller gets:

DriverError(CUDA_ERROR_NOT_FOUND, "named symbol not found")

The name that would make this actionable was in scope at the point of failure
and was thrown away.

Why this is worse than it looks

Two things conspire to make it hard to recover the information afterwards.

A backtrace does not identify the kernel. The error propagates upward as a
value through ?; the eventual panic happens at whatever unwrap/expect
consumes it, often several layers up in user code. RUST_BACKTRACE=1 names
that consumer, not the get_or_load_func call site. The natural first
debugging step yields nothing.

A missing symbol is a legitimate, reachable state, not a corrupt build.
candle-kernels guards kernels by target architecture, and the guards are not
uniform across a dtype. For F8E4M3 specifically, the casts sit under one
threshold and the elementwise and indexing ops under a higher one:

kernels guard
cast_f8_e4m3_f32, cast_f8_e4m3_bf16, … (cast.cu) __CUDA_ARCH__ >= 800
affine_f8_e4m3 (affine.cu) __CUDA_ARCH__ >= 890
is_*_f8_e4m3, gather_*_f8_e4m3, scatter_add_f8 (indexing.cu) __CUDA_ARCH__ >= 890

So a build for, say, sm_86 has a genuinely partial F8E4M3 kernel set: some
operations on such a tensor work and others fail at lookup time. That is a
reasonable design — but it means "symbol not found" is a routine outcome of
running a supported dtype on a supported device, and the one thing the user
needs in order to understand it is the name that is currently discarded.

Concretely: we hit CUDA_ERROR_NOT_FOUND while building a model on an
sm_86 device and, after reading the guards above and tracing the load path by
hand, still cannot say which kernel is missing. That is the entire content of
this report — not that the lookup failed, but that a failed lookup is
undiagnosable from its own error.

Why not fix this in cudarc

DriverError is a newtype over the raw driver result:

pub struct DriverError(pub sys::CUresult);

It has a public field, and its std::error::Error impl is behind
#[cfg(feature = "std")]. Carrying a symbol name would require an owned
String — allocation in a no_std-friendly FFI binding, a change to a
pattern-matchable public shape, and a breaking change across every API that
returns it. cudarc already surfaces the driver's own description; the caller's
identifier is the caller's to attach.

Shape of the fix

Attach the name at both load_function sites, and the module with it, since
"which module was searched" is the natural follow-up question:

cuda: kernel `affine_f8_e4m3` not found in module `affine`
  (this kernel may not be compiled for the target architecture)

The parenthetical matters as much as the name: for a user on a pre-sm_89
device, "not found" reads as a broken installation, when it is in fact the
architecture guard doing its job.

Acceptance

  • A failed lookup names the kernel and the module it was searched in.
  • The underlying DriverError remains in the source chain, so
    CUDA_ERROR_NOT_FOUND is still distinguishable from other driver failures.
  • Both call sites are covered — the cached-module path and the
    load-then-lookup path fail identically today and must keep doing so.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions