candle-vllm --p 8000 --d 0,1,2,3 --w /data/Qwen3.5-35B-A3B-GPTQ-Int4 --ui-server --prefix-cache
2026-05-06T03:28:17.505045Z INFO candle_vllm::openai::pipelines::pipeline: Loading Qwen3_5MoeForConditionalGeneration model.
2026-05-06T03:28:17.505268Z INFO candle_vllm::openai::communicator: build command channel for the main process!
2026-05-06T03:28:17.882063Z WARN candle_vllm::openai::pipelines::pipeline: create nccl comm channel rank 0, shards 4, id Id { id: ncclUniqueId { internal: [-77, -17, -8, -32, -46, 85, -4, 44, 2, 0, -40, 73, -84, 16, 120, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] } }
2026-05-06T03:28:18.164611Z WARN candle_vllm::openai::pipelines::pipeline: nccl comm created for rank 0
2026-05-06T03:28:18.382914Z INFO candle_vllm::openai::communicator: accept one daemon process in command channel!
2026-05-06T03:28:18.383724Z INFO candle_vllm::openai::communicator: accept one daemon process in command channel!
2026-05-06T03:28:18.499552Z INFO candle_vllm::openai::communicator: accept one daemon process in command channel!
2026-05-06T03:28:18.500943Z INFO candle_vllm::openai::communicator: accept one daemon process in command channel!
2026-05-06T03:28:18.503909Z INFO candle_vllm::openai::communicator: accept one daemon process in command channel!
2026-05-06T03:28:18.503952Z INFO candle_vllm::openai::communicator: one daemon process connected to the command channel!
2026-05-06T03:28:18.503970Z INFO candle_vllm::openai::communicator: one daemon process connected to the command channel!
2026-05-06T03:28:18.503986Z INFO candle_vllm::openai::communicator: one daemon process connected to the command channel!
2026-05-06T03:28:18.503996Z WARN candle_vllm::openai::communicator: command channel is built!
2026-05-06T03:28:18.504020Z INFO candle_vllm::backend::heartbeat: enter heartbeat processing loop (Ok(DaemonManager { daemon_streams: Some([Stream(Stream(UnixStream { fd: FileDesc(OwnedFd { fd: 42 }), local: "command_heartbeat_candle_vllm.sock" (abstract), peer: (unnamed) })), Stream(Stream(UnixStream { fd: FileDesc(OwnedFd { fd: 103 }), local: "command_heartbeat_candle_vllm.sock" (abstract), peer: (unnamed) })), Stream(Stream(UnixStream { fd: FileDesc(OwnedFd { fd: 105 }), local: "command_heartbeat_candle_vllm.sock" (abstract), peer: (unnamed) }))]), main_stream: None, mpi_rank: None, mpi_size: None }))
2026-05-06T03:28:18.505255Z INFO candle_vllm::openai::communicator: accept one daemon process in command channel!
2026-05-06T03:28:18.505292Z INFO candle_vllm::openai::communicator: one daemon process connected to the command channel!
2026-05-06T03:28:18.505310Z INFO candle_vllm::openai::communicator: one daemon process connected to the command channel!
2026-05-06T03:28:18.505323Z INFO candle_vllm::openai::communicator: one daemon process connected to the command channel!
2026-05-06T03:28:18.505332Z WARN candle_vllm::openai::communicator: command channel is built!
Loading model in 4 ranks!
2026-05-06T03:28:19.507429Z WARN candle_vllm::backend::progress: all ranks finished model loading!
2026-05-06T03:28:19.507894Z INFO candle_vllm: parallel model: multiprocess!
thread 'main' (1378) panicked at src/main.rs:394:19:
Failed to load Qwen3-VL model for arch Qwen3_5MoeForConditionalGeneration on rank 0: DriverError(CUDA_ERROR_INVALID_PTX, "a PTX JIT compilation failed") when loading cast_bf16_f16
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
candle-vllm --p 8000 --d 0,1,2,3 --w /data/Qwen3.5-35B-A3B-GPTQ-Int4 --ui-server --prefix-cache
2026-05-06T03:28:17.505045Z INFO candle_vllm::openai::pipelines::pipeline: Loading Qwen3_5MoeForConditionalGeneration model.
2026-05-06T03:28:17.505268Z INFO candle_vllm::openai::communicator: build command channel for the main process!
2026-05-06T03:28:17.882063Z WARN candle_vllm::openai::pipelines::pipeline: create nccl comm channel rank 0, shards 4, id Id { id: ncclUniqueId { internal: [-77, -17, -8, -32, -46, 85, -4, 44, 2, 0, -40, 73, -84, 16, 120, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] } }
2026-05-06T03:28:18.164611Z WARN candle_vllm::openai::pipelines::pipeline: nccl comm created for rank 0
2026-05-06T03:28:18.382914Z INFO candle_vllm::openai::communicator: accept one daemon process in command channel!
2026-05-06T03:28:18.383724Z INFO candle_vllm::openai::communicator: accept one daemon process in command channel!
2026-05-06T03:28:18.499552Z INFO candle_vllm::openai::communicator: accept one daemon process in command channel!
2026-05-06T03:28:18.500943Z INFO candle_vllm::openai::communicator: accept one daemon process in command channel!
2026-05-06T03:28:18.503909Z INFO candle_vllm::openai::communicator: accept one daemon process in command channel!
2026-05-06T03:28:18.503952Z INFO candle_vllm::openai::communicator: one daemon process connected to the command channel!
2026-05-06T03:28:18.503970Z INFO candle_vllm::openai::communicator: one daemon process connected to the command channel!
2026-05-06T03:28:18.503986Z INFO candle_vllm::openai::communicator: one daemon process connected to the command channel!
2026-05-06T03:28:18.503996Z WARN candle_vllm::openai::communicator: command channel is built!
2026-05-06T03:28:18.504020Z INFO candle_vllm::backend::heartbeat: enter heartbeat processing loop (Ok(DaemonManager { daemon_streams: Some([Stream(Stream(UnixStream { fd: FileDesc(OwnedFd { fd: 42 }), local: "command_heartbeat_candle_vllm.sock" (abstract), peer: (unnamed) })), Stream(Stream(UnixStream { fd: FileDesc(OwnedFd { fd: 103 }), local: "command_heartbeat_candle_vllm.sock" (abstract), peer: (unnamed) })), Stream(Stream(UnixStream { fd: FileDesc(OwnedFd { fd: 105 }), local: "command_heartbeat_candle_vllm.sock" (abstract), peer: (unnamed) }))]), main_stream: None, mpi_rank: None, mpi_size: None }))
2026-05-06T03:28:18.505255Z INFO candle_vllm::openai::communicator: accept one daemon process in command channel!
2026-05-06T03:28:18.505292Z INFO candle_vllm::openai::communicator: one daemon process connected to the command channel!
2026-05-06T03:28:18.505310Z INFO candle_vllm::openai::communicator: one daemon process connected to the command channel!
2026-05-06T03:28:18.505323Z INFO candle_vllm::openai::communicator: one daemon process connected to the command channel!
2026-05-06T03:28:18.505332Z WARN candle_vllm::openai::communicator: command channel is built!
Loading model in 4 ranks!
2026-05-06T03:28:19.507429Z WARN candle_vllm::backend::progress: all ranks finished model loading!
2026-05-06T03:28:19.507894Z INFO candle_vllm: parallel model: multiprocess!
thread 'main' (1378) panicked at src/main.rs:394:19:
Failed to load Qwen3-VL model for arch Qwen3_5MoeForConditionalGeneration on rank 0: DriverError(CUDA_ERROR_INVALID_PTX, "a PTX JIT compilation failed") when loading cast_bf16_f16
note: run with
RUST_BACKTRACE=1environment variable to display a backtrace