Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom CUDA dispatch sample broken #19662

Open
maxbartel opened this issue Jan 10, 2025 · 2 comments
Open

Custom CUDA dispatch sample broken #19662

maxbartel opened this issue Jan 10, 2025 · 2 comments
Labels
bug 🐞 Something isn't working

Comments

@maxbartel
Copy link
Contributor

What happened?

Hi! I have noticed that the CUDA example for custom dispatch is broken.

This is the stack trace:

iree-compile: iree/third_party/llvm-project/llvm/include/llvm/Support/Casting.h:572: decltype(auto) llvm::cast(From &) [To = mlir::iree_compiler::PartitionableLoopsInterface, From = mlir::Operation]: Assertion `isa<To>(Val) && "cast<Ty>() argument of incompatible type!"' failed.
Please report issues to https://github.com/iree-org/iree/issues and include the crash backtrace.
Stack dump:
0.	Program arguments: iree/build/tools/iree-compile --output-format=vm-bytecode --mlir-print-op-on-diagnostic=false --iree-hal-target-backends=cuda --iree-link-bitcode=cuda_ukernel.bc iree/samples/custom_dispatch/cuda/kernels/ukernel_example.mlir -o check_cuda_ukernel_ukernel_example.mlir_module.vmfb --iree-hal-executable-object-search-path=\"iree/build\" --mlir-disable-threading
 #0 0x00007045a746e768 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) iree/third_party/llvm-project/llvm/lib/Support/Unix/Signals.inc:723:13
 #1 0x00007045a746c980 llvm::sys::RunSignalHandlers() iree/third_party/llvm-project/llvm/lib/Support/Signals.cpp:106:18
 #2 0x00007045a746ee08 SignalHandler(int) iree/third_party/llvm-project/llvm/lib/Support/Unix/Signals.inc:413:1
 #3 0x00007045a13b5320 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x45320)
 #4 0x00007045a140eb1c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
 #5 0x00007045a140eb1c __pthread_kill_internal ./nptl/pthread_kill.c:78:10
 #6 0x00007045a140eb1c pthread_kill ./nptl/pthread_kill.c:89:10
 #7 0x00007045a13b526e raise ./signal/../sysdeps/posix/raise.c:27:6
 #8 0x00007045a13988ff abort ./stdlib/abort.c:81:7
 #9 0x00007045a139881b _nl_load_domain ./intl/loadmsgcat.c:1177:9
#10 0x00007045a13ab507 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x3b507)
#11 0x00007045a92f839e mlir::iree_compiler::setRootConfig(mlir::iree_compiler::IREE::GPU::TargetAttr, mlir::FunctionOpInterface, mlir::Operation*) iree/compiler/src/iree/compiler/Codegen/LLVMGPU/KernelConfig.cpp:0:0
#12 0x00007045a92f2157 llvm::LogicalResult::failed() const iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#13 0x00007045a92f2157 llvm::failed(llvm::LogicalResult) iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#14 0x00007045a92f2157 mlir::iree_compiler::initGPULaunchConfig(mlir::FunctionOpInterface) iree/compiler/src/iree/compiler/Codegen/LLVMGPU/KernelConfig.cpp:2555:7
#15 0x00007045a92f169f llvm::LogicalResult::failed() const iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#16 0x00007045a92f169f llvm::failed(llvm::LogicalResult) iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#17 0x00007045a92f169f mlir::iree_compiler::(anonymous namespace)::LLVMGPUSelectLoweringStrategyPass::runOnOperation() iree/compiler/src/iree/compiler/Codegen/LLVMGPU/LLVMGPUSelectLoweringStrategy.cpp:74:9
#18 0x00007045a76bd6c2 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1::operator()() const iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:0:17
#19 0x00007045a76bd6c2 void llvm::function_ref<void ()>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1>(long) iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:46:12
#20 0x00007045a76bd6c2 llvm::function_ref<void ()>::operator()() const iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:69:12
#21 0x00007045a76bd6c2 void mlir::MLIRContext::executeAction<mlir::PassExecutionAction, mlir::Pass&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, mlir::Pass&) iree/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:280:7
#22 0x00007045a76bd6c2 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:520:21
#23 0x00007045a76be217 llvm::LogicalResult::failed() const iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#24 0x00007045a76be217 llvm::failed(llvm::LogicalResult) iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#25 0x00007045a76be217 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:592:9
#26 0x00007045a76bfad1 llvm::LogicalResult::failed() const iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#27 0x00007045a76bfad1 llvm::failed(llvm::LogicalResult) iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#28 0x00007045a76bfad1 mlir::detail::OpToOpPassAdaptor::runOnOperationImpl(bool) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:733:13
#29 0x00007045a76bdb39 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1::operator()() const iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:527:22
#30 0x00007045a76bdb39 void llvm::function_ref<void ()>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1>(long) iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:46:12
#31 0x00007045a76bdb39 llvm::function_ref<void ()>::operator()() const iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:69:12
#32 0x00007045a76bdb39 void mlir::MLIRContext::executeAction<mlir::PassExecutionAction, mlir::Pass&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, mlir::Pass&) iree/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:280:7
#33 0x00007045a76bdb39 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:520:21
#34 0x00007045a76be217 llvm::LogicalResult::failed() const iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#35 0x00007045a76be217 llvm::failed(llvm::LogicalResult) iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#36 0x00007045a76be217 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:592:9
#37 0x00007045a76c261a llvm::LogicalResult llvm::function_ref<llvm::LogicalResult (mlir::OpPassManager&, mlir::Operation*)>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_0>(long, mlir::OpPassManager&, mlir::Operation*) iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:46:5
#38 0x00007045a932c399 llvm::LogicalResult::failed() const iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#39 0x00007045a932c399 llvm::failed(llvm::LogicalResult) iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#40 0x00007045a932c399 mlir::iree_compiler::IREE::HAL::(anonymous namespace)::ConfigureTargetExecutableVariantsPass::runOnOperation() iree/compiler/src/iree/compiler/Dialect/HAL/Transforms/ConfigureExecutables.cpp:71:9
#41 0x00007045a76bd6c2 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1::operator()() const iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:0:17
#42 0x00007045a76bd6c2 void llvm::function_ref<void ()>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1>(long) iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:46:12
#43 0x00007045a76bd6c2 llvm::function_ref<void ()>::operator()() const iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:69:12
#44 0x00007045a76bd6c2 void mlir::MLIRContext::executeAction<mlir::PassExecutionAction, mlir::Pass&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, mlir::Pass&) iree/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:280:7
#45 0x00007045a76bd6c2 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:520:21
#46 0x00007045a76be217 llvm::LogicalResult::failed() const iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#47 0x00007045a76be217 llvm::failed(llvm::LogicalResult) iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#48 0x00007045a76be217 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:592:9
#49 0x00007045a76bfad1 llvm::LogicalResult::failed() const iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#50 0x00007045a76bfad1 llvm::failed(llvm::LogicalResult) iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#51 0x00007045a76bfad1 mlir::detail::OpToOpPassAdaptor::runOnOperationImpl(bool) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:733:13
#52 0x00007045a76bdb39 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1::operator()() const iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:527:22
#53 0x00007045a76bdb39 void llvm::function_ref<void ()>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1>(long) iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:46:12
#54 0x00007045a76bdb39 llvm::function_ref<void ()>::operator()() const iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:69:12
#55 0x00007045a76bdb39 void mlir::MLIRContext::executeAction<mlir::PassExecutionAction, mlir::Pass&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, mlir::Pass&) iree/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:280:7
#56 0x00007045a76bdb39 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:520:21
#57 0x00007045a76be217 llvm::LogicalResult::failed() const iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#58 0x00007045a76be217 llvm::failed(llvm::LogicalResult) iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#59 0x00007045a76be217 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:592:9
#60 0x00007045a76c261a llvm::LogicalResult llvm::function_ref<llvm::LogicalResult (mlir::OpPassManager&, mlir::Operation*)>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_0>(long, mlir::OpPassManager&, mlir::Operation*) iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:46:5
#61 0x00007045a932d03a llvm::LogicalResult::failed() const iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#62 0x00007045a932d03a llvm::failed(llvm::LogicalResult) iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#63 0x00007045a932d03a mlir::iree_compiler::IREE::HAL::(anonymous namespace)::ConfigureExecutablesPass::runOnOperation() iree/compiler/src/iree/compiler/Dialect/HAL/Transforms/ConfigureExecutables.cpp:110:9
#64 0x00007045a76bd6c2 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1::operator()() const iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:0:17
#65 0x00007045a76bd6c2 void llvm::function_ref<void ()>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1>(long) iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:46:12
#66 0x00007045a76bd6c2 llvm::function_ref<void ()>::operator()() const iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:69:12
#67 0x00007045a76bd6c2 void mlir::MLIRContext::executeAction<mlir::PassExecutionAction, mlir::Pass&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, mlir::Pass&) iree/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:280:7
#68 0x00007045a76bd6c2 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:520:21
#69 0x00007045a76be217 llvm::LogicalResult::failed() const iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#70 0x00007045a76be217 llvm::failed(llvm::LogicalResult) iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#71 0x00007045a76be217 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:592:9
#72 0x00007045a76bfad1 llvm::LogicalResult::failed() const iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#73 0x00007045a76bfad1 llvm::failed(llvm::LogicalResult) iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#74 0x00007045a76bfad1 mlir::detail::OpToOpPassAdaptor::runOnOperationImpl(bool) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:733:13
#75 0x00007045a76bdb39 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1::operator()() const iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:527:22
#76 0x00007045a76bdb39 void llvm::function_ref<void ()>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1>(long) iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:46:12
#77 0x00007045a76bdb39 llvm::function_ref<void ()>::operator()() const iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:69:12
#78 0x00007045a76bdb39 void mlir::MLIRContext::executeAction<mlir::PassExecutionAction, mlir::Pass&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, mlir::Pass&) iree/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:280:7
#79 0x00007045a76bdb39 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:520:21
#80 0x00007045a76be217 llvm::LogicalResult::failed() const iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#81 0x00007045a76be217 llvm::failed(llvm::LogicalResult) iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#82 0x00007045a76be217 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:592:9
#83 0x00007045a76c0769 mlir::PassManager::run(mlir::Operation*) iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:885:0
#84 0x00007045a73bf453 llvm::LogicalResult::failed() const iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#85 0x00007045a73bf453 llvm::failed(llvm::LogicalResult) iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#86 0x00007045a73bf453 mlir::iree_compiler::embed::(anonymous namespace)::Invocation::runPipeline(iree_compiler_pipeline_t) iree/compiler/src/iree/compiler/API/Internal/CompilerDriver.cpp:1008:7
#87 0x00007045a73bf453 ireeCompilerInvocationPipeline iree/compiler/src/iree/compiler/API/Internal/CompilerDriver.cpp:1447:23
#88 0x00007045a75f89fd mlir::iree_compiler::runIreecMain(int, char**)::$_2::operator()(iree_compiler_source_t*) const iree/compiler/src/iree/compiler/Tools/iree_compile_lib.cc:254:11
#89 0x00007045a75f825a mlir::iree_compiler::runIreecMain(int, char**) iree/compiler/src/iree/compiler/Tools/iree_compile_lib.cc:0:10
#90 0x00007045a139a1ca __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:74:3
#91 0x00007045a139a28b call_init ./csu/../csu/libc-start.c:128:20
#92 0x00007045a139a28b __libc_start_main ./csu/../csu/libc-start.c:347:5
#93 0x00006344d0ac76b5 _start (iree/build/tools/iree-compile+0x16b5)

The failing line is not in there, so jumping in with the debugger, this line failes:

auto interfaceOp = cast<PartitionableLoopsInterface>(*op);

The op it tries to cast is the ukernel op

%8 = iree_codegen.ukernel.generic "simple_mul_workgroup" ins(%5, %6 : tensor<?xf32>, tensor<?xf32>) outs(%7 : tensor<?xf32>) (%4 : index) strided_outer_dims(0) -> tensor<?xf32>

Seems like some special casing for that op is missing in

static LogicalResult setRootConfig(IREE::GPU::TargetAttr target,

It hink this PR broke it, because it removed the lowering config #19504 @bjacob I know there is this new approach to ukernels on GPU. Is the plan to fully remove the old one?

Also @benvanik, because you seem to have build the example.

Steps to reproduce your issue

Compile IREE with CUDA and samples on.

What component(s) does this issue relate to?

Compiler

Version information

No response

Additional context

No response

@maxbartel maxbartel added the bug 🐞 Something isn't working label Jan 10, 2025
@bjacob
Copy link
Contributor

bjacob commented Jan 10, 2025

Oh, we don't care much about that sample anymore. It was mostly just a stepping stone towards ukernels. We could just remove it, or we could not build it on CUDA.

@ScottTodd
Copy link
Member

Ah, I can't reproduce due to #17933:

# TODO(#17933): fix cuda_ukernel_unkernel.cu.bc compilation on MSVC (use compiler-rt?)
if(MSVC)
message(STATUS "IREE custom_dispatch/cuda/kernels example ignored -- #17933 required to make MSVC work")
return()
endif()

Would be nice to have CI coverage for CUDA on Linux again: #18814.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug 🐞 Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants