Thank you so much for all your hard work, which has delivered tremendous breakthroughs to the community. Without the ATOM project, advances in LLM research would not have reached where we stand today.
I need to run the Kimi K3 model on multiple AMD MI308X servers using either the vLLM or SGLang framework, but I have run into several issues. I would greatly appreciate your guidance and assistance. Thank you very much.
The official guide at https://recipes.vllm.ai/moonshotai/Kimi-K3?hardware=mi300x states that two AMD MI308X servers are required to run inference tests on the Kimi K3 model with MXFP4 precision. However, the issue at ROCm/ATOM#1582 mentions that AMD MI308X does not support MXFP4 precision. This seems contradictory.
Additionally, the SGLang and TokenSpeed frameworks cannot run inference tests for the Kimi K3 model on MI308X servers; they only support MI355X (gfx950) hardware.
Could you clarify which inference framework and quantization precision I should use to perform inference testing on the Kimi K3 model across two AMD MI308X servers?
Thank you so much for all your hard work, which has delivered tremendous breakthroughs to the community. Without the ATOM project, advances in LLM research would not have reached where we stand today.
I need to run the Kimi K3 model on multiple AMD MI308X servers using either the vLLM or SGLang framework, but I have run into several issues. I would greatly appreciate your guidance and assistance. Thank you very much.
The official guide at https://recipes.vllm.ai/moonshotai/Kimi-K3?hardware=mi300x states that two AMD MI308X servers are required to run inference tests on the Kimi K3 model with MXFP4 precision. However, the issue at ROCm/ATOM#1582 mentions that AMD MI308X does not support MXFP4 precision. This seems contradictory.
Additionally, the SGLang and TokenSpeed frameworks cannot run inference tests for the Kimi K3 model on MI308X servers; they only support MI355X (gfx950) hardware.
Could you clarify which inference framework and quantization precision I should use to perform inference testing on the Kimi K3 model across two AMD MI308X servers?