[Bug]: FP8 accuracy issue

[yuzho-amd]

Your current environment

vllm branch: llama_fp8_12062024, commit: d7fefdf
you can use this docker:
yuzho/private:ali1209_llama_fp8_12062024_d7fefdf03

🐛 Describe the bug

the output token id of fp8 models are not same when using different TP:

pastedImage_2_28_2025__9_52_30_796.jpeg

and this is the script to test models quantized by quark:

import os
from vllm import LLM, SamplingParams

if name == 'main':
llm = LLM("/mnt/raid0/pretrained_model/amd/Llama-2-7b-hf-FP8-KV", tensor_parallel_size=4, quantization="fp8", kv_cache_dtype="fp8")
sampling_params = SamplingParams(temperature=0, top_k=1)
print(llm.generate("What is batch inference?", sampling_params))

this is the script to test models quantized by vllm:

import os
from vllm import LLM, SamplingParams

if name == 'main':
llm = LLM("/mnt/raid0/pretrained_model/meta-llama/Llama-2-7b-hf", tensor_parallel_size=4, quantization="fp8", kv_cache_dtype="fp8")
sampling_params = SamplingParams(temperature=0, top_k=1)
print(llm.generate("What is batch inference?", sampling_params))

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.