New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Best Performance for a single card for Llama-2-7b-chat-hf #196

Open

AdityaKulshrestha opened this issue Jul 29, 2024 · 0 comments

AdityaKulshrestha commented Jul 29, 2024

Hi team,

We are currently working on getting the best performance on a single card for Gaudi. Following is the configuration we are looking for:

Target - Maximize currency without losing TTFT
Model - meta-llama/Llama-2-7b-chat-hf
Input Tokens - 1024
Output - 256
Acceptable TTFT - 2-2.5 sec

The best concurrency we could achieve was 16 on a single card. Can someone please help if this is the best possibility for this case scenario?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment