Performance testing

[reluctantfuturist]

🚀 Describe the new functionality needed

Performance testing and meeting some self-defined SLAs around Latency/Throughput/Concurrency for a defined baseline setup

💡 Why is this needed? What if we don't build it?

Needed for enterprise deployment reliability

Other thoughts

No response

[ehhuang]

some early results on chat completions API

[mattf]

@ehhuang what's that evaluating?

[ehhuang]

@ehhuang what's that evaluating?

chat completions API. will share more.

[ehhuang]

Merged a telemetry fix: #3070

Put up #3096 for benchmarking setup.

Identified scaling issues around inference store, and looking

[ehhuang]

Benchmark script added in #3160

Load Impact Comparison:

Metric	10 Users	50 Users
Llama Stack Throughput	10.06	31.56 req/s
vLLM Throughput	10.97	54.52 req/s
------------------------	---------------------	--------------------------
Performance Gap	9%	73%

Low concurrency is now similar to vllm but still need better scaling.

[ehhuang]

Optimize inference store writes: #3383
Telemetry background logger: #3393

[ehhuang]

GuideLLM benchmarks #3421

[ehhuang]

#3462 to enable multi process with uvicorn