feat: add compare_with_vllm.py

Author: viraatc

examples/03_BenchmarkComparison/README.md

@@ -0,0 +1,70 @@
+# Benchmark Comparison Example
+
+Compare `inference-endpoint` with vLLM's benchmarking tool using identical prompts.
+
+## Prerequisites
+
+1. **Running inference server** (OpenAI-compatible):
+
+   ```bash
+   vllm serve Qwen/Qwen2.5-0.5B-Instruct --port 8000
+   ```
+
+2. **Pre-generated dataset files** (included in this directory):
+   - `dummy_prompts_vllm.jsonl` - vLLM format: `{"prompt": "..."}`
+   - `dummy_prompts_ie.jsonl` - inference-endpoint format: `{"text_input": "..."}`
+
+## Usage
+
+```bash
+cd examples/03_BenchmarkComparison
+python compare_with_vllm.py --model "Qwen/Qwen2.5-0.5B-Instruct" --endpoint http://localhost:8000
+```
+
+### Options
+
+| Option                | Description                      | Default                 |
+| --------------------- | -------------------------------- | ----------------------- |
+| `--model`, `-m`       | Model name (required)            | -                       |
+| `--num-prompts`, `-n` | Number of prompts                | 100                     |
+| `--endpoint`          | Server URL                       | `http://localhost:8000` |
+| `--max-output-tokens` | Max output tokens                | 2000                    |
+| `--timeout`           | Timeout in seconds               | 900                     |
+| `--workers`           | Number of workers                | 1                       |
+| `--dry`               | Print commands without executing | -                       |
+
+### Example
+
+```bash
+python compare_with_vllm.py \
+    --model "Qwen/Qwen2.5-0.5B-Instruct" \
+    --num-prompts 200 \
+    --max-output-tokens 1000
+```
+
+## Output
+
+The script runs both benchmarks and displays a comparison table:
+
+```
+====================================================================================================
+Metric                              | Inference Endpoint        | vLLM Benchmark
+----------------------------------------------------------------------------------------------------
+Test Duration (s)                   | 8.03                      | 8.05
+----------------------------------------------------------------------------------------------------
+Throughput (req/s)                  | 12.45                     | 12.38
+Output Token Throughput (tok/s)     | 1234.56                   | 1230.45
+----------------------------------------------------------------------------------------------------
+Mean TTFT (ms)                      | 45.23                     | 46.12
+Median TTFT (ms)                    | 43.56                     | 44.89
+P99 TTFT (ms)                       | 78.90                     | 80.34
+----------------------------------------------------------------------------------------------------
+Mean TPOT (ms)                      | 12.34                     | 12.56
+Median TPOT (ms)                    | 11.89                     | 12.01
+P99 TPOT (ms)                       | 18.45                     | 18.90
+----------------------------------------------------------------------------------------------------
+Mean ITL (ms)                       | 12.34                     | 12.56
+Median ITL (ms)                     | 11.89                     | 12.01
+P99 ITL (ms)                        | 18.45                     | 18.90
+====================================================================================================
+```