|
| 1 | +# Memory monitoring and control via kvcached CLI |
| 2 | + |
| 3 | +kvcached includes a built-in CLI tool that allows you to monitor GPU memory usage and manage memory limits across different applications. A command `kvctl` is installed along with kvcached package: |
| 4 | + |
| 5 | +```bash |
| 6 | +kvctl |
| 7 | + |
| 8 | +# if kvcached is installed in source |
| 9 | +cd <kvcached-source-dir>/kvcached/cli |
| 10 | +python kvctl.py |
| 11 | +``` |
| 12 | + |
| 13 | +Once inside the CLI, type `help` to view all supported commands: |
| 14 | + |
| 15 | +``` |
| 16 | +kvcached> help |
| 17 | +Available commands: |
| 18 | + list [ipc ...] List IPC segments and usage |
| 19 | + limit <ipc> <size> Set absolute limit (e.g. 512M, 2G) |
| 20 | + limit-percent <ipc> <pct> Set limit as percentage of total GPU RAM |
| 21 | + watch [-n sec] [ipc ...] Continuously display usage table |
| 22 | + kvtop [ipc ...] [--refresh r] Launch curses kvtop UI (q to quit) |
| 23 | + !<shell cmd> Run command in system shell |
| 24 | + help Show this help message |
| 25 | + delete <ipc> Delete IPC segment and its limit entry |
| 26 | + exit | quit Exit the shell |
| 27 | +
|
| 28 | +kvcached> |
| 29 | +``` |
| 30 | + |
| 31 | +Use the `kvtop` command for real-time visualization of memory usage: |
| 32 | + |
| 33 | +<!-- KVCache memory monitor (muted colours) --> |
| 34 | +<pre> |
| 35 | +<span style="color:#009ACD; font-weight:bold;">KVCache Memory Usage</span> |
| 36 | + |
| 37 | +<span style="color:#009ACD;">IPC: SGLANG</span> |
| 38 | +<span style="color:#009ACD;">[</span><span style="color:#B7A800;">==</span><span style="color:#009E8F;">##################</span><span style="color:#666666;">----------------------------------------</span><span style="color:#009ACD;">]</span> |
| 39 | +Prealloc: 792.0 MB | Used: 11.2 GB / 39.9 GB (30.1%) | Free: 27.9 GB |
| 40 | + |
| 41 | +<span style="color:#009ACD;">IPC: VLLM</span> |
| 42 | +<span style="color:#009ACD;">[</span><span style="color:#B7A800;">==</span><span style="color:#009E8F;">#######</span><span style="color:#666666;">--------------------------------------------------- </span><span style="color:#009ACD;">]</span> |
| 43 | +Prealloc: 768.0 MB | Used: 3.6 GB / 37.4 GB (11.7%) | Free: 33.0 GB |
| 44 | + |
| 45 | +<span style="color:#009ACD;">GPU Memory Usage</span> |
| 46 | +<span style="color:#009ACD;">[</span><span style="color:#B7A800;">########################################</span><span style="color:#666666;">--------------------</span><span style="color:#009ACD;">]</span> |
| 47 | +Used: 52.9 GB / 79.2 GB (66.8%) | Free: 26.3 GB |
| 48 | + |
| 49 | +<span style="color:#555555;">Press 'q' to quit</span> |
| 50 | +</pre> |
0 commit comments