Skip to content

Commit 61f0eab

Browse files
authored
[Example] Add memory control example (#152)
1 parent c9bdca6 commit 61f0eab

2 files changed

Lines changed: 50 additions & 0 deletions

File tree

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Memory monitoring and control via kvcached CLI
2+
3+
kvcached includes a built-in CLI tool that allows you to monitor GPU memory usage and manage memory limits across different applications. A command `kvctl` is installed along with kvcached package:
4+
5+
```bash
6+
kvctl
7+
8+
# if kvcached is installed in source
9+
cd <kvcached-source-dir>/kvcached/cli
10+
python kvctl.py
11+
```
12+
13+
Once inside the CLI, type `help` to view all supported commands:
14+
15+
```
16+
kvcached> help
17+
Available commands:
18+
list [ipc ...] List IPC segments and usage
19+
limit <ipc> <size> Set absolute limit (e.g. 512M, 2G)
20+
limit-percent <ipc> <pct> Set limit as percentage of total GPU RAM
21+
watch [-n sec] [ipc ...] Continuously display usage table
22+
kvtop [ipc ...] [--refresh r] Launch curses kvtop UI (q to quit)
23+
!<shell cmd> Run command in system shell
24+
help Show this help message
25+
delete <ipc> Delete IPC segment and its limit entry
26+
exit | quit Exit the shell
27+
28+
kvcached>
29+
```
30+
31+
Use the `kvtop` command for real-time visualization of memory usage:
32+
33+
<!-- KVCache memory monitor (muted colours) -->
34+
<pre>
35+
<span style="color:#009ACD; font-weight:bold;">KVCache Memory Usage</span>
36+
37+
<span style="color:#009ACD;">IPC: SGLANG</span>
38+
<span style="color:#009ACD;">[</span><span style="color:#B7A800;">==</span><span style="color:#009E8F;">##################</span><span style="color:#666666;">----------------------------------------</span><span style="color:#009ACD;">]</span>
39+
Prealloc: 792.0&nbsp;MB | Used: 11.2&nbsp;GB / 39.9&nbsp;GB (30.1%) | Free: 27.9&nbsp;GB
40+
41+
<span style="color:#009ACD;">IPC: VLLM</span>
42+
<span style="color:#009ACD;">[</span><span style="color:#B7A800;">==</span><span style="color:#009E8F;">#######</span><span style="color:#666666;">--------------------------------------------------- </span><span style="color:#009ACD;">]</span>
43+
Prealloc: 768.0&nbsp;MB | Used: 3.6&nbsp;GB / 37.4&nbsp;GB (11.7%) | Free: 33.0&nbsp;GB
44+
45+
<span style="color:#009ACD;">GPU Memory Usage</span>
46+
<span style="color:#009ACD;">[</span><span style="color:#B7A800;">########################################</span><span style="color:#666666;">--------------------</span><span style="color:#009ACD;">]</span>
47+
Used: 52.9&nbsp;GB / 79.2&nbsp;GB (66.8%) | Free: 26.3&nbsp;GB
48+
49+
<span style="color:#555555;">Press 'q' to quit</span>
50+
</pre>

examples/02_memory_control/__placeholder__

Whitespace-only changes.

0 commit comments

Comments
 (0)