Skip to content
This repository was archived by the owner on May 3, 2023. It is now read-only.

Commit 1b63177

Browse files
pothospothos
committed
contrib: integrate with systemd units
The benchmark script in contrib/ already used ExecStartPre to automatically set up systemd unit tracing. This was quite verbose and and also lacked the "+" prefix to refer to the host filesystem and run as root even if the unit processes are sandboxed. Document how to modify systemd units so that they register with traceloop. For convenience this is based on a traceloop service file and new helper tool traceloopctl in the contrib folder. The traceloopctl tool has a speical systemd mode which follows a naming scheme for a better integration but the traceloopctl can also be used in the generic mode to interact with any trace.
1 parent cd13f83 commit 1b63177

File tree

3 files changed

+232
-0
lines changed

3 files changed

+232
-0
lines changed

‎README.md

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,95 @@ $ sudo curl --unix-socket /run/traceloop.socket 'http://localhost/dump-by-cgroup
6363
6464
```
6565

66+
## With systemd services using traceloopctl and the HTTP interface to integrate with systemd
67+
68+
The `contrib/traceloopctl` helper is a command line tool to manage traceloop logs and has a special mode for systemd units.
69+
This works when systemd unit traces have a name that consits of the systemd unit file name combined with the systemd service invocation ID through a `_` character (`%n_$INVOCATION_ID`).
70+
71+
```
72+
$ contrib/traceloopctl
73+
Usage: contrib/traceloopctl COMMAND|-h|--help
74+
Commands:
75+
list-all
76+
dump-id ID
77+
dump-name NAME
78+
close-id ID
79+
close-name NAME
80+
81+
list-sd-units
82+
list-sd-traces SERVICE
83+
dump-sd SERVICE INVOCATION|-1
84+
close-sd SERVICE INVOCATION|-1|all
85+
86+
The *-sd commands assume the trace name format is systemd_UNIT_INVOCATIONID which can be automated with an ExecStartPre command (see README).
87+
88+
Needs to be run with access to /run/traceloop.socket (e.g., with sudo).
89+
```
90+
91+
First you need to make sure that traceloop runs as a service:
92+
93+
```
94+
sudo cp contrib/traceloop.service /etc/systemd/system/traceloop.service
95+
# You can enable it always or just start it on demand pulled in as dependency: sudo systemctl enable --now traceloop.service
96+
```
97+
98+
Now add `Requires=traceloop.service` and `After=traceloop.service` directives to the `[Unit]` section of your service.
99+
In the `[Service]` section you have to add a special command that registers the unit CGroup with traceloop: add `ExecStartPre=+/PATH/TO/kinvolk/traceloop/contrib/traceloopctl add-current-cgroup-sd "%n" "$INVOCATION_ID"` as very first `ExecStartPre` line.
100+
The `+` prefix means to ignore any `User=` directives but run as root user and also ignore any filesystem changes like `ProtectSystem=`.
101+
This allows us to use the system's exec path and write to `/run/traceloop.socket` regardless of the restrictions applying for the regular `ExecStartPre=`/`ExecStart=` processes.
102+
103+
An example unit `my-service.service` looks like this:
104+
105+
```
106+
[Unit]
107+
Description=My Service
108+
# Add:
109+
Requires=traceloop.service
110+
# Add:
111+
After=traceloop.service
112+
113+
[Service]
114+
User=1000
115+
Group=1000
116+
ProtectSystem=strict
117+
NoNewPrivileges=yes
118+
119+
# Add:
120+
ExecStartPre=+/PATH/TO/kinvolk/traceloop/contrib/traceloopctl add-current-cgroup-sd "%n" "$INVOCATION_ID"
121+
122+
ExecStart=/bin/echo Hello World
123+
```
124+
125+
Instead of modifying the original `my-service.service` unit file you can also do the traceloop registration through a small drop-in unit file in `/etc/systemd/system/my-service.service.d/10-traceloop.conf`:
126+
127+
```
128+
[Unit]
129+
Requires=traceloop.service
130+
After=traceloop.service
131+
132+
[Service]
133+
# The + prefix means to ignore the User= but run as root and ignore filesystem changes like ProtectSystem=, this allows us to use the system's curl and write to /run/
134+
ExecStartPre=+/PATH/TO/kinvolk/traceloop/contrib/traceloopctl add-current-cgroup-sd "%n" "$INVOCATION_ID"
135+
```
136+
137+
Start the service with `sudo systemctl daemon-reload; sudo systemctl restart my-service.service` and observe the traces:
138+
139+
```
140+
# List the traced systemd units:
141+
$ list-sd-units
142+
Traces Units
143+
------- -----
144+
1 my-service.service
145+
# Now list the traces:
146+
$ contrib/traceloopctl list-sd-traces my-service.service
147+
a72e5d0f2b7e405894ca4664ddf205b1
148+
# Dump the trace:
149+
$ contrib/traceloopctl dump-sd my-service.service -1 | less
150+
# Clean up afterwards:
151+
$ contrib/traceloopctl close-sd my-service.service all
152+
closed
153+
```
154+
66155
### Talk at Linux Plumbers Conference 2020
67156

68157
A comprehensive presentation was held at LPC 2020 in the Networking and BPF Summit.

‎contrib/traceloop.service

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
[Unit]
2+
Description=Traceloop
3+
4+
[Service]
5+
Type=notify
6+
NotifyAccess=all
7+
ExecStartPre=/bin/rm -f /run/traceloop.socket
8+
ExecStart=/bin/sh -c "/home/kai/kinvolk/traceloop/traceloop serve & while ! curl -fsS --unix-socket /run/traceloop.socket 'http://localhost/list' > /dev/null; do sleep 1; echo Waiting for traceloop to start up; done ; systemd-notify --ready; wait"
9+
ExecStopPost=/bin/rm -f /run/traceloop.socket
10+
11+
[Install]
12+
WantedBy=multi-user.target

‎contrib/traceloopctl

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
#!/bin/sh
2+
CMD="$1"
3+
ARG1="$2"
4+
ARG2="$3"
5+
set -euo pipefail
6+
7+
if [ $# -lt 1 ] || [ "$CMD" = "-h" ] || [ "$CMD" = "--help" ]; then
8+
echo "Usage: $0 COMMAND|-h|--help"
9+
echo "Needs to be run with access to /run/traceloop.socket (e.g., with sudo)."
10+
echo "Commands:"
11+
echo " list-all"
12+
echo " dump-id ID"
13+
echo " dump-name NAME"
14+
echo " close-id ID"
15+
echo " close-name NAME"
16+
echo " add-current-cgroup NAME"
17+
echo
18+
echo " list-sd-units"
19+
echo " list-sd-traces SERVICE"
20+
echo " dump-sd SERVICE INVOCATION|-1"
21+
echo " close-sd SERVICE INVOCATION|-1|all"
22+
echo " add-current-cgroup-sd SERVICE INVOCATION"
23+
echo
24+
echo "The *-sd commands assume the trace name format is systemd_UNIT_INVOCATIONID which can be automated with:"
25+
echo ' ExecStartPre=+/…/contrib/traceloopctl add-current-cgroup-sd "%n" "$INVOCATION_ID"'
26+
exit 1
27+
fi
28+
29+
SCRIPT_FOLDER="$(dirname "$(readlink -f "$0")")"
30+
CURL="curl -fsS --unix-socket /run/traceloop.socket"
31+
if [ "$CMD" = list-all ]; then
32+
$CURL "http://localhost/list"
33+
elif [ "$CMD" = dump-id ]; then
34+
if [ "$ARG1" = "" ]; then
35+
echo "Expected ID argument" > /dev/stderr
36+
exit 1
37+
fi
38+
ID="$ARG1"
39+
$CURL "http://localhost/dump?id=${ID}"
40+
elif [ "$CMD" = dump-name ]; then
41+
if [ "$ARG1" = "" ]; then
42+
echo "Expected NAME argument" > /dev/stderr
43+
exit 1
44+
fi
45+
NAME="$ARG1"
46+
$CURL "http://localhost/dump-by-name?name=${NAME}"
47+
elif [ "$CMD" = close-id ]; then
48+
if [ "$ARG1" = "" ]; then
49+
echo "Expected ID argument" > /dev/stderr
50+
exit 1
51+
fi
52+
ID="$ARG1"
53+
$CURL "http://localhost/close?id=${ID}"
54+
elif [ "$CMD" = close-name ]; then
55+
if [ "$ARG1" = "" ]; then
56+
echo "Expected NAME argument" > /dev/stderr
57+
exit 1
58+
fi
59+
NAME="$ARG1"
60+
$CURL "http://localhost/close-by-name?name=${NAME}"
61+
elif [ "$CMD" = add-current-cgroup ]; then
62+
if [ "$ARG1" = "" ]; then
63+
echo "Expected NAME argument" > /dev/stderr
64+
exit 1
65+
fi
66+
NAME="$ARG1"
67+
CURRENT_CGROUP=$("${SCRIPT_FOLDER}"/current-cgroup)
68+
$CURL "http://localhost/add?name=${NAME}&cgrouppath=${CURRENT_CGROUP}"
69+
elif [ "$CMD" = list-sd-units ]; then
70+
echo " Traces Units"
71+
echo "------- -----"
72+
$CURL "http://localhost/list" | grep -o '\[systemd_.*\]' | cut -d _ -f 2 | sort | uniq -c
73+
elif [ "$CMD" = list-sd-traces ]; then
74+
if [ "$ARG1" = "" ]; then
75+
echo "Expected SERVICE argument" > /dev/stderr
76+
exit 1
77+
fi
78+
SERVICE="$ARG1"
79+
$CURL "http://localhost/list" | grep -o "\[systemd_${SERVICE}.*\] " | cut -d _ -f 3- | cut -d ] -f 1
80+
elif [ "$CMD" = dump-sd ]; then
81+
if [ "$ARG1" = "" ]; then
82+
echo "Expected SERVICE argument" > /dev/stderr
83+
exit 1
84+
fi
85+
SERVICE="$ARG1"
86+
if [ "$ARG2" = "" ]; then
87+
echo "Expected INVOCATION argument" > /dev/stderr
88+
exit 1
89+
fi
90+
INVOCATION="$ARG2"
91+
if [ "$INVOCATION" = "-1" ]; then
92+
INVOCATION=$($CURL "http://localhost/list" | grep -o "\[systemd_${SERVICE}.*\] " | cut -d _ -f 3- | cut -d ] -f 1 | tail -n 1)
93+
fi
94+
$CURL "http://localhost/dump-by-name?name=systemd_${SERVICE}_${INVOCATION}"
95+
elif [ "$CMD" = close-sd ]; then
96+
if [ "$ARG1" = "" ]; then
97+
echo "Expected SERVICE argument" > /dev/stderr
98+
exit 1
99+
fi
100+
SERVICE="$ARG1"
101+
if [ "$ARG2" = "" ]; then
102+
echo "Expected INVOCATION argument" > /dev/stderr
103+
exit 1
104+
fi
105+
if [ "$ARG2" = "-1" ]; then
106+
INVOCATIONS=$($CURL "http://localhost/list" | grep -o "\[systemd_${SERVICE}.*\] " | cut -d _ -f 3- | cut -d ] -f 1 | tail -n 1)
107+
elif [ "$ARG2" = all ]; then
108+
INVOCATIONS=$($CURL "http://localhost/list" | grep -o "\[systemd_${SERVICE}.*\] " | cut -d _ -f 3- | cut -d ] -f 1)
109+
else
110+
INVOCATIONS="$ARG2"
111+
fi
112+
for ID in $INVOCATIONS; do
113+
$CURL "http://localhost/close-by-name?name=systemd_${SERVICE}_${ID}"
114+
done
115+
elif [ "$CMD" = add-current-cgroup-sd ]; then
116+
if [ "$ARG1" = "" ]; then
117+
echo "Expected SERVICE argument" > /dev/stderr
118+
exit 1
119+
fi
120+
SERVICE="$ARG1"
121+
if [ "$ARG2" = "" ]; then
122+
echo "Expected INVOCATION argument" > /dev/stderr
123+
exit 1
124+
fi
125+
INVOCATION="$ARG2"
126+
CURRENT_CGROUP=$("${SCRIPT_FOLDER}"/current-cgroup)
127+
$CURL "http://localhost/add?name=systemd_${SERVICE}_${INVOCATION}&cgrouppath=${CURRENT_CGROUP}"
128+
else
129+
echo "Unknown command \"$CMD\"" > /dev/stderr
130+
exit 1
131+
fi

0 commit comments

Comments
 (0)