Usage of `systemd-cgroup` flag produces substantial load on DBus

[mattnappo]

Description

At modal.com we run a custom multi-tenant container runtime which can use runc or runsc (gVisor). We use the systemd-cgroup flag on all of our containers. Recently, we noticed that DBus on the host machine becomes unresponsive when many containers spin up/down rapidly. For example, when a host spins up sufficiently many containers, the following command will take 10+ seconds, often timing out entirely after 25s:

busctl get-property org.freedesktop.systemd1 /org/freedesktop/systemd1 \
           org.freedesktop.systemd1.Manager Version

However, when we don't use systemd-cgroup, DBus is much more responsive, notably with much shorter tail latencies.

We ran an experiment which spins up n containers while timing the above command in a loop every second. We repeated this experiment on increasing values of n with and without the systemd-cgroup flag enabled. Each container just does sleep inf. This data covers a few seconds of host idling, container startup, 30 seconds of container runtime, and container shutdown. Ran on a AWS c6i.8xlarge.

My understanding is that when systemd-cgroup is not used, runc configures cgroups directly via cgroupfs which places zero additional load on DBus. How does runc configure cgroups when the flag is used? Does it write a systemd DBus message to the DBus socket, or does it write to the systemd private socket?

It seems like a performance concern that the difference in D-Bus lag between using and not using the systemd-cgroup flag is so large.

Steps to reproduce the issue

1. In a loop, measure how long a simple DBus method_call takes, such as org.freedesktop.systemd1.Manager Version on the host machine.
2. Spin up n containers that sleep inf, where n is large enough to create DBus load as measured by (1)
3. Let the containers run for a few seconds, then tear them down
4. Observe the DBus lag measurements.

Describe the results you received and expected

We expect similar DBus responsiveness whether or not we use systemd-cgroup.

What version of runc are you using?

runc version 1.2.5
commit: v1.2.5-0-g59923ef
spec: 1.2.0
go: go1.23.7
libseccomp: 2.5.2

Host OS information

Ran on a AWS c6i.8xlarge

NAME="Oracle Linux Server"
VERSION="9.6"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="9.6"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Oracle Linux Server 9.6"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:9:6:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://github.com/oracle/oracle-linux"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 9"
ORACLE_BUGZILLA_PRODUCT_VERSION=9.6
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=9.6

Host kernel information

Linux 5.15.0-309.180.4.el9uek.x86_64 #2 SMP Wed May 21 06:56:22 PDT 2025 x86_64 x86_64 x86_64 GNU/Linux

[alam0rt]

Wanting to add onto this, we are seeing the same.

ftrace

/sys/kernel/debug/tracing# cat trace
# tracer: function
#
# entries-in-buffer/entries-written: 38/38   #P:64
#
#                                _-----=> irqs-off/BH-disabled
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
            runc-1756534 [044] ..... 529725.338844: cgroup_kn_lock_live <-__cgroup_procs_write
            runc-1756534 [044] ..... 529725.338848: kernfs_break_active_protection <-cgroup_kn_lock_live
            runc-1756533 [047] ..... 529725.338916: cgroup_kn_lock_live <-__cgroup_procs_write
            runc-1756533 [047] ..... 529725.338917: kernfs_break_active_protection <-cgroup_kn_lock_live
            runc-1756534 [030] ..... 529725.349460: cgroup_kn_unlock <-__cgroup_procs_write
            runc-1756533 [047] ..... 529725.349526: cgroup_kn_unlock <-__cgroup_procs_write
            runc-1756636 [044] ..... 529725.911796: cgroup_kn_lock_live <-__cgroup_procs_write
            runc-1756636 [044] ..... 529725.911799: kernfs_break_active_protection <-cgroup_kn_lock_live
            runc-1756636 [006] ..... 529725.925457: cgroup_kn_unlock <-__cgroup_procs_write
            runc-1756676 [042] ..... 529726.163752: cgroup_kn_lock_live <-__cgroup_procs_write
            runc-1756676 [042] ..... 529726.163756: kernfs_break_active_protection <-cgroup_kn_lock_live
            runc-1756676 [030] ..... 529726.174516: cgroup_kn_unlock <-__cgroup_procs_write
            runc-1756718 [061] ..... 529726.337664: cgroup_kn_lock_live <-__cgroup_procs_write
            runc-1756718 [061] ..... 529726.337667: kernfs_break_active_protection <-cgroup_kn_lock_live
            runc-1756719 [049] ..... 529726.337966: cgroup_kn_lock_live <-__cgroup_procs_write
            runc-1756719 [049] ..... 529726.337969: kernfs_break_active_protection <-cgroup_kn_lock_live


...

         systemd-1       [037] .N... 530000.002701: cgroup_kn_unlock <-cgroup_rmdir
            runc-1815966 [015] ..... 530000.006202: cgroup_kn_lock_live <-__cgroup_procs_write
            runc-1815966 [015] ..... 530000.006205: kernfs_break_active_protection <-cgroup_kn_lock_live
            runc-1815946 [061] ..... 530000.006924: cgroup_kn_lock_live <-__cgroup_procs_write
            runc-1815946 [061] ..... 530000.006926: kernfs_break_active_protection <-cgroup_kn_lock_live
         systemd-1       [005] ..... 530000.007092: cgroup_kn_lock_live <-cgroup_rmdir
         systemd-1       [005] ..... 530000.007094: kernfs_break_active_protection <-cgroup_kn_lock_live

it appears to be lock contention, mainly around starting/stopping of pods.

Tracing cgroup kernel function timings... Hit Ctrl-C to end and report.
<tid:1> Function: cgroup_kn_lock_live, Latency: 10558 ms

<tid:1> Function: cgroup_kn_lock_live, Latency: 21217 ms
<tid:1> Function: cgroup_kn_lock_live, Latency: 32721 ms
<tid:1> Function: cgroup_kn_lock_live, Latency: 22338 ms
<tid:1> Function: cgroup_kn_lock_live, Latency: 3249 ms
<tid:1> Function: cgroup_kn_lock_live, Latency: 32730 ms
<tid:1> Function: cgroup_kn_lock_live, Latency: 15 ms
<tid:1> Function: cgroup_kn_lock_live, Latency: 32749 ms
<tid:1> Function: cgroup_kn_lock_live, Latency: 30 ms
<tid:1> Function: cgroup_kn_lock_live, Latency: 25 ms
<tid:1> Function: cgroup_kn_lock_live, Latency: 20 ms
<tid:1> Function: cgroup_kn_lock_live, Latency: 86 ms
<tid:1> Function: cgroup_kn_lock_live, Latency: 105 ms
<tid:1> Function: cgroup_kn_lock_live, Latency: 186 ms
<tid:1> Function: cgroup_kn_lock_live, Latency: 218 ms
<tid:1> Function: cgroup_kn_lock_live, Latency: 184 ms
<tid:1> Function: cgroup_kn_lock_live, Latency: 237 ms

using this bpftrace script

BEGIN
{
    printf("Tracing cgroup kernel function timings... Hit Ctrl-C to end and report.\n")
}

kprobe:cgroup_kn*
{
    @start[tid, reg("sp")] = nsecs;
}

kretprobe:cgroup_kn*
{
    if (@start[tid, reg("sp")]) {
        $latency_ns = nsecs - @start[tid, reg("sp")];
        $latency_ms = $latency_ns / 1000000;
        @latency_ns[func] = hist($latency_ns);
        if ($latency_ms > 10) {
            printf("<tid:%d> Function: %s, Latency: %d ms\n", tid, func, $latency_ms);
        }
    }
    delete(@start[tid, reg("sp")]);
}

Appears to be once our nodes hit about 200+ pods (16xlarge in our case).

runc version 1.3.0
commit: v1.3.0-0-g4ca628d1
spec: 1.2.1
go: go1.23.8
libseccomp: 2.5.6

6.8.0-1033-aws

[cyphar]

--systemd-cgroup does use DBus to tell systemd to configure a TransientUnit for the container with the same resource constraints as we would configure (well, the ones systemd supports at least). This is usually necessary on systemd-based systems because systemd "owns" the cgroup tree and unless you explicitly tell it to delegate it to you, it can end up silently removing resource restrictions from containers when it decides to re-apply changes to the cgroup tree. We use a library for this (github.com/coreos/go-systemd), and it probably uses the global DBus.

One possible workaround would be to start containers with the cgroupfs manager but inside a systemd scope that the administrator has configured with Delegate=yes (by setting cgroupParent -- I think kubelet has a way of configuring this, but I'm not sure). That (in principle) should avoid the issues you could encounter otherwise without requiring the use of DBus.

As for the general lock contention issues in cgroups, CLONE_INTO_CGROUP is the only way for us to avoid that particular problem (and it is being worked on #4812 for runc exec at least but there are some issues with getting it working).

[alam0rt]

Thanks @cyphar for that information. I may give the cgroupfs-in-a-scope configuration a go. Worst case, we will need to limit how many pods we can run (and probably revert to smaller instances). Best case (aside from runc exec changes you mention)... It "just works" using the cgroupfs driver.

[kolyshkin]

Yes, as it was noted above by @cyphar, runc --systemd-cgroup start/stop uses dbus to tell systemd to start/stop transient unit. When --systemd-cgroup flag is not used, runc doesn't use dbus at all.

I don't think there's a way around it, if you want containers to be systemd units. If you don't, delegate a cgroup and run all your containers in it, without using --systemd-cgroup (in this case, runc just writes directly to cgroupfs).

As to the dbus contention, it seems it's a problem with dbus rather than systemd. Having said that, if you can do more analysis in the area it would be interesting to know your findings.

[alam0rt]

@kolyshkin, I think i've found the issue. I turned on debug logging and

# jobs
[1]+  Running                 journalctl -b -t systemd -f --grep limit &

Aug 18 00:30:40 systemd[1]: Event source 0x5ec19d0f92a0 (mount-monitor-dispatch) entered rate limit state.

# while true; do timeout 1 busctl get-property org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager Version &>/dev/null; echo "$?"; sleep 1; done
124 # returns non-zero if timeout
124
124
124
124
Aug 18 00:30:57 systemd[1]: Event source 0x5ec19d0f92a0 (mount-monitor-dispatch) left rate limit state.
0 # working.
0
0

Rate limited by systemd fairly sure. This makes sense as

• dbus process never appeared to be under much load
• using busctl to query non systemd buses etc never had any issue

Going to try to play with SYSTEMD_DEFAULT_MOUNT_RATE_LIMIT_BURST and SYSTEMD_DEFAULT_MOUNT_RATE_LIMIT_INTERVAL_SEC.

Edit: unfortunately I am on v249.11 and those tuneables are not available

[mattnappo]

Hi @alam0rt. Thank you for your helpful bpftrace script! Can you please clarify whether the output you posted is using systemd or cgroupfs to configure cgroups? I was only able to reproduce kernel lock contention when using the cgroupfs driver. I assume systemd does its own write serialization. Thank you.

[cyphar]

I would naively expect both to see similar lock contention because in both cases we (runc) are adding a process to a cgroup (and creating / destroying cgroups in the hierarchy). I was also mistaken earlier -- CLONE_INTO_CGROUP still takes cgroup_mutex so it won't help with this lock contention issue AFAICS.

[alam0rt]

Hi @alam0rt. Thank you for your helpful bpftrace script! Can you please clarify whether the output you posted is using systemd or cgroupfs to configure cgroups? I was only able to reproduce kernel lock contention when using the cgroupfs driver. I assume systemd does its own write serialization. Thank you.

This was using the systemd driver.

[kolyshkin]

So, is there anything we can do about it?

What runc does can be seen from the source code (or via dbus-monitor). It's basically using StartTransientUnit, SetUnitProperties, StopTransientUnit, ResetFailedUnit, and, since a recent PR #4822, exec also uses AttachProcessesToUnit.

I ran a trace using

dbus-monitor --system"type='method_call',path='/org/freedesktop/systemd1',interface='org.freedesktop.systemd1.Manager'"

and did not find anything that's bad, excessive, or suspicious.