Launch cockpit-session via socket activation on /run/cockpit/session

[allisonkarlitskaya]

This paves the way for removing the static cockpit users in #16811, and fixes cockpit on bootc images.

A nice side effect of this is that we can now connect to unix sockets from cockpitauth, which is useful for https://github.com/allisonkarlitskaya/cockpit-cloud

• cockpit.spec: drop --with-selinux-config-type #16810
• ws: add support for connecting to unix sockets #16819
• Preparations for launching cockpit-session via unix socket activation #21258
• session: Drop cgroupsv1 compatibility for certificate authentication #21260 and simplify to a read() call (no stream FILE* any more)
• ws: Fix reported error if the authentication process is not available; containers/ws: Drop cockpit-session #21291
• session: Pass remote peer address in authorize message, add authorize response timeout, robustify client certificate mapping errors #21292
• pass memfd with login information from ws to session not required. That information comes from cockpit-session to cockpit-bridge, which is unaffected by this change.
• pass $COCKPIT_REMOTE_PEER from ws to session
• fix cgroup detection for cert login
• SELinux rejections for TestLogin.testSELinuxRestrictedUser
• cockpit-session@<id>.service fails with error 5/NOTINSTALLED initially (e.g. TestTerminal.testBasic)
• Clean up Debian statoverride on upgrades
• fix reporting "no cockpit" error -- it fails cockpit-session@*.service which shows up as "Authentication failed" and leaves a failed unit behind. (TestConnection.testBasic)
• fix running cockpit-ws outside of socket activation failure

Fixes #21201

Web server: Increased sandboxing, setuid removal, bootc support

Cockpit's web server already had low privilege levels, but previously used a setuid helper program for user logins. This helper program had restricted permissions and was only executable by Cockpit (through group ownership). Its sole purpose was to run at the system level and immediately drop permissions to log in to a specific user account. However, the binary was still setuid, and setuid should be minimzed for security reasons.

This release removes the setuid flag from the binary. The helper now starts via systemd socket activation, with the Cockpit web server connecting to it using a protected UNIX socket in the /run directory. This approach enables tighter sandbox security by preventing the login session from being a direct descendant of the web server process. It also resolves an issue with Cockpit running on bootc images.

All Cockpit components now run as dynamic users created at startup using the DynamicUser= systemd feature. Existing systems may still have a cockpit-ws user (and very old systems might even have associated TLS certificates). However, this cockpit-ws user is no longer required and can be safely deleted.

[allisonkarlitskaya]

Thanks for allowing me to be pedantic about this <3

[mvollmer]

Thanks!

[mvollmer]

This gets removed in a later commit. Can we squash that?

[martinpitt]

This is mostly me being picky: Technically, that commit (session: Support Unix socket mode for client certificates) still requires the fallback. It is made obsolete by "ws: connect to cockpit-session via socket", but that one also cannot land before. So it's a very short transitionary period (just over two commits), but nevertheless for "I can build and test or revert a random commit" correctness I think we should keep it.

[mvollmer]

Thanks! Then let me review it: errno might be overwritten here, no? And it prints a FD instead of a PID, is that intended?

[martinpitt]

ah yes, that's in the wrong order, the debug() should come before the open_proc_pid() of course. Thanks, let me fix that completely dead piece of code!

[martinpitt]

🌟

[bluca]

You are probably already aware, but just in case, in polkit there were CVEs due to comparing processes in this way for authentication purposes, as there are ways for an attacker to control the start timestamp as recorded by the kernel, by holding the forking process at a particular time. See comment at: https://github.com/polkit-org/polkit/blob/main/src/polkit/polkitunixprocess.c#L59

These days we use pid fds which cannot be recycled, and only fallback on heuristics on old kernels.

[allisonkarlitskaya]

Thanks for this comment. We discussed this an awful lot. Please see this thread:

#16808 (comment)

[allisonkarlitskaya]

It would be helpful to know a bit more about what the pidfd peercred thing actually solves: does it mean that for as long as the kernel is willing to return that pidfd to us that it then won't reissue that pid? That would be a very big help indeed, but it would also mean that we don't actually need to use the pidfd to gain that benefit...

Or (in the case that the process that called connect() is gone) will it issue us a pidfd that points to a process that doesn't exist anymore and possibly the pid of that process already got used?

I admittedly stopped looking into the polkit code after @martinpitt determined that it wasn't available on many of our target platforms and expressed reservations about implementing the fallback, but it would be useful to know how this ought to work, and I do believe that it will be a worthwhile thing to pursue in the future.

In the meantime, though, please understand that this code is only relevant on systems that support logging in with TLS client certificates, and this is only one part of a defence-in-depth strategy:

• only cockpit-ws can connect to the socket, as we control access to it via its group ownership
• we use a "nonce" system: a randomly-generated certificate file name is created in cockpit-tls. That nonce is passed only to the process that's supposed to be logging in using that client certificate. That file is written to a temporary directory only accessible by root, and lives only as long as the TCP connection is open. In order to convince cockpit-session that you are allowed to login, you need to know the name of the file in that directory.
• the first line of that file contains the name of the cgroup of the process that is expected to be logging in using that file and we perform that double check based on the cgroup lookup we're discussing here.

We also limit the time that the login process is allowed to linger to 60 seconds in an attempt to make it more difficult to wait for a recycled PID.

One of the attack scenarios that we consider possible in Cockpit is that one legitimate user could exploit a weakness in the large C code base of cockpit-ws to take over control of the process and use it to gain control over a logged-in session of another user. This is the main reason for why we go out of our way to isolate client-certificate-identified sessions from each other. But: I notice that we could be doing more limiting what cockpit-ws is able to do, via the large number of sandboxing options available in systemd unit files. I'm not sure, for example, that it wouldn't be possible for one cockpit-ws instance to gain control of another via ptrace() — they're running as the same user. We restrict things quite a lot with our custom selinux policy, but it would be good to look into locking things down with systemd. I'll open an issue for that.

[bluca]

It would be helpful to know a bit more about what the pidfd peercred thing actually solves: does it mean that for as long as the kernel is willing to return that pidfd to us that it then won't reissue that pid? That would be a very big help indeed, but it would also mean that we don't actually need to use the pidfd to gain that benefit...

A pidfd can never be recycled, when the process exits, it will simply become invalid and can no longer be resolved to a pid, even if a process with the same pid has since appeared. With recent kernels you can use statx() and compare directly two pidfds for equality, we do this in systemd now for example. On slightly older kernels you get two pidfds, you resolve each pidfd -> pid, compare the pid, and then afterwards check again that both pidfds are still valid - more laborious than statx(), but equally safe against all races.
You can get the pidfd of a socket peer via SO_PEERPIDFD, this is safe because it's the kernel that gives you the pidfd, so you can be always sure it refers to the process that started the communication over the socket, it cannot be faked.

In summary, it is a feature designed exactly to avoid the kind of race conditions and vulnerabilities that have plagued polkit and other system software that need to authenticate processes.

In the meantime, though, please understand that this code is only relevant on systems that support logging in with TLS client certificates, and this is only one part of a defence-in-depth strategy

Of course, it's entirely up to you, and it is dependent on use cases, threat model, etc. I only mentioned because I found the link to this PR on Mastodon, and having had to deal with all of this recently in polkit, I thought of giving a heads-up, as it's a fairly recent kernel feature so it's not very well known yet. But if it's not necessary for the use case here, that's entirely fine, this was just intended as an FYI, not an RFE.

[martinpitt]

@bluca Thanks! My main woe is that just getting a pidfd doesn't help us much. We need to know the peer's startup time (currently parsed from /proc/pid/stat) and cgroup (/proc/pid/cgroup), and pidfds have neither these APIs nor "give me the corresponding /proc/pid".

If SO_PEERPIDFD is guaranteed to give the pid fd at the time of connect(), and resolving it fails once it's gone, that is a good hardening. We'd still have to resolve it to a pid and open /proc/pid, but it would replace the time comparison on newer kernels -- and we'd keep the time comparison as a fallback on older ones.

[martinpitt]

We won't need /stat any more, but we very much need the cgroup. That's the reason for doing all this dance 😁 (we use that to identify the correct service instance for the TLS certificate that we receive, and encode the cert's SHA into the unit's instance name)

Thanks for confirming the pidfd mechanics! That's how I understood it, and that will actually be helpful.

[bluca]

Right, in the future we'll hopefully have a SO_PEERCGROUP or so, work was in progress. For now one can ask systemd to translate from pidfd to cgroup in various ways, like sd_pidfd_get_cgroup()

[bluca]

In fact if you want the unit, there's APIs to get that directly, via D-Bus

[martinpitt]

We don't have D-Bus in cockpit-session. It's a minimal program which should have essentially zero dependencies. But we'll do the pidfd robustification, thanks!

[martinpitt]

Done in #21341 . Thanks @bluca!