Add PUN-based mTLS proxy

[mattmix]

This is an attempt to address #1306. I wasn’t sure this was going to work, so I wrote a PoC first, which I’m submitting for feedback here.

It adds a PUN-based mTLS proxy configuration with the PUN managing the PKI infrastructure. The intended flow to the upstream server on the compute node looks like:

                              ┌──────────────────────────────┐
                              │ PUN Proxy                    │
                              │ /pun/proxy/<host>/<port>/... │
                              │                              │
                              └───────────────┬──────────────┘
                                              │               
                                              │               
                                            mTLS              
                                              │               
┌─────────────────────────────────────────────┼──────────────┐
│ Compute Node Job                            │              │
│                                             ▼              │
│                                      ┌─────────────┐       │
│ ┌────────────────────────────────────┤ slirp4netns ├─────┐ │
│ │ Net+User Namespace                 └───────┬─────┘     │ │
│ │                                            ▼           │ │
│ │  ┌──────────────────────────┐   ┌───────────────────┐  │ │
│ │  │ Backend Server           │ ◄─┤ Caddy             │  │ │
│ │  │ 127.0.0.1:<backend port> │   │ *:<frontend port> │  │ │
│ │  └──────────────────────────┘   └───────────────────┘  │ │
│ │                                                        │ │
│ └────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────┘

I had several things I wanted to do with this feature, addressing the requirements from #1306 and a few of my own:

• secure. No-one should be able to proxy to another user's interactive job. (taken from issue)
  • Per-user mTLS ensures that no one can use the proxy to successfully connect to another user's backend.
  • Caddy is the only bridge into the network namespace and requires mTLS, other users with jobs on the same compute node cannot connect to the backend.
• easy to install & maintain (taken from issue)
  • Only adds 2 new dependencies for compute nodes: caddy and slirp4netns. These packages are widely available on Ubuntu and RHEL derivatives (EPEL). Neither are services that need to run at the system level.
• Easy transport encryption for compliance requirements and for apps that do not easily support HTTPS.
  • As an example, this makes it easy to add transport encryption to RStudio, which locks TLS behind the Pro version
• Being able to use trusted headers for applications that support them
  • Some applications support HTTP header authentication, which can only be used safely if we can trust the provenance of those headers. mTLS + running the app in a network namespace allows us to do that.
• No privileges or firewall manipulation are required for any part of the setup

I chose Caddy because I felt it was the easiest to install and configure reverse proxy option that supports mTLS authentication. This could be replaced with something else.

You can see the bash functions that automate the mTLS/Caddy/netns setup in this simplified interactive app: https://github.com/mattmix/bc_mtls_headers/blob/main/template/script.sh.erb. If the direction of this PR is acceptable, I would make a PR for ood_core to add them to the bash_helpers generation.

The simplest usage in script.sh of an interactive app looks like:

start_mtls_caddy <outer port> <inner port> <server invocation with args>

outer port is the port reported back to connection.yml and which slirp4netns will forward through to Caddy inside the network namespace. inner port is what port the application server is listening on. This can be any port since it is in its own network namespace and cannot conflict with anything listening in the main netns. This function will generate a 35 day cert signed by the user's CA, create the netns, and start Caddy and the backend server.

If Caddy injected headers are desired, those can be added in script.sh as:

add_caddy_header X-Custom "Custom header example"

The jankiest part of this solution is handling NGINX's need for a defined resolver for this kind of proxying (also noted in the issue). To try to make this easy to deal with, I have the PUN read out of /etc/resolv.conf and grab the first nameserver listed there unless configured otherwise. I don't really like that, but I think it will work almost all of the time.

Things that I know are missing from this PR:

• Tests
• Documentation
• ood_core changes

I just want to confirm that this is acceptable in principle before putting more time in.

[johrstrom]

Hi thanks for the pull request. I can't quite look at it at the moment, because we're busy prepping the 4.2 release with WCAG fixes. So I'm not sure when we'll get to this, but won't likely get into 4.2.