How to manage CA certificate files for git, curl, and other open source tools

[MikeFultonDev]

I want to improve the zopen-setup tool to have it really be a 'works out of the box' experience.
There are probably a few other clunky areas, but one I'm keen to fix is around CA certificates that are needed to use tools like git and curl.
The current approach is ok but certainly not ideal and requires manual steps to get things setup.

Here are the goals from my perspective of a good solution (opinions wanted)

• Someone can upload zopen-setup, run it, and git and curl 'just work' after they run .bootenv to set up their environment
• Someone can get just the pax file for curl and/or git and it 'just works' - they do not need any other packages
• People can provide their own CA certificates file as an alternative to the one shipped
• People can use zopen download to download curl or git into their prod directory and they just work
• People can build their own version of curl or git into their dev directory and it just works
• When a CA certificate file gets updated (which is required because the certificates expire), it's easy to refresh the new CA cert file

Here's my proposal - alternatives welcome, and opinions around security and such also welcome.

• Change the meta package to no longer have a CA certificate in the downloadable pax file for prod/boot, but have it continue to own the primary version of the CA certificate file (perhaps called metacacert.pem), which can be updated periodically. The prod/boot version of meta will delegate to git and curl to provide their associated CA certificates. Ideally this would be a generated file that we download from the cURL site as part of the build process, but we could cache it if it takes too long to download.
• Have the build of the curl and git packages copy the metacacert.pem file from meta as part of their build process into their installation tree. Since zopen build is packaged in meta, we know the metacacert.pem file will always be available.
• Have the curl and git .env files set up their respective environment variables, but only if the environment variables are not already set. The curl .env will set up SSL_CERT_FILE to point to ${CURL_HOME}/curlcacert.pem IFF SSL_CERT_FILE is not already set. The git .env will set up GIT_SSL_CAINFO to point to ${GIT_HOME}/gitcacert.pem IFF SSL_CERT_FILE is not already set.

We will continue to have zopen update-cacert for people to use if they want to manage their CA certificates and do not want to upgrade their software package

Advanced users can also have just one common cacert.pem and manually export their environment variable to point to it, if they want full control over the CA certificate.

I think this satisfies the goals at the top, but perhaps there are other goals to consider?

The reason this is different than other development platforms is that typically the web browser on the system owns and manages the CA certs file and tools like git and curl can just use them, but we don't have that luxury on z/OS.
I am not sure how AIX tackles this or a stripped down Linux system that doesn't have a browser?

Everyone's opinions wanted. I would appreciate opinions specifically from @IgorTodorovskiIBM @AnthonyGiorgio @ejratl @phaumer @drbruce-git given your interest in the topic.

[IgorTodorovskiIBM]

That sounds like a good solution to me. As an alternative, we could consider modifying the buildenv of curl and git (maybe in zopen_post_install) to call 'zopen update-cacert', which would download the latest cacert.pem file. We might need to modify zopen update-cacert to be able to provide an output path. This way, when the package file is created, it would always include the latest version of the ca cert.

[MikeFultonDev]

I like that. It makes sense because meta doesn’t really need to be involved.

[AnthonyGiorgio]

What does the scenario look like for a system that doesn't have internet connectivity?

[MikeFultonDev]

By ‘no internet’ do you also mean no ability to connect out at all? If so there wouldn’t be a need for them to install git or curl.
if ‘no internet’ means you can connect to internal sites for the equivalent of GitHub.ibm.com then that’s interesting. In that scenario they would need to provide their own pem file. If so they could set their environment variables to point to it before they source .env. Is there a simpler way?

[v1gnesh]

Can this be moved to a separate config file, so that sites with their own pem packages (TLS intercept proxies at work) can fetch from some internal URL instead?
Or maybe feed that in as user input, asking for a URL where pem can be fetched from.
I don't know if many/most zOS shops can directly contact domains on the internet.

[MikeFultonDev]

I considered that. My concern was that developers would end up with their own pem files which the sysprog might not like. Or are you thinking a global config under /var ?

[v1gnesh]

Yeah, global is better. This thing is best configured & maintained by sysprog/security.

[AnthonyGiorgio]

I feel like z/OS is missing a feature here, where something like the Web Toolkit would own and manage certificates. It's as if we are having to implement this feature on our own.

[MikeFultonDev]

Web toolkit does do this but only through racf and not with pem files. A z/OS enhancement we might want to consider would be adding support for SAF CA certs.

[waichoi22]

From the Rocket forum: RACF keyring support for curl is being considered.

[ejratl]

Ubuntu/Debian have a separate package ca-certificates which can be a dependency for packages which rely on having the certificates in place. See https://packages.ubuntu.com/kinetic/ca-certificates for the package or this article for a description with more words. In this way a browser is not needed for a server build.
With that said, I like the suggestion that Igor proposed above and will add my +1 to it.

[MikeFultonDev]

I like @ejratl 's answer here after some thinking... @IgorTodorovskiIBM and others - what do you think? Taking an approach similar to the Linux distros has a lot of value to it.

[v1gnesh]

It'll be great if it works... would this mean that sites will now have to package their CAs into rpm or deb?
And zopen somehow add a pre-req / dependency on a site-supplied package?
Allowing the installation to point to a pem file seems quicker & more in line with what one can expect from sites, I think.

[AnthonyGiorgio]

I like the separate package idea. This way it's managed like any other dependency.

[IgorTodorovskiIBM]

Brew has the same thing here: https://formulae.brew.sh/formula/ca-certificates

We have the runtime dependency support in zopen now, so this is doable

[waichoi22]

Hi all, Any thoughts to point to a RACF keyring instead of a CA file package? Wai

…

-------- Original Message -------- From: Igor Todorovski ***@***.***> Date: Thu, March 16, 2023 9:52 AM -0700 To: ZOSOpenTools/meta ***@***.***> CC: Wai Choi ***@***.***>, Comment ***@***.***> Subject: [EXTERNAL] Re: [ZOSOpenTools/meta] How to manage CA certificate files for git, curl, and other open source tools (Discussion #130) Brew has the same thing here: https: //formulae. brew. sh/formula/ca-certificates We have the runtime dependency support in zopen now, so this is doable — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Brew has the same thing here: https://formulae.brew.sh/formula/ca-certificates<https://formulae.brew.sh/formula/ca-certificates> We have the runtime dependency support in zopen now, so this is doable — Reply to this email directly, view it on GitHub<https://github.com/orgs/ZOSOpenTools/discussions/130#discussioncomment-5336787>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AXTWEPXPFGFR2MV6FSAYXBTW4NAN7ANCNFSM6AAAAAAUCU323I>. You are receiving this because you commented.Message ID: ***@***.***>

[MikeFultonDev]

I think we are now at the point where we don't actually need certs anymore? Can we close this?

[waichoi22]

Would you clarify "we don't actually need certs anymore" ?

[IgorTodorovskiIBM]

We ship curl's ca certificate along with Git now and if no override is present (git config setting or GIT_SSL_CERT environment variable), we set git to to use curl's ca certificate.

[waichoi22]

So I can specify a RACF keyring using GIT_SSL_CERT now?

[MikeFultonDev]

no - not yet. @waichoi22 i would recommend you open up an issue - right now the choices are to use the cert we provide, or to specify your own with an env var. But having RACF support would be a nice enhancement. Probably should be opened under the 'meta' repo since this is really a cross-tools enhancement