Puller not working with (custom) system certs

[abergmeier]

We have custom certs at my company. These are installed in /usr/local/share/ca-certificates and work fine with Browsers and most tools (on Ubuntu 16.04.3).
Sadly puller is the exception. It fails with:

File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/__main__.py", line 93, in <module>
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/__main__.py", line 80, in main
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/containerregistry/client/v2_2/docker_image_.py", line 324, in __enter__
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/containerregistry/client/v2_2/docker_http_.py", line 191, in __init__
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/containerregistry/client/v2_2/docker_http_.py", line 217, in _Ping
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/containerregistry/transport/transport_pool_.py", line 62, in request
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/httplib2/__init__.py", line 1659, in request
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/httplib2/__init__.py", line 1399, in _request
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/httplib2/__init__.py", line 1319, in _conn_request
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/httplib2/__init__.py", line 1092, in connect
httplib2.SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

Is there any way of telling that to use the system certificates?
If not, where does puller/httplib2 store the certificates?

[abergmeier]

Trying to fix this in google/containerregistry#50.

[calder]

This fixed the error for me:

sudo pip install httplib2.system-ca-certs-locater

[roblg]

I'm still investigating and trying to wrap my head around everything, but I believe we're experiencing issues (CERTIFICATE_VERIFY_FAILED) connecting to private registry secured by a certificate signed by a real root CA (Comodo) as well, so this issue may not be limited to custom certs. My current theory is that the root CA that signed our cert isn't bundled with httplib2, way down in the depths of containerregistry.

curl, docker push, my browser, and other non-containerregistry interactions with the private registry succeed w/ no issue.

Will report back with more info as I find it.

Update: Ultimately, I think our issue is the same as described above, even though it's not a custom cert -- we have a cert signed by a root CA that isn't included in the cacerts.txt bundled with httplib2. We're having issues with both container_pull and container_push though. Seems like the issue is in containerregistry, so I'll go bug them. :)

[sbueringer]

We're having the same problem with our registry with custom certs. That's our solution:

pip install httplib2.system-ca-certs-locater
# You may have to install the custom certificate in your OS truststore
# If your OS stores the certs in `/etc/ssl/certs/ca-certificates.crt` you're done
# If not, set the env var `HTTPLIB_CA_CERTS_PATH`, e.g.:
export HTTPLIB_CA_CERTS_PATH=/etc/ssl/certs/ca-bundle.crt

[roblg]

@sbueringer thanks for the suggestion. The pip install doesn't actually work for us, in part because httplib2.system-ca-certs-locater makes a pretty limited, specific assumption about where the default certificates file lives that for whatever reason doesn't match the way our environment is set up. I also see the HTTPLIB_CA_CERTS_PATH in the pypy page for the package, but I don't see anything in the code itself of the locater package, or of httplib2 core (and couldn't get it to make a difference) update: the code that's actually installed looks different than the github repo that's linked from the pypy page, so I dunno why it wasn't working. I'll take another crack at that, I guess.

I'm curious how folks in this thread feel about the proposal in #539? (assuming google/containerregistry#89 is merged first)

[sbueringer]

Yes I thought the same as I saw the code at the pypy page, but the installed code makes a lot more sense :).
So the difference between the different setups in our team is just the default location of the cert bundle in the OS. That's why Archlinux for example works per default, but Fedora needs the environment variable.

[nlopezgi]

Hi everyone,
we are working towards fixing this in #594 for push and will followup with a PR to fix for pull by enabling use of a custom docker client config.

[ali5h]

any update on this?

[nlopezgi]

it's implemented, but we have not updated the docs yet. You want to try it out? you need to import docker_toolchain_configure as here: https://github.com/bazelbuild/rules_docker/blob/master/testing/custom_toolchain_auth/WORKSPACE#L32. Note the client_config should point to the absolute path where the config is located (in the example we replace the value of client_config in the e2e tests here: https://github.com/bazelbuild/rules_docker/blob/master/testing/e2e.sh#L418.
And then import the custom container_pull as here https://github.com/bazelbuild/rules_docker/blob/master/testing/custom_toolchain_auth/WORKSPACE#L60 (note the name of the def there has 'local' just for testing purposes, but it does not need to use a local registry). (To use container_push with custom auth all you need is adding the docker_toolchain_rule to workspace). Do let me know if it works or you run into issues.

[ali5h]

For now, I fixed my problem by building a custom puller.par based on httplib2==0.12.0, and used that to override what rules_docker uses. This needs injecting proper env variable using action_env

[smukherj1]

We added some documentation on how to configure the docker toolchain rule to use custom authentication credentials specified via a docker client config.json in #686

You can specify a custom config.json when configuring the docker toolchain rule that will be used by every container_push rule automatically. container_pull requires a little extra work and our updated docs specifies what that is.

Note: All this does is make containerregistry load the config.json specified in the toolchain configuration. It's up to containerregistry to read what's in that file and interpret it appropriately.

Docker's official documentation on the client configuration file is at https://docs.docker.com/engine/reference/commandline/cli/#configuration-files