How can I use a custom SSLContext / PyOpenSSLContext when creating a Client?

[kafonek]

Checklist

• [x ] I searched the HTTPX documentation but couldn't find what I'm looking for. I looked at:
  • https://www.python-httpx.org/api/#client
  • https://www.python-httpx.org/advanced/#ssl-certificates
• [x ] I looked through similar issues on GitHub, but didn't find anything. I looked at:
  • How to force Client to use a specific TLS version? #469
  • Where is SSLConfig? #904
  • Transport API #768
• [x ] I looked up "How to do ... in HTTPX" on a search engine and didn't find any information.
• I asked the community chat for help but didn't get an answer.

Question

Can I create a httpx.Client with my own ssl.SSLContext or urllib3.contrib.pyopenssl.PyOpenSSLContext instead of passing in cert/key/verify?

Background

I found httpx while looking for requests syntax + asyncio support. It looks like a great project, thanks for all the work you've put into it.

One very oft-asked feature for requests.py was making requests with user-provided SSLContexts 2118. Eventually that was resolved by allowing us to pass SSLContexts to Adapters, then mounting the adapter onto a session. As a real world example, I have used pypki2 and requests_pkcs12 at different times to create Session's from PKCS12/X509 certificates instead of PEM format that I believe vanilla requests and httpx require. There is a requests_pkcs12 author blog post with more background.

My understanding from reading httpx documentation is that a httpx.Client is roughly similar to a requests.Session and the Dispatcher API will be roughly similar to Adapters. @tomchristie mentions configuring an ssl_context in 768 - Dispatcher API but I didn't understand how to use that in practice.

@sethmlarson suggests httpx.Client(verify=ssl_context) in 469, which looked similar to my use-case but not identical. When I tried that with httpx 0.12.1 and 0.13.0.dev, I got TypeError: expected str, bytes or os.PathLike object, not PyOpenSSLContext on Client init.

Thanks.

(courtesy tagging @rashley-iqt (requests_pkcs12) and @gershwinlabs (pypki2) )

[sethmlarson]

Thanks for bringing this use-case to my attention. Got me thinking how this use case can be reconciled with my other thoughts on safe high level TLS APIs.

[tomchristie]

Heya,

How can I use a custom SSLContext / PyOpenSSLContext when creating a Client?

Yup, the verify argument optionally takes an SSLContext instance. So let's thrash out if there's an issue here or not.

One thing that I did notice prompted by this ticket, is that even though that's a supported usage, and is type annotated as bool|str|SSLContext, the SSLContext usage isn't actually currently noted in the docs. https://www.python-httpx.org/advanced/#ssl-certificates

That's something we should clearly improve. We ought to include a very minimal possible example, such as...

>>> import httpx
>>> import ssl
>>> import certifi
>>> context = ssl.create_default_context()
>>> context.load_verify_locations(cafile=certifi.where())
>>> httpx.get('https://www.example.com', verify=context)
<Response [200 OK]>

I got TypeError: expected str, bytes or os.PathLike object, not PyOpenSSLContext on Client init.

Could you include a traceback with that?

We don't have any instances of PathLike in the httpx codebase, and the example above works as expected, so it's not immediately obvious how to replicate this?

[kafonek]

Thanks for the reply @tomchristie. I went back and did some more tests on httpx 0.12.1 and 0.13.0.dev. It looks like the verify= syntax does work for both httpx.get(url, verify=context) and httpx.Client(verify=context).get(url) for ssl.SSLContext objects generated by pypki2. I should have tested that initially.

For requests_pkcs12 generated urllib3.contrib.pyopenssl.PyOpenSSLContext objects, I get the same general error. Here's the full traceback from httpx 0.12.1:

import requests_pkcs12
content = open('/home/jovyan/.devpki/mrkafon.p12', 'rb').read()
context = requests_pkcs12.create_ssl_context(content, b"changeme")
context.load_verify_locations(cafile='/home/jovyan/.devpki/vast-ca.pem')
context
>>> <urllib3.contrib.pyopenssl.PyOpenSSLContext at 0x7fe95baafa90>

url = "https://internal-site.com"
resp = httpx.get(url, verify=context)
resp
>>>

TypeError                                 Traceback (most recent call last)
<ipython-input-3-2fa41aa34910> in <module>
      1 url = "https://internal-site.com"
----> 2 resp = httpx.get(url, verify=context)
      3 resp

/opt/conda/lib/python3.7/site-packages/httpx/_api.py in get(url, params, headers, cookies, auth, allow_redirects, cert, verify, timeout, trust_env)
    166         verify=verify,
    167         timeout=timeout,
--> 168         trust_env=trust_env,
    169     )
    170 

/opt/conda/lib/python3.7/site-packages/httpx/_api.py in request(method, url, params, data, files, json, headers, cookies, auth, timeout, allow_redirects, verify, cert, trust_env)
     80     """
     81     with Client(
---> 82         cert=cert, verify=verify, timeout=timeout, trust_env=trust_env,
     83     ) as client:
     84         return client.request(

/opt/conda/lib/python3.7/site-packages/httpx/_client.py in __init__(self, auth, params, headers, cookies, verify, cert, proxies, timeout, pool_limits, max_redirects, base_url, dispatch, app, trust_env)
    470             dispatch=dispatch,
    471             app=app,
--> 472             trust_env=trust_env,
    473         )
    474         self.proxies: typing.Dict[str, SyncDispatcher] = {

/opt/conda/lib/python3.7/site-packages/httpx/_client.py in init_dispatch(self, verify, cert, pool_limits, dispatch, app, trust_env)
    499 
    500         return URLLib3Dispatcher(
--> 501             verify=verify, cert=cert, pool_limits=pool_limits, trust_env=trust_env,
    502         )
    503 

/opt/conda/lib/python3.7/site-packages/httpx/_dispatch/urllib3.py in __init__(self, proxy, verify, cert, trust_env, pool_limits)
     33     ):
     34         ssl_config = SSLConfig(
---> 35             verify=verify, cert=cert, trust_env=trust_env, http2=False
     36         )
     37         hard_limit = pool_limits.hard_limit

/opt/conda/lib/python3.7/site-packages/httpx/_config.py in __init__(self, cert, verify, trust_env, http2)
     69         self.trust_env = trust_env
     70         self.http2 = http2
---> 71         self.ssl_context = self.load_ssl_context()
     72 
     73     def __eq__(self, other: typing.Any) -> bool:

/opt/conda/lib/python3.7/site-packages/httpx/_config.py in load_ssl_context(self)
     92 
     93         if self.verify:
---> 94             return self.load_ssl_context_verify()
     95         return self.load_ssl_context_no_verify()
     96 

/opt/conda/lib/python3.7/site-packages/httpx/_config.py in load_ssl_context_verify(self)
    121         elif isinstance(self.verify, bool):
    122             ca_bundle_path = self.DEFAULT_CA_BUNDLE_PATH
--> 123         elif Path(self.verify).exists():
    124             ca_bundle_path = Path(self.verify)
    125         else:

/opt/conda/lib/python3.7/pathlib.py in __new__(cls, *args, **kwargs)
   1020         if cls is Path:
   1021             cls = WindowsPath if os.name == 'nt' else PosixPath
-> 1022         self = cls._from_parts(args, init=False)
   1023         if not self._flavour.is_supported:
   1024             raise NotImplementedError("cannot instantiate %r on your system"

/opt/conda/lib/python3.7/pathlib.py in _from_parts(cls, args, init)
    667         # right flavour.
    668         self = object.__new__(cls)
--> 669         drv, root, parts = self._parse_args(args)
    670         self._drv = drv
    671         self._root = root

/opt/conda/lib/python3.7/pathlib.py in _parse_args(cls, args)
    651                 parts += a._parts
    652             else:
--> 653                 a = os.fspath(a)
    654                 if isinstance(a, str):
    655                     # Force-cast str subclasses to str (issue #21127)

TypeError: expected str, bytes or os.PathLike object, not PyOpenSSLContext

If it helps, I'm using my own internal CA to sign the .p12 certs, and that CA is used by the internal server I'm connecting to. I can get to the internal site using the same .p12 and CA with requests_pkcs12. It's still possible I'm messing up the cafile loading though?

Thanks.

[tomchristie]

If you want custom SSL context support, we do have that, but you need to be using an SSLContext instance, from the stdlib.

We don't have support for PyOpenSSLContext, and the SSL configuration is (correctly) falling through the if isinstance(self.verify, ssl.SSLContext) check, and raising an error at the point that it attempts to treat the context as a string.

httpx/httpx/_config.py

Lines 107 to 115 in d34c89a

	if isinstance(self.verify, ssl.SSLContext):
	# Allow passing in our own SSLContext object that's pre-configured.
	context = self.verify
	self._load_client_certs(context)
	return context
	elif isinstance(self.verify, bool):
	ca_bundle_path = self.DEFAULT_CA_BUNDLE_PATH
	elif Path(self.verify).exists():
	ca_bundle_path = Path(self.verify)

I'd suggest you start by looking into setting up a custom SSLContext instance, and passing that to verify=....

If there's some capabilities exposed by the third party pyopenssl package, that aren't also present in Python 3's ssl implementation, then it's feasible that there's a valid feature request in here that we could work towards, tho it looks like pyopenssl is more aimed at improving SSL capabilities for Python 2.7-3.5.

[beneshed]

Seconding this, pyopenssl allows the use of loading certificates from memory instead of from disk

[whiteowl3]

If there's some capabilities exposed by the third party pyopenssl package, that aren't also present in Python 3's ssl implementation, then it's feasible that there's a valid feature request in here that we could work towards, tho it looks like pyopenssl is more aimed at improving SSL capabilities for Python 2.7-3.5.

There absolutely is such functionality.

from urllib3.contrib.pyopenssl import inject_into_urllib3
from urllib3 import PoolManager
from OpenSSL import SSL

#cert, key = generate with pyopenssl, or convert from cryptography, generate with certauth or other tools
#cert is a SSL.X509, key is a SSL.PKey

ctx = SSL.Context()
ctx.use_cert(cert)
ctx.use_privatekey(key)

inject_into_urllib3()

with PoolManager(ssl_context=ctx) as pool:
	pool.request(...)

This demonstrates the generation and use of an ssl context which never writes its certs or keys to disk, something that, in all of pythonland, is only possible using pyopenssl, because only pyopenssl uses the cffi bindings to the openssl functionality. important peps were proposed and withdrawn, and to date, we appear to be no closer to loading keys from memory in std python than we were ten years ago. httpx solving for this use case, especially async, would be a big deal, as people usually move to other languages to get this done.