Not verifying on Windows

[senpro-ingwersenk]

Hello there!

I am trying to use Narrowlink and it uses the native verifier to allow users to use custom CAs - which in my case, I have to, as our firewall uses that for TLS/SSL traffic inspection.

However, whenever I try to connect to a server, I get the UnknownIssuer message back, which to me sounds like it couldn't verify the certificate against what Windows had stored.

The firewall CA is valid untill 2037 and is self-signed - hence why it needed to be added. cURL and friends can easily use that certificate, but this library can not? I must be missing something.

Any ideas? I tried to look for some way to debug that but had no success...

Kind regards!

[senpro-ingwersenk]

After more digging, I found this: https://learn.microsoft.com/en-us/troubleshoot/windows-server/certificates-and-public-key-infrastructure-pki/valid-root-ca-certificates-untrusted

...and I followed some instructions I found for validating the certificate using certutil - which works, flawlessly so.

Since there are two stores (local user and system), which one does the verifier actually use?

[complexspaces]

Hi there, thanks for the issue report.

AFAIK the platform verifier uses the local user's store. We don't set any flags to configure it otherwise and CurrentUser seems to be the default based on the docs of CERT_CHAIN_ENGINE_CONFIG. Do you know which store your custom CA is currently installed in?

[senpro-ingwersenk]

It is installed in "This Maschine"; so I assume this is the system-wide store (think /etc/ssl/... but the Windows-thing). I will see if it works by purposely importing it into my local profile, if that is even possible with a root CA

[senpro-ingwersenk]

Not exactly my finest idea but I imported it into neigh every "category" in the "This user" selection to see if any of those worked. Unfortunately, it did not. Sadly, due to Windows being Windows, I don't really have any other idea where to put it.

[complexspaces]

Hey again @senpro-ingwersenk, I apologize for my silence. As I had a moment to come by again, I put together a short debugging script based on this blog earlier today. Would you mind running it with a server that failed with UnknownIssuer with the platform verifier? I'm hoping it helps describe where the root Windows "proper" finds is located so I know more of what to look for searching the Win32 documentation. Please feel free to redact any certificate information it prints that is internal/sensitive to your setup.

# Must be run using PowerShell that ships with Windows as such: `powershell .\chain_dump.ps1 -serverName 1password.com`

param ($serverName)

if (!$servername) {
    throw "Provide a server name with -serverName"
}

$MethodDefinition = @'
[StructLayout(LayoutKind.Sequential)]
public struct CERT_CONTEXT
{
    public uint dwCertEncodingType;
    public IntPtr pbCertEncoded;
    public uint cbCertEncoded;
    public IntPtr pCertInfo;
    public IntPtr hCertStore;
}
'@
$WinTypes = Add-Type -MemberDefinition $MethodDefinition -Name 'Win32' -NameSpace '' -PassThru

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
$url = "https://${serverName}:443"

Write-Host "Checking certificates of ${url}"

$WebRequest = [Net.WebRequest]::CreateHttp($url)
$WebRequest.AllowAutoRedirect = $true
$chain = New-Object -TypeName System.Security.Cryptography.X509Certificates.X509Chain
[Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}

#Request website
try {$Response = $WebRequest.GetResponse()}
catch {}

#Creates Certificate
$Certificate = $WebRequest.ServicePoint.Certificate.Handle

$Issuer = $WebRequest.ServicePoint.Certificate.Issuer
$Subject = $WebRequest.ServicePoint.Certificate.Subject

Write-Host
Write-Host "Certificate issuer is '${Issuer}'"
Write-Host "Certificate is issued for: '${Subject}'"

#Build chain
$validChain = $chain.Build($Certificate)

Write-Host
Write-Host "Chain sucessfully built: ${validChain}"
if (!$validChain) {
    Write-Warning "Windows was not able to verify the validility of the URL's certificate. This may impact your ability to use app functionality!"
}
Write-Host "------------------------------------"
Write-Host "Fully built chain is: "
$chain.ChainElements.Certificate

foreach ($cert in $chain.ChainElements) {
    # Is this the root?
    if ($cert.Certificate.Issuer -eq $cert.Certificate.Subject) {
        
        Write-Host
        Write-Host "Root information: "
        $handlePtr = $cert.Certificate.Handle
        $certContext = [System.Runtime.InteropServices.Marshal]::PtrToStructure($handlePtr, [System.Type][Win32+CERT_CONTEXT])

        $certStore = New-Object -TypeName System.Security.Cryptography.X509Certificates.X509Store -ArgumentList $certContext.hCertStore

        $certStore

        Write-Host "Root was stored in $($certStore.Location)"
        Write-Host "Store of root is named '$($certStore.Name)'"
    }
}

Write-Host "------------------------------------"

[senpro-ingwersenk]

Hello!

No problem - I know how it is, things pile up and work just never stops ;) So, no worries.

I ran the script and, as expected, I see the DPI certificate:

> ./main.ps1 -serverName birb.it
WARNUNG: Der generierte Typ definiert keine öffentlichen Methoden oder Eigenschaften.
Checking certificates of https://birb.it:443

Certificate issuer is 'E=support@sophos.com, CN=Sophos SSL CA_CDnUQltAXJc4S83, OU=NSG, O=Sophos, S=Oxfordshire, C=GB'
Certificate is issued for: 'CN=birb.it'

Chain sucessfully built: False
WARNUNG: Windows was not able to verify the validility of the URL's certificate. This may impact your ability to use app functionality!
------------------------------------
Fully built chain is:

Thumbprint                                Subject
----------                                -------
2115D337566467B31044EAFEE83372A399D316CE  CN=birb.it
7431E3718A7B72A9CB82FB0D7C98A1412519CF86  E=support@sophos.com, CN=Sophos SSL CA_CDnUQltAXJc4S83, OU=NSG, O=Sophos, S=Oxfordshire, C=GB

Root information:

StoreHandle  : 2591085465008
Location     : 0
Name         :
Certificates : {[Subject]
                 O=Sophos, CN=Sophos Endpoint RSA Root

               [Issuer]
                 O=Sophos, CN=Sophos Endpoint RSA Root

               [Serial Number]
                 81544A25BD5A3B1C5664C327584C9505

               [Not Before]
                 31.01.2023 19:43:32

               [Not After]
                 31.01.2030 12:27:56

               [Thumbprint]
                 E7C4EF904F68D558901E46C9B703776628FCE97A
               , [Subject]
                 CN=SENST-NB-KEIN.senpro.it

               [Issuer]
                 CN=SENST-NB-KEIN.senpro.it

               [Serial Number]
                 5C44CD25E19FEA9B4E07F6EEBB51BE30

               [Not Before]
                 08.07.2024 06:52:12

               [Not After]
                 09.11.3023 05:52:12

               [Thumbprint]
                 CEE1700D93FC131D405BB548DBCD7DE4DD5F25CF
               , [Subject]
                 CN=Microsoft Root Certificate Authority, DC=microsoft, DC=com

               [Issuer]
                 CN=Microsoft Root Certificate Authority, DC=microsoft, DC=com

               [Serial Number]
                 79AD16A14AA0A5AD4C7358F407132E65

               [Not Before]
                 10.05.2001 01:19:22

               [Not After]
                 10.05.2021 01:28:13

               [Thumbprint]
                 CDD4EEAE6000AC7F40C3802C171E30148030C072
               , [Subject]
                 CN=Thawte Timestamping CA, OU=Thawte Certification, O=Thawte, L=Durbanville, S=Western Cape, C=ZA

               [Issuer]
                 CN=Thawte Timestamping CA, OU=Thawte Certification, O=Thawte, L=Durbanville, S=Western Cape, C=ZA

               [Serial Number]
                 00

               [Not Before]
                 01.01.1997 01:00:00

               [Not After]
                 01.01.2021 00:59:59

               [Thumbprint]
                 BE36A4562FB2EE05DBB3D32323ADF445084ED656
               ...}

Root was stored in 0
Store of root is named ''
------------------------------------

Hope this info helps!

[complexspaces]

Oh that's pretty interesting; the .NET bindings to the Windows certificate verification also failed to find a trusted root CA issuer:

Chain sucessfully built: False
WARNUNG: Windows was not able to verify the validility of the URL's certificate. This may impact your ability to use app functionality!

This is weird to me because you mentioned certutil didn't have any problems with verifying a certificate (do you remember if you were testing the birb.it certificate or the root one?)

Is the Sophos SSL CA_CDnUQltAXJc4S83 certificate the root you expect (I suspect it is but just checking) or is that an intermediate CA?

If you change this line in the script:

$chain = New-Object -TypeName System.Security.Cryptography.X509Certificates.X509Chain

to have this addition:

$chain = New-Object -TypeName System.Security.Cryptography.X509Certificates.X509Chain -ArgumentList $true

is there a behavior difference? This tells it to use the machine context instead of the current user's.

[senpro-ingwersenk]

This is weird to me because you mentioned certutil didn't have any problems with verifying a certificate (do you remember if you were testing the birb.it certificate or the root one?)

I should have verified against birb.it - but it's been a bit ago, so I will try and see if I can rerun that, just to be safe. I also regret not putting the commands and output in my earlier comment - because that has long left my shell history. Oh well...

Is the Sophos SSL CA_CDnUQltAXJc4S83 certificate the root you expect (I suspect it is but just checking) or is that an intermediate CA?

It should be the root CA. The firewall supposedly generates certificates in order to do "deep packet inspection" by effectively MITM'ing TLS/SSL connections. So, to do so efficiently, it has it's own root CA and issues certificates for remotes using that. At the very least, this is as far as my collegues could tell me (and I am quite sure they're right from all I have seen so far).

Here is the output with the modified line (I just commented out the old one)

> .\test-ssl-v2.ps1 -serverName birb.it
Checking certificates of https://birb.it:443

Certificate issuer is 'E=support@sophos.com, CN=Sophos SSL CA_CDnUQltAXJc4S83, OU=NSG, O=Sophos, S=Oxfordshire, C=GB'
Certificate is issued for: 'CN=birb.it'

Chain sucessfully built: False
WARNUNG: Windows was not able to verify the validility of the URL's certificate. This may impact your ability to use app functionality!
------------------------------------
Fully built chain is:

Thumbprint                                Subject
----------                                -------
D7FC811069242A651DF8CB6F4F64165C9B930ECE  CN=birb.it
7431E3718A7B72A9CB82FB0D7C98A1412519CF86  E=support@sophos.com, CN=Sophos SSL CA_CDnUQltAXJc4S83, OU=NSG, O=Sophos, S=Oxfordshire, C=GB

Root information:

StoreHandle  : 2296259200576
Location     : 0
Name         :
Certificates : {[Subject]
                 O=Sophos, CN=Sophos Endpoint RSA Root

               [Issuer]
                 O=Sophos, CN=Sophos Endpoint RSA Root

               [Serial Number]
                 4B2525AAC9A0BEBE76AEF8CA6E9AEC6B

               [Not Before]
                 18.02.2023 19:05:39

               [Not After]
                 18.02.2030 11:50:03

               [Thumbprint]
                 F9BE86B9EE203D4F22BC8231B27C10F110CAA3E9
               , [Subject]
                 CN=SENST-NB-KEIN.senpro.it

               [Issuer]
                 CN=SENST-NB-KEIN.senpro.it

               [Serial Number]
                 5C44CD25E19FEA9B4E07F6EEBB51BE30

               [Not Before]
                 08.07.2024 06:52:12

               [Not After]
                 09.11.3023 05:52:12

               [Thumbprint]
                 CEE1700D93FC131D405BB548DBCD7DE4DD5F25CF
               , [Subject]
                 CN=Microsoft Root Certificate Authority, DC=microsoft, DC=com

               [Issuer]
                 CN=Microsoft Root Certificate Authority, DC=microsoft, DC=com

               [Serial Number]
                 79AD16A14AA0A5AD4C7358F407132E65

               [Not Before]
                 10.05.2001 01:19:22

               [Not After]
                 10.05.2021 01:28:13

               [Thumbprint]
                 CDD4EEAE6000AC7F40C3802C171E30148030C072
               , [Subject]
                 CN=Thawte Timestamping CA, OU=Thawte Certification, O=Thawte, L=Durbanville, S=Western Cape, C=ZA

               [Issuer]
                 CN=Thawte Timestamping CA, OU=Thawte Certification, O=Thawte, L=Durbanville, S=Western Cape, C=ZA

               [Serial Number]
                 00

               [Not Before]
                 01.01.1997 01:00:00

               [Not After]
                 01.01.2021 00:59:59

               [Thumbprint]
                 BE36A4562FB2EE05DBB3D32323ADF445084ED656
               ...}

Root was stored in 0
Store of root is named ''
------------------------------------

I did notice two oddities: The Microsoft certificate at the bottom (CN=Microsoft Root Certificate Authority, DC=microsoft, DC=com) and the one having my laptop's hostname (CN=SENST-NB-KEIN.senpro.it) have odd dates. The former is simply too old and the latter has it's dates seemingly backwards?

I also took a look at my trust store settings:

So that's clearly where it should be. ...right? It's a trusted root certificate and has the proper dates set.

I'll dig a little further and try to be verbose; perhaps I find something this time around.

[senpro-ingwersenk]

Well, well, well. ChatGPT must've had a good day, it suggested a flag for cURL: curl.exe -v --ssl-no-revoke https://birb.it

Looks like one of my certificates is in fact revoked, breaking the whole chain. Interesting.

[senpro-ingwersenk]

I found it!

It turns out that the certificate lacks URLs for revocation and locking mechanisms (OCSP, AIA) - and thus, standard Windows schannel API will not validate the certificate. For example:

> curl.exe -v --cacert SecurityAppliance_SSL_CA.crt https://birb.it
* Host birb.it:443 was resolved.
* IPv6: (none)
* IPv4: 49.13.8.103
*   Trying 49.13.8.103:443...
* Connected to birb.it (49.13.8.103) port 443
* schannel: disabled automatic use of client certificate
* ALPN: curl offers http/1.1
* schannel: added 1 certificate(s) from CA file 'C:\Users\ingwersenk\Downloads\SecurityAppliance_SSL_CA.crt'
* schannel: CertGetCertificateChain trust error CERT_TRUST_REVOCATION_STATUS_UNKNOWN
* closing connection #0
curl: (60) schannel: CertGetCertificateChain trust error CERT_TRUST_REVOCATION_STATUS_UNKNOWN
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.

That CERT_TRUST_REVOCATION_STATUS_UNKNOWN is the root cause. Even self-signed certificates must provide those mechanisms to schannel - and this one does not, and likely never will. There is a registry hack to disable this revocation check - but I am kinda unsure if I should, or shouldn't enable it (given that this is what TLS/SSL is supposed to do).

So for now, I guess I can't use anything that very directly verifies against the schannel store...? Honestly I am taking quite a crash-course over here; apologies for the little ramblings. :)

[complexspaces]

Thank you for taking the time to dig through this! These findings and descriptions are great, so no worries about the rambling.

What you're describing is actually something we've had to think about before here, but on Android specifically. See #69. I didn't know that Windows cared about the same things TBH, and macOS was reported to not mind as it presumably checks if something was issued by a public CA or not by itself so I didn't think too hard about the other desktop OS.

Given that the cause has been mostly identified, I think we can start looking at solutions to the problem. One thing I want to make sure I'm clear on though is the error today with the same certificate chain. Is the following correct?:

• rustls-platform-verifier gives you UnknownIssuer which is this line of error handling
• curl via schannel gives you CERT_TRUST_REVOCATION_STATUS_UNKNOWN

If so, rustls-platform-verifier might also have a secondary issue with how this error is reported since I would have hoped it comes back as CertificateError::Revoked. The last time I'm aware of this changing is #17. CERT_TRUST_REVOCATION_STATUS_UNKNOWN was ignored before that PR in our code but I suspect the same issue would happen anyway.

Unlike Android however determining if something is a public root might be messier or harder on Windows because the platform doesn't have as clean of a line between OS-included roots and custom ones.

[complexspaces]

Hey again! I tried to reproduce this using mkcert but I haven't been able to get the same behavior yet. I issued a certificate for foobar.internal.domain using it and served a localhost HTTPS server with it. I also made sure to move the mkcert root from my user certificate store to the machine one to match your environment. This test code passes however and I see the HTML from the fake server:

use reqwest::ClientBuilder;
use crate::ConfigVerifierExt;

#[tokio::test]
async fn try_connect() {
    let client = ClientBuilder::new()
        .use_preconfigured_tls(rustls::ClientConfig::with_platform_verifier())
        .build()
        .expect("nothing should fail");
    
    let response = client.get("https://foobar.internal.domain").send().await.unwrap();
    println!("{}", response.text().await.unwrap());
}

If I try running curl on that it fails with a revocation error but its a different one then you have so I think there's a difference between how the certificates are being issued here:

curl -v https://foobar.internal.domain
* Host foobar.internal.domain:443 was resolved.
* IPv6: (none)
* IPv4: 127.0.0.1
*   Trying 127.0.0.1:443...
* Connected to foobar.internal.domain (127.0.0.1) port 443
* schannel: disabled automatic use of client certificate
* ALPN: curl offers http/1.1
* schannel: next InitializeSecurityContext failed: CRYPT_E_NO_REVOCATION_CHECK (0x80092012) - The revocation function was unable to check revocation for the certificate.
* closing connection #0
curl: (35) schannel: next InitializeSecurityContext failed: CRYPT_E_NO_REVOCATION_CHECK (0x80092012) - The revocation function was unable to check revocation for the certificate.

[complexspaces]

Do the certificate fields/properties on your chain in a browser look significantly different then these ones from mkcert?

Website cert:

mkcert CA root: