macOS Security Framework fails to import modern PKCS#12 created by OpenSSL 3

[danni-m]

Hi,

Im having the following error when running on macbook (M1) with a k3s cluster that was created by k3d:

 cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.19s
     Running `target/debug/test_kube`
Error: SslError: MAC verification failed during PKCS12 import (wrong password?)

This doesn't happen if I use GKE or token based authentication.

I have a repository that reproduces this on my machine: https://github.com/danni-m/PKCS12_issue.
The kubeconfig file im using is:

---
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkakNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTXpVNU1qa3pNRGd3SGhjTk1qRXhNVEF6TURnME9ESTRXaGNOTXpFeE1UQXhNRGcwT0RJNApXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTXpVNU1qa3pNRGd3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFUblVCS1NsT2FpN1J6UTVEandKQktaZFNFYThSTWsvRXpONEJ4N1pnSkQKNjlFS2xRZXZERkl3Mm1rMVpSNzNFNytxR2VpRVlLLzhadW5Tb0tCNGtwdjdvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVUU0bjlOUzBUODdWaWx2d0hnZ2dSCjU1R1RhWXN3Q2dZSUtvWkl6ajBFQXdJRFJ3QXdSQUlnVlhBRUxBZW5IbVRhVU1GTjViaWZzaE9qYTN3VmRRVm4KTTRXZnhNT0VHWWtDSUdRRE1JT1Vtb2xtd2dEVmZabXo2bGt0Y2lxUHBtRHVxenNrZG9YZ0hiVXUKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://0.0.0.0:58362
  name: k3d-testing
contexts:
- context:
    cluster: k3d-testing
    user: admin@k3d-testing
  name: k3d-testing
current-context: k3d-testing
kind: Config
preferences: {}
users:
- name: admin@k3d-testing
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRlZ0F3SUJBZ0lJT01iMWYvOHk4a293Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOak0xT1RJNU16QTRNQjRYRFRJeE1URXdNekE0TkRneU9Gb1hEVEl5TVRFdwpNekE0TkRneU9Gb3dNREVYTUJVR0ExVUVDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhGVEFUQmdOVkJBTVRESE41CmMzUmxiVHBoWkcxcGJqQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJQbHMxY2ZYdGpQVWIxdUMKZFhCbHVydTBZU3pqR0pGWlEzYXRUTHFDT1FTNlFDQUNMcW5NY29scy82aGhBL2RXY3JJdmFFS1VpWHZIMUs5dApzQkw5OEZxalNEQkdNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBakFmCkJnTlZIU01FR0RBV2dCU0VzRDdNZElVQnExNGJWdFoybjJ0S1pOMnY4REFLQmdncWhrak9QUVFEQWdOSUFEQkYKQWlBRzB2Yjk4dzV4ekVIL2tORTNBOGh1TmMwRG42N08yMS9WUmtzbFloSWN6Z0loQU9SZlVnSVpaWm40WU54egoydGpUMkhHdUlXS0QvVzVRdGp3Uk5pZjFWWEZjCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdFkyeHAKWlc1MExXTmhRREUyTXpVNU1qa3pNRGd3SGhjTk1qRXhNVEF6TURnME9ESTRXaGNOTXpFeE1UQXhNRGcwT0RJNApXakFqTVNFd0h3WURWUVFEREJock0zTXRZMnhwWlc1MExXTmhRREUyTXpVNU1qa3pNRGd3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFUMWVDVlhyeTc4V3RSaS9iekQ4c1pZbkIwOUVTZDJCK0lHdDR6c0tTdXMKbWJERGZOWTdvSjhwWGJTeUpyNWkvdCs5VXBoTWljbGVYcHZFUjQycHhLaEZvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVWhMQSt6SFNGQWF0ZUcxYldkcDlyClNtVGRyL0F3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUloQU94bmIrSTZYWEdFNFJ0RWpnelhacG9nOVZxMWJ0cGgKSklWcTRNbllseWJvQWlCNUQ3anFudjRnZGUrRFJFLzRtQ2Uxdk16d1JUSm9ZbnJTcUx3a2VNeGpodz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUtiU3B6N0NXdFNLZ3FaUHhHWm9tZTZCa1Z6RGxEbkxCRjF4MzFMZEh5dDBvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFK1d6Vng5ZTJNOVJ2VzRKMWNHVzZ1N1JoTE9NWWtWbERkcTFNdW9JNUJMcEFJQUl1cWN4eQppV3ovcUdFRDkxWnlzaTlvUXBTSmU4ZlVyMjJ3RXYzd1dnPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=

[clux]

Because you are using rustls, this looks like a duplicate of #542. I'd suggest trying native-tls for now.

[danni-m]

@clux Its also happens with native-tls

[danni-m]

@clux I pushed the change to my repo to reflect how i use native-tls.

[clux]

Hm. Damn. It runs successfully on linux against my k3d with rustls-tls (EDIT: and native-tls).

[danni-m]

A workaround for this will be to add openssl-sys = "=0.9.66" to your dependencies to avoid the implicit upgrade to openssl3.

[clux]

As noted on discord, this is a regression from kube 0.58.0, but only because 0.63 pulls in the new patch release of openssl-sys that pulls in the openssl 3 from their release on halloween: https://github.com/sfackler/rust-openssl/blob/master/openssl-sys/CHANGELOG.md#v0969---2021-10-31 (apparently openssl major version changes is not consider breaking for them).

It would be good to have some more eyes on this particularly upstream to find the underlying issue. Pinning openssl-sys is probably only liable to work for so long.

It's strange that it appears on rustls variant of that test case, but the test case is also not disabling default features so both dependencies will be present.

[clux]

Purging openssl from the dependency tree:

[dependencies]
log = "0.4.14"
anyhow = "1.0.44"
kube = { version = "0.63.2", default-features=false, features = ["derive", "rustls-tls", "client"]}
tokio = { version = "1.0.1", features = ["rt-multi-thread", "time", "fs", "macros", "net"] }
k8s-openapi = { version = "0.13.1", features = ["v1_20"], default-features=false }

gives the error

Error: SslError: No valid private key was found

from https://github.com/kube-rs/kube-rs/blob/120d0001ce58eb18ac59d2a9da1bf8514bc8058c/kube-client/src/client/tls.rs#L90-L117 - so that's the rustls EC issue again.

So it looks like this issue here is related solely to openssl 3.0 (additionally) breaking it on native-tls (and only on mac).

[clux]

This is a problem because that means that soon there could be no good way to run kube against k3d on mac's since both tls stacks have issues (at least if the pinning of openssl-sys stops being available). If anyone is able to dig in here and look for some workarounds it would be really appreciated.

[kazk]

As noted on discord, this is a regression from kube 0.58.0, but only because 0.63 pulls in the new patch release of openssl-sys that pulls in the openssl 3 from their release on halloween: sfackler/rust-openssl@master/openssl-sys/CHANGELOG.md#v0969---2021-10-31 (apparently openssl major version changes is not consider breaking for them).

• Upgraded the vendored OpenSSL to 3.0.0.

I don't think we're using vendored feature. Actually, enabling vendored feature might be a possible workaround too (avoiding openssl-sys).

A workaround for this will be to add openssl-sys = "=0.9.66" to your dependencies to avoid the implicit upgrade to openssl3.

The openssl-sys crate will automatically detect OpenSSL installations via Homebrew on macOS and vcpkg on Windows. Additionally, it will use pkg-config on Unix-like systems to find the system installation.

openssl - Rust

v0.9.67 added support for LibreSSL 3.4.0.

@danni-m What's the output of openssl version?

[kazk]

I don't think we're using vendored feature.

Never mind, I just found @danni-m's repo that enables it.