Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions ai-docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ Request IDs work as follows. The gateway contracts assign each ID and bind it to

The meta store is in-memory only, so a reboot of the core forgets every known request and session ID. The KMS connector keeps the state of each request in its [persistent database](https://github.com/zama-ai/fhevm/tree/main/kms-connector/connector-db); its [kms-worker](https://github.com/zama-ai/fhevm/tree/main/kms-connector/crates/kms-worker) marks a request as sent and only polls for the result on a retry, so a request is not run more often than necessary across core reboots.

P2P TLS requires TLS 1.3 with `X25519MLKEM768` or `secp256r1MLKEM768` key exchange, with X25519 first. The server and client use an explicit AWS-LC provider restricted to these hybrid groups. `threshold_networking::tls::build_p2p_tls_config` constructs both configurations, and `kms-server` supplies the attested verifier and certificate resolver. This policy applies to manual and automatic certificates. Classical-only and pure ML-KEM peers cannot connect. Certificate authentication uses classical signatures. Every peer must support at least one permitted hybrid group before deployment.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I could read from the NIST documents MLKEM768 provides level 3, equivalent to 192 bit AES, whereas secp256r1 (and by extension) x25519 provides around 128 bits security. So I guess we either want to use P384 or MLKEM512 to have the levels consistent.
However, we already used MLKEM1024 with P384, so we have had a tendency of selecting PQ parameters higher than their classical counterparts.


Consequences for agents:

- The threat model assumes that at most `t` of the `n` parties are malicious. An attack that needs more than `t` malicious parties is out of scope. Every attack that works with at most `t` malicious parties is in scope: bypassing TLS, attestation or sender binding on the core-to-core interface, or breaking the confidentiality of the key material or the correctness of a result.
Expand Down
15 changes: 4 additions & 11 deletions core/service/src/bin/kms-server.rs
Original file line number Diff line number Diff line change
Expand Up @@ -41,15 +41,14 @@ use kms_lib::{
use observability::health::register_process_health;
use std::{net::ToSocketAddrs, num::NonZero, sync::Arc, thread};
use thread_handles::init_rayon_thread_pool;
use threshold_networking::tls::AttestedVerifier;
use threshold_networking::tls::{AttestedVerifier, build_p2p_tls_config};
use tokio::net::TcpListener;
use tokio_rustls::rustls::{
client::{ClientConfig, danger::DangerousClientConfigBuilder},
client::ClientConfig,
crypto::{CryptoProvider, aws_lc_rs::default_provider as aws_lc_rs_default_provider},
pki_types::{CertificateDer, PrivateKeyDer},
server::ServerConfig,
sign::{CertifiedKey, SingleCertAndKey},
version::TLS13,
};

#[derive(Parser)]
Expand Down Expand Up @@ -132,6 +131,7 @@ async fn make_mpc_listener(threshold_config: &ThresholdPartyConf) -> TcpListener
/// instead of using the wrapper from tonic::transport because we need to
/// provide our own certificate verifier that can validate bundled attestation
/// documents and that can receive new trust roots on the context change.
/// Uses [`build_p2p_tls_config`] for the P2P transport policy.
async fn build_tls_config(
peers: &Option<Vec<PeerConf>>,
tls_config: &TlsConf,
Expand Down Expand Up @@ -292,14 +292,7 @@ async fn build_tls_config(
// We do not need to add context to verifier here
// because it'll be added using [ensure_default_threshold_context_in_storage].

let server_config = ServerConfig::builder_with_protocol_versions(&[&TLS13])
.with_client_cert_verifier(verifier.clone())
.with_cert_resolver(cert_resolver.clone());
let client_config = DangerousClientConfigBuilder {
cfg: ClientConfig::builder_with_protocol_versions(&[&TLS13]),
}
.with_custom_certificate_verifier(verifier.clone())
.with_client_cert_resolver(cert_resolver.clone());
let (server_config, client_config) = build_p2p_tls_config(verifier.clone(), cert_resolver)?;
Ok((server_config, client_config, verifier))
}

Expand Down
215 changes: 211 additions & 4 deletions core/threshold-networking/src/tls.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,22 @@ use tfhe_versionable::{Versionize, VersionsDispatch};
use tokio_rustls::rustls::{
DigitallySignedStruct, DistinguishedName, Error, RootCertStore, SignatureScheme,
client::{
WebPkiServerVerifier,
danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier},
ClientConfig, ResolvesClientCert, WebPkiServerVerifier,
danger::{
DangerousClientConfigBuilder, HandshakeSignatureValid, ServerCertVerified,
ServerCertVerifier,
},
},
crypto::{
CryptoProvider, WebPkiSupportedAlgorithms,
aws_lc_rs::{default_provider, kx_group},
},
crypto::{CryptoProvider, WebPkiSupportedAlgorithms},
pki_types::{CertificateDer, ServerName, UnixTime},
server::{
WebPkiClientVerifier,
ResolvesServerCert, ServerConfig, WebPkiClientVerifier,
danger::{ClientCertVerified, ClientCertVerifier},
},
version::TLS13,
};
use x509_parser::{certificate::X509Certificate, parse_x509_certificate, pem::Pem};

Expand Down Expand Up @@ -154,6 +161,32 @@ impl std::fmt::Debug for AttestedVerifier {
}
}

/// Constructs mutually authenticated P2P TLS configurations with hybrid-only key exchange.
///
/// Both endpoints require TLS 1.3 and prefer [`kx_group::X25519MLKEM768`] over [`kx_group::SECP256R1MLKEM768`].

@titouantanguy titouantanguy Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason why we support both ? and in that order ?
I reckon compliance frameworks might enforce SECP256R1MLKEM768 as that's what NIST suggests ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason was that rustls supports both, and whatever is the default came out first. Will pin the NIST recommendation, unless someone objects.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

x25519 is generally consider a safer choice than NIST, in particular when it comes to NIST's *R1 curves as there has been a bit of fear of weaknesses in the randomness.
So I think it depends on whether we want to stick to NIST recommendations or we rather trust the DJ Bernstein and co gang.
I do not have a strong opinion but for a personal project I would pick X25519. But I think we at some point agreed to go with NIST recommendations whenever possible?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, under further investigation X25519 is not FIPS approved. So we should probably go with secp256r1 as the default

/// Returns an error if the provider cannot support TLS 1.3.
pub fn build_p2p_tls_config<R>(
verifier: Arc<AttestedVerifier>,
cert_resolver: Arc<R>,
) -> Result<(ServerConfig, ClientConfig), Error>
where
R: ResolvesServerCert + ResolvesClientCert + 'static,
{
let mut provider = default_provider();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR talks about handshake only, but I'm wondering do we want to force AES 256 as well ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, will restrict session ciphers too.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this would be greater than the classical signature (128 bits) and PQ (192 bits). Still good but might be a bit overkill with current paramters

provider.kx_groups = vec![kx_group::X25519MLKEM768, kx_group::SECP256R1MLKEM768];
let provider = Arc::new(provider);
let server_config = ServerConfig::builder_with_provider(provider.clone())
.with_protocol_versions(&[&TLS13])?
.with_client_cert_verifier(verifier.clone())
.with_cert_resolver(cert_resolver.clone());
let client_config = DangerousClientConfigBuilder {
cfg: ClientConfig::builder_with_provider(provider).with_protocol_versions(&[&TLS13])?,
}
.with_custom_certificate_verifier(verifier)
.with_client_cert_resolver(cert_resolver);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude pointed out that session resuming skips the attested verifier, and hence leaves a potential attack vector. It should be easy to solve with client.resumption = Resumption::disabled(), server.session_storage = Arc::new(NoServerSessionStorage {}) and server.send_tls13_tickets = 0.

Ok((server_config, client_config))
}

impl AttestedVerifier {
pub fn new(
user_data_verifier: Option<Arc<UserDataVerifier>>,
Expand Down Expand Up @@ -886,6 +919,180 @@ pub async fn generate_mock_tls_cert_with_attestation(
}
}

#[cfg(test)]
mod p2p_tests {
use super::*;
use rcgen::{
CertificateParams, DnType, ExtendedKeyUsagePurpose, KeyPair, PKCS_ECDSA_P256_SHA256,
};
use std::{collections::HashMap, time::Duration};
use threshold_types::{party::MpcIdentity, session_id::SessionId};
use tokio_rustls::{
TlsAcceptor, TlsConnector,
rustls::{
NamedGroup, PeerIncompatible,
pki_types::PrivateKeyDer,
sign::{CertifiedKey, SingleCertAndKey},
},
};

async fn handshake(
server: ServerConfig,
client: ClientConfig,
) -> (
std::io::Result<tokio_rustls::server::TlsStream<tokio::io::DuplexStream>>,
std::io::Result<tokio_rustls::client::TlsStream<tokio::io::DuplexStream>>,
) {
let (server_io, client_io) = tokio::io::duplex(16384);
let acceptor = TlsAcceptor::from(Arc::new(server));
let connector = TlsConnector::from(Arc::new(client));
tokio::time::timeout(Duration::from_secs(5), async {
tokio::join!(
acceptor.accept(server_io),
connector.connect("p2p-test".try_into().unwrap(), client_io),
)
})
.await
.expect("TLS handshake timed out")
}

#[tokio::test]
async fn p2p_tls_requires_hybrid_key_exchange() {
let _ = default_provider().install_default();
let key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256).unwrap();
let mut params = CertificateParams::new(vec!["p2p-test".to_string()]).unwrap();
params
.distinguished_name
.push(DnType::CommonName, "p2p-test");
params.extended_key_usages = vec![
ExtendedKeyUsagePurpose::ServerAuth,
ExtendedKeyUsagePurpose::ClientAuth,
];
let cert = params.self_signed(&key).unwrap();
let cert_resolver = Arc::new(SingleCertAndKey::from(
CertifiedKey::from_der(
vec![cert.der().clone()],
PrivateKeyDer::try_from(key.serialize_der()).unwrap(),
&default_provider(),
)
.unwrap(),
));
let verifier = Arc::new(
AttestedVerifier::new(
None,
false,
#[cfg(feature = "insecure")]
false,
)
.unwrap(),
);
let (server, client) = build_p2p_tls_config(verifier.clone(), cert_resolver).unwrap();
verifier
.add_context(
SessionId::new(&"hybrid-only").unwrap(),
HashMap::from([(
MpcIdentity("p2p-test".to_string()),
x509_parser::pem::parse_x509_pem(cert.pem().as_bytes())
.unwrap()
.1,
)]),
None,
)
.unwrap();

for config_groups in [
&server.crypto_provider().kx_groups,
&client.crypto_provider().kx_groups,
] {
assert_eq!(
config_groups
.iter()
.map(|group| group.name())
.collect::<Vec<_>>(),
vec![NamedGroup::X25519MLKEM768, NamedGroup::secp256r1MLKEM768],
);
}

for group in [
kx_group::X25519MLKEM768,
kx_group::SECP256R1MLKEM768,
kx_group::X25519,
kx_group::SECP256R1,
kx_group::SECP384R1,
kx_group::MLKEM768,
kx_group::MLKEM1024,
] {
let mut provider = default_provider();
provider.kx_groups = vec![group];
let provider = Arc::new(provider);
let peer_server = ServerConfig::builder_with_provider(provider.clone())
.with_protocol_versions(&[&TLS13])
.unwrap()
.with_client_cert_verifier(verifier.clone())
.with_cert_resolver(server.cert_resolver.clone());
let peer_client = DangerousClientConfigBuilder {
cfg: ClientConfig::builder_with_provider(provider)
.with_protocol_versions(&[&TLS13])
.unwrap(),
}
.with_custom_certificate_verifier(verifier.clone())
.with_client_cert_resolver(client.client_auth_cert_resolver.clone());

for (server_result, client_result) in [
handshake(server.clone(), peer_client).await,
handshake(peer_server, client.clone()).await,
] {
if matches!(
group.name(),
NamedGroup::X25519MLKEM768 | NamedGroup::secp256r1MLKEM768
) {
let server_stream = server_result.unwrap();
let client_stream = client_result.unwrap();
assert_eq!(
server_stream.get_ref().1.protocol_version(),
Some(tokio_rustls::rustls::ProtocolVersion::TLSv1_3)
);
assert_eq!(
server_stream
.get_ref()
.1
.negotiated_key_exchange_group()
.unwrap()
.name(),
group.name()
);
assert_eq!(
client_stream
.get_ref()
.1
.negotiated_key_exchange_group()
.unwrap()
.name(),
group.name()
);
assert!(server_stream.get_ref().1.peer_certificates().is_some());
assert!(client_stream.get_ref().1.peer_certificates().is_some());
} else {
let error = server_result.unwrap_err();
assert!(
matches!(
error
.get_ref()
.and_then(|error| error.downcast_ref::<Error>()),
Some(Error::PeerIncompatible(
PeerIncompatible::NoKxGroupsInCommon
))
),
"unexpected rejection for {:?}: {error}",
group.name(),
);
assert!(client_result.is_err(), "client accepted {:?}", group.name());
Comment on lines +1076 to +1089

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, why an assert on the client's error but an unwrap on the server's one ?

}
}
}
}
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude pointed out that it would be a good idea to also add a test to validate the HelloRetryRequest path, to ensure that the next release can work gracefully with the 0.15 during a rolling upgrade. In theory it should work without issue, but would be good to have a test


#[cfg(all(test, feature = "insecure"))]
mod tests {
use super::*;
Expand Down
2 changes: 2 additions & 0 deletions docs/explanations/network_doc.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ The networking service uses a session-based model. For each MPC computation (i.e

### TLS Configuration and Requirements

P2P connections require TLS 1.3 and hybrid key exchange. The permitted groups are `X25519MLKEM768` and `secp256r1MLKEM768`, with X25519 first. Both client and server reject classical-only and pure ML-KEM exchanges. Manual and automatic certificate configurations use the same policy. Every peer must support at least one permitted group before deployment. Certificate signatures remain classical.

#### Identity and Authentication

Each MPC node is identified by its **Common Name (CN)** within its certificate. This identity is validated against the certificate's **Subject Alternative Name (SAN)** list.
Expand Down
1 change: 1 addition & 0 deletions docs/explanations/trust_model.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ The core-to-core interface is reachable by other KMS cores, which other operator

- **Mutual TLS.** Every party holds its own TLS certificate. The peer list and every MPC context carry the certificates of all parties, and these certificates form the trust store of a node. A connection that presents a certificate outside this set fails the TLS handshake. The node accepts messages only from the allowlisted set of peers.
- **Sender binding.** Every MPC message names its sender. The receiver compares this name with the Common Name of the peer certificate and rejects a mismatch with `Unauthenticated`. A peer cannot impersonate another peer.
- **Hybrid key exchange.** P2P TLS requires TLS 1.3 with `X25519MLKEM768` or `secp256r1MLKEM768`. Nodes prefer `X25519MLKEM768` and reject classical-only and pure ML-KEM exchanges. Both manual and automatic certificate configurations use this policy. Certificate authentication uses classical signatures. Every peer must support at least one permitted hybrid group before deployment.
- **Release attestation.** In an AWS Nitro Enclave deployment (`tls.auto`), the TLS certificate embeds an attestation document that the Nitro Security Module signs. The document carries the Platform Configuration Registers (PCRs) of the running enclave: SHA-384 measurements that the Nitro hypervisor computes when it boots the enclave image. The custom verifier checks three of them against the `trusted_releases` list, so a node only talks to a peer that runs an allowlisted release of the software. A context whose PCR allowlist is empty is rejected in an enclave deployment. The three measured values are:
- **PCR0** is the hash of the complete enclave image file (EIF). It changes whenever any part of the image changes, so it identifies one exact release build.
- **PCR1** is the hash of the Linux kernel and the bootstrap ramdisk inside the image. It identifies the enclave runtime and stays the same across releases that only change the application.
Expand Down
Loading