One-way DTLS Handshake Succeeds Even Without -R trust_casfile. Expected behaviour?

[ShvaykaD]

Environment

• Build System: Make
• Operating System: Linux
• Operating System Version: Ubuntu 22.04
• Hosted Environment: None

libcoap Configuration Summary:

coap-client-openssl -version
coap-client-openssl v4.3.5 -- a small CoAP implementation
Copyright (C) 2010-2024 Olaf Bergmann <[email protected]> and others

Build: v4.3.5
TLS Library: OpenSSL - runtime 3.0.0b-dev, libcoap built for 3.0.0b-dev
(DTLS and TLS support; PSK, PKI, PKCS11, no RPK and no CID support)
(Have OSCORE)
(Have WebSockets)

Problem Description

According to the coap-client-openssl man page:

From this description, I would expect that if -R is not provided, the DTLS handshake should fail because the server's certificate cannot be verified.

Actual Behavior

When running the following command without -R:

coap-client-openssl -v 6 -V 7 -m POST coaps://localhost:5684/api/v1/g8WD4xiQWPclKezkoVph/telemetry -t json -e "{temperature:25}"

I get the following output:

Feb 21 16:25:03.667 INFO    127.0.0.1:35046 <-> 127.0.0.1:5684 DTLS: unable to get local issuer certificate: overridden: 'localhost' depth=0
Feb 21 16:25:03.667 INFO    127.0.0.1:35046 <-> 127.0.0.1:5684 DTLS: unable to get certificate CRL: overridden: 'localhost' depth=0
Feb 21 16:25:03.667 INFO    127.0.0.1:35046 <-> 127.0.0.1:5684 DTLS: unable to verify the first certificate: overridden: 'localhost' depth=0

Despite these warnings, the handshake continues successfully:

Feb 21 16:25:03.669 Info *  127.0.0.1:35046 <-> 127.0.0.1:5684 DTLS: Using cipher: ECDHE-ECDSA-AES256-GCM-SHA384

Questions

• Is this behavior expected?
• Why does the handshake succeed despite failing to verify the server certificate?
• Am I missing any configuration that explicitly enables strict certificate validation?

[mrdeep1]

Is this behavior expected?

Yes. As you are not defining any certificates, the OpenSSL client logic generates its own certificates.

Why does the handshake succeed despite failing to verify the server certificate?

As no certificates are defined, it has no way of validating the server certificate as no root servers are defined, so the logic allows this to continue.

Am I missing any configuration that explicitly enables strict certificate validation?

You need to provide the client with suitable PKI certificates that both the client and server can work with. It is possible at the (liboap using) server end to continue working with client certificates it does not like (including the internally generated client certificates), but that requires server certificates to be provided and the use of the -n option.

[ShvaykaD]

Hi @mrdeep1,

Thanks for the update. As I understand it, this behavior is due to how libcoap interacts with OpenSSL in this scenario.

I would like to clarify:

1. Which TLS/DTLS library used with libcoap provides strict certificate validation by default?
   Specifically, if the -R option is not set (i.e., no trusted CA is provided), which of the supported TLS backends will ensure that the DTLS handshake fails due to an unverified server certificate?

2. If a server certificate is issued by a widely trusted Certificate Authority (e.g., Let's Encrypt), does the CoAP client still need the trusted CA explicitly provided via -R?
   Or would the system's default trust store be used to validate the certificate automatically?

3. The -n flag explicitly disables remote peer certificate checking, which I do not want to use. I want to ensure that the server certificate is always validated. Based on the man page description, I expected that if neither -R nor -c is provided, the DTLS handshake would fail. However, from your explanation, it seems that OpenSSL may allow the handshake to proceed even without a trusted anchor. This behavior seems counterintuitive to me, as it raises the question: What is the purpose of such a DTLS handshake if the server’s identity is not verified?

[mrdeep1]

Which TLS/DTLS library used with libcoap provides strict certificate validation by default?

None of them do for the client. The -R option to define the trusted CAs is required.

If a server certificate is issued by a widely trusted Certificate Authority (e.g., Let's Encrypt), does
the CoAP client still need the trusted CA explicitly provided via -R?

Yes, because the system's default trust store is not loaded (as provided by ca-certificates package for example on Ubuntu)

Based on the man page description, I expected that if neither -R nor -c is provided, the DTLS handshake would fail.

I will get the man page updated to be clearer.

Back in time, someone requested that they wanted the client to be able to connect to a PKI enabled server, but without any PKI certificates provided. The question about the client validating the server's certificate did not come up then, but using the -R option gave the ability to force the client to be checking the server certificate against the provided list of trusted CAs.

I agree that nowadays trust is important. Not every CoAP environment will want the default trust store loaded - especially closed systems with little RAM. I need to think how best to implement loading trust store without breaking any API.

[ShvaykaD]

@mrdeep1, thanks for all of these clarifications. This explanation helps a lot in understanding the current behavior of libcoap with OpenSSL and the reasoning behind it.

Would you like to keep this issue open to track the potential implementation of loading the trust store without breaking any API, or should it be closed for now?

Looking forward to your thoughts.

[mrdeep1]

I would leave this open for now.

As a comment, SSL_CTX_set_default_verify_store() and SSL_CTX_load_verify_store() were only added in OpenSSL 3.0.0, so not sure about what happened with the trust store before then (i.e. with OpenSSL 1.1.1) - trust store was probably always loaded.