Make changes to integrate with MbedTLSv3.5.1 and remove OTA cmake tar…

…gets
aws · Dec 12, 2024 · 719da5f · 719da5f
1 parent 86cf6e0
commit 719da5f
Show file tree

Hide file tree

Showing 26 changed files with 55 additions and 951 deletions.
diff --git a/.github/workflows/update-submodules.yml b/.github/workflows/update-submodules.yml
@@ -8,7 +8,7 @@ on:
         required: true
         default: main
       allowed:
-        description: 'Optional regex pattern passed to `grep` to update only the specified library submodules, e.g. "ota\|jobs" updates only libraries with "ota" or "jobs" in the name.'
+        description: 'Optional regex pattern passed to `grep` to update only the specified library submodules, e.g. "jobs" updates only libraries with "jobs" in the name.'
         required: false
         default: .*
 

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,8 +4,10 @@
 
 ### Major Changes
 
-- [#1929](https://github.com/aws/aws-iot-device-sdk-embedded-C/pull/1929) This release includes 202406.01-LTS versions of coreMQTT, corePKCS11, coreHTTP, coreJSON, backoffAlgorithm, AWS IoT Device Shadow, AWS IoT Jobs, AWS IoT Device Defender, AWS IoT Fleet Provisioning and SigV4 libraries. These libraries have gone through code quality checks including verification that no function has a [GNU Complexity](https://www.gnu.org/software/complexity/manual/complexity.html) score greater than 8, checks against deviations from the mandatory rules in the [MISRA coding standard](https://www.misra.org.uk/), static code analysis from [Coverity static analysis](https://scan.coverity.com/) and validation of memory safety through the [CBMC automated reasoning tool](http://www.cs.cmu.edu/~modelcheck/cbmc/).
-- [#1929](https://github.com/aws/aws-iot-device-sdk-embedded-C/pull/1929)  Removes the OTA library and OTA demo . They are planned to be replaced by [aws-iot-core-mqtt-file-streams-embedded-c](https://github.com/aws/aws-iot-core-mqtt-file-streams-embedded-c) in future releases.
+- [#1929](https://github.com/aws/aws-iot-device-sdk-embedded-C/pull/1929) This release brings the following changes:
+  - Includes [202406.01-LTS](https://github.com/FreeRTOS/FreeRTOS-LTS/releases/tag/202406.01-LTS) versions of coreMQTT, corePKCS11, coreHTTP, coreJSON, backoffAlgorithm, AWS IoT Device Shadow, AWS IoT Jobs, AWS IoT Device Defender, AWS IoT Fleet Provisioning and SigV4 libraries.
+  - Removes the OTA library and OTA demo . They are planned to be replaced by [aws-iot-core-mqtt-file-streams-embedded-c](https://github.com/aws/aws-iot-core-mqtt-file-streams-embedded-c) in future releases following [this announcement](https://aws.amazon.com/about-aws/whats-new/2023/12/freertos-modular-composable-ota-libraries/). To learn more about FreeRTOS OTA libraries, visit the [FreeRTOS OTA page](https://freertos.org/freertos-core/over-the-air-updates/index.html). To get started, see the new OTA [reference demo](https://freertos.org/freertos-core/over-the-air-updates/mqtt-ota-agent-orchestrator.html). 
+  - Update readme instructions for demos for better user experience.
 
 ### Minor Changes
 
@@ -16,7 +18,6 @@
 - [#1875](https://github.com/aws/aws-iot-device-sdk-embedded-C/pull/1875) Add a Fleet provisioning demo to get certificate and private key via CreateKeysAndCertificate API.
 - [#1899](https://github.com/aws/aws-iot-device-sdk-embedded-C/pull/1899) Add Tunnelmole as an open source tunneling option in addition to ngrok.
 - [#1901](https://github.com/aws/aws-iot-device-sdk-embedded-C/pull/1901) Add a HTTP demo to generate a pre-signed URL to an S3 object file.
-- Update readme instructions for demos for better user experience.
 
 ## 202211.00 (November 2022)
 

diff --git a/MISRA.md b/MISRA.md
@@ -36,4 +36,4 @@ Deviations from the MISRA standard are listed below:
 | Rule 11.8 | Required | An OpenSSL API `SSL_set_tlsext_host_name`, which is used in the TLS transport implementation, internally casts a string literal to a `void *` pointer. |
 | Rule 13.4 | Required | A POSIX-specific macro utility `FD_SET` is flagged for this violation. This macro utility, whose implementation is supplied by the system, is used in the transport implementation. |
 | Rule 14.4 | Required | A POSIX-specific macro utility `FD_ZERO` is flagged for this violation. This macro utility, whose implementation is supplied by the system, is used in the transport implementation. |
-| Rule 21.6 | Required | The Standard Library input/output functions for opening and closing files are used by the OpenSSL transport implementation, since the OpenSSL API `PEM_read_X509` to read PEM files takes `FILE *` as an argument. The standard C library file handling functions are also used in POSIX platform implementation of OTA. |
+| Rule 21.6 | Required | The Standard Library input/output functions for opening and closing files are used by the OpenSSL transport implementation, since the OpenSSL API `PEM_read_X509` to read PEM files takes `FILE *` as an argument. |
diff --git a/README.md b/README.md
@@ -488,13 +488,7 @@ The following creates a job that specifies a Linux Kernel link for downloading.
         --targets arn:aws:iot:us-west-2:<account-id>:thing/<thing-name> \
         --document '{"url":"https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.8.5.tar.xz"}'
 ```
-3. Execute the following command:
-```
-/build/bin/jobs_demo_mosquitto
-```
-This command will provide detailed instructions for running the demo.
-
-4. When prompted, run the demo using this command:
+3. Run the demo using this command:
 ```    
 ./build/bin/jobs_demo_mosquitto \
 -n <thing-name> \

diff --git a/demos/CMakeLists.txt b/demos/CMakeLists.txt
@@ -26,8 +26,6 @@ check_symbol_exists(fork "unistd.h" HAVE_FORK)
 if(${LIB_RT} STREQUAL "LIB_RT-NOTFOUND")
     set(librt_demos
             "http_demo_s3_download_multithreaded"
-            "ota_demo_core_http"
-            "ota_demo_core_mqtt"
     )
     message( WARNING "rt library could not be found. Demos that use it will be excluded from the default target." )
     foreach(demo_name ${librt_demos})
@@ -46,8 +44,6 @@ if(NOT ${OpenSSL_FOUND})
             "mqtt_demo_basic_tls"
             "mqtt_demo_mutual_auth"
             "mqtt_demo_subscription_manager"
-            "ota_demo_core_http"
-            "ota_demo_core_mqtt"
             "shadow_demo_main"
             "greengrass_demo_local_auth"
     )
@@ -56,16 +52,6 @@ if(NOT ${OpenSSL_FOUND})
         set_target_properties(${demo_name} PROPERTIES EXCLUDE_FROM_ALL true)
     endforeach()
 endif()
-if(NOT ${Threads_FOUND})
-    set(thread_demos
-            "ota_demo_core_http"
-            "ota_demo_core_mqtt"
-    )
-    message( WARNING "Threads library could not be found. Demos that use it will be excluded from the default target." )
-    foreach(demo_name ${thread_demos})
-        set_target_properties(${demo_name} PROPERTIES EXCLUDE_FROM_ALL true)
-    endforeach()
-endif()
 if(NOT HAVE_FORK)
     set(fork_demos
             "http_demo_s3_download_multithreaded"

diff --git a/demos/fleet_provisioning/fleet_provisioning_keys_cert/pkcs11_operations.c b/demos/fleet_provisioning/fleet_provisioning_keys_cert/pkcs11_operations.c
@@ -44,11 +44,11 @@
 /* MbedTLS include. */
 #include "mbedtls/ctr_drbg.h"
 #include "mbedtls/entropy.h"
-#include "mbedtls/entropy_poll.h"
+#include "entropy_poll.h"
 #include "mbedtls/error.h"
 #include "mbedtls/oid.h"
 #include "mbedtls/pk.h"
-#include "mbedtls/pk_internal.h"
+#include "pk_wrap.h"
 #include "mbedtls/sha256.h"
 #include "mbedtls/x509_crt.h"
 #include "mbedtls/x509_csr.h"
@@ -537,10 +537,15 @@ static CK_RV provisionPrivateKey( CK_SESSION_HANDLE session,
     mbedtls_pk_type_t mbedKeyType = MBEDTLS_PK_NONE;
     int mbedResult = 0;
     mbedtls_pk_context mbedPkContext = { 0 };
+    mbedtls_ctr_drbg_context ctr_drbg;
+    mbedtls_entropy_context entropy;
 
     mbedtls_pk_init( &mbedPkContext );
+    mbedtls_entropy_init(&entropy);
+    mbedtls_ctr_drbg_init(&ctr_drbg);
+    mbedtls_ctr_drbg_seed(&ctr_drbg, mbedtls_entropy_func, &entropy, NULL, 0);
     mbedResult = mbedtls_pk_parse_key( &mbedPkContext, ( const uint8_t * ) privateKey,
-                                       privateKeyLength, NULL, 0 );
+                                       privateKeyLength, NULL, 0, mbedtls_ctr_drbg_random, &ctr_drbg );
 
     if( mbedResult != 0 )
     {

diff --git a/demos/fleet_provisioning/fleet_provisioning_keys_cert/pkcs11_operations.h b/demos/fleet_provisioning/fleet_provisioning_keys_cert/pkcs11_operations.h
@@ -27,6 +27,11 @@
 #include <stdlib.h>
 #include <stdbool.h>
 
+/**
+ * @brief To access the private members of the MbedTLS structs
+ */
+ #define MBEDTLS_ALLOW_PRIVATE_ACCESS
+
 /* corePKCS11 include. */
 #include "core_pkcs11.h"
 

diff --git a/demos/fleet_provisioning/fleet_provisioning_with_csr/pkcs11_operations.c b/demos/fleet_provisioning/fleet_provisioning_with_csr/pkcs11_operations.c
@@ -44,11 +44,11 @@
 /* MbedTLS include. */
 #include "mbedtls/ctr_drbg.h"
 #include "mbedtls/entropy.h"
-#include "mbedtls/entropy_poll.h"
+#include "entropy_poll.h"
 #include "mbedtls/error.h"
 #include "mbedtls/oid.h"
 #include "mbedtls/pk.h"
-#include "mbedtls/pk_internal.h"
+#include "pk_wrap.h"
 #include "mbedtls/sha256.h"
 #include "mbedtls/x509_crt.h"
 #include "mbedtls/x509_csr.h"
@@ -243,11 +243,12 @@ static int extractEcPublicKey( CK_SESSION_HANDLE p11Session,
  * @param[in] pRng Unused.
  * @param[in] pRngContext Unused.
  */
-static int32_t privateKeySigningCallback( void * pContext,
+static int32_t privateKeySigningCallback( mbedtls_pk_context * pContext,
                                           mbedtls_md_type_t mdAlg,
                                           const unsigned char * pHash,
                                           size_t hashLen,
                                           unsigned char * pSig,
+                                          size_t sig_size,
                                           size_t * pSigLen,
                                           int ( * pRng )( void *, unsigned char *, size_t ),
                                           void * pRngContext );
@@ -636,10 +637,15 @@ static CK_RV provisionPrivateKey( CK_SESSION_HANDLE session,
     mbedtls_pk_type_t mbedKeyType = MBEDTLS_PK_NONE;
     int mbedResult = 0;
     mbedtls_pk_context mbedPkContext = { 0 };
+    mbedtls_ctr_drbg_context ctr_drbg;
+    mbedtls_entropy_context entropy;
 
     mbedtls_pk_init( &mbedPkContext );
+    mbedtls_entropy_init(&entropy);
+    mbedtls_ctr_drbg_init(&ctr_drbg);
+    mbedtls_ctr_drbg_seed(&ctr_drbg, mbedtls_entropy_func, &entropy, NULL, 0);
     mbedResult = mbedtls_pk_parse_key( &mbedPkContext, ( const uint8_t * ) privateKey,
-                                       privateKeyLength, NULL, 0 );
+                                       privateKeyLength, NULL, 0, mbedtls_ctr_drbg_random, &ctr_drbg );
 
     if( mbedResult != 0 )
     {
@@ -900,11 +906,12 @@ static int extractEcPublicKey( CK_SESSION_HANDLE p11Session,
 
 /*-----------------------------------------------------------*/
 
-static int32_t privateKeySigningCallback( void * pContext,
+static int32_t privateKeySigningCallback( mbedtls_pk_context * pContext,
                                           mbedtls_md_type_t mdAlg,
                                           const unsigned char * pHash,
                                           size_t hashLen,
                                           unsigned char * pSig,
+                                          size_t sig_size,
                                           size_t * pSigLen,
                                           int ( * pRng )( void *, unsigned char *, size_t ),
                                           void * pRngContext )

diff --git a/demos/fleet_provisioning/fleet_provisioning_with_csr/pkcs11_operations.h b/demos/fleet_provisioning/fleet_provisioning_with_csr/pkcs11_operations.h
@@ -27,6 +27,11 @@
 #include <stdlib.h>
 #include <stdbool.h>
 
+/**
+ * @brief To access the private members of the MbedTLS structs
+ */
+ #define MBEDTLS_ALLOW_PRIVATE_ACCESS
+
 /* corePKCS11 include. */
 #include "core_pkcs11.h"
 

diff --git a/demos/http/common/src/http_demo_s3_utils.c b/demos/http/common/src/http_demo_s3_utils.c
@@ -36,6 +36,7 @@
 
 /* MBEDTLS API header. */
 #include "mbedtls/sha256.h"
+#include "mbedtls/compat-2.x.h"
 
 /* OpenSSL transport header. */
 #include "openssl_posix.h"

diff --git a/demos/lexicon.txt b/demos/lexicon.txt
@@ -428,13 +428,6 @@ optionalquery
 org
 os
 ota
-otaagentstatestopped
-otafile
-otahttpinitfailed
-otahttprequestfailed
-otahttpsuccess
-otamqttsuccess
-otapalimagestatevalid
 outcsrlength
 outform
 outgoingpublishpackets

diff --git a/docs/doxygen/building.dox b/docs/doxygen/building.dox
@@ -23,8 +23,7 @@ Linux platform. This SDK builds with [CMake](https://cmake.org/), a cross-platfo
 @section aws_iot_setup AWS IoT Account Setup
 @brief Setting up AWS IoT to run demos.
 
-You need to setup an AWS account and access the AWS IoT console for running the AWS IoT Device Shadow library, AWS IoT Device Defender library, AWS IoT Jobs library,
-AWS IoT OTA library and coreHTTP S3 download demos.
+You need to setup an AWS account and access the AWS IoT console for running the AWS IoT Device Shadow library, AWS IoT Device Defender library, AWS IoT Jobs library, and coreHTTP S3 download demos.
 Also, the AWS account can be used for running the MQTT mutual auth demo against AWS IoT broker.
 Note that running the AWS IoT Device Defender, AWS IoT Jobs and AWS IoT Device Shadow library demos require the setup of a Thing resource for the device running the demo.
 -  [Setup an AWS account](https://portal.aws.amazon.com/billing/signup#/start).
@@ -111,56 +110,6 @@ Note: A pre-signed GET URL is required for all HTTP S3 demos. For upload demos,
 	- `--key` : The name of the existing object you wish to download (GET), or the name of the object you wish to upload (PUT).
 	- `--region` : The name of the region in which your S3 bucket was created. If this is not passed in, the region configured in AWS CLI (or us-east-1, if not configured) will be used.
 
-
-@section configuring_ota_demos Configuring the Over-the-air Update Demos
-@brief Configurations and Prerequisites for the OTA Demo
-
-## Requirements for OTA demo
-
-### Prerequisites
-<ol>
-	<li> To perform a successful OTA update you would need to complete the prerequisites mentioned here: https://docs.aws.amazon.com/freertos/latest/userguide/ota-prereqs.html </li>
-	<li> A code signing certificate is required to authenticate the update. A code signing certificate based on the SHA-256 ECDSA algorithm will work with the current demos. An example of how to generate this kind of certificate can be found here: https://docs.aws.amazon.com/freertos/latest/userguide/ota-code-sign-cert-esp.html </li>
-</ol>
-
-### Configuration
-You will need to specify the following parameters in the file demo_config.h (located in `demos/ota/ota_demo_[mqtt/http]/`)
-
-<ol>
-<li> AWS_IOT_ENDPOINT: This is the endpoint for your account. The endpoint should be for the region that your thing was created in. This can be found by going to the AWS IoT Core console, pressing the “Settings” tab on the left hand side of the page. It is under the section labeled “Custom endpoint”. </li>
-<li> CLIENT_CERT_PATH: This is the client certificate that was downloaded when you created your thing. The path specified needs to be either an absolute path, or a path that is relative to the “cloned-repo-root-dir/build/bin" directory where the demo will be ran from.</li>
-<li> CLIENT_PRIVATE_KEY_PATH: This is the client private key that was downloaded when you created your thing. The path specified needs to be either an absolute path, or a path that is relative to the “cloned-repo-root-dir/build/bin" directory where the demo will be ran from.</li>
-<li> CLIENT_IDENTIFIER: This is the name of the thing you created during the OTA Prerequisites section.</li>
-</ol>
-
-An example expected output:
-@code{sh}
-#define AWS_IOT_ENDPOINT           "endpoint##-ats.iot.us-west-2.amazonaws.com"
-#define CLIENT_CERT_PATH           "/home/ubuntu/certificates/thing#-certificate.pem.crt"
-#define CLIENT_PRIVATE_KEY_PATH    "/home/ubuntu/certificates/thing#-private.pem.key"
-#define CLIENT_IDENTIFIER          "testclient"
-@endcode
-
-### Scheduling an OTA Update Job
-After you build and run the initial executable you will have to create another executable and schedule an OTA update job with this image.
-<ol>
-	<li> Increase the version of the application by setting macro `APP_VERSION_BUILD` in demos/ota_demo_core_[mqtt/http]/demo_config.h to a different version than what is running.</li>
-	<li> Rebuild the application using @ref building_demo_commmandline from below into a different directory, say build-dir-2</li>
-	<li> Rename the demo executable to reflect the change, e.g. `mv ota_demo_core_mqtt ota_demo_core_mqtt2`</li>
-	<li> Create an OTA job:
-		<ol>
-		<li> Go to the AWS IoT Core console console.aws.amazon.com/iot/ (https://console.aws.amazon.com/iot/)</li>
-		<li> Manage → Jobs → Create → Create a FreeRTOS OTA update job → Select `testclient` from the thing list</li>
-		<li> Sign a new firmware → Create a new profile → Select any SHA-ECDSA signing platform → Upload the code signing certificate(from prerequisites) and provide its path on the device.</li>
-		<li> Select the image → Select the bucket you created in prerequisites → Upload the binary build-dir-2/bin/ota_demo2</li>
-		<li> The path on device should be the complete path to place the executable and the binary name: eg /home/ubuntu/aws-iot-device-sdk-embedded-C-staging/build-dir/bin/ota_demo_core_mqtt2</li>
-		<li> Select the IAM role created in prerequisites</li>
-		<li> Create the Job
-		</ol>
-	</li>
-	<li>Run the initial executable again with the following command: `sudo ./ota_demo_core_mqtt or sudo ./ota_demo_core_http`</li>
-</ol>
-
 @section building_demo_commmandline Build Steps
 @brief How to build the demo applications on the command-line.
 

diff --git a/docs/doxygen/demos/demos_main.dox b/docs/doxygen/demos/demos_main.dox
@@ -9,6 +9,5 @@ Demos for the following libraries are provided:
 - @subpage shadow_demo
 - @subpage jobs_demo
 - @subpage defender_demo
-- @subpage ota_demos
 - @subpage fleet_provisioning_demo
 */
diff --git a/docs/doxygen/demos/ota_demo.dox b/docs/doxygen/demos/ota_demo.dox