From c71240b0b94bbc15153eac03f7385798756e3d23 Mon Sep 17 00:00:00 2001 From: Leandro Damascena Date: Fri, 1 Aug 2025 11:52:37 +0100 Subject: [PATCH 1/4] Updating release documentation --- docs/maintainers.md | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/docs/maintainers.md b/docs/maintainers.md index 879a9a7e9e3..1acfb452e84 100644 --- a/docs/maintainers.md +++ b/docs/maintainers.md @@ -151,15 +151,33 @@ Some examples using our initial and new RFC templates: #92, #94, #95, #991, #122 ### Releasing a new version -Firstly, make sure the commit history in the `develop` branch **(1)** it's up to date, **(2)** commit messages are semantic, and **(3)** commit messages have their respective area, for example `feat(logger): `, `chore(ci): ...`). +!!! note "Only maintainers with write permissions to this repository can trigger the pipeline to release a new version" -**Looks good, what's next?** +Releasing a new version is a multi-step process that takes up to 2 hours to complete. Most steps are automated - you provide inputs to trigger the release and monitor progress. -Kickoff the [`Release` workflow](https://github.com/aws-powertools/powertools-lambda-python/blob/6db9079d21698b72f5d36d72c993c1aad7276db6/.github/workflows/release.yml#L3) with the intended version - this might take around 25m-30m to complete. +**Prerequisites**: Ensure the commit history in the `develop` branch is up to date, commit messages are semantic, and include their respective area (e.g., `feat(logger): `, `chore(ci): ...`). -Once complete, you can start drafting the release notes to let customers know **what changed and what's in it for them (a.k.a why they should care)**. We have guidelines in the [release notes section](#drafting-release-notes) so you know what good looks like. +**Release Steps**: -> **NOTE**: Documentation might take a few minutes to reflect the latest version due to caching and CDN invalidations. +1. **Run end-to-end tests** and ensure they pass +2. **Trigger Release v3 workflow** - Run the [`Release v3`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/release-v3.yml) workflow with two inputs: the new Powertools version (check [latest version](https://pypi.org/project/aws-lambda-powertools/)) and the new Lambda Layer version number +3. **Monitor the release workflow** - it runs tests, publishes to PyPI, deploys Lambda layers to Beta and Prod environments across all commercial regions, runs canary tests, and updates documentation. If it fails, see the [Re-run partial failed Release workflow](#re-run-partial-failed-release-workflow) section +4. **Review and merge documentation/version PRs** - two PRs will be created to update documentation and bump version files. Review, approve and merge both (order doesn't matter) +5. **Deploy GovCloud layers** - Run the [`Layer Deployment (GovCloud)`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/layer_govcloud.yml) workflow with `develop` branch, `Prod` environment, and the Layer version number from step 2. Contact the Powertools team via internal tools if this fails +6. **Deploy China layers** - Run the [`Layer Deployment (Partitions)`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/layers_partitions.yml) workflow with `develop` branch, `Prod` environment, and the Layer version number from step 2. Contact the Powertools team via internal tools if this fails +7. **Update documentation** - Run the [`Rebuild latest docs`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/rebuild_latest_docs.yml) workflow with `develop` branch and the PyPI package version + +Once complete, you can start drafting the release notes to let customers know **what changed and what's in it for them (a.k.a why they should care)**. We have guidelines in the [release notes](#drafting-release-notes) section so you know what good looks like. + +#### Re-run partial failed Release workflow + +While workflows are designed to be stable, failures can occur during the release process. If the Release v3 workflow fails, you have two recovery options: + +**Option 1: Re-run failed jobs** +If layer deployments fail due to CloudFormation errors (we deploy ~600 layers per release and can't control CloudFormation quotas), you can re-run only the failed jobs. This will retry the deployment and typically resolves quota-related issues. + +**Option 2: Re-trigger the entire workflow** +If the release fails due to workflow modifications or permission issues that prevent re-running failed jobs, trigger the Release v3 workflow again. In this case, check the `Skip publishing to PyPI as it can't publish more than once. Useful for semi-failed releases` option to avoid PyPI publishing conflicts. #### Release process visualized @@ -224,6 +242,11 @@ section Docs Documentation release : milestone, m4, 10:28,1m +section SSM Parameters + Update SSM Parameters : active, 8s + +SSM Parameters : milestone, m5, 10:28,1m + section Post-release Close pending issues : active, 8s From f15d7aa76a42f822b9dbf2ed26a24e657ba58560 Mon Sep 17 00:00:00 2001 From: Leandro Damascena Date: Fri, 1 Aug 2025 11:56:57 +0100 Subject: [PATCH 2/4] Updating release documentation --- docs/maintainers.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/docs/maintainers.md b/docs/maintainers.md index 1acfb452e84..fa6551a4826 100644 --- a/docs/maintainers.md +++ b/docs/maintainers.md @@ -240,17 +240,28 @@ section Docs Create PR (Layer ARN) : active, after layer_prod, 8s Release versioned docs : active, 2.2m -Documentation release : milestone, m4, 10:28,1m +Documentation release : milestone, m5, 10:28,1m section SSM Parameters Update SSM Parameters : active, 8s -SSM Parameters : milestone, m5, 10:28,1m +SSM Parameters : milestone, m6, 10:28,1m section Post-release Close pending issues : active, 8s -Release complete : milestone, m6, 10:31,2m +section GovCloud + Publish GovCloud layers (Gamma) : active, 8s + Publish GovCloud layers (Prod) : active, 8s + GovCloud layers published : milestone, m7 + + +section China + Publish China layers (Gamma) : active, 8s + Publish China layers (Prod) : active, 8s + China layers published : milestone, m8 + +Release complete : milestone, m9, 10:31,2m ``` #### Drafting release notes From c6cf97f7bf723f758ab77990004574b70b5a37e1 Mon Sep 17 00:00:00 2001 From: Leandro Damascena Date: Wed, 6 Aug 2025 14:20:36 +0100 Subject: [PATCH 3/4] Addressing Andrea's feedback --- docs/maintainers.md | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/docs/maintainers.md b/docs/maintainers.md index fa6551a4826..637a02534c6 100644 --- a/docs/maintainers.md +++ b/docs/maintainers.md @@ -151,33 +151,37 @@ Some examples using our initial and new RFC templates: #92, #94, #95, #991, #122 ### Releasing a new version -!!! note "Only maintainers with write permissions to this repository can trigger the pipeline to release a new version" +!!! note "Only maintainers with write access to this repository can release a new version" Releasing a new version is a multi-step process that takes up to 2 hours to complete. Most steps are automated - you provide inputs to trigger the release and monitor progress. **Prerequisites**: Ensure the commit history in the `develop` branch is up to date, commit messages are semantic, and include their respective area (e.g., `feat(logger): `, `chore(ci): ...`). + **Release Steps**: -1. **Run end-to-end tests** and ensure they pass -2. **Trigger Release v3 workflow** - Run the [`Release v3`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/release-v3.yml) workflow with two inputs: the new Powertools version (check [latest version](https://pypi.org/project/aws-lambda-powertools/)) and the new Lambda Layer version number +1. **Run [end-to-end tests](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/.github/workflows/run-e2e-tests.yml)** and ensure they pass +2. **Trigger Release v3 workflow** - Run the [`Release v3`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/release-v3.yml) workflow with two inputs: the new Powertools version (check [latest version](https://pypi.org/project/aws-lambda-powertools/)) and the new Lambda Layer version number. To find the new Lambda Layer version number, go to this [page](https://docs.powertools.aws.dev/lambda/python/latest/#python-313) and increase the version by one (N + 1). 3. **Monitor the release workflow** - it runs tests, publishes to PyPI, deploys Lambda layers to Beta and Prod environments across all commercial regions, runs canary tests, and updates documentation. If it fails, see the [Re-run partial failed Release workflow](#re-run-partial-failed-release-workflow) section 4. **Review and merge documentation/version PRs** - two PRs will be created to update documentation and bump version files. Review, approve and merge both (order doesn't matter) -5. **Deploy GovCloud layers** - Run the [`Layer Deployment (GovCloud)`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/layer_govcloud.yml) workflow with `develop` branch, `Prod` environment, and the Layer version number from step 2. Contact the Powertools team via internal tools if this fails -6. **Deploy China layers** - Run the [`Layer Deployment (Partitions)`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/layers_partitions.yml) workflow with `develop` branch, `Prod` environment, and the Layer version number from step 2. Contact the Powertools team via internal tools if this fails +5. **Deploy GovCloud layers** - Run the [`Layer Deployment (GovCloud)`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/layer_govcloud.yml) workflow with `develop` branch, `Prod` environment, and the Layer version number from step 2. +6. **Deploy China layers** - Run the [`Layer Deployment (Partitions)`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/layers_partitions.yml) workflow with `develop` branch, `Prod` environment, and the Layer version number from step 2. 7. **Update documentation** - Run the [`Rebuild latest docs`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/rebuild_latest_docs.yml) workflow with `develop` branch and the PyPI package version Once complete, you can start drafting the release notes to let customers know **what changed and what's in it for them (a.k.a why they should care)**. We have guidelines in the [release notes](#drafting-release-notes) section so you know what good looks like. #### Re-run partial failed Release workflow -While workflows are designed to be stable, failures can occur during the release process. If the Release v3 workflow fails, you have two recovery options: +While workflows are designed to be stable, failures can occur during the release process. If the release workflow fails, you have two recovery options: -**Option 1: Re-run failed jobs** -If layer deployments fail due to CloudFormation errors (we deploy ~600 layers per release and can't control CloudFormation quotas), you can re-run only the failed jobs. This will retry the deployment and typically resolves quota-related issues. +**Option 1: Re-run failed jobs** The Release v3 pipeline contains several steps, and one of them is `publish_layer`. This step is responsible for building and deploying public Lambda layers. If this step fails due to CloudFormation errors (we deploy approximately 600 layers per release and cannot control CloudFormation quotas), you can re-run only the failed jobs. This will retry the deployment and typically resolves quota-related issues. + **Option 2: Re-trigger the entire workflow** -If the release fails due to workflow modifications or permission issues that prevent re-running failed jobs, trigger the Release v3 workflow again. In this case, check the `Skip publishing to PyPI as it can't publish more than once. Useful for semi-failed releases` option to avoid PyPI publishing conflicts. +If the release fails due to workflow modifications or permission issues that prevent re-running failed jobs, trigger the Release v3 workflow again. + +!!! important "Avoid PyPI errors" + Make sure to select `Skip publishing to PyPI` as it can't publish more than once. This is useful for semi-failed releases when rerunning the entire workflow to avoid duplicate publishes #### Release process visualized From 935db3b1a36e2210119a81c5074038ec1b801c22 Mon Sep 17 00:00:00 2001 From: Leandro Damascena Date: Wed, 6 Aug 2025 17:25:55 +0100 Subject: [PATCH 4/4] Addressing Andrea's feedback --- docs/maintainers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/maintainers.md b/docs/maintainers.md index 637a02534c6..3ddea9fa924 100644 --- a/docs/maintainers.md +++ b/docs/maintainers.md @@ -160,7 +160,7 @@ Releasing a new version is a multi-step process that takes up to 2 hours to comp **Release Steps**: -1. **Run [end-to-end tests](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/.github/workflows/run-e2e-tests.yml)** and ensure they pass +1. **Run [end-to-end tests](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/run-e2e-tests.yml)** and ensure they pass 2. **Trigger Release v3 workflow** - Run the [`Release v3`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/release-v3.yml) workflow with two inputs: the new Powertools version (check [latest version](https://pypi.org/project/aws-lambda-powertools/)) and the new Lambda Layer version number. To find the new Lambda Layer version number, go to this [page](https://docs.powertools.aws.dev/lambda/python/latest/#python-313) and increase the version by one (N + 1). 3. **Monitor the release workflow** - it runs tests, publishes to PyPI, deploys Lambda layers to Beta and Prod environments across all commercial regions, runs canary tests, and updates documentation. If it fails, see the [Re-run partial failed Release workflow](#re-run-partial-failed-release-workflow) section 4. **Review and merge documentation/version PRs** - two PRs will be created to update documentation and bump version files. Review, approve and merge both (order doesn't matter)