From a49c3648882961212c272c2ec9d4ee8db86e2f36 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Mon, 2 Dec 2024 18:11:06 -0800 Subject: [PATCH 1/8] Add lifecycle document --- README.md | 2 ++ lifecycle.md | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) create mode 100644 lifecycle.md diff --git a/README.md b/README.md index 409e4b8a..7eb196cd 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,8 @@ The Dockerfiles in this repository are used for building and testing the .NET product. As such there are Dockerfiles for the various supported Linux distributions which setup the necessary prerequisites to build and test the .NET product. +The images are provided and removed according to [Prereq container image lifecycle](./lifecycle.md). + ## Where are the published images The images produced from the Dockerfiles in this repository are published to the `mcr.microsoft.com/dotnet-buildtools/prereqs` container repository. diff --git a/lifecycle.md b/lifecycle.md new file mode 100644 index 00000000..6ab0e3bc --- /dev/null +++ b/lifecycle.md @@ -0,0 +1,54 @@ +# Prereq container image lifecycle + +The container images in this repository are provided for building .NET source and testing the resultant binaries. The images are built with OS base images and must be updated per the OS lifecycle. This document describes the required patterns. + +Container images are produced for each of the supported Linux distros listed in [Supported OS Versions](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md) files. + +## Build images + +Examples: + +- [dotnet-buildtools-prereqs-docker #1263](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1263) +- [dotnet/runtime #110198](https://github.com/dotnet/runtime/pull/110198) + +Build images are exclusively built with Azure Linux, supporting both `glibc` and `musl` build flavors. They are built for x64 only, per [Linux build methodology](https://github.com/dotnet/runtime/blob/main/docs/project/linux-build-methodology.md). + +Build images are uniquely created for each new .NET version with the latest Azure Linux available at that time. The image tags include both the given .NET and Azure Linux versions. + +The Azure Linux lifecycle is shorter than the .NET LTS lifecycle. It is expected that build images will need to be (initially) created with Azure Linux n and then replaced with Azure Linux n+1 mid-way through the .NET version lifecycle. + +Build images are deleted after a .NET version is no longer supported. It is possible that images will be required for a while after the published EOL date. + +## Test images + +Examples: + +- [dotnet-buildtools-prereqs-docker #1227](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1227) +- [dotnet-buildtools-prereqs-docker #1260](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/issues/1260) + +Test images are provided for each of the supported OS versions. They are built for each supported architecture. They are shared across .NET versions. + +Test images should be deleted after: + +- The underlying OS is EOL. +- Test images for a new version are available. +- All references to the old OS versions have been replaced by a newer in-support version + +Test images should be be built according to established patterns: + +- [dotnet-buildtools-prereqs-docker #1147](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1147) +- [dotnet-buildtools-prereqs-docker #1142](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1142) +- [dotnet-buildtools-prereqs-docker #1070](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1070) + +## Image references + +Build and test images are referenced in repo infra files, across a variety of `main` and `release/*` branches. Updating these references is a multi-step detail-oriented task. It is a pain, but necessary. + +We used version specific references in infra to ensure that our CI builds are reliable. Floating tags (for OS versions) would be guaranteed to break our build. We know that since we often see build and test breaks that need addressing in PRs where we update build and test images. + +The following locations are examples of infra that gets updated when new images are available. + +- https://github.com/dotnet/runtime/blob/main/eng/pipelines/libraries/helix-queues-setup.yml +- https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/helix-queues-setup.yml +- https://github.com/dotnet/arcade/tree/main/eng/common/templates-official/ +- https://github.com/dotnet/aspnetcore/tree/main/.azure/pipelines From ba354fa4e7c240249527cdecee5129e90e984d26 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Tue, 3 Dec 2024 09:14:27 -0800 Subject: [PATCH 2/8] Update per feedback --- lifecycle.md | 37 ++++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/lifecycle.md b/lifecycle.md index 6ab0e3bc..47a854ee 100644 --- a/lifecycle.md +++ b/lifecycle.md @@ -2,31 +2,27 @@ The container images in this repository are provided for building .NET source and testing the resultant binaries. The images are built with OS base images and must be updated per the OS lifecycle. This document describes the required patterns. -Container images are produced for each of the supported Linux distros listed in [Supported OS Versions](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md) files. +The images are a mix of Microsoft- and community-supported images. ## Build images -Examples: +The build images are exclusively built with Azure Linux, supporting both `glibc` and `musl` build flavors. They are built for x64 only, per [Linux build methodology](https://github.com/dotnet/runtime/blob/main/docs/project/linux-build-methodology.md). Microsoft designates the set of images it used for its official build per [The Official Runtime Docker Images](https://github.com/dotnet/runtime/blob/main/docs/workflow/using-docker.md#the-official-runtime-docker-images). -- [dotnet-buildtools-prereqs-docker #1263](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1263) -- [dotnet/runtime #110198](https://github.com/dotnet/runtime/pull/110198) +Build images are uniquely created for each new .NET version with the latest Azure Linux available at that time. The image tags include both the given .NET and Azure Linux versions. The tag may also contain a target OS and architecture. -Build images are exclusively built with Azure Linux, supporting both `glibc` and `musl` build flavors. They are built for x64 only, per [Linux build methodology](https://github.com/dotnet/runtime/blob/main/docs/project/linux-build-methodology.md). +The Azure Linux lifecycle is shorter than the .NET LTS lifecycle. It is expected that build images will need to be (initially) created with Azure Linux n and then replaced with Azure Linux n+1 mid-way through the .NET version lifecycle. They may also need to be updated if the target OS is EOL. -Build images are uniquely created for each new .NET version with the latest Azure Linux available at that time. The image tags include both the given .NET and Azure Linux versions. +Official build images are deleted after a .NET version is no longer supported. It is possible that images will be required for a while after the published EOL date. -The Azure Linux lifecycle is shorter than the .NET LTS lifecycle. It is expected that build images will need to be (initially) created with Azure Linux n and then replaced with Azure Linux n+1 mid-way through the .NET version lifecycle. +Examples: -Build images are deleted after a .NET version is no longer supported. It is possible that images will be required for a while after the published EOL date. +- [dotnet-buildtools-prereqs-docker #1263](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1263) +- [dotnet/runtime #110198](https://github.com/dotnet/runtime/pull/110198) +- [dotnet-buildtools-prereqs-docker #1272](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1272) ## Test images -Examples: - -- [dotnet-buildtools-prereqs-docker #1227](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1227) -- [dotnet-buildtools-prereqs-docker #1260](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/issues/1260) - -Test images are provided for each of the supported OS versions. They are built for each supported architecture. They are shared across .NET versions. +Test images are produced for the supported Linux distros listed in [Supported OS Versions](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md) files. In some cases, a [compatible derivative](https://github.com/dotnet/core/blob/main/support.md#compatible-derivatives) will be used. Images are typically built for just `x64` or for all of `Arm32`, `Arm64`, and `x64`. They are shared across .NET versions. Test images should be deleted after: @@ -34,17 +30,28 @@ Test images should be deleted after: - Test images for a new version are available. - All references to the old OS versions have been replaced by a newer in-support version +Examples: + +- [dotnet-buildtools-prereqs-docker #1227](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1227) +- [dotnet-buildtools-prereqs-docker #1260](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/issues/1260) + +## Test image construction + Test images should be be built according to established patterns: - [dotnet-buildtools-prereqs-docker #1147](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1147) - [dotnet-buildtools-prereqs-docker #1142](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1142) - [dotnet-buildtools-prereqs-docker #1070](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1070) +Note: this content should be moved to another location as it is not lifecycle related. + ## Image references Build and test images are referenced in repo infra files, across a variety of `main` and `release/*` branches. Updating these references is a multi-step detail-oriented task. It is a pain, but necessary. -We used version specific references in infra to ensure that our CI builds are reliable. Floating tags (for OS versions) would be guaranteed to break our build. We know that since we often see build and test breaks that need addressing in PRs where we update build and test images. +We use version specific references in infra to ensure that our CI builds are reliable. One can imagine using floating OS tags (such as `debian-oldest` and `debian-latest`), however such an approach would be guaranteed to break our build. We know that since we often see build and test breaks that need addressing in PRs where we update build and test images. + +At times, it may be neccessary to use a [fixed image reference](https://github.com/dotnet/runtime/pull/110199#discussion_r1859075989) for build reliability. If this is ever done, a tracking issue should be created (before the PR is merged) so that we remember to resolve the underlying issue and update the image reference. The following locations are examples of infra that gets updated when new images are available. From 06409d4afce7625fa18d7a18578dae13bef01509 Mon Sep 17 00:00:00 2001 From: Jan Kotas Date: Tue, 3 Dec 2024 10:58:16 -0800 Subject: [PATCH 3/8] Update lifecycle.md Co-authored-by: Rich Lander --- lifecycle.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lifecycle.md b/lifecycle.md index 47a854ee..d0d08e4f 100644 --- a/lifecycle.md +++ b/lifecycle.md @@ -6,7 +6,7 @@ The images are a mix of Microsoft- and community-supported images. ## Build images -The build images are exclusively built with Azure Linux, supporting both `glibc` and `musl` build flavors. They are built for x64 only, per [Linux build methodology](https://github.com/dotnet/runtime/blob/main/docs/project/linux-build-methodology.md). Microsoft designates the set of images it used for its official build per [The Official Runtime Docker Images](https://github.com/dotnet/runtime/blob/main/docs/workflow/using-docker.md#the-official-runtime-docker-images). +The build images are primarily built with Azure Linux, supporting both `glibc` and `musl` build flavors. They are built for x64 only, per [Linux build methodology](https://github.com/dotnet/runtime/blob/main/docs/project/linux-build-methodology.md). Microsoft designates the set of images it used for its official build per [The Official Runtime Docker Images](https://github.com/dotnet/runtime/blob/main/docs/workflow/using-docker.md#the-official-runtime-docker-images). Build images are uniquely created for each new .NET version with the latest Azure Linux available at that time. The image tags include both the given .NET and Azure Linux versions. The tag may also contain a target OS and architecture. From de4b589a6517026a6bc8d428c4e752d785744f6f Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Tue, 3 Dec 2024 11:31:44 -0800 Subject: [PATCH 4/8] Update lifecycle.md Co-authored-by: Sven Boemer --- lifecycle.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lifecycle.md b/lifecycle.md index d0d08e4f..f054a095 100644 --- a/lifecycle.md +++ b/lifecycle.md @@ -51,7 +51,7 @@ Build and test images are referenced in repo infra files, across a variety of `m We use version specific references in infra to ensure that our CI builds are reliable. One can imagine using floating OS tags (such as `debian-oldest` and `debian-latest`), however such an approach would be guaranteed to break our build. We know that since we often see build and test breaks that need addressing in PRs where we update build and test images. -At times, it may be neccessary to use a [fixed image reference](https://github.com/dotnet/runtime/pull/110199#discussion_r1859075989) for build reliability. If this is ever done, a tracking issue should be created (before the PR is merged) so that we remember to resolve the underlying issue and update the image reference. +At times, it may be necessary to use a [fixed image reference](https://github.com/dotnet/runtime/pull/110199#discussion_r1859075989) for build reliability. If this is ever done, a tracking issue should be created (before the PR is merged) so that we remember to resolve the underlying issue and update the image reference. The following locations are examples of infra that gets updated when new images are available. From 8a85dd75302012fcd53695ea1c8d094e9ce0addd Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Tue, 3 Dec 2024 12:07:14 -0800 Subject: [PATCH 5/8] Update lifecycle.md --- lifecycle.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lifecycle.md b/lifecycle.md index f054a095..6b7e9605 100644 --- a/lifecycle.md +++ b/lifecycle.md @@ -22,7 +22,7 @@ Examples: ## Test images -Test images are produced for the supported Linux distros listed in [Supported OS Versions](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md) files. In some cases, a [compatible derivative](https://github.com/dotnet/core/blob/main/support.md#compatible-derivatives) will be used. Images are typically built for just `x64` or for all of `Arm32`, `Arm64`, and `x64`. They are shared across .NET versions. +Test images are produced for the supported Linux distros listed in [Supported OS Versions](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md) files. In some cases, a [compatible derivative](https://github.com/dotnet/core/blob/main/support.md#compatible-derivatives) will be used. Images are built for a variety of architectures based on need. They are shared across .NET versions. Test images should be deleted after: From 6ec48efbeab42baf7f05758cd3e9d4197fc8b045 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Tue, 3 Dec 2024 14:16:04 -0800 Subject: [PATCH 6/8] Update per feedback --- lifecycle.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/lifecycle.md b/lifecycle.md index 6b7e9605..4485462e 100644 --- a/lifecycle.md +++ b/lifecycle.md @@ -4,7 +4,7 @@ The container images in this repository are provided for building .NET source an The images are a mix of Microsoft- and community-supported images. -## Build images +## Microsoft build images The build images are primarily built with Azure Linux, supporting both `glibc` and `musl` build flavors. They are built for x64 only, per [Linux build methodology](https://github.com/dotnet/runtime/blob/main/docs/project/linux-build-methodology.md). Microsoft designates the set of images it used for its official build per [The Official Runtime Docker Images](https://github.com/dotnet/runtime/blob/main/docs/workflow/using-docker.md#the-official-runtime-docker-images). @@ -12,15 +12,14 @@ Build images are uniquely created for each new .NET version with the latest Azur The Azure Linux lifecycle is shorter than the .NET LTS lifecycle. It is expected that build images will need to be (initially) created with Azure Linux n and then replaced with Azure Linux n+1 mid-way through the .NET version lifecycle. They may also need to be updated if the target OS is EOL. -Official build images are deleted after a .NET version is no longer supported. It is possible that images will be required for a while after the published EOL date. +Build images are deleted after a .NET version is no longer supported. It is possible that images will be required for a while after the published EOL date. Examples: - [dotnet-buildtools-prereqs-docker #1263](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1263) - [dotnet/runtime #110198](https://github.com/dotnet/runtime/pull/110198) -- [dotnet-buildtools-prereqs-docker #1272](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1272) -## Test images +## Microsoft test images Test images are produced for the supported Linux distros listed in [Supported OS Versions](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md) files. In some cases, a [compatible derivative](https://github.com/dotnet/core/blob/main/support.md#compatible-derivatives) will be used. Images are built for a variety of architectures based on need. They are shared across .NET versions. @@ -35,7 +34,7 @@ Examples: - [dotnet-buildtools-prereqs-docker #1227](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1227) - [dotnet-buildtools-prereqs-docker #1260](https://github.com/dotnet/dotnet-buildtools-prereqs-docker/issues/1260) -## Test image construction +### Test image construction Test images should be be built according to established patterns: @@ -59,3 +58,7 @@ The following locations are examples of infra that gets updated when new images - https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/helix-queues-setup.yml - https://github.com/dotnet/arcade/tree/main/eng/common/templates-official/ - https://github.com/dotnet/aspnetcore/tree/main/.azure/pipelines + +## Community images + +Community images are constructed per the needs of the given community. From 9f1dcdc8ae97081cc81d2eca39f947621cfc4e77 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Tue, 3 Dec 2024 14:19:31 -0800 Subject: [PATCH 7/8] Resolve linter warnings --- lifecycle.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lifecycle.md b/lifecycle.md index 4485462e..bcdc5e1f 100644 --- a/lifecycle.md +++ b/lifecycle.md @@ -54,10 +54,10 @@ At times, it may be necessary to use a [fixed image reference](https://github.co The following locations are examples of infra that gets updated when new images are available. -- https://github.com/dotnet/runtime/blob/main/eng/pipelines/libraries/helix-queues-setup.yml -- https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/helix-queues-setup.yml -- https://github.com/dotnet/arcade/tree/main/eng/common/templates-official/ -- https://github.com/dotnet/aspnetcore/tree/main/.azure/pipelines +- [dotnet/arcade](https://github.com/dotnet/arcade/tree/main/eng/common/templates-official/) +- [dotnet/aspnetcore](https://github.com/dotnet/aspnetcore/tree/main/.azure/pipelines) +- [dotnet/runtime (libraries)](https://github.com/dotnet/runtime/blob/main/eng/pipelines/libraries/helix-queues-setup.yml) +- [dotnet/runtime (coreclr)](https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/helix-queues-setup.yml) ## Community images From 454178383d42dcd04b5943758a1093012ce0b7ac Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Tue, 3 Dec 2024 14:19:59 -0800 Subject: [PATCH 8/8] Update lifecycle.md Co-authored-by: Jan Kotas --- lifecycle.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lifecycle.md b/lifecycle.md index bcdc5e1f..af1aa4f8 100644 --- a/lifecycle.md +++ b/lifecycle.md @@ -6,7 +6,7 @@ The images are a mix of Microsoft- and community-supported images. ## Microsoft build images -The build images are primarily built with Azure Linux, supporting both `glibc` and `musl` build flavors. They are built for x64 only, per [Linux build methodology](https://github.com/dotnet/runtime/blob/main/docs/project/linux-build-methodology.md). Microsoft designates the set of images it used for its official build per [The Official Runtime Docker Images](https://github.com/dotnet/runtime/blob/main/docs/workflow/using-docker.md#the-official-runtime-docker-images). +The build images are built with Azure Linux, supporting both `glibc` and `musl` build flavors. They are built for x64 only, per [Linux build methodology](https://github.com/dotnet/runtime/blob/main/docs/project/linux-build-methodology.md). Microsoft designates the set of images it used for its official build per [The Official Runtime Docker Images](https://github.com/dotnet/runtime/blob/main/docs/workflow/using-docker.md#the-official-runtime-docker-images). Build images are uniquely created for each new .NET version with the latest Azure Linux available at that time. The image tags include both the given .NET and Azure Linux versions. The tag may also contain a target OS and architecture.