Make published container images reproducible - #55689
Conversation
Publishing the same application twice produces two different image digests, because image creation is a function of the current time in three places: - every layer tar entry is stamped with the moment it happened to be written, as `PaxTarEntry` defaults its modification time to `DateTime.UtcNow` and the file's own timestamp is never read; - the image config's `created` field and the generated history entries each sample `DateTime.UtcNow` independently; - the generated `org.opencontainers.image.created` label uses `UtcNow`. The layer entries dominate: because each entry samples the clock separately, two publishes of byte-identical content yield different layer digests, so the registry cannot deduplicate the blobs and downstream tooling treats a rebuild of an unchanged commit as a brand new artifact. That is what motivated this change: retrying a publish of one commit created a second, spurious artifact in a GitOps promotion pipeline. These sites now use SOURCE_DATE_EPOCH when it is set, the cross-ecosystem convention for this exact problem (https://reproducible-builds.org/docs/source-date-epoch/). When it is unset the behavior is unchanged, so this is opt-in. Following the specification, a value that cannot be interpreted is ignored rather than failing the build. Directory enumeration order is also filesystem-defined, so entries are now sorted by their path in the container to keep the tar stream stable across machines. The digest remains a function of the content: layers built from different content still differ, which is covered by a test.
With the timestamps pinned, publishing the same content twice still produced different layer digests. Only 13 bytes of the layer differed, and they all followed from one field: `TarWriter` names the pax extended header entry that precedes each entry `./PaxHeaders.<process id>/.`, so the archive depends on the process that produced it. The remaining bytes were the header checksums that the name change shifts. POSIX suggests the process id so that concurrent extractions cannot collide over a temporary name, but the name is not meaningful to an extractor: the path an extended header applies to is carried in its `path` record, and the header always applies to the entry that immediately follows it. It is replaced here with a constant while writing the layer. The rewrite is done with a small write-through stream placed between the tar writer and the stream that hashes the layer, so the digest is computed over the normalized bytes. It reassembles the 512 byte blocks itself, since a caller can write across block boundaries, and it tracks each header's size so that file content which happens to look like a header is never rewritten. Verified end to end by publishing the same project twice to a registry: previously the two digests differed, now they are identical, while changing the source or the base image still changes the digest as expected.
|
Azure Pipelines: Successfully started running 1 pipeline(s). 2 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
This PR makes PublishContainer outputs reproducible across repeated publishes of identical inputs by removing time/process-id nondeterminism from layer tar generation, image config creation timestamps, and the org.opencontainers.image.created label. This supports registry-side deduplication and enables downstream workflows (e.g., GitOps retry/promotion) to treat identical rebuilds as the same artifact.
Changes:
- Honor
SOURCE_DATE_EPOCH(when set) for layer entry mtimes, image configcreated/history timestamps, and the generated OCIcreatedlabel. - Normalize pax extended header entry names to remove process-id variance and sort directory enumeration by container path to stabilize tar entry order.
- Add unit tests covering
SOURCE_DATE_EPOCHparsing, pax header name normalization correctness, and layer digest reproducibility.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/Microsoft.NET.Build.Containers.UnitTests/SourceDateEpochTests.cs | Adds unit coverage for SOURCE_DATE_EPOCH parsing behavior (valid/invalid/unset/culture). |
| test/Microsoft.NET.Build.Containers.UnitTests/PaxHeaderNameNormalizingStreamTests.cs | Adds tests ensuring pax header renaming is deterministic and does not corrupt tar archives. |
| test/Microsoft.NET.Build.Containers.UnitTests/LayerReproducibilityTests.cs | Adds tests asserting identical inputs yield identical layer digests and entries use the pinned timestamp. |
| src/Containers/packaging/build/Microsoft.NET.Build.Containers.targets | Updates generated org.opencontainers.image.created label to use SOURCE_DATE_EPOCH when available. |
| src/Containers/Microsoft.NET.Build.Containers/SourceDateEpoch.cs | Introduces a helper to compute a stable UTC timestamp from SOURCE_DATE_EPOCH with fallback behavior. |
| src/Containers/Microsoft.NET.Build.Containers/PaxHeaderNameNormalizingStream.cs | Introduces a stream filter to rewrite pax header names deterministically while preserving tar validity. |
| src/Containers/Microsoft.NET.Build.Containers/Layer.cs | Makes layer tar generation deterministic (pax name normalization, stable mtimes, sorted entries). |
| src/Containers/Microsoft.NET.Build.Containers/ImageConfig.cs | Makes config created and history timestamps deterministic and consistent within the blob. |
Suppressed comments (2)
test/Microsoft.NET.Build.Containers.UnitTests/LayerReproducibilityTests.cs:79
- This test sets SOURCE_DATE_EPOCH and then clears it to null in the finally block, which can clobber a pre-existing value in the test process. Capture and restore the original environment variable value instead.
finally
{
Environment.SetEnvironmentVariable("SOURCE_DATE_EPOCH", null);
}
test/Microsoft.NET.Build.Containers.UnitTests/LayerReproducibilityTests.cs:97
- This test sets SOURCE_DATE_EPOCH and then clears it to null in the finally block, which can clobber a pre-existing value in the test process. Capture and restore the original environment variable value instead.
finally
{
Environment.SetEnvironmentVariable("SOURCE_DATE_EPOCH", null);
}
Bound the digit count accepted by the targets file. DateTimeOffset.FromUnixTimeSeconds throws outside its supported range, so a value such as 99999999999999999999 failed evaluation with MSB4186 instead of being ignored, contradicting the documented behavior of ignoring values that cannot be interpreted. The C# helper already guarded this. Also restore any pre-existing SOURCE_DATE_EPOCH in the layer tests rather than clearing it.
|
Thanks, one of these was a real bug. Out-of-range
So the comma and whitespace cases were already safe: the regex rejects them and evaluation falls back to Fixed by bounding the digit count, which keeps every value inside the supported range: I also quoted the property in the Test environment variable restore: applied. Agreed, and it is cheap to be correct. The three sites in Full suite after both changes: 320 total, 314 passed, 6 skipped, 0 failed. |
baronfel
added
the
Area-Containers
baronfel
left a comment
baronfel
left a comment
There was a problem hiding this comment.
I think a few changes to the way the SOURCE_DATE_EPOCH is managed in the Task code itself are necessary to align with some MSBuild Task authoring best practices, but otherwise this looks quite good!
|
Hopefully 6cf7e45 reads easier |
|
/ba-g unrelated templating test build failure |
|
maybe update PR description to mark a few issues as closed? 🤔 |
GitHub Actions
Failure
|
/backport to release/11.0.1xx-rc1 |
|
Started backporting to |
|
/backport to release/11.0.1xx |
|
Started backporting to |
Publishing the same application twice produces two different image digests. Nothing about the build changed, but the registry cannot deduplicate the blobs and anything watching it treats the rebuild as a new artifact. This came up in a GitOps pipeline, where retrying a failed publish of one commit created a second, spurious release of an unchanged application.
Image creation is a function of the current time in four places:
PaxTarEntrydefaults its modification time toDateTime.UtcNow, and the file's own timestamp is never read, so each entry samples the clock separately;DateTime.UtcNowtwice, once forcreatedand once for the generated history entries;UtcNowin the targets file;TarWriternames each pax extended header./PaxHeaders.<process id>/..The first and last dominate: either alone changes the layer digest even when the content is byte-identical.
What this changes
The MSBuild
$(SOURCE_DATE_EPOCH)property is passed explicitly toCreateNewImage. The task parses it once using invariant integer parsing and computes onecreatedAtvalue. That single value is used for every layer entry, the image config'screatedfield, generated history entries, and theorg.opencontainers.image.createdandorg.opencontainers.artifact.createdlabels.When
SOURCE_DATE_EPOCHis unset, malformed, negative, or outside the supported date range, the task falls back to one current UTC timestamp rather than failing the build. Reproducibility therefore remains opt-in, while all timestamps produced by one publish stay consistent.The pax header name is replaced with a constant. The name is not meaningful to an extractor: the path an extended header applies to is carried in its
pathrecord, not in its entry name, and the header always applies to the entry that immediately follows it. POSIX suggests the process id so that two concurrent extractions cannot collide over a temporary name, which does not apply to an archive being written to a stream.The normalization is contained within the layer-writing stream. Before each
TarWriter.WriteEntry, the layer arms normalization for the next 512-byte tar header. If that header is a pax extended header, its name and checksum are normalized before the bytes are hashed and compressed. Normalization is only armed at an entry boundary, so file content that happens to look like a tar header is never rewritten.Directory enumeration order is filesystem-defined, so entries are also sorted by their path in the container to keep the tar stream stable across machines and builds.
Note that the pax change affects every publish, not only opted-in builds. Digests will shift once for everyone. That seemed right given it is fixing a defect rather than adding a behavior, but I am happy to gate it if you would prefer.
Verification
Published the same project twice to a registry and compared the manifest digests.
Before, the two publishes differed. Investigating showed only 13 bytes of the layer differed, and all of them followed from the pax entry name, the rest being the header checksums it shifts.
After:
sha256:2e726658…→sha256:2e726658…identicalsha256:6354b493…differs10.0-noblesha256:006bdce…differs10.0-alpinesha256:b0c7746…differsSo the digest is still a function of the inputs; it is reproducible, not frozen. With
SOURCE_DATE_EPOCHunset, the generated creation labels still use the current time.Tests
Added coverage for parsing valid, malformed, negative, and out-of-range
SOURCE_DATE_EPOCHvalues; applying a supplied creation time to the image config and history; and layer reproducibility.The layer tests verify that identical content produces the same digest, different content still produces a different digest, every entry receives the supplied timestamp, the process id is absent from pax header names, and file content that resembles a pax header is preserved unchanged.
Addresses dotnet/sdk-container-builds#34 (determinism for layers) and dotnet/sdk-container-builds#585 (
SOURCE_DATE_EPOCHsupport), which are called out as prerequisites in #54038.Also covers most of #52256, which asks for control over the creation timestamps for the same reproducibility reason. Of the options suggested there, this implements the
SOURCE_DATE_EPOCHconvention, which subsumes "set it to the commit timestamp" (SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)) without the SDK needing to shell out to git.One thing that issue reports is worth separating out. Setting the label directly still gets overwritten while automatic creation-label generation is enabled:
CreateNewImageadds the generated creation labels after the supplied labels. There is already a supported way to yield to the user:With that set, the task does not generate either OCI creation label and the user-specified value survives to the image config. That part is therefore a discoverability problem rather than a missing feature, and this PR leaves the precedence behavior unchanged.
Fixes dotnet/sdk-container-builds#34
Fixes dotnet/sdk-container-builds#585
Fixes #52256