Convert ref and runtime packs to use the SharedFramework SDK and Arcade's installer tooling (#58612)

* Add an sfxproj for the ref pack for Microsoft.AspNetCore.App. IT has the same contents except for some simplification of the analyzer layout and the PlatformManifest (which will be implemented correctly as part converting the runtime pack)

* Add sfxproj-based runtime framework project

* Produce the targeting pack archives

* Build the runtime archive. There's some extra rebuilding now, so need to look into that.

* Add a (mostly correct, some TODOs) composite runtime image and archive.

* Use traversal projects to ensure that building the runtime archive projects directly restores the sfxprojs.

* Add the new .deb packaging to the targeting pack

* Add the aspnetcore runtime deb package to the sfxproj

* Enable building RPM packages

* Publish from the new-style projects to the correct subfolders for consuming projects to pick up.

* Remove a TODO that's being addressed

* Add support for partial R2R rooting based on dotnet/sdk#44436

* Reuse global properties and remove TODO addresed in corresponding Arcade PR.

* Replace old shared framework projects and Linux installers with the new sfxproj and archive-based projects

* Remove the NoTargets SDK, ended up using the Traversal SDK instead.

* Fix path to runtime pack for manifest

* Don't reference the installers package on Windows. We don't want to build MSIs with this tooling (yet)

* Fix reference from aspnetcore-runtime and sequence the projects into the build better

* Profive projects to build when only -buildinstallers is passed. This ensures we don't try to build AspNetCore.sln

* Fix the path of the runtime list for testing.

* Set property with file name

* Disable APICompat for the archive projects.

* Fix expected output directory for deb/rpm/targz/zip to be the shipping packages output.

* Fix including zips (as wixpacks are still in the intsallers folder)

* Build the shared frameworks via the transitive SDK proj only and make the native project references "just work" without a separate target.

* Add PGO data via direct references to the MIBC files instead of flowing them through a different manner (that doesn't work)

* Don't exclude the runtime archive traversal proj from the source-only build.

* Add some makedirs

* Don't try to extract the runtime we skipped downloading

* Always prefer the ref assemblies in the transport pack over the impl assemblies when building the ref pack.

* Add the right PackageOverrides.txt entries and make sure we exclude analyzers from the expected PackageOverrides list (they aren't supposed to be there)

* Enable building all projects using the sharedfx sdk in source-build

* Don't exclude the composite archive from source build.

* Write out the aspnetcore_base_runtime.version file

* Update aspnetcore-base-runtime.proj

* Don't R2R when explicitly requested not to

* Update .NET SDK to get composite image fix

* Use the new RPM tooling and build debs/rpms in the regular build step (instead of as separate steps)

* Remove rpmpkg references and update ref pack baseline version

* Fix version string

* Remove installers group on Build.props as the "managed" build will now correctly trigger debs/rpms when requested

* Use Arcade RPM tooling for the Runtime sfxproj

* Revert "Update .NET SDK to get composite image fix"

This reverts commit b1b144d.

* Don't pack the ref pack when only packing rid-specific packages

* Remove RpmOwnedDirectory items as they're now provided by the SharedFramework SDK.

* Don't build installers in source-build

* Pass linux installer/archive artifacts over to the signing jobs

* Don't publish the unsigned build if we're not doing post-build signing. We'll publish from the signing job.

* Bulid installers on Linux-arm64

* Set the InstallerPackageDescription property to also set the description for installers (after dotnet/arcade#15308 flows)

---------

Co-authored-by: Igor Velikorossov <[email protected]>

Author: 2 people

.azure/pipelines/ci-public.yml

@@ -79,14 +79,6 @@ variables:
   value: true
 - template: /eng/common/templates/variables/pool-providers.yml
 
-resources:
-  containers:
-  - container: debpkg
-    image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-debpkg
-
-  - container: rpmpkg
-    image: mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-fpm
-
 stages:
 - stage: build
   displayName: Build
@@ -313,43 +305,13 @@ stages:
             --ci
             --arch x64
             --pack
+            --build-installers
             --all
             --no-build-java
             -p:OnlyPackPlatformSpecificPackages=true
             $(_BuildArgs)
             $(_InternalRuntimeDownloadArgs)
         displayName: Run build.sh
-      - script: ./eng/build.sh
-            --ci
-            --nobl
-            --arch x64
-            --build-installers
-            --no-build-deps
-            --no-build-nodejs
-            -p:OnlyPackPlatformSpecificPackages=true
-            -p:BuildRuntimeArchive=false
-            -p:LinuxInstallerType=deb
-            $(_BuildArgs)
-            $(_InternalRuntimeDownloadArgs)
-        displayName: Build Debian installers
-        target: debpkg
-      - script: ./eng/build.sh
-            --ci
-            --nobl
-            --arch x64
-            --build-installers
-            --no-build-deps
-            --no-build-nodejs
-            --publish
-            -p:OnlyPackPlatformSpecificPackages=true
-            -p:BuildRuntimeArchive=false
-            -p:LinuxInstallerType=rpm
-            -p:AssetManifestFileName=aspnetcore-Linux_x64.xml
-            $(_BuildArgs)
-            $(_PublishArgs)
-            $(_InternalRuntimeDownloadArgs)
-        displayName: Build RPM installers
-        target: rpmpkg
       artifacts:
       - name: Linux_x64_Logs_Attempt_$(System.JobAttempt)
         path: artifacts/log/
@@ -404,29 +366,13 @@ stages:
             --ci
             --arch arm64
             --pack
+            --build-installers
             --all
             --no-build-java
             -p:OnlyPackPlatformSpecificPackages=true
             $(_BuildArgs)
             $(_InternalRuntimeDownloadArgs)
         displayName: Run build.sh
-      - script: ./eng/build.sh
-            --ci
-            --nobl
-            --arch arm64
-            --build-installers
-            --no-build-deps
-            --no-build-nodejs
-            --publish
-            -p:OnlyPackPlatformSpecificPackages=true
-            -p:BuildRuntimeArchive=false
-            -p:LinuxInstallerType=rpm
-            -p:AssetManifestFileName=aspnetcore-Linux_arm64.xml
-            $(_BuildArgs)
-            $(_PublishArgs)
-            $(_InternalRuntimeDownloadArgs)
-        displayName: Build RPM installers
-        target: rpmpkg
       artifacts:
       - name: Linux_arm64_Logs_Attempt_$(System.JobAttempt)
         path: artifacts/log/

.azure/pipelines/ci.yml

@@ -38,9 +38,13 @@ variables:
 - ${{ if or(startswith(variables['Build.SourceBranch'], 'refs/heads/release/'), startswith(variables['Build.SourceBranch'], 'refs/heads/internal/release/'), eq(variables['Build.Reason'], 'Manual')) }}:
   - name: PostBuildSign
     value: false
+  - name: _ArcadePublishNonWindowsArg
+    value: ''
 - ${{ else }}:
   - name: PostBuildSign
     value: true
+  - name: _ArcadePublishNonWindowsArg
+    value: --publish
 - name: _UseHelixOpenQueues
   value: ${{ ne(variables['System.TeamProject'], 'internal') }}
 - ${{ if and(notin(variables['Build.Reason'], 'PullRequest'), eq(variables['Build.SourceBranch'], 'refs/heads/main')) }}:
@@ -99,12 +103,6 @@ variables:
 - template: /eng/common/templates-official/variables/pool-providers.yml@self
 
 resources:
-  containers:
-  - container: debpkg
-    image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-debpkg
-
-  - container: rpmpkg
-    image: mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-fpm
   repositories:
   # Repo: 1ESPipelineTemplates/1ESPipelineTemplates
   - repository: 1esPipelines
@@ -299,7 +297,7 @@ extends:
             --pack
             --all
             --no-build-java
-            --publish
+            $(_ArcadePublishNonWindowsArg)
             -p:OnlyPackPlatformSpecificPackages=true
             -p:AssetManifestFileName=aspnetcore-MacOS_arm64.xml
             $(_BuildArgs)
@@ -329,7 +327,7 @@ extends:
             --pack
             --all
             --no-build-java
-            --publish
+            $(_ArcadePublishNonWindowsArg)
             -p:OnlyPackPlatformSpecificPackages=true
             -p:AssetManifestFileName=aspnetcore-MacOS_x64.xml
             $(_BuildArgs)
@@ -355,48 +353,15 @@ extends:
           jobDisplayName: "Build: Linux x64"
           agentOs: Linux
           useHostedUbuntu: false
-          steps:
-          - script: ./eng/build.sh
-                --ci
-                --arch x64
-                --pack
-                --all
-                --no-build-java
-                -p:OnlyPackPlatformSpecificPackages=true
-                $(_BuildArgs)
-                $(_InternalRuntimeDownloadArgs)
-            displayName: Run build.sh
-          - script: ./eng/build.sh
-                --ci
-                --nobl
-                --arch x64
-                --build-installers
-                --no-build-deps
-                --no-build-nodejs
-                -p:OnlyPackPlatformSpecificPackages=true
-                -p:BuildRuntimeArchive=false
-                -p:LinuxInstallerType=deb
-                $(_BuildArgs)
-                $(_InternalRuntimeDownloadArgs)
-            displayName: Build Debian installers
-            target: debpkg
-          - script: ./eng/build.sh
-                --ci
-                --nobl
-                --arch x64
-                --build-installers
-                --no-build-deps
-                --no-build-nodejs
-                --publish
-                -p:OnlyPackPlatformSpecificPackages=true
-                -p:BuildRuntimeArchive=false
-                -p:LinuxInstallerType=rpm
-                -p:AssetManifestFileName=aspnetcore-Linux_x64.xml
-                $(_BuildArgs)
-                $(_PublishArgs)
-                $(_InternalRuntimeDownloadArgs)
-            displayName: Build RPM installers
-            target: rpmpkg
+          buildArgs:
+            --arch x64
+            --pack
+            --all
+            --build-installers
+            --no-build-java
+            -p:OnlyPackPlatformSpecificPackages=true
+            $(_BuildArgs)
+            $(_InternalRuntimeDownloadArgs)
           artifacts:
           - name: Linux_x64_Logs_Attempt_$(System.JobAttempt)
             path: artifacts/log/
@@ -421,7 +386,7 @@ extends:
             --pack
             --all
             --no-build-java
-            --publish
+            $(_ArcadePublishNonWindowsArg)
             -p:OnlyPackPlatformSpecificPackages=true
             -p:AssetManifestFileName=aspnetcore-Linux_arm.xml
             $(_BuildArgs)
@@ -446,34 +411,15 @@ extends:
           jobName: Linux_arm64_build
           jobDisplayName: "Build: Linux ARM64"
           agentOs: Linux
-          steps:
-          - script: ./eng/build.sh
-                --ci
-                --arch arm64
-                --pack
-                --all
-                --no-build-java
-                -p:OnlyPackPlatformSpecificPackages=true
-                $(_BuildArgs)
-                $(_InternalRuntimeDownloadArgs)
-            displayName: Run build.sh
-          - script: ./eng/build.sh
-                --ci
-                --nobl
-                --arch arm64
-                --build-installers
-                --no-build-deps
-                --no-build-nodejs
-                --publish
-                -p:OnlyPackPlatformSpecificPackages=true
-                -p:BuildRuntimeArchive=false
-                -p:LinuxInstallerType=rpm
-                -p:AssetManifestFileName=aspnetcore-Linux_arm64.xml
-                $(_BuildArgs)
-                $(_PublishArgs)
-                $(_InternalRuntimeDownloadArgs)
-            displayName: Build RPM installers
-            target: rpmpkg
+          buildArgs:
+            --arch arm64
+            --pack
+            --all
+            --build-installers
+            --no-build-java
+            -p:OnlyPackPlatformSpecificPackages=true
+            $(_BuildArgs)
+            $(_InternalRuntimeDownloadArgs)
           artifacts:
           - name: Linux_arm64_Logs_Attempt_$(System.JobAttempt)
             path: artifacts/log/
@@ -500,7 +446,7 @@ extends:
             --pack
             --all
             --no-build-java
-            --publish
+            $(_ArcadePublishNonWindowsArg)
             -p:OnlyPackPlatformSpecificPackages=true
             -p:AssetManifestFileName=aspnetcore-Linux_musl_x64.xml
             $(_BuildArgs)
@@ -534,7 +480,7 @@ extends:
             --pack
             --all
             --no-build-java
-            --publish
+            $(_ArcadePublishNonWindowsArg)
             -p:OnlyPackPlatformSpecificPackages=true
             -p:AssetManifestFileName=aspnetcore-Linux_musl_arm.xml
             $(_BuildArgs)
@@ -567,7 +513,7 @@ extends:
             --pack
             --all
             --no-build-java
-            --publish
+            $(_ArcadePublishNonWindowsArg)
             -p:OnlyPackPlatformSpecificPackages=true
             -p:AssetManifestFileName=aspnetcore-Linux_musl_arm64.xml
             $(_BuildArgs)

.azure/pipelines/jobs/codesign-xplat.yml

@@ -19,12 +19,20 @@ jobs:
       inputs:
         artifactName: ${{ parameters.inputName }}_Packages
         downloadPath: $(Build.StagingDirectory)/downloaded_packages/
-        itemPattern: '**/*.nupkg'
+        itemPattern: |
+          **/*.nupkg
+          **/*.tar.gz
+          **/*.deb
+          **/*.rpm
     - task: CopyFiles@2
       displayName: Copy packages to ArtifactsShippingPackagesDir
       inputs:
         sourceFolder: $(Build.StagingDirectory)/downloaded_packages/
-        contents: '**/*.nupkg'
+        contents: |
+          **/*.nupkg
+          **/*.tar.gz
+          **/*.deb
+          **/*.rpm
         targetFolder: $(Build.SourcesDirectory)/artifacts/packages/$(BuildConfiguration)/shipping/
         flattenFolders: true
     - powershell: .\eng\common\build.ps1

AspNetCore.sln

@@ -972,7 +972,7 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Framework", "Framework", "{
 EndProject
 Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "App.Runtime", "App.Runtime", "{F1A6BA04-853E-43B5-B27E-54558603837B}"
 EndProject
-Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.AspNetCore.App.Runtime", "src\Framework\App.Runtime\src\Microsoft.AspNetCore.App.Runtime.csproj", "{3163C56E-99BC-4BD1-8880-4F94A2407741}"
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.AspNetCore.App.Runtime", "src\Framework\App.Runtime\src\Microsoft.AspNetCore.App.Runtime.sfxproj", "{3163C56E-99BC-4BD1-8880-4F94A2407741}"
 EndProject
 Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.AspNetCore.Razor.Test", "src\Razor\Razor\test\Microsoft.AspNetCore.Razor.Test.csproj", "{776BFCDC-8591-4E3D-BD0D-858A188A033D}"
 EndProject
@@ -1354,7 +1354,7 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Extensions.FilePr
 EndProject
 Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Extensions.FileProviders.Embedded.Tests", "src\FileProviders\Embedded\test\Microsoft.Extensions.FileProviders.Embedded.Tests.csproj", "{B06ADD57-E855-4D8C-85DC-B323509AE540}"
 EndProject
-Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.AspNetCore.App.Ref", "src\Framework\App.Ref\src\Microsoft.AspNetCore.App.Ref.csproj", "{BAD47859-95DF-4C8F-9AF7-C48B68F478A1}"
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.AspNetCore.App.Ref", "src\Framework\App.Ref\src\Microsoft.AspNetCore.App.Ref.sfxproj", "{BAD47859-95DF-4C8F-9AF7-C48B68F478A1}"
 EndProject
 Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.AspNetCore.App.UnitTests", "src\Framework\test\Microsoft.AspNetCore.App.UnitTests.csproj", "{010A9638-F20E-4FE6-A186-85732BFC9CB0}"
 EndProject

Directory.Build.BeforeCommonTargets.targets

@@ -6,7 +6,7 @@
         Condition="'$(ExcludeFromSourceOnlyBuild)' == '' and
             '$(DotNetBuildSourceOnly)' == 'true' and
             '$(IsAspNetCoreApp)' != 'true' and
-            '$(MSBuildProjectName)' != '$(TargetingPackName)' and
+            '$(UsingMicrosoftDotNetSharedFrameworkSdk)' != 'true' and
             '$(IsAnalyzersProject)' != 'true' and
             '$(IsProjectTemplateProject)' != 'true'">true</ExcludeFromSourceOnlyBuild>
 

Directory.Build.targets

@@ -180,6 +180,7 @@
   <Import Project="eng\targets\FSharp.Common.targets"  Condition="'$(MSBuildProjectExtension)' == '.fsproj'" />
   <Import Project="eng\targets\Wix.Common.targets"  Condition="'$(MSBuildProjectExtension)' == '.wixproj'" />
   <Import Project="eng\targets\Java.Common.targets"  Condition="'$(MSBuildProjectExtension)' == '.javaproj'" />
+  <Import Project="eng\targets\Sfx.Common.targets"  Condition="'$(MSBuildProjectExtension)' == '.sfxproj'" />
   <Import Project="eng\testing\linker\trimmingTests.targets" Condition="'$(IsPublishedAppTestProject)' == 'true'" />
   <Import Project="eng\targets\Helix.targets" Condition=" $(IsTestProject) " />
   <Import Project="eng\targets\FunctionalTestWithAssets.targets"

eng/Build.props

@@ -115,8 +115,11 @@
           the entire pattern will silently fail to evaluate correctly.
         -->
         <DotNetProjects Include="
-                          $(RepoRoot)src\Framework\App.Ref\src\Microsoft.AspNetCore.App.Ref.csproj;
+                          $(RepoRoot)src\Framework\App.Ref\src\Microsoft.AspNetCore.App.Ref.sfxproj;
                           $(RepoRoot)src\Framework\App.Ref.Internal\src\Microsoft.AspNetCore.App.Ref.Internal.csproj;
+                          $(RepoRoot)src\Framework\App.Runtime\src\aspnetcore-runtime.proj;
+                          $(RepoRoot)src\Framework\App.Runtime\src\aspnetcore-runtime-composite.proj;
+                          $(RepoRoot)src\Framework\App.Runtime\src\aspnetcore-base-runtime.proj;
                           $(RepoRoot)src\Framework\AspNetCoreAnalyzers\test\Microsoft.AspNetCore.App.Analyzers.Test.csproj;
                           $(RepoRoot)src\Framework\test\Microsoft.AspNetCore.App.UnitTests.csproj;
                           $(RepoRoot)src\Assets\**\*.*proj;

eng/DotNetBuild.props

@@ -78,12 +78,9 @@
       <!--
         Add the internal installers artifacts required by dotnet/installer.
       -->
-      <IntermediateNupkgArtifactFile Include="$(InstallersArtifactsDir)aspnetcore-runtime-*.tar.gz" />
       <IntermediateNupkgArtifactFile Include="$(InstallersArtifactsDir)aspnetcore_base_runtime.version" />
       <IntermediateNupkgArtifactFile Include="$(InstallersArtifactsDir)aspnetcore-runtime-*.msi" />
       <IntermediateNupkgArtifactFile Include="$(InstallersArtifactsDir)aspnetcore-runtime-*.pkg" />
-      <IntermediateNupkgArtifactFile Include="$(InstallersArtifactsDir)aspnetcore-runtime-*.deb" />
-      <IntermediateNupkgArtifactFile Include="$(InstallersArtifactsDir)aspnetcore-runtime-*.rpm" />
     </ItemGroup>
   </Target>
 

eng/Publishing.props

@@ -12,7 +12,7 @@
                                        '$(PostBuildSign)' != 'true' and
                                        '$(DotNetBuildRepo)' != 'true'">false</EnableDefaultArtifacts>
 
-    <PublishInstallerBaseVersion Condition="'$(PublishInstallerBaseVersion)' == '' and 
+    <PublishInstallerBaseVersion Condition="'$(PublishInstallerBaseVersion)' == '' and
                                             ('$(OS)' == 'Windows_NT' or '$(DotNetBuildOrchestrator)' == 'true')">true</PublishInstallerBaseVersion>
     <!-- This avoids creating VS.*.symbols.nupkg packages that are identical to the original package. -->
     <AutoGenerateSymbolPackages>false</AutoGenerateSymbolPackages>
@@ -31,13 +31,15 @@
     <_InstallersToPublish Include="$(ArtifactsDir)installers\**\*.version" UploadPathSegment="Runtime" Condition="'$(PublishInstallerBaseVersion)' == 'true'" />
 
     <!-- The following installers create checksums -->
-    <_InstallersToPublish Include="$(ArtifactsDir)installers\**\*.deb" UploadPathSegment="Runtime" ChecksumPath="%(FullPath).sha512" />
+    <_InstallersToPublish Include="$(ArtifactsDir)packages\**\*.deb" UploadPathSegment="Runtime" ChecksumPath="%(FullPath).sha512" />
+    <_InstallersToPublish Include="$(ArtifactsDir)packages\**\*.rpm" UploadPathSegment="Runtime" ChecksumPath="%(FullPath).sha512" />
+    <_InstallersToPublish Include="$(ArtifactsDir)packages\**\*.tar.gz" UploadPathSegment="Runtime" ChecksumPath="%(FullPath).sha512" />
+    <_InstallersToPublish Include="$(ArtifactsDir)packages\**\*.zip" UploadPathSegment="Runtime" />
+
     <_InstallersToPublish Include="$(ArtifactsDir)installers\**\*.exe" UploadPathSegment="Runtime" ChecksumPath="%(FullPath).sha512" />
     <_InstallersToPublish Include="$(ArtifactsDir)installers\**\*.msi" UploadPathSegment="Runtime" ChecksumPath="%(FullPath).sha512" />
-    <_InstallersToPublish Include="$(ArtifactsDir)installers\**\*.rpm" UploadPathSegment="Runtime" ChecksumPath="%(FullPath).sha512" />
-    <_InstallersToPublish Include="$(ArtifactsDir)installers\**\*.tar.gz" UploadPathSegment="Runtime" ChecksumPath="%(FullPath).sha512" />
     <_InstallersToPublish Include="$(ArtifactsDir)installers\**\*.wixlib" UploadPathSegment="Runtime" ChecksumPath="%(FullPath).sha512" />
-    <_InstallersToPublish Include="$(ArtifactsDir)installers\**\*.zip" UploadPathSegment="Runtime" >
+    <_InstallersToPublish Include="$(ArtifactsDir)installers\**\*.zip" UploadPathSegment="Runtime">
       <!-- Exclude wixpack.zip files from checksum generation -->
       <ChecksumPath Condition="$([System.String]::Copy('%(Filename)%(Extension)').EndsWith('.wixpack.zip')) != 'true'">%(FullPath).sha512</ChecksumPath>
     </_InstallersToPublish>

eng/Signing.props

@@ -19,7 +19,7 @@
     <CommonFilesToSign Include="$(ArtifactsPackagesDir)**\*.nupkg" />
     <CommonFilesToSign Include="$(VisualStudioSetupOutputPath)**\*.vsix" />
     <CommonFilesToSign Include="$(ArtifactsPackagesDir)**\*.jar"  />
-    <CommonFilesToSign Include="$(ArtifactsDir)installers\$(Configuration)\**\*.zip" />
+    <CommonFilesToSign Include="$(ArtifactsDir)packages\$(Configuration)\**\*.zip" />
     <CommonFilesToSign Remove="$(ArtifactsDir)installers\$(Configuration)\**\*.wixpack.zip" Condition="'$(PostBuildSign)' != 'true'" />
   </ItemGroup>
   <Choose>