Skip to content

Commit b830904

Browse files
snazysnazy
committed
WIP Release infra stuff
Introduce a couple of shell scripts to automate the release process. Some docs are included under `releases/README.md`. Generally, releases at the ASF follow the following workflow: 1. Draft a release 2. Start a VOTE on the dev mailing list 3. If the VOTE fails, the release has failed - "go to step 1" 4. If the VOTE passes, publish the release The above process is, without release branches, reflected in the scripts: 1. `releases/bin/draft-release.sh --major <major-version-number> --minor <minor-version-number> --commit <Git-commit-ID>` 2. if the vote passes: `releases/bin/publish-release.sh --major <major-version-number> --minor <minor-version-number>` 3. if the vote fails, just run `draft-release.sh` again The change includes scripts to handle version branches, however, using those is not required for the two release scripts above. It's important to know that the scripts handle changes to the `version.txt` file and that a specific syntax for Git tags is expected, which is required to automatically use/generate RC and final versions, including the artifact publishing to Nexus.
1 parent 6560453 commit b830904

21 files changed

+2481
-148
lines changed

‎NEXT_RELEASE_NOTES.md

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
19+
20+
Content to be copied into the next release notes, after the Contributors section.
21+
This file is maintained manually!
22+
-->

‎build-logic/src/main/kotlin/publishing/PublishingHelperPlugin.kt

-7
Original file line numberDiff line numberDiff line change
@@ -47,15 +47,8 @@ import org.gradle.plugins.signing.SigningPlugin
4747
* The task `sourceTarball` (available on the root project) generates a source tarball using `git
4848
* archive`.
4949
*
50-
* The task `releaseEmailTemplate` generates the release-vote email subject + body. Outputs on the
51-
* console and in the `build/distributions/` directory.
52-
*
5350
* Signing tip: If you want to use `gpg-agent`, set the `useGpgAgent` Gradle project property
5451
*
55-
* The following command publishes the project artifacts to your local maven repository, generates
56-
* the source tarball - and uses `gpg-agent` to sign all artifacts and the tarball. Note that this
57-
* requires a Git tag!
58-
*
5952
* ```
6053
* ./gradlew publishToMavenLocal sourceTarball -Prelease -PuseGpgAgent
6154
* ```

‎build-logic/src/main/kotlin/publishing/digest-task.kt

+1-1
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ abstract class GenerateDigest @Inject constructor(objectFactory: ObjectFactory)
3939
objectFactory.fileProperty().convention {
4040
val input = file.get().asFile
4141
val algo = algorithm.get()
42-
input.parentFile.resolve("${input.name}-${algo.replace("-", "").lowercase()}")
42+
input.parentFile.resolve("${input.name}.${algo.replace("-", "").lowercase()}")
4343
}
4444

4545
@TaskAction

‎build-logic/src/main/kotlin/publishing/rootProject.kt

+12-129
Original file line numberDiff line numberDiff line change
@@ -19,14 +19,11 @@
1919

2020
package publishing
2121

22-
import io.github.gradlenexus.publishplugin.NexusPublishExtension
2322
import io.github.gradlenexus.publishplugin.NexusPublishPlugin
24-
import io.github.gradlenexus.publishplugin.internal.StagingRepositoryDescriptorRegistryBuildService
2523
import org.gradle.api.Project
26-
import org.gradle.api.services.BuildServiceRegistration
24+
import org.gradle.api.tasks.Delete
2725
import org.gradle.api.tasks.Exec
2826
import org.gradle.kotlin.dsl.apply
29-
import org.gradle.kotlin.dsl.named
3027
import org.gradle.kotlin.dsl.register
3128
import org.gradle.plugins.signing.Sign
3229

@@ -41,11 +38,22 @@ internal fun configureOnRootProject(project: Project) =
4138
val isRelease = project.hasProperty("release")
4239
val isSigning = isRelease || project.hasProperty("signArtifacts")
4340

41+
val cleanDistributionsDir = tasks.register<Delete>("cleanDistributionsDir")
42+
cleanDistributionsDir.configure {
43+
outputs.cacheIf { false }
44+
45+
val e = project.extensions.getByType(PublishingHelperExtension::class.java)
46+
delete(e.distributionDir)
47+
}
48+
4449
val sourceTarball = tasks.register<Exec>("sourceTarball")
4550
sourceTarball.configure {
4651
group = "build"
4752
description =
4853
"Generate a source tarball for a release to be uploaded to dist.apache.org/repos/dist"
54+
outputs.cacheIf { false }
55+
56+
dependsOn(cleanDistributionsDir)
4957

5058
val e = project.extensions.getByType(PublishingHelperExtension::class.java)
5159
doFirst { mkdir(e.distributionDir) }
@@ -85,129 +93,4 @@ internal fun configureOnRootProject(project: Project) =
8593
}
8694
sourceTarball.configure { finalizedBy(signSourceTarball) }
8795
}
88-
89-
val releaseEmailTemplate = tasks.register("releaseEmailTemplate")
90-
releaseEmailTemplate.configure {
91-
group = "publishing"
92-
description =
93-
"Generate release-vote email subject + body, including the staging repository URL, if run during the Maven release."
94-
95-
mustRunAfter("initializeApacheStagingRepository")
96-
97-
doFirst {
98-
val e = project.extensions.getByType(PublishingHelperExtension::class.java)
99-
val asfName = e.asfProjectName.get()
100-
101-
val gitInfo = MemoizedGitInfo.gitInfo(rootProject)
102-
val gitCommitId = gitInfo["Apache-Polaris-Build-Git-Head"]
103-
104-
val repos = project.extensions.getByType(NexusPublishExtension::class.java).repositories
105-
val repo = repos.iterator().next()
106-
107-
val stagingRepositoryUrlRegistryRegistration =
108-
gradle.sharedServices.registrations.named<
109-
BuildServiceRegistration<StagingRepositoryDescriptorRegistryBuildService, *>
110-
>(
111-
"stagingRepositoryUrlRegistry"
112-
)
113-
val staginRepoUrl =
114-
if (stagingRepositoryUrlRegistryRegistration.isPresent) {
115-
val stagingRepositoryUrlRegistryBuildServiceRegistration =
116-
stagingRepositoryUrlRegistryRegistration.get()
117-
val stagingRepositoryUrlRegistryService =
118-
stagingRepositoryUrlRegistryBuildServiceRegistration.getService()
119-
if (stagingRepositoryUrlRegistryService.isPresent) {
120-
val registry = stagingRepositoryUrlRegistryService.get().registry
121-
try {
122-
val stagingRepoDesc = registry.get(repo.name)
123-
val stagingRepoId = stagingRepoDesc.stagingRepositoryId
124-
"https://repository.apache.org/content/repositories/$stagingRepoId/"
125-
} catch (e: IllegalStateException) {
126-
"NO STAGING REPOSITORY ($e)"
127-
}
128-
} else {
129-
"NO STAGING REPOSITORY (no registry service) !!"
130-
}
131-
} else {
132-
"NO STAGING REPOSITORY (no build service) !!"
133-
}
134-
135-
val asfProjectName = fetchAsfProjectName(asfName)
136-
137-
val versionNoRc = version.toString().replace("-rc-?[0-9]+".toRegex(), "")
138-
139-
val subjectFile = e.distributionFile("vote-email-subject.txt").relativeTo(projectDir)
140-
val bodyFile = e.distributionFile("vote-email-body.txt").relativeTo(projectDir)
141-
142-
val emailSubject = "[VOTE] Release $asfProjectName $version"
143-
subjectFile.writeText(emailSubject)
144-
145-
val emailBody =
146-
"""
147-
Hi everyone,
148-
149-
I propose that we release the following RC as the official
150-
$asfProjectName $versionNoRc release.
151-
152-
* This corresponds to the tag: apache-$asfName-$version
153-
* https://github.com/apache/$asfName/commits/apache-$asfName-$version
154-
* https://github.com/apache/$asfName/tree/$gitCommitId
155-
156-
The release tarball, signature, and checksums are here:
157-
* https://dist.apache.org/repos/dist/dev/incubator/$asfName/apache-$asfName-$version
158-
159-
You can find the KEYS file here:
160-
* https://dist.apache.org/repos/dist/release/incubator/$asfName/KEYS
161-
162-
Convenience binary artifacts are staged on Nexus. The Maven repository URL is:
163-
* $staginRepoUrl
164-
165-
Please download, verify, and test.
166-
167-
Please vote in the next 72 hours.
168-
169-
[ ] +1 Release this as Apache $asfName $version
170-
[ ] +0
171-
[ ] -1 Do not release this because...
172-
173-
Only PPMC members and mentors have binding votes, but other community members are
174-
encouraged to cast non-binding votes. This vote will pass if there are
175-
3 binding +1 votes and more binding +1 votes than -1 votes.
176-
177-
NB: if this vote pass, a new vote has to be started on the Incubator general mailing
178-
list.
179-
180-
Thanks
181-
Regards
182-
"""
183-
184-
logger.lifecycle(
185-
"""
186-
187-
188-
The email for your release vote mail:
189-
-------------------------------------
190-
191-
Suggested subject: (also in file $subjectFile)
192-
193-
$emailSubject
194-
195-
Suggested body: (also in file $bodyFile)
196-
197-
$emailBody
198-
199-
"""
200-
.trimIndent()
201-
)
202-
bodyFile.writeText(emailBody.trimIndent())
203-
}
204-
}
205-
206-
if (isRelease) {
207-
sourceTarball.configure { finalizedBy(releaseEmailTemplate) }
208-
}
209-
210-
afterEvaluate {
211-
tasks.named("closeApacheStagingRepository") { mustRunAfter(releaseEmailTemplate) }
212-
}
21396
}

‎build.gradle.kts

+3
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,9 @@ tasks.named<RatTask>("rat").configure {
110110
excludes.add("client/python/.openapi-generator/**")
111111
excludes.add("regtests/output/**")
112112

113+
excludes.add("releases/test/bats/**")
114+
excludes.add("releases/test/test_helper/**")
115+
113116
excludes.add("**/*.ipynb")
114117
excludes.add("**/*.iml")
115118
excludes.add("**/*.iws")

‎releases/README.md

+123
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# TODOs
21+
22+
* Allow draft-release.sh to publish to Apache
23+
* Close staging repo when RC is abandoned
24+
* Adopt website (top bar menu, maintained releases, left site menu)
25+
26+
# Apache Polaris Release Infrastructure
27+
28+
This directory holds all the infrastructure parts that are required to draft a Polaris release (RC) and to eventually
29+
release it.
30+
31+
Releases can be created from any Git commit, version branches are supported.
32+
33+
Generally, releases at the ASF follow the following workflow:
34+
35+
1. Draft a release
36+
2. Start a VOTE on the dev mailing list
37+
3. If the VOTE fails, the release has failed - "go to step 1"
38+
4. If the VOTE passes, publish the release
39+
40+
The Polaris project decided to run all release related via GitHub actions - a release is never drafted nor actually
41+
released from a developer machine.
42+
43+
## Technical process
44+
45+
In the `release/bin/` directory are the scripts that are required to draft a release and to publish it as "GA". All
46+
scripts can be called with the `--help` argument to get some usage information. There is also a `--dry-run` option
47+
to inspect what _would_ happen.
48+
49+
The technical process for a release follows the workflow above:
50+
51+
1. `releases/bin/draft-release.sh --major <major-version-number> --minor <minor-version-number> --commit <Git-commit-ID>`
52+
creates a release-candidate. The `--commit` argument is optional and defaults to the HEAD of the local Git
53+
worktree. The patch version number and RC-number are generated automatically. This means, that RC-numbers are
54+
automatically incremented as long as there is no "final" release tag. In that case, the patch version number
55+
is incremented and the RC-number is set to 1.
56+
57+
The Git tag name that will be used follows the pattern `polaris-<major>.<minor>.<patch>-RC<rc-number>`
58+
59+
The content of the `version.txt` file is set to the full release version, for example `1.2.3`.
60+
61+
Gradle will run with the arguments `-Prelease publishToApache closeApacheStagingRepository sourceTarball`.
62+
Both the release artifacts (jars) and the source tarball are signed. The signing and Nexus credentials must be
63+
provided externally using the `ORG_GRADLE_PROJECT_*` environment variables. In dry-run mode, this step runs
64+
Gradle with the arguments `-PjarWithGitInfo -PsignArtifacts -PuseGpgAgent publishToMavenLocal sourceTarball publishToMavenLocal`.
65+
66+
The staging repository ID, which is needed to release the staging repository, and URL are extracted from the
67+
Gradle output and memoized in the files `releases/current-release-staging-repository-id` and
68+
`releases/current-release-staging-repository-url`. The source Git commit ID is memoized in the file
69+
`releases/current-release-commit-id`.
70+
71+
The source tarball will be uploaded to the Apache infrastructure.
72+
73+
Release notes will be generated and included in the file `site/content/in-dev/unreleasd/release-notes.md`. This
74+
makes the release notes available later on the project website within the versioned docs.
75+
76+
The suggested release VOTE email subject and body with the correct links and information are provided.
77+
78+
The last step is to push the Git tag with a Git commit containing the above file changes.
79+
2. Once the release VOTE passed: `releases/bin/publish-release.sh --major <major-version-number> --minor <minor-version-number>`.
80+
81+
The command will find the latest RC tag for the latest patch release of the given major/minor version.
82+
83+
The final Git tag name that will be used follows the pattern `polaris-<major>.<minor>.<patch>` and created from
84+
the latest RC-tag for that version.
85+
86+
The Git tag is then pushed.
87+
88+
Documentation pages for the release will then be copied from the `site/content/in-dev/unreleased` into the
89+
`site/content/releases/<major>.<minor>.<patch>` folder within the `versioned-docs` branch.
90+
The changes for the updated `versioned-docs` branch are then pushed to Git.
91+
92+
The suggested ANNOUNCEMENT email subject and body are provided.
93+
94+
Creating a release in GitHub is the last step.
95+
96+
## Technical requirements
97+
98+
To use the scripts in the `releases/bin/` folder, it is required to have a _full_ clone with all tags and branches.
99+
Shallow clones, which is the default when checking out a Git repository in GitHub actions, will _not_ work properly!
100+
101+
On top, all scripts need privileges to be able to push tags and branches to the GitHub repository - this is an
102+
essential requirement for the scripts to do their job.
103+
104+
The `draft-release.sh` and `publish-release.sh` scripts also require the following environment variables providing
105+
the necessary secrets:
106+
107+
* `ORG_GRADLE_PROJECT_signingKey`
108+
* `ORG_GRADLE_PROJECT_signingPassword`
109+
* `ORG_GRADLE_PROJECT_sonatypeUsername`
110+
* `ORG_GRADLE_PROJECT_sonatypePassword`
111+
112+
GitHub actions running the scripts must provide those secrets and privileges.
113+
114+
## Version branches
115+
116+
The Polaris project may use maintenance version branches following the pattern `release/<major>.x` and
117+
`release/<major>.<minor>`. The scripts mentioned above are already support version branches and have validations for
118+
this use case. Using version branches is not a requirement for the release scripts.
119+
120+
The two scripts `releases/bin/create-release-branch.sh` plus the informative `releases/bin/list-release-branches.sh`
121+
are there to help with version branches. The former must be invoked on the main branch and creates a new major-version
122+
branch using the `release/<major>.x` pattern. The latter must be invoked on a major-version branch and creates a new
123+
minor version branch using the `release/<major>.<minor>` pattern.

0 commit comments

Comments
 (0)