KAFKA-19174 POC for a new :distribution submodule

[dejan2609]

Prologue: #19513

This POC tries to explain why a new :distribution submodule is required for Gradle 8 -->> 9 upgrade

The thing is that Gradle 9 is very strict when it comes to a submodule boundaries: https://docs.gradle.org/9.1.0/userguide/viewing_debugging_dependencies.html#sub:resolving-unsafe-configuration-resolution-errors

Root cause (see this commit: 09c8d9f): :core module releaseTarGz task is not able to directly access other submodules runtimeClasspath configuration:

from(project(':trogdor').configurations.runtimeClasspath) { into("libs/") }

And that is why I tried to bypass that with:

subprojects {
...
  def  submodulesToIncludeInDistribution = ['tools', 'trogdor', 'shell', 'api', 'runtime', 'transforms',
                                              'json', 'file', 'basic-auth-extension', 'mirror', 'mirror-client',
                                              'streams', 'streams-scala', 'test-utils', 'examples', 'tools-api']
                                                
  def shouldModuleResolveRuntimeClasspath = (submodulesToIncludeInDistribution.contains(project.name))

    if (shouldModuleResolveRuntimeClasspath) {
        configurations.create('resolvedRuntimeClasspath') {
            extendsFrom(configurations.runtimeClasspath)
            canBeResolved = true
        }
    } 
}
...
project(':core') {
 ...
  tasks.create(name: "releaseTarGz", dependsOn: configurations.archives.artifacts, type: Tar) {
      ...
      from(project(':trogdor').configurations.resolvedRuntimeClasspath) { into("libs/") }
      ...
  }
}

but the build still fails ❌

---

Why build works in #19513 ? Because in addition to the changes above configuration for :distribution is extended: a new submodule :distribution expands Gradle setup like this:

project(':distribution') {
....
  configurations {
    includeIntoDistribution
  }

  dependencies {
    ...
    includeIntoDistribution project(path: ':trogdor', configuration: 'resolvedRuntimeClasspath')
    ...
  }

  tasks.create(name: "releaseTarGz", dependsOn: project(':core').configurations.archives.artifacts, type: Tar) {
    ...
    from(configurations.includeIntoDistribution) { into("libs/") }
    ...
  }

Pro et contra :distribution:

• build command stays the same: ./gradlew releaseTarGz ✔️
• :distribution Gradle submodule has no folder in Git ✔️ (🪧 folder gets created on ./gradlew releaseTarGz but Git ignores it)
• generated distributions (i.e. two .tgz files) paths also stays the same (they will be created in the same/previous :core path) ✔️
• some additional paragraph (no more than two lines) could be added into README.md ℹ️
• build does get slightly complicated ⚠️
• build works ✅

Pro et contra :core:

• build stays the same ✔️
• build fails (at the moment) ❌

Disclaimer: certainly, there is a way to do things differently.
Gradle provides some recommendations (not sure about effort/feasibility):
https://docs.gradle.org/9.1.0/userguide/how_to_share_outputs_between_projects.html

[dejan2609]

FYI @chia7712

[chia7712]

@dejan2609 Is this patch available? I encounter the following error

> Task :core:releaseTarGz FAILED

[Incubating] Problems report is available at: file:///home/chia7712/project/kafka/build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':core:releaseTarGz'.
> Resolution of the configuration ':trogdor:resolvedRuntimeClasspath' was attempted without an exclusive lock. This is unsafe and not allowed.

[dejan2609]

@chia7712 Yes, a complete solution is available in KAFKA-19174 branch / #19513 PR; this branch/PR (KAFKA-19174-POC-for-distribution-submodule) is just an attempt to show why :distribution submodule is required.

KAFKA-19174 build works fine:

dejan@dejan-HP-ProBook-450-G7:~/kafka$ date
Thu 25 Sep 10:58:26 CEST 2025
dejan@dejan-HP-ProBook-450-G7:~/kafka$ git status 
On branch KAFKA-19174
Your branch is up-to-date with 'origin/KAFKA-19174'.

nothing to commit, working tree clean
dejan@dejan-HP-ProBook-450-G7:~/kafka$ git log -10 --oneline 
dcfda259ca (HEAD -> KAFKA-19174, origin/KAFKA-19174) KAFKA-19174 slightly changing configuration for `releaseTarGz` task (it should be clearly visible now which submodules configurations had to be extended)
41e2d73856 KAFKA-19174 settling down with `com.gradleup.shadow` version 8.3.9 (gradle task `:jmh-benchmarks:shadowJar` is failing with shadow plugin 9+ versions)
744d7bbcef KAFKA-19174 tidying up things: 'distribution' dummy folder is removed
44950bb6da KAFKA-19174 Gradle version upgrade: 9.0.0 -->> 9.1.0
62649e7f22 KAFKA-19654 do not fail Github Actions flaky/new build matrix build when no test are discovered (i.e. solution for KAFKA-16801 is expanded)
c754acd8ca KAFKA-16801 do not fail Gradle 9 build when no test are discovered in a `upgrade-system-tests*` submodules
04eeb043c1 KAFKA-19174 (note: optional commit) version upgrades for Gradle build dependencies/plugins and GitHub Actions
07cf3e9e5c KAFKA-19174 Gradle version upgrade (8 -->> 9); build logic is refactored
79f04159d9 KAFKA-19591 solving issues with `gradlew` content: an old template version (i.e. file `unixStartScript.txt`) was being referenced
5919762009 (gooler/trunk, gooler/HEAD) MINOR: Remove exitMessage.set() call in TopicBasedRemoteLogMetadataManagerTest (#20563)
dejan@dejan-HP-ProBook-450-G7:~/kafka$ ./gradlew clean releaseTarGz -i
...
> Task :distribution:releaseTarGz
Caching disabled for task ':distribution:releaseTarGz' because:
  Build cache is disabled
  Not worth caching
Task ':distribution:releaseTarGz' is not up-to-date because:
  Output property 'archiveFile' file /home/dejan/kafka/core/build/distributions/kafka_2.13-4.2.0-SNAPSHOT.tgz has been removed.

[Incubating] Problems report is available at: file:///home/dejan/kafka/build/reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with Gradle 10.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/9.1.0/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 4m 22s
228 actionable tasks: 228 executed
Consider enabling configuration cache to speed up this build: https://docs.gradle.org/9.1.0/userguide/configuration_cache_enabling.html
dejan@dejan-HP-ProBook-450-G7:~/kafka$

[github-actions]

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

[dejan2609]

Main takeaway: from my POV it is mandatory to add a new Gradle :distribution submodule into Gradle 9 proposed solution (#19513) because we need to accommodate build to the strict rules imposed by Gradle 9.

I managed to keep paths for the generated distribution kafka-*.tgz files (they will still be added to the old :core submodule). Also: a new :distribution Gradle submodule has no folder.

@chia7712 I assume this is an acceptable solution for you, so I'll will close this POC (but please let me know if you have other thoughts).