Shared GRADLE_USER_HOME via node-local hostPath: journal cache lock timeout across pods (Owner PID == Our PID); Jenkins same-host parallel OK

[zhengyansheng]

Summary

We share a writable Gradle user home across concurrent Argo Workflow pods using node-local disk (hostPath, not NFS). Concurrent ./gradlew runs often fail at Gradle startup with a journal cache lock timeout. The lock file frequently reports Owner PID == Our PID.

The same Gradle usage on a Jenkins agent (multiple processes, one PID namespace, local ~/.gradle) does not hit this failure: parallel builds proceed (or wait briefly on locks).

We are filing this to clarify whether this is a known limitation of sharing build-tool home directories across pods, and whether docs should call it out. We are not asking for a specific mitigation design in this issue—primarily want the behavior documented / confirmed.

Environment

• Argo Workflows: v3.5.13 (CLI; BuildDate 2024-12-02, GitCommit 06c761b8cc993aa6ab60f8c35c3c95bb334f3da0)
• Kubernetes (EKS): client v1.34.2-eks-ecaa3a6, server v1.35.6-eks-bca9cf6
• Gradle home volume: node-local disk (hostPath or equivalent), e.g. /opt/workflow-data/gradlew
• Storage: not NFS
• Build command: Gradle Wrapper 7.4, e.g. ./gradlew --no-daemon clean <module>:build -x test
• Multiple Workflows / parallel steps mount the same host path as Gradle user home / caches

Error

FAILURE: Build failed with an exception.

* What went wrong:
Gradle could not start your build.
> ... (service creation chain) ...
  > Could not create service of type FileAccessTimeJournal using GradleUserHomeScopeServices.createFileAccessTimeJournal().
     > Timeout waiting to lock journal cache (/opt/workflow-data/gradlew/caches/journal-1). It is currently in use by another Gradle instance.
       Owner PID: 41
       Our PID: 41
       Owner Operation:
       Our operation:
       Lock file: /opt/workflow-data/gradlew/caches/journal-1/journal-1.lock

[tomm1990]

this is expected / a known footgun, not an Argo scheduler bug.

why Owner PID == Our PID and Jenkins doesn’t do this

• file locks on that hostPath journal are host-visible (same directory, same lock file).
• each pod has its own PID namespace. pid 41 in pod A and pid 41 in pod B are different processes, but Gradle’s lock metadata only stores the pid number.
• so pod B sees “owned by 41”, checks “am I 41?”, says yes, and still can’t take the lock / times out. classic cross-container shared ~/.gradle (or GRADLE_USER_HOME) failure mode.
• Jenkins on one agent is usually one pid namespace + many processes, so the lock owner pid is meaningful and wait/contend works.

so: sharing a writable Gradle user home across concurrent pods on node-local disk is unsafe for journal/cache locks, even when the volume is not NFS. docs calling that out would be accurate.

[zhengyansheng]

Thanks for confirming the PID-namespace / Owner PID == Our PID analysis — that matches what we observed.

We want the same operational behavior as a Jenkins agent: multiple concurrent Gradle builds on one node, still sharing one writable node-local GRADLE_USER_HOME (hostPath, not NFS) so dependency caches stay warm.

Proposed approach on the Argo side:

1. Keep GRADLE_USER_HOME=/opt/workflow-data/gradlew on node-local hostPath (unchanged).
2. Set hostPID: true on the build pods so Gradle processes on that node use the host PID namespace.
3. Do not wrap gradlew with a node-level flock (that would serialize builds; we want parallelism).

Expectation: lock metadata PIDs become host-unique (like Jenkins), so journal lock wait/contend works instead of Owner PID == Our PID timeout.

Could you confirm whether this is a sound mitigation for this footgun?

Specifically:

1. With hostPID: true + shared writable GRADLE_USER_HOME on local hostPath, should concurrent pods on the same node behave like multiple processes on one Jenkins agent w.r.t. Gradle journal locks?
2. Any known caveats (PID reuse, permissions, security policies, Gradle versions) we should watch for?
3. If hostPID is undesirable, is the recommended pattern still “never share a writable Gradle user home across pods” (per-pod home / remote build cache / serialize), rather than trying to mimic Jenkins on Kubernetes?

Happy to document whichever guidance you recommend.

[tomm1990]

@zhengyansheng short answers:

1. yes, for the specific Owner PID == Our PID bug - with hostPID: true, Gradle processes on that node share the host pid namespace, so lock metadata pids are unique the way they are on a Jenkins agent. journal wait/contend should start behaving instead of the false “that’s me” path. you still want --no-daemon (or be very careful with daemons spanning pods).

2. caveats to watch

   • security / policy: hostPID is a big privilege bump (see/signal other host PIDs). many clusters block it via Pod Security / PSA / admission. treat it as an exception, not default.
   • permissions: shared writable GRADLE_USER_HOME still needs a consistent uid/gid across pods or you’ll trade lock weirdness for permission errors on caches.
   • not a silver bullet: hostPID fixes pid-namespace confusion on file locks. it does not make every Gradle cache interaction magically safe under heavy parallel writers - you can still hit contention, corrupt caches, or other lock timeouts; just not the “Owner PID == Our PID” lie.
   • PID reuse: rare edge case if a process dies and the host reuses the pid while a stale lock remains - Gradle usually recovers, but it’s why people still prefer isolation for CI.

3. if hostPID is undesirable (most locked-down K8s): recommended pattern is still don’t share one writable Gradle user home across pods. usual shapes:

   • per-pod GRADLE_USER_HOME (emptyDir) + remote build cache / dependency cache
   • or node-local read-mostly artifact cache + per-pod writable home for journals/locks
   • or serialize on that node if you insist on one writable home without hostPID

so: your proposed hostPID: true + shared local GRADLE_USER_HOME is a sound mitigation for this footgun when policy allows it; otherwise stay on “no shared writable home.”

[ahmetoguzer]

One correction to the diagnosis above, because it changes whether hostPID: true will help.

Gradle doesn't use the PID to arbitrate these locks. The PID in the lock file is only written for the error message. When a process can't get a lock, it reads the port of the current owner from the lock file and sends it an "unlock request" over UDP to a local address. The owner then releases the lock (for caches like journal-1 it's normal to hold the lock and give it up only when asked). You can see this in DefaultFileLockManager#lockStateRegion (it calls maybePingOwner(lockInfo.port, …)) and in DefaultFileLockCommunicator, which is a plain DatagramSocket. The address comes from InetAddressFactory#getLocalBindingAddress():
https://github.com/gradle/gradle/blob/master/platforms/core-execution/persistent-cache/src/main/java/org/gradle/cache/internal/DefaultFileLockManager.java

Every pod has its own network namespace, so its own loopback. Pod B sends the unlock request to localhost:<port>, but that's B's own loopback, so the request never gets to pod A. A never lets go of the lock, and B keeps retrying until the 60s timeout. On a Jenkins agent every process shares one loopback, so the request arrives and the lock is handed over. Owner PID: 41 / Our PID: 41 is just a side effect of both JVMs being PID 41 in their own namespaces. It doesn't explain why the lock times out.

So hostPID: true on its own will most likely not fix this. It only changes the PID numbers in the error message. Making the lock handover work would take a shared network namespace, i.e. hostNetwork: true. I haven't tested that, and it's an even bigger privilege bump than hostPID, so I wouldn't recommend it.

The approach Gradle supports for "warm caches, many concurrent containers" is the read-only dependency cache:

1. Seed a cache once, e.g. a nightly job that runs a build with its own GRADLE_USER_HOME. Then copy only caches/modules-2 into a node-local dir, for example /opt/gradle-ro-cache/modules-2. Drop the *.lock and gc.properties files from the copy.
2. Mount that dir read-only into each pod and set GRADLE_RO_DEP_CACHE=/opt/gradle-ro-cache.
3. Give every pod its own writable GRADLE_USER_HOME (emptyDir). That's where journals, locks and anything missing from the RO cache go.

env:
  - name: GRADLE_RO_DEP_CACHE
    value: /opt/gradle-ro-cache
  - name: GRADLE_USER_HOME
    value: /home/gradle/.gradle   # emptyDir, per pod
volumeMounts:
  - name: gradle-ro-cache
    mountPath: /opt/gradle-ro-cache
    readOnly: true

Docs: https://docs.gradle.org/current/userguide/dependency_caching.html#sec:shared-readonly-cache. Two caveats from the docs: the RO cache must not be written to while builds read from it (refresh it by swapping directories, not in place), and you shouldn't point it at a live Gradle user home. The feature is still marked incubating, but Gradle 7.4 already supports it. For task outputs, add a remote build cache. Together these give you warm dependencies and parallel builds without sharing a writable home.