Hey! Thanks for reporting.

It does look like your build ends up resulting in multiple different versions of the same packages (different build IDs). I expect this is not intended behavior; but since your builds succeeded with v1.0.2, then it most likely is a wrong assumption I had made.

I have an idea for a small enough fix (basically removing this particular assumption), let me give this a shot.

Alright, I have a candidate fix, would you be able to try with:

$ go get github.com/DataDog/orchestrion@b50f449

And then let me know whether this fixes your builds?

Yes sure @RomainMuller, let me try!

I am trying your fix, but after 35 minutes, the build seems frozen, and nothing seems to be done (cpu is not doing anything). Maybe there is a deadlock or something ? I will update my message if it finishes, but normally with v1.0.2 it does not take so much time.

Alright yeah I think you are probably correct... There is probably a deadlock and that might be related to how you ended up with multiple builds of the same package (just different build IDs)... It seems like my assumption that this is not "expected behavior" might actually have been correct.

I suspect a child build somehow results in different build configuration being used in v1.0.3 while it did not do so in v1.0.2... I'm going to see if I can set up a similar build to yours to trigger the v1.0.3 bug... That'd make things a lot easier for me to investigate 😅

@RomainMuller
I have created a reproducible example. Just launch foo/build.sh from the archive

foo.tar.gz

I have created a reproducible example. Just launch foo/build.sh from the archive

Haha I managed to get a reproduction on my side about the same time. Yours is a little smaller than mine though 🫣 Thanks nonetheless!

Alright so... this is a little surprising; but it would appear that the issue manifests itself ONLY if your go.mod has a dependency on github.com/DataDog/orchestrion v1.0.2 (1.0.2), but you are installing (and hence, executing) github.com/DataDog/[email protected]. The use of CGO_ENABLED, GOOS and GOARCH does not influence this, and I can reproduce it on macOS (and also Linux, but this is not a surprise).

What happens in this case, is that the -toolexec initially spawns 1.0.3 which detects you're pinned to v1.0.2 and re-spawns this (using go run github.com/DataDog/orchestrion <argv...>).

I have a theory on why this causes an issue which I'm going to test out...

Oh yes nice catch! Indeed the versions downloaded in the Dockerfile and the one in the go.mod mismatch. Previously I had @latest in the Dockerfile go install command, and the v.1.0.2 hardcoded in the go.mod, which will eventually lead to conflicts. I guess this bug report is not one in the end, sorry for the inconvenience !

Well I think there is an issue... Even though there is an "easy" workaround for you here.

Here's the culprit if you're interested (beware, lots of information ahead):

1. In order to collaborate with the go toolchain's build cache, orchestrion appends information to the outputs of compile -V=full and link -V=full that are called by the toolchain to determine the toolchain versions.
2. The data that orchestrion appends to these contains:
   • The version of orchestrion itself
   • A hash of the injection configuration
3. The <tool> -V=full commands are invoked using a working directory that does not allow orchestrion to perform version pinning checks; so the "root" build uses whatever version is running regardless of go.mod (in our case, v1.0.3)
4. Child builds, however, have a job server available (this would always be the case if you did orchestrion go build instead of go build -toolexec='orchestrion toolexec'), which allows the version check to happen, so these get v1.0.2 and NOT v1.0.3
5. The result is that child builds use a different logical toolchain, meaning different build IDs, which results in the error you observed (and otherwise could result in a deadlock or infinite loop; or a link-time fingerprint mismatch error).

I reckon this teaches us that we probably shouldn't do the version check + respawn... Maybe instead what we want to do is:

• If $CI is false-ish, emit a stark warning if the running version of orchestrion isn't the pinned one
• If $CI is true-ish, turn the above warning into an error (what is running is not what is in your go.mod so your build cannot be guaranteed to be reproductible, which usually is desirable in CI).

We might want to preserve the "auto restart" feature, but only when running in what we call "driver mode", which is when you do orchestrion go ... and NOT when you're using it in "toolexec mode", which is when you specify the -toolexec argument on your own (like you do).

By the way (and I mean to document this somewhere), in your Dockerfile, I would recommend using the following commands instead (from your reproduction):

FROM golang:1.23-bookworm AS base

# 🚮 Don't install a hard-coded orchestrion release...
# RUN go install github.com/DataDog/[email protected]

WORKDIR /bin

WORKDIR /go/src

COPY common ./common

ARG MODULE="bar"

COPY ${MODULE} ./${MODULE}

WORKDIR /go/src/${MODULE}

# Fetch dependencies.
RUN go vet && \
    go get -v

# 🆕 Installs whatever version of orchestrion is in `go.mod`
RUN go install github.com/DataDog/orchestrion

RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -toolexec="orchestrion toolexec" \
    -ldflags='-w -s -extldflags "-static"' -a -o /bin/foo . && \
    DD_TRACE_STARTUP_LOGS="false" go test -toolexec="orchestrion toolexec";

If you prefer to go install github.com/DataDog/orchestrion@latest, then you probably want to go get github.com/DataDog/orchestrion@latest && go install github.com/DataDog/orchestrion, or you perhaps should run orchestrion pin before running your build (this would "upgrade" your go.mod to the currently running orchestrion version if it's newer).

I've re-titled & pinned this issue as we need to make a decision on how exactly we permanently fix this situation. I believe this isn't specific to v1.0.3 though (just it manifests in a different way on v1.0.3).