[ci] `SNAPSHOT=false` overridden in Helm packaging step

[ebeahan]

Summary

SNAPSHOT=false is being overridden by a later step in settings initialization.

Example of the v9.5.0 tag triggering the Helm Chart pipeline step, but the artifact is named with the -SNAPSHOT suffix: https://buildkite.com/elastic/elastic-agent-helm-charts/builds/6261#019fccca-6047-444f-b2a9-9304758b6261/L259

Simplified Claude analysis

setPackagingDefaults() sets s.Packaging.UsePackageVersion = true at line 744 in settings.go.

parseBoolEnv("USE_PACKAGE_VERSION", s.Packaging.UsePackageVersion) at line 1564 only overrides that if USE_PACKAGE_VERSION is explicitly present in the environment.

If the env var is absent, parseBoolEnv returns the existing default (true) unchanged.

The Buildkite helm chart pipeline never sets USE_PACKAGE_VERSION, so the default true is never overridden. The if s.Packaging.UsePackageVersion guard at line 1595 is therefore entered in every BK run.

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[ebeahan]

@swiatekm this possibly relates to #14805 and #15435. Only appeared to affect the new minor and not the patch releases.

[swiatekm]

It's caused by #14871. The root cause is that USE_PACKAGE_VERSION=true is the default, so setting SNAPSHOT=false by itself doesn't do anything. This is correct for the elastic-agent package, because having an inconsistency between the dependencies and the package in the snapshot setting causes the package to be broken - though perhaps this should error instead of quietly ignoring the setting.

But this is only true for the full package. It's not true for the Helm Chart or the core package, neither of which have any dependencies. But all mage targets use the same config resolution, so they need to explicitly disable USE_PACKAGE_VERSION. The binary dra job already does this, but the Helm Chart one slipped through the cracks.

The reason we don't see it in earlier branches is that It's caused by #14871 wasn't backported there yet.

For now, I'll do the same workaround for the Helm Chart as I did for the core package, but there should be a more systemic solution to this problem. At the very least, SNAPSHOT=false shouldn't be quietly ignored.

[swiatekm]

#15972 was another example where reading from .package-version caused a bug. I'm increasingly convinced this should be a default packaging targets are opted into, rather than opted out of.