Variable interpolation: behavior changes between balena-cli v24 and v25

[klutchell]

FYI for docs. Three cases where a docker-compose.yml that built fine under v24 either fails or silently behaves differently at runtime under v25. All three trace back to the same change: v24 deferred variable resolution to the runtime shell inside /bin/sh -c …, v25 resolves at parse time and freezes the result into the image config.

Want to confirm each one before we write it up so the docs are right and we know which are intentional vs known rough edges.

All outputs below come from calling @balena/compose-parser's parse() directly with referenced env vars unset. See repro.

---

1. Bare $VAR for runtime-shell expansion

services:
  test:
    entrypoint: ["/bin/sh", "-c", "echo MOD_PATH_IS=[$MOD_PATH]"]

v24: parser passes $MOD_PATH through verbatim. The runtime /bin/sh -c sees the literal $MOD_PATH and expands it from the container env at exec time. Works if MOD_PATH is set on the device via environment:, env_file:, fleet/device service vars, etc.

v25: parser interpolates against process.env at parse time. With MOD_PATH unset on the host, the entrypoint becomes:

["/bin/sh", "-c", "echo MOD_PATH_IS=[]"]

The empty string is baked into the image config before the device sees it, so container-side env vars no longer have a chance to fill it in.

Workaround: escape the dollar so it survives parse-time interpolation and reaches the runtime shell. Use $$MOD_PATH.

Question: is there a parser option callers can pass to opt out of interpolation for a specific field? Would document that as the escape hatch if it exists.

---

2. Service-level environment: no longer feeds entrypoint vars

services:
  test:
    environment:
      MY_VAR: from-service-environment-block
    entrypoint: ["/bin/sh", "-c", "echo seen=[$MY_VAR]"]

v24: parser passes $MY_VAR through verbatim. The supervisor launches the container with MY_VAR=from-service-environment-block in its env, the runtime sh -c substitutes, output is seen=[from-service-environment-block].

v25: parser interpolates $MY_VAR at parse time, only against host env. Service-level environment: is runtime-only and not visible to the interpolation pass, so the entrypoint becomes:

["/bin/sh", "-c", "echo seen=[]"]

The container still launches with MY_VAR in its env, but the entrypoint string has already lost the reference.

Same underlying change as case 1, but worth flagging separately. People who set a variable in environment: because their entrypoint references it had a working config in v24 that silently degrades in v25.

Workaround: same as case 1, use $$MY_VAR so the dollar reaches the shell. Or move the assignment into the entrypoint payload itself: ["/bin/sh", "-c", "MY_VAR=from-service-environment-block; echo seen=[$MY_VAR]"].

---

3. ${VAR:?msg} hard-fails at parse instead of at runtime

services:
  test:
    command: ["echo", "${MY_REQUIRED_VAR:?must-be-set}"]

v24: parser passes the whole ${VAR:?msg} token through verbatim. The :? enforcement happens inside the runtime shell. If the shell ever sees an unset variable at exec time, it errors with msg. Compose files that used ${VAR:?msg} for documentation purposes ("you should set this") rather than strict enforcement still built and ran fine as long as the variable was eventually present in container env.

v25: parser enforces :? at parse time, against host env. With MY_REQUIRED_VAR unset on the host, parsing fails:

ComposeError: Failed to parse compose file:
  error while interpolating services.test.command.[]:
  required variable MY_REQUIRED_VAR is missing a value: must-be-set

balena build and balena push exit non-zero before any image is built.

Workaround: either ensure the host env has the variable set during CLI invocation, or drop the :? and validate the variable's presence inside the container's entrypoint script.

---

Repro

// repro.mjs : node repro.mjs <fixture-dir>
import { parse } from '@balena/compose-parser';
import path from 'node:path';

for (const k of ['MOD_PATH','MY_VAR','MY_REQUIRED_VAR']) delete process.env[k];

const dir = path.resolve(process.argv[2] ?? '.');
process.chdir(dir);
try {
  const c = await parse(path.join(dir, 'docker-compose.yml'));
  const out = {};
  for (const [name, svc] of Object.entries(c.services ?? {})) {
    out[name] = { command: svc.command, entrypoint: svc.entrypoint };
  }
  console.log(JSON.stringify(out, null, 2));
} catch (e) {
  console.log('ParseError:', e.message);
}

Tested against @balena/compose-parser bundled in balena-cli@25.1.6. v24 baseline established against @balena/compose bundled in balena-cli@24.1.4, which doesn't do interpolation in the parser layer at all. grep for interpolat|substitute in both node_modules/@balena/compose and build/utils/compose_ts.js returns nothing.

---

Research and test cases developed with Claude (Anthropic). All outputs reviewed manually.

[cywang117]

Thanks @klutchell for looking into this. I took a look at the issue and reproduction, and this is indeed a behavior change between v24 and v25 CLI, but a change in favor of parity with docker compose. All three cases and workarounds match docker compose:

• 1 & 2: $VAR interpolates at parse time against the parser's env, not the container's runtime env. Escape with $$VAR to defer expansion to the runtime shell. Service-level environment: is runtime-only and intentionally invisible to interpolation in both tools.
• Case 3: ${VAR:?msg} is enforced at parse time and errors when unset, matching docker compose config command. Use $$ to defer to runtime.

I confirmed all three reproduce and that $$ reaches the runtime shell as expected. So the changes between CLI v24 & v25 are spec-compliant instead of rough edges.

Regarding a parser option to opt out of interpolation per field: compose-go only exposes a global, all-or-nothing SkipInterpolation (ref), and we don't surface any toggle through parse(). I'd rather not add one as docker compose doesn't expose such a toggle, and since $$ already gives per-token control.

We don't currently have any variable interpolation docs at the moment, I can look at creating those.

[klutchell]

Worth flagging that this is worse in practice right now than the repro suggests, because the builder hasn't picked up the v25 parser yet. Only the CLI and the deploy-to-balena action have.

So the $$VAR workaround isn't portable across build paths:

• Build with the CLI or deploy-to-balena: new parser collapses $$ to $, runtime shell sees $VAR. Works.
• Push to the balena builder: old parser passes $$ through untouched, so the literal $$ gets baked into the image config.

I confirmed the builder case on a device via the dashboard. With command: ["/bin/bash","-c","if [[ $${DISABLE,,} =~ ... ]]"], the $$ reaches the container literally instead of being collapsed. bash then reads $$ as the shell PID, so the check never evaluates DISABLE.

So today there's no single escaping that's correct for both. $VAR breaks the CLI (interpolates to empty at parse time), $$VAR breaks the builder (literal $$ at runtime). I worked around it in my case with printenv VAR, since there's no $ to interpolate either way, but that only helps for shell commands.

[cywang117]

Thanks for the additional info @klutchell . I suppose the workaround until we get the builder issue fixed is to stay on v24 CLI. Is this blocking in any way?

[klutchell]

Thanks for the additional info @klutchell . I suppose the workaround until we get the builder issue fixed is to stay on v24 CLI. Is this blocking in any way?

Yes, if users want to avoid variable interpolation in their compose file I would suggest they stay on CLI v24 if possible.

Internally it's not as simple, because we also need to stay on deploy-to-balena-action < v2.2.7 and Flowzone < v22.12.19 but we always track master.

So we could pin Flowzone to an older release of the action to mitigate the effected projects, but I don't know how many that is at this point. Some might be silently failing with vars substituted with empty strings before being passed to the builder, and we aren't aware of it. Though an aligned builder wouldn't solve that either I guess.