bosh-enable-monit-access exits 1 during stemcell update - `ErrMonitJobsChainNotFound` check is dead code

[neddp]

When updating ubuntu-noble stemcells, any job whose pre-start calls bosh-enable-monit-access fails with exit 1. The BOSH director aborts the deployment at the canary with 1 of N pre-start scripts failed.

Maybe related to #406

Observed behaviour

BOSH director task output:

Task 27879319 | 06:35:42 | L executing pre-start: diego-cell/486348c4 (canary)
               L Error: Action Failed get_task: Task 3ee6f9a5 result: 1 of 22 pre-start scripts failed.
                 Failed Jobs: runtime-evacuation.
                 Successful Jobs: telegraf-bosh-team, base_rules, sysctl, loggregator_agent,
                 silk-cni, garden-cni, cfdot, bpm, install_laurel, garden, silk-daemon,
                 vxlan-policy-agent, bosh-dns, harden_env, syslog_forwarder, epf, harden_sshd,
                 mapfs, cflinuxfs3-rootfs-setup, fssnfsdriver, cflinuxfs4-rootfs-setup.
Task 27879319 | 06:35:59 | Error: Action Failed get_task: ...

Pre-start stdout log on the affected instance - output stops immediately after bosh-enable-monit-access, before the rest of the script body executes:

+ '[' -x /var/vcap/bosh/etc/bosh-enable-monit-access ']'
+ /var/vcap/bosh/etc/bosh-enable-monit-access
[enable-monit-access] 2026/06/24 06:35:42 INFO - Setting up monit firewall rule

The job config file still contained dns_server: auto after the failure - the sed replacement that follows bosh-enable-monit-access in the pre-start never ran.

bosh-enable-monit-access was confirmed to still exit 1 hours after the failed deployment:

$ sudo /var/vcap/bosh/etc/bosh-enable-monit-access; echo "exit code: $?"
[enable-monit-access] 2026/06/24 07:31:11 INFO - Setting up monit firewall rule
exit code: 1

The bosh_agent nftables table exists but the monit_access_jobs chain is absent and monit_access is empty:

table inet bosh_agent {
    chain nats_access {
        type filter hook output priority filter - 1; policy accept;
        ct state established,related accept
        ...
    }
    chain monit_access { }   # empty — no rules, no jump to monit_access_jobs
                             # monit_access_jobs chain is missing entirely
}

Used Claude for the investigation.

[neddp]

I think I managed to find the root cause: cloudfoundry/bosh-linux-stemcell-builder#672

I'm still wondering if changes to the enable-monit-access to gracefully handle this are a good idea or we should just consider this as unsupported config.

[klapkov]

@neddp I am looking at the same issue and I also think that maybe enable-monit-access should handle this case better. In our case, we need to aways call it to ensure that we can access monit, if we need to also check if the stemcell is jammy or noble, this adds unnecessary complexity.

[aramprice]

Resolved by referenced PR.