[FEATURE] opt-in flag to preserve drain-on-SIGTERM behaviour (v3.110.0+)

[aodanxin]

Is your feature request related to a problem? Please describe.

Prior to v3.110.0, agent-stack-k8s handled SIGTERM by draining gracefully. The agent waited for the current job to finish before disconnecting.

We've heard from our Buildkite contact that starting in v3.110.0, this behaviour changed. The agent now immediately cancels the running job on SIGTERM instead of draining.

This would cause issues for us.
We run Buildkite agents on Kubernetes with Karpenter using the WhenEmptyOrUnderutilized consolidation policy. When Karpenter identifies an underutilized node, it evicts pods via SIGTERM.
With the new v3.110.0+ behaviour, this causes running CI jobs to be cancelled rather than completing.

Describe the solution you'd like

Add an opt-in flag or config option to agent-stack-k8s to restore the previous drain-on-SIGTERM behaviour: when a job pod receives SIGTERM, the agent should attempt to finish the current job rather than cancelling it immediately.

Describe alternatives you've considered

N/A

Affiliation (optional)

Faire

Additional context

Conversation within our Buildkite Slack channel: https://faire-wholesale.slack.com/archives/C06HR118GSW/p1782334084676339.

[DrJosh9000]

Thanks for raising @aodanxin. Let me see if I understand:

With the previous behaviour, the agent swallowed the SIGTERM sent when Kubernetes deletes a Pod resource, and so nothing happened to the running job. Some time later, the pod would be sent a SIGKILL, and abruptly stop running. This was a problem for two reasons:

• Artifact upload phase, and post-command and pre-exit hooks had no time to run.
• The agent was not able to call the FinishJob or Disconnect API methods, and would be considered lost a few minutes later.

With the current behaviour, the agent observes SIGTERM, passes the SIGTERM along to the command so the command can start winding up if it wants to. If the command finishes within the signal grace period then it can run the post-command steps and exit properly. If the command doesn't terminate within the signal grace period, then the agent sends a SIGKILL so it still has time to run those post-command steps. This all happens before Kubernetes sends SIGKILL to the whole pod, because the signal grace period is computed based on terminateGracePeriodSeconds that agent-stack-k8s is configured with, unless the agent has hung for some reason.

The current behaviour is more aligned with how the agent works outside of Kubernetes.

If you want to ignore the SIGTERM from your command in order to let it finish naturally (I'm assuming you also set terminateGracePeriodSeconds to be quite large) then your command seems like the most reasonable place to trap SIGTERM and ignore it if that's what you want. But perhaps there's a case for the agent having config where it would run jobs without ever signalling the command.

[DrJosh9000]

Would annotating the job pods with karpenter.sh/do-not-disrupt: "true" solve it for you? This would prevent Karpenter from disrupting each job, allowing them to complete.

From here: https://karpenter.sh/docs/concepts/disruption/#pod-level-controls

This can be set using config.default-metadata in the Helm values.yaml, e.g.

agentToken: "yourtokenhere"
config:
  default-metadata:
    annotations:
      karpenter.sh/do-not-disrupt: "true"

I haven't tested it but it should solve your problem?