Jobs launching in k8s, but they are not removed when completed. #73691
Description
Helm Chart Version
1.9.1
What step the error happened?
During the Sync
Relevant information
airbyte version 2.0.1
helm 1.9.1
Im having an issue where my job pods arent being removed in k8s after the job completes. The workflow api server indicates that its unable to update the workload.
I had search for similar issues that mentioned dataplane_group_id updates on the workspace table. I have verified taht the dataplane group id is correct.
Relevant log output
2026-02-20 16:48:28,412 [io-executor-thread-2] ERROR i.a.w.s.WorkloadService(runningWorkload):274 - Failed to update workload (ac5b26bb-0d38-45df-9ae1-1d08d17ca666_184641_0_sync) to running. Current status is running.
2026-02-20 16:48:46,596 [io-executor-thread-2] ERROR i.a.w.s.WorkloadService(launchWorkload):223 - Failed to update workload (ea0f42fb-af32-4ccf-b812-6f7a61cb6336_184637_0_sync) to launched. Current status is launched.
2026-02-20 16:50:18,119 [io-executor-thread-5] ERROR i.a.w.s.WorkloadService(launchWorkload):223 - Failed to update workload (0fa6edda-f091-421a-9f84-ba1192430647_184647_0_sync) to launched. Current status is launched.
2026-02-20 16:50:31,168 [io-executor-thread-3] ERROR i.a.w.s.WorkloadService(launchWorkload):223 - Failed to update workload (e36a1e08-f12b-476f-9dbc-a235d83582a8_184646_0_sync) to launched. Current status is launched.
2026-02-20 16:50:35,251 [io-executor-thread-2] ERROR i.a.w.s.WorkloadService(runningWorkload):274 - Failed to update workload (e36a1e08-f12b-476f-9dbc-a235d83582a8_184646_0_sync) to running. Current status is running.
2026-02-20 16:50:48,278 [io-executor-thread-5] ERROR i.a.w.s.WorkloadService(launchWorkload):223 - Failed to update workload (950e27cf-b1e7-4ed3-ad50-d1ad216f981e_184642_0_sync) to launched. Current status is launched.
2026-02-20 16:50:52,520 [io-executor-thread-6] ERROR i.a.w.s.WorkloadService(runningWorkload):274 - Failed to update workload (950e27cf-b1e7-4ed3-ad50-d1ad216f981e_184642_0_sync) to running. Current status is running.
2026-02-20 16:51:12,464 [io-executor-thread-8] ERROR i.a.w.s.WorkloadService(launchWorkload):223 - Failed to update workload (4c14f733-4241-4c56-8618-52dfc6c51984_184660_0_sync) to launched. Current status is launched.
2026-02-20 16:51:12,509 [io-executor-thread-9] ERROR i.a.w.s.WorkloadService(launchWorkload):223 - Failed to update workload (6c31d8e6-380c-4bca-9f9c-e23c4e2c3c75_184662_0_sync) to launched. Current status is launched.
2026-02-20 16:51:24,507 [io-executor-thread-5] ERROR i.a.w.s.WorkloadService(launchWorkload):223 - Failed to update workload (d403fb47-75ee-4842-b4fe-60831f3cfb82_184657_0_sync) to launched. Current status is launched.
2026-02-20 16:51:28,530 [io-executor-thread-3] ERROR i.a.w.s.WorkloadService(runningWorkload):274 - Failed to update workload (d403fb47-75ee-4842-b4fe-60831f3cfb82_184657_0_sync) to running. Current status is running.
2026-02-20 16:51:53,698 [io-executor-thread-8] ERROR i.a.w.s.WorkloadService(launchWorkload):223 - Failed to update workload (fd44cc46-c4b1-40f6-a71c-162813f385ea_184649_0_sync) to launched. Current status is launched.
2026-02-20 16:52:34,922 [io-executor-thread-9] ERROR i.a.w.s.WorkloadService(launchWorkload):223 - Failed to update workload (d854c8f9-3afd-477a-8e4d-6696dcf104a0_184656_0_sync) to launched. Current status is launched.
2026-02-20 16:56:47,188 [io-executor-thread-8] ERROR i.a.w.s.WorkloadService(launchWorkload):223 - Failed to update workload (32d24a5a-a762-4723-8e6a-0e69feaf8e6d_184661_0_sync) to launched. Current status is launched.
2026-02-20 16:57:28,396 [io-executor-thread-6] ERROR i.a.w.s.WorkloadService(launchWorkload):223 - Failed to update workload (1d512491-4206-4b8d-b0ea-6c3805ce1805_184672_0_sync) to launched. Current status is launched.
2026-02-20 17:35:23,080 [io-executor-thread-3] ERROR i.a.w.s.WorkloadService(launchWorkload):223 - Failed to update workload (838467e4-3d48-41cd-a0cf-1a9443e6a99a_184648_2_check) to launched. Current status is launched.
2026-02-20 17:35:26,041 [io-executor-thread-1] ERROR i.a.w.s.WorkloadService(runningWorkload):274 - Failed to update workload (838467e4-3d48-41cd-a0cf-1a9443e6a99a_184648_2_check) to running. Current status is running.
2026-02-20 17:36:35,252 [io-executor-thread-6] ERROR i.a.w.s.WorkloadService(launchWorkload):223 - Failed to update workload (d76d6a24-d1d2-4cd9-8466-b82661ffd5d0_184657_2_check) to launched. Current status is launched.
2026-02-20 17:36:39,896 [io-executor-thread-2] ERROR i.a.w.s.WorkloadService(runningWorkload):274 - Failed to update workload (d76d6a24-d1d2-4cd9-8466-b82661ffd5d0_184657_2_check) to running. Current status is running.
2026-02-20 17:36:43,293 [io-executor-thread-10] ERROR i.a.w.s.WorkloadService(launchWorkload):223 - Failed to update workload (f8b19ec8-3b92-49bd-aed7-950b9ad8050c_184648_2_check) to launched. Current status is launched.
2026-02-20 17:36:47,499 [io-executor-thread-3] ERROR i.a.w.s.WorkloadService(runningWorkload):274 - Failed to update workload (f8b19ec8-3b92-49bd-aed7-950b9ad8050c_184648_2_check) to running. Current status is running.
2026-02-20 17:51:47,236 [io-executor-thread-6] ERROR i.a.w.s.WorkloadService(launchWorkload):223 - Failed to update workload (f8b19ec8-3b92-49bd-aed7-950b9ad8050c_184657_2_check) to launched. Current status is launched.
2026-02-20 17:51:50,736 [io-executor-thread-5] ERROR i.a.w.s.WorkloadService(runningWorkload):274 - Failed to update workload (f8b19ec8-3b92-49bd-aed7-950b9ad8050c_184657_2_check) to running. Current status is running.
2026-02-20 18:21:12,808 [io-executor-thread-10] INFO i.a.w.s.WorkloadService(failWorkload$lambda$11):193 - Workload a87d747a-9fdf-4e35-aa4e-f59624e9d52d_184658_0_sync is already failed. Failing an already failed workload is a noop
2026-02-20 18:21:22,664 [io-executor-thread-7] INFO i.a.w.s.WorkloadService(failWorkload$lambda$11):193 - Workload b000a466-2329-42ec-b06e-8087406d6a32_184664_0_sync is already failed. Failing an already failed workload is a noop
2026-02-20 18:21:35,108 [io-executor-thread-5] INFO i.a.w.s.WorkloadService(failWorkload$lambda$11):193 - Workload f7d55304-e4b5-49dc-9c32-3a856bfe0f73_184667_0_sync is already failed. Failing an already failed workload is a noop
2026-02-20 18:21:54,042 [io-executor-thread-3] INFO i.a.w.s.WorkloadService(failWorkload$lambda$11):193 - Workload d4b0e5b9-6ddd-4574-8009-350ef39278d9_184670_0_sync is already failed. Failing an already failed workload is a noopInternal Tracking: https://github.com/airbytehq/oncall/issues/11384