Error observed: AirflowRuntimeError in airflow version 3.1.2

[pruyai]

Hello people , I am facing an issue in my airflow code. I am working with celery executer and airflow version 3.1.2. It seems like when multiple tasks are rendering fields in parallel, the insert conflicts and raises the error above. I have once observed this happening when single task tries to execute as well but usually seen with parallel executions.

AirflowRuntimeError: API_SERVER_ERROR: {'status_code': 409, 'message': 'Server returned error', 'detail': {'detail': {'reason': 'Unique constraint violation', 'statement': 'INSERT INTO rendered_task_instance_fields (dag_id, task_id, run_id, map_index, rendered_fields, k8s_pod_yaml) VALUES (%(dag_id)s, %(task_id)s, %(run_id)s, %(map_index)s, %(rendered_fields)s, %(k8s_pod_yaml)s)', 'orig_error': 'duplicate key value violates unique constraint "rendered_task_instance_fields_pkey"

This issue seems to be intermittent and happens with multiple tasks, sometimes even 3 attempts for task resolution results into the same error causing entire dag failure.

Is this a known issue and would be resolved sooner in next couple of releases? Could you please guide if there is any way to fix this up. Any suggestions would be highly appreciated.

[pavelekshin]

same as #53905

[bkarankar]

This looks like a race condition around inserts into rendered_task_instance_fields, where multiple processes attempt to write the same (dag_id, task_id, run_id, map_index) record concurrently. We've seen similar behavior in highly parallel environments (e.g., CeleryExecutor with high worker concurrency).

As a workaround, try disabling rendered task field storage:

AIRFLOW__CORE__STORE_RENDERED_TI_FIELDS=False

Also check your Airflow version, scheduler/API server replica count, and worker concurrency settings. It would be helpful to know whether this is reproducible on the latest Airflow patch release, as there may already be a fix or ongoing work related to this race condition.