-
Notifications
You must be signed in to change notification settings - Fork 16.6k
Description
Apache Airflow Provider(s)
standard
Versions of Apache Airflow Providers
apache-airflow-providers-standard==1.8.0
Apache Airflow version
2.11.0
Operating System
linux
Deployment
Other Docker-based deployment
Deployment details
No response
What happened
When using ExternalTaskSensor in reschedule mode, we need to use timeout field as explained in BaseSensorOperator
:param timeout: Time elapsed before the task times out and fails.
Can betimedeltaorfloatseconds.
This should not be confused withexecution_timeoutof the
BaseOperatorclass.timeoutmeasures the time elapsed between the
first poke and the current time (taking into account any
reschedule delay between each poke), whileexecution_timeout
checks the running time of the task (leaving out any reschedule
delay). In case that themodeispoke(see below), both of
them are equivalent (as the sensor is never rescheduled), which is not
the case inreschedulemode.
Given the docs section above, seems like all sensors should use timeout instead of execution_timeout to have a predictable behaviour in all modes.
But, when we set the sensor in deferrable mode, it passes execution_timeout to the base class defer implementation. This creates an undocumented difference in behaviour between poke and reschedule mode against the deferred mode.
What you think should happen instead
ExternalTaskSensor should use timeout first and fallback to execution_timeout for deferred execution timeouts.
How to reproduce
ExternalTaskSensor(
task_id='check_sample_task',
external_dag_id='sample_dag',
external_task_id='sample_task',
timeout=60,
poke_interval=300,
deferrable=True,
)Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct