Do not execute removed cron workloads #17
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We need to check whether a workload was enqueued with a scheduler key, but no longer is in the cron table. If that is the case (we are trying to execute a workload which has a scheduler key, but the scheduler does not know about that key) it means that the workload has been removed from the cron table and must not run. Moreover: running it can be dangerous because it was likely removed from the table for a reason. Should that be the case, mark the job "finished" and return
nil
to get to the next poll. If the deployed worker still has the workload in its scheduler table, but a new deploy removed it - this is a race condition, but we are willing to accept it.Note that we are already "just not enqueueing" that job when the cron table gets loaded - but it is not enough.
If our system can be in a state of partial deployment:
...and we remove the crontabled workloads during app boot, it does not give us a guarantee that release 1 won't reenqueue them. For example, via the "reinsert next scheduled" feature when the job is executing. This is why this safeguard is needed.
This protects us from a very dangerous failure mode where we would remove an entry from the cron table, deploy the change, and then still have the workload run a day later.