Better scheduling of jobs in case of server overloading

[casella]

Currently, we have a number of periodic tasks that are scheduled on a regular basis. If for some reason there is some server overloading (e.g. due to heavy on-demand jobs running), these tasks keep piling up in the task queue, until it becomes unmanageable.

One simple strategy to avoid this problem at the root is that before starting a new job, we check if a previous instance of the same job is still running; if it is, we skip it. By construction, this avoids task of the same type piling up on the queue. I would recommend implementing this as a first quick step to avoid the current chronic overloading of our server pool.

Another strategy is to avoid scheduling a new job if nothing has changed at the scheduling time compared to the previous run.