Task timeouts

[RealOrangeOne]

One of the reasons to move a task to the background is to prevent it slowing down page requests. However, tasks shouldn't be allowed to run forever, and so should have a timeout.

How will timeouts interact with timeouts and retries (#6)?

[jribbens]

This can solve #46 of course...

[RealOrangeOne]

A timeout doesn't necessarily solve limbo tasks. It only solves it if the timeout is used to re-schedule a task with some kind of padding in-between. But that's a very separate feature.

[jribbens]

I'm not following you. It solves it because on timeout the task moves from RUNNING to FAILED.

[RealOrangeOne]

A "timeout" in this context is more about how long the task can be running. Retries are only solved if a task also has a "TTL", ie a point after which the task shouldn't be run (and thus marked failed as you say).

[jribbens]

Ok so it solves it because after the timeout the task will be killed, and everything continues as if it had thrown an exception.

[mgax]

Consider this code:

import time
from django_tasks.task import task

@task()
def highlander_1986():
    while True:
        try:
            time.sleep(100)
        except BaseException:
            pass

Worker.shutdown can raise SystemExit, or any exception really, but the function will just catch it and continue running.

AFAIK, the only reliable way to kill a runaway task is to first run it in a subprocess. Forking a subprocess for each task would be costly, but that might not be necessary; after a successful task, the subprocess can be reused.

[RealOrangeOne]

I've seen a few other implementations handle it in a somewhat sensible way. Because the worker process is only ever handling 1 command, we can do similar to what gunicorn does - terminate the process.

That is, have a separate thread (threading.Timer) running, and when the timeout is reached, mark the task as failed, and terminate the current process. That way, we can more easily guarantee the code actually stops, unless it's doing terrible things (eg setting signal handlers itself).

[mgax]

So then the whole worker process (./manage.py db_worker) stops and needs to be restarted, perhaps by the infrastructure? That seems a bit dramatic; if this happens too often, maybe the infrastructure decides that the worker is unstable and leaves it stopped. I'm not sure what gunicorn does, but maybe it keeps its parent process running, so it can restart workers that terminate themselves?

[RealOrangeOne]

The worker does indeed keep running, which I guess from an infrastructure perspective avoids that issue. I think RQ works in a similar way, but not sure what it does.

A task timing out shouldn't happen regularly anyway, which tends to be when most process managers get unhappy. With that said, if for whatever reason there isn't a process manager watching and capable of restarting, it'd be quite the foot-gun.

Using multiprocessing.Process as a wrapper should let the code be terminated consistently, at the cost of more memory usage and complexity. It might be necessary to opt-out of the multiprocessing (as you can with runserver) if needed, at which time timeouts would be ignored.

[mgax]

The worker does indeed keep running, which I guess from an infrastructure perspective avoids that issue. I think RQ works in a similar way, but not sure what it does.

Yep, it's using multiprocessing to run a pool of workers:
https://github.com/rq/rq/blob/5cfd853546485c9fbda36db38bd1873fa5c766c9/rq/worker_pool.py#L144-L164

A task timing out shouldn't happen regularly anyway

I'd push back on that. It's common to dispatch many similar tasks on a queue. If there's a systematic problem that causes timeouts, e.g. because the task needs to do several network calls and the network is particularly slow that day, it's likely to affect many tasks.