Implement Backoff

[dlecocq]

As @mreiferson pointed out, we should implement the backoff mechanism for failed job processing.

[dlecocq]

As described in the docs, we should throttle our consumption if message processing fails. This is easy to determine when using callbacks, but consuming messages through an iterator puts more responsibility on the consumer itself. A couple of ideas come to mind.

In the case of using an iterator, we could check it around the yield:

def __iter__(self):
    ...
    try:
        yield message
    except:
        # Throttle

In the case of using a map-like function, we could provide a Client.map which would use a threadpool, gevent.pool.Pool or callbacks in the case of tornado.

[mreiferson]

The other option would be to require that the user call message.{finish,requeue} to signal success/fail (it isn't clear what nsq-py should do with uncaught exceptions, but it could assume REQ for example).

In retrospect, I wish this is what we would have done with pynsq from the start. We're sort of stuck having to rely on (what I've come to believe) is a less-than-desirable situation of responding to return values unless you "enable async".

[dlecocq]

This is what qless does, incidentally, and it could definitely be doable. One the one hand, it seems like allowing messages to time out and automatically be requeued is viable.

[mreiferson]

The problem with just allowing messages to time out is that it could have the side effect of RDY state tracking inconsistencies.

That's why I think that just assuming REQ is probably best.