[LRUG] Chat Digest, Vol 68, Issue 14

[Sam Livingston-Gray]

> Date: Wed, 7 Sep 2011 11:04:17 +0100
> From: Neil Middleton <neil.middleton at gmail.com>
> To: London Ruby Users Group <chat at lists.lrug.org>
> Subject: Re: [LRUG] Queuing systems
>
> Incidentally - I've seen a few people mentioning the Resque tolerance thing. What sort of numbers can you expect here? Are we talking 99.9% OK or what?
>
> Neil

The correct answer is "it depends on how crash-prone your workers are."  (=

Resque is designed for a different use case than what you're talking
about -- it solves a specific problem set that GitHub has, and as such
it embeds a number of assumptions based on their application and user
base.

Resque is consistent with the consensus opinion of the Ruby Rogues
that you should be able to rebuild the entire contents of your work
queue based on the contents of your database.  (In other words, queues
should operate on records that have associated state machines, and if
the queue falls over, you can just recreate all of the jobs in it
based on the states of their records.  Using a state machine also
gives you a nice way to provide idempotence; workers can abort a job
if they see it's not in the state they expect it to be in, either
before they start or when they attempt to write their results back to
the databse.)  If you use a high-volume front end to create records in
some initial state, and then use some other monitor process to
generate jobs for those records, you might be able to get away with
treating your queue as disposable.

As for durability, I know of two ways that Resque can lose jobs;
technically, both are Redis issues.  First, it's possible for a worker
to pop a job off the queue and then die.  I don't recall whether
Resque pops jobs off of one queue and pushes them onto a different "in
progress" queue, but even if it does, Redis doesn't offer that as an
atomic operation, so it would be possible to lose a job in that
handoff.  Second, Redis itself could lose data if it crashes before
writing to disk.  Redis' configuration options offer a tradeoff
between durability and performance that is thoroughly explained in
their documentation:  http://redis.io/topics/persistence

Hope this helps,
-Sam