[LRUG] Distributed delayed_job / resque

[Kenneth Lee]

I'm using Resque using a cluster of 3+ with no problems and it's on EC2. 
Obviously there is a concern of possible data loss if outstanding jobs 
are lying in the queue in memory because of Redis.

Anything that is critical I tend to use a service in between to persist 
it to ensure something is "safe" before throwing it in the queue and an 
after job to go clean up the persisted queue once I've verified a job is 
safely completed. It's expensive though under load but you can have this 
only for "critical" stuff. DJ is better for this kind of thing, but in 
the end it's how you plumb together you can still use Resque and Redis.

To answer the question a distributed setup shouldn't make a difference 
for either DJ or Resque, the fact you can run multiple works on one 
machine, almost means you can run it on many machines.

Ken

Ed James (Alt) wrote:
> Hi all
>
> I've used both DJ and Resque before, so I'm familiar with both. 
> However, I've never used them in a distributed architecture, where you 
> have workers running on multiple machines. Resque seems to tout this 
> as a feature a bit more then DJ, although the DJ docs say it is possible.
>
> Has anyone had any experience in running either Resque or DJ in a 
> distributed setup?
>
> The primary reason for wanting to do this is for the ability to 
> auto-scale (on AWS).
>
> Thanks.
>
> -- 
> Ed James
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20120703/b3eba8a6/attachment-0003.html>