[LRUG] Distributed delayed_job / resque

[Frederick Cheung]

On 3 Jul 2012, at 16:30, Ed James (Alt) wrote:

> Hi all
> 
> I've used both DJ and Resque before, so I'm familiar with both. However, I've never used them in a distributed architecture, where you have workers running on multiple machines. Resque seems to tout this as a feature a bit more then DJ, although the DJ docs say it is possible.
> 
> Has anyone had any experience in running either Resque or DJ in a distributed setup?
> 
> The primary reason for wanting to do this is for the ability to auto-scale (on AWS).
> 
I've done this with DJ. Most of the time we have one instance running a bunch of workers. A delayed job hook updates cloud watch with the number of items to be processed and a scaling policy attached to that starts extra instances when the backlog is too big.
We've never very large numbers of workers attached (we run delayed job on c1.medium instances so it doesn't take that many to fully utilise the instance) but we've not run into any particular problems. spreading the workers across multiple boxes doesn't really make any difference to how it works (other than the throughput of course). If you're having a look at job processors in general, sidekiq is also worth a look.

Fred

> Thanks.
> 
> -- 
> Ed James
> Sent with Sparrow
> 
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4399 bytes
Desc: not available
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20120703/05600ccd/attachment-0004.bin>