[LRUG] Queuing systems

[Chris Rode]

Yes something like that. But that is dependent on what you are doing with it and your environment 

The only holes that I have found is when the worker itself dies, or is terminated, or if there are network issues. Unfortunately this can be common especially in a dynamic environment. 

Resque relies on the worker node re-queuing the failed job, so if a worker is prevented from excuting that path somehow, the job will simply disappear. 

AMQP in transaction mode simply posts the job to another worker if the original worker goes offline of NACKs it. Making the queue itself responsible for fault tolerance, the worker is free to fail in whatever way it sees fit. 

Idempotence of tasks becomes important at this stage however...


On 07/09/2011, at 12:04, Neil Middleton <neil.middleton at gmail.com> wrote:

> Incidentally - I've seen a few people mentioning the Resque tolerance thing.  What sort of numbers can you expect here?  Are we talking 99.9% OK or what?
> 
> Neil
> On Wednesday, 7 September 2011 at 10:38, Chris Rode wrote:
> 
>> Ultimately it comes down to how scalable and how important each message is to you. 
>> 
>> If losing a message here or there is not palatable, you will need to look at an AMQP compliant solution (we have clocked reads and writes on RabbitMQ at 1000's per second).
>> 
>> Another option is the redis-backed resque which is great. It is not guaranteed to be fault tolerant though due to the fact that redis is not designed as a message queuing system, although it goes to pains to reduce the loss. 
>> 
>> I hope that helps
>> 
>> Chris
>> 
>> On 07/09/2011, at 11:18, Neil Middleton <neil.middleton at gmail.com> wrote:
>> 
>>> Hundreds, possibly thousands of new jobs per second.
>>> 
>>> Neil
>>> On Wednesday, 7 September 2011 at 10:17, Graham Ashton wrote:
>>> 
>>>> On 7 Sep 2011, at 10:14, Neil Middleton wrote:
>>>> 
>>>>> a crap load of data from an API
>>>> 
>>>> How much is a crap load?
>>>> _______________________________________________
>>>> Chat mailing list
>>>> Chat at lists.lrug.org
>>>> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>> 
>>> _______________________________________________
>>> Chat mailing list
>>> Chat at lists.lrug.org
>>> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>> 
>> _______________________________________________
>> Chat mailing list
>> Chat at lists.lrug.org
>> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
> 
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20110907/bf42d82a/attachment-0003.html>