[LRUG] Queue-related war stories

[Najaf Ali]

Thanks for the responses everyone, gives me plenty to think about :-)

-Ali

On Mon, Mar 16, 2015 at 1:06 PM, javier ramirez <
javier.ramirez.gomara at gmail.com> wrote:

>  Hi,
>
>
> I'm trying to identify some general good practices (based on real-life
> problems) when it comes to working with async job queues (think DJ, Resque
> and Sidekiq).
>
>
> A classic, particularly with Rails, is queuing your messages AFTER SAVE,
> so in many cases the message will be enqueued and processed out of the
> queue before the COMMIT to database has been issued, causing all sorts of
> interesting errors and side-effects. This one is easily saved using after
> commit instead.
>
> Also, having messages that need to be processed on a given order, then
> having multiple consumers, so the order is not ensured.
>
> Another one I've seen often is not having separate queues/topics for
> different needs, or having separate queues, but then have all your
> consumers listening to all of the topics, so in effect it is like having a
> single one. I like to keep at least two types of queue/topic in my system.
> One for messages that should be processed as soon as possible and are fast
> to process, and one for things that can wait and might take more time
> (indexing a document, downloading a link and parsing it, rescaling a
> picture...). In a normal environment I keep at least 5 or 6 types of
> specialized consumers, so I can decide to scale up/down their number to
> improve a part of the system without affecting others.
>
> For developers not used to the async world, one of the walls I see being
> hit once and again is having your development environment (or even worse,
> integration/staging) working synchronously, then everything falling apart
> when you go live and async kicks in. If my system has an async queue on
> production, all my environments (including development) work asynchronously
> too.
>
> Cheers,
>
> j
>
> --
>
> javier ramírez
>
> javier ramirez's home page (http://javier-ramirez.com)
> <http://javier-ramirez.com>
> javier ramirez's blog (http://formatinternet.com)
> <http://formatinternet.com>
>
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
> Manage your subscription: http://lists.lrug.org/options.cgi/chat-lrug.org
> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20150318/d102e0a6/attachment-0003.html>