[LRUG] Queue-related war stories

[javier ramirez]

Hi,

>
> I'm trying to identify some general good practices (based on real-life 
> problems) when it comes to working with async job queues (think DJ, 
> Resque and Sidekiq).
>

A classic, particularly with Rails, is queuing your messages AFTER SAVE, 
so in many cases the message will be enqueued and processed out of the 
queue before the COMMIT to database has been issued, causing all sorts 
of interesting errors and side-effects. This one is easily saved using 
after commit instead.

Also, having messages that need to be processed on a given order, then 
having multiple consumers, so the order is not ensured.

Another one I've seen often is not having separate queues/topics for 
different needs, or having separate queues, but then have all your 
consumers listening to all of the topics, so in effect it is like having 
a single one. I like to keep at least two types of queue/topic in my 
system. One for messages that should be processed as soon as possible 
and are fast to process, and one for things that can wait and might take 
more time (indexing a document, downloading a link and parsing it, 
rescaling a picture...). In a normal environment I keep at least 5 or 6 
types of specialized consumers, so I can decide to scale up/down their 
number to improve a part of the system without affecting others.

For developers not used to the async world, one of the walls I see being 
hit once and again is having your development environment (or even 
worse, integration/staging) working synchronously, then everything 
falling apart when you go live and async kicks in. If my system has an 
async queue on production, all my environments (including development) 
work asynchronously too.

Cheers,

j

-- 

javier ramírez

javier ramirez's home page (http://javier-ramirez.com) 
<http://javier-ramirez.com>
javier ramirez's blog (http://formatinternet.com) 
<http://formatinternet.com>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20150316/4e53d119/attachment-0003.html>