<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi,<br>
<br>
</div>
<blockquote
cite="mid:20150314171233.3668.8540@domU-12-31-39-0A-A0-4F"
type="cite">
<div dir="ltr"><br>
<div>I'm trying to identify some general good practices (based
on real-life problems) when it comes to working with async job
queues (think DJ, Resque and Sidekiq).</div>
<div><br>
</div>
</div>
</blockquote>
<br>
A classic, particularly with Rails, is queuing your messages AFTER
SAVE, so in many cases the message will be enqueued and processed
out of the queue before the COMMIT to database has been issued,
causing all sorts of interesting errors and side-effects. This one
is easily saved using after commit instead. <br>
<br>
Also, having messages that need to be processed on a given order,
then having multiple consumers, so the order is not ensured.<br>
<br>
Another one I've seen often is not having separate queues/topics for
different needs, or having separate queues, but then have all your
consumers listening to all of the topics, so in effect it is like
having a single one. I like to keep at least two types of
queue/topic in my system. One for messages that should be processed
as soon as possible and are fast to process, and one for things that
can wait and might take more time (indexing a document, downloading
a link and parsing it, rescaling a picture...). In a normal
environment I keep at least 5 or 6 types of specialized consumers,
so I can decide to scale up/down their number to improve a part of
the system without affecting others.<br>
<br>
For developers not used to the async world, one of the walls I see
being hit once and again is having your development environment (or
even worse, integration/staging) working synchronously, then
everything falling apart when you go live and async kicks in. If my
system has an async queue on production, all my environments
(including development) work asynchronously too.<br>
<br>
Cheers,<br>
<br>
j<br>
<br>
<div class="moz-signature">-- <br>
<p>javier ramÃrez</p>
<p> <a href="http://javier-ramirez.com" title="javier ramirez">javier
ramirez's home page (http://javier-ramirez.com)</a><br>
<a href="http://formatinternet.com" title="format internet :
ruby on rails blog by javier ramirez">javier ramirez's blog
(http://formatinternet.com)</a><br>
</p>
</div>
</body>
</html>