<div dir="ltr">Hey!<br><br>Friend created some kind of geo-coding mechanism with sidekiq with caching. He used singletons "pattern" in his geo-coder logic. All tests were green, core review passed and everything was considered to be all right. Until they deployed the code and they discovered that executed code gets shared. He ended up wrongly geo-coding few million records. So. Be careful when dealing with "threads", singletons and background workers on MRI.<div><br></div><div>Cheers!</div><div><br></div><div>- Oto<br><div><br></div></div></div><br><div class="gmail_quote">On Wed, Mar 18, 2015 at 7:18 PM Najaf Ali <<a href="mailto:ali@happybearsoftware.com">ali@happybearsoftware.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks for the responses everyone, gives me plenty to think about :-)</div><div dir="ltr"><div><br></div><div>-Ali</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 16, 2015 at 1:06 PM, javier ramirez <span dir="ltr"><<a href="mailto:javier.ramirez.gomara@gmail.com" target="_blank">javier.ramirez.gomara@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi,<br>
<br>
</div><span>
<blockquote type="cite">
<div dir="ltr"><br>
<div>I'm trying to identify some general good practices (based
on real-life problems) when it comes to working with async job
queues (think DJ, Resque and Sidekiq).</div>
<div><br>
</div>
</div>
</blockquote>
<br></span>
A classic, particularly with Rails, is queuing your messages AFTER
SAVE, so in many cases the message will be enqueued and processed
out of the queue before the COMMIT to database has been issued,
causing all sorts of interesting errors and side-effects. This one
is easily saved using after commit instead. <br>
<br>
Also, having messages that need to be processed on a given order,
then having multiple consumers, so the order is not ensured.<br>
<br>
Another one I've seen often is not having separate queues/topics for
different needs, or having separate queues, but then have all your
consumers listening to all of the topics, so in effect it is like
having a single one. I like to keep at least two types of
queue/topic in my system. One for messages that should be processed
as soon as possible and are fast to process, and one for things that
can wait and might take more time (indexing a document, downloading
a link and parsing it, rescaling a picture...). In a normal
environment I keep at least 5 or 6 types of specialized consumers,
so I can decide to scale up/down their number to improve a part of
the system without affecting others.<br>
<br>
For developers not used to the async world, one of the walls I see
being hit once and again is having your development environment (or
even worse, integration/staging) working synchronously, then
everything falling apart when you go live and async kicks in. If my
system has an async queue on production, all my environments
(including development) work asynchronously too.<br>
<br>
Cheers,<br>
<br>
j<span><font color="#888888"><br>
<br>
<div>-- <br>
<p>javier ramÃrez</p>
<p> <a href="http://javier-ramirez.com" title="javier ramirez" target="_blank">javier
ramirez's home page (http://javier-ramirez.com)</a><br>
<a href="http://formatinternet.com" title="format internet :
ruby on rails blog by javier ramirez" target="_blank">javier ramirez's blog
(http://formatinternet.com)</a><br>
</p>
</div>
</font></span></div>
<br>_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" target="_blank">http://lists.lrug.org/pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/options.cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
<br></blockquote></div><br></div>
______________________________<u></u>_________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org" target="_blank">Chat@lists.lrug.org</a><br>
Archives: <a href="http://lists.lrug.org/pipermail/chat-lrug.org" target="_blank">http://lists.lrug.org/<u></u>pipermail/chat-lrug.org</a><br>
Manage your subscription: <a href="http://lists.lrug.org/options.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/options.<u></u>cgi/chat-lrug.org</a><br>
List info: <a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/<u></u>listinfo.cgi/chat-lrug.org</a><br>
</blockquote></div>