[LRUG] Queue-related war stories

[Oto Brglez]

Hey!

Friend created some kind of geo-coding mechanism with sidekiq with caching.
He used singletons "pattern" in his geo-coder logic. All tests were green,
core review passed and everything was considered to be all right. Until
they deployed the code and they discovered that executed code gets shared.
He ended up wrongly geo-coding few million records. So. Be careful when
dealing with "threads", singletons and background workers on MRI.

Cheers!

- Oto


On Wed, Mar 18, 2015 at 7:18 PM Najaf Ali <ali at happybearsoftware.com> wrote:

> Thanks for the responses everyone, gives me plenty to think about :-)
>
> -Ali
>
> On Mon, Mar 16, 2015 at 1:06 PM, javier ramirez <
> javier.ramirez.gomara at gmail.com> wrote:
>
>>  Hi,
>>
>>
>> I'm trying to identify some general good practices (based on real-life
>> problems) when it comes to working with async job queues (think DJ, Resque
>> and Sidekiq).
>>
>>
>> A classic, particularly with Rails, is queuing your messages AFTER SAVE,
>> so in many cases the message will be enqueued and processed out of the
>> queue before the COMMIT to database has been issued, causing all sorts of
>> interesting errors and side-effects. This one is easily saved using after
>> commit instead.
>>
>> Also, having messages that need to be processed on a given order, then
>> having multiple consumers, so the order is not ensured.
>>
>> Another one I've seen often is not having separate queues/topics for
>> different needs, or having separate queues, but then have all your
>> consumers listening to all of the topics, so in effect it is like having a
>> single one. I like to keep at least two types of queue/topic in my system.
>> One for messages that should be processed as soon as possible and are fast
>> to process, and one for things that can wait and might take more time
>> (indexing a document, downloading a link and parsing it, rescaling a
>> picture...). In a normal environment I keep at least 5 or 6 types of
>> specialized consumers, so I can decide to scale up/down their number to
>> improve a part of the system without affecting others.
>>
>> For developers not used to the async world, one of the walls I see being
>> hit once and again is having your development environment (or even worse,
>> integration/staging) working synchronously, then everything falling apart
>> when you go live and async kicks in. If my system has an async queue on
>> production, all my environments (including development) work asynchronously
>> too.
>>
>> Cheers,
>>
>> j
>>
>> --
>>
>> javier ramírez
>>
>> javier ramirez's home page (http://javier-ramirez.com)
>> <http://javier-ramirez.com>
>> javier ramirez's blog (http://formatinternet.com)
>> <http://formatinternet.com>
>>
>> _______________________________________________
>> Chat mailing list
>> Chat at lists.lrug.org
>> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
>> Manage your subscription: http://lists.lrug.org/options.cgi/chat-lrug.org
>> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>
>>
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> Archives: http://lists.lrug.org/pipermail/chat-lrug.org
> Manage your subscription: http://lists.lrug.org/options.cgi/chat-lrug.org
> List info: http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20150318/d680825e/attachment-0003.html>