[LRUG] Queuing systems

[Sam Livingston-Gray]

Very interesting responses on this thread.  I haven't read all the way
through Sean's yet, so apologize for any conceptual duplication.

A couple of the guys from NewRelic gave a presentation at pdxruby a
while back.  One of the things I found particularly memorable was
their performance numbers on their own front-end stats collector
(i.e., the thing that the NewRelic agent reports to).  They had
response times consistently in the 6-8ms range, which was a bit
jaw-dropping... but they explained that that process (a) is written in
Java, and (b) basically just takes data from the wire and stuffs it
directly into the database with as little processing as they can get
away with (probably just the requisite type conversion and
marshaling).

Apparently they talked about using a queue to handle these, but that
was overridden by their head of engineering, on the basis that if they
didn't have capacity to *accept* the jobs fast enough, there was no
reason to think that they were going to be able to process them in
time either.

(Side note:  if you're as allergic to Java as I am, you might consider
writing the front end to this in node.js or EventMachine; those
event-based systems are specifically designed for directing large
volumes of I/O traffic.)

Anyway, if you're concerned about infrastructure (and rightly so; I've
never worked in a shop that had a proper queuing system, because setup
was a pain), you might look into Amazon's Simple Queueing Service.
I've used it for small apps in the past, and it's got a reasonable API
and tolerable semantics... and, of course, the fact that you don't
have to do any IT chores makes it easy to experiment with.

Ruby Rogues did an episode about queues and background processing
about a month ago:
http://rubyrogues.com/queues-and-background-processing/

More on Redis/Resque in a separate reply.

-Sam




> Date: Wed, 7 Sep 2011 10:14:23 +0100
> From: Neil Middleton <neil.middleton at gmail.com>
> To: London Ruby Users Group <chat at lists.lrug.org>
> Subject: [LRUG] Queuing systems
>
>  Simply put - I've got a requirement to capture a crap load of data from an API and stuff it all into a queue to be processed later on. We would be receiving hundreds of new items per second, over a sustained period.
>
> We're looking at hosting on EngineYard so ideally need to stick with what's available there.
>
> We cannot really lose any of the received data before processing.
>
> Any ideas?
>
> Neil