[LRUG] Multi-threading, Ruby & Rails

[Mr Jaba]

As no one else has mentioned it yet, I'm going to recommend both Sidekiq
and Celluloid. If you want to use Ruby Threading then Celluloid makes it
really easy to spawn a thread pool and monitor what's going on with
supervisors etc. It follows an Erlangy/Akka approach and is very easy to
use and setup if you wanted to test it out quickly.

Sidekiq is a worker based approach that is resque compatible, but uses
Celluloid. It's a lot more efficient than Resque as well (although I've got
no benchmarks I'm afraid) as it uses threads, not entire processes. As it
says in the docs you'll see some improvement even with MRI 1.9 but
Rubinius/JRuby would be better. If you are I/O bound though it'll still
help.

Some nice Railscasts to explain it better than me!
http://railscasts.com/episodes/367-celluloid
http://railscasts.com/episodes/366-sidekiq

Hope that helps a little


Tom

On 18 September 2012 08:52, Riccardo Tacconi <rtacconi at gmail.com> wrote:

> Rubinius supports multithreading too, not only JRuby. However Rubinius 2
> is still in beta and it is the only version supporting MRI 1.9 syntax. I
> tried ZeroMQ to distribute messages to avoid install other servers
> (RabbitMQ).
>
>
> On 18 September 2012 00:30, Tim Cowlishaw <tim at timcowlishaw.co.uk> wrote:
>
>> On 17 September 2012 22:38, Roland Swingler <roland.swingler at gmail.com>
>> wrote:
>>
>> >> b) Throw it out to cloud-like infrastructures like Hadoop/MapReduce,
>> but the problems needs direct SQL access and that can get messy
>> >
>> > I've not tried it and I don't know whether you need SQL or "SQL-like"
>> > but there are things like hive http://hive.apache.org/ built on top of
>> > hadoop that may be of some use?
>> >
>>
>> If I recall correctly, Hive provides a SQL-like querying layer on top
>> of information that's stored in a Hadoop (HDFS cluster), rather than
>> providing integration with a SQL db. However, there's a DB input
>> format [1] for hadoop that allows you to use the rows returned by a DB
>> query as the input to a mapreduce job which might be helpful in this
>> case. It depends a little on the complexity of the query - in my
>> fairly limited experience, doing complex joins can get rather messy
>> (although there are patterns for writing MR jobs that alleviate this -
>> Nathan Marz's 'Big Data' book [2] which is in Manning EAP at the
>> moment is in its infancy, but it looks like it's going to become a
>> good reference for this sort of stuff when it's published, as is their
>> Hadoop book [3])
>>
>> Of course, using Hadoop would mean embracing some Java-ish
>> infrastructure to a greater or lesser extent (you could use MRI ruby
>> to run your jobs with hadoop streaming, but hadoop itself is still a
>> Java tool. Alternatively you can use JRuby to access the Java apis
>> directly, and if you're going down this road then:
>>
>> > JRuby threads are Java threads, so you you get their benefits - i.e.
>> > proper use of all cores, no global interpreter lock.
>>
>> ...which might give you the performance increase you need without the
>> extra overhead of setting up and maintaining a hadoop cluster. If you
>> go don this route but are keen to use some sort of higher-level
>> concurrency primitive than threads, locks, mutexes etc then you might
>> want to take a look at akka [4], an erlang-ish   library for
>> actor-based concurrency on the JVM.  It's written and maintained by
>> Typesafe, the scala guys, but is usable from any other JVM language
>> too (and it looks like people have had some success using it with
>> JRuby [5]), so it might prove fruitful if you decide that JRuby's the
>> way you want to go.
>>
>> Hope this helps!
>>
>> Tim
>>
>> REFERENCES
>> ------------------
>>
>> [1]
>> http://hadoop.apache.org/docs/mapreduce/current/api/org/apache/hadoop/mapreduce/lib/db/DBInputFormat.html
>> [2] http://www.manning.com/marz/
>> [3] http://www.manning.com/lam/
>> [4] http://akka.io/
>> [5]
>> http://metaphysicaldeveloper.wordpress.com/2010/12/16/high-level-concurrency-with-jruby-and-akka-actors/
>> _______________________________________________
>> Chat mailing list
>> Chat at lists.lrug.org
>> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>
>
>
>
> --
> Riccardo Tacconi
> Ruby on Rails and PHP development - System Administration
> VIRTUELOGIC LIMITED <http://virtuelogic.net/>
>
> http://github.com/rtacconi
> http://riccardotacconi.blogspot.com
> http://twitter.com/rtacconi
>
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20120918/88ae9279/attachment.html>