[LRUG] Using statistical tables with Rails

[Richard Livsey]

 I'd recommend taking a look at mongodb [1], your use-case sounds like what it was made for and mongo is really easy to get up and running to play with. 

Because it's document oriented ('schema-less') you've got a lot of flexibility on how you structure the data you want to store. You're not stuck with a table with tonnes of NULL fields because not every event contains the same information for example.

It's very good at throwing large amounts of event/log style information into, whilst at the same time has atomic modifiers so you can build up aggregates as you go. You still get indexes and the ability to run ad-hoc queries, so you can just start by dumping data in there and work on the aggregate counters over time.

The raw Ruby driver is really nice and you don't need to use an ORM for most tasks, if you have a more complex model then it's worth looking at MongoMapper [2] or Mongoid [3].

Hope that helps!

[1] - http://mongodb.org 
[2] - http://mongomapper.com
[3] - http://mongoid.org

-- 
Richard Livsey
Co-Founder, MinuteBase 
Meeting collaboration made easy
http://minutebase.com
+44 (0) 7841 260 797

On Friday, 23 September 2011 at 11:37, Neil Middleton wrote:

> I'm building an app that needs to store a fair amount of events that the users carry out. (Think LOTS as in millions per month).
> 
> I need to report on the these events (total of type x in the last month, etc) and need something resilient and fast.
> 
> I've toyed with Redis etc to store aggregates of the data, but this could just mean that I'm building up a massive store of single figure aggregates that aren't rebuildable.
> 
> Whilst this isn't a bad solution, I'm looking at storing the raw event data in tables that I can then query on a needs basis, and potentially generate aggregate counters on a periodic basis. This would thus give me the ability to add counters over time, and also carry out ad-hoc inspections on what is going on, something which aggregates don't allow.
> 
> Question is, how is best to do this? I obviously don't want to have to create a model for each table (which is what Rails would prefer), so do I just create the tables and interact with raw SQL on a needs basis, or is there some other choice for dealing with this sort of data?
> 
> It would be interesting to know what thoughts you guys have.
> 
> Cheers
> 
> Neil
> 
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org (mailto:Chat at lists.lrug.org)
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org