[LRUG] User activity tracking

[Gerhard Lazu]

For the past 6 months I've been working for GoSquared, a real-time website
analytics service, and the available tech choices are pretty
straightforward.

*Redis* for real-time, short-term storage. It shines for aggregates and
counters in particular.

I've spent a fair amount of time with *Graphite* and there is no other open
source project which gives you more time series goodness out of the box.
MongoDB is nice and easy up to a certain point, but you find yourself
needing to write a lot of code to do something that Graphite does
brilliantly by default. This should help you get
started<https://github.com/gchef/graphite-cookbook>
.

Your biggest challenge will be the long-term storage and post-processing
(ie. map reduce). *Cassandra* is my favourite, but *MongoDB* will do the
job. Pre Mongo 1.8 map reduce was buggy and very difficult to work with,
but I've heard that since it got a lot better. Pick whatever you're most
comfortable with, they're both fine choices.

The challenge ahead of you is immense. Teams of many sharp developers have
been working on the same problems for many years now. If you'll want some
of the work cut out for you, there's always services with good APIs which
you can build on. My first recommendation will have to be gosquared.com (in
particular the February release), but chartbeat.com and gaug.es have their
strengths too. For something more DIY-ish, I would check out Librato's
metrics <https://metrics.librato.com/> and
Splunk<http://www.splunk.com/view/whats-new/SP-CAAAFD2>
.

Good luck!

Gerhard

------------------------------
Twitter <http://twitter.com/#!/gerhardlazu> Github<https://github.com/gerhard>
 Blog <http://gerhardlazu.com/>


On Wed, Jan 25, 2012 at 12:34 PM, Adam Carlile <adam at benchmedia.co.uk>wrote:

> Hola El Rug
>
> I've been tasked with creating a user activity tracking engine, in order
> to build more complete profiles on our users, their preferences and common
> activities.
>
> But I'm not really sure in which direction to go, I'm pretty sure that
> it's got to be a separate app, with a fast write database, something like
> MongoDB perhaps. Along with an exposed REST API to post and retrieve data.
>
> I also want the tracking engine to send calls to various other external
> services, such as the Facebook Graph API.
>
> It's something I've never tackled before so any advice or articles you
> would recommend would be great!
>
> Thanks
> Adam
>
> -
> *Twitter* @frozenproduce
> *
> *
>
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20120125/3603d3a2/attachment-0003.html>