[LRUG] e-petitions site

[steven shingler]

Thanks for that straight-up explanation, Chris. This has been a really
interesting thread.

Seems pretty clear then, sticking your outbound email in a background job
runner would sort it. Easy to miss when you're on a deadline - I'm actually
really impressed you had time to do the jmeter-ing before launch! :)

Quite a few of the posts on the thread have mentioned unicorn and nginx,
here's a write-up of how we're using them, together with bluepill for
monitoring and resque for background jobs:
http://www.devmull.net/articles/unicorn-resque-bluepill - hope that's
useful, would be cool to know if others have found better configs.

Cheers,
Steven


On Sun, Aug 7, 2011 at 5:02 PM, Chris Parsons <chris.p at rsons.org> wrote:

> Hi Paul,
>
> On 5 Aug 2011, at 09:33, Paul Robinson wrote:
>
> It would be interesting to hear what caused the e-petitions site to go
> under. What cacheing was in use? Which tier gave in first? Was the load
> profile unexpected in that it was biased to some corner of the app that
> nobody thought people would be much interested in?
>
>
> 1000/minute is less than 17 requests/second, and to my mind doesn't seem
> too absurd: almost 60ms per request on a single thread. A moderate server in
> terms of CPU with decent RAM, local MySQL instance and 4-6 Passenger threads
> should be able to handle that no problem at all, I think. So it would be
> interesting to hear which part of it failed first.
>
>
> Sorry for a bit of delay in responding to this thread. I'm planning to blog
> this soon, but here are the facts:
>
> First up, the site didn't 'crash' as was reported: we had a few stuck rails
> processes and the load balancer is only a basic round robin, so you
> sometimes had to refresh a few times to get your request through. The site
> stayed up throughout if you were persistent.
>
> The Rails processes got stuck waiting on SMTP connections for sending
> emails. As soon as the email sending load decreased, the site popped right
> back up.
>
> The main lesson is not to trust your hosting environment's SMTP server to
> be responsive, and ensure that you send emails asynchronously right out of
> the gate rather than making the whole request dependent on it. We didn't
> quite get time to do this as part of main dev, but next time I'd proactively
> break this out of the critical path.
>
> Re stress testing: We used JMeter to test the site load extensively, and
> pre-launch we were getting 50 req/second without much trouble. However, the
> one difference being that we were using dummy email addresses in our JMeter
> scripts, which the hosting environments SMTP service was discarding very
> quickly, and it could therefore take a much higher volume than in real life.
> Should've thought of that possibility, too :)
>
> Hope that clears a few things up: happy to answer any other questions
> people have.
>
> Thanks
> Chris
>
>
>
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>
>
> --
> Chris Parsons
> chris.p at rsons.org
> http://twitter.com/chrismdp
> http://pa.rsons.org
>
>
>
>
>
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20110807/c59f17d9/attachment-0003.html>