[LRUG] e-petitions site

[Chris Parsons]

On 7 Aug 2011, at 18:29, steven shingler wrote:

> Thanks for that straight-up explanation, Chris. This has been a really interesting thread.

Thanks. Also +1 to what Jolyon said: didn't see his email until after I wrote mine. He was "on point" through the whole day (I was away) and did a great job.

> Seems pretty clear then, sticking your outbound email in a background job runner would sort it. Easy to miss when you're on a deadline - I'm actually really impressed you had time to do the jmeter-ing before launch! :)

That was Alan our PM/tester: he can work wonders with a jmeter config!

> Quite a few of the posts on the thread have mentioned unicorn and nginx, here's a write-up of how we're using them, together with bluepill for monitoring and resque for background jobs: http://www.devmull.net/articles/unicorn-resque-bluepill - hope that's useful, would be cool to know if others have found better configs.

Nice. I'd love to share my chef setup at some point, pending clearance from GDS. Will possibly blog about that too in future.

Chris

> 
> Cheers,
> Steven
> 
> 
> On Sun, Aug 7, 2011 at 5:02 PM, Chris Parsons <chris.p at rsons.org> wrote:
> Hi Paul,
> 
> On 5 Aug 2011, at 09:33, Paul Robinson wrote:
> 
>> It would be interesting to hear what caused the e-petitions site to go under. What cacheing was in use? Which tier gave in first? Was the load profile unexpected in that it was biased to some corner of the app that nobody thought people would be much interested in?
>> 
>> 1000/minute is less than 17 requests/second, and to my mind doesn't seem too absurd: almost 60ms per request on a single thread. A moderate server in terms of CPU with decent RAM, local MySQL instance and 4-6 Passenger threads should be able to handle that no problem at all, I think. So it would be interesting to hear which part of it failed first.
> 
> Sorry for a bit of delay in responding to this thread. I'm planning to blog this soon, but here are the facts:
> 
> First up, the site didn't 'crash' as was reported: we had a few stuck rails processes and the load balancer is only a basic round robin, so you sometimes had to refresh a few times to get your request through. The site stayed up throughout if you were persistent.
> 
> The Rails processes got stuck waiting on SMTP connections for sending emails. As soon as the email sending load decreased, the site popped right back up. 
> 
> The main lesson is not to trust your hosting environment's SMTP server to be responsive, and ensure that you send emails asynchronously right out of the gate rather than making the whole request dependent on it. We didn't quite get time to do this as part of main dev, but next time I'd proactively break this out of the critical path.
> 
> Re stress testing: We used JMeter to test the site load extensively, and pre-launch we were getting 50 req/second without much trouble. However, the one difference being that we were using dummy email addresses in our JMeter scripts, which the hosting environments SMTP service was discarding very quickly, and it could therefore take a much higher volume than in real life. Should've thought of that possibility, too :)
> 
> Hope that clears a few things up: happy to answer any other questions people have.
> 
> Thanks
> Chris
> 
> 
> 
>> _______________________________________________
>> Chat mailing list
>> Chat at lists.lrug.org
>> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
> 
> --
> Chris Parsons
> chris.p at rsons.org
> http://twitter.com/chrismdp
> http://pa.rsons.org
> 
> 
> 
> 
> 
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
> 
> 
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org

--
Chris Parsons
chris.p at rsons.org
http://twitter.com/chrismdp
http://pa.rsons.org




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20110807/68bc0d19/attachment.html>