[LRUG] e-petitions site

[Tim Harding]

Warning: I'm mostly a programmer.

Sendmail can be configured locally on front end servers to store and forward
so you can talk to a local _process_ that can forward on to a centralised
mail gateway out. You don't have to make SMTP connections from your front
ends and you can queue mail locally and let your MTA take care of async
delivery.

Just a thought.

On 7 August 2011 20:52, Chris Parsons <chris.p at rsons.org> wrote:

> On 7 Aug 2011, at 18:29, steven shingler wrote:
>
> Thanks for that straight-up explanation, Chris. This has been a really
> interesting thread.
>
>
> Thanks. Also +1 to what Jolyon said: didn't see his email until after I
> wrote mine. He was "on point" through the whole day (I was away) and did a
> great job.
>
> Seems pretty clear then, sticking your outbound email in a background job
> runner would sort it. Easy to miss when you're on a deadline - I'm actually
> really impressed you had time to do the jmeter-ing before launch! :)
>
>
> That was Alan our PM/tester: he can work wonders with a jmeter config!
>
> Quite a few of the posts on the thread have mentioned unicorn and nginx,
> here's a write-up of how we're using them, together with bluepill for
> monitoring and resque for background jobs:
> http://www.devmull.net/articles/unicorn-resque-bluepill - hope that's
> useful, would be cool to know if others have found better configs.
>
>
> Nice. I'd love to share my chef setup at some point, pending clearance from
> GDS. Will possibly blog about that too in future.
>
> Chris
>
>
> Cheers,
> Steven
>
>
> On Sun, Aug 7, 2011 at 5:02 PM, Chris Parsons <chris.p at rsons.org> wrote:
>
>> Hi Paul,
>>
>> On 5 Aug 2011, at 09:33, Paul Robinson wrote:
>>
>> It would be interesting to hear what caused the e-petitions site to go
>> under. What cacheing was in use? Which tier gave in first? Was the load
>> profile unexpected in that it was biased to some corner of the app that
>> nobody thought people would be much interested in?
>>
>>
>> 1000/minute is less than 17 requests/second, and to my mind doesn't seem
>> too absurd: almost 60ms per request on a single thread. A moderate server in
>> terms of CPU with decent RAM, local MySQL instance and 4-6 Passenger threads
>> should be able to handle that no problem at all, I think. So it would be
>> interesting to hear which part of it failed first.
>>
>>
>> Sorry for a bit of delay in responding to this thread. I'm planning to
>> blog this soon, but here are the facts:
>>
>> First up, the site didn't 'crash' as was reported: we had a few stuck
>> rails processes and the load balancer is only a basic round robin, so you
>> sometimes had to refresh a few times to get your request through. The site
>> stayed up throughout if you were persistent.
>>
>> The Rails processes got stuck waiting on SMTP connections for sending
>> emails. As soon as the email sending load decreased, the site popped right
>> back up.
>>
>> The main lesson is not to trust your hosting environment's SMTP server to
>> be responsive, and ensure that you send emails asynchronously right out of
>> the gate rather than making the whole request dependent on it. We didn't
>> quite get time to do this as part of main dev, but next time I'd proactively
>> break this out of the critical path.
>>
>> Re stress testing: We used JMeter to test the site load extensively, and
>> pre-launch we were getting 50 req/second without much trouble. However, the
>> one difference being that we were using dummy email addresses in our JMeter
>> scripts, which the hosting environments SMTP service was discarding very
>> quickly, and it could therefore take a much higher volume than in real life.
>> Should've thought of that possibility, too :)
>>
>> Hope that clears a few things up: happy to answer any other questions
>> people have.
>>
>> Thanks
>> Chris
>>
>>
>>
>> _______________________________________________
>> Chat mailing list
>> Chat at lists.lrug.org
>> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>
>>
>> --
>> Chris Parsons
>> chris.p at rsons.org
>> http://twitter.com/chrismdp
>> http://pa.rsons.org
>>
>>
>>
>>
>>
>> _______________________________________________
>> Chat mailing list
>> Chat at lists.lrug.org
>> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>
>>
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>
>
> --
> Chris Parsons
> chris.p at rsons.org
> http://twitter.com/chrismdp
> http://pa.rsons.org
>
>
>
>
>
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20110808/cfe9e9e8/attachment-0003.html>