[LRUG] Large Slow Test Suites

[Mr Jaba]

Wow, thank you all for such detailed responses! That's a whole lot of food
for thought. Hopefully we'll get the time down and maybe come and do a talk
on what approach we took!

Cheers

Tom


On 5 December 2013 08:07, Mark Burns <markthedeveloper at gmail.com> wrote:

> Another suggestion I just remembered that might help with the feedback
> loop at least, and I was surprised to only learn about it recently is zeus.
> https://github.com/burke/zeus/
>
> The tag line, if I had to think of one would be spork but less flakey.
>
>
> On 5 December 2013 07:50, Graham Ashton <graham at effectif.com> wrote:
>
>> On 4 Dec 2013, at 22:35, Mark Burns <markthedeveloper at gmail.com> wrote:
>>
>> > Was going to suggest the --profile flag until I realised that is just
>> for RSpec.
>> >
>> > Might sound like a silly question, but have you looked at profiling the
>> suite for the slowest ones? It could help two-fold, help spot the actual
>> tests that are taking up the time, or maybe even better, help spot the
>> pattern behind what is taking the most time across all the tests.
>>
>> You’ve reminded me of one of my favourite features of Tconsole (a
>> test-unit test runner).
>>
>> It has a !timings command that will report how long each of the tests you
>> just ran took, sorting the output so the slowest tests appear at the top.
>> You can see it in the video at around 2 minutes 50 seconds:
>>
>> http://vimeo.com/37641415
>>
>> Quite apart from showing timings, Tconsole really comes into its own when
>> you’re working with a slow test suite, giving you some of your productivity
>> back immediately.
>>
>> You can ask it to:
>>
>> - Only run the tests that failed in your last run of the suite
>> (especially useful after you’ve run a 30 minute test suite, to find that a
>> handful failed).
>>
>> - Run the tests for a list of test classes (e.g. just those that cover
>> the area you’re working on); you specify them by name on the command line.
>>
>> It also has a fast mode, where it stops as soon as it gets the first test
>> failure (though I find this less useful than running the specific tests
>> that are currently relevant, it’s great when you’ve got a lot of failures).
>>
>> In terms of a process for dealing with a 30 minute test suite, I’d
>> definitely be exploring speed improvements, but I’d investigate whether I
>> needed them all first.
>>
>> I’ve had great success with deleting less useful (or duplicated) tests,
>> and not just the slow ones; every test needs to give more value to the
>> project than the cost of keeping it around.
>>
>> What stage is the app at? Is it live with lots of customers, or is it
>> early days? Where the business is up to would have an impact on how
>> ruthless I’d be with carving off bits of the test suite.
>>
>> I once took over a project in which we deleted all the acceptance tests
>> on the first afternoon. There were loads of them, and they kept breaking as
>> we were adding new features. We were wasting a lot of time keeping them
>> running.
>>
>> The project was on an extremely tight budget, and keeping the acceptance
>> tests would have wasted a lot of the cash. Controller tests were written in
>> RSpec, so we turned on view rendering (which would catch any failures to
>> render the templates - the most likely failures caught by the integration
>> tests) and deleted the acceptance tests. Nothing went wrong, and we were
>> able to get a lot done quickly on a tiny budget. A few integration tests
>> were written later on to cover the happy path of the most important parts
>> of the site (e.g. the sign up pages).
>>
>> We might not have done it that way if we’d had thousands of users, but it
>> worked great in that context.
>>
>> Cheers,
>> Graham
>>
>> --
>> Graham Ashton
>> Founder, Agile Planner
>> https://www.agileplannerapp.com | @agileplanner | @grahamashton
>>
>> _______________________________________________
>> Chat mailing list
>> Chat at lists.lrug.org
>> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>>
>
>
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20131205/74ed517f/attachment-0003.html>