[LRUG] Fwd: Large Slow Test Suites

[Sam Livingston-Gray]

(Oops, forgot to reply all for posterity.)

---------- Forwarded message ----------
From: Sam Livingston-Gray <geeksam at gmail.com>
Date: Wed, Dec 4, 2013 at 3:03 PM
Subject: Re: [LRUG] Large Slow Test Suites
To: Mr Jaba <the.jaba at gmail.com>


Stopping the world (or setting aside one developer on a parallel track) to
rewrite a large existing body of tests for speed is almost certainly the
Wrong Thing To Do(tm).  With any significant legacy Rails app, I think the
most pragmatic thing you can do is to decide to change course, then commit
the entire team to implementing that change incrementally.

Here are some things you might try:
- Write fast (read: isolated) unit tests around new code.
- Add fast characterization tests around code you're about to change.
- Use a quick metaprogramming hack to mark some of the existing tests as
being part of a "full" test suite, with only a few essential tests as being
part of a "smoke test" or "happy path" test suite.

In fact, on that last point, something like this might be all you need
(plus, obviously, a few additional Rake tasks to set up the environment
appropriately):

class Test::Unit::TestCase
  def self.in_suite(suite_type, &block)
    return unless HandwavyEnvironmentThing.run_tests_of_type?(suite_type)
    class_eval &block
  end
end

With that in place, you can wrap "in_suite :full do ... end" or "in_suite
:smoke do ... end" around your existing test definitions, and Ruby will
just skip the bits you don't want based on the answer it gets from
HandwavyEnvironmentThing.  (Coming up with a better name for that is left
as an exercise for the reader.)

On one of the Rails projects I joined well into the "red giant" phase of
its life cycle, I noticed that there were already a few tests that were
marked as "fast", but that ran as part of the main suite.  I spent half a
day moving them out into their own Rake task "test:fast", made that task a
prerequisite of "test:units", and told my teammates about it.  It started
with about 20-30 tests, and now runs about about (one moment while I go run
them...) 1,500 tests in ~8 seconds of wall clock time (T::U reports 1.9
seconds).

A few months later, I also spent a highly enjoyable afternoon hacking out a
custom TestRunner class that would record the time spent by all tests,
determine the median execution time, and print out a warning with the names
of any tests that had taken more than 100x the median value.
 Unfortunately, people found that warning a little bit too easy to
ignore—if I had that to do over again, I'd probably make it actually fail
the build on slow fast tests.  (That code has since been disabled in favor
of a different test runner that offers pretty-printed results.  But I hope
a few people got the point.)  Writing a custom test runner isn't that hard
to do, and might give you enough information to figure out where your time
is going.  (I saw the word Capybara as other replies came in, though, so I
think that might be a suspect!  Full-stack tests are great, but slooooooow.)

When I say "fast" tests, what I really mean is "DO NOT LOAD RAILS" (or
anything else that crosses a process boundary or does network I/O).  Then,
when you find yourself typing "module ActiveRecord; class Base; end; end"
so you can test a new feature, stop, hit yourself over the head, and figure
out how you can add your feature using a small independent collaborator
that can interact with your AR model.  ;>  Then, back up your fast tests on
the collaborator with one or two model tests that exercise the collaborator
in the context of a full round trip to the database.

You'll probably find that you need to employ other strategies to make
things *better*, but this advice should at least help keep them from
getting exponentially *worse*.  :)

Hope this helps,
-Sam


On Wed, Dec 4, 2013 at 2:20 PM, Mr Jaba <the.jaba at gmail.com> wrote:

> Hi Everyone,
>
> I've recently taken ownership of a new project with a large test suite
> (2000ish tests), and the overall run time is around 30 minutes which is
> certainly less than ideal!
>
> Now I know the general approach to "Fast Rails Tests" but taking the time
> to refactor the whole test suite is a bit too much right now. I'm wondering
> if anyone has experience of transforming a test suite of this magnitude to
> something a bit speedier? If so how did you go about it? What tips, tricks
> and techniques can you share?
>
> A bit more info:
>
> - Test::Unit tests, moving to Minitest
> - Rails 3.2.16
> - Running parallel_tests shaved 5 mins off the time
> - Using Spring to reduce Rails load time.
>
> Any advice gratefully received!
>
> Tom
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20131204/966c1ebe/attachment.html>