[LRUG] Making your tests run fast enough?

[Sam Livingston-Gray]

On Tue, Jan 24, 2012 at 1:57 AM, Joel Chippindale
<joel.chippindale at econsultancy.com> wrote:
> Test speed seems to be a perennial issue for our team.
>
> Currently running a single spec file in our rails (v3.0, running under REE)
> app takes about 20 seconds and running a single cucumber scenario 30+
> seconds. This is too slow for comfortable test driven development/design.
>
[snip]
>
> How fast are your tests/specs/cucumber? Are they fast enough for you? If
> they are, what have you done to make this so?

Hello, all-

Interesting thread!  Being 8 hours behind, I'll consolidate my
responses into one epic post.

--- Ruby Interpreters ---

TL;DR:
  case ruby_interpreter
  when :mri_192 then return
  when :mri_187 then switch_to_ree
  when :ree then optimize_gc_settings
  when :jruby, :rubinius then figure_out_how_to_disable_jit
  else i_have_no_idea
  end

About five months ago, I tried running the test suite for our biggest
app under a variety of different Ruby interpretations.  This app runs
on 1.8.7, and we'd switched to Bundler+RVM a month or two beforehand,
so it was relatively easy to try different Rubies.  Our baseline was
MRI 1.8.7p302.

I got the fastest results using REE and the GC parameters described in
this post (I tried tweaking GC params, but didn't get any results more
dramatic than the values listed):
http://smartic.us/2010/10/27/tune-your-ruby-enterprise-edition-garbage-collection-settings-to-run-tests-faster/

Results, sorted from slowest to fastest:

  JRuby 1.6.2:  I got bored waiting and killed the job.
  Rubinius 2.0.0pre:  16:02
  Rubinius 1.2.4:  15:04
  MRI 1.8.7 p302:  11:18  <--- baseline
  MRI 1.8.7 p302:  9:07  (running GC at minimum intervals of 1 second)
  REE-1.8.7-2011.03 (p334):  8:01
  MRI 1.8.7 p302:  7:50  (running GC at minimum intervals of 30 seconds)
  REE-1.8.7-2011.03 (p334) with tweaks:  5:30

NOTE:  JRuby and Rubinius both operate at a significant disadvantage
for tests.  Both use just-in-time compilation to optimize production
performance, which is actually a pessimization in a test suite!  In
production, you're likely to run a subset of execution paths over and
over, so the cost of doing the JIT is amortized across the relatively
long runtime of the Ruby process.  In your tests, you're (ideally)
executing a different path each time, so the compiler might JIT
something and then never use it again.

Ultimately, we switched to using REE 1.8.7 on our development
machines, while production executes on MRI 1.8.7.  In theory, this
could expose us to subtle platform-specific bugs; in practice, we
haven't noticed any... yet?

--- Amortizing Rails load time ---

It's not surprising that a single Cucumber spec takes ~30 seconds,
especially if it's using Selenium -- the cost of launching Rails
itself is high, and the cost of launching and controlling a browser is
significant.

RSpec (which I assume you're using) has a "--profile" command-line
flag that will print out the 10 slowest tests from your test run;

It might be an interesting experiment to compare the following:
- Time to run a single model spec             $ time rspec --profile
spec/models/towel.rb:42
- Time to run all specs for a single model    $ time rspec --profile
spec/models/towel.rb
- Time to run all model specs                 $ time rspec --profile
spec/models/

--- Controller tests ---

...are, by and large, a waste of time.  In keeping with "skinny
controller", logic from controllers should be pushed down into things*
that can be tested in isolation, and mocked out in most controller
tests.  One or two full-stack tests may be appropriate to cover
critical responsibilities that really do belong in your controller.
Also, if you have decent Cucumber coverage, those will (obviously)
exercise the full Rails stack, meaning they'll act both as acceptance
and integration tests.

* (Note: "things" does not necessarily mean models, and definitely
doesn't mean subclasses of ActiveRecord::Base; see the Facade pattern
for one possible approach.)

--- I/O vs. CPU ---

I'm fortunate enough to have an employer-provided MacBook Pro with
SSD.  Disk I/O is not the problem for us; your mileage may vary.  We
saw a ~2x speedup switching to REE with GC tweaks; I expect this is
mostly attributable to the high execution cost of garbage collection.

--- Testing Library ---

MiniTest is insanely fast, and RSpec 2.8 (released in the last few
weeks) is supposed to be considerably faster than 2.7.  I upgraded a
more recent project from 2.7 to 2.8, and saw test execution times drop
by a modest amount (<10%).  However, for most Rails projects, the
speed of even a slow testing framework is dwarfed by the cost of
loading and then executing Rails.

Which leads me to...

--- OOP vs. Rails ---

As I believe someone else noted in this thread, the Rails stack itself
is rather large; if you're walking all the way through it, you're
sacrificing a lot of execution time even if you never save your models
(though that, too, is low-hanging fruit you may be able to address,
especially if you're already using FactoryGirl or similar).

Corey Haines's approach of putting as much logic as possible into
mixins and testing them outside of the Rails context will give you a
dramatic speed boost, and *may* help you somewhat with design
improvements.  While I used to absolutely adore mixins and was
thrilled to see ActiveSupport::Concern included in Rails 3, I'm
starting to regard mixins as an insidiously evil feature of Ruby:
nothing else lets you violate the Single Responsibility Principle so
quickly and efficiently.  DCI (Data, Context, Interaction) helps
mitigate this somewhat; I haven't quite made up my mind on whether
it's useful in its own right, or whether it just seems better because
working in Rails for so long has warped my brain.  ;>

The single best resource I've seen to date is "Objects on Rails" by
Avdi Grimm (author of Exceptional Ruby).  This will eventually be a
free website, but you can pay USD$5 for early access, plus a few
extras once the site is ready for launch.  More here:
http://avdi.org/devblog/2011/11/15/early-access-beta-of-objects-on-rails-now-available-2/

I also started reading "Growing OO Software, Guided by Tests", which
looks promising, but bogged down when I hit the Java examples, which
I'd like to work along with while transliterating them into Ruby.
"Objects on Rails" has the advantage of being Ruby-specific, so can
make use of some advanced Ruby-fu.

Last but definitely not least...

--- Improving Acceptance Tests ---

With our largest app, we've committed a wide variety of grievous sins
in Cucumber tests and step definitions.  These include:  configuring
ActiveRecord models directly in step definitions, writing large and
complex step definitions, writing tests at a too-low level of
granularity (see
http://aslakhellesoy.com/post/11055981222/the-training-wheels-came-off),
putting non-acceptance tests in Cucumber just because that's the stack
we already had that could drive a browser (my personal favorite line:
"Then I should see 99 passing QUnit tests"), and more!

My team has taken a new approach on a recent app, and several months
in, it's still working quite well.  Basically, our Cucumber step
definitions are an extremely thin adapter layer that just calls into
an automation framework that exercises our application.  Wherever
possible, this framework talks to the app using its API over
Rack::Test to set up the "Given" state -- this can be an order of
magnitude faster than setting up the same state by driving a real
browser through the UI, and is especially useful for testing later
stages of a long and complex workflow.

More on the tool we've written at:
http://johnwilger.com/blog/2012/01/21/acceptance-and-integration-testing-with-kookaburra/

One happy and unanticipated (at least by me) side effect of using this
tool for acceptance testing is that it's also sped up integration
tests:  we've been able to reuse this driver in our RSpec-based
integration tests, giving us the ability to set up complicated model
state with a one-line #before block.

Hope this helps,
-Sam