[LRUG] Making your tests run fast enough?

[Joel Chippindale]

Thanks for all the responses to this question, it gives me plenty to digest
and areas to investigate further, not least how we isolate more of our app
from rails.

J.

P.s. +1 for Garbage Collection tuning - we spent a few days tuning our GC
for REE early last year and we were amazed to discover how much time our
Ruby was spending in GC (60-70%). Tuning had a huge payoff in terms of
performance both in tests and on production (roughly halving render times
for production). This is well worth doing if you are running a version of
ruby which allows you to tune GC.




On 24 January 2012 18:25, Sam Livingston-Gray <geeksam at gmail.com> wrote:

> On Tue, Jan 24, 2012 at 1:57 AM, Joel Chippindale
> <joel.chippindale at econsultancy.com> wrote:
> > Test speed seems to be a perennial issue for our team.
> >
> > Currently running a single spec file in our rails (v3.0, running under
> REE)
> > app takes about 20 seconds and running a single cucumber scenario 30+
> > seconds. This is too slow for comfortable test driven development/design.
> >
> [snip]
> >
> > How fast are your tests/specs/cucumber? Are they fast enough for you? If
> > they are, what have you done to make this so?
>
> Hello, all-
>
> Interesting thread!  Being 8 hours behind, I'll consolidate my
> responses into one epic post.
>
> --- Ruby Interpreters ---
>
> TL;DR:
>  case ruby_interpreter
>  when :mri_192 then return
>  when :mri_187 then switch_to_ree
>  when :ree then optimize_gc_settings
>  when :jruby, :rubinius then figure_out_how_to_disable_jit
>  else i_have_no_idea
>  end
>
> About five months ago, I tried running the test suite for our biggest
> app under a variety of different Ruby interpretations.  This app runs
> on 1.8.7, and we'd switched to Bundler+RVM a month or two beforehand,
> so it was relatively easy to try different Rubies.  Our baseline was
> MRI 1.8.7p302.
>
> I got the fastest results using REE and the GC parameters described in
> this post (I tried tweaking GC params, but didn't get any results more
> dramatic than the values listed):
>
> http://smartic.us/2010/10/27/tune-your-ruby-enterprise-edition-garbage-collection-settings-to-run-tests-faster/
>
> Results, sorted from slowest to fastest:
>
>  JRuby 1.6.2:  I got bored waiting and killed the job.
>  Rubinius 2.0.0pre:  16:02
>  Rubinius 1.2.4:  15:04
>  MRI 1.8.7 p302:  11:18  <--- baseline
>  MRI 1.8.7 p302:  9:07  (running GC at minimum intervals of 1 second)
>  REE-1.8.7-2011.03 (p334):  8:01
>  MRI 1.8.7 p302:  7:50  (running GC at minimum intervals of 30 seconds)
>  REE-1.8.7-2011.03 (p334) with tweaks:  5:30
>
> NOTE:  JRuby and Rubinius both operate at a significant disadvantage
> for tests.  Both use just-in-time compilation to optimize production
> performance, which is actually a pessimization in a test suite!  In
> production, you're likely to run a subset of execution paths over and
> over, so the cost of doing the JIT is amortized across the relatively
> long runtime of the Ruby process.  In your tests, you're (ideally)
> executing a different path each time, so the compiler might JIT
> something and then never use it again.
>
> Ultimately, we switched to using REE 1.8.7 on our development
> machines, while production executes on MRI 1.8.7.  In theory, this
> could expose us to subtle platform-specific bugs; in practice, we
> haven't noticed any... yet?
>
> --- Amortizing Rails load time ---
>
> It's not surprising that a single Cucumber spec takes ~30 seconds,
> especially if it's using Selenium -- the cost of launching Rails
> itself is high, and the cost of launching and controlling a browser is
> significant.
>
> RSpec (which I assume you're using) has a "--profile" command-line
> flag that will print out the 10 slowest tests from your test run;
>
> It might be an interesting experiment to compare the following:
> - Time to run a single model spec             $ time rspec --profile
> spec/models/towel.rb:42
> - Time to run all specs for a single model    $ time rspec --profile
> spec/models/towel.rb
> - Time to run all model specs                 $ time rspec --profile
> spec/models/
>
> --- Controller tests ---
>
> ...are, by and large, a waste of time.  In keeping with "skinny
> controller", logic from controllers should be pushed down into things*
> that can be tested in isolation, and mocked out in most controller
> tests.  One or two full-stack tests may be appropriate to cover
> critical responsibilities that really do belong in your controller.
> Also, if you have decent Cucumber coverage, those will (obviously)
> exercise the full Rails stack, meaning they'll act both as acceptance
> and integration tests.
>
> * (Note: "things" does not necessarily mean models, and definitely
> doesn't mean subclasses of ActiveRecord::Base; see the Facade pattern
> for one possible approach.)
>
> --- I/O vs. CPU ---
>
> I'm fortunate enough to have an employer-provided MacBook Pro with
> SSD.  Disk I/O is not the problem for us; your mileage may vary.  We
> saw a ~2x speedup switching to REE with GC tweaks; I expect this is
> mostly attributable to the high execution cost of garbage collection.
>
> --- Testing Library ---
>
> MiniTest is insanely fast, and RSpec 2.8 (released in the last few
> weeks) is supposed to be considerably faster than 2.7.  I upgraded a
> more recent project from 2.7 to 2.8, and saw test execution times drop
> by a modest amount (<10%).  However, for most Rails projects, the
> speed of even a slow testing framework is dwarfed by the cost of
> loading and then executing Rails.
>
> Which leads me to...
>
> --- OOP vs. Rails ---
>
> As I believe someone else noted in this thread, the Rails stack itself
> is rather large; if you're walking all the way through it, you're
> sacrificing a lot of execution time even if you never save your models
> (though that, too, is low-hanging fruit you may be able to address,
> especially if you're already using FactoryGirl or similar).
>
> Corey Haines's approach of putting as much logic as possible into
> mixins and testing them outside of the Rails context will give you a
> dramatic speed boost, and *may* help you somewhat with design
> improvements.  While I used to absolutely adore mixins and was
> thrilled to see ActiveSupport::Concern included in Rails 3, I'm
> starting to regard mixins as an insidiously evil feature of Ruby:
> nothing else lets you violate the Single Responsibility Principle so
> quickly and efficiently.  DCI (Data, Context, Interaction) helps
> mitigate this somewhat; I haven't quite made up my mind on whether
> it's useful in its own right, or whether it just seems better because
> working in Rails for so long has warped my brain.  ;>
>
> The single best resource I've seen to date is "Objects on Rails" by
> Avdi Grimm (author of Exceptional Ruby).  This will eventually be a
> free website, but you can pay USD$5 for early access, plus a few
> extras once the site is ready for launch.  More here:
>
> http://avdi.org/devblog/2011/11/15/early-access-beta-of-objects-on-rails-now-available-2/
>
> I also started reading "Growing OO Software, Guided by Tests", which
> looks promising, but bogged down when I hit the Java examples, which
> I'd like to work along with while transliterating them into Ruby.
> "Objects on Rails" has the advantage of being Ruby-specific, so can
> make use of some advanced Ruby-fu.
>
> Last but definitely not least...
>
> --- Improving Acceptance Tests ---
>
> With our largest app, we've committed a wide variety of grievous sins
> in Cucumber tests and step definitions.  These include:  configuring
> ActiveRecord models directly in step definitions, writing large and
> complex step definitions, writing tests at a too-low level of
> granularity (see
> http://aslakhellesoy.com/post/11055981222/the-training-wheels-came-off),
> putting non-acceptance tests in Cucumber just because that's the stack
> we already had that could drive a browser (my personal favorite line:
> "Then I should see 99 passing QUnit tests"), and more!
>
> My team has taken a new approach on a recent app, and several months
> in, it's still working quite well.  Basically, our Cucumber step
> definitions are an extremely thin adapter layer that just calls into
> an automation framework that exercises our application.  Wherever
> possible, this framework talks to the app using its API over
> Rack::Test to set up the "Given" state -- this can be an order of
> magnitude faster than setting up the same state by driving a real
> browser through the UI, and is especially useful for testing later
> stages of a long and complex workflow.
>
> More on the tool we've written at:
>
> http://johnwilger.com/blog/2012/01/21/acceptance-and-integration-testing-with-kookaburra/
>
> One happy and unanticipated (at least by me) side effect of using this
> tool for acceptance testing is that it's also sped up integration
> tests:  we've been able to reuse this driver in our RSpec-based
> integration tests, giving us the ability to set up complicated model
> state with a one-line #before block.
>
> Hope this helps,
> -Sam
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lrug.org/pipermail/chat-lrug.org/attachments/20120125/8135b2fe/attachment-0003.html>