Thanks for all the responses to this question, it gives me plenty to digest and areas to investigate further, not least how we isolate more of our app from rails.<div><div><br></div><div>J.</div><div><br></div><div>P.s. +1 for Garbage Collection tuning - we spent a few days tuning our GC for REE early last year and we were amazed to discover how much time our Ruby was spending in GC (60-70%). Tuning had a huge payoff in terms of performance both in tests and on production (roughly halving render times for production). This is well worth doing if you are running a version of ruby which allows you to tune GC.</div>
<div><div><br></div><div><br></div><div><div><br><br><div class="gmail_quote">On 24 January 2012 18:25, Sam Livingston-Gray <span dir="ltr"><<a href="mailto:geeksam@gmail.com">geeksam@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On Tue, Jan 24, 2012 at 1:57 AM, Joel Chippindale<br>
<<a href="mailto:joel.chippindale@econsultancy.com">joel.chippindale@econsultancy.com</a>> wrote:<br>
> Test speed seems to be a perennial issue for our team.<br>
><br>
> Currently running a single spec file in our rails (v3.0, running under REE)<br>
> app takes about 20 seconds and running a single cucumber scenario 30+<br>
> seconds. This is too slow for comfortable test driven development/design.<br>
><br>
</div>[snip]<br>
<div class="im">><br>
> How fast are your tests/specs/cucumber? Are they fast enough for you? If<br>
> they are, what have you done to make this so?<br>
<br>
</div>Hello, all-<br>
<br>
Interesting thread! Being 8 hours behind, I'll consolidate my<br>
responses into one epic post.<br>
<br>
--- Ruby Interpreters ---<br>
<br>
TL;DR:<br>
case ruby_interpreter<br>
when :mri_192 then return<br>
when :mri_187 then switch_to_ree<br>
when :ree then optimize_gc_settings<br>
when :jruby, :rubinius then figure_out_how_to_disable_jit<br>
else i_have_no_idea<br>
end<br>
<br>
About five months ago, I tried running the test suite for our biggest<br>
app under a variety of different Ruby interpretations. This app runs<br>
on 1.8.7, and we'd switched to Bundler+RVM a month or two beforehand,<br>
so it was relatively easy to try different Rubies. Our baseline was<br>
MRI 1.8.7p302.<br>
<br>
I got the fastest results using REE and the GC parameters described in<br>
this post (I tried tweaking GC params, but didn't get any results more<br>
dramatic than the values listed):<br>
<a href="http://smartic.us/2010/10/27/tune-your-ruby-enterprise-edition-garbage-collection-settings-to-run-tests-faster/" target="_blank">http://smartic.us/2010/10/27/tune-your-ruby-enterprise-edition-garbage-collection-settings-to-run-tests-faster/</a><br>
<br>
Results, sorted from slowest to fastest:<br>
<br>
JRuby 1.6.2: I got bored waiting and killed the job.<br>
Rubinius 2.0.0pre: 16:02<br>
Rubinius 1.2.4: 15:04<br>
MRI 1.8.7 p302: 11:18 <--- baseline<br>
MRI 1.8.7 p302: 9:07 (running GC at minimum intervals of 1 second)<br>
REE-1.8.7-2011.03 (p334): 8:01<br>
MRI 1.8.7 p302: 7:50 (running GC at minimum intervals of 30 seconds)<br>
REE-1.8.7-2011.03 (p334) with tweaks: 5:30<br>
<br>
NOTE: JRuby and Rubinius both operate at a significant disadvantage<br>
for tests. Both use just-in-time compilation to optimize production<br>
performance, which is actually a pessimization in a test suite! In<br>
production, you're likely to run a subset of execution paths over and<br>
over, so the cost of doing the JIT is amortized across the relatively<br>
long runtime of the Ruby process. In your tests, you're (ideally)<br>
executing a different path each time, so the compiler might JIT<br>
something and then never use it again.<br>
<br>
Ultimately, we switched to using REE 1.8.7 on our development<br>
machines, while production executes on MRI 1.8.7. In theory, this<br>
could expose us to subtle platform-specific bugs; in practice, we<br>
haven't noticed any... yet?<br>
<br>
--- Amortizing Rails load time ---<br>
<br>
It's not surprising that a single Cucumber spec takes ~30 seconds,<br>
especially if it's using Selenium -- the cost of launching Rails<br>
itself is high, and the cost of launching and controlling a browser is<br>
significant.<br>
<br>
RSpec (which I assume you're using) has a "--profile" command-line<br>
flag that will print out the 10 slowest tests from your test run;<br>
<br>
It might be an interesting experiment to compare the following:<br>
- Time to run a single model spec $ time rspec --profile<br>
spec/models/towel.rb:42<br>
- Time to run all specs for a single model $ time rspec --profile<br>
spec/models/towel.rb<br>
- Time to run all model specs $ time rspec --profile<br>
spec/models/<br>
<br>
--- Controller tests ---<br>
<br>
...are, by and large, a waste of time. In keeping with "skinny<br>
controller", logic from controllers should be pushed down into things*<br>
that can be tested in isolation, and mocked out in most controller<br>
tests. One or two full-stack tests may be appropriate to cover<br>
critical responsibilities that really do belong in your controller.<br>
Also, if you have decent Cucumber coverage, those will (obviously)<br>
exercise the full Rails stack, meaning they'll act both as acceptance<br>
and integration tests.<br>
<br>
* (Note: "things" does not necessarily mean models, and definitely<br>
doesn't mean subclasses of ActiveRecord::Base; see the Facade pattern<br>
for one possible approach.)<br>
<br>
--- I/O vs. CPU ---<br>
<br>
I'm fortunate enough to have an employer-provided MacBook Pro with<br>
SSD. Disk I/O is not the problem for us; your mileage may vary. We<br>
saw a ~2x speedup switching to REE with GC tweaks; I expect this is<br>
mostly attributable to the high execution cost of garbage collection.<br>
<br>
--- Testing Library ---<br>
<br>
MiniTest is insanely fast, and RSpec 2.8 (released in the last few<br>
weeks) is supposed to be considerably faster than 2.7. I upgraded a<br>
more recent project from 2.7 to 2.8, and saw test execution times drop<br>
by a modest amount (<10%). However, for most Rails projects, the<br>
speed of even a slow testing framework is dwarfed by the cost of<br>
loading and then executing Rails.<br>
<br>
Which leads me to...<br>
<br>
--- OOP vs. Rails ---<br>
<br>
As I believe someone else noted in this thread, the Rails stack itself<br>
is rather large; if you're walking all the way through it, you're<br>
sacrificing a lot of execution time even if you never save your models<br>
(though that, too, is low-hanging fruit you may be able to address,<br>
especially if you're already using FactoryGirl or similar).<br>
<br>
Corey Haines's approach of putting as much logic as possible into<br>
mixins and testing them outside of the Rails context will give you a<br>
dramatic speed boost, and *may* help you somewhat with design<br>
improvements. While I used to absolutely adore mixins and was<br>
thrilled to see ActiveSupport::Concern included in Rails 3, I'm<br>
starting to regard mixins as an insidiously evil feature of Ruby:<br>
nothing else lets you violate the Single Responsibility Principle so<br>
quickly and efficiently. DCI (Data, Context, Interaction) helps<br>
mitigate this somewhat; I haven't quite made up my mind on whether<br>
it's useful in its own right, or whether it just seems better because<br>
working in Rails for so long has warped my brain. ;><br>
<br>
The single best resource I've seen to date is "Objects on Rails" by<br>
Avdi Grimm (author of Exceptional Ruby). This will eventually be a<br>
free website, but you can pay USD$5 for early access, plus a few<br>
extras once the site is ready for launch. More here:<br>
<a href="http://avdi.org/devblog/2011/11/15/early-access-beta-of-objects-on-rails-now-available-2/" target="_blank">http://avdi.org/devblog/2011/11/15/early-access-beta-of-objects-on-rails-now-available-2/</a><br>
<br>
I also started reading "Growing OO Software, Guided by Tests", which<br>
looks promising, but bogged down when I hit the Java examples, which<br>
I'd like to work along with while transliterating them into Ruby.<br>
"Objects on Rails" has the advantage of being Ruby-specific, so can<br>
make use of some advanced Ruby-fu.<br>
<br>
Last but definitely not least...<br>
<br>
--- Improving Acceptance Tests ---<br>
<br>
With our largest app, we've committed a wide variety of grievous sins<br>
in Cucumber tests and step definitions. These include: configuring<br>
ActiveRecord models directly in step definitions, writing large and<br>
complex step definitions, writing tests at a too-low level of<br>
granularity (see<br>
<a href="http://aslakhellesoy.com/post/11055981222/the-training-wheels-came-off" target="_blank">http://aslakhellesoy.com/post/11055981222/the-training-wheels-came-off</a>),<br>
putting non-acceptance tests in Cucumber just because that's the stack<br>
we already had that could drive a browser (my personal favorite line:<br>
"Then I should see 99 passing QUnit tests"), and more!<br>
<br>
My team has taken a new approach on a recent app, and several months<br>
in, it's still working quite well. Basically, our Cucumber step<br>
definitions are an extremely thin adapter layer that just calls into<br>
an automation framework that exercises our application. Wherever<br>
possible, this framework talks to the app using its API over<br>
Rack::Test to set up the "Given" state -- this can be an order of<br>
magnitude faster than setting up the same state by driving a real<br>
browser through the UI, and is especially useful for testing later<br>
stages of a long and complex workflow.<br>
<br>
More on the tool we've written at:<br>
<a href="http://johnwilger.com/blog/2012/01/21/acceptance-and-integration-testing-with-kookaburra/" target="_blank">http://johnwilger.com/blog/2012/01/21/acceptance-and-integration-testing-with-kookaburra/</a><br>
<br>
One happy and unanticipated (at least by me) side effect of using this<br>
tool for acceptance testing is that it's also sped up integration<br>
tests: we've been able to reuse this driver in our RSpec-based<br>
integration tests, giving us the ability to set up complicated model<br>
state with a one-line #before block.<br>
<br>
Hope this helps,<br>
-Sam<br>
<div class="HOEnZb"><div class="h5">_______________________________________________<br>
Chat mailing list<br>
<a href="mailto:Chat@lists.lrug.org">Chat@lists.lrug.org</a><br>
<a href="http://lists.lrug.org/listinfo.cgi/chat-lrug.org" target="_blank">http://lists.lrug.org/listinfo.cgi/chat-lrug.org</a><br>
</div></div></blockquote></div><br></div></div></div></div>