[LRUG] TDD (was: Ruby Contracting)

[Eleanor McHugh]

On 19 Feb 2009, at 10:03, Stephen Bartholomew wrote:
> Hey all,
>
> Further to the last few replies, it would be interesting to hear  
> what the specific issues are with the TDD process.

Well the most obvious issue is that tests are themselves code and  
therefore prone to the same proportionate defect count as any other  
software: that's about one defect every three hundred SLOC for your  
average shrink-wrap application, one every 3 KSLOC for a well- 
engineered embedded system and one every 30 KSLOC for an aviation  
control system. Incidentally I've worked at all three levels of  
tolerance so when I sound contrarian it's not complete hooey, I am  
experienced ;)

If you build a test suite of 6 KSLOC for a shrink-wrap you'll have 20  
defects which may not seem like many until you realise that mainstream  
TDD lacks tools for identifying those defects. It's not self-testing.  
Now a defect isn't necessarily a bug in the 'this line of code is  
incorrect' sense although if they were it would make our lives much  
easier. Unfortunately it's often the result of misunderstood or  
missing requirements and that's beyond the competence of a machine to  
identify.

As Chris demonstrates in his post, in the hands of a skilled developer  
with experience this still isn't a problem: poor requirements leave  
certain kinds of code smells and once those are identified you can do  
another round of client interrogation and analysis. That's where  
manual testing enters the frame. But this is an aesthetic sense that  
many younger developers lack and which a heavy reliance on automated  
testing can prevent them developing.

You don't become a good programmer by writing code, you become a good  
programmer by reading it. Lots of it.

Writing meaningful tests is likewise a difficult aesthetic to develop.  
I started off with something of an advantage as I studied physics at  
university and spent a lot of time performing experiments (many of  
which were intensely dull) and analysing the results. When you do that  
day in and day out for three or four years you soon realise that  
accurate testing is a very different skill to framing hypotheses: the  
experimentalist and the theoretician both have their place and by only  
utilising the skills of one or the other science wouldn't progress  
very far. Our theoreticians are the analysts, architects whilst our  
experimenters used to be the QA testers. Programmers straddle the  
divide, but in general they're more on the theoretician side and our  
career path generally supports that.

As a result programmers are in general the worst people in the world  
to be testing code - at least their own code - as their theoretical  
understanding of what needs to be done can and will interfere with the  
impartiality of those tests. There are ways around that such as having  
developers write tests for one set of requirements and then write code  
to fulfil the tests written by one of their colleagues but that's only  
practical if they're either working in a relatively large team (like a  
ThoughtWorks pod) or a very disciplined environment.

Once you start moving to this approach though a lot more can be  
achieved by using collaborative offline code reviews (i.e. printing  
the code out and going through it by hand, marking up the bits that  
don't make sense with a nice bright highlighter). Most large companies  
that championed code reviews made them unworkable by tying them to  
performance metrics but I know quite a few old school hackers who  
never debug any other way. Indeed for some of the embedded systems  
I've worked on where the emulators were buggy as hell this was the  
only way to accurately reason about the code and much of what I take  
for granted as common instinct is the result of following that process.

So why should code reviews be more effective than tests? Surely tests  
likewise allow us to reason about code? Because code reviews get  
straight to the point: identifying bad code and doing something about  
it. Test suites on the other hand focus on the much more nebulous and  
costly process of identify good code.

Say we have a codebase which is 30 KSLOC and we want to turn it into  
something solid and reliable. Basic informal console debugging will  
reduce the defect count to 100 defects (1 every 300 SLOC), so from  
that point onwards any method for improving the code has to be  
proportionate to the impact of those 100 defects. Writing tests which  
cover all 30 KSLOC is going to result in a test suite of similar order  
of magnitude and perhaps that's a good investment.

But what if eliminating ninety percent of those defects can be  
achieved without a test suite (which in my experience it can) then the  
role of testing in a 30K codebase is restricted even further to  
finding just 10 defects. How well that expense can then be justified  
will vary from project to project, just as the defect count varies  
based on team experience, but I don't believe it's a priori  
demonstrable for all projects that a test suite is a desirable  
investment of developer effort.

I know there's an argument that such test suites make regression  
testing easier, so perhaps if we expand the design brief to specify  
that the system design must be mutable that alone becomes a good  
argument in favour of [B|T]DD methodologies. I'd personally say that  
depends on the level of coupling required by the system architecture -  
sometimes components need to be tightly coupled for performance  
reasons - but that's a different discussion.

I know that [B|T]DD can produce some amazing results when applied  
intelligently but as the de facto methodology of our age it is as  
blind and dumbheaded as the waterfall and SSADM. Ultimately this  
industry-wide obsessions are nothing more than ways of disguising a  
basic truth: programmers are a lot more like writers than machines and  
the quality of their code has more to do with personal passion than  
anything to do with established process.

A couple of years ago there was a lot of passion in Agile and [B|T]DD  
is one of the fruits of that. As usual early adopters received big  
initial gains compared to larger players and so everyone has rushed to  
join the bandwagon. These methodologies in the hands of someone who  
loves and nurtures them will work well but that tells us more about  
the developer than about the methodology.

And if that sounds highly cynical, that's because I am ;)


Ellie

Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net
----
raise ArgumentError unless @reality.responds_to? :reason