« Time to go home? Leave a broken test | Main | The Office »

February 07, 2007

Ignored test are OK; too many ignored tests too long are a smell

I'm sometimes startled by zealots that insist that tests should never be ignored. Ignored tests are OK; too many ignored tests or long-ignored tests are a smell.

Tests marked as ignored are very useful: they remind you of work you need to do and allow you to check in work-in-progress without breaking the build.

The number of ignored tests is directly proportional to your code volatility. The greater the amount of work being done on your codebase the greater number of ignored tests are to be expected.

ignore1.GIF

This implies that as you approach the end of your project ignroed tests should decrease. After the system is feature complete there should be very few ignored tests. Before going into production all tests should be enabled.

Long-Ignored Tests Are A Smell

Tests that have been ignored for more than a few days are a smell. Obsolete tests, unclear requirements, or just bad code hygene will lead to long-ignored tests.

To combat this I put the date on which I ignored the tests in the Ignore attribute and give precidence to those tests that have been left alone the longest.

[Test]
[Ignore("Garrett: 20 Dec. waiting for requirements on final file format")]
public void ValidateOutputFileFormat() ...

Too Many Ignored Tests Are A Smell

If a large refactoring or requirements change requires a huge number of tests to be ignored it is typically a sign of either two things: tests which test the implementation rather than the contract of a class (behavioral mocking is a big culprit here) or tests which are not fine-grained enough.

Posted by gsmith at February 7, 2007 07:16 AM

Comments

"Tests that have been ignored for more than a few days are a smell."

Is this a code smell or a hint that the requirements/specifications may have to be revisited. Can we claim to be feature complete with ignored tests in the system. By feature complete I mean a clean iteration close.
-shaun



p.s. I thought I would take a break from all that financial number crunching and come and say hi to you guys ;)

Posted by: Shaun at February 7, 2007 08:32 AM