February 07, 2007
Ignored test are OK; too many ignored tests too long are a smell
I'm sometimes startled by zealots that insist that tests should never be ignored. Ignored tests are OK; too many ignored tests or long-ignored tests are a smell.
Tests marked as ignored are very useful: they remind you of work you need to do and allow you to check in work-in-progress without breaking the build.
The number of ignored tests is directly proportional to your code volatility. The greater the amount of work being done on your codebase the greater number of ignored tests are to be expected.
This implies that as you approach the end of your project ignroed tests should decrease. After the system is feature complete there should be very few ignored tests. Before going into production all tests should be enabled.
Long-Ignored Tests Are A Smell
Tests that have been ignored for more than a few days are a smell. Obsolete tests, unclear requirements, or just bad code hygene will lead to long-ignored tests.
To combat this I put the date on which I ignored the tests in the Ignore attribute and give precidence to those tests that have been left alone the longest.
[Ignore("Garrett: 20 Dec. waiting for requirements on final file format")]
public void ValidateOutputFileFormat() ...
Too Many Ignored Tests Are A Smell
If a large refactoring or requirements change requires a huge number of tests to be ignored it is typically a sign of either two things: tests which test the implementation rather than the contract of a class (behavioral mocking is a big culprit here) or tests which are not fine-grained enough.