The benefits of using <strong>unit tests</strong> are recognized throughout the software development industry. Unit tests help us verify that written code works as intended, prevent regression bugs, and can even be used as a design methodology (TDD). The problem with using unit tests begins when previously written tests fail to run or even compile.
There are valid reasons for unit tests to fail - namely a software bug was introduced or a feature changed. There are also quite a few wrong reasons why a certain test fails - most of which can be avoided by following a few simple guidelines.
In a perfect world, a test only breaks if the code under the tests stopped working. In the real world, poorly written tests fail just because the code changed.
As the overhead of fixing and re-writing these fragile tests increase, their effectiveness decreases. And so developers stop paying attention to these tests and no longer fix them. A few weeks down the road someone deletes the tests simply because they no longer compile.
How to Write Robust Unit Tests
The simple truth is that writing good unit tests is not exactly like writing good code. Some of the principles are similar while others are completely different. Writing good unit tests that won't break on every single code change is not difficult and can be achieved easily by following a few simple practices:
1. Unit Tests Should be Atomic
A unit test should not be dependent on environmental settings, other tests, order of execution or specific order of running. Running the same unit test 1000 times should return the same result.
Using global state such as static variables, external data (i.e. registry, database) or environment settings may cause "leaks" between tests. The order of the test run should not affect the test result, and so make sure to properly initialize and clean each global state between test runs or avoid using it completely.
2. Unit Tests Should be Deterministic
As stated before, a test should return the same result no matter how many times it runs. Using multithreading related objects (i.e. timers, threads) as part of the test should be avoided where possible and managed properly when used to make sure that the test won't fail due to different scheduling.
Time related operations such as Sleep are to be avoided since it's difficult or even impossible to make sure that a test won't fail due to high CPU utilization on the running machine.
Some developers write unit tests using random numbers/data in a test – introducing uncertainty to the test and making sure that when the test fails, it would be impossible to reproduce it since the test data changes every time it runs.
3. Know What You're Testing
There is nothing wrong with testing every aspect of a specific scenario/object. The problem is that developers tend to gather several such tests into a single method, creating very complex and fragile "unit tests".
One trick I found is to use the scenario tested and expected result as part of the test method name. When a developer has problems naming his test, it means that she's not sure what is being tested.
Testing only one thing has the additional benefit of a more readable test. When a simple test fails it's easier to find the failure cause and fix it than with a long complex test.
4. Mind the Test Scope
There is a direct correlation between the scope of the test and its fragility. Most tests have an object under test that responds or calls other objects as part of the test. The inner workings of these external dependencies are not important to the test and they can be faked. Using isolation (aka mocking), so that the test does not have to initialize and set objects that are used in the test but are not the actual object being tested, can solve this so that when one of these objects change it would not affect the test.
4. Test the What (Result) not the How
Since unit tests are usually written by the same developer that wrote the code and know how the solution was implemented, it is hard not to test the inner workings of how a feature was implemented.
The problem is that implementation tends to change and the test will fail even if the end result is the same.
Another related issue arises when testing internal/private methods and objects. There's a reason that these methods are private – they are not meant to be "seen" outside of the class and are part of the internal mechanics of the class. Only test private methods if you have a very good reason to do so, since trivial refactoring can cause complication errors and failures in the tests.
5. Avoid Over Specification
It's very tempting to create a well-defined, controlled and strict test that observes the exact process flow during the test, by setting every single object and testing every single aspect under test. The problem is that doing so "locks" the scenario under test preventing it to change in ways that do not affect the end result.
For example, don’t create a test that expects a certain method to be called exactly three times or those only two specific methods are called (and no other). There are reasons to write very precise tests but usually such micromanagement of test execution would only lead to a very fragile test. Use an Isolation framework to set default behavior of external objects and make sure that it's not set to throw an exception if an unexpected method was called (strict).
Writing good robust unit tests is not hard. It just takes a little practice. The list above is far from being complete but it outlines a few key points that would help you write better unit tests. And remember that if a specific test keeps failing - investigate the root cause and find a better way to test that feature.