◆ Problem
You have a test case class with some tests that need to execute in a particular order and others that do not. You want to specify the order of the order-dependent tests without worrying about the order of the rest of the tests.
◆ Background
If you follow the instructions in recipe 4.2, “Collect a specific set of tests,” then you are forced to specify the order of all the tests in your test case class; otherwise, you remain at the mercy of JUnit’s arbitrary ordering of your tests.20 This can be annoying if only 3 of your 22 tests need to execute in a certain order. What you would really like to do is identify the order of a few tests, and let JUnit do the rest.
Because JUnit doesn’t seem to handle this situation very well, it is worth asking,
“How did we get into this mess in the first place?” Actually, it’s not very unusual.
Let’s say you have a suite of order-dependent tests. Maybe you’re writing tests for a legacy system, perhaps you have some End-to-End or Integration Tests in which multiple tests use a fixed fixture,21 or you might have inherited tests from an uninspired programmer—it could be anything. Your goal is to refactor the tests in the direction of total test isolation, considered a good practice among JUnit practitioners. If you build your test suite using the suite() method, you have no way to know whether your incremental changes are really moving you in the direction of test isolation, because the order of the tests in your suite() method determines the order in which JUnit executes the tests. You’re not get- ting that warm, fuzzy feeling that you’re headed in the right direction.
Even if you can’t refactor this way, you’d like to have as much test isolation as you can have, given the current tests. JUnit doesn’t seem to support what you’re trying to do.
◆ Recipe
GSBase to the rescue! Mike Bowler ran into this problem on a project and decided to write a test suite builder that guarantees the order of some tests but
20If you think you “know” the order in which JUnit automatically extracts tests from a test case class, then throw that information away as quickly as possible. JUnit makes no test-order guarantee. If you rely on its current implementation, and then it changes, you’ll wish you hadn’t “known” quite so much!
21Try saying that five times in a row.
124 CHAPTER 4 Managing test suites
leaves the rest to the order in which JUnit decides to execute them. He calls this an OrderedTestSuite. Using the OrderedTestSuite class is simple:
1 If your test case class does not already have a suite() method, create one.
Refer to recipe 4.2, “Collect a specific set of tests,” if you need detailed instructions.
2 Code your suite() method to create an OrderedTestSuite from your test case class, specifying as a list the names of the tests you wish to execute in a particular order. Return this OrderedTestSuite object.
It sounds simple, as it should, because it is. Here is an example:
public static Test suite() {
String[] orderDependentTests = new String[] { "testQueryWithNoAccounts",
"testInsert",
"testInsertAccountExists", "testQueryWithOneAccount", "testDelete"
};
return new OrderedTestSuite(
AccountDataStoreTest.class, orderDependentTests);
}
Let us look at how we arrived at this solution. Consider a test case class from an online banking application that verifies some basic operations on a data store for bank accounts. The person writing these tests decided it would be best to have the tests execute in a particular order; that way, each test does not need to set up and tear down its own fixture each time.22 After some trial and error, you have deter- mined that there are only five tests that need to run in a certain order:
1 Querying the accounts when there are none should return an empty collec- tion of accounts.
2 Inserting a new account should work.
3 Inserting the same account a second time should fail with a duplicate key exception.
4 Querying the accounts with some accounts in the data store ought to return those accounts.
5 Deleting an existing account should work.
22The performance benefit seems tempting, and the apparent simplicity of the tests seems tempting, but believe us, you will feel much more pain later on. See recipe 4.1, “Let JUnit build your test suite.”
125 Control the order of some of your tests
There are other tests, though, that could be run in any order:
■ Querying a certain account that isn’t in the data store should fail with
“object not found.”
■ Deleting an account that doesn’t exist should fail with “object not found.”
■ Inserting a number of accounts in a row and then performing a mass delete should work.
■ Performing a mass delete on no accounts should do nothing.
There may be more, but this second list of tests has a common fixture: an empty account data store. JUnit could execute them before or after executing the tests you care about—either works. By using OrderedTestSuite, JUnit can execute the first set of tests in the order you specify them and then execute the second set of tests in whatever order it chooses. Our example shows specifying the first set of tests as order-dependent but makes no explicit mention of the second set. It sim- ply lets JUnit do its job. This is exactly what we are looking for.
◆ Discussion
We have said before—and will say again—that you should strive for 100% test isola- tion. Every time we have decided that test isolation is not important, it eventually slapped us in the face and showed us who is boss. We also admit that it is not always easy to achieve total test isolation. In particular, we may not have it right now. Even if you are an avid Test-Driven Development practitioner, you may inherit some tests that are order dependent. You can use OrderedTestSuite as a tool to help you refactor in the direction of total test isolation. Here is the basic approach:
1 Create an OrderedTestSuite that specifies the order of all the tests in the test case class, just as they are specified in the current suite() method.
2 Select a test to fix, and remove its dependence on the behavior of the pre- ceding tests.
3 Remove the test you have fixed from the list of order-dependent tests.
4 Repeat until the list of order-dependent tests is empty.
5 Remove the suite() method, and let JUnit build the default test suite for you.
This may be time consuming, but it certainly works. Without OrderedTestSuite, you would not be able to perform this delicate code surgery incrementally with the tests as a safety net. Until you experience how calm you are when you make big changes with tests as a safety net, you don’t know how empowering it can
126 CHAPTER 4 Managing test suites
feel.23 Using OrderedTestSuite in this situation enables a gradual transition to total test isolation without the risk of breaking the system along the way. In this way, OrderedTestSuite is invaluable.
Remember that OrderedTestSuite executes the order-dependent tests first, then the rest. The rest of your tests may assume an empty test fixture—that is, one in which no data has been changed by a test. If the last of the order-dependent tests leaves your fixture in an unknown state, then the remaining tests just won’t work. To overcome this problem, add a fixture barrier at the end of the list of order- dependent tests. It is a barrier in that it protects the “good tests” from whatever havoc the “bad tests” might wreak. This is a method that cleans up the fixture, placing it in the state that the remaining tests expect. We usually call this method resetFixture(). Add this method to your class and code it to reset the fixture, as in this example:
public static Test suite() {
String[] orderDependentTests = new String[] { "testQueryWithNoAccounts",
"testInsert",
"testInsertAccountExists", "testQueryWithOneAccount", "testDelete",
"resetFixture"
};
return new OrderedTestSuite(
AccountDataStoreTest.class, orderDependentTests);
}
This change causes JUnit to call resetFixture() after executing the order-depen- dent tests you’ve specified, creating a sparkling clean environment for the rest of the tests to use. Once you have refactored the remaining tests to have total test iso- lation, remove resetFixture(). This solution is a hack, and JUnit will erroneously report resetFixture() as a passed test; but because it is a temporary solution you’re working toward removing, we promise not to call the JUnit Police.
It seems strange to view such a useful piece of software as something that you hope never to use and as something you want out of your code as soon as possible, but it is still an important tool in your arsenal as a JUnit practitioner.
23J. B. once spent 9 days changing 60% of a system because of massive database schema restructuring. The only reason it could be done in 9 days was because the system had a comprehensive suite of JUnit tests. Oth- erwise, large sections would had to have been rewritten, and no one knows how long that would have taken.
127 Build a data-driven test suite
◆ Related
■ 4.1—Let JUnit build your test suite
■ 4.2—Collect a specific set of tests