◆ Problem
You have written a test with multiple assertions, and JUnit stops executing your test after the first assertion fails. You want to execute all the assertions, even if the first one fails.
◆ Background
This is not so much a problem with JUnit as it is a general misunderstanding of the way JUnit works or a misunderstanding of the philosophy behind JUnit’s design. Either way, you might have a fundamentally different notion of how JUnit ought to work when compared to the way it does work. JUnit was designed to fail a test at any point that an assertion fails.
◆ Recipe
You have written a test in such a way that you want to execute the entire test even after an assertion has failed. Seeing this, many JUnit practitioners would say,
“What you really have is multiple tests, so move each assertion into its own test.”
The issue is not having multiple assertions in one test, but rather wanting to con- tinue execution even after an assertion has failed. If the assertions are “different enough” that the failure of one does not render the others meaningless (for that test run), then we believe they belong in different tests. In this case, we recom- mend moving them to separate tests.
Follow these instructions to perform the required change:
1 Factor out your multiassertion test’s fixture into the appropriate instance- level fields and setUp() code.
2 Move each assertion into its own test method.
3 Remove the old multiassertion method.
We claim that your multiassertion “test” was really a series of tests that share a common fixture. The usual way to implement this in JUnit is to factor out the common fixture into the test case class as we have recommended here. You might find yourself extracting the newly formed tests in a separate test case class, particularly if your test case class contains more tests than just the one you have refactored. See recipe 3.4, “Factor out a test fixture,” for further discussion about this technique.
245 Your test stops after
the first assertion fails
If you really want to allow multiple assertions to fail in a single test, you need to modify a considerable amount of code, so be prepared to dig in and “get a little dirty” with the JUnit source. We outline one approach here:
1 Change junit.framework.TestResult.run(TestCase test) so that when it invokes TestCase.runBare(), it passes a reference to itself (this) to the TestCase object.
2 Due to the previous change, you need to add the method TestCase.run- Bare(TestResult testResult). This method needs to pass the TestResult object along when it invokes TestCase.runTest(). (Still with us?)
3 Due to the previous change, you need to add the method TestCase.run- Test(TestResult testResult). This method does essentially the same thing as TestCase.runTest(), but its InvocationTargetException handler adds a failure to the TestResult object rather than rethrow the originat- ing exception.
This should do the trick. Notice that we say should because we have never tried it.
We have never tried it because we have never wanted this feature. Such is the nature of open source.8
◆ Discussion
First, let us explain our recommendation, because you might feel that it is a com- paratively large amount of work for not much gain. For this, remember that JUnit was created to support fine-grained, object-level testing in the style of Test- Driven Development. As such, the goal is to focus each test on a single, predict- able behavior. This leads to a larger number of shorter tests with (at times) a con- siderable amount of common test fixture.9 This kind of test is desirable because it promotes orthogonality: being able to determine the problem by identifying the specific failing test. If each test verifies a single aspect of the system’s behavior then it ought to be easier—and it generally is—to pinpoint the problem behind a failing test. In addition, the TDD style of programming naturally lends itself to many tests, each adding features incrementally, so the tendency is for a TDD
8 Open source is built on the notion that he who has an itch ought to be the one to scratch it. Karl Fogel, Open Source Development with CVS (freeware document) 9. http://cvsbook.red-bean.com/OpenSourceDev WithCVS_2E. tar.gz
9 We mean the same fixture as the starting point for each test, and not the case where the second test relies on the results of the first test. Total test isolation is still important!
246 CHAPTER 8
Troubleshooting JUnit
practitioner to have many small tests, just as a matter of course. We understand that not everyone is a TDD practitioner, but by the same token you need to remember that the people who created JUnit are TDD practitioners. It is simply human nature for a person to build software that solves her problems according to her preferences, even if that goes against what others might do.
There are common situations in which you might want to write some code according to the techniques of TDD but still have multiple assertions fail in a test.
We will discuss one such situation here and describe an alternate approach that works with JUnit rather than against it. You might be able to find similar solutions to other, similar problems. Consider the case of building a web application. The server handles an HTTP request, passing it to the Controller, which selects the cor- responding business logic to execute. As the Controller executes this business logic, it collects any messages that the system decides it should display to the end user. After it has finished this, the Controller bundles these messages into an object.
It places that message collection on a web page template (such as a JSP or a Veloc- ity template) and the web page template processor generates the final web page to send to the end user as a response. This pattern is so commonplace in web applications that the Struts (http://jakarta.apache.org/struts) web application framework provides direct support for this. The test you want to write verifies the collection of messages for a particularly complex bit of business logic. Suppose the user of an e-commerce system attempts to check out, that is, submits his order for processing. In so doing, the following is true for this user:
■ He qualifies for a 10% discount if he purchases another $25 worth of merchandise.
■ He has an item in his shopping cart that might need to be back-ordered.
■ He has provided an invalid credit card number.
There are three messages to display to the end user, in addition to the other pro- cessing taking place. Your test focuses on those three messages. Your first thought is to verify these messages like so:
public void testMessagesForComplexCheckout() { // Set up shopcart...
// Submit order...
assertEquals("volume.discount", messages.getMessage(0).getKey());
assertEquals("maybe.backorder", messages.getMessage(1).getKey());
assertEquals(
"invalid.credit.card.number", messages.getMessage(2).getKey());
}
247 Your test stops after
the first assertion fails
Here you are comparing the message keys you expect to the ones in the corre- sponding position in the list of messages. After you write this test you decide that perhaps it is not a good idea to make an assumption about the order in which the messages appear. This is a brittle test, as it stands: if the messages come out in a different order, the test fails, even though the behavior is as expected. This is a case where your expectations are too specific. All that matters is that each message show up once and that there be three messages. If both those things are correct, then the test should pass. You rewrite the test to look like the following:
public void testMessagesForComplexCheckout() { // Set up shopcart
// Submit order
Collection messageKeys = messages.getKeys();
assertTrue(messageKeys.contains("volume.discount"));
assertTrue(messageKeys.contains("maybe.backorder"));
assertTrue(messageKeys.contains("invalid.credit.card.number"));
assertEquals(3, messageKeys.size());
}
This is clearly better: now the messages can come in any order and the test will pass. When you execute this test, it fails, because the volume discount message does not show up. Once you add the production code to make that pass, the test continues to fail because the credit card number message does not show up. You add the production code to fix that problem then the test still fails because you are adding each message twice.10 Finally you fix this last problem and the test passes.
You look back and think, if I could have seen all those failures at once, I could have fixed all those problems at once. That would have been better. We happen to think it is better to concentrate on one problem at a time, but apart from that, there is an even bet- ter way to write this test that shows all the problems at once and does not require multiple failures for a single test. Use the technique we recommend in recipe 2.9,
“Let collections compare themselves”. Rather than verify each item in a collection for correctness, we build the collection we expect, then compare that to the col- lection we get. The key here is the kind of collection you build: because you don’t care about the order of the messages and you don’t want any duplicate messages, you want to use a Set rather than a List. We rewrite the test as follows:
public void testMessagesForComplexCheckout() { // Set up shopcart
// Submit order
Set expectedMessageKeys = new HashSet(Arrays.asList(new Object[] { "volume.discount",
10If you wondered why we checked the number of messages, that’s why.
248 CHAPTER 8
Troubleshooting JUnit
"maybe.backorder",
"invalid.credit.card.number"
}));
Set actualMessageKeys = new HashSet(messages.getKeys());
assertEquals(expectedMessageKeys, actualMessageKeys);
}
The way we have constructed our set of expected message keys might look odd, so here is an explanation. First, we want a Set, so we build a HashSet, the “default”
implementation of Set. Next, we want a simple way to define the list of message keys we expect, and the only way to do that in one line of code is by building an array. To build a Set requires invoking add() multiple times, which is a consider- able amount of duplication. Java does not provide a built-in way to get from an array to a Set, but it does provide a way to build a List from an array:
Arrays.asList(). Java also provides a way to go from virtually any collection class to another: in this case, making a Set from a List. So at last we build a HashSet from a List from an array. This is the only way we know to do this without the unwieldy duplication of invoking add() multiple times.11
Now it is easy: two Sets are equal if they contain the same entries. We use assertEquals() to verify the contents of the checkout messages collection, and, if there are any differences, the failure message shows us a string representation of each collection. This allows us to see that we are missing two messages, and the one that is there is there twice. We can now take the necessary steps to fix each problem.
The recommendation here is to find ways to implement your tests that go along with JUnit’s design, rather than fight against its design. You might think that this is unfair: “My way or the highway,” you might call it. Far from it. If you want multiple assertion failures per test, you have a few choices: you can build the feature your- self12 or you can find an existing JUnit extension package that already does what you need. JUnitX (www.extreme-java.de/junitx) is one such project. We under- stand that sometimes you really do want multiple assertion failures per test. Shane Celis has this to say on the subject:
Having multiple failures per test allows me to provide much more detailed reports. In my environment, running each test is very expensive, so I want to get the most out of it as possible. If the assertion failure means the rest of my assertions aren’t run, I’m missing a lot of information. Initially, I broke them all up into separate tests, but that proved to be clumsy, so I modified the code
11Of course, we should hide this nonsense behind a creation method called makeSet() that takes an array. Once we get the test to pass, we can refactor it.
12Remember, this is open source.
249 Your test stops after
the first assertion fails
to allow for multiple failures (basically, a failed assertion wouldn’t throw an exception, it would simply be added to the TestResult’s failures), and writing the tests proved much easier and provided much more information than previ- ously available.
We have seen this before and it has always boiled down to the following: someone has a series of tests that are difficult or annoying to implement as separate test methods, that use some expensive external resource (such as a database), and which often fail because of frequent changes in the application’s environment (such as continual changes in the data in that database). These people reason that adding multiple assertion failures per test is the way to solve the problem. Let us offer another recommendation.
In this particular case, there are a number of tests that share a test fixture and are order dependent. We recommend:
1 Factoring out the test fixture (see recipe 3.4, “Factor out a test fixture”)
2 Building a custom test suite (see recipe 4.2, “Collect a specific set of tests”) that executes those tests in the required order
3 Using one-time setup (see recipe 5.10, “Set up your fixture once for the entire suite”) to put the shared fixture into the correct state before the test suite executes
4 Moving each assertion (or block of assertions) into its own test method We can derive all the benefits that Shane derives from adding the “multiple asser- tion failures per test” feature to JUnit. The difference? Our approach uses fea- tures that JUnit already provides, rather than adding to it. We can implement this kind of test using a smaller framework than Shane.13 All other things being equal, we prefer to use less code, because that way less can go wrong. Still, if you feel you need this feature, we recommend you try both and measure the difference. If you decide that multiple assertion failures per test is worth the cost, then implement it and share it with the community. If you do, please make it configurable so we can turn it off. Thanks.
◆ Related
■ 3.4—Factor out a test fixture
■ 4.2—Collect a specific set of tests
■ 5.10—Set up your fixture once for the entire suite
13This is not a knock on Shane. We like Shane, really.
250 CHAPTER 8
Troubleshooting JUnit