◆ Problem
You want a real-time report of the tests as they execute, perhaps including the name of the test, the suite to which it belongs, and the result.
◆ Background
The text-based test runner that comes with JUnit provides only a compact and very brief report of the test run as it happens. Specifically, it prints a dot for each test it executes and adds an “E” for an error or an “F” for a failure. It does not report the name of each test as the execution proceeds. You might decide that
600 CHAPTER 16 JUnit-addons
you want this feature, if for no other reason than to have a sense that the tests you expect to execute are actually doing so.
Beyond this there is one situation we have encountered where we needed to know the names of the tests as they executed.3 If you write a test that uncovers a deadlock situation, it is impossible to use the JUnit text-based test runner to deter- mine which test reproduces the deadlock. At this point you have a few options, none of them particularly nice.
■ Use a graphical test runner, wait for the deadlock situation to happen, and then look at the test runner’s status bar. It shows you name of the last test to execute completely, from which you might be able to deduce which test contains the deadlock.4
■ Remove test suites from and add them back to your test run, hoping to iso- late the test suite that contains the offending test. From there you can remove test methods from and add them back to the one test suite, hoping to isolate the test method that contains the offending test. Binary search will not help you here: you cannot rely on the order of execution of tests.
■ Isolate the test suite that contains the offending test, as in the previous option, and then add System.out.println(getName()); into the setUp() method so that the name of the test prints to the console before the test executes. Remember to take it out when you have finished! Remember to put it back when you need it next time!
Forget it! None of these is useful on an ongoing basis, and each requires special- ized knowledge of JUnit to do. You want a solution that lasts, that’s easy to turn on and off and that anyone can use.
◆ Recipe
What you need is a better test runner. Gathering information about which test is currently executing is not the responsibility of the test itself, but of the test run- ner, so you should add this feature to the test runner. Fortunately, you can add a simple TestRunListener to the JUnit-addons TestRunner whose job is to print to the console the name of the test about to be executed. Listing 16.9 shows a quick- and-dirty implementation of such a test run listener.
3 Undoubtedly there are more, but we prefer to recount our experience rather than speculate.
4 Of course, Murphy's Law says that the next test—the one that uncovers the deadlock—is in a different test suite, and you have no idea which test suite the test runner is executing now!
TE AM FL Y
Team-Fly®
601 Report the name of each test
as it executes
package junit.cookbook.addons.listener;
import junit.framework.Test;
import junit.framework.TestResult;
import junitx.runner.listener.AbstractRunListener;
public class DumpTestNameListener extends AbstractRunListener {
public void testStarted(Test test, TestResult result) { System.out.println("> " + test);
}
// The remaining event handler methods do nothing }
To use this extra listener, specify it in a “runner properties” file. This is a proper- ties file that the JUnit-addons runner uses to register listeners at runtime. Because we want the default output as well as our small amount of customized reporting, we want the following runner properties file:
# runner.properties
junitx.runner.listener.0=junitx.runner.listener.DefaultConsole
junitx.runner.listener.1=junit.cookbook.addons.listener.DumpTestNameListener
The DefaultConsole listener is the one that the JUnit-addons runner registers if you do not specify your own listeners. If you specify custom listeners but still want the default listener to run, then you need to include it, probably in the first posi- tion in the listener list.
Now that you have created the listener and placed it in the runner properties file, you need to specify that properties file when you launch the JUnit-addons test runner. You specify a runner properties file by passing the option -runner.prop- erties <filename> to the test runner, as in the following command.
> java -classpath <your classpath> junitx.runner.TestRunner
➾ -runner.properties <runner properties file>
➾ -class <test suite or test case class>
NOTE A minor defect in the JUnit-addons test runner documentation—Unfortunately, at press time, the JUnit-addons test runner documentation was incorrect in describing how to specify the runner.properties file. It mentions using –runner.properties=<filename>, but our experiments showed that this does not work. The way we specify the runner properties in this recipe is correct. Other than this minor problem, the JUnit-addons docu- mentation is quite good!
Listing 16.9 DumpTestNameListener
toString() includes the test name
602 CHAPTER 16 JUnit-addons
The following is some sample output from a test run using the DumpTestNameListener.
> testRunHeader(junit.cookbook.listener.test.TestRunReporterTest)
*> testRunFooter_RunEnds(junit.cookbook.listener.test.TestRunReporterTest)
*> testRunFooter_RunStops(junit.cookbook.listener.test.TestRunReporterTest)
*> testTestStarted_TestCase(junit.cookbook.listener.test.TestRunReporterTest)
*> testTestIgnored(junit.cookbook.listener.test.TestRunReporterTest)
*> testTestFailure(junit.cookbook.listener.test.TestRunReporterTest)
*> testTestError(junit.cookbook.listener.test.TestRunReporterTest)
*> testTestSuccess(junit.cookbook.listener.test.TestRunReporterTest)
*> testTestStarted_TestSuite(junit.cookbook.listener.test.TestRunReporterTest)
*> testTestStarted_TestSuite_TwoTests
➾ (junit.cookbook.listener.test.TestRunReporterTest)
*> testTestStarted_TestSuite_ZeroTests
➾ (junit.cookbook.listener.test.TestRunReporterTest)
*
Elapsed time: 0.061 sec (11 tests)
The default run listener DefaultConsole provides the asterisks (*) and the “Elapsed time” message. The run listener DumpTestNameListener provides the name of each test as it executes.
◆ Discussion
The architecture of the JUnit-addons test runner is very similar to JUnit’s stan- dard test runners, with one key exception that interests us now: JUnit’s test run- ners do not provide a method to register additional TestListeners when running tests. Internally, JUnit’s test runners send events to a TestListener, which can report on each test as it executes. The text-based test runner prints dots and “E”s and “F”s to the console; the graphical test runner advances the progress bar and decides whether to turn it red (the result of a failure or error). You could cer- tainly write your own test runner to report test execution the way you want, but it would take much more work than it should: you just want to add one more TestListener to the TestRunner!
◆ Related
■ 6.1—See the name of each test as it executes
■ 6.2—See the name of each test as it executes with a text-based test runner
603
Odds and ends
This chapter covers
■ Testing file-based features without disrupting the file system
■ Testing the syntax of your tests
■ Customizing assertions for test readability
■ Testing hidden behavior
604 CHAPTER 17 Odds and ends
There are always a few more things that authors want to say, but cannot find the appropriate chapter in which to say them. At that point, there are essentially two options: leave those things out of the book or create a catchall chapter in which to put them. We opted for the latter. This chapter contains recipes that simply did not make their way into one of the other chapters, for one reason or another.
We do not recommend writing tests that rely heavily on the file system; but if you need to do it, then we provide a recipe that describes some of the unexpected problems with cleaning up files between tests (see recipe 17.1, “Clean up the file system between tests”). We further describe one way to reduce the degree to which your tests depend on the file system in recipe 17.2, “Test your file-based application without the file system.”
Some of the problems that novice JUnit users experience have to do with incor- rect JUnit syntax. We see at least one or two such messages on the mailing lists per month. In recipe 17.3, “Verify your test case class syntax,” we describe a tool that helps you avoid spending time hunting down a problem related to a typo, rather than a “real” problem. If you have to program alone, then this recipe helps elimi- nate one source of problems.
The more tests you write, the more complex your assertions become. Patterns emerge. Some people have questions about refactoring test code, wondering if it is any different from refactoring production code. Read recipe 17.4, “Extract a custom assertion” to see that there really is no difference: a custom assertion is sim- ply the result of removing certain kinds of duplication from your tests.
Finally, we offer some recipes that use JUnitX (www.extreme-java.de/junitx/), a package that allows you to test the non-public parts of your classes. While we strongly recommend that you test entirely through publicly accessible methods, we recognize that not everyone agrees with this sentiment. Moreover, especially when testing legacy code, it is often advantageous to write tests for private methods before extracting them to a public interface or refactoring them some other way.
This chapter contains a few recipes about testing non-public parts: testing a leg- acy method with no return value and testing a private method. Relying on pri- vate implementation details is discouraged, in general, but if you need to do it, then JUnitX helps you do it.
605 Clean up the file system between tests