Testing Computer Software phần 8 docx

197 198 Printer compatibility table There are over 1000 printers on the market, and most of them emulate (work just like) other printers. If 50 printers all emulate the Hewlett Packard LaserJet II, don't test all 50 printers. We've found it useful to create charts, along the lines of Figure 12.16, that show what printers are compatible with what others. (You can create similar charts for other devices but here, as in Chapter 8, we'll stick with printers as our model device.) Formats vary widely. All charts contain some of the following columns: • Printer: make and model. • Mode: some printers can be set in their own native mode, or to emulate a few other, different, printers. • Compatibility: make and model of the printer this one emulates. • Source: how do you know this printer emulates that one? Name the magazine article, the person, the advertisement. Some reports are much more trustworthy than others. • Tested: indicate whether your lab has tested this printer's compatibility and what test was used. Have you confirmed graphic mode compatibility? Escape sequence command compat ibility? Character set compatibility? • Notes: list incompatibilities, doubts, customer reports, etc. Boundary Chart See "Equivalence classes and boundary values" in Chapter 7 for a detailed discussion of equivalence class and boundary chart development. Figure 12.17 illustrates a boundary analysis of some of the input variables used in the Problem tracking system. Don't expect to complete this chart early in testing. It takes a long time to do this chart well. Start by listing every input field (use the input variable list if you made one). Identify their functions. Assign limit values and further information as you learn more, but let yourself experiment with the variables from a full list from as early a time as you can. OUTLINES—THE FUNCTION LIST Function lists outline what the program can do. They express your organization of the program's functions. Organize them in a way that you'll find convenient for testing and test planning. We use a function list as the core of our notes. You can develop these lists to almost any level of completeness and detail. We suggest an incremental approach. Start simple; add detail as you go. Figure 12.18 summarizes our approach. You' 11 find it much easier and faster to add items and reorganize the list if you use an outline processor, rather than a word processor. / Figure 12.8 shows a basic, first draft, top-level list of the problem tracking system's functions. It shows everything the program is supposed to do. This first step toward a more thorough outline is valuable in its own right. Keep a copy at your desk while testing the program; check it as you go to make sure that you've tested each function on the list. You can summarize the stability of different parts of the program by highlighting working functions in one color and failing ones in another. 199 Make your next draft more organized. Choose an organization that seems natural and easy to work with. Try an alphabetic listing for a command language that invokes each function with a separate command. Try a hierarchical layout for menu-driven programs. Alternatively, you could organize around the program's conceptual structure, grouping all functions that do related work, rely on related input data, or generate related outputs. Figure 12.19 expands the first draft list of Figure 12.8. Many of the main functions have been left alone. The figure breaks one function, 5. Work with a holding file, into two subfunctions. It explores one of these, 5.1 Read data from a file, in further detail. You can keep adding detail to a function list. For example, you can expand 5.1.3.6 (Optional) Print record to show that the program will (should) check whether the printer is online and ready to print. Expand the list gradually. If you try to make a full list before starting to test, you may never have time to start testing. Figure 12.20 magnifies section 3. Enter new Problem Reports from Figure 12.19. You can call a list that's as detailed as this a Junction or program outline. This list is so detailed that in many cases, if you do the operation listed, the computer will do exactly one thing. This is a test case. Expected results belong here. Include the following types of information as you add detail to the outline: • Every Junction of the program. • Every visible subfunction • Every command you can issue to the program. • The effect of pressing each command key at each place that the program accepts input. 200 201 202 • Every menu and every choice on every menu. Testers often create menu maps to show this information. These are free-form diagrams. Show where each menu choice leads. It often helps to name each menu or screen, showing the names on the map. The names should directly reflect the menu or screen as displayed. • Every entry to each part of the program. How can you get there? Does how you got there make any difference or does the program behave in the same way no matter what menu, prompt screen, etc., you came from? • Every exit from each part of the program. How do you go to the next menu, dialog, form, state, whatever? How do you quit entering data? How do you get back to where you came from? How do you halt the program from here? • Every data entry screen, dialog box, and message box. Analyze the data values in a boundary chart. Here, show how to get to each screen, any sequence-dependencies (such as different behavior the first and second times you reach the screen), special commands, how to exit with and (ifppssible) without saving the data you entered, and where you get to from here. • Error handling in this part of the program. We often find it easier to treat error handling as a separate section rather than trying to show it under each topic. MATRICES A matrix is like a table. Both have rows and columns. The top row of each (the heading), shows what belongs in each column. The first column of many tables, and of all matrices, shows what belongs in each row. As we use the terms, a table and a matrix differ in the following ways: • The table's main function is descriptive. It summarizes the behavior of the program or (as in Figure 12.16) of hardware. A person with a complete enough specification could completely fill in the table without doing any testing. The tester would then compare the program's behavior against the information in the table. • The matrix's main function is data collection. It provides a structure for testing the effect of combining two or more variables, circumstances, types of hardware, or events. The row and column headings identify test conditions. After she runs the test, the tester enters the result into the corresponding cell of the matrix. (For example if she combines the condition listed in the third row with the one listed in the fifth column, she'd enter the result in the matrix cell 3 rows down and 5 columns across.) Often, the cell entry is just a checkmark, indicating that the program behaved correctly. Disk input/output matrix The disk I/O matrix is a classic example of a widely used test matrix. Suppose that your program can use the following functions to send data to the disk: • Save: a copy of the data in memory to the disk. Save As: save a fresh copy of the data in memory under a new name. 203 • Print to disk (ASCII): format the output as if you were going to send it to an ASCII printer (no formatting codes fancier than tabs, line feeds, or carriage returns), but direct the output to a disk file rather than to a printer. • Print to disk (formatted): format the output as if you were going to send it to your current printer. Include all formatting codes for that printer. Direct the output to a disk file rather than to a printer. Suppose it can use the following functions to read data from the disk: • Open: erase what's in memory (perhaps after saving it), then load this file into memory. • Append: load the contents of this file into memory, placing them at the end of the data already in memory. • Insert: load the contents of this file into memory, placing them at the current cursor position. • Import text: open a text file that was created by some other application and is not in this program's native file format. Bring the text, and perhaps some of its formatting, into memory. • Import graphics: load a picture into memory. Suppose you're trying to save or read one of the following types of files: • A very small file (one or two bytes). • A typical file. • A large file (more bytes in the file than available in RAM). Suppose the program can run on computers that can use any of the following types of disks: • Low density floppy. • High density floppy. • Hard disk. • Network volume. 'Optical disk. • RAM drive. And finally, suppose any of the following events can happen: • Disk already full when you try to write to it (via Save, Save as, or Print to disk). • Disk almost full, fills during the attempted write or when creating a temporary file during the attempted read. • Disk is write-protected. • Time-out: the disk (probably the network) takes too long to respond. • Power failure or power turned off. 204 • Keyboard entry: hit keys during the read or write. • Mouse activity: move the mouse or click its buttons during the read or write. There are lots more possibilities in each of these categories, but these cases illustrate the four categories involved in an I/O matrix analysis: • File operation (such as Save As or Open). • File characteristics, such as type, format, and size. • Hardware, such as disk drive types, but also including individually listed computers if compat ibility with specific models is suspect. • Failure condition, such as a full disk or a hardware failure. There are many ways to organize these categories into one or more I/O matrices. Figure 12.21 shows part of one possible matrix. Using the matrix as a guide, you would run such tests as attempting to save a medium-size file (2nd row) to a write-protected low-density floppy disk (3rd column). If the program responded with a write-protected disk error message, and behaved reasonably thereafter, you would put a checkmark in the cell that at the intersection of the 2nd row and 3rd column. This is the starred (***) cell in Figure 12.21. The disk I/O matrix is one of your most important testing charts. We recommend that you fill out the four category lists to match your application, and then make an input table and an output table. Other hardware-related matrices Figure 8.4 is a test matrix that varies the type of printer across columns and the print features across rows. Another printer matrix would show the types of printer across columns (such as LaserJet II, LaserJet III, LBP-8, etc.) and the amount of printer memory down the rows (such as 0.5,1.0,1.5,2.0, and 2.5 megabytes). Perhaps the test case is a three-quarter-page graphic image. The program passes the test if it can print the image without running the printer out of memory. 205 206 Environment matrix In this case the rows (or columns) show environment variables, such as type or version of windowing environment, type or version of memory manager, operating system version, language, or country. The columns (or rows) could also show environment variables, or they might list individual features, types of hardware, error states, or anything else you want to test in combination with a set of environment alternatives. Input combination matrix Some bugs are triggered by a combination of events. If the program crashes only when you type a 60 character string on the third line of the screen, then press the up-arrow (this was a real bug), you won't find the problem by testing single actions or variables. Unfortunately, in most programs, the number of combinations of actions and variables' values is nearly infinite. Therefore, the question is not how to test all combinations of inputs (you can't). The question is, how to find a few interesting combinations. Myers (1979) describes one powerful but complex approach called Cause-Effect Graphing. We will not describe it here, but we recommend it for further study. V Our approach is more experiential. We learn a lot about input combinations as we test. We learn about natural combinations of these variables, and about variables that seem totally independent. We also go to the programmers with the list of variables and ask which ones are supposed to be totally independent. Their memories aren't perfect, and some programmers think it's funny to throw in the odd falsehood, so check a few combinations of allegedly independent variables just to reassure yourself. Once we develop a feel for the relatedness among variables, we test different combina- tions of values of related variables. Error message and keyboard matrix Earlier in this chapter, we described a keyboard convention table that showed all the program's responses to all possible keyboard inputs. Here we also look at program responses to keystrokes, but in a more tightly focused matrix. In a graphical user interface (GUI), error messages appear in message boxes. The only valid response is an acknowledgment, by mouse-clicking on <OK> or by pressing <Ent er >. Mouse clicks in other places and other keystrokes are (supposedly) invalid and ignored. In practice, even on systems with extensive, standard- ized message box building support, programmers often add special case diagnostics or other responses to specific keys in individual message boxes. We often hear the claim that all message boxes in a given Macintosh application always work the same way, so if you test one message box thoroughly, you've tested them all. This is pablum, not suitable for real testers. We've found message-specific ways to crash Macintosh applications, Amiga applications, Windows applications, and DOS applications. You don't have enough time to check every possible keystroke with every message box, but a keystroke that has no effect in one message box may crash another. We drive this testing with a test matrix that lists messages on rows and groups of related keys in columns. For each row (message), we try a few keys from each column. Figure 12.22 shows some of the ways that we group characters. [...]... summarize IEEE Standard 82 9-1 983 for Software Test Documentation, which attempts to define a common set of test documents, to be used across the industry Schulmeyer (1 987 ) summarizes many other test documentation specifications You can order Standard 82 9-1 983 , which includes examples and much more detailed definitions, for a few dollars from: Computer Society of the IEEE P.O Box 80 452 Woridway Postal... than descriptions of them in isolation 2 08 • Dependency flags For example, if the program can read only 80 bytes of data at a time, you'll test it with 80 and 81 bytes If the program is enhanced to deal with 256 bytes, the old tests are obsolete Now you want to try 256- and 257-byte streams State explicitly that the test assumes that the program processes only 80 bytes, perhaps in a special section called... resumption requirements List anything that would cause you to stop testing until it's fixed What would have to be done to get you to restart testing? What tests should be redone at this point? • Test detiverables List all of the testing documents that will be written for this product • Testing tasks List all tasks necessary to prepare for and do testing Show dependencies between tasks, special skills (orpeople)... the details of the program's behavior or specification Test script for the inexperienced tester This person might be experienced with computers (a programmer, manager, or hardware or software support technician) or he might be a computer novice In either case, he has no testing experience and probably noi much familiarity with the program being tested A test script will guide him through each test, step... must pass when submitted for testing If it passes, the Testing Group runs the item through a full test cycle Otherwise they reject it as too unstable for testing Such tests should take less than half an hour—never more than two hours 213 If you use an acceptance test, write a document that describes it exactly Circulate it to programmers, preferably before the first cycle of testing Make the document... is a chronological record of the test executions and events that happened during testing According to Standard 82 9, it includes the following sections: • Test log identifier • Description What's being tested, including Version ID, where testing is being done, what hardware (printer, amount of available memory, type of computer, etc.), and all other configuration information (for example, operating... anticipate and justify the types of testing you should propose and conduct as the project moves forward 219 OVERVIEW The chapter considers the following topics: • Software development tradeoffs • Software development models • Quality-related costs • The development time line • Product design • Analyze customer data • First functionality • Almost alpha • Alpha • Depth vs breadth of testing • Pre-beta ■ Beta •... cash this quarter, and it's not going to get it, you have a big problem SOFTWARE DEVELOPMENT MODELS The project manager's plan for ordering tasks is his development model These models are discussed at length in the literature Some good sources are DeGrace & Stahl (1991), Evans & Marciniak (1 987 ), Gause & Weinberg (1 989 ), and Ould (1 989 ) We discussed these models in Chapter 12 ("Evolutionary Development... through some testing The test group (or someone else) tests the core as if it were a final product; testing and fixing continue until the core appears stable From here, the programmer adds single features or small groups of features Each time a new feature or group goes into the program, the program goes through another round of testing and fixing until it stabilizes The programming and testing team... on test plans and specifications Test summary report This is a summary of a series of tests, of the type that you might issue after completing a cycle of testing It briefly describes the testing done and evaluates the results According to Standard 82 9, it includes the following sections: • Test summary report identifier • Summary Say what was tested (including Version ID), in what environment, and summarize . Standard 82 9-1 983 , which includes examples and much more detailed definitions, for a few dollars from: Computer Society of the IEEE P.O. Box 80 452 Woridway Postal Center Los Angeles, CA 90 080 Or. summarize IEEE Standard 82 9-1 983 for Software Test Documentation, which attempts to define a common set of test documents, to be used across the industry. Schulmeyer (1 987 ) summarizes many other. might be experienced with computers (a programmer, manager, or hardware or software support technician) or he might be a computer novice. In either case, he has no testing experience and probably