Testing Computer Software phần 5 pptx

107 • You will be asked to retract every query because these are not error reports. Don't expect to get answers to the queries either. • You will be asked to retract all design suggestions and most design issues. After all, if the program's behavior matches a reviewed specification, it would hardly be fair to count it as a bug. Our impression is that, over a few releases, perhaps 15% of the design changes suggested by testers are implemented. In practice, this contributes strongly to the polish and usability of the program. Do you really want to lose this information from your database? • Plan to spend days arguing whether reports point to true bugs or just to design errors. This is especially likely if you try to keep design issues in the database by agreeing to count only coding errors in the employee performance monitoring statistics. If you're already sick of arguing with people who say "but it's supposed to crash," just wait until their raise depends on whether you class reports as coding errors or design issues. • Expect your staff to be criticized every time they report a "bug" that turns out to be a user error. • Expect to be asked to retract every irreproducible Problem Report It shouldn't count against the programmer if the problem is truly irreproducible. There are lots of non-programming-errorreasons for these problems (user error, power wobbles, hardware wobbles, etc.). If the programmer does track down the coding error underlying an "irreproducible" problem, this report now counts against his statistics. If he can convince you that it's irreproducible, it won't count against his statistics. How hard should he look for coding errors underlying these reports? • Don't expect any programmer or project manager to report any bugs they find in any product under development. • And someday, you 'II be sued. Many people who are fired or who quit under pressure sue their former employer for wrongful dismissal. If you're the test manager, and your database provided performance monitoring that contributed to the departure of an employee who sues the company, you may be sued along with the company. This tactic lets the lawyer ask you more questions before trial more easily than if you're just a witness. Sound like fun? Who's going to pay your legal bills? Think before you say, "The Company." Probably they'll be glad to let you use the company's lawyer, but if you and the company are both defendants in the same trial, and the company's lawyer sees a way to help the company that hurts you, what do you think will happen? Maybe it depends on the company and the lawyer. The objective of the database is to get bugs fixed, not to generate nice management statistics. LAWYERS Everything in the problem tracking database is open to investigation in any relevant lawsuit by or against your company (also see Chapter 14): • Problem Reports that include tester comments raging against programmer-improfessionalism can be very damaging evidence, even if the comments are entirely unjustified. 108 • The company might gain credibility if the database gives evidence of thorough testing and thorough, customer-sensitive consideration of each problem. • It is illegal to erase Problem Reports from the database in order to prevent them from being used as evidence. MECHANICS OF THE DATABASE At some point you get to design your own system or to suggest extensive revisions to someone else's. From here, we'll assume that the design is yours to change. These are our implementation suggestions for a problem tracking system. Many other systems will satisfy your needs just as well, but variants on this one have worked well for us. REPORTING NEW PROBLEMS The Problem Report (Figure 5.1) is the standard form for reporting bugs. Chapter 5 describes it in detail. We recommend that anyone in the company can file a Problem Report. Your group allows some people to enter problems into the computer directly. Others write reports on paper (as in Figure 1.1), which you enter into the computer. 109 The system checks some aspects of the report as it's entered. It does not accept reports that it classifies as incomplete or incorrect If someone doesn't know how to fill in all the required fields, ask her to report the problem on paper. The Testing Group (you) will replicate the problem, flesh out the report, and enter it into the computer. On a single-user system, and in some multi-user systems, when you enter a new Problem Report, the computer prints at least 3 copies of it. One goes to the person who reported the problem. The second goes to the programmer, perhaps via his manager. The third copy is the Testing Group's file copy. (If your disk ever crashes, you'll be glad you kept a copy of each report on paper. Your paper files don't have to be elaborate, but they must include each Problem Report.) WEEKLY STATUS REPORTS At the end of each week, issue status reports. Be consistent: circulate the reports to the same people, week in, week out. The Weekly Summary of New Problem Reports tells everyone on the project what new problems were found this week. Figure 6.1 shows tbe new problems sorted by FUNCTIONAL AREA . Figure 6.2 shows the same problem sorted by SEVERITY . Some project managers have strong preferences for one order over the other. Be flexible. The Weekly Status Report (Figure 6.3) shows the state of the project, and how this has changed since last week. These is a popular and useful report, but don't present the numbers without careful commentary explaining unusual jumps in the counts. 110 END OF A TESTING CYCLE At the end of each cycle of testing, issue the Testing Cycle Complete report (Figure 6.4). A testing cycle includes all tests of one version of the product. For example, if you are testing CalcDog 2.10, one cycle of testing covers VERSION 2.10g and another covers VERSION 2.10h. The Test Cycle Complete report summarizes the state of the project, in much the same way as the weekly summary. The weekly report is convenient because it comes out every week, but comparing different weeks' data can be difficult because more testing is done in some weeks than others. Test Cycle Complete reports are more comparable because each covers one full cycle of testing. R ESOLVED AND UNRESOLVED PROBLEMS Problem Reports come back to you when they're resolved. Some problems are fixed, others set aside (deferred), and others are rejected. Try to recreate problems marked Fixed, before accepting them as fixed. 111 If the problem is only partially fixed, close this report, then write a new one that cross- references this one. If the problem wasn't fixed at all, re-open the report with a polite note. For each unfixed problem (Can't be fixed, As designed, and Disagree with suggestion),decidewhethertosay Yes to TREAT AS DEFERRED (seeChapter5,"Contentof the problem report: Treat as deferred"). Distribute copies of all resolved reports to the people who reported the problems. They may respond to unfixed problems with follow-up reports. Some Problem Reports are misplaced or ignored. Periodically—perhaps every two weeks—distribute a Summary of Unresolved Problems (Figure 6.5). Your goal is to keep these problems visible, but in a way that looks routine, impersonal, and impartial. Figure 6.5 organizes the problems by severity, without mentioning who's responsible for them. Figure 6.6 is a more personal variation on the Summary of Unresolved Problems. It organizes everything around who's supposed to fix each problem. Don't circulate this report publicly. Use it during private discussions with individual managers. DEFERRED PROBLEMS If your company doesn't hold regular review meetings for deferred Problem Reports, distribute the Summary of Deferred Problems (Figure 6.7) biweekly. This report describes every problem that the programmers 112 113 deferred or that you said should be treated as deferred. Senior managers see these reports and sometimes insist that certain deferred bugs be fixed. Also, this report keeps deferred problems visible. Programmers who see that these problems are still of concern sometimes find simple solutions to them. If you do have regular review meetings, this summary is still useful for the meetings, but only show the problems that were deferred since the last meeting. Also, add the PROBLEM AND How TO REPRODUCE IT field and the COMMENTS field, or print this summary but append full copies of each summarized report. Distribute the report a few days in advance of each meeting. PROGRESS SUMMARIES The Weekly Totals (Figure 6.8) summarize the project's progress over time. A similar report shows one line per cycle of testing instead of one line per week. A third useful report shows how many minor, serious, and fatal problems were reported each week. A fourth tracks reports of problems within each functional area. Each of these reports gives you a base of historical data. Summaries from old projects are handy for comparison to today's project. For example, you can use them to demonstrate that: • The project requires months of further testing. The number of new Problem Reports (per week or per cycle) usually increases, peaks, then declines. It is unwise to ship the product before reaching a stable, low rate of discovery of new problems. 114 • It doesn 'tpay to cut off testing a week or two early or without notice. Testers often make an extra effort during the last cycle(s) of testing. Summary reports reflect this by showing a jump in the number of serious problems found and fixed at the end of the project. * A sea of reports of user interface errors is normal at the current (e.g., early) stage of the project. Always generate one of these reports at the end of a project, for future use. Beyond that, the report is discretionary—generate it when you need it, and give a copy to whoever wants one. Many project groups like to see these data in a graph, distributed with the Weekly Status report. WHEN DEVELOPMENT IS COMPLETE When the product is almost ready for release to customers, tie up loose ends. Get unresolved Problem Reports fixed or signed off as deferred. Once the paperwork is tidy, and the product is ready to ship, circulate the Final Release Report (Figure 6.9). The report shows the number of deferrals. Attach a copy of the Summary of Deferred Problems (FiguTe 6.7). Because this is a last-chancc-for-changes report, consider adding the PROBLEM AND HOW TO REPRODUCE IT field from the Problem Reports to the description of each deferred problem. The report goes to everyone who has to sign it. Circulate a draft copy, with XXXs through the signature areas, a day in advance. Give readers the day to review the deferrals and scream if they should. The next day, visit each person and have them sign the final copy of the report (all signatures on the same copy). 115 Senior management, not you, decides who signs this report. Anyone who must approve the release of the product before it goes to manufacturing (and thence to the customer) should sign this release. Don't ask anyone who can't veto the release for their signature. Note that a tester's (your) signature appears at the bottom of the report, beside PREPARED BY. The Testing Group prepares this report but does not approve a product for release. You provide technical'input. Management decides to hold or release the product. If you feel that testing was inadequate, say so, »nd say why, in an attached memo. R EOPEN DEFERRED BUGS FOR THE NEXT RELEASE * You finally close the books on Release 2.10 and ship it. the company begins planning Release 3. As part of the planning or early development process, you should reopen the bugs that were marked Deferred, Treat as deferred, and, perhaps As designed too. 116 This is one of the system's most important functions. Deferred bugs are just that, deferred, set aside until later. The normal expectation is that they will be fixed in the next release. The tracking system must ensure that they are not forgotten. Your database management software should be able to copy these reports to a temporary file, modify them as listed below, move them to the main data file for the next release, and print copies of each report. Modify each report as follows: • Reset the RESOLUTION CODE to Pending. •Change RELEASE and VERSION ID (for example, to 3 .00a). • Assign a new PROBLEM REPORT # . • Clear any signatures (except for the report's author) and the associated dates. • Clear the COMMENTS . Leave the rest of the report as it was. After entering them into the database, circulate these reports in the usual way. In practice, some companies review the bugs before reopening them, and carry only a selection of the deferred bugs forward. The three of us are split on this issue, reflecting our different situations. Company practices vary widely. TRACKING PATCHES Some companies respond to customer complaints with patches. A patch is a small change made to fix a specific error. It's easy to miss side effects because the rest of the code isn't thoroughly retested. The patched [...]... between 1 and 45 Work backwards What input could make it print something bigger than 45 or less than 1? Create test cases to try them Look for equivalent operating environments The program is specified to work if the computer has between 64 and 256 K of available memory That's an equivalence class Another class includes RAM configurations of less than 64K A third includes more than 256 K Some well-known... also illustrates a practical problem Look at outline section 1.2 .5. 2 dealing with arithmetic operators Conceptually "arithmetic operators" is an equivalence class of its own and the programmer might in fact treat this group ay an equivalence class by testing inputs against a list of every arithmetic operator Now consider 1.2 .5. 3.1 and 1.2 .5. 3.2 These also include all the arithmetic operators How should... correctly 1 25 OVERVIEW The chapter starts by considering the characteristics of good test case Next It asks how to come up with powerful test cases It discusses five techniques: • Equivalence class analysis • Boundary analysis • Testing state transitions • Testing race conditions and other time dependencies • Doing error guessing It considers a class of automation techniques called function equivalence testing. .. In a system that has shown itself vulnerable to races, execute a full cycle of testing under load Don't be dissuaded by project managers who whine that you're testing the program unfairly People will run the program concurrently with others, even on supposedly non-concurrent computers People will use cheaper versions of the computer that have slower clocks, slower memory, and less memory Your task is... pathway to every option in every menu You might be able to reach Menu 15 from a choice on Menu 14 and from another on Menu 27 If so, you should test every choice in Menu 15 twice—once after reaching the menu from Menu 14, again after reaching it from Menu 27 If there are ten ways to get to Menu 14, there are at least ten ways to get to menu 15, each a different route that takes you through Menu 14 If you... automation techniques called function equivalence testing It describes an absolutely required testing technique, regression testing Regression test cases may or may not be as efficient as the rest, but they are indispensable Finally, there are a few notes on executing test cases Sometimes testers have great testing Ideas but they miss the bugs because they don't conduct their tests effectively Here... you've pushed the machinery too hard Load testing is boundary condition testing Run a test that the program should be able to pass (such as maximum number of terminals) and another test that the program should fail (one too many terminals) Test many limits in combination It may not be able to cope with more than one limiting case at once Also, do enough general testing while you've got the program operating... of the programs demands keyboard input However, you can send input remotely to most programs on most computers Computers usually pass input they receive through a modem to application programs in a way that makes the modem input look just like keyboard input To simulate keyboard input, program a second computer to send data to the first by modem It's a little clumsy, but it works There are costs here... once, because you'll find bugs and have to redo testing in the next test cycle Estimate five to eight cycles of testing (Or use the average number of cycles needed by your company in the past.) • Estimate how much time the tools will save you Allow for planning, programming, and debugging the automated tests Be realistic If you are allowed to automate testing, this estimate will be quoted back to you... that you automate function equivalence testing You can now execute many more tests than you could by hand However, you still have to select test cases with care: the number of values that can be passed to the function under test is probably too large (maybe infinite) for complete testing Naturally you should test boundary values, but now you have the luxury of testing the function more thoroughly How . unusual jumps in the counts. 110 END OF A TESTING CYCLE At the end of each cycle of testing, issue the Testing Cycle Complete report (Figure 6.4). A testing cycle includes all tests of one version. analysis • Testing state transitions • Testing race conditions and other time dependencies • Doing error guessing It considers a class of automation techniques called function equivalence testing. . fields, ask her to report the problem on paper. The Testing Group (you) will replicate the problem, flesh out the report, and enter it into the computer. On a single-user system, and in some multi-user