Testing Computer Software phần 9 pptx

225 • Dollars. Management can try to add features more quickly, rather than dropping them, by spending money. However, because new features haven't yet been fully specified or designed, the project's designers may become bottlenecks. • Release date. The power of the evolutionary approach is best seen in the degree of schedule control that it gives management. The project manager can always postpone the release date. Often, though, he will choose to ship the product on time rather than adding those last 15 features. Because the program is always stabilized before the next wave of features is added, a proj ect manager who decides to stop adding features can probably finish the project within a few weeks. • Reliability. The reliability of evolutionary products is high. Because the product is tested and stabilized as each new piece is added, most of the project's testing budget is spent before the end of the schedule. If management says stop adding features and get the thing ready to ship, it won't take much additional final testing before the product is ready forrelease. The incentive to skimp on testing and release something buggy is gone. The testing cost under the evolutionary model threatens to be large because testing starts early. However, much of that cost is recouped because there isn't so much testing at the end. Further, all those features that weren't specified or coded weren't the subject of test plans or tests either. One way to make a mess of an evolutionary project is to have the programmers keep writing fresh code rather than fixing the problems discovered during testing. This is very tempting when the project is running behind schedule, but the appearance of progress is deceiving. The problems will take longer to fix and retest later. The project manager will face last minute tradeoffs between reliability and release date. Another risk of adopting an evolutionary approach is that an inexperienced project manager may imagine that he doesn't have to do much initial planning, because the product will evolve over time. This is a big mistake. If the core is built inflexibly, it will need extensive reworking before key features can be added. Fundamental work might have to be redone many times. Also, at what should be the end of the project, the marketing group might realize that they forgot to identify some features as critical, so others were done instead. Now the product has to wait until those features are added. Marketing and sales staff sometimes don't understand this approach well enough to realize that when the project manager says that some features might be in the product or they might not, no one should sell the product as if it has a given feature, until that feature has been added. The test manager has another way to ruin the project. If he doesn't assign test resources to the project early (perhaps because they're still assigned to some other project that's behind schedule), then the core program doesn't go through the full test cycle, nor do the incremental changes. No one sees this slippage on the schedule until the testing staff are finally assigned, perhaps near what should be the very end of the project. At that point, the testers are in the traditional system test situation, with a big pile of untested code and very little remaining testing time left in the schedule. The result is something that looks like a badly managed waterfall project— testing costs more, the schedule falls apart, and lots of bugs are missed. The Testing Group probably has more ability to destroy an evolutionary project than anyone else. Be considerate of this risk. 226 A DEVELOPMENT MODEL'S IMPLICATIONS FOR TESTING Once you understand the project manager's development model from his point of view, think about its implications for the testing effort. Here are some implications of the waterfall method: • Review the user interface early, perhaps by testing prototypes. If the user interface is specified and approved before the code is written, usability tests near the end of the project won't have much effect. • Start writing the test plan as early as possible because this will force you to critically analyze the specifications (or drafts of them). Any risks they pose for testability of the project must be raised early. Theoretical models are important, but you must also be effective when managers throw out the rule book. Think about this Figure as you read the tidy sequence of tasks in this chapter. 227 • You cannot start testing until late in the project The odds are that you will start later than the initial schedule projects because design and coding will fall behind schedule. Plan to spend extra time crafting your staffing plan. For example, you want to unleash a trained group of testers as soon as the product is ready for testing, but what should they do if the program is so unstable that you kick it back to the programmers for rework almost immediately? You should anticipate this and have alternate work ready—perhaps you can use the time to automate some tests or create additional test data files. As another example, develop a strategy for incorporating new people at the last minute. You can delegate many simple but time-consuming tasks to new staff, making maximum use of your experi enced testers during the limited time available. You and your testing team can make it easy to hand off these tasks by identifying them and organizing your work accordingly. If you don't do this planning in advance, you'll find it much harder to incorporate new testers at the end of a behind- schedule project, when no one has time to think. • By the time you start testing, your work is on the critical path. Coding is complete, or nearly so. As soon as you stop finding bugs, the program ships. On a project large enough for a few testers, consider dedicating one to guerrilla raids and other relatively unstructured testing. This person's goal is to find one or two bugs each week that are so obviously bad that no one would ship the project with these problems in it. The time it takes to fix these bugs is time the rest of the test group can use to plod through the program more systematically. Projects that need this tactic for buying testing time are no fun, but this tactic might buy you several months of testing time, slipping the schedule week after week, one week at a time. (Warning: never delay reporting a critical bug. It might be tempting to keep a bug in reserve, to be reported when you can't find anything else that week. This buys you that one last week of testing (the week needed to fix this bug). Don't try this unless you like the idea of being hated, fired, and blacklisted. Don't do it. Don't think about it. And never, never joke about it.) In contrast, consider these (quite different) implications of the evolutionary method: • Plan to staff the project with testers very early, starting reliability testing as soon as the program reaches its first level of functionality. • Plan waves of usability tests as the project grows more complex. You can probably argue successfully for design changes in recently added code. • Plan to write the test plan as you go, rather than as a big effort before testing. • Plan to do your most powerful testing as early as possible. Be careful about gambling that you' 11 have time later to conduct tests that you know are critical. The project manager might stop development at any time. We don't mean "^n/time after the planned release date." We mean any time—for example, two of us shipped a product three months early by stopping adding features earlier than planned. QUALITY-RELATED COSTS Quality-related costs include the costs of preventing, searching for, and coping with product errors and failures. From a business point of view, you have a testing budget because the cost of testing is lower than the cost of dealing with customer complaints about undiscovered bugs. If you can show that the company will save money if you conduct a certain type of test at a certain point in the project, you'll probably get funding to do that work. As you envision ways to tailor your team's work to the project manager's development model, you'll want to propose testing assignments that might be unusual in your company. For example, your company might 228 not be used to assigning testers to review the product's external design—or to giving them enough time to do the job competently. Other tasks that might require justification include analyzing customer support call records, automating test cases, delegating some types of last minute work to newly added junior testers or administrative staff, testing compatibility with a broader range of printers, or conducting an outside beta test. In each case, you can dramatically strengthen your argument with data that shows that this work will prevent significantly larger expenses later. 229 The more you know about your company's quality-related expenditures, the better you'll do at evaluating and advocating new testing procedures. Quality-related costs are normally discussed in terms of four categories (Campanella, 1990): • Prevention costs: everything the company spends to prevent software and documentation errors. • Appraisal costs: all testing costs and the costs of everything else the company does to look for errors. • Internal failure costs: all costs of coping with errors discovered during development and testing. • External failure costs: all costs of coping with errors discovered, typically by your customers, after the product is released. Figure 13.2 shows examples of the different types of costs. Feigenbaum (1991) estimates that the typical company spends 5 to 10 cents of every quality cost dollar on prevention, another 20 to 25 cents on appraisal, and the remaining 65 to 75 cents on internal and external failure costs. Quality Assurance (QA) groups often systematically collect quality-related cost information. The typical software testing group has a narrower focus and mandate than a QA group. You will probably find much information hard to obtain. Don't despair. Even if you can't learn everything, you can often learn a lot just by pooling data with your company's technical support group. As to the rest of the data, make (but be ready to explain) educated guesses as needed. What you collect or reasonably estimate will often be sufficient to justify a proposal to reduce failure costs by increasing prevention or testing costs. THE DEVELOPMENT TIME LINE The typical project manager will publish schedules that contain a series of milestones, the most common of which are called "alpha" and "beta." The exact definitions vary (widely) from company to company, but in essence, alpha software is preliminary, buggy, but usable, while beta software is almost complete. Figure 13.3 is an example of a project time line, showing the milestones. This milestone-based approach is pragmatic. It recognizes that programming, testing, manual writ- ing, and many other activities are done in parallel, and it maps them all onto the same time line. In some companies, requirements writing, prototyp- ing, specifying, etc., might be mapped in parallel with all of these tasks (e.g., under an evolutionary model) whereas in others, this work might be con- sidered preliminary and be put earlier on the line. But however the work is to be done in theory, the approach in practice involves a set of agreements— 230 231 these people will get this work done by this time and these people will be doing these things with the results of that work. Figures 13.4 (13.4a, 13.4b, 13.4c, and 13.4d) are our rendition of a milestone chart. We stress that this is only an example, and that no company that we know of conforms exactly to this illustration. Every company defines its milestones its own way and organizes the work in its own way, but these ways are often reasonably close to what we show here. We think this is a useful way to think about the ordering of the tasks. For the rest of this chapter, we will explore this table and map your testing and test planning strategy and priorities onto the time line. (Note: many of these terms were defined and discussed in Chapter 3.) PRODUCT DESIGN This is the start of the project, when all parties concerned figure out what this product should be. This phase includes requirements analysis and internal and external design. For thoughtful discussions, see DeMarco (1979), Gause & Weinberg (1989), Gilb (1988), Gould & Lewis (1985), Ould (1990), Weinberg (1982), Yourdon (1975), and Yourdon & Constantine (1979). PROGRAMMING ACTIVITIES DURING PRODUCT DESIGN If requirements documents, proposals, and contracts are written, they are written during this phase. Coding begins at the end of this phase. In waterfall projects, the internal and external specifications are written during this phase. Detailed internal and external design might be done in this phase or later. MARKETING ACTIVITIES DURING PRODUCT DESIGN The marketing department does research during this phase, to help define the product and to communicate the vision of the product to management and to the programmers. Marketing might run ideas or very early prototypes through small groups of customers, in focus groups. They might also survey customers of this product (or competitors), asking what features these people like in this type of product, how they perceive competitors' quality (and why), and how much they would pay for a product they liked in this category. You may be asked to help with some of these early consumer tests. DOCUMENTATION ACTIVITIES DURING PRODUCT DESIGN Some documentation groups help draft specifications during this phase. TESTING ACTIVITIES DURING PRODUCT DESIGN If you're lucky, you might be asked to review the product design documents as they are written. This way, you'll learn about the product and be prepared for some early test planning. See Chapter 3, "Testing during the design stages" for thoughts on design flaws that you can find in the various specifying documents. In practice, many testers don't make many useful comments during this review. 232 233 In many companies, these reviews are not high priorities for testers. If you are invited into this process, read Gause & Weinberg (1989) and Freedman & Weinberg (1982). Under a strict waterfall model, user interface design reviews might be your only effective opportunity to challenge the design. In this case, you need time to study the design before the review. Look to customer support for statistics on the costs of user interface inconsistencies in previously shipped products and use these to justify setting aside the time you need. Prepare for test automation Often, the most important thing you can do during the design period is to identify testing support code that you want in the program (or with it). You might not get this support, but if you ask early, and explain the value of the individual tools clearly, you stand a good chance of getting some of them. Here are some examples for your wish list: • Printer automation, you want command line control for test automation. The program will probably need the following parameters: name of the input file that the program is to print, name of the output file if printing is to be redirected to a file, name of the printer (possibly including the printer driver), and setup information such as the initial font or printer output resolution. The program should exit after printing. Given this facility, you can build a batch file that feeds a file to the program, selects the printer, tells it to print the file to disk, compares the file to one saved previously and prints the result, then starts the next test. See "Some Tips On Automated Testing" in Chapter 8 for further discussion. • Memory meter. You want to be able to press a key at any point in the program and get a display or printout of the program's memory usage. Something as simple as the number of bytes free is handy. This might be more useful if it also lists the sizes of the five or ten largest contiguous blocks of memory. It is amazing (testers never believe it until they try it) how many new tests you will invent and how many hard to reproduce bugs become easily reproducible once you know how much memory is available before you do something, while you're doing it, and after you stop (or after you erase what you did). • Cheat keys. These take you to a specific place in the program. This is essential for testing arcade games and role playing games and it might be handy in other programs. • Screen dump. You need an easy way to copy everything on the screen to a printer or file (or serial port if you analyze the data in real time on another machine). The screen dump utility should capture cursor and mouse position. The tester will want to capture the screen put up by the program as it crashed, so the screen dump utility should be available even when the program under test crashes. If the computer you test on doesn't have excellent programs for this, ask the programmers to make one. Create contractual acceptance tests If your company is custom developing a product, it is extremely important to include an acceptance test in the sales contract. You and your company's customer should define a set of tests that the customer will run when development is complete. The customer must agree that if the program passes the tests, the contract has been fulfilled and any further changes or bug fixes are to be paid for separately. The test must be so clearly stated that you can execute the test, and confirm that the product is ready for delivery, before sending the product to the customer. It is important that you (Testing) work with the customer to define this test. The risk is too high 234 [...]... NOTE ON TESTING CYCLES The ideal cycle of testing includes a complete round of tests of one version of the product In practice, the amount of testing varies from version to version We still call the testing of Version 1.2 Ob a cycle of testing, and the testing of 1.2 Oc another cycle, even if we skipped many tests in each In many companies, programmers submit a new version of the product for testing. .. 238 TESTING ACTIVITIES AFTER FIRST FUNCTIONALITY Somebody starts testing at this point The programmer does unit testing Someone (programmer, tester, programmer's assistant, someone) should start testing the program from the outside This might merely involve playing with the program In an evolutionary environment, the goal is to make this core software as reliable as a finished product, so formal testing. .. the level of testing • If you need it, develop and publish an acceptance test (An acceptance test is a test suite each version of the program must pass before being subjected to more detailed testing See Chapter 3, "Black box testing: The usual black box sequence of events: Acceptance testing" and Chapter 12, "Documenting test materials: Types of test documents: Criteria for acceptance into testing. ")... a complete test cycle When the Testing Group decides they've tested this version sufficiently, they close this cycle of testing and accept the next version for testing In many other companies, programmers submit a new version for testing after they've made so many changes to the program that they (and you) feel that it's wiser to test the new code than the old Early in testing, expect delays of two... write as a tool for thinking about testing, while you 240 write Always test while you write: it keeps your test notes accurate and it keeps you testing while you're doing your best thinking about tests Don't give in to the temptation to spend days writing about how to test the program, instead of testing it The objective is to find errors, not to make notes Do mainstream testing Do the obvious tests, don't... fewer implications that fix will have for other parts of the program Your goal in early testing is to find all easily found problems in every area of the program Your testing continues to be broad, shallow, mainstream testing As soon as you can get a draft of it, stroke the manual Try everything in the manual at the computer Try every program path it describes plus any other obvious ones Check every example... valuable skill 237 Negotiate early testing milestones A good development team will want you to start testing before all the code is written A disorganized team will waste a tremendous amount of your time in early testing Negotiate some structure with the project manager A project manager is often delighted to prioritize programming tasks in order to make your early testing effective With just a little... say about your testing? Finally, during alpha, you should start laying out your automated tests Automated tests are regression tests that the computer either runs for you or helps you run They hold out the promise of saved testing time The less time you spend re-executing old tests, the more time you'll have to create new ones • Archive non-trivial data files as you create them for testing purposes... pay for itself for ten cycles of testing Some project managers insist that their bugfree wonder needs only two or three testing cycles They are probably wrong (always budget for at least eight cycles) but they can be offended by too much visibly long-term work Automate some tests, but not so many that you're chastised for delaying testing Even managers who expect many testing cycles may question your... level of testing What if management halts testing tomorrow? How many weakly tested areas will there be? Guerrilla raids are brief and informal, but much stronger than mainstream tests They are much faster than the more focused tests, so you can test more areas of the program to this level Test as many areas of the program as possible at this level before testing is halted Intense planned testing Choose . discussions, see DeMarco ( 197 9), Gause & Weinberg ( 198 9), Gilb ( 198 8), Gould & Lewis ( 198 5), Ould ( 199 0), Weinberg ( 198 2), Yourdon ( 197 5), and Yourdon & Constantine ( 197 9). PROGRAMMING ACTIVITIES. of four categories (Campanella, 199 0): • Prevention costs: everything the company spends to prevent software and documentation errors. • Appraisal costs: all testing costs and the costs of everything. and specify later), and programming. 2 39 TESTING ACTIVITIES AFTER FIRST FUNCTIONALITY Somebody starts testing at this point. The programmer does unit testing. Someone (programmer, tester, programmer's