3 principles of testing [compatibility mode]

Software Testing ISEB Foundation Certificate Course Principles of Testing Principles Lifecycle Dynamic test Management techniques Static testing Tools Principles ISEB Foundation Certificate Course Contents Why testing is necessary Fundamental test process Psychology of testing Re-testing and regression testing Expected results Prioritisation of tests Testing terminology n n No generally accepted set of testing definitions used world wide New standard BS 7925-1 - Glossary of testing terms (emphasis on component testing) - most recent - developed by a working party of the BCS SIGIST - adopted by the ISEB What is a “bug”? n n n Error: a human action that produces an incorrect result Fault: a manifestation of an error in software - also known as a defect or bug - if executed, a fault may cause a failure Failure: deviation of the software from its expected delivery or service - (found defect) Failure is an event; fault is a state of the software, caused by an error Error - Fault - Failure A person makes an error … that creates a fault in the software … that can cause a failure in operation Reliability versus faults n Reliability: the probability that software will not cause the failure of the system for a specified time under specified conditions - Can a system be fault-free? (zero faults, right first faulttime) - Can a software system be reliable but still have faults? - Is a “fault-free” software application always “faultreliable? Why faults occur in software? n n n software is written by human beings - who know something, but not everything - who have skills, but aren’t perfect - who make mistakes (errors) under increasing pressure to deliver to strict deadlines - no time to check but assumptions may be wrong - systems may be incomplete if you have ever written software What software faults cost? n n n huge sums - Ariane ($7billion) - Mariner space probe to Venus ($250m) - American Airlines ($50m) very little or nothing at all - minor inconvenience - no visible or physical detrimental impact software is not “linear”: - small input may have very large effect Safety-critical systems n software faults can cause death or injury - radiation treatment kills patients (Therac-25) (Therac- train driver killed - aircraft crashes (Airbus & Korean Airlines) - bank system overdraft letters cause suicide So why is testing necessary? - because software is likely to have faults - to learn about the reliability of the software - to fill the time between delivery of the software and the release date - to prove that the software has no faults - because testing is included in the project plan - because failures can be very expensive - to avoid being sued by customers - to stay in business 10 Re-testing after faults are fixed n n n n Run a test, it fails, fault reported New version of software with fault “fixed” Re-run the same test (i.e re-test) - must be exactly repeatable - same environment, versions (except for the software which has been intentionally changed!) - same inputs and preconditions If test now passes, fault has been fixed correctly - or has it? 53 Re-testing (re-running failed tests) New faults introduced by the first fault fix not found during re-testing x x x x ü Fault now fixed Re-test to check 54 Regression test n to look for any unexpected side-effects x x x x ü Can’t guarantee to find them all 55 Regression testing n n n n n misnomer: "anti-regression" or "progression" standard set of tests - regression test pack at any level (unit, integration, system, acceptance) well worth automating a developing asset but needs to be maintained 56 Regression testing n n Regression tests are performed - after software changes, including faults fixed - when the environment changes, even if application functionality stays the same - for emergency fixes (possibly a subset) Regression test suites - evolve over time - are run often - may become rather large 57 Regression testing n Maintenance of the regression test pack - eliminate repetitive tests (tests which test the same test condition) - combine test cases (e.g if they are always run together) - select a different subset of the full regression suite to run each time a regression test is needed - eliminate tests which have not found a fault for a long time (e.g old fault fix tests) 58 Regression testing and automation n n n n Test execution tools (e.g capture replay) are regression testing tools - they re-execute tests which have already been executed Once automated, regression tests can be run as often as desired (e.g every night) Automating tests is not trivial (generally takes to 10 times longer to automate a test than to run it manually Don’t automate everything - plan what to automate first, only automate if worthwhile 59 Principles ISEB Foundation Certificate Course Contents Why testing is necessary Fundamental test process Psychology of testing Re-testing and regression testing Expected results Prioritisation of tests 60 Expected results n n Should be predicted in advance as part of the test design process - ‘Oracle Assumption’ assumes that correct outcome can be predicted Why not just look at what the software does and assess it at the time? - subconscious desire for the test to pass - less work to do, no incident report to write up - it looks plausible, so it must be OK - less rigorous than calculating in advance and comparing 61 A test inputs expected outputs A Program: Read A IF (A = 8) THEN PRINT (“10”) ELSE PRINT (2*A) Source: Carsten Jorgensen, Delta, Denmark 62 Principles ISEB Foundation Certificate Course Contents Why testing is necessary Fundamental test process Psychology of testing Re-testing and regression testing Expected results Prioritisation of tests 63 Prioritising tests n n n We can’t test everything There is never enough time to all the testing you would like So what testing should you do? 64 Most important principle Prioritise tests so that, whenever you stop testing, you have done the best testing in the time available 65 How to prioritise? n Possible ranking criteria (all risk based) - test where a failure would be most severe - test where failures would be most visible - test where failures are most likely - ask the customer to prioritise the requirements - what is most critical to the customer’s business - areas changed most often - areas with most problems in the past - most complex areas, or technically critical 66 Principles ISEB Foundation Certificate Course Summary: Key Points Testing is necessary because people make errors The test process: planning, specification, execution, recording, checking completion Independence & relationships are important in testing Re-test fixes; regression test for the unexpected Expected results from a specification in advance Prioritise to the best testing in the time you have 67 ... Prioritisation of tests Testing terminology n n No generally accepted set of testing definitions used world wide New standard BS 7925-1 - Glossary of testing terms (emphasis on component testing) -... your system 13 How much testing? n It depends on RISK - risk of missing important faults risk of incurring failure costs risk of releasing untested or under-tested software underrisk of losing credibility... fewer faults left in 45 The testing paradox Purpose of testing: to find faults Finding faults destroys confidence Purpose of testing: destroy confidence Purpose of testing: build confidence The