Software Testing and Quality Assurance• Ron Patton. ppt

• Advantages:Easy to test clear requirements • Disadvantages:High bugs fixing cost bugs creep, cus-tomer is far from this till the end • Disadvantages:Need risk-assessments expertise 23

Software Testing and Quality Assurance

Dang Van Hung

College of Technology

Vietnam National University

Reading Materials

• Ron Patton.Software Testing, second edition, Pearson Education, 2006

• Roger S Pressman.Software Engineering - A practitioner’s approach, sixth

edition, McGraw-Hill Companies, 2005

• Shari Vance Pfleeger.Software Engineering - Theory and Practice- secondedition, Pearson Education, 2001

• Kshirasagar Naik and Priyadarshi Tripathy.Software Testing and Quality Assurance, Theory and Practice A John Wiley & Sons, Inc., Publication, 2008.

• Sommerville,Software Engineering, 8th edition, Pearson Education Limitted,

• Software Testing Background

• The software Development Process

• The Realities of Software Testing

4

Software Testing Background

Topics:

• How software bugs impact our lives

• What bugs are and why they occur

• Who software tester are and what they do

Infamous Software Error Case Studies

• Disney’s Lion King, 1994-1995

• Intel Pentium Floating-Point Division Bug, 1994

• NASA Mars Polar Lander, 1999

• Patriot Missile Defense System, 1991

• The Y2K Bug, circa 1974

• Dangerous Viewing Ahead, 2004

What is a Bug?

• Terms for software failures:Defect, Variance, Fault, Failure, Problem,Inconsistency, Error, Feature, Incident, Bug, Anomaly

• Generic terms:Problem, error, bugs (those cause software failure)

• Formal definition of Software Bug:

1 The software doesn’t do something that the product specification says itshould do

2 The software does something that the product specification says itshouldn’t do

3 The software does something that the product specification doesn’tmention

4 The software doesn’t do something that the product specification doesn’tmention, but it should

5 The software is difficult to understand, hard to use, slow, or–in the softwaretester’s eyes–will be viewed by the end-user as just plain not right

7Why do Bugs Occur?

Bugs are caused by numerous reasons, but the main cause can be traced to the

specification

8The Cost of Bugs

What Does a Software Tester Do?

• The goal of a software tester isto find bugs

• The goal of a software tester isto find bugs and find them as soon as possible

• The goal of a software tester isto find bugs and find them as soon as possible

and make sure they get fixed!

What make a Good Software Tester?

• They are explorers

• They are trouble shooter

• They are relentless (stronger and stronger)

• They are creative

• They are perfectionists

• They exercise good judgment

• They are tactful and diplomatic

• They are persuasive

11Example: A Square Root Program

Program SquareRoot Var r,s,x,y: integer ; input y ;

{y > 0}

r:=o; s:=-1; x:= 0;

while r <= y do begin

Automatic Testing (with Spin)

Test Case Generator:

chan inpt = [0] of {short};

active proctype inputgen()

short r = 0,s = -1, x = 0;

short y;

end: do:: inpt? y ->

r = 0; s = -1; x = 0;

do:: r <= y -> s = s + 2; r = r + s; x = x + 1;

:: else -> breakod;

x = x-1;

assert((! (x*x > y)) && (y < (x+1)*(x+1)));

printf ("if y = %d then x = %d\n",y, x)

od}

15

Testing Results

Full statespace search for:

hash conflicts: 0 (resolved)

(max size 2ˆ18 states)

1.596 memory usage (Mbyte)

16

The Software Development Process

Topics:

• What major components go into a software product

• How software progress from an idea to a final product

What Effort Goes into a Software Product

(architec-• Test documents (test plan, test cases,bug reports, test tools and automation)

The Software Development Lifecycle Models

P eople + M oney + Energy =

P erf ect Sof tware or N othing!

• Advantages:Simple, needs less planning, lessformal development process

Waterfall Model

• The Approach:

Discrete phases, a review after eachphase will decide to move next or stay atthis phase till it’s ready

• Advantages:Easy to test (clear requirements)

• Disadvantages:High bugs fixing cost (bugs creep), cus-tomer is far from this till the end

• Disadvantages:Need risk-assessments expertise

23

The Realities of Software Testing

Topics:

• Why software can never be perfect

• Why software testing isn’t just a technical problem

• The terms commonly used by software testers

24

Testing Axioms

• It’s impossible to test a program completely

– large input space

– large output space

– large state space

– large number of possible execution paths

– subjectivity of specifications

Testing Axioms

Large Input/State Space

int exF unction(int x, int y){

elsex[i] = x[i]/2;

}

• nthe number of decisions

• Linumber of times a decision can loop

• Xthe number of decisions that cause loops

• Vi the number of possible values each input variable could have

• Y the number of input variables

28

Testing Axioms

• Software testing is a risk based exercise

– need to balance cost and risk of missingbugs

• Testing can’t prove the absence of bugs

• The more bugs you find, the more bugs thereare

• The pesticide paradox

– a system tends to build resistance to a ticular testing technique

par-Testing Axioms

• Not all bugs found are fixed

• Difficult to say when a bug is a bug

– only when observed (latent otherwise)

• Product specifications are never final

• Software testers aren’t the most popular members of a project team

• Software testing is a disciplined technical profession that requires training

Software Testing Terms and Definitions

• Precision and accuracy

• Verification and validation (V&V)

• Quality and reliability

• Testing and quality assurance

31

What Is The V&V

• Verification:Are we building the product right? - Confirming that something

(software) meets its specification

• Validation: Are we building the right product? - Confirming that it meets the

user’s requirements

• The two major V&V activities are reviews, and testing So,testing is a part of

V& V

• Validation testing

– alpha testing: test conducted at the developer’s site by end-user

– beta testing: test conducted at the end-user site

32

Testing and Quality Assurance

• The goal of a software tester isto find bugs and find them as soon as possible and make sure they get fixed!

• A software quality assurance person’s main responsibility isto create and enforce standards and methodsto improve the development process and toprevent bugs from ever occurring!

Part 2: Testing Fundamentals

Topics:

• Examining the specification

• Testing the Software with Blinders On

• Examining the code

• Testing the Software with X-Ray Classes

Examining The Specification

Topics:

• What is black-box and white-box testing

• How static and dynamic testing differ

• What high-level techniques can be used for reviewing a product specification

• What specific problem you should look for when reviewing productspecification in detail

35

Black-box and White-Box Testing

36

Black-box and White-Box Testing

• Black-box(functional, behavioral, data-driven)

– The software under test considered as a black-box and no knowledge ofthe internal structure or how the software actually works is utilized in testing

– Testing based on inputs and respective outputs

– The size of the black box can vary from one class or component to a wholesystem

• White-box(structural, logic-driven)

– Testing based on knowing the inner structure of the system and theprogram logic

Dynamic and Static Testing

• Dynamic methods - testing - executes code

– Software testing in the traditional sense: running and using software

– Dynamic analysis methods

• Static methods do not execute code

– Reviews, inspections, static analysis; testing the specification is black-box

static testing

– Some authors name these static testing

Performing a High-level Review of The Specification

• Pretend to be the customer

• Research existing standards and guidelines

– Corporate terminology and conventions

• Specification Terminology Check lists

– Always, every, all, none, never (think of cases thatviolate them!)

– certainly, Therefore, Clearly, Obviously, Evidently(Don’t fall to the trap)

– Some, sometimes, often, usually, ordinarily, tomarily, most, mostly (too vague to be tested)

cus-– Etc, and so forth, so on, such as (aren’t testable)

– Good, fast, cheap, efficient, small, stable tifiable, need to explain)

(unquan-– Handled, processed, rejected, skipped, eliminated(these hide some functionalities that need to bespecified)

– If then (no else) (incomplete)

40

Testing The Software with Blinders On

Topics:

• What is dynamic black box testing

• How to reduce the number of test cases by equivalence partitioning

• How to identify troublesome boundary condition

• Good data values to use to induce bugs

• How to test software states and state transitions

• How to use repetition, stress, and high loads to locate bugs

• A few secret places where bugs hide

Dynamic Black-box Testing: Testing The

Software While Blindfold

• Testing software without having an insight into the details of underlying code;

program is running - testing is dynamic

• Test cases are specific inputs to try, and the procedure to follow

• Test case generation techniques described here belong to the black-box

testing category, using the software requirement/specification

• These techniques are useful during functional testing where the objective is to

test whether or not an application, unit, system, or subsystem, correctly

implements the functionality as per the given requirements

Test-to-Pass and Test-to-Fail

• Test-to-pass to assure that the softwaredoes what it’s specified to do inordinary circumstances

• Test-to-failis to design and run test caseswith the sole purpose of breaking thesoftware

Use test-to-pass to reveal bugs before you test-to-fail!

43

Equivalence Partitioning

• Selecting test cases is the single most important task for the software testers

• Equivalence classing is a mean to select test cases that divide input domain of

a program into classes of data Equivalence Partitioning strives to define a

test case that uncoversclasses of errorsto reduce the total number of test

cases that must be developed

• Example: windows calculator

44

Equivalence Classes Definition

Equivalence classesdefined with the guidelines:

• if an input condition specifies a range, one valid and two invalid equivalenceclasses are defined

• if an input condition requires a specific value, one valid and two invalidequivalence classes are defined

• if an input condition specifies a member of a set, one valid and one invalidequivalence classes are defined

• if an input condition is Boolean, one valid and one invalid equivalence classesare defined

Data Testing

Software can be viewed as to consists of two part: the data (its domain) and the

program Testing on the data is to check that information the user inputs, results

he receives and any interim results internal to the software are handle correctly

Examples of data would be

1 The words you type into the word processor

2 The numbers entered into a spreadsheet

3 The number of shots you have remaining in your space game

4 The pictures printed by your photo software

5 The backup files stored on your floppy disk

6 The data being sent by your modem over the phone line

Data Testing II

• Boundary is the edges of software capabilities; ifthe software can work on the boundary, it certainlycan work on normal condition

• Boundary conditions are special because ming by its nature, is susceptible to problems at itsedges

program-Types of boundary Conditions

• When you’re presented with a software test problem that involves identifyingboundaries, look for: Numeric, speed, character, location, position, size,quantity

• And think about the following characteristics: First/Last, Min/Max, Start/Finish,Over/Under, Empty/Full, Shortest/Longest, Slowest/Fastest, Soonest/Latest,Largest/Smallest, Highest/Lowest, Next-to/Farthest-from

47

Testing The Boundary Edges

• Software is susceptible to bugs at the boundaries, choosing data from the

boundaries to include in the equivalence partition will help to find more bugs

• Create two equivalence partitions:

– one contains data expected to work properly (valid points inside the

boundaries)

– the other contains data expected to cause errors (invalid points outside the

boundaries)

48

Other Special Data Values

• Sub-Boundary Conditions (internal to the software, but need to be check bythe testers)

• Values Default, Empty, Blank, Null, Zero, and None

• Invalid, Wrong, Incorrect, and Garbage Data

State Testing I

• A software state is a condition or mode that the software can be in, the current

values of the program variables

• Testing the Software’s Logic Flow: to traverse all paths to all states - it is

impossible for complex software Apply equivalence partition techniques to

select states/paths

State Testing II

• Creating a State Transition Map, A state transition map should show:

– Each unique state that the software can be in

– The input or condition that takes it from one state to the next

– Set conditions and produced output when a state is entered or exited

• Reducing the Number of States and Transitions to Test

• What to Specifically Test: defining test cases

51

Testing States to Fail

Find test cases that test the software to fail, examples are:

• Race Conditions and Bad Timing (e.g in multitasking OS)

• Repetition, Stress, and Load (repeated load and save, low, high conditions on

resources, etc.)

52

Other Black-Box Test Techniques

• Behave like a dumb user

• Look for bugs where you’ve already found them

• Think like a hacker

• Follow expertise, intuition and hunches

Examining The Code

Topics:

• The benefits of static white-box testing

• The different types of static white-box reviews

• Coding guidelines and standards

• How to generically review the code for errors

Static White-Box Testing: Examining The Design and Code

• White-box (or clear-box) testingimplies having access to the code, being able

to see it and review it

• Static white-box testingis the process of carefully and methodically reviewingthe software design, architecture, or code for bugs without executing it It’ssometimes referred to as structural analysis

• Benefits:finding bugs early, and those bugs difficult to uncover and isolate withdynamic black-box testing; giving the team’s black-box tester ideas for testcases to apply

• The unfortunate things:not always done, time-consuming, too costly, etc Now,more and more companies carry out white-box-testing (model checking, etc.)

• Follow Rules (e.g the amount of code for reviewing)

• Prepare (contributions to the review)

• Write a Report

56

Formal Review II

In addition to finding problems, holding formal reviews has a few indirect results

• Communications (e.g to the black box testers)

• Quality (improvement of code)

• Team Camaraderie (better understanding each other’s job for testers andprogrammers)

• SolutionsFormal review is the first net for catching bugs!

Formal Review III

• Peer Reviews (informal discussion: programmers who designed and testers)

• Walkthroughs (the programmer who wrote code presents to a small group of

4,5 programmers and testers)

• Inspections (most formal, presenters are not original programmers)

What Is The Characteristics of ’Peer reviews’?

• Informal

• Audience: Owner(author) and peer(s) of similar expertise (no managers)

• It requires guidelines and checklist

59

What Is The Characteristics of The

’Walkthrough’?

• More formal

• Audience: 3-5 , at least 1 senior, the owner act as a reader and recorder

• The owner should send a copy of things that need review prior to the meeting

• Reader should write a report with the feedback and how to address it

• 2nd meeting can be conducted to follow up

60

What Is The Characteristics of The

’Inspection’?

• The most formal

• Audience: typically 3-8 people including a moderator, reader, and a recorder

to take notes

• The reader should not be the owner

• The owner should send a copy of things that need review prior to the meeting

• Each Inspectors should review the code for a different perspective

• After the inspection meeting the moderator call for another meeting to discussthe feedback and how the owner will address them

• Require training

Coding Standards and Guidelines

There are problems where the code may operate properly but may not be written

to meet a specific standard or guideline or some criteria Three reasons for

adherence to a standard or guideline:

• Reliability: code written to a specific standard or guideline is more reliable and

secure that code that isn’t

• Readability/Maintainability: code written to a specific standard or guideline is

easier to read and understand

• Portability: easier to move it to different platform

Example of Programming Standards and Guidelines

For main parts of the standard:

• Titledescribes what topic the standard covers

• Standard(guideline) describes the standard or guideline explaining exactlywhat’s allowed and not allowed

• Justificationgives the reasoning behind the standard

• Exampleshow simple programming samples

63

Examples

TOPIC 3.05 Control-reduction on control structures

STANDARD

Thegotostatement should not be used

Thewhileloop should be used instead ofdo-whileloop

If a singleif-elsecan replace a continue, anif-elseshould be used

• American National Standards Institute (ANSI), www.ansi.org

• International Engineering Consortium (IEC), www.iso.org

• International Organization for Standardization (ISO), www.iso.ch

• National Committee for Information Technology Standards (NCITS),www.ncits.org

• Association for Computing Machinery (ACM), www.acm.org

• Institute of Electrical and Electronics Engineers, Inc (IEEE), www.ieee.org

Generic Code Review Checklist

• Data Reference Errors

• Data Declaration Errors

• Computation Errors

• Comparison Errors

• Control Flow Errors

• Subroutine Parameter Errors

• Other Checks

Testing The Software with X-ray Glasses

Topics:

• What dynamic white-box testing is

• The difference between debugging and dynamic white-box testing

• What unit and integration testing are

• How to test low-level functions

• The data areas that need to be tested at a low level

67Dynamic White Box Testing

• Examining the code, and watching it as

it runs - testing the software with X-ray

glasses!

• Other name: structural testing (you can see

and use the underlying structure of the code

to design and run your tests)

• Knowing how the software operate will influence how you test!

• Four areas that dynamic white-box testing encompasses:

– Directly testing low-level functions, procedures, etc

– Testing the software at the top-level as a completed program

– Gaining access to read variables helps to determine if your tests are doing

as your thought

– measuring how much of the code you hit during running a test

68Dynamic White-Box Testing Versus Debugging

• Both Dynamic White-Box Testing and Debugging involve dealing with softwarebugs and looking at the code, but are different in their goals

• Dynamic White-Box Testing is to find bugs

• Debugging is to fix bugs

Testing The Pieces

• Why Testing the Pieces? It is because testing the whole with dynamic black-box

testing is very costly, and bugs are found late, and may hide each others with

the techniques

• Unit and Integration Testing

– unit testing (or module testing) occurs at the lowest level

– after the lowest level bugs are found and fixed, units are integrated and

integration testing is performed (against the group of modules)

– the process of incremental testing continues until the entire product is

tested -system testing

• The problem of NASA Mars Polar Lander, 1999 caused by poor integration

testing

V-Model and Incremental Testing

Unit and Intergration Testing

System Testing

Requirement Analysis

System Design

Program Design

Coding

Acceptance Testing

and Maintenance Operation

component

• Types of components for testing: individual functions or methods within anobject, object classes that have several methods& attributes, compositecomponents made up of several different objects or functions

• Unit test considerations: module interface, local data structures, boundaryconditions, independent paths, error handling paths

Example: WeatherStation State Diagram

Calibrating

Testing

TransmitingSummarising

weather summarycompletedone test completeOperation

State Diagram for the Object

Example: WeatherStation Object

WeatherStationidentifier

reportWeather()calibrate (instruments)test()

startup (instruments)shutdown (instruments)

The Object

75

What to be tested?

• We need to define test cases for individual methodsreportWeather, calibrate,

test, startup, shutdown(in isolation or test sequences when necessary)

• We need to test the states We have to identify sequences of state transitions

to be tested:

Shutdown → Waiting → Shutdown

Waiting → Calibrating → Testing → Transmiting → Waiting

Waiting → Calibrating → Waiting → Summarising → Transmiting → Waiting

etc

76Interface Testing

Test cases

C

Types of Interfaces

1 Parameter interfaces (data and function references are passed between

components)

2 Shared memory interfaces

3 Procedural interfaces (one component encapsulates a set of procedures that

can be called by other component)

4 Message passing interfaces (services are requested by sending messages)

Types of Interface Errors

1 Interface misused

2 Interface misunderstanding

3 Timing errors

79

Guidelines for Interface Testing

1 Examine the code to be tested and explicitly list each call to an external

component Design a set of test cases where the values of the parameters to

the external components are at the extreme ands of their range

2 Where the pointers are passed across an interface, always test the interface

with null pointer parameters

3 For a procedural interface, design tests that cause the encapsulating

component to fail

4 use stress testing for message passing interfaces

5 For sharing memory interfaces of a set of components, design tests that vary

the order in which the components are activated

80

System TestingTwo distinct phases to system testing:

1 Integration testing

2 Release testing

Integration Testing

• Integration testing is a systematic technique for constructing the software

architecture while at the same time conducting tests to uncover errors

associated with interfacing (to make sure components actually work together)

• Big-bang approach and incremental integration approach

• Bottom-up integration, top-down integration, and regression testing

Incremental Integration Testing

Datais one of the two parts of Software, data includes all the variables, data

structures, input/output devices

P = principle amount r = annual interest rate

n = number of times the t = number of years

interest is compound per year A = amount of money after time t

• n = 0? Precondition is not satisfied

• n 6= 0? Precondition is satisfied

• Depending on how is the function used?