Software testing introduction (full)

The main purpose of this test is to verify the requirements for the 9000 Advanced Color Module as set forth in References 2 and 3 Advanced Color Module SRS and Common Image Generator SRS

SOFTWARE TESTING

Nguyen Thanh Binh

Danang, 2010

Test Plan for (Project name)

Confidential and Proprietary Information of Datacard Worldwide

1.1 Objective

The Test Plan is an aggregation of information, which describes the entire test activity for this project It covers the entire testing effort (unit, development test, system verification test, and Beta) It identifies the product requirements, schedules, resource requirements (people, effort and equipment), quality, assumptions, exclusions, and risks

A preliminary Test Plan is prepared for the Project Team during the System Phase of PEAQ Process This Test Plan will be updated in the earliest possible time of the Implementation Phase, so that progress can be tracked during implementation

Specify also what types of tests are planned for the project Types of testing include specification, functional, limits, stress, destructive, compatibility, performance, documentation, network and system Referenced Documents

[All references used to produce this document should be listed here Examples are Software Requirements Specification, Product Definition Document, Software Development Plan These references should be listed with part numbers.]

2.0 Assumptions/Dependencies

[All assumptions for carrying out this test effort successfully are listed here Some requirements assumptions might be necessary in order to scope the test activities Also, assumption of responsibility to conduct unit, integration, SVT, regression, and beta tests

Also listed here are the external dependencies, such as code completion by a certain date in order to meet the test schedule Other dependencies might include prototype available and functional by a certain date.]

3.0 Test Requirements

[Itemize a list of system test requirements as extracted from Software Requirements Spec, Product Definition Document, and possibly Software Design Document Every requirement should be listed as an item This list will be used to trace test cases and procedures that verify/validate the requirements

developed or purchased tools If some software tools need to be developed, describe the process to be used The schedule and resource requirements must be identified and included in the sections that follow.]

Effort expended during DVT, SVT and Regression

# of defects uncovered during DVT, SVT and Regression, and development phase each defect is attributable to

Test tracking S-Curve

PTR S-Curve

After shipment:

# of defects uncovered and development phase each defect is attributable to

Size of software

[To estimate the resource, all test activities must be identified and resources needed to accomplish the activities estimated Detailed estimates will be shown here This consists of identifying all project test activities by the Test Group and the number of hours estimated to accomplish these activities Be specific Show specific responsible test engineer’s names, if possible A grand total of the effort must be shown here, as well as in Section 5.0.]

[Attach two charts, viz Gantt and PERT In Gantt, main activities are shown as a list on the Y-column with bars parallel to the X-axis, showing the timeframe to perform activities In PERT, dependencies of each activity must be identified.]

System Verification Test Plan for

Advanced Color Module

Rev 2.0

2/22/00

Appendix A - Test Plan Requirements Matrix

Appendix B - Test Cases

This document describes the test plan for the 9000 Advanced Color Module and includes information on what is

to be tested, and how the testing is to be accomplished (test methodology) Specifically, this document describes the tests to be performed, the testing schedule, resources required, entry criteria, exit criteria, dependencies, test tools, metrics and the Test Plan Requirements Matrix This is a living test plan and must be changed to reflect Core Team needs and requirements as they arise

The main purpose of this test is to verify the requirements for the 9000 Advanced Color Module as set forth in

References 2 and 3 (Advanced Color Module SRS and Common Image Generator SRS)

2 Test Description

2.1 Description

Much of the functionality that was handled by the module firmware and DLL has been relegated to the Common Image Generator, and much of what is in the CIG is being tested by the UltraGraphics Module tests Thus every attempt will be made not to overlap too much with what is being covered by UltraGraphics For example, all the image positioning commands are handled by the CIG and will be tested by UltraGraphics, making it unnecessary

to stress this aspect of color photo printing here

System Verification Test will be broken into three phases:

Entrance Testing will verify that the new Advanced Color Module can process a standard set of color images that use the 4 hybrids of Version A and B headers See Appendix B.2 for a description of the Entrance Test cases

Main Test will thoroughly verify the operation of the Advanced Color Module software and hardware

Main Test determines that all the requirements in the SRS have been satisfied

The Main Tests will consist of both new tests written for this SVT effort as well as tests selected from the current

9000 Library of test cases The following types of testing, as specified in Reference 4, will be included:

 Specification testing

 Functional testing

 Compatibility testing

 Documentation review

See Appendix B.3 for a description of the Main Testing test cases

Regression Test will be performed as a subset of Main Test to verify the integrity of the Advanced Color Module after all problems found during Main Testing have been attended to This means that all Severity 1 and 2 defects have been fixed and verified and that the product is ready for release

See Appendix B.4 for a description of the Regression Test cases

2.2 References

1 Software Development Process Handbook (no revision, no date)

2 Software Requirements Specification for the Advanced Color Module, May 13, 1999

3 Software Requirements Specification for the Common Image Generator, May 28, 1999

The testing defined in this document was completed as follows:

System Verification Test will require the following resources:

The following consumables are required in order to complete testing:

5 Dependencies

In order to begin testing of the Advanced Color Module, the following needs to happen:

In addition, effort, size and defect data will be collected prior to and after product shipment Once data from enough projects has been collected, estimates of testing progress and duration will become more meaningful

11 Test Plan Requirements Matrix

The Test Plan Requirements Matrix lists all the testable requirements for the Advanced Color Module, as

determined from Reference 2, and indicates which test cases verify those requirements The complete test plan requirements matrix is in Appendix A; the associated Test Case Matrix is in Appendix B.3

12 Document History

Rev P1 - 07/08/99

Initial Draft

Rev 1.0 - 07/30/99

Color Management requirements were pared down per a conversation with Colleen

The entire Appendix B was filled in

Entries in Appendix A were clarified

Rev 1.1 - 12/7/99

Updated the Main Test matrix in Section B.3

13 Definitions and Acronyms

Requirement

number

section(s) of SRS

Notes

profile

3.9, 3.9.1, 3.9.2, 3.9.3

ACM-02-18 through -29

6-H Ability to import an Image color profile 3.14.1 See File Management test cases

listed anywhere that I can tell ACM-06-11 through -13

Requirement

number

section(s) of SRS

Notes

Appendix B.2 Entrance Test

The Entrance Test Case Matrix for the Advanced Color Module SVT is in a separate document titled acmentr.doc There are a total of 72 test cases defined

Appendix B.3 Main Test

The Main Test Case Matrix for this SVT is in a separate document titled acmsvt.doc There are a total of 508 test cases defined

Appendix B.4 Regression Test

Regression Testing will consist of a subset of Main Testing based on those areas where problems were found The Test Case Matrix for ACM Regression Test is in a separate element titled acmreg.doc; a total of 132 test cases are defined

Appendix B.5 Test Case Procedures

The Test Case Procedures for Entrance Testing and Main Testing are in ACM-nn.DOC where nn is 01, 02,

03, 04, 05 or 06, depending on the requirement

Table of Contents

1 TEST REQUIREMENTS 1

1.1 OBJECTIVE 1

1.2 EXPLICIT REQUIREMENTS 1

1.3 IMPLICIT REQUIREMENTS 1

1.4 REQUIREMENTS MATRIX 1

1.5 TEST CASE MATRIX 2

1.6 TRACEABILITY MATRIX 3

1.7 REFERENCE DOCUMENTS 3

1.8 DEFINITIONS AND ACRONYMS 3

2 TEST CASES 4

2.1 DESCRIPTION 4

2.2 TEST CASE TEMPLATE 4

3 APPENDIX A 4

3.1 SAMPLE REQUIREMENTS,TEST CASE AND TRACEABILITY MATRICES 4

4 APPENDIX B 4

4.1 SAMPLE TEST CASES 4

1 Test Requirements

1.1 Objective

The purpose of the Test Requirements section is to list ALL hardware and software test requirements, whether explicitly determined from any relevant documents or implicitly determined from experience and product knowledge For most projects, the documents referred to may be the Product Definition Document, Software/Hardware Requirements Specification and perhaps the

Software/Hardware Design Specification The format used to list the requirements will be that of a Requirements Matrix and associated Traceability Matrices (TM) The Requirements Matrix shows which section of the requirements document the

requirement may be found in, and the Traceability Matrix shows which test case(s) verify the various requirements

In addition, a Test Case Matrix is provided that simply lists all the test cases by title or description, and includes a method of tracking when the test case was run and whether it passed or not

1.2 Explicit Requirements

Explicit requirements for the product can usually be recognized by the use of such helping verbs as “will/will not”,

“should/should not”, “must/must not”, etc All explicit requirements should be listed in the Requirements Matrix along with the section(s) of the document where the requirement is specified Once listed, a requirement should not be deleted from the matrix If the requirement is removed from the SRS, for example, change the section designation to read “Removed in Ver xx.x of the SRS”, where xx.x denotes the relevant version of the document Similarly, added requirements could be annotated

as “Added in Ver xx.x of the SRS” along with the section number

This particular section of the Test Requirements/Test Cases document may be used to indicate how the explicit requirements were determined

1.3 Implicit Requirements

Implicit, or implied, requirements are generally determined by experience Typical implicit requirements are performing

Endurance Testing and verifying Documentation updates While not specifically mentioned in the SRS, they are nevertheless important parts of product completeness Implicit requirements should be listed in the matrix with a notation of “Imp” to indicate that there is no specific section of the document that specifies this requirement

This section of the Test Requirements/Test Cases document may be used to indicate how the implicit requirements were determined

1.4 Requirements Matrix

This section contains the Requirements Matrix; a template for such a matrix is shown below The columns of the Requirements Matrix have the following meanings:

document is used to determine requirements, the column heading could specify something like

“All references are from the SRS unless otherwise specified”, or the appropriate document could

be listed with each section designation

relevant to this requirement

1.5 Test Case Matrix

This section contains the Test Case Matrix for the project The purpose of the Test Case Matrix is to list all test cases defined for this project, including Entrance Test Cases, Main Test Cases and Regression Test Cases A separate matrix may be used for each phase of SVT

The columns of the Test Case Matrix have the following meanings:

2.1), the module ID should be incorporated in the Test Case ID

case of that type Some Test Case types are:

Other examples may be found in the Structured Software Test Planning at DataCard, participant manual from

Benchmark Laboratories Incorporated, August 1996

Date last attempted / PTR#: The date that this test case was last attempted, along with the relevant PTR number if the test

case failed

The template for each Test Case Matrix is as follows:

Date Complete

The order in which test cases are listed in the matrix is not intended to indicate the order in which they must be run

An example of an actual Test Case Matrix is shown in Appendix A

Describe in full all documents used for determining requirements

1.8 Definitions and Acronyms

List any technical terms or acronyms used in the document, along with their meanings

Examples for this document:

2 Test Cases

2.1 Description

Test Cases are developed to verify the various requirements listed in the Requirements Matrix Various Traceability Matrices are then constructed to show the correspondence between Requirements and Test Cases For convenience, test cases may be logically grouped together in what are called Test Modules, based on areas of testing The use of Test Modules helps make the Traceability Matrices and archiving Test Cases more manageable

2.2 Test Case Template

Test Case ID Test Case Title

The underlined Test Case ID may be any convenient identifier, as decided upon by the tester Identifiers should follow a consistent pattern within Test Cases, and a similar consistency should apply across Test Modules written for the same project See Appendix B for an example

method of recording whether or not the expected results actually occurred, i.e if the test case, or even individual steps of the test case, passed

should be mentioned here Similarly, any dependency on factors outside the immediate test environment should also be mentioned

to succeed, such initialization should be mentioned here

description of the test case, along with a Test Procedure, which in turn can be specified by test case steps, tables of values or configurations, further narrative, or whatever is most appropriate to the type of testing taking place

At this point, the actual Test Case steps may be listed, or relevant data tables, or any other information that will help in

executing the Test Case successfully See Appendix B for examples Note that Appendix B gives examples of various ways of specifying the test procedure associated with a test case

Owner: Test/Tools Group

Expected Results: The data will be loaded without errors, and will be stored correctly

as follows: START /n HOSTPOLL -s10

Step 1: Click on 2 Data In on the main CSM menu bar

Step 2: Select an <Input Devices> of HOSTIN

Select a <Data Setup> of DEFAULT

Press ALT + V on the keyboard, then press the Up and Down Arrow keys until HOST

appears in the <Device Setup> drop-down list

Click on the Exit button

Step 3: Switch to an OS/2 window

Copy drive:<directory>\htest.01 to C:\csm\datain\host\htest.01

Type ‘exit’ and press Enter

Step 4: Click on 3 Status on the main CSM menu

loaded into the CSM

Step 5: Click on the Exit button

Step 6: After the data has loaded, click on 1 Production

Step 7: Click on HOST in the <Production Groups> list box

sequence number beginning with ‘H’

Step 8: Click on the Exit button

Step 9: Switch to an OS/2 window

Do a DIR of directory C:\csm\datain\host

values will be honored

[On test platform Continuation 2, Job Setup ‘2image’ takes care of the above Card and Data Setups, along with Machine Setup ‘2image-a’.]

On Continuation 2, the Machine Setup is ‘2image-a’ and the appropriate Module Profile is hs- 01-07-00.ocp

job of 16 cards each as shown in Table 3 below The jobs are then produced on the 9000

Table 3

Window Title Bar Test Case(s) Findings

Citibank Queue Detail

will be generated”

“Insert Recs” should be “Insert Count”

Under the Begin button description, if you are denied access via security, the Button isn’t even present

Data Input Summary

Device Information

Device Utilization by Shift

Under RECORDS AND FIELDS: “have a separate fields” → “have

separate fields”

FIR Item Descriptions

Global Processing Exceptions

Global Tagged and Held

Job Processing Exceptions

Item 4 under DELETING SPLIT JOBS: Actually, it’s the OK button that gets pressed more than once

Job Status

Job Tagged and Held

Part B: Color Header data on an HSACM

Part A: UltraGraphics Language commands on the Advanced Color Module

Part B: UltraGraphics Language Color commands

Requirement

number

section(s) of SRS

Test Cases

profile

3.9, 3.9.1, 3.9.2, 3.9.3

ACM-02-18 through -29

6-I Ability to import a color filter 3.14.2 See File Management test cases

listed anywhere that I can tell ACM-06-11 through -13

<<Date of Last Revision Here (Should match last entry below)>>

<<Author’s Name Here>>

<<Company/Business Unit Name Here>>

Test Group

When printing this document, the “Comments” checkbox (via Tools-Options-Print tab) is checked so that

you’ll get the “instructions” that the comments contain You should disable this option when doing the

version of the Test Report the is made publicly available Of course, you will also remove this text

Template Revision History (Note: to be cleared out when issue a Test Report as don’t

need readers seeing the history for this Template version)

Gawarecki

Original distribution 1.1 Feb 1998 Dan G Incorporate suggestions by others, & misc additions and corrections

1.2 Oct 14,

1998

Dan G Added Section 1.1 “Size Measurements”, Title Page, & Table of Contents Section 4.3, item 2

changed to report Actual as well as Estimate test hours

1.3 Nov 2,

1998

Dan G Section 3- changed “Opened” to “Total Opened” and added new “Still Open, break by severity”

item, Made table out of information asked for in Section 4.2 “Builds”

1.4 Nov 3, 1998 Lyn B Changed Header Title text from “Test Plan” to Test Report”

1.5 99-Sep-03 Dan G Implemented Test Council Recommendations:
§1.1, Remove item 1 as no one using PEAQ

category sizes anymore §2, added more details to original items- they are now # 1-4, and 6, added item that is now #5 §3, added “at the time…” to item 6 §4.1, made into table adding Planned and Actual for Column Names §4.3, for item #2, modified to be a table so Planned and Actual are easy to read. Added italic instructions to title page  Revised §5 1st

paragraph, added 2nd paragraph

Document Location

1 OVERVIEW 1

a general comment about the testing status (passed with little difficulty, had lots of problems, etc.)

Hopefully the entire report, excluding any attached charts (e.g., from Section 3) is only 1-2 pages long

Software Size Estimates vs Actual

(identify units of measure- Thousand Source Lines of Code- KSLOC, number of

classes, objects, Function Points, etc)

This section records the status of Test Cases for System Verification Test This provides some “sizing” of

the project and helps future estimates Feel free to record any comments about Test Cases or related issues

here Ideally, you’d have an S-curve to embed in the document!

1 # Test Cases Planned (i.e., written up in plan)

2 # Test Cases Executed (i.e., attempted to run)

5 # Executions of Test Cases (i.e., summation of the number of times executed each test case-

do this because may execute a test case more than once This metric needed to show effort

and explain why a test cycle may be longer than planned This item could be considered

optional as it is something that a mature process would provide- which I’m not sure we have

entirely The other 5 items should be done first and done well before attempting to provide

this metric)

6 What test cases were planned but *NOT* executed (perhaps explain why?)

Similar to the previous Test Case section, this area records the status of PTRs Again, feel free to record

any comments about the PTRs you wish You should be able to get charts for items #1, 2 and the data for

items #3 & 4 from Tracker

1 PTR Trendline-

2 Severity breakdown/distribution

3 #PTRs Total Opened

4 #PTRs Closed

5 #PTRs Still Open (breakdown by severity)

6 Any open issues still remaining at the time of release (e.g., explain why you might be shipping

with an “open” Severity 2 PTR) Important to reference the actual PTR number(s) here

4.1 Dates

1 Date testing Started:

this is the first day of Entrance Testing

2 Date Testing Ended:

this is the last day of Regression Testing

Please fill in the 5 cells in the following table:

VERSION DATE DONE

1 First Build done

2 Last Build done

3 # Builds done during testing period

1 List all those people involved in testing the product during this project

2 Record Estimated and Actual number of hours spent testing (this is time executing test cases,

not “playing around” with the product or testing by “exploration”

PLANNED ACTUAL

Actual Hours Spent Test

Note if there was or was not a Beta test and if so, any results (good or bad) Refer to separate Beta

Test Report if available If there is none, mention any testing customers performed

This is a “catch-all” section for you to add any comments you feel important You might

re-emphasize any open issues listed in section 3, major stumbling blocks in the testing process, any

areas NOT tested, unique situations, etc Mention what documents, manuals, specifications etc

that were not tested

This is also your opportunity to comment on what improvements should be made (back it up with

data, or at least a rationalization) that will improve product quality in future versions of this

product or other products

4-28-00

Neil Bitzenhofer Datacard Worldwide

1 OVERVIEW 1

performed for Color Management, even though Color Management was not released with 9000 Ver 7.0

The size of the software involved is unknown at the time of this writing

1 Advanced Color SVT consists of 508 test cases 499 of those test cases have been run, and have passed

2 The 9 test cases that have not been run are as follows:

ACM-03-193 through -196: Image Distribution test cases No system has been set up with more than one

front or more than one back Advanced Color Module

ACM-05-13: Blind Spot testing Since the hardware is not significantly different from what it was for a

High Speed Color Module, this testing has not been performed

ACM-06-10: High Speed Advanced Color Module endurance testing The endurance of a single stand-

alone Advanced Color Module did not seem worth testing It is anticipated that this type of testing will be performed in a much larger system, e.g the System Integration test system

ACM-06-11, -12, -13: These refer to 3 document reviews Although the tester has not reviewed these

documents, it is expected that various development personnel have done so

1 PTR Trendline- See the attachment

4.1 Dates

1

Date testing Started:

this is the first day of Entrance Testing

2

Date Testing Ended:

this is the last day of Regression Testing

Please fill in the 5 cells in the following table:

VERSION DATE DONE

1 Personnel involved in testing the Advanced Color Module were Robert Ang, Neil Bitzenhofer, Mike Hazzard and Colleen Lanhart

2 Actual time spent testing , based solely on number of days used, times 8 hrs per day, times 80% load:

PLANNED ACTUAL

No attempt was made to add together the test hours for all 4 test personnel This information may be available to management via time card data

There were no Beta Test sites for the Advanced Color Module

1 One very big hindrance to this testing effort was that, on the Friday before SVT was to start, the new Print On Image Generation Error was successfully built This feature was not included in any of the currently-existing test cases, and hence caused a complete overhaul of the associated test cases (mainly all the error test cases) and associated requirements and test case matrices Under no circumstances should extensive new features (or perhaps features of any kind) be introduced at the very end of DVT

Software testing

Nguyen Thanh Binh

IT Faculty, Danang University of Technology

Email: ntbinh@dut.udn.vn Website: www.dut.edu.vn/itf/ntb

2

Prerequisites

phases of a project would be helpful

References

1 Paul Jorgensen, Software Testing-A Craftsman's Approach, CRC

Press, 1995.

2 Spyos Xanthakis, Pascal Régnier, Constantin Karapoulios, Le test des

logiciels, Hermes Science, 2000.

3 Hung Q Nguyen and al., Testing application on the Web, John Wiley &

Sons, 2004.

4 Ilene Burnstein, Practical Software Testing, Springer, 2003.

5 Glenford J Myers, The art of software testing, Wiley, 2004.

6 Cem Kaner, Jack Falk, Hung Q Nguyen, Testing Computer Software,

2nd Edition, John Wiley & Sons, 1999.

7 Boris Beizer, Software Testing Techniques, International Thomson

Computer Press, Second Edition, 1990.

8 Neil Bitzenhofer, Software Testing and Verification, Course, MSSE, 2008.

9 Paul Ammann and Jeff Offutt, Introduction to Software Testing,

Cambridge University Press, Cambridge, UK, ISBN 0-52188-038-1,

2008.

Write a test plan

Introductions and expectations

Course overview

Contents

Contents

Definitions, Principles, Axioms

Stages of testing

Perspectives on Software Testing

Contents

Software Development Life Cycle (SDLC)

Software Development stage

Testing and requirements

Learn to think like a tester

White box testing / Structural testing

Graph Theory

Control flow criteria

Data flow criteria

Graph Coverage for Source Code

Testing State Behavior