1. Trang chủ
  2. » Luận Văn - Báo Cáo

Luận án tiến sĩ: Development and analysis of comprehensive website software test plan information packages

91 0 0
Tài liệu đã được kiểm tra trùng lặp

Đang tải... (xem toàn văn)

Tài liệu hạn chế xem trước, để xem đầy đủ mời bạn chọn Tải xuống

THÔNG TIN TÀI LIỆU

Thông tin cơ bản

Tiêu đề Development and Analysis of Comprehensive Website Software Test Plan Information Packages
Tác giả Diane L. Stottlemyer
Người hướng dẫn Al Endres, Ph.D, Robert Haussmann, Ph.D, Michael Ewald, Ph.D, Joey Wallace, Ph.D
Trường học Northcentral University
Chuyên ngành Business & Technology Management
Thể loại Dissertation
Năm xuất bản 2006
Thành phố Prescott
Định dạng
Số trang 91
Dung lượng 8,77 MB

Nội dung

Using the current IEEE and augmented test planrequirements’ guidelines, a comparison was made between the current standard and theaugmented test plans’ information packages relative perc

Trang 1

DEVELOPMENT AND ANALYSIS OF COMPREHENSIVE WEBSITE SOFTWARE

TEST PLAN INFORMATION PACKAGES

A Dissertation submitted to

The graduate faculty of the Department of Business & Technology Management

In candidacy for the degree of

DOCTOR OF PHILOSOPHY

By

DIANE L STOTTLEMYERPrescott, Arizona

November 2006

Trang 2

alignment can adversely affect reproduction.

In the unlikely event that the author did not send a complete manuscript

and there are missing pages, these will be noted Also, if unauthorizedcopyright material had to be removed, a note will indicate the deletion

®UMI

UMI Microform 3243181 Copyright 2007 by ProQuest Information and Learning Company All rights reserved This microform edition is protected against unauthorized copying under Title 17, United States Code.

ProQuest Information and Learning Company

300 North Zeeb Road

P.O Box 1346 Ann Arbor, MI 48106-1346

Trang 3

Diane L Stottlemyer

Trang 4

NORTHCENTRAL UNIVERSITY

APPROVAL

As Dissertation Chair, I certify that the committee listed below haveread this dissertation and approve it as adequate in scope and

quality for the degree of Doctor of Philosophy

Candidate: Diane Stottlemyer

Title of Dissertation: DEVELOPMENT AND ANALYSIS OF COMPREHENSIVE

WEBSITE SOFTWARE TEST PLAN INFORMATION PACKAGES

Trang 5

Development and Analysis of Comprehensive Website Software Test Plan Information

a series of development steps and evaluation rubrics used as a means for test plan

generation and review These steps can be used in addition to the current Institute ofElectrical and Electronics Engineers standards Developer and evaluator participants were

randomly chosen from students and mentors at Northcentral University Findings from

the experimental design indicated that there was evidence of an increase (p=.15) inquality associated with the addition of the proposed maintenance and measurementstandards The additional standards could be used to enhance the website test planning

generation and review process

Trang 6

I would like to thank Dr Al Endres my committee chair for his patience as we wentthrough this process He is an excellent source of information and undying commitmentand understanding I would like to thank Dr Ewald and Dr Haussmann for taking thetime to review my paper I would also like to thank Donna Bridgham, Ned Campbell and

Dr Joey Wallace for their support and patience as I progressed through this process

Trang 7

IiIL9)0.1332)09)(02.1010717577 xi

®; 8Ð) 0n 1Statement of Problem 0n 2Definition of Key TT€TTT HH HT TT gà HT ng 3Brief Review of Related LIterafUF€ cty TH ng ng nh nh ngờ 5Highlights and Limitations of Methodology cong Hy Hy HH ki Ho §Research EXpEeCfalOTNS cong Hà TT Hà nà g0 01p g0 gà 9

Chapter 2: Review of Related Literature sành ng HH ng ng nen 10

90 ca, 10Software Test Strategy and PÏlanning «ngọn ng ng kg 10Software Mainf€'iafiC€ HH HT TH HT TT nàn Hàng 15RUïA 5-80 J2 2130041, NA 17SÌUTTTTAV nh nọ TH HH TT HT TH HH nv 18

Chapter 3: Methodology 0 TT 19

0 2,2 nố ốố ố ố ố ẻ ố ẻ 19Restatement of the Problem - - - sọ ng HT ng ng ng nu ca HH gi re 19Statement of HypotheSeS, HH HH HH TH Tà ngàn nem 20Description of Research Design ‹ HH HH HH HH nàng ve20Application of Power 20.1) 0006 22TPQ power 0 e6 /((/dd431 22TSU power nh “6e 22Operational Definition of Constructs and Key Variables nen 23Description of Materials and Instrurnerif§ - ng ng cư24Selection of ))1:)1.v: DA 270.201 NA na 27

vi

Trang 8

Discussion of Data PfoC€SSINB HH HH The 0kg 28Methodological Assumptions and LimifafÍO'S nhàn ng ngưng 29Ethical A SSUTäTIC€S HH ng họ TT HH HT ii 30

Chapter 4: Flindings ch go gi Tà net 32OVCTVICN, Q0 nh HH HH Tu TH TT họ Ti 9E 32FEndinngSs PP 32Research Question 1 and Associated Hypotheses án nen 32Research Question 2 and Associated Hypotheses - Hee 33Data Collection and RÑ€SUÏIS - cuc HH HH TH Hà nàn Hà 34Summary of results for research quesfion .- co vn vs key 38Summary of results for research question - «cu snvnvsns 38Analysis and Evaluation of Findings HH ng ng ng 40SUMIMALY ÊPP0Ẽ7ẼẺỀ 8 6 4I

Chapter 5: Summary, Conclusions and RecommendatiOns - se, 43QUITITđFY HT HH họ Họ ch T1 001 0 000909 43COncÏUSIOTS Út HH HH TH in 47RecomrmmnendafiOT0S - sọ TH Tp 49

R€fET€TC€S LH ng HH nh HH như Error! Bookmark not defined

Appendix A: IEEE Standards Test Plan Information Package co 58Appendix B: IEEE+ Standards Test Plan Information Package sscseee 64Jin 69Exhibit C1: Application for Approval for the Use of Animals or Humans 69Exhibit C2: Approval for the Use of Animals or Humanss ssssssecseesseeseneeseneesenee 70Exhibit C3: Letter of Inquiry sent to NCU students, staff and mentors for Study 71Exhibit C4: Online Consent Form for Research Project ssssssssssseseseesesessseeseseenenee 72Appendix D: Test Plan Specification Information Package Instructions 74Appendix E: Results for TPQ Data Collected for IEEE and IEEE Enhanced TPSIP 75Appendix F: Instructions for Evaluating IEEE Standards - son 77Appendix G:Instructions for Evaluating IEEE+ Standards csescessssssssscsesessensenees 78

Appendix H: MINITAB TM Results for TPQ and TSU Hypothesis Testing 79

vii

Trang 9

Table 1 Input Parameters/Sample Size Requirements for TSU Power Analysis 22Table E1 IEEE Enhanced Spreadsheet Data Results scecceesesseeeseneeeennes 75Table E2 IEEE+ Spreadsheet Data Results for TPQ Test Plan Specification InformationJ2 e e 76Table H1.Test and CI for Two PrOpOTIOTNS HH HH HH HH fees 79Table H2.Test for Equal Variances for IEEE and IEEE+ Test Results 79

viii

Trang 10

LIST OF FIGURES

Figure 1 Flow diagram of research Study - cu HH HH ng Hang kg 26

Figure 2 Boxplot template comparing the IEEE and IEEE+ TPIP quality data sets 36

Figure 3 Normality probability plot of IEEE totals for 31 samples -.-«- 37

Figure 4 Normality probability plot of IEEE+ totals for 32 samples 37

Figure 5 F-Test for equality of variances for TSU IEEE and IEEE+ data 39 Figure 6 Test for equal variances TSU for IEEE and IEEE+ .0 cc eecssecssssesceseeseneeenes 40

Trang 11

ETPDP — Enhanced Test Plan Development PToC€SS HH HH Hye 3IEEE - The Institute of Electrical and Electronics Engin€€rS «co neeeeesree 3IEEE+ - The Institute of Electrical and Electronics Engineers’ Standards with Two

Additional Proposed Standards - cà HH HH HT Hà Hà TH BH 4I9) 2.9121 4TPDP - Test Plan Development Process LH HH0 1v 4TPSIP —Test Plan Specification Information Package - -‹ Hy gu ng, 5TSU - 8 ‹o À0 ii an 5World Wide Web Consortium (W3C) «Hà nàn HH HH HH Hà Hàng 5

Trang 12

LIST OF APPENDICES

Appendix A: IEEE Standards Test Plan Information Package seo 58Appendix B: IEEE+ Standards Test Plan Information Package con 64Appendix C, - 5 HH HH TT HT Hà TH HH TT HT TT Hà Hà nàn Họng 69Exhibit C1: Application for Approval for the Use of Animals or Humans 69Exhibit C2: Approval for the Use of Animals or Humans Ăn 70Exhibit C3: Letter of Inquiry sent to NCU students, staff and mentors for Study 71Exhibit C4: Online Consent Form for Research PTOJ€C( 0 cúng 72Appendix D: Test Plan Specification Information Package Instructions 74Appendix E: Results for TPQ Data Collected for IEEE and IEEE Enhanced TPSIP 75Appendix F: Instructions for Evaluating IEEE Standards ccsscscsesecsesereeeenenees 77Appendix G: Instructions for Evaluating IEEE+ Standards ccccccsseseeses 78Appendix H: MINITAB TM Results for TPQ and TSU Hypothesis Testing 79

xi

Trang 13

This dissertation focused on the development of a new series of web site test plan

specification guidelines for use as an extension to existing standards and guidelines outlined

in the Institute of Electrical and Electronics Engineers (IEEE) standards number 829(Schmidt, 2000) The IEEE provided standards which include guidelines for developingand verifying software and web sites The augmented software standards include

guidelines for software developers and testers to use in designing test plans that have

consistency in appearance, form, structure and content The resultant dissertation has

provided a series of development steps and evaluation rubrics that can be used as a means

to establish a process for test plan generation and review

The research has identified and analyzed key areas as enhancements to the IEEE

standards The additions have addressed maintenance and measurement test planning.Both of these subject areas were analyzed for their potential to improve the current IEEEtest plan guidelines for developing, examining, and testing web sites The purpose of the

research was to evaluate, analyze, and enhance the existing IEEE guidelines, and to

provide an enhanced test plan specification development and assessment process for

developers The additional maintenance and measurement guidelines may also be useful

for verifying, validating, and improving preexisting requirements and/or test plans after

the initial guidelines from the IEEE have been utilized

A web-site test plan specification information package development process was

designed to provide developers with a series of steps for preparing a test plan The series

of steps reflected a reference to both the current IEEE standards as well as the additionsfor the standards and guidelines An assessment rubric was designed for both the current

Trang 14

and augmented sets of standards, and used to evaluate and critique the test plan

specification information packages Using the current IEEE and augmented test planrequirements’ guidelines, a comparison was made between the current standard and theaugmented test plans’ information packages relative perceived quality, and the relative

uniformity of the test plan information packages quality scores

Statement ofProblem

Since there are no maintenance or measurement standards in the IEEE test

planning, developers and testers do not measure the percentage of errors during

development or conduct maintenance after the application is released So therefore it is

necessary to have unified and/or standardized processes/guidelines that can be used to

provide standards for consistency throughout the Information Technology industry TheIEEE is considered a leader in the field of engineering standards and guidelines for

development and testing “The Institute of Electrical and Electronics Engineers (IEEE)

Software Engineering Standards provides a comprehensive set of standards for

developing and validating software” (Schmidt, 2000, p 1)

Research was undertaken to examine if a relationship exists between the IEEE

standards and the additional proposed standards, maintenance and measurements Thisstudy was used to determine whether the inclusion of an additional set of test plan

specification information package guidelines and assessment rubrics was significantly

correlated with an increase in measured perceived test plan specification informationpackage quality, and increased uniformity of the test plan specification information

package evaluation process

This study therefore addressed the following research questions:

Trang 15

be positively correlated with an increase in perceived test plan specification information

package (TPSIP) quality?

2 To what extent will the use of a test plan specification information package

assessment rubric and the associated test plan guidelines increase the uniformity of the

evaluations of test plan quality?

The focus of this quantitative study was to examine if a relationship exists

between the IEEE standards and the addition of maintenance and measurements The

study was used to address the issue of increasing test planning quality by includingmeasurements and maintenance which provided a more comprehensive and uniform

approach to developing test plans An enhanced test planning process in the maintenance

phase of the software/web site lifecycle can provide designers, developers and testerswith a methodology for both preventing and detecting errors that may not have been

detected by the existing IEEE standards in earlier phases of the lifecycle Juran (1999)

indicated that this may be the case when he stated that “with the increasing complexity

and criticality of computer programs, traditional ad hoc methods for testing have become

inadequate” (p 5)

Definition of Key Terms

Assessment Rubric —The forms used to grade each test plan specification information

package and to assign a point value for any “yes” answer The forms consisted of 8 or 10

specific standards depending on the test plan specifications that were evaluated The

forms were intended to provide a uniform and consistent approach to evaluation

ETPDP — Enhanced Test Plan Development Process — This is the test plan process thatwas used to facilitate the preparation of the test plan information packages that were

Trang 16

presented in an online format using the enhanced standards along with the IEEE

standards

IEEE - The Institute of Electrical and Electronics Engineers — This organization setstandards for Engineers for standardization of practices IEEE’s mission is: “The Institute

of Electrical and Electronics Engineers (IEEE) promotes the development of

electro-technology and allied sciences, the application of those technologies for the benefit ofhumanity, the advancement of the profession, and the well-being of its Members.” (IEEEMission, 2005, p.1)

IEEE+ - The Institute of Electrical and Electronics Engineers’ Standards with TwoAdditional Proposed Standards - IEEE+ was the development process of a test plan

template which consisted of standards from the IEEE web site augmented by two

additional standards developed for measurement and maintenance, and a detailed

development template

TPQ - Test Plan Quality — TPQ is a construct that was used to measure the scores from

the assessments that were tallied A spreadsheet containing the headings, answers andscaled results was used for entering the results for statistical analysis Hypothesis testing

was used to assess the significance of the differences in TPQ conformance percentages

associated from using the IEEE test planning standards and the results obtained from theuse of IEEE+ standards with the additional test planning standards and the developmenttemplate respectively

TPDP - Test Plan Development Process - TPDP is a construct that was used to

operationally define a method that was used for developing a test plan specification

information package for web-site software development It included various components

Trang 17

tested, the test plan methodology required to perform the test, specific testing tasks,equipment required and those responsible for conducting the testing process.

TPSIP —Test Plan Specification Information Package — TPSIP is a construct that was

used to define a grouping of information that was needed to evaluate the development ofthe test plan specifications

TSU - Test Scoring Uniformity - TSU is a construct that was used to determine whetherthe test scoring process was associated with any significant difference for the variance ofthe TPQ scores

World Wide Web Consortium (W3C) — The World Wide Web Consortium (W3C) is an

international consortium where Member organizations, a full-time staff, and the public

work together to develop Web standards W3C's mission is: “To lead the World Wide

Web to its full potential by developing protocols and guidelines that ensure long-term

growth for the Web” (W3C Consortium, 2004, p.1)

BriefReview of Related Literature

In addressing the general rationale for test planning, Juran (1999) discussed the

importance of defining quality as a part of the process for testing and validation “Testingmust ensure that the system operates correctly in the actual environment or, where such

testing is not possible, in an environment that simulates the conditions of actual use.” (p

5) For this reason it is necessary to have a test process that can simulate a real-time

environment With careful test planning and structure, this type of testing would help

prevent and reduce errors, and inconsistent processes, by following a formalized test plan

development and implementation process Juran asserted that test planning should be

Trang 18

conducted in conjunction with a product development plan so that the product planspecifications can be built into the test plan more accurately (p 18).

Juran also stated that a typical product test plan outline would include the

following elements: safety review and test, certifications, test to product requirements,test to application requirements, maintainability/serviceability and documentation (p 18)

The addition of control inputs and product readout conditions also need to be considered

as a part of software test planning These elements focus on testing the hardware

components of products

Schmidt (2000) discussed the importance of establishing standards for test

planning and how the IEEE has established standards for software The IEEE is

responsible for developing software engineering standards such as customer and

terminology standards, process standards, product standards and resources and techniquestandards He stated that, “Proper use of the standards can help organizations improve

their software development and validation process and help implement an effective

software quality system” (p 1)

Schulmeyer (1999) discussed how a software quality assurance plan should be

designed by including the following information; “Objectives, References, Risks,

Responsibilities, Quality assurance tasks, Schedule, Metrics, Supplier, Customer andSubcontractor SQA support, Quality records and reports and Problem solving”

(Schulmeyer, 1999, p 339) “The plan will serve as the driving force for how the SQA

(Software Quality Assurance) organization will function on the project and what

activities they will take in parallel with other software development activities” (p 343)

Trang 19

the software development lifecycle process This template provides a "planning stage of

the software system tests" (p 228) The template can be used as a tool to help sort

information into various phases such as the scope of the tests, the testing environment,the test details such as objectives, requirements and the schedule for the testing This type

of data collection is important for ensuring that the objectives and goals of the testing arecarried out in a procedural fashion

Microsoft (2005) has suggested that test plans are necessary for documenting test

cases for application development Two types of test plans are suggested, the detailed test

plan document and the detailed test case document Both types of documentation are

examples used to verify and validate test cases and also for assuring coverage of specific

aspects of testing which include the design, review and testing of an application

Kaner (2004) discussed the importance of test planning and documentation as a

process to record and document the testing process and present the results of testing

process "Testers should document manual tests in great procedural detail so that they can

be handed down to less experienced or less skilled testers" (p 3) Kaner also suggested

that test case documentation is important for managing the testing process and for

documenting the results of the testing

It was apparent from the literature that there is a need to prepare test plans that

can be a part of a standardized product development process The literature indicates that

there are several different approaches to test planning and there is not a single standard

that is used throughout the industry,

Trang 20

In summary, based upon the initial literature review, there was a general

agreement that it was beneficial to prepare a standardized test plan as a part of both thehardware and software/web site lifecycle process Furthermore, the expected benefits will

be manifested through reduced defects and the concomitant increase in percentages of

higher-quality hardware and software web sites released within budget and schedule

Highlights and Limitations ofMethodologyThis study has analyzed the potential efficacy of the additional areas of test

planning, and provided a uniform web site test plan development process The

hypothesized efficacy of the enhanced standards, the uniform test plan developmentprocess, and the assessment rubrics can provide web site developers with both a more

comprehensive and standardized process for developing web site test plans A factorialexperimental design methodology was chosen to provide the most assurance of

identifying the dependent and independent variables’ actual relationships The followingconstructs and variables were reflected in the final experimental design: Test Plan Quality(TPQ) — Dependent Variable, Test Plan Development Process (TPDP) — Independent

Variable, Test Scoring Uniformity (TSU) — Dependent Variable and Test Plan

Specification Information Package (TPSIP) — Independent Variable The data was

collected and analyzed using the variables associated with the constructs as a method for

analyzing their effects on TPQ and TSU

This research analyzed the effects of the independent variables for preparing and

evaluating web site test plans This research did not address all possible types of test

plans required for all possible types of web sites It may not have been possible to

eliminate all potential bias from this study since there was subjective evaluation of thetest plans The research was restricted to studying the selected factors and other factors

Trang 21

Also only NCU learners and faculty participated in the evaluations.

arch ExpectationsThis study was expected to provide the following results:

1 An increased understanding of the relationships between the Test PlanDevelopment Processes and perceived Test Plan Quality

2 An increased understanding of relationships between the use of Test

Plan Evaluation Rubrics and the uniformity of the test plan informationpackage scoring

The reliability of this study was assured through the use of the original standardswhich were the IEEE test planning guidelines, and uniform operational guidelines forusing the addition of the maintenance and measurement test planning guidelines Thesestandards were used to determine their usage as test plan specifications The purpose ofthis research was to evaluate the efficacy of using the additional standards, a test plan

development process, and scoring rubrics, that can potentially be used by designers,

developers and testers for developing more uniform and effective web site test plans

Internal validity was assured through the use of a formal experimental design thatwas used to test and analyze the relationships between the dependent variables TPQ andTSU and the independent variables External validity was addressed by including

reasonably flexible procedures for the test plan information package development

processes, test plan quality, and finally randomly selected test plan developers and

evaluators with diverse experience levels

Trang 22

Chapter 2: Review of Related Literature

OverviewThe following literature review presents various methods and strategies for test

planning The literature review demonstrates the rationale and current approaches used

for hardware and software test plan development The literature review was organized

into the following subject areas: Software Test Strategy and Planning, Software

Maintenance and Software Measurements, and the Summary

Software Test Strategy and PlanningSoftware documentation is an important part of the testing process It was clearfrom the literature that there are many different forms of documentation However test

planning may be considered a foundation for the testing process Lengel (2001) has stated

“Test coverage is about insuring that test plans and test cases include information vital forsuccessful testing of the program in the areas of functionality, performance, and theoverall quality of the software In addition, test managers who prepare test plans that

provide proper test coverage can avoid the wrath of a project manager whose

implementation has just gone sour or an angry customer whose system has just crashed”

(p 1)

Lengel (2001) posted that adequate test planning should include test scenarios thatrelate to the requirements and specifications of the testing Lengel (2001) also stated as

you prepare a test plan that the initial focus should involve a process that captures

scenarios from the available documentation This would involve collecting the

requirements that are relative to the development of the software or application andmaking sure that testing is performed to address the necessary requirements for the best

results

10

Trang 23

Software QA Associates (2001) has suggested that it is important to ensure thatthere are adequate plans for testing that provide detailed processes and procedures.

Software QA Associates (2001) has also stated that “Documentation of the software testprocess should generally describe: 1) the plans and priorities for the testing, 2) the test to

be performed, and 3) results of those tests” (p 3)

The results of the testing should be used to improve product performance andapplications Software QA Associates (2001) asserted that it is important to use metrics

when testing to determine whether there is a relationship between two or more variables

and to determine the effectiveness of a process By using metrics as a part of the testplanning process it becomes possible to determine the effectiveness of the testing process

and the overall effectiveness and usefulness of the application or product

Software QA Associates (2001) has provided examples of metrics that would beuseful as a part of the test planning process They include test coverage analysis, defect

analysis, bugs and test cases By collecting and analyzing the above types of metrics theremay be a decrease in the error ratio and an increase in the usability and reliability of thesoftware It is therefore clear that by incorporating measurements and maintenance

requirements in test plans this may lead to better software

Careful test planning is critical and essential for software development and

maintenance Campbell (1993) stated that “We must carefully plan the testing effort in

order to best use the time and money we do have for testing” (p 2) This is an important

concept Often when test time is budgeted it is generally one of the first things cut fromthe budget due to lack of time and resources Many managers do not see the need to

Trang 24

budget for adequately testing software; therefore it is essential to have an effective test

plan that outlines the key objectives necessary for implementing and testing software

Campbell (1993) also stated that it is important to write a test plan that is easy touse The test plan is a useful document to provide guidance and direction for the testingprocess Test planning is critical and essential to ensure that full and effective testing is

performed that will adequately reflect the usability, reliability and implementation of thesoftware for customers’ use

It is important that there be evaluation criteria which can be used to analyzetesting results Campbell stated that “Results must be examined and evaluated You candetermine results by running a baseline test Perform, the test on the current software,

make changes, perform the test again and compare the results” (p 4) This type of testingshould be a part of the test planning process and account for selective measurements thatwill be used to evaluate the baseline run of the test with successive test runs to account

for variations and changes This will also help with maintenance testing because thebaseline data will be a means for making test result comparisons for the software duringactual use and also be used to help improve the testing process

There are many different types of software documentation that are useful forsoftware development and test processing Delgado (1992) asserted that there are four

types of plans that are useful and necessary for quality software: Quality assurance plans,

Development plans, Quality control/Test plans and Configuration plans Each type of

plan provides an essential component for software development The Quality Assurance

plan is used for process improvement This plan is useful for focusing on the vision,

continuous quality improvement, cost of quality, staffing issues, resources and

Trang 25

measurements (Delgado, 1992) The Development plan is used for project managementand technical aspects such as the "life cycle process, process analysis, process analysis,

technology, requirements, quality factors, productivity and standards" (Delgado, 1992,p.10) The Quality control test plan is used to determine if the system is meeting the

requirements This plan is used for testing, evaluating the test environment, test strategy,

error recording and test analysis (Delgado, 1992) The Configuration plan is used toidentify and order the process for creating the product It is important to identify, control,evaluate and audit the product using this plan

After reviewing the different types of test planning documentation references it

was apparent that there was an overlapping theme of required: information that was

needed to fulfill the software documentation for each plan The QA plan presents

measurements as a necessary part in providing a quantitative approach to techniques forquality assurance Delgado (1992) stated that a QA plan provides a method that can be

used to work well with the observation of errors and system efficiency which correspondwith the quality of the test The testing results, such as error counts and system efficiencydata, can be gathered as a result of QA plan development, test planning and processing

The quality control/test plan should require error recording as necessary for adequate

tracking and as means for defining and driving necessary corrective action(s).

Test strategies and adequate planning are necessary for good testing Craig (1999)

suggested that in planning for testing it is not critical to create a lot of test cases, but

instead to ensure that the test strategy and the risks and priorities of the test are

adequately addressed The process of planning a test strategy involves the review of

business requirements and application specifications In order to perform good test

Trang 26

planning it is important to understand that the goals and objectives of the test plan must

support the project plan

Craig (1999) also stated that “in test planning, the process is ultimately moreimportant than the product” (p 1) Since the process is considered an important part of

test planning then all parts of the software development process should be considered for

inclusion in the test planning process This would include measurements and

maintenance Furthermore, Craig (1999) has stated that the test plan template should beappropriately modified to meet the needs of the organization Craig (1999) has also statedthat although the IEEE test plan is a good template to use, all areas may not be applicable

to the organization’s goals and objectives In addition, Craig (1999) mentioned that the

plan should be considered a special process for any given project and may need to bemodified to meet each specific test process Therefore it is important to consider

additional aspects of test planning that will work well for different organizations

Test-driven development is a good tool for ensuring that all phases of

development incorporate a form of testing Goldsmith (2002) asserted that test planning

as presented by the IEEE may be viewed as no more than a “counterproductive

paper-generator” (p 1) Although a test plan needs to be a part of the development process, it

also needs to be useful and help the testing efforts This may be accomplished by treatingthe test plan as a working document that provides a step-by-step current approach to

testing Goldsmith (2002) asserted that it is essential to record test plan information to

enhance reuse and continuous improvement for testing

Goldsmith (2002) has also concluded that many developers viewa test plan as a

collection of test cases This should not be the case, the test plan should be written as a

Trang 27

guide for conducting the test process that can be used throughout the entire development

and testing lifecycle Goldsmith (2002) considered this “proactive testing” (p 2) This

form of test planning is useful for guiding the testing process through each phase of

development which may produce a higher quality software product

Software Maintenance

In addressing the general rationale for maintenance software planning, Favre

(1999) defined eight phases of the software maintenance life cycle process These eight

phases are; modification request, estimation, planification, implementation, test,

document, delivery and operation It is noted that during the test phase, testing strategiesare defined and tools may be chosen

Support for software maintenance follows a maintenance process which includes;

production version, problem report, problem analysis, maintenance design, maintenance

implementation, test version, regression test, documentation, acceptance test and

delivery Each activity contributes to the overall maintenance process for software

Testing is considered a core maintenance activity and contributes to the overall

maintenance process

During the problem analysis, the development of the test plan is considered

critical to ensuring that the maintenance request contains information about the test

process During the maintenance phase of software development, testing is a critical part

of the overall planning and implementation of software development and should be a part

of the test planning process

Favre (1999) stated that “any software maintenance organization must define a

testing plan, create a base of test cases, automatize the testing process when possible (e.g

testing tools)” (p 42) This suggests the importance and need for test plans and their use

Trang 28

during the testing process Favre (1999) also considered metrics to be a critical part of the

support for software maintenance and are important considerations for how to measure

those measurements that are relevant to software maintenance According to Favre

(1999), measurements are useful for evaluating quality, estimating costs, and provide ameans to control and improve the software maintenance process for long term projects

Because of the importance of metrics and measurements careful test planning is

necessary.

The University of Toronto (2005) has developed a software maintenance processthat contains a presentation on software maintenance, cost, types of maintenance,

problems facing maintainers, approaches to maintenance and reuse According to the

University of Toronto, “Maintenance consumes 40% - 80% of software costs” (p 3) Due

to the large investment in the cost of software maintenance there should be a software testplan that incorporates a process that includes testing throughout the maintenance

planning phase

Since there are different types of software maintenance such as corrective

maintenance, adaptive maintenance, perfective maintenance and preventative

maintenance, test planning is a critical part of each phase A test plan should be used to

outline and accommodate a testing process for each maintenance phase

According to the University of Toronto (2005) information that is useful for

helping the maintenance process requires documentation that supports the release, testing

and the ability to debug and test as the maintenance process is performed This indicatesthat a formal test plan would be beneficial for the maintenance phase of the software

process Maintenance is considered a time intensive effort for software development It is

Trang 29

noted in the article from the University of Guelph (1995) that “The maintenance of

existing software can account for over 60 percent of all development change” (p 1)

Because a large percentage of effort is spent on this effort, it is important to consider howthe maintenance of software is planned and implemented into the software development

life process and software test planning The University of Guelph (1995) has also stated

that “70 per cent of the cost of software is devoted to maintenance” (p 1) Due to thehigh percentage of cost involved in the maintenance of software applications, it is

essential to fully plan and document the process and procedures that are needed for

testing and maintaining software within the test planning process

Software MeasurementsMetrics are critical to the software development process The University of

Guelph (1996, p.1) noted that “Metrics are management tools which are used to estimate

the cost and resource requirements of a project” Because of the criticality of metrics, it is

essential to understand the types of metrics that adequately reflect the project’s goals andobjectives for the type and phase of software development that are necessary

The type of measurements that may be used for software projects should be based

on the requirements for the project Metrics can be used to “estimate costs, project tasks,and a project schedule” (University of Guelph, p.1) This is all essential information that

requires adequate planning Furthermore, the University of Guelph (1996) stated thathistorical data can be used to successfully predict and assist in planning and estimation ofmeasurements, This type of data collection is necessary to adequately measure and

analyze data which means careful planning is necessary to testing and development

Mcquaid (2004) discussed the importance of measurements and also mentioned

that the measurements used for a project should match the goals and outcomes for a

Trang 30

project Not all measurements are useful indicators for software projects and Mcquaid(2004) concluded that careful planning is necessary Mcquaid (2004) presented the adagethat “no one plans to fail, yet many fail to plan“ (p 2)

The use of a successful measurement program should become a required feature

of the planning process for the software development lifecycle process which also shouldinclude the software testing planning process This should not be viewed as “an overheadprocess” (Mcquaid, 2004, p 2) but a vital part of the development process for software

development and testing Mcquaid (2004) also concluded that careful planning is

essential and that measurements can assist in the planning process in many cases

Summary

In summary, based upon the literature review, there was an apparent consensus

that it is beneficial to include a standardized test plan as a part of both the hardware andsoftware/web site lifecycle process Based on the findings from the literature review, itwas apparent that there is a need for at least some form of measurement and maintenance

process to be integrated with the software test plan preparation Furthermore, the

expected benefits of adding a form of measurement and maintenance will be manifestedthrough reduced defects and a higher percentage of higher-quality hardware and software

web sites being implemented within budget, and delivered on schedule

Trang 31

In this chapter a description of the research problem was discussed and the

research questions are presented The research design, research instruments, and themethods of data analysis are also defined

Restatement of the ProblemSince there are no maintenance or measurement standards in the IEEE test

planning, developers and testers do not measure the percentage of errors during

development or conduct maintenance after the application is released So therefore it is

necessary to have unified and/or standardized processes/guidelines that can be used to

provide standards for consistency throughout the Information Technology industry Thisresearch was used to determine whether the inclusion of an additional set of test planguidelines, a test plan development process, and assessment rubrics was positively

correlated with an increase in measured perceived test plan information package quality,and increased uniformity of the test plan package evaluation process The effect of each

of these factors, and their potential interactions, was estimated to determine if their use

was associated with an increase in perceived test plan information package quality andevaluation uniformity This study addressed the following research questions:

1 To what extent will the use of the augmented testing and maintenance standards

be positively correlated with an increase in perceived test plan specification information

package (TPSIP) quality?

2 To what extent will the use of a test plan specification information package

assessment rubric and the associated test plan guidelines increase the uniformity of theevaluations of test plan quality?

19

Trang 32

Statement of Hypotheses

This study presented two test plan formats (Appendix A and Appendix B) andused an evaluation to determine if there was a significant increase in perceived test planinformation package quality by adding a formal development process with maintenance

and measurements to the test plan development process Following are the null

hypotheses and alternative hypothesis statements to reflect the study:

H01 ~ The addition of the standards and development process for IEEE+ TPSIP

preparation will be associated with either the same or decreased levels of TPSIPperceived quality

H11 — The addition of the additional standards and development process for

IEEE+ TPSIP preparation will be associated with a significant increase in TPSIP.H02 - The addition of the standards for IEEE+ test plan preparation,

use of a test plan assessment rubric and test plan guidelines, will be associated

with either the same TSU or a decrease in TSU

H12 - The addition of the additional standards for IEEE+ test plan preparation,use of a test plan assessment rubric and test plan guidelines, will be associatedwith a significant increase in TSU

Description of Research Design

A factorial experimental design methodology was chosen to provide the greatest

assurance of properly addressing the research questions, and for identifying the truedependent and independent variables’ relationships “The importance of experimental

design also stems from the quest for inference about causes or relationships as opposed tosimply description” (Sytsma, 1999, p 1) The research for cause and effect was at thefoundation of research The following constructs and variables were reflected in the

Trang 33

experimental design: Test Plan Quality (TPQ) — Dependent Variable, Test Plan

Development Process (TPDP) - Independent Variable, Test Scoring Uniformity (TSU) —

Dependent Variable and Test Plan Specification Information Package (TPSIP) —

Independent Variable Data generated from the experimental design were:

1 The distributions of the perceived TPQ conformance scores

2 The variances of the TSU (Test Score Uniformity) scores

To address the first research question and its associated hypothesis, the

randomized experimental design was used as the basis for determining the data collection

sequence and identifying any significant differences between the effects on perceived testplan quality (TPQ) associated with the use of two sets of standards (one for the IEEE and

one for the IEEE+) and using a design development template TPQ scoring data collected

from the evaluation groups was used as the dependent variable This information wasused to determine if the use of the augmented testing and maintenance standards was

positively and significantly correlated with an increase in perceived test plan quality

To address the second research question and its associated hypothesis, the secondstatistical analysis for the dissertation focused on the test scores’ uniformity TSU (Test

Score Uniformity) The relative variation in the TSU scores was used to determine

whether there was an advantage (characterized by lower scoring variation) associatedwith using a detailed test plan development template and an enhanced standard scoring

rubric versus the scoring variation associated with the IEEE test plan results and a

macro-level scoring rubric The reason that a one-tailed F-test was selected for TSU was to

determine if the combination of the enhanced test plan development process and scoring

rubric were associated with an increase in TSU uniformity Therefore only an increase in

Trang 34

TSU was of interest, and a one-tailed F-test was chosen to test the null hypotheses

However for both the TPQ and TSU hypotheses, P values were listed for both the onetailed and two-tailed alternatives

The evaluation procedures used two sets of standards and two evaluation rubrics

The statistical analysis utilized the data generated by evaluating the test plans informationpackages perceived quality from using the current JEEE standards and the additionalstandards using the IEEE+ which reflected the use of a test plan development template.Two sets of test plans were prepared and evaluated to determine if there were any

statistically significant improvement in either perceived TPQ and/or TSU associated withusing the new standards, augmented by a test plan development template, and the

enhanced scoring rubric

Application of Power Analysis

TPQ power analysis For TPQ ,MINITAB TM was used to evaluate the power for

the sample size for two proportions using a test for two proportions The results from

MINITAB TM one-tailed power analysis for a one-sided test for two- proportions were

calculated as 0.8 power with an alpha of 05 using a sample size of 32

TSU power analysis G* Power (Erdfelder, 1996) was used for the final poweranalysis and sample size determination for TSU This software tool was used to analyzethe power for the TSU F Test by estimating the sample size required for the final targeted

power level for the F test for comparing TSU evaluation test scores’ variability The Ftest method employed G*Power The results are listed in Table 1:

Table 1

Input Parameters/Sample Size Requirements for TSU Power Analysis

| Effect size f | 36 |

Trang 35

Alpha 05

Power 8

Reguired Sample Size | 64 (32 per group)

As shown in Table 1, for TSU, the sample size requirements were 32 test plan

information packages for IEEE and 32 test plans for IEEE+ One group was used todevelop test plans information packages for IEEE, and the second group was used todevelop test plan information packages for IEEE+ using an enhanced test plan

development template

Operational Definition of Constructs and Key Variables

Test Plan Development Process (TPDP) - A test plan development process wasoperationally defined as the method that was used for developing the specific areas of a

test It includes various components such as an introduction to the web site or application,

defining the specific features to be tested, the test plan methodology required to perform

the test, specific testing tasks, equipment required and those responsible for conducting

the testing process

Test Scoring Uniformity (TSU) - Test score uniformity was operationally defined

as the consistency of scoring of the evaluating test plan information packages It reflected

the use of an enhanced standard scoring rubric for evaluating the test plan results or the

scoring variation associated with the IEEE test plan results and a macro-level scoring

rubric

Test Plan Quality (TPQ) - Test plan quality was operationally defined as theperceived percentage conformance scores of the Test Plan Specification InformationPackages

Trang 36

Test Plan Specification Information Package (TPSIP) — Test Plan SpecificationInformation Package was operationally defined as a grouping of information that wasneeded to evaluate the development of the test plan specification package elements

Description of Materials and InstrumentsThe materials and instruments for the experiment included an online survey

document which was used to enter data, accumulate the results, and provided a resultspage of the data for input to the subsequent statistical analysis Instrumentation for the

experimentation consisted of a data collection web site which was used for the designed

test plan templates, one for the IEEE and one for the IEEE+ test plans There were twoweb sites designed where test plan information packages were prepared, entered, and the

data collected for the IEEE and IEEE+ test plan packages The IEEE test plan

information package development process was implemented using a macro level test planinformation package development guideline process The IEEE+ test plan process was

implemented using a detailed level test plan development guideline process Both web

sites http://www.leesol.addr.com/dissertation5/index.html, and

http://www.leesol.addr.com/dissertation6/index.html, were used to gather data associated

with the 64 test plan information packages Thirty two packages were prepared and

collected from the IEEE+ and 32 test packages were prepared and collected from theIEEE The IEEE+ test plan package template was located at:

http://www leesol.addr.com/dissertation5/standards.htm, and the IEEE test plan packagetemplate were located at: http://www leesol.addr.com/dissertation6/standards.htm

The IEEE template consisted of standards that were contained in the current IEEE

test plan standards The IEEE+ test plan development template consisted of standards

from the IEEE web site augmented by two additional standards developed for

Trang 37

measurement and maintenance, and a detailed development template TSU scores were

generated from the evaluations by eight separate participants who evaluated eight testplans (four IEEE and four IEEE+) using a basic scoring rubric for the IEEE test plans,and an enhanced scoring rubric for the IEEE+ test plans

The templates used on the data collection web site were used only to record the

test plan information for each category The web site was designed so that all the

experimental design data were entered, collected, recorded, and summarized The design

for the data collection web site was capable of producing a results page that was used to

automatically send an email of the results for each test plan entry to an assigned emailaddress

Each of the eight evaluators evaluated the same eight test plans using the scoringrubrics that were designed so that they were able to allocate scores for each test plan

category This evaluation was used, in conjunction with the development and evaluation

templates, to provide percentage conformance scores

The IEEE scoring rubric process was developed using macro level scoring

guidelines The IEEE+ scoring rubric process was developed using detailed scoring levelguidelines A macro-level flow diagram of the entire research study’s instrumentation and

procedures is contained in Figure 1

Trang 38

Figure 1, Flow diagram of research study

Administrative _ Ins trumentation

Results Accumulated and

Results Accumulated and sent for rubric evaluation sent for rubric evaluation

Rubric evaluation for Rubric evaluation forIEEE test plans IEEE+ test plans

Results Accumulated Results Accumulated

L

Results Analyzed |

The instructions were presented on the web site that described the purpose, design

of the experiment, and a discussion of the procedures for filling out the test plan template.The web site was used to accumulate the data and results of the experiment The process

Trang 39

and materials were pre tested to determine operational feasibility, and to ensure ease of

understanding of the scoring and data entry processes

Selection of SubjectsSixty four test plan developers were randomly selected The participants were

randomly chosen from students and professionals at NCU Ideal candidates were thosewho were working in the field of information technology as web developers,

programmers or testers It was deemed acceptable to include a few participants outside

the field of web development, but they were limited in their understanding of testing and

web design

Procedures

The web sites were tested before the experimentation to make sure that all linkswere working, that data could be collected and stored for reuse The process involved

ensuring that both URL's for the test plan could be viewed in a browser Simulated data

were entered into both test plans to ensure that when the submit button was clicked that aconfirmation page displayed and also that the information was emailed to a central

repository for collecting the data Each web page had its own email address so there was

not any mixing of the data The rubrics’ results also had their own email addresses forcollecting the data Data for comparing the relative quality of the test planning processes

and the uniformity of evaluation processes using the constructs TPQ, TPDP, TPSIP and

TSU was obtained by using:

1 Sixty four randomly selected participants who were assigned to use the current

or augmented standards, and use, or not use, the test plan development template/flow

diagram The participants were provided instructions for the evaluation (Appendix D)

Trang 40

2 Assigning randomly selected participant reviewers to score test plans againstboth sets of standards by first using the developed rubric with a rating from one to eightfor the IEEE standards and one to ten for the IEEE standards augmented with the

standards for each test plan category/ element The first scoring process that was used

was a simple percentage scale; that reflected the inclusion of whether or not the standard

versus category/element of standard was included, and to assess the appropriate usage ofthe standard

3 Assigning a corresponding proper subset of the reviewers to score a set of thetest plan information packages developed from both the current and augmented standardsusing the basic and enhanced scoring rubrics that were developed for assessing test plans

developed from the current IEEE standards, and test plans that was developed from the(IEEE+) augmented standards

Each of the randomly selected TPQ evaluators submitted their results to the datacollection web site When the user submitted the form they were tracked by their remotename The page links were inactive so the user could not go outside of the immediate

window and were only able to submit the actual form that was provided with the URL

TPQ scores were tallied, and a spreadsheet containing the headings, answers and scaledresults was used for collecting the results for statistical analysis

Discussion of Data Processing

MINITAB TM and an Excel TM spreadsheet from Aczel and Sounderpandian

(2002) were used for the evaluation of the data resulting from the experiment MINITAB

TM was used for analyzing the TPQ data The input data were the 32 individual test plans

for IEEE and 32 individual test plans for IEEE+ TPQ attribute scores were used to testfor significant differences in the two-proportions of the plans’ maximum scores using a

Ngày đăng: 02/10/2024, 01:45

TÀI LIỆU CÙNG NGƯỜI DÙNG

TÀI LIỆU LIÊN QUAN