Using the current IEEE and augmented test planrequirements’ guidelines, a comparison was made between the current standard and theaugmented test plans’ information packages relative perc
Trang 1DEVELOPMENT AND ANALYSIS OF COMPREHENSIVE WEBSITE SOFTWARE
TEST PLAN INFORMATION PACKAGES
A Dissertation submitted to
The graduate faculty of the Department of Business & Technology Management
In candidacy for the degree of
DOCTOR OF PHILOSOPHY
By
DIANE L STOTTLEMYERPrescott, Arizona
November 2006
Trang 2alignment can adversely affect reproduction.
In the unlikely event that the author did not send a complete manuscript
and there are missing pages, these will be noted Also, if unauthorizedcopyright material had to be removed, a note will indicate the deletion
®UMI
UMI Microform 3243181 Copyright 2007 by ProQuest Information and Learning Company All rights reserved This microform edition is protected against unauthorized copying under Title 17, United States Code.
ProQuest Information and Learning Company
300 North Zeeb Road
P.O Box 1346 Ann Arbor, MI 48106-1346
Trang 3Diane L Stottlemyer
Trang 4NORTHCENTRAL UNIVERSITY
APPROVAL
As Dissertation Chair, I certify that the committee listed below haveread this dissertation and approve it as adequate in scope and
quality for the degree of Doctor of Philosophy
Candidate: Diane Stottlemyer
Title of Dissertation: DEVELOPMENT AND ANALYSIS OF COMPREHENSIVE
WEBSITE SOFTWARE TEST PLAN INFORMATION PACKAGES
Trang 5Development and Analysis of Comprehensive Website Software Test Plan Information
a series of development steps and evaluation rubrics used as a means for test plan
generation and review These steps can be used in addition to the current Institute ofElectrical and Electronics Engineers standards Developer and evaluator participants were
randomly chosen from students and mentors at Northcentral University Findings from
the experimental design indicated that there was evidence of an increase (p=.15) inquality associated with the addition of the proposed maintenance and measurementstandards The additional standards could be used to enhance the website test planning
generation and review process
Trang 6I would like to thank Dr Al Endres my committee chair for his patience as we wentthrough this process He is an excellent source of information and undying commitmentand understanding I would like to thank Dr Ewald and Dr Haussmann for taking thetime to review my paper I would also like to thank Donna Bridgham, Ned Campbell and
Dr Joey Wallace for their support and patience as I progressed through this process
Trang 7IiIL9)0.1332)09)(02.1010717577 xi
®; 8Ð) 0n 1Statement of Problem 0n 2Definition of Key TT€TTT HH HT TT gà HT ng 3Brief Review of Related LIterafUF€ cty TH ng ng nh nh ngờ 5Highlights and Limitations of Methodology cong Hy Hy HH ki Ho §Research EXpEeCfalOTNS cong Hà TT Hà nà g0 01p g0 gà 9
Chapter 2: Review of Related Literature sành ng HH ng ng nen 10
90 ca, 10Software Test Strategy and PÏlanning «ngọn ng ng kg 10Software Mainf€'iafiC€ HH HT TH HT TT nàn Hàng 15RUïA 5-80 J2 2130041, NA 17SÌUTTTTAV nh nọ TH HH TT HT TH HH nv 18
Chapter 3: Methodology 0 TT 19
0 2,2 nố ốố ố ố ố ẻ ố ẻ 19Restatement of the Problem - - - sọ ng HT ng ng ng nu ca HH gi re 19Statement of HypotheSeS, HH HH HH TH Tà ngàn nem 20Description of Research Design ‹ HH HH HH HH nàng ve20Application of Power 20.1) 0006 22TPQ power 0 e6 /((/dd431 22TSU power nh “6e 22Operational Definition of Constructs and Key Variables nen 23Description of Materials and Instrurnerif§ - ng ng cư24Selection of ))1:)1.v: DA 270.201 NA na 27
vi
Trang 8Discussion of Data PfoC€SSINB HH HH The 0kg 28Methodological Assumptions and LimifafÍO'S nhàn ng ngưng 29Ethical A SSUTäTIC€S HH ng họ TT HH HT ii 30
Chapter 4: Flindings ch go gi Tà net 32OVCTVICN, Q0 nh HH HH Tu TH TT họ Ti 9E 32FEndinngSs PP 32Research Question 1 and Associated Hypotheses án nen 32Research Question 2 and Associated Hypotheses - Hee 33Data Collection and RÑ€SUÏIS - cuc HH HH TH Hà nàn Hà 34Summary of results for research quesfion .- co vn vs key 38Summary of results for research question - «cu snvnvsns 38Analysis and Evaluation of Findings HH ng ng ng 40SUMIMALY ÊPP0Ẽ7ẼẺỀ 8 6 4I
Chapter 5: Summary, Conclusions and RecommendatiOns - se, 43QUITITđFY HT HH họ Họ ch T1 001 0 000909 43COncÏUSIOTS Út HH HH TH in 47RecomrmmnendafiOT0S - sọ TH Tp 49
R€fET€TC€S LH ng HH nh HH như Error! Bookmark not defined
Appendix A: IEEE Standards Test Plan Information Package co 58Appendix B: IEEE+ Standards Test Plan Information Package sscseee 64Jin 69Exhibit C1: Application for Approval for the Use of Animals or Humans 69Exhibit C2: Approval for the Use of Animals or Humanss ssssssecseesseeseneeseneesenee 70Exhibit C3: Letter of Inquiry sent to NCU students, staff and mentors for Study 71Exhibit C4: Online Consent Form for Research Project ssssssssssseseseesesessseeseseenenee 72Appendix D: Test Plan Specification Information Package Instructions 74Appendix E: Results for TPQ Data Collected for IEEE and IEEE Enhanced TPSIP 75Appendix F: Instructions for Evaluating IEEE Standards - son 77Appendix G:Instructions for Evaluating IEEE+ Standards csescessssssssscsesessensenees 78
Appendix H: MINITAB TM Results for TPQ and TSU Hypothesis Testing 79
vii
Trang 9Table 1 Input Parameters/Sample Size Requirements for TSU Power Analysis 22Table E1 IEEE Enhanced Spreadsheet Data Results scecceesesseeeseneeeennes 75Table E2 IEEE+ Spreadsheet Data Results for TPQ Test Plan Specification InformationJ2 e e 76Table H1.Test and CI for Two PrOpOTIOTNS HH HH HH HH fees 79Table H2.Test for Equal Variances for IEEE and IEEE+ Test Results 79
viii
Trang 10LIST OF FIGURES
Figure 1 Flow diagram of research Study - cu HH HH ng Hang kg 26
Figure 2 Boxplot template comparing the IEEE and IEEE+ TPIP quality data sets 36
Figure 3 Normality probability plot of IEEE totals for 31 samples -.-«- 37
Figure 4 Normality probability plot of IEEE+ totals for 32 samples 37
Figure 5 F-Test for equality of variances for TSU IEEE and IEEE+ data 39 Figure 6 Test for equal variances TSU for IEEE and IEEE+ .0 cc eecssecssssesceseeseneeenes 40
Trang 11ETPDP — Enhanced Test Plan Development PToC€SS HH HH Hye 3IEEE - The Institute of Electrical and Electronics Engin€€rS «co neeeeesree 3IEEE+ - The Institute of Electrical and Electronics Engineers’ Standards with Two
Additional Proposed Standards - cà HH HH HT Hà Hà TH BH 4I9) 2.9121 4TPDP - Test Plan Development Process LH HH0 1v 4TPSIP —Test Plan Specification Information Package - -‹ Hy gu ng, 5TSU - 8 ‹o À0 ii an 5World Wide Web Consortium (W3C) «Hà nàn HH HH HH Hà Hàng 5
Trang 12LIST OF APPENDICES
Appendix A: IEEE Standards Test Plan Information Package seo 58Appendix B: IEEE+ Standards Test Plan Information Package con 64Appendix C, - 5 HH HH TT HT Hà TH HH TT HT TT Hà Hà nàn Họng 69Exhibit C1: Application for Approval for the Use of Animals or Humans 69Exhibit C2: Approval for the Use of Animals or Humans Ăn 70Exhibit C3: Letter of Inquiry sent to NCU students, staff and mentors for Study 71Exhibit C4: Online Consent Form for Research PTOJ€C( 0 cúng 72Appendix D: Test Plan Specification Information Package Instructions 74Appendix E: Results for TPQ Data Collected for IEEE and IEEE Enhanced TPSIP 75Appendix F: Instructions for Evaluating IEEE Standards ccsscscsesecsesereeeenenees 77Appendix G: Instructions for Evaluating IEEE+ Standards ccccccsseseeses 78Appendix H: MINITAB TM Results for TPQ and TSU Hypothesis Testing 79
xi
Trang 13This dissertation focused on the development of a new series of web site test plan
specification guidelines for use as an extension to existing standards and guidelines outlined
in the Institute of Electrical and Electronics Engineers (IEEE) standards number 829(Schmidt, 2000) The IEEE provided standards which include guidelines for developingand verifying software and web sites The augmented software standards include
guidelines for software developers and testers to use in designing test plans that have
consistency in appearance, form, structure and content The resultant dissertation has
provided a series of development steps and evaluation rubrics that can be used as a means
to establish a process for test plan generation and review
The research has identified and analyzed key areas as enhancements to the IEEE
standards The additions have addressed maintenance and measurement test planning.Both of these subject areas were analyzed for their potential to improve the current IEEEtest plan guidelines for developing, examining, and testing web sites The purpose of the
research was to evaluate, analyze, and enhance the existing IEEE guidelines, and to
provide an enhanced test plan specification development and assessment process for
developers The additional maintenance and measurement guidelines may also be useful
for verifying, validating, and improving preexisting requirements and/or test plans after
the initial guidelines from the IEEE have been utilized
A web-site test plan specification information package development process was
designed to provide developers with a series of steps for preparing a test plan The series
of steps reflected a reference to both the current IEEE standards as well as the additionsfor the standards and guidelines An assessment rubric was designed for both the current
Trang 14and augmented sets of standards, and used to evaluate and critique the test plan
specification information packages Using the current IEEE and augmented test planrequirements’ guidelines, a comparison was made between the current standard and theaugmented test plans’ information packages relative perceived quality, and the relative
uniformity of the test plan information packages quality scores
Statement ofProblem
Since there are no maintenance or measurement standards in the IEEE test
planning, developers and testers do not measure the percentage of errors during
development or conduct maintenance after the application is released So therefore it is
necessary to have unified and/or standardized processes/guidelines that can be used to
provide standards for consistency throughout the Information Technology industry TheIEEE is considered a leader in the field of engineering standards and guidelines for
development and testing “The Institute of Electrical and Electronics Engineers (IEEE)
Software Engineering Standards provides a comprehensive set of standards for
developing and validating software” (Schmidt, 2000, p 1)
Research was undertaken to examine if a relationship exists between the IEEE
standards and the additional proposed standards, maintenance and measurements Thisstudy was used to determine whether the inclusion of an additional set of test plan
specification information package guidelines and assessment rubrics was significantly
correlated with an increase in measured perceived test plan specification informationpackage quality, and increased uniformity of the test plan specification information
package evaluation process
This study therefore addressed the following research questions:
Trang 15be positively correlated with an increase in perceived test plan specification information
package (TPSIP) quality?
2 To what extent will the use of a test plan specification information package
assessment rubric and the associated test plan guidelines increase the uniformity of the
evaluations of test plan quality?
The focus of this quantitative study was to examine if a relationship exists
between the IEEE standards and the addition of maintenance and measurements The
study was used to address the issue of increasing test planning quality by includingmeasurements and maintenance which provided a more comprehensive and uniform
approach to developing test plans An enhanced test planning process in the maintenance
phase of the software/web site lifecycle can provide designers, developers and testerswith a methodology for both preventing and detecting errors that may not have been
detected by the existing IEEE standards in earlier phases of the lifecycle Juran (1999)
indicated that this may be the case when he stated that “with the increasing complexity
and criticality of computer programs, traditional ad hoc methods for testing have become
inadequate” (p 5)
Definition of Key Terms
Assessment Rubric —The forms used to grade each test plan specification information
package and to assign a point value for any “yes” answer The forms consisted of 8 or 10
specific standards depending on the test plan specifications that were evaluated The
forms were intended to provide a uniform and consistent approach to evaluation
ETPDP — Enhanced Test Plan Development Process — This is the test plan process thatwas used to facilitate the preparation of the test plan information packages that were
Trang 16presented in an online format using the enhanced standards along with the IEEE
standards
IEEE - The Institute of Electrical and Electronics Engineers — This organization setstandards for Engineers for standardization of practices IEEE’s mission is: “The Institute
of Electrical and Electronics Engineers (IEEE) promotes the development of
electro-technology and allied sciences, the application of those technologies for the benefit ofhumanity, the advancement of the profession, and the well-being of its Members.” (IEEEMission, 2005, p.1)
IEEE+ - The Institute of Electrical and Electronics Engineers’ Standards with TwoAdditional Proposed Standards - IEEE+ was the development process of a test plan
template which consisted of standards from the IEEE web site augmented by two
additional standards developed for measurement and maintenance, and a detailed
development template
TPQ - Test Plan Quality — TPQ is a construct that was used to measure the scores from
the assessments that were tallied A spreadsheet containing the headings, answers andscaled results was used for entering the results for statistical analysis Hypothesis testing
was used to assess the significance of the differences in TPQ conformance percentages
associated from using the IEEE test planning standards and the results obtained from theuse of IEEE+ standards with the additional test planning standards and the developmenttemplate respectively
TPDP - Test Plan Development Process - TPDP is a construct that was used to
operationally define a method that was used for developing a test plan specification
information package for web-site software development It included various components
Trang 17tested, the test plan methodology required to perform the test, specific testing tasks,equipment required and those responsible for conducting the testing process.
TPSIP —Test Plan Specification Information Package — TPSIP is a construct that was
used to define a grouping of information that was needed to evaluate the development ofthe test plan specifications
TSU - Test Scoring Uniformity - TSU is a construct that was used to determine whetherthe test scoring process was associated with any significant difference for the variance ofthe TPQ scores
World Wide Web Consortium (W3C) — The World Wide Web Consortium (W3C) is an
international consortium where Member organizations, a full-time staff, and the public
work together to develop Web standards W3C's mission is: “To lead the World Wide
Web to its full potential by developing protocols and guidelines that ensure long-term
growth for the Web” (W3C Consortium, 2004, p.1)
BriefReview of Related Literature
In addressing the general rationale for test planning, Juran (1999) discussed the
importance of defining quality as a part of the process for testing and validation “Testingmust ensure that the system operates correctly in the actual environment or, where such
testing is not possible, in an environment that simulates the conditions of actual use.” (p
5) For this reason it is necessary to have a test process that can simulate a real-time
environment With careful test planning and structure, this type of testing would help
prevent and reduce errors, and inconsistent processes, by following a formalized test plan
development and implementation process Juran asserted that test planning should be
Trang 18conducted in conjunction with a product development plan so that the product planspecifications can be built into the test plan more accurately (p 18).
Juran also stated that a typical product test plan outline would include the
following elements: safety review and test, certifications, test to product requirements,test to application requirements, maintainability/serviceability and documentation (p 18)
The addition of control inputs and product readout conditions also need to be considered
as a part of software test planning These elements focus on testing the hardware
components of products
Schmidt (2000) discussed the importance of establishing standards for test
planning and how the IEEE has established standards for software The IEEE is
responsible for developing software engineering standards such as customer and
terminology standards, process standards, product standards and resources and techniquestandards He stated that, “Proper use of the standards can help organizations improve
their software development and validation process and help implement an effective
software quality system” (p 1)
Schulmeyer (1999) discussed how a software quality assurance plan should be
designed by including the following information; “Objectives, References, Risks,
Responsibilities, Quality assurance tasks, Schedule, Metrics, Supplier, Customer andSubcontractor SQA support, Quality records and reports and Problem solving”
(Schulmeyer, 1999, p 339) “The plan will serve as the driving force for how the SQA
(Software Quality Assurance) organization will function on the project and what
activities they will take in parallel with other software development activities” (p 343)
Trang 19the software development lifecycle process This template provides a "planning stage of
the software system tests" (p 228) The template can be used as a tool to help sort
information into various phases such as the scope of the tests, the testing environment,the test details such as objectives, requirements and the schedule for the testing This type
of data collection is important for ensuring that the objectives and goals of the testing arecarried out in a procedural fashion
Microsoft (2005) has suggested that test plans are necessary for documenting test
cases for application development Two types of test plans are suggested, the detailed test
plan document and the detailed test case document Both types of documentation are
examples used to verify and validate test cases and also for assuring coverage of specific
aspects of testing which include the design, review and testing of an application
Kaner (2004) discussed the importance of test planning and documentation as a
process to record and document the testing process and present the results of testing
process "Testers should document manual tests in great procedural detail so that they can
be handed down to less experienced or less skilled testers" (p 3) Kaner also suggested
that test case documentation is important for managing the testing process and for
documenting the results of the testing
It was apparent from the literature that there is a need to prepare test plans that
can be a part of a standardized product development process The literature indicates that
there are several different approaches to test planning and there is not a single standard
that is used throughout the industry,
Trang 20In summary, based upon the initial literature review, there was a general
agreement that it was beneficial to prepare a standardized test plan as a part of both thehardware and software/web site lifecycle process Furthermore, the expected benefits will
be manifested through reduced defects and the concomitant increase in percentages of
higher-quality hardware and software web sites released within budget and schedule
Highlights and Limitations ofMethodologyThis study has analyzed the potential efficacy of the additional areas of test
planning, and provided a uniform web site test plan development process The
hypothesized efficacy of the enhanced standards, the uniform test plan developmentprocess, and the assessment rubrics can provide web site developers with both a more
comprehensive and standardized process for developing web site test plans A factorialexperimental design methodology was chosen to provide the most assurance of
identifying the dependent and independent variables’ actual relationships The followingconstructs and variables were reflected in the final experimental design: Test Plan Quality(TPQ) — Dependent Variable, Test Plan Development Process (TPDP) — Independent
Variable, Test Scoring Uniformity (TSU) — Dependent Variable and Test Plan
Specification Information Package (TPSIP) — Independent Variable The data was
collected and analyzed using the variables associated with the constructs as a method for
analyzing their effects on TPQ and TSU
This research analyzed the effects of the independent variables for preparing and
evaluating web site test plans This research did not address all possible types of test
plans required for all possible types of web sites It may not have been possible to
eliminate all potential bias from this study since there was subjective evaluation of thetest plans The research was restricted to studying the selected factors and other factors
Trang 21Also only NCU learners and faculty participated in the evaluations.
arch ExpectationsThis study was expected to provide the following results:
1 An increased understanding of the relationships between the Test PlanDevelopment Processes and perceived Test Plan Quality
2 An increased understanding of relationships between the use of Test
Plan Evaluation Rubrics and the uniformity of the test plan informationpackage scoring
The reliability of this study was assured through the use of the original standardswhich were the IEEE test planning guidelines, and uniform operational guidelines forusing the addition of the maintenance and measurement test planning guidelines Thesestandards were used to determine their usage as test plan specifications The purpose ofthis research was to evaluate the efficacy of using the additional standards, a test plan
development process, and scoring rubrics, that can potentially be used by designers,
developers and testers for developing more uniform and effective web site test plans
Internal validity was assured through the use of a formal experimental design thatwas used to test and analyze the relationships between the dependent variables TPQ andTSU and the independent variables External validity was addressed by including
reasonably flexible procedures for the test plan information package development
processes, test plan quality, and finally randomly selected test plan developers and
evaluators with diverse experience levels
Trang 22Chapter 2: Review of Related Literature
OverviewThe following literature review presents various methods and strategies for test
planning The literature review demonstrates the rationale and current approaches used
for hardware and software test plan development The literature review was organized
into the following subject areas: Software Test Strategy and Planning, Software
Maintenance and Software Measurements, and the Summary
Software Test Strategy and PlanningSoftware documentation is an important part of the testing process It was clearfrom the literature that there are many different forms of documentation However test
planning may be considered a foundation for the testing process Lengel (2001) has stated
“Test coverage is about insuring that test plans and test cases include information vital forsuccessful testing of the program in the areas of functionality, performance, and theoverall quality of the software In addition, test managers who prepare test plans that
provide proper test coverage can avoid the wrath of a project manager whose
implementation has just gone sour or an angry customer whose system has just crashed”
(p 1)
Lengel (2001) posted that adequate test planning should include test scenarios thatrelate to the requirements and specifications of the testing Lengel (2001) also stated as
you prepare a test plan that the initial focus should involve a process that captures
scenarios from the available documentation This would involve collecting the
requirements that are relative to the development of the software or application andmaking sure that testing is performed to address the necessary requirements for the best
results
10
Trang 23Software QA Associates (2001) has suggested that it is important to ensure thatthere are adequate plans for testing that provide detailed processes and procedures.
Software QA Associates (2001) has also stated that “Documentation of the software testprocess should generally describe: 1) the plans and priorities for the testing, 2) the test to
be performed, and 3) results of those tests” (p 3)
The results of the testing should be used to improve product performance andapplications Software QA Associates (2001) asserted that it is important to use metrics
when testing to determine whether there is a relationship between two or more variables
and to determine the effectiveness of a process By using metrics as a part of the testplanning process it becomes possible to determine the effectiveness of the testing process
and the overall effectiveness and usefulness of the application or product
Software QA Associates (2001) has provided examples of metrics that would beuseful as a part of the test planning process They include test coverage analysis, defect
analysis, bugs and test cases By collecting and analyzing the above types of metrics theremay be a decrease in the error ratio and an increase in the usability and reliability of thesoftware It is therefore clear that by incorporating measurements and maintenance
requirements in test plans this may lead to better software
Careful test planning is critical and essential for software development and
maintenance Campbell (1993) stated that “We must carefully plan the testing effort in
order to best use the time and money we do have for testing” (p 2) This is an important
concept Often when test time is budgeted it is generally one of the first things cut fromthe budget due to lack of time and resources Many managers do not see the need to
Trang 24budget for adequately testing software; therefore it is essential to have an effective test
plan that outlines the key objectives necessary for implementing and testing software
Campbell (1993) also stated that it is important to write a test plan that is easy touse The test plan is a useful document to provide guidance and direction for the testingprocess Test planning is critical and essential to ensure that full and effective testing is
performed that will adequately reflect the usability, reliability and implementation of thesoftware for customers’ use
It is important that there be evaluation criteria which can be used to analyzetesting results Campbell stated that “Results must be examined and evaluated You candetermine results by running a baseline test Perform, the test on the current software,
make changes, perform the test again and compare the results” (p 4) This type of testingshould be a part of the test planning process and account for selective measurements thatwill be used to evaluate the baseline run of the test with successive test runs to account
for variations and changes This will also help with maintenance testing because thebaseline data will be a means for making test result comparisons for the software duringactual use and also be used to help improve the testing process
There are many different types of software documentation that are useful forsoftware development and test processing Delgado (1992) asserted that there are four
types of plans that are useful and necessary for quality software: Quality assurance plans,
Development plans, Quality control/Test plans and Configuration plans Each type of
plan provides an essential component for software development The Quality Assurance
plan is used for process improvement This plan is useful for focusing on the vision,
continuous quality improvement, cost of quality, staffing issues, resources and
Trang 25measurements (Delgado, 1992) The Development plan is used for project managementand technical aspects such as the "life cycle process, process analysis, process analysis,
technology, requirements, quality factors, productivity and standards" (Delgado, 1992,p.10) The Quality control test plan is used to determine if the system is meeting the
requirements This plan is used for testing, evaluating the test environment, test strategy,
error recording and test analysis (Delgado, 1992) The Configuration plan is used toidentify and order the process for creating the product It is important to identify, control,evaluate and audit the product using this plan
After reviewing the different types of test planning documentation references it
was apparent that there was an overlapping theme of required: information that was
needed to fulfill the software documentation for each plan The QA plan presents
measurements as a necessary part in providing a quantitative approach to techniques forquality assurance Delgado (1992) stated that a QA plan provides a method that can be
used to work well with the observation of errors and system efficiency which correspondwith the quality of the test The testing results, such as error counts and system efficiencydata, can be gathered as a result of QA plan development, test planning and processing
The quality control/test plan should require error recording as necessary for adequate
tracking and as means for defining and driving necessary corrective action(s).
Test strategies and adequate planning are necessary for good testing Craig (1999)
suggested that in planning for testing it is not critical to create a lot of test cases, but
instead to ensure that the test strategy and the risks and priorities of the test are
adequately addressed The process of planning a test strategy involves the review of
business requirements and application specifications In order to perform good test
Trang 26planning it is important to understand that the goals and objectives of the test plan must
support the project plan
Craig (1999) also stated that “in test planning, the process is ultimately moreimportant than the product” (p 1) Since the process is considered an important part of
test planning then all parts of the software development process should be considered for
inclusion in the test planning process This would include measurements and
maintenance Furthermore, Craig (1999) has stated that the test plan template should beappropriately modified to meet the needs of the organization Craig (1999) has also statedthat although the IEEE test plan is a good template to use, all areas may not be applicable
to the organization’s goals and objectives In addition, Craig (1999) mentioned that the
plan should be considered a special process for any given project and may need to bemodified to meet each specific test process Therefore it is important to consider
additional aspects of test planning that will work well for different organizations
Test-driven development is a good tool for ensuring that all phases of
development incorporate a form of testing Goldsmith (2002) asserted that test planning
as presented by the IEEE may be viewed as no more than a “counterproductive
paper-generator” (p 1) Although a test plan needs to be a part of the development process, it
also needs to be useful and help the testing efforts This may be accomplished by treatingthe test plan as a working document that provides a step-by-step current approach to
testing Goldsmith (2002) asserted that it is essential to record test plan information to
enhance reuse and continuous improvement for testing
Goldsmith (2002) has also concluded that many developers viewa test plan as a
collection of test cases This should not be the case, the test plan should be written as a
Trang 27guide for conducting the test process that can be used throughout the entire development
and testing lifecycle Goldsmith (2002) considered this “proactive testing” (p 2) This
form of test planning is useful for guiding the testing process through each phase of
development which may produce a higher quality software product
Software Maintenance
In addressing the general rationale for maintenance software planning, Favre
(1999) defined eight phases of the software maintenance life cycle process These eight
phases are; modification request, estimation, planification, implementation, test,
document, delivery and operation It is noted that during the test phase, testing strategiesare defined and tools may be chosen
Support for software maintenance follows a maintenance process which includes;
production version, problem report, problem analysis, maintenance design, maintenance
implementation, test version, regression test, documentation, acceptance test and
delivery Each activity contributes to the overall maintenance process for software
Testing is considered a core maintenance activity and contributes to the overall
maintenance process
During the problem analysis, the development of the test plan is considered
critical to ensuring that the maintenance request contains information about the test
process During the maintenance phase of software development, testing is a critical part
of the overall planning and implementation of software development and should be a part
of the test planning process
Favre (1999) stated that “any software maintenance organization must define a
testing plan, create a base of test cases, automatize the testing process when possible (e.g
testing tools)” (p 42) This suggests the importance and need for test plans and their use
Trang 28during the testing process Favre (1999) also considered metrics to be a critical part of the
support for software maintenance and are important considerations for how to measure
those measurements that are relevant to software maintenance According to Favre
(1999), measurements are useful for evaluating quality, estimating costs, and provide ameans to control and improve the software maintenance process for long term projects
Because of the importance of metrics and measurements careful test planning is
necessary.
The University of Toronto (2005) has developed a software maintenance processthat contains a presentation on software maintenance, cost, types of maintenance,
problems facing maintainers, approaches to maintenance and reuse According to the
University of Toronto, “Maintenance consumes 40% - 80% of software costs” (p 3) Due
to the large investment in the cost of software maintenance there should be a software testplan that incorporates a process that includes testing throughout the maintenance
planning phase
Since there are different types of software maintenance such as corrective
maintenance, adaptive maintenance, perfective maintenance and preventative
maintenance, test planning is a critical part of each phase A test plan should be used to
outline and accommodate a testing process for each maintenance phase
According to the University of Toronto (2005) information that is useful for
helping the maintenance process requires documentation that supports the release, testing
and the ability to debug and test as the maintenance process is performed This indicatesthat a formal test plan would be beneficial for the maintenance phase of the software
process Maintenance is considered a time intensive effort for software development It is
Trang 29noted in the article from the University of Guelph (1995) that “The maintenance of
existing software can account for over 60 percent of all development change” (p 1)
Because a large percentage of effort is spent on this effort, it is important to consider howthe maintenance of software is planned and implemented into the software development
life process and software test planning The University of Guelph (1995) has also stated
that “70 per cent of the cost of software is devoted to maintenance” (p 1) Due to thehigh percentage of cost involved in the maintenance of software applications, it is
essential to fully plan and document the process and procedures that are needed for
testing and maintaining software within the test planning process
Software MeasurementsMetrics are critical to the software development process The University of
Guelph (1996, p.1) noted that “Metrics are management tools which are used to estimate
the cost and resource requirements of a project” Because of the criticality of metrics, it is
essential to understand the types of metrics that adequately reflect the project’s goals andobjectives for the type and phase of software development that are necessary
The type of measurements that may be used for software projects should be based
on the requirements for the project Metrics can be used to “estimate costs, project tasks,and a project schedule” (University of Guelph, p.1) This is all essential information that
requires adequate planning Furthermore, the University of Guelph (1996) stated thathistorical data can be used to successfully predict and assist in planning and estimation ofmeasurements, This type of data collection is necessary to adequately measure and
analyze data which means careful planning is necessary to testing and development
Mcquaid (2004) discussed the importance of measurements and also mentioned
that the measurements used for a project should match the goals and outcomes for a
Trang 30project Not all measurements are useful indicators for software projects and Mcquaid(2004) concluded that careful planning is necessary Mcquaid (2004) presented the adagethat “no one plans to fail, yet many fail to plan“ (p 2)
The use of a successful measurement program should become a required feature
of the planning process for the software development lifecycle process which also shouldinclude the software testing planning process This should not be viewed as “an overheadprocess” (Mcquaid, 2004, p 2) but a vital part of the development process for software
development and testing Mcquaid (2004) also concluded that careful planning is
essential and that measurements can assist in the planning process in many cases
Summary
In summary, based upon the literature review, there was an apparent consensus
that it is beneficial to include a standardized test plan as a part of both the hardware andsoftware/web site lifecycle process Based on the findings from the literature review, itwas apparent that there is a need for at least some form of measurement and maintenance
process to be integrated with the software test plan preparation Furthermore, the
expected benefits of adding a form of measurement and maintenance will be manifestedthrough reduced defects and a higher percentage of higher-quality hardware and software
web sites being implemented within budget, and delivered on schedule
Trang 31In this chapter a description of the research problem was discussed and the
research questions are presented The research design, research instruments, and themethods of data analysis are also defined
Restatement of the ProblemSince there are no maintenance or measurement standards in the IEEE test
planning, developers and testers do not measure the percentage of errors during
development or conduct maintenance after the application is released So therefore it is
necessary to have unified and/or standardized processes/guidelines that can be used to
provide standards for consistency throughout the Information Technology industry Thisresearch was used to determine whether the inclusion of an additional set of test planguidelines, a test plan development process, and assessment rubrics was positively
correlated with an increase in measured perceived test plan information package quality,and increased uniformity of the test plan package evaluation process The effect of each
of these factors, and their potential interactions, was estimated to determine if their use
was associated with an increase in perceived test plan information package quality andevaluation uniformity This study addressed the following research questions:
1 To what extent will the use of the augmented testing and maintenance standards
be positively correlated with an increase in perceived test plan specification information
package (TPSIP) quality?
2 To what extent will the use of a test plan specification information package
assessment rubric and the associated test plan guidelines increase the uniformity of theevaluations of test plan quality?
19
Trang 32Statement of Hypotheses
This study presented two test plan formats (Appendix A and Appendix B) andused an evaluation to determine if there was a significant increase in perceived test planinformation package quality by adding a formal development process with maintenance
and measurements to the test plan development process Following are the null
hypotheses and alternative hypothesis statements to reflect the study:
H01 ~ The addition of the standards and development process for IEEE+ TPSIP
preparation will be associated with either the same or decreased levels of TPSIPperceived quality
H11 — The addition of the additional standards and development process for
IEEE+ TPSIP preparation will be associated with a significant increase in TPSIP.H02 - The addition of the standards for IEEE+ test plan preparation,
use of a test plan assessment rubric and test plan guidelines, will be associated
with either the same TSU or a decrease in TSU
H12 - The addition of the additional standards for IEEE+ test plan preparation,use of a test plan assessment rubric and test plan guidelines, will be associatedwith a significant increase in TSU
Description of Research Design
A factorial experimental design methodology was chosen to provide the greatest
assurance of properly addressing the research questions, and for identifying the truedependent and independent variables’ relationships “The importance of experimental
design also stems from the quest for inference about causes or relationships as opposed tosimply description” (Sytsma, 1999, p 1) The research for cause and effect was at thefoundation of research The following constructs and variables were reflected in the
Trang 33experimental design: Test Plan Quality (TPQ) — Dependent Variable, Test Plan
Development Process (TPDP) - Independent Variable, Test Scoring Uniformity (TSU) —
Dependent Variable and Test Plan Specification Information Package (TPSIP) —
Independent Variable Data generated from the experimental design were:
1 The distributions of the perceived TPQ conformance scores
2 The variances of the TSU (Test Score Uniformity) scores
To address the first research question and its associated hypothesis, the
randomized experimental design was used as the basis for determining the data collection
sequence and identifying any significant differences between the effects on perceived testplan quality (TPQ) associated with the use of two sets of standards (one for the IEEE and
one for the IEEE+) and using a design development template TPQ scoring data collected
from the evaluation groups was used as the dependent variable This information wasused to determine if the use of the augmented testing and maintenance standards was
positively and significantly correlated with an increase in perceived test plan quality
To address the second research question and its associated hypothesis, the secondstatistical analysis for the dissertation focused on the test scores’ uniformity TSU (Test
Score Uniformity) The relative variation in the TSU scores was used to determine
whether there was an advantage (characterized by lower scoring variation) associatedwith using a detailed test plan development template and an enhanced standard scoring
rubric versus the scoring variation associated with the IEEE test plan results and a
macro-level scoring rubric The reason that a one-tailed F-test was selected for TSU was to
determine if the combination of the enhanced test plan development process and scoring
rubric were associated with an increase in TSU uniformity Therefore only an increase in
Trang 34TSU was of interest, and a one-tailed F-test was chosen to test the null hypotheses
However for both the TPQ and TSU hypotheses, P values were listed for both the onetailed and two-tailed alternatives
The evaluation procedures used two sets of standards and two evaluation rubrics
The statistical analysis utilized the data generated by evaluating the test plans informationpackages perceived quality from using the current JEEE standards and the additionalstandards using the IEEE+ which reflected the use of a test plan development template.Two sets of test plans were prepared and evaluated to determine if there were any
statistically significant improvement in either perceived TPQ and/or TSU associated withusing the new standards, augmented by a test plan development template, and the
enhanced scoring rubric
Application of Power Analysis
TPQ power analysis For TPQ ,MINITAB TM was used to evaluate the power for
the sample size for two proportions using a test for two proportions The results from
MINITAB TM one-tailed power analysis for a one-sided test for two- proportions were
calculated as 0.8 power with an alpha of 05 using a sample size of 32
TSU power analysis G* Power (Erdfelder, 1996) was used for the final poweranalysis and sample size determination for TSU This software tool was used to analyzethe power for the TSU F Test by estimating the sample size required for the final targeted
power level for the F test for comparing TSU evaluation test scores’ variability The Ftest method employed G*Power The results are listed in Table 1:
Table 1
Input Parameters/Sample Size Requirements for TSU Power Analysis
| Effect size f | 36 |
Trang 35Alpha 05
Power 8
Reguired Sample Size | 64 (32 per group)
As shown in Table 1, for TSU, the sample size requirements were 32 test plan
information packages for IEEE and 32 test plans for IEEE+ One group was used todevelop test plans information packages for IEEE, and the second group was used todevelop test plan information packages for IEEE+ using an enhanced test plan
development template
Operational Definition of Constructs and Key Variables
Test Plan Development Process (TPDP) - A test plan development process wasoperationally defined as the method that was used for developing the specific areas of a
test It includes various components such as an introduction to the web site or application,
defining the specific features to be tested, the test plan methodology required to perform
the test, specific testing tasks, equipment required and those responsible for conducting
the testing process
Test Scoring Uniformity (TSU) - Test score uniformity was operationally defined
as the consistency of scoring of the evaluating test plan information packages It reflected
the use of an enhanced standard scoring rubric for evaluating the test plan results or the
scoring variation associated with the IEEE test plan results and a macro-level scoring
rubric
Test Plan Quality (TPQ) - Test plan quality was operationally defined as theperceived percentage conformance scores of the Test Plan Specification InformationPackages
Trang 36Test Plan Specification Information Package (TPSIP) — Test Plan SpecificationInformation Package was operationally defined as a grouping of information that wasneeded to evaluate the development of the test plan specification package elements
Description of Materials and InstrumentsThe materials and instruments for the experiment included an online survey
document which was used to enter data, accumulate the results, and provided a resultspage of the data for input to the subsequent statistical analysis Instrumentation for the
experimentation consisted of a data collection web site which was used for the designed
test plan templates, one for the IEEE and one for the IEEE+ test plans There were twoweb sites designed where test plan information packages were prepared, entered, and the
data collected for the IEEE and IEEE+ test plan packages The IEEE test plan
information package development process was implemented using a macro level test planinformation package development guideline process The IEEE+ test plan process was
implemented using a detailed level test plan development guideline process Both web
sites http://www.leesol.addr.com/dissertation5/index.html, and
http://www.leesol.addr.com/dissertation6/index.html, were used to gather data associated
with the 64 test plan information packages Thirty two packages were prepared and
collected from the IEEE+ and 32 test packages were prepared and collected from theIEEE The IEEE+ test plan package template was located at:
http://www leesol.addr.com/dissertation5/standards.htm, and the IEEE test plan packagetemplate were located at: http://www leesol.addr.com/dissertation6/standards.htm
The IEEE template consisted of standards that were contained in the current IEEE
test plan standards The IEEE+ test plan development template consisted of standards
from the IEEE web site augmented by two additional standards developed for
Trang 37measurement and maintenance, and a detailed development template TSU scores were
generated from the evaluations by eight separate participants who evaluated eight testplans (four IEEE and four IEEE+) using a basic scoring rubric for the IEEE test plans,and an enhanced scoring rubric for the IEEE+ test plans
The templates used on the data collection web site were used only to record the
test plan information for each category The web site was designed so that all the
experimental design data were entered, collected, recorded, and summarized The design
for the data collection web site was capable of producing a results page that was used to
automatically send an email of the results for each test plan entry to an assigned emailaddress
Each of the eight evaluators evaluated the same eight test plans using the scoringrubrics that were designed so that they were able to allocate scores for each test plan
category This evaluation was used, in conjunction with the development and evaluation
templates, to provide percentage conformance scores
The IEEE scoring rubric process was developed using macro level scoring
guidelines The IEEE+ scoring rubric process was developed using detailed scoring levelguidelines A macro-level flow diagram of the entire research study’s instrumentation and
procedures is contained in Figure 1
Trang 38Figure 1, Flow diagram of research study
Administrative _ Ins trumentation
Results Accumulated and
Results Accumulated and sent for rubric evaluation sent for rubric evaluation
Rubric evaluation for Rubric evaluation forIEEE test plans IEEE+ test plans
Results Accumulated Results Accumulated
L
Results Analyzed |
The instructions were presented on the web site that described the purpose, design
of the experiment, and a discussion of the procedures for filling out the test plan template.The web site was used to accumulate the data and results of the experiment The process
Trang 39and materials were pre tested to determine operational feasibility, and to ensure ease of
understanding of the scoring and data entry processes
Selection of SubjectsSixty four test plan developers were randomly selected The participants were
randomly chosen from students and professionals at NCU Ideal candidates were thosewho were working in the field of information technology as web developers,
programmers or testers It was deemed acceptable to include a few participants outside
the field of web development, but they were limited in their understanding of testing and
web design
Procedures
The web sites were tested before the experimentation to make sure that all linkswere working, that data could be collected and stored for reuse The process involved
ensuring that both URL's for the test plan could be viewed in a browser Simulated data
were entered into both test plans to ensure that when the submit button was clicked that aconfirmation page displayed and also that the information was emailed to a central
repository for collecting the data Each web page had its own email address so there was
not any mixing of the data The rubrics’ results also had their own email addresses forcollecting the data Data for comparing the relative quality of the test planning processes
and the uniformity of evaluation processes using the constructs TPQ, TPDP, TPSIP and
TSU was obtained by using:
1 Sixty four randomly selected participants who were assigned to use the current
or augmented standards, and use, or not use, the test plan development template/flow
diagram The participants were provided instructions for the evaluation (Appendix D)
Trang 402 Assigning randomly selected participant reviewers to score test plans againstboth sets of standards by first using the developed rubric with a rating from one to eightfor the IEEE standards and one to ten for the IEEE standards augmented with the
standards for each test plan category/ element The first scoring process that was used
was a simple percentage scale; that reflected the inclusion of whether or not the standard
versus category/element of standard was included, and to assess the appropriate usage ofthe standard
3 Assigning a corresponding proper subset of the reviewers to score a set of thetest plan information packages developed from both the current and augmented standardsusing the basic and enhanced scoring rubrics that were developed for assessing test plans
developed from the current IEEE standards, and test plans that was developed from the(IEEE+) augmented standards
Each of the randomly selected TPQ evaluators submitted their results to the datacollection web site When the user submitted the form they were tracked by their remotename The page links were inactive so the user could not go outside of the immediate
window and were only able to submit the actual form that was provided with the URL
TPQ scores were tallied, and a spreadsheet containing the headings, answers and scaledresults was used for collecting the results for statistical analysis
Discussion of Data Processing
MINITAB TM and an Excel TM spreadsheet from Aczel and Sounderpandian
(2002) were used for the evaluation of the data resulting from the experiment MINITAB
TM was used for analyzing the TPQ data The input data were the 32 individual test plans
for IEEE and 32 individual test plans for IEEE+ TPQ attribute scores were used to testfor significant differences in the two-proportions of the plans’ maximum scores using a