Test strategy

1 Test Strategy 1 Introduction This Document entails you towards the better insight of the Test Strategy and its methodology. It is the role of test management to ensure that new or modified service products meet the business requirements for which they have been developed or enhanced. The Testing strategy should define the objectives of all test stages and the techniques that apply. The testing strategy also forms the basis for the creation of a standardized documentation set, and facilitates communication of the test process and its implications outside of the test discipline. Any test support tools introduced should be aligned with, and in support of, the test strategy. Test Approach/Test Architecture are the acronyms for Test Strategy. Test management is also concerned with both test resource and test environment management. 2 Key elements of Test Management: Test organization –the set-up and management of a suitable test organizational structure and explicit role definition. The project framework under which the testing activities will be carried out is reviewed, high level test phase plans prepared and resource schedules considered. Test organization also involves the determination of configuration standards and the definition of the test environment. Test planning – the requirements definition and design specifications facilitate in the identification of major test items and these may necessitate the test strategy to be updated. A detailed test plan and schedule is prepared with key test responsibilities being indicated. Test specifications – required for all levels of testing and covering all categories of test. The required outcome of each test must be known before the test is attempted. Unit, integration and system testing – configuration items are verified against the appropriate specifications and in accordance with the test plan. The test environment should also be under configuration control and test data and results stored for future evaluation. Test monitoring and assessment – ongoing monitoring and assessment of the integrity of the development and construction. The status of the configuration items should be reviewed against the phase plans and test progress reports prepared providing some assurance of the verification and validation activities. Product assurance – the decision to negotiate the acceptance testing program and the release and commissioning of the service product is subject to the ‘product assurance’ role being satisfied with the outcome of the verification activities. Product assurance may oversee some of the test activity and may participate in process reviews. A common criticism of construction programmers is that insufficient time is frequently allocated to the testing and commissioning of the building systems together with the involvement and subsequent training of the Facilities Management team. Testing and commissioning is often considered by teams as a secondary activity and given a lower priority particularly as pressure builds on the program towards completion. Sufficient time must be dedicated to testing and commissioning as ensuring the systems function correctly is fairly fundamental to the project’s success or failure. Traditionally the responsibility for testing and commissioning is buried deep within the supply chain as a sub-contract of a sub-contract. It is possible to gain greater control of this process and the associated risk through the use of specialists such as Systems Integration who can be appointed as part of the professional team. The time necessary for testing and commissioning will vary from project to project depending upon the complexity of the systems and services that have been installed. The Project Sponsor should ensure that the professional team and the contractor consider realistically how much time is needed. Fitness for purpose checklist: • Is there a documented testing strategy that defines the objectives of all test stages and the techniques that may apply, e.g. non-functional testing and the associated techniques such as performance, stress and security etc? • Does the test plan prescribe the approach to be taken for intended test activities, identifying: • the items to be tested, • the testing to be performed, • test schedules, • resource and facility requirements, • reporting requirements, • evaluation criteria, • risks requiring contingency measures? • Are test processes and practices reviewed regularly to assure that the testing processes continue to meet specific business needs? For example, e-commerce testing may involve new user interfaces and a business focus on usability may mean that the organization must review its testing strategies. 3 Test Strategy Flow : Test Cases and Test Procedures should manifest Test Strategy. Test Strategy – Selection Selection of the Test Strategy is based on the following factors  Product Test Strategy based on the Application to help people and teams of people in making decisions.  Based on the Key Potential Risks  Suggestion of Wrong Ideas.  People will use the Product Incorrectly  Incorrect comparison of scenarios.  Scenarios may be corrupted.  Unable to handle Complex Decisions.  Determination of Actual Risk.  Understand the underlying Algorithm.  Simulate the Algorithm in parallel.  Capability test each major function.  Generate large number of decision scenarios.  Create complex scenarios and compare them.  Review Documentation and Help.  Test for sensitivity to user Error. Test Strategy Execution: Understand the decision Algorithm and generate the parallel decision analyzer using the Perl or Excel that will function as a reference for high volume testing of the app.  Create a means to generate and apply large numbers of decision scenarios to the product. This will be done using the GUI test Automation system or through the direct generation of Decide Right scenario files that would be loaded into the product during test.  Review the Documentation, and the design of the user interface and functionality for its sensitivity to user error.  Test with decision scenarios that are near the limit of complexity allowed by the product  Compare complex scenarios.  Test the product for the risk of silent failures or corruptions in decision analysis.  Issues in Execution of the Test Strategy  The difficulty of understanding and simulating the decision algorithm  The risk of coincidal failure of both the simulation and the product.  The difficulty of automating decision tests 4 General Testing Strategies • Top-down • Bottom-up • Thread testing • Stress testing • Back-to-back testing 5 Need for Test Strategy The objective of testing is to reduce the risks inherent in computer systems. The strategy must address the risks and present a process that can reduce those risks. The system concerns on risks then establish the objectives for the test process. The two components of the testing strategy are the Test Factors and the Test Phase.  Test Factor – The risk or issue that needs to be addressed as part of the test strategy. The strategy will select those factors that need to be addressed in the testing of a specific application system.  Test Phase – The Phase of the systems development life cycle in which testing will occur. Not all the test factors will be applicable to all software systems. The development team will need to select and rank the test factors for the specific software systems being developed. The test phase will vary based on the testing methodology used. For example the test phases in as traditional waterfall life cycle methodology will be much different from the phases in a Rapid Application Development methodology. 6 Developing a Test Strategy The test Strategy will need to be customized for any specific software system. Analysis Coding Errors 36% and design Errors 64% The applicable test factors would be listed as the phases in which the testing must occur. Four test steps must be followed to develop a customized test strategy.  Select and rank Test Factors  Identify the System Developmental Phases  Identify the Business risks associated with the System under Development.  Place risks in the Matrix TestFactors\T est Phase Requirements Design Build Dynamic Test Integrate Maintain 7 Conclusion: Test Strategy should be developed in accordance with the business risks associated with the software when the test team develop the test tactics. Thus the Test team needs to acquire and study the test strategy that should question the following:  What is the relationship of importance among the test factors?  Which of the high level risks are the most significant?  What damage can be done to the business if the software fails to perform correctly?  What damage can be done to the business if the business if the software is not completed on time?  Who are the individuals most knowledgeable in understanding the impact of the identified business risks? Hence the Test Strategy must address the risks and present a process that can reduce those risks. The system accordingly focuses on risks thereby establishes the objectives for the test process. Risks Factors 2 TEST PLAN 1 What is a Test Plan? A Test Plan can be defined as a document that describes the scope, approach, resources and schedule of intended test activities. It identifies test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning. The main purpose of preparing a Test Plan is that everyone concerned with the project are in sync with regards to the scope, responsibilities, deadlines and deliverables for the project. It is in this respect that reviews and a sign-off are very important since it means that everyone is in agreement of the contents of the test plan and this also helps in case of any dispute during the course of the project (especially between the developers and the testers). Purpose of preparing a Test Plan A Test Plan is a useful way to think through the efforts needed to validate the acceptability of a software product. The completed document will help people outside the test group understand the 'why' and 'how' of product validation. It should be thorough enough to be useful but not so thorough that no one outside the test group will read it. Contents of a Test Plan 1. Purpose 2. Scope 3. Test Approach 4. Entry Criteria 5. Resources 6. Tasks / Responsibilities 7. Exit Criteria 8. Schedules / Milestones 9. Hardware / Software Requirements 10. Risks & Mitigation Plans 11. Tools to be used 12. Deliverables 13. References a. Procedures b. Templates c. Standards/Guidelines 14. Annexure 15. Sign-Off 2 Contents (in detail) Purpose This section should contain the purpose of preparing the test plan Scope This section should talk about the areas of the application which are to be tested by the QA team and specify those areas which are definitely out of scope (screens, database, mainframe processes etc). Test Approach This would contain details on how the testing is to be performed and whether any specific strategy is to be followed (including configuration management). Entry Criteria This section explains the various steps to be performed before the start of a test (i.e.) pre-requisites. For example: Timely environment set up, starting the web server / app server, successful implementation of the latest build etc. Resources This section should list out the people who would be involved in the project and their designation etc. Tasks / Responsibilities This section talks about the tasks to be performed and the responsibilities assigned to the various members in the project. Exit criteria Contains tasks like bringing down the system / server, restoring system to pre-test environment, database refresh etc. Schedules / Milestones This sections deals with the final delivery date and the various milestone dates to be met in the course of the project. Hardware / Software Requirements This section would contain the details of PC’s / servers required (with the configuration) to install the application or perform the testing; specific software that needs to be installed on the systems to get the application running or to connect to the database; connectivity related issues etc. Risks & Mitigation Plans This section should list out all the possible risks that can arise during the testing and the mitigation plans that the QA team plans to implement incase the risk actually turns into a reality. Tools to be used This would list out the testing tools or utilities (if any) that are to be used in the project (e.g.) WinRunner, Test Director, PCOM, WinSQL. Deliverables This section contains the various deliverables that are due to the client at various points of time (i.e.) daily, weekly, start of the project, end of the project etc. These could include Test Plans, Test Procedure, Test Matrices, Status Reports, Test Scripts etc. Templates for all these could also be attached. References Procedures Templates (Client Specific or otherwise) Standards / Guidelines (e.g.) QView Project related documents (RSD, ADD, FSD etc) Annexure This could contain embedded documents or links to documents which have been / will be used in the course of testing (e.g.) templates used for reports, test cases etc. Referenced documents can also be attached here. Sign-Off This should contain the mutual agreement between the client and the QA team with both leads / managers signing off their agreement on the Test Plan. 3 Test Data Preparation - Introduction A System is programmed by its data. Functional testing can suffer if data is poor, and good data can help improve functional testing. Good test data can be structured to improve understanding and testability. Its contents, correctly chosen, can reduce maintenance effort and allow flexibility. Preparation of the data can help to focus the business where requirements are vague. The first stage of any recogniser development project is data preparation. Test data should however, be prepared which is representative of normal business transactions. Actual customer names or contact details should also not be used for such tests. It is recommended that a full test environment be set up for use in the applicable circumstances. Each separate test should be given a unique reference number which will identify the Business Process being recorded, the simulated conditions used, the persons involved in the testing process and the date the test was carried out. This will enable the monitoring and testing reports to be co-coordinated with any feedback received. Tests must be planned and thought out a head of time; you have to decide such things as what exactly you are testing and testing for, the way the test is going to be run and applied, what steps are required, etc. Testing is the process of creating, implementing and evaluating tests. Effective quality control testing requires some basic goals and understanding: You must understand what you are testing; if you're testing a specific functionality, you must know how it's supposed to work, how the protocols behave, etc. You should have a definition of what success and failure are. In other words, is close enough good enough? You should have a good idea of a methodology for the test, the more formal a plan the better; you should design test cases. You must understand the limits inherent in the tests themselves. You must have a consistent schedule for testing; performing a specific set of tests at appropriate points in the process is more important than running the tests at a specific time. Roles of Data in Functional Testing Testing consumes and produces large amounts of data. Data describes the initial conditions for a test, forms the input, is the medium through which the tester influences the software. Data is manipulated, extrapolated, summarized and referenced by the functionality under test, which finally spews forth yet more data to be checked against expectations. Data is a crucial part of most functional testing. This paper sets out to illustrate some of the ways that data can influence the test process, and will show that testing can be improved by a careful choice of input data. In doing this, the paper will concentrate most on data-heavy applications; those which use databases or are heavily influenced by the data they hold. The paper will focus on input data, rather than output data or the transitional states the data passes through during processing, as input data has the greatest influence on functional testing and is the simplest to manipulate. The paper will not consider areas where data is important to non-functional testing, such as operational profiles, massive datasets and environmental tuning. A SYSTEM IS PROGRAMMED BY ITS DATA Many modern systems allow tremendous flexibility in the way their basic functionality can be used. Configuration data can dictate control flow, data manipulation, presentation and user interface. A system can be configured to fit several business models, work (almost) seamlessly with a variety of cooperative systems and provide tailored experiences to a host of different users. A business may look to an application's configurability to allow them to keep up with the market without being slowed by the development process, an individual may look for a personalized experience from commonly-available software. FUNCTIONAL TESTING SUFFERS IF DATA IS POOR Tests with poor data may not describe the business model effectively, they may be hard to maintain, or require lengthy and difficult setup. They may obscure problems or avoid them altogether. Poor data tends to result in poor tests, that take longer to execute. GOOD DATA IS VITAL TO RELIABLE TEST RESULTS An important goal of functional testing is to allow the test to be repeated with the same result, and varied to allow diagnosis. Without this, it is hard to communicate problems to coders, and it can become difficult to have confidence in the QA team's results, whether they are good or bad. Good data allows diagnosis, effective reporting, and allows tests to be repeated with confidence,. GOOD DATA CAN HELP TESTING STAY ON SCHEDULE An easily comprehensible and well-understood dataset is a tool to help communication. Good data can greatly assist in speedy diagnosis and rapid re-testing. Regression testing and automated test maintenance can be made speedier and easier by using good data, while an elegantly-chosen dataset can often allow new tests without the overhead of new data. A formal test plan is a document that provides and records important information about a test project, for example: project and quality assumptions project background information resources schedule & timeline entry and exit criteria test milestones tests to be performed use cases and/or test cases 1 Criteria for Test Data Collection This section of the Document specifies the description of the test data needed to test recovery of each business process.  Identify Who is to Conduct the Tests In order to ensure consistency of the testing process throughout the organization, one or more members of the Business Continuity Planning (BCP) Team should be nominated to co-ordinate the testing process within each business unit, a nominated testing and across the organization. Each business process should be thoroughly tested and the coordinator should ensure that each business unit observes the necessary rules associated with ensuring that the testing process is carried out within a realistic environment. This section of the BCP should contain the names of the BCP Team members nominated to co-ordinate the testing process. It should also list the duties of the appointed co-ordinators.  Identify Who is to Control and Monitor the Tests In order to ensure consistency when measuring the results, the tests should be independently monitored. This task would normally be carried out by a nominated member of the Business Recovery Team or a member of the Business Continuity Planning Team. This section of the BCP will contain the names of the persons nominated to monitor the testing process throughout the organization. It will also contain a list of the duties to be undertaken by the monitoring staff.  Prepare Feedback Questionnaires It is vital to receive feedback from the persons managing and participating in each of the tests. This feedback will hopefully enable weaknesses within the Business Recovery Process to be identified and eliminated. Completion of feedback forms should be mandatory for all persons participating in the testing process. The forms should be completed either during the tests (to record a specific issue) or as soon after finishing as practical. This will enable observations and comments to be recorded whilst the event is still fresh in the persons mind. This section of the BCP should contain a template for a Feedback Questionnaire. Prepare Budget for Testing Phase Each phase of the BCP process which incurs a cost requires that a budget be prepared and approved. The 'Preparing for a Possible Emergency' Phase of the BCP process will involve the identification and implementation of strategies for back up and recovery of data files or a part of a business process. It is inevitable that these back up and recovery processes will involve additional costs. Critical parts of the business process such as the IT systems, may require particularly expensive back up strategies to be implemented. Where the costs are significant they should be approved separately with a specific detailed budget for the establishment costs and the ongoing maintenance costs. This section of the BCP will contain a list of the testing phase activities and a cost for each. It should be noted whenever part of the costs is already incorporated with the organization’s overall budgeting process.  Training Core Testing Team for each Business Unit In order for the testing process to proceed smoothly, it is necessary for the core testing team to be trained in the emergency procedures. This is probably best handled in a workshop environment and should be presented by the persons responsible for developing the emergency procedures. This section of the BCP should contain a list of the core testing team for each of the business units who will be responsible for coordinating and undertaking the Business Recovery Testing process. It is important that clear instructions are given to the Core Testing Team regarding the simulated conditions which have to be observed. Conducting the Tests The tests must be carried out under authentic conditions and all participants must take the process seriously. It is important that all persons who are likely to be involved with recovering a particular business process in the event of an emergency should participate in the testing process. It should be mandatory for the management of a business unit to be present when that unit is involved with conducting the tests. Test each part of the Business Recovery Process In so far as it is practical, each critical part of the business recovery process should be fully tested. Every part of the procedures included as part of the recovery process is to be tested to ensure validity and relevance. This section of the BCP is to contain a list of each business process with a test schedule and information on the simulated conditions being used. The testing co-ordination and monitoring will endeavor to ensure that the simulated environments are maintained throughout the testing process, in a realistic manner. Test Accuracy of Employee and Vendor Emergency Contact Numbers During the testing process the accuracy of employee and vendor emergency contact information is to be re- confirmed. All contact numbers are to be validated for all involved employees. This is particularly important for management and key employees who are critical to the success of the recovery process. This activity will usually be handled by the HRM Department or Division. Where, in the event of an emergency occurring outside of normal business hours, a large number of persons are to be contacted, a hierarchical process could be used whereby one person contacts five others. This process must have safety features incorporated to ensure that if one person is not contactable for any reason then this is notified to a nominated controller. This will enable alternative contact routes to be used. Assess Test Results Prepare a full assessment of the test results for each business process. The following questions may be appropriate: Were objectives of the Business Recovery Process and the testing process met - if not, provide further comment Were simulated conditions reasonably "authentic" - if not, provide further comment Was test data representative - if not, provide further comment Did the tests proceed without any problems - if not, provide further comment What were the main comments received in the feedback questionnaires Each test should be assessed as either fully satisfactory, adequate or requiring further testing. Training Staff in the Business Recovery Process All staff should be trained in the business recovery process. This is particularly important when the procedures are significantly different from those pertaining to normal operations. This training may be integrated with the training phase or handled separately. The training should be carefully planned and delivered on a structured basis. The training should be assessed to verify that it has achieved its objectives and is relevant for the procedures involved. Training may be delivered either using in-house resources or external resources depending upon available skills and related costs. Managing the Training Process For the BCP training phase to be successful it has to be both well managed and structured. It will be necessary to identify the objective and scope for the training, what specific training is required, who needs it and a budget prepared for the additional costs associated with this phase. Develop Objectives and Scope of Training The objectives and scope of the BCP training activities are to be clearly stated within the plan. The BCP should contain a description of the objectives and scope of the training phase. This will enable the training to be consistent and organized in a manner where the results can be measured, and the training fine tuned, as appropriate. The objectives for the training could be as follows : "To train all staff in the particular procedures to be followed during the business recovery process". The scope of the training could be along the following lines : "The training is to be carried out in a comprehensive and exhaustive manner so that staff become familiar with all aspects of the recovery process. The training will cover all aspects of the Business Recovery activities section of the BCP including IT systems recovery". Consideration should also be given to the development of a comprehensive corporate awareness program for communicating the procedures for the business recovery process. Training Needs Assessment The plan must specify which person or group of persons requires which type of training. It is necessary for all new or revised processes to be explained carefully to the staff. For example it may be necessary to carry out some process manually if the IT system is down for any length of time. These manual procedures must be fully understood by the persons who are required to carry them out. For larger organizations it may be practical to carry out the training in a classroom environment, however, for smaller organizations the training may be better handled in a workshop style. This section of the BCP will identify for each business process what type of training is required and which persons or group of persons need to be trained. Training Materials Development Schedule Once the training needs have been identified it is necessary to specify and develop suitable training materials. This can be a time consuming task and unless priorities are given to critical training programmes, it could delay the organization in reaching an adequate level of preparedness. This section of the BCP contains information on each of the training programmes with details of the training materials to be developed, an estimate of resources and an estimate of the completion date. Prepare Training Schedule Once it has been agreed who requires training and the training materials have been prepared a detailed training schedule should be drawn up. This section of the BCP contains the overview of the training schedule and the groups of persons receiving the training. Communication to Staff Once the training is arranged to be delivered to the employees, it is necessary to advise them about the training programmes they are scheduled to attend. This section of the BCP contains a draft communication to be sent to each member of staff to advise them about their training schedule. The communication should provide for feedback from the staff member where the training dates given are inconvenient. A separate communication should be sent to the managers of the business units advising them of the proposed training schedule to be attended by their staff. Each member of staff will be given information on their role and responsibilities applicable in the event of an emergency. Prepare Budget for Training Phase Each phase of the BCP process which incurs a cost requires that a budget be prepared and approved. Depending upon the cross charging system employed by the organization, the training costs will vary greatly. However, it has to be recognized that, however well justified, training incurs additional costs and these should be approved by the appropriate authority within the organization. This section of the BCP will contain a list of the training phase activities and a cost for each. It should be noted whenever part of the costs is already incorporated with the organization’s overall budgeting process. Assessing the Training The individual BCP training programmes and the overall BCP training process should be assessed to ensure its effectiveness and applicability. This information will be gathered from the trainers and also the trainees through the completion of feedback questionnaires. Feedback Questionnaires Assess Feedback Feedback Questionnaires It is vital to receive feedback from the persons managing and participating in each of the training programmes. This feedback will enable weaknesses within the Business Recovery Process, or the training, to be identified and eliminated. Completion of feedback forms should be mandatory for all persons participating in the training process. The forms should be completed either during the training (to record a [...]... visibility over the test coverage of requirements  Test Director links requirements to test cases and test cases to defects  Manages Both Manual and Automated Testing  Test Director can manage both manual and automated tests (Win Runner)  Scheduling of automated tests can be effectively done using Test Director Test Report Standards - Defining the components that should be included in a test report Statistical... all of the following information Status Report Word Processing Tests or Keypad Tests Basic Skills Tests or Data Entry Tests Progress Graph Game Scores Test Report for each test Test Director:  Facilitates consistent and repetitive testing process  Central repository for all testing assets facilitates the adoption of a more consistent testing process, which can be repeated throughout the application... system testing, once the software has been integrated and functional testing is complete Average CPU utilization Average memory Utilization Measured I/O transaction rate Test Reporting A final test report should be prepared at the conclusion of each test activity This includes the following  Individual Project Test Report  Integration Test Report  System Test Report  Acceptance test Report These test. .. components Platform- The hardware and Software environment in which the software system will operate Test Transactions, Test Suites, and Test Events These are the test products produced by the test team to perform testing Test transactions/events: The type of tests that will be conducted during the execution of tests, which will be based on software requirements Inspections – A verification of process deliverables... characteristics: Safe area  Used for enquiry tests, usability tests etc  No test changes the data, so the area can be trusted  Many testers can use simultaneously Change area  Used for tests which update/change data  Data must be reset or reloaded after testing  Used by one test/ tester at a time Scratch area  Used for investigative update tests and those which have unusual requirements... improvements  System Test Reports A System Test plan standard that identified the objective of testing , what was to be tested, how was it to be tested, and when tests should occur The system test Report should present the results of executing the test plan If these details are maintained Electronically , then it need only be referenced , not included in the report  Acceptance Test Report There are... following: 1 =Test is needed, but not performed 2 =Test currently being performed 3=MINOR DEFECT NOTED 4=Major defect noted 5 =Test complete and function is defect free for the criteria included in this test TEST FUNCTION 1 2 3 4 5 6 7 8 A B C D E Function Test Matrix 1 9 Methods of Test Reporting Reporting Tools - Use of word processing, database, defect tracking, and graphic tools to prepare test reports... Analysis - Ability to draw statistically valid conclusions from quantitative test results Testing Data used for metrics Testers are typically responsible for reporting their test status at regular intervals The following measurements generated during testing are applicable: Total number of tests Number of Tests executed to date Number of tests executed successfully to date Data concerning software defects... for use of testers, test managers, and the software development team The frequency of the test reports should be based on the discretion of the team and extensiveness of the test process Use of Function /Test matrix: This shows which tests must be performed in order to validate the functions and also used to determine the status of testing Many organizations use spreadsheet package to maintain test results... an Y2K lawsuit  Individual Project Test Report These reports focus on the Individual projects(software system),when different testers should test individual projects, they should prepare a report on their results  Integration Test Report Integration testing tests the interfaces between individual projects A good test plan will identify the interfaces and institute test conditions that will validate . test strategy. Test Approach /Test Architecture are the acronyms for Test Strategy. Test management is also concerned with both test resource and test environment. Test Strategy Flow : Test Cases and Test Procedures should manifest Test Strategy. Test Strategy – Selection Selection of the Test Strategy is based on