Tài liệu Java Testing and Design- P9 doc

379 Chapter 12 Turning Test Agent Results into Actionable Knowledge he stock trading information system in Chapter 11 presents a methodol- ogy, infrastructure, software design, and protocol design to implement a Web-enabled application with great scalability, reliability, and functionality. We designed user archetypes, wrote multiprotocol intelligent test agents, and made requests to an application host. First, we checked for the correct func- tional results, then we checked the host’s ability to serve increasing numbers of concurrent users. All of this activity provides a near-production experience from which we can uncover scalability problems, concurrency problems, and reliability problems. It also usually generates a huge amount of logged data. Looking into the logged data allows us to see many immediate problems with the Web-enabled application under test. The log file is one of many places you can observe problems and find places to optimize the Web- enabled application. This chapter shows how to understand and analyze a Web-enabled application while the test is running and how to analyze the results data after the test is finished. With the method presented in this chap- ter, you will be able to demonstrate the system’s ability to achieve scalability, reliability, and functionality. Chapter 11 took the techniques presented in earlier chapters to command services over a variety of protocols (HTTP, HTTPS, SOAP, XML-RPC) and build a test modeled after a set of user archetypes. It presented the test goals, user archetypes, and test agents for an example stock trading firm. The mas- ter component of the test handled configuration, test agent thread creation, T PH069-Cohen.book Page 379 Monday, March 15, 2004 9:00 AM Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 380 Chapter 12 Turning Test Agent Results into Actionable Knowledge and recorded the results to a special log file. This chapter explains how to turn the recorded results into actionable knowledge. What to Expect from Results Analysis Chapter 11 ended with a strong word of caution. You may be tempted to con- duct a cursory review of test results for actionable knowledge. In this regard Alexander Pope had it right when he wrote: “A little learning is a dangerous thing; Drink deep, or taste not the Pierian spring.” Thoroughly analyzing test results produces actionable knowledge, whereas looking only at the surface of the test result data can lead to terrible problems for yourself, your com- pany, and your project. So, before showing how to analyze the test result data generated from the intelligent test agents in the previous chapter, this section presents what we can reasonably expect to uncover from conducting a test. Results data provides actionable knowledge, but the meaning may be con- tingent on your role in the software process. Consider the following tests and how the actionable knowledge changes depending on who is running the test. Table 12–1 describes this in detail. In each case, the same intelligent test agents may stage a test but the results log is analyzed to find different actionable knowledge. For example, Table 12–1 Actionable Knowledge Changes Depending on Who is Running the Test Activity Test Who Actionable knowledge A software developer writes a new function Functional test Developer Determines that the func- tion works and the new module is ready for testing. Delivery of new software build Scalability and concurrency test QA technician Identifies optimization pos- sibilities to improve perfor- mance and reduce resource needs (CPU, disk, mem- ory, database). Production servers upgraded Rollout test IT manager Determines when the data- center infrastructure is capable of serving fore- casted user levels. PH069-Cohen.book Page 380 Monday, March 15, 2004 9:00 AM Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. What to Expect from Results Analysis 381 when a QA analyst looks at the log file on a server undergoing a scalability and concurrency test, the analyst will be looking for log entries that indicate when a thread becomes deadlocked because it is waiting on resources from another thread. The developer looking at the same results log would be satis- fied that the module under test functioned. Therefore, a starting point in analyzing results is to understand the goals of the test and see how the goals can be translated to results. Following are a few test goals and how the goals may be translated to actionable results. Goal: Our New Web Site Needs to Handle Peak Loads of 50 Concurrent Users Imagine a company Web site redesign that added several custom functions. Each function is driven by a Java servlet. The goal identifies the forecasted total number of concurrent users. The definition for concurrency is covered later in this chapter. For the moment concurrency means the state where two or more people request a function at the same time. One technique to translate the goal into an actionable result is to look at the goal in reverse. For example, how would we know when the system is not able to handle 50 concurrent users? Imagine running multiple copies of an intelligent test agent concurrently for multiple periods of time. Each test period increases the number of concurrently running agents. As the test agents run, system resources (CPU time, disk space, memory) are used and the overall performance of the Web-enabled application slows. The logged results will show that the total number of transactions decreases as more con- current agents run. Charting the results enables us to set criteria for acceptable performance under peak loads. For example, at 100 concurrent test agents the total num- ber of transactions completed might be three times smaller than when 50 concurrent test agents are run. Charting the transactions completed under an increasing number of concurrent test agents enables us to pick a number between 50 and 100 concurrent test agents where system throughput is still acceptable. Goal: The Web Site Registration Page Needs to Work Flawlessly Imagine a company that promotes a new fiction book. The company Web site provides a Web page for prospective customers to register to receive PH069-Cohen.book Page 381 Monday, March 15, 2004 9:00 AM Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 382 Chapter 12 Turning Test Agent Results into Actionable Knowledge announcements when the book is published. A simple HTML form enables prospective customers to enter their contact information, including their email address. A Microsoft ASP.NET object serves the HTML form. When a user posts his or her contact information, the ASP.NET object needs to record the information to a database and redirect the user to a download page. This reminds me of a time I waited for a phone call from a woman I invited out to dinner on a date. The longer I waited, the more I thought the phone might not be working. To my chagrin, I lift the phone receiver and find that the phone was indeed working. Doing so, of course, prevented her call from getting through to me. The analog to this is testing an HTML form. Until you actually click the submit button in a browser interface, you don’t really know that the server is working. Yet, clicking the button causes the server to do actual work for you that takes resources away from real users. One technique to translate the goal into an actionable result is to under- stand the duration of the goal. Consider that the only way to know that the HTML form and ASP.NET object are working flawlessly is use them. And each time they are used and perform correctly we have met the goal. So how long do you keep testing to achieve the goal of “flawless performance”? Understanding the goal of the test can be translated into a ratio of successes to failures. The service achieves the goal when the ratio of successful tests of the HTML form and the ASP.NET object exceed by a set among the tests that failed. For example, over a period of 24 hours the goal is achieved if the ratio of successful tests to tests with failures always exceeds 95%. Searching the logged results for the ratio is fairly straightforward. Alrighty then! Goal: Customer Requests for Month-End Reports Must Not Slow Down the Order-Entry Service A common system architecture practice puts a load-balanced group of appli- cation servers in front of a single database server. Imagine the application server providing two types of functions: one function uses many database queries to produce a month-end sales report to salespeople and the second uses database insert commands to enter new orders into the database. In a Web environment both types of functions may be used at the same time. One technique to translate the goal into an actionable result is to look at the nature of the goal. When the goal speaks of multiple concurrent activi- ties, then an actionable result provides feedback to tune the application. The tuning shows system performance when the ratio of activity types changes. PH069-Cohen.book Page 382 Monday, March 15, 2004 9:00 AM Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. The Big Five Problem Patterns 383 In this example, the goal betrays that the system slows down toward the end of the month as customers increasingly request database query-intensive reports. In this case the goal can be translated into actionable results by using a combination of two test agents: one agent requests month-end reports and the second places orders. Testing the system with 100 total agents and a changing mix of test agents shows a ratio of overall system performance to the mix of agents. For example, with 60 agents requesting month-end reports and 40 agents placing orders, the system performed twice as fast as with 80 agents requesting month-end reports and 20 agents placing orders. The changing mix of agent types and its impact of overall performance makes it possible to take action by optimizing the database and improving computing capacity with more equipment. Goal Summary The examples and goals presented here are meant to show you a way to think through the goals to determine a course of action to get actionable knowl- edge from test results. Many times simple statistics from logged results are presented. While these statistics might look pretty, the actionable knowledge from the test results is the true goal you are after. The Big Five Problem Patterns Looking through raw logged results data often gives a feeling of staring up at the stars on a cold, clear winter night. The longer one looks into the stars, the more patterns emerge. In testing Web-enabled applications for scalability, performance, and reliability, five patterns emerge to identify problems and point to solutions. Resource Problems While there may be new software development techniques on the way, today’s Web-enabled application software is built on a “just-in-time” architec- ture. An application responds to requests the moment it receives the request. Web-enabled applications written to run on a host typically wait until a given resource (CPU bandwidth, disk space, memory, network bandwidth) becomes available. This is a major cause for application latency. At any moment the hosting server must be able to provide the needed resource to the application, including resources from Web service hosts. The PH069-Cohen.book Page 383 Monday, March 15, 2004 9:00 AM Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 384 Chapter 12 Turning Test Agent Results into Actionable Knowledge log results record the latency that occurs while the host and Web services provide the needed resources to the application. The host running a Web-enabled application provides resources in the form of memory, disk space, processor bandwidth, and network connectivity. Two strategies emerged over the years to understand resource allocation in server-side applications. Remote agent programs running separately monitor resources and provide you with a remote programmatic interface for retriev- ing the resource information. The leader in this space is the System Network Monitor Protocol (SNMP) standard. Java developers have access to SNMP agent data through the Java Management Extensions (JMX) library. Details on both are found at http://java.sun.com/jmx. Management consoles, such as HP OpenView, provide a dashboard that collects, collates, and displays SNMP agent data. The second strategy for understanding resource allocation is to build resource monitoring into the Web-enabled application. This is accomplished by providing the software developer with a set of APIs that provide live data about the resources available on the host machine and those used by the application. The developer writes the Web-enabled application to write the resource usage to the results log as each transaction is being handled. As we found in the test agent presented in Chapter 11, we chose the later strategy for understanding resource allocation. The test agents logged the transaction data to a log file. For example, the agents could also use the built- in resource reporting APIs to save current memory size, time from the last Java Virtual Machine garbage collection, and amount of free disk space. Most test tools, including those introduced in Chapter 5, have methods available to scripts to learn about current resource usage and availability. Writing code in a test agent to check resource allocation is the better strat- egy because the developer writing the test agent knows what resource infor- mation is important. Generically recording system resources using a management console and SNMP agents has a tendency to produce extrane- ous log results. While any unexpected latency reported in a results log might appear to be caused by a lack of resources, latency is oftentimes caused by one of the other four big types of problems. Concurrency Problems The days when a computer’s video display froze for a few seconds when the floppy disk drive started up ended when multiple dedicated processors were PH069-Cohen.book Page 384 Monday, March 15, 2004 9:00 AM Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. The Big Five Problem Patterns 385 added to the motherboard. Even on a single processor-equipped desktop or server machine, the motherboard contains separate processors for handling disk, sound, video, and network operations. The modern computer is a multi- tasking machine by design. Event-driven applications, on the server or client side, are built to handle multiple concurrent tasks. Concurrency is a measurement taken when more than one user operates a Web-enabled application. One can say a Web-enabled application’s concur- rency is good when the Web service can handle multiple users’ operating functions and making requests at the same time with little speed degradation while handling each user’s operations. Concurrency measurements are recorded in two ways. When an applica- tion runs on multiple load-balanced machines, a simple analysis of the com- bined results log shows the concurrency of the applications running on the machines behind the load balancer as a unit. Second, as each machine han- dles concurrent requests for the application’s functions, the results log shows how the operating system and application handle threads and context switches. In both ways, I measure concurrency by determining when a transaction’s start time is after a second transaction’s start time, but before the second transaction’s end time. A test agent script can parse through the log results and tally the number of concurrent transactions. In today’s world of datacenters where so few people understand what is impacting system performance but almost everyone is a user of Web-enabled applications, it has become popular opinion that “concurrency” is the cause of bad things. Contrary to popular opinion, concurrency is not a bad thing. Few applications require all of the host’s resources all the time. Unfortu- nately for server applications, concurrency has gotten a bad reputation as the root of increasingly slow application performance. In reality, measuring con- currency is a great way to measure how efficiently the system shares its resources. The typical pattern that identifies concurrency problems is to measure concurrency over a period of time. During this time period, the test increases the number of concurrently running transactions. For an indicator of a con- currency problem, look at the ratio of TPS at the start and end of the test. If you observe that the number of transactions completed decreases as the number of concurrent transactions increases, then you can suspect a concur- rency problem. PH069-Cohen.book Page 385 Monday, March 15, 2004 9:00 AM Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 386 Chapter 12 Turning Test Agent Results into Actionable Knowledge Concurrency problems are usually caused when bottlenecks are built into a system. Multiple requests stack up waiting for a single synchronized method to complete its function for the previously received requests. An analysis of the results log also shows a ratio of transaction times for transactions that are not running concurrently to those that are running concurrently. Understanding the pattern for concurrency problems is a significant asset when solving scalability and performance problems. However, sometimes concurrency problems mask a component problem. Component Problems The problem patterns identified so far appear because they reoccur in the results log. The more activity the system processes, the more the problem occurs. On the other hand, component problem patterns are nonrecurring or occur seldom enough that there is no repeated pattern that becomes obvious from analyzing results logs. Additionally, component problems appear in a results log as errors, whereas the majority of results log entries are successful transactions. When the component fails, it fails rarely. For component prob- lems we need a different strategy for results log analysis. The top priority for developers, QA analysts, and IT managers when solv- ing component problems is to determine which component fails and what scenario of actions, use, and load contributes to the failure. For example, consider a private collaborative extranet service that offers a large enterprise sales organization the ability to share documents, chat, and participate in bul- letin-board-style threaded messages. The extranet service, hosted by Inclu- sion Technologies, uses an XML single-sign-on mechanism described on IBM developerWorks at http://www-106.ibm.com/developerworks/webser- vices/library/ws-single/. After signing in, a salesperson reads briefing materi- als and posts questions to a group discussion list. The salespeople may optionally subscribe to participate in the group discussion through their email client. As messages are posted to the discussion list, the Web users see the messages in a threaded list and the email subscribers receive a copy of the posted message in their email account. Replies to the email messages are posted back to the discussion group. Looking through the logs of the collaborative extranet system showed that approximately every three days a salesperson was not receiving a day’s worth of email messages. The problem was intermittent and appeared to resolve on its own, only to fail later. PH069-Cohen.book Page 386 Monday, March 15, 2004 9:00 AM Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. The Big Five Problem Patterns 387 The solution was to build and run an intelligent test agent modeled after a salesperson’s behavior. The agent signs in, posts messages, receives, and replies to email messages. When the system fails, the agent marks the time and steps that caused the problem. Identifying the scenario that causes the problem shows the software development team where a thread is not han- dling exceptions thrown by the email server. The next day when the user signs in, a new thread is instantiated to replace the hung one and the email delivery problem is magically solved. Understanding what part of the Web- enabled application failed leads the developers to build and deploy a fix to the code. The key to solving component problems is finding the malfunctioning component. Test agents help to hit a component in just the right way and observe the failure. Contention Problems Competition between Web-enabled applications, and on a lesser scale the threads in a single application, lead to the system making decisions on which thread gets the focus and when. Contention problems happen when one type of thread predominantly gets the focus. For example, in the Stock Trading Company example in Chapter 11, the system responded to many requests concurrently. Imagine what would happen when the system gave preference to database queries. More requests from workers doing stock market research would cause the stock trader’s requests to slow down. Running multiple concurrent intelligent test agents against an information system provides a unique view of the contention problems in the system. The test agents can change the mix of concurrently running test agent types and analyze the impact on performance and scalability. For example, consider the test results in Table 12–2. Table 12–2 Test Results from Multiple Concurrent Test Agents Test agent Quantity running concurrently Average transaction time Mira 40 16 seconds Simon 30 22 seconds Doree 30 35 seconds PH069-Cohen.book Page 387 Monday, March 15, 2004 9:00 AM Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 388 Chapter 12 Turning Test Agent Results into Actionable Knowledge Now, imagine we adjust the mix of concurrently running test agents so that more Simon agents run. Table 12–3 shows the test results. Note that we doubled the number of Simon agents from the previous test and the average transaction time for Simon decreased. At the same time we ran half the number of Mira agents, but the average transaction for Mira slowed to 50 seconds. While many problems might lead to these perfor- mance inconsistencies, experience tells us to look for contention problems: Look at the system memory usage, look at the database resource logs, and look at the application server settings. Crash Recovery Problems When a desktop or laptop computer crashes, our first urge is to restart the computer to solve any lingering problems brought on by the crash. Servers are meant to run for a long time and handle even exceptional cases like appli- cation crashes. So restarting the server is usually not an option. Instead, we must build high-quality crash handling code into Web-enabled applications. Intelligent test agents have a place to play in detecting faulty crash recovery code. Test agents are in a unique position to be able to cause crashes and drive the system to handle additional requests concurrently. For example, consider a test scenario where the Simon test agent from Chapter 11 makes dozens of requests for research reports. The system makes dozens of queries to the data- base. If one or more queries throws an exception, the server must handle the exception and release any used resources. The test agent is in a unique position to learn what happens if the exception handling code does not release the thread’s memory, open file handles, and database connections. In summary of the Big Five Problem Patterns, we found general patterns emerge when analyzing results logs. The patterns provide a way to detect prob- Table 12–3 Notice the Change from Changing the Test Agent Mix Test agent Quantity running concurrently Average transaction time Mira 20 50 seconds Simon 60 20 seconds Doree 20 35 seconds PH069-Cohen.book Page 388 Monday, March 15, 2004 9:00 AM Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... business and home user’s access to the Internet For every ISP, automation is no longer an option They require cost-effective solutions to automate basic configuration and management tasks, including DHCP and NAT routing configuration and DNS service configuration and security configuration Plus, customer understanding of Internet network configuration and management has increased expectations of ease of use and. .. future CMS systems enable 2Wire customers to handle large deployments Understanding how CMS solutions perform in large deployments is the key to 2Wire’s strategy to provide scalable solutions to ISPs 2Wire’s focus on reliability, scalability, and extensibility demands a deep understanding of the system Understanding What Happens over the Wire To get an understanding of the 2Wire system architecture, I... reliable and scalable in increasingly large deployments While the individual 2Wire system components are standard off-the-shelf proven Web infrastructure, 2Wire needed to develop and use new test methodologies, techniques, and tools to understand the overall system performance To address the advanced nature of the 2Wire CMS solution design, 2Wire developed two test methodologies: • • User goal-oriented testing Rather... performance, and reliability Experience shows that the final report usually focuses on two areas: throughput and reliability Throughput is measured in terms of transactions per second and answers the question, “Did I buy enough servers?” Reliability is measured by a SPI and answers the question, “Does this system allow users to achieve their goals?” Measurements of concurrency, latency, availability, and performance... problems The TPS and SPI methods for measuring concurrency, latency, availability, and performance provide a set of metrics that software developers, QA analysts, and IT managers use to manage system reliability Next, we see how these measurements may provide misleading results and how to avoid making such mistakes Scenarios Where Results Data Misleads Over the years of pouring over results logs and implementing... with the 2Wire datacenter over HTTPS and TCP/IP on the device’s DSL connection to the 2Wire datacenter The gateway device executes a “bootstrap” command code The CMS datacenter responds with the device’s IP address, DNS information, NAT and DHCP configuration information, and a number of other settings All of this communication happens between the gateway device and the CMS datacenter invisibly to the... TCP/IP, DNS, NAT, and DHCP values erability and security in an open environment, 2Wire knows the gateway devices would always only communicate with a CMS datacenter So it only needs to worry about its own interoperability Each communication from a gateway device to the CMS datacenter takes three or more steps to authenticate the gateway device, pass a command request and its supporting data, and receive... these connections are a potent source of poor performance because of TCP’s slow startup and the round trips required to open and close each TCP connection The World Wide Web Consortium (W3C) developed HTTP/1.1 persistent connections and pipelining to reduce network traffic and the amount of TCP overhead caused by opening and closing TCP connections However, HTTP/ 1.1 has its own drawbacks for the 2Wire architecture... authenticates the gateway device and returns an authenticated session identifier to the gateway device The 2Wire system uses a command protocol to enable the gateway devices to request updates, receive its initial settings, and provide registration information The protocol is extensible so 2Wire may add additional commands without redeploying the gateway devices 2Wire Datacenter Request: command: authorize, keycode:... Concurrency and Scalability in a High-Volume Datacenter test environment based on the TestMaker open source project and to conduct an independent audit of CMS system performance and scalability: • • • • We wrote a scalability study proposal to define the criteria for good performance and scalability of the backend HomePortal/ CMS support system We interviewed the software developers, QA technicians, and IT . (SNMP) standard. Java developers have access to SNMP agent data through the Java Management Extensions (JMX) library. Details on both are found at http:/ /java. sun.com/jmx Determines that the func- tion works and the new module is ready for testing. Delivery of new software build Scalability and concurrency test QA technician