the art of scalability scalable web architecture processes and organizations for the modern enterprise phần 3 potx

ptg5994185 BUILDING TEAMS—A SPORTS ANALOGY 93 Building Teams—A Sports Analogy Professional football team coaches and management know that having the right team to accomplish the mission is critical to reaching the Super Bowl in any given season. Furthermore, they understand that the right team today might not be the right team for next season; rookie players enter the sport stronger and faster than ever before; offensive strategies and needs change; injuries plague certain players; and salary caps create downward pressure on the total value of talent that can exist within any team in any year. Managing team skill sets and skill levels in professional sports is a constant job requiring the upgrading of talent, moving personnel to different positions, manage- ment of depth and bench strength, selection of team captains, recruiting new talent, and coaching individual high performance players. Imagine a coach or general manager faced with the difficult task of needing to bring in a new player at a high salary to fill a specific weakness in his team. That coach is likely already at or near the team’s salary cap. The choices are to remove an existing player, renegotiate one or more players’ salaries to make room for the new player’s salary, or not hire the necessary player into the critical position. What do you think would happen to the coach who decides to take no action and not hire the new player? If his owners find out, they would likely remove him and if they didn’t find out sooner or later, the team would atrophy and consistently turn out substandard seasons resulting in lower ticket sales and unhappy shareholders (owners). Our jobs as managers and executives are really no different than the jobs of the coaches of professional football teams. Our salary caps are the budgets that are developed by the executive management team and are reviewed and approved by our boards of directors. In order to ensure that we are cost effectively doing our jobs with the highest possible throughput and an appropriate level of quality, we too must con- stantly look for the best talent available at a price that we can afford. Yet most of us don’t actively manage the skills, people, and composition of our teams, which in turn means that we aren’t doing the right thing for our company and our shareholders. Scalability in professional sports means scaling the output of individuals; profes- sional football, for instance, will not allow you to add a twelfth player. In your orga- nization, scaling individuals might mean the same thing. The output of your organization is dependent both on the output of any individual as well as the size of your team. Efficiency in output, another component of scale (or at least scaling cost effectively), is a measurement of getting more for the same amount of money or (bet- ter yet) more for less money. Scaling with people then is a function both of the indi- vidual people, the number of people, and the organization of people. Now think about a coach who refused to spend time improving his players. Can you imagine such a coach keeping her job? Similarly, can you imagine walking into ptg5994185 94 CHAPTER 5MANAGEMENT 101 your next board of directors meeting and stating that part of your job is not to grow and maintain the best team possible? Think about that last point for a minute. In our last chapter on leadership, we made the case that everything you do needs to be focused on shareholder value creation. Here, we have just identified a test to help you know when you are not creating shareholder value. For any major action that you make, would you go in and present it to the board of directors as something that must be done? Remember that a decision to not do something is the same as deciding to do something. Further, ignoring something that should be done is a decision not to do it. If you have not spent time with the members of your team for weeks on end, you have decided not to spend time with them and that is absolutely inexcusable and not something that you would likely feel comfortable discussing with your board of directors. The parallels in professional sports to the responsibilities of team building for cor- porate executives are clear but all too commonly ignored. To get our jobs done, we must have the best talent (the best people) possible for our board authorized budgets. We must constantly evaluate and coach our team to ensure that each member is add- ing value appropriate to his level of compensation, find new and higher performing talent, and coach the great talent that we have to even higher levels of performance. Upgrading Teams—A Garden Analogy Even a novice gardener knows that gardening is about more than just raking some soil, throwing some seeds, and praying for rain. Unfortunately, if you are like most managers, rake, throw, and pray is probably exactly what you do with your team. Our team is a garden and our garden expects more of us than having manure spread upon it at times convenient to us. As importantly, the scalability of our organization as we described in our last metaphor is largely tied to how great our talent is on a per person basis and how consistent their behaviors are with our corporate culture. Gardens should be designed and so should our teams. Designing our teams means finding the right talent that matches the needs of our vision and mission. Before planting our garden or inserting new seeds or seedlings in our garden, we evaluate how the different plants and flowers will interact. We should do the same with our teams. Will certain team members steal too many nutrients? Will the soil (our cul- ture) properly support their needs? Should the garden be full of only bright and bril- liant flowers or will it be more pleasing with robust and healthy foliage to support the flowers? Managers in hyper-growth companies often spend a lot of time interviewing and selecting candidates but usually not much time on a per candidate basis. Worst still, these managers often don’t take the time to determine where they’ve gone wrong ptg5994185 UPGRADING TEAMS—A GARDEN ANALOGY 95 with past hiring decisions and what they’ve done well in certain decisions. Finding the right individual for your job requires paying attention to and correcting your past failures and repeating your past hiring successes. We might interview for skills but overlook critical items like cultural or team fit. Why have you had to remove people? Why have people decided to leave? Candidate selection also requires paying attention to the needs of the organization from a productivity and quality perspective. Do you really need another engineer or product manager, or do your pipeline inefficiencies indicate additional process defini- tion needs, tools engineers, or quality assurance personnel? Too often, we try to make hiring decisions after we’ve spent 30 to 60 minutes with a candidate. We encourage you to spend as much time as possible with the candidate and try to make a good hire the first time. Seek help in interviewing by adding people whom you trust and who have great interviewing skills to your interview team. Call previous managers and peers and be mindful to ask and prod for weaknesses of indi- viduals in your background checks. Pay attention to more than just the skills and determine whether you and your team will like spending a lot of time with the indi- vidual. Interview the person to make certain that she will be a fit with your cultures and that her behaviors are consistent with the behavioral expectations of the company. The Cultural Interview One of the most commonly overlooked components of any interview is interviewing a candidate to ensure that he is a cultural and behavioral fit for your company. We recommend picking up a book or taking a class on behavioral interviewing, but here are some things that you can do in your next interview to find the right cultural and behavior fit for your company: • Make a list of your company’s beliefs regarding people. They may be on the back of your identification badge or on your intranet. Identify questions around these beliefs and dis- tribute them to interview members. • Identify interviewers who are both high performers within your team and a good match with the cultures, beliefs, and behaviors of your company (or the behaviors to which your company aspires). • Gather after the interview and discuss the responses to the questions and the feelings of the team. It is as important to make the right cultural hire as it is to hire the right talent and experience. Can you spend 9 to 12 hours a day with this person? Can the team do the same? Can you learn from him? Will the candidate allow the team to teach him? ptg5994185 96 CHAPTER 5MANAGEMENT 101 Feeding your garden means spending time growing your team. Of all the practices in tending to your team, this is the one that is most often overlooked for lack of time. We might spend time picking new flowers (though not enough on a per flower basis), but we often forget about the existing flowers needing nourishment within our garden. The intent of feeding is to help grow the members of your team who are producing to the expectations of your shareholders. Feeding consists of coaching, praising, cor- recting technique or approach, adjusting compensation and equity, and anything else that creates a stronger and more productive employee. Feeding your garden also means taking individuals who might not be performing well in one position and putting them into positions where they can perform well. However, if you find yourself moving an employee more than once, it is likely that you are avoiding the appropriate action of weeding. Finally, feeding your garden means raising the bar on the team overall and helping employees achieve greater levels of success. Great teams enjoy aggressive but achiev- able challenges, and it is your job as a manager to challenge them to be the best they can be. Although you should invest as much as possible in seeding and feeding, we all know that underperforming and nonperforming individuals choke team productivity just as surely as weeds steal vital nutrients from the flowers within your garden. The nutrients in this case are the time that you spend attempting to coach underperform- ing individuals to an acceptable performance level and the time your team spends compensating for an underperforming individual’s poor results. Weeding our gardens is often the most painful activity for most managers and executives, and as a result it is often the one to which we tend last. Although you must abide by your company’s practices regarding the removal of people who are not performing (these practices vary not only by country but by state as well), it is vital that you find ways to quickly remove personnel who are keeping you and the rest of your team from achieving your objectives. The sooner you remove them, the sooner you can find an appropriate replacement and get your team where it needs to be. When considering performance as a reason for termination, one should always include an evaluation of the person’s behaviors. It is possible to have an individual within an organization who creates more and gets more done than any other team member, but whose actions and behaviors bring the total output of the team down. This is typically pretty obvious in the case of an employee creating a hostile work environment, but it can also be the case for an employee who simply does not work well with others. For instance, you might have an employee who gets a lot done, but does so in a manner that absolutely no one wants to work with him. The result might be that you spend a great deal of time soothing hurt feelings or finding out how to assign the employee work that does not require teamwork. If the employee’s actions ptg5994185 UPGRADING TEAMS—A GARDEN ANALOGY 97 are such that she limits the output of the team, that limitation is by definition a scale limitation and one upon which you should immediately act. We’ve found that it’s often useful to use the concept of a two-dimensional axis with defined actions such as in Figure 5.1. The x-axis here is the behavior of the employee and the y-axis is the employee’s performance. Many employee reviews, when done properly, identify the actions on the y-axis. But many such reviews do not consider the impact of the behavioral x-axis. The idea here is that the employees you want to keep are in the upper-right portion of our graph. Those that should be imme- diately “weeded” are in the bottom-left portion of the graph. You should coach those individuals in the upper-left and lower-right portion of the graph, but be prepared to weed them should they not respond to coaching. And of course, you want all of your seeds or new employees to be targeted in the upper-right portion of the graph. One thing that we have learned over time is that you will always wish you had acted earlier in removing underperformers. There are a number of reasons why you just can’t act quickly enough, including company travel, competing requests, meet- ings, and so on. You shouldn’t waste time agonizing over whether you are acting too quickly—that never happens. You will always wish you had acted even sooner when you have completed the termination. Figure 5.1 Evaluating Behaviors and Performance Performance Unsatisfactory Behavior Superior Great Performance Bad Behavior Poor Performance Great Behavior Poor Performance Bad Behavior Weed Immediately! Coach or “Feed”— If Unsuccessful, Weed Coach or “Feed”— If Unsuccessful, Weed Feed! Target for New Hires Great Performance Great Behavior ptg5994185 98 CHAPTER 5MANAGEMENT 101 Seed, Feed, and Weed to Succeed To continually upgrade or improve our team’s performance, we need to perpetually perform three individual activities: • Seeding is the addition of new and better talent to our organization. • Feeding is the development of the people within our organization we want to retain. • Weeding is the removal of underperforming individuals within our organization. As managers, we often spend too little time interviewing and selecting our new employees, too little time developing and coaching our high performing employees, and act too late to remove employees who do not display behaviors consistent with our culture or have the drive and motivation to create shareholder wealth. Measurement, Metrics, and Goal Evaluation We’re not certain who first said it, but one of our favorite sayings is “You can’t improve that which you do not measure.” Amazingly, we’ve found ourselves in a number of arguments regarding this statement. These arguments range from “Mea- surement is too expensive” to “I know intuitively whether I’ve improved something.” You can get away with both of these statements if you are the only shareholder of your company, though we would still argue that your results are going to be suboptimal. If you happen to be a manager in a company with external shareholders, however, you must be able to prove that you are creating shareholder value, and the only way to do that is with data. Data in return requires measurements in order to be produced. We believe in creating cultures that support measurement of nearly everything that is related to the creation of shareholder value. With respect to scale, however, we believe in bundling our measurements thematically. The themes we most often rec- ommend for scale related purposes are cost, availability and response times, engineer- ing productivity and efficiency, and quality. As we’ve previously indicated, cost has a direct impact to the scalability of your platform. You undoubtedly are either given or have helped develop a budget for the company’s engineering initiatives. A portion of that budget in a growth company ide- ally is dedicated to the scalability of your platform or services. This alone is an inter- esting value to measure over time as we would expect that good managers will be able to reduce the cost of scaling their platforms over time. Let’s assume that you inherit a platform with scalability problems that manifest themselves as availability issues. You might decide that you need to spend 30% to 50% of your engineering time and a significant amount of capital to fix a majority of these issues in the first ptg5994185 MEASUREMENT, METRICS, AND GOAL EVALUATION 99 two to 24 months of your job. However, something is wrong if you can’t slowly start giving more time back to the business for business initiatives (customer features) over time. We recommend measuring the cost of scale as both a percentage of total engi- neering spending and as a cost per transaction. Cost of scale as a percentage of engineering time should go down over time. But it’s easy to “game” this number. If in year 1 you have a team of 20 engineers and dedicate 10 to scalability initiatives, you are spending 50% of your engineering headcount related budget on scalability. If in year 2 you hire 10 more engineers but still only dedi- cate the original 10 to scale, you are now spending only 33% of your budget. Although it would appear that you’ve reduced the cost of scale, you’ve really kept it constant, which could argue for measuring and reporting on the relative and absolute cost of scale. Rather than reporting the absolute cost of scale (10 engineers, or $1.2M per annum), we often recommend normalizing the value by the activities that create shareholder value. If you are a Software as a Service platform (SaaS) provider and make money on a per transaction basis, either through advertising or the charging of transaction fees, this might be accomplished by reporting the cost of scale on a per transaction basis. For instance, if you have 1.2 million transactions a year and spend 1.2 million in headcount on scale initiatives, your cost of scale would be $1/transac- tion. Ouch! That’s really painful if you don’t make at least a dollar a transaction! Availability is another obvious choice when figuring out what to measure. If you see a primary goal of scalability initiatives as eliminating scalability related down- time, you must measure availability and report on how much of your downtime is associated with scalability problems within your platforms or systems. The intent here is to eliminate lost opportunity associated with users not being able to complete their transactions. In the Internet world, this most often is a real impact to revenue; whereas in the back office information technology world, it might result in a greater cost of operations as people are required to work overtime to complete jobs when systems become available again. Closely related to measuring availability for the purposes of scalability is measuring response time of your systems. In most systems, increasing user perceived response times often escalate to brownouts followed by blackouts or downtime for the system. Brownouts are typically caused by systems performing so slowly that most users will abandon their efforts, whereas blackouts are a result of a system that completely fails under high demand. The measurement of response times should be against an abso- lute service level agreement (SLA), even if that agreement isn’t published to the end users. Ideally, the measurement is performed using actual end-user transactions rather than proxies for their interaction. In addition to the absolute measurement against internal or external service levels, relative measurement against past month values should be tracked over time for critical transactions. This data can later be used to justify projects if a slowing of any given critical transaction is proven to be tightly correlated with revenue associated with that transaction, abandon rates, and so on. ptg5994185 100 CHAPTER 5MANAGEMENT 101 Engineering productivity and efficiency is another important measurement when considering scalability. Your first reaction may be that these two things have abso- lutely nothing to do with the scalability of a platform. Consider an organization that measures and improves the productivity of its engineers over time versus that of an organization that has no such measurements. You would expect that the former will start to produce more products and complete more initiatives at an equivalent cost to the latter or that they would start to produce the same at a lower cost. Either of these will help us in our scalability initiatives because if we produce more, by allocating an equivalent percentage of our engineering team, we can get more done more quickly and thereby reduce future scale demands on our engineering team. And if we can produce the same at lower cost, we are increasing shareholder value as the net decrease in cost structure to produce a scalable platform means greater profitability for the company. The real trick in figuring out how to measure engineering productivity and effi- ciency is to split it up into at least two component parts. The first part has to do with whether your engineering teams are using as much of the available engineering days as possible for engineering related tasks. To do this, assume that an engineer is avail- able for 200 days/year minus your company’s sick time, vacation time, training time, and so on. Maybe your company has 15 days of paid time off a year and expects engineers to be in 10 days of training a year resulting in 175 engineering days/engi- neer. This becomes the denominator within our equation. Then, subtract from this denominator all of the hours and days spent “blocked” on issues related to unavail- able build environments, nonworking test environments, broken tools or build envi- ronments, missing source code or documentation, and so on. It shouldn’t surprise you if you haven’t measured such value destroyers in the past to find out that you are only getting to make use of 60% to 70% of your engineering days. The second component part of engineering productivity and efficiency is to mea- sure how much you get out of each of your engineering days. This is a much harder exercise as it requires you to choose among a set of unattractive options. These options range from measuring thousands of lines of code (KLOC) produced by an engineer, to stories produced, function points produced, or use cases produced. The options are unattractive as they all have “failures” within their implementation. For instance, you may produce 100 lines of code per engineer per day, but what if you really only need to write 10 to get the same job done? Function points on the other hand are difficult and costly to calculate. Stories and use cases don’t really contain a measure of complexity within their evaluation or use. As such, they all sound like bad options. But a worse option is to decide not to measure this area at all. Training pro- grams, after all, are intended to help increase individual output, and without some sort of measurement of their effectiveness, there is no way to prove to a shareholder that the money spent on training was well spent. Quality rounds out our scalability management measurement suite. Quality has a positive or negative impact on many of the other measurements. Poor product quality ptg5994185 THE GOAL TREE 101 can cause scalability issues in the production environment and as a result can increase downtime and decrease availability. Poor product quality causes an increase in cost and a reduction in productivity and efficiency as rework is needed to meet the appro- priate scalability needs. Although you obviously need to look at such typical metrics as bugs KLOC in production and per release, absolute bug numbers for your entire product, and the cost of your product quality initiatives, we also recommend further breaking these out into the issues that affect scale. How many defects cause scalabil- ity (response time or availability) problems for your team? How many do you release per major or minor release of your code and how are you getting these to trend down over time? How many do you catch in your quality assurance initiatives versus those that are found in production, and so on? The Goal Tree One easy way to map organizational goals to company goals is through a goal tree. A goal tree takes as its root one or more company or organizational goals and breaks it down into the subordinate goals to achieve that major goal. Here, we will use the computer science inverted view of a tree, where the root is at the top of the tree rather than the bottom. For instance, AllScale may have a goal to “Achieve Profit- ability by Q1.” As you can see in Figure 5.2, this company goal is at the “root” of the tree. Johnny Fixer decides that the two ways he can increase profitability is by creating more monetization opportunities and creating greater revenue at a reduced cost base. Figure 5.2 Example Goal Tree for AllScale Networks AllScale Goals Achieve Profitability by Q1 Greater Revenue at a Reduced Cost BaseMore Monetization Opportunities Availability CostQuality Efficiency Bugs per Push 0 P1 < 5 P2 Time to Verify Bugs P1 < 1d P2 < 2d P3 < 5d Coverage 100% regression per push Bugs/KLOC 0.15 Adserving 99.9% Registration 99.99% Scalability 99.99% Response SLA 99.9% $/1K Impression Current $0.56 Goal $0.25 Days/Engineer 175/200 Days KLOC/Day .20/Day Time to Market Production SLA New h/w 1wk Hot fix 1d Code push 1wk ptg5994185 102 CHAPTER 5MANAGEMENT 101 Johnny determines that quality and availability affect the opportunity to monetize AllScale’s platform and adds a number of quality and availability goals. One avail- ability goal has to do with scalability (no more than .01% downtime for the quarter due to scalability), and he also adds a 99.9% adherence to the internal response time SLAs for the platform. Quality goals are to reduce the number of bugs per push (with measurable amounts), reduce the time to verify bugs, increase test suite coverage for regression tests, and have fewer than .15 bugs/KLOC outstanding in production. From a cost perspective, Johnny desires to reduce the cost per thousand pages delivered by over 50%. Johnny also wants to impact time to market (TTM), thereby decreasing the cost of delivery, and has specific goals for that. Finally, he desires to increase his engineering productivity and decides to count both used man days versus available man days and KLOC produced per day. Paving the Path for Success So far, we’ve painted the picture of a manager as being equal parts task master, tacti- cian, gardener, and measurement guru. But a manager’s job isn’t done there. Besides being responsible for ensuring the team is up to the job, deciding on the path to take to a goal, and measuring progress, a manager is also responsible for ensuring that the path to that goal is bulldozed and paved. A manager who allows a team to struggle unnecessarily over rough terrain on the way to an objective when he can easily pave the way means reducing the output of the team. This reduction in output means the team can’t scale efficiently, as less work is applied to the end goal. Less efficiency means lower shareholder return for an investment. Bulldozed is a rather aggressive term and we don’t mean to imply that a manager should act as a fullback attempting to lay out a linebacker so that a halfback can make a touchdown. Although that type of aggressive play might be required from time to time, employing it all the time will get you a reputation that you’d rather not have. Additionally, it may be absolutely unacceptable in some cultures. What we mean here is that managers are responsible for removing obstacles to the success of an organization and its objectives. It is very easy for people to confuse this idea with “anything that stands in my way is an obstacle to my success and should be removed.” Sometimes, the obstacles in your way serve to ensure that you are performing the correct functions. For instance, if you have a need to release something to your production environment, you might see the quality assurance organization as an obstacle. This observation is at odds with our definition of obstacle, as the QA organization serves to help you ensure that you are meeting the shareholder’s needs for a higher quality product. The obstacle in this case is actually you and your perception. [...]... get to know you The best teams in the world spend thousands of hours training with each other and maybe even living together When you think of great teams, you probably think of professional sports teams or Navy SEALS or Delta Force The one thing all of these organizations have in common is a set of shared experiences and shared trials created over thousands and thousands of hours of training time... list of ideas: 1 Form relationships 2 Set the example 3 Educate other executives 4 Use the RASCI model 5 Speak in business terms 6 Get them involved 7 Scare the executive team with facts D EFEATING THE C ORPORATE M INDSET Forming Relationships One of the best ways to start changing the business executives is to begin forming a relationship with them As discussed in the section “Understanding the Experiential... costing hundreds of thousands of dollars that no one could explain the need for The old CTO would rely on the threat of “we either do this or we will die” to get all the other executives in line Although this worked for the short term, it left all the other executives feeling that something was not right with the decision that had to be forced on them Christine quickly saw this situation and put an end... databases, and so on would save a lot of engineering time and energy How Complex The second part of establishing the right process at the right time for your organization is the level of complexity of the process As we discussed previously, organizations are constantly changing, new employees are added, others leave, people mature, people learn lessons, and sometimes they forget lessons Choosing the right... “yes” to all the questions in the preceding subsection “Why the Business Executives Might Be the Problem.” Unfortunately, these executives are also probably not insightful or self-reflective enough to think they could be part of the problem and therefore need to be fixed Having worked in a few and having been a spectator to many of these types of environments, our first reaction is to run away And if you... the case that you just do not understand what they are saying There are two ways to resolve that: You can either gain a better understanding of what it is they are telling you, or you can work with them to speak a language that you better understand Better yet, do both! Why the Technology Executive Might Be the Problem For nearly all of the reasons that the business executives are responsible for their... retention, and so on It is critical for the technology executive to understand how the business makes money, the drivers of that revenue equation, the current financial reality within the business, and the current year’s financial goals for the business In AllScale, as discussed in Chapter 2, Roles for the Scalable Technology Organization, the previous CTO was promoted based on his technical acumen As the. .. considered the outage percentage and could be used in the calculation of downtime and cost associated with it However, notice the dashed line from 9:00 PM to 12:00 AM goes much higher than the normal traffic in solid This is typical of sites for consumer user bases where there is a pent-up demand for the service and a spike usually occurs afterward Unfortunately, this is a busy time of the year for AllScale’s... the use of processes that govern, control, suggest, guide, teach, and advise team members In Part II, we are going to spend time explaining various essential processes and the roles they should play in your organizations, depending on the size, maturity, culture, and duration of your business We will cover this in much more detail but we believe there is a right time and right place for processes and. .. several, of the preceding points hits home We do not mean to imply that the business leaders are the only problem However, if they absolutely refuse to accept culpability in the problem, this is a huge warning sign The best leaders accept that they are at least part of the problem, and we believe that the very best leaders believe that they are the source of most problems It absolutely may be the case, and . measurement of getting more for the same amount of money or (bet- ter yet) more for less money. Scaling with people then is a function both of the indi- vidual people, the number of people, and the organization. the disaggregation of goals into their associated projects and tasks, the assignment of individuals and organizations to those tasks, the measurement of progress, communi- cation of status, and. Navy SEALS or Delta Force. The one thing all of these organizations have in common is a set of shared experiences and shared trials created over thousands and thousands of hours of training time.