Building Web Reputation Systems- P14 docx

Help system display, giving unknown users extra navigation help Lockout of potentially abused features, such as content editing, until the user has demonstrated familiarity with the application and lack of hostility to it Deciding when to route new contributions to customer care for moderation Pros Allows for a significantly lower barrier for some user contributions than otherwise possible, for example, not requiring registration or login. Provides for corporate (internal use) karma. No user knows this score, and the site operator can change the application’s calculation method freely as the situation evolves and new proxy reputations become available. Helps render your application impervious to accidental damage caused by drive-by users. Cons Inferred karma is, by construction, unreliable. For example, since people can share an IP address over time without knowing it or each other, including it in a reputation can undervalue an otherwise excellent user by accident. However, though it might be tempting for that reason to remove IP reputation from the model, IP address is the strongest indicator of bad users; such users don’t usually go to the trouble of getting a new IP address whenever they want to attack your site. Inferred karma can be expensive to generate. How often do you want to update the supporting reputations, such as IP or cookie reputation? It would be too expensive to update them at very single HTTP roundtrip, so smart design is required. Inferred karma is weak. Don’t trust it alone for any legally or socially significant actions. Practitioner’s Tips: Negative Public Karma Because an underlying karma score is a number, product managers often misunder- stand the interaction between numerical values and online identity. The thinking goes something like this: • In our application context, the user’s value will be represented by a single karma, which is a numerical value. • There are good, trustworthy users and bad, untrustworthy users, and everyone would like to know which is which, so we will display their karma. • We should represent good actions as positive numbers and bad actions as negative, and we’ll add them up to make karma. • Good users will have high positive scores (and other users will interact with them), and bad users will have low negative scores (and other users will avoid them). This thinking—though seemingly intuitive—is impoverished, and is wrong in at least two important ways: • There can be no negative public karma—at least for establishing the trustworthi- ness of active users. A bad enough public score will simply lead to that user’s abandoning the account and starting a new one, a process we call karma bank- ruptcy. This setup defeats the primary goal of karma—to publicly identify bad actors. Assuming that a karma starts at zero for a brand-new user that an Practitioner’s Tips: Negative Public Karma | 161 application has no information about, it can never go below zero, since karma bankruptcy resets it. Just look at the record of eBay sellers with more than three red stars. You’ll see that most haven’t sold anything in months or years, either because the sellers quit or they’re now doing business under different account names. • It’s not a good idea to combine positive and negative inputs in a single public karma score. Say you encounter a user with 75 karma points and another with 69 karma points. Who is more trustworthy? You can’t tell; maybe the first user used to have hundreds of good points but recently accumulated a lot of negative ones, while the second user has never received a negative point at all. If you must have public negative reputation, handle it as a separate score (as in the eBay seller feedback pattern). Even eBay, with the most well-known example of public negative karma, doesn’t rep- resent how untrustworthy an actual seller might be; it only gives buyers reasons to take specific actions to protect themselves. In general, avoid negative public karma. If you really want to know who the bad guys are, keep the score separate and restrict it to internal use by moderation staff. The Dollhouse Mafia, or “Don’t Display Negative Karma” The Sims Online was a multiplayer version of the popular Sims games by Electronic Arts and Maxis in which the user controlled an animated character in a virtual world with houses, furniture, games, virtual currency (called Simoleans), rental property, and social activities. You could call it playing dollhouse online. One of the features that supported user socialization in the game was the ability to declare that another user was a trusted friend. The feature involved a graphical display that showed the faces of users who had declared you trustworthy outlined in green, attached in a hub-and-spoke pattern to your face in the center. People checked each other’s hubs for help in deciding whether to take certain in-game actions, such as becoming roommates in a house. Decisions like these are costly for a new user—the ramifications of the decision stick with a newbie for a long time, and “backing out” of a bad decision is not an easy thing to do. The hub was a useful decision- making device for these purposes. That feature was fine as far as it went, but unlike other social networks, The Sims Online allowed users to declare other users untrustworthy, too. The face of an untrust- worthy user appeared circled in bright red among all the trustworthy faces in a user’s hub. It didn’t take long for a group calling itself the Sims Mafia to figure out how to use this mechanism to shake down new users when they arrived in the game. The dialog would go something like this: “Hi! I see from your hub that you’re new to the area. Give me all your Simoleans or my friends and I will make it impossible to rent a house.” 162 | Chapter 6: Objects, Inputs, Scope, and Mechanism “What are you talking about?” “I’m a member of the Sims Mafia, and we will all mark you as untrustworthy, turning your hub solid red (with no more room for green), and no one will play with you. You have five minutes to comply. If you think I’m kidding, look at your hub—three of us have already marked you red. Don’t worry, we’ll turn it green when you pay….” If you think this is a fun game, think again. A typical response to this shakedown was for the user to decide that the game wasn’t worth $10 a month. Playing dollhouse doesn’t usually involve gangsters. It’s hard to estimate the final cost to EA & Maxis for such a simple design decision, in terms of lost users, abandoned accounts, and cancelled subscriptions. In your own community and application design, think twice about overtly displaying negative reputation, or putting such direct means in the hands of the community to affect other’s reputations. You risk enabling your own mafias to flourish. Draw Your Diagram With your goals, objects, inputs, and reputation patterns in hand, you can draw a draft reputation model diagram and sketch out the flows in enough detail to generate the following questions: what data will I need to formulate these reputation scores cor- rectly?; how will I collect the claims and transform them into inputs?; which of those inputs will need to be reversible, and which will be disposable? If you’re using this book as a guide, try sketching out a model now, before you consider creating screen mock-ups. One approach we’ve often found helpful is to start on the right side of the diagram—with the reputations you want to generate—and work your way back to the inputs. Don’t worry about the calculations at first; just draw a process box with the name of the reputation inside and a short note on the general nature of the formulation, such as aggregated acting average or community player rank. Once you’ve drawn the boxes, connect them with arrows where appropriate. Then consider what inputs go into which boxes, and don’t forget that the arrows can split and merge as needed. Then, after you have a good rough diagram, start to dive into the details with your development team. Many mathematical and performance-related details will affect your reputation model design. We’ve found that reputation systems diagrams make excellent requirements documentation and make it easier to generate the technical specification, while also making the overall design accessible to nonengineers. Of course, your application will consist of displaying or using the reputations you’ve diagrammed (Chapters 7 and 8). Project engineers, architects, and operational team members may want to review Chapter 9 first, as it completes the schedule focused, development-cycle view of any reputation project. Draw Your Diagram | 163 CHAPTER 7 Displaying Reputation In Chapter 6, we described how to create a custom reputation model by identifying the objects in your application, selecting appropriate inputs, and developing the processes you’ll need to generate your reputations. But your work doesn’t end there. Far from it. Now you have decisions to make about how to use the reputations that your system is tabulating. In this chapter and the next, we discuss the many options for using reputation to im- prove the user experience of your site, enrich content quality, and provide incentives for your users to become better, more active participants. In this chapter specifically, we discuss options for displaying reputation, to whom to display it, how to display it, and help you decide which display forms are right for your application. How to Use a Reputation: Three Questions For each reputation you are creating to display or use, you should ask each of these questions before proceeding: 1. Who will be able to see the reputation? • Is it personal—hidden from other users but visible to the reputation holder? • Is it public—displayed to friends or strangers, or visible to search engines? • Is it corporate—limited to internal use—for improving the site or discreetly rec- ognizing outliers in ways that may not be visible to the community? 2. How will the reputation be used to modify your site’s output? • Will you use the reputation to filter the lowest- or highest-quality items in a set? • Will you use the reputation to sort or rank items? • And/or will this score be used to make other decisions about how the site flows or your business operates? 3. Is this reputation for a content item or a person? Each requires a fundamentally different approach. 165 Though you may choose multiple answers from this list for each reputation, try to keep it simple at first: don’t try to do too much with a single reputation. Confounding the purposes of a reputation—by, for example, surfacing participation points in a public karma score—can encourage undesirable user behavior and may even backfire by dis- couraging participation. Read Chapters 7 and 8 completely for a solid understanding of the issues related to overloading a single reputation. Resist the temptation to treat a single reputation score as the cure-all for your user-generated content incentive ills. Remember the lesson of the FICO score in “FICO: A Study in Global Reputation and Its Chal- lenges” on page 10. Who Will See a Reputation? So far, the reputation you’re calculating is little more than a cold numerical score rolled up from the aggregate actions of people interacting with your site. You’ve carefully determined the scope of the reputation, chosen the inputs that contribute to it, and thought at length about the effect that you want the reputation to generate in the community. Now you must decide whether it makes sense to display the reputation on your site at all and, if so, to whom. How you display reputation information—how much and how prominently—will influence the actions that users take on your site, their trust in your site and one another, and their long-term satisfaction with your community. To Show or Not to Show? Compelling reasons exist to keep reputations hidden from users. In fact, in some cir- cumstances, you may want to obscure the fact that you’re tracking them at all. It may sound rather Machiavellian, but the truth of the matter is this: a community under public scrutiny behaves differently (and, in many ways, less honestly) than one in bliss- ful ignorance. Several trade-offs are involved. Displaying reputations takes up significant page real estate, requires user interface design and testing, and can compete with your content for the user’s attention and understanding. Quickly, show Digg.com (Figure 7-1) to 10 of your friends and ask them, “What kind of site is this? News? Entertainment? Com- munity?” Odds are good that at least a few of them will answer: “This appears to be some sort of contest.” The impression that Digg makes is not a bad thing; it just demonstrates that Digg made a conscious decision to display content reputation prominently. In fact, the display of reputation is the central interaction mechanism on the site. It’s practically impossible to interact with Digg, or get any use out of it, without some understanding of how community voting affects the selection and display of popular items on the site. (Digg 166 | Chapter 7: Displaying Reputation is perhaps the most well-known example of a site that employs the Vote-to-Promote pattern. See Chapter 6.) Juxtapose Digg’s approach with that of Flickr. The popular photo-sharing and discov- ery service also makes use of reputation to surface quality content, but it does not display explicit reputations, rather it prominently displays items that achieve a certain reputation and that can be browsed (daily, weekly, or monthly) in the “Explore” gallery (at http://www.flickr.com/explore); see Figure 7-2. The result is a very consistent and impressive display of high-quality photos with very little indication of how those photos were selected. Flickr’s interestingness algorithm determines which photos make it into the “Explore” gallery and which don’t. The same algorithm lets users sort their own photos by interestingness. Figure 7-1. Digg’s site design puts overt reputation scores front and center. Who Will See a Reputation? | 167 Digg and Flickr represent two very different approaches to reputation display, but the results are very much the same. Theoretically, you can always glance at the front page of Digg or Flickr’s “Explore” gallery to see where the good stuff is—what people are watching, commenting on, or interacting with the most on the site. How do you decide whether to display reputations on your site? And how prominently? Generally, follow the rule of least disclosure: do not display a reputation that doesn’t add specific value to the objects being evaluated. Figure 7-2. Flickr’s “Explore” gallery is also based on reputation, but you never see a score associated with a photo. 168 | Chapter 7: Displaying Reputation Likewise, don’t bother asking users for reputation input (see Chapter 6) that you’ll never use; you’ll confuse users and encourage undesired patterns of “invented signifi- cance,” including abuse. Avoid collecting reputation for display only. Orkut allowed users to rate other users explicitly on iconic criteria like “trusty,” “cool,” and “sexy” for no use other than display. This use of reputation caused all kinds of social backlash. People were either disappointed that they weren’t rated “cool” by more people, or they were creeped out by people of the same gender calling them sexy. Eventually, Orkut removed the display of individual friends’ ratings and kept only the aggregate scores. Irrelevant reputations are meaningless and consume valuable resources. If you don’t have a relevant use for a reputation, beware of sticking yourself later with the tough choice of either awkwardly removing a failed feature or having to support it as a costly legacy element. Personal Reputations: For the Owner’s Eyes Only Are you tracking a reputation primarily to keep users informed about how well they or their creations are performing in the community? Consider displaying that reputation only to its owner, as a personal communication between site and user. Personal Reputation Is Not Private We use the word personal very deliberately here, distinguishing it from private. No reputation system is truly private; at least one other party (typically the site operator) will almost always have access to the actions, inputs, and roll-ups that formulate a user’s score. In fact, you may store internally used reputations (see “Corporate Reputations Are Internal Use Only: Keep Them Hush-hush” on page 172) that are largely based on the exact same data. In other words, reputations may be displayed in a personal context, but that’s no guar- antee that they’re private. As a service provider, you should acknowledge that distinc- tion and account for it in your terms of service. Personal reputations are used extensively for applications such as social bookmarking, lists of favorites, training recommendation systems, sorting and filtering news feeds, providing content quality and feedback, fine-grained experience point tracking, and other performance metrics. Most of the same user interface patterns for displaying public reputation apply to personal ones, too, but take care to ensure that each user knows when her reputations will and will not be displayed to others. Who Will See a Reputation? | 169 Keep a reputation personal when its owner gains some significant ben- efit from it—when it either improves his experience of the site (that is, personalizes it) or provides a tool for increasing self-satisfaction. For example, by selecting news stories about various sports teams over time, a user might generate a geographic region reputation that can be used to target advertising displayed to the user. Clearly that reputation should not be public information, but it might be surfaced privately so that the user can correct it—“I’m a fan of Northern California sports teams, but I’m going to MIT and I really want ads for electronics stores in the Bos- ton area.” Google Analytics (see Figure 7-3) is an example of rich personal reputation information. It provides detailed information about the performance of your website, across a known range of score types, and it is available only to you, the site owner (or others to whom you grant access). While that information is invaluable to you in gauging the response of a community (in this case, the entire Web) to your content, exposing it to everyone would offer very little practical benefit. In fact, it would be a horrible idea. Figure 7-3. Google’s Analytics interface shows information that is clearly best kept between you and Google. It’s personal. 170 | Chapter 7: Displaying Reputation [...]... with one or more reputations attached to it All kinds of reputation score types and all kinds of displays and use patterns might seem equally valid for content reputation and karma, but usually they’re not To highlight the differences between content reputation and karma, we’ve categorized them by the ways in which they’re typically calculated: Simple reputation Simple reputation is any reputation score... potential as a dating hangout Who Will See a Reputation? | 171 Public reputations are helpful for allowing users to compare like items Public karma reputations also serve as an effective extension of a person’s identity Corporate Reputations Are Internal Use Only: Keep Them Hush-hush Almost every website with a large volume of user-generated content is using hidden reputation scores internally—as a means... example, simple reputation is used on most ratings-and-reviews sites Simple reputation is direct and easy to understand Complex reputation Complex reputation is a score aggregated from multiple evaluations, including evaluations of different but related targets, calculated with an opaque method Email IP spammer, Google PageRank, and eBay feedback reputations are examples of complex reputation It’s... focus on 172 | Chapter 7: Displaying Reputation filtering and decisions If you’re going to display a lot of public reputation over many entities, focus on ranking and sorting to help users explore your content We’ll cover patterns for making use of the reputation of entities in Chapter 8 Reputation Filtering At its simplest, filtering consists of sorting by one or more reputation dimensions and looking... actually is Online games usually report only the level to other users and the exact experience points to the player Public Reputations: Widely Visible When the whole community would benefit from knowing the reputations of either people or content, consider displaying public reputations Public reputations may be displayed to everyone in the community or only to users who are members of a group, are connected... Perhaps the easiest way to think about a filter reputation is that, if it is not ever displayed to users, they don’t have to understand what it’s made up of If users can see a reputation indicator, they’ll want to know what it means and how it’s calculated In fact, algorithm speculation has become almost a spectator sport on the Web Name any popular reputation- heavy site (Digg, Amazon, YouTube, and... successful your reputation ranking is, the more value it creates, and the more some people will want to game your design for their own benefit The lesson is a reputation- based display that may work well when a community is small may need to be modified over time, as it becomes more successful This is a success paradox: the more popular your reputation system becomes, the more likely you’ll see reputation. .. content item or another user: • When users click the Spam button in a webmail application, they contribute to a database of IP addresses for abusive mail servers • Web crawlers constantly scan the Web to examine what sites link to what other sites and to calculate a hidden score such as Google’s PageRank • Yahoo! Answers tracks corporate reputation for users who are particularly good at identifying bad... costs How Will You Use Reputation to Modify Your Site’s Output? After deciding which reputation scores to display to whom, you’ll need to decide how to use the scores to change the way your application works It’s easy to think that all you need to do is display a few stars here or a few points there—but if you stopped there, you wouldn’t capture the most value from reputation To use reputation without... restaurants, show today’s gallery of the most interesting photos, to display leaderboards of the top-scoring reputation targets, and much more Over time, public reputations can evolve to represent your community’s understanding of its own zeitgeist And there’s the rub: depending on how you use public reputation, you can alienate users who aren’t part of the in crowd For example, Yelp is all about public . displaying public reputation apply to personal ones, too, but take care to ensure that each user knows when her reputations will and will not be displayed to others. Who Will See a Reputation? | 169 Keep a reputation. player. Public Reputations: Widely Visible When the whole community would benefit from knowing the reputations of either people or content, consider displaying public reputations. Public reputations. a Reputation? | 171 Public reputations are helpful for allowing users to compare like items. Public karma reputations also serve as an effective extension of a per- son’s identity. Corporate Reputations