Building Web Reputation Systems- P4 pptx

Molecules: Constructing Reputation Models Using Messages and Processes Just as molecules are often made up of many different atoms in various combinations to produce materials with unique and valuable qualities, what makes reputation models so powerful is that they aggregate reputation statements from many sources and often statements of different types. Instead of concerning ourselves with valence and Van der Waals forces, in reputation models we bind the atomic units—the reputation statements—together with messages and processes. In the simple reputation model presented in Figure 2-3, messages are represented by arrows and flow in the direction indicated. The boxes are the processes and contain descriptions of the processes that interpret the activating message to update a reputa- tion statement and/or send one or more messages onto other processes. As in chemistry, the entire process is simultaneous; messages may come in at any time, and multiple messages may take different paths through a complex reputation model at the same time. People often become confused about the limited scope of reputation, and where to draw the lines between multiple reputations, so we need a couple of definitions: Reputation model A reputation model describes all of the reputation statements, events, and pro- cesses for a particular context. Usually a model focuses on a single type of reputable entity. Yahoo! Local, Travel, Movies, TV, etc. are all examples of ratings-and-reviews reputation models. eBay’s Seller Feedback model, in which users’ ratings of trans- actions are reflected in sellers’ profiles, is a karma reputation model. The example in Figure 2-3 is one of the simplest possible models and was inspired by the Digg it vote-to-promote reputation model (see Chapter 6) made popular by Digg.com. Reputation Context A reputation context is the relevant category for a specific reputation. By definition, the reputation’s reuse is limited to related contexts. A high ranking for a user of Yahoo! Chess doesn’t really tell you whether you should buy something from that user on eBay, but it might tell you something about how committed the user is to board gaming tournaments. See the sections “Reputation Takes Place Within a Context” on page 4 and “FICO: A Study in Global Reputation and Its Chal- lenges” on page 10 for a deeper consideration of the limiting effects of context. Messages and Processes Again, look at the simplest reputation model diagram shown in Figure 2-3. The input reputation statement appears on the left and is delivered as a message to the reputation 26 | Chapter 2: A (Graphical) Grammar for Reputation process box. Messages and processes make up the working mechanics of the reputation model: Reputation message Reputation messages, represented by the flow lines in reputation model diagrams, are information supplied to a reputation process for some sort of computational action. In this example, the input is a reputation statement delivered as a message to the Raw Sum of Votes process. These messages may come from other processes, explicit user action, or external autonomous software. Don’t confuse the party sending the messages with the reputation source; they are usually unrelated. For example, even when the source of a reputation input is a user, the message usually comes from an application—allowing the reputation system to take different ac- tions based on the sender’s identity. Input event (reputation message) In this book, we call initial messages—those that start the execution flow of a reputation model—input events; we show them at the start of a model diagram. Input events are said to be transient when the reputation message does not need to be undone or referenced in the future. Transient input events are not stored. If, on the other hand, an event may need to be displayed or reversed in the future (e.g., if a user abuses the reputation model), it is said to be reversible and must be stored either in an external file such as a log or as a reputation statement. Most rating-and-review models have reversible inputs. But for very large-scale systems, such as IP address reputations that identify mail spammers, it’s too costly to store a separate input event for every email received. For those reputation models, the transient input method is appropriate. Reputation process The large boxes represent one or more reputation processes. Using the message parameters, these processes normalize, transform, store, decide how to route new messages, or, most often, calculate a value. Although this book describes several common process types, the logic in these processes is usually customized to the Figure 2-3. Users endorse articles, and the sum of their votes is displayed by that article. Molecules: Constructing Reputation Models Using Messages and Processes | 27 application’s requirements for more engineering-level detail about a specific im- plementation example, see Appendix A. Stored Reputation Value A stored reputation value is simply a reputation statement that may be read as part of a reputation process that calculates a new claim value for itself or another rep- utation statement. Many reputation processes use message input to transform a reputation statement. In our example in Figure 2-3, when a user clicks “I Digg this URL,” the application sends the input event to a reputation process that is a simple counter: CountOfVotes. This counter is a stored reputation value that is read for its current value, then incremented by one, and then is stored again. This brings the reputation database up to date and the application may use the target identifier (in Digg’s case, a URL) to get the claim value. Roll-ups A roll-up is a specific kind of stored reputation value—any aggregated reputation score that incorporates multiple inputs over time or from multiple processes. Sim- ple average and simple accumulator are examples of roll-ups. Reputation Model Explained: Vote to Promote Figure 2-3 is the first of many reputation model diagrams in this book. We follow each diagram with a detailed explanation. This model is the simple accumulator: It counts votes for a target object. It can count clickthroughs or thumbs-ups, or it can mark an item as a favorite. Even though most models allow multiple input messages, for clarity we’re presenting a simplified model that has only one, in the form of a single reputation statement: • As users take actions, they cause votes to be recorded and start the reputation model by sending them as a messages (represented by the arrows) to the Raw Sum of Votes process. Likewise, whereas models typically have many processes, this example has only one: • Raw Sum of Votes: When vote messages arrive, the CountOfVotes counter is incre- mented and stored in a reputation statement of the claim type Counter, set to the value of CountOfVotes and with the same target as the originating vote. The source of this statement is said to be aggregate because it is a roll-up—the product of many inputs from many different sources. See Chapter 3 for a detailed list of common reputation process patterns, and see Chap- ters 6 and 7 for a discussion of the effects of various score classes on a user interface. 28 | Chapter 2: A (Graphical) Grammar for Reputation Building on the Simplest Model Figure 2-4 shows a fuller representation of a Digg.com-like vote-to-promote reputation model. This example adds a new element to determining community interest in an article: adding a reputation for the level of user activity, measured by comments left on the target entity. These two are weighted and combined to produce a rating. Figure 2-4. Now articles are ranked not only according to endorsements, but also the amount of discussion they generate. The input messages take the form of two reputation statements: • When a user endorses an article, the thumbs-up vote is represented as a 1.0 and sent as a message to the Raw Sum of Votes process. If a previous thumbs-up vote is withdrawn, a score of 0.0 is sent instead. Molecules: Constructing Reputation Models Using Messages and Processes | 29 • When a user comments on an article, it is counted as activity and represented as a 1.0 and sent as a message to the Level of Activity process. If the comment is later deleted by the user, a score of 0.0 is sent to undo the earlier vote. This model involves the following reputation processes: Raw Sum of Votes This process either increments (if the input is 1.0) or decrements a roll-up reputa- tion statement containing a simple accumulator called CountOfVotes. The process stores the new value and sends it in a message to another process, Community Interest Rank. Level of Activity This process either increments (if the input is 1.0) or decrements a roll-up reputa- tion statement containing a simple accumulator called ActivityLevel. It stores the new value back into the statement and sends it in a message to another process, Community Interest Rank. Community Interest Rank This process always recalculates a roll-up reputation statement containing a weigh- ted sum called interest, which is the value that the application uses to rank the target article in search results and in other page displays. The calculation uses a local constant—Weighting—to combine the values of CountOfVotes and Activity Level scores disproportionately; in this example, an endorsement is worth 10 times the interest score of a single comment. The resulting Interest score is stored in a typical roll-up reputation statement: aggregate source, numeric score, and target shared by all of the inputs. Complex Behavior: Containers and Reputation Statements As Targets Just as there exist some interesting-looking molecules in nature, and much like hydro- gen bonds are especially strong, certain types of reputation statements called contain- ers join multiple closely related statements into one super-statement. Most websites with user-generated ratings and comments for products or services provide examples of this kind of reputation statement: they clump together different star ratings with a text comment into an object formally called a review. See Figure 2-5 for a typical ex- ample, restaurant reviews. Containers are useful devices for determining the order of reputation statements. Al- though it’s technically true that each individual component of a container could be represented and addressed as a statement of its own, that arrangement would be se- mantically sloppy and lead to unnecessary complexity in your model. The container model maps well to real life. For example, you probably wouldn’t think of the series of statements about Dessert Hut made by user Zary22 as a rapid-fire stream of individual 30 | Chapter 2: A (Graphical) Grammar for Reputation opinions; you’d more likely consider them related and influenced by one another. Taken as a whole, the statements express Zary22’s experience at Dessert Hut. A container is a compound reputation statement with multiple claims for the same source and target. Once a reputation statement exists in your system, consider how you might make it a reputable entity itself, as in Figure 2-6. This indirection provides for subtle yet powerful feedback. For example, people regularly form their own opinions about the opinions of others based on some external criteria or context. (“Jack hated The Dark Knight, but he and I never see eye to eye anyway.”) Another feature of review-based reputation systems is that they often incorporate a form of built-in user feedback about reviews written by other users. We’ll call this feature the Was This Helpful? pattern. (See Figure 2-7.) When a user indicates whether a review was helpful, the target is a review (container) written earlier by a different user. Figure 2-6. A reputation statement can itself be the target of a helpful vote. Here, MuvyLuvr has written a review that others can then rate. Figure 2-5. A reputation container: some claims make more sense when considered together than when standing alone. Complex Behavior: Containers and Reputation Statements As Targets | 31 Figure 2-7. Helpful votes for a review are rolled up into an aggregate HelpfulScore. It’s often more efficient to just store this score back to the review container for easy retrieval. The input message takes the form of a single reputation statement: • A user votes on the quality of another reputation statement, in this case a review: a thumbs-up vote is represented by a 1.0 value, and a thumbs-down vote by a 0.0 value. This model includes only one reputation process: • Calculate Helpful Score: When the message arrives, load the TotalVotes stored reputation value, increment, and store it. If the vote is not zero, the process incre- ments HelpfulVotes. Finally, set the HelpfulScore to a text representation of the score suitable for display: “HelpfulVotes out of TotalVotes.” This representation is usually stored in the very same review container that the voter was judging (i.e., had targeted) as helpful. This configuration simplifies indexing and retrieval, e.g., “Retrieve a list of the most helpful movie reviews by MuvyLuvr” and “Sort the list of movie reviews of Aliens by helpful score.” Though the original review writer isn’t the author of his helpful votes, his review is responsible for them and should con- tain them. 32 | Chapter 2: A (Graphical) Grammar for Reputation You’ll see variations on this simple pattern of reputation-statements-as-targets repeated throughout this book. It makes it possible to build fairly advanced meta-moderation capabilities into a reputation system. Not only can you ask a community “What’s good?”; you can also ask “…and whom do you believe?” Solutions: Mixing Models to Make Systems In one more invocation of our chemistry metaphor, consider that physical materials are rarely made of a single type of molecule. Chemists combine molecules into solu- tions, compounds, and mixtures to get the exact properties they want. But not all sub- stances mix well—oil and water, for example. The same is true in combining multiple reputation model contexts in a single reputation system. It’s important to combine only those models with compatible reputation contexts. Reputation system A reputation system is a set of one or more interacting reputation models. An ex- ample is SlashDot.com, which combines two reputation models: an entity repu- tation model of users’ evaluations of individual message board postings and a karma reputation model for determining the number of moderation opportunities granted to users to evaluate posts. It is a “the best users have the most control” reputation system. See Figure 2-8. Reputation framework The reputation framework is the execution environment for one or more reputation systems. It handles message routing, storing, and retrieving statements, and maintaining and executing the model processes. Appendix A describes an imple- mentation of a reputation framework. Figure 2-8 shows a simple abuse reporting system that integrates two different repu- tation models in a single weighted voting model that takes weights for the IP addresses of the abuse reporters from an external karma system. This example also illustrates an explicit output, common in many implementations. In this example, the output is an event sent to the application environment suggesting that the target comment be dealt with. In this case, the content would need to be reviewed and either hidden or de- stroyed. Many services also consider punitive action against the content creator’s ac- count, such as suspension of access. For the reporter trustworthiness context, the inputs and mechanism of this external reputation model are opaque—we don’t know how the model works—because it is on a foreign service, namely TrustedSource.org by McAfee, Inc. Its service provides us one input, and it is different from previous examples: 1. When a reputation system is prompted to request a new trust score for a particular IP address—perhaps by a periodic timer or on demand by external means—it re- trieves the TrustedSourceReputation as input using the web service API, represented here as a URL. The result is one of the following categories: Trusted, Solutions: Mixing Models to Make Systems | 33 Neutral, Unverified, Suspicious, or Malicious, which the system passes to the McAfee TrustedSource Web Reputation process. This message arrives at a reputation process that transforms the external IP reputation, in a published API format, into the reputation system’s normalized range: 2. McAfee TrustedSource Web Reputation: Using a transformation table, the system normalizes the TrustedSourceReputation into WebReputation with a range from 0.0 (no trust) to 1.0 (maximum trust). The system stores the normalized score in a reputation statement with a source of TrustedSource.org, claim type simple karma, with the claim value equal to the normalized WebReputation, and a target of the IP address being evaluated. The main context of this reputation is content quality, which is designed to collect flags from users whenever they think that the comment in question violates the site’s terms of service. When enough users whose web providers have a high enough Figure 2-8. Two or more separate models can work in symphony to create a larger, more robust reputation system. 34 | Chapter 2: A (Graphical) Grammar for Reputation reputation flag the content, the reputation system sends out a special event. This rep- utation model is a weighted voting model. This model has one input: 3. A user, connected using a specific IP address, flags a target comment as violating the site’s terms of service. The value of the flag is always 1.0 and is sent to the Abusive Content Score process. This model involves two processes—one to accumulate the total abuse score, and an- other to decide when to alert the outer application: 4. The Abusive Content Score process uses one external variable: WebReputation. This variable is stored in a reputation statement with the same target IP address as was provided with the flag input message. The AbuseScore starts at 0 and is increased by the value of Flag multiplied by WebReputation. The system stores the score in a reputation statement with an aggregate source, numeric score type, and the com- ment identifier as the target, then passes the statement in a message to the Is Abu- sive? process. 5. Is Abusive? then tests the AbuseScore against an internal constant, called AbuseThreshold, in order to determine whether to highlight the target comment for special attention by the application. In a simple (request-reply) framework imple- mentation, this reputation system returns the result of the Is Abusive? process as TRUE or FALSE to indicate whether the comment is considered to be abusive. For high-performance asynchronous (fire-and-forget) reputation platforms such as the one described in Appendix A, an alert is triggered only when the result is TRUE. From Reputation Grammar to… This chapter defined the graphical reputation grammar, from bottom (reputation state- ments made of sources, claims, and targets) to top (models, systems, and frameworks). All members of any team defining a reputation-enhanced application should find this reference material helpful in understanding their reputation systems. But at this point, readers with different team functional roles might want to look at different parts of the book next: Product managers, application designers Chapter 3 expands the grammar with a comprehensive review of common source, claim, target, and process types that serve as the basis of most reputation models. Think of these as the building blocks that you would start with when thinking about designing your customized reputation model. Solutions: Mixing Models to Make Systems | 35 [...]... “Practitioner’s Tips: Reputation Is Tricky” on page 57 for some detailed insights before skipping ahead to Chapter 4, which provides a detailed look at common deployed reputation systems 36 | Chapter 2: A (Graphical) Grammar for Reputation PART II Extended Elements and Applied Examples CHAPTER 3 Building Blocks and Reputation Tips By now you should feel fairly conversant in the lingua franca of reputation systems... Functionally, different types of roll-ups yield very different types of reputations, so we offer guidance on when to use which roll-ups We end the chapter with practical advice in a section of “practitioner’s tricks.” Extending the Grammar: Building Blocks Though understanding the elements of the reputation grammar is essential, building reputation models from the atoms up every time is time consuming and.. .Reputation framework engineers, software architects If you’re interested in technical details and want to know more about implementing reputation systems and dealing with issues such as reliability, reversibility, and scale, look at Appendix A for a detailed discussion of reputation frameworks before proceeding Reputation enthusiasts If you’re considering... to developing tools and shorthand for common patterns and using those to configure reputation statements and templates for well-understood reputation models The Data: Claim Types Remember, a fundamental component of a reputation statement is the claim—the assertion of quality that a source makes about a target In “The Reputation Statement” on page 6, we discussed how claims can be either explicit (a... you’ve had some exposure to their constituent bits and pieces We’ve gone over reputation statements, messages, and processes, and you’ve become familiar with some rudimentary but serviceable models In this chapter, we “level up” and explore reputation claims in greater detail We describe a taxonomy of claim types, exploring reputation roll-ups: actual computations on incoming messages that affect a particular... some simple cumulative score (or “aggregator”—we discuss the various reputation roll-ups later in “Processes: Computing Reputation on page 46) So while we can’t necessarily assign an evaluative score to a user-contributed text comment, for instance (at least not without the rest of the community involved), it’s quite common on the Web to see a count of the number of comments left about an entity, as... quantitative claims, which are claims that can be measured (and, in fact, are largely generated, communicated, and read back as numbers of some kind) Reputation statements have claim values, which you can generally think of as “what you get back when you ask for a reputation s current state.” So, for instance, we can always query the system for Movies.Review.Overall.Average and get back a normalized score... feature of the user-generated content on the Web Though a debate rages about the value of such comments (which in any case differs from site to site and from community to community), no one denies that the ability to leave text comments about an item of content—whether it’s a blog entry, an article, or a YouTube video—is a wildly popular form of expression on the Web Text comment fields typically are provided... fill in any way they choose However, better social sites will attempt to direct comments by providing guidelines or suggestions about what may be considered on- or off-topic 40 | Chapter 3: Building Blocks and Reputation Tips ... concrete actions associated with a target entity) These fundamentally different approaches are important because the combination of implicit and explicit claims can yield some very nuanced and robust reputation models In other words, we should pay attention to what people say, but we should give equal weight to what they do to determine which entities hold the community’s interest 39 Claims also can . vote-to-promote reputation model (see Chapter 6) made popular by Digg.com. Reputation Context A reputation context is the relevant category for a specific reputation. By definition, the reputation s. working mechanics of the reputation model: Reputation message Reputation messages, represented by the flow lines in reputation model diagrams, are information supplied to a reputation process for. combining multiple reputation model contexts in a single reputation system. It’s important to combine only those models with compatible reputation contexts. Reputation system A reputation system