Đang tải... (xem toàn văn)
Explanation of the task When a user starts typing a query in a search engine, the system can often guess what they were trying to type and fill in the rest of the query, saving the user some work The[.]
Explanation of the task When a user starts typing a query in a search engine, the system can often guess what they were trying to type and fill in the rest of the query, saving the user some work The part of the query the user has typed so far is called the prefix The system’s guess about what the entire query would be if the user had finished typing it is called the search completion, or just completion for short You will be provided with a prefix along with the context information (such as geo-location and search date) and two lists of completions presented side by side After understanding the prefix, you will be asked to answer your preference as one side over the other as well as helping us flagging any issues you see for the completion(s) Important Concepts We will often use an arrow to indicate a prefix getting a particular completion For example, “[fac] → [facebook]” means the user typed the prefix “fac” and the system suggested the completion “facebook.” prefix — the part of a search query that the user has typed so far o A prefix will often end in the middle of a word or phrase o A prefix may also contain spelling and/or typographical errors o Example: A few possible prefixes you might see for users who intended to type the query "facebook" are [fac], [face], [fcaebo], [facebook] intended query — the entire query the user had in mind when they started typing o Sometimes most users typing the same prefix will have the same intended query, such as [fac] → [facebook] We call this a dominant intent o Other times different users will have different intended queries even though they’ve typed the same prefix For example, if the prefix is [wal], some users might have [walgreens] in mind, while others were thinking of [walmart] We call this a split intent completion — the system’s prediction about the intended query o Occasionally the completion will be the same as the prefix This means the system’s best guess is that the user was done typing the query e.g [instagram] → [instagram] matching — how much the text of the prefix overlaps with the text of the completion Although users typically prefer completions that start with and include the entire prefix (e.g [ama] → [amazon]), there are some special cases where a good completion may change the text, matching conceptually but not letter-forletter: o Abbreviations: A good completion may expand or collapse initials: [uc davis] → [university of california davis] o Corrections: A good completion may correct spelling or typographical errors: [fcaebook] → [facebook] o Special Character Equivalents: In some languages, a user may type a combination of letters instead of a single accented character A good completion may replace the combination with the correct character For example, in German, “oe” is another way to write the character ö, so a good completion in locale de-DE might be: [moe] → [möllersdorf] context — everything we know about the query besides the prefix itself, such as the date of the query, the user’s location when the query was issued, and so on When deciding how good a completion is, you’ll need to take the context info account o Example: The completion “[cafe in cam] → [cafe in campbell]“ might be good if the searcher is in San Jose, CA, but not if they are in Boston, MA (where ”cafe in cambridge“ would be a more likely completion) General Guideline For Likelihood Rating Before you begin the task, make sure your language and country settings align with the task’s designated locale Then, for each prefix and completion, you’ll the following: Try to determine the intended query for the prefix Mark any potential problems you see with the given prefix Decide how likely you think it is that the completion shown is the intended query.See the sections below for details about how to each step Determining the Intended Query To determine the intended query, you’ll need to get a sense of what some possible completions are Click on the links to the web search engines and see what some of their suggestions are This will give you a sense of some possible queries the user might have had in mind However, remember that the user may have been thinking of query that is not shown Remember to consider the context when thinking about the intended query In particular: Part of the context is the user’s location Think about how that might make a certain completion more or less likely Part of the context is the date of the query Consider what happened on or just before that date If there was a major news event, that might change which completion is most likely Identifying Completion Problems Indicate whether the completion has any of the following problems by checking as many choices as apply: Spelling Errors: Spelling, typographical, or grammatical errors Inappropriate: Inappropriate language, including profanity, explicit sexual references, and hate speech Wrong language: Completion is in the wrong language, i.e a language that is not the locale language, AND not the name of an entity (person, place, title, etc.) that the user was looking for Incomplete: Completion ends in the middle of a word, phrase, or name Intention too specific: Overly specific completion for broad prefix Gibberish: Completion is gibberish Deciding the Likelihood of a Completion Grade each completion using the following scale, which answers the question “How likely you think it is that a user who typed this prefix had this completion in mind?” (Or, in some cases where different users may have different intents, “How likely you think it is that many users who typed this prefix had this completion in mind?”) Highly Likely: It means search completion almost fully satisfies the user's typing intent for the given prefix Usually a completion is highly likely with respect to a prefix when there are no other completions you can think about or the it is the dominant intent e.g [ama] > [amazon], [fac] > [facebook], [earring] > [earrings], [google.c] >[google.com], [Vans] > [Vans shoes] Likely: It means the search completion satisfies partial intent of the given prefix but such prefix probably does not has a best possibility Usually a completion is likely with repsect to a prefix when there are only very few other completions you can think about at first glance e.g [wal] > [walmart],[walgreen], [fetal alcohol] > [fetal alchohol syndrome], [fetal alchohol definition] Somewhat Likely: The search completion matches with the prefix However, the semantic meaning of the completion deviates too much from prefix, which makes it only satisfy the search intent for a few people Usually a completion is somewhat likely with repsect to a prefix when there are more than a few other completions you can think of e.g [in] > [instgram], [indeed], [internet], [india], [integer], [how old is o] > [how old is obama],[how old is oprah], [how old is olivia], [how old is ozzy], [how old is oj], [1122] > [1122 Brook St.], [1122 Ocean Rd], [1122 angel number], [1122 Certral Ave] Unlikely: Generally if the prefix is very broad, while the search completion is too specific; e.g [thu] > [thursday night football] (too specific); [dese] > [deseandate frankie ruiz lyrics]); [what does it mean w] >[what does it mean when you left eye jumps], [what]->[what is dogecoin] Highly Unlikely: If the search completion is gibberish, incomplete, has severe spelling error (missing space in website domain name, profane, wrong language) e.g [how to] > [how what] (non sense completion), [esdfdofdfowngf] > [esdfdofdfowngf] (gibberish), [fucking big a] > [fucking big a] (inappropriate), [top 30 college coahce] > [top 30 college coahce] (spelling error), [bridge health home and ] > [bridge health home and ] (Incomplete), [fel] > [felicidad] (happiness in Spanish) [Wrong language], [gl] > [glück] (happiness in German) [Wrong language] How to Handle Errors Remember that a completion is trying to predict what the user intended their search query to be Because people don’t intend to make spelling/typographical/grammatical errors, a completion containing an error is Highly Unlikely to be what the user wanted However, there are a few exceptions where an error is so minor that you not need to grade it Highly Unlikely Instead, simply assign a grade one level lower than you would have if the error were not present These exceptions are: When a word has multiple valid spellings, even though only one is technically correct in the given context o Example: [huntington theater c] → [huntington theater company] The actual name is “Huntington Theatre Company,” but since “theater” and “theatre” are both valid spellings of the word in US English, this counts as only a minor error When a word has regional/national variations in spelling across countries sharing that language o Example: [driver’s lic] → [driver’s licence] This completion uses a British/Canadian spelling of the word, instead of the American “license.” When the completion and the intended query differ only by a grammatical article like “the” or “a” o Example: [andy weir ma] → [andy weir martian] The actual name of the book is “The Martian,” but this completion left out “the.” When the completion and the intended query differ only by punctuation o Example: [olympics womens s] → [olympics womens soccer] The correct punctuation would be “women’s”.[Note: All the above examples assume locale is en-US.] Nonstandard/Dialect Usage A deliberate nonstandard, informal, slang or dialect usage is not an error Mistakenly “correcting” such usage is an error Example: [50 cent in d] → [50 cent in da club] o A valid completion uses the intended song title, even though “da” is not standard spelling/usage Task Starts! The search prefix is [target deodant pack womens] The search was completed in au/tasmania/launceston pt_b; and the search date is 2022-01-28 Please use the checkbox to select all issue(s) that apply for each suggestion (if any of issues applies, the intent rating for the auto-completion should not be better than "Unlikely") Left Only in this list Prefix: [target deodant pack womens] > Search Completion: [target deodant pack womens] Inappropriate Wrong Language Too specific Incomplete Gibberish Spelling error How likely would the auto-completion satisfy the intent of the prefix? (Rule of thumb is the more the meaning of auto-completion deviates from the given prefix, the less likely the auto-completion would satisfy the prefix's intent ) N/AHighly UnlikelyUnlikelySomewhat LikelyLikelyHighly Likely First use following instrustions to rate the likelihood of each side separately: Option 1: If the completion is obviously (Highly) Likely (you think there is a dominant intent for the prefix) or (Highly) Unlikely (reasons can be too specific, incomplete, non-sense, spelling errors, gibberish, inappropriate, wrong language) to you > ignore Option 2/3 and give a likelihood rating accordingly Option 2: If you are not sure about the answer > please click here: 'Third party search link' to further look into this Once landed, click on the search box, type or copy the prefix into the box (DON'T click search button or press 'enter') and you should see multiple suggestions in the dropdown list If all of suggestions contain the completion, it's almost certain that the completion is 'Highly Likely' If at least half of suggestions contain the completion, it's safe to rate the completion as 'Likely' If less than half of suggestions contain the completion, the completion is 'Somewhat Likely' If no suggestions contain the completion, the completion should be 'Unlikely' If there are applicable issue flags other than 'Too specific', the completion should 'Highly Unlikely' Option (Exception): If you don't think Option 1-2 apply to the case, use your best judgement and explain your thought in the comment section Second compare the difference in ratings that you gave for both sides and give a side-by-sde preference rating: SBS rating scale instructions: When the degree of likelihood from one side is two degree higher then "Much better" When the degree of likelihood from one side is one degree higher then "Better" When the degree of likelihood from one side fall into the same bucket, most of the time, you can choose "About the same" However, even if they are in the same bucket of likelihood, but you still have your preference, then "Slightly better" When one side is non-completion (empty), use the following rule: Non-completion side is 'Slightly Better' than a 'Somewhat Likely' completion side Non-completion side is 'Better' than an 'Unlikely' completion side Non-completion side is 'Much better' than a 'Highly Unlikely' completion side A 'Likely' completion is 'Better' than non-completion side A 'Highly Likely' completion is 'Much Better' than non-completion side Considering the overall search experience, which side would you prefer? Much better Better Slightly better About the same Slightly better Better Much better