ºÝºÝߣ

Editor's Notes

• #3: ? A standard practice of evaluating IR effectiveness is¡­? Relevance labels are subjective, leading to assessor disagreements¡­? Recently, preference judging... ¡­more natural user task ¡­higher inter-assessor agreement levels ¡­increased measurement sensitivity. ¡­desirable with the increasing adoption of crowdsourcing? Various reports on assessor disagreement, but little work has been done on characterizing disagreement, e.g.:? Why preference judging reduces assessor disagreement? ? Does better agreement among assessors also means better agreement with user satisfaction, ¡­user clicks? ? What is the role of user intent in judging behavior? E.g., judges may agree better when rating pairs of documents that satisfy more similar or more diverse intents.
• #5: Experiment setup¡­examine the relationship between assessor disagreement and click based measures: click preference strength and user intent similarityfor judgments collected from editorial judges and crowd workers using absolute and preference based methodsMeasuring both inter-assessor and user-assessor agreementNote that user-assessor agreement is referred to as click-agreement in the paper.
• #6: Click preference is defined as the proportion of times web users prefer one search result, URL u, over another, URL v, for a given query, by solely clicking on u even though both results are observed. In order to identify the cases where both u and v are observed by the user, we focus only on the cases where u and v are presented in consecutiverank positions (regardless of the order in which they are presented) and where at least one result (u or v) is clicked by the user. We then compute the click preference strength of u over v, (Puv), as the proportion of times only u is clicked by the user minus the proportion of times only v is clicked. We use c?uv to denote the number of times when the two results were shown with u immediately above v (e.g., u at rank 2 and v at rank 3), where u was clicked and v was not clicked. Similarly, let cu?v be the number of times v was clicked and u was not, and c?u?v be the number of times when both results were clicked.Intent similarity is measured by 3 different metrics in the paper. Here we only present the dupe score.
• #7: Data:We use three months of click logs (Sept - Nov 2011) and select pairs of URLs that were shown to users in adjacent rank positions, where both orderings appeared in different impressions, e.g., u ranked above v in some impressions and v ranked above u in others, and where at least one of the two results was clicked.Our final sample set consists of 1,068 (q, u, v) tuples with 830 unique queries, 1,757 unique URLs and 1,915 unique query-URL pairs.¡­pairwise judging experiments: randomly swap the two URLs¡­single judging task, we separate the 1,068 pairs of URLs in our sample set into 1,915 unique query-URL pairs¡­ We collect relevance labels using three different HIT designs¡­ sequential pairwise is not shown.
• #10: ³§±ð±ô´Ú-±ð³æ±è±ô²¹²Ô²¹³Ù´Ç°ù²â¡­
• #13: X = click pref scoreY=kappaTrend lines are the message.. They show whether judges agree better with each other as click-preference strength increases, i.e., when there is stronger user preference for one URLCrowd workers do not show any relationshipFor trained judges, we see that they agree with each other more when users have stronger preference for one URL, especially for pairs of URLs that have a lot of traffic
• #14: All judges agree better with user signals with increasing click preference scorePairwise UI helps to boost judge-user agreement and the relationship with preference strength is strongerTrained judges demonstrate a stronger relationship, especially for high traffic URLs
• #16: This is just dup score, see paper for other intent similarity scores¡­Weak relationships, barely positive trend, except pairwise-crowd (PC)No clear relationship for crowd, they do not agree with each other any more when judging URLs that are near dupes of each other or URLs that are not-dupes
• #17: The dupe score highlights the difference between the single and pairwise judging methods: in singleUI-crowd (SC), the more interchangeable the pairs of web pages are, the more likely that crowd workers disagree with user clicks, while this trend flips for the pairwise UI.the pairwise UI exposes properties of the web pages that can then improve judging quality when faced with more interchangeable documents, leading to better agreement with web users (even if not with other assessors) ¨C see prev slide.
• #18: Sampling: url pairs that were both shown to users next to each other in the ranking -> for informational these will be close calls, since ranked close