News Article Ranking : Leveraging the Wisdom of Bloggers
Download as PPTX, PDF
1 like338 views
Paper presented at RIAO 2010 by Richard McCreadie entitled 'News Article Ranking: Leveraging the Wisdom of Bloggers'
News Article Ranking : Leveraging the Wisdom of Bloggers
- 1. News Article Ranking:Leveraging the Wisdom of BloggersRichard McCreadie, Craig Macdonald & IadhOunis
- 2. IntroductionBackground:Bloggers react to news as it happens
- 3. Thelwall explored how bloggers reacted to the London bombings
- 4. 30% of bloggersblog on news-related topics (Technorati poll 2008)
- 5. Hence, the blogosphere is valuable as a source of news-related information
- 6. Kniget al. & Sayyadiet al. have exploited the blogosphere for event detectionObama VictoryNumber of blog postsDay (November 2008)M. Thelwall WWW06Knig et al. SIGIR09Sayyadi et al. ICWSM09
- 8. Every day newspaper editors select articles for placement within their newspapers.
- 9. This can be seen as a ranking problem.
- 10. Rank articles by readership interestFrontPagePage2NewspaperEditor . . .We investigate how such a ranking can be approximated using evidence from the blogosphere
- 11. Introduction
- 12. The News Article Ranking Problem
- 14. Evaluating Votes
- 18. News Article RankingProblem Definition:Rank news articles by their inherent importance.
- 19. Given a day of interest dQ we wish to score each news article a by its predicted importance, score(a,dQ) using evidence from the blogosphere.=29Day dQ=23=14=13News ArticleRanker=4=4ImportanceScores
- 20. Idea:The more blog posts about an article the more important the subject must be.
- 21. Score by blog post volumeApproachTwo Stages:Score each news article a for all days d based on related blog post volume for day d. News articles are represented by their headlinesGiven a query day dQ rank A based on the score for each news article on day dQ, i.e. score(a, dQ)-> a voting processThe Votes Approach
- 22. Votes Approach : Stage 1Stage 1: Score days for each news story11234234Ranking of days for ablog postranking4) Rank days by votes received2) Select the top 1000 blog posts for a3) Each post votes for a dayDaysvotes = 2votes = 1votes = 2votes = 2For eachnews articlea1) Use its representation (headline) as a queryvotes = 0votes = 1votes = 2votes = 0TerrierVotesVoting Model : Count* Craig Macdonald PhD thesis 2009
- 23. Votes Approach : Stage 2Stage 2: Rank news articles for day dQvotes = 22Stage 1votes = 2votes = 242votes = 1votes = 2News article aNews article aNews article a14123votes = 0votes = 131votes = 03votes = 64votes = 2votes = 634QueryDay 2votes = 3votes = 2News article a132votes = 1votes = 321votes = 12votes = 91votes = 7votes = 931votes = 5votes = 72News article a3votes = 03votes = 542votes = 04Ranking of Articles
- 24. Introduction
- 25. The News Article Ranking Problem
- 27. Evaluating Votes
- 31. Hypothesis:The volume of relevant blog posts published on a news article is a strong indicator of that articles importance (from an editors perspective).Research Questions:Can the number of related blog posts to a news article published on day dQ provide a comparative ranking to that which an editor might make?Evaluating Votes
- 32. TaskTREC 2009:Blog Track : top news stories identification task
- 33. Rank news articles by predicted importance
- 35. 100k Articles provided by the New York Times
- 36. e.g. In a Decisive Victory, Obama Reshapes the Electoral MapBaselines:Random ranking
- 37. Inlinks (hyperlink evidence vs Votes textual evidence)
- 38. TREC best systemsSetup :TREC 2009 Blog track top news stories identification task
- 39. 100k news headlines from the New York Times to represent articles
- 40. E.g. In a Decisive Victory, Obama Reshapes the Electoral Map
- 41. Uses blog posts from the Blogs08 blog post corpus (28 million posts)
- 42. Judgments for 50 days of interest (dQs)
- 43. E.g. 2008-05-22 : headline1 headline34 headline35 headline38Evaluation:Mean Average Precision (MAP)Experimental SetupdQImportant headlines on dQ
- 44. Evaluating VotesEvaluation:TREC2009 Blog Track
- 45. Top stories identification task
- 46. Blogs08 blog post corpus
- 47. News Stories from New York TimesJudgementsParticipants were asked to label queries as being important or not
- 48. Criteria:
- 49. Timing : Favour stories that cover live events
- 50. Significance : Favour stories that effect many people
- 51. Proximity : Favour stories that are local to the reader (USA)
- 52. Prominence : Favour stories about celebrities, politicians etc.TREC Task:Each participating system needs to rank a set of news articles for a day dQ based upon evidence from the Blogs08 collection.
- 53. Ranking performance is measured in terms of Mean Average Precision (MAP).Indexing & Retrieval:Indexed Blogs08 using Terrier (stemming, stopwords)
- 54. Secondary index holds blog post -> day relations
- 55. Retrieve 1000 blogposts for headlines.
- 56. DPH (DFR)
- 57. BM25Baselines:Random ranking : average over 10 runs
- 58. Inlinks : hyperlink evidence
- 59. TREC 2009 best systemsExperimental Setup
- 60. Votes PerformanceBetter performance than TREC 2009 best systemsResults:BM25<DPH (DFR)Votes + extrasHyperlink evidence is of less value than textual evidenceVotes ApproachTREC 2009 Best Systems
- 61. Conclusions:Blog post volume is a decent indicator of editorial importance
- 62. Can be effectively leveraged to rank news articles by their importance
- 63. However, still room for improvement (0.17 map)Votes PerformanceHow can we improve Votes performance?
- 64. Introduction
- 65. The News Article Ranking Problem
- 67. Evaluating Votes
- 71. IdeaRe-score for each news article using evidence from days before and after dQ.IntuitionImportant stories will be discussed before or after the eventE.g. Run up to an electionTemporal PromotionBoth articles receive the same score for dQ under VotesdQNumVotesDays
- 72. Hypothesis:An article which is highly blogged about either before or after dQ should be scored more highly than one which is not.Approach:Promote articles which were highly blogged before or after dQ
- 73. Two Techniques
- 74. NDayBoost
- 76. ApproachLinearly combines the scores for day dQ with the n days before or after dQ.NDayBoostdQN = -2NumVotesScore=11Score=6Days
- 77. Idea:Evidence will weaken as the distance from dQ increases
- 78. NDayBoost might over-estimate the importance of days distant from dQApproach:Linearly combine scores as with NDayBoost, but weight each day by its distance from dQ using a Gaussian curve.GaussBoostDistance of days d Weight
- 79. GaussBoostWeightdWeightingThe weight for each article is calculated as :
- 80. d is the distance (in days) from dQ
- 81. w is the width of the gaussian curve
- 82. Controls the score decay as d increasesGaussBoostExample:n = -2, w = 1
- 83. Weights downward the scores for each day dependent on w.ScoreGaussBoost(B,4) = (1*4)+(0.79*1)+(0.18*1) = 4.970ScoreGaussBoost(A,4) = (1*4)+(0.79*4)+(0.18*3) = 7.700dQN = -2Score=7.700NumVotesScore=11Score=4.970Score=6Days
- 84. Hypothesis:An article which is highly blogged about either before or after dQ is more likely to be important than one which is not.Research Questions:Can the promotion of articles which are highly blogged about before or after dQ improve article ranking performance?
- 85. Does the quality of evidence decrease as distance from dQ increases?
- 86. Is historical or future (before or after dQ) blog post evidence more useful?Research Questions
- 87. GaussBoostApproachLinearly combine scores as with NDayBoost, but weight each day by its distance from d using a Gaussian curve.
- 88. The parameter w determines the width of the Gaussian curve, and as such, the weights d for the days.( n = -2, w = 0.5 )ScoreGaussBoost(A,4) = (1*4)+(0.38*4)+(0.01*3) = 4.608ScoreGaussBoost(B,4) = (1*4)+(0.38*1)+(0.01*1) = 4.390( n = -2, w = 1 )ScoreGaussBoost(A,4) = (1*4)+(0.79*4)+(0.18*3) = 7.700ScoreGaussBoost(B,4) = (1*4)+(0.79*1)+(0.18*1) = 4.970Temporal Promotion
- 89. NDayBoost PerformanceFuture blog postings does provide useful evidenceBaseline DPH+VotesMAPHistorical evidence is not useful for NDayBoostn value (days)
- 90. GaussBoost PerformanceFuture blog postings provide stronger evidence than historical postingsHistorical blog postings are useful for days close to dQBaseline DPH+VotesMAPw value (not days!)
- 91. Conclusions
- 92. Both historical and future evidence is useful to improve Votes ranking performance
- 93. Can use this evidence to generate a better ranking for editors if the data is available
- 94. Future evidence is more powerful than historical evidence
- 95. Not too useful if we want to rank in real-time though
- 96. NDayBoost is only effective for future evidence
- 97. GaussBoost is effective for both future and historical evidence
- 98. The most effective of the techniques
- 99. Does not over emphasise evidence from days distant from dQTemporal Promotion
- 100. Introduction
- 101. The News Article Ranking Problem
- 102. The Votes Approach
- 103. Evaluating Votes
- 104. Temporal Promotion
- 106. ConclusionsTalk OutlineCan we improve upon the news article representation?
- 107. Issue:News articles are represented with headlines
- 108. e.g. In a Decisive Victory, Obama Reshapes the Electoral Map
- 109. Headlines are a sparse representation of an article
- 110. Many headlines are not `news-worthy
- 111. Editors dont even consider these
- 112. e.g. paid death noticesApproach:Enrich the headlines using related terms extracted from blog posts and Wikipedia.
- 113. Prune headlines less likely to be news-worthyImproving the Article Representation
- 114. News Article EnrichmentIdea:Improve the news-article representation (headline)
- 115. Add related terms (counter sparsity)Approach:Select retrieve top 3 blog posts from: Blogs08 (query expansion , K. L. Kwok and M. S. Chan. SIGIR 1998)Wikipedia(collection enrichment, F. Diaz and D. Metzler. SIGIR 2006) using DPH (DFR)Expand query with the top 10 terms identified using Bo1 (G. Amati, Thesis 2003) from those documents.aTerrierTopTermsDPHBo1Blogs08/WikipediaQuery expansion/External Query expansion/Collection Enrichment
- 116. Related but generic termsCase specific terms
- 117. Article Enrichment:News headlines while being good quality representations are still ambigious
- 118. Collection enrichment helps find the blog posts that are related.Article Improvement PerformanceCollection enrichment with Wikipedia significantly increases performanceMAP
- 119. Article PruningIdea:Editors have lots of latent knowledge to draw upon
- 120. Try simulating this within the system
- 121. Prune away articles that an editor would not even considerNon-stories:Remove news articles which follow editorially defined patternsNoisy headlines:Remove misleading dates
- 122. Remove uppercase category termsPatterns List: New York TimesPaid Notice
- 123. Corrections for the Record
- 124. Comments of the Week
- 125. Inside the Times
- 126. Best Sellers
- 127. The Week Ahead
- 128. Movie Review
- 129. Arts Briefly
- 130. The Listings
- 131. Dance Review
- 132. Whats on Today
- 133. Critics Choice
- 134. Book of the Times
- 135. Music ReviewE.g. Inside the Times, November 6, 2008E.g. N.F.L. ROUNDUP; Giants Shut Down Tyree for Season; Raiders Cut Hall
- 136. Article Pruning:Removing non-news-worthy articles makes the ranking of articles easier.Article Pruning PerformanceDates and Uppercase further increase performance when combined.Patterns significantly increase performance over Votes aloneMAP
- 139. Headline pruning (All Heuristics)
- 140. Headline enrichment (Collection Enrichment)Results:Significant increase in performance over
- 141. DPH+Votes
- 142. DPH+Votes + Single techniquesVotes:The volume of blog posts about a news story is a useful measure for the importance from an editorial perspective
- 143. Can be used to automatically rank news stories for a newspaper editor
- 144. The Voting model provides strong baseline ranking performaceTemporal Promotion:Can be beneficial to look at blog post volume either before or after the day of interest
- 145. More useful to look at tomorrows blog posts than yesterdays blog posts
- 146. Evidence diminishes as we look further from the day of interest, evidence should be weighted accordinglyArticle representation ImprovementsEditorshold much in the way of latent knowledge that we need to simulate
- 147. i.e. they can disregard whole classes of articles as not being news-worthy
- 148. By pruning away such articlesapriori, ranking performance is improved
- 149. Headlines are sparse representations of news articles
- 150. Enrichment with terms from Wikipedia can help find more representative blog postsConclusions
- 151. TREC 2010:Blog track top stories identification task is running again in 2010
- 152. Focus on real-time ranking of news (no future evidence)
- 153. Uses a larger news article collection from ReutersFuture WorkQuestions?