ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo

Sigir 2014-mobile-eye-tracking-slides

?
?
Dmitry Lagun
Dmitry Lagun

This document discusses two studies on measuring user attention and satisfaction in mobile search. The first study found that viewport and page metrics can indicate whether a knowledge graph result is relevant and that users are not satisfied when an irrelevant knowledge graph is shown. The second study found that viewport time closely correlates with gaze time, users prefer the top half of the mobile screen, and that the "short scroll effect" influences attention patterns differently between desktop and mobile.

1 of 26
Download to read offline
Towards Better Measurement of Attention and Satisfaction in Mobile Search Dmitry Lagun, Chih-Hung Hsieh, Dale Webster, Vidhya Navalpakkam
Thanks! Vidhya Navalpakkam Dale WebsterChih-Hung Hsieh 2
Mobile is popular! ? 25% of Web page visits come from mobile [Statcounter.com, 2014] ? Mobile browsing grew five fold since 2010 (5%) [Statcounter.com, 2014] ? One in every 5 search queries is issued from a mobile device [RKG Digital Marketing Report, 2013] 3
Our Study Attention MeasurementSatisfaction with Rich Results Knowledge Graph Result 4
Satisfaction with Rich Results on Mobile: Background ? Long history of using clicks for measurement of search satisfaction and result relevance [Joachims et al., SIGIR 2005; Agichtein et al., SIGIR 2006] ? Result relevance and implicit indicators (mouse cursor hover, touch & swipe) [Huang et al., CHI 2011; Lagun et al., SIGIR 2011; Guo et al., SIGIR 2013] ? Rich Answers do not require to click and mouse hovers do not exist on mobile ? ¨C What other implicit metrics can we use to infer result relevance/satisfaction without clicks/hovers? 5
User Study Design ? Two Factor (within Subject) ¨C Relevance ¨C Presence ? 20 Search Tasks ? Users were asked to provide explicit satisfaction score for each task (1-7 scale) KG Relevant KG Not Relevant KG Present 5 Tasks 5 Tasks KG Absent 5 Tasks 5 Tasks 6
User Study Details ? Participants ¨C 24 users (diverse background, age, occupation) ? Mobile Eye Tracker Setup ? Calibration Directly on Phone Screen 7
Can Implicit User Metrics Indicate Answer Relevance? ? Page and Task metrics ¨C Time on SERP ¨C Number of Scrolls ¨C Time on Task ? Gaze Metrics ¨C Time on Rich Result (and %) ¨C Total Time below Rich Result (and %) ? Viewport Metrics ¨C Time on Rich Result (and %) ¨C Total Time below Rich Result (and %) KnowledgeGraphResult 8
KG is Not Relevant ? More Scrolling 10
KG is Relevant ? Faster Search (answer is found in KG without a click) 11
No Impact on User Satisfaction when KG is Not Relevant! 12
Gaze Metrics vs. KG Relevance 13 Relevant Not Relevant More Time Below the KG Result
%Viewport Time Below vs. KG Relevance 15 0 5 10 15 20 25 30 Not Relevant Relevant %ViewportTimeBelowKG More time on results below Not Relevant KG
Satisfaction with Rich Results: Summary ? We can use Page and Viewport metrics to infer KG relevance and satisfaction ? No impact on user satisfaction when Not Relevant KG is shown ? Users view more results below the KG, when it is Not Relevant 16
Our Study Attention MeasurementSatisfaction with Rich Results Knowledge Graph Result 17
Attention Measurement in Search: Background ? Eye Tracking ¨C accurate, but limited in scale [Granka et al., WWW 2004; Buscher et al., SIGIR, CHI 2008- 2010] ? Mouse Cursor Tracking ¨C less accurate, but scalable ? [Huang et al., CHI 2011, 2012; Lagun et al., SIGIR 2011; Guo et al., CHI 2010, WWW 2012; Navalpakkam et al., WWW 2013] ? Viewport Tracking ¨C accurate (???), scalable (on mobile) 18
Viewport Time Calculation: Primer ? Display Time = 10 sec ? ViewportTime(R1) = ? ? Coverage ¨C % of screen area occupied by the result (e.g. Coverage(KG) > Coverage(R1)) ? Exposure ¨C % of result area visible on the screen (e.g. Exposure (R2) < 1.0) ? ViewportTime(R) = DisplayTime * Coverage(R) * Exposure(R) 19 KG R1 R2
%Gaze Time %ViewportTime ViewportTime Gaze Time Can we use Viewport Time to measure time spent on each result? Pearson R = 0.57 Pearson R = 0.69 one search result Correlation is high ? can use Viewport Time to accurately measure time spent on individual search result at scale 20
Are attention patterns similar on desktop and mobile? ? Granka et al., WWW 2004 ? 21
Viewing Time vs. Result Position Granka et al., WWW 2004 On desktop: Why? 22
Short Scroll Effect 23
Short Scroll Effect 24
Short Scroll Effect 25
Do users have position preference when reading on a mobile phone? 26
Conclusions ? Viewport and Page metrics can be used to measure Rich Answer Relevance and Satisfaction ? Viewport time provides accurate (R=0.69) estimate on time spent on search result ? Users prefer to position content on top half of the phone¡¯s screen 27
Results Summary Attention MeasurementSatisfaction with Rich Results 28 %Gaze Time %ViewportTime Viewport ¡Ö Gaze (on mobile) Pearson R = 0.69 Top half of the screen receives more Attention ¡°Short-Scroll¡± effect Granka et al., WWW 2004 DesktopMobile Relevant Not Relevant More results are viewed if Answer is Not Relevant No Impact on User Satisfaction when KG is Not Relevant!

More Related Content

Sigir 2014-mobile-eye-tracking-slides

  • 1. Towards Better Measurement of Attention and Satisfaction in Mobile Search Dmitry Lagun, Chih-Hung Hsieh, Dale Webster, Vidhya Navalpakkam
  • 2. Thanks! Vidhya Navalpakkam Dale WebsterChih-Hung Hsieh 2
  • 3. Mobile is popular! ? 25% of Web page visits come from mobile [Statcounter.com, 2014] ? Mobile browsing grew five fold since 2010 (5%) [Statcounter.com, 2014] ? One in every 5 search queries is issued from a mobile device [RKG Digital Marketing Report, 2013] 3
  • 4. Our Study Attention MeasurementSatisfaction with Rich Results Knowledge Graph Result 4
  • 5. Satisfaction with Rich Results on Mobile: Background ? Long history of using clicks for measurement of search satisfaction and result relevance [Joachims et al., SIGIR 2005; Agichtein et al., SIGIR 2006] ? Result relevance and implicit indicators (mouse cursor hover, touch & swipe) [Huang et al., CHI 2011; Lagun et al., SIGIR 2011; Guo et al., SIGIR 2013] ? Rich Answers do not require to click and mouse hovers do not exist on mobile ? ¨C What other implicit metrics can we use to infer result relevance/satisfaction without clicks/hovers? 5
  • 6. User Study Design ? Two Factor (within Subject) ¨C Relevance ¨C Presence ? 20 Search Tasks ? Users were asked to provide explicit satisfaction score for each task (1-7 scale) KG Relevant KG Not Relevant KG Present 5 Tasks 5 Tasks KG Absent 5 Tasks 5 Tasks 6
  • 7. User Study Details ? Participants ¨C 24 users (diverse background, age, occupation) ? Mobile Eye Tracker Setup ? Calibration Directly on Phone Screen 7
  • 8. Can Implicit User Metrics Indicate Answer Relevance? ? Page and Task metrics ¨C Time on SERP ¨C Number of Scrolls ¨C Time on Task ? Gaze Metrics ¨C Time on Rich Result (and %) ¨C Total Time below Rich Result (and %) ? Viewport Metrics ¨C Time on Rich Result (and %) ¨C Total Time below Rich Result (and %) KnowledgeGraphResult 8
  • 9. KG is Not Relevant ? More Scrolling 10
  • 10. KG is Relevant ? Faster Search (answer is found in KG without a click) 11
  • 11. No Impact on User Satisfaction when KG is Not Relevant! 12
  • 12. Gaze Metrics vs. KG Relevance 13 Relevant Not Relevant More Time Below the KG Result
  • 13. %Viewport Time Below vs. KG Relevance 15 0 5 10 15 20 25 30 Not Relevant Relevant %ViewportTimeBelowKG More time on results below Not Relevant KG
  • 14. Satisfaction with Rich Results: Summary ? We can use Page and Viewport metrics to infer KG relevance and satisfaction ? No impact on user satisfaction when Not Relevant KG is shown ? Users view more results below the KG, when it is Not Relevant 16
  • 15. Our Study Attention MeasurementSatisfaction with Rich Results Knowledge Graph Result 17
  • 16. Attention Measurement in Search: Background ? Eye Tracking ¨C accurate, but limited in scale [Granka et al., WWW 2004; Buscher et al., SIGIR, CHI 2008- 2010] ? Mouse Cursor Tracking ¨C less accurate, but scalable ? [Huang et al., CHI 2011, 2012; Lagun et al., SIGIR 2011; Guo et al., CHI 2010, WWW 2012; Navalpakkam et al., WWW 2013] ? Viewport Tracking ¨C accurate (???), scalable (on mobile) 18
  • 17. Viewport Time Calculation: Primer ? Display Time = 10 sec ? ViewportTime(R1) = ? ? Coverage ¨C % of screen area occupied by the result (e.g. Coverage(KG) > Coverage(R1)) ? Exposure ¨C % of result area visible on the screen (e.g. Exposure (R2) < 1.0) ? ViewportTime(R) = DisplayTime * Coverage(R) * Exposure(R) 19 KG R1 R2
  • 18. %Gaze Time %ViewportTime ViewportTime Gaze Time Can we use Viewport Time to measure time spent on each result? Pearson R = 0.57 Pearson R = 0.69 one search result Correlation is high ? can use Viewport Time to accurately measure time spent on individual search result at scale 20
  • 19. Are attention patterns similar on desktop and mobile? ? Granka et al., WWW 2004 ? 21
  • 20. Viewing Time vs. Result Position Granka et al., WWW 2004 On desktop: Why? 22
  • 24. Do users have position preference when reading on a mobile phone? 26
  • 25. Conclusions ? Viewport and Page metrics can be used to measure Rich Answer Relevance and Satisfaction ? Viewport time provides accurate (R=0.69) estimate on time spent on search result ? Users prefer to position content on top half of the phone¡¯s screen 27
  • 26. Results Summary Attention MeasurementSatisfaction with Rich Results 28 %Gaze Time %ViewportTime Viewport ¡Ö Gaze (on mobile) Pearson R = 0.69 Top half of the screen receives more Attention ¡°Short-Scroll¡± effect Granka et al., WWW 2004 DesktopMobile Relevant Not Relevant More results are viewed if Answer is Not Relevant No Impact on User Satisfaction when KG is Not Relevant!