Investigating Self-Reporting Behavior in Long-Term Studies
Self-reporting techniques, such as data logging or a diary, are frequently used in long-term studies, but prone to subjects forgetfulness and other sources of inaccuracy. We conducted a six-week self-reporting study on smartphone usage in or- der to investigate the accuracy of self-reported information, and used logged data as ground truth to compare the sub- jects reports against. Subjects never recorded more than 70% and, depending on the requested reporting interval, down to less than 40% of actual app usages. They significantly over- estimated how long they used apps. While subjects forgot self-reports when no automatic reminders were sent, a high reporting frequency was perceived as uncomfortable and bur- densome. Most significantly, self-reporting even changed the actual app usage of users and hence can lead to deceptive measures if a study relies on no other data sources. With this contribution, we provide empirical quantitative long-term data on the reliability of self-reported data col- lected with mobile devices. We aim to make researchers aware of the caveats of self-reporting and give recommenda- tions for maximizing the reliability of results when conduct- ing large-scale, long-term app usage studies.
More Related Content
More from Distributed Multimodal Information Processing Group (14)
Recently uploaded (20)
Investigating Self-Reporting Behavior in Long-Term Studies
- 1. INVESTIGATING SELF-REPORTING BEHAVIOR IN LONG-TERM STUDIES Andreas M旦ller , Matthias Kranz , Barbara Schmid , Stefan Diewald , Luis Roalter Technische Universit辰t M端nchen, Germany Universit辰t Passau, Germany
- 4. RESEARCH QUESTIONS Accuracy? Change over time? In鍖uence of reporting frequency? Reliability maximization? A. M旦ller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
- 5. BACKGROUND Electronic diaries show higher compliance (Hufford & Shields, 2002) Mobile phone as survey tool (Consolvo et al., 2007) Consolvo et al., 2007 A. M旦ller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
- 6. CONTRIBUTIONS Empirical quantitative long-term data on reliability of information collected with mobile devices Recommendations for maximizing result reliability A. M旦ller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
- 7. A. M旦ller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris METHODOLOGY Evaluate reporting behavior Ground truth Can be gained in automated way Limited effort Smartphone usage GOAL REQUIRE- MENTS SOLUTION Frequently used apps Installed by everyone
- 8. Mail Facebook Frequently used apps Installed by everyone A. M旦ller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
- 9. A. M旦ller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris PROCEEDING Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 6 + 4 Pre-Questionnaire Reminder Emails Post-Questionnaire Post-Post- Questionnaire Requested Self-Reports & Logging
- 10. TASK Answer questionnaire after Facebook or Mail usage Report as accurate as possible 1. How long did you use the app? 2. How many times did you use the app without answering a questionnaire? 1 direct report n indirect reports A. M旦ller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
- 11. SELF-REPORTING AND EXPERIENCE SAMPLING ASSISTANT A. M旦ller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
- 12. SERENA App usage logging Server upload Questionnaire triggers Event-based Time-based Manually A. M旦ller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
- 13. A. M旦ller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris 3 CONDITIONS Voluntary Interval Event No trigger Daily trigger Trigger after app usage 30 Participants 3,631 Mail usages 3,181 Facebook usages
- 14. SESSIONS Voluntary Interval Event Amount of reported Facebook usages 37.6% 63.8% 54.3% Indirect reports Direct reports A. M旦ller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
- 15. DURATIONS Voluntary Interval Event 1:29 1:29 1:22 2:52 3:02 3:35 Facebook sessions Real Self-Reported A. M旦ller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
- 16. OVER TIME Self-report ratio decreases Actual usage decreases Answering the questionnaire changed my Facebook usage habits. Voluntary 2.3 Interval 2.2 Event 3.5 5 = strongly agree A. M旦ller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
- 17. A. M旦ller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris DISCUSSION Subjects reported max. 70% of usage Commitment decreased Reports may in鍖uence actual behavior Subjects overestimated session length Reminder emails pushed commitment in 2nd phase Behavior change with increasing burden Little control: forgetting High control: burden Trigger in鍖uence lower than hypothesized App usage decreased Most subjects would report max. 4 weeks
- 19. Please cite this work as follows: A. M旦ller, M. Kranz, B. Schmid, L. Roalter, S. Diewald Investigating Self-Reporting Behavior In Long-Term Studies In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2013), pp. 2931-2940, Paris, France, April-May 2013. If you use BibTex, please use the following entry: @inproceedings{chi2013selfreport, author = {Andreas M"{o}ller and Matthias Kranz and Barbara Schmid and Luis Roalter and Stefan Diewald}, title = {Investigating Self-Reporting Behavior In Long-Term Studies}, booktitle = {Proceedings of the 2013 ACM annual conference on Human Factors in Computing Systems}, pages = {2931--2940}, series = {CHI '13}, year = {2013}, isbn = {978-1-4503-1899-0}, location = {Paris, France}, numpages = {10}, publisher = {ACM}, address = {New York, NY, USA}, }