Tips for conducting long-term diary studies - UPA 2009
?Download as PPT, PDF?
5 likes?1,653 views
This document discusses conducting longitudinal diary studies. It provides tips for running a better diary study, including creating an effective template, understanding the data collected, and keeping users engaged. Some key tips are to recruit extra participants to offset dropouts, use a recruiter to save time, communicate clear deadlines, make the template easy for participants to use with sample entries and questions separated from answers, require daily entries to maintain engagement, and talk to participants directly for clarification and support. Diary studies provide unique insights into infrequent tasks, help usage, issue resolution over time, and can be conducted without travel at a lower cost than site visits.
Tips for conducting long-term diary studies - UPA 2009
- 1. Conducting Longitudinal Diary Studies Amanda Nance Sage June 10, 2009
- 3. Goal of this presentation ? How to run a better diary study ¨C Create effective template ¨C Understand the data ¨C Keep users engaged ? See benefits of the method
- 4. Topics for this presentation ? Reason for the project ? About the diary study ? Diary study tips ? Unique contributions of the diary method
- 5. Reasons for the project ? Assess ¡°getting started¡± experience of our desktop accounting software ? Focus on tasks after initial set up ? Create requirements for next release
- 6. Used multiple methods to get truer picture Site visits Support calls Survey analysis Researcher calls Diary study Diary from: http://www.flickr.com/photos/bdorfman/15846725/
- 7. About the diary study ? 5 participants: small business owners, bookkeepers ? Diaries kept for up to 8 weeks beginning at install ? Diaries submitted weekly by email ? Compensation ¨C at least $40/week
- 8. About the diary study ? Provided template in Word to record ¨C Daily Work: tools they used, other tasks ¨C Issues and Comments: problems with our product
- 9. Tips
- 10. Recruit extra participants ? Recruited 12 ? 5 completed ? Reasons for drop outs were interesting findings, too
- 11. Use a recruiter to save time ? Pros and cons ? They can recruit and more ¨C Email documents ¨C Remind of deadlines ¨C Collect diaries
- 12. Communicate deadlines ? Communicate due date for first submission ? Disqualify participants who don¡¯t respond ? Close the loop
- 13. Template tips
- 15. Make it easy for participants ? Keep it short ¨C This study: 3 questions daily, 6-7 per incident ? Separate questions from answers
- 16. Separate questions from answers
- 17. Show sample entries What did you do for your business today? ? I answered several customer phone calls. ? Called automated bank system to make sure we have enough money for payroll. ? Called in payroll to ADP (we outsource payroll). ? Ordered parts for one of our jobs by phone.
- 18. Ask for screenshots ¡°Looking at Aged Receivables¡. They do not match. Why are they different?¡±
- 19. Tips for screenshots ? If they are important, require them ? Include instructions
- 20. Structure template for easy analysis ? Separate daily entries from critical incidents ? Include participant name and week number in footer and file name
- 21. Iterate on the template ? Prior to launching ? As you receive user feedback
- 22. Process tips
- 23. Require daily entries ? Keeps participants engaged ? Gives you a reason to pay them ? Opportunity to learn about the broader work context
- 24. Talk to the participants yourself ? Intro call to understand business and product expectations ? Follow up on unclear entries ? Provide support
- 25. Research tips ? Read entries as soon as possible ? Leave time to research
- 26. Data analysis ? No different from other methods ? Evaluate data ¨C Excel to categorize theme, second theme, week # ? Share and act on data ¨C Summarize key findings, provide recommendation for each ¨C Use customer quotes ¨C Share complete diaries
- 27. Unique contributions of diary study ? Infrequent tasks ? Help use ? Users resolving issues ? Tasks over time
- 28. Reasons to run a diary study ? No travel required ? Non-intrusive, in context ? Cheaper than site visits
- 29. Engagement and meaningful data ? Pay for more complete responses ? Encourage useful responses through feedback on recent entries ? Continue participants who are useful
- 30. Key Points ? Daily entries ? Explain what you expect ? Sample entries ? Reasonable number of questions ? Personal contact with researcher ? Questions?
- 31. Contact More tips in conference proceedings anance@gmail.com Twitter: @amandaux
Editor's Notes
- #12: Pros/cons. I started by recruiting myself: I thought I could identify appropriate customers more accurately than a screener. Also thought customers more likely to participate if contacted directly by Sage. Recruitment continued... Was taking time away from analyzing the data, so I hired a recruiter. Recruiters can also handle some of the admin work. (give examples from bullets) All of this frees up your time for analysis.
- #13: When deadline passes, let potential participants know they cannot complete the study.
- #15: Full template is in proceedings. Tell participants how to use the document. This includes: 1. Explain what the diary is 2. How to complete it ¨C Be clear about what you expect. Story here: participant who didn¡¯t use product one week so didn¡¯t fill in diary 3. Remind when and where to send it 4. Contact info ¨C in case they have questions.
- #17: Makes it easier for participant to fill out Easier to analyze later Can paste it into Excel and rotate it to look across participants
- #18: With appropriate level of detail Full example in the proceedings
- #19: This is a quote from one of the diaries. I had no idea what it meant, but she included screenshots. She felt there was a discrepancy between 2 screens showing the same information, but looking at the screenshots, I realized the label on one of them was very confusing.
- #20: This study: only 1 person included screenshots, probably because template said, ¡°include if it helps you explain the problem¡±
- #25: Engages users Their expectations about the product ¨C found out that people wanted a certain feature through these Follow up: by email or phone depending on type of issue Provide support after you understand problem.
- #26: Because you¡¯re following up w/ participants, it¡¯s important to read entries quickly. It allows you to get clarification while incidents are fresh on their minds. Our product is complex. I didn¡¯t always understand how the system works. Had to understand that to interpret some of the entries.
- #28: Captured infrequent tasks Use of Help documented, reasons people call support Saw how users resolve issues on their own and how long it takes Gave a sense for overall getting started process
- #30: ¡°Participants who provide especially helpful information (determined at our discretion) will be given a bonus of $140 for a total of $300.¡± But didn¡¯t want to affect how often people used the software. I continued 2 participants for 8 weeks