Usability Testing Online Surveys (and forms)
1 like545 views
The document discusses the importance of surveys in user research, outlining their advantages over traditional qualitative methods. It covers methodologies for effective survey design and testing, including a four-stage questionnaire model and cognitive interviewing techniques. The document also highlights the relevance of mixed-methods research to achieve a comprehensive understanding of user behavior.
Usability Testing Online Surveys (and forms)
- 1. @uxordie Text fields and dropdowns and radios oh my! USABILITY TESTING ONLINE SURVEYS (AND FORMS) Eva Kaniasty UXPA International Conference @uxordie
- 2. @uxordie@uxordie 1. Why surveys? Isnt user research enough? 2. Why pre-test surveys? Let me count the ways. 3. Four stage questionnaire model The Cognitive Psych (Tourangeau) 4. The cognitive interview Usability testing just for surveys. (Willis) 5. Survey Pre-Testing Process 3 scenarios. AGENDA? AGENDA. 2
- 4. @uxordie@uxordie Validate qualitative research results Get feedback from a larger sample Build a participant database Bridge gap between quant and qual Build a strong case for change Kill features WHY SURVEYS? 4 Doesnt everybody love a survey? Execs Business People Marketing People Sales People
- 5. @uxordie@uxordie WHY SURVEYS? 5 Surveys are easier to do Surveys are faster to do Surveys are less biased Surveys are more objective Survey data is more valid Survey data is statistically significant Why are surveys better than user research? THE TRUTH ABOUT SURVEYS? Survey methodology is well-researched Best practices are well- established THE TRUTH ABOUT SURVEYS
- 6. @uxordie@uxordie MIXED METHODS RESEARCH 6 Full picture of user behavior Mitigate methodological weaknesses Understand the what and the why Make better decisions aka Triangulation
- 7. @uxordie@uxordie Complex onboarding Complex customization Data-entry driven applications Multi-step data collection workflows Skills testing WHAT ABOUT FORMS? 7 What kinds of forms benefit from testing? B2B eCommerce Healthcare Education Government
- 9. @uxordie@uxordie WHY TEST SURVEYS? 9 ERROR TYPES Sampling Measurement Coverage Non-Response There are no stupid questions or are there?
- 10. @uxordie@uxordie WHY TEST SURVEYS? 10 Stop doing survey research. - Jared Spool https://articles.uie.com/on_surveys DO SURVEY SUCK? Bias is incredibly easy to introduce. Bad questions can yield misleading answers. You cant adjust a survey mid-stream. Surveys are easy to manipulate. Surveys are not a substitute for qualitative research. Surveys are not a shortcut. Surveys lack context. Surveys are subjective. How many researchers does it take to write a bad survey question? Surveys are the most dangerous research toolmisunderstood and misused. They frequently straddle the qualita>ve and quan>ta>ve, and at their worst represent the worst of both.. - Erika Hall
- 12. @uxordie@uxordie 1. Comprehension Do you understand the question? 2. Retrieval Do you know or remember the answer? 3. Judgement Can you formulate a truthful, accurate response? 4. Response Can you map your answer to the response options provided? FOUR STAGE QUESTIONNAIRE MODEL 12 Roger Tourangeau
- 13. @uxordie@uxordie How many times have you participated in an online discussion about photography in the last year? 0 | Once or twice | 3-5 times | 6-10 times | more than 10 times FOUR STAGES 13 COGNITIVE PROCESS Well, Ive commented on things on Facebook, but I dont think thats what they mean. I also took an online class, which definitely included discussion, so that counts. (Comprehension) It was still cold when I took the class, so thats within the last year. (Retrieval) There were 12 classes, but I know I missed a couple of them. (Judgement) Overall, I was pretty active, so lets just say that its more than 10. (Response)
- 15. @uxordie@uxordie 1. COMPREHENSION 15 How many times have you seen a provider in the last year? Whats a provider? Does it mean a doctor? If so, does this include chiropractors therapists pharmacists? Is this calendar year or last 12 months?
- 16. @uxordie@uxordie 2. RETRIEVAL 16 How many times have you seen a primary care doctor in the last 12 months? I had a checkup a while ago, but it may be more than a year at this point.
- 17. @uxordie@uxordie 3.JUDGEMENT 17 How many times have you seen a primary care doctor in the last 6 months? I guess its 0. I probably should be taking better care of my health though. I did talk to her on the phone about a refill, so Ill just say 1.
- 18. @uxordie@uxordie 4. RESPONSE 18 How many times have you spoken to your primary care doctor in the last 6 months? Through email On the phone At the office Wait a second my doctor has this new telemedicine thingy, so I did have a live chat as well. Should I list that as part of email? 1 0 0
- 19. @uxordie@uxordie Are we done with this question yet? 19 How many times have you spoken to your primary care doctor in the last 6 months? Through email On the phone At the office Wait researcher, you also forgot to randomize the answer options, but you wont notice this until you watch a few people take the survey. 1 0 0 Other 0 Enter all that apply: time(s) time(s) time(s) time(s)
- 20. @uxordie@uxordie COGNITIVE BURDEN 20 Are 30 clear questions better than 20 difficult ones? Perhaps Yes Maybe Dont know
- 22. @uxordie@uxordie The administration of draft survey questions while collecting additional verbal information about the survey responses, which is used to evaluate the quality of the response or to help determine whether the question is generating the information that its author intends. COGNITIVE INTERVIEWING 22 Beatty & Willis (2007) What is cognitive interviewing?
- 23. @uxordie@uxordie Think-Aloud + Cognitive probes = Cognitive Interview USABILITY TESTING SURVEYS 23 How is cognitive interviewing different from usability testing?
- 24. @uxordie@uxordie 1. Comprehension/Interpretation What does the term leaderboard mean to you? 2. Paraphrasing Tell me in your own words what this is asking you. 3. Recall How do you know that you used it 10 times in the past 3 months? 4. Confidence Judgement How sure are you that you used it at least 10 times? 5. Specific What information do you review when you use the leaderboard? 6. General I noticed you changed your answer. Tell me what you were thinking. COGNITIVE PROBES 24 Gordon B. Willis How frequently do you use the leaderboard feature?
- 26. @uxordie@uxordie Q. When was the last time you visited your mother to help her with her computer? COGNITIVE INTERVIEW 26 A. I live with my mother actually, so I wouldnt be visiting her. P. OK, so do you remember the last time you helped her with her computer at home? (General) A. Yes it was about a month ago. P. How sure are you it was a month ago? (Confidence Judgement) A. Im pretty sure, because it was the weekend of the big snow storm at the end of March. P. Does anyone else help your mom with her computer? (Specific) A. Not really. I dont have any siblings, and my dad is just as bad at technology as she is.
- 27. @uxordie@uxordie Q. How concerned are you about lead contamination in your community garden? COGNITIVE INTERVIEW 27 A. I dont really know. This is the first time Ive thought about it honestly. Now that you brought it up though, I feel like I should be very concerned. P. Are you familiar with the term lead contamination? (Comprehension) A. I am I know that lead paint was used in old buildings, so sometimes it can get in the soil, but Im not sure why this would affect a community garden if its not near a building. P. You said that you should be very concerned How sure are you that youre very concerned? (Confidence Judgement) A. I guess I just assumed that this wouldnt be a problem in my community garden. So maybe Im not that concerned. Honestly, I dont know.
- 28. @uxordie@uxordie Q. Did you go to the beach last weekend? COGNITIVE INTERVIEW 28 A. I did actually. I went snorkeling. P. Tell me about it. (General) A. I was at a conference in Puerto Rico, and I got there a few days before. P. Was this the weekend that just passed, or the weekend before? (Interpretation) A. It was Saturday, so two days ago. P. Do you remember the last time you went to beach back home? (Recall) A.
- 29. @uxordie@uxordie Q. How often do you use our website to book a dog sitter? COGNITIVE INTERVIEW 29 A. OK In the past, Ive traveled a few times per year, but if its just a weekend my friend would watch him, so maybe 6 times per year so thats. once every 2-3 months P. I noticed you hesitated before choosing your answer. (General) A. Yeah I was trying to figure out what it would be on average. P. So how sure are you that its once every 2-3 months? (Confidence Judgement) A. Well, to be honest, my dog died a few months ago so I havent used it since then, obviously. Once a month or more / Once every 2-3 months / Every six months / Once a year / Less than once a year
- 31. @uxordie@uxordie Exploratory interview incorporating cognitive probes. MODERATED INTERVIEW 31 HOW USE IT TO EXPLORE Comprehension | Retrieval | Judgement Asking the right questions Understanding respondent background knowledge Exposing assumptions Setting the right response intervals PRO TIP Doubles as a round of qualitative research.
- 32. @uxordie@uxordie Observation and cognitive interview of draft survey. MODERATED DRY RUN 32 HOW USE IT TO EXPLORE Comprehension | Retrieval | Judgement | Response Terminology and interpretation issues Fixing confusing response categories Layout issues PRO TIP Probe on response categories as well as the questions.
- 33. @uxordie@uxordie Self-administration of survey to hand-picked, sample before official launch. PILOT 33 HOW USE IT TO EXPLORE Comprehension | Response Some terminology issues Missing response categories Layout issues PRO TIP Include open-ended responses for reporting problems in each survey section.