How to create surveys that don't suck
23 likes5,440 views
The document discusses the importance of minimizing researcher bias in surveys by creating clear, concise questions that all participants will interpret the same way. It notes that surveys are inherently subjective as they are designed by people. However, awareness of this subjectivity during question design allows researchers to reduce "noise" in data by avoiding ambiguous, vaguely defined response options that different people may interpret differently. The document provides examples of poor question design and advocates for response options that are logically spaced and intuitively understood the same by all participants. Clear, unambiguous questions, though not guaranteeing valid results, offer more assurance that the responses reflect the phenomena measured rather than uncertainties introduced by unclear questioning.
More Related Content
What's hot (20)
Similar to How to create surveys that don't suck (20)
Recently uploaded (20)
![[Webinar] Scaling Made Simple: Getting Started with No-Code Web Apps](https://cdn.slidesharecdn.com/ss_thumbnails/webinarscalingmadesimplegettingstartedwithno-codewebapps-mar52025-250305183437-f03c78a3-thumbnail.jpg?width=560&fit=bounds)
![Formal Methods: Whence and Whither? [Martin Fr辰nzle Festkolloquium, 2025]](https://cdn.slidesharecdn.com/ss_thumbnails/mf2025-250305164811-a0930761-thumbnail.jpg?width=560&fit=bounds)
How to create surveys that don't suck
- 2. "They've done studies, you know. 60% of the time, it works every time. - Brian Fantana 03/18/12 Will Evans & Yana Kuchirko 2
- 3. Surveys Arent Objective They are created by people who are biased and analyze their results based on their own (mis)perceptions. 03/18/12 Will Evans & Yana Kuchirko 3
- 4. Epistemological Dead End? No. Awareness that the researcher plays an integral role in the process of measuring any given phenomena by deciding how to measure is key. There are ways to minimize researcher bias by creating better questions. 03/18/12 Will Evans & Yana Kuchirko 4
- 5. Methods are not just methods How you measure what you are studying shapes what you find. 03/18/12 Will Evans & Yana Kuchirko 5
- 6. Lets say you are studying How often teenagers use your website 03/18/12 Will Evans & Yana Kuchirko 6
- 7. You might ask teenagers How often do you my website? c.Very rarely d.Rarely e.Occasionally f.Frequently g.Very frequently Response options source: http://www.dataguru.org 03/18/12 Will Evans & Yana Kuchirko 7
- 8. You might ask teenagers How often do you my website? c.Very rarely d.Rarely And whats wrong e.Occasionally with these???? f.Frequently g.Very frequently Response options source: http://www.dataguru.org 03/18/12 Will Evans & Yana Kuchirko 8
- 9. You might ask teenagers How often do you my website? ! !! VI TY EC TI c.Very rarely BJ d.Rarely SU And whats wrong e.Occasionally with these???? f.Frequently g.Very frequently Response options source: http://www.dataguru.org 03/18/12 Will Evans & Yana Kuchirko 9
- 10. Hmmm, for me frequently really means. A few times a week A few times a day So what are we really measuring? NOISE. 03/18/12 Will Evans & Yana Kuchirko 10
- 11. What is noise? What we observe 03/18/12 Will Evans & Yana Kuchirko 11
- 12. What is noise? We WANT this! We DONT WANT this! 03/18/12 Will Evans & Yana Kuchirko 12
- 13. An imaginary scenario Lets really stretch our thinking a bit here to provide a more concrete understanding of noise in your data. Imagine you have AT&T phone service and youre trying to make a call. The signal isnt clearly going through and you end up hearing everything but the other persons voice. Frustrating, right? Thank God this is only a pretend scenario. 03/18/12 Will Evans & Yana Kuchirko 13
- 14. How is AT&T related to Noise? When you develop bad questions, you dont hear the message of your data clearly. 03/18/12 Will Evans & Yana Kuchirko 14
- 15. Sowhy is noise bad??? Badly structured and poorly worded questions that obfuscate meaning for participants provide bad data. Bad data Doesnt answer your research question Makes it difficult to interpret results Is pretty much useless. 03/18/12 Will Evans & Yana Kuchirko 15
- 16. More bad examples of frequently used response options a. Completely satisfied a. Totally like b. Very satisfied b. Very much like c. Fairly well satisfied c. Moderately like d. Somewhat dissatisfied d. Somewhat like e. Very dissatisfied e. Not like Whats the difference between Whats the difference between fairly well and somewhat? moderately like and somewhat like? 03/18/12 Will Evans & Yana Kuchirko 16
- 17. More bad examples of frequently used response options a. Completely satisfied a. Totally like b. Very satisfied b. Very much like c. Fairly well satisfied c. Moderately like d. Somewhat dissatisfied d. Somewhat like e. Very dissatisfied e. Not like Whats the difference between Whats the difference between fairly well and somewhat? moderately like and somewhat like? Semantics 03/18/12 Will Evans & Yana Kuchirko 17
- 18. Words like. Sometimes Often Moderately Very Not very much Are subjective = they mean different things for different people 03/18/12 Will Evans & Yana Kuchirko 18
- 19. A better alternative? Actually asking people how often they do certain activities. 03/18/12 Will Evans & Yana Kuchirko 19
- 20. In order to gather objective data your questions must mean the same thing for everyone. How often do you use my website? d.Never e.A few times a year f.Once a month g.2-3 times a month h.Once a week i.A few times a week j.Every day Never means never for everyone! 03/18/12 Will Evans & Yana Kuchirko 20
- 21. Importance of Anchors Responses options serve as anchors for each question, determining the location of each responses as qualitatively distinct from the other. 03/18/12 Will Evans & Yana Kuchirko 21
- 22. Importance of Anchors Responses options serve as anchors for each question, determining the location of each responses as qualitatively distinct from the other. Strongly Strongly disagree Disagree Neutral Agree Agree 03/18/12 Will Evans & Yana Kuchirko 22
- 23. (Ideal) Response Symmetry Good questions aim to have symmetrical quantitative/qualitative distance between anchors a. Never b. A few times a year The distance between never and c. Once a month a few times a year is proportionate to d. 2-3 times a month every day and a few times a week. e. Once a week f. A few times a week g. Every day 03/18/12 Will Evans & Yana Kuchirko 23
- 24. Adherence to Logic & Linearity People are accustomed to paradigms that are intuitive, and often linear. Anchors should not be the exception. 03/18/12 Will Evans & Yana Kuchirko 24
- 25. Intuitive Anchor Directions Never Always Disagree Agree Very poor Very good Not very Very important Important 03/18/12 Will Evans & Yana Kuchirko 25
- 26. Value of Clarity If questions are clear and concise, participants would spend less time analyzing the questions themselves and more time on answering them. Mystic Arts, LLC 03/18/12 Will Evans & Yana Kuchirko 26
- 27. So do good questions guarantee valid results? No. But good questions offer more assurance that you are listening to signal and not the noise. But answering your research question can be done in other ways Behavioral Observations Open ended qualitative questions Many more 03/18/12 Will Evans & Yana Kuchirko 27
- 28. The End. So was this presentation simply oh-my-fucking-god-awesome? a. Hell yeah! b. Definitely! c. Totally rocked! 03/18/12 Will Evans & Yana Kuchirko 28
- 29. Thanks!