際際滷

際際滷Share a Scribd company logo

Visually Integrative Representation of User Types in Surveys (Ricardo Carvalho & Joseph Luchman & Michael Paraloglou & Vanessa Patterson & Ron Vega)

uxpa-dc
uxpa-dc

- The document describes research analyzing survey response patterns of 370 respondents who skipped a question. Through mixture modeling, they identified 3 distinct "user types": Quitters, Returners, and Completers. - A visualization tool was created mapping response patterns and identifying troublesome questions and areas. This allowed easy identification of user types and usability issues. - Key conclusions were that the tool provides an effective way to examine survey effectiveness, pinpoint problem questions and respondents, and suggest improvements to the design and administration process.

1 of 23
Download to read offline
Why Do Respondents Skip Questions in Surveys: A Visually Integrative Representation of User Types Ricardo Carvalho Joseph Luchman Michael Paraloglou Vanessa Patterson Ron Vega 1
Outline Background Our Research Our Findings Our Conclusions & Recommendations Future Research 2
Background 3
Background DoD Youth Poll December 2011 survey Mailed to 50,000 youth ages 16 to 24 with no prior or current military experience through stratified, probability-based sampling Address-based sample drawn from list frame estimated to cover 92% of target population Standard mailing methodology (Dillman 2007) Scantron survey; double-entered and verified Up-front and contingent monetary incentives upon completion Response Rate 3: 17%, Contact Rate 2: 92%; n= 7,210 Note: although our study was specific to completing a paper survey, much of the theory (and more importantly, the tool) can be applied to other survey modes and experiences 4
Background 5
Background The Issue # Refusals to Q29 and Q30 Amount of refusals per item was very small 1-18 Refusals (one to many but not all items) (<1.5%) up to Q29 19 Refusals (all items) But at Q30 and thereafter, it increases to 6% 6% 5% about 5-6% 5% 5% 4% Behavior of certain users changes in consistent manner 3% 2% Can we understand the users 1% 1% experience and behavior? 0% Q29 Q30 6
Our Research 7
Our Research We noticed 370 respondents whose behavior seemed fundamentally different Are these different user types? Or was there a usability issue with the survey (troublesome areas)? How can we identify the user type or a troublesome area? Does this kind of information tell us what to change in the experience? No 8
Our Research The behavior we noticed is characteristic of satisficing (Simon 1957) Economic phenomenon: satisfying and sacrificing We exercise an acceptable level of effort to achieve a satisfactory but less than optimal outcome Example: driving around for the cheapest gas price There is substantial literature written on this topic and how it applies to surveys (Krosnick 1991) Behavior points to this phenomenon, but very difficult to be certain The focus of our presentation is NOT on exploring this behavior but on understanding and visualizing different user types How does the survey experience impact users? Are there usability issues we can notice or isolate? Can we build a tool to help improve the overall user experience and hence obtain more complete and accurate information? 9
Our Findings 10
Our Findings What we did: Examined only the 370 respondents who refused all of Q30 Determined if unique user types existed through mixture modeling Wrote code to visually map these user types and their refusals for the remainder of the survey Marked page breaks and grid questions in this map What we found: 3 distinct user types The Quitters The Returners The Completers Map allows us to easily identify these user types Map also allows us to easily identify troublesome areas 11
Our Findings Black line = pages The 370 Respondents Orange line = grid question Colored squares = question was ANSWERED
Our Findings: The Quitters Black line = pages Orange line = grid question Last page of survey Colored squares = question was ANSWERED Demographics Engagement clearly breaks off and users flip to back of survey Paper survey immediately presents users with workload Completes Demographic questions (essential and easy items) for token of appreciation
Our Findings: The Returners Black line = pages Orange line = grid question Colored squares = question was ANSWERED Grid questions Engagement terminates after long second grid question (Q30) but returns Selectively respond to taskful questions (i.e., grid questions) to minimize effort
Our Findings: The Completers Black line = pages Orange line = grid question Colored squares = question was ANSWERED Most conscientious and engaged group Engagement terminates for only Q30 Occasional refusals Only a few questions are left unanswered by this user type
Our Findings: Profiling the User Types Other Asian Hispanic Asians show more Completers Black Whites show almost all of the Returners White 16
Our Findings: Profiling the User Types Mixture model: seeks homogenous distributions within data based on number of questions refused after Q30 Predictive model based on census block sociodemographic data linked to respondent scores Exploratory predictive model (i.e., empirically driven) Hispanic, Native Hawaiian or Pacific Islander Civilian Population in Labor Force Employed; Age 16 Overall Population Median Household Income Population and up Population with less than 9th Grade Education; Age Population in Labor Force Unemployed; Age 16 and Population aged 16-17 Hispanic, Other Population 25 and up up Population with some High School Education; Age Population aged 18-20 Hispanic Population Population not in Labor Force; Age 16 and up 25 and up Population with High School Education; Age 25 and Percent of Population in Labor Force Unemployed; Population aged 21-24 Population in Nursing Home up Age 16 and up Population in other Institutionalized Group Population with some College Education; Age 25 Population employed in Private, for Profit; Age 16 and Median Age Quarters and up up Population Employed in Private, not-for Profit; Age 16 Non-Hispanic, White Population Population in College Dorms Population with Associates Degree; Age 25 and up and up Population Employed in Local Government; Age 16 Non-Hispanic, Black Population Population in Military Barracks Population with Bachelors Degree; Age 25 and up and up Population in Non-Institutionalized Group Population Employed in State Government; Age 16 Non-Hispanic, American Indian Population Population with Masters Degree; Age 25 and up Quarters and up Population Employed in Federal Government; Age 16 Non-Hispanic, Asian Population Average Household Size Population with Professional Degree; Age 25 and up and up Non-Hispanic, Native Hawaiian or Pacific Islander Average Household Size Non-Family Household Population with Doctorate Degree; Age 25 and up Population Self-Employed; Age 16 and up Population Non-Hispanic, Other Population Average Household Size Family Household Families at Poverty Level Population Unpaid Family Work; Age 16 and up Population Speaking only English at Home; Age 5 Population Employed Blue Collar Work; Age 16 and Hispanic, White Population Families at Poverty Level with Children and Older up Population Speaking Spanish at Home; Age 5 and Population Employed White Collar Work; Age 16 and Hispanic, Black Population Families above Poverty Level Older up Population Employed Service and Farm Work; Age 16 Hispanic, American Indian Population Housing Units Owned by Occupant Families above Poverty Level with Children and up Population in Labor Force Employed by Armed Hispanic, Asian Population Housing Units Rented by Occupant Population Male Forces; Age 16 and up Average Length of Residence Population Female 17
Our Findings: Profiling the User Types Unemploy % with Median tired of being surveyed! Im Summary of other socio-economic User Type -mentwasting our Bachelors Income The government is variables Rate time/money! Degree Quitters $63,000 7.1% 13.4% Government and private, not-for (n=111) profit employment with large Im doing the best I can, but youre asking a lot household size conditions Returners $58,000 8.5% 10.7% More transient, socioeconomically (n=180) I should do a good job at this, my disadvantageous conditions opinions are helping Completers $61,000 8.1% 11% Less transient, socioeconomically (n=79) advantageous conditions 18
Our Conclusions & Recommendations 19
Our Conclusions The tool gives us immediately visual, easy to interpret results that clearly bring out patterns Knew the user types before we modeled it Very easy to explain to clients or share across professionals With visual mapping, we can: Easily see the entire user experience Determine if unique user types exist See if usability problems exist with certain questions and people Length interactions Placement issues Factual vs attitudinal questions 20
Our Conclusions Provides great alternative for complex statistical investigation May not give you anything useful or helpful Usually empirically driven, so results can change frequently Hard to determine what to ACTUALLY do! A simple and effective way to communicate and examine a surveys effectiveness to clients and other researchers Can overlay respondent behavior with survey design Natural extension of pilot-testing and cognitive testing 21
Our Conclusions Most concerns are with total non-response. But this suggests specific item non-response patterns Allows us to pinpoint the characteristics of those items As well as the people behind that non-response This suggests that item-level non-response adjustments may be necessary if variable is of key interest Weigh option against client interests Complexity can grow exponentially 22
Our Recommendations 1. Everyone knows the value of pre-testing a survey. This emphasizes it and the need for true test conditions. 2. Avoid areas where respondents are forced to engage for long periods 3. The design of a survey are critical and should not be left just to the statisticians! Example: paper surveys and interaction of questions and page location 4. Different users may required different persuasions techniques Incentive levels Customized invitations Survey instructions Different layout 5. Remember a reassessment of your key variables is always a good idea and can uncover significant issues (try this new tool!) 23

More Related Content

Visually Integrative Representation of User Types in Surveys (Ricardo Carvalho & Joseph Luchman & Michael Paraloglou & Vanessa Patterson & Ron Vega)

  • 1. Why Do Respondents Skip Questions in Surveys: A Visually Integrative Representation of User Types Ricardo Carvalho Joseph Luchman Michael Paraloglou Vanessa Patterson Ron Vega 1
  • 2. Outline Background Our Research Our Findings Our Conclusions & Recommendations Future Research 2
  • 4. Background DoD Youth Poll December 2011 survey Mailed to 50,000 youth ages 16 to 24 with no prior or current military experience through stratified, probability-based sampling Address-based sample drawn from list frame estimated to cover 92% of target population Standard mailing methodology (Dillman 2007) Scantron survey; double-entered and verified Up-front and contingent monetary incentives upon completion Response Rate 3: 17%, Contact Rate 2: 92%; n= 7,210 Note: although our study was specific to completing a paper survey, much of the theory (and more importantly, the tool) can be applied to other survey modes and experiences 4
  • 6. Background The Issue # Refusals to Q29 and Q30 Amount of refusals per item was very small 1-18 Refusals (one to many but not all items) (<1.5%) up to Q29 19 Refusals (all items) But at Q30 and thereafter, it increases to 6% 6% 5% about 5-6% 5% 5% 4% Behavior of certain users changes in consistent manner 3% 2% Can we understand the users 1% 1% experience and behavior? 0% Q29 Q30 6
  • 8. Our Research We noticed 370 respondents whose behavior seemed fundamentally different Are these different user types? Or was there a usability issue with the survey (troublesome areas)? How can we identify the user type or a troublesome area? Does this kind of information tell us what to change in the experience? No 8
  • 9. Our Research The behavior we noticed is characteristic of satisficing (Simon 1957) Economic phenomenon: satisfying and sacrificing We exercise an acceptable level of effort to achieve a satisfactory but less than optimal outcome Example: driving around for the cheapest gas price There is substantial literature written on this topic and how it applies to surveys (Krosnick 1991) Behavior points to this phenomenon, but very difficult to be certain The focus of our presentation is NOT on exploring this behavior but on understanding and visualizing different user types How does the survey experience impact users? Are there usability issues we can notice or isolate? Can we build a tool to help improve the overall user experience and hence obtain more complete and accurate information? 9
  • 11. Our Findings What we did: Examined only the 370 respondents who refused all of Q30 Determined if unique user types existed through mixture modeling Wrote code to visually map these user types and their refusals for the remainder of the survey Marked page breaks and grid questions in this map What we found: 3 distinct user types The Quitters The Returners The Completers Map allows us to easily identify these user types Map also allows us to easily identify troublesome areas 11
  • 12. Our Findings Black line = pages The 370 Respondents Orange line = grid question Colored squares = question was ANSWERED
  • 13. Our Findings: The Quitters Black line = pages Orange line = grid question Last page of survey Colored squares = question was ANSWERED Demographics Engagement clearly breaks off and users flip to back of survey Paper survey immediately presents users with workload Completes Demographic questions (essential and easy items) for token of appreciation
  • 14. Our Findings: The Returners Black line = pages Orange line = grid question Colored squares = question was ANSWERED Grid questions Engagement terminates after long second grid question (Q30) but returns Selectively respond to taskful questions (i.e., grid questions) to minimize effort
  • 15. Our Findings: The Completers Black line = pages Orange line = grid question Colored squares = question was ANSWERED Most conscientious and engaged group Engagement terminates for only Q30 Occasional refusals Only a few questions are left unanswered by this user type
  • 16. Our Findings: Profiling the User Types Other Asian Hispanic Asians show more Completers Black Whites show almost all of the Returners White 16
  • 17. Our Findings: Profiling the User Types Mixture model: seeks homogenous distributions within data based on number of questions refused after Q30 Predictive model based on census block sociodemographic data linked to respondent scores Exploratory predictive model (i.e., empirically driven) Hispanic, Native Hawaiian or Pacific Islander Civilian Population in Labor Force Employed; Age 16 Overall Population Median Household Income Population and up Population with less than 9th Grade Education; Age Population in Labor Force Unemployed; Age 16 and Population aged 16-17 Hispanic, Other Population 25 and up up Population with some High School Education; Age Population aged 18-20 Hispanic Population Population not in Labor Force; Age 16 and up 25 and up Population with High School Education; Age 25 and Percent of Population in Labor Force Unemployed; Population aged 21-24 Population in Nursing Home up Age 16 and up Population in other Institutionalized Group Population with some College Education; Age 25 Population employed in Private, for Profit; Age 16 and Median Age Quarters and up up Population Employed in Private, not-for Profit; Age 16 Non-Hispanic, White Population Population in College Dorms Population with Associates Degree; Age 25 and up and up Population Employed in Local Government; Age 16 Non-Hispanic, Black Population Population in Military Barracks Population with Bachelors Degree; Age 25 and up and up Population in Non-Institutionalized Group Population Employed in State Government; Age 16 Non-Hispanic, American Indian Population Population with Masters Degree; Age 25 and up Quarters and up Population Employed in Federal Government; Age 16 Non-Hispanic, Asian Population Average Household Size Population with Professional Degree; Age 25 and up and up Non-Hispanic, Native Hawaiian or Pacific Islander Average Household Size Non-Family Household Population with Doctorate Degree; Age 25 and up Population Self-Employed; Age 16 and up Population Non-Hispanic, Other Population Average Household Size Family Household Families at Poverty Level Population Unpaid Family Work; Age 16 and up Population Speaking only English at Home; Age 5 Population Employed Blue Collar Work; Age 16 and Hispanic, White Population Families at Poverty Level with Children and Older up Population Speaking Spanish at Home; Age 5 and Population Employed White Collar Work; Age 16 and Hispanic, Black Population Families above Poverty Level Older up Population Employed Service and Farm Work; Age 16 Hispanic, American Indian Population Housing Units Owned by Occupant Families above Poverty Level with Children and up Population in Labor Force Employed by Armed Hispanic, Asian Population Housing Units Rented by Occupant Population Male Forces; Age 16 and up Average Length of Residence Population Female 17
  • 18. Our Findings: Profiling the User Types Unemploy % with Median tired of being surveyed! Im Summary of other socio-economic User Type -mentwasting our Bachelors Income The government is variables Rate time/money! Degree Quitters $63,000 7.1% 13.4% Government and private, not-for (n=111) profit employment with large Im doing the best I can, but youre asking a lot household size conditions Returners $58,000 8.5% 10.7% More transient, socioeconomically (n=180) I should do a good job at this, my disadvantageous conditions opinions are helping Completers $61,000 8.1% 11% Less transient, socioeconomically (n=79) advantageous conditions 18
  • 19. Our Conclusions & Recommendations 19
  • 20. Our Conclusions The tool gives us immediately visual, easy to interpret results that clearly bring out patterns Knew the user types before we modeled it Very easy to explain to clients or share across professionals With visual mapping, we can: Easily see the entire user experience Determine if unique user types exist See if usability problems exist with certain questions and people Length interactions Placement issues Factual vs attitudinal questions 20
  • 21. Our Conclusions Provides great alternative for complex statistical investigation May not give you anything useful or helpful Usually empirically driven, so results can change frequently Hard to determine what to ACTUALLY do! A simple and effective way to communicate and examine a surveys effectiveness to clients and other researchers Can overlay respondent behavior with survey design Natural extension of pilot-testing and cognitive testing 21
  • 22. Our Conclusions Most concerns are with total non-response. But this suggests specific item non-response patterns Allows us to pinpoint the characteristics of those items As well as the people behind that non-response This suggests that item-level non-response adjustments may be necessary if variable is of key interest Weigh option against client interests Complexity can grow exponentially 22
  • 23. Our Recommendations 1. Everyone knows the value of pre-testing a survey. This emphasizes it and the need for true test conditions. 2. Avoid areas where respondents are forced to engage for long periods 3. The design of a survey are critical and should not be left just to the statisticians! Example: paper surveys and interaction of questions and page location 4. Different users may required different persuasions techniques Incentive levels Customized invitations Survey instructions Different layout 5. Remember a reassessment of your key variables is always a good idea and can uncover significant issues (try this new tool!) 23