際際滷

際際滷Share a Scribd company logo
Designing Surveys to Determine Impact of Online Social Networks on Students Social and Academic Interactions Nicola Ritter, M.Ed. Data Analyst, Student Life Studies  @nicolalritter  #naspatech8
Take Away Points Outline the facets of survey development Pose areas to explore in online social networks (OSN) Engage in the survey construction process
I. Tailored Design Dillman, Smyth, & Christian (2009) Reduces Survey Error Develops procedures that are respondent-centered Includes variety of features that build positive social exchange
1. Reducing Survey Error   Dillman, Smyth, & Christian (2009) Coverage Error  when some members of the population are excluded AND those excluded are different from those included  (e.g. students with text messaging vs. students without text messaging) Sampling Error  surveying only a portion of the population  (e.g. full time vs. part-time students)  (ie. Ravert, Calix, & Sullivan (2010)
1. Reducing Survey Error  Dillman, Smyth, & Christian (2009) Non-response Error  respondents different from non-respondents  (e.g. missing data) Measurement Error  inaccurate or imprecise responses  (e.g. item construction) (ie. Ravert, Calix, & Sullivan (2010))
2. Develops procedures that are respondent-centered  Dillman, Smyth, & Christian (2009) Increase the Benefits of Participation Decrease Cost of Participation Establish Trust
Increase the Benefits of Participation Provide information about the survey Ask for help or advise Show positive regard Say thank you Support group values Give tangible rewards Make the questionnaire interesting Provide social validation Inform people that opportunities to respond are limited
Decrease Cost of Participation Make it convenient to respond Avoid subordinating language Make the questionnaire short and easy to complete Minimize requests to obtain personal or sensitive information Emphasize similarity to other requests or task to which a person has already responded
Establish Trust Obtain sponsorship by legitimate authority Provide a token of appreciation in advance Make the task appear important Ensure confidentiality and security of information
3. Includes variety of features that build positive social exchange Dillman, Smyth, & Christian (2009) Outside the Survey Invitations Reminder s Thank yous Follow-up that offers results Inside the Survey Clickable graphics Analog Scales
II. Research Questions: Beyond Usage (Ritter, in review) Insight to student perspectives  (Martinez Aleman & Wartman, 2009) Social and Academic Interactions Student/Student, Student/Faculty-Staff Interactions  (Kord & Wolf-Wendel, 2009; Teclehaimanot & Hickman, 2011) Student/Organization Interactions Impact of New Features  (ie. Groups/Circles, geo-location tools) Privacy  (Eberhardt, 2005; Eberhardt, 2006; Koleck & Saunders, 2008; Kirschner & Karpinski, 2010)  Student Affair Professionals Usage Students Organization Other
Brainstorm Topic/subtopics
Overall Survey Construction What survey mode(s) will be used to ask the questions? Is this question being repeated from another survey, and/or will answers be compared to previously collected data? Will respondents be willing and motivated to answer accurately? What type of information is the question asking for?
Guidelines for Word  Choice & Question Format  (Dillman, et. al. (2009) Make sure the question applies to the respondent. Make sure the question is technically accurate. Ask one question at a time. Use similar and familiar words Use specific and concrete words to specify the concepts clearly.
Guidelines for Word  Choice & Question Format  (Dillman, et. al. (2009) Use as few words as possible to pose the question. Use complete sentences with simple sentence structures. Make sure yes means yes and no means no. Be sure the question specifies the response task.
1. Make sure the question applies to the respondent. A question that does not require an answer from every respondent If you have a Facebook profile, about how many hours per day do you spend on Facebook? hours per day
1. Make sure the question applies to the respondent. A question that uses a filter Do you have a Facebook profile? Yes  No About how many hours per day do  you spend on Facebook?   hours per day
2. Make sure the question is technically accurate. Which Facebook features/activities listed below do you use? (Check all that apply) Mini-feed News feed Check-in Posting on other peoples walls Searching for people
2. Make sure the question is technically accurate. Which Facebook features listed below do you use? (Check all that apply) News feed Check-in on Places Wall-to-Wall Search bar
3. Ask one question at a time. Two questions in one From which one of these sources did you first learn about  A&M moving to the SEC? University website Facebook Twitter Someone at work While traveling to campus
3. Ask one question at a time. From which one of these sources did you first learn about  A&M moving to the SEC? University website Facebook Twitter Radio TexAgs Another person Where were you when you first heard about it? At work At home Traveling to work Somewhere else:
4. Use similar and familiar words From which one of these sources did you first learn about  A&M moving to the SEC? University website Facebook Twitter Radio TexAgs Another person
4. Use similar and familiar words From which one of these sources did you first learn about  Texas A&M moving to the Southeastern Conference? University website Facebook Twitter Radio TexAgs Another person
5. Use specific and concrete words to specify the concepts clearly. How many times did you log on to Facebook as a student affairs professional last week? # of times
5. Use specific and concrete words to specify the concepts clearly. How many times did you log on to Facebook as a student affairs professional last week? # of times To work with students
5. Use specific and concrete words to specify the concepts clearly. How many times did you log on to Facebook as a student affairs professional to connect with students last week? # of times
6. Use as few words as possible to pose the question. According to Strauss and Howe (1992), the United States is made up of different generations. Based on their definitions, which generation do you belong to? Silent Generation (born in years 1924 through 1942) Boom Generation (born in years 1943 through 1960) Generation X (born in years 1961 through 1981) Millennial Generation (born in years 1982 through 2002)
6. Use as few words as possible to pose the question. Which age range were you born?  1924 - 1942 1943 -1960 1961 - 1981 1982 - 2002 What year were you born?
7. Use complete sentences with simple sentence structures. Number of years lived in Texas Years Your city or town City or Town
7. Use complete sentences with simple sentence structures. Number of years lived in Texas Years Your city or town City or Town About 25% of respondents listed the # of years they lived in the city or town
7. Use complete sentences with simple sentence structures. How many years have you lived in Texas? Years In what city or town do you live? Name of City or Town
8. Make sure yes means yes and no means no. A question with a double negative Do you favor or oppose not allowing Texas A&M to use information from Facebook and Twitter during proceedings? Favor Oppose
8. Make sure yes means yes and no means no. Revision with no double negative Do you favor or oppose allowing Texas A&M to use information from Facebook and Twitter during proceedings? Favor Oppose
8. Make sure yes means yes and no means no. Revision with that preserves important wording In the upcoming student elections, you will be asked to vote on this referendum: [Insert referendum] If the election were held today, would you vote for or against approval? For Against
9. Be sure the question specifies the response task. A question specifying different tasks How many days a week do you log in to Facebook? More than once a day Once a day A few times a week Once every other week Once a month Less than once a month
9. Be sure the question specifies the response task. Revision How many days per week do you log in to Facebook? days per week
References Dillman, D. A., Smyth, J.D., & Christian, L. M., (2009).  Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method  (3 rd  ed.). Hoboken, NJ: John Wiley and Sons, Inc. Eberhardt, D. (2005). Should institutions respect students on-line privacy of facebook?.  Journal of College and Character, 6 (7). Eberhardt, D. (2006). The facebook/myspace era: A help or hinderance to college students and administrators?.  Journal of College and Character, 7 (7). Kirschner, P. A., and Karpinski, A. C. (2010). Facebook and academic performance,  Computers in Human Behavior, 26 (6), 1237-1245. doi: 10.1016/j.chb.2010.03.024 Kolek, E. A. & Saunders, D. (2008). Online disclosure: An empirical examination of undergraduate facebook profiles.  NASPA Journal, 45  (1), 1-25. Kord, J., & Wolf-Wendel, L. (2009). The relationship between online social networking and academic and social integration.  College Student Affairs Journal, 28 (1), 103. Martinez-Aleman, A. M. and Wartman, K. L.  Online Social Networking on Campus: Understanding What Matters in Student Culture , Routledge, New York, NY, 2009. Ravert, R. D., Calix, S. I.,  Sullivan, M. J. (2010). Research in brief: Using mobile phones to collect daily experience data from college undergraduates.  Journal of College Student Development , 51 (3) 343-352. Ritter, N. (in review). Journal sponsored by Student Affairs Administrators in Higher Education. Teclehaimanot, B., & Hickman, T. (2011). Student-teacher interaction on facebook: What students find appropriate.  TechTrends: Linking Research and Practice to Improve Learning, 55 (3), 19-30.

More Related Content

Designing Surveys to Determine the Impact of Online Social Networks #naspatech8

  • 1. Designing Surveys to Determine Impact of Online Social Networks on Students Social and Academic Interactions Nicola Ritter, M.Ed. Data Analyst, Student Life Studies @nicolalritter #naspatech8
  • 2. Take Away Points Outline the facets of survey development Pose areas to explore in online social networks (OSN) Engage in the survey construction process
  • 3. I. Tailored Design Dillman, Smyth, & Christian (2009) Reduces Survey Error Develops procedures that are respondent-centered Includes variety of features that build positive social exchange
  • 4. 1. Reducing Survey Error Dillman, Smyth, & Christian (2009) Coverage Error when some members of the population are excluded AND those excluded are different from those included (e.g. students with text messaging vs. students without text messaging) Sampling Error surveying only a portion of the population (e.g. full time vs. part-time students) (ie. Ravert, Calix, & Sullivan (2010)
  • 5. 1. Reducing Survey Error Dillman, Smyth, & Christian (2009) Non-response Error respondents different from non-respondents (e.g. missing data) Measurement Error inaccurate or imprecise responses (e.g. item construction) (ie. Ravert, Calix, & Sullivan (2010))
  • 6. 2. Develops procedures that are respondent-centered Dillman, Smyth, & Christian (2009) Increase the Benefits of Participation Decrease Cost of Participation Establish Trust
  • 7. Increase the Benefits of Participation Provide information about the survey Ask for help or advise Show positive regard Say thank you Support group values Give tangible rewards Make the questionnaire interesting Provide social validation Inform people that opportunities to respond are limited
  • 8. Decrease Cost of Participation Make it convenient to respond Avoid subordinating language Make the questionnaire short and easy to complete Minimize requests to obtain personal or sensitive information Emphasize similarity to other requests or task to which a person has already responded
  • 9. Establish Trust Obtain sponsorship by legitimate authority Provide a token of appreciation in advance Make the task appear important Ensure confidentiality and security of information
  • 10. 3. Includes variety of features that build positive social exchange Dillman, Smyth, & Christian (2009) Outside the Survey Invitations Reminder s Thank yous Follow-up that offers results Inside the Survey Clickable graphics Analog Scales
  • 11. II. Research Questions: Beyond Usage (Ritter, in review) Insight to student perspectives (Martinez Aleman & Wartman, 2009) Social and Academic Interactions Student/Student, Student/Faculty-Staff Interactions (Kord & Wolf-Wendel, 2009; Teclehaimanot & Hickman, 2011) Student/Organization Interactions Impact of New Features (ie. Groups/Circles, geo-location tools) Privacy (Eberhardt, 2005; Eberhardt, 2006; Koleck & Saunders, 2008; Kirschner & Karpinski, 2010) Student Affair Professionals Usage Students Organization Other
  • 13. Overall Survey Construction What survey mode(s) will be used to ask the questions? Is this question being repeated from another survey, and/or will answers be compared to previously collected data? Will respondents be willing and motivated to answer accurately? What type of information is the question asking for?
  • 14. Guidelines for Word Choice & Question Format (Dillman, et. al. (2009) Make sure the question applies to the respondent. Make sure the question is technically accurate. Ask one question at a time. Use similar and familiar words Use specific and concrete words to specify the concepts clearly.
  • 15. Guidelines for Word Choice & Question Format (Dillman, et. al. (2009) Use as few words as possible to pose the question. Use complete sentences with simple sentence structures. Make sure yes means yes and no means no. Be sure the question specifies the response task.
  • 16. 1. Make sure the question applies to the respondent. A question that does not require an answer from every respondent If you have a Facebook profile, about how many hours per day do you spend on Facebook? hours per day
  • 17. 1. Make sure the question applies to the respondent. A question that uses a filter Do you have a Facebook profile? Yes No About how many hours per day do you spend on Facebook? hours per day
  • 18. 2. Make sure the question is technically accurate. Which Facebook features/activities listed below do you use? (Check all that apply) Mini-feed News feed Check-in Posting on other peoples walls Searching for people
  • 19. 2. Make sure the question is technically accurate. Which Facebook features listed below do you use? (Check all that apply) News feed Check-in on Places Wall-to-Wall Search bar
  • 20. 3. Ask one question at a time. Two questions in one From which one of these sources did you first learn about A&M moving to the SEC? University website Facebook Twitter Someone at work While traveling to campus
  • 21. 3. Ask one question at a time. From which one of these sources did you first learn about A&M moving to the SEC? University website Facebook Twitter Radio TexAgs Another person Where were you when you first heard about it? At work At home Traveling to work Somewhere else:
  • 22. 4. Use similar and familiar words From which one of these sources did you first learn about A&M moving to the SEC? University website Facebook Twitter Radio TexAgs Another person
  • 23. 4. Use similar and familiar words From which one of these sources did you first learn about Texas A&M moving to the Southeastern Conference? University website Facebook Twitter Radio TexAgs Another person
  • 24. 5. Use specific and concrete words to specify the concepts clearly. How many times did you log on to Facebook as a student affairs professional last week? # of times
  • 25. 5. Use specific and concrete words to specify the concepts clearly. How many times did you log on to Facebook as a student affairs professional last week? # of times To work with students
  • 26. 5. Use specific and concrete words to specify the concepts clearly. How many times did you log on to Facebook as a student affairs professional to connect with students last week? # of times
  • 27. 6. Use as few words as possible to pose the question. According to Strauss and Howe (1992), the United States is made up of different generations. Based on their definitions, which generation do you belong to? Silent Generation (born in years 1924 through 1942) Boom Generation (born in years 1943 through 1960) Generation X (born in years 1961 through 1981) Millennial Generation (born in years 1982 through 2002)
  • 28. 6. Use as few words as possible to pose the question. Which age range were you born? 1924 - 1942 1943 -1960 1961 - 1981 1982 - 2002 What year were you born?
  • 29. 7. Use complete sentences with simple sentence structures. Number of years lived in Texas Years Your city or town City or Town
  • 30. 7. Use complete sentences with simple sentence structures. Number of years lived in Texas Years Your city or town City or Town About 25% of respondents listed the # of years they lived in the city or town
  • 31. 7. Use complete sentences with simple sentence structures. How many years have you lived in Texas? Years In what city or town do you live? Name of City or Town
  • 32. 8. Make sure yes means yes and no means no. A question with a double negative Do you favor or oppose not allowing Texas A&M to use information from Facebook and Twitter during proceedings? Favor Oppose
  • 33. 8. Make sure yes means yes and no means no. Revision with no double negative Do you favor or oppose allowing Texas A&M to use information from Facebook and Twitter during proceedings? Favor Oppose
  • 34. 8. Make sure yes means yes and no means no. Revision with that preserves important wording In the upcoming student elections, you will be asked to vote on this referendum: [Insert referendum] If the election were held today, would you vote for or against approval? For Against
  • 35. 9. Be sure the question specifies the response task. A question specifying different tasks How many days a week do you log in to Facebook? More than once a day Once a day A few times a week Once every other week Once a month Less than once a month
  • 36. 9. Be sure the question specifies the response task. Revision How many days per week do you log in to Facebook? days per week
  • 37. References Dillman, D. A., Smyth, J.D., & Christian, L. M., (2009). Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method (3 rd ed.). Hoboken, NJ: John Wiley and Sons, Inc. Eberhardt, D. (2005). Should institutions respect students on-line privacy of facebook?. Journal of College and Character, 6 (7). Eberhardt, D. (2006). The facebook/myspace era: A help or hinderance to college students and administrators?. Journal of College and Character, 7 (7). Kirschner, P. A., and Karpinski, A. C. (2010). Facebook and academic performance, Computers in Human Behavior, 26 (6), 1237-1245. doi: 10.1016/j.chb.2010.03.024 Kolek, E. A. & Saunders, D. (2008). Online disclosure: An empirical examination of undergraduate facebook profiles. NASPA Journal, 45 (1), 1-25. Kord, J., & Wolf-Wendel, L. (2009). The relationship between online social networking and academic and social integration. College Student Affairs Journal, 28 (1), 103. Martinez-Aleman, A. M. and Wartman, K. L. Online Social Networking on Campus: Understanding What Matters in Student Culture , Routledge, New York, NY, 2009. Ravert, R. D., Calix, S. I., Sullivan, M. J. (2010). Research in brief: Using mobile phones to collect daily experience data from college undergraduates. Journal of College Student Development , 51 (3) 343-352. Ritter, N. (in review). Journal sponsored by Student Affairs Administrators in Higher Education. Teclehaimanot, B., & Hickman, T. (2011). Student-teacher interaction on facebook: What students find appropriate. TechTrends: Linking Research and Practice to Improve Learning, 55 (3), 19-30.

Editor's Notes

  • #5: Coverage error when not all members of the population have a known, non-zero chance of being included in the sample and when those excluded are different from those who are included on measures of interest Sampling error transfer students vs non-transfer students; leads to re-defining the population; lack of random sampling -> decreases generalizability
  • #6: Non-response error solutions forced responses or offering an option prefer not to answer
  • #12: Online social networks have become a part of everyday life to the majority of college students (Texas A&M University Student Life Studies, 2009; Kord and Wolf-Wendel, 2009; Texas A&M University Student Life Studies, 2011). Online social networks such as Facebook offer users a medium to stay connected to individuals and groups around the clock. As Facebook approaches a decade of social connectivity and as popularity continues to grow with later generations of students, higher education needs to have a better understanding of online social networks. While online social media is seen in headlines, little research has been conducted on the impact of online social networks on students academic and social interaction in higher education. It is not clear whether social media is a help or a hindrance to college students and higher education (Teclehaimanot and Hickman, 2011; Munoz & Strotmeyer, 2010; Eberhardt, 2006). In addition, the dearth of literature that is available neglects student perspectives (Martinez Aleman and Wartman, 2009). Although college students are using this media to stay connected with peers, few students interact (Kolek & Saunders, 2008) or feel comfortable (Teclehaimanot, et. al, 2011) interacting with campus faculty and staff. Kord & Wolf-Wendel (2009) found that as students spend more time with online social networks the less students perceived faculty were concerned about student development and teaching and were less likely to positively perceive growth in their academic and intellectual development. These findings are of particular concern because these variables have been associated with student persistence (Tinto, 1993; Pacarella & Terenzini, 1980; Pacarella & Terenzini, 1998). Since this study provides perspectives of freshman on campus, it is not clear if online social networking has the same impact on sophomores, juniors, and seniors. Future studies should examine the relationship between online social networking and undergraduate perceptions of academic and social integration to see if perceptions of academic interactions differ across student classification. Such a study would provide information on whether or not online social networking is a help or hindrance to students academic and social integration on campus. Another concern is student awareness of privacy in online social networks (Koleck, et. al, 2008; Eberhardt, 2006; Ebderhardt, 2005). Koleck, et. al (2008) found only 11% of student profiles had restricted access so that researchers could not view the students profile, while women were more likely than men to disclose personal information. Further studies should investigate whether or not universities are seeing an increase in student Facebook usage as well as explore students perspectives on disclosing information on Facebook. As new features are added to social media site, student usage of these features and their impact on student identities and privacy also need to be investigated.
  • #13: Objective: Attendees will discuss their own campus research questions regarding social media.
  • #14: Objective: Attendees will determine poor items given a sample survey and pilot study data. Objective: Attendees will propose suggested revisions to poor items.
  • #19: Facebook jargon should be clear to respondents Mini-feed not clear on which feature this choice is referring to. Check-in application called Places Actions vs. features Posting on other peoples walls could be changed to Wall-to-Wall feature Searching for people search tool bar