際際滷

際際滷Share a Scribd company logo

Web panel surveys patrick sturgis

Alf Fyhrlund
Alf Fyhrlund

The document summarizes findings from an inquiry into inaccurate polling leading up to the 2015 UK general election. The inquiry examined potential causes of error, including late swings in voter preference, unrepresentative samples, turnout misreporting, and how polls treated undecided voters. The inquiry found little evidence that late swings or turnout modeling affected results. Rather, polls likely suffered from sampling biases, as their samples underrepresented older, less-educated groups who favored the Conservatives. A similar pattern of biased samples emerged in EU referendum polls.

1 of 42
Download to read offline
Quality of Internet Surveys: what can we learn from election polling? Patrick Sturgis Web Panel Surveys, Methods and Experiences Statistics Sweden, 2 November 2016
What Ill talk about The 2015 polling disaster History of polling (in)accuracy (GB) The 2015 Inquiry Hypotheses Evidence Conclusions Herding (if time) EU referendum polls How random probability surveys fared 2
3
The final polls 4 Pollster Mode Fieldwork n Con Lab Lib UKIP Green Other Populus O 56 May 3,917 34 34 9 13 5 6 Ipsos-MORI P 56 May 1,186 36 35 8 11 5 5 YouGov O 46 May 10,30 7 34 34 10 12 4 6 ComRes P 56 May 1,007 35 34 9 12 4 6 Survation O 46 May 4,088 31 31 10 16 5 7 ICM P 36 May 2,023 34 35 9 11 4 7 Panelbase O 16 May 3,019 31 33 8 16 5 7 Opinium O 45 May 2,960 35 34 8 12 6 5 TNS UK O 30/44/5 1,185 33 32 8 14 6 6 Ashcroft* P 56 May 3,028 33 33 10 11 6 8 BMG* O 35 May 1,009 34 34 10 12 4 6 SurveyMonkey* O 30/4-6/5 18,13 1 34 28 7 13 8 9 Result 37.8 31.2 8.1 12.9 3.8 6.3
Election Result v Average of final Polls (GB) 34 33 13 8 5 7 37,8 31,2 12,9 8,1 3,8 6,3 0 5 10 15 20 25 30 35 40 Tory Labour UKIP Lib Dem Green Other V o t e S h a r e % Party
Had the forecasts been different, then the nightly news bulletins would surely have concentrated rather more on the vast spending cuts to come, and rather less on the potential role of Scottish nationalists in a hung parliament. That might have influenced the result. Editorial 14/5/15
Web panel surveys patrick sturgis
2-month moving average poll vote intention estimates 2010-2015 8
Some history 9
Error on Conservative vote share 10
Error on Labour vote share 11
Error on Con/Lab lead 12
Frequency of GB Polls 1940-2015 0 50 100 150 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015 13 N of election polls 1945-2010 = 3,500 N of election polls 2010-2015 = 1,942
Methodology of the polls All polls were done using phone or online methods 1. Non-random recruitment + match sample to population totals (e.g. age, sex, region, social grade, past vote) 2. Allocation of Dont Knows/refusals 3. Apply turnout weight 14
Quota sampling methodology For quota sampling to work, following conditions should by met: 1. Within levels of the quota/weighting variables, sample should have the same vote intention as same group in population 2. Assigned probabilities of turnout should be accurate 3. Respondents stated vote intentions should agree with how they actually voted 4. Treatment of DK/refusals should be accurate
Evidence Considered Three polls from each BPC pollster 1st poll of short campaign Penultimate poll Final poll Plus re-contact surveys, if undertaken Contemporaneous probability surveys British Election Study British Social Attitudes survey All published estimates replicated using micro-data 16
Published estimates: Con-Lab difference 17
Whodunnit?
The suspects Late swing
The suspects Late swing Sampling/weighting
The suspects Late swing Sampling/weighting Turnout misreporting
The suspects Late swing Sampling/weighting Turnout misreporting Dont Know/refusal
The suspects Dont Know/refusal Late swing Sampling/weighting Turnout misreporting Question wording
The suspects Dont Know/refusal Late swing Sampling/weighting Turnout misreporting Question wording Registration/postal
Final polls: Con-Lab after different weightings 25
Unlikely to have had an effect Postal voting Voter registration Overseas voters Question wording/framing Turnout weighting Deliberate misreporting 26
Late swing Main evidence from post-election re-contact polls, where respondents of pre-election polls interviewed after election Compare reported vote pre and post election No strong evidence of late swing to Conservatives
Turnout weighting Different types of evidence on the effects of turnout weighting: Pre-election vote intention using known voters in re-contact polls Modelling validated turnout by party*ltv Sensitivity of estimates to specifications of turnout probabilities None of these makes a notable difference 28
Reported vote before and after election (known voters)
Difference in Con lead phone-online2011 2012 2013 2014 2015 2016 10 Date Online Telephone 2011 2012 2013 2014 2015 2016 42024 Mode of Data Collection Difference Date PhoneOnline(ConLabMargin) Final Polls
Unrepresentative samples 31
Final polls vs. Post-election surveys 32
Reported vote before and after election (known voters) 33
Conservative lead by age group, polls v BES/BSA 34
Conservative lead by GB region, polls v election result 35
Age among those aged 65+ (three polls) 36
self-reported 2010 turnout by age band (polls v BES) 37
Early-call respondents in BES and BSA
EU Referendum polls 44
The final EU Ref polls Fieldwork Sample Remain Leave MAE ORB 1419 June 877 54 46 5.9 Survation 20 June 1003 51 49 2.9 ComRes 17-22 June 1032 54 46 5.9 Opinium 20-22 June 3011 49 51 0.9 YouGov 20-23 June 3766 51 49 2.9 Ipsos MORI 21-22 June 1592 52 48 3.9 Populus 21-22 June 4740 55 45 6.9 TNS* 16-22 June 2320 48.8 51.2 0.7 Result 48 52 Average MAE 3.8 *TNS did not pre-announce as their final poll, so were not included in the British Polling Council list of final polls (where the average error is 4.3).
-15 -10 -5 0 5 10 15 14-April 28-April 12-May 26-May 9-June 23-June Online Telephone Online vs. phone
Random probability estimates Survey Mode/fieldwork RR Remain % Leave % ICM Face-to-Face 1-19 June 60% 54% 46% NatCen Online/phone 16 May -12 June 19% 53% 47% ??? Face-to-Face 27 May-19 June ~60% 55% 45% 47

More Related Content

Web panel surveys patrick sturgis

  • 1. Quality of Internet Surveys: what can we learn from election polling? Patrick Sturgis Web Panel Surveys, Methods and Experiences Statistics Sweden, 2 November 2016
  • 2. What Ill talk about The 2015 polling disaster History of polling (in)accuracy (GB) The 2015 Inquiry Hypotheses Evidence Conclusions Herding (if time) EU referendum polls How random probability surveys fared 2
  • 3. 3
  • 4. The final polls 4 Pollster Mode Fieldwork n Con Lab Lib UKIP Green Other Populus O 56 May 3,917 34 34 9 13 5 6 Ipsos-MORI P 56 May 1,186 36 35 8 11 5 5 YouGov O 46 May 10,30 7 34 34 10 12 4 6 ComRes P 56 May 1,007 35 34 9 12 4 6 Survation O 46 May 4,088 31 31 10 16 5 7 ICM P 36 May 2,023 34 35 9 11 4 7 Panelbase O 16 May 3,019 31 33 8 16 5 7 Opinium O 45 May 2,960 35 34 8 12 6 5 TNS UK O 30/44/5 1,185 33 32 8 14 6 6 Ashcroft* P 56 May 3,028 33 33 10 11 6 8 BMG* O 35 May 1,009 34 34 10 12 4 6 SurveyMonkey* O 30/4-6/5 18,13 1 34 28 7 13 8 9 Result 37.8 31.2 8.1 12.9 3.8 6.3
  • 5. Election Result v Average of final Polls (GB) 34 33 13 8 5 7 37,8 31,2 12,9 8,1 3,8 6,3 0 5 10 15 20 25 30 35 40 Tory Labour UKIP Lib Dem Green Other V o t e S h a r e % Party
  • 6. Had the forecasts been different, then the nightly news bulletins would surely have concentrated rather more on the vast spending cuts to come, and rather less on the potential role of Scottish nationalists in a hung parliament. That might have influenced the result. Editorial 14/5/15
  • 8. 2-month moving average poll vote intention estimates 2010-2015 8
  • 10. Error on Conservative vote share 10
  • 11. Error on Labour vote share 11
  • 12. Error on Con/Lab lead 12
  • 13. Frequency of GB Polls 1940-2015 0 50 100 150 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015 13 N of election polls 1945-2010 = 3,500 N of election polls 2010-2015 = 1,942
  • 14. Methodology of the polls All polls were done using phone or online methods 1. Non-random recruitment + match sample to population totals (e.g. age, sex, region, social grade, past vote) 2. Allocation of Dont Knows/refusals 3. Apply turnout weight 14
  • 15. Quota sampling methodology For quota sampling to work, following conditions should by met: 1. Within levels of the quota/weighting variables, sample should have the same vote intention as same group in population 2. Assigned probabilities of turnout should be accurate 3. Respondents stated vote intentions should agree with how they actually voted 4. Treatment of DK/refusals should be accurate
  • 16. Evidence Considered Three polls from each BPC pollster 1st poll of short campaign Penultimate poll Final poll Plus re-contact surveys, if undertaken Contemporaneous probability surveys British Election Study British Social Attitudes survey All published estimates replicated using micro-data 16
  • 20. The suspects Late swing Sampling/weighting
  • 21. The suspects Late swing Sampling/weighting Turnout misreporting
  • 22. The suspects Late swing Sampling/weighting Turnout misreporting Dont Know/refusal
  • 23. The suspects Dont Know/refusal Late swing Sampling/weighting Turnout misreporting Question wording
  • 24. The suspects Dont Know/refusal Late swing Sampling/weighting Turnout misreporting Question wording Registration/postal
  • 25. Final polls: Con-Lab after different weightings 25
  • 26. Unlikely to have had an effect Postal voting Voter registration Overseas voters Question wording/framing Turnout weighting Deliberate misreporting 26
  • 27. Late swing Main evidence from post-election re-contact polls, where respondents of pre-election polls interviewed after election Compare reported vote pre and post election No strong evidence of late swing to Conservatives
  • 28. Turnout weighting Different types of evidence on the effects of turnout weighting: Pre-election vote intention using known voters in re-contact polls Modelling validated turnout by party*ltv Sensitivity of estimates to specifications of turnout probabilities None of these makes a notable difference 28
  • 29. Reported vote before and after election (known voters)
  • 30. Difference in Con lead phone-online2011 2012 2013 2014 2015 2016 10 Date Online Telephone 2011 2012 2013 2014 2015 2016 42024 Mode of Data Collection Difference Date PhoneOnline(ConLabMargin) Final Polls
  • 32. Final polls vs. Post-election surveys 32
  • 33. Reported vote before and after election (known voters) 33
  • 34. Conservative lead by age group, polls v BES/BSA 34
  • 35. Conservative lead by GB region, polls v election result 35
  • 36. Age among those aged 65+ (three polls) 36
  • 37. self-reported 2010 turnout by age band (polls v BES) 37
  • 40. The final EU Ref polls Fieldwork Sample Remain Leave MAE ORB 1419 June 877 54 46 5.9 Survation 20 June 1003 51 49 2.9 ComRes 17-22 June 1032 54 46 5.9 Opinium 20-22 June 3011 49 51 0.9 YouGov 20-23 June 3766 51 49 2.9 Ipsos MORI 21-22 June 1592 52 48 3.9 Populus 21-22 June 4740 55 45 6.9 TNS* 16-22 June 2320 48.8 51.2 0.7 Result 48 52 Average MAE 3.8 *TNS did not pre-announce as their final poll, so were not included in the British Polling Council list of final polls (where the average error is 4.3).
  • 41. -15 -10 -5 0 5 10 15 14-April 28-April 12-May 26-May 9-June 23-June Online Telephone Online vs. phone
  • 42. Random probability estimates Survey Mode/fieldwork RR Remain % Leave % ICM Face-to-Face 1-19 June 60% 54% 46% NatCen Online/phone 16 May -12 June 19% 53% 47% ??? Face-to-Face 27 May-19 June ~60% 55% 45% 47