How A/B Tests Lie to Us and How to Drive Genuine Improvement
0 likes665 views
Smart marketers know that A/B testing helps take the guesswork out of website optimizations. Often times though, A/B testing doesn't give you a true view of how your site or web page is performing. In this presentation, WP Engine Founder & CTO Jason Cohen covers why tools that show statistical significance are often wrong, and how you can correct it, how to use insights from Google Analytics to drive A/B tests and what elements of your marketing campaigns should be A/B tested.
How A/B Tests Lie to Us and How to Drive Genuine Improvement
- 1. SIX PUZZLES How A/B tests lie to us, and how to drive genuine improvement Jason Cohen, founder & CTO http://wpengine.com
- 3. 654 out of 24,483 visits 85% chance-to-beat
- 4. What do stats really tell us?
- 6. 654 out of 24,483 visits 85% chance-to-beat
- 7. Finding the Cheaters 105 athletes. 100 are clean. 5 are juicing. Test is 95% accurate.
- 8. Finding the Cheaters IN REALITY Clean (100) Juicing (5) TEST RESULT (95% accuracy) Clean Juicing 5 095 5
- 9. Insight: Rare events are indistinguishable from random noise
- 10. 758 out of 24,483 visits 85% chance-to-beat
- 11. Easy significance testing with hamsters Trust in the Hamster. bit.ly/abhamster
- 12. Easy significance testing N = A+B = total # of conversions. D = (AB)/2 = half the difference. Significant if D2 > N. bit.ly/abhamster N = 758 + 685 = 1443 D = ( 758 685 ) / 2 = 37 1369 > 1443? No! Not significant.
- 13. Final Result: As predicted Wasted another month
- 14. 30 40 50 60 70 12/2/09 12/16/09 1/5/10 1/13/10 2/2/10 4/9/10 4/27/10 9/24/10 9/14/11 11/8/11 OtherInbox Splash Page
- 16. Lesson #1 Beware the Rare
- 17. Lesson #2 Trust the Hamster
- 19. 41 Shades of Blue
- 20. 41 Shades of Blue Does this... ...get more clicks than this?
- 21. 2 Shades of Blue with 1 winner at a 95% confidence level, chance of a false-positive: 5%
- 22. 3 Shades of Blue with 1 winner at a 95% confidence level, chance of a false-positive: 9%
- 23. 10 Shades of Blue with 1 winner at a 95% confidence level, chance of a false-positive: 40%
- 24. 41 Shades of Blue with 1 winner at a 95% confidence level, chance of a false-positive: 88%
- 26. Oops, youre doing this.
- 28. Lesson #3 Test theories, not headlines
- 30. Bible Code!
- 31. Bible Code!
- 33. War and Peace
- 35. GA reveals ideas, not facts
- 36. Lesson #4 Exploration yields new theories, not new truth.
- 38. Headwinds to improvement significance huge N N/A high failure rate many tests false-positives rare events worse on both!
- 39. Summary: Subtly is the Enemy
- 40. Insight: You dont want Subtle anyway!
- 41. Lesson #5 Seek Huge Effects
- 42. Can you A/B test your way to big results?
- 44. Wisdom of the Crowd Im as clueless as you, but in the equal and opposite direction.
- 45. Whats the funniest joke?
- 46. And the winner 庄壊
- 47. Jokes, Pizza, Design, Features, Voice, Values, Culture
- 48. Lesson #6 Crowds find objective truth but kill creativity and greatness
- 49. Think for yourself. Test theories. Seek big effects. Jason Cohen, founder & CTO http://wpengine.com
- 50. Register now for our next webinar! Your Workflow, Your Way, with WP Engine Register now: http://wpeng.in/ywyw
- 51. Q & A Jason Cohen, founder & CTO http://wpengine.com