Practical Introduction to A/B Testing
Download as PPTX, PDF1 like800 views
A/B testing involves showing different versions of a digital product or interface to different user groups in order to determine which version performs better according to predefined metrics. This document provides an overview of how to set up an A/B test, including defining the user funnel, designing test variants, setting appropriate metrics, filtering traffic, collecting statistically significant data, and determining when a test yields a significant result. The goal of A/B testing is to observe user behavior objectively in order to optimize conversions, engagement, or other goals through datadriven experimentation.
More Related Content
What's hot (20)
Similar to Practical Introduction to A/B Testing (20)
Recently uploaded (20)
Practical Introduction to A/B Testing
- 1. Practical Introduction to A/B Testing Alex Alwan
- 2. Alex Alwan Sn. Product manager / Consultant Worked at Rackspace, Airbrake, and Spark Networks Over 8 years of experience in product management Data driven Interested in growth strategies, data science, and Clash Royale
- 3. What is A/B testing?
- 8. A/B testing is observing what people do, not what they say.
- 9. Where do we start?
- 10. Start With The User Funnel Homepage visits Exploring the site Register Make payment Clicked ad Landing page visits Register Make payment
- 11. User Funnel - SaaS Product Homepage visits Pricing page visits Select plan Make payment Register / Setup Repeat
- 12. SaaS Product - Airbrake.io
- 13. SaaS Product - Airbrake.io
- 14. SaaS Product - Airbrake.io
- 15. Airbrake.io - Define User Funnel Homepage visits Pricing page visits Select plan Make payment Register / Setup Repeat 1,000 900 500 200 5040
- 16. Make payment Homepage visits Pricing page visits Select plan Register / Setup Repeat 1,000 900 500 200 5040 90% 55.56% 40% 25%80% Airbrake.io - Calculate Conversion Rates * Sample numbers *
- 17. Make payment Homepage visits Pricing page visits Select plan Register / Setup Repeat 1,000 900 500 200 5040 90% 55.56% 40% 25%80% Airbrake.io - Define Opportunities * Sample numbers *
- 19. Improve Sign Up - Design The Experiment 1 page form 2 page form 22.2% 18% Completion 22% Completion Control Variant * Sample numbers *
- 20. Set the Metrics
- 21. Set The Metrics Click Through Rate Click Through ProbabilityVs CTR = # Page Visits # Signups CTP = Unique Page Visits Unique Signups Which metric should we use?
- 22. Example CTR = 2.5 CTP = 0.5 1 1 Conversions Visits 0 5
- 23. Select the best metric for each experiment Experiment Metric CTR on the buttonUpdate the content of pricing page. Change the color of Check Pricing Button on Homepage. Remove credit card info from sign up form. CTP on the button Probability of completing the registration Probability of of starting the registration process
- 24. Difficult Metrics - Hard to Calculate Lifetime Value Virality Customer Satisfaction
- 25. Complex Metrics Viral Growth Factor = X * Y * Z X = % of users who invite other people Y = average # of people that they invited Z = % of users who accepted an invitation Example: Viral Growth Factor = 0.5 * 8 * 37.5% Viral Growth Factor = 1.5
- 27. A/B Testing - Filter Traffic Spam Internal traffic
- 28. A/B Testing - Segment Traffic By platform By traffic source By registration status By user-active status
- 29. Collect Data
- 30. Collect Data - How long should I run the experiment? The goal is to be certain = Statistically significant We keep it running until we have statistically significant data.
- 31. A/B Significance Test - Kissmetrics Tool
- 32. A/B Significance Test - Kissmetrics Tool
- 33. Test 1 --> 15% improvement Collect Data - Long vs Short Testing Period 0 Month 6 Month Test 1 --> 10% improvement 0 Month 2 Month Test 2 --> 5% improvement Test 3 --> 15% improvement 2 Month 4 Month 4 Month 6 Month