Big Data, Little Devices: Mobile A/B Testing
12 likes1,988 views
A/B testing applies the scientific method and principles of randomized experimentation to digital marketing and product design. In this presentation, Zac Aghion from Splitforce will share his experience with A/B testing in the unique context of mobile applications, as well as two case studies featuring Marks & Spencer - among the UK's largest retailers, and Spot the Difference - a kids' game for iPhone and iPad.
Big Data, Little Devices: Mobile A/B Testing
- 1. Big Data, Little Devices Mobile A/B testing in theory and practice Thursday, October 23, 14
- 2. 2 truths, 1 lie 1. Co-founder of Splitforce, Inc. mobile A/B testing software 2. Previously product manager @ Google 3. Speak Mandarin better than Zuck Thursday, October 23, 14
- 3. 2 truths, 1 lie 1. Co-founder of Splitforce, Inc. mobile A/B testing software 2. Previously product manager @ Google 3. Speak Mandarin better than Zuck Thursday, October 23, 14
- 4. 2 truths, 1 lie 1. Co-founder of Splitforce, Inc. mobile A/B testing software 2. Previously product manager @ Google 3. Speaks Mandarin better than Zuck Thursday, October 23, 14
- 5. 2 truths, 1 lie 1. Co-founder of Splitforce, Inc. mobile A/B testing software 2. Previously product manager @ Google 3. Speaks Mandarin better than Zuck Thursday, October 23, 14
- 6. What is A/B testing? Why? To confirm or reject a (null) hypothesis What? Randomized experiment in which two variants, A and B, represent the control and treatment groups Synonyms: bucket testing, split testing People also mean... multivariate testing, A/B/n Thursday, October 23, 14
- 7. +500% increase in A/B testing as a search term on Google since 2009 Google Trends Thursday, October 23, 14
- 8. Testing in Web vs. App Web Mobile App Javascript Obj-C, Java Responsive, lightweight Native UI objects, UX conventions (Near) instant collection Blind spots More context Data Design Tech Thursday, October 23, 14
- 9. How Marks & Spencer Boosted Engagement by +20% Thursday, October 23, 14
- 10. Goal: Increase usage of Cooking Mode Thursday, October 23, 14
- 11. Test: Call to action copy on a button 50% Original (control group) 50% Treatment (test group) Thursday, October 23, 14
- 12. Result: +19.56% usage rate Thursday, October 23, 14
- 13. As an agile product team within a larger company, we are constantly treading the line between moving quickly and making well informed decisions. -Neil Kimmett, Marks & Spencer Labs Thursday, October 23, 14
- 14. How MyBook Increased In-App Revenue +25% Thursday, October 23, 14
- 15. Goal: Increase monthly subscriptions There are free books, and paid books When a user arrives at a paywall, she is presented with an two subscription offers Three subscription types: Weekly, Monthly, or Annual Thursday, October 23, 14
- 16. Test: Secondary option as price anchor 50% Original (Weekly price) 50% Treatment (Yearly price) Thursday, October 23, 14
- 17. Result: +15.2% conversion rate Thursday, October 23, 14
- 18. We saw a measurable improvement in payments, and revenue from the winning variation represented a 25-35% increase when measured against the baseline. -Ivan Pestunov, MyBook Thursday, October 23, 14
- 19. Matt Boosted Conversions +125% Matt Mayer Founder, ReignDesign Thursday, October 23, 14
- 20. Users Buy & Spend Stars to Play... ...Matt Makes $ $ $ Thursday, October 23, 14
- 21. Only 0.5% of users purchased... In-app message prompting to buy Thursday, October 23, 14
- 22. Matt decided to test something new... 50% Thursday, October 23, 14
- 23. Matt decided to test something new... 50% 50% Thursday, October 23, 14
- 24. Matt decided to test something new... 50% 50% Original = 0.5% New = 1.13% Thursday, October 23, 14
- 25. Thank You! Zac Aghion CEO & Co-Founder Splitforce, Inc. zac@splitforce.com Thursday, October 23, 14