際際滷

際際滷Share a Scribd company logo

Intro ab-taguchi

Zac Aghion
Zac Aghion

The document discusses A/B testing and multivariate testing. It explains how to efficiently test multiple variations of website elements through techniques like orthogonal arrays. This allows testing all pairwise combinations of variables while reducing the total number of tests needed. The document provides an example of reducing tests from 54 combinations to just 9 combinations through an L9 orthogonal array design. It also discusses analyzing test results through visualizing data plots and statistical methods like analysis of variance.

1 of 47
Download to read offline
Prepared for: NYC UX + DATA Meetup March 12, 2014 Pivotal Labs, NewYork A/B and PairwiseTesting How I Learned to Stop Worrying and Love Data-Driven Decisions Wednesday, March 12, 14
About Me Founded Splitforce in 2013 - Data is power, and it should be easy to leverage Marketing for Chinese media company in Shanghai Designed experiments and predictive analytics for ILABS in Montreal Studied in economics and statistics at McGill University in Montreal Wednesday, March 12, 14
User Base Wednesday, March 12, 14
User Base Publish two different versions of your app... Wednesday, March 12, 14
User Base Publish two different versions of your app... 50% sees version B50% sees version A Wednesday, March 12, 14
User Base ...and see which one is driving desirable user behavior. Publish two different versions of your app... 50% sees version B50% sees version A Wednesday, March 12, 14
WhichVersion Won? Version A Version B Wednesday, March 12, 14
Version B: 114% Improvement Version A Version B Marketers Surprise: FREE Loses Wednesday, March 12, 14
How Obama Raised $60 Million Wednesday, March 12, 14
Four ButtonVariations Wednesday, March 12, 14
Six MediaVariations Wednesday, March 12, 14
24 Combinations! Wednesday, March 12, 14
And the Winner is... +40% increase in conversion rate 2.9 million additional donators $60 million value of additional donations Wednesday, March 12, 14
Obamalytics Original Conversion Rate: 8.3% Wednesday, March 12, 14
Obamalytics Original Conversion Rate: 8.3% New Conversion Rate: 11.6% Wednesday, March 12, 14
Obamalytics Original Conversion Rate: 8.3% New Conversion Rate: 11.6% 10 million signups from NewVersion would have been 7.12 million signups with the OriginalVersion Wednesday, March 12, 14
Obamalytics Original Conversion Rate: 8.3% New Conversion Rate: 11.6% 10 million signups from NewVersion would have been 7.12 million signups with the OriginalVersion +2.88 million additional signups Wednesday, March 12, 14
Obamalytics Original Conversion Rate: 8.3% New Conversion Rate: 11.6% 10 million signups from NewVersion would have been 7.12 million signups with the OriginalVersion +2.88 million additional signups $21 average donation per signup Wednesday, March 12, 14
Obamalytics Original Conversion Rate: 8.3% New Conversion Rate: 11.6% 10 million signups from NewVersion would have been 7.12 million signups with the OriginalVersion +2.88 million additional signups $21 average donation per signup Approximately $60 million in additional donations Wednesday, March 12, 14
InterpretingTest Results Wednesday, March 12, 14
MultivariateTesting Every screen has X components (ex: Marilyns hair) For each, we can test Y variations (ex.: Green) In total, we have [Y1 x Y2 x Y3] combinations Wednesday, March 12, 14
Costs ofTesting Risk of false positives (Type I error, saying something is there when its not) Need for adequate sample size Testing presents an opportunity cost Wednesday, March 12, 14
Design of Experiments Lets say we have four variables: Header Banner (A, B, C) Main Copy (1, 2, 3) Button Color (Cyan, Magenta,Yellow) Call to Action (Buy!, Check Out) Wednesday, March 12, 14
Design of Experiments Option 1: Full factorial design - multiply out for all different combinations Wednesday, March 12, 14
Design of Experiments Option 1: Full factorial design - multiply out for all different combinations Example: (3 header banners) x (3 main copy) x (3 button colors) x (2 CTAs) = 54 combinations Wednesday, March 12, 14
Design of Experiments Option 1: Full factorial design - multiply out for all different combinations Example: (3 header banners) x (3 main copy) x (3 button colors) x (2 CTAs) = 54 combinations Can we get similar information with fewer tests? Wednesday, March 12, 14
Design of Experiments Option 2: Orthogonal arrays tests pairs of combinations instead of all combinations Wednesday, March 12, 14
Design of Experiments Option 2: Orthogonal arrays tests pairs of combinations instead of all combinations Risk: pairing will hide some combinations, and the effects that paired variables have on each other Wednesday, March 12, 14
Design of Experiments Option 2: Orthogonal arrays tests pairs of combinations instead of all combinations Risk: pairing will hide some combinations, and the effects that paired variables have on each other Mitigation: pair variables that are unlikely to in鍖uence each other Wednesday, March 12, 14
L9 Array Compare any pair of variables across all combinations and youll see that theyre all represented! Wednesday, March 12, 14
Design of Experiments Lets say we have four variables: Header Banner (A, B, C) Main Copy (1, 2, 3) Button Color (Cyan, Magenta,Yellow) Call to Action (Buy!, Check Out) Wednesday, March 12, 14
Design of Experiments Four variables: Header Banner (A, B, C) Main Copy (1, 2, 3) Button Color (Cyan, Magenta,Yellow) Call to Action (Buy, Purchase) Combo # HB MC BC CTA 1 A 1 Cyan Buy 2 A 2 Magenta Purchase 3 A 3 Yellow 4 B 1 Magenta 5 B 2 Yellow Buy 6 B 3 Cyan Purchase 7 C 1 Yellow Purchase 8 C 2 Cyan 9 C 3 Magenta Buy Wednesday, March 12, 14
Design of Experiments Four variables: Header Banner (A, B, C) Main Copy (1, 2, 3) Button Color (Cyan, Magenta,Yellow) Call to Action (Buy, Purchase) Combo # HB MC BC CTA 1 A 1 Cyan Buy 2 A 2 Magenta Purchase 3 A 3 Yellow Buy 4 B 1 Magenta Purchase 5 B 2 Yellow Buy 6 B 3 Cyan Purchase 7 C 1 Yellow Purchase 8 C 2 Cyan Buy 9 C 3 Magenta Buy Wednesday, March 12, 14
Design of Experiments Four variables: Header Banner (A, B, C) Main Copy (1, 2, 3) Button Color (Cyan, Magenta,Yellow) Call to Action (Buy, Purchase) Combo # HB MC BC CTA 1 A 1 Cyan Buy 2 A 2 Magenta Purchase 3 A 3 Yellow Buy 4 B 1 Magenta Purchase 5 B 2 Yellow Buy 6 B 3 Cyan Purchase 7 C 1 Yellow Purchase 8 C 2 Cyan Buy 9 C 3 Magenta Buy Weve reduced need to collect data on 54 combinations to just 9 (6x ef鍖ciency increase) Wednesday, March 12, 14
FROM 54 COMBINATIONS A1CyanBuy, A1CyanPurchase, A1MagentaBuy, A1MagentaPurchase, A1YellowBuy, A1YellowPurchase, A2CyanBuy, A2CyanPurchase, A 2 M a g e n t a B uy, A 2 M a g e n t a P u rc h a s e , A 2 Ye l l ow B uy, A2YellowPurchase, A3CyanBuy, A3CyanPurchase, A3MagentaBuy, A3MagentaPurchase, A3YellowBuy, A3YellowPurchase, B1CyanBuy, B1CyanPurchase, B1MagentaBuy, B1MagentaPurchase, B1YellowBuy, B1YellowPurchase, B2CyanBuy, B2CyanPurchase, B 2 M a g e n t a B uy, B 2 M a g e n t a P u rc h a s e , B 2 Ye l l ow B uy, B2YellowPurchase, B3CyanBuy, B3CyanPurchase, B3MagentaBuy, B3MagentaPurchase, B3YellowBuy, B3YellowPurchase, C1CyanBuy, C1CyanPurchase, C1MagentaBuy, C1MagentaPurchase, C1YellowBuy, C1YellowPurchase, C2CyanBuy, C2CyanPurchase, C 2 M a g e n t a B uy, C 2 M a g e n t a P u rc h a s e , C 2 Ye l l ow B uy, C2YellowPurchase, C3CyanBuy, C3CyanPurchase, C3MagentaBuy, C3MagentaPurchase, C3YellowBuy, C3YellowPurchase Wednesday, March 12, 14
TO JUST 9 (+6X EFFICIENCY) A1CyanBuy, A1CyanPurchase, A1MagentaBuy, A1MagentaPurchase, A1YellowBuy, A1YellowPurchase, A2CyanBuy, A2CyanPurchase, A 2 M a g e n t a B uy, A 2 M a g e n t a P u rc h a s e , A 2 Ye l l ow B uy, A2YellowPurchase, A3CyanBuy, A3CyanPurchase, A3MagentaBuy, A3MagentaPurchase, A3YellowBuy, A3YellowPurchase, B1CyanBuy, B1CyanPurchase, B1MagentaBuy, B1MagentaPurchase, B1YellowBuy, B1YellowPurchase, B2CyanBuy, B2CyanPurchase, B 2 M a g e n t a B uy, B 2 M a g e n t a P u rc h a s e , B 2 Ye l l ow B uy, B2YellowPurchase, B3CyanBuy, B3CyanPurchase, B3MagentaBuy, B3MagentaPurchase, B3YellowBuy, B3YellowPurchase, C1CyanBuy, C1CyanPurchase, C1MagentaBuy, C1MagentaPurchase, C1YellowBuy, C1YellowPurchase, C2CyanBuy, C2CyanPurchase, C 2 M a g e n t a B uy, C 2 M a g e n t a P u rc h a s e , C 2 Ye l l ow B uy, C2YellowPurchase, C3CyanBuy, C3CyanPurchase, C3MagentaBuy, C3MagentaPurchase, C3YellowBuy, C3YellowPurchase Wednesday, March 12, 14
Design of Experiments Where do orthogonal arrays come from? Derived by hand (like playing Sudoku!) Look them up (U Michigan, UYork, Hexawise.com) Wednesday, March 12, 14
Design of Experiments Where do orthogonal arrays come from? Derived by hand (like playing Sudoku!) Look them up (U Michigan, UYork, Hexawise.com) How to choose a design? Number of variables Number of states for each variable Wednesday, March 12, 14
Design of Experiments Where do orthogonal arrays come from? Derived by hand (like playing Sudoku!) Look them up (U Michigan, UYork, Hexawise.com) How to choose a design? Number of variables Number of states for each variable How to analyze results? Plot data,Analysis ofVariance (ANOVA), binning Wednesday, March 12, 14
Analyzing Results Plot data and look at it Some things you dont need statistics to tell you, its just there Your eye is a pretty good analysis tool Wednesday, March 12, 14
Analyzing Results Plot data and look at it Some things you dont need statistics to tell you, its just there Your eye is a pretty good analysis tool Analysis ofVariance (ANOVA) One-way ANOVAs to 鍖nd in鍖uence of a one variable on the result (assume that other variables have minimal in鍖uence) Two-way ANOVAs to 鍖nd in鍖uence of two variables on result at once Wednesday, March 12, 14
Analyzing Results Plot data and look at it Some things you dont need statistics to tell you, its just there Your eye is a pretty good analysis tool Analysis ofVariance (ANOVA) One-way ANOVAs to 鍖nd in鍖uence of a one variable on the result (assume that other variables have minimal in鍖uence) Two-way ANOVAs to 鍖nd in鍖uence of two variables on result at once Binning Group combinations based on results (high vs. low) How many Header Banner As have high result? low result? Wednesday, March 12, 14
Analyzing Results Plot data and look at it Some things you dont need statistics to tell you, its just there Your eye is a pretty good analysis tool Analysis ofVariance (ANOVA) One-way ANOVAs to 鍖nd in鍖uence of a one variable on the result (assume that other variables have minimal in鍖uence) Two-way ANOVAs to 鍖nd in鍖uence of two variables on result at once Binning Group combinations based on results (high vs. low) How many Header Banner As have high result? low result? Takeaway: You can extrapolate data from a subset of combinations to make a conclusion about a full factorial set Wednesday, March 12, 14
Design of Experiments Can get pretty complex, but super ef鍖cient! L36 array - reducing ~94 million combinations to 36 Wednesday, March 12, 14
Comparison of A/BTesting Platforms Google Analytics Optimizely Splitforce Platform Web / mWeb X X Platform Native Mobile X A/BTesting X X Experiment Design Multivariate X X Automation X X Other In-Browser Editor X X Other Consulting X X Wednesday, March 12, 14
In-House vs.Agency In-House Agency Pros Lower initial costs More control over testing process Better understanding of business objectives No need for internal resources Faster results as agency provides specialized expertise Learn best practices and accelerate internal competency Cons Long time to build expertise from scratch Longer time to start achieving great test results Higher initial costs Less understanding of complexities / nuances of your business Less control over testing Wednesday, March 12, 14
ThankYou! For more information: Zac Aghion, CEO & Co-Founder zac@splitforce.com China: (+86)1592-1631-924 USA: (+1)617-750-6684 www.splitforce.com Wednesday, March 12, 14

More Related Content

Intro ab-taguchi

  • 1. Prepared for: NYC UX + DATA Meetup March 12, 2014 Pivotal Labs, NewYork A/B and PairwiseTesting How I Learned to Stop Worrying and Love Data-Driven Decisions Wednesday, March 12, 14
  • 2. About Me Founded Splitforce in 2013 - Data is power, and it should be easy to leverage Marketing for Chinese media company in Shanghai Designed experiments and predictive analytics for ILABS in Montreal Studied in economics and statistics at McGill University in Montreal Wednesday, March 12, 14
  • 4. User Base Publish two different versions of your app... Wednesday, March 12, 14
  • 5. User Base Publish two different versions of your app... 50% sees version B50% sees version A Wednesday, March 12, 14
  • 6. User Base ...and see which one is driving desirable user behavior. Publish two different versions of your app... 50% sees version B50% sees version A Wednesday, March 12, 14
  • 7. WhichVersion Won? Version A Version B Wednesday, March 12, 14
  • 8. Version B: 114% Improvement Version A Version B Marketers Surprise: FREE Loses Wednesday, March 12, 14
  • 9. How Obama Raised $60 Million Wednesday, March 12, 14
  • 13. And the Winner is... +40% increase in conversion rate 2.9 million additional donators $60 million value of additional donations Wednesday, March 12, 14
  • 14. Obamalytics Original Conversion Rate: 8.3% Wednesday, March 12, 14
  • 15. Obamalytics Original Conversion Rate: 8.3% New Conversion Rate: 11.6% Wednesday, March 12, 14
  • 16. Obamalytics Original Conversion Rate: 8.3% New Conversion Rate: 11.6% 10 million signups from NewVersion would have been 7.12 million signups with the OriginalVersion Wednesday, March 12, 14
  • 17. Obamalytics Original Conversion Rate: 8.3% New Conversion Rate: 11.6% 10 million signups from NewVersion would have been 7.12 million signups with the OriginalVersion +2.88 million additional signups Wednesday, March 12, 14
  • 18. Obamalytics Original Conversion Rate: 8.3% New Conversion Rate: 11.6% 10 million signups from NewVersion would have been 7.12 million signups with the OriginalVersion +2.88 million additional signups $21 average donation per signup Wednesday, March 12, 14
  • 19. Obamalytics Original Conversion Rate: 8.3% New Conversion Rate: 11.6% 10 million signups from NewVersion would have been 7.12 million signups with the OriginalVersion +2.88 million additional signups $21 average donation per signup Approximately $60 million in additional donations Wednesday, March 12, 14
  • 21. MultivariateTesting Every screen has X components (ex: Marilyns hair) For each, we can test Y variations (ex.: Green) In total, we have [Y1 x Y2 x Y3] combinations Wednesday, March 12, 14
  • 22. Costs ofTesting Risk of false positives (Type I error, saying something is there when its not) Need for adequate sample size Testing presents an opportunity cost Wednesday, March 12, 14
  • 23. Design of Experiments Lets say we have four variables: Header Banner (A, B, C) Main Copy (1, 2, 3) Button Color (Cyan, Magenta,Yellow) Call to Action (Buy!, Check Out) Wednesday, March 12, 14
  • 24. Design of Experiments Option 1: Full factorial design - multiply out for all different combinations Wednesday, March 12, 14
  • 25. Design of Experiments Option 1: Full factorial design - multiply out for all different combinations Example: (3 header banners) x (3 main copy) x (3 button colors) x (2 CTAs) = 54 combinations Wednesday, March 12, 14
  • 26. Design of Experiments Option 1: Full factorial design - multiply out for all different combinations Example: (3 header banners) x (3 main copy) x (3 button colors) x (2 CTAs) = 54 combinations Can we get similar information with fewer tests? Wednesday, March 12, 14
  • 27. Design of Experiments Option 2: Orthogonal arrays tests pairs of combinations instead of all combinations Wednesday, March 12, 14
  • 28. Design of Experiments Option 2: Orthogonal arrays tests pairs of combinations instead of all combinations Risk: pairing will hide some combinations, and the effects that paired variables have on each other Wednesday, March 12, 14
  • 29. Design of Experiments Option 2: Orthogonal arrays tests pairs of combinations instead of all combinations Risk: pairing will hide some combinations, and the effects that paired variables have on each other Mitigation: pair variables that are unlikely to in鍖uence each other Wednesday, March 12, 14
  • 30. L9 Array Compare any pair of variables across all combinations and youll see that theyre all represented! Wednesday, March 12, 14
  • 31. Design of Experiments Lets say we have four variables: Header Banner (A, B, C) Main Copy (1, 2, 3) Button Color (Cyan, Magenta,Yellow) Call to Action (Buy!, Check Out) Wednesday, March 12, 14
  • 32. Design of Experiments Four variables: Header Banner (A, B, C) Main Copy (1, 2, 3) Button Color (Cyan, Magenta,Yellow) Call to Action (Buy, Purchase) Combo # HB MC BC CTA 1 A 1 Cyan Buy 2 A 2 Magenta Purchase 3 A 3 Yellow 4 B 1 Magenta 5 B 2 Yellow Buy 6 B 3 Cyan Purchase 7 C 1 Yellow Purchase 8 C 2 Cyan 9 C 3 Magenta Buy Wednesday, March 12, 14
  • 33. Design of Experiments Four variables: Header Banner (A, B, C) Main Copy (1, 2, 3) Button Color (Cyan, Magenta,Yellow) Call to Action (Buy, Purchase) Combo # HB MC BC CTA 1 A 1 Cyan Buy 2 A 2 Magenta Purchase 3 A 3 Yellow Buy 4 B 1 Magenta Purchase 5 B 2 Yellow Buy 6 B 3 Cyan Purchase 7 C 1 Yellow Purchase 8 C 2 Cyan Buy 9 C 3 Magenta Buy Wednesday, March 12, 14
  • 34. Design of Experiments Four variables: Header Banner (A, B, C) Main Copy (1, 2, 3) Button Color (Cyan, Magenta,Yellow) Call to Action (Buy, Purchase) Combo # HB MC BC CTA 1 A 1 Cyan Buy 2 A 2 Magenta Purchase 3 A 3 Yellow Buy 4 B 1 Magenta Purchase 5 B 2 Yellow Buy 6 B 3 Cyan Purchase 7 C 1 Yellow Purchase 8 C 2 Cyan Buy 9 C 3 Magenta Buy Weve reduced need to collect data on 54 combinations to just 9 (6x ef鍖ciency increase) Wednesday, March 12, 14
  • 35. FROM 54 COMBINATIONS A1CyanBuy, A1CyanPurchase, A1MagentaBuy, A1MagentaPurchase, A1YellowBuy, A1YellowPurchase, A2CyanBuy, A2CyanPurchase, A 2 M a g e n t a B uy, A 2 M a g e n t a P u rc h a s e , A 2 Ye l l ow B uy, A2YellowPurchase, A3CyanBuy, A3CyanPurchase, A3MagentaBuy, A3MagentaPurchase, A3YellowBuy, A3YellowPurchase, B1CyanBuy, B1CyanPurchase, B1MagentaBuy, B1MagentaPurchase, B1YellowBuy, B1YellowPurchase, B2CyanBuy, B2CyanPurchase, B 2 M a g e n t a B uy, B 2 M a g e n t a P u rc h a s e , B 2 Ye l l ow B uy, B2YellowPurchase, B3CyanBuy, B3CyanPurchase, B3MagentaBuy, B3MagentaPurchase, B3YellowBuy, B3YellowPurchase, C1CyanBuy, C1CyanPurchase, C1MagentaBuy, C1MagentaPurchase, C1YellowBuy, C1YellowPurchase, C2CyanBuy, C2CyanPurchase, C 2 M a g e n t a B uy, C 2 M a g e n t a P u rc h a s e , C 2 Ye l l ow B uy, C2YellowPurchase, C3CyanBuy, C3CyanPurchase, C3MagentaBuy, C3MagentaPurchase, C3YellowBuy, C3YellowPurchase Wednesday, March 12, 14
  • 36. TO JUST 9 (+6X EFFICIENCY) A1CyanBuy, A1CyanPurchase, A1MagentaBuy, A1MagentaPurchase, A1YellowBuy, A1YellowPurchase, A2CyanBuy, A2CyanPurchase, A 2 M a g e n t a B uy, A 2 M a g e n t a P u rc h a s e , A 2 Ye l l ow B uy, A2YellowPurchase, A3CyanBuy, A3CyanPurchase, A3MagentaBuy, A3MagentaPurchase, A3YellowBuy, A3YellowPurchase, B1CyanBuy, B1CyanPurchase, B1MagentaBuy, B1MagentaPurchase, B1YellowBuy, B1YellowPurchase, B2CyanBuy, B2CyanPurchase, B 2 M a g e n t a B uy, B 2 M a g e n t a P u rc h a s e , B 2 Ye l l ow B uy, B2YellowPurchase, B3CyanBuy, B3CyanPurchase, B3MagentaBuy, B3MagentaPurchase, B3YellowBuy, B3YellowPurchase, C1CyanBuy, C1CyanPurchase, C1MagentaBuy, C1MagentaPurchase, C1YellowBuy, C1YellowPurchase, C2CyanBuy, C2CyanPurchase, C 2 M a g e n t a B uy, C 2 M a g e n t a P u rc h a s e , C 2 Ye l l ow B uy, C2YellowPurchase, C3CyanBuy, C3CyanPurchase, C3MagentaBuy, C3MagentaPurchase, C3YellowBuy, C3YellowPurchase Wednesday, March 12, 14
  • 37. Design of Experiments Where do orthogonal arrays come from? Derived by hand (like playing Sudoku!) Look them up (U Michigan, UYork, Hexawise.com) Wednesday, March 12, 14
  • 38. Design of Experiments Where do orthogonal arrays come from? Derived by hand (like playing Sudoku!) Look them up (U Michigan, UYork, Hexawise.com) How to choose a design? Number of variables Number of states for each variable Wednesday, March 12, 14
  • 39. Design of Experiments Where do orthogonal arrays come from? Derived by hand (like playing Sudoku!) Look them up (U Michigan, UYork, Hexawise.com) How to choose a design? Number of variables Number of states for each variable How to analyze results? Plot data,Analysis ofVariance (ANOVA), binning Wednesday, March 12, 14
  • 40. Analyzing Results Plot data and look at it Some things you dont need statistics to tell you, its just there Your eye is a pretty good analysis tool Wednesday, March 12, 14
  • 41. Analyzing Results Plot data and look at it Some things you dont need statistics to tell you, its just there Your eye is a pretty good analysis tool Analysis ofVariance (ANOVA) One-way ANOVAs to 鍖nd in鍖uence of a one variable on the result (assume that other variables have minimal in鍖uence) Two-way ANOVAs to 鍖nd in鍖uence of two variables on result at once Wednesday, March 12, 14
  • 42. Analyzing Results Plot data and look at it Some things you dont need statistics to tell you, its just there Your eye is a pretty good analysis tool Analysis ofVariance (ANOVA) One-way ANOVAs to 鍖nd in鍖uence of a one variable on the result (assume that other variables have minimal in鍖uence) Two-way ANOVAs to 鍖nd in鍖uence of two variables on result at once Binning Group combinations based on results (high vs. low) How many Header Banner As have high result? low result? Wednesday, March 12, 14
  • 43. Analyzing Results Plot data and look at it Some things you dont need statistics to tell you, its just there Your eye is a pretty good analysis tool Analysis ofVariance (ANOVA) One-way ANOVAs to 鍖nd in鍖uence of a one variable on the result (assume that other variables have minimal in鍖uence) Two-way ANOVAs to 鍖nd in鍖uence of two variables on result at once Binning Group combinations based on results (high vs. low) How many Header Banner As have high result? low result? Takeaway: You can extrapolate data from a subset of combinations to make a conclusion about a full factorial set Wednesday, March 12, 14
  • 44. Design of Experiments Can get pretty complex, but super ef鍖cient! L36 array - reducing ~94 million combinations to 36 Wednesday, March 12, 14
  • 45. Comparison of A/BTesting Platforms Google Analytics Optimizely Splitforce Platform Web / mWeb X X Platform Native Mobile X A/BTesting X X Experiment Design Multivariate X X Automation X X Other In-Browser Editor X X Other Consulting X X Wednesday, March 12, 14
  • 46. In-House vs.Agency In-House Agency Pros Lower initial costs More control over testing process Better understanding of business objectives No need for internal resources Faster results as agency provides specialized expertise Learn best practices and accelerate internal competency Cons Long time to build expertise from scratch Longer time to start achieving great test results Higher initial costs Less understanding of complexities / nuances of your business Less control over testing Wednesday, March 12, 14
  • 47. ThankYou! For more information: Zac Aghion, CEO & Co-Founder zac@splitforce.com China: (+86)1592-1631-924 USA: (+1)617-750-6684 www.splitforce.com Wednesday, March 12, 14