際際滷

際際滷Share a Scribd company logo

Make Or Break Customer Satisfaction

O
oanaman

This document discusses limitations of current customer satisfaction models and proposes a new "make or break" model. The current models use linear regression to assign attribute importance weights, but may miss non-compensatory effects where one attribute alone causes (dis)satisfaction. The new model incorporates rewards and penalties based on attributes identified as alone making or breaking the experience. A test study collected customer satisfaction data on mobile phone retailers using the new approach and found it generated attribute weights along with bonuses and penalties that provide additional insights over traditional models.

1 of 30
Downloaded 50 times
Make or Break Customer Satisfaction Improving Customer Satisfaction Measurement With New Methods Keith Chrzan Chief Research Officer, Maritz Research PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 1
Current practice for customer satisfaction modeling Driver analysis using linear, compensatory customer satisfaction models (regression, correlation, PLS, SEM) Each attribute has an importance weight Sum of attribute importances times their respective performance scores reflects overall satisfaction When done well, we account for the multicollinearity thats pervasive in customer satisfaction research (e.g. Theils information-theoretic averaging-over-orderings regression model) Entered the marketing and economics fields in the 1950s and 1960s and never left Fits well with statistical models But how realistic is this model? PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 2
A visit to <NAME REMOVED> hotel Accurate reservation Quick check in process Nice room, upgraded bath Larger TV than most movie theater screens Easy internet access Excellent workout room Tasty room service Comfortable bed, neatly folded PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 3
A visit to <NAME REMOVED> hotel Accurate reservation Quick check in process Nice room, upgraded bath Larger TV than most movie theater screens Easy internet access Excellent workout room Tasty room service Comfortable bed, neatly folded with dead cockroaches between the sheets PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 4
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 5
Non-compensatory effects Sometimes performance on an attribute is so bad that, all by itself, it causes dissatisfaction It doesnt matter how well the brand performs on other attributes, poor performance on just one ruins the entire experience This is a non-compensatory effect because adding the good effects still cant overcome the effect of the poor performance on overall satisfaction Of course the opposite is also possible great performance on one attribute outweighs shortcomings on others Standard regression models miss these kinds of effects PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 6
A non-compensatory model Joffre Swait (1997) developed a questionnaire method that allowed analysts to incorporate non-compensatory effects to conjoint modeling His method created better-fitting models, models that explain brand choice better His models generated additional insights not otherwise available Which are must have attribute levels and for which and how many people? Which are dealbreakers for which and how many people? Within three years, however, the advent of HB analysis made this model obsolete, for conjoint studies PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 7
A non-compensatory customer satisfaction model Adapting Swaits approach to customer satisfaction modeling creates a non-compensatory model featuring rewards and penalties The result is Make or Break customer satisfaction PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 8
Test case Online survey, May 2010 599 respondents rating a recent trip to a mobile phone retail store Overall satisfaction Nine attributes identified as drivers of retailer satisfaction in qualitative research PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 9
Sample design Two cells Control cell (n=305): Standard customer sat survey Test cell (n=294): Penalty/Reward approach PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 10
Questionnaire Outline First respondents rate their overall satisfaction We ask of satisfied respondents If any attributes were so wonderful that, all by themselves, they made the experience great We use a checklist, to make this easy for respondents We ask dissatisfied respondents to check any attributes that were so terrible as to ruin, by themselves, the overall experience We ask respondents to rate only the attributes not checked above Thus there are no additional keystrokes required from respondents PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 11
Questionnaire Q1. Please indicate how satisfied you were with your mobile phone shopping experience at <INSERT MOBILE PHONE RETAIL STORE> using the scale below. [ ] Completely satisfied [ ] Somewhat satisfied [ ] Neither satisfied nor dissatisfied [ ] Somewhat dissatisfied [ ] Completely dissatisfied Q2c. ASK IF Q1< 3. Was the performance on any of these aspects so good as, all by itself, to make your overall mobile phone shopping experience satisfactory? Q2d. ASK IF Q1> 3. Was the performance on any of these aspects so bad as, all by itself, to make your overall mobile phone shopping experience satisfactory? Aspect Yes (for the good) Yes (for the bad) Store location [] [] Speed of service [] [] Friendliness of sales representative [] [] Professionalism sales representative [] [] Information the sales representative had for me [] [] Phone prices [] [] Network coverage [] [] Availability of the phone I wanted [] [] Price of the calling plans [] [] None of these CANNOT BE CHOSEN WITH ANY OTHER [] [] Q2e. Please indicate how much you agree or disagree with each of the following statements about your mobile phone shopping experience at <MOBILE PHONE RETAIL STORE>. SHOW ONLY ATTRIBUTES NOT CHECKED IN 2C OR 2D. RANDOMIZE ORDER. Neither Strongly agree nor Strongly Agree Agree disagree Disagree Disagree The store was conveniently located [] [] [] [] [] I was waited on quickly [] [] [] [] [] The sales representative was friendly [] [] [] [] [] The sales representative was professional [] [] [] [] [] The sales representative had the information I needed [] [] [] [] [] The phones were reasonably priced [] [] [] [] [] The network has adequate coverage [] [] [] [] [] The phone I wanted was available [] [] [] [] [] The calling plans were reasonably priced [] [] [] [] [] PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 12
Results We get these results Basic coefficients (weights) for each attribute An additional bonus weight for those people saying each attribute was wonderful and made their experience great An additional penalty, a negative weight, that detracts from the overall rating for those people reporting attributes that ruined their experience Patterns of which attributes were particularly wonderful or terrible vary Not all respondents get the same attribute weights The model accommodates respondent heterogeneity This test case study uses regression analysis and shows the statistically significant attribute coefficients PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 13
Standard regression model Attribute Coefficient Rep had info I needed - My phone was available - Price of phone .13 Price of plan - Coverage .15 Rep friendly .38 Rep professionalism - Quick service - PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 14
Base of Make or Break model Attribute Coefficient Rep had info I needed .18 My phone was available .07 Price of phone .15 Price of plan Coverage Rep friendly Rep professionalism Quick service PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 15
Adding in penalties % reporting attribute ruined the overall Attribute Coefficient experience Penalty Rep had info I needed .18 5 -.40 My phone was available .07 5 -.68 Price of phone .15 Price of plan 4 -.48 Coverage 1 -.64 Rep friendly Rep professionalism Quick service 6 -.96 PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 16
Topping it off with gains % % reporting reporting attribute attribute ruined the perfected overall the overall Attribute Coefficient experience Penalty experience Reward Rep had info I needed .18 5 -.40 My phone was available .07 5 -.68 31 .26 Price of phone .15 Price of plan 4 -.48 Coverage 1 -.64 30 .25 Rep friendly 39 .22 Rep professionalism 34 .17 Quick service 6 -.96 PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 17
Correcting for multicollinearity Using Theils model (True Driver Analysis) shows the impact of all the penalties and gains, taking into account shared variance among the scale questions, the penalties and the rewards -10% -5% 0% 5% 10% Represenative - had info I needed Representative - professional Representative - friendly Waited on quickly Phone i wanted was Reward 32% available Phones were Scale Questions 42% reasonably priced Penalty 26% Calling plans were reasonably priced Network has adequate coverage Conveniently located PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 18
Evaluation The non-compensatory Make or Break model yields additional insights The model incorporates respondent heterogeneity Different respondents can have different patterns of penalties and rewards In this case, 47 distinct patterns The model VASTLY improves prediction Control cell with standard customer sat questions: R2 = 30% Above plus non-compensatory penalties/rewards for ruining/making my experience: R2 = 65% PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 19
Maybe that was too easy? Experiences at mobile phone retailers vary quite a bit (long waits for service, phone availability, etc.) How does the model perform when most respondents are happy with their experiences? Will a shortage of penalties allow the model enough to work with? Lets try retail banking PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 20
Case studies 2-6 Web surveys fielded October 2010 Control groups doing standard ratings and test cells identifying non-compensatory penalty/rewards Five surveys of banking satisfaction Branch satisfaction ATM satisfaction Call center satisfaction Customer service representative satisfaction Online banking satisfaction Overall satisfaction and 3-11 attributes, depending on study PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 21
Sample sizes case studies 2-6 Control Aspect Test Sample Sample Branch 395 377 ATM 367 369 Call center 113 128 CSR 181 180 Online 363 363 PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 22
Bank branch satisfaction model % reporting % reporting attribute attribute ruined the perfected Standard overall the overall Attribute regression Base experience Penalty experience Reward Wait time in line .15 .10 2 -1.42 Staff courteous 41 .27 Speed of completing .19 .17 request Staff knowledgeable .28 .22 Staff provides .18 .17 1 -.40 accurate answers R2 .62 .68 .70 Significant effects for 4/10 attributes Model improves with addition of penalties and then again with addition of rewards PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 23
ATM satisfaction model % reporting % reporting attribute attribute ruined the perfected Standard overall the overall Attribute regression Base experience Penalty experience Reward Safe and secure .14 .14 71 .11 Ease of transaction .37 .19 2 -.99 42 .36 Wait time .13 .13 1 -1.61 R2 .51 .61 .64 Significant effects for all three attributes Model improves with addition of penalties and rewards Service failures are uncommon and catastrophic 1.61 points on a 5 point scale! PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 24
Automated call satisfaction model % reporting % reporting attribute attribute ruined the perfected Standard overall the overall Attribute regression Base experience Penalty experience Reward Easy to get live rep 8 -.53 Easy to navigate .31 .13 Useful response .53 .42 5 -.72 options Reasonable hold 4 -.85 37 .50 time R2 .74 .81 .84 Significant effects for all four attributes Model improves with addition of penalties and of rewards Service failures are more common Captures eight different varieties of customer experience PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 25
Phone rep satisfaction model % reporting % reporting attribute attribute ruined the perfected Standard overall the overall Attribute regression Base experience Penalty experience Reward Authority to address .22 .18 1 -.49 your issue Explains things 2 -.80 clearly Takes responsibility .28 .22 4 -.58 to resolve your issue Handles call quickly 43 .31 Provides complete .27 .19 answers R2 .71 .75 .77 Significant effects for 5/11 attributes Three significant and injurious penalties Improved model fit PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 26
Online banking satisfaction model % reporting % reporting attribute attribute ruined the perfected Standard overall the overall Attribute regression Base experience Penalty experience Reward Safe and secure 1 -1.45 Easy to navigate .21 .19 Easy to complete .15 .14 50 .17 tasks Able to conduct .19 .13 1 -1.22 desired transactions Helps you manage .11 .11 your finances R2 .57 .61 .62 Significant effects for 5/10 attributes Rewards are less good than penalties are bad (but more common) Improved model fit PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 27
Location of penalty/reward questions If before attribute ratings, as in Study 1, an adaptive survey flow can keep the amount of respondent effort (in terms of number of keystrokes) the same as standard customer satisfaction studies If after, we require additional work from respondents We split respondents, half with penalty/reward questions before and half with after Similar models, same R2 either way Adaptive set-up doesnt seem to hurt the resulting data PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 28
Summary In all six cases tested, the Make or Break model significantly outperforms the standard customer satisfaction ratings measurements Higher R2 explains more variance on overall satisfaction Additional insight about non-linear penalties and boosts for excellent/poor performance Additional insight about respondent heterogeneity These benefits remain even if we only use the penalties part of the model PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 29
PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 30

More Related Content

Make Or Break Customer Satisfaction

  • 1. Make or Break Customer Satisfaction Improving Customer Satisfaction Measurement With New Methods Keith Chrzan Chief Research Officer, Maritz Research PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 1
  • 2. Current practice for customer satisfaction modeling Driver analysis using linear, compensatory customer satisfaction models (regression, correlation, PLS, SEM) Each attribute has an importance weight Sum of attribute importances times their respective performance scores reflects overall satisfaction When done well, we account for the multicollinearity thats pervasive in customer satisfaction research (e.g. Theils information-theoretic averaging-over-orderings regression model) Entered the marketing and economics fields in the 1950s and 1960s and never left Fits well with statistical models But how realistic is this model? PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 2
  • 3. A visit to <NAME REMOVED> hotel Accurate reservation Quick check in process Nice room, upgraded bath Larger TV than most movie theater screens Easy internet access Excellent workout room Tasty room service Comfortable bed, neatly folded PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 3
  • 4. A visit to <NAME REMOVED> hotel Accurate reservation Quick check in process Nice room, upgraded bath Larger TV than most movie theater screens Easy internet access Excellent workout room Tasty room service Comfortable bed, neatly folded with dead cockroaches between the sheets PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 4
  • 5. PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 5
  • 6. Non-compensatory effects Sometimes performance on an attribute is so bad that, all by itself, it causes dissatisfaction It doesnt matter how well the brand performs on other attributes, poor performance on just one ruins the entire experience This is a non-compensatory effect because adding the good effects still cant overcome the effect of the poor performance on overall satisfaction Of course the opposite is also possible great performance on one attribute outweighs shortcomings on others Standard regression models miss these kinds of effects PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 6
  • 7. A non-compensatory model Joffre Swait (1997) developed a questionnaire method that allowed analysts to incorporate non-compensatory effects to conjoint modeling His method created better-fitting models, models that explain brand choice better His models generated additional insights not otherwise available Which are must have attribute levels and for which and how many people? Which are dealbreakers for which and how many people? Within three years, however, the advent of HB analysis made this model obsolete, for conjoint studies PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 7
  • 8. A non-compensatory customer satisfaction model Adapting Swaits approach to customer satisfaction modeling creates a non-compensatory model featuring rewards and penalties The result is Make or Break customer satisfaction PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 8
  • 9. Test case Online survey, May 2010 599 respondents rating a recent trip to a mobile phone retail store Overall satisfaction Nine attributes identified as drivers of retailer satisfaction in qualitative research PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 9
  • 10. Sample design Two cells Control cell (n=305): Standard customer sat survey Test cell (n=294): Penalty/Reward approach PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 10
  • 11. Questionnaire Outline First respondents rate their overall satisfaction We ask of satisfied respondents If any attributes were so wonderful that, all by themselves, they made the experience great We use a checklist, to make this easy for respondents We ask dissatisfied respondents to check any attributes that were so terrible as to ruin, by themselves, the overall experience We ask respondents to rate only the attributes not checked above Thus there are no additional keystrokes required from respondents PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 11
  • 12. Questionnaire Q1. Please indicate how satisfied you were with your mobile phone shopping experience at <INSERT MOBILE PHONE RETAIL STORE> using the scale below. [ ] Completely satisfied [ ] Somewhat satisfied [ ] Neither satisfied nor dissatisfied [ ] Somewhat dissatisfied [ ] Completely dissatisfied Q2c. ASK IF Q1< 3. Was the performance on any of these aspects so good as, all by itself, to make your overall mobile phone shopping experience satisfactory? Q2d. ASK IF Q1> 3. Was the performance on any of these aspects so bad as, all by itself, to make your overall mobile phone shopping experience satisfactory? Aspect Yes (for the good) Yes (for the bad) Store location [] [] Speed of service [] [] Friendliness of sales representative [] [] Professionalism sales representative [] [] Information the sales representative had for me [] [] Phone prices [] [] Network coverage [] [] Availability of the phone I wanted [] [] Price of the calling plans [] [] None of these CANNOT BE CHOSEN WITH ANY OTHER [] [] Q2e. Please indicate how much you agree or disagree with each of the following statements about your mobile phone shopping experience at <MOBILE PHONE RETAIL STORE>. SHOW ONLY ATTRIBUTES NOT CHECKED IN 2C OR 2D. RANDOMIZE ORDER. Neither Strongly agree nor Strongly Agree Agree disagree Disagree Disagree The store was conveniently located [] [] [] [] [] I was waited on quickly [] [] [] [] [] The sales representative was friendly [] [] [] [] [] The sales representative was professional [] [] [] [] [] The sales representative had the information I needed [] [] [] [] [] The phones were reasonably priced [] [] [] [] [] The network has adequate coverage [] [] [] [] [] The phone I wanted was available [] [] [] [] [] The calling plans were reasonably priced [] [] [] [] [] PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 12
  • 13. Results We get these results Basic coefficients (weights) for each attribute An additional bonus weight for those people saying each attribute was wonderful and made their experience great An additional penalty, a negative weight, that detracts from the overall rating for those people reporting attributes that ruined their experience Patterns of which attributes were particularly wonderful or terrible vary Not all respondents get the same attribute weights The model accommodates respondent heterogeneity This test case study uses regression analysis and shows the statistically significant attribute coefficients PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 13
  • 14. Standard regression model Attribute Coefficient Rep had info I needed - My phone was available - Price of phone .13 Price of plan - Coverage .15 Rep friendly .38 Rep professionalism - Quick service - PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 14
  • 15. Base of Make or Break model Attribute Coefficient Rep had info I needed .18 My phone was available .07 Price of phone .15 Price of plan Coverage Rep friendly Rep professionalism Quick service PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 15
  • 16. Adding in penalties % reporting attribute ruined the overall Attribute Coefficient experience Penalty Rep had info I needed .18 5 -.40 My phone was available .07 5 -.68 Price of phone .15 Price of plan 4 -.48 Coverage 1 -.64 Rep friendly Rep professionalism Quick service 6 -.96 PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 16
  • 17. Topping it off with gains % % reporting reporting attribute attribute ruined the perfected overall the overall Attribute Coefficient experience Penalty experience Reward Rep had info I needed .18 5 -.40 My phone was available .07 5 -.68 31 .26 Price of phone .15 Price of plan 4 -.48 Coverage 1 -.64 30 .25 Rep friendly 39 .22 Rep professionalism 34 .17 Quick service 6 -.96 PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 17
  • 18. Correcting for multicollinearity Using Theils model (True Driver Analysis) shows the impact of all the penalties and gains, taking into account shared variance among the scale questions, the penalties and the rewards -10% -5% 0% 5% 10% Represenative - had info I needed Representative - professional Representative - friendly Waited on quickly Phone i wanted was Reward 32% available Phones were Scale Questions 42% reasonably priced Penalty 26% Calling plans were reasonably priced Network has adequate coverage Conveniently located PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 18
  • 19. Evaluation The non-compensatory Make or Break model yields additional insights The model incorporates respondent heterogeneity Different respondents can have different patterns of penalties and rewards In this case, 47 distinct patterns The model VASTLY improves prediction Control cell with standard customer sat questions: R2 = 30% Above plus non-compensatory penalties/rewards for ruining/making my experience: R2 = 65% PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 19
  • 20. Maybe that was too easy? Experiences at mobile phone retailers vary quite a bit (long waits for service, phone availability, etc.) How does the model perform when most respondents are happy with their experiences? Will a shortage of penalties allow the model enough to work with? Lets try retail banking PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 20
  • 21. Case studies 2-6 Web surveys fielded October 2010 Control groups doing standard ratings and test cells identifying non-compensatory penalty/rewards Five surveys of banking satisfaction Branch satisfaction ATM satisfaction Call center satisfaction Customer service representative satisfaction Online banking satisfaction Overall satisfaction and 3-11 attributes, depending on study PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 21
  • 22. Sample sizes case studies 2-6 Control Aspect Test Sample Sample Branch 395 377 ATM 367 369 Call center 113 128 CSR 181 180 Online 363 363 PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 22
  • 23. Bank branch satisfaction model % reporting % reporting attribute attribute ruined the perfected Standard overall the overall Attribute regression Base experience Penalty experience Reward Wait time in line .15 .10 2 -1.42 Staff courteous 41 .27 Speed of completing .19 .17 request Staff knowledgeable .28 .22 Staff provides .18 .17 1 -.40 accurate answers R2 .62 .68 .70 Significant effects for 4/10 attributes Model improves with addition of penalties and then again with addition of rewards PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 23
  • 24. ATM satisfaction model % reporting % reporting attribute attribute ruined the perfected Standard overall the overall Attribute regression Base experience Penalty experience Reward Safe and secure .14 .14 71 .11 Ease of transaction .37 .19 2 -.99 42 .36 Wait time .13 .13 1 -1.61 R2 .51 .61 .64 Significant effects for all three attributes Model improves with addition of penalties and rewards Service failures are uncommon and catastrophic 1.61 points on a 5 point scale! PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 24
  • 25. Automated call satisfaction model % reporting % reporting attribute attribute ruined the perfected Standard overall the overall Attribute regression Base experience Penalty experience Reward Easy to get live rep 8 -.53 Easy to navigate .31 .13 Useful response .53 .42 5 -.72 options Reasonable hold 4 -.85 37 .50 time R2 .74 .81 .84 Significant effects for all four attributes Model improves with addition of penalties and of rewards Service failures are more common Captures eight different varieties of customer experience PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 25
  • 26. Phone rep satisfaction model % reporting % reporting attribute attribute ruined the perfected Standard overall the overall Attribute regression Base experience Penalty experience Reward Authority to address .22 .18 1 -.49 your issue Explains things 2 -.80 clearly Takes responsibility .28 .22 4 -.58 to resolve your issue Handles call quickly 43 .31 Provides complete .27 .19 answers R2 .71 .75 .77 Significant effects for 5/11 attributes Three significant and injurious penalties Improved model fit PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 26
  • 27. Online banking satisfaction model % reporting % reporting attribute attribute ruined the perfected Standard overall the overall Attribute regression Base experience Penalty experience Reward Safe and secure 1 -1.45 Easy to navigate .21 .19 Easy to complete .15 .14 50 .17 tasks Able to conduct .19 .13 1 -1.22 desired transactions Helps you manage .11 .11 your finances R2 .57 .61 .62 Significant effects for 5/10 attributes Rewards are less good than penalties are bad (but more common) Improved model fit PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 27
  • 28. Location of penalty/reward questions If before attribute ratings, as in Study 1, an adaptive survey flow can keep the amount of respondent effort (in terms of number of keystrokes) the same as standard customer satisfaction studies If after, we require additional work from respondents We split respondents, half with penalty/reward questions before and half with after Similar models, same R2 either way Adaptive set-up doesnt seem to hurt the resulting data PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 28
  • 29. Summary In all six cases tested, the Make or Break model significantly outperforms the standard customer satisfaction ratings measurements Higher R2 explains more variance on overall satisfaction Additional insight about non-linear penalties and boosts for excellent/poor performance Additional insight about respondent heterogeneity These benefits remain even if we only use the penalties part of the model PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 29
  • 30. PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010 30