�ݺ�ߣ

1. Webinar Four Country Mobile Operator Study:a competitive benchmark of leading mobile operator sites19th January 2011

2. OverviewIntroductions

3. Study Background

4. What do we mean by UX Benchmarking? Why is it important?

5. Methodology and Technology

6. Findings and Opportunities

7. Questions and Comments

8. UserZoom and UX Alliance Partners at a glanceIntroductions Arthur MoanCountry Manager UK & Ireland at UserZoomLouiselle MorandUser Experience Consultantat TelonoMartje van der LindeUser Experience Consultant at User IntelligenceJavier JarribaManaging Director at Xperience Consulting

9. Today’s Session GoalsSee how users in four countries look for:

10. the monthly cost to surf the internet daily with an iPad

11. the fax number/address for a high street shop

12. To gain insight into the European mobile operator market

13. Introduction and review of UX Benchmarking methodology

14. Introduction and review of UserZoom software solution for conducting this type of researchStudy Background

15. What is UX benchmarking and why is it important?OK……. So how do we define UX benchmarking?

16. What is UX benchmarking and why is it important?Definition of Benchmarking:

17. The process of comparing the business processes and performance metrics including cost, cycle time, productivity, or quality to another that is widely considered to be an industry standard benchmark or best practice.

18. It provides a snapshot of the performance of your business and helps you understand where you are in relation to a particular standard.What is UX benchmarking and why is it important?Definition of UX:

19. It’s about how a person feels about using a system. It highlights the experiential, affective, meaningful and valuable aspects of HCI and product ownership, but it also covers a person’s perceptions of the practical aspects such as utility, ease of use and efficiency of the system.

20. User experience is subjective in nature, as it is about an individual’s feelings and thoughts about the system. User experience is dynamic, because it changes over time as the circumstances change.What is UX benchmarking and why is it important?Some may wonder: is UX benchmarking:?

21. Not really…..Here’s how we’d define UX Benchmarking:The process of comparing a website’s performance, specifically in relation to UX metrics, including among others effectiveness, efficiency, overall satisfaction and quality perception (ISO-9241-11 standard), to another that is widely considered to be an industry standard benchmark or best practice

22. UX benchmarking provides a snapshot of the performance of your website and helps you understand where you are in relation to a particular standard.10 must-dos with UX BenchmarkingDefine measurable & comparable UX-related KPIs (key performance indicators)Select competitors to benchmark againstDefine target audienceIdentify common site activities/tasks (between-subjects design)Conduct research using scenario-based testingGather both quantitative & qualitative dataUse the quantitative data to test for statistically significant differences between means or proportionsUse qualitative data to support your findings and understand the why Research at least twice a year to measure design changes and trends in expectationsUse the right research solution/tool

23. What is UX benchmarking and why is it important?‘You can’t manage what you can’t measure’Attributed to Peter Drucker, Mgt Guru The question is:How are you actually measuring UX and defining your benchmark?

24. Methodology and Technology

25. Common UX research challengesDifficulties obtaining objective, statistically significant, actionable data

26. Obtaining idea adoption from stakeholders

27. Geographic/demographic limitations

28. Research is costly and time-consumingHow does UserZoom add value?By allowing you to see the BIG picture

29. Testing large samples of geographically spread participants

30. Combining statistically-significant usability + user behavior data

31. Measuring Customer Experience cost-effectively

32. Getting actionableinsightsBehaviorUsability Lab StudiesWeb Traffic AnalysisOnline UXStudiesAttitudesOnline SurveysFocus GroupsQualitativeQuantitative

33. Online Usability Testing SolutionUZ Self-Serve Edition

34. 100% on-demand, web-based software solution

35. To manage & conduct various types of online UX research projectsHundreds of users participate in a study

36. In their natural context

37. From geographically spread locations

38. Users try to complete tasks(or goals) + answer questions

39. No human moderation needed

40. Our browser bar connects users with our secure serversOur method

41. Analyze data real-time using UZ AnalyticsEffectiveness + efficiency ratios

42. Analyze data using UZ AnalyticsQualitative feedback collected. Easy visualization through ‘text clouds’

43. Analyze data using UZ Analytics

44. UserZoom Results - Behavioural

45. Study structureIntroduction:For each study, participants were randomly assigned two tasks on two popular mobile operator websites (four tasks in total)

46. Task: Find a monthly iPad data plan

47. Task: Locate a store address/fax numberPost-task:Mobile operator site evaluation ratings included -Ease of use

48. Perceived complexity of sites

49. Perceived consistency of sites

50. Clear where to start

51. Confidence in using the sites

52. Usability Satisfaction (SUS score)

53. What was liked most about the sites

54. What was liked least about the sitesWrap-up:Mobile Operator Site Preference

55. Likelihood to recommend: (Net Promoter Score)Switzerland Study – Participant DemographicsParticipants:N= 78 participantsDemographic data collected:Gender

56. Current mobile operator

57. Age

58. Individuals between 18-55 years of age were invited to complete the study.GENDER45 females completed the study33 males completed the study

59. Switzerland Study – Participant DemographicsMOBILE OPERATORSwisscom, Sunrise, and Orange were the most common current providers among participants.“Other”: yallo, vtx, vodafoneAGEThe highest percentage of Swiss participants are 36 - 45 years of age.14 participants are 18-25

60. 19 participants are 26-35

62. 20 participants are 46-55Netherlands Study – Participant DemographicsParticipants:N= 100 participantsDemographic data collected:Gender

64. Age

66. Netherlands Study – Participant DemographicsMOBILE OPERATORVodafone, KPN, and T-Mobile were the most common current providers among participants.“Other”: hi, simyo, lebara, ziggo, rabo mobielAGEThe highest percentage of Dutch participants are 36 - 45 years of age.21 participants are 18-25

69. 23 participants are 46-55Spain Study – Participant DemographicsParticipants:N= 100 participantsDemographic data collected:Gender

71. Age

73. Spain Study – Participant DemographicsMOBILE OPERATORMovistar, Vodafone, and Orange were the most common current providers among participants.AGEThe highest percentage of Spanish participants are 26-35 years of age.27 participants are 18-25

76. 08 participants are 46-55United Kingdom Study – Participant DemographicsParticipants:N= 101 participantsDemographic data collected:Gender

78. Age

80. United Kingdom Study – Participant DemographicsMOBILE OPERATORVodafone, O2, and Orange were the most common current providers among participants.“Other”: asda, tesco, virginAGEThe highest percentage of UK participants are 36 - 45 years of age.21 participants are 18-25

83. 18 participants are 46-55Executive Summary

84. Executive Summary Orange UK and Orange Switzerland (CH) had highest task success rates for both iPad (UK 75%) and Address (CH 87%) tasks.

85. Orange Spain (SP) had the lowest success rate across tasks 24%, the lowest participants satisfaction scores, and substantially lower preference (25%) vs. Vodafone (75%).

86. All mobile providers tested in each country received low usability satisfaction scores by participants.

87. The lowest system usability scale scores (SUS) were received by Vodafone Spain (51) and Orange Spain (40).

88. All four mobile providers received low Net Promoter Scores (between -46% and -75%). There were more detractors than promoters in every instance.Findings and Opportunities

89. Top level overview of Task 1 across all countries with deep dive into Switzerland & UK results “Find the monthly cost to surf the internet daily with an iPad”

90. Task Metrics: iPad Monthly Plan

91. Switzerland: SwisscomSuccessful process

92. ��Գٱ��Ա��”

93. “Internet with your ��ǳٱ��Ǵǰ�”

94. “iPad tariffs”

95. Can be found in 3 clicks if browsingHeatmaps First Click: Swisscom - Switzerland (DE)Task: iPad Monthly PlanOverall 30% were successful 6% used the search bar- 40% rolled over to the side navigation bar and then clicked on the links on the right

96. 15% clicked “Kombi-Angebote” on the side navigation barHeatmaps First Click: Orange – Switzerland (DE)Task: iPad Monthly PlanOverall 24% were successful 14% used the search bar

97. 16% clicked “Mobile” on the top navigation bar

98. 14% clicked ��Գٱ��Ա��” on the top navigation barHeatmaps First Click: Orange - SpainTask: iPad Monthly PlanOverall 28% were successful 21% used the search bar

99. 14% clicked “movil” on the top navigation bar

100. 65% clicked in other placesHeatmaps First Click: Vodafone - SpainTask: iPad Monthly PlanOverall 61% were successful 7% used the search bar

101. 31% clicked “Tienda Online” on the top navigation bar

102. 14% clicked in the center promotional areaHeatmaps First Click: Vodafone - NetherlandsTask: iPad Monthly PlanOverall 27% were successful 4% used the search bar

103. 14% clicked ��Գٱ��Ա��” on the top navigation bar

104. 19% clicked in the center promotional areaHeatmaps First Click: T-Mobile - NetherlandsTask: iPad Monthly PlanOverall 40% were successful 15% used the search bar

105. 13% clicked “Abonnement” on the top navigation bar

106. 27% clicked ��Գٱ��Ա��” on the top navigation barHeatmaps First Click: Orange - United Kingdom Task: iPad Monthly PlanOverall 75% were successful 13% used the search bar

107. 37% clicked “iPad” on the left navigation bar

108. 50% clicked in other placesVodafone > iPad: UKOnly half of the participants were able to successfully find the monthly cost of an iPad subscription on the Vodafone website.

109. Successful Participants spent an average of 5 minutes 24 seconds and 19 clicks completing the task. Task Effectiveness

110. Orange > iPad: UKThe majority of participants were able to find the correct price for the plan (75%).Successful Participants spent an average of 1 minute 11 seconds and 7 clicks completing the task. Task Effectiveness

111. Top level overview of Task 2 across all countries with deep dive into Netherlands & Spain’s results “Locate a store address/fax number”

112. Task Metrics: Locating Store Address/Phone number

113. Success rates for NetherlandsTask 2 : « Find the closest shop to the Amsterdam railway station and write down its address. »

114. The T-mobile site provided the best results regarding this task, 77% of the participants was successful.

115. On average it took slightly over one minute to find the right information, with an average of only 4 page views and 8 clicks.

116. On the Vodafone website 52% was successful in achieving this information.

117. Remarkable result is that it took an average of 3 minutes and 29 clicks to complete this task.Success questionnaire - NetherlandsNevertheless, the rate of success regarding a task is not necessarily an indication of the ease of use:

118. The Vodafone website was less successful, but only 28% of the participants regarded it to be difficult to acquire the needed information.

119. On the T-mobile website, people were far more successful, but also less satisfied with the level of difficulty to get to the right URL: 42% of the participants thought this was (very) hard to achieve.

120. Vodafone

121. T-mobileAbandon questionnaire - NetherlandsMain reasons for abandoning a task were:

122. Page not loading: Especially the Vodafone website was unable to load the required page. Remarkable is that people wait up to 5 minutes!

123. Information not in expected location: Both Vodafone and T-mobile present information in an unexpected manner. Participants have certain expectations on where and how to find the required information. In case these expectations are not met, people easily abandon the website.QUOTES“I was looking for it under Customer service and expected to be able to fill in my location and that it would automatically give the closest store location.”“ I was unable to open the page.”

124. Final questionnaire - NetherlandsUsers of both the Vodafone as well as the T-mobile website are neither positive nor negative about the websites:

125. They regard it as easy to use for inexperienced users and there is no need for extra help.

126. Nevertheless, the lack of success will have some influence on the use of the website.

127. Most users indicate that they will not likely be using these websites in the near future again. Task 2 Spain : Find the closest shop to the Atocha railway station and write down its fax number.

128. Task 2 Spain – success ratiosOn Orange 53% of users completed the task successfully.

129. On Vodafone 68% of users completed the task successfully .Vodafone analysis - SpainFirst click of users was mostly focused on the search engine and the help section. There were several users that tried to find the shops using the vodafone logotipe. Vodafone analysis - SpainThe reason was users got the answer incorrect was due to a problem on the web. Although as happened in the previous task, the information that users found was not clear enough.The site is not clear nor intuitive, you have to use the search engine, the information provided is not preciseThe section with the Vodafone shops is easy, the problem is one you have to find one in MadridToo much information on the map

130. Vodafone analysis - SpainThe reasons because participants abandoned the task was mainly due to the information being incomplete (67%). Other reasons were:

131. The information presentation is not clear enough

132. The information, options and menus are difficult to understand

133. The information wasn’t in the expected location

134. Few search options

135. As it has been seen in the previous task users again had problems with the information provided. I have entered province “Madrid”, city “Madrid” and no results appeared with this search options. I have tried with different options and it always happened the same.

136. Orange Analysis - SpainMost of the users went to the tab “Orange shop” in order to look for the addresses of the physical shops.Orange Analysis - SpainThe reason for the incorrect answer was not being able to find the correct information due to options and menus being difficult to understand.

137. Users explained that all the site is focusing on the iphone whereas ipad information is quite difficult to find.

138. The system could be improve using an accessible and easy to find search engine. There is no way to understand what is going on here.There is no way to find anything. What a fuss!

139. Orange Analysis - SpainThose users who didn’t finish the task was mainly because they couldn’t find the information where they were expecting it to be.

140. The links to the information were not clear enough.

141. The site is very focused on selling products or services instead of giving clients the service that they need. I don’t have anything against Orange but its website is a disaster

142. Final Questionnaire – all countries: Which mobile operator do you prefer?NetherlandsSwitzerlandSpainUK

143. Metrics: SUS ScoreThe System Usability Scale (SUS) is a set of 10 questions that is used to measure the perception of a systems overall usability. Participants rate each question with a 1 to 5 scale (5 being strongly agree). 1. I think that I would like to use this system frequently. 2. I found the system unnecessarily complex. 3. I thought the system was easy to use. 4. I think that I would need the support of a technical person to be able to use this system. 5. I found the various functions in this system were well integrated. 6. I thought there was too much inconsistency in this system. 7. I would imagine that most people would learn to use this system very quickly. 8. I found the system very cumbersome to use. 9. I felt very confident using the system. 10. I needed to learn a lot of things before I could get going with this system.

144. Metrics: SUS ScoreAfter completing the tasks on each mobile provider website participants were asked to complete the SUS questionnaire. The averages for each question are used to calculate an overall score (0-100). SUS Score Guide*: 35=poor, 50=OK, 70=good, 85=excellentBangor, A. (2009, May). Determining What Individual SUS Scores Mean: Adding and Adjective Rating Scale. Journal of Usability Studies, 4(3), 114-123.

145. Net Promoter Score (NPS®)After completing the four tasks, participants were asked how likely they would be to recommend each mobile provider’s website. PromotersDetractors-=% of Promoters(9s and 10s)Detractors(0 through 6)NPS

146. Top 5 findings / Learning'sThe satisfaction is directly related with the success of the users in obtaining the information that they are looking for. In general the websites for mobile operators in all countries are not performing optimal when consumers have a task in mind. The performance rate was dependent on the task, not of the website.

147. Across the board the main difficulties were in participants finding specific information. The normal way users tried to find the information was through the menu structure. When they realized it was to complicated they tried to use ‘search’ although the results weren’t always clear enough. A much clearer architecture and clearer results in search engines are therefore needed. Furthermore the use of promotional ads made the navigation much longer and difficult for most of the users.Top 5 findings / Learning'sAnalysing NPS , SUS, success on task and preference data it is clear that the brand halo effect has a bearing on site preference in each market irrespective of task success.

148. UK - although participants had notably higher task success rates on the Orange site (75% and 71%) vs. the Vodafone site (50% and 49%), when asked for site preference at the end of the study, the Orange website was selected only 8% more than the Vodafone site (54% vs 46%) respectively. Out of the Orange customers (n=17) more than half (9) preferred the Vodafone website.

149. Orange is not strong in Spain

150. Swisscom the national provider is stronger in Switzerland

151. Vodafone marginally outperforms T-mobile in the NetherlandsTop 5 findings / Learning'sWhen participants were asked whether they would recommend the sites in each market all operators scored negatively showing room for improvement . Although these scores are low previous research has shown that telecommunication companies have the lowest net promoter scores within the technology sector.

152. As Vodafone and Orange were both tested in 3 of the 4 markets we can benchmark their experiences.

153. Vodafone outperforms Orange on aggregated SUS (53 versus 51.3)

154. Vodafone outperforms Orange on aggregated NPS (-55% versus -64.3%)

155. Task 1 success aggregated : Vodafone outperforms Orange (46% versus 42%)

156. Task 2 success aggregated : Orange outperforms Vodafone (70% versus 56%)Q & A

157. Information on UIUser Intelligence is a user experience consultancy and was founded in 2002

158. Offices in Amsterdam and Helsinki, additional service offering in Sweden and Norway

159. Our team consists 17 consultants with a passion for people and interactive technologyAbout TelonoTelono is a Swiss consultancy specialising in usability, user experience and web accessibility.

160. We help companies implement a systematic user-centred design process, from user experience strategy and user research to interaction design and usability evaluations.

161. Telono is the Swiss representative of the User Experiene Alliance (Uxalliance)About Xperience Consulting10 years of experience

162. 2 offices (Barcelona)

163. 150 AI projects

164. 300 usability studies

165. 6000 users analyzed

166. 150 clients (15 countries)Usability and user experience experts, we create interfaces that suit user needs and objectives of your company. Intuitive and easy to use webs.

167. UX Alliance - Who are we?The UX Alliance is an international network for usability testing and user experience research consisting of core partners, associates, and affiliate members.

168. Back in 2005 the leading user experience companies from France, Germany, Spain, UK and US met in Barcelona with a single common goal in mind: how to provide effective international user experience services to our clients.

169. Having worked together on international projects for many years, we recognised that we were ideally placed to lead international user experience research. Thus, the UXalliance was born… www.uxalliance.com

170. Who are we?20 Countries – 22 Companies  Over 100 global projects per year together.SirValUseGermanyUser IntelligenceNetherlands/FinlandThink UserKoreaUIDesign GroupRussiaExperienceLabUnited KingdomAxanceFranceUser ExperienceChinaExperienceUCzech RepublicYu CentrikCanadaSnitker & Co.DenmarkAyer ViernesChileXperience ConsultingSpainUXa Partners, Associates, & AffiliatesSymetriaPolandTelonoSwitzerlandUser Centric, Inc.United StatesOptimal UsabilityNew ZealandMitsue LinksJapanOptimal ExperienceAustraliaMercedes SanchezBrazil www.uxalliance.

171. About UserZoomLeading online (remote) user experience research software company

172. 9 years experience in UX research & consulting

173. Offices in Sunnyvale (CA), London (UK) and Barcelona (Spain)

174. Created because traditional UX research methods are great but not enoughThanks so much for your time!Checkoutourupcomingevents at www.userzoom.comSunnyvale (USA)440 N. Wolfe Rd. Sunnyvale,CA 94085 Phone: +1 (408) 524 7445 Contact: Alfonso de la NuezBarcelona (Spain)Av. Diagonal 419 3º 2ª 08008 Barcelona Phone: +34 93 414 7554 Fax: +34 93 209 8380 Contact: Xavier MestresCheshire (UK)Booths Park. Chelford Road, Knutsford, Cheshire WA16 8GSPhone: + 44 (0)1565 759890Cell: + 44 (0)7900 472 920Contact: Arthur Moan

�ݺ�ߣ

Mobile Operator UX Benchmarking Study

More Related Content

Mobile Operator UX Benchmarking Study

Editor's Notes