ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo

Gears of War 3 Analytics: Optimizing the Online Experience, or How I Learned to Stop Worrying and Love the Beta

?
?
J
joe_graf

The document summarizes analytics from the Gears of War 3 beta test, which was conducted to optimize the online experience for launch. Key findings from the beta test data included issues with quality of service probes overburdening servers, too few dedicated servers being available for matchmaking, and experiments adjusting the matchmaking algorithm to prioritize player experience level over other factors. The data was analyzed to address issues and ensure a smooth launch for the full game.

1 of 61
Gears of War 3 Analytics: Optimizing the Online Experience, or How I Learned to Stop Worrying and Love the Beta
Gears of War 3 Analytics: Optimizing the Online Experience, or How I Learned to Stop Worrying and Love the Beta Joe Graf (Dr. Statslove) Game Connection 2011 December 8, 2011
About Me ? Been at Epic over 8 years ? Primarily focused on online features for our games and engine ? Lead Gameplay Programmer for Gears of War 3
About You You are at this talk because you are a ¨C Programmer responsible for matchmaking ¨C Designer that wants to know more ¨C Producer that wants to know where the Gears team invested
About Gears of War ? First two games were very popular multiplayer games ? Gears of War 2 ¨C 6 million units sold ¨C More than 1 million simultaneous players ? Gears of War 3 looked to expand upon that via the inclusion of dedicated servers
Overview ? Network infrastructure of Gears ? Types of metrics the Gears beta captured ? How that data was used to pinpoint issues ? How the Gears matchmaking system works ? What we adjusted post beta for a great launch
Why do a beta? ? Gears 2 launch wasn¡¯t seamless ¨C Overburdened Xbox Live services caused slow matchmaking at launch ¨C Took about 1 month post ship to get everything solid ? Unanswered questions ¨C Tested dedicated server support in Gears 2, but not at launch levels of players
Why worry about a beta? ? Risk!!! ¨C Betas are seen as ¡°Demos¡± not work in progress ? Do it too early and pre-orders are cancelled ¨C Schedule impact ? Essentially shipping the game twice ? Reduced output from team while beta is running ¨C Throwaway work is polish lost on final product ? Remove content not included in the beta ? Custom UI flow and exposure to different TCRs
Goals for the Beta ? Find any scalability issues in matchmaking ¨C Gears 2 did not exhibit issues with < 150k simultaneous players ¨C Gears 2 test validated approach but not scalability ? Determine how many datacenters we need and where to locate them ? Test our data collection pipeline beginning to end ? Verify gameplay balancing with real players ? Test alternative matchmaking algorithms
Network Infrastructure
Unreal MCP ? Epic¡¯s Xbox Live Server Platform (XLSP) ¨C Determines best datacenter for each player ¨C Handles the playlist population tracking ¨C Allows the Gears team to change data on the Xbox client without requiring a patch ¨C Responsible for processing all uploads from the game
Data Captured ? Player data ¨C Each player uploads their data as XML once per day ? Matchmaking data ¨C Party host uploads an XML payload with the details ? Gameplay data ¨C Each host uploads a compressed binary file containing all of the events that were not filtered out ? 1 Terrabyte of data collected in first week of beta
Player Data ? Population counts and location data ? Aggregate view of user preferences ¨C MP character ¨C Starting weapons and weapon skins ¨C Button and stick configuration ? Inverse, southpaw, legacy, etc. ? Player statistics and progress ¨C Nearly 2,000 different data points
Player Data
Player Data (cont.)
Player Data (cont.) ? NAT (network address table) types ¨C Data is different from Gears 2 and final product ¨C More Open NATs, fewer Moderate or Strict
Matchmaking Data ? Search times for each type of search ? Party details ¨C Size, unique ids, average skill, average XP level ? Returned host data ¨C Ping, match quality, average skill ? Number of searches ? Number of results per search
Matchmaking Data
Matchmaking Data (cont.)
Matchmaking Data (cont.)
Gameplay Data ? Map and game mode played ? Weapon data ¨C Fired, melee, picked up, dropped ? Player locations recorded for all events ? Each recorded event has a granularity level for throttling via Unreal MCP
Gameplay Data
Gameplay Data (cont.)
Gameplay Data (cont.)
Gameplay Data (cont.) ? Goal was 10 minutes per match ¨C Needed to tweak game rules to bring them closer to that target
Gameplay Data (cont.)
Gameplay Data (cont.)
Gameplay Data (cont.)
Gears Matchmaking ? Gears 3 used the same algorithm as Gears 2 ¨C Minor changes to accommodate dedicated servers ? Player hosted matches are still required ¨C Things can and will go wrong ? Player¡¯s are generally not network engineers ? Servers can fail and network outages can prevent access to datacenter ? Some regions have poor infrastructure
Terminology ? Match Quality ¨C How likely a match ends in a draw ? QoS ¨C Quality of Service ¨C Network quality between client and host ? Best search ¨C Exact match based upon party size ? Any search ¨C Match based upon space greater than party size ? Empty search ¨C Find a new server to configure for the desired mode
Matchmaking Goals Matchmaking Time Match Quality Network Quality
Matchmaking Process Request Read Skill Search for Rate Join Match Datacenter Data Matches Matches
Matchmaking Process (cont.) Request Read Skill Search for Rate Join Match Datacenter Data Matches Matches ? Use IP to location to find closest datacenter ¨C Unreal MCP returns an ID for this datacenter ? Done once per player signin ? Saved in the player¡¯s profile in case the look up fails
Matchmaking Process (cont.) Request Read Skill Search for Rate Join Match Datacenter Data Matches Matches ? Each playlist has its own skill table ? Done once per search initiation ? Used to provide Xbox Live with a single value to rate advertised sessions against ? Used to calculate match quality
Matchmaking Process (cont.) Request Read Skill Search for Rate Join Match Datacenter Data Matches Matches ¡°Best¡± Rate Found Search Matches ? ¡°Any¡± Search ¡°Empty¡± Search
Matchmaking Process (cont.) Request Read Skill Search for Rate Join Match Datacenter Data Matches Matches Sort Into Ping Buckets Sort Buckets By Match Quality Request Ok? Reservation
Beta Launch Plan ? Access to beta was in multiple phases ¨C Epic employees, Microsoft employees, friends and family ¨C Bulletstorm Epic Edition purchasers ¨C Gears of War Pre-orders ? Phased approach allowed us to manage player population growth ¨C Thousands, tens of thousands, hundreds of thousands
Beta Launch ? Employees, friends and family ¨C Everything is fine ? Bulletstorm players join in ¨C Forums and Twitterverse indicate matchmaking times are slow ? War Room ¨C We gather to go over the data and drill into what¡¯s going on
QoS Data Indicates a Problem
Too Many Servers Are Unreachable
Leading To Slow Matchmaking
What¡¯s Going On? ? Contention on QoS results ¨C Dedicated servers were configured like Xbox clients ¨C Except all clients were funneled to similar sets of servers: different pattern from player hosted ? The fix? ¨C Increase the configurable max number of simultaneous QoS probes
QoS Problem Resolved
QoS Problem Over Time
Production QoS Data
QoS was improved, but¡­ ? Matchmaking charts show clients are host starved ¨C Not enough hosts are being found ¨C Resulting in player hosted sessions
Too Few Search Results
Less is More
QoS and Search Results Change
QoS and Search Results Change
Matchmaking Algorithm Experiment ? Prefer matches by distance from party¡¯s average XP Level ¨C Sorting has to be on the backend where the data is available ¨C Risky, because it overrides all standard Xbox Live sorting mechanisms ? Why do this? ¨C Player perception is XP Level means skill
Experiment Results ? <SearchResult ¡­ Ping="50" Quality="0.924" /> <SearchResult ¡­ Ping="100" Quality="0.668" /> <SearchResult ¡­ Ping="100" Quality="0.409" /> ? <Reservation> <Player ¡­ Skill="23" XpLevel="92" /> <Player ¡­ Skill="28" XpLevel="89" /> <Player ¡­ Skill="25" XpLevel="86" /> </Reservation> <Reservation> <Player ¡­ Skill="19" XpLevel="90" /> </Reservation> <Reservation> <Player ¡­ Skill="28" XpLevel="87" /> </Reservation>
Experiment Results ? <SearchResult ¡­ Ping="50" Quality="0.924" /> <SearchResult ¡­ Ping="100" Quality="0.668" /> <SearchResult ¡­ Ping="100" Quality="0.409" /> ? <Reservation> <Player ¡­ Skill="23" XpLevel="92" /> <Player ¡­ Skill="28" XpLevel="89" /> <Player ¡­ Skill="25" XpLevel="86" /> </Reservation> <Reservation> <Player ¡­ Skill="19" XpLevel="90" /> </Reservation> <Reservation> <Player ¡­ Skill="28" XpLevel="87" /> </Reservation>
Experiment Results ? <SearchResult ¡­ Ping="50" Quality="0.924" /> <SearchResult ¡­ Ping="100" Quality="0.668" /> <SearchResult ¡­ Ping="100" Quality="0.409" /> ? <Reservation> <Player ¡­ Skill="23" XpLevel="92" /> <Player ¡­ Skill="28" XpLevel="89" /> <Player ¡­ Skill="25" XpLevel="86" /> </Reservation> <Reservation> <Player ¡­ Skill="19" XpLevel="90" /> </Reservation> <Reservation> <Player ¡­ Skill="28" XpLevel="87" /> </Reservation>
Beta Results
Production Changes ? Added the ability to place servers in buckets ? Added looping in addition to bucketing ? Added new QoS configuration options ? Changed Quick Match searching to be XP Level based
Production Matchmaking Process Request Read Skill Search for Rate Join Match Datacenter Data Matches 0..N Matches ¡°Best¡± Rate Found Search 0..X Matches ? ¡°Any¡± Search 0..Y ¡°Empty¡± Search 0..Z
Matchmaking Times Beta Production
Takeaways ? Record all input matchmaking variables ? Record all output results ¨C Don¡¯t assume a detail isn¡¯t important ? You can never have too many configurable knobs to turn ¨C We couldn¡¯t change some values in the beta
Why I learned to love the beta ? The extra effort up front was offset by removing the need for emergency patches ? Instead of going from 0 to 1 million, we could learn while managing the growth ? We discovered and addressed bottlenecks in every phase of our data collection pipeline ¨C The data doesn¡¯t do you any good if you can¡¯t look at it ? Gears of War 3 launch was our best ever
Gears of War 3 Analytics: Optimizing the Online Experience, or How I Learned to Stop Worrying and Love the Beta Special thanks to: Wes Hunt Josh Markiewicz Joe Graf Justin Rowland Epic Games, Inc. Ian Thomas Sam Zamani Twitter: @EpicCog

More Related Content

Gears of War 3 Analytics: Optimizing the Online Experience, or How I Learned to Stop Worrying and Love the Beta

  • 2. Gears of War 3 Analytics: Optimizing the Online Experience, or How I Learned to Stop Worrying and Love the Beta Joe Graf (Dr. Statslove) Game Connection 2011 December 8, 2011
  • 3. About Me ? Been at Epic over 8 years ? Primarily focused on online features for our games and engine ? Lead Gameplay Programmer for Gears of War 3
  • 4. About You You are at this talk because you are a ¨C Programmer responsible for matchmaking ¨C Designer that wants to know more ¨C Producer that wants to know where the Gears team invested
  • 5. About Gears of War ? First two games were very popular multiplayer games ? Gears of War 2 ¨C 6 million units sold ¨C More than 1 million simultaneous players ? Gears of War 3 looked to expand upon that via the inclusion of dedicated servers
  • 6. Overview ? Network infrastructure of Gears ? Types of metrics the Gears beta captured ? How that data was used to pinpoint issues ? How the Gears matchmaking system works ? What we adjusted post beta for a great launch
  • 7. Why do a beta? ? Gears 2 launch wasn¡¯t seamless ¨C Overburdened Xbox Live services caused slow matchmaking at launch ¨C Took about 1 month post ship to get everything solid ? Unanswered questions ¨C Tested dedicated server support in Gears 2, but not at launch levels of players
  • 8. Why worry about a beta? ? Risk!!! ¨C Betas are seen as ¡°Demos¡± not work in progress ? Do it too early and pre-orders are cancelled ¨C Schedule impact ? Essentially shipping the game twice ? Reduced output from team while beta is running ¨C Throwaway work is polish lost on final product ? Remove content not included in the beta ? Custom UI flow and exposure to different TCRs
  • 9. Goals for the Beta ? Find any scalability issues in matchmaking ¨C Gears 2 did not exhibit issues with < 150k simultaneous players ¨C Gears 2 test validated approach but not scalability ? Determine how many datacenters we need and where to locate them ? Test our data collection pipeline beginning to end ? Verify gameplay balancing with real players ? Test alternative matchmaking algorithms
  • 11. Unreal MCP ? Epic¡¯s Xbox Live Server Platform (XLSP) ¨C Determines best datacenter for each player ¨C Handles the playlist population tracking ¨C Allows the Gears team to change data on the Xbox client without requiring a patch ¨C Responsible for processing all uploads from the game
  • 12. Data Captured ? Player data ¨C Each player uploads their data as XML once per day ? Matchmaking data ¨C Party host uploads an XML payload with the details ? Gameplay data ¨C Each host uploads a compressed binary file containing all of the events that were not filtered out ? 1 Terrabyte of data collected in first week of beta
  • 13. Player Data ? Population counts and location data ? Aggregate view of user preferences ¨C MP character ¨C Starting weapons and weapon skins ¨C Button and stick configuration ? Inverse, southpaw, legacy, etc. ? Player statistics and progress ¨C Nearly 2,000 different data points
  • 16. Player Data (cont.) ? NAT (network address table) types ¨C Data is different from Gears 2 and final product ¨C More Open NATs, fewer Moderate or Strict
  • 17. Matchmaking Data ? Search times for each type of search ? Party details ¨C Size, unique ids, average skill, average XP level ? Returned host data ¨C Ping, match quality, average skill ? Number of searches ? Number of results per search
  • 21. Gameplay Data ? Map and game mode played ? Weapon data ¨C Fired, melee, picked up, dropped ? Player locations recorded for all events ? Each recorded event has a granularity level for throttling via Unreal MCP
  • 25. Gameplay Data (cont.) ? Goal was 10 minutes per match ¨C Needed to tweak game rules to bring them closer to that target
  • 29. Gears Matchmaking ? Gears 3 used the same algorithm as Gears 2 ¨C Minor changes to accommodate dedicated servers ? Player hosted matches are still required ¨C Things can and will go wrong ? Player¡¯s are generally not network engineers ? Servers can fail and network outages can prevent access to datacenter ? Some regions have poor infrastructure
  • 30. Terminology ? Match Quality ¨C How likely a match ends in a draw ? QoS ¨C Quality of Service ¨C Network quality between client and host ? Best search ¨C Exact match based upon party size ? Any search ¨C Match based upon space greater than party size ? Empty search ¨C Find a new server to configure for the desired mode
  • 31. Matchmaking Goals Matchmaking Time Match Quality Network Quality
  • 32. Matchmaking Process Request Read Skill Search for Rate Join Match Datacenter Data Matches Matches
  • 33. Matchmaking Process (cont.) Request Read Skill Search for Rate Join Match Datacenter Data Matches Matches ? Use IP to location to find closest datacenter ¨C Unreal MCP returns an ID for this datacenter ? Done once per player signin ? Saved in the player¡¯s profile in case the look up fails
  • 34. Matchmaking Process (cont.) Request Read Skill Search for Rate Join Match Datacenter Data Matches Matches ? Each playlist has its own skill table ? Done once per search initiation ? Used to provide Xbox Live with a single value to rate advertised sessions against ? Used to calculate match quality
  • 35. Matchmaking Process (cont.) Request Read Skill Search for Rate Join Match Datacenter Data Matches Matches ¡°Best¡± Rate Found Search Matches ? ¡°Any¡± Search ¡°Empty¡± Search
  • 36. Matchmaking Process (cont.) Request Read Skill Search for Rate Join Match Datacenter Data Matches Matches Sort Into Ping Buckets Sort Buckets By Match Quality Request Ok? Reservation
  • 37. Beta Launch Plan ? Access to beta was in multiple phases ¨C Epic employees, Microsoft employees, friends and family ¨C Bulletstorm Epic Edition purchasers ¨C Gears of War Pre-orders ? Phased approach allowed us to manage player population growth ¨C Thousands, tens of thousands, hundreds of thousands
  • 38. Beta Launch ? Employees, friends and family ¨C Everything is fine ? Bulletstorm players join in ¨C Forums and Twitterverse indicate matchmaking times are slow ? War Room ¨C We gather to go over the data and drill into what¡¯s going on
  • 39. QoS Data Indicates a Problem
  • 40. Too Many Servers Are Unreachable
  • 41. Leading To Slow Matchmaking
  • 42. What¡¯s Going On? ? Contention on QoS results ¨C Dedicated servers were configured like Xbox clients ¨C Except all clients were funneled to similar sets of servers: different pattern from player hosted ? The fix? ¨C Increase the configurable max number of simultaneous QoS probes
  • 46. QoS was improved, but¡­ ? Matchmaking charts show clients are host starved ¨C Not enough hosts are being found ¨C Resulting in player hosted sessions
  • 47. Too Few Search Results
  • 49. QoS and Search Results Change
  • 50. QoS and Search Results Change
  • 51. Matchmaking Algorithm Experiment ? Prefer matches by distance from party¡¯s average XP Level ¨C Sorting has to be on the backend where the data is available ¨C Risky, because it overrides all standard Xbox Live sorting mechanisms ? Why do this? ¨C Player perception is XP Level means skill
  • 52. Experiment Results ? <SearchResult ¡­ Ping="50" Quality="0.924" /> <SearchResult ¡­ Ping="100" Quality="0.668" /> <SearchResult ¡­ Ping="100" Quality="0.409" /> ? <Reservation> <Player ¡­ Skill="23" XpLevel="92" /> <Player ¡­ Skill="28" XpLevel="89" /> <Player ¡­ Skill="25" XpLevel="86" /> </Reservation> <Reservation> <Player ¡­ Skill="19" XpLevel="90" /> </Reservation> <Reservation> <Player ¡­ Skill="28" XpLevel="87" /> </Reservation>
  • 53. Experiment Results ? <SearchResult ¡­ Ping="50" Quality="0.924" /> <SearchResult ¡­ Ping="100" Quality="0.668" /> <SearchResult ¡­ Ping="100" Quality="0.409" /> ? <Reservation> <Player ¡­ Skill="23" XpLevel="92" /> <Player ¡­ Skill="28" XpLevel="89" /> <Player ¡­ Skill="25" XpLevel="86" /> </Reservation> <Reservation> <Player ¡­ Skill="19" XpLevel="90" /> </Reservation> <Reservation> <Player ¡­ Skill="28" XpLevel="87" /> </Reservation>
  • 54. Experiment Results ? <SearchResult ¡­ Ping="50" Quality="0.924" /> <SearchResult ¡­ Ping="100" Quality="0.668" /> <SearchResult ¡­ Ping="100" Quality="0.409" /> ? <Reservation> <Player ¡­ Skill="23" XpLevel="92" /> <Player ¡­ Skill="28" XpLevel="89" /> <Player ¡­ Skill="25" XpLevel="86" /> </Reservation> <Reservation> <Player ¡­ Skill="19" XpLevel="90" /> </Reservation> <Reservation> <Player ¡­ Skill="28" XpLevel="87" /> </Reservation>
  • 56. Production Changes ? Added the ability to place servers in buckets ? Added looping in addition to bucketing ? Added new QoS configuration options ? Changed Quick Match searching to be XP Level based
  • 57. Production Matchmaking Process Request Read Skill Search for Rate Join Match Datacenter Data Matches 0..N Matches ¡°Best¡± Rate Found Search 0..X Matches ? ¡°Any¡± Search 0..Y ¡°Empty¡± Search 0..Z
  • 58. Matchmaking Times Beta Production
  • 59. Takeaways ? Record all input matchmaking variables ? Record all output results ¨C Don¡¯t assume a detail isn¡¯t important ? You can never have too many configurable knobs to turn ¨C We couldn¡¯t change some values in the beta
  • 60. Why I learned to love the beta ? The extra effort up front was offset by removing the need for emergency patches ? Instead of going from 0 to 1 million, we could learn while managing the growth ? We discovered and addressed bottlenecks in every phase of our data collection pipeline ¨C The data doesn¡¯t do you any good if you can¡¯t look at it ? Gears of War 3 launch was our best ever
  • 61. Gears of War 3 Analytics: Optimizing the Online Experience, or How I Learned to Stop Worrying and Love the Beta Special thanks to: Wes Hunt Josh Markiewicz Joe Graf Justin Rowland Epic Games, Inc. Ian Thomas Sam Zamani Twitter: @EpicCog