際際滷

際際滷Share a Scribd company logo
Maximizing Revenue Through
Logging
Albert Ho
About Rumble
Funded by:
Mission:
 Bring AAA quality to F2P
 Play together cross-device/platform
Rumbles Platform
 Played in >200 countries
 Six data centers worldwide
 Hybrid infrastructure
 Play together on:
What this talk is about
User
Acquisition
Funnel Loss Retention
Loss
Your Game Paid
Users
Free
Users
Logging Distinctions
 User behavior
 Game design, retention, virality, monetization
 System behavior
 Maximize potential
 Lower CPA
Thesis
You really arent maximizing revenue
unless you can measure system behavior
Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging
Challenges
 24/7 operations
 Game teams ship code daily
 Being up is not good enough
The Long Tail of Revenue Lost:
Daily Site Problems Affect Weeks of Revenue
Outage
Problem
Recovery
New Installs/Sales
Revenue
Days
How to Detect Problems
 Solving is Harder than it sounds
Signal to noise ratio
+ Indexing and search
+ Multiple game teams shipping code daily
= BIG Headache and worlds biggest chore
The answers are in the logs!
Rumble Logging 1.0
 Graylog, Elasticsearch ftw?
 1 day to setup
 Collecting for 2 game servers
 Few hundred MB a day / 7-day retention time
 Scaling is easy right?
- Bigger hard disks, more ES nodes
Rumble Logging 1.5
 7 days to setup ES cluster
 > 20 servers
 Logs @ TB a day / 3-day retention time
 Log noise incomprehensible
 Very expensive!!!
 Logging cluster broke every day
Key Learnings Log Management is really
Undifferentiated Heavy Lifting
We were spending more on
LOGGING than HOSTING
There must be an expert that
does it better, faster, and cheaper
Rumble Logging 2.0 Requirements
 Must support JSON
 Easy to integrate
 Can scale up and meet retention time goals
 Cost-effective
Enter.
 No more agents!
 REST API  21st century please/no syslogs
 Scalable and Responsive
 Superb partner relationship
Sample Log
Loggly Use Cases
 Log clean-up
 Release monitoring and debugging
 Customer service requests
 Trend analysis
 3rd-party developers
Log Clean-up
 1000s errors per minute noise
 DEBUG, INFO, WARN, ERROR?
 Reduce TBs of logs to GBs
Release Monitoring
 Log snapshots before and after
Release Monitoring
 Game teams own
process
 All logs in one place
 Easy for PMs
Customer Service
Reactive problem solving
 Why is customer x slow?
 Why did I not get my gems?
!
Proactive debugging
 These 20 customers were affected by
todays issue.
Trend Analysis
Game performance analysis over time
Client Lag
Days
3rd Party Development
 Bake logging into games at the beginning
 Including 3rd party ones
Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging
Loggly Live Demo
Loggly Before And After
Before After
Log Retention 3 days 15 days
Log Availability 50% uptime >99.9%
Debugging Speed Days Minutes
Happiness Factor
Questions?
albert.ho@rumbleentertainment.com
KingsRoad
http://kingsroad.com
http://apps.facebook.com/kingsroadgame
Ballistic
https://apps.facebook.com/ballistic
Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging
Ad

Recommended

Space Ape's Analytics Stack
Space Ape's Analytics Stack
Simon Hade
Technical Disruption and a New Golden Era of Games
Technical Disruption and a New Golden Era of Games
James Gwertzman
Epic Fails in LiveOps
Epic Fails in LiveOps
James Gwertzman
How To Manage Zero Clients
How To Manage Zero Clients
Teradici
Captain Strike Post-mortem
Captain Strike Post-mortem
JOY Entertainment
6 Critical SaaS Engineering Mistakes to Avoid
6 Critical SaaS Engineering Mistakes to Avoid
SolarWinds Loggly
Why @Loggly Loves Apache Kafka, and How We Use Its Unbreakable Messaging for ...
Why @Loggly Loves Apache Kafka, and How We Use Its Unbreakable Messaging for ...
SolarWinds Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
SolarWinds Loggly
[NDC 2011] 蟆 螳覦襯 一危磯
[NDC 2011] 蟆 螳覦襯 一危磯
Hoon Park
[5Rocks] 蟆一危 覿 碁碁
[5Rocks] 蟆一危 覿 碁碁
Tapjoy X 5Rocks
GA襦 蟆 襦蠏 覿蠍
GA襦 蟆 襦蠏 覿蠍
Alan Kang
NDC 2015 求メ釈п釈求メメ & 殊瑚伎
NDC 2015 求メ釈п釈求メメ & 殊瑚伎
Paul Jung
[磯Μ螳 一危磯ゼ 磯 覯] 覈覦 蟆 襦蠏 一危 覿 伎手鍵 - 磯危 螻旧覦
[磯Μ螳 一危磯ゼ 磯 覯] 覈覦 蟆 襦蠏 一危 覿 伎手鍵 - 磯危 螻旧覦
Dylan Ko
[NDC 覦] 覈覦 蟆一危磯 覦 れ
[NDC 覦] 覈覦 蟆一危磯 覦 れ
Tapjoy X 5Rocks
Loggly - Tools and Techniques For Logging Microservices
Loggly - Tools and Techniques For Logging Microservices
SolarWinds Loggly
Loggly - 5 Popular .NET Logging Libraries
Loggly - 5 Popular .NET Logging Libraries
SolarWinds Loggly
Loggly - IT Operations in a Serverless World (Infographic)
Loggly - IT Operations in a Serverless World (Infographic)
SolarWinds Loggly
Loggly - Case Study - Loggly and Docker Deliver Powerful Monitoring for XAPPm...
Loggly - Case Study - Loggly and Docker Deliver Powerful Monitoring for XAPPm...
SolarWinds Loggly
Loggly - Case Study - Stanley Black & Decker Transforms Work with Support fro...
Loggly - Case Study - Stanley Black & Decker Transforms Work with Support fro...
SolarWinds Loggly
Loggly - Case Study - Loggly and Kubernetes Give Molecule Easy Access to the ...
Loggly - Case Study - Loggly and Kubernetes Give Molecule Easy Access to the ...
SolarWinds Loggly
Loggly - Case Study - Datami Keeps Developer Productivity High with Loggly
Loggly - Case Study - Datami Keeps Developer Productivity High with Loggly
SolarWinds Loggly
Loggly - Case Study - BEMOBI - Bemobi Monitors the Experience of 500 Million ...
Loggly - Case Study - BEMOBI - Bemobi Monitors the Experience of 500 Million ...
SolarWinds Loggly
Loggly - How to Scale Your Architecture and DevOps Practices for Big Data App...
Loggly - How to Scale Your Architecture and DevOps Practices for Big Data App...
SolarWinds Loggly
Loggly - Benchmarking 5 Node.js Logging Libraries
Loggly - Benchmarking 5 Node.js Logging Libraries
SolarWinds Loggly
Humans vs AI Call Agents - Qcall.ai's Special Report
Humans vs AI Call Agents - Qcall.ai's Special Report
Udit Goenka
Heat Treatment Process Automation in India
Heat Treatment Process Automation in India
Reckers Mechatronics
Which Hiring Management Tools Offer the Best ROI?
Which Hiring Management Tools Offer the Best ROI?
HireME
Test Case Design Techniques Practical Examples & Best Practices in Software...
Test Case Design Techniques Practical Examples & Best Practices in Software...
Muhammad Fahad Bashir
Modern Platform Engineering with Choreo - The AI-Native Internal Developer Pl...
Modern Platform Engineering with Choreo - The AI-Native Internal Developer Pl...
WSO2
Best Practice for LLM Serving in the Cloud
Best Practice for LLM Serving in the Cloud
Alluxio, Inc.

More Related Content

Viewers also liked (6)

[NDC 2011] 蟆 螳覦襯 一危磯
[NDC 2011] 蟆 螳覦襯 一危磯
Hoon Park
[5Rocks] 蟆一危 覿 碁碁
[5Rocks] 蟆一危 覿 碁碁
Tapjoy X 5Rocks
GA襦 蟆 襦蠏 覿蠍
GA襦 蟆 襦蠏 覿蠍
Alan Kang
NDC 2015 求メ釈п釈求メメ & 殊瑚伎
NDC 2015 求メ釈п釈求メメ & 殊瑚伎
Paul Jung
[磯Μ螳 一危磯ゼ 磯 覯] 覈覦 蟆 襦蠏 一危 覿 伎手鍵 - 磯危 螻旧覦
[磯Μ螳 一危磯ゼ 磯 覯] 覈覦 蟆 襦蠏 一危 覿 伎手鍵 - 磯危 螻旧覦
Dylan Ko
[NDC 覦] 覈覦 蟆一危磯 覦 れ
[NDC 覦] 覈覦 蟆一危磯 覦 れ
Tapjoy X 5Rocks
[NDC 2011] 蟆 螳覦襯 一危磯
[NDC 2011] 蟆 螳覦襯 一危磯
Hoon Park
[5Rocks] 蟆一危 覿 碁碁
[5Rocks] 蟆一危 覿 碁碁
Tapjoy X 5Rocks
GA襦 蟆 襦蠏 覿蠍
GA襦 蟆 襦蠏 覿蠍
Alan Kang
NDC 2015 求メ釈п釈求メメ & 殊瑚伎
NDC 2015 求メ釈п釈求メメ & 殊瑚伎
Paul Jung
[磯Μ螳 一危磯ゼ 磯 覯] 覈覦 蟆 襦蠏 一危 覿 伎手鍵 - 磯危 螻旧覦
[磯Μ螳 一危磯ゼ 磯 覯] 覈覦 蟆 襦蠏 一危 覿 伎手鍵 - 磯危 螻旧覦
Dylan Ko
[NDC 覦] 覈覦 蟆一危磯 覦 れ
[NDC 覦] 覈覦 蟆一危磯 覦 れ
Tapjoy X 5Rocks

More from SolarWinds Loggly (10)

Loggly - Tools and Techniques For Logging Microservices
Loggly - Tools and Techniques For Logging Microservices
SolarWinds Loggly
Loggly - 5 Popular .NET Logging Libraries
Loggly - 5 Popular .NET Logging Libraries
SolarWinds Loggly
Loggly - IT Operations in a Serverless World (Infographic)
Loggly - IT Operations in a Serverless World (Infographic)
SolarWinds Loggly
Loggly - Case Study - Loggly and Docker Deliver Powerful Monitoring for XAPPm...
Loggly - Case Study - Loggly and Docker Deliver Powerful Monitoring for XAPPm...
SolarWinds Loggly
Loggly - Case Study - Stanley Black & Decker Transforms Work with Support fro...
Loggly - Case Study - Stanley Black & Decker Transforms Work with Support fro...
SolarWinds Loggly
Loggly - Case Study - Loggly and Kubernetes Give Molecule Easy Access to the ...
Loggly - Case Study - Loggly and Kubernetes Give Molecule Easy Access to the ...
SolarWinds Loggly
Loggly - Case Study - Datami Keeps Developer Productivity High with Loggly
Loggly - Case Study - Datami Keeps Developer Productivity High with Loggly
SolarWinds Loggly
Loggly - Case Study - BEMOBI - Bemobi Monitors the Experience of 500 Million ...
Loggly - Case Study - BEMOBI - Bemobi Monitors the Experience of 500 Million ...
SolarWinds Loggly
Loggly - How to Scale Your Architecture and DevOps Practices for Big Data App...
Loggly - How to Scale Your Architecture and DevOps Practices for Big Data App...
SolarWinds Loggly
Loggly - Benchmarking 5 Node.js Logging Libraries
Loggly - Benchmarking 5 Node.js Logging Libraries
SolarWinds Loggly
Loggly - Tools and Techniques For Logging Microservices
Loggly - Tools and Techniques For Logging Microservices
SolarWinds Loggly
Loggly - 5 Popular .NET Logging Libraries
Loggly - 5 Popular .NET Logging Libraries
SolarWinds Loggly
Loggly - IT Operations in a Serverless World (Infographic)
Loggly - IT Operations in a Serverless World (Infographic)
SolarWinds Loggly
Loggly - Case Study - Loggly and Docker Deliver Powerful Monitoring for XAPPm...
Loggly - Case Study - Loggly and Docker Deliver Powerful Monitoring for XAPPm...
SolarWinds Loggly
Loggly - Case Study - Stanley Black & Decker Transforms Work with Support fro...
Loggly - Case Study - Stanley Black & Decker Transforms Work with Support fro...
SolarWinds Loggly
Loggly - Case Study - Loggly and Kubernetes Give Molecule Easy Access to the ...
Loggly - Case Study - Loggly and Kubernetes Give Molecule Easy Access to the ...
SolarWinds Loggly
Loggly - Case Study - Datami Keeps Developer Productivity High with Loggly
Loggly - Case Study - Datami Keeps Developer Productivity High with Loggly
SolarWinds Loggly
Loggly - Case Study - BEMOBI - Bemobi Monitors the Experience of 500 Million ...
Loggly - Case Study - BEMOBI - Bemobi Monitors the Experience of 500 Million ...
SolarWinds Loggly
Loggly - How to Scale Your Architecture and DevOps Practices for Big Data App...
Loggly - How to Scale Your Architecture and DevOps Practices for Big Data App...
SolarWinds Loggly
Loggly - Benchmarking 5 Node.js Logging Libraries
Loggly - Benchmarking 5 Node.js Logging Libraries
SolarWinds Loggly
Ad

Recently uploaded (20)

Humans vs AI Call Agents - Qcall.ai's Special Report
Humans vs AI Call Agents - Qcall.ai's Special Report
Udit Goenka
Heat Treatment Process Automation in India
Heat Treatment Process Automation in India
Reckers Mechatronics
Which Hiring Management Tools Offer the Best ROI?
Which Hiring Management Tools Offer the Best ROI?
HireME
Test Case Design Techniques Practical Examples & Best Practices in Software...
Test Case Design Techniques Practical Examples & Best Practices in Software...
Muhammad Fahad Bashir
Modern Platform Engineering with Choreo - The AI-Native Internal Developer Pl...
Modern Platform Engineering with Choreo - The AI-Native Internal Developer Pl...
WSO2
Best Practice for LLM Serving in the Cloud
Best Practice for LLM Serving in the Cloud
Alluxio, Inc.
Why Every Growing Business Needs a Staff Augmentation Company IN USA.pdf
Why Every Growing Business Needs a Staff Augmentation Company IN USA.pdf
mary rojas
Microsoft-365-Administrator-s-Guide1.pdf
Microsoft-365-Administrator-s-Guide1.pdf
mazharatknl
Canva Pro Crack Free Download 2025-FREE LATEST
Canva Pro Crack Free Download 2025-FREE LATEST
grete1122g
Simplify Task, Team, and Project Management with Orangescrum Work
Simplify Task, Team, and Project Management with Orangescrum Work
Orangescrum
On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?
On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?
Hassan Abid
AI for PV: Development and Governance for a Regulated Industry
AI for PV: Development and Governance for a Regulated Industry
Biologit
Azure AI Foundry: The AI app and agent factory
Azure AI Foundry: The AI app and agent factory
Maxim Salnikov
ERP Systems in the UAE: Driving Business Transformation with Smart Solutions
ERP Systems in the UAE: Driving Business Transformation with Smart Solutions
dheeodoo
MOVIE RECOMMENDATION SYSTEM, UDUMULA GOPI REDDY, Y24MC13085.pptx
MOVIE RECOMMENDATION SYSTEM, UDUMULA GOPI REDDY, Y24MC13085.pptx
Maharshi Mallela
Streamlining CI/CD with FME Flow: A Practical Guide
Streamlining CI/CD with FME Flow: A Practical Guide
Safe Software
Why Edge Computing Matters in Mobile Application Tech.pdf
Why Edge Computing Matters in Mobile Application Tech.pdf
IMG Global Infotech
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
arabelatso
Enable Your Cloud Journey With Microsoft Trusted Partner | IFI Tech
Enable Your Cloud Journey With Microsoft Trusted Partner | IFI Tech
IFI Techsolutions
Sysinfo OST to PST Converter Infographic
Sysinfo OST to PST Converter Infographic
SysInfo Tools
Humans vs AI Call Agents - Qcall.ai's Special Report
Humans vs AI Call Agents - Qcall.ai's Special Report
Udit Goenka
Heat Treatment Process Automation in India
Heat Treatment Process Automation in India
Reckers Mechatronics
Which Hiring Management Tools Offer the Best ROI?
Which Hiring Management Tools Offer the Best ROI?
HireME
Test Case Design Techniques Practical Examples & Best Practices in Software...
Test Case Design Techniques Practical Examples & Best Practices in Software...
Muhammad Fahad Bashir
Modern Platform Engineering with Choreo - The AI-Native Internal Developer Pl...
Modern Platform Engineering with Choreo - The AI-Native Internal Developer Pl...
WSO2
Best Practice for LLM Serving in the Cloud
Best Practice for LLM Serving in the Cloud
Alluxio, Inc.
Why Every Growing Business Needs a Staff Augmentation Company IN USA.pdf
Why Every Growing Business Needs a Staff Augmentation Company IN USA.pdf
mary rojas
Microsoft-365-Administrator-s-Guide1.pdf
Microsoft-365-Administrator-s-Guide1.pdf
mazharatknl
Canva Pro Crack Free Download 2025-FREE LATEST
Canva Pro Crack Free Download 2025-FREE LATEST
grete1122g
Simplify Task, Team, and Project Management with Orangescrum Work
Simplify Task, Team, and Project Management with Orangescrum Work
Orangescrum
On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?
On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?
Hassan Abid
AI for PV: Development and Governance for a Regulated Industry
AI for PV: Development and Governance for a Regulated Industry
Biologit
Azure AI Foundry: The AI app and agent factory
Azure AI Foundry: The AI app and agent factory
Maxim Salnikov
ERP Systems in the UAE: Driving Business Transformation with Smart Solutions
ERP Systems in the UAE: Driving Business Transformation with Smart Solutions
dheeodoo
MOVIE RECOMMENDATION SYSTEM, UDUMULA GOPI REDDY, Y24MC13085.pptx
MOVIE RECOMMENDATION SYSTEM, UDUMULA GOPI REDDY, Y24MC13085.pptx
Maharshi Mallela
Streamlining CI/CD with FME Flow: A Practical Guide
Streamlining CI/CD with FME Flow: A Practical Guide
Safe Software
Why Edge Computing Matters in Mobile Application Tech.pdf
Why Edge Computing Matters in Mobile Application Tech.pdf
IMG Global Infotech
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
arabelatso
Enable Your Cloud Journey With Microsoft Trusted Partner | IFI Tech
Enable Your Cloud Journey With Microsoft Trusted Partner | IFI Tech
IFI Techsolutions
Sysinfo OST to PST Converter Infographic
Sysinfo OST to PST Converter Infographic
SysInfo Tools
Ad

Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging