際際滷

際際滷Share a Scribd company logo

Erlang factory layered architecture - final

Dennis Docter
Dennis Docter

The document describes a new layered architecture for a social and casual gaming platform. The key points are: 1) The previous architecture had inconsistent designs, unclear responsibilities, technical debt, and scaling issues with 7 layers of caching. 2) The new architecture aims to be flexible, predictable, scalable, simple with clean separation and no horizontal dependencies. 3) It has four main layers - client, interface, service, and storage - each with clear responsibilities around integration, authorization, composition, and persistence. Building blocks are small, simple applications and a platform to abstract complexity.

1 of 43
Downloaded 14 times
A Layered Architecture Robustness & Efficiency dennis.docter@spilgames.com March 22nd, 2013
2
A legacy of growth
Social and casual gaming platform 50+ portals 190+ countries 4
Casual games Multi player games Social games Native games
200+ million unique users per month 250" 200" Traffic growth 150" over 1600% 100" 50" 5/1/12" 1/1/12" 9/1/11" 0" 5/1/11" 1/1/11" 9/1/10" 5/1/10" 1/1/10" 9/1/09" 5/1/09" 1/1/09" 9/1/08" 5/1/08" 1/1/08" 9/1/07" 5/1/07" 1/1/07" 90" 80" 70" 60" Tech dept. 50" 40" growth 30" over 2000% 20" 10" 0" 1/1/08" 1/1/09" 1/1/10" 1/1/11" 1/1/12"
XML Json Interfacing Interfacing + plain GET Intefacing Intefacing Caching SQL Business Caching Authorization + Authentication Business Caching Caching logic logic Authorization Business SQL Business Business logic Caching Caching logic logic Business SQL logic Caching SQL SQL Business SQL logic Swift SQLSchema API Schema Schema Schema Mysql Write DB Mysql DB Mysql DB Mysql DB Mysql Reader Memcache Mysql Mysql DB Cluster Readers 7
Problems Inconsistent design & interfaces Unclear responsibili4es Technical debt Independent teams developing the same features. 7 layers of caching hell Scaling
Problems Inconsistent design & interfaces Unclear responsibili4es Technical debt Independent teams developing the same features. 7 layers of caching hell Scaling not a problem :)
9
A new approach
New architecture expectations Flexible Predictable Scalable TM d wor t B uzz lian p C om 11
Choices made Simple Clean separa4on No horizontal dependencies Fault tolerant Distributed 12
Layers Responsibilities Client Integra4on State Authen4ca4on Interface Aggrega4on Valida4on Authoriza4on Service Composi4on Transforma4on Caching Storage Persistence Sharding 13
Layers Responsibilities Client Integra4on State Authen4ca4on Interface Aggrega4on Stateless Valida4on Authoriza4on Service Composi4on Transforma4on Caching Storage Persistence Sharding 13
Building blocks Application Small & Simple Limited scope Quick to deploy Platform Abstracts complexity Low change rate Long lifetime 14
Walkthrough
Interfaces https post /user/get/ 3rd Party {auth: {token: UwAA_AWeff...} Highscore Backend } Service list 1 get([friends]) Application User 2 User 2 Interface Interface Service list get(me) 1 count /application/highscores/ 1 Friends Service 1 User Service Awards Service 16
Widgets
Widgets logo widget profile widget search widget navigation widget recommendation widget new_games widget most_played widget recent_games widget social_games widget
definition User profile logic Service template + Award html, Service [properties] 18
header definition User logo profile Service logic template navigation search + Award html, Service [properties] 18
ame.com / htt p://www.ag preprocess ... widget platform footer ... home_page new games header body ... definition User logo profile Service logic template navigation search + Award html, Service [properties] 18
Services user {context:{ get interface guid: 62623423, siteid : 88, locale : en_US },guid: 324112535} { gid: 324112535, user service username : -fooba-, avatarId : http://agame.com/avatars/99142.png } check fetch 鍖lter modify { gid: 324112535, username : -fooba-, profile firstName : john storage avatarId : 99142 }
Storage platform API instead of SQL Buckets of data = Key-足Value with Schema Abstracts storage Bucket Bucket Bucket Bucket Bucket Bucket Storage Node 1 Storage Node 2 Cache Cache Shard 1 Shard 2 Shard .. 20
Job handlers Account interface-account.register Audit Interface {guid:7734} Handler Audit User service-user.register Log Service {result:success,guid:73324} Welcome Profile Handler bucket rabbitmq send email 21
Fitting it together
interface JSON Load balancer Auth User Award Pro鍖le Game in@erface interface interface widget widget Protobuf Widget PlaGorm service User User Game Award Award service service service service service Protobuf storage User Game Pro鍖les Pro鍖les Games Award Award bucket bucket bucket bucket bucket Storage PlaGorm 23
How we do it Strictly de鍖ne EVERYTHING input/output format, urls, rpc func4ons, types, valida4on Strict rules on changes Everything op4onal Unless youre really, really sure (and have proof!) Generate code and documenta4on Interface derived from speci鍖ca4on Documenta4on derived from speci鍖ca4on 24
XML: De鍖ni4on XSLT: Transforma4on Piqi: Protobuf & Json encoding/decoding HTTP RPC (h`p://piqi.org/) 25
What we ended up with Small applica4ons running on their own nodes Scale up where necessary Many of them on the same machine Small cluster already contains hundreds of nodes 26
Keeping it connected Connec4ng all nodes is a bad idea: Not needed Number of open connec4ons Solu4on: use hidden nodes Each node speci鍖es the nodes it needs. How does it scale? 27
Hidden nodes vs. visible nodes Hidden" Visible" 160000" Number of TCP connections 145530$ 140000" 120000" 114960$ 100000" 87990$ 80000" 64620$ 60000" 44850$ 40000" 28680$ 16110$ 20000" 7140$ 1770$ 240$ 320$ 400$ 480$ 560$ 640$ 720$ 0" 2" 4" 6" 8" 10" 12" 14" 16" 18" Number of servers (~30 nodes / server) 28
Router Each applica4on has its own router Scans & Monitors Balances requests across nodes Parallelizes requests as much as possible Applica4on can only talk to what it has de鍖ned 29
But what about response times? Use proper transports (Erlang RPC vs. HTTP) Yes there is some transport overhead Typically < 1ms per layer (+network latency) But each layer has predictable response 4mes As long as enough downward resources are available 30
The old ... 31
.. and the new. 32
Closing Words
Where are we now.. Interface layer has been de鍖ned and largely implemented ImplemenMng the service & storage layer step by step Meanwhile we have adapters in place Several of the new APIs are in use in producMon First portals on the new widget plaGorm will be deployed in the coming months 34
... And where were going Deploying to mul4ple datacenters this year Bring elas4city to our deployments Decommission our old architecture Erlang game plahorm 35
Lessons Learned An architectural vision is crucial Integra4ng into a mess is a mess Layers with clear responsibili4es Strict communica4on Package small Change is the only constant Op4mize for predictability 鍖rst 36
Questions 37
Thank you! 38

More Related Content

Erlang factory layered architecture - final

  • 1. A Layered Architecture Robustness & Efficiency dennis.docter@spilgames.com March 22nd, 2013
  • 2. 2
  • 3. A legacy of growth
  • 4. Social and casual gaming platform 50+ portals 190+ countries 4
  • 5. Casual games Multi player games Social games Native games
  • 6. 200+ million unique users per month 250" 200" Traffic growth 150" over 1600% 100" 50" 5/1/12" 1/1/12" 9/1/11" 0" 5/1/11" 1/1/11" 9/1/10" 5/1/10" 1/1/10" 9/1/09" 5/1/09" 1/1/09" 9/1/08" 5/1/08" 1/1/08" 9/1/07" 5/1/07" 1/1/07" 90" 80" 70" 60" Tech dept. 50" 40" growth 30" over 2000% 20" 10" 0" 1/1/08" 1/1/09" 1/1/10" 1/1/11" 1/1/12"
  • 7. XML Json Interfacing Interfacing + plain GET Intefacing Intefacing Caching SQL Business Caching Authorization + Authentication Business Caching Caching logic logic Authorization Business SQL Business Business logic Caching Caching logic logic Business SQL logic Caching SQL SQL Business SQL logic Swift SQLSchema API Schema Schema Schema Mysql Write DB Mysql DB Mysql DB Mysql DB Mysql Reader Memcache Mysql Mysql DB Cluster Readers 7
  • 8. Problems Inconsistent design & interfaces Unclear responsibili4es Technical debt Independent teams developing the same features. 7 layers of caching hell Scaling
  • 9. Problems Inconsistent design & interfaces Unclear responsibili4es Technical debt Independent teams developing the same features. 7 layers of caching hell Scaling not a problem :)
  • 10. 9
  • 12. New architecture expectations Flexible Predictable Scalable TM d wor t B uzz lian p C om 11
  • 13. Choices made Simple Clean separa4on No horizontal dependencies Fault tolerant Distributed 12
  • 14. Layers Responsibilities Client Integra4on State Authen4ca4on Interface Aggrega4on Valida4on Authoriza4on Service Composi4on Transforma4on Caching Storage Persistence Sharding 13
  • 15. Layers Responsibilities Client Integra4on State Authen4ca4on Interface Aggrega4on Stateless Valida4on Authoriza4on Service Composi4on Transforma4on Caching Storage Persistence Sharding 13
  • 16. Building blocks Application Small & Simple Limited scope Quick to deploy Platform Abstracts complexity Low change rate Long lifetime 14
  • 18. Interfaces https post /user/get/ 3rd Party {auth: {token: UwAA_AWeff...} Highscore Backend } Service list 1 get([friends]) Application User 2 User 2 Interface Interface Service list get(me) 1 count /application/highscores/ 1 Friends Service 1 User Service Awards Service 16
  • 20. Widgets logo widget profile widget search widget navigation widget recommendation widget new_games widget most_played widget recent_games widget social_games widget
  • 21. definition User profile logic Service template + Award html, Service [properties] 18
  • 22. header definition User logo profile Service logic template navigation search + Award html, Service [properties] 18
  • 23. ame.com / htt p://www.ag preprocess ... widget platform footer ... home_page new games header body ... definition User logo profile Service logic template navigation search + Award html, Service [properties] 18
  • 24. Services user {context:{ get interface guid: 62623423, siteid : 88, locale : en_US },guid: 324112535} { gid: 324112535, user service username : -fooba-, avatarId : http://agame.com/avatars/99142.png } check fetch 鍖lter modify { gid: 324112535, username : -fooba-, profile firstName : john storage avatarId : 99142 }
  • 25. Storage platform API instead of SQL Buckets of data = Key-足Value with Schema Abstracts storage Bucket Bucket Bucket Bucket Bucket Bucket Storage Node 1 Storage Node 2 Cache Cache Shard 1 Shard 2 Shard .. 20
  • 26. Job handlers Account interface-account.register Audit Interface {guid:7734} Handler Audit User service-user.register Log Service {result:success,guid:73324} Welcome Profile Handler bucket rabbitmq send email 21
  • 28. interface JSON Load balancer Auth User Award Pro鍖le Game in@erface interface interface widget widget Protobuf Widget PlaGorm service User User Game Award Award service service service service service Protobuf storage User Game Pro鍖les Pro鍖les Games Award Award bucket bucket bucket bucket bucket Storage PlaGorm 23
  • 29. How we do it Strictly de鍖ne EVERYTHING input/output format, urls, rpc func4ons, types, valida4on Strict rules on changes Everything op4onal Unless youre really, really sure (and have proof!) Generate code and documenta4on Interface derived from speci鍖ca4on Documenta4on derived from speci鍖ca4on 24
  • 30. XML: De鍖ni4on XSLT: Transforma4on Piqi: Protobuf & Json encoding/decoding HTTP RPC (h`p://piqi.org/) 25
  • 31. What we ended up with Small applica4ons running on their own nodes Scale up where necessary Many of them on the same machine Small cluster already contains hundreds of nodes 26
  • 32. Keeping it connected Connec4ng all nodes is a bad idea: Not needed Number of open connec4ons Solu4on: use hidden nodes Each node speci鍖es the nodes it needs. How does it scale? 27
  • 33. Hidden nodes vs. visible nodes Hidden" Visible" 160000" Number of TCP connections 145530$ 140000" 120000" 114960$ 100000" 87990$ 80000" 64620$ 60000" 44850$ 40000" 28680$ 16110$ 20000" 7140$ 1770$ 240$ 320$ 400$ 480$ 560$ 640$ 720$ 0" 2" 4" 6" 8" 10" 12" 14" 16" 18" Number of servers (~30 nodes / server) 28
  • 34. Router Each applica4on has its own router Scans & Monitors Balances requests across nodes Parallelizes requests as much as possible Applica4on can only talk to what it has de鍖ned 29
  • 35. But what about response times? Use proper transports (Erlang RPC vs. HTTP) Yes there is some transport overhead Typically < 1ms per layer (+network latency) But each layer has predictable response 4mes As long as enough downward resources are available 30
  • 37. .. and the new. 32
  • 39. Where are we now.. Interface layer has been de鍖ned and largely implemented ImplemenMng the service & storage layer step by step Meanwhile we have adapters in place Several of the new APIs are in use in producMon First portals on the new widget plaGorm will be deployed in the coming months 34
  • 40. ... And where were going Deploying to mul4ple datacenters this year Bring elas4city to our deployments Decommission our old architecture Erlang game plahorm 35
  • 41. Lessons Learned An architectural vision is crucial Integra4ng into a mess is a mess Layers with clear responsibili4es Strict communica4on Package small Change is the only constant Op4mize for predictability 鍖rst 36