Scaling Source Control for the Next Generation of Game Development
1 like298 views
EA tested different source control configurations to improve scalability for next generation game development. Linux with XFS outperformed Windows, with database operations 5-10x faster and less degradation under load. The new configuration provided major improvements to stability, performance and productivity across EA. Future work includes clustering, RAID 50 and integrating binary delta copying.
![[KGC 2012] Online Game Server Architecture Case Study Performance and Security](https://cdn.slidesharecdn.com/ss_thumbnails/kgc2012-20121009-new-18-f-sladeshare-130927202856-phpapp01-thumbnail.jpg?width=560&fit=bounds)
![Fast Screen Recorder v2.1.0.11 Crack Updated [April-2025]](https://cdn.slidesharecdn.com/ss_thumbnails/dataanalysisforbusiness-250322061148-eeff8a831-250401123246-f36be9ca-250401141614-87b01766-thumbnail.jpg?width=560&fit=bounds)
More Related Content
Similar to Scaling Source Control for the Next Generation of Game Development (6)
Recently uploaded (20)
Scaling Source Control for the Next Generation of Game Development
- 1. Scaling Source Control for the Next Generation of Game Development Mike Sundy Toby Roberts 2007 Perforce User¡¯s Conference Scaling Source Control for NextGen Game Development 1
- 2. Overview ? Current Gen vs. NextGen ? Test Methodology ? Test Results ? Lessons Learned ? Future of P4 at EA Scaling Source Control for NextGen Game Development 2
- 3. Challenges in Game Industry ? CurrentGen vs. NextGen ? 9 GB for DVD's for CurrentGen. (Xbox, PS2) ? 30-50 GB for NextGen (Blu-Ray, HD-DVD ¨C XBox 360, PS3) - 3-5x increase in storage capacity ? Asset Volume Explosion ? 50k clientspec grew to 500k clientspec ? Build Farms ? 20-30 machines grew to 100+ ? Team Sizes ? Outsourcing Scaling Source Control for NextGen Game Development 3
- 4. EA Redwood Shores (EARS) P4 Installation Metrics ? 11 Perforce Servers, 1200 users (90+ servers, 4,000+ users at EA) ? 7 million files on one server, 80 GB db.have ? Store all source code and source data in P4 ? 1.5 TB on one server, 7 TB total P4 storage ? Avg. client workspace ¨C 80-120GB and 250k-500k files ? 2,500 ¨C 66,000 syncs per day ? Titles include Sims, Godfather, Simpsons, and Spore. Scaling Source Control for NextGen Game Development 4
- 5. Problems with Windows ? 2 GB per process memory limit ? Filehandle leak (crashed up to 6 times per day) ? Performance during large concurrent syncs ? 40 GB sync went from 40 min. to 3+ hours ? NTFS ? Threads and troubleshooting ? Patching Scaling Source Control for NextGen Game Development 5
- 6. Goals of Project ? Improve stability ? Improve scalability ? Solution that works for our dev environment, infrastructure, and team cultures Scaling Source Control for NextGen Game Development 6
- 7. Parameters of Project ? Dell and STK hardware ? Intel ? RedHat ? Windows Desktops ? 95% of user machines ? Gig network Scaling Source Control for NextGen Game Development 7
- 8. Test Methodology ? Single operation vs. load ? P4 benchmarking tools ? fstst and locktest ? Homegrown scripts ? Force sync of 39 GB data, 180k files ? p4.exe on Windows Scaling Source Control for NextGen Game Development 8
- 9. Configuration Overview ? Tested 21 different server configurations. ? Windows vs. Linux ? 32-bit vs. 64-bit ? NTFS v. ext3 v. ReiserFS v. XFS ? 10k vs. 15k drives Scaling Source Control for NextGen Game Development 9
- 10. db locktest 6 5 4 Seconds 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 10
- 11. db 8k read-write 600 500 400 MBs 300 200 100 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 11
- 12. rcs 64k write 900 800 700 600 500 MBs 400 300 200 100 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 12
- 13. rcs 64k read 3500 3000 2500 2000 MBs 1500 1000 500 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 13
- 14. 1 force sync avg. 21 19 17 15 13 config 11 9 7 5 3 1 0 5 10 15 20 25 30 35 40 45 50 minutes Scaling Source Control for NextGen Game Development 14
- 15. 3 force syncs avg. 21 19 17 15 13 config 11 9 7 5 3 1 0 10 20 30 40 50 60 70 80 minutes Scaling Source Control for NextGen Game Development 15
- 16. 5 force syncs avg. 21 19 17 15 13 config 11 9 7 5 3 1 0 10 20 30 40 50 60 70 80 minutes Scaling Source Control for NextGen Game Development 16
- 17. % degradation from 1 to 3 syncs 21 19 17 15 13 config 11 9 7 5 3 1 0 0.1 0.2 0.3 0.4 0.5 0.6 percent Scaling Source Control for NextGen Game Development 17
- 18. edit loop avg. (176 files) 21 19 17 15 13 config 11 9 7 5 3 1 0 0.2 0.4 0.6 0.8 1 1.2 seconds Scaling Source Control for NextGen Game Development 18
- 19. Test Results Summary ? DB operations 5-10x faster on 64-bit Linux on XFS than Windows 32-bit. ? Linux sync performance under load degraded only 5% instead of the 450% we saw with Windows. ? XFS outperforms ext3 and even edges ReiserFS for data syncs from the repository volume. Scaling Source Control for NextGen Game Development 19
- 20. dm-CommitSubmit config linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 seconds Scaling Source Control for NextGen Game Development 20
- 21. avg. edit config linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 seconds Scaling Source Control for NextGen Game Development 21
- 22. avg. fstat config linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.1 0.2 0.3 0.4 0.5 0.6 seconds Scaling Source Control for NextGen Game Development 22
- 23. avg. sync compute phase config linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 seconds Scaling Source Control for NextGen Game Development 23
- 24. Configuration Lessons Learned ? OS ? 2.6 kernel was 2 ¨C 4x faster than 2.4 in some tests. ? RAID ? RAID 5 best for RCS due to the high number of syncs/reads. ? DB benefited from RAID 1 or RAID 10. ? Filesystem ? XFS proved to be significantly faster than ext3 and beat Reiser by 30% in some tests. ? Drive Speed ? 20% less degradation with 15k drives during multiple concurrent forced syncs due to higher I/O capacity. Scaling Source Control for NextGen Game Development 24
- 25. Performance Principles ? Keep your db¡¯s on their own disks ? Proxy heavy sync users such as build farms or remote locations ? Best place to improve hardware performance is memory and disk I/O for db¡¯s ? There is a big difference in filesystem speed Scaling Source Control for NextGen Game Development 25
- 26. Success Stories ? New standard has had a huge impact on our local team productivity and P4 performance across EA. ? New platform has scaled well for 1,000 user team and could scale to larger teams. ? Stability is great! ? No more memory or file descriptor limits ? Can identify and kill individual threads. ? $5 million saved per year in productivity across EA. Scaling Source Control for NextGen Game Development 26
- 27. NextGen EA P4 Architecture ? P4 Server setup ? Distributed Development ? Proxy Build Farm (30% gain) ? Case-insensitivity ? Disk to disk mirroring Scaling Source Control for NextGen Game Development 27
- 28. The Future of Perforce at EA ? RAID 50 (2x faster locking than RAID 10) ? SAS SFF drives ? AD2P4 ? Multi-Node Cluster ? Perforce + binary delta copy technology ? Workspace mirroring ? Single View of Perforce Servers Scaling Source Control for NextGen Game Development 28
- 29. Thanks ? Michael Shields ? Brett Taylor ? Kenny Wu Scaling Source Control for NextGen Game Development 29
- 30. Trivia for Schwag Questions? Scaling Source Control for NextGen Game Development 30