際際滷

際際滷Share a Scribd company logo

Scaling Source Control for the Next Generation of Game Development

Toby Roberts
Toby Roberts

EA tested different source control configurations to improve scalability for next generation game development. Linux with XFS outperformed Windows, with database operations 5-10x faster and less degradation under load. The new configuration provided major improvements to stability, performance and productivity across EA. Future work includes clustering, RAID 50 and integrating binary delta copying.

1 of 30
Download to read offline
Scaling Source Control for the Next Generation of Game Development Mike Sundy Toby Roberts 2007 Perforce Users Conference Scaling Source Control for NextGen Game Development 1
Overview Current Gen vs. NextGen Test Methodology Test Results Lessons Learned Future of P4 at EA Scaling Source Control for NextGen Game Development 2
Challenges in Game Industry CurrentGen vs. NextGen 9 GB for DVD's for CurrentGen. (Xbox, PS2) 30-50 GB for NextGen (Blu-Ray, HD-DVD XBox 360, PS3) - 3-5x increase in storage capacity Asset Volume Explosion 50k clientspec grew to 500k clientspec Build Farms 20-30 machines grew to 100+ Team Sizes Outsourcing Scaling Source Control for NextGen Game Development 3
EA Redwood Shores (EARS) P4 Installation Metrics 11 Perforce Servers, 1200 users (90+ servers, 4,000+ users at EA) 7 million files on one server, 80 GB db.have Store all source code and source data in P4 1.5 TB on one server, 7 TB total P4 storage Avg. client workspace 80-120GB and 250k-500k files 2,500 66,000 syncs per day Titles include Sims, Godfather, Simpsons, and Spore. Scaling Source Control for NextGen Game Development 4
Problems with Windows 2 GB per process memory limit Filehandle leak (crashed up to 6 times per day) Performance during large concurrent syncs 40 GB sync went from 40 min. to 3+ hours NTFS Threads and troubleshooting Patching Scaling Source Control for NextGen Game Development 5
Goals of Project Improve stability Improve scalability Solution that works for our dev environment, infrastructure, and team cultures Scaling Source Control for NextGen Game Development 6
Parameters of Project Dell and STK hardware Intel RedHat Windows Desktops 95% of user machines Gig network Scaling Source Control for NextGen Game Development 7
Test Methodology Single operation vs. load P4 benchmarking tools fstst and locktest Homegrown scripts Force sync of 39 GB data, 180k files p4.exe on Windows Scaling Source Control for NextGen Game Development 8
Configuration Overview Tested 21 different server configurations. Windows vs. Linux 32-bit vs. 64-bit NTFS v. ext3 v. ReiserFS v. XFS 10k vs. 15k drives Scaling Source Control for NextGen Game Development 9
db locktest 6 5 4 Seconds 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 10
db 8k read-write 600 500 400 MBs 300 200 100 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 11
rcs 64k write 900 800 700 600 500 MBs 400 300 200 100 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 12
rcs 64k read 3500 3000 2500 2000 MBs 1500 1000 500 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 13
1 force sync avg. 21 19 17 15 13 config 11 9 7 5 3 1 0 5 10 15 20 25 30 35 40 45 50 minutes Scaling Source Control for NextGen Game Development 14
3 force syncs avg. 21 19 17 15 13 config 11 9 7 5 3 1 0 10 20 30 40 50 60 70 80 minutes Scaling Source Control for NextGen Game Development 15
5 force syncs avg. 21 19 17 15 13 config 11 9 7 5 3 1 0 10 20 30 40 50 60 70 80 minutes Scaling Source Control for NextGen Game Development 16
% degradation from 1 to 3 syncs 21 19 17 15 13 config 11 9 7 5 3 1 0 0.1 0.2 0.3 0.4 0.5 0.6 percent Scaling Source Control for NextGen Game Development 17
edit loop avg. (176 files) 21 19 17 15 13 config 11 9 7 5 3 1 0 0.2 0.4 0.6 0.8 1 1.2 seconds Scaling Source Control for NextGen Game Development 18
Test Results Summary DB operations 5-10x faster on 64-bit Linux on XFS than Windows 32-bit. Linux sync performance under load degraded only 5% instead of the 450% we saw with Windows. XFS outperforms ext3 and even edges ReiserFS for data syncs from the repository volume. Scaling Source Control for NextGen Game Development 19
dm-CommitSubmit config linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 seconds Scaling Source Control for NextGen Game Development 20
avg. edit config linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 seconds Scaling Source Control for NextGen Game Development 21
avg. fstat config linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.1 0.2 0.3 0.4 0.5 0.6 seconds Scaling Source Control for NextGen Game Development 22
avg. sync compute phase config linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 seconds Scaling Source Control for NextGen Game Development 23
Configuration Lessons Learned OS 2.6 kernel was 2 4x faster than 2.4 in some tests. RAID RAID 5 best for RCS due to the high number of syncs/reads. DB benefited from RAID 1 or RAID 10. Filesystem XFS proved to be significantly faster than ext3 and beat Reiser by 30% in some tests. Drive Speed 20% less degradation with 15k drives during multiple concurrent forced syncs due to higher I/O capacity. Scaling Source Control for NextGen Game Development 24
Performance Principles Keep your dbs on their own disks Proxy heavy sync users such as build farms or remote locations Best place to improve hardware performance is memory and disk I/O for dbs There is a big difference in filesystem speed Scaling Source Control for NextGen Game Development 25
Success Stories New standard has had a huge impact on our local team productivity and P4 performance across EA. New platform has scaled well for 1,000 user team and could scale to larger teams. Stability is great! No more memory or file descriptor limits Can identify and kill individual threads. $5 million saved per year in productivity across EA. Scaling Source Control for NextGen Game Development 26
NextGen EA P4 Architecture P4 Server setup Distributed Development Proxy Build Farm (30% gain) Case-insensitivity Disk to disk mirroring Scaling Source Control for NextGen Game Development 27
The Future of Perforce at EA RAID 50 (2x faster locking than RAID 10) SAS SFF drives AD2P4 Multi-Node Cluster Perforce + binary delta copy technology Workspace mirroring Single View of Perforce Servers Scaling Source Control for NextGen Game Development 28
Thanks Michael Shields Brett Taylor Kenny Wu Scaling Source Control for NextGen Game Development 29
Trivia for Schwag Questions? Scaling Source Control for NextGen Game Development 30

More Related Content

Scaling Source Control for the Next Generation of Game Development

  • 1. Scaling Source Control for the Next Generation of Game Development Mike Sundy Toby Roberts 2007 Perforce Users Conference Scaling Source Control for NextGen Game Development 1
  • 2. Overview Current Gen vs. NextGen Test Methodology Test Results Lessons Learned Future of P4 at EA Scaling Source Control for NextGen Game Development 2
  • 3. Challenges in Game Industry CurrentGen vs. NextGen 9 GB for DVD's for CurrentGen. (Xbox, PS2) 30-50 GB for NextGen (Blu-Ray, HD-DVD XBox 360, PS3) - 3-5x increase in storage capacity Asset Volume Explosion 50k clientspec grew to 500k clientspec Build Farms 20-30 machines grew to 100+ Team Sizes Outsourcing Scaling Source Control for NextGen Game Development 3
  • 4. EA Redwood Shores (EARS) P4 Installation Metrics 11 Perforce Servers, 1200 users (90+ servers, 4,000+ users at EA) 7 million files on one server, 80 GB db.have Store all source code and source data in P4 1.5 TB on one server, 7 TB total P4 storage Avg. client workspace 80-120GB and 250k-500k files 2,500 66,000 syncs per day Titles include Sims, Godfather, Simpsons, and Spore. Scaling Source Control for NextGen Game Development 4
  • 5. Problems with Windows 2 GB per process memory limit Filehandle leak (crashed up to 6 times per day) Performance during large concurrent syncs 40 GB sync went from 40 min. to 3+ hours NTFS Threads and troubleshooting Patching Scaling Source Control for NextGen Game Development 5
  • 6. Goals of Project Improve stability Improve scalability Solution that works for our dev environment, infrastructure, and team cultures Scaling Source Control for NextGen Game Development 6
  • 7. Parameters of Project Dell and STK hardware Intel RedHat Windows Desktops 95% of user machines Gig network Scaling Source Control for NextGen Game Development 7
  • 8. Test Methodology Single operation vs. load P4 benchmarking tools fstst and locktest Homegrown scripts Force sync of 39 GB data, 180k files p4.exe on Windows Scaling Source Control for NextGen Game Development 8
  • 9. Configuration Overview Tested 21 different server configurations. Windows vs. Linux 32-bit vs. 64-bit NTFS v. ext3 v. ReiserFS v. XFS 10k vs. 15k drives Scaling Source Control for NextGen Game Development 9
  • 10. db locktest 6 5 4 Seconds 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 10
  • 11. db 8k read-write 600 500 400 MBs 300 200 100 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 11
  • 12. rcs 64k write 900 800 700 600 500 MBs 400 300 200 100 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 12
  • 13. rcs 64k read 3500 3000 2500 2000 MBs 1500 1000 500 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 13
  • 14. 1 force sync avg. 21 19 17 15 13 config 11 9 7 5 3 1 0 5 10 15 20 25 30 35 40 45 50 minutes Scaling Source Control for NextGen Game Development 14
  • 15. 3 force syncs avg. 21 19 17 15 13 config 11 9 7 5 3 1 0 10 20 30 40 50 60 70 80 minutes Scaling Source Control for NextGen Game Development 15
  • 16. 5 force syncs avg. 21 19 17 15 13 config 11 9 7 5 3 1 0 10 20 30 40 50 60 70 80 minutes Scaling Source Control for NextGen Game Development 16
  • 17. % degradation from 1 to 3 syncs 21 19 17 15 13 config 11 9 7 5 3 1 0 0.1 0.2 0.3 0.4 0.5 0.6 percent Scaling Source Control for NextGen Game Development 17
  • 18. edit loop avg. (176 files) 21 19 17 15 13 config 11 9 7 5 3 1 0 0.2 0.4 0.6 0.8 1 1.2 seconds Scaling Source Control for NextGen Game Development 18
  • 19. Test Results Summary DB operations 5-10x faster on 64-bit Linux on XFS than Windows 32-bit. Linux sync performance under load degraded only 5% instead of the 450% we saw with Windows. XFS outperforms ext3 and even edges ReiserFS for data syncs from the repository volume. Scaling Source Control for NextGen Game Development 19
  • 20. dm-CommitSubmit config linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 seconds Scaling Source Control for NextGen Game Development 20
  • 21. avg. edit config linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 seconds Scaling Source Control for NextGen Game Development 21
  • 22. avg. fstat config linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.1 0.2 0.3 0.4 0.5 0.6 seconds Scaling Source Control for NextGen Game Development 22
  • 23. avg. sync compute phase config linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 seconds Scaling Source Control for NextGen Game Development 23
  • 24. Configuration Lessons Learned OS 2.6 kernel was 2 4x faster than 2.4 in some tests. RAID RAID 5 best for RCS due to the high number of syncs/reads. DB benefited from RAID 1 or RAID 10. Filesystem XFS proved to be significantly faster than ext3 and beat Reiser by 30% in some tests. Drive Speed 20% less degradation with 15k drives during multiple concurrent forced syncs due to higher I/O capacity. Scaling Source Control for NextGen Game Development 24
  • 25. Performance Principles Keep your dbs on their own disks Proxy heavy sync users such as build farms or remote locations Best place to improve hardware performance is memory and disk I/O for dbs There is a big difference in filesystem speed Scaling Source Control for NextGen Game Development 25
  • 26. Success Stories New standard has had a huge impact on our local team productivity and P4 performance across EA. New platform has scaled well for 1,000 user team and could scale to larger teams. Stability is great! No more memory or file descriptor limits Can identify and kill individual threads. $5 million saved per year in productivity across EA. Scaling Source Control for NextGen Game Development 26
  • 27. NextGen EA P4 Architecture P4 Server setup Distributed Development Proxy Build Farm (30% gain) Case-insensitivity Disk to disk mirroring Scaling Source Control for NextGen Game Development 27
  • 28. The Future of Perforce at EA RAID 50 (2x faster locking than RAID 10) SAS SFF drives AD2P4 Multi-Node Cluster Perforce + binary delta copy technology Workspace mirroring Single View of Perforce Servers Scaling Source Control for NextGen Game Development 28
  • 29. Thanks Michael Shields Brett Taylor Kenny Wu Scaling Source Control for NextGen Game Development 29
  • 30. Trivia for Schwag Questions? Scaling Source Control for NextGen Game Development 30