$460 mn loss in 45 min, and similar stories. will devops save our world
0 likes107 views
Knight Capital lost $460 million in 45 minutes on August 1, 2012 due to a software deployment issue. A new version of code was deployed to only 7 of 8 servers, activating old obsolete "power-peg" functionality that moved stock prices unexpectedly. This caused major disruptions and Knight's stock fell 70%. The incident highlights the importance of proper code deployment practices like reviewing all deployments, addressing errors, and testing impact before manual undeployment. Adopting DevOps practices around configuration, deployments, testing, and monitoring could help prevent such issues.
$460 mn loss in 45 min, and similar stories. will devops save our world
- 1. $460Mn loss in 45 min, and similar stories. Will DevOps save our world?
- 2. Savinder Puri @savinderpuri | savinderpuri@gmail.com
- 3. USD $460 million loss in 45 min! US $210 billion lost in one night!! World War 3 just about averted!!!
- 4. When markets opened on 1st Aug at 9:30am, blood-bath happened in 45 min! On 31st July 2012, Knight deployed a new version of software to Production The new code re-purposed an obsoleted functionality power-pegs SMARS is its high speed routing system for equity orders Knight Capital WAS a global financial services firm. It specialized with its high frequency trading algorithms
- 5. Total, USD $460 min lost. All this in 45 min! Knight Capitals stock fell by over 70%. It was officially termed as a technology breakdown A manual rollback caused power-peg to being activated on all 8 SMARS! This caused a major disruption in prices of 148 Companies listed on NYSE. 4 million child execution orders were inadvertently transacted A technician forgot to copy the new code to one off 8 SMARS server This activated the old PP, that moved stock prices up/down to verify trading behaviour
- 9. Issue 1: Code deployed to only 7 off 8 SMARS servers
- 10. Issue 2: No review of deployments or gates
- 11. Issue 2: Notifications were not marked as errors, thus not taken seriously
- 12. Issue 4: Manual un-deploy carried out without taking cognizance of it's impact
- 13. DevOps a possible saviour!!!
- 14. US $210 billion, or 86 perfect of cash in Asia's 3rd largest economy was scrapped with this single decision.
- 15. We all saw this on TV and in reality!
- 16. Scaling Reliability Frequent Changes Out Experiment This is what was expected, to react successfully to this sudden change
- 17. DevOps a possible saviour!!! Cloud adoption Config-based deployments Canary Deployments Alfa-Beta testing
- 18. Its a fast changing world, today!
- 19. Even with the worlds best tech, it still takes a human with the right values
- 20. We see ourselves as a technology company, with a banking license Why every Company Is A Technology Company Every company is a tech company now (yes, even Blue Apron) Silicon Valley will be competing with us very soon. We are ready
- 21. All Tesla cars have the hardware to be driverless. All we need is software updates!
- 22. Biggest car company in the world. No. of cars owned?
- 23. No. of properties owned?
- 24. Any customer can have a car painted any colour that he wants so long as it is black. Henry Ford