Deco3
Download as key, pdf1 like736 views
The document discusses failures and repairs in datacenters. It describes categorizing faults by severity and cause. Machine-level failures are also examined, specifically what causes machine crashes and how faults can be predicted. Repair processes are then covered, with an emphasis on tolerating rather than hiding faults. Software-based fault tolerance and automated repair systems are key to minimizing downtime due to failures.
Deco3
- 1. The Datacenter as a Computer Chapter 7 2009/12/20 id:daisukebe
- 2. Agenda ? 7 Dealing with Failures and Repairs ? 7.1 Implications of Software-Based Fault Tolerance ? 7.2 Categorizing Faults ? 7.2.1 Fault Severity ? 7.2.2 Causes of Service-Level Faults ? 7.3 Machine-Level Failures ? 7.3.1 What Causes Machine Crashes? ? 7.3.2 Predicting Faults ? 7.4 Repairs ? 7.5 Tolerating Faults, Not Hiding Them
- 3. ? 7 ? 7.1 Implications of Software-Based Fault Tolerance ? 7.2 Categorizing Faults ? 7.2.1 Fault Severity ? 7.2.2 Causes of Service-Level Faults ? 7.3 Machine-Level Failures ? 7.3.1 What Causes Machine Crashes? ? 7.3.2 Predicting Faults ? 7.4 Repairs ? 7.5 Tolerating Faults, Not Hiding Them
- 4. ? => ? H/W ? WSC H/W ? MTBF 30 -> 10000 1 1 ? WSC
- 5. ? 7 Dealing with Failures and Repairs ? 7.1 ? 7.2 Categorizing Faults ? 7.2.1 Fault Severity ? 7.2.2 Causes of Service-Level Faults ? 7.3 Machine-Level Failures ? 7.3.1 What Causes Machine Crashes? ? 7.3.2 Predicting Faults ? 7.4 Repairs ? 7.5 Tolerating Faults, Not Hiding Them
- 6. ? 1 2 ? ? ?
- 7. ? RAID => ? => ? RAID GFS ? ? OS
- 8. ? ? ? => ?
- 9. Google ? DRAM ? 2000 ? => ? 100% ? ECC DRAM
- 10. ECC DRAM ? ECC Error Correction Code 1 2 via http://www.nec.co.jp/products/express/tech/memory/index.shtml
- 11. ? 7 Dealing with Failures and Repairs ? 7.1 Implications of Software-Based Fault Tolerance ? 7.2 ? 7.2.1 Fault Severity ? 7.2.2 Causes of Service-Level Faults ? 7.3 Machine-Level Failures ? 7.3.1 What Causes Machine Crashes? ? 7.3.2 Predicting Faults ? 7.4 Repairs ? 7.5 Tolerating Faults, Not Hiding Them
- 12. ~ ? ? ? WSC ?
- 13. ? 7 Dealing with Failures and Repairs ? 7.1 Implications of Software-Based Fault Tolerance ? 7.2 Categorizing Faults ? 7.2.1 ? 7.2.2 Causes of Service-Level Faults ? 7.3 Machine-Level Failures ? 7.3.1 What Causes Machine Crashes? ? 7.3.2 Predicting Faults ? 7.4 Repairs ? 7.5 Tolerating Faults, Not Hiding Them
- 14. ? ? Corrupted: ? Unreachable: ? Degraded: ? Masked: ?
- 15. ? => 99.0% => 99.0% ? ?
- 16. ? 7 Dealing with Failures and Repairs ? 7.1 Implications of Software-Based Fault Tolerance ? 7.2 Categorizing Faults ? 7.2.1 Fault Severity ? 7.2.2 ? 7.3 Machine-Level Failures ? 7.3.1 What Causes Machine Crashes? ? 7.3.2 Predicting Faults ? 7.4 Repairs ? 7.5 Tolerating Faults, Not Hiding Them
- 17. ? Oppenheimer 500 ? ? H/W 10-25% ? Gray Tandem ? H/W -> 10% -> 60% -> 20% ?
- 18. Google ? Oppenheimer ?
- 19. ? 7 Dealing with Failures and Repairs ? 7.1 Implications of Software-Based Fault Tolerance ? 7.2 Categorizing Faults ? 7.2.1 Fault Severity ? 7.2.2 Causes of Service-Level Faults ? 7.3 ? 7.3.1 What Causes Machine Crashes? ? 7.3.2 Predicting Faults ? 7.4 Repairs ? 7.5 Tolerating Faults, Not Hiding Them
- 20. H/W ? Google ? 95% 1 reboot ? 1% reboot
- 21. ? reboot 55% 6 ? 25% 6 30 1% 1 ? 3 ? reboot
- 22. ? 7 Dealing with Failures and Repairs ? 7.1 Implications of Software-Based Fault Tolerance ? 7.2 Categorizing Faults ? 7.2.1 Fault Severity ? 7.2.2 Causes of Service-Level Faults ? 7.3 Machine-Level Failures ? 7.3.1 ? 7.3.2 Predicting Faults ? 7.4 Repairs ? 7.5 Tolerating Faults, Not Hiding Them
- 23. ? DRAM ? ECC ? ? ?
- 24. ? 7 Dealing with Failures and Repairs ? 7.1 Implications of Software-Based Fault Tolerance ? 7.2 Categorizing Faults ? 7.2.1 Fault Severity ? 7.2.2 Causes of Service-Level Faults ? 7.3 Machine-Level Failures ? 7.3.1 What Causes Machine Crashes? ? 7.3.2 ? 7.4 Repairs ? 7.5 Tolerating Faults, Not Hiding Them
- 25. ? 10 100% ? WSC ? Pinheiro Google ? WSC
- 26. ? 7 Dealing with Failures and Repairs ? 7.1 Implications of Software-Based Fault Tolerance ? 7.2 Categorizing Faults ? 7.2.1 Fault Severity ? 7.2.2 Causes of Service-Level Faults ? 7.3 Machine-Level Failures ? 7.3.1 What Causes Machine Crashes? ? 7.3.2 Predicting Faults ? 7.4 ? 7.5 Tolerating Faults, Not Hiding Them
- 27. WSC ? WSC ? ? => ? =>
- 28. Google ? System Health ? ?
- 29. ? 7 Dealing with Failures and Repairs ? 7.1 Implications of Software-Based Fault Tolerance ? 7.2 Categorizing Faults ? 7.2.1 Fault Severity ? 7.2.2 Causes of Service-Level Faults ? 7.3 Machine-Level Failures ? 7.3.1 What Causes Machine Crashes? ? 7.3.2 Predicting Faults ? 7.4 Repairs ? 7.5
- 30. ? => ?
- 31. IT ? 24 5-15% / ? Google ? WSC ? 40000 5% 200
- 32. Thank you!