Talk-Foutse-SrangeLoop.pdf
0 likes44 views
NeuraLint and TheDeepChecker are tools for finding bugs in deep learning programs. NeuraLint works by statically analyzing the code structure and checking for potential issues based on a taxonomy of common deep learning faults. It can find bugs quickly and less expensively than testing the full program. TheDeepChecker monitors programs as they train and checks for issues based on defined verification rules related to parameters, activations, and optimization. It can find 30% more bugs than AWS SageMaker and captures defects during training rather than just on the final model. Both tools aim to improve quality and catch bugs earlier in the development process.
![Parameters-related Issues Untrained Parameters
Given a layer and iterations
0
=
1
,
0
=
1
1
=
2
,
1
=
2
1
=
,
1
=
Issue
Poor Weight Initialization
Parameters Values Divergence
Given a layer and an iteration
+1
+1
[0, 1]
Verification
Routine
Parameters Unstable Learning
TheDeepChecker verification rules](https://image.slidesharecdn.com/talk-foutse-srangeloop-220924031458-547e8f4a/85/Talk-Foutse-SrangeLoop-pdf-12-320.jpg)
![13
Activation-related Issues Activations out of Range
Given a layer
基基 [, ]
Issue
Neuron Saturation
Given a layer
基基
Verification
Routine
Dead ReLU
Specification of verification rules](https://image.slidesharecdn.com/talk-foutse-srangeloop-220924031458-547e8f4a/85/Talk-Foutse-SrangeLoop-pdf-13-320.jpg)
![Testing concepts [3] - Software Testing Techniques (CIS640)](https://cdn.slidesharecdn.com/ss_thumbnails/testing-concepts-170926174731-thumbnail.jpg?width=560&fit=bounds)
![[DSC Europe 22] Testing Machine Learning Systems: What it is and why you shou...](https://cdn.slidesharecdn.com/ss_thumbnails/ninamrzeljtestingmlsystems-221129232305-3a4f10b4-thumbnail.jpg?width=560&fit=bounds)
More Related Content
Similar to Talk-Foutse-SrangeLoop.pdf (20)
More from Foutse Khomh (11)
Recently uploaded (20)
Talk-Foutse-SrangeLoop.pdf
- 1. Finding Bugs in Deep Learning Programs SoftWare Analytics and Technologies Lab S.W.A.T Foutse Khomh, PhD, Ing. foutse.khomh@polymtl.ca @SWATLab Canada CIFAR AI Chair on Trustworthy Machine Learning Systems Trustworthy Engineering of AI Software 1
- 2. Quality Assurance of ML-enabled systems System evolution & continuous delivery 2 Reliable system Data Health Model Health Code Health Time Distribution shift Data imbalance Coding errors Noisy labels Underspecification Bias Vulnerabilities
- 3. Deep Learning is being rapidly adopted in industry 3
- 4. Preliminary preparation Data Collection Data Preprocessing Model Implementation Model Training Model Evaluation Model Tuning Data postprocessing Model Prediction DL development phases produce a lot of code! Han et al., What do Programmers Discuss about Deep Learning Frameworks
- 5. DL programs can be faulty! 5
- 6. 6 Multi-dimensional space of DL bugs Model Issues not enough learning capacity non-optimal regularization correct incorrect feature extraction incorrect gradient computation Implementation Issues correct
- 7. 7 TOSEM21 NeuraLint : A linter for DL programs Capture defects early, so saves rework cost. Less expensive, because it doesnt require execution. Find defects in seconds. NeuraLint is fast and effective! It achieves an accuracy of 91.7 % . It correctly reported 18 additional bugs that were not found by developers. The average execution time of NeuraLint for the studied TensorFlow and Keras based programs are 2.892 and 3.197 seconds respectively. Try it out!
- 8. NeuraLint has two pillars 8 A meta-model of DL programs Taxonomy of common DL faults Gunel Jahangirova, Nargiz Humbatova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. 2019. Taxonomy of Real Faults in Deep Learning Systems. arXiv preprint arXiv:1910.11015
- 9. NeuraLint: Execution Flow 9 Graph transformation Rules Potential issues Program Original program Model Extraction Model Run List of detected Issues
- 10. TheDeepChecker outperforms AWS SMD DL coding bugs and misconfigurations are detected with (precision, recall), respectively, equal to (90%, 96.4%) and (77%, 83.3%). Finds 30% more defects than AWS SageMaker. 10 TOSEM22 TheDeepChecker : Dynamic testing of DL programs Capture defects during the training process. Less expensive than testing the resulting model. Some overhead on the training process. Try it out!
- 11. 11 TheDeepChecker verification rules Parameters-related Issues Untrained Parameters Poor Weight Initialization Parameters Values Divergence Parameters Unstable Learning Activation-related Issues Activations out of Range Neuron Saturation Dead ReLU Optimization-related Issues Unable to fit a small sample Zero Loss Diverging Loss Slow or Non decreasing Loss Loss Fluctuations Unstable Gradient: Exploding Unstable Gradient: Vanishing
- 12. Parameters-related Issues Untrained Parameters Given a layer and iterations 0 = 1 , 0 = 1 1 = 2 , 1 = 2 1 = , 1 = Issue Poor Weight Initialization Parameters Values Divergence Given a layer and an iteration +1 +1 [0, 1] Verification Routine Parameters Unstable Learning TheDeepChecker verification rules
- 13. 13 Activation-related Issues Activations out of Range Given a layer 基基 [, ] Issue Neuron Saturation Given a layer 基基 Verification Routine Dead ReLU Specification of verification rules
- 14. 14 Optimization-related Issues Unable to fit a small sample The DNN could not properly minimize the loss. Issue Zero Loss Diverging Loss Slow or Non decreasing Loss The DNN (with regularization off) should overfit a tiny sample of data. Given N iterations = 0 Verification Routine Loss Fluctuations Unstable Gradient: Exploding Unstable Gradient: Vanishing TheDeepChecker verification rules
- 15. 15 Program Pre-processing Post-processing Verification Routines Potential issues Program Original program Monitoring Monitored Program Run Sanity Check of Program TheDeepChecker: Execution Flow
- 16. TheDeepChecker outperforms AWS SMD DL coding bugs and misconfigurations are detected with (precision, recall), respectively, equal to (90%, 96.4%) and (77%, 83.3%). Finds 30% more defects than AWS SageMaker. 16 TOSEM22 TheDeepChecker : Dynamic testing of DL programs Capture defects during the training process. Less expensive than testing the resulting model. Some overhead on the training process. Try it out!
- 17. 17 Foutse Khomh, PhD, Ing. foutse.khomh@polymtl.ca @SWATLab Canada CIFAR AI Chair Emmanuel Thepie Fapi
- 18. 18 Try it out!