際際滷

際際滷Share a Scribd company logo

Review: DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems

Download as PPTX, PDF
J
Jinhan Kim

DeepGauge proposes new testing criteria for deep learning systems at multiple granularities including the neuron, layer, and model levels. The criteria include k-multisection neuron coverage, strong neuron activation coverage, neuron boundary coverage, top-k neuron coverage, and top-k neuron patterns. An evaluation of these criteria on image classification models found they provided more fine-grained coverage than existing neuron coverage techniques. In particular, the new criteria were able to detect differences in model behavior not captured by prior approaches.

1 of 26
Downloaded 20 times
DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue , Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang will be published @ ASE 2018 Jinhan Kim 2018.8.3
Review: DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems
How to Test Deep Learning System Systematically?
Not Like Traditional Software
Neuron Coverage of DeepXplore
Neuron
Neuron Coverage # of neurons activated / # total neurons -11 Car Face hedge vedge ... Nose Eyes Wheel ... ... ... f: ReLU 3 1 2 1 2 3 f(1) 1 Neuron coverage: 4/7=57% when activation threshold=0.75
Fine-grained Neuron Coverage
Neuron-Level Coverage k-multisection Neuron Coverage Neuron Inputs 0.1 0.13 0.17 0.2 0.43 0.15 Outputs K-Section bucketing low high
Neuron-Level Coverage Neuron Boundary Coverage K-Section bucketing low high Lower Corner Upper Corner Neuron
Neuron-Level Coverage Strong Neuron Activation Coverage K-Section bucketing low high Lower Corner Upper Corner Neuron
Layer-Level Coverage Top-k Neuron Coverage 6 7 1 2 3 4 5 0.5 0.2 0.6 Pick Top-2 Neuron 3 and 5 are covered
Layer-Level Coverage Top-k Neuron Patterns 6 7 1 2 3 4 5 Pick Top-1 for each layer 1 4 7 Pattern
Study
Dataset and Model
Configuration
Result: k-multisection Neuron Coverage
Result: Strong Neuron Activation Coverage
Result: Neuron Boundary Coverage
Result: Top-k Neuron Coverage
Result: Top-k Neuron Patterns
Remarks Adversarial test-sets boost coverage but.. Note that increasing the test coverage does not necessarily imply that new defects could be detected in traditional software testing.
Comparison with DeepXplores NC
Result: DeepXplores Neuron Coverage (DNC)
Why?
Investigation on DNC DNC uses the same threshold as the activation evaluation for all the neurons. DNC normalizes the dynamic range of neuron outputs according to max and min output of neurons on the corresponding layer for each input image under analysis.

More Related Content

Review: DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems