ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo
A Quantitative Comparison of
Coverage-Based Greybox Fuzzers
Natsuki Tsuzuki, Nagoya University, Japan
Norihiro Yoshida, Nagoya University, Japan
Koji Toda, Fukuoka Institute of Technology, Japan
Kenji Fujiwara, National Institute of Technology, Toyota College , Japan
Ryota Yamamoto, Nagoya University, Japan
Hiroaki Takada, Nagoya University, Japan
1
Many Coverage-Based Greybox Fuzzers
? AFL (originally developed by Zalewski in ¡®13)
? AFLFast (B?hme et al., CCS ¡®16, TSE ¡®19)
? AFLGo (B?hme et al., CCS¡¯ 17)
? FairFuzz (Lemieux & Sen, ASE ¡®18)
https://www.pikrepo.com/
2
Questions
- The newest fuzzer is always better than the others?
- How each fuzzer works better than the use of default test suite?
3
Fuzzers are evaluated by different criteria
Fuzzers Fuzzing targets Criteria
AFLFast - Binutils 2.26
(c++filt, nm, objdump, readelf, size, strings)
- Coreutils 8.25
- # unique crashes
- ground truth
- line coverage
AFLGo - Binutils (detailed information is not found)
- Diffutils
- libPNG
- basic block coverage
- ground truth
FairFuzz - Binutils 2.28 (c++filt, nm,objdump, readelf )
- tcpdump
- xmllint
- mutool draw
- djpeg
- readpng
- basic block translations covered
- # occurrences of specific
sequences
It is difficult to compare the experimental results in these papers.
4
Research Overview
We prepared a unified collection of fuzzing targets
and then compared the existing fuzzers.
Evaluation measures:
- The number of executed paths
- Branch coverage
5
Research Questions
RQ1 Is a newer AFL-based fuzzer able to execute significantly
a larger number of paths?
RQ2 Does an AFL-based fuzzer improve branch coverage?
RQ3 Does a newer AFL-based fuzzer always achieve higher
coverage?
6
Fuzzers and Fuzzing Targets
- AFL 1.94b
- AFL 2.40b
- AFL 2.49b
- AFL 2.51b
- AFL 2.52b
- AFLFast
- AFLGo
- FairFuzz
- Binutils 2.26
(c++filt, nm, objdump, readelf)
- Binutils 2.28
(c++filt, nm, objdump, readelf)
- Binutils 2.32
(c++filt, nm, objdump, readelf)
Each execution of a fuzzer is terminated after 6 hours. 7
Significance test (# paths)
We used Steel-Dwass test for judging the significance.
AFL
1.94b
AFL
2.40b
AFL
2.49b
AFL
2.51b
AFL
2.52b
AFLFast AFLGo FairFuzz
AFL 1.94b - ?
AFL 2.40b - ?
AFL 2.49b - ?
AFL 2.51b - ?
AFL 2.52b - ?
AFLFast -
AFLGo - ?
FairFuzz ? ? ? ? ? ? -
Answer to RQ1: In most cases, the newest fuzzer FairFuzz executes
significantly larger number of paths. 8
Branch coverage
in the non-use and use of fuzzers
Answer to RQ2: The fuzzers can improve branch coverage. 9
Significance test (branch coverage)
We used Steel-Dwass test for judging the significance.
AFL
1.94b
AFL
2.40b
AFL
2.49b
AFL
2.51b
AFL
2.52b
AFLFast AFLGo FairFuzz
AFL 1.94b -
AFL 2.40b -
AFL 2.49b -
AFL 2.51b -
AFL 2.52b -
AFLFast -
AFLGo -
FairFuzz -
Answer to RQ3: The newer fuzzer does not always achieve
higher branch coverage. 10
Discussion
The results are different between the number of paths and
branch coverage.
Newer fuzzers are unoptimized for quality assurance
process based on branch coverage.
The use of fuzzers can improve branch coverage.
11
Thank you for listening!
E-mail: yoshida AT ertl.jp
12

More Related Content

Similar to A Quantitative Comparison of Coverage-Based Greybox Fuzzers (14)

2.6 latifs 17 dramix pisos sobre pilotes
2.6 latifs 17 dramix pisos sobre pilotes 2.6 latifs 17 dramix pisos sobre pilotes
2.6 latifs 17 dramix pisos sobre pilotes
Latifs Chile
?
Jain-DeCO-FCCM-2016
Jain-DeCO-FCCM-2016Jain-DeCO-FCCM-2016
Jain-DeCO-FCCM-2016
Abhishek Jain
?
OIF CEI 56-G-FOE-April2015
OIF CEI 56-G-FOE-April2015OIF CEI 56-G-FOE-April2015
OIF CEI 56-G-FOE-April2015
Deborah Porchivina
?
JGI_HMMER.pptx
JGI_HMMER.pptxJGI_HMMER.pptx
JGI_HMMER.pptx
William Arndt
?
MAKER2
MAKER2MAKER2
MAKER2
Shaojun Xie
?
Porting and Optimization of Numerical Libraries for ARM SVE
Porting and Optimization of Numerical Libraries for ARM SVEPorting and Optimization of Numerical Libraries for ARM SVE
Porting and Optimization of Numerical Libraries for ARM SVE
Linaro
?
Low Overhead System Tracing with eBPF
Low Overhead System Tracing with eBPFLow Overhead System Tracing with eBPF
Low Overhead System Tracing with eBPF
Akshay Kapoor
?
Hands-on Experience with IPv6 Routing and Services
Hands-on Experience with IPv6 Routing and ServicesHands-on Experience with IPv6 Routing and Services
Hands-on Experience with IPv6 Routing and Services
Cisco Canada
?
×ÔÂÉÒƶ¯¥í¥Ü¥Ã¥ÈÏò¤±¥Ï©`¥É?¥½¥Õ¥ÈЭµ÷¤Î¤¿¤á¤Î¥³¥ó¥Ý©`¥Í¥ó¥ÈÉè¼ÆÖ§Ô®¥Ä©`¥ë
×ÔÂÉÒƶ¯¥í¥Ü¥Ã¥ÈÏò¤±¥Ï©`¥É?¥½¥Õ¥ÈЭµ÷¤Î¤¿¤á¤Î¥³¥ó¥Ý©`¥Í¥ó¥ÈÉè¼ÆÖ§Ô®¥Ä©`¥ë×ÔÂÉÒƶ¯¥í¥Ü¥Ã¥ÈÏò¤±¥Ï©`¥É?¥½¥Õ¥ÈЭµ÷¤Î¤¿¤á¤Î¥³¥ó¥Ý©`¥Í¥ó¥ÈÉè¼ÆÖ§Ô®¥Ä©`¥ë
×ÔÂÉÒƶ¯¥í¥Ü¥Ã¥ÈÏò¤±¥Ï©`¥É?¥½¥Õ¥ÈЭµ÷¤Î¤¿¤á¤Î¥³¥ó¥Ý©`¥Í¥ó¥ÈÉè¼ÆÖ§Ô®¥Ä©`¥ë
Kazushi Yamashina
?
Usage, Performance and Future Of PL1 at NRB Benoit Ebner
Usage, Performance and Future Of PL1 at NRB Benoit EbnerUsage, Performance and Future Of PL1 at NRB Benoit Ebner
Usage, Performance and Future Of PL1 at NRB Benoit Ebner
NRB
?
Fuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation TestingFuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation Testing
Lionel Briand
?
New features in Ruby 2.5
New features in Ruby 2.5New features in Ruby 2.5
New features in Ruby 2.5
Ireneusz Skrobi?
?
Odoo Performance Limits
Odoo Performance LimitsOdoo Performance Limits
Odoo Performance Limits
Odoo
?
Dynamische Routingprotokolle Aufzucht und Pflege - OSPF
Dynamische Routingprotokolle Aufzucht und Pflege - OSPFDynamische Routingprotokolle Aufzucht und Pflege - OSPF
Dynamische Routingprotokolle Aufzucht und Pflege - OSPF
Maximilan Wilhelm
?
2.6 latifs 17 dramix pisos sobre pilotes
2.6 latifs 17 dramix pisos sobre pilotes 2.6 latifs 17 dramix pisos sobre pilotes
2.6 latifs 17 dramix pisos sobre pilotes
Latifs Chile
?
Porting and Optimization of Numerical Libraries for ARM SVE
Porting and Optimization of Numerical Libraries for ARM SVEPorting and Optimization of Numerical Libraries for ARM SVE
Porting and Optimization of Numerical Libraries for ARM SVE
Linaro
?
Low Overhead System Tracing with eBPF
Low Overhead System Tracing with eBPFLow Overhead System Tracing with eBPF
Low Overhead System Tracing with eBPF
Akshay Kapoor
?
Hands-on Experience with IPv6 Routing and Services
Hands-on Experience with IPv6 Routing and ServicesHands-on Experience with IPv6 Routing and Services
Hands-on Experience with IPv6 Routing and Services
Cisco Canada
?
×ÔÂÉÒƶ¯¥í¥Ü¥Ã¥ÈÏò¤±¥Ï©`¥É?¥½¥Õ¥ÈЭµ÷¤Î¤¿¤á¤Î¥³¥ó¥Ý©`¥Í¥ó¥ÈÉè¼ÆÖ§Ô®¥Ä©`¥ë
×ÔÂÉÒƶ¯¥í¥Ü¥Ã¥ÈÏò¤±¥Ï©`¥É?¥½¥Õ¥ÈЭµ÷¤Î¤¿¤á¤Î¥³¥ó¥Ý©`¥Í¥ó¥ÈÉè¼ÆÖ§Ô®¥Ä©`¥ë×ÔÂÉÒƶ¯¥í¥Ü¥Ã¥ÈÏò¤±¥Ï©`¥É?¥½¥Õ¥ÈЭµ÷¤Î¤¿¤á¤Î¥³¥ó¥Ý©`¥Í¥ó¥ÈÉè¼ÆÖ§Ô®¥Ä©`¥ë
×ÔÂÉÒƶ¯¥í¥Ü¥Ã¥ÈÏò¤±¥Ï©`¥É?¥½¥Õ¥ÈЭµ÷¤Î¤¿¤á¤Î¥³¥ó¥Ý©`¥Í¥ó¥ÈÉè¼ÆÖ§Ô®¥Ä©`¥ë
Kazushi Yamashina
?
Usage, Performance and Future Of PL1 at NRB Benoit Ebner
Usage, Performance and Future Of PL1 at NRB Benoit EbnerUsage, Performance and Future Of PL1 at NRB Benoit Ebner
Usage, Performance and Future Of PL1 at NRB Benoit Ebner
NRB
?
Fuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation TestingFuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation Testing
Lionel Briand
?
Odoo Performance Limits
Odoo Performance LimitsOdoo Performance Limits
Odoo Performance Limits
Odoo
?
Dynamische Routingprotokolle Aufzucht und Pflege - OSPF
Dynamische Routingprotokolle Aufzucht und Pflege - OSPFDynamische Routingprotokolle Aufzucht und Pflege - OSPF
Dynamische Routingprotokolle Aufzucht und Pflege - OSPF
Maximilan Wilhelm
?

More from Norihiro Yoshida (12)

¥Õ¥¡¥¸¥ó¥°¥Ä©`¥ë´¡¹ó³¢¤ÎÀûÓäòÖ§Ô®¤¹¤ë¥Ä©`¥ë¹ó³Ü³ú³ú4µþ¤Ë¤è¤ë¥Õ¥¡¥¸¥ó¥°½ÌÓý¤ÎÊÔ¤ß
¥Õ¥¡¥¸¥ó¥°¥Ä©`¥ë´¡¹ó³¢¤ÎÀûÓäòÖ§Ô®¤¹¤ë¥Ä©`¥ë¹ó³Ü³ú³ú4µþ¤Ë¤è¤ë¥Õ¥¡¥¸¥ó¥°½ÌÓý¤ÎÊԤߥե¡¥¸¥ó¥°¥Ä©`¥ë´¡¹ó³¢¤ÎÀûÓäòÖ§Ô®¤¹¤ë¥Ä©`¥ë¹ó³Ü³ú³ú4µþ¤Ë¤è¤ë¥Õ¥¡¥¸¥ó¥°½ÌÓý¤ÎÊÔ¤ß
¥Õ¥¡¥¸¥ó¥°¥Ä©`¥ë´¡¹ó³¢¤ÎÀûÓäòÖ§Ô®¤¹¤ë¥Ä©`¥ë¹ó³Ü³ú³ú4µþ¤Ë¤è¤ë¥Õ¥¡¥¸¥ó¥°½ÌÓý¤ÎÊÔ¤ß
Norihiro Yoshida
?
Extracting a Micro State Transition Table Using KLEE
Extracting a Micro State Transition Table Using KLEEExtracting a Micro State Transition Table Using KLEE
Extracting a Micro State Transition Table Using KLEE
Norihiro Yoshida
?
¥½¥Õ¥È¥¦¥§¥¢é_°k¤Ë¤ª¤±¤ë®bѧ…f„“¥Õ¥©©`¥é¥à ¥ª©`¥×¥Ë¥ó¥°ÙYÁÏ¥½¥Õ¥È¥¦¥§¥¢é_°k¤Ë¤ª¤±¤ë®bѧ…f„“¥Õ¥©©`¥é¥à ¥ª©`¥×¥Ë¥ó¥°ÙYÁÏ
¥½¥Õ¥È¥¦¥§¥¢é_°k¤Ë¤ª¤±¤ë®bѧ…f„“¥Õ¥©©`¥é¥à ¥ª©`¥×¥Ë¥ó¥°ÙYÁÏ
Norihiro Yoshida
?
¥³©`¥É¥¯¥í©`¥ó —ʳö?‰ä¸ü¹ÜÀí¥Ä©`¥ëȺ¤Îé_°k
¥³©`¥É¥¯¥í©`¥ó —ʳö?‰ä¸ü¹ÜÀí¥Ä©`¥ëȺ¤Îé_°k¥³©`¥É¥¯¥í©`¥ó —ʳö?‰ä¸ü¹ÜÀí¥Ä©`¥ëȺ¤Îé_°k
¥³©`¥É¥¯¥í©`¥ó —ʳö?‰ä¸ü¹ÜÀí¥Ä©`¥ëȺ¤Îé_°k
Norihiro Yoshida
?
Proactive Clone Recommendation System for Extract Method Refactoring
 Proactive Clone Recommendation System for Extract Method Refactoring Proactive Clone Recommendation System for Extract Method Refactoring
Proactive Clone Recommendation System for Extract Method Refactoring
Norihiro Yoshida
?
Code Search Based on Deep Neural Network and Code Mutation
Code Search Based on Deep Neural Network and Code MutationCode Search Based on Deep Neural Network and Code Mutation
Code Search Based on Deep Neural Network and Code Mutation
Norihiro Yoshida
?
»úеѧϰ¥·¥¹¥Æ¥à¤Ë¤ª¤±¤ë¼¼ÊõµÄ¸ºÕ®¤Ë¤Ä¤¤¤Æ
»úеѧϰ¥·¥¹¥Æ¥à¤Ë¤ª¤±¤ë¼¼ÊõµÄ¸ºÕ®¤Ë¤Ä¤¤¤Æ»úеѧϰ¥·¥¹¥Æ¥à¤Ë¤ª¤±¤ë¼¼ÊõµÄ¸ºÕ®¤Ë¤Ä¤¤¤Æ
»úеѧϰ¥·¥¹¥Æ¥à¤Ë¤ª¤±¤ë¼¼ÊõµÄ¸ºÕ®¤Ë¤Ä¤¤¤Æ
Norihiro Yoshida
?
When, why and for whom do practitioners detect technical debts?: An experienc...
When, why and for whom do practitioners detect technical debts?: An experienc...When, why and for whom do practitioners detect technical debts?: An experienc...
When, why and for whom do practitioners detect technical debts?: An experienc...
Norihiro Yoshida
?
Revisiting the Relationship Between Code Smells and Refactoring
Revisiting the Relationship Between Code Smells and RefactoringRevisiting the Relationship Between Code Smells and Refactoring
Revisiting the Relationship Between Code Smells and Refactoring
Norihiro Yoshida
?
Mining the Modern Code Review Repositories: A Dataset of People, Process and ...
Mining the Modern Code Review Repositories: A Dataset of People, Process and ...Mining the Modern Code Review Repositories: A Dataset of People, Process and ...
Mining the Modern Code Review Repositories: A Dataset of People, Process and ...
Norihiro Yoshida
?
IWESEP 2013
IWESEP 2013IWESEP 2013
IWESEP 2013
Norihiro Yoshida
?
MSR2013
MSR2013MSR2013
MSR2013
Norihiro Yoshida
?
¥Õ¥¡¥¸¥ó¥°¥Ä©`¥ë´¡¹ó³¢¤ÎÀûÓäòÖ§Ô®¤¹¤ë¥Ä©`¥ë¹ó³Ü³ú³ú4µþ¤Ë¤è¤ë¥Õ¥¡¥¸¥ó¥°½ÌÓý¤ÎÊÔ¤ß
¥Õ¥¡¥¸¥ó¥°¥Ä©`¥ë´¡¹ó³¢¤ÎÀûÓäòÖ§Ô®¤¹¤ë¥Ä©`¥ë¹ó³Ü³ú³ú4µþ¤Ë¤è¤ë¥Õ¥¡¥¸¥ó¥°½ÌÓý¤ÎÊԤߥե¡¥¸¥ó¥°¥Ä©`¥ë´¡¹ó³¢¤ÎÀûÓäòÖ§Ô®¤¹¤ë¥Ä©`¥ë¹ó³Ü³ú³ú4µþ¤Ë¤è¤ë¥Õ¥¡¥¸¥ó¥°½ÌÓý¤ÎÊÔ¤ß
¥Õ¥¡¥¸¥ó¥°¥Ä©`¥ë´¡¹ó³¢¤ÎÀûÓäòÖ§Ô®¤¹¤ë¥Ä©`¥ë¹ó³Ü³ú³ú4µþ¤Ë¤è¤ë¥Õ¥¡¥¸¥ó¥°½ÌÓý¤ÎÊÔ¤ß
Norihiro Yoshida
?
Extracting a Micro State Transition Table Using KLEE
Extracting a Micro State Transition Table Using KLEEExtracting a Micro State Transition Table Using KLEE
Extracting a Micro State Transition Table Using KLEE
Norihiro Yoshida
?
¥½¥Õ¥È¥¦¥§¥¢é_°k¤Ë¤ª¤±¤ë®bѧ…f„“¥Õ¥©©`¥é¥à ¥ª©`¥×¥Ë¥ó¥°ÙYÁÏ¥½¥Õ¥È¥¦¥§¥¢é_°k¤Ë¤ª¤±¤ë®bѧ…f„“¥Õ¥©©`¥é¥à ¥ª©`¥×¥Ë¥ó¥°ÙYÁÏ
¥½¥Õ¥È¥¦¥§¥¢é_°k¤Ë¤ª¤±¤ë®bѧ…f„“¥Õ¥©©`¥é¥à ¥ª©`¥×¥Ë¥ó¥°ÙYÁÏ
Norihiro Yoshida
?
¥³©`¥É¥¯¥í©`¥ó —ʳö?‰ä¸ü¹ÜÀí¥Ä©`¥ëȺ¤Îé_°k
¥³©`¥É¥¯¥í©`¥ó —ʳö?‰ä¸ü¹ÜÀí¥Ä©`¥ëȺ¤Îé_°k¥³©`¥É¥¯¥í©`¥ó —ʳö?‰ä¸ü¹ÜÀí¥Ä©`¥ëȺ¤Îé_°k
¥³©`¥É¥¯¥í©`¥ó —ʳö?‰ä¸ü¹ÜÀí¥Ä©`¥ëȺ¤Îé_°k
Norihiro Yoshida
?
Proactive Clone Recommendation System for Extract Method Refactoring
 Proactive Clone Recommendation System for Extract Method Refactoring Proactive Clone Recommendation System for Extract Method Refactoring
Proactive Clone Recommendation System for Extract Method Refactoring
Norihiro Yoshida
?
Code Search Based on Deep Neural Network and Code Mutation
Code Search Based on Deep Neural Network and Code MutationCode Search Based on Deep Neural Network and Code Mutation
Code Search Based on Deep Neural Network and Code Mutation
Norihiro Yoshida
?
»úеѧϰ¥·¥¹¥Æ¥à¤Ë¤ª¤±¤ë¼¼ÊõµÄ¸ºÕ®¤Ë¤Ä¤¤¤Æ
»úеѧϰ¥·¥¹¥Æ¥à¤Ë¤ª¤±¤ë¼¼ÊõµÄ¸ºÕ®¤Ë¤Ä¤¤¤Æ»úеѧϰ¥·¥¹¥Æ¥à¤Ë¤ª¤±¤ë¼¼ÊõµÄ¸ºÕ®¤Ë¤Ä¤¤¤Æ
»úеѧϰ¥·¥¹¥Æ¥à¤Ë¤ª¤±¤ë¼¼ÊõµÄ¸ºÕ®¤Ë¤Ä¤¤¤Æ
Norihiro Yoshida
?
When, why and for whom do practitioners detect technical debts?: An experienc...
When, why and for whom do practitioners detect technical debts?: An experienc...When, why and for whom do practitioners detect technical debts?: An experienc...
When, why and for whom do practitioners detect technical debts?: An experienc...
Norihiro Yoshida
?
Revisiting the Relationship Between Code Smells and Refactoring
Revisiting the Relationship Between Code Smells and RefactoringRevisiting the Relationship Between Code Smells and Refactoring
Revisiting the Relationship Between Code Smells and Refactoring
Norihiro Yoshida
?
Mining the Modern Code Review Repositories: A Dataset of People, Process and ...
Mining the Modern Code Review Repositories: A Dataset of People, Process and ...Mining the Modern Code Review Repositories: A Dataset of People, Process and ...
Mining the Modern Code Review Repositories: A Dataset of People, Process and ...
Norihiro Yoshida
?

Recently uploaded (20)

Water Industry Process Automation & Control Monthly - March 2025.pdf
Water Industry Process Automation & Control Monthly - March 2025.pdfWater Industry Process Automation & Control Monthly - March 2025.pdf
Water Industry Process Automation & Control Monthly - March 2025.pdf
Water Industry Process Automation & Control
?
Cyber Security_ Protecting the Digital World.pptx
Cyber Security_ Protecting the Digital World.pptxCyber Security_ Protecting the Digital World.pptx
Cyber Security_ Protecting the Digital World.pptx
Harshith A S
?
04 MAINTENANCE OF CONCRETE PAVEMENTS.ppt
04  MAINTENANCE OF CONCRETE PAVEMENTS.ppt04  MAINTENANCE OF CONCRETE PAVEMENTS.ppt
04 MAINTENANCE OF CONCRETE PAVEMENTS.ppt
sreenath seenu
?
Frankfurt University of Applied Science urkunde
Frankfurt University of Applied Science urkundeFrankfurt University of Applied Science urkunde
Frankfurt University of Applied Science urkunde
Lisa Emerson
?
only history of java.pptx real bihind the name java
only history of java.pptx real bihind the name javaonly history of java.pptx real bihind the name java
only history of java.pptx real bihind the name java
mushtaqsaliq9
?
Industrial Construction shed PEB MFG.pdf
Industrial Construction shed PEB MFG.pdfIndustrial Construction shed PEB MFG.pdf
Industrial Construction shed PEB MFG.pdf
PLINTH & ROOFS
?
Lecture -3 Cold water supply system.pptx
Lecture -3 Cold water supply system.pptxLecture -3 Cold water supply system.pptx
Lecture -3 Cold water supply system.pptx
rabiaatif2
?
TM-ASP-101-RF_Air Press manual crimping machine.pdf
TM-ASP-101-RF_Air Press manual crimping machine.pdfTM-ASP-101-RF_Air Press manual crimping machine.pdf
TM-ASP-101-RF_Air Press manual crimping machine.pdf
ChungLe60
?
Syntax Directed Definitions Synthesized Attributes and Inherited Attributes
Syntax Directed Definitions  Synthesized Attributes  and  Inherited AttributesSyntax Directed Definitions  Synthesized Attributes  and  Inherited Attributes
Syntax Directed Definitions Synthesized Attributes and Inherited Attributes
GunjalSanjay
?
US Patented ReGenX Generator, ReGen-X Quatum Motor EV Regenerative Accelerati...
US Patented ReGenX Generator, ReGen-X Quatum Motor EV Regenerative Accelerati...US Patented ReGenX Generator, ReGen-X Quatum Motor EV Regenerative Accelerati...
US Patented ReGenX Generator, ReGen-X Quatum Motor EV Regenerative Accelerati...
Thane Heins NOBEL PRIZE WINNING ENERGY RESEARCHER
?
Integration of Additive Manufacturing (AM) with IoT : A Smart Manufacturing A...
Integration of Additive Manufacturing (AM) with IoT : A Smart Manufacturing A...Integration of Additive Manufacturing (AM) with IoT : A Smart Manufacturing A...
Integration of Additive Manufacturing (AM) with IoT : A Smart Manufacturing A...
ASHISHDESAI85
?
Mathematics_behind_machine_learning_INT255.pptx
Mathematics_behind_machine_learning_INT255.pptxMathematics_behind_machine_learning_INT255.pptx
Mathematics_behind_machine_learning_INT255.pptx
ppkmurthy2006
?
Env and Water Supply Engg._Dr. Hasan.pdf
Env and Water Supply Engg._Dr. Hasan.pdfEnv and Water Supply Engg._Dr. Hasan.pdf
Env and Water Supply Engg._Dr. Hasan.pdf
MahmudHasan747870
?
Lectureof nano 1588236675-biosensors (1).ppt
Lectureof nano 1588236675-biosensors (1).pptLectureof nano 1588236675-biosensors (1).ppt
Lectureof nano 1588236675-biosensors (1).ppt
SherifElGohary7
?
Piping-and-pipeline-calculations-manual.pdf
Piping-and-pipeline-calculations-manual.pdfPiping-and-pipeline-calculations-manual.pdf
Piping-and-pipeline-calculations-manual.pdf
OMI0721
?
How to Make an RFID Door Lock System using Arduino
How to Make an RFID Door Lock System using ArduinoHow to Make an RFID Door Lock System using Arduino
How to Make an RFID Door Lock System using Arduino
CircuitDigest
?
How to Build a Maze Solving Robot Using Arduino
How to Build a Maze Solving Robot Using ArduinoHow to Build a Maze Solving Robot Using Arduino
How to Build a Maze Solving Robot Using Arduino
CircuitDigest
?
Sachpazis: Foundation Analysis and Design: Single Piles
Sachpazis: Foundation Analysis and Design: Single PilesSachpazis: Foundation Analysis and Design: Single Piles
Sachpazis: Foundation Analysis and Design: Single Piles
Dr.Costas Sachpazis
?
GM Meeting 070225 TO 130225 for 2024.pptx
GM Meeting 070225 TO 130225 for 2024.pptxGM Meeting 070225 TO 130225 for 2024.pptx
GM Meeting 070225 TO 130225 for 2024.pptx
crdslalcomumbai
?
US Patented ReGenX Generator, ReGen-X Quatum Motor EV Regenerative Accelerati...
US Patented ReGenX Generator, ReGen-X Quatum Motor EV Regenerative Accelerati...US Patented ReGenX Generator, ReGen-X Quatum Motor EV Regenerative Accelerati...
US Patented ReGenX Generator, ReGen-X Quatum Motor EV Regenerative Accelerati...
Thane Heins NOBEL PRIZE WINNING ENERGY RESEARCHER
?
Cyber Security_ Protecting the Digital World.pptx
Cyber Security_ Protecting the Digital World.pptxCyber Security_ Protecting the Digital World.pptx
Cyber Security_ Protecting the Digital World.pptx
Harshith A S
?
04 MAINTENANCE OF CONCRETE PAVEMENTS.ppt
04  MAINTENANCE OF CONCRETE PAVEMENTS.ppt04  MAINTENANCE OF CONCRETE PAVEMENTS.ppt
04 MAINTENANCE OF CONCRETE PAVEMENTS.ppt
sreenath seenu
?
Frankfurt University of Applied Science urkunde
Frankfurt University of Applied Science urkundeFrankfurt University of Applied Science urkunde
Frankfurt University of Applied Science urkunde
Lisa Emerson
?
only history of java.pptx real bihind the name java
only history of java.pptx real bihind the name javaonly history of java.pptx real bihind the name java
only history of java.pptx real bihind the name java
mushtaqsaliq9
?
Industrial Construction shed PEB MFG.pdf
Industrial Construction shed PEB MFG.pdfIndustrial Construction shed PEB MFG.pdf
Industrial Construction shed PEB MFG.pdf
PLINTH & ROOFS
?
Lecture -3 Cold water supply system.pptx
Lecture -3 Cold water supply system.pptxLecture -3 Cold water supply system.pptx
Lecture -3 Cold water supply system.pptx
rabiaatif2
?
TM-ASP-101-RF_Air Press manual crimping machine.pdf
TM-ASP-101-RF_Air Press manual crimping machine.pdfTM-ASP-101-RF_Air Press manual crimping machine.pdf
TM-ASP-101-RF_Air Press manual crimping machine.pdf
ChungLe60
?
Syntax Directed Definitions Synthesized Attributes and Inherited Attributes
Syntax Directed Definitions  Synthesized Attributes  and  Inherited AttributesSyntax Directed Definitions  Synthesized Attributes  and  Inherited Attributes
Syntax Directed Definitions Synthesized Attributes and Inherited Attributes
GunjalSanjay
?
Integration of Additive Manufacturing (AM) with IoT : A Smart Manufacturing A...
Integration of Additive Manufacturing (AM) with IoT : A Smart Manufacturing A...Integration of Additive Manufacturing (AM) with IoT : A Smart Manufacturing A...
Integration of Additive Manufacturing (AM) with IoT : A Smart Manufacturing A...
ASHISHDESAI85
?
Mathematics_behind_machine_learning_INT255.pptx
Mathematics_behind_machine_learning_INT255.pptxMathematics_behind_machine_learning_INT255.pptx
Mathematics_behind_machine_learning_INT255.pptx
ppkmurthy2006
?
Env and Water Supply Engg._Dr. Hasan.pdf
Env and Water Supply Engg._Dr. Hasan.pdfEnv and Water Supply Engg._Dr. Hasan.pdf
Env and Water Supply Engg._Dr. Hasan.pdf
MahmudHasan747870
?
Lectureof nano 1588236675-biosensors (1).ppt
Lectureof nano 1588236675-biosensors (1).pptLectureof nano 1588236675-biosensors (1).ppt
Lectureof nano 1588236675-biosensors (1).ppt
SherifElGohary7
?
Piping-and-pipeline-calculations-manual.pdf
Piping-and-pipeline-calculations-manual.pdfPiping-and-pipeline-calculations-manual.pdf
Piping-and-pipeline-calculations-manual.pdf
OMI0721
?
How to Make an RFID Door Lock System using Arduino
How to Make an RFID Door Lock System using ArduinoHow to Make an RFID Door Lock System using Arduino
How to Make an RFID Door Lock System using Arduino
CircuitDigest
?
How to Build a Maze Solving Robot Using Arduino
How to Build a Maze Solving Robot Using ArduinoHow to Build a Maze Solving Robot Using Arduino
How to Build a Maze Solving Robot Using Arduino
CircuitDigest
?
Sachpazis: Foundation Analysis and Design: Single Piles
Sachpazis: Foundation Analysis and Design: Single PilesSachpazis: Foundation Analysis and Design: Single Piles
Sachpazis: Foundation Analysis and Design: Single Piles
Dr.Costas Sachpazis
?
GM Meeting 070225 TO 130225 for 2024.pptx
GM Meeting 070225 TO 130225 for 2024.pptxGM Meeting 070225 TO 130225 for 2024.pptx
GM Meeting 070225 TO 130225 for 2024.pptx
crdslalcomumbai
?

A Quantitative Comparison of Coverage-Based Greybox Fuzzers

  • 1. A Quantitative Comparison of Coverage-Based Greybox Fuzzers Natsuki Tsuzuki, Nagoya University, Japan Norihiro Yoshida, Nagoya University, Japan Koji Toda, Fukuoka Institute of Technology, Japan Kenji Fujiwara, National Institute of Technology, Toyota College , Japan Ryota Yamamoto, Nagoya University, Japan Hiroaki Takada, Nagoya University, Japan 1
  • 2. Many Coverage-Based Greybox Fuzzers ? AFL (originally developed by Zalewski in ¡®13) ? AFLFast (B?hme et al., CCS ¡®16, TSE ¡®19) ? AFLGo (B?hme et al., CCS¡¯ 17) ? FairFuzz (Lemieux & Sen, ASE ¡®18) https://www.pikrepo.com/ 2
  • 3. Questions - The newest fuzzer is always better than the others? - How each fuzzer works better than the use of default test suite? 3
  • 4. Fuzzers are evaluated by different criteria Fuzzers Fuzzing targets Criteria AFLFast - Binutils 2.26 (c++filt, nm, objdump, readelf, size, strings) - Coreutils 8.25 - # unique crashes - ground truth - line coverage AFLGo - Binutils (detailed information is not found) - Diffutils - libPNG - basic block coverage - ground truth FairFuzz - Binutils 2.28 (c++filt, nm,objdump, readelf ) - tcpdump - xmllint - mutool draw - djpeg - readpng - basic block translations covered - # occurrences of specific sequences It is difficult to compare the experimental results in these papers. 4
  • 5. Research Overview We prepared a unified collection of fuzzing targets and then compared the existing fuzzers. Evaluation measures: - The number of executed paths - Branch coverage 5
  • 6. Research Questions RQ1 Is a newer AFL-based fuzzer able to execute significantly a larger number of paths? RQ2 Does an AFL-based fuzzer improve branch coverage? RQ3 Does a newer AFL-based fuzzer always achieve higher coverage? 6
  • 7. Fuzzers and Fuzzing Targets - AFL 1.94b - AFL 2.40b - AFL 2.49b - AFL 2.51b - AFL 2.52b - AFLFast - AFLGo - FairFuzz - Binutils 2.26 (c++filt, nm, objdump, readelf) - Binutils 2.28 (c++filt, nm, objdump, readelf) - Binutils 2.32 (c++filt, nm, objdump, readelf) Each execution of a fuzzer is terminated after 6 hours. 7
  • 8. Significance test (# paths) We used Steel-Dwass test for judging the significance. AFL 1.94b AFL 2.40b AFL 2.49b AFL 2.51b AFL 2.52b AFLFast AFLGo FairFuzz AFL 1.94b - ? AFL 2.40b - ? AFL 2.49b - ? AFL 2.51b - ? AFL 2.52b - ? AFLFast - AFLGo - ? FairFuzz ? ? ? ? ? ? - Answer to RQ1: In most cases, the newest fuzzer FairFuzz executes significantly larger number of paths. 8
  • 9. Branch coverage in the non-use and use of fuzzers Answer to RQ2: The fuzzers can improve branch coverage. 9
  • 10. Significance test (branch coverage) We used Steel-Dwass test for judging the significance. AFL 1.94b AFL 2.40b AFL 2.49b AFL 2.51b AFL 2.52b AFLFast AFLGo FairFuzz AFL 1.94b - AFL 2.40b - AFL 2.49b - AFL 2.51b - AFL 2.52b - AFLFast - AFLGo - FairFuzz - Answer to RQ3: The newer fuzzer does not always achieve higher branch coverage. 10
  • 11. Discussion The results are different between the number of paths and branch coverage. Newer fuzzers are unoptimized for quality assurance process based on branch coverage. The use of fuzzers can improve branch coverage. 11
  • 12. Thank you for listening! E-mail: yoshida AT ertl.jp 12