狠狠撸

Boston
USA
Kazuki Nishiuraa
Yuta Maezawaa
Hironori Washizakib
Shinichi Honidena,c
06/27/2013
Mutation Analysis for
JavaScript Web
Application Testing
SEKE
2013 aThe University of Tokyo, Japan
bWaseda University, Japan
cNational Institute for Informatics, Japan

? Background
? Mutation Analysis: Test adequacy criteria
? JavaScript Web Applications (JSWAs)
? AjaxMutator: Mutation Analysis tool for JSWAs
(http://knishiura-lab.github.io/AjaxMutator/)
? Feature analysis on JSWA
? Definition of Mutation Operators
? Bug survey/case study using real-world apps.
2
Table of Contents
Improvement of test cases; exposed 10% more faults

is a method to evaluate adequacy of test cases
3
Mutation Analysis
Ticket t = issueTicket(true, true);
assertEquals(600, t.price);
Do we have
enough test ?
Ticket issueTicket(
bool isStudent,
bool extraBanquet){
Ticket t;
if (isStudent) {
t = {price:500, numBanquet:0};
} else {
}
if (extraBanquet) {
t.numBanquet = t.numBanquet+1;
t.price = t.price + 100;
}
return t;
}
Ticket t = issueTicket(false, false);
Program under test Test case #1
Test case #2

4
Mutation Analysis
Do we have
enough test ?
Ticket issueTicket(
bool isStudent,
bool extraBanquet){
Ticket t;
if (isStudent) {
} else {
}
if (extraBanquet) {
t.numBanquet = t.numBanquet+1;
}
return t;
}
Program under test Test case #1
Test case #2
Test #1 Test #2
100% coverage

Mutation Analysis
Ex) Program to calculate ?2 ? ?2
int diffOfSquares(int x, int y){
int sum = x + y;
int diff = x – y;
return sum * diff;
}
Ticket issueTicket(
bool isStudent,
bool extraBanquet){
Ticket t;
if (isStudent) {
} else {
}
if (extraBanquet) {
t.numBanquet = t.numBanquet-1;
}
return t;
}
Mutant #1
by injecting artificial fault
Mutation operator
+ -
? Mutant guides improvement of test cases
? Mutation operator determine mutant
Test case #1
Test case #2
Unkilled by test #1,2

Mutation Analysis
Ex) Program to calculate ?2 ? ?2
int diffOfSquares(int x, int y){
int sum = x + y;
int diff = x – y;
return sum * diff;
}
Ticket issueTicket(
bool isStudent,
bool extraBanquet){
Ticket t;
if (isStudent) {
} else {
}
if (extraBanquet) {
t.numBanquet = t.numBanquet-1;
}
return t;
}
Mutant #1
by injecting artificial fault
Mutation operator
+ -
assertEquals(1, t.numBanquet);
Test case #3
? Mutant guides improvement of test cases
? Mutation operator determine mutant
Test case #1
Test case #2
Unkilled by test #1,2killed by test #3

7
How JavaScript Web Applications Work
Web Page
Async. comm.Event invocation
e.g., click, mouseover
Update page
Send DataDisplay
Server
Client
HTML/CSS
XML/XSLT
DOM
XHR
JavaScript
Can we test JavaScript program adequately?
Target of our research

8
Motivating Example: E-Commerce app.
open(INDEX_URL);
wait(untilElementAppeared(“#detailBtn1”));
click(“#detailBtn1”);
assertVisible(“#detailWindow”);
wait(untilElementAppeared(“.discount”));
assertEnabled(“#purchaseBtn”);
Test script
No exception doesn’t mean no fault Insufficiency of coverage

9
open(INDEX_URL);
Test script
On detail loaded

10
open(INDEX_URL);
Test script
On detail loaded
Click

11
open(INDEX_URL);
Test script
On detail loaded
Click

12
open(INDEX_URL);
Test script
On detail loaded
Click After a while

13
open(INDEX_URL);
Test script
Statement coverage：100%
On detail loaded
Click After a while

14
open(INDEX_URL);
Test script
Statement coverage：100%
On detail loaded
Click After a while
Fault: wrong property name
// should be data.price
elm.setText(data.value);

15
Proposal: Mutation Analysis on JSWAs
Web Applications
JavaScript
(1)Implement Test
！

16
Web Applications
JavaScript
(1)Implement Test
Mutant
！
AjaxMutator
(2) Generate mutants

17
Web Applications
JavaScript
(1)Implement Test
Mutant
Unkilled Mutant
！
AjaxMutator

18
Web Applications
JavaScript
(1)Implement Test
(4)Add Test
Mutant
Unkilled Mutant
！
AjaxMutator

(3) Execute
Test
19
Web Applications
JavaScript
(1)Implement Test
(4)Add Test
Mutant
Unkilled Mutant
！
AjaxMutator
1. Analyze feature of JSWA
2. Define mutation operators
Approach(2) Generate mutants

①
②
③
Web Page Send DataDisplay
Server
Client
How JSWA works
XHRXML/XSLTDOMHTML, etc..
JavaScript
20
Three Main Characteristics of JSWAs [1]
[1] Justin Gehtland, et al. “Pragmatic Ajax: A Web 2.0 Primer”, O’REILLY, 2006
① ②
③
Async. comm.
Update page
Event invocation
e.g., click, mouseover
JavaScript Web Application
Asynchronous communicationsEvent DOM Manipulation
① ② ③

Asynchronous communications
Feature Diagram for Event Registration
Event Registration
Interval
Callback Function
Timer Event Registration
Repeat?
Target
Event Type
User Event Registration
Callback Function
Window DOM Element Body Return Value
Click Etc…Focus
21
Mandatory Optional
Alternative Or

Event Registration
Timer Event Regi
22
detailButton1.click(showDetail1);
Original Program
loginButton.click(showDetail1);
Mutant #1
Target Event Type
Callback Function
Window DOM Element Body Return ValueClick Etc…Focus
Defining Mutation Operators Based on Features
Click

Event Registration
Timer Event Regi
23
Original Program
Mutant #1
Mutant #2
Target Event Type
Callback Function
Click
detailButton1.mouseover(showDetail1);

Event Registration
Timer Event Regi
24
Original Program
Mutant #1
Mutant #2
Mutant #3
Target Event Type
Callback Function
Click

Mutant #1
Event Registration
Timer Event Regi
25
Click
Original Program
Mutant #2
Mutant #3
Target Event Type
Callback Function
Test in
motivating example
Require additional test cases to find faults

26
Timer event
Mutant #2
setTimeout(showDetail, 1000);
setTimeout(applyDiscount1, 1000);
Original Program Mutant #1
setTimeout(applyDiscount1, 2000);
Event Registration
Interval
Callback Function
Repeat?
nt Registration Timer Event Registration

27
Request
Response
Asynchronous Communication
Callback Function
URL Method Body Return Value
Status Body
Parameter
Event DOM Manipulati
Asynchronous Communication
Mutant #2
$.get(“buy.php”, showButtons);
$.get(“items.php”, showMsg);
$.get(“items.php”, showButtons);
Original Program
Focus on client-side features
Mutant #1

ommunication
Web Application
28
Target Selection Manipulation
Relative
Position
Tag Name
Creation Deletion
Insertion
Attribute
Value Attribute Assignment
DOM Manipulation
Mutant #1
Mutant #2
$(“.detailBtn”).parent();
btn.width = 300;
$ (“.detailBtn”);
Original Program #1
Mutant #1
btn.height = 100;btn.width = 100;
Original Program #2
DOM Manipulation

1. Can JavaScript features really cause faults?
2. Can developers improve test cases
with AjaxMutator to find unexposed faults?
3. Can AjaxMutator be used in a reasonable time?
29
Evaluation: Research Questions

30
RQ1: Actual faults in JSWAs related to features
[2] Frolin S. Ocariza Jr. et al. “Autoflox: An automatic fault localizer for client-side javascript”
Ticket # Related Feature Brief Explanation
1895 User Event Don’t handle on-focus event properly
2184 Don’t handle on-resize event properly
8812 DOM
Manipulation
Generating span element where p should be generated
9740 Selecting some unwanted elements
Search from bug repository of
JavaScript features can really cause actual faults
“JavaScript”, “JS”, or “console” [2]Keyword

Quizzy (5561LOC)
JavaScript(311LOC)
(2)Add Tests to kill
more mutants
Mutant
Unkilled
Mutant
AjaxMutator
(1)Implement Test
* Undergraduate students who has been developed JSWAs for two years in industry
? Normal
? Invalid
RQ2, 3: Case study setup
scenario

Faulty
program
×20
Faulty
program
×20
32
Quizzy (5561LOC)
JavaScript(311LOC)
more mutants
Mutant
Unkilled
Mutant
AjaxMutator
Faulty
program
×20
(1)Implement Test
Participant*
(3) Seed
fault
? Normal
? Invalid
scenario

Faulty
program
×20
Faulty
program
×20
33
Quizzy (5561LOC)
JavaScript(311LOC)
more mutants
Mutant
Unkilled
Mutant
AjaxMutator
Faulty
program
×20
(1)Implement Test
Participant*
(3) Seed
fault
? Normal
? Invalid
(4) Check if test
can find faults
scenario

×20×20
34
Quizzy (5561LOC)
JavaScript(311LOC)
(2)Add Test to kill
more mutants
Mutant
Unkilled
Mutant
×20
(1)Implement
Test
Participant*
(3) Seed
faults
? Normal use case
? Another invalid scenario
(4) Check if test
can find faults
RQ2: Improvement of test cases
We could improve test cases to find unexposed faults
Test suites # of test cases Coverage Killed mutants Fault finding capability
Initial 2 95% 45.9% 89.5%
Improved +5 100% 67.0% 100%

35
Limitations
5 5 4 4
20
8
4
2 2
1
16
1
1
1
2
17
4
6
2
2
2
0
10
20
30
40
50
60
Semantically Equivalent (5.5%)
Unkilled (27.5%)
Killed by improved test (21.1%)
Killed by initial test (45.9%)
Equivalent mutants
Browser robustness
* target program deal with only click-type event
*
elm.width=100 -1
Browser ignore invalid
value and infer correct one
further challenge
Details of generated mutants
val = 100
elm.width=100 val

36
RQ3: Time for Mutation Analysis
Web Applications
JavaScript
109 Mutants
59 Unkilled
AjaxMutator
20 min.
109 mutants × 2 tests
20 min.
1 hour
Implement Test
3 hour
Add Test
Reasonable time; automation/optimization can be applied
59 mutants × 5 tests

37
Related Work
? For HTML and JSP [Praphamontripong’10]
? For JSWA but limited features [Mirshokraie’13]
? Faults inWeb apps [Marchetto’07][Guo’09]
? JavaScript run-time errors [Ocariza’11]
Survey on real-world faults
Mutation analysis on Web Apps
AjaxMuator: comprehensive mutation operators for features of JSWA
Definition of mutation operators by a top-down way
No bug reports can be used for defining mutation operators

? Proposal of mutation operators by feature analysis of JSWAs
? Publicly available implementation: AjaxMutator
? http://knishiura-lab.github.io/AjaxMutator/
? Survey and case study using real-world apps:
? AjaxMutator helps aa
? More case study and survey
? Automated test case generation
38
Conclusion & Future Work
Conclusion
Future Work
Improvement of test cases to expose faults

40
RQ3: Time for Mutation Analysis
Initial Test Additional Test
Preparing test cases
(manual)
1 hour 3 hours
Mutation Analysis
(Automated)
20 minutes
(2 Test cases * 109 mutants)
20 minutes
(5 Test cases×59 mutants)
To reduce manual cost
→ We can utilize automated-test case generation technique
To reduce mutation analysis cost (computational time)
→ We can parallelize execution
? AjaxMutator can be used in reasonable time
? Automation/Optimization can be applied

41
Defined Mutation Operators
JS Features Operation target Mutation Example
User Event Event target cancelBtn.click(requestBuy);
Event type buyBtn.mouseover(requestBuy);
Callback function buyBtn.click(closeModal);
Timer Event Timer interval setTimeout(anotherCallback, 1000);
Callback function setTimeout(callback, 2000);
Asynchronous
Communications
Request target $.get(“server.php”, callback);
Request on-success
callback function
$.get(“server.php”, callback);
DOM Manipulation Selection of element $(“.btn”).id = “btn1”;
Assigned property of DOM $(“#btn”).className = “btn1”;
Assigned value to DOM $(“#btn”).id = “btn0”;

Category Mutation Target Ours Mirshokraie’s[3]
User event Target, type, callback ? NA
Timer event Interval, callback ? NA
Asynchronous
communications
Request destination URL ? ?
Asynchronous or Synchronouns NA(Out of scope) ?
Callback ?(on-success
callback)
?(condition on which
callback is invoked)
DOM Element selection ? ?
Attribute assignment ? ?
Element insertion ?(Implemented
after submission)
?
Other Common mistakes for
inexperienced programmers
NA ?
42
Comparison between Mirshokraie et al.’13[3]
[3] A. Mirshokraie, Shabnam, et al.
“Efficient Javascript Mutation Testing,” Proc. of ICST, 2013

狠狠撸

Mutation Analysis for JavaScript Web Applicaiton Testing SEKE2013

More Related Content

Mutation Analysis for JavaScript Web Applicaiton Testing SEKE2013