�ݺ�ߣ

Department of Computer Sience, Software Technology Group
Valencia, 31.05.2013
Towards Systematic Model-based Testing of
Self-adaptive Software
Georg Püschel, Sebastian Götz, Claas Wilke, Uwe Aßmann

Towards Systematic MBT of SAS Folie Nr. 2 von XYZ
Outline
1) SAS requires a new MBT foundation
2) Failure Analysis -- Method and Application
3) Result: SAS Testing Requirements
4) Conclusion and Future Work
02.06.2013

Motivation I: Why do we need…?
 Energy Auto Tuning models and architecture:
„CoolSoftware“
 Multi-NFP Auto-Tuning: ”Highly Adaptive Energy-Efficient
Computing” (HAEC)
 Role-based SAS architecture: „Smart Application Grids“
(SMAGS)
 Cyber-physical Systems including SAS behaviour: „Visual
and Interactive Cyber-Physical Systems Control and
Integration“ (VICCI)
 All projects research about requirements and design.
 To complete the SE process, testing is also required.
02.06.2013

Motivation II: Why should we generally….?
 SAS conform to a feedback loop principle (e.g., MAPE-K [KC03])
 Challenge:
“Creating validation and verification techniques to
test and evaluate control loops’ behaviour and
automatically detect unintended interactions.” [Cheng+08]
 Are SAS special to testing?
 As in design much more complex information has to be
specified.
 As SAS run control loops, testing methods can be
specifically tailored to it.
 Because of both, models should be to generate test cases.
(MBT – Model-based Testing)
02.06.2013

Related Work: What was done before?
 Gap (1): Besides challenge statements, analysis is rare.
 Gap (2): Only a minority considers the interaction between running system
processes and adaptation. (Transactions? Order of actions?)
 Gap (3): Generalization?
 Gap (4): Many approaches base on @runtime self-testing.
 But: For industry-ready and safe systems (e.g., with certification) we
require black box testing in advance (but before deployment)!
Some related approaches/methods:
 DiVA offers a comprehensive strategy, split in two phases [DiVA]:
 Early validation@design time: adaptation logic and context model are
executed in a simulation
 Operational validation: deals with context changes, uses Multi-
dimensional Coverage Arrays (MDCA)
 Testing strategy for context-adaptive systems: „Artificial Shake Table Testing“
coverage strategy to produce context test data [MB09]
 Further strategies generate test cases from context and control flows
[TSE+04][Wang+07]
 Cheng„s explicit test model of the total state space of an SAS. [Cheng+08]
02.06.2013

Long-term goals
02.06.2013
here
• failure analysis
• requirements
future
• instantiation
• generalized SAS testing

Failure Analysis prerequisites:
Requirements and Assumptions
 Following steps are performed in Failure Mode
and Effects Analysis (FMEA) [FMEA]:
1. Build Failure Domain Model (dimensions&properties)
2. Investigate scenarios
(3.) visualize fault propagation in a Fault Dependency
Tree/Graph
But: We require a common architectural
perspective of SAS!
 Assumption: We do only consider SAS fulfilling the
MAPE-K principle.
02.06.2013

Failure Analysis, Step (1): Failure Domain Model for SAS
02.06.2013
Deviation from exepected
behaviour at a system‘s interface.
Inconsistent part of total system
state (inner state+perceivable
external state).
An error‘s cause.
Propagation to
another component.

Failure Analysis, Step (1): Failure Domain Model for SAS
02.06.2013
dimensions/properties
Standard dimensions, but important
to keep track of.
With or without inclusion of
knowledge model?
Which components of the control
loop are affected.
Especially if the failure includes
manipulating the outer reality.
Failures can also be associated with
non-optimal fulfilled goals.
Event-driveness/Determintation of
adaptation initiation.

Failure Analysis, Step (2): Scenarios
02.06.2013
check
Monitor Plan
Scheduler
Configuration
planner
(Planner)
Adaptation
logic execution
(Analyzer)
Event
monitoring
& processing
(Monitor)
Analyse
Action queue
synch
System
actions
PRE
PLAN
SCHED
Sensors Effectors
SENS
EFFECT
POST
ADAPT
Execute
Executor
RECONF
TRIG
EVENT
„K“

Failure Analysis, Step (3): Fault Dependency Graph
 the FDG visualizes the casual chain of
failure propagation
 the cyclic propagation is estabilished
through the stateful knowledge memory
and potential interdependencies of sensing
manipulated outer/physical reality
(dashed arrow)
 PRE and TRIG are coupled in both
directections such that these scenarios
must be tested together
02.06.2013

 Task: State requirements which can be mapped to respective scenarios.
 Requirements as assurance tasks/goals.
Assure…
1) …correct sensor interpretation. (SENS)
2) …correct adaptation initiation. (TRIG/PRE/ADAPT)
3) …correct adaptation planning. (PLAN)
4) …consistent interaction between adaptation and
system behavior. (SCHED)
5) …consistent adaptation execution. (POST/RECONF)
6) …correct system behavior. (EVENT/EFFECT)
Extra:
7) Find contructive (to generate) and analytic (to measure) coverage criteria for
SAS.
Result: SAS Testing Requirements
02.06.2013

Conclusion and Future Work
 We applied FMEA to the MAPE-K concept.
 Failure Domain Model: properties of failures in
SAS
 Fault Dependency Graph shows some special
causal interdependencies in SAS
 6+1 derived requirements for SAS testing
 its a basic toolset for SAS testers
 Future Work:
 instantiate requirements for our (initially named)
projects
 … other to existing SAS frameworks
 therefore, develop appropriate testing models and
coverage criteria
02.06.2013

FIN– Q/A?
02.06.2013
Thank you for your attendence.

Funding
This research has received funding…
• …within the project #100084131 by the
European Social Fund (ESF) and the German
Federal State of Saxony,
• …by Deutsche Forschungsgemeinschaft
(DFG) within CRC 912 (HAEC)
• …as well as T-Systems Multimedia Solutions
GmbH.
02.06.2013

References
[KC03] J. O. Kephart and D. M. Chess, “The vision of autonomic computing,” Computer, vol. 36, no. 1, Jan.
2003, pp. 41–50.
[Cheng+08] B. H. C. Cheng, D. Lemos, H. Giese, P. Inverardi, and J. M. et al., “Software engineering for self-
adaptive systems: A research roadmap,” in Dagstuhl Seminar 08031 on Software Engineering for Self-
Adaptive Systems, 2008.
[FMEA] H. E. Roland and B. Moriarty, System Safety Egnineering and Managemnent, 2nd edn. John Wiley &
Sons, Chichester, 1990, ch. Failure Mode and Effect Analysis.
[STA07] H. Sozer, B. Tekinerdogan, and M. Aksit, Archtitecting dependable systems IV. Springer, 2007, ch.
Extending failure models and effects analysis approach for reliability analysis at the software architecture
design level.
[TSA08] B. Tekinerdogan, H. Sozer, and M. Aksit, “Software architecture reliability analysis using failure
scenarios,” Journal of Systems and Software, vol. 81 (4), 2008, pp. 558–575.
[MB09] F. Munoz and B. Baudry, “Artificial table testing dynamically adaptive systems,” 2009.
[DiVA] A. Maaß, D. Beucho, and A. Solberg, “Adaptation model and validation framework final version (DiVA
deliverable D4.3),” 2010.
[Tse+04] T. Tse, S. Yau, W. Chan, H. Lu, and T. Chen, “Testing context-sensitive middleware-based software
applications,” 28th Annual International Computer Software and Applications Conference, 2004, pp. 458–466.
[Wang+07] Z. Wang, S. Elbaum, and D. S. Rosenblum, “Automated generation of context-aware tests,” 29th
International Conference on Software Engineering (ICSE), 2007, pp. 406–415.
02.06.2013

�ݺ�ߣ

Adaptive13 test

More Related Content

Adaptive13 test