Functional and non-functional testing with IoT-Testware
1 like205 views
The Internet of Things (IoT) is omnipresent. More and more hardware devices get connected and will collect and share huge amounts of data in the near future. This progress will lead to a digital and hyper-connected world. Though, in such growing networks of interconnected things, quality assurance (QA) will become a continuous challenge. Especially aspects like conformance, interoperability and security but also performance and robustness will require an increased attention from QA perspective.
![6
Connectivity
options
Throughput
Latency
Power efficiency
Packet size
Wide portfolio of competences required
Devices (sensors, HW, embedded SW)
Platforms (Cloud, platform domain knowledge)
Applications (SW, dashboard, business logic)
TECHNICAL SCOPE
IoT platforms
360+ worldwide
IoT protocols
Rich selection
IP-based
non-IP based
Cellular
netw.
Cellular: 4G, NB-IoT, Cat-M1,EC-GSM[5G];
non-cellular: Wifi, LoRa, Sigfox ,Zigbee, BLE, connectivity
MQTT MQTT-SN CoAPAMQP3OPC
UA
XMPPHTTP1 Web
Socket21|2|3
applicationIoT application logic
TCP UDP
SMS
DTLSDTLS
IPv4/IPv6(6loWPAN)
transport
TLS/SSL
IoT services layer services](https://image.slidesharecdn.com/20180116iot-ttestwareswqd-190119122536/85/Functional-and-non-functional-testing-with-IoT-Testware-6-320.jpg)
![14
TTCN-3 is the Testing and Test Control Notation
Internationally standardized testing language for formally defining test
scenarios.
Designed purely for testing
CHALLENGE TEST AUTOMATION
testcase Hello_Bob () {
p.send(How do you do?);
alt {
[]p.receive(Fine!);
{setverdict( pass )};
[else]
{setverdict( inconc )} //Bob asleep!
}
}](https://image.slidesharecdn.com/20180116iot-ttestwareswqd-190119122536/85/Functional-and-non-functional-testing-with-IoT-Testware-14-320.jpg)
Functional and non-functional testing with IoT-Testware
- 1. Axel Rennoch, Alexander Kaiser, Sascha Hackel Software Quality Days 2019 16. January 2019, Vienna, Austria FUNCTIONAL AND NON-FUNCTIONAL TESTING WITH IOT-TESTWARE
- 2. 2 IoT Testing Challenges and scope IoT test language: TTCN-3 Project IoT-T Eclipse IoT-Testware Standardization & Certification Summary and outlook AGENDA
- 3. 3 Mirai botnet, October 2016: botnet using insecure configured IoT-devices (~100.000) attack causes blackout and disruption (e.g. Amazon, Netflix, Twitter, Github) Wannacry, May 2017 ransomeware affecting the whole world (e.g. hospitals in the U.K.) KRACK: Key Reinstallation Attack, October 2017 Replay attack on Wi-Fi Protected Access protocol Spectre and Meltdown, January 2018 Spectre: vulnerability that allows observable side effects from mispredicted speculative executions Meltdown: hardware vulnerability that allows to read all memory MOTIVATION FOR QUALITY
- 5. 5 IOT ARCHITECTURE The Three Software Stacks Required for IoT Architectures, Eclipse IoT Working Group, September 2016 telemetry commands telemetry commands
- 6. 6 Connectivity options Throughput Latency Power efficiency Packet size Wide portfolio of competences required Devices (sensors, HW, embedded SW) Platforms (Cloud, platform domain knowledge) Applications (SW, dashboard, business logic) TECHNICAL SCOPE IoT platforms 360+ worldwide IoT protocols Rich selection IP-based non-IP based Cellular netw. Cellular: 4G, NB-IoT, Cat-M1,EC-GSM[5G]; non-cellular: Wifi, LoRa, Sigfox ,Zigbee, BLE, connectivity MQTT MQTT-SN CoAPAMQP3OPC UA XMPPHTTP1 Web Socket21|2|3 applicationIoT application logic TCP UDP SMS DTLSDTLS IPv4/IPv6(6loWPAN) transport TLS/SSL IoT services layer services
- 7. 7 IoT devices, Mikrocontroller (MCU), Gateways (Bosch XDK, IoT starterkits) IoT platforms RIOT, relayr, Thread, mbed service layer (oneM2M, FiWare) IoT protocols Constrained Application Protocol (CoAP) MQ Telemetry Transport (MQTT) IoT challenges: complexity, asynchronism, resource constraints, long operation phase STARTING: TEST OBJECTS LPWAN LoRa oneM2M
- 8. 8 After the acceptance and system tests there will be a long operation phase => new test phase operation LONG OPERATION LIFETIME
- 9. 9 INTEGRATION OF SEVERAL TESTING APPROACHES IoT Testing Software Testing System Testing Security Testing Test Automation Protocol Testing
- 10. 10 Less resources needed (time and money) Avoid human mistakes due to manually testing During test development and execution Speed-up of regression tests and product time-to-market TEST AUTOMATION
- 11. 11 MULTIPLE TEST CONFIGURATION (SAMPLES)
- 12. 12 Toolset (selection of available means) Protocol tester/monitor (Eclipse Titan, Wireshark) Test devices (RFID kit, Bluetooth test device) GUI tester (Selenium, SikuliX, Chrome headless) Web services tester (soapUI) Public Testsuites (in development) Application of a standardized notation Abstract and platform-independent TESTWARE
- 14. 14 TTCN-3 is the Testing and Test Control Notation Internationally standardized testing language for formally defining test scenarios. Designed purely for testing CHALLENGE TEST AUTOMATION testcase Hello_Bob () { p.send(How do you do?); alt { []p.receive(Fine!); {setverdict( pass )}; [else] {setverdict( inconc )} //Bob asleep! } }
- 15. 15 One test technology for different tests Distributed, platform-independent testing Integrated graphical test development, documentation and analysis Adaptable, open test environment Areas of Testing Conformance and functional testing Interoperability and integration testing Real-time, performance, load and stress testing Security testing Regression testing Used for system and product qualification and certification DESIGN PRINCIPLES OF TTCN-3
- 16. Eclipse IoT-Testware THE IOT-T PROJECT
- 17. 17 息 Fraunhofer FOKUS IOT QUALITY 1. Conformance 2. Robustness / Security 3. Performance Protocol conformance Test Suites
- 18. 18 Take available software and tools and adding public testuites as a result of insights from IoT testing: IOT-TESTWARE https://projects.eclipse.org/projects/technology.iottestware
- 19. 19 Supplement to running and active Eclipse projects Paho, OM2M, Titan New project at Eclipse Foundation: https://projects.eclipse.org/projects/technology.iottestware TTCN-3 test suites for CoAP, MQTT, OPC-UA, LoRa? Assured licenses for users Currently in cooperation with relayr GmbH, Ericsson, LAAS/CNRS, itemis AG, Spirent Communications, Easy Global Market, Iskratel/Sintesio, THE ECLIPSE PROJECT
- 20. 20 SAMPLE TESTSUITE STRUCTURE: MQTT Broker as SUT All mandatory message data fields Regular and illegal data (Fixed/variable header, payload) Protocol features General Connect/disconnect (session) Subscribe/unsubscribe Immediate publish Last will and Testament (LWT) Heartbeats keepAlive values Topic Error handling Client as SUT
- 21. 21 TEST DEVELOPMENT SAMPLE: MQTT TESTZIEL-KATALOG Test configurations Test Suite Structure Test purpose (catalogue) Test implementation (TTCN-3)
- 22. 22 MQTT BROKER EVALUATION (CONFORMANCE, APRIL 2018) Broker PASS FAIL INCONCLUSIVE # % # % # % HiveMQ ? 39 86,67% 4 8,89% 2 4,44% Mosquitto 1.4.15 38 84,44% 5 11,11% 2 4,44% VerneMQ 1.3.1 37 82,22% 5 11,11% 3 6,67% EMQ 2.2 35 77,78% 8 17,78% 2 4,44% lannister ? 31 68,89% 12 26,67% 2 4,44% ActiveMQ 5.15.3 29 64,44% 14 31,11% 2 4,44% aedes v0.33.0 26 57,78% 17 37,78% 2 4,44% RSMB ? 26 57,78% 17 37,78% 2 4,44% RabbitMQ 3.7.4 19 42,22% 26 57,78% 0 0,00% Mosca 2.8.1 19 42,22% 24 53,33% 2 4,44% HBMQTT 0.9 17 37,78% 28 62,22% 0 0,00% Moquette 0.10 16 35,56% 29 64,44% 0 0,00%
- 23. 23 息 Fraunhofer FOKUS BROKER CONFORMANCE OVER TIME 40 38 39 37 36 35 19 19 3 5 3 5 7 8 24 24 2 2 3 3 2 2 2 2 0 5 10 15 20 25 30 35 40 45 50 Mosquitto 1.4.14 Mosquitto 1.4.15 VerneMQ 1.1.0 VerneMQ 1.3.1 EMQ 2.0 EMQ 2.2 Mosca 2.5.1Mosca 2.8.1 Pass Fail Inconclusive Testcases -2 -2 -1 0
- 24. 24 Vulnerability scanner: in particular for web applications, zero-day/fuzzing, consideration of data bases, traffic/network analyser, program code scanner Penetration tester, e.g. SQL injection Intrusion detection tools Load test/Scalability Further utilities: Model-based testing (UML testing profile) and risk modelling TESTWARE: SECURITY
- 25. 25 FUZZING APPROACH CoAP ATS CoAP ETS Fuzzed Data SUT ATS: Abstract Test Suite ETS: Executable TS
- 26. 26 Results for CoAP: - Initially, 4421 fuzzed test data for CoAP were generated - After sending the data to a (local) CoAP server, it crashed after date 1107 https://www.fokus.fraunhofer.de/de/sqc/security_testing https://github.com/fraunhoferfokus/Fuzzino/blob/master/doc/Fuzzino_XML_Description.pdf FUZZINO RESULTS AND RESOURCES
- 27. 27 PERFORMANCE TESTING: HIGH-LEVEL VIEW Eclipse Titan RIoT SUTLoad htop
- 28. 28 息 Fraunhofer FOKUS PERFORMANCE TESTING: WORKFLOW (1) (1) Create/choose State Machine (2) Configure scenario (3) Start Performance Test Suite a. Observe RIoT verdicts b. Observe IUT process with htop (CPU / Mem. Usage) (2) Titan / RIoT SUT htop (3) Load
- 29. 29 1. Control Logic 1. State Machine 2. Configuration Files 2. Applib 1. CoAP / MQTT / LwM2M / HTTP 2. Model-base Testing 3. CLL 1. Loadtest Framework 2. Functiontest Framework 3. Etc. 息 Fraunhofer FOKUS RIOT UNDER THE HOOD TTCN-3 & Titan User space
- 30. 30 COAP STATE MACHINE: EXAMPLE S0 S1 Start Traffic Case CoAP rsp: 2.05: verdict = pass CoAP rsp: not 2.05: verdict = fail Timeout : verdict = fail 息 Fraunhofer FOKUS S0: idle state S1: initiated, test case execution of => CoAP GET /some/resource/123
- 31. 31 COAP STATE MACHINE: EXAMPLE CONT. S0 S1 init 息 Fraunhofer FOKUS S2: CoAP GET /some/resource/123 S4: CoAP DELETE /some/resource/123 S3: CoAP PUT /some/resource/123 S2 set verdict !2.05 S3 !2.01 S4 2.02 2.05 2.01 rnd* rnd* Scenario: randomly GET / PUT or DELETE a single resource from multiple/ parallel CoAP Clients4.04
- 32. 32 Source: Eclipse Titan user guide 息 Fraunhofer FOKUS EXCURSUS: TTCN-3 PARALLEL EXECUTION
- 33. 33 TS COAP PERFORMANCE: EXAMPLE 息 Fraunhofer FOKUS SUT PTC PTC MTC CoAP: Req / Resp GET | PUT | DELETE
- 35. Standardization & Certification THE IOT-T PROJECT
- 36. 36 New Working Group (TST) will develop IoT test catalogues and specifications (not covered elsewhere) The types of testing include conformance, interoperability, security and performance testing The initial technical focus will be: IoT network layer (communication protocols, node connectivity, edge computing etc.), Basic security of IoT devices ETSI TC MTS
- 37. 37 MTS TST WORK PROGRAMME IEC 62443-4-2 CoAP MQTT LoRaWAN Vul. database https://portal.etsi.org/tb.aspx?tbid=860&SubTB=860
- 38. 38 BASE SECURITY CERTIFICATION SCOPE The Three Software Stacks Required for IoT Architectures, Eclipse IoT Working Group, September 2016 IoT-Testlab Scope (basic security level certification) telemetry commands telemetry commands
- 39. Putting everything together SUMMARY AND OUTLOOK
- 40. 40 IOT-TESTWARE BIG PICTURE ETS Reporting Logging TP: Test Purpose TSS: Test Suite Structure ATS: Abstract Test Suite ETS: Executable TS SUT: System Under Test TSS TP ATS
- 41. 41 Advanced testing technology: Open source IoT-Testware (code): External (open source) SW Standardized IoT test purposes: SUMMARY
- 42. 42 Adding more protocols to IoT-Testware AMQP, LWM2M, 6LoWPAN, LPWAN Increased security level for certification Cooperation/liaisons (in preparation) with ETSI TC Cyber/SmartM2M, oneM2M, OPC Foundation ... OUTLOOK
- 43. 43 CONTACTS Thank you for your attention! https://www.fokus.fraunhofer.de/en/sqc https://relayr.io/