20220914-MBT-Experiences-SB1-final.pptx
Download as PPTX, PDF
0 likes19 views
Minh Nguyen presented experiences applying model-based testing (MBT) for automated test case generation and execution. MBT uses models of requirements, rules, and test filters to automatically generate test cases from an MBT tool and execute them against a system under test via a test automation tool. Key benefits are adjustable test coverage tailored to changes, reduced maintenance costs, and traceability between requirements and tests. Completeness of models and domain knowledge are critical. The roadmap is to expand MBT use with tools like ReadyAPI and integrate with DevOps pipelines.
More Related Content
Similar to 20220914-MBT-Experiences-SB1-final.pptx (20)
More from Minh Nguyen (11)
Recently uploaded (20)
20220914-MBT-Experiences-SB1-final.pptx
- 1. Experiences in applying model based testing for automated test case generation and execution Minh Nguyen Founder/CEO/Quality Engineer
- 2. About me Holds a Ph.D. degree in Computer Science in 1997 from Norwegian University of Technology 25 years work experience in software testing and QA. Founder / CEO / Quality Engineer at Questigage. Currently working as contractor at SpareBank 1 Master Data Management team 束Pursuing and seeking for new and innovative way of doing test and QA in smart and effective manner損 Test automation; continuous test execution; model- based test; ML supported test; etc
- 3. About the talk 01 Context and motivation 02 MBT implementation 03 Lesson learned
- 4. Test efficiency vs. effectiveness Testing exploring and examining the system to evaluate and prove quantitatively if the quality requirements are met. Challenges: Too much focus on gaining efficiency by test automation and demonstrating test coverage Less attention paid on fact-based identifying test areas which give highest possible test effectiveness
- 5. MDM at SpareBank 1 (SB1) SB1-MDM Public registers Updates Consolidation SB1 Fagsystemer SB1 Fagsystemer Enterprise systems MDM-usage: 7 millions Customer records 25 Consumers and 2 millions requests/day 12.000 daily updates from public registers at real-time Producers Exposing SB1 Systemer SB1 Systemer Systems MDM-API Consumers Producers MDM-solution: 12 micro-services exposed to Consumers 10 enterprise systems integrated 300 rules: data validation, transformation and merge consolidation Heavy batch transactions 束The most updated and best quality of enterprise data stored in one single place損
- 6. Challenges Extremely high requirement to data quality Technical complexity - many integration interfaces High frequency of changes to business rules and domain models Increasing number of Consumers with new requirements Increasing number of Enterprise systems joining the consolidated platform Automated regression test suites are constantly growing and changing Frequent hot-fix releases raise a need for rapid verification of the actual change
- 7. What we want Improve test effectiveness by smart deriving of test scope caused by particular changes and their intrinsic risk Reduce cost of developing and maintaining tests due to constant requirement changes Better control of traceability between business requirements and tests MBT Model based testing ???
- 8. Model based testing MBT Rules / Requirements Model Test- oracle Test cases Executable tests SUT Manual modeling Automated generation Automated generation Automated binding Automated execution Automated evaluation Test tool Expected benefits: Automated (smart) test case generation Traceability between requirements and tests Adjustable test coverage 束right tests for particular change損 Cost-efficient maintenance of tests by changing the models instead of tests
- 9. Example Input parameter: 32 input parameters. Average 3-4 possible input values per parameter which are needed to be verified. Business rules: 60 Field-rule: 1. If SSN is set then CustomerType = PER 2. LastName length shall not exceed 256 characters Cross-field rules: 3. Secondary Citizenship shall not be set if Primary Citizenship is empty 4. If SSN is D-No && PersonStatus = 3 then Sector = 9800 Behaviour-rules: Changes from system A will override changes from system B if the trust level of B is downgraded after X-days
- 10. Model based testing MBT Rules / Requirements Model Test- oracle Test cases Executable tests SUT Manual modeling Automated generation Automated generation Automated binding Automated execution Automated evaluation Test tool
- 11. Input parameter Test case filter Rule Equivalence Class Boundary PublicIdentificationNo: regularSSN d-No orgNo invalid_IdentNo LastName: length_20 length_255 length_256 length_257 Rule_2: if (Citizenship1=null && Citizenship2 != null) { Outcome = false Error_msg = 束Citizenship2 rejected損} Decision Table Logical Operator Rule_1: if (SSN=dNo && PersonStatus=3 && SectorCode = 9800) Outcome = true Exclude(IdentNo.d-no || SectorCode = 9800 || Citizenship1 = NOR ) Fixed_value(IdentNo = orgNo && MaritalStatus = Single && PersonStatus = Resident) Modeling
- 14. Model based testing MBT Rules / Requirements Model Test- oracle Test cases Executable tests SUT Manual modeling Automated generation Automated generation Automated binding Automated execution Automated evaluation Test tool
- 15. Test case generation Parameter = 20 No. possible values 3 No. of possible permutations = 320 (some combinations are not relevant) Still 束Test case explosion損 Objective of test case generation: To derive a set of test cases when being executed will give a necessary and sufficient coverage for a particular purpose. Genererated test cases: o (o1, a1, b3, c2, ...) o (o2, a2, b1, c2, ...) o ... Combinatorial algorithm: o NWise 2 | 3 | .. | n o Cartesian (full coverage) RISK Model element: o Parameter o Rule o TC-filter SCOPE
- 16. Model based testing MBT Rules / Requirements Model Test- oracle Test cases Executable tests SUT Manual modeling Automated generation Automated generation Automated binding Automated execution Automated evaluation Test tool
- 17. Test case execution SoapUI test EcFeed Test case generation Algorithms + Rules + TC-filters Expected_output, input_1, input_2, ..., input_n Expected_output, input_1, input_2, ..., input_n Expected_output, input_1, input_2, ..., input_n Expected_output, input_1, input_2, ..., input_n ... Test parameter binding Evaluate test result Export test result CSV DB tables Slack Execute service API
- 18. Evaluation of benefits Expected benefits: Automated (smart) test case generation Traceability between requirements and tests Adjustable test coverage 束right tests for particular change損 Cost-efficient maintenance of tests by changing the models instead of tests
- 19. Lesson learned Need to have a suitable use case for MBT implementation. Completeness/correctness of the model is crucial hard to find errors in models. Might lead to defect leakage or false alarm. MBT technique Require good domain knowledge and requirement -> Involve PO? Define sufficient number of rules and filters to facilitate effective test Automated tests must be parameterized for easy binding Require modelling skill and anticipate overhead Implementation Smooth integration between MBT-tool ecFeed and SoapUI Improvement areas: make MBT-tool more intuitive and 束test-friendly損 Tool integration
- 20. Our roadmap Manual test Automated test Ad-hoc Rapid and time efficient Adaptive test Smart and learning Continuous test Model based test Timely and event-based Targeted and effective ReadyAPI + integration support with Azure Devops
- 21. Thank you Email: minh.nguyen@questigate.no Web: www.questigate.no LinkedIn: https://www.linkedin.com/in/minhng67/