際際滷

際際滷Share a Scribd company logo

AI Assisted Continuous Testing - Talk Track v2.pdf

Marc Hornbeek
Marc Hornbeek

In my talk today I will explain how to transform an organization that is currently using "Continuous Testing" as part of their DevSecOps solution to "AI-Assisted Continuous Testing". I will cover: What AI-Assisted Continuous Testing is. I will compare AI Assisted Testing to existing Continuous Testing. I will highlight some of key challenges that I found during different types of transformation projects fusng A I enabled tools. and I will explain some tools and transformation practices that I find useful to ensure success with AI-Assisted Continuous Test transformations for DevSecOps. Finally I will conclude with my view of benefits, and a radical prediction for the future of DevSecOps enabled by AI-Assisted Continuous Testing.

1 of 21
Download to read offline
AI Assisted Continuous Testing - Talk Track v2.pdf
AI-Assisted Continuous Testing Comparison to existing Continuous Testing Challenges for transformation Transformation practices and tools Benefits Future Prediction DALL-E generated image Navigating to AI-Assisted Continuous Testing
Continuous Testing Blueprint A strategy designed to reduce lead times and failure rates in continuous delivery pipelines and in continuous operations, through automated and iterative testing processes, aiming for decreased time from code commit to production deployment and reduced failures in production. CT is much more than simply automating tests. Continuous Testing Predicted Autonomous Continuous Improvement.
Test Management Test Tools Test Automation Leadership and Culture Continuous Testing Blueprint Test Strategies and Plans Test Results Analysis Test Environment Management Continuous Testing Tenets Continuous Testing Blueprint Artifacts Development (Dev) Artifacts Artifacts Artifacts Artifacts Integration (CI) Delivery (CD) Deployment (Deploy) Production (Prod) Planning (Plan) Test Data Management AI can assist every aspect of Continuous Testing.
AI/ML Game-Changer for Testing Software analysis Test plan and script generation Test data generation Defect & vulnerabilities detection / correction Test optimization Test maintenance Report generation &analysis Test scheduling and management Testability Time from Test Strategy to Test Results reduced to hours instead of days. AI-Assist
Effectiveness of AI depends on the wisdom and skills of domain experts Deep Domain Knowledge Contextual and subject matter expertise Create templates and select tools Innovation and strategic planning Knows how to leverage AI Training and guiding AI models Bridge gaps between technology and applications Ensure compliance and ethical standards Continual learning and model improvement Risk assessment and management Training and development Facilitate collaboration AI / ML Agents Capabilities Specialist AI Agent roles Ingest guidance and use cases Identify information from adjacent domains Accumulate and organize information Analyze dat and pattern recognition Predictive analytics Language, visual and audio processing Machine Learning for Cybersecurity Automated problem solving Customization and personalization Process automation Real-time monitoring and alerts Roles with AI-Assisted Continuous Testing
Deployment Tests Delivery Testing Integration Testing Dev Test Test Strategies Test Plans Test Cases Test Scripts Test Notation Definitions Test Environment Notation Test Results Test Metrics and Analytics Version- Managed Templates and Artifacts AI Results and Analysis AI- Assisted Test Workflow AI Test Environment Orchestration and Test Automation AI-Assisted Continuous Testing Blueprint
CodeThreat - AI SAST Keploy Data to Test Generator Examples of AI-Enabled Testing Tools *Testbytes, Feb,2024 * https://www.linkedin.com/pulse/10-ai-tools-software-testing-you-should-know-softwaretestingcompany-uaxvc/ Refraction Coding and Scanning Synthetic Users User testing UFT One Function test automation Aqua ALM test management Eggplant visual element testing Katalon Regression test generation ACCELQ zero code test automation Perfecto Scriptless no code generation Tricentis Tosca test generation Functionize test generation Sapient.ai unit test generation There is no universal AI- tool solution for DevSecOps.
Traditional Hours Days AI-Assist Seconds - Minutes 1. Requirements Analysis 2. Test Strategy 3. Test Plans 4. Test cases 5. Test scripts 6. Test environments 7. Test campaigns 8. Test Executions 9. Test Reports 10.Test Reviews 11.Test Remedies 12.Test Regress Macro-Flow Steps: Continuous Test Workflows Savings (Reduction) Lead Time: 50 - 90% Costs: 70 - 95% Micro-Flow Steps
Technology Transformations AI Integration Data Infrastructure Tool Modernization Procedural Transformations Test creation and execution Data-driven decisions Feedback loops Organizational Transformations Skill development Team structures Culture shifts Quality Assurance Transformations Enhanced coverage and accuracy Continuous improvement Compliance and security Transformation Use Cases
What Can Go Wrong? Consistency Readability Understandability Relevance Environment Controllability Observability Ethical and Bias Intuition Repeatability Coverage Compatibility Acceptance Data Quality Continuous Learning A strategic transformation approach is needed !
Strategic Goals Model Applications Maturity Model Goals Topics Priorities Strategic Assessment Discovery Survey Gap Assessment Current state VSM Baseline metrics Strategic Plan Future state VSM Themes, Epics, Stories, Tasks Tools selection Dependencies Timeline Governance Implementations Pipeline Migrations Integration Dashboard Governance Checkpoint Assessments 5,6,7 REALIZE, OPERATIONALIZE EXPAND 4 SOLUTION ALIGNMENT 3 ASSESSMENT 1,2 VISION/GOAL ALIGNMENT Discovery Surveys Value Stream tools Topics and Practices Metrics and Analytics Deliverable templates AI-Assisted Tools & Templates AI-assisted Analysis and Results Alignment Transformation Consultant Process Leadership and Collaboration AI-Assisted CT Strategic Transformation Blueprint
Example Strategic Transformation Roadmap Theme 1 Model AI Platform MVP -------------- Test environment, generation, and automation Templates & Tools Selected test types Metrics Workflows Training Retrospective Theme 2 Consolidation -------------- Test Results side Standard pipelines Migrate other application Center of Excellence Guilds and Training Re-assessment Theme 3 Coverage Expansion -------------- Types of tests Advance Training More applications Re-assessment Release auomation Theme 4 Optimization -------------- Deployment automation Accelerate test creation Accelerate test execution testing Accelerate test analysis Continuous Experimentation 1 2 3 4 Every transformation needs it own strategic roadmap!
Transformation Implementation Patterns that Work Dedicated Platform Team Embedded Team Outsourced Team Hybrid Dedicated/Outsource Team
Transformation Patterns to Avoid
Motivating Continuous Testing Transformations Educate and Align Demonstrate Value Develop Skills and Infrastructure Cultural Shift Toward Testing Automation Incorporate Feedback Mechanisms Adapt Project Management and Processes Encourage Innovation Highlight ROI and Performance Metrics Strengthen Vendor Partnerships Manage Change Effectively
Measuring Transformation Success DORA Metrics: Deployment Frequency (DF) Lead Time for Changes (LT) Change Failure Rate (CFR) Time to Restore Service (TRS) Additional Metrics: Test Coverage Defect Detection Efficiency Automated Test Pass Rate Flakiness Score Test Execution Time Cost per Defect Found Feedback Loop Duration ROI of Testing
Measuring Transformation ROI Costs: Development/Integration Costs Software Costs Hardware Costs Training Costs Maintenance Costs Benefits: Increased Efficiency Reduced Labor Costs Improved Quality Enhanced Test Coverage Intangible Benefits: Improved Team Morale and Productivity Enhanced Market Reputation
Benefits of Continuous Testing On-time delivery Improved predictability Reduced cost of quality Improved customer experience Improved productivity Better quality Faster releases Fewer security events Improved compliance scores up to 70% efficiency gains due to the reduction of wait-time and toil - Without AI ! What will it be with AI-Assist ?
The Future DevInOps No Shift, No Lead Time Enablers Observability AI-Automation Feature Flag Rollouts No-Shift Faster Lead time DevSecOps Continuous Delivery Shift-Left Production Development in Production At some point smart automation will make it so there will be nothing Left to shift!
AI-Assisted: DevSecOps and SRE Transformations Continuous Testing Consulting EngineeringDevOps.com mhornbeek@engineeringdevops.com

More Related Content

AI Assisted Continuous Testing - Talk Track v2.pdf

  • 2. AI-Assisted Continuous Testing Comparison to existing Continuous Testing Challenges for transformation Transformation practices and tools Benefits Future Prediction DALL-E generated image Navigating to AI-Assisted Continuous Testing
  • 3. Continuous Testing Blueprint A strategy designed to reduce lead times and failure rates in continuous delivery pipelines and in continuous operations, through automated and iterative testing processes, aiming for decreased time from code commit to production deployment and reduced failures in production. CT is much more than simply automating tests. Continuous Testing Predicted Autonomous Continuous Improvement.
  • 4. Test Management Test Tools Test Automation Leadership and Culture Continuous Testing Blueprint Test Strategies and Plans Test Results Analysis Test Environment Management Continuous Testing Tenets Continuous Testing Blueprint Artifacts Development (Dev) Artifacts Artifacts Artifacts Artifacts Integration (CI) Delivery (CD) Deployment (Deploy) Production (Prod) Planning (Plan) Test Data Management AI can assist every aspect of Continuous Testing.
  • 5. AI/ML Game-Changer for Testing Software analysis Test plan and script generation Test data generation Defect & vulnerabilities detection / correction Test optimization Test maintenance Report generation &analysis Test scheduling and management Testability Time from Test Strategy to Test Results reduced to hours instead of days. AI-Assist
  • 6. Effectiveness of AI depends on the wisdom and skills of domain experts Deep Domain Knowledge Contextual and subject matter expertise Create templates and select tools Innovation and strategic planning Knows how to leverage AI Training and guiding AI models Bridge gaps between technology and applications Ensure compliance and ethical standards Continual learning and model improvement Risk assessment and management Training and development Facilitate collaboration AI / ML Agents Capabilities Specialist AI Agent roles Ingest guidance and use cases Identify information from adjacent domains Accumulate and organize information Analyze dat and pattern recognition Predictive analytics Language, visual and audio processing Machine Learning for Cybersecurity Automated problem solving Customization and personalization Process automation Real-time monitoring and alerts Roles with AI-Assisted Continuous Testing
  • 7. Deployment Tests Delivery Testing Integration Testing Dev Test Test Strategies Test Plans Test Cases Test Scripts Test Notation Definitions Test Environment Notation Test Results Test Metrics and Analytics Version- Managed Templates and Artifacts AI Results and Analysis AI- Assisted Test Workflow AI Test Environment Orchestration and Test Automation AI-Assisted Continuous Testing Blueprint
  • 8. CodeThreat - AI SAST Keploy Data to Test Generator Examples of AI-Enabled Testing Tools *Testbytes, Feb,2024 * https://www.linkedin.com/pulse/10-ai-tools-software-testing-you-should-know-softwaretestingcompany-uaxvc/ Refraction Coding and Scanning Synthetic Users User testing UFT One Function test automation Aqua ALM test management Eggplant visual element testing Katalon Regression test generation ACCELQ zero code test automation Perfecto Scriptless no code generation Tricentis Tosca test generation Functionize test generation Sapient.ai unit test generation There is no universal AI- tool solution for DevSecOps.
  • 9. Traditional Hours Days AI-Assist Seconds - Minutes 1. Requirements Analysis 2. Test Strategy 3. Test Plans 4. Test cases 5. Test scripts 6. Test environments 7. Test campaigns 8. Test Executions 9. Test Reports 10.Test Reviews 11.Test Remedies 12.Test Regress Macro-Flow Steps: Continuous Test Workflows Savings (Reduction) Lead Time: 50 - 90% Costs: 70 - 95% Micro-Flow Steps
  • 10. Technology Transformations AI Integration Data Infrastructure Tool Modernization Procedural Transformations Test creation and execution Data-driven decisions Feedback loops Organizational Transformations Skill development Team structures Culture shifts Quality Assurance Transformations Enhanced coverage and accuracy Continuous improvement Compliance and security Transformation Use Cases
  • 11. What Can Go Wrong? Consistency Readability Understandability Relevance Environment Controllability Observability Ethical and Bias Intuition Repeatability Coverage Compatibility Acceptance Data Quality Continuous Learning A strategic transformation approach is needed !
  • 12. Strategic Goals Model Applications Maturity Model Goals Topics Priorities Strategic Assessment Discovery Survey Gap Assessment Current state VSM Baseline metrics Strategic Plan Future state VSM Themes, Epics, Stories, Tasks Tools selection Dependencies Timeline Governance Implementations Pipeline Migrations Integration Dashboard Governance Checkpoint Assessments 5,6,7 REALIZE, OPERATIONALIZE EXPAND 4 SOLUTION ALIGNMENT 3 ASSESSMENT 1,2 VISION/GOAL ALIGNMENT Discovery Surveys Value Stream tools Topics and Practices Metrics and Analytics Deliverable templates AI-Assisted Tools & Templates AI-assisted Analysis and Results Alignment Transformation Consultant Process Leadership and Collaboration AI-Assisted CT Strategic Transformation Blueprint
  • 13. Example Strategic Transformation Roadmap Theme 1 Model AI Platform MVP -------------- Test environment, generation, and automation Templates & Tools Selected test types Metrics Workflows Training Retrospective Theme 2 Consolidation -------------- Test Results side Standard pipelines Migrate other application Center of Excellence Guilds and Training Re-assessment Theme 3 Coverage Expansion -------------- Types of tests Advance Training More applications Re-assessment Release auomation Theme 4 Optimization -------------- Deployment automation Accelerate test creation Accelerate test execution testing Accelerate test analysis Continuous Experimentation 1 2 3 4 Every transformation needs it own strategic roadmap!
  • 14. Transformation Implementation Patterns that Work Dedicated Platform Team Embedded Team Outsourced Team Hybrid Dedicated/Outsource Team
  • 16. Motivating Continuous Testing Transformations Educate and Align Demonstrate Value Develop Skills and Infrastructure Cultural Shift Toward Testing Automation Incorporate Feedback Mechanisms Adapt Project Management and Processes Encourage Innovation Highlight ROI and Performance Metrics Strengthen Vendor Partnerships Manage Change Effectively
  • 17. Measuring Transformation Success DORA Metrics: Deployment Frequency (DF) Lead Time for Changes (LT) Change Failure Rate (CFR) Time to Restore Service (TRS) Additional Metrics: Test Coverage Defect Detection Efficiency Automated Test Pass Rate Flakiness Score Test Execution Time Cost per Defect Found Feedback Loop Duration ROI of Testing
  • 18. Measuring Transformation ROI Costs: Development/Integration Costs Software Costs Hardware Costs Training Costs Maintenance Costs Benefits: Increased Efficiency Reduced Labor Costs Improved Quality Enhanced Test Coverage Intangible Benefits: Improved Team Morale and Productivity Enhanced Market Reputation
  • 19. Benefits of Continuous Testing On-time delivery Improved predictability Reduced cost of quality Improved customer experience Improved productivity Better quality Faster releases Fewer security events Improved compliance scores up to 70% efficiency gains due to the reduction of wait-time and toil - Without AI ! What will it be with AI-Assist ?
  • 20. The Future DevInOps No Shift, No Lead Time Enablers Observability AI-Automation Feature Flag Rollouts No-Shift Faster Lead time DevSecOps Continuous Delivery Shift-Left Production Development in Production At some point smart automation will make it so there will be nothing Left to shift!
  • 21. AI-Assisted: DevSecOps and SRE Transformations Continuous Testing Consulting EngineeringDevOps.com mhornbeek@engineeringdevops.com