際際滷

際際滷Share a Scribd company logo

It Doesn't Do What You Think It Does

Justin Stoller
Justin Stoller

This document discusses strategies for gaining confidence in applications and systems. It begins by defining what is meant by "gaining confidence" and why it is important. It then outlines several topics to be covered, including using language paradigms like imperative, object-oriented, and functional programming to perform static and dynamic analysis. It will discuss testing at different levels from unit tests up to integration and system-level tests. It will also look at simulation testing to analyze emergent behaviors and how monitoring in production helps continue analysis.

1 of 20
Download to read offline
IT DOESN'T DO WHAT YOU THINK IT DOES A Survey of Strategies for Gaining Confidence in Applications and Systems Justin Stoller @juststoller justin.stoller@gmail.com
YOUR TITLE NEEDS A TL;DR way to go...
So what do we mean by "Gaining Confidence" and why should you care?
First that we have a clear idea of what we are trying to gain confidence in? What is the value of our software? What are the acceptance criteria? Inputs Outputs Performance Characteristics Visual Style
Does it meet the acceptance criteria?
What are the Rewards? What are the Costs? Where are the Risks?
A Few of the Topics Out of Scope Organization & Communication Architecture & Design
The Presentation 1. We will start with Language Paradigms and how they affect our abilities farther up the stack 2. Using Language constructs to perform both static analysis (type systems) and dynamic analysis (error handling) 3. Working our way up the stack to explicit low level tests 4. And farther up to higher level integration tests 5. We will look at combining what we've seen above into a simulation testing and the emergent behaviors that are found with these 6. And finally continuing the analysis from simulation into production via monitoring
You forgot to say why should care...
Language Paradigms Imperative vs Declarative Low Level vs High Level
Procedural Type Systems that Model the Components of a Computer Tests assert the state of the world Error Handling by way of return codes Goto like statements
Object Oriented Type Systems that Model the Application Domain Tests assert the state of an object Tests assert the passing of messages More reliance on type systems Limit exceptional behaviors to the boundaries of the system
Functional Type Systems that Model the Application Domain Tests assert the state of a returned data structure Tests assert the distribution returned data structures More reliance on type systems
Logical/Relational Type Systems as Theorem Provers Tests assert the end state of the application
Testing Layers
A single unit test of logic: conditional, etc Simple units of logic require simple tests However low level tests tend to be very tied to implementations Good design facility but rarely good at ensuring quality Often written in the language of the software and contains the pros and cons of the software's language
The behaviors of Integrated Units Tests are tied to behaviors of the systems Increasing complexity of tests and prerequisites Requires decent design, good at gaining confidence
The System as a Whole What properties does it display Analysis of emergent behaviors
A wide ranging spectrum Higher level languages start farther up the level of strategies The performance characteristics of the tests determine how often they are ran Higher level tests with simple ways to instantiate them provide more value as ways to gain confidence in the software.
Novelty in combining concepts Instrumented or "monitored" development environments

More Related Content

It Doesn't Do What You Think It Does

  • 1. IT DOESN'T DO WHAT YOU THINK IT DOES A Survey of Strategies for Gaining Confidence in Applications and Systems Justin Stoller @juststoller justin.stoller@gmail.com
  • 2. YOUR TITLE NEEDS A TL;DR way to go...
  • 3. So what do we mean by "Gaining Confidence" and why should you care?
  • 4. First that we have a clear idea of what we are trying to gain confidence in? What is the value of our software? What are the acceptance criteria? Inputs Outputs Performance Characteristics Visual Style
  • 5. Does it meet the acceptance criteria?
  • 6. What are the Rewards? What are the Costs? Where are the Risks?
  • 7. A Few of the Topics Out of Scope Organization & Communication Architecture & Design
  • 8. The Presentation 1. We will start with Language Paradigms and how they affect our abilities farther up the stack 2. Using Language constructs to perform both static analysis (type systems) and dynamic analysis (error handling) 3. Working our way up the stack to explicit low level tests 4. And farther up to higher level integration tests 5. We will look at combining what we've seen above into a simulation testing and the emergent behaviors that are found with these 6. And finally continuing the analysis from simulation into production via monitoring
  • 9. You forgot to say why should care...
  • 10. Language Paradigms Imperative vs Declarative Low Level vs High Level
  • 11. Procedural Type Systems that Model the Components of a Computer Tests assert the state of the world Error Handling by way of return codes Goto like statements
  • 12. Object Oriented Type Systems that Model the Application Domain Tests assert the state of an object Tests assert the passing of messages More reliance on type systems Limit exceptional behaviors to the boundaries of the system
  • 13. Functional Type Systems that Model the Application Domain Tests assert the state of a returned data structure Tests assert the distribution returned data structures More reliance on type systems
  • 14. Logical/Relational Type Systems as Theorem Provers Tests assert the end state of the application
  • 16. A single unit test of logic: conditional, etc Simple units of logic require simple tests However low level tests tend to be very tied to implementations Good design facility but rarely good at ensuring quality Often written in the language of the software and contains the pros and cons of the software's language
  • 17. The behaviors of Integrated Units Tests are tied to behaviors of the systems Increasing complexity of tests and prerequisites Requires decent design, good at gaining confidence
  • 18. The System as a Whole What properties does it display Analysis of emergent behaviors
  • 19. A wide ranging spectrum Higher level languages start farther up the level of strategies The performance characteristics of the tests determine how often they are ran Higher level tests with simple ways to instantiate them provide more value as ways to gain confidence in the software.
  • 20. Novelty in combining concepts Instrumented or "monitored" development environments