際際滷

際際滷Share a Scribd company logo
Traffic Lights, Targets, & Tolstoy:
Introduction to Tolstoy Targets
Dennis Sweitzer
www.dennis-sweitzer.com
denswei@gmail.com
(Enterprise dashboard
with 4 dredging projects)
Outline
 Principles:
 Targets, Tolstoy & Traffic Lights
 Conventions
 The Waterline
 Symbols
 Practical Examples
 Project Results, Statuses
 Massively Parallel Outcomes
 Reporting
PRINCIPLES
Testing Drug Izitwerkin: Traffic Light
The predefined objectives
of the study are:
# 1 , #2. #3.. #4.. #5..
For X$ within Y months
Cut to the chase: we know
what are the objectives.
Did we meet them?
We need to make plans
for lunch..
Testing Drug Izitwerkin: Green Light
Great, we can quit
early
& celebrate
over lunch
Met all predefined objectives
for efficacy, safety, etc.
And also budget, timelines,.
Testing Drug Izitwerkin: Red Light
Clearly failed on predefined
objectives for efficacy, safety,
budget, timeline, etc.
Which failed? .
If its critical, we cancel the rest.
Well commiserate over lunch.
Testing Drug Izitwerkin: Yellow Light
Mostly met objectives, but.
Neither clear success
nor clear failure
But what? .
Should we have had more patients?
A little over budget? A little late?
Some bad luck? Unusual circumstances?
What can we salvage?
What do we have to redo?
Get lunch delivered,
its going to be a long day..
Targets: Testing Drug Izitwerkin
Multiple Objectives, Multiple Shots:
Timelines Budget
Adverse Events
Abnormal Lab Changes
Pain Relief
Drug Stability
Speed of Relief
Ready..
Targets: Aiming for Multiple Objectives
Timelines Budget
Adverse Events
Abnormal Lab Changes
Pain Relief
Drug Stability
Speed of
Relief
FIRE!!!Aim..
Targets: Traffic light color code
Timelines Budget
Adverse Events
Abnormal Lab Changes
Pain Relief
Drug Stability
Speed of
Relief
Targets: 1 Radial Axis per Objective
Timelines Budget
Adverse Events
Abnormal Lab Changes
Pain Relief
Drug Stability
Speed of
Relief
Targets: Simplify
Timelines Budget
Adverse Events
Abnormal Lab Changes
Pain Relief
Drug Stability
Speed of
Relief
Target: Minimize Clutter
Tolstoy?
Visual, rapid, high level understanding
without having to read & interpret
 Click on each target to drill down for details
Happy
families
are all alike;
Every failed project
fails
in its own way.
--Not Tolstoy
Every unhappy
family
is unhappy
in its own way.
--Leo Tolstoy
(Anna Karenina)
Successful projects
are all alike;
CONVENTIONS
Convention: The waterline
Above the waterline:
 Known unknowns
 Things we know
we dont know
 Eg, efficacy
 Things to get us off the ground
(MUST have some Green)
Below the waterline:
 Unknown unknowns
 Things we dont know
we dont know
 Eg, safety
Things to sink us
(MUST NOT have any Red)
Grouping Attributes by Direction
Easy to see
general
areas of
success
and failure!
Adding Confidence/Certainty Ranges
Elicited opinions,
Statistical Calculations
 Best Case
/Worst Case
 Hi/Med/Low
Uncertainty
 Best likely
/ Worst Likely
 Optimistic
/ Pessimistic
 Estimates
+ 95% Confidence Intervals
Piling on the Symbols
Circles
and pluses
and Xes,
oh my!
Perhaps:
 Splats are point estimates
O  Past Estimates
+  Optimal (nice to have)?
X  The competition?
Extreme Example (24 axes x 6 values)
 Complex but
Interpretable
 Additional
Symbols take
some effort
 Splats arent
bad
 Use sparingly
 Point estimates
 Range
O Past Estimates
+ Optimal
X The competition
PRACTICAL EXAMPLES
Comparing Projects in the ABC program
At a
glance,
can see
successes
& failures!
--And
Where!
Study ABC-OhNoStudy ABC-GoGo
Study ABC-GoSlo Study ABC-NoNo
Larger project with some issues 
apparently including damaged
equipment
Big Picture: Multiple Project Dashboard
(Randomly generated dredging examples)
Small project
with many
problems, but on
schedule and in
budget
Larger project with a
couple of possible
problems, but overall
doing well
Metrics for Dredging Projects
& Summary over all Projects
Bad weather, so a
little behind
schedule,
but in budget
Bars indicate range of metric over all projects
Mass Screening of Evolving Enzymes
Criteria:
At each step:
 Pick best candidates for each criteria
 Recombine those to generate new candidates
 Repeat until optimal
Alkalinity, Acidity, Yield, Salinity, Metal Tolerance,
Durability, Km (Michaelsons constant)
T.Targets provide comprehensive & visual
feedback on process
GenerationX
Generation1
Ex. Transfusion Risks
Whole plot
 One Patient
Each Radial Axis
 Blood type Group
Each Dot
 Patient Risk
Green Dot  Low Risk
 Normal procedures
Red Dot  High Risk
 Clear & identified risk
 Special Procedures
Yellow Dot  Uncertain Risk
 Moderate Risks
 Caution
 Further testing?
No Dot  No Information
臓No confusion with Low Risk
!
Transfusion Risk of Multiple Patients
Risks for 16 patients
 Each Legible
3 Redundancies:
 Color  Traffic-Light
coding
{Red, Yellow, Green}
 Location  Targets
Red on Rim, next to label
 Easier to identify
 Size  Proportional to Risk
The Tolstoy principal:
"All happy families are alike;
Every unhappy family is unhappy
in it's own way. (Anna Karenina)
Low Risk patients look alike
High Risk patients are distinct!
 What way & How much
Predicted Risks vs Outcomes
 Unclear connections between
Risks & Outcomes
 Add feedback on outcome
 Some risks may have a
stronger connection with bad
outcomes
Circle the target
(Same conventions)
Good Outcome  Thin Green
Mixed outcome  Yellow
Bad outcome  Thick Red
Unknown outcome  No Circle
Example: 
One Bad Outcome
 Patient with 2 risk factors
Mostly Good Outcomes
 Even with Risk Factors
Assessment Scores
 For 12 patients
 Factor scores from
3 assessments
 Mania,
Depression,
Schizophrenia
 3 Total Scores
+ 11 subscales
Ex. Gallup Well-Being Index
Pennsylvania
PA City Rankings as Tolstoy Targets
For more information, see my website:
www.dennis-sweitzer.com
My linked-In profile:
http://www.linkedin.com/in/dennissweitzer
Or email me:
denswei@gmail.com

More Related Content

TolstoyTarget,AnimatedExpl,v5

  • 1. Traffic Lights, Targets, & Tolstoy: Introduction to Tolstoy Targets Dennis Sweitzer www.dennis-sweitzer.com denswei@gmail.com (Enterprise dashboard with 4 dredging projects)
  • 2. Outline Principles: Targets, Tolstoy & Traffic Lights Conventions The Waterline Symbols Practical Examples Project Results, Statuses Massively Parallel Outcomes Reporting
  • 4. Testing Drug Izitwerkin: Traffic Light The predefined objectives of the study are: # 1 , #2. #3.. #4.. #5.. For X$ within Y months Cut to the chase: we know what are the objectives. Did we meet them? We need to make plans for lunch..
  • 5. Testing Drug Izitwerkin: Green Light Great, we can quit early & celebrate over lunch Met all predefined objectives for efficacy, safety, etc. And also budget, timelines,.
  • 6. Testing Drug Izitwerkin: Red Light Clearly failed on predefined objectives for efficacy, safety, budget, timeline, etc. Which failed? . If its critical, we cancel the rest. Well commiserate over lunch.
  • 7. Testing Drug Izitwerkin: Yellow Light Mostly met objectives, but. Neither clear success nor clear failure But what? . Should we have had more patients? A little over budget? A little late? Some bad luck? Unusual circumstances? What can we salvage? What do we have to redo? Get lunch delivered, its going to be a long day..
  • 8. Targets: Testing Drug Izitwerkin Multiple Objectives, Multiple Shots: Timelines Budget Adverse Events Abnormal Lab Changes Pain Relief Drug Stability Speed of Relief Ready..
  • 9. Targets: Aiming for Multiple Objectives Timelines Budget Adverse Events Abnormal Lab Changes Pain Relief Drug Stability Speed of Relief FIRE!!!Aim..
  • 10. Targets: Traffic light color code Timelines Budget Adverse Events Abnormal Lab Changes Pain Relief Drug Stability Speed of Relief
  • 11. Targets: 1 Radial Axis per Objective Timelines Budget Adverse Events Abnormal Lab Changes Pain Relief Drug Stability Speed of Relief
  • 12. Targets: Simplify Timelines Budget Adverse Events Abnormal Lab Changes Pain Relief Drug Stability Speed of Relief
  • 14. Tolstoy? Visual, rapid, high level understanding without having to read & interpret Click on each target to drill down for details Happy families are all alike; Every failed project fails in its own way. --Not Tolstoy Every unhappy family is unhappy in its own way. --Leo Tolstoy (Anna Karenina) Successful projects are all alike;
  • 16. Convention: The waterline Above the waterline: Known unknowns Things we know we dont know Eg, efficacy Things to get us off the ground (MUST have some Green) Below the waterline: Unknown unknowns Things we dont know we dont know Eg, safety Things to sink us (MUST NOT have any Red)
  • 17. Grouping Attributes by Direction Easy to see general areas of success and failure!
  • 18. Adding Confidence/Certainty Ranges Elicited opinions, Statistical Calculations Best Case /Worst Case Hi/Med/Low Uncertainty Best likely / Worst Likely Optimistic / Pessimistic Estimates + 95% Confidence Intervals
  • 19. Piling on the Symbols Circles and pluses and Xes, oh my! Perhaps: Splats are point estimates O Past Estimates + Optimal (nice to have)? X The competition?
  • 20. Extreme Example (24 axes x 6 values) Complex but Interpretable Additional Symbols take some effort Splats arent bad Use sparingly Point estimates Range O Past Estimates + Optimal X The competition
  • 22. Comparing Projects in the ABC program At a glance, can see successes & failures! --And Where! Study ABC-OhNoStudy ABC-GoGo Study ABC-GoSlo Study ABC-NoNo
  • 23. Larger project with some issues apparently including damaged equipment Big Picture: Multiple Project Dashboard (Randomly generated dredging examples) Small project with many problems, but on schedule and in budget Larger project with a couple of possible problems, but overall doing well Metrics for Dredging Projects & Summary over all Projects Bad weather, so a little behind schedule, but in budget Bars indicate range of metric over all projects
  • 24. Mass Screening of Evolving Enzymes Criteria: At each step: Pick best candidates for each criteria Recombine those to generate new candidates Repeat until optimal Alkalinity, Acidity, Yield, Salinity, Metal Tolerance, Durability, Km (Michaelsons constant) T.Targets provide comprehensive & visual feedback on process GenerationX Generation1
  • 25. Ex. Transfusion Risks Whole plot One Patient Each Radial Axis Blood type Group Each Dot Patient Risk Green Dot Low Risk Normal procedures Red Dot High Risk Clear & identified risk Special Procedures Yellow Dot Uncertain Risk Moderate Risks Caution Further testing? No Dot No Information 臓No confusion with Low Risk !
  • 26. Transfusion Risk of Multiple Patients Risks for 16 patients Each Legible 3 Redundancies: Color Traffic-Light coding {Red, Yellow, Green} Location Targets Red on Rim, next to label Easier to identify Size Proportional to Risk The Tolstoy principal: "All happy families are alike; Every unhappy family is unhappy in it's own way. (Anna Karenina) Low Risk patients look alike High Risk patients are distinct! What way & How much
  • 27. Predicted Risks vs Outcomes Unclear connections between Risks & Outcomes Add feedback on outcome Some risks may have a stronger connection with bad outcomes Circle the target (Same conventions) Good Outcome Thin Green Mixed outcome Yellow Bad outcome Thick Red Unknown outcome No Circle Example: One Bad Outcome Patient with 2 risk factors Mostly Good Outcomes Even with Risk Factors
  • 28. Assessment Scores For 12 patients Factor scores from 3 assessments Mania, Depression, Schizophrenia 3 Total Scores + 11 subscales
  • 29. Ex. Gallup Well-Being Index Pennsylvania
  • 30. PA City Rankings as Tolstoy Targets
  • 31. For more information, see my website: www.dennis-sweitzer.com My linked-In profile: http://www.linkedin.com/in/dennissweitzer Or email me: denswei@gmail.com

Editor's Notes

  • #5: The History of every major Galactic Civilization tends to pass through three distinct and recognizable phases, those of Survival, Inquiry and Sophistication, otherwise known as the How, Why, and Where phases. For instance, the first phase is characterized by the question 'How can we eat?' the second by the question 'Why do we eat?' and the third by the question 'Where shall we have lunch?
  • #11: For pdf version (replaces previous 2 slides)
  • #20: Mutated from WLGores management principals. Plimsoll mark Above & below the waterline is easy. Is there any other ways to categorize? (say, left-right) because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns - - the ones we don't know we don't know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones. Donald Rumsfield Plimsoll mark Above & below the waterline is easy. Is there any other ways to categorize? (say, left-right)
  • #23: These were just convenient to do in Excel. Additional elements should be added sparingly
  • #24: Very sparingly, as appropriate: Say, Os for last budget & Timelines, Xs for competitors in select outcomes +s for ???? For epidemiology: Os might be observed parameters (infectivity, etc) for isolated cases, Splats & bars might be
  • #26: Abc-GoGo is a happy family: objectives were met. Abc-GoSlo is pretty average. Got some issues Abc-OhNo has some problems in some areas Abc-NoNo has lots of problems
  • #28: Evolving Enzymes Must meet criteria for Durability, Metal Tolerance, Salinity, Yield Seek best criteria for Acidity, Alkalinity, Km (Michaelsons constant)
  • #29: We had brainstormed about blood bank uses of the plot on my Business card, so I made a few to see how they would look. Attached are 3 4 5 figures, one with a single plot, two with 16, and one with 100. (here I explored about how to use them for transfusion risk, but they can be used in many ways) The basic plot for a transfusion candidate (below) has blood group typing systems on radial axes, with ABO and Rh on the horizontal axes for easy identifiably, less frequent ones below, and more frequent ones above. The interpretation of this is that: Green dots (toward the center) indicate normal procedures can be used with low risk to the patient, Red dots (outermost) indicate that special handling is required because of a clear & identified risk to the patients (say, known to be heterozygous), and Yellow dots (middle band) indicate there are risk factors (say, the patient is part of an at-risk ethnic group). NB: the plot would change for a patient as more information became known. An Rh- patient might automatically be in the yellow, and an Rh- patient who has been exposed to Rh antigens might automatically be classified in the red. There is also different levels within each band (3 risk bands x 3 levels each). For instance, on the ABO axis, an AB patient would be on the innermost level because they are "universal recipients", while an O patient would be on the outermost green level, an A or B type would be on the middle level, but an O patient exposed to A or B antigens might go into the Red. The 9 levels are relatively intuitive since people are conditioned to rate things on a scale of 0 to 10. I usually ask people to imagine the best possible outcome (score=0), the worst possible outcome (score = 0), where the subject falls among Good, Bad, and Mediocre (eg, Low, High, and Medium risk), and then how do they compare to others within the color category (so an extremely Low risk patient might score =1, while a relatively Low risk patient might score=3). Note that the size of the dot increases with the level of risk, so there is 3 redundancies in the ranking (color, location, and size) that make it very easy to rapidly interpret the plots: the color (Red, Yellow, Green, just like a traffic light); the distance from the center (like a target), and the size. Consequently, the highest risks cases are the most obvious, low risk cases are clearly seen as such, and there is little risk of confusing a lack of data (no dot) with a low risk case (small green dot). The graph also follows the Tolstoy principal: "All happy families are alike; Every unhappy family is unhappy in it's own way." (the 1st line of Anna Karenina). Panels of targets can present a rapid overview for many subjects, with the goal of identifying subjects that need special attention. The figure with 16 targets is fairly readable: you can still read off the blood group typing on the labels; in the figure with 100 targets, you can clearly see the category reds & yellows, and can still distinguish the Rh & ABO axes because they are horizontal. I envision a data display in which one clicks on a target out of 100's to drill down for a detailed examination. (It could also automatically select the high risk ones by various criteria). An attending physician might only need see the plot for an individual patient to get an immediate & intuitive grasp of there risks. The floor nurse might review a panel of all patients on the floor to identify those needing special attention; An administrator might review panels of dozens for QC purposes. The last figure includes the outcomes of the transfusion, in order to complete the picture of the represented cases. This might be used to refine the risk assessment process itself (perhaps a green dot should have been red or yellow), or to identify procedural problems (did the staff follow the proper procedures for a yellow dot case? Or was there a flaw in the procedures?). The figure follows the same conventions as the rest of the plot: a thin green line indicates a good outcome; a thick red one, bad; a yellow circle, a mixed outcome; no circle, no outcome. Additional information can be included on the plot, such a a solid line indicating the range of uncertainty in an estimate, circles for the last prior risk assessment (if it has changed), etc. This plot can be used in many other ways: For instance, in a plot of blood product supplies in the entire blood bank the axes could represent blood types & products, the dots represent current stocks, an overlaid circle could indicate predicted future stocks (perhaps several weeks ahead, taking into account expected needs & supplies), and confidence limits as an overlaid solid bar indicating the range of uncertainty in predicted future stocks (perhaps from simulations based on historical data. In this use, if a confidence limit enters a red band, contingency plans might be implemented to reduce that risk).
  • #30: I envision a data display in which one clicks on a target out of 100's to drill down for a detailed examination. (It could also automatically select the high risk ones by various criteria). An attending physician might only need see the plot for an individual patient to get an immediate & intuitive grasp of there risks. The floor nurse might review a panel of all patients on the floor to identify those needing special attention; An administrator might review panels of dozens for QC purposes. The last figure includes the outcomes of the transfusion, in order to complete the picture of the represented cases. This might be used to refine the risk assessment process itself (perhaps a green dot should have been red or yellow), or to identify procedural problems (did the staff follow the proper procedures for a yellow dot case? Or was there a flaw in the procedures?). The figure follows the same conventions as the rest of the plot: a thin green line indicates a good outcome; a thick red one, bad; a yellow circle, a mixed outcome; no circle, no outcome. Additional information can be included on the plot, such a a solid line indicating the range of uncertainty in an estimate, circles for the last prior risk assessment (if it has changed), etc. This plot can be used in many other ways: For instance, in a plot of blood product supplies in the entire blood bank the axes could represent blood types & products, the dots represent current stocks, an overlaid circle could indicate predicted future stocks (perhaps several weeks ahead, taking into account expected needs & supplies), and confidence limits as an overlaid solid bar indicating the range of uncertainty in predicted future stocks (perhaps from simulations based on historical data. In this use, if a confidence limit enters a red band, contingency plans might be implemented to reduce that risk).
  • #31: The last figure includes the outcomes of the transfusion, in order to complete the picture of the represented cases. This might be used to refine the risk assessment process itself (perhaps a green dot should have been red or yellow), or to identify procedural problems (did the staff follow the proper procedures for a yellow dot case? Or was there a flaw in the procedures?). The figure follows the same conventions as the rest of the plot: a thin green line indicates a good outcome; a thick red one, bad; a yellow circle, a mixed outcome; no circle, no outcome. Row1, Col3: bad outcome in a patient with 2 red risks. Are either of those 2 risks particularly associated a bad outcome? * the upper right risk (red) also occurred in 2 other patients with good outcomes (row1, cols 1&2) * the upper left red risk occurred in 1 other patient with a mixed outcome (42, c2). these 2 risk factors might be synergistic