ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo

Ahab's Leg Dilemma

?Download as PPTX, PDF?
?
Luca Sabatucci
Luca Sabatucci

Luca Sabatucci Ahab's Leg Dilemma - on the Design of a Controlled Experiment Empire 2011, Trento, Italy

1 of 18
Download to read offline
Ahab's Leg Dilemma: on the Design of a Controlled Experiment Luca Sabatucci Mariano Ceccato Alessandro Marchetto Angelo Susi
The Ahab¡¯s Leg dilemma ? When changing media (or communication style) we need to add details to a story, to keep the story engaging Umberto Eco 1956 Ray Bradbury & John Huston.
The Ahab¡¯s Leg dilemma ? The peg leg is fundamental for the story ¨C Deciding which leg is a peg one has no bearing on it ¨C When the peg-leg is instantiated, this decision may generate of lot of consequences 1930 Warner Bros. 1956 Ray Bradbury & John Huston. Directed by Lloyd Bacon.
Ahab¡¯s Leg in RE ? Often, narrative scenarios are used to validate requirements with stakeholders in focus groups ? Narrative scenarios are derived from requirements (change of communication style) ? Details must be added during translation to instantiate generic requirements into a concrete spatial-temporal context ? Stakeholders might be distracted by irrelevant details
An example from our experience Requirement: the system communicates with caregivers with low and high priority signals The camera PDA displays identifies Fall on that an the event staircase unknown person and sends a is fallen in the signal to staircase caregiver¡¯s PDA Maria falls on the staircase ? The focus group was proceeding well until a nurse commented on the PDA (Ahab¡¯s Leg) ¨C PDA is intrusive (to carry around, battery¡­) and it would change working practices ¨C Lively discussion on less intrusive devices ¨C This was pointless, because the kind of device was not yet decided Credits: picture (c) By Chiara Leonardi
Mandatory Vs Optional ? Some are mandatory to make the story concrete and believable (e.g., PDA) ¨C Concreteness is important for stakeholders to envisage functionalities ? Some are optional and choreographic, needed just to increase the stakeholders engagement (e.g., name of the patient) ? Does mandatory and optional ALs affect in the same way requirement validation sessions? The camera PDA displays identifies Fall on that an the event staircase unknown person and sends a is fallen in the signal to staircase caregiver¡¯s PDA Maria falls on the staircase Credits: picture (c) By Chiara Leonardi
Cardinality ? Optional ALs can be removed ¨C Many: scenarios very concrete but with the risk of high distraction ¨C Few: scenarios very abstract, difficult to present to stakeholders ? Influence of personal and contextual background: ¨C Melville did not specify if all the member of the crew had two legs. But the reader assumes it based on his/her common knowledge of the real world ¨C Abstract scenarios make stakeholders mentally complete missing details ? The initial scenario is corrupted ? No possibility to control ALs by the analyst The camera PDA displays identifies Fall on that an the event staircase unknown person and sends a is fallen in the signal to staircase caregiver¡¯s PDA Maria falls on the staircase
Stakeholder awareness ? Not realistic to compare scenario with and without ALs (the latter does not make sense) ? If the discussion is moderated by a facilitator, he/she could highlight irrelevant details to avoid spending time in discussing about them ¨C Risk of attracting even more attention on them ¨C Just mention that there are more and less important details (with some example) ? Exploiting stakeholder awareness is probably the more realistic approach and analyst would take, to limit distraction.
The plan ? The role of Ahab¡¯s Leg has been observer during a real project validation session ? We conjecture that this is not due to the specific project, but the problem is more general ? Test this conjecture in a controlled and repeatable in-lab experiment ¨C We control/measure all the relevant variables ¨C We change just one variable and we study the effect
Research questions ? Ahab¡¯s Legs are often unavoidable ? Not a problem, unless they divert the attention from important aspects of the story. RQ1: What is the actual impact of Ahab¡¯s Legs on the distraction of stakeholders during a requirement validation sessions? RQ2: Is there a reliable way to reduce their impact on the distraction?
Context of the experiment ? Subjects: Computer science master students. ¨C Background on software and requirement engineering ¨C Some actual development experience ? Objects: 2 software system found on the internet ¨C MyBanking: home banking application for mobile devices, designed to replace credit cards and cash. ¨C MyShopping: augmented reality application for mobile devices that display information on items pointed by the camera.
Hypothesis formulation ? H0: explicitly mentioning what are the over- specified details (Ahab¡¯s Legs) in application scenarios does not significantly reduce the distraction in a requirement validation session. ? HA: explicitly mentioning what are the over- specified details (Ahab¡¯s Legs) in application scenarios significantly reduces the distraction in a requirement validation session.
Variable selection ? Independent variable: explicitly telling that there are details irrelevant for the discussion (Ahab¡¯s Legs) or without such explicit mention. ? Dependent variable: distraction observed during the requirement validation phase. ¨C The stakeholder comment addresses a topic that is not part of the requirement (e.g., Ahab¡¯s Leg) distractioni =1 ¨C The comment requires to fix/reconsider a requirement distractioni =0 ¨C Disrtaction = distractioni
Co-factors that we measure ? Learning effect between the two labs ? System for which requirements are validated ? Subjects¡¯ academic merit as the average of exam score ? Subjects¡¯ background measured as they attended relevant courses ? Previous subjects¡¯ experience ¨C On requirement validation ¨C On industrial software development
Experimental design Group1 Group2 Group3 Group4 Lab1 MyBanking AL MyBanking No MyShopping AL MyShopping No Lab2 MyShopping No MyShopping AL MyBanking No MyBanking AL ? Fill the profiling pre-questionnaire ? Lab 1 ¨C Read the description of the first application ¨C For each of the 4 scenarios ? Read a scenario ? Write a comment/question for the scenario ? Lab 2 ¨C Read the description of the first application ¨C For each of the 4 scenarios ? Read a scenario ? Write a comment/question for the scenario ? Fill the feedback questionnaire.
Missing aspects? ? Other strategies to control the influence of Ahab¡¯s Leg dilemma? ? Trade-off between distraction and level of abstraction? ? What is the influence of subject background?
Conclusions
Questions?

More Related Content

Ahab's Leg Dilemma

  • 1. Ahab's Leg Dilemma: on the Design of a Controlled Experiment Luca Sabatucci Mariano Ceccato Alessandro Marchetto Angelo Susi
  • 2. The Ahab¡¯s Leg dilemma ? When changing media (or communication style) we need to add details to a story, to keep the story engaging Umberto Eco 1956 Ray Bradbury & John Huston.
  • 3. The Ahab¡¯s Leg dilemma ? The peg leg is fundamental for the story ¨C Deciding which leg is a peg one has no bearing on it ¨C When the peg-leg is instantiated, this decision may generate of lot of consequences 1930 Warner Bros. 1956 Ray Bradbury & John Huston. Directed by Lloyd Bacon.
  • 4. Ahab¡¯s Leg in RE ? Often, narrative scenarios are used to validate requirements with stakeholders in focus groups ? Narrative scenarios are derived from requirements (change of communication style) ? Details must be added during translation to instantiate generic requirements into a concrete spatial-temporal context ? Stakeholders might be distracted by irrelevant details
  • 5. An example from our experience Requirement: the system communicates with caregivers with low and high priority signals The camera PDA displays identifies Fall on that an the event staircase unknown person and sends a is fallen in the signal to staircase caregiver¡¯s PDA Maria falls on the staircase ? The focus group was proceeding well until a nurse commented on the PDA (Ahab¡¯s Leg) ¨C PDA is intrusive (to carry around, battery¡­) and it would change working practices ¨C Lively discussion on less intrusive devices ¨C This was pointless, because the kind of device was not yet decided Credits: picture (c) By Chiara Leonardi
  • 6. Mandatory Vs Optional ? Some are mandatory to make the story concrete and believable (e.g., PDA) ¨C Concreteness is important for stakeholders to envisage functionalities ? Some are optional and choreographic, needed just to increase the stakeholders engagement (e.g., name of the patient) ? Does mandatory and optional ALs affect in the same way requirement validation sessions? The camera PDA displays identifies Fall on that an the event staircase unknown person and sends a is fallen in the signal to staircase caregiver¡¯s PDA Maria falls on the staircase Credits: picture (c) By Chiara Leonardi
  • 7. Cardinality ? Optional ALs can be removed ¨C Many: scenarios very concrete but with the risk of high distraction ¨C Few: scenarios very abstract, difficult to present to stakeholders ? Influence of personal and contextual background: ¨C Melville did not specify if all the member of the crew had two legs. But the reader assumes it based on his/her common knowledge of the real world ¨C Abstract scenarios make stakeholders mentally complete missing details ? The initial scenario is corrupted ? No possibility to control ALs by the analyst The camera PDA displays identifies Fall on that an the event staircase unknown person and sends a is fallen in the signal to staircase caregiver¡¯s PDA Maria falls on the staircase
  • 8. Stakeholder awareness ? Not realistic to compare scenario with and without ALs (the latter does not make sense) ? If the discussion is moderated by a facilitator, he/she could highlight irrelevant details to avoid spending time in discussing about them ¨C Risk of attracting even more attention on them ¨C Just mention that there are more and less important details (with some example) ? Exploiting stakeholder awareness is probably the more realistic approach and analyst would take, to limit distraction.
  • 9. The plan ? The role of Ahab¡¯s Leg has been observer during a real project validation session ? We conjecture that this is not due to the specific project, but the problem is more general ? Test this conjecture in a controlled and repeatable in-lab experiment ¨C We control/measure all the relevant variables ¨C We change just one variable and we study the effect
  • 10. Research questions ? Ahab¡¯s Legs are often unavoidable ? Not a problem, unless they divert the attention from important aspects of the story. RQ1: What is the actual impact of Ahab¡¯s Legs on the distraction of stakeholders during a requirement validation sessions? RQ2: Is there a reliable way to reduce their impact on the distraction?
  • 11. Context of the experiment ? Subjects: Computer science master students. ¨C Background on software and requirement engineering ¨C Some actual development experience ? Objects: 2 software system found on the internet ¨C MyBanking: home banking application for mobile devices, designed to replace credit cards and cash. ¨C MyShopping: augmented reality application for mobile devices that display information on items pointed by the camera.
  • 12. Hypothesis formulation ? H0: explicitly mentioning what are the over- specified details (Ahab¡¯s Legs) in application scenarios does not significantly reduce the distraction in a requirement validation session. ? HA: explicitly mentioning what are the over- specified details (Ahab¡¯s Legs) in application scenarios significantly reduces the distraction in a requirement validation session.
  • 13. Variable selection ? Independent variable: explicitly telling that there are details irrelevant for the discussion (Ahab¡¯s Legs) or without such explicit mention. ? Dependent variable: distraction observed during the requirement validation phase. ¨C The stakeholder comment addresses a topic that is not part of the requirement (e.g., Ahab¡¯s Leg) distractioni =1 ¨C The comment requires to fix/reconsider a requirement distractioni =0 ¨C Disrtaction = distractioni
  • 14. Co-factors that we measure ? Learning effect between the two labs ? System for which requirements are validated ? Subjects¡¯ academic merit as the average of exam score ? Subjects¡¯ background measured as they attended relevant courses ? Previous subjects¡¯ experience ¨C On requirement validation ¨C On industrial software development
  • 15. Experimental design Group1 Group2 Group3 Group4 Lab1 MyBanking AL MyBanking No MyShopping AL MyShopping No Lab2 MyShopping No MyShopping AL MyBanking No MyBanking AL ? Fill the profiling pre-questionnaire ? Lab 1 ¨C Read the description of the first application ¨C For each of the 4 scenarios ? Read a scenario ? Write a comment/question for the scenario ? Lab 2 ¨C Read the description of the first application ¨C For each of the 4 scenarios ? Read a scenario ? Write a comment/question for the scenario ? Fill the feedback questionnaire.
  • 16. Missing aspects? ? Other strategies to control the influence of Ahab¡¯s Leg dilemma? ? Trade-off between distraction and level of abstraction? ? What is the influence of subject background?

Editor's Notes

  • #4: 1956, John Houston directed a film adaptation of the book starring Gregory Peck as Captain Ahab