際際滷

際際滷Share a Scribd company logo

Issues in experimental design for the study of atypical language developmentDavid session 1 handout

David Salda単a Sage
David Salda単a Sage

The document discusses issues in experimental design for studying atypical language development. It covers the differences between controlled experiments, field experiments, and natural experiments. Controlled experiments allow for precise control but lack generalizability, while field and natural experiments are more reflective of real life but have less control. The document also addresses within-subject repeated measures designs versus between-group independent measures designs. It notes the importance of counterbalancing to control for order effects. Additional topics covered include matching groups on relevant variables, checking the distribution of data to select appropriate statistical tests, and keeping thorough lab notes.

1 of 18
Download to read offline
08/10/2013 Issues in experimental design for the study of atypical language development David Salda単a DESIGNING YOUR STUDY: CONTROL 1
08/10/2013 Controlled experiments This type of experiment is conducted in a well-controlled environment not necessarily a laboratory and therefore accurate measurements are possible. Easier to replicate Precise control of extraneous and independent variables. The artificiality of the setting and lack of generalization Demand characteristics or experimenter effects may bias the results McLeod, S. A. (2012). Field experiments The experimenter still manipulates the independent variable, but in a reallife setting (so cannot really control extraneous variables). More likely to reflect real life because of it natural setting Less control over extraneous variables Less likelihood of demand characteristics affecting the results Less replicable Can be used in situations in which it would be ethically unacceptable to manipulate the independent variable, e.g. researching stress. McLeod, S. A. (2012). 2
08/10/2013 Natural experiments Natural Experiments are conducted in the everyday environment of the participants but here the experimenter has no control over the IV as it occurs naturally in real life. More likely to reflect real life because No control over extraneous variables of it natural setting Less replicable Less likelihood of demand Possible more expensive characteristics affecting the results Can be used in situations in which it would be ethically unacceptable to manipulate the independent variable, e.g. researching stress. Less likelihood of demand characteristics affecting the results, as participants may not know they are being studied. McLeod, S. A. (2012). A perhaps silly research question Does vocabulary influence the comprehension of questions more than the comprehension of imperatives? 3
08/10/2013 A question that is a little more complicated Do children with SLI understand questions better than typically developing children? Do children with SLI understand questions better than imperatives? DESIGNING YOUR STUDY: CONDITIONS 4
08/10/2013 How is your IV distributed? Independent Measures Repeated Measures Matched Pairs Independent measures (between-group) Each group gets one condition Different participants in each group Avoids practice More people needed Participant variables could affect results. 5
08/10/2013 Repeated measures (within-group) Task 1 Task 2 Measurement Measurement - Less participants - Precision determined by variation within same subject - May be the only design that answers the questions of interest. - Order effects: practice and fatigue effect Counterbalancing Moment 1 Moment 2 A B B A 6
08/10/2013 Back to our silly study Lets design the tasks and the data collection Does vocabulary influence the comprehension of questions more than the comprehension of imperatives? A little bit more interesting (perhaps) research question Does vocabulary influence the comprehension of questions more than the comprehension of imperatives in SLI? 7
08/10/2013 Mixed-design Group 1 Task 1 Task 2 Measurement Measurement Group 2 Task 1 Task 2 Measurement Measurement Some imaginative results SLI Typical 120 100 Standard Scores 100 99 95 92 90 80 77 60 70 73 90 90 90 75 40 20 0 IQ Vocabulary Comprehension Questions Word Reading SES Comprehension Imperatives 8
08/10/2013 MATCHING Matching your groups: why? SLI Typical 120 100 Standard Scores 100 99 95 92 90 80 77 60 70 73 90 90 90 75 40 20 0 IQ Vocabulary Comprehension Questions Word Reading SES Comprehension Imperatives 9
08/10/2013 Matching your groups: on what? 6 8 10 2 4 6 8 2 4 6 8 Imp.Comp 6 8 10 IQ 8 5 10 15 QuestComp 4 6 Voc 8 10 2 4 6 SES 2 4 6 8 5 10 15 4 6 8 10 Your groups: three group design Group of interest Control group, matched on chronological age Control group, matched on variable of interest 10
08/10/2013 Some issues Match on variables of interest to the study Match average and distribution Study carefully your exclusions Recruit clinical sample Decide variables of matching Recruit age controls Recruit level controls Lets do it! Matching exercise Some issues Recruit clinical sample Match on variables of interest to the study Match average and distribution Study carefully your exclusions Decide variables of matching Recruit age controls Recruit level controls Check and report 11
08/10/2013 What statistical test? An ideal world Facon et al. (2011) What statistical test? An ideal world 12
08/10/2013 What statistical test? An ideal world The real world Non-normal distributions Different range and variance Different distributions What statistical test? None: look at the picture first! Back-to-back histogram Back-to-back stem-and-leaf 13
08/10/2013 What statistical test? What statistical test? Mann-Whitney? Two-sample Kolmogorov-Smirnov test For small sample sizes: exact version Does not assume prior shape in distributions Tests for the differences in distribution 14
08/10/2013 Matching your groups: how much? The easy-peasy way Matching your groups: how much? A brief recall of significance testing I say No Match I say Match Match No Match http://intuitor.com/statistics/T1T2Errors.html 15
08/10/2013 Casual acceptance of the null hypothesis (Harcum, 1990) < .2 too low Frick (1995) .2 to .5 ambiguous > .5 fine http://intuitor.com/statistics/T1T2Errors.html Some things to remember Population comparisons are group comparisons (Paradis, 2010) 16
08/10/2013 Some things to remember Each group must be there for a reason (Paradis, 2010) An example: why each group? Two groups of deaf adolescents SAL 16 deaf (13-21) good oral language SBL 16 deaf (13-21) poor oral language Three groups of hearing adolescents OCS 20 adolescents (13-21) task with sound OSS 20 adolescents (13-21) task with no sound NO 20 children (6-11) oral language equivalent to SAL Torres (2013) 17
08/10/2013 Some things to remember Check matching while you are recruiting (Paradis, 2010) Keep your lab notes up to date Some important decisions to record: Changes and discussions on criteria Subjects in subjects out. Transcriptions and criteria if using MLU (Paradis, 2010) 18

More Related Content

Issues in experimental design for the study of atypical language developmentDavid session 1 handout

  • 1. 08/10/2013 Issues in experimental design for the study of atypical language development David Salda単a DESIGNING YOUR STUDY: CONTROL 1
  • 2. 08/10/2013 Controlled experiments This type of experiment is conducted in a well-controlled environment not necessarily a laboratory and therefore accurate measurements are possible. Easier to replicate Precise control of extraneous and independent variables. The artificiality of the setting and lack of generalization Demand characteristics or experimenter effects may bias the results McLeod, S. A. (2012). Field experiments The experimenter still manipulates the independent variable, but in a reallife setting (so cannot really control extraneous variables). More likely to reflect real life because of it natural setting Less control over extraneous variables Less likelihood of demand characteristics affecting the results Less replicable Can be used in situations in which it would be ethically unacceptable to manipulate the independent variable, e.g. researching stress. McLeod, S. A. (2012). 2
  • 3. 08/10/2013 Natural experiments Natural Experiments are conducted in the everyday environment of the participants but here the experimenter has no control over the IV as it occurs naturally in real life. More likely to reflect real life because No control over extraneous variables of it natural setting Less replicable Less likelihood of demand Possible more expensive characteristics affecting the results Can be used in situations in which it would be ethically unacceptable to manipulate the independent variable, e.g. researching stress. Less likelihood of demand characteristics affecting the results, as participants may not know they are being studied. McLeod, S. A. (2012). A perhaps silly research question Does vocabulary influence the comprehension of questions more than the comprehension of imperatives? 3
  • 4. 08/10/2013 A question that is a little more complicated Do children with SLI understand questions better than typically developing children? Do children with SLI understand questions better than imperatives? DESIGNING YOUR STUDY: CONDITIONS 4
  • 5. 08/10/2013 How is your IV distributed? Independent Measures Repeated Measures Matched Pairs Independent measures (between-group) Each group gets one condition Different participants in each group Avoids practice More people needed Participant variables could affect results. 5
  • 6. 08/10/2013 Repeated measures (within-group) Task 1 Task 2 Measurement Measurement - Less participants - Precision determined by variation within same subject - May be the only design that answers the questions of interest. - Order effects: practice and fatigue effect Counterbalancing Moment 1 Moment 2 A B B A 6
  • 7. 08/10/2013 Back to our silly study Lets design the tasks and the data collection Does vocabulary influence the comprehension of questions more than the comprehension of imperatives? A little bit more interesting (perhaps) research question Does vocabulary influence the comprehension of questions more than the comprehension of imperatives in SLI? 7
  • 8. 08/10/2013 Mixed-design Group 1 Task 1 Task 2 Measurement Measurement Group 2 Task 1 Task 2 Measurement Measurement Some imaginative results SLI Typical 120 100 Standard Scores 100 99 95 92 90 80 77 60 70 73 90 90 90 75 40 20 0 IQ Vocabulary Comprehension Questions Word Reading SES Comprehension Imperatives 8
  • 9. 08/10/2013 MATCHING Matching your groups: why? SLI Typical 120 100 Standard Scores 100 99 95 92 90 80 77 60 70 73 90 90 90 75 40 20 0 IQ Vocabulary Comprehension Questions Word Reading SES Comprehension Imperatives 9
  • 10. 08/10/2013 Matching your groups: on what? 6 8 10 2 4 6 8 2 4 6 8 Imp.Comp 6 8 10 IQ 8 5 10 15 QuestComp 4 6 Voc 8 10 2 4 6 SES 2 4 6 8 5 10 15 4 6 8 10 Your groups: three group design Group of interest Control group, matched on chronological age Control group, matched on variable of interest 10
  • 11. 08/10/2013 Some issues Match on variables of interest to the study Match average and distribution Study carefully your exclusions Recruit clinical sample Decide variables of matching Recruit age controls Recruit level controls Lets do it! Matching exercise Some issues Recruit clinical sample Match on variables of interest to the study Match average and distribution Study carefully your exclusions Decide variables of matching Recruit age controls Recruit level controls Check and report 11
  • 12. 08/10/2013 What statistical test? An ideal world Facon et al. (2011) What statistical test? An ideal world 12
  • 13. 08/10/2013 What statistical test? An ideal world The real world Non-normal distributions Different range and variance Different distributions What statistical test? None: look at the picture first! Back-to-back histogram Back-to-back stem-and-leaf 13
  • 14. 08/10/2013 What statistical test? What statistical test? Mann-Whitney? Two-sample Kolmogorov-Smirnov test For small sample sizes: exact version Does not assume prior shape in distributions Tests for the differences in distribution 14
  • 15. 08/10/2013 Matching your groups: how much? The easy-peasy way Matching your groups: how much? A brief recall of significance testing I say No Match I say Match Match No Match http://intuitor.com/statistics/T1T2Errors.html 15
  • 16. 08/10/2013 Casual acceptance of the null hypothesis (Harcum, 1990) < .2 too low Frick (1995) .2 to .5 ambiguous > .5 fine http://intuitor.com/statistics/T1T2Errors.html Some things to remember Population comparisons are group comparisons (Paradis, 2010) 16
  • 17. 08/10/2013 Some things to remember Each group must be there for a reason (Paradis, 2010) An example: why each group? Two groups of deaf adolescents SAL 16 deaf (13-21) good oral language SBL 16 deaf (13-21) poor oral language Three groups of hearing adolescents OCS 20 adolescents (13-21) task with sound OSS 20 adolescents (13-21) task with no sound NO 20 children (6-11) oral language equivalent to SAL Torres (2013) 17
  • 18. 08/10/2013 Some things to remember Check matching while you are recruiting (Paradis, 2010) Keep your lab notes up to date Some important decisions to record: Changes and discussions on criteria Subjects in subjects out. Transcriptions and criteria if using MLU (Paradis, 2010) 18