Clariana AERA 2023 presentation.pptx
- 1. The Influence of Essay Prompt Directedness on the Content Quality of Summary Writing by Roy B. Clariana (rbc4@psu.edu), College of Education Ryan Solnosky (rls5008@psu.edu), College of Engineering The Pennsylvania State University A presentation at the Annual Meeting of the American Educational Research Association (AERA), Chicago, IL, April 14, 2023. Funding from the Division of Undergraduate Education of the National Science Foundation (Award Abstract #2215807), Roy B. Clariana (PI). link
- 2. The Influence of Essay Prompt Directedness on the Content Quality of Summary Writing As an important step in the development of GIKS browser-based writing-to-learn software that provides immediate structural feedback, we seek ways to improve the quality of students essays and to optimize the software analysis algorithm. This quasi-experimental investigation compares the quality of students summary writing under three writing prompt conditions, otherwise identical prompts add either 0, 14, or 26 key terms. Results show that key terms matter substantially the summary essays of those given the prompt without key terms had longer essays and the resulting networks of those essays were more like the expert referent and like their peers essays. Although tentative, these results indicate that writing prompts should NOT include key terms. 2 of 10
- 3. GIKS browser-based tool (new version) 3 of 10 You can play with GIKS at: https://tinyurl.com/MyGIKs
- 4. 4 of 10 Network feedback screen
- 5. Participants Participating students in two separate sections of the 3-credit residential course AE 222 Building Modeling and Documentation were asked to complete one of three similar essay prompts, called Focus, Full, and no terms Focus group n = 44 Full group, n = 40 No terms group = 73 Note: this quasi-experimental design adds a control no terms condition Focus the central nodes in the expert network Full central + peripheral nodes in the expert network 5 of 10 Dissertation by Yu Wang (2022), randomly assigned to groups Sample of convenience one year later See next slide for details
- 6. Materials Essay prompt: Reflect back on the readings and lecture this week regarding wood-frame building. Write a broad description of this in your own words (about 300 words long). Here are some key terms that you could include in your summary: fastener, floor, framing, joists, nails, opening, plate, platform, roof truss, residential, roof, sheathing, stud, walls, blocking, bottom plate, commercial, door, fire stop, floor truss, header, rafter, subfloor, top plate, window, wood 6 of 10 In the question (2) Focus terms (14) Full terms (14 +12) Instructors network Focus Full
- 7. General Findings Essay length Open > Focus (p < .001, ES = 1.21), and Open > Full (p < .001, ES = 1.06), Focus fell midway between the two, adding key terms to the writing prompt decreased the total number of words in the essays. Central terms including key terms (full, focus, none) in the writing prompt made no difference on the number of central terms in the essays. (see term freq. slide 9) Peripheral terms Including key terms (full, focus, none) in the writing prompt made no difference on the number of peripheral terms in the essays. (see term freq. slide 9) Essay network similarity with the expert network as a measure of essay quality Counterintuitively, including only the central key terms in the writing prompt decreased the essay network similarity to the expert. Open > Focus (p = .001, ES = .61), Full fell between the two Essay network similarity with peers averaged network as a measure of peer-peer knowledge convergence Open > Focus (p = .003, ES = .64), Full fell between the two 7 of 10 NSD
- 8. Findings 8 of 10 Essay length Open > Focus (p < .001, ES = 1.21), and Open > Full (p < .001, ES = 1.06), Focus fell midway between the two, adding key terms to the writing prompt decreased the total number of words in the essays. Essay network similarity with the expert network as a measure of essay quality Counterintuitively, including only the central key terms in the writing prompt decreased the essay network similarity to the expert. Open > Focus (p = .001, ES = .61), Full fell between the two Essay network similarity with peers averaged network as a measure of peer-peer knowledge convergence Open > Focus (p = .003, ES = .64), Full fell between the two In this case, adding terms to the essay prompt messed up the essay networks structure relative to the expert network and to peers essay networks
- 9. 9 of 10 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 student essay term freq. H focus H full H none Term average frequency Full group (solid) Focus (dashed) Prompt (3) + 14 focus terms 12 full terms Remaining 21 high frequency terms no terms group (dashed) framing wood floors walls roof building studs joists sheathing platform residential plates nails opening roof truss fastener timber construction foundation top plate rafter commercial subfloor window fire header bottom plate door blocking floor truss type member light support wood framing balloon structure floor joist step exterior light wood material structural concrete platform framing interior wall framing roof framing typically truss component 2 Central terms including key terms in the writing prompt (full, focus, none) made no difference on the number of central terms in the essays. Peripheral terms Including key terms in the writing prompt (full, focus, none) made no difference on the number of peripheral terms in the essays. You would expect that the Focus group essays would have higher term frequency for the focus terms and the Full group essays would have higher term frequency for the Full terms This did NOT occur, term frequency in essays was consistent across all treatments (full, focus, none) Term frequencies
- 10. Discussion/Conclusion Reason and common sense would suggest that including key terms in a writing prompt should scaffold writing and would thus result in better essays. For example, low-directed concept mapping, where no key terms are provided, tended to better reveal conceptual understanding (both explanations and errors) than did high-directed mapping that provided some or all terms and some structure (Ruiz-Primo et al. 2001). This did NOT happen, no terms was best. Why? Pre-existing group differences are the most likely explanation of the findings (quasi-experimental design) But perhaps including key terms in the writing prompt disrupted retrieval during writing in this case because the alphabetical list of terms misaligns with both the writers conceptual structure of this content as well as with the actual lesson conceptual structure. Research into part-list cuing inhibition (Slamecka, 1968) has shown that providing several items as cues counterintuitively decreases recall memory relative to no cues. Basden, Basden, and Stephens (2002) explain this inhibition by showing that cue order maters. Inhibition could occur when the structure of the cue (i.e., here, the alphabetical list of key terms in the prompt) mismatches the undergirding conceptual structure of the lesson. Alternately, terms lists in the prompt may influence regulation regarding, is my essay complete now?, i..e, once you include all the terms, you are done Further research is needed to determine if key terms inhibition observed in this investigation, as well as part- list cuing inhibition, are mediated by matching or mismatching of conceptual structures of the learner, the cue, and the lesson. 10 of 10
- 11. Questions?