How to ensure robust assessment in the light of Generative AI developments
Download as PPTX, PDF
0 likes262 views
Presented Liz Hardie and Mychelle Pride (Open University, UK) as part of the EADTU Webinar Week 'AI in Education: Assessment, Ethics, and the Future of Learning'
More Related Content
Similar to How to ensure robust assessment in the light of Generative AI developments (20)
More from EADTU (20)
Recently uploaded (20)
How to ensure robust assessment in the light of Generative AI developments
- 1. How to ensure robust assessment in the light of Generative AI developments Presenters: Liz Hardie and Mychelle Pride Research authors: Liz Hardie, Jonquil Lowe, Mychelle Pride, Kevin Waugh, Mirjam Hauck, Francine Ryan, Daniel Gooch, Volker Patent, Kieran McCartney, Claire Maguire, Mike Richards, Heather Richardson The Open University and NCFE 25 October 2024
- 2. Agenda Research outline How well does GAI perform? Detecting GAI and impact of training Findings and recommendations
- 3. Research outline Research questions: Which question types are most/least easily answered by GAI? Does that vary across disciplines and levels? Does training markers improve GAI script detection? Method: Select questions (59) representative of different types (17), levels (4) and disciplines (17) For each question, markers (43) mark a batch of 8 scripts, mixed student (5) and GAI (3). Markers undergo short online training For each question, markers mark second batch of 8 scripts, mixed student (5) and GAI (3). Total 944 scripts (472 in each batch; 354 GAI scripts across both batches) Analysis Mixed methods (quantitative and qualitative): marking records; focus groups; training quizzes and survey
- 4. 4 How well does GAI perform by question type? 5 = Pass 1 4 = Pass 2 (Upper second) 3 = Pass 3 (Lower second) 2 = Pass 4 1 = Fail N = 944 scripts
- 5. Activity Plan 2nd year course Economic in Practice Imagine that you work in a health-policy consultancy firm. You have been hired by the government of a small country that is completely new to evaluating its policies. You must provide advice on a plan to impose a tax on fat in take-away foods. Write a briefing to be submitted to the government that: describes in general terms the purpose of doing evaluations and the types of evaluation the council might want to consider articulates a plausible ultimate goal of the policy and suggests a theory of change for how a tax on fat in take-away foods might achieve that goal suggests a design for an impact or economic evaluation, including the type of data that would need to be gathered and any distributional and ethical issues you have considered. Make sure you read the Student notes for Part A which contain important instructions about the format and content of your briefing.
- 6. 6 Guidance
- 7. 7 Detection of GAI scripts and impact of training For each question, detection of GAI can range from 0 to 3 scripts correctly detected. N = 58 questions
- 8. 8 Detection of student scripts and impact of training For each question, detection of student can range from 0 to 5 scripts correctly detected. N = 58 questions It's increased my confidence, however the decline in my score between the pre- and post-training quiz emphasises that identifying the GenAI tool outputs is more an art than a science.
- 9. 9 GAI hallmarks and weaknesses I could see how the ways in which GAI cannot really come to a conclusion is different from the ways in which students just fail to put forward an argument GAI tends to provide both sides of an argument and then its not coming to a conclusion, while students just fail to put forward a convincing argument throughout. N = 287 correct GAI scripts; 59 False positive scripts
- 10. 10 Question-types that may be more robust
- 11. 11 Key findings - assessment Training improves GAI detection but also increases false positives. Detection might not be feasible false positives and time it takes. No question types are currently robust but some types are definitely less robust and others are amenable to embedding more robust design. The amount of time that I was spending contemplating whether or not these questions were AI-generated or not, now, if I was to then apply that to my day-to-day role as a tutor, the amount of time that I would need to spend to really think about it in detail, it becomes impractical.
- 12. Recommendations - assessment Hallmarks of GAI: key value is in informing question design, marking guidance, and student skills interventions, rather than GAI-misuse detection. Question design and marking guidance: risk assess question design (eg University of Greenwich AI Risk Measure Scale); use clear rubric and explicit criteria to determine a fail; design curriculum to support robust assessment. Institutional approach: Use a team approach to foster responsible and ethical usage of GAI in order to ensure clarity, consistency and fairness to all students (non-GAI users, responsible GAI-users, irresponsible GAI- users).
- 13. 13 Recommendations academic conduct and skills Skills development o Equip teaching staff to use GAI to strengthen students study skills. o Before referring for academic misconduct, engage with students with a focus on support and developing skills; maintain a record of these interventions. Conduct referrals o Provide teaching staff with clarity about their role and with clear academic conduct referral guidance. o Ensure referral decisions are proportionate and timely.
- 14. 14 Recommendations GAI training Who to train: Markers; staff who produce curriculum and set assessments; students. When to train: needs to be ongoing. GAI is not standing still, so neither can we. Content of training: o Staff: GAI hallmarks, question and marking guidance design, discipline specific issues, when to refer to skills intervention. o Students: GAI hallmarks, GAI skills for study and employability.
- 15. Further information Open University: https://law-school.open.ac.uk/research/developing-robust-assessment-light-generative-ai- developments NCFE: our Assessment Innovation Fund pilot with The Open University: https://www.ncfe.org.uk/help-shape-the-future-of-learning-and-assessment/aif-pilots/the- open-university/ Training resource (as used in the research): https://www5.open.ac.uk/stem/computing-and-communications/research/generative-ai-st aff-development Liz.Hardie@open.ac.uk Mychelle.Pride@open.ac.uk