On the Validity of Peer Grading and a Cloud Teaching Assistant System
Download as pptx, pdf3 likes1,259 views
The document discusses the validity of peer grading and cloud teaching assistant systems, focusing on differences in student performance and grading across various criteria. Key research questions include how grades from peers, cloud teaching assistants (CTAs), and professors compare, and the validity of these grading methods. Results are presented through various data analyses, including scoring distributions, correlations, and potential biases in peer evaluations.
On the Validity of Peer Grading and a Cloud Teaching Assistant System
- 1. On the Validity of Peer Grading and a Cloud Teaching Assistant System Tim Vogelsang - March 2015 Lara Ruppertz Paper (ACM): http://dx.doi.org/10.1145/2723576.2723633
- 2. On the Validity of Peer Grading and a Cloud Teaching Assistant System Tim Vogelsang - March 2015 Lara Ruppertz Paper (ACM): http://dx.doi.org/10.1145/2723576.2723633
- 3. Q3: How do students performances and the grading types differ amend different scoring criteria? Q4: How valid are CTAS and peer grading? MOOC INTRODUCTION MOOC, every letter is negotiable by Mathieu Plourde is licensed under CC-BY.
- 7. On The Validity of Peer Grading and a Cloud Teaching Assistant System On The Validity of Peer Grading and a Cloud Teaching Assistant System
- 12. Peer Grading Systems RELATED WORK
- 13. Peer Grading Systems RELATED WORK SWoRD Peer Grader
- 14. On The Validity of Peer Grading and a Cloud Teaching Assistant System On The Validity of Peer Grading and a Cloud Teaching Assistant System
- 15. Cloud Teaching Assistants (CTAs) METHODS Platform
- 16. Cloud Teaching Assistants (CTAs) METHODS Platform
- 17. Cloud Teaching Assistants (CTAs) METHODS Platform
- 18. Cloud Teaching Assistants (CTAs) METHODS Platform
- 19. Cloud Teaching Assistants (CTAs) METHODS Platform
- 20. On The Validity of Peer Grading and a Cloud Teaching Assistant System On The Validity of Peer Grading and a Cloud Teaching Assistant System
- 21. Reliability and Validity METHODS Validity & Reliability by Nevit Dilmen is licensed under CC BY-SA 3.0.
- 22. Reliability and Validity METHODS Validity & Reliability by Nevit Dilmen is licensed under CC BY-SA 3.0.
- 24. Reliability and Validity INTRODUCTION Few Profs Assignment
- 25. Reliability and Validity INTRODUCTION Many Peers Few Profs Assignment
- 26. Reliability and Validity INTRODUCTION Many Peers Some CTAs Few Profs Assignment
- 27. Reliability and Validity INTRODUCTION Many Peers Some CTAs Few Profs Assignment Compute Reliability
- 28. Reliability and Validity INTRODUCTION Many Peers Some CTAs Few Profs Assignment Compute Reliability
- 29. Reliability and Validity INTRODUCTION Many Peers Some CTAs Few Profs Assignment Compute Reliability
- 30. Reliability and Validity INTRODUCTION Many Peers Some CTAs Few Profs Compute Validity Assignment Compute Reliability
- 31. On The Validity of Peer Grading and a Cloud Teaching Assistant System Experiment Setup & Methods
- 32. Course: Public Privacy EXPERIMENT SETUP Source: 11 pt, regular 32
- 33. Professor: Anja Mihr EXPERIMENT SETUP Screenshot taken from iversity.org platform. Image courtesy Anja Mihr.
- 34. Sample Size EXPERIMENT SETUP Many Peers Some CTAs Few Profs Compute Validity Compute Reliability Final Exam
- 35. Many Peers Some CTAs Few Profs Compute Validity Compute Reliability - 25 paid exam participants - 16 multiple choice questions - 476 exam participants - 4620 active students - 30k enrolments - 1 essay question Sample Size Final Exam EXPERIMENT SETUP
- 36. Many Peers (476) Some CTAs (4) Few Profs (1) Final Exam Compute Validity Compute Reliability - 25 paid exam participants - 16 multiple choice questions - 476 exam participants - 4620 active students - 30k enrolments - 1 essay question 25 Sample Size EXPERIMENT SETUP
- 37. Final Exam EXPERIMENT SETUP Screenshot taken from iversity.org platform. Image courtesy Anja Mihr. Source: 11 pt, regular 37
- 39. THESIS / CENTRAL IDEA Grading Rubrics EXPERIMENT SETUP I
- 40. THESIS / CENTRAL IDEA ORGANISATION Grading Rubrics EXPERIMENT SETUP I II
- 41. THESIS / CENTRAL IDEA ORGANISATION CONTENT Grading Rubrics EXPERIMENT SETUP I II III
- 42. THESIS / CENTRAL IDEA ORGANISATION CONTENT BALANCED ARGUMENTATION Grading Rubrics EXPERIMENT SETUP I II III IV
- 43. I II III IV V THESIS / CENTRAL IDEA ORGANISATION CONTENT BALANCED ARGUMENTATION SOURCES Grading Rubrics EXPERIMENT SETUP
- 44. Q3: How do students performances and the grading types differ amend different scoring criteria? Q4: How valid are CTAS and peer grading? Research Questions METHODS
- 45. Q1: How are grades given by peers, the CTAs and the instructor distributed? Q3: How does the grading differ between the grading rubrics? Q4: How valid are CTAS and peer grading? Research Questions METHODS
- 46. Q1: How are grades given by peers, the CTAs and the instructor distributed? Q2: How does the grading vary among the grading rubrics? Q4: How valid are CTAS and peer grading? Research Questions METHODS
- 47. Q1: How are grades given by peers, the CTAs and the professor distributed? Q2: How does the grading vary among the grading rubrics? Q3: How valid are peer and CTA grading? Research Questions METHODS
- 48. On The Validity of Peer Grading and a Cloud Teaching Assistant System Result s
- 49. Q1: How are grades given by peers, the CTAs and the professor distributed?
- 50. RESULTS Histogram of grades given by peers, CTAs and professor Range: 0 (worst grade) - 4 (best grade) Grade CTAs Peers Professor How are grades distributed?
- 51. Q2: How does the grading vary among the grading rubrics?
- 52. RESULTS Thesis Organisation Content Argumentation Sources Histogram of grades given by peers, CTAs and professor. Range: 0 (worst grade) - 4 (best grade). Distinguished by grading rubrics. CTAs Peers Professor Grade How does the grading vary among the rubrics? 52
- 53. RESULTS Average grade assigned by peers, CTAs and professor to a specific criterion. Range: 0 (worst grade) - 4 (best grade). Peers CTAs Professor How does the grading vary among the rubrics? 53
- 54. Q3: How valid are peer and CTA grading?
- 55. Many Peers (476) Some CTAs (4) Few Profs (1) Final Exam Compute Validity Compute Reliability - 25 paid exam participants - 16 multiple choice questions - 476 exam participants - 4620 active students - 30k enrolments - 1 essay question 25 RESULTS How valid are peer and CTA grading?
- 56. RESULTS How valid are peer and CTA grading? Correlation matrix using Pearsons correlation. 375 data points: (25 exams) x (5 criteria) x (3 grading groups).
- 57. RESULTS Scatterplot : One dot represents one final exam graded by peers, CTAs and professor. Range: 0 (worst grade) - 4 (best grade) Average CTA Grade How valid are peer and CTA grading?
- 58. RESULTS How valid are peer and CTA grading? Correlation matrix using Pearsons correlation. 375 data points: (25 exams) x (5 criteria) x (3 grading groups).
- 59. RESULTS How valid are peer and CTA grading? Peers CTAs Prof. Peers CTAs Prof. Correlation matrix using Pearsons correlation. 375 data points: (25 exams) x (5 criteria) x (3 grading groups).
- 60. Many Peers (476) Some CTAs (4) Few Profs (1) Final Exam Compute Validity Compute Reliability - 25 paid exam participants - 16 multiple choice questions - 476 exam participants - 4620 active students - 30k enrolments - 1 essay question 25 RESULTS How valid are peer and CTA grading?
- 61. TheCooperReview.com.Image courtesy: Sarah Cooper. 61
- 62. DISCUSSION Hypotheses / Future Work Repeat Experiment Adjust Experiment 62
- 63. Hypotheses / Future Work DISCUSSION Repeat Experiment Adjust Experiment Disagreement on argumentation? 63
- 64. Hypotheses / Future Work DISCUSSION Repeat Experiment Adjust Experiment Disagreement on argumentation? Peers bias? 64
- 65. Hypotheses / Future Work DISCUSSION Repeat Experiment Adjust Experiment Disagreement on argumentation? Peers bias? Advanced CTA selection process. 65
- 66. Hypotheses / Future Work DISCUSSION Repeat Experiment Adjust Experiment Disagreement on argumentation? Peers bias? Advanced CTA selection process. Peer calibration process. 66
- 67. How to define validity in the context of massive grading? Hypotheses / Future Work DISCUSSION Repeat Experiment Adjust Experiment Disagreement on argumentation? Peers bias? Advanced CTA selection process. Peer calibration process. 67
- 68. Disagreement on argumentation? Peers bias? Advanced CTA selection process. Peer calibration process. How to define validity in the context of massive grading? How to design a valid grading process including professor, peers and CTAs? Hypotheses / Future Work DISCUSSION Repeat Experiment Adjust Experiment 68
- 69. Hypotheses / Future Work DISCUSSION 69