際際滷

際際滷Share a Scribd company logo

13-itemanalysis-161127124647.pptx

Download as PPTX, PDF
L
LuisSalenga1

Item analysis is a process used to evaluate the quality and performance of individual test items. It involves analyzing students' responses to separate test items and comparing performance on individual items to overall test scores. The main tasks of item analysis are assessing item discrimination and difficulty. Item discrimination examines how well an item distinguishes between students who know the material well versus those who do not. Item difficulty looks at the proportion of students who answered an item correctly. Electronic and manual methods are used to calculate discrimination and difficulty indices to evaluate item quality. Item response theory is an alternative statistical approach to item analysis.

1 of 19
Download to read offline
ITEM ANALYSIS Ann Meredith Garcia, MD
What is item analysis? Judging the quality of test items by examining the students responses Competent vs. less competent students? Difficulty of items?
How is item analysis done? Administer the test Check the students responses to separate items Check the total scores
Tasks of item analysis 1st task: Item discrimination Sort the students who know the topic well from those who do not Correlate performance on a single test item with total test performance (+) Correlation better discrimination 2nd task: Item difficulty
Electronic item analysis Student Item 1 Score Total Score / 30 A 1 25 B 1 19 C 1 18 D 0 16 E 1 12 F 0 10 Average total score (correct Item 1) = 18.5 Average total score (incorrect Item 1) = 13 Computed correlation coefficient = 0.53 Item is to some extent related to the total score Correct = 1 Incorrect = 0
Electronic item analysis r = correlation of an option with the total score p = percentage of students who chose that option (n = 65) Correct options should show positive correlations; distractors should show negative correlations Item 1: r = 0.25 low correlation Item 2: r = 0.49 fairly good correlation Item 3: r = 0.34 modest correlation r 0.15 course content is not being assessed well eliminate the item (OR revise) Item A B C 1 r = -0.27 p = 13.89 r = 0.25 p = 50.00 r = -0.06 p = 36.11 2 r = -0.46 p = 5.56 r = 0.49 p = 88.86 r = -0.22 p = 5.56 3 r = -0.30 p = 16.67 r = -0.13 p = 27.78 r = 0.34 p = 55.56
Electronic item analysis r = correlation of an option with the total score p = percentage of students who chose that option (n = 65) Standard error (SE) = 1 / (number of students 1) = 0.12 Any r > 2(SE) will be accepted as other than a chance relationship between the item and the total score Item 1: r = 0.25 > 0.24 [2(SE)] very marginal but acceptable Item A B C 1 r = -0.27 p = 13.89 r = 0.25 p = 50.00 r = -0.06 p = 36.11 2 r = -0.46 p = 5.56 r = 0.49 p = 88.86 r = -0.22 p = 5.56 3 r = -0.30 p = 16.67 r = -0.13 p = 27.78 r = 0.34 p = 55.56
Item analysis by hand Step 1: Arrange the students papers according to their test scores (highest to lowest). Step 2: Divide these into high scorers vs. low scorers. Step 3: Tabulate the number of students who chose each option in both groups. Step 4: Compute for the discrimination index. Item 1 A B C D Total High scorers 2 4 0 16 22 Low scorers 12 7 0 4 23
Item analysis by hand Step 4: Compute for the discrimination index (DI). DI = (NumHigh NumLow) Number of students in larger group = (16 4) / 23 = 0.52 * Ranges from 0 1.00 * Can also be negative (for distractors) Item 1 A B C D Total High scorers 2 4 0 16 22 Low scorers 12 7 0 4 23
Item analysis by hand Alternative: Straight difference method Steps 1 3: Same Step 4: Compute for NumHigh NumLow. If 0.10(n) adequate (used across all items) 16 4 = 12 0.10(45) = 4.5 Item 1 A B C D Total High scorers 2 4 0 16 22 Low scorers 12 7 0 4 23
Analysis of distractors Distractor C was not chosen by any student 3-option item (0.33 instead of 0.25 chance level of guessing the item correctly) Good item each distractor will be chosen more often by the low scorers Item 1 A B C D High scorers 4 13 0 3 Low scorers 7 9 0 4 p 27.5 55.0 0.0 18.5
Tasks of item analysis 2nd task: Item difficulty Difficulty index/facility index = proportion of students who get an item correctly Step 1: Award a score to each student. Step 2: Arrange the scored tests from highest to lowest. Step 3: Identify the upper and lower 27%. Step 4: Count the response counts in each group. Item 1 A B C D Total High scorers 2 4 0 16 22 Low scorers 11 7 0 4 22
Tasks of item analysis 2nd task: Item difficulty Difficulty index/facility index = proportion of students who get an item correctly Step 5: Calculate the difficulty index. = H + L N H = no. of students in the high group with a correct answer L = no. of students in the low group with a correct answer N = total no. of students Item 1 A B C D Total High scorers 2 4 0 16 22 Low scorers 11 7 0 4 22
Tasks of item analysis 2nd task: Item difficulty Difficulty index/facility index = proportion of students who get an item correctly Ranges from 0 1.00 Best 0.50 (0.30 0.70) Larger index easier item; smaller index more difficult item DI = (16 + 4) / 80 = 0.25 Criteria: 0.35 = excellent question 0.25 0.34 = good question 0.15 0.24 = marginal question revise <0.15 = poor question discard Item 1 A B C D Total High scorers 2 4 0 16 22 Low scorers 11 7 0 4 22
Item analysis for essay tests Step 1: Identify the upper and lower 25% of the students. Step 2: Compute for the following: Disc. = (Sum of scores for highs sum of score for lows) N x (max. possible score on item) Diff. = (Sum of scores for highs + sum of score for lows) 2N x (max. possible score on item) N = 25% of the number tested Item Score High Group Low Group No. of Students No. of Students x Score No. of Students No. of Students x Score 10 9 90 1 10 8 6 48 0 0 6 2 12 4 24 4 3 12 7 28 2 0 0 8 16 Total 20 162 20 78
Item analysis for essay tests Step 2: Compute for the following: Disc. = (Sum of scores for highs sum of score for lows) N x (max. possible score on item) = (162 78) / [(0.25 x 80) x 10] = 0.42 Satisfactory discrimination Item Score High Group Low Group No. of Students No. of Students x Score No. of Students No. of Students x Score 10 9 90 1 10 8 6 48 0 0 6 2 12 4 24 4 3 12 7 28 2 0 0 8 16 Total 20 162 20 78
Item analysis for essay tests Step 2: Compute for the following: Diff. = (Sum of scores for highs + sum of score for lows) 2N x (max. possible score on item) = (162 + 78) / [(2 x 0.25 x 80) x 10] = 0.60 Satisfactory difficulty Item Score High Group Low Group No. of Students No. of Students x Score No. of Students No. of Students x Score 10 9 90 1 10 8 6 48 0 0 6 2 12 4 24 4 3 12 7 28 2 0 0 8 16 Total 20 162 20 78
Item response theory and item analysis Calculates the odds of getting an item right converts this number to a natural logarithm Allows faculty to equate the item difficulty scale on one test to the scale of another test and across different student groups Useful system for building item banks
Thank you!

More Related Content

13-itemanalysis-161127124647.pptx

  • 2. What is item analysis? Judging the quality of test items by examining the students responses Competent vs. less competent students? Difficulty of items?
  • 3. How is item analysis done? Administer the test Check the students responses to separate items Check the total scores
  • 4. Tasks of item analysis 1st task: Item discrimination Sort the students who know the topic well from those who do not Correlate performance on a single test item with total test performance (+) Correlation better discrimination 2nd task: Item difficulty
  • 5. Electronic item analysis Student Item 1 Score Total Score / 30 A 1 25 B 1 19 C 1 18 D 0 16 E 1 12 F 0 10 Average total score (correct Item 1) = 18.5 Average total score (incorrect Item 1) = 13 Computed correlation coefficient = 0.53 Item is to some extent related to the total score Correct = 1 Incorrect = 0
  • 6. Electronic item analysis r = correlation of an option with the total score p = percentage of students who chose that option (n = 65) Correct options should show positive correlations; distractors should show negative correlations Item 1: r = 0.25 low correlation Item 2: r = 0.49 fairly good correlation Item 3: r = 0.34 modest correlation r 0.15 course content is not being assessed well eliminate the item (OR revise) Item A B C 1 r = -0.27 p = 13.89 r = 0.25 p = 50.00 r = -0.06 p = 36.11 2 r = -0.46 p = 5.56 r = 0.49 p = 88.86 r = -0.22 p = 5.56 3 r = -0.30 p = 16.67 r = -0.13 p = 27.78 r = 0.34 p = 55.56
  • 7. Electronic item analysis r = correlation of an option with the total score p = percentage of students who chose that option (n = 65) Standard error (SE) = 1 / (number of students 1) = 0.12 Any r > 2(SE) will be accepted as other than a chance relationship between the item and the total score Item 1: r = 0.25 > 0.24 [2(SE)] very marginal but acceptable Item A B C 1 r = -0.27 p = 13.89 r = 0.25 p = 50.00 r = -0.06 p = 36.11 2 r = -0.46 p = 5.56 r = 0.49 p = 88.86 r = -0.22 p = 5.56 3 r = -0.30 p = 16.67 r = -0.13 p = 27.78 r = 0.34 p = 55.56
  • 8. Item analysis by hand Step 1: Arrange the students papers according to their test scores (highest to lowest). Step 2: Divide these into high scorers vs. low scorers. Step 3: Tabulate the number of students who chose each option in both groups. Step 4: Compute for the discrimination index. Item 1 A B C D Total High scorers 2 4 0 16 22 Low scorers 12 7 0 4 23
  • 9. Item analysis by hand Step 4: Compute for the discrimination index (DI). DI = (NumHigh NumLow) Number of students in larger group = (16 4) / 23 = 0.52 * Ranges from 0 1.00 * Can also be negative (for distractors) Item 1 A B C D Total High scorers 2 4 0 16 22 Low scorers 12 7 0 4 23
  • 10. Item analysis by hand Alternative: Straight difference method Steps 1 3: Same Step 4: Compute for NumHigh NumLow. If 0.10(n) adequate (used across all items) 16 4 = 12 0.10(45) = 4.5 Item 1 A B C D Total High scorers 2 4 0 16 22 Low scorers 12 7 0 4 23
  • 11. Analysis of distractors Distractor C was not chosen by any student 3-option item (0.33 instead of 0.25 chance level of guessing the item correctly) Good item each distractor will be chosen more often by the low scorers Item 1 A B C D High scorers 4 13 0 3 Low scorers 7 9 0 4 p 27.5 55.0 0.0 18.5
  • 12. Tasks of item analysis 2nd task: Item difficulty Difficulty index/facility index = proportion of students who get an item correctly Step 1: Award a score to each student. Step 2: Arrange the scored tests from highest to lowest. Step 3: Identify the upper and lower 27%. Step 4: Count the response counts in each group. Item 1 A B C D Total High scorers 2 4 0 16 22 Low scorers 11 7 0 4 22
  • 13. Tasks of item analysis 2nd task: Item difficulty Difficulty index/facility index = proportion of students who get an item correctly Step 5: Calculate the difficulty index. = H + L N H = no. of students in the high group with a correct answer L = no. of students in the low group with a correct answer N = total no. of students Item 1 A B C D Total High scorers 2 4 0 16 22 Low scorers 11 7 0 4 22
  • 14. Tasks of item analysis 2nd task: Item difficulty Difficulty index/facility index = proportion of students who get an item correctly Ranges from 0 1.00 Best 0.50 (0.30 0.70) Larger index easier item; smaller index more difficult item DI = (16 + 4) / 80 = 0.25 Criteria: 0.35 = excellent question 0.25 0.34 = good question 0.15 0.24 = marginal question revise <0.15 = poor question discard Item 1 A B C D Total High scorers 2 4 0 16 22 Low scorers 11 7 0 4 22
  • 15. Item analysis for essay tests Step 1: Identify the upper and lower 25% of the students. Step 2: Compute for the following: Disc. = (Sum of scores for highs sum of score for lows) N x (max. possible score on item) Diff. = (Sum of scores for highs + sum of score for lows) 2N x (max. possible score on item) N = 25% of the number tested Item Score High Group Low Group No. of Students No. of Students x Score No. of Students No. of Students x Score 10 9 90 1 10 8 6 48 0 0 6 2 12 4 24 4 3 12 7 28 2 0 0 8 16 Total 20 162 20 78
  • 16. Item analysis for essay tests Step 2: Compute for the following: Disc. = (Sum of scores for highs sum of score for lows) N x (max. possible score on item) = (162 78) / [(0.25 x 80) x 10] = 0.42 Satisfactory discrimination Item Score High Group Low Group No. of Students No. of Students x Score No. of Students No. of Students x Score 10 9 90 1 10 8 6 48 0 0 6 2 12 4 24 4 3 12 7 28 2 0 0 8 16 Total 20 162 20 78
  • 17. Item analysis for essay tests Step 2: Compute for the following: Diff. = (Sum of scores for highs + sum of score for lows) 2N x (max. possible score on item) = (162 + 78) / [(2 x 0.25 x 80) x 10] = 0.60 Satisfactory difficulty Item Score High Group Low Group No. of Students No. of Students x Score No. of Students No. of Students x Score 10 9 90 1 10 8 6 48 0 0 6 2 12 4 24 4 3 12 7 28 2 0 0 8 16 Total 20 162 20 78
  • 18. Item response theory and item analysis Calculates the odds of getting an item right converts this number to a natural logarithm Allows faculty to equate the item difficulty scale on one test to the scale of another test and across different student groups Useful system for building item banks