�ݺ�ߣ

Editor's Notes

• #3: Assessing the validity of an assessment includes the 3 elements: Construct validity; Content validity; Face validity. The first two of these ought to be assured through the normal awarding body processes relating to unit validation.The third is more subjective and has several dimensions. It is important that the tutor and the student should be confident that the processes and demands of the assessment are in line with their expectations.Any assessment process should not require the candidate to possess knowledge and/or skills that are not directly relevant to the assessment (a complaint often levied against essay writing in exams!). In the case of computer-enabled assessment it is important that the knowledge of how to operate a computer and/or the skills to use software do not become a barrier (or a distraction) for the candidateThe measuring of reliability (on an item-by-item basis) is technically difficult and usually requires lots of data. So measuring reliability was always going to be a challenge with the small number of centres each with a small number of candidates
• #4: There are National Assessment Bank items for each of the units in the study – these exemplify the standards as set out in the Unit Specification (and have been validated by SQA’s processes)The challenge for the study was to be assured that the GBA provided an equivalent demand (and remained in line with the standards as set out in the Unit Specification)Making judgements about face validity and extraneous factors was based around observation of candidate behaviour and subsequent interviews with both candidates and tutorsThe initial thought around measuring reliability was to randomise the order in which pupils attempted; (1) the paper NAB; and (2) the GBA
• #5: At SCQF Levels 4 and 5 (Intermediate 1 and Intermediate 2) assessment instruments tend to cluster around the Knowledge level in Bloom’s taxonomy. Most questions ask candidates to ‘identify’ or ‘describe’ – both of which are satisfied by the selection/matching exercises in the GBAsIn some instances (eg open-book testing) the requirement to ‘explain’ can be taken to be equivalent to matching or selectingBut in other cases these mechanisms in the GBA do not match the requirement to ‘explain’. In these cases a text-entry field is required (not available in the GBA versions in the study)In some cases the candidate evidence was obtained through performance (with associated assessor checklist) – this was more difficult to achieve by GBA selection and/or matchingWhere the candidate was required to produce a report (eg risk assessment) the GBA approach was to select and/or match and then incorporate these answers into a template that became the report
• #6: Measuring reliability was made considerably more difficult by the common practices of tutors to make assessment more manageable and/or more holisticIn carrying out the fieldwork, none of the centres used a NAB as a single assessment event. Tutors broke NABs into manageable chunksWithin these chunks there was a fairly common strategy of assess / remediate / re-assess Some tutors treated the NAB templates as instruments that had to be followed to the letter (down to the number of lines in an answer box!)
• #7: There were implementation issues to be overcome in almost every centreIn spite of this tutors were really enthusiastic about the GBAs and what they offered in the way of engagement & motivation especially for the more disengaged studentsThey would really like scripts to be computer-marked – but accepted that computer-produced scripts did save them having to read students' handwritingThey were disappointed that it was not possible to present students with only part of the game
• #8: Students LOVED the absence of writing in the completion of these assessmentsThey agreed that the contexts conveyed in the gameplay were helpful to them in answering the questions100& agreement that they had no difficulty in playing the GBA or reading the screenStudents with learning difficulties (and requiring readers/scribes) appeared to cope well unaided in playing the game
• #9: The linear nature of the game was a real issue for studentsTo correct a mistake they were asked to play the whole game again. This was seen as a strong negative for students who pointed out that this facility (ie going back to previous answer) was readily available in paper NAB
• #10: Adjacent screens and identical screen questions and layouts raised issues around cheating