First introduced in 2013-2014, the assessment has already undergone one phase of evaluation. Key changes made in response to this first phase of participant feedback included: (1) decreasing the ratio of short answer to multiple-choice style questions to address students’ concerns about issues associated with typing answers including keyboarding proficiency and noise, and (2) organising the questions according to marks in order to support time management.
This presentation will focus on the results of the second phase of evaluation. In this phase of evaluation participant feedback was solicited after the mock examination and again after the final examination in order to factor out the influence of students’ perceptions of their performance on the final examination. A combination of surveys and focus groups was used to provide a rich picture of engagement with the online assessment methods. The survey adapted items from previous studies led by Dermo (2009) and Hillier (2014) and sought to investigate students’ perceptions of the assessment experience, exploring affective variables, perceptions of equity/validity, technical reliability, keyboarding and general ICT proficiency, as well as the pedagogic appropriateness of the assessment methods to the research methods course. Transcripts from the focus groups were analysed and compared with the open comments from the surveys to provide a rich picture of the student assessment experience.
The data suggests that typed responses are seen to act as both a discriminator and a leveller, with slower typists feeling at a disadvantage, and those with poor handwriting feeling that computer-based assessment is fairer than a paper-based assessment. Students also valued the ease with which they could edit their responses to open questions and the accuracy of the automated marking of closed questions.
On the basis of these results, and students’ recommendations, we will present for discussion a transferable framework for e-assessment coordinators to use, addressing the student engagement issues which need to be considered when designing computer-based assessments and preparing students for them. The latter issue, preparation or socialisation, is perhaps one of the most important considerations, with a large number of our students requesting increased opportunities for practice, and training in the skills necessary to succeed in the examination, including keyboarding, general ICT skills and exam technique.
Dermo (2009). British Journal of Educational Technology, 40 (2).
Hillier (2014). Proceedings ascilite Dunedin, pp. 77-88.