\nAt the Mathematics Education Centre at Loughborough University we have been exploring an alternative approach to marking to help with the assessment of nebulous knowledge and skills. Comparative judgement (Pollitt 2012) involves no marking and no assessment criteria. Instead, an assessor is presented with two pieces of students\u2019 work and asked which is better in terms of a construct such as conceptual understanding. Many such pairings are presented to several assessors and the outcomes are statistically analysed to construct a scaled rank order of students\u2019 work from worst to best. The rank order can then be used for the assessment procedures such as grading.<\/p>\n
Comparative judgement is based on the long standing principle that human beings are very reliable when comparing one object to another, but are often very unreliable at gauging a single object\u2019s position on a scale (Laming 1984). For instance, it is likely that you can state whether the temperature of the room in which you are reading this article is higher or lower than that outside the building. However, you are probably unable to produce an accurate estimate of the temperature in degrees Celsius.<\/p>\n
Recent technology developments mean the principle of comparative judgement is now viable for assessing nebulous learning outcomes. Student work can be delivered online to assessors for pairwise judging without the need for printing and sending round large numbers of hardcopies. Judgement decisions can be recorded in real-time, and used to select pairings of scripts thereby reducing the number of judgements required to construct a scaled rank order.<\/p>\n
For example, consider a scenario in which there are 100 scripts. For every script to be compared once with every other script would require almost 5000 individual pairwise judgements. Fortunately, in practice this can be reduced to as few as 500 judgements using adaptive algorithms (Pollitt, 2012) or random sampling (Suzuki, Yasui and Ojima 2010). Even so, several times more judgements are always required than there are scripts for the statistical modelling to work.<\/p>\n
Peer assessment<\/strong> The resource demands of comparative judgements can be addressed through peer assessment. Students can be set a task and then comparatively judge one another\u2019s work. This satisfies the requirement for a group of assessors rather than an individual assessor. Returning to the above example, if 100 students are allocated 20 judgements then a total of 2000 judgements will be recorded. This satisfies the requirement for several times more judgements than there are scripts.<\/p>\n A further attraction of peer-based comparative judgement is that it might encourage higher-order learning. When undertaking comparative judgement, students need to gauge the relative quality of peers\u2019 understanding and communication of the assessed construct. This can be particularly beneficial for nebulous constructs that lend themselves to open-ended, unstructured assessment tasks. Students, often used to structured test questions, can be unsettled when presented with an unstructured assessment. Viewing and judging how others attempted the same task can help build confidence and strategies for tackling future open-ended, unstructured tasks.<\/p>\n A recent example<\/strong> The responses to the question were anonymised and scanned, then uploaded to a secure comparative judgement website. Students aged 13 to 15 from three other schools then conducted comparative judgement on the responses. They were asked to decide, for each pairing, which student had the better understanding of fractions. Their decision were based on the ordering of the fractions and the quality of the written explanations. Once the judging was completed a scaled rank order of responses was constructed for each school. The three scaled rank orders were very similar and correlated strongly with one another, showing that the assessment was reliable.<\/p>\n We also recruited experts (teachers and mathematics education researchers) to assess the students\u2019 answers using comparative judgement. This enabled us to compare the rank orders produced by the students with those of the experts. We found that the student and expert rank orders correlated strongly and the students\u2019 collective interpretation of the better understanding of fractions was very similar to that of the experts.<\/p>\n Upcoming work<\/strong> We are also currently developing technological tools for implementing comparative judgement. Readers interested in using comparative judgement should contact Ian Jones at i.jones@lboro.ac.uk<\/a><\/p>\n
\nA potential limitation of the comparative judgement approach is that it can be more resource intensive than marking. It requires a group of assessors rather than just one. This is fine for large scale contexts, such as national school tests, that traditionally need many examiners. However, it is a barrier in small scale contexts such as a university lecturer or school teacher marking his or her own students\u2019 work.<\/p>\n
\nPeer assessment needs to be reliable and valid to be of practical use. We recently evaluated school students\u2019 performance when using comparative judgement to assess conceptual understanding. A mathematics teacher of 12-15 year olds provided us with 24 responses to the test question shown in the box. We wanted to find out whether students the same age could assess the responses using comparative judgement.<\/p>\n![\"Test](\"https:\/\/i0.wp.com\/newsletter.alt.ac.uk\/wp-content\/uploads\/2013\/07\/Jones-300x225.jpg?resize=300%2C225\" "\\"Jones\\"")
<\/a>
\nIn the coming months we will undertake a study to investigate the learning benefits of students undertaking peer assessment using comparative judgement. Students will sit a specially designed conceptual test similar to that outlined above. The students will then undertake pairwise judgements of their peers\u2019 responses before sitting the test again. We will compare the gains made from pre- to post-test with reference to a control group who will not undertake the peer assessment exercise. We anticipate that the learning gains will be statistically significant, and that children\u2019s conceptual development will be qualitatively evident in the content of the post-tests compared to the pre-tests.<\/p>\n