Imagine a daily scene at your typical sporting facility designed to meet the needs of athletes across many different sports. Each day three particular athletes arrive for their routine training session. Each athlete is committed to success and trains hard. They have recruited expert coaches and seek guidance from specialists in sports medicine to ensure that their daily training schedule and overall healthcare plan aligns with their goals. They are equally passionate about their sports and this is plain for all to see in their approach to training.
Alex is into athletics and in particular is training in the hope of making the Olympics. Alex likes short to middle distance events and is focussing currently on training for the challenging 400 metre event. Alex spends time working on starts, running corners and accelerating hard over the last 50 metres to ensure the line is crossed at full speed. Alex’s coach has set up a training routine that ensures every muscle in Alex’s body is finely tuned to this task.
Bobbie is into Football (Soccer) and considers it to be the one true sport. Bobbie trains hard and is working on endurance and speed as areas of growth. Bobbie also spends time on drills to enhance ball control and passing. Bobbie hopes that all this effort will ensure a full game can be played with fitness left for the final ten minutes of play. Bobbie knows that improvements in strike rate come from a combination of skill with the ball, speed in open play and endurance.
Charlie is into weightlifting and has a training regimen designed to increase power and build muscle. Charlie is keen to find the limits of human strength and is continually increasing the weights lifted. Charlie spends long hours hoisting steel into the air and likes to feel the burn that comes from pushing muscles to new limits. Charlie’s coach works hard to ensure that the limits are not exceeded too rapidly and builds rest and recovery time into the schedule. The untrained onlooker could easily overlook the science behind Charlie’s training schedule.
After many weeks the three athletes are asked to participate in an assessment of their training methods. The researcher wants to find out which training method works best and will use this to inform coaches nationally of the method with the greatest research based effect size. The assessment is designed to be easy to administer and provide clear quantitative data. A few days later, Alex, Bobbie and Charlie are lined up on the starting line for a quick run around an oval, the time each records will reveal which athlete has chosen an effective training regime and who has been wasting their time.
With the assessment over each athlete is given their score. Alex is very happy. Bobbie is pleased. Charlie is devastated, clearly all that effort has been for nought.
In this example, the folly of the scenario is plain to see. Alex’s preparation was ideally suited to the assessment and the result achieved was as might have been expected. Bobbie’s training schedule was also sound preparation for the assessment; although the time spent on drills is probably the cause of a result that was less than what Alex achieved. Charlie’s preparation probably hindered performance and although it has built a powerful body, it is clearly a slow moving one. But in this situation the assessment is clearly not aligned with each athletes differing purposes. It suited Alex very well. It failed to reveal that Bobbie still has limited ball control and it gave no indication of Charlie’s immense strength.
The trouble is that we do this sort of thing all of the time. We rely on an assessment measure without taking a close look at what it is measuring and we obfuscate the information we need to evaluate the utility of these measures by reducing the results to numerical values.
Take as an example an assessment of learning in the sciences. One set of students participate in a classroom where direct instruction is the norm. They are presented with detailed information over a number of weeks. They transcribe information from texts and teacher instruction into their books, they study their notes at home and in small groups. They take mock exams and revise information that they missed. In another classroom a group of students is approaching similar content through an inquiry process. They understand that their goal is to master the process of scientific inquiry. They ask lots of questions, design experiments to test their hypotheses and share their results. Their teacher guides them through the inquiry process, models how a scientist works and provides feedback on the methods they are using. Both methods have their place in learning, both have value.
The complication comes when the student’s learning is assessed. If the test is based on recall of knowledge then the first group is more likely to succeed. The method of instruction they experienced is aimed at achieving the type of learning that the assessment measures. If the assessment requires students to design an experiment that will test a hypothesis, the second group has a clear advantage as their learning has prepared them for this exact task. The problem becomes significant when the assessment is used to endorse claims that one teaching method is more effective than another without any reference to what was assessed.
It is easy to point the finger at standardised assessments but the problem is more expansive than that. We all have a certain concept of what a summative assessment should look like and of the sort of items that belong within one. The biases that stem from our experience with tests and from the relative ease with which we assess recall of facts compared to creativity, critical thinking and problem finding/solving result in assessments that overvalue particular modes of teaching.
We need to consider the purposes of our teaching, the goals that we hope to achieve with and for our learners and in each instance, what success might look like. Only when we ensure that the assessments we value align with the learning we hope to engender, will we begin to have an accurate perception of what works and what does not. Until then Alex will be very happy, Bobbie will be blindly pleased and Charlie will be mightily devastated.
by Nigel Coutts