I am wanting to take a slightly different approach to this weeks post. The past week has seen the latest round of PISA results and the media has had a field day. Headlines have routinely attacked students, educators and education systems in equal measure. The Canberra Times reported that “Australian school scores plummet on world stage”, the Sydney Morning Herald led with "Alarm bells': Australian students record worst result in global tests” and The Weekend Australian went with "PISA global educational rankings: Schools fail on maths, science”.
Everyone seems to be an expert when it comes to education. The singular criteria appears to be any sort of experience in a school. If in the past century you have been a school student you qualify to comment on the complexity of modern education systems and their even more complex place within society. It is well noted that the same logic is not applied to medicine, where having been in a hospital does not certify one to offer medical advice or to aviation where frequent flyer miles do not qualify one as a pilot. It may also be noted that although most people have read a newspaper, we do not imagine ourselves to be journalists even if one might be compelled to question where journalistic integrity lives in the advertising driven market of todays media conglomerates. It is nevertheless, disappointing that the highly skilled task of educating young people to thrive in a constantly changing world, is not given the level of respect that it deserves.
PISA is the OECD's Programme for International Student Assessment. The OECD is the Organisation for Economic Co-operation and Development. The stated goal of the OECD is "to shape policies that foster prosperity, equality, opportunity and well-being for all”, but we should be clear that these goals are driven by economic imperatives and development or growth is seen as of inherent value. The OECD is not free of agendas and aims to shape global policy in alignment with a central vision for how the world might be. It does not exist purely to document the nature of global policy or in the case of PISA to reveal what students have or have not learned. If you are truly interested in the findings of the latest round of PISA results you can go directly to the source and avoid the media hype. The results are online here.
Regardless of how you engage with the PISA results, if you are genuinely interested in what they might mean, here are some questions I hope you might ask about them:
What motivates the OECD to conduct the PISA testing? - What purposes does PISA testing serve for the OECD and its agenda?
Who does PISA serve? Who is the programme aimed at? What needs within this target audience does PISA serve?
What does PISA aim to measure? Why does it measure these things and not others?
Does PISA measure things which matter?
Can we trust PISA to measure what it claims to measure? How do we know this? How does PISA compare to other similar assessments?
Is PISA valid across systems? How do we know if it is or isn’t?
Is PISA valid for comparisons over time? Are changes in scores from one assessment round due to changes in education systems or changes in the assessment? How do we know?
How granular is the data? At what point do differences in scores between nations or across time indicate a significant difference?
If we answer the questions above and still feel that the PISA data is a valuable measure of our education system’s progress, here are some further questions to ask:
If there is a change, what has caused the change and how do we know it was this and not something else?
If nothing has changed in a nations education system from one round of testing to the next, what has changed to produce different results? Might the change in results indicate a change in other parts of society?
Do the PISA results indicate something about all parts of the education system? Are there parts where a different story is told? Why might this be?
If some parts of a national education system show growth or even stable results, while others show negative growth, why might this be?
What do the results reveal about equity issues in a society? Are all people able to achieve in an equitable way? If not why not and what is being done about this?
Why is the distribution of scores across a nation, or across parts of an education system not discussed? What does the relative absence of debate about this distribution reveal about national priorities?
What is the long term plan? What commitment are our politicians making to drive long term growth based on sound educational practices and knowledge?
Why is the voice of educators largely missing from and ignored when there is discussion of PISA? What drives and shapes the reporting of PISA results?
National education systems are inherently complex and they sit within the greater complexity of a nation's society, culture and history. They are shaped by and shapers of culture and national imperatives at all levels from macro to micro. PISA is a single measure of something that is very complex. To trust that this single score provides sufficient information to establish policy is surely foolhardy. We must ask hard questions about PISA before we trust that it measures what it claims to and our media should do the same.
By Nigel Coutts