Problems with PISA: Why Canadians should be skeptical of the global test
This article is republished with permission of the authors and the journal which you can read by clicking here
The Program for International Student Assessment (PISA) — the Organization for Economic Co-operation and Development’s (OECD) global standardized test of student achievement — is frequently used by commentators to compare and rank national or provincial education systems.
PISA, which has now spread into 80 countries as a best education practice, presents itself as a tool to help countries make their systems more inclusive leading to equitable outcomes. But PISA is far more ambiguous and controversial.
Many academics and educators critique PISA as an economic measurement, not an educational one. The media generally use PISA results to blame and shame school systems. And the way that some politicians, policy-makers and researchers have used PISA is more closely aligned to a political process than an educational one.
There are many reasons to be skeptical about PISA rankings and how they’re used to compare student achievement or to identify best practices or solutions for educational problems.
A narrow measurement
PISA numbers are limited in what they can explain and the conclusions they can support.
PISA measures math, science and reading skills, not more holistic educational goals or understanding of literacy as defined by the United Nations Educational, Scientific and Cultural Organization (UNESCO). PISA is a narrow measure of educational achievement.
PISA typically samples 5,000 to 10,000 15-year-old students from about 500 randomly selected schools in each country every three years. Up to 53 students are randomly selected in each of these schools.
In small countries with fewer than 5,000 students, all 15-year-old students are sampled. The two-hour test relies heavily on multiple choice and rating scale questions.
PISA ignores the importance of engagement and positive attitudes to learning for future success.
Relation to Indigenous schools, special needs students
In Canada, federally funded Indigenous schools are not considered in PISA data and would likely impact provincial scores. The absence of Indigenous schools may help mask disparities in educational equity in Canada and marginalize the significance of chronic under-funding of Indigenous-controlled education and basic infrastructure necessary to Indigenous child well-being.
Differing levels of how special needs students are integrated in distinct jurisdictions also confound PISA results.
PISA tests a sample of students and the results are then adjusted to reflect a whole population of 15-year-old students. The scores therefore include a measure of statistical uncertainty and PISA can only report a range of positions (upper rank and lower rank) where a country can be placed.
There is a co-relation between poverty and lower test results: the OECD notes that up to 46 per cent of the differences in PISA mathematics scores among OECD countries can be explained by socio-economic disadvantage.
Psychometric experts, who examine the fitness and effects of particular methodological choices and the validity and reliability of modelling and calculations, criticize PISA for downplaying the problematic nature of its calculations, and its lack of transparency in reporting uncertainties.
The majority of nations fall in the middle PISA rankings. However, small differences in mean scores can shift rankings by 10 to 20 places.
Relying on a small number of questions also means scores are highly affected by completion rates. In some jurisdictions, higher scores may result from greater significance being placed on PISA completion by parents and school authorities.
Pressure to narrow curricula
As an instrument of international comparison, PISA has created pressure for states to narrow curricula, relegating subjects such as the arts and social studies to second-class status, and to introduce testing cultures to monitor performance and achievement
Testing culture and curriculum narrowing have been linked to students dropping out, students and teachers cheating, students undergoing stress and anxiety disorders, teachers leaving the profession, a fear of failing and a dislike of schools and learning.
Overlooking inquiry-based learning
As school systems narrow curricula to focus on testable concepts, students may reach high levels of proficiency in a few subjects but lose out on programs of study based on active, inquiry-based processes and content.
Countries with the highest PISA scores appear to have the lowest levels of inquiry-based learning. High levels of inquiry-based science appear to have a negative association with PISA science scores. Focusing on PISA may increase skill levels but cause students to miss out on learning that generates higher-order thinking.
Even though PISA use is spreading globally, and is translated into national languages, it is still framed by Western understandings and may distort results from students with diverse social and cultural histories. This will become more of a challenge as PISA’s ambition is to move beyond testing skills to assess attitudes that promote student success.
Corporate partnership in the age of digital surveillance
Finally, journalists and researchers have expressed concerns about PISA’s partnership with Pearson, a global educational business enterprise that boasts it operates more than 70 countries worldwide, reaching 100 million people.
The partnership is a worrisome conflict of interest. PISA assesses and ranks school systems and Pearson is a provider of global and onlinecharter schools, tests and education consulting. Additionally, in an age of rising concern about digital surveillance and data privacy, it is reasonable to ask how data on students and teachers that’s collected globally may be used, and to what end.
What needs to change
Other democratically governed organizations such as UNESCO should play a larger role in the collection of international educational data. This would ensure students’ and teachers’ rights are protected and children have the vibrant and democratic educational structures, processes and relationships they are entitled to.
PISA has shifted education discussion globally in alarming ways. The OECD needs to listen to critical voices and rethink its PISA strategy and framework.
Sarfaroz Niyozov is an Associate Professor at OISE, University of Toronto. He received funding from SSHRC in 2009.
Wendy Hughes is an EdD student at OISE, University of Toronto. She does not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond this academic appointment.