The results of the Progress in International Reading Literacy Study (PIRLS) and Programme for International Student Assessment (PISA) have stirred discussions in many nations about how well they are preparing students for life in an increasingly technological world in which ever-increasing levels of literacy are needed. The two surveys, which were both released in late 2007, offer insights on educational trends within individual nations and worldwide.
PIRLS, which is directed by the Trends in International Mathematics and Science Study (TIMSS) and PIRLS International Study Center at Boston College, surveyed the reading literacy of approximately 215,000 fourth-grade level students across 40 countries, including 26 countries that also participated in PIRLS 2001. In addition, PIRLS collected extensive information about home, school, and national influences on how well students learn to read, and queried parents and caregivers about their children’s early literacy activities.
Meanwhile, after focusing in 2000 on reading skills and in 2003 on mathematics, PISA 2006 tested 400,000 15-year-old students in 57 countries on science-related knowledge. PISA is administered by the Organisation for Economic Co-operation and Development (OECD), and it sampled students in 30 OECD nations and 27 other countries. Although the focus was on science, the assessment also included a reading section, and the survey also collected data on student, family, and institutional factors that can help to explain differences in performance.
That said, most news reports still focused on where various countries ranked, and government officials in many countries rushed to take credit (or assign blame) based on the results. The Russian Federation, Hong Kong Special Administrative Region of the People’s Republic of China (Hong Kong SAR), and Singapore were the three top-performing countries in reading literacy in the PIRLS survey, whereas South Korea, Finland, and Hong Kong-China ranked highest among 15-year-olds in the reading portion of PISA.
“This report contains a wealth of data countries can use to improve schooling in reading, while confirming the family as children’s first, and perhaps most important, reading teachers,” said PIRLS codirectors Ina V.S. Mullis and Michael O. Martin of Boston College.
The report also demonstrated that countries can make progress in improving children’s academic achievement. According to the report, most of the highest achieving countries in 2006 showed significant improvement since 2001, including the three top-performing countries. Among the countries with the largest gains, Hong Kong SAR and Singapore both enacted systemic reforms since 2001 in their reading curriculum, instructional materials, and teacher education. Two other countries that made significant gains, the Russian Federation and Slovenia, underwent structural changes, so that students now receive one more year of primary schooling.
Here are some of the key findings from the PIRLS study.
Gerry Shiel and William G. Brozo of IRA’s PISA/PIRLS Task Force, which is charged with monitoring and analyzing PIRLS and PISA results, commented on the PIRLS findings. They noted that the Russian Federation, Hong Kong SAR, and Singapore had made significant gains between 2001 and 2006, while the Netherlands, Sweden, England, and Romania had significantly lower average scores.
They noted that the United States had dropped in rank, despite the fact that its overall average reading score had not changed. The drop came because a number of other countries had made achievement gains.
Shiel and Brozo pointed out the positive relationship between students’ reading achievement in fourth grade and parents’ having engaged their children in early literacy activities before starting school. Parents in 14 countries reported increases in engagement in 2006 relative to 2001. The number of children’s books in homes was also associated with achievement. Children in homes with the greatest number of books (more than 100) did much better than those in homes with few books (0-10).
The PISA survey focused on scientific literacy, but there was also a reading component. Here are some of the interesting findings:
Andreas Schleicher, head of the Indicators and Analysis Division of the OECD Directorate for Education, offered some thoughts regarding the PISA findings in an interview with Reading Today. He noted that globally there hasn’t been much progress, despite quite significant increases in expenditures in education. However, there are some notable increases, such as Poland, which raised achievement levels by about three-fourths of a school year since 2000. They were able to accomplish this, he said, “by remarkably reducing the proportion of poor performers.” South Korea, on the other hand, increased its overall score primarily by raising the performance of the students at the top of the scale.
These examples, said Schleicher, “show that over short periods of time, great progress can be achieved.” He noted that both Poland and South Korea have made changes in their education focus that contributed to these results.
Commenting on the PISA survey, OECD Secretary-General Angel Gurria said that it is a tool to assist governments in their policy choices on education. “In today’s competitive global economy, quality education is one of the most valuable assets that a society and an individual can have,” he said in a PISA news release.
In that context, he added, “PISA is much more than just a ranking. It is about how well individual education systems are equipping their young people for the world of tomorrow. First and foremost, it tells countries where their strengths and weaknesses lie.”
Reading Today invited comment on the PISA/PIRLS results from several prominent researchers. We received the following responses (comments have been condensed in some places because of space constraints).
Keith Topping, professor of educational and social research, School of Education, Social Work and Community Education, University of Dundee, Scotland, offered the following thoughts about the PIRLS results.
At first sight, the PIRLS results for 2006 for England and Scotland seem disappointing. England fell from 3rd to 19th. Scotland fell from 14th to 26th. Closer inspection reveals that Scotland’s score stayed much the same from 2001 to 2006 (528 to 527), while England’s score declined (553 to 539). Both countries remain above the OECD average, but not by much.
Because of increased participation in PIRLS 2006, a country’s ranking could be considerably lower despite no significant difference in its scale scores in the two studies. Scotland, New Zealand, and the United States fall into this category.
In 2001, only 12 countries had a mean score significantly higher than Scotland, compared with 19 in 2006. However, five of these 19 countries were new to the study in 2006. The three top-scoring countries in 2006 (Russian Federation, Hong Kong SAR, and Singapore) were ranked equally with Scotland in 2001 but have since overtaken all other countries in the survey. The remaining 11 countries were already above Scotland in 2001, but their relative positions have changed.
A further complication is that in Scotland children were tested in Primary 5 (mean age 9.9 years) and in England in Year 5 (which is one year ahead of P5) at a mean age of 10.3 years. Obviously, this discrepancy would make a difference. The PIRLS reports mistakenly assert that both countries were tested at the same grade level.
The Scottish government has taken a low-key approach to these findings, expressing concern mainly about the wide gap between high and low achievers in Scotland (although this has not changed since 2001). The English government has expressed concern but tried to blame the decline on parents, citing results about the increase in time spent on computers at home (although the gap between high and low achievers is wider in England than in Scotland). Opposition parties have, of course, pointed out that the English government’s own school policies might be to blame.
The National Foundation for Educational Research (which conducted the English and Scottish testing) has been alternatively analyzing the English data (of course finding more favorable results). This analysis has not been done for Scotland.
In regard to PISA 2006, the United Kingdom was entered as a single entity. Fifteen-year-olds in the United Kingdom ranked slightly above average on reading among the OECD countries (13th among 30), a comparable result to the PIRLS outcome. There are no comparable data available for the United Kingdom from PISA 2003 because of a low response rate.
Allan Luke, professor, Faculty of Education, Queensland University of Technology, had the following comments. The PISA data have proven useful in Australian and international debates around curriculum reform in literacy, despite the near universal tendency of the media and political officials to report and misreport data as it suits. While other countries’ reporting of the data and the OECD’s various discussion papers consistently report that Australian literacy results are in the top tier of postindustrial countries, the national press insists that there is a “crisis” and misuses PISA data to support calls for more testing, “back to the basics,” parental choice, funding of private schools, and a range of other reforms. It is an object lesson in the use and abuse of testing data. There are several caveats and challenges that PISA raises.
Yes, PISA is large-scale standardised testing, with all of the inherent technical limits of such systems. But the technical quality of the assessment instruments is high, and the international development processes have been consultative and generative. For example, the writing assessment tasks and procedures are well in advance of many of the very narrow, multiple-choice, skill-based tests currently used in many countries.
There is always a danger of the misuse of PISA and PIRLS-style results-that PISA domain specifications, however generic, risk becoming a de facto transnational curriculum, with some European Union states attempting to realign their national curricula to match PISA. The success or failure of a national or regional educational system cannot be judged principally on its capacity to generate better standings on international league-tables-there is so much more to the goals and outcomes of a nation’s education and schooling than this.
This said, the analysis of the PISA results provide three very interesting insights on the impacts of curriculum reform on equity.
This has given us some grounds for state-level discussions of the characteristics of “high quality/high equity” systems. It is worth noting that these systems typically eschew reform via high-stakes testing. As Andreas Schleicher of OECD argues, these systems have achieved a crucial and delicate balance between “informed prescription” and “informed professionalism” in which teachers are well supported to engage in local curriculum development, interpretation of standards, and school and classroom-based assessment. Readers should consult the early 2005 PISA reports on equity available from the OECD.
This is what the school reform and teacher professionalism literature has been telling us for some time, and it is reiterated in various recent studies of the effects of NCLB, which found that test-driven “hard prescription” with strict accountabilities runs the risk of deskilling teachers, narrowing the curriculum, and discouraging local curriculum making and innovation, with potentially deleterious effects on the equity gap. For the U.S. audience, then, PISA also provides some data that several systems that have gone down the current neoliberal policy route of increased high-stakes testing, marketized schools, scripted and commodified pedagogy, and heavy accountability sanctions and pressures continue to have widening equity gaps-with steep regression slopes indicating that the replicating effects of social class are locked in early.
From the perspective of government policy-making, we now have ample evidence that such reforms are extremely blunt policy instruments, with significant unanticipated and collateral effects on a broad range of schooling processes and practices, and that they are unlikely to close the equity gap. Finally, PISA shows us yet again that we can get very sound systemic data for guiding reform through sample testing, as NAEP does-as opposed to the high-stakes census testing approach of NCLB.
So for those of us committed to redistributive social justice and universal access to literacy, PISA has provided us with an important “talking stick” around equity that goes beyond the very crude, technically naïve, and politically motivated approaches to testing, reporting, and policy formation currently used in many states.
Karen Wixson, professor of education, University of Michigan, notes that although the U.S. reading scores for PISA were invalidated, there is at least one aspect of the results in mathematics and science that would likely have applied to reading as well. This is the finding, as reported in Education Week (December 7, 2007), that poverty had a greater effect on U.S. scores than it did for other nations. The bottom line here is that the PISA data indicate that the United States does not do as good a job as other countries of compensating for nonschool achievement factors.
This is a finding that U.S. researchers such as John Fantuzzo at the University of Pennsylvania have been trying to address in spite of the overwhelming belief in the United States that the source of persistent achievement gaps is entirely within the schools. Although there is always room for improvement in schools, especially in areas with high concentrations of low-income students, school improvement by itself is not likely to bring about the desired results.
An aspect of PIRLS that interests me is the comparison in performance between the literary and informational subscales. Although the differences between the 2001 and 2006 PIRLS results on these subscales are not statistically significant, there is sufficient difference to suggest a possible trend. In 2001, the United States had the largest difference between scores on these subscales (17 points), and out of 35 countries ranked 4th on the literary subscale and 13th on the informational subscale. In 2006, the United States had only a 3-point difference in performance between these subscales. The smaller difference comes from a decrease on the literary subscale and an increase on the informational subscale.
Although the gap between literary and informational reading is narrowing, the difference is still statistically significant. Of concern, however, is the downward trend in literary reading.
On the good news front, for each of the four international benchmarks for PIRLS performance (low, intermediate, high, and advanced), the percentage of U.S. students that reached the benchmark was higher than the international median percentage. Importantly, the percentage of U.S. students achieving at least the low international benchmark showed a statistically significant increase between 2001 and 2006 from 94% to 96%. Students at this level were able to recognize, locate, and reproduce explicitly stated details and make straightforward inferences from literary and informational texts.
These results confirm what most reading experts have known for some time. Generally speaking, U.S. students are not lacking in basic reading skills.
PISA, PIRLS spotlight global trends. (February 2008). Reading Today, 25(4), 1, 4, 5.
Journals
Reading Today