New Directions in Research
In the Press to Scale up, What Is at Risk?

Anne McGill-Franzen, University of Tennessee, Knoxville, USA

About the author

In the commentary that follows, I identify what has been left out of the discussion of scale up. I am particularly concerned that the rhetoric of scaling up, and concomitant assertion of scientific underpinnings for programs brought to scale, may overstate the relation of the developers' programs and program materials to children's learning and undervalue the contribution of teachers' adaptive implementation. Further, an overreliance on standardized test scores as proxy for comprehension and an underspecification of the contribution of various teachers may create faulty contextual associations with learning. I argue for the funding of close-up, focused study of teacher variation along dimensions that count, and for a more elaborated view of the context of comprehension instruction and assessment.

At risk: Independent program evaluation

As a researcher of literacy teaching and learning, particularly in low-income settings, I find I often must attend to the larger state and federal polices within which classroom practice is situated. At the present time, given the mantra of “scientifically based reading instruction” among state and federal policymakers in the United States, I am incredulous that administrators and teachers in low-socioeconomic-status schools are forced to buy one of a dozen or so “core reading programs,” all of which are poorly validated for the target population and none of which have demonstrated effectiveness with children most at risk—presumably those children who are the intended beneficiaries of Reading First. Poor children, particularly low-achieving poor children, and their teachers are thrust into “forced choice” standardized curricula that, at worst, may limit opportunities to achieve at grade level. This situation represents an inappropriate “scale up” of leviathan proportions that has not been adequately studied or evaluated.

As a case in point, the state of Florida identified six reading programs that were deemed to be “research-based” and, as such, eligible for adoption by Reading First schools. A statewide survey of principals with a 58% response rate yielded data appropriate for statistical analyses (1,052 out of 1,835 schools) on the relation of program to performance. A simple ANOVA with poverty as covariate, conducted at the University of Florida, revealed that more than 20% of third graders in the most effective core program failed the state assessment (and third grade), and more than 30% of students in the least effective program failed (McGill-Franzen, Zmach, Solic, & Love-Zeig, in press). The average failure rate for all third graders across core programs was approximately 25%. After a second year in third grade, and the same core program, over 40% of these students failed the state test, and failed third grade again! Failure rates on the test were relatively stable across the years prior to the policy mandate and subsequent to it, even though a quarter of the students were repeating the same grade in the second year of implementation (C. Zmach, personal communication, February 3, 2005). Surely, in pursuit of social justice, a policy that mandates (or is interpreted at state levels to mandate) the purchase of a core program with a demonstrated failure rate of one quarter of the children it is designed to serve should be subjected to empirical and ethical evaluation.

This is not to imply that core programs or commercial interventions are never beneficial for children, nor educative for teachers. Reading First schools are needy schools and, in my view, are often staffed by the least experienced teachers and are often without stable leadership or a coherent literacy curriculum. Core programs may help guide the planning of novice teachers, who have reported feeling “lost at sea” in an environment of little support, overwhelming need, and pressing accountability (Kauffman, Johnson, Kardos, Liu, & Peske, 2002). On the other hand, comprehensive, tightly structured core programs may “shackle” the professional growth of beginning teachers, as suggested by Valencia, Place, Martin, and Grossman (in press). In reality, we know virtually nothing about actual program implementation, the contexts that diminish or support the feasibility of putting core programs into place, or what is lost—what economists call “opportunity cost”—in the replacement of, for example, a homegrown curriculum with one off the shelf. There is much to learn from rigorous evaluations, and major funding for field-initiated evaluation of education innovations has been available for some time through the Institute of Education Sciences (IES). But to evaluate a core reading program against no curriculum at all is simply to compare something to nothing—to paraphrase Slavin's (1991) comments in an evaluation study comparing IBM's Writing to Read program with no curriculum in kindergarten—and there will be an effect size.