Just as the nation was absorbing Education Secretary Richard W. Riley’s glum August news of the widening achievement gap between white and minority students, academic researchers were delivering much more encouraging news about narrowing that gap by as much as one-third, based on one-year and two-year results from school voucher intervention programs in four different cities.
Although the researchers warned “it remains to be seen” whether the gains of black students continue to increase over time, the results are significant, since the gains reported are larger than those from either class-size reduction studies or rigorous statewide testing programs.
According to a three-city research project led by Harvard University’s Program on Education Policy and Governance, African-American students who received vouchers to transfer from a public school to a private school in New York City, the District of Columbia, and Dayton, Ohio, scored an average of 6.3 percentile points higher on the Iowa Test of Basic Skills in reading and mathematics after two years in the program, compared to students who applied for vouchers but stayed in the public schools.
Gains of 6.5 percentile points in reading and 5.9 points in math achieved by a similar program in Charlotte, North Carolina, were documented in a report from the Manhattan Institute for Policy Research.
The voucher programs involved in these studies are not publicly funded measures, but privately funded partial tuition scholarship initiatives for low-income, largely minority families. Recent programs established under the umbrella of the Children’s Scholarship Fund have become a rich environment for research on vouchers.
The research study on Dayton, New York City, and Washington, DC was conducted by William G. Howell of the University of Wisconsin, Patrick J. Wolf of Georgetown University, and Paul E. Peterson and David E. Campbell, both of Harvard University.
Although the voucher initiatives involve all ethnic groups, the gains in all four cities appear to have occurred only among the African-American students. Senior Manhattan Institute Fellow Jay P. Greene, who conducted the Charlotte study, notes this is consistent with the widespread impression that these students may be particularly poorly served by public schools.
“Whatever the reasons for the gains, [these] studies of school choice programs make clear that choice has significant academic benefits, particularly for African-American students,” said Greene. “Given the frustration with previous efforts to increase African-American student achievement, it is time that we consider new approaches for addressing persistent problems.”
Questionable Attack
Opponents of vouchers weren’t ready to concede either of Greene’s points and quickly launched a campaign to discredit the results, the quality of the research, and the integrity of the researchers. “Biased,” “incomplete,” “overstated,” and “inconsistent” were some of the pejorative terms they used to describe the studies.
One of the most questionable attempts to discredit the research findings came from The New York Times after a researcher from Mathematica Policy Research–which conducted the research for the Harvard University team–issued a statement saying the claims of success in the New York City experiment were “premature.” Looking at the results by grade level and ethnic group, Mathematica researcher David Myers said there was no difference between students who were offered scholarships and those who weren’t. Since the gains appeared concentrated in sixth-graders, he urged caution.
Myers later agreed there was nothing in the Harvard University study that either overstated or misrepresented the New York City findings. But in the meantime, The New York Times transformed Myers’ caution into a feature article by Kate Zernike warning “New Doubt Is Cast on Study That Backs Voucher Efforts.”
Voucher opponents quickly used the broad outlines of Zernike’s story in “the newspaper of record” to pour discredit not just on the results from New York City but on the results from other cities, too. But what Zernike reported as a breathless scoop–no difference in test scores!–was simply a statement of the obvious for statisticians: that is, smaller sample sizes necessarily mean less confidence in the findings. That is why the researchers did not base their conclusions on findings from breakout-level data where differences had no statistical significance.
To explain the researchers’ position, Greene suggested this analogy. Consider if The New York Times had reported a national opinion poll of 1,000 respondents in the following way: On one day, the newspaper reports the poll’s national results showing that Al Gore is 6 points ahead of George W. Bush; but then the next day the newspaper cautions its readers that it is premature to conclude that Gore has a 6-point lead because, when the same poll is reported by state, Gore’s lead over Bush becomes statistically insignificant and is concentrated in just a few states.
The Harvard-Wisconsin-Georgetown researchers responded to some of the other criticisms by gently reminding the critics to go back and read the report. In other cases, they pointed out that what the critics claimed the report had said was “simply wrong.” Several other criticisms came from apparent misunderstandings about how the research was conducted.
Significant Findings
In all of the studies, the evaluations were designed as randomized field trials, a technique Caroline Hoxby of Harvard University has called the “gold standard” of social science research. This technique, which was used in the widely touted Tennessee STAR class-size study, is most often associated with the testing of new drugs and medical treatments.
Students for the voucher programs were selected from the pool of applicants by lottery, allowing researchers to compare the performance of students who received a voucher with that of similar students who also applied for a voucher but didn’t get one.
On test-score performance, the Harvard-Wisconsin-Georgetown researchers found no significant difference between non-African-American voucher students and non-African-American students in the control group for all three cities. However, they found “moderately large” effects for African-American students.
Taking all three cities together, the researchers found that African-American students who switched from public to private schools scored 6.3 percentile points higher on the reading test and 6.2 points higher on the math test than did African-American students in the control group. These differences were statistically significant.
At the city level, the differences in test scores for voucher students versus control group students showed more variability, but all effects were consistently positive and all were statistically significant except for math scores in Dayton. The percentile point gain in reading and math scores for each city were, respectively, New York City (4.5, 4.1), Dayton (7.6, 5.3), and Washington, DC (8.1, 9.9).
“When similar results emerge from evaluations of school voucher programs in three cities in different parts of the United States, they provide a stronger basis for drawing conclusions and generalizing to a larger context,” note the authors of the three-city study. “Thus, the average impact across the three sites may provide a reasonable estimate of the likely impact of a school voucher initiative elsewhere.”
Beats Other Reforms
The researchers point out that a difference of 6.3 percentile points in overall test performance is 0.33 standard deviations, generally regarded to be a “moderately large effect.” By comparison, test scores for African-Americans improved only 4.9 percentile points, or 0.21 standard deviations, in the much-promoted randomized field trial of class-size reduction in Tennessee. Two-year gains of 0.12 to 0.14 standard deviations from rigorous statewide testing programs were reported as “remarkable” in the recent RAND study, Improving School Achievement.
“The effects of vouchers after two years, as observed here, are over twice as large,” comment the researchers.
George A. Clowes is managing editor of School Reform News.
For more information . . .
The research reports from Harvard University’s Program on Education Policy and Governance are available at the PEPG Web site at http://data.fas.harvard.edu/pepg. The site also includes the researchers’ responses to criticisms raised by voucher opponents. Other responses to critics are available at the Web site of the Black Alliance for Educational Options at http://www.schoolchoiceinfo.org.
The research report on the Charlotte Children’s Scholarship Fund Program is available at the Website of the Manhattan Institute for Policy Research at http://www.manhattan-institute.org.
George Clowes is former Heartland senior fellow addressing education policy. He served as founding managing editor of School Reform News between November 1996 and January 2005.
