Credibility of Indiana U. Voucher Study Questioned

Published May 1, 1998

Although a new evaluation of the Cleveland Scholarship and Tutoring Pilot Program shows that 94 out of 3,000 voucher students score no better than students in public schools, a researcher is questioning the credibility of the base test scores used by the study, which show all Cleveland public school second-graders performing at the national average–a remarkable achievement for poor, minority, inner-city students. If true, such achievement levels would suggest that the Cleveland Public Schools are not failing at all.

Yet it was to address the failure of the city’s public schools that the voucher program was established two years ago, providing 2,000 K-3 students from low-income families with tax-funded scholarships of up to $2,250 to attend private or religious schools. The program now serves 3,000 students in grades K-4, including 450 students at two new HOPE schools established by industrialist David Brennan in just 15 days to accommodate 360 initial scholarship students who could not find places at 57 existing private schools.

The HOPE Academies were unique in that they tested their students at the beginning of the school year. With this initial measure of academic performance available, first-year achievement gains were analyzed last summer by researchers Paul E. Peterson of Harvard University, Jay P. Greene of the University of Texas at Austin, and William Howell of Stanford University. They found that students in grades K-3 were an average of 5.4 percentile points higher on the reading test and 15.0 percentile points higher on the math concepts test. (See “Cleveland Choice Program Shows Student Gains,” School Reform News, September 1997.)

The new study, commissioned by the State of Ohio and conducted by researchers at Indiana University, specifically excludes the HOPE students and focuses instead on a tiny group of third-graders who had transferred from the Cleveland Public Schools at the end of second grade. The new study concludes that there is no statistically significant difference between the scores of the selected third-grade voucher students and a control group of public school students.

That’s hardly a surprising result, says Greene, because sampling “fewer students makes it harder statistically to find any significant effects, positive or negative.” The Indiana University researchers could have increased their sample size by including 31 HOPE students through a standard conversion of their test scores, but they chose not to.

Even without the HOPE students, Greene points out that the unadjusted scores of the voucher students are better than those of the public school control group. And even after making adjustments for earlier scores the students received in the Cleveland Public Schools, the scholarship students still outperformed the public schools students in all but one subject area–but none of the differences was statistically significant.

“The Indiana University conclusion depends entirely on adjustments made for the scores that students received in second grade when they were in the Cleveland Public Schools,” explains Greene. “The problem is that these second-grade results simply are not credible. It’s hard to believe that Cleveland Public School students were performing at the national average in second grade, given that central city, poor, minority students tend to perform less well on average.”

Since the second-grade results lack credibility, notes Greene, the conclusion of the study also lacks credibility.

George A. Clowes is managing editor of School Reform News. His email address is [email protected].