Indiana U. Voucher Analysis Labeled “Faulty”

Published June 1, 1998

Contradicting the findings of an analysis conducted by Indiana University’s School of Education, Harvard University researchers conclude from a re-analysis of the data that, in fact, students in Cleveland’s choice schools show academic gains in all subject areas.

The Harvard researchers raise concerns that conclusions based on “faulty analysis–such as Indiana’s” may lead to far-reaching public policy decisions affecting the lives of children in inner-city public schools.

When a report last summer from Harvard University’s Program on Education Policy and Governance (PEPG) showed significant gains in reading and math for 263 students who participated in the Cleveland pilot voucher program at two new Hope schools, teacher unions challenged the findings, claiming that the gains would fade over the summer break. Although the claim was refuted when students were re-tested at the start of the 1997 school year, a new study from Indiana University showed no test-score gains for 94 third-grade voucher students.

Like the teacher union’s challenge, Indiana University’s claim also does not withstand scrutiny, according to Paul E. Peterson, director of Harvard’s PEPG program, and program associates Jay P. Greene and William G. Howell. In a May 6 paper detailing new findings from a re-analysis of the data, they report that “the Indiana University evaluation suffers from several limitations, including the use of implausible second-grade test scores supplied by the Cleveland Public Schools.”

Correcting the deficiencies and removing the implausible scores profoundly affects the results of the analysis: Instead of being indistinguishable from students in the Cleveland Public Schools, choice student show gains in all subjects. Students in the voucher program score 4.7 percentile points higher in science, 4.1 points higher in language, 2.5 points higher in reading, 2.5 points higher in social studies, and 0.6 points higher in math. All except math are statistically significant.

“Even the most conservative estimate of choice-school effects observed in Cleveland are comparable to those observed in Milwaukee after one year,” note the Harvard authors.

According to the Harvard researchers, the Indiana University study has four major weaknesses: only third-grade students were evaluated; implausible second-grade scores from the Cleveland Public Schools were used; students from the Hope schools were excluded; and an inappropriate statistical analysis procedure was employed. In addition, the selected comparison group had “a more advantageous learning environment than the one available to students in the typical Cleveland public school.”

While such disagreements may appear strictly academic, Peterson and Greene point out that policy decisions based on such studies “have a tremendous effect on the lives of those trapped in America’s failing inner-city public schools.”

“Drawing a conclusion based on faulty analysis–such as Indiana’s–can make the difference between empowering poor parents to exercise control over their children’s education and killing that opportunity,” they note.

George A. Clowes is managing editor of School Reform News. His email address is [email protected].