An analysis of National Assessment of Educational Progress (NAEP) test results, released this summer by RAND researchers, touted optimistic conclusions about the effectiveness of several popular public school reform strategies. But testing experts familiar with NAEP protocols question whether such conclusions can be supported by the thin trend lines produced from NAEP’s limited and fragmented data set.
“Our results certainly challenge the traditional view of public education as ‘unreformable,'” asserted David W. Grissmer, the lead researcher for RAND, a think tank based in Santa Monica, California. “But the achievement of disadvantaged students is still substantially affected by inadequate resources. Stronger federal compensatory programs are required to address this inequity.”
Grissmer’s conclusion has profound implications for public policy. The federal government has spent approximately $130 billion on compensatory education for disadvantaged children since passage of the Elementary and Secondary Education Act in 1965, with studies to date showing little or no gain in achievement as a result. But by ranking 44 states by their NAEP data and attempting to compare students from similar family backgrounds, RAND concluded that at-risk students in relatively high-achieving states benefitted from such factors as lower pupil-teacher ratios and higher participation in prekindergarten programs.
The RAND study, titled “Improving Student Achievement: What NAEP Scores Tell Us,” purported to look at a representative sample of NAEP mathematics tests administered in 44 participating states in 1990, 1992, 1994, and 1996. Tests of reading were given in just two years, so reading results were not analyzed by the RAND researchers.
However, only 28 states took part in all the NAEP math tests, and some took as few as two. Idaho, Ohio, and Oklahoma–which RAND counted among the 44–have not participated in NAEP since 1992, and New Hampshire hasn’t since 1994.
The only fourth-grade NAEP math scores came from tests administered in 1992 and 1996; only in eighth-grade were math data available for three years (1990, 1992, and 1996). Such limited data make for the thinnest of trend lines and weaken RAND’s claim to be presenting a nationwide picture of student achievement.
Another problem, as University of Missouri economist Michael J. Podgursky has noted, is that the RAND researchers used grouped state-level data.
“It’s very difficult to test hypotheses about whether particular education policies work or don’t work with aggregated state-level data,” explained Podgursky. “The best you can really say is that something is going on in state A that makes its performance significantly better or worse than comparable states, but it’s hard to isolate what that something is with so few observations. To identify the effect of particular policies, you really need school or student-level longitudinal data.”
Data quality is another issue. Kentucky education activist and testing expert Richard Innes pointed to potential problems with the quality of NAEP random sampling; RAND appeared to acknowledge the dataset’s weaknesses by attempting to use Census data to correct uneven student reporting information about family background.
It was Innes who discovered that widely touted reading gains by certain states in the 1998 NAEP testing were largely generated by dramatically increased exclusions of disabled students from the testing.
Dr. Alan Bonsteel, a San Francisco physician and president of California Parents for Educational Choice, agreed that the RAND authors had addressed the “downward creep” in NAEP participation that has led to an “upward creep” in NAEP scores over the past six years. However, Bonsteel added, RAND “skirted the central problem, namely that NAEP participation rates are self-reported and unaudited.”
The RAND study reported, for example, that Louisiana excluded zero percent of its Limited English Proficiency (LEP) students and had 100 percent participation in NAEP four years in a row.
“These utterly implausible numbers have been simply accepted at face value in the RAND study,” Bonsteel noted.
Democrats and Republicans alike found support in the RAND report for their education proposals.
Democratic Presidential candidate Al Gore and fellow liberals touted the report’s finding that reduced class sizes and expanded preschool made a difference for poor children. Those are two objectives in Gore’s plan to increase federal education spending by $115 billion.
Conservatives found support in the RAND conclusion that hiking teacher salaries and requiring advanced degrees for teachers don’t affect student achievement. Republican Presidential nominee George W. Bush took pride in Texas’ ranking near the top when 1990-96 NAEP gains were compared with those for students of similar race and income in other states. While Bush didn’t become Governor until 1995, he has continued an education reform drive launched by Ross Perot in the 1980s.
The RAND report and press release took great pains to draw an unfavorable contrast between Texas and California, pointedly observing that the Golden State ranked “dead last” in cross-state comparisons of achievement by disadvantaged students.
That conclusion would appear to contradict lead RAND researcher Grissmer’s crediting popular “systemic reforms” for gains. In California, the systemic changes of the late 1980s that would have affected the 1990-96 NAEP tests were the wholesale adoption of “fuzzy math” standards and the sacking of phonics in favor of whole language.
More recently, led by Mathematically Correct and similar organizations, California has restored computation skill to math and phonics to beginning reading. The effects of those research-proven “systemic reforms” have not yet kicked in for NAEP testing.
Robert Holland is a senior fellow at the Lexington Institute, a think tank in Arlington, Virginia. His e-mail address is [email protected].
For more information . . .
The full text of the RAND report, Improving Student Achievement: What NAEP State Test Scores Tell Us, by David W. Grissmer, Ann Flanagan, Jennifer Kawata, and Stephanie Williamson can be ordered for $20 from the RAND Web site at http://www.rand.org/centers/education/pubsnav.html; click on What’s New. The table of contents is available on the Web site at http://www.rand.org/publications/MR/MR924.
Robert Holland, a journalist and author who has championed school choice throughout his career, is a Heartland Institute Senior Fellow addressing education policy.
