The Mission of the Manhattan Institute is
to develop and disseminate new ideas that
foster greater economic choice and
individual responsibility.

The New York Times.

New Ammunition for Backers of Do-or-Die Exams
April 23, 2003

By Greg Winter

The rocky terrain of educational research has gotten even rockier.

Two new studies make the case that do-or-die exams -- which decide whether students graduate, teachers are dismissed or schools are shut in more than half the states in the nation -- have brought about at least a modicum of academic progress, especially for minority students who may get scant attention otherwise.

The studies entirely contradict what some other scholars have found and are bound to feed an already fiery debate over the phenomenon known as high-stakes testing, a course of educational change that teachers resent, the Bush administration embraces and states are hurriedly adopting.

Neither the authors nor their peers contend that the new research ends the dispute, given the many remaining open questions. But taken together, the studies appear to push the research pendulum away from critics who have argued that the fixation with make-or-break exams undermines teachers, stifles analytical learning and squeezes out struggling students, all without providing any clear benefits.

"If I were gambling on whether to put in a high-stakes system or not, I would put one in," said Martin Carnoy, the Stanford University professor who co-wrote one of the studies. "There's some probability I would be wrong. But if I were to put my money on something right now, I would try this."

In his study, published next month in Educational Evaluation and Policy Analysis, a peer-reviewed journal, Mr. Carnoy and a colleague, Susanna Loeb, examined whether states with serious test consequences did better on a nationwide math assessment than their counterparts bearing none at all.

While there seemed to be little to no difference in the performance of white students, the study found that the more consequences a state imposed, the better its minority students typically did.

In fact, for every additional layer of sanction or reward placed on schools, teachers and children, about 3.5 percent more black students and 3 to 4 percent more Latinos grasped the basics of eighth-grade math.

The same pattern did not prove true for Latinos in math in the lower grades, but for black students it did, leading Mr. Carnoy to speculate that the threat of consequences may compel schools to demand more from students whom they may have otherwise written off.

No less importantly, the study found that do-or-die exams did not lead to more dropouts, as other researchers have argued. Still, there was no evidence that they improved graduation rates.

"If that's not the aim, what is all this about?" asked Mr. Carnoy, whose study was financed by the federal government. "Why do we care about raising test scores if more people aren't going to finish high school and go on to college?"

Another peer-reviewed study to be published next month found that when states imposed consequences on their own exams, their students tend to do better on nationwide math assessments as well.

Margaret E. Raymond and Eric A. Hanushek of the Hoover Institution at Stanford, two supporters of high-stakes tests, found that national math scores between 1996 and 2000 rose an average of seven-tenths of 1 percent in states with no consequences, 1.2 percent in those that simply published the results in the newspaper and 1.6 percent in states that either rewarded success or penalized failure.

Small though they may seem, the differences between the gains are meaningful, the authors argue, because national math scores have barely moved in decades.

"The systems that we had in place by 2000 are not going to revolutionize our schools, that's clear," Mr. Hanushek said. "But they are an element in moving our schools forward."

The study was paid for by the Packard Humanities Institute and the Smith Richardson Foundation, neither of which claims a position on the issue.

There is a caveat to all this data, educational researchers warn. Just because a correlation may exist between make-or-break exams and achievement does not mean the exams are to thank for any progress. Indeed, there are so many factors influencing test scores, be they economic or curricular, that proving a causal link between consequences and results may never be done.

"That goes for both sides of the debate," said Robert L. Linn, president of the American Educational Research Association. "To be able to definitively attribute results to the stakes of a test is a stretch for any of them."

Underneath the academic dispute over the efficacy of make-or-break exams is a strong distaste for them among teachers. A survey of more than 4,000 teachers last February found that about three quarters of them, whether or not they taught under the threat of serious consequences, said that state testing programs "were not worth the time and money involved."

Perhaps more worrisome, 76 percent of teachers facing the highest stakes and 63 percent of those encountering the lowest said that mandatory testing led to teaching in ways that contradicted their own ideas of sound educational practice.

"Teachers are saying look, we're not opposed to high standards, but when you put so much emphasis on this one testing measure, it really becomes problematic," said Joseph J. Pedulla, who helped coordinate the survey for National Board on Educational Testing and Public Policy at Boston College. It was financed by The Atlantic Philanthropies, which takes no position on the testing question.

The idea of teaching to the test bothers many educators, but others argue that it does not preclude learning thinking skills that can be transferred from one context to another.

A study last February by the Manhattan Institute, which was not peer-reviewed, found that the results on state exams closely paralleled those on other, independent tests.

Though it looked at less than 10 percent of the nation's schools, the study suggested that whatever controversy surrounds make-or-break exams, they can be reliable instruments for gauging student learning.

"Which should we believe, the teachers or the tests?" said Jay P. Greene, the study's author. "We might wonder whether the teachers have it right and the test has it wrong. But it's also quite possible that the test has it right."

GRAPHIC: Chart: "A Carrot-and-Stick Approach to Testing"
Make-or-break tests have produced improvement in students math scores, according to a new study. The tests carry consequences ranging from publishing results in newspapers to schools potentially being shut down.
Map of the U.S. shows Degree of sanctions or rewards for schools, districts and students, per state.
What this means for national math scores
Below are the largest changes in students reaching competence when an additional layer of consequences was placed on them and their schools.
White -- 1.37%
Black -- 3.65
Hispanic -- 4.06
White -- -0.19%*
Black -- 3.42
Hispanic -- 1.69*
*Not statistically signficant, according to researchers.
(Source: Stanford University study to be published in Educational Evaluation and Policy Analysis)

©2003 The New York Times



Home | About MI | Scholars | Publications | Books | Links | Contact MI
City Journal | CAU | CCI | CEPE | CLP | CMP | CRD | ECNY
Thank you for visiting us.
To receive a General Information Packet, please email
and include your name and address in your e-mail message.
Copyright © 2009 Manhattan Institute for Policy Research, Inc. All rights reserved.
52 Vanderbilt Avenue, New York, N.Y. 10017
phone (212) 599-7000 / fax (212) 599-3494