No. 38 February 2005
Do Charter Schools Help Their Students?
Caroline Hoxby is a professor of economics at Harvard University and program director of the Economics of Education Program at the National Bureau of Economic Research. This Civic Bulletin is adapted from the transcript of a luncheon co-sponsored by CEI-PEA and the Manhattan Institute in New York City on October 21, 2004.
PROFESSOR HOXBY: I'd like to go back to July 2004 and tell you about my outlook on charter schools at that time. At that time, there were more than 3,000 charter schools in the United States, and they didn't enroll very many students in the U.S.-only about 1.5 percent of the students in the U.S. But they were important schools. We researchers thought we would learn a lot from them, and educational innovators from all parts of the political spectrum were involved in making charter schools work across the United States.
Although most charter schools are in urban areas like New York City and although most disproportionately enroll low-income and minority students, there are also charter schools in rural areas. For instance, there are a good many charter schools in Alaska, which mainly enroll Native Americans. There are also a number of charter schools in rural Texas, rural Minnesota, and rural California that cater mainly to the children of farm workers.
In short, in July 2004, there was a diverse set of charter schools. Despite that diversity, there was tremendous consensus in the research community about how to evaluate charter schools. The consensus was that we needed to have randomized studies, which are easy to do with charter schools because most of them have more applicants than they have places.
In order to do a randomized study, we start with a charter school's entire pool of applicants. When the school holds a lottery among its applicants, some will randomly be admitted (or "lotteried in") and consequently attend charter schools. Some will randomly not be admitted (the "lotteried-out") and consequently continue to attend regular public schools. Researchers follow both groups of students, the lotteried-in and lotteried-out, over several subsequent years. Nearly all researchers agree that this type of study is the gold standard because randomization ensures that you're comparing apples with apples. We compare students who go to charter schools to students who wanted to go to charter schools. The two groups are not only the same in terms of family background, they also are equally motivated, doing equally well in regular public school, and so on.
The U.S. Department of Education has funded several studies that use randomization, and they will produce findings in a couple of years. I myself am leading some studies that use randomization, and, with my co-author Jonah Rockoff of Columbia University, released the first randomization-based study in May 2004. Let me give you a sense of what such a study looks like. We studied the largest charter school system in Illinois, which covers many of the charter school students in Chicago. The charter schools in question are about 80 percent black.
Most of the rest of their students are Hispanic. About 82 percent of their students come from such low-income families that they would qualify for free or reduced-price lunch. All of the students come from inner-city neighborhoods.
We were able to ascertain that the Chicago charter school lotteries were indeed fair. The lotteried-in and lotteried-out students were almost identical in terms of race, ethnicity, family income, home location, special education status, limited English proficiency, and prior achievement in the regular public schools.
After following both groups of students, we found that the lotteried-in students who attended charter schools had math and reading achievement that was about 6 percentile points higher than lotteried-out students who continued in the regular public schools. These gains were achieved after only a couple of years. If we extrapolate the gains to an entire primary and secondary education, a student could gain almost four grade equivalents by being in the charter schools as opposed to the regular public schools. Ultimately, we won't need to extrapolate because we'll continue to follow the students until they graduate from high school. Studies like ours, which use the randomized method and follow children over a long period of time, are undoubtedly the best way to do research on charter schools. Moreover, such studies are also a good way for the charter schools to learn about themselves. We researchers communicate with the schools so that they learn where they're succeeding and where they're not—math versus reading, grades one through three versus grades four through six, and so on.
I'm hoping that we researchers are able to continue doing studies of this type, but the future can be potentially undermined by articles like the one that appeared on the front page of the New York Times in August of this year. It was one of the most irresponsible examples of education reporting that I have ever seen.
The New York Times gave uncritical promotion to a "study" conducted by the American Federation of Teachers (AFT) that compared students in charter schools in the United States with students in regular public schools, using the National Assessment of Educational Progress test. That test is the only exam given to a representative sample of students across the entire United States. It is a very good, useful exam for plotting national trends and for following states in a general way.
However, the National Assessment of Educational Progress is administered to only 3 percent of American students and is designed to be representative of the entire student population of the U.S. This means that it is not appropriate for the use that the AFT put it to. Charter-school students are only 1.5 percent of students in the United States, and the test covers only 3 percent of students. Take 3 percent of 1.5 percent, and you realize that the charter school results touted by the AFT depended on a mere 0.045 percent of the students in the United States. That is, the results depending on about four one-hundredths of one percent of students in America.
That is an incredibly small number of students on whom to rely. To give you a sense, four one-hundreds of one percent of students in New York is just thirty-two students. In other words, we're talking about relying on the equivalent of one classroom of students to evaluate charter schools for the entire state of New York. The situation is even worse for smaller states, where the results may depend on fewer than a dozen students! The National Assessment of Educational Progress is just not appropriate for evaluating charter schools.
Even worse, the AFT study compared apples to oranges. It compared the average charter school student with the average regular public school student. That's not a fair comparison because most charter school students are economically disadvantaged students from inter-city or rural areas. They're disproportionately likely to be minorities. In contrast, the typical regular public school student in the United States is a suburban student who is not a racial or ethnic minority and who comes from a family with average income.
Not only did the New York Times not criticize the AFT for comparing apples to oranges, the New York Times did not even point out that when the AFT compared black students to black students, or Hispanic students to Hispanic students, charter school and regular public school students performed equally well. The New York Times article was not only a very irresponsible piece of reporting; it was deliberately misleading.
My initial response to the AFT study was that it was such a lousy study that it did not deserve a response. It did not deserve to be considered for even a few minutes. Serious education scholars all over the United States agreed. Their unanimous response was that the AFT study was poor research and highly unscientific. Let me give you just one piece of evidence about the unanimity of the response. Several people have asked me to debate another scholar about the AFT study. I've replied each time by saying that, if they could find me a scholar who will support the study, I'd be glad to debate him or her. Despite scouring the scholarly world, they've not been able to come up with a single serious researcher who will defend the AFT report.
In America, we need to have education reform based on science if we're going to make progress. Science is the standard we set for medical research. Why should we accept quackery for education?
Unfortunately, the AFT study received a lot of attention, probably owing to the New York Times’s irresponsible coverage. The attention eventually convinced me that it would be a good idea for someone to produce evidence that addressed the AFT study's most egregious failings.
Although charter schools are fee-based and more autonomous than regular public schools, they are public schools. Therefore, in every state in the United States, charter school students take exactly the same exams, under the same conditions, as regular public school students take. Charter school students are assessed against the same proficiency levels as are regular public school students.
Therefore, there is no reason to rely on a minuscule sample of students. You can get test scores for all students in charter schools in the United States, and that's exactly what I did.
I obtained data on each elementary charter school in the United States and matched each one to a sensible comparison school—the nearest regular public school or the nearest regular public school with a similar racial composition. These schools (which are the same one 93 percent of the time) make a good, sensible comparison. Why? Think about where a charter school student who is, say, a fourth-grader would be if he or she weren't in the charter school? He or she would probably be in the nearest regular public school. Put another way, what decision do the parents of a disadvantaged, elementary school-aged child really get to make? They decide whether to send their child to the local regular public school or a nearby charter school. They don't get to consider elite suburban schools or schools that are twenty-five miles away.
In short, what I did was compare charter schools with the nearest regular public schools, which shared not only a neighborhood, but also were very similar on race, ethnicity, family income, parent's education, and so on. When I did this, I found that, on average, charter school students are about 5 percent more likely to be proficient in reading and 3 percent more likely to be proficient in math.
These are not dramatic improvements in proficiency, but they are significant. Furthermore, in states that have stronger charter school laws and that have had charter schools operating longer, the proficiency gains are larger. Here are some examples:
In Arizona, the state with the largest share of students enrolled in charter schools, charter school students are about 10 percent more likely to be proficient in reading and math. In California, Massachusetts, and Pennsylvania, charter school students are 8 to 9 percent more likely to be proficient in reading. In Alaska, Colorado, the District of Columbia, New Jersey, Hawaii, Illinois, Louisiana, New Jersey, and Oregon, charter school students are at least 12 percent more likely to be proficient in reading. North Carolina is the only state in which the charter school students were statistically likely to be performing worse than regular public school students in both reading and math. There, they were about 4 percent less likely to be proficient.
That's a picture of the nation's charter school students as we have it now. You'll notice that I didn't mention New York. Why? New York, like many other states, has too few charter school students enrolled at this point for us to figure out what's happening. I laud Chancellor Klein's efforts to improve charter school availability in New York City, but the fact is that the state of New York has only 0.33 percent (one third of one percent) of its students enrolled in charter schools. With such a small share of students, it's too early to draw conclusions.
Of course, the goal of education reform is not to have a sea of mediocre public schools in which there are a few high-performing charter schools. What we'd like to see in the long run is high performance at both charter schools and regular public schools. A charter school can exhibit pedagogical and management innovations, but it can also be a threat. There's nothing like a combination of an inspiration and a threat to wake up a management team and that includes the management teams at regular public schools. Thus, while we'd like charter schools to innovate and set performance standards, we'd also like to see all regular public schools rise to the challenge and match them.
There is evidence that regular public schools are already rising to the challenge in Michigan and Arizona, where charter schools represent a significant share of enrollment. In those states, we see that regular public schools improve when they face a neighboring charter school that can take, say, 6 percent of local students. We want all American schools to experience similar challenges and respond similarly. I cannot emphasize enough that the goal is to improve all schools, not just to have good charter schools in the United States.
Improving education in America could not be more important. I'm an economist by training and I worry a great deal about the skills of the country's population. We live in a global environment in which we cannot protect American workers forever from skilled workers overseas. I wish we could, but that is not in the cards.
Therefore, we need all students in the U.S. to become skilled enough to compete in the global economy. It's not acceptable to have students who are unskilled because they are minorities, or are disadvantaged, or just go to lousy schools. Education reform is all about ensuring that every child has the opportunity to succeed.
Education reform in the United States also ought to be based on science. We do not accept drugs that do not go through randomized trials, and we certainly don't publish quack studies about drugs on the front page of the New York Times. Returning to my first theme, I want to reiterate that randomized studies are the best way to understand charter schools. We should have the patience to wait for randomized studies to produce results. Charter schools are doing well enough - better on average than the regular public schools with which they compete—to make us wait and find out what works and why.