Manhattan Institute for Policy Research.
search  
 
Subscribe   Subscribe   MI on Facebook Find us on Twitter Find us on Instagram      
 
   
 

Civic Bulletin
No. 27 October 2002


Rising to the Challenge: The Effect of School Choice on Public Schools in Milwaukee and San Antonio

Jay P. Greene, Senior Fellow at the Manhattan Institute
Greg Forster, Senior Research Associate at the Manhattan Institute

Introduction

A considerable body of high-quality research exists on whether students who are given the opportunity to attend private school with a voucher benefit academically. Random assignment experiments, the “gold standard” for research designs, have been conducted in Charlotte, Dayton, Milwaukee, New York, and Washington, D.C. to address whether school choice improves outcomes for students who attend private school with a voucher. All of those experiments show significant positive results, particularly for African-American students.

Whether public schools improve in response to the challenge of voucher programs, however, has been less thoroughly studied. The initial results from a number of studies suggest that rather than being debilitated by a loss of revenue and talent to choice programs, public schools are energized to improve and retain their students. As traditional public schools are likely to educate the bulk of students under any realistic choice arrangement, more research on the public school response to the challenge of vouchers would be useful.

This study provides new evidence on the broader question of how public schools fare when faced with the challenge of school choice. It finds that some public schools exposed to competition through school choice programs in Milwaukee and San Antonio significantly improved their academic performance. Controlling for demographic factors of race and income as well as for local school spending, public schools exposed to competition showed more improvement in student test scores than other public schools. The Edgewood school district in San Antonio, the one Texas district where public schools were exposed to a large-scale privately funded voucher program, did as well as or better than 85% of Texas school districts after controlling for demographics and local resources. In Milwaukee, private-school competition (at the 4th grade level) and charter-school competition (at the 10th grade level) were found to cause significant improvements in public school test scores, again controlling for demographics and resources. The comparison between the effects of private-school and charter-school competition is most appropriate at the elementary level, the level at which both private and charter competition are widely available; at that level, only private competition was found to cause improvements.

Previous Research

The claim most often made for school choice is that it benefits the students who participate, by enabling them to attend whichever school—public or private—will provide the best education. The question of whether school choice benefits participating students has been well-studied; there have been five studies of school choice programs that have taken advantage of random assignment methods. Random assignment provides the most reliable study results, because it allows researchers to compare groups of students that are, for all practical purposes, identical in every respect except one: students in one group used school choice programs and students in the other group did not. All five of the random assignment studies of school choice found that participating students had significantly better test scores than non-participating students (see Rouse 1998; Greene, Peterson, and Du 1999; Greene 2001a; and Howell and Peterson 2002).

However, school choice may have significant effects on more than just the students who participate. As research has accumulated showing that school choice benefits participating students, its critics have relied more and more heavily on the argument that school choice will hurt public schools and the students who remain there after participating students have left for private schools. In response to these criticisms, advocates claim that school choice programs indirectly benefit public schools and their students by forcing public schools to compete with private schools, providing a strong incentive for those public schools to improve. According to this argument, because public schools do not want to lose students (and the revenue students generate) to private schools, they can be expected to respond constructively to the presence of school choice programs, providing better educational services in order to reduce the number of students who choose to exercise their option to leave the public school system. The alternative claim is that the departure of resources and talent from public schools under choice programs debilitates public schools, harming their academic quality.

The effect of school choice programs on public schools has not been studied as well as its effect on participating students, so less is known about this question. Of the few studies that have been done of U.S. public schools exposed to school choice, none have ever found a decrease in the academic performance of public school students, and a few have found academic gains. Studies of school choice outside the U.S. have reached less encouraging conclusions about its benefits for public education, but there are questions regarding whether these conclusions are applicable to U.S. schools.

A previous study by Jay Greene found that failing Florida schools improved when threatened with exposure to school choice. Under Florida law, every school is graded each year based on its students’ performance on a statewide exam. If a school has received a grade of “F” in two of the past four years, students assigned to that school become eligible for scholarships to attend private schools. Thus, when a school receives its first “F” grade, it knows it must provide better educational services or risk losing students to school choice. Greene found that schools that received an “F” in the first year of the program did remarkably better in the second year. For example, their average math score showed an improvement so large that it would have taken a spending increase of $3,484 per pupil—a 60% increase over existing funds—to achieve the same improvement by expanding school budgets (see Greene 2001b). This suggests that public schools are both able and willing to respond constructively to the prospect of losing their students to a choice program.

Caroline Hoxby has also studied public school responses to school choice programs. Studying charter schools in Michigan and Arizona, she found that public schools exposed to competition from charter schools (defined as public schools operating in areas where at least 6% of students were enrolled in charter schools) made significantly better improvements in test scores than public schools not exposed to charter competition.

In Arizona, she found that 4th grade math scores improved by 3 percentile points in public schools exposed to charter competition, and only by 1 percentile point in public schools not exposed to competition. Hoxby also found that public schools exposed to choice had significantly higher gains in 4th grade reading, 7th grade math, and 7th grade reading. She wrote that if schools exposed to competition in Phoenix continue to improve so much faster than schools in the Phoenix suburbs, the achievement gap between them would close in less than ten years. In Michigan, she found that in public schools exposed to charter competition, 4th grade reading scores climbed 2.4 scale points faster than in other public schools, and 4th grade math scores climbed 2.5 scale points faster. She wrote that if this trend continued, the achievement gap between public schools in Detroit and public schools in the wealthy suburb of Grosse Pointe would close in less than twenty years (see Hoxby 2001).

Hoxby has also measured the effects of competition on public schools by comparing test scores in areas served by many different school districts with test scores in areas served by only one district, such as Miami or Hawaii. In an area served by many districts, parents will exercise “school choice” on their own by finding out where the best schools are and moving there. Hoxby found that school districts faced with public-school competition had consistently higher test scores while spending less per student. In one such study, an increase in public school competition equal to one standard deviation was associated with a 17% decrease in per-pupil spending and a 3 percentile point improvement in test scores (see Hoxby 1998).

Most importantly, Hoxby has studied public school response to school choice in Milwaukee, one of the cities included in the present study. She found that schools most exposed to competition from private school scholarships had larger increases in math, science, and language test scores than schools less exposed to competition. For math scores, schools more exposed to competition improved by 7.1 percentile points, while other schools improved by 3.7 percentile points (see Hoxby 2001). Hoxby did not study public schools forced to compete with charter schools, as opposed to private schools, and includes fewer years in the period studied than does the present study. Also, she studied only 4th grade test scores, whereas the present study also includes 8th grade and 10th grade test scores. As we will see, our results for 4th grade scores are consistent with Hoxby’s, but we found different results for 8th grade and 10th grade scores.

A study by Christopher Hammons of century-old school choice programs in Maine and Vermont found that such programs produced gains in public-school test scores. In those states, school districts that do not operate their own schools pay to send their students to public schools in other districts or to non-religious private schools, a practice known as “tuitioning.” Hammons found that high schools located in areas with larger concentrations of tuitioning districts had better test scores than high schools with lower exposure to tuitioning. He found that if a town one mile away from a school started tuitioning, that school could expect a 3.4 point increase in test scores, a gain of 12% over existing scores (see Hammons 2001).

Studies examining public school response to school choice in other countries have found more mixed results. However, there are serious questions about whether the programs covered by these studies are analogous to school choice programs in the U.S., both because of differing national conditions and because the programs themselves are different. A school choice program in New Zealand, studied by Helen Ladd and Edward Fiske, did not create many new schools and provided choice only within the traditional public school system; furthermore, teachers’ contracts and the academic curriculum were both controlled by the national government and kept uniform across all schools, allowing for little meaningful choice even within the public school system (see Ladd and Fiske 2000). A private-school scholarship program in Chile, studied by Patrick McEwan and Martin Carnoy, provided so little money for the scholarships that in practice students had only limited opportunities to use them; what’s more, the study failed to control adequately for student demographics and background, making it impossible to separate the effects of school choice from the effects of poverty, family status, and other factors (see McEwan and Carnoy 1999).

Frederick Hess has conducted case studies suggesting that while U.S. public schools do respond to school choice, institutional and cultural factors limit the significance of these responses. Hess did not look at test scores, focusing instead on whether public schools undertake institutional reforms, and most of his explanations for public school resistance to change are theoretical. In addition, he studied cases in which school choice had not been in effect for long periods of time. However, his work does shed light on potential barriers to change in public schools (see Hess, 2002).

Method

Edgewood is a small school district in San Antonio, Texas, serving a low-income, Hispanic section of the city. Since 1998 a private organization has offered a scholarship to every student in the Edgewood district with which students could attend a private school or a public school in another district. This scholarship program effectively creates a system of school choice in Edgewood in which all schools in the district are exposed to private-school competition.

In order to compare schools that were exposed to competition (“treated” schools) with schools not exposed to competition (“control” schools), we had to compare the performance of the Edgewood school district with that of other school districts in Texas. This required us to develop a model that would control for the demographic characteristics of each district’s population, as well as the level of resources available to public schools in each district.

To accomplish this, our model calculates an “expected gain” for each district in test scores on the statewide TAAS test between 1998 and 2001, the period covered in the study. This expected gain is calculated based on the demographic characteristics of the district population (including race and poverty) and local resources (including the district’s per-pupil spending and teacher-student ratio). For each of these factors we took into account both the district’s characteristics at the beginning of the study period and the change in those characteristics over the study period. The expected gain figure for each district allows us to control for the effects of population demographics and local resources. By comparing a district’s actual test score gain (or loss) with its expected gain, we can isolate changes in test scores that are not attributable to changes in population or resources.

To the extent that Edgewood outperforms its expected gain, some factor other than changes in population and resources has pushed up its test scores. Likewise, to the extent that Edgewood underperforms its expected gain, some other factor has driven down test scores. And by ranking all districts in Texas according to how well they outperformed (or how badly they underperformed) their expected gains, we can see whether Edgewood is outperforming its expected gain significantly better (or underperforming it significantly worse) than most Texas districts. To the extent that Edgewood performs significantly stronger or weaker than most Texas districts, this would be consistent with the hypothesis that school choice has had a significant effect on the quality of public education. Of course other factors, including mere chance, may play a role in determining Edgewood’s statewide rank. However, the further Edgewood ranks towards the top or bottom of the scale, the more likely it is that school choice had some significant effect.

Milwaukee has the largest school voucher program, and, with the exception of “tuitioning” programs in Maine and Vermont, is the longest running voucher program. In Milwaukee, unlike in Edgewood, not every school was equally exposed to competition from choice programs. This allowed us to adopt a different (and better) research design in which we were able to examine whether schools that were more exposed to competition from choice programs experienced greater academic improvement. We measured both the extent to which each school was exposed to private-school competition and the extent to which each school was exposed to charter-school competition.

To measure the extent to which each school was exposed to charter-school choice, we used an index of the distance between the school and the three nearest charter schools. The closer a public school is to charter schools, the more it is exposed to charter-school competition.1 Distance to charter schools is the relevant measurement of exposure to charter-school competition because all charter schools in Milwaukee are free and all students are eligible to attend them. The only variation in access to charter schools is logistical: how hard is it for students to get to the charter school? If a public school is located close to charter schools, less extra effort is required for students to attend those charter schools rather than the public school. The easier it is for students to get to charter schools, the more likely the public schools are to face competition from those schools.2

We faced a challenge in measuring exposure to private-school choice, because all students participating in the federal free and reduced-price lunch program were qualified to receive private school scholarships. We had to use participation in the federal lunch program both to measure exposure to private-school choice and to control for the effects of poverty in the school population. To solve this problem, for each school we used the percentage of the school’s population that participated in the federal lunch program in the 1995-96 school year as a measurement of the extent to which that school was exposed to private-school choice, and the percentage of the school’s population that participated in the federal lunch program in the 2000-01 school year as a measurement of poverty in the school population. We did so because 1995-96 was one year before the city’s private-school choice program expanded to full scale. Thus the federal lunch program participation figure for 1995-96 represents the number of students in each school who were about to become eligible for private-school choice, while the 2000-01 figure represents the number of impoverished students remaining in each school after school choice became widely available.

Our study measured whether changes in each school’s test scores for 4th grade, 8th grade, and 10th grade students on the Wisconsin Knowledge and Concepts Examinations between the 1996-97 and 2000-01 school years were related to our measurements of exposure to private-school and charter-school competition. We used the 1996-97 test scores, rather than earlier test scores, because changes in the test rendered earlier scores non-comparable with 2000-01 scores. In the model, we controlled for race and poverty in each school’s student population, as well as for projected per-pupil spending for the 2002-03 school year. A statistically significant relationship between exposure to competition and changes in test scores over the study period, controlling for other factors, indicates that competition had a significant effect.

Results

Table 1:
Statewide Rank of Edgewood School District
(Actual Gain Relative to Expected Gain)

Population

Edgewood Statewide Rank

All Students

85th Percentile

Hispanic Students

73rd Percentile

Impoverished Students

75th Percentile

Our findings for Edgewood are summarized in Table 1. We found that when Texas school districts were ranked according to the difference between actual gains and expected gains over the study period, the Edgewood school district ranked at the 85th percentile. This means that Edgewood’s performance relative to its expected gain was equal or superior to that of 85% of all Texas school districts. Among Hispanic students, who made up 97% of Edgewood’s population, Edgewood ranked at the 73rd percentile statewide. Among lower-income students (those in the federal lunch program), who made up 93% of Edgewood’s population, Edgewood ranked at the 75th percentile statewide.

This indicates that, after the effects of population demographics and local resources were isolated and removed, Edgewood performed well above the average Texas school district among all students, Hispanic students, and low-income students. This is consistent with the hypothesis that public schools are responding to competition from school choice by improving educational services. Of course, other factors may be at work, including random chance. However, given Edgewood’s unusually strong performance, the data suggest that school choice probably made an important difference in student outcomes.

Table 2:
Average Test Score Change In Public Schools Associated With Competition

Expected Test Score Change

4th Grade

50% of students eligible for private school scholarships

-5 points

100% of students eligible for private school scholarships

+10 points

10th Grade

Charter school 5 km away

+3.5 points

Charter school 1 km away

+9 points

Our results for Milwaukee are summarized in Table 2. In 4th grade, we found a statistically significant relationship between private-school competition and public school test scores, but not between charter-school competition and public school test scores.3 The effects of private-school competition were such that if a public school had only 50% of its students eligible for private-school scholarships in 1995-96, and was in every other way average, it could expect its average test score to decrease by just over 5 percentile points over the study period. On the other hand, if the same public school had 100% of its students eligible for private-school scholarships in 1995-96, it could expect its average test score to increase by just over 10 percentile points—a 15-point difference. (On average, the actual level of exposure to private-school competition was about 77% for elementary schools.)4

However, our results for middle and high school tests were not identical to our results for 4th grade tests. In 8th grade, we found no statistically significant relationship between exposure to either private-school or charter-school competition and test scores. And in 10th grade, our findings for 4th grade were actually reversed: we found a significant positive relationship between charter-school competition and public school test scores, but no significant relationship between private-school competition and test scores. The effects of charter-school competition were such that if a school faced competition from only one charter school located 5 km away, and was in every other way average, it could expect its average test score to increase by only about 3.5 points between 1996-97 and 2000-01. But if we were to pick up that competing charter school and move it 4 km closer, such that it was only 1 km away, the public school could expect its average test score to increase by about 9 points over the study period.5

In interpreting these results, two important facts need to be kept in mind. The first is that there is a much greater supply of private elementary schools than of private middle and high schools. This is due primarily to the significantly greater per-pupil spending needed to operate a middle or high school. In light of this fact, the discrepancy between our results for 4th grade, 8th grade, and 10th grade is less mystifying; private-school competition seems to apply more competitive pressure on public schools at the elementary level, where there are more private elementary schools for parents to choose from, and less competitive pressure at the middle and high school level, where charter schools take up some of the slack caused by the smaller number of private options available. At the elementary level, the level at which both private and charter competition are available, private schools rather than charter schools seem to drive public school improvements.

The other fact to be borne in mind is that our results for 8th grade and 10th grade tests are hindered by the low number of public middle and high schools in Milwaukee. While we had data from 115 public elementary schools in the city, we had data from only 33 middle schools and 20 high schools. This smaller number of schools reduces the chance that we will discover any relationship that may exist between exposure to competition and test scores.

Conclusion

The data in this study seem to indicate that public schools respond to competition from school choice programs by improving educational services. In Texas, the one school district where public schools were exposed to private-school competition did as well as or better than 85% of Texas school districts, after controlling for population demographics and local resources. In Milwaukee, private-school competition (at the 4th grade level) and charter-school competition (at the 10th grade level) were found to cause significant improvements in public school test scores, again controlling for demographics and resources. At the elementary level, for which both private and charter competition are widely available, only private competition was found to cause improvements.

There is, of course, much that these data do not tell us. Further study of the subject is badly needed, especially since much less research has been done on this question than on the question of school choice’s benefits for students who directly participate in choice programs. Given that whether school choice helps or hurts public schools is a question of large and growing political importance, discovering more about this phenomenon should be a high research priority.

References

Endnotes

  1. We measured exposure to charter-school competition by taking the sum of 1/X + 1/Y + 1/Z, where X, Y, and Z are the distances in kilometers between the school and each of the three closest charter schools. Thus, the further away the three closest charter schools were, the lower the figure for charter school competition.
  2. This research design uses the distance from a public school to a charter school as a proxy for the distance between that charter school and the homes of the students who attend that public school. For this to be true, it must be the case that most Milwaukee public school students attend schools close to where they live. Aquine Jackson, director of the Milwaukee Public Schools Office of Neighborhood Schools indicates that this is the case (phone interview, 1:15 pm, Sept. 30, 2002). According to the Office of Neighborhood Schools, only 63% of Milwaukee public school students are bused to school. Elementary school students are bused if they live one mile from school, middle school students are bused if they live two miles from school, and high school students are bused if they live outside the school’s attendance area. Since students living as close as one mile away from school can be bused, the 63% figure includes many students who live close to their schools; clearly most Milwaukee public school students go to school close to where they live.
  3. Following standard practice, for all regression analyses we considered a result statistically significant if p was less than 0.05.
  4. For our 4th grade results, the standardized coefficient for exposure to private-school choice was 0.49. This means that an increase of one standard deviation in exposure to private-school choice can be expected to produce an increase of about half a standard deviation in a school’s average test score gain.
  5. For our 10th grade results, the standardized coefficient for exposure to charter-school choice was 0.59. This means that a difference of one standard deviation in exposure to charter-school choice can be expected to produce a difference of about six-tenths of a standard deviation in a school’s average test score gain.

 


Center for Civic Innovation.

EMAIL THIS | PRINTER FRIENDLY

CB 27 PDF

WHAT THE PRESS SAID:

Six Sigma Schools? Wall Street Journal, 1-15-03
Choice Helps Public Schools by Jay P. Greene, New York Sun, 1-14-03
School choice is good move for all students by Jay P. Greene & Greg Forster, Austin American-Statesman, 1-14-03
Choice proves beneficial for public schools, too by Jay P. Greene and Greg Forster, Milwaukee Journal-Sentinel, 10-17-02

SUMMARY:
This report assesses the impact of charter school and school voucher competition on public schools. Controlling for demographic factors of race and income, and for local school spending, the report analyzes the effect of varying levels of school choice competition on student test scores of the Milwaukee public school system, and of the Edgewood school district in San Antonio. It finds that Edgewood, the one Texas district where public schools were exposed to a privately funded voucher program, showed as much or more test score improvement than 85% of Texas school districts, while in Milwaukee private-school competition (at the 4th grade level) and charter-school competition (at the 10th grade level) were found to cause significant test score improvements in public schools.

TABLE OF CONTENTS:

Introduction

Previous Research

Method

Results

Table 1: Statewide Rank of Edgewood School District (Actual Gain Relative to Expected Gain)

Table 2: Average Test Score Change In Public Schools Associated With Competition

Conclusion

References

Endnotes

 


Home | About MI | Scholars | Publications | Books | Links | Contact MI
City Journal | CAU | CCI | CEPE | CLP | CMP | CRD | ECNY
Thank you for visiting us.
To receive a General Information Packet, please email support@manhattan-institute.org
and include your name and address in your e-mail message.
Copyright © 2014 Manhattan Institute for Policy Research, Inc. All rights reserved.
52 Vanderbilt Avenue, New York, N.Y. 10017
phone (212) 599-7000 / fax (212) 599-3494