Civic Report No. 12 August 2000 The Effect of School Choice:an Evaluation of the Charlotte Children’s Scholarship Fund Jay P. Greene Senior Fellow, The Manhattan Institute for Policy Research Executive Summary Does providing low-income families vouchers or scholarships with which they can select a private school improve student achievement? The evidence from the Children’s Scholarship Fund (CSF) program in Charlotte suggests that providing low-income families with scholarships has significant benefits for those families. This finding is consistent with the results from similar evaluations of scholarship programs in New York, Washington, D.C., and Dayton, Ohio as well as the results of evaluations of publicly funded school choice programs in Milwaukee and Cleveland. The main findings from this evaluation of the Charlotte CSF Program are: Receiving a scholarship to attend private school improves scores on standardized math tests by between 5.9 and 6.2 national percentile ranking points, depending on the type of analysis performed. Receiving a scholarship to attend a private school improves scores on standardized reading tests by between 5.4 and 7.7 national percentile ranking points, depending on the type of analysis performed. Parents were asked to assign their child’s school a letter grade, A through F. Nearly twice as many choice parents gave their child’s school an A (53%), compared to the public school parents (26%). Choice parents were also nearly twice as likely to report being “very satisfied” with virtually all aspects of their children’s school: location, safety, teaching quality, course content, class size, facilities, student respect for teachers, information on student progress, religious observance, parental support for school, discipline, clarity of school goals, teamwork among staff, teaching moral values, academic quality, and teacher respect for students. Roughly two in five students would give their choice school an A compared to 32% of public school students. When students were asked how they feel about going to school each day, 24% of the public school students said that they did not want to go compared to 9% of choice students. And 24% of non-scholarship students agreed that they did not feel safe at school compared to 9% of choice students. Parental reports confirm student perceptions about safety at school. More than a third of public school parents reported problems with fighting in school (36%) compared to 16% of choice parents. One-quarter of public school parents reported problems with racial conflict compared to 12% of choice parents. 22% of public school parents reported problems with guns or weapons at their children’s elementary schools compared to 11% of choice parents. And 25% of public school parents reported problems with destruction of property at school compared to 12% of choice parents. Because the private schools examined operate with far less money per pupil than do the public schools, it is not surprising to discover that the private schools have more sparse facilities and fewer services to offer. For example, only 70% of choice parents described their school as having a library compared to 90% of the public school parents. Only 63% of choice parents said that their school had a gym compared to 91% of public school parents. Only 71% of choice parents said that their school had a cafeteria compared to 89% of public school parents. Parents also reported fewer school services at the private schools. Only 18% of choice parents said that their school had a program for students learning to speak English compared to 50% of public school parents. Only 49% of choice parents said that their school had a program for learning disabilities compared to 71% of public school parents. Only 51% of choice parents reported programs for gifted students at their schools compared to 72% of public school parents. Choice parents were also less likely to report that their school had a counselor, nurse, music program, art program, or prepared lunches. The Charlotte CSF Program successfully targeted disadvantaged families. In general, choice schools were accepting students with scholarships who were considerably more disadvantaged than typical students in Charlotte. Three-quarters of the choice students were African-American, while a little more than one-third of all students in the Charlotte-Mecklenburg school district are African-American. As of 1990 the average family income in Charlotte was nearly $34,000, almost $10,000 more than the average family income of choice students 10 years later. Almost one-third (32%) of choice families report that they receive some kind of public assistance, such as food stamps or welfare, while the 1990 census reports that only 5% of households in Charlotte were on public assistance. And even after one year of the scholarship, choice students were still scoring well below the national average on standardized tests (although they were scoring significantly better than they would have had they not received the scholarship). There is no evidence to support the claim that the private schools were “creaming” the best students or “dumping” those students whom they found undesirable. The private schools accepting scholarship students were smaller and had smaller class sizes, on average, than the public schools. But small class size does not “explain” the higher student achievement observed in private schools. Adding class size to the multivariate model predicting student test scores shows that class size has no effect on student achievement in our sample.   The Effect of School Choice: An Evaluation of the Charlotte Children’s Scholarship Fund Program Introduction Does providing low-income families vouchers or scholarships with which they can select a private school improve student achievement?1  The evidence from the Children’s Scholarship Fund (CSF) program in Charlotte suggests that providing low-income families with scholarships has significant benefits for those families. This finding is consistent with the results from similar evaluations of scholarship programs in New York, Washington, D.C., and Dayton, Ohio as well as the results of evaluations of publicly funded school choice programs in Milwaukee and Cleveland. The findings of those studies have been summarized and discussed elsewhere.2  This report will focus on presenting the results from Charlotte. Research Design The CSF program offered partial scholarships to low-income families in Charlotte with a maximum value of $1,700 to attend private schools in the 1999-2000 academic year. To ration limited funds, scholarships were awarded by lottery to families that had completed an application process. This study examined only students enrolled in grades 2 through 8. In that age group, 388 students had been awarded scholarships by lottery and were enrolled in private school, 342 students were not offered scholarships by lottery, and 413 students had won the lottery to receive a scholarship but did not enroll in private school. All of these students and their parents were sent invitations to attend four testing sessions on a Saturday or Sunday between March 18 and April 30, 2000, where parents completed surveys while students took the Iowa Test of Basic Skills survey version. Older students also completed a survey. Families whose children were not using scholarships were offered $20 and an opportunity to win a new scholarship as incentives to participate and to defray the transportation and other expenses involved. Families whose children were using scholarships were simply asked to participate without compensation. Despite these relatively modest incentives, our response rate was quite good. Of the 1,143 students who were sent invitations to attend a testing session, 452, or 40%, participated in the study. The participation rate among the students who won the lottery and were using scholarships, whom we will call “choice students,” was 53%. The participation rate among the students who applied but failed to win a scholarship in the lottery, whom we will call “control students,” was 49%. The participation rate among the students who won a scholarship but did not use it to attend a private school, whom we will call “non-complying students,” was 20%. Various explanations account for the level of participation. The contact information available for all students was over a year old. Given the high mobility of urban, low-income populations, it is likely that many invitations never reached their target. In addition, we only offered four testing opportunities on Saturdays or Sundays, which may not have accommodated the work and social schedules of a number of families. Other factors that may have influenced participation include transportation issues, family motivation, and student cooperation with sacrificing a weekend day to take a standardized test. These obstacles to participation were obviously most severe among the group that we call non-complying students. Many of those families did not use the scholarship that was offered to them because they moved, exacerbating the difficulty of inviting them to participate in this study. Other students who were offered scholarships but did not use them (and thus did not “comply” with a lottery research design), may have declined the scholarships because they obtained access to a desired public school, such as a magnet school or other public school choice program. If these students were doing well in their public school they would have little reason to participate in the study where the primary incentive was the opportunity to win a private school scholarship. Other students may not have used a scholarship that was offered because they were unable to find a satisfactory private school. Yet other students did not use a scholarship that was offered because their families did not have the financial resources to pay the tuition charges above the $1,700 value of the scholarship. Families that do not believe that they will be able to use a new scholarship are unlikely to be enticed by an offer of a scholarship to participate in the study. Non-compliance and non-participation are issues in all evaluations, including random-assignment or lottery based studies, such as this evaluation and most medical studies. People are always free to cease cooperating with researchers and they are always free to refuse the treatment they are offered. Lotteries in research do not ensure identical treatment and control groups, but they certainly help get closer to achieving comparable groups than other methods of selecting subjects. To the extent that non-compliance and non-participation produce non-identical treatment and control groups, the differences can be adjusted statistically with little difficulty, as was done in this study. Comparability of Groups All applicants for scholarships were asked to provide their family income at the time of application. More complete demographic information was collected during the testing sessions, but, as noted, not all applicants participated in the study. By looking at the income information provided at the time of application we can see a number of things: 1) the lottery produced two groups that were not significantly different in income (this helps confirm that the lottery was properly conducted); 2) those applicants who participated in the study had somewhat higher incomes than those that did not; and 3) the differences between the incomes of study participants and non-participants are roughly equal for lottery winners and lottery losers as well as for choice, control, and non-complying students. In other words, while those who participated in our study differed somewhat from those who did not, those differences do not appear to have biased the comparability of our groups. The family income of applicants who won the lottery to be offered a scholarship was $23,449 compared to $23,689 for those who lost the lottery. The difference in income is not statistically significant, helping to confirm that the lottery was fairly conducted. The family income of students who participated in the study was $25,313, which is significantly different from the $22,441 reported at the time of application for those families who did not later participate in testing. This gap of roughly $3,000 between participants and non-participants exists among those who won the lottery (combining the choice and non-complying students) as well as among those who were in the control group. Lottery winners who participated in the study had average family incomes of $25,323, while lottery winners who did not participate had average family incomes of $22,517. Control group students who participated in the study had average family incomes of $25,297, while control group students who did not participate had average family incomes of $22,215. Whatever factors influenced participation in the study appear to have operated equally on lottery winners and lottery losers. This claim is further supported by the demographic similarity of the treatment and control groups who participated in the study and completed our survey. As can be seen in Table 1, the lottery winners and losers who participated in the study did not differ from each other very much in their demographic characteristics. The control group had slightly better educated mothers, but the difference was not significant, while those offered a scholarship were more likely to have mothers born outside of the United States. Those offered scholarships were more likely to receive Supplemental Security Income (SSI) for a family disability, while control group mothers were more likely to work outside of the home. All of these differences are modest and we can expect some significant differences to be produced by chance when comparing a large number of demographic characteristics. The overall picture is that despite non-participation in our study, we managed to preserve the similarity of the lottery winners and losers. Table 1: Demographic Characteristics of Lottery Winners and Losers Variable Lottery Winners Lottery Losers Significance Mother’s Education (11 point scale) 6.9 7.8 0.23 Mother U.S. Born 89% 96% 0.02 Attend Religious Services (5 point scale) 3.3 3.4 0.44 Receive Food Stamps 19% 19% 0.99 Receive Welfare 28% 24% 0.41 Receive Social Security 13% 11% 0.44 Receive Supplemental Security Income 28% 15% 0.01 Receive HUD Housing Vouchers 2% 5% 0.19 Family Income (from application) $25,323 $25,297 0.97 Family Income (from survey) $23,150 $24,800 0.14 Children in Household 2.4 2.2 0.12 Family Member in Jail 2% 1% 0.88 Student Has Physical Handicap 3% 1% 0.25 Student Has Learning Disability 9% 11% 0.55 Student is a Native English Speaker 97% 97% 0.77 African-American Mother 81% 80% 0.75 Two Parent Household 36% 33% 0.53 Male Student 49% 41% 0.1 Year of Mother’s Birth 1962 1963 0.28 Mother Employed Full-Time 60% 68% 0.03 Mother Single, Never Married 27% 29% 0.14 Mother is Baptist 38% 42% 0.96 N 206-267 135-161   Significance below .05 is conventionally considered statistically significant. But we are not primarily interested in comparing outcomes of lottery winners to lottery losers. That is, our primary interest is in identifying the effect of using a scholarship to attend private school, not the effect of being offered a scholarship even if one does not use it. We therefore want to compare choice students to the other groups. As can be seen in Table 2, choice students differ from the other two groups of students (control and non-complying) whom we are calling “public” students. Even though some of the differences are statistically significant, the substantive differences are modest. The overall picture is of the choice students and comparison groups being quite similar, although clearly not identical. Table 2: Demographic Characteristics of Choice and Public School Students Variable Choice Public Significance Mother’s Education (11 point scale) 7.1 7.3 0.75 Mother U.S. Born 88% 94% 0.02 Attend Religious Services (5 point scale) 3.4 3.2 0.08 Receive Food Stamps 13% 22% 0.03 Receive Welfare 6% 9% 0.26 Receive Social Security 13% 12% 0.79 Receive Supplemental Security Income 7% 6% 0.87 Receive HUD Housing Vouchers 1% 5% 0.07 Family Income (from application) $26,084 $24,714 0.24 Family Income (from survey) $23,450 $23,850 0.88 Children in Household 2.4 2.2 0.12 Family Member in Jail 1% 2% 0.15 Student Has Physical Handicap 3% 2% 0.72 Student Has Learning Disability 4% 13% 0.00 Student is a Native English Speaker 98% 97% 0.45 African-American Mother 76% 85% 0.04 Two Parent Household 42% 29% 0.01 Male Student 49% 44% 0.34 Year of Mother’s Birth 1961 1963 0.01 Mother Employed Full-Time 51% 73% 0.00 Mother Single, Never Married 23% 31% 0.00 Mother is Baptist 33% 45% 0.00 N 145-189 197-239   Significance below .05 is conventionally considered statistically significant. We employ two strategies in this study for comparing the outcomes of choice students to those of the other groups. The first strategy employs what is called a quasi-experimental research design in which observed differences between the groups that are theoretically expected to be related to the outcomes are controlled statistically.3  Because the groups are already very similar, we have less reason to fear that unobserved differences between the groups bias our estimates of the effect of using a scholarship. Concern about the unobserved differences between families that send their children to public and private schools has always limited our ability to draw conclusions from comparisons of the outcomes of students enrolled in public and private schools. Even after controlling for observed demographic differences, researchers could always wonder whether unobserved differences that were not being controlled statistically, such as parental motivation or the intellectual richness of home life, actually accounted for the differences in student outcomes instead of the schools. In our case, however, the application process and lottery have produced groups for comparison that are already quite similar on observed as well as (in all likelihood) unobserved characteristics. All families had to be sufficiently motivated to complete an application for a scholarship. All families had to be low-income to qualify for a scholarship. A lottery was used to select who would be offered scholarships, creating, as we have confirmed, two groups that were nearly identical. While non-compliance and non-participation have caused the groups we are comparing to stray from being identical in their background characteristics, they are still quite similar so that controlling for observed characteristics is likely to produce results in which we can have high confidence. The second strategy to identify the effect of using a scholarship is to use the lottery as an “instrument” to estimate who uses the scholarships that are offered.4  That is, we first predict who will use a scholarship, using whether someone won the lottery to help us make that prediction, and then we determine whether the students we predict used a scholarship have better outcomes. By using the predicted users of scholarships rather than the actual users, we remove the bias that may be introduced by the fact that the students who used the vouchers may differ (in unobserved ways) from the students who were offered a voucher but did not use them. Our estimated scholarship users will be nearly identical in their background characteristics to the groups against which we are comparing them. This technique, known as an instrumental analysis or a two-stage Heckman analysis, is a widely used strategy among economists that can produce very reliable findings. Test Score Outcomes Using these two strategies we can estimate the benefit of receiving a scholarship to attend a private school in Charlotte on student standardized test scores after one year. Using the quasi-experimental technique, we compute the effect of using a scholarship controlling for a host of background characteristics, including mother’s education, mother’s race, family income, two-parent household, and sex of student. These background characteristics are widely thought to be strongly related to student achievement in education research.5  We could control for additional background characteristics, but we would lose additional students from our analyses due to the fact that not all parents completed all questions on their surveys without gaining much explanatory power. The benefit of receiving a scholarship on students’ math scores is 5.9 percentile points at the end of the first year (see Table 3). The benefit of using a scholarship to attend a private school on reading scores is 6.5 percentile points after one year. Gains in both math and reading are statistically significant at the conventional p < .05 level. Table 3: The Effect of Attending a Private School with a Scholarship on Test Scores The Effect of Attending a Private School with a Scholarship on Math Scores   Quasi-Experimental Instrumental w/o background controls Instrumental w/ background controls Variable Effect Significance Effect Significance Effect Significance Choice 5.9 0.04 6.1 0.01 6.2 0.10 African-American Mother -13.5 0.00     -13.4 0.00 Mother’s Education 2.4 0.00     2.4 0.00 Family Income (in $5,000 increments) 2.0 0.01     2.0.01 0   Two-Parent Household 2.8 0.40     2.6 0.44 Male Student 0.4 0.86     0.4 0.87 Non-Complying Student 0.7 0.85         Constant 10.3 0.09 29.1 0.00 10.3 0.10 N 357   436   357   Adjusted R-Square 0.14   0.01   0.14   The Effect of Attending a Private School with a Scholarship on Reading Scores   Quasi-Experimental Instrumental w/o background controls Instrumental w/ background controls Variable Effect Significance Effect Significance Effect Significance Choice 6.5 0.03 5.4 0.00 7.7 0.05 African-American Mother -11.0 0.00     -10.7 0.00 Mother’s Education 2.8 0.00     2.7 0.00 Family Income (in $5,000 increments) 1.6 0.06     1.7 0.05 Two-Parent Household 10.0 0.01     9.3 0.01 Male Student -5.7 0.04     -5.7 0.04 Non-Complying Student 3.3 0.42         Constant 13.0 0.04 34.7 0.04 12.8 0.05 N 357   436   357   Adjusted R-Square 0.17   0.01   0.17   Significance below .05 is conventionally considered statistically significant. When using the instrumental analysis it is arguable that it is not necessary to control for background characteristics because we have re-captured the nearly identical comparison groups produced by the lottery to award scholarships. The advantage of not controlling for any background characteristics is that we avoid losing any cases due to missing data from the parent surveys. An instrumental analysis without controlling for any background characteristics shows the benefit of using a scholarship to be 6.1 percentile points for math and 5.4 percentile points for reading. Both results are statistically significant. The estimated effect of using a scholarship from the instrumental analysis increases somewhat if we add controls for background characteristics, although we do lose nearly 100 cases because of missing data on one or more variable. The benefit of receiving a scholarship on math scores in this analysis is 6.2 percentile points, while the benefit for reading is 7.7 percentile points. The math effect is statistically significant at p < .1 and the reading effect is significant at p < .05. The test score results across these analyses are consistently positive and significant. Having access to a private school with a scholarship improves student performance on standardized test scores by between 5.4 and 7.7 percentile points for math and reading after only one year’s time. On average, a scholarship makes the difference between students scoring in the low 30s and the high 30s. This gain is fairly large. Using within sample variance, the benefit is approximately .25 standard deviations for math and reading, which education researchers generally consider large. To put the gain in perspective, the difference between minority and white students nationwide is approximately 1 standard deviation. The benefits observed from the Charlotte CSF program are roughly one-quarter as large at the end of the first year. Parental and Student Satisfaction Another important indicator of the benefit of a program on students is how parents describe those benefits. While parents’ judgments may be distorted by the desire to affirm their decision, parents are particularly well-positioned to assess effects on their own children given how much more contextual information they have about how their children are doing. According to parents, having a scholarship to attend private school is clearly beneficial. Parents were asked to assign their child’s school a letter grade, A through F. Nearly twice as many choice parents gave their child’s school an A (53%), compared to the public school parents (26%). (See Table 4) Choice parents were also much more likely to report being “very satisfied” with virtually all aspects of their children’s school: location, safety, teaching quality, course content, class size, facilities, student respect for teachers, information on student progress, religious observance, parental support for school, discipline, clarity of school goals, teamwork among staff, teaching moral values, academic quality, and teacher respect for students. Table 4: Parental Satisfaction Variable Choice Public Significance Would Give School an A 53% 26% 0.00 Percentage Very Satisfied With…       School Location 47% 29% 0.00 School Safety 58% 32% 0.00 Teaching Quality 54% 27% 0.00 What is Taught 64% 33% 0.00 Class Size 61% 24% 0.00 Facilities 53% 25% 0.00 Students Respect Teachers 61% 31% 0.00 Information on Student Progress 60% 29% 0.00 Observe Religion 65% 25% 0.00 Parental Support for School 58% 27% 0.00 Discipline 53% 30% 0.00 Clarity of School Goals 50% 25% 0.00 Teamwork Among Staff 54% 26% 0.00 Teaching Values 62% 27% 0.00 Academic Quality 55% 27% 0.00 Teachers Respect Students 58% 26% 0.00 N 185-190 231-242   Significance below .05 is conventionally considered statistically significant. The older students who completed a survey during the testing sessions similarly reported significantly more positive assessments of their private school than did those students who did not receive a scholarship. Roughly two in five students would give their choice school an A compared to 32% of public school students. (See Table 5) When students were asked how they feel about going to school each day 24% of the public school students said that they did not want to go compared to 9% of choice students. And 24% of non-scholarship students agreed that they did not feel safe at school compared to 9% of choice students. Table 5: Student Assessments of Schools Variable Choice Public Significance Would Give School an A 40% 32% 0.05 Do Not Want to Go to School 9% 24% 0.00 Do Not Feel Safe at School 9% 24% 0.03 Strongly Agree that…       Teachers are Interested in Students 52% 28% 0.00 Teachers Listen 44% 26% 0.01 Teaches are Fair 35% 22% 0.00 Agree that…       Students Get Along with Teachers 66% 38% 0.00 N 96-98 107-109   Significance below .05 is conventionally considered statistically significant. Parental reports confirm student perceptions about safety at school. More than a third of public school parents reported problems with fighting in school (36%) compared to 16% of choice parents. (See Table 6) One-quarter of public school parents reported problems with racial conflict compared to 12% of choice parents. 22% of public school parents