Manhattan Institute for Policy Research.
search  
 
Subscribe   Subscribe   MI on Facebook Find us on Twitter Find us on Instagram      
 
   
 

Education Working Paper
No. 6  September 2004


The Teachability Index:
Can Disadvantaged Students Learn?

Jay P. Greene, Ph.D.
Senior Fellow, Manhattan Institute for Policy Research
Greg Forster, Ph.D.
Senior Research Associate, Manhattan Institute for Policy Research

Executive Summary

Student “teachability”—the personal advantages and disadvantages that students bring to school with them—plays an important role in public discussion of education policy. Huge increases in resources are producing no improvements in student achievement: inflation-adjusted spending per pupil has doubled in the last thirty years while academic outcomes are flat. Defenders of the status quo claim the reason is that students are less teachable than they used to be; problems like poverty and social dysfunction have made the schools’ job harder. They also claim that systematic reforms like school choice and accountability testing won’t help, because students with low teachability levels can’t be expected to learn better even with reforms unless the disadvantages that students bring to school are also addressed.

These claims are rarely subjected to serious scrutiny. This study, the first of its kind, systematically measures the teachability of students by examining sixteen social factors that researchers agree affect student teachability. Combining these factors into a single Teachability Index provides the first-ever valid measurement of whether schools are facing a student population with greater challenges to learning.

The Teachability Index shows that students today are actually somewhat easier to teach than they were thirty years ago. Overall, student disadvantages that pose challenges to learning have declined 8.7% since 1970. Children’s physical health and economic security have substantially improved, and preschool enrollment has grown dramatically. While other factors have presented increased challenges—broken homes and students whose native language isn’t English are more common—these changes have been more than offset by ongoing improvements in children’s well-being. This means that student teachability cannot be a valid excuse for the failure of vastly increased spending to produce better results (see Figures 1a–1c).

The states with the highest scores on the Teachability Index were North Dakota, Maine, New Hampshire, Vermont, and South Dakota. These states had student populations with the lowest levels of disadvantages that present obstacles to learning, as well as the highest levels of advantages. The states with the lowest scores on the Teachability Index were Louisiana, Texas, Arizona, New Mexico, and the District of Columbia.

We also compare the teachability levels of students in each state with their academic outcomes. The School Performance Index gives the level of student achievement in each state expressed as a percentage of the level that would be predicted by the teachability of its students. We find that some states with low student teachability perform much better than their students’ problems would lead us to expect, while other states do not rise to this challenge (see Figure 10). This indicates that what schools do still matters even when students are facing obstacles to learning.

In particular, states with more school choice or stronger accountability testing demonstrate better school performance. Our statistical analyses find significant relationships between both of these reforms and the School Performance Index, meaning that these reforms produce higher levels of student achievement relative to student teachability.

The states with the highest scores on the School Performance Index were Montana, Colorado, Kansas, Texas, and North Carolina. Students in these states had the highest levels of academic achievement relative to their teachability—that is, these states had actual achievement levels that were the furthest above the levels we would expect to see, given the disadvantages that students faced. The states with the lowest scores on the School Performance Index were California, Alabama, Mississippi, Hawaii, and the District of Columbia.

Finally, we calculate a School Efficiency Index to determine which states are getting the best results for their education dollars. This index gives the level of student achievement in each state expressed as a percentage of the level that would be predicted by the teachability of its students and its level of education spending. We find that some states get substantially more education for each dollar they spend (see Figure 12).

The states with the highest scores on the School Efficiency Index were Utah, Idaho, Oklahoma, Kentucky, and Arkansas. Students in these states had the highest levels of academic performance relative to their teachability and the states’ education spending—that is, these states had actual achievement levels that were the furthest above the levels we would expect to see, based on their students’ teachability and their spending. The states with the lowest scores on the School Efficiency Index were Alaska, New York, Connecticut, New Jersey, and the District of Columbia.

This study indicates that teachability cannot serve as an excuse for the education system’s failure to perform, and it provides evidence that student disadvantages are not destiny: some schools do much better than others at educating students with low levels of teachability.

*********************************************

Introduction

Public discussion of education often focuses on the various kinds of advantages and disadvantages that children bring with them when they walk in the schoolhouse door. We might call this the issue of “teachability”: some students arrive at school more prepared to learn than others. Many commentators put forward confident assertions about how teachable students are and the extent to which the less teachable students can be expected to learn; low student achievement is often blamed on the students’ lack of teachability rather than on the performance of the schools. But these claims are rarely backed up by systematic evidence regarding the actual level of teachability in the student population or the ability of school systems to effectively educate students with low levels of teachability.

This is the first study to systematically evaluate both the overall teachability of the student population and the performance of schools serving students with high and low levels of teachability. There have been many previous efforts to evaluate levels of well-being among children, but these measures cannot be used as indexes of student teachability because they include student achievement alongside such factors as health and family structure. Student achievement levels reflect a combination of the teachability of students and the effectiveness of the school system. While measuring student achievement is necessary when researchers are evaluating levels of child well-being, it must be excluded from our measurements if our goal is to measure only the teachability of students, independent of the school system’s performance.

Student teachability has become a particularly important issue in debates over the merits of systematic school reforms like school choice and accountability testing. Reformers argue that the school system needs systematic reform because it is not performing as well as it should in light of the enormous increases in resources that have been poured into it: inflation-adjusted education spending per pupil has doubled in the past thirty years, while student achievement and graduation rates have remained flat.[1] Defenders of the public school system often respond by pointing to the difficulty of educating students who face disadvantages that make them less teachable, arguing that these challenges consume the system’s resources and hinder its performance.

The most prominent champion of this view is Richard Rothstein. For years, Rothstein has defended the education status quo against all types of systematic reform by arguing that social problems are the cause of inadequate student achievement. “Parents today, by a ratio of three to one, tell pollsters that all children should be held to identical standards,” he laments in one article, saying that the old myth that black children couldn’t learn had been replaced by “a new, equally dangerous one: that ... all children, regardless of background, can achieve to the same high standards if only schools demand it.” Rothstein dismisses this view: “We cannot seriously believe this. Consider how typical middle-class families raise children. Infants’ first toys are ‘touch and feel’ books. Toddlers soon ‘read’ stories from memory. Magnetic letters decorate refrigerator doors. Sitting on parents’ laps, children ‘help’ compose on computers before they can talk.” It is unreasonable, he writes, to expect schools to teach poor and minority students to read and do math up to a defined minimum standard when those students have not benefited from the superior parenting skills and resources of middle-class families.[2] In other articles, Rothstein blames low student performance on the poor health, disabilities, economic hardship, social disadvantages, and dysfunctional communities of the students.[3]

Despite his across-the-board opposition to serious reform, Rothstein is in some ways a voice of moderation on the issue of teachability because he sometimes concedes that other factors, including school effectiveness, can raise student achievement. Other mainstream education commentators actually argue that schools are helpless in the face of social problems—that schools can’t make any difference for students with low levels of teachability. “The doleful statistics that really matter are produced even before a kid gets to school,” writes Richard Cohen. “Low-income minority children are already four months behind the national average in reading and math scores by the time they arrive in kindergarten. By the 12th grade, they’re four years behind. There are many reasons for this—too much TV time, too few books in the home, the father’s taken a hike, the mother doesn’t read to her kids—but none of these factors is the fault of the schools.”[4] Cohen does not consider the possibility that one reason poor students graduate from school even further behind than they started might be that their schools are less effective than the schools attended by middle-class children. Alfie Kohn goes further still, arguing in effect that schools are already doing the best they possibly can: “Some observers ... assume that all we need are forceful demands to ‘raise the bar.’ The implication would seem to be that teachers and students could be doing a better job but have for some reason chosen not to do so and need only be bribed or threatened into improvement.” Either teachers are lazy reprobates, Kohn seems to be saying, or else there is no room at all for improvement in their performance. Kohn opposes holding schools responsible for their students’ achievement levels, arguing that “explanations about very real obstacles such as racism, poverty, fear of crime, low teacher salaries, inadequate facilities, and language barriers are sometimes written off as mere ‘excuses.’ This is at once naive and callous.”[5]

Everyone agrees that some students are more teachable than others. The issue is not whether some students have disadvantages that affect their academic performance, but how large this effect is and to what extent it can be mitigated by good schools. Specifically, the most important claims of the education reformers raise two questions that the opponents of reform must answer: Has the student population’s overall teachability truly declined dramatically in the last thirty years? And do students with similar levels of teachability actually achieve similar levels of academic performance regardless of whether their schools are excellent or mediocre?

If teachability is going to explain why large increases in spending have not produced any increases in student achievement, it must be the case that teachability has declined. It is not sufficient to show that social problems exist or that some students face great disadvantages; these things might have been equally true thirty years ago. Since the phenomenon to be explained is an increase in resources over time accompanied by student achievement that remains unchanged over time, it would have to be the case that teachability levels have declined over time in order for them to have nullified the effects of the extra revenue flowing into the system. What’s more, the increase in funds has been quite large, so the decline in teachability required to explain its failure to produce results would have to be equally large.

In fact, few of those who invoke teachability as the explanation for stagnant student achievement actually claim that teachability has declined over time. The most prominent exceptions are David Berliner and Bruce Biddle, whose book The Manufactured Crisis claims that students have higher disability rates, are more likely to speak a native language other than English, have greater health-care needs, and are more likely to be poor, rendering them less teachable than they used to be.[6] But most of these claims don’t withstand scrutiny. Berliner and Biddle present no evidence that the health-care needs of students have worsened; children’s physical health has actually improved quite a bit over the past thirty years. The evidence they put forward to show that student disabilities and poverty have worsened—evidence that has also been cited by Rothstein—is based on misleading measurements of those factors.[7] Disability levels haven’t actually changed much, while poverty has declined considerably. Of the factors cited by Berliner and Biddle, only the percentage of students not speaking English as their native language is substantially worse than it used to be (see below for further discussion).

Most important, Berliner and Biddle do not attempt to systematically measure the overall teachability of students. Instead, they isolate a few factors on which they claim conditions have worsened and present only those factors to the reader. As a result, they present a highly selective depiction of students’ teachability. An accurate evaluation of students’ overall teachability would have to include an examination of other factors, such as preschool attendance, on which conditions may have improved. The relevant question is not whether the level of poverty (or any other particular problem) has gone up but whether students are or are not less teachable on the whole than they used to be.

While the first question—how student teachability has changed over time—is relevant to evaluating  whether there is a need for systematic reform, the second question—the extent to which teachability determines final student outcomes—is relevant to evaluating whether reform can be successful. Opponents of reform claim that schools cannot be expected to produce much improvement in the academic performance of students with low levels of teachability. This view requires us to believe that a student’s teachability level imposes very restrictive limits on how much schools will be able to teach him. No matter how high-quality the school may be, it is the student’s life situation that will really determine his level of achievement.

For this view to be true, it would have to be the case that students with low teachability levels consistently end up with low levels of achievement, while students with high teachability levels consistently end up with high levels of achievement. The argument of reform opponents—expressed most directly by Cohen but also implicit in the positions taken by Rothstein and others—is that when it comes to student outcomes, the effect of school quality is swamped by the effect of student teachability. If hard-to-teach students can’t be reliably brought up to minimal levels of basic skills even by schools that have benefited from reforms, while easy-to-teach students will pick up the same skills even in mediocre schools, then clearly reform is a waste of time. But if schools can be effective in teaching even hard-to-teach students, or even if easy-to-teach students learn more when they attend more effective schools, then there is hope that reform can make a difference in student achievement levels. The way to test these hypotheses is to see whether student achievement levels vary relative to student teachability levels. If students with similar teachability levels do substantially better in some places than in others, then school quality probably makes a difference in student outcomes.

Method

The Teachability Index

To evaluate the level of students’ teachability, we developed an index combining measurements of numerous factors that affect students’ capacity to learn. Of course, a comprehensive index of every relevant factor would be impossible. This index collects sixteen factors that are very important for determining how teachable students are. While this is not a perfect measurement of teachability, it is a reasonably good measurement that will allow us to track upward and downward trends in teachability and to get an idea of their magnitude.

The Teachability Index is made up of six component indexes. These are the Readiness Index, the Economics Index, the Community Index, the Health Index, the Race Index, and the Family Index. Each of these indexes is made up of a number of factors—such as family incomes, single parenthood rates, and preschool attendance rates—affecting students’ teachability. We track these factors from 1970 to 2001 in order to measure teachability over the period during which inflation-adjusted education spending has doubled. Where data were not available for all years, data for missing years were imputed from available data; where data were not available all the way back to 1970, the earliest available figure was imputed to previous years.[8]

First, we converted each of the sixteen factors into a standardized scale. On this scale, the value for each year is set to the difference between that year’s value and the original 1970 value expressed as a percentage of the original 1970 value. That is, the difference between that year’s value and the 1970 value is divided by the original 1970 value, and the resulting fraction is then multiplied by 100 and expressed as a percentage. Using this method, we always produce a value of zero for the year 1970 itself, followed by upward or downward movement based on subsequent changes in the actual value. If an increase in a given factor would result in greater student teachability (e.g., family income or preschool enrollment), the difference is obtained by subtracting the original 1970 value from the later year’s value; thus, increases in the factor will result in positive index scores while decreases will result in negative index scores. But if an increase in a given factor would result in lower student teachability (e.g., disabilities or single parenthood), the difference is obtained by subtracting the later year’s value from the 1970 value, so that an increase in the factor would result in a negative index score, and vice versa.

For example, in 1970 the violent-crime victimization rate for children ages 12 to 19 was 81.75 victims per 1,000 persons. An increase in violent crime would result in lower student teachability, so for each subsequent year we calculated the difference by subtracting the later year’s victimization rate from 81.75. Then we divided this difference by 81.75 and multiplied the resulting fraction by 100. For 1970, the standardized value is zero, of course, since there is not yet any change. In 1978, the violent crime rate had increased to 87.7; subtracting this from 81.75 gives us a difference of -5.95; dividing this by 81.75 gives us -0.073; multiplying by 100 gives us a standardized value of -7.3%. In 1994, the victimization rate was 121.25; subtracting this from 81.75 gives us a difference of -39.5; dividing this by 81.75 gives us -0.483; multiplying by 100 gives us a standardized value of -48.3%. In 2001, the victimization rate was 55.5; subtracting this from 81.75 gives us a difference of 26.25; dividing this by 81.75 gives us 0.321; multiplying this by 100 gives us a standardized value of 32.1%.

The Teachability Index

Readiness

Economics

Preschool enrollment

Income

Language other than English

Poverty

Parents’ education

 

Community

Health

Crime victimization

Disabilities

Drug use

Mortality

Religious observance

Low birth-weight survival

Residential mobility

Suicide

Race

Family

Non-Hispanic white

Teenage birth

 

Single parenthood

Once we converted the factors into this standardized scale, we calculated the six component indexes by taking the average of the factors assigned to that index. For example, the Family Index is the average of the selected measurements for teenage birth and single parenthood, while the Community Index is the average of the selected measurements for crime victimization, drug use, religious observance, and residential mobility. The overall Teachability Index was calculated by taking the average of the six component indexes.

We used unweighted averages to compute the six component indexes and the main Teachability Index because there was no valid a priori method for weighting the factors based on their relative importance. The measured factors that make up the index were chosen based on preexisting research consensus on what affects the teachability of students. No doubt some of these factors have a larger effect than others, but any attempt to account for this by weighting some factors more heavily than others in our calculations could not be done scientifically and would only reflect our subjective opinions. Because we had no valid way to weight the various factors, the final Teachability Index is only a rough indicator of student teachability. Aware of this concern, we did “reality checks” to ensure that the index is a valid measure of the teachability of students (see below).

The Readiness Index measures the amount of academic preparation and support that students receive before or outside of school. It includes three factors: preschool enrollment, children speaking native languages other than English, and parents’ education. Our measurement of preschool enrollment is the full-time equivalent rate of enrollment in preprimary academic programs for children ages three and four, taken from U.S. Department of Education data.[9] Students who attend preschool enter the K-12 system more academically prepared. Our measurement of non-English native languages is the percentage of children ages five to 17 who speak another language at home and have difficulty speaking English, taken from U.S. Census data.[10] Students whose first language is not English will have greater difficulty reaching the same level of achievement in reading and writing English as native speakers and may have greater difficulty learning other subjects as well. Our measurement of parents’ education is the percentage of children ages five to 17 whose parents have at least a bachelor’s degree, based on U.S. Department of Education data.[11] Children of better-educated parents are more likely to receive early mental stimulation and ongoing academic support at home.

The Economics Index measures the material well-being of students. Greater family wealth leads to higher levels of student teachability because students from wealthier families will have fewer material challenges in their lives to hinder their learning and will receive more material support to aid their learning (in the form of academic supplies, tutoring, etc.). The Economic Index includes two factors: income and poverty. Our measurement of income is the median family income in constant 2001 dollars, based on U.S. Census data.[12] This provides a broad-based measurement of the material well-being of students’ families. Our measurement of poverty is the mean family income for families in the lowest quintile of income, based on U.S. Census data.[13] This allows us to specifically track the material well-being of poor families.

It is important to separately track the income of poor families, since material well-being presents the largest challenge for those families, and their incomes may or may not change in the same way or at the same rate as those of the general population. However, it is important to define poverty in the right way. The “poverty lines” set by the U.S. Census and many other statistical sources change over time, going up when society as a whole gets richer and going down when society as a whole gets poorer. The same material standard of living might have been considered somewhat wealthy in 1900, middle class in 1950, and borderline poor in 2000. If our purpose were to measure the social phenomenon of poverty, it would be appropriate simply to ask what percentage of people live below the poverty line. But when our purpose is to measure people’s actual material well-being rather than their social class as such, we should not ask how many families are poor but rather how much income families on the lowest end of the spectrum are making. The average poor person in 1900 lived at a much lower level of material well-being than the average poor person in 2000, but if we examine the poverty rate we will not see this, since the definition of what counts as “poverty” has changed. Our measurement of poverty provides information on the changing state of material well-being among those who have the least amount of wealth.

Berliner and Biddle present misleading data on this point. When explaining the effects of poverty on teachability, they speak in terms of material well-being: “It is very difficult to provide good schooling for impoverished students who may come to school hungry or in cast-off and torn clothing, who suffer from untreated medical problems, who live in neighborhoods that are rife with crime violence, or who come from homes that lack even basic amenities—let alone books and other supports for education.”[14] But they begin their analysis of poverty by presenting data based on relative rather than absolute poverty, defining a family as poor if its income is less than half the median income for all families.[15] Similarly, Rothstein claims that higher relative poverty rates mean that more students are poorly fed, but he presents no evidence on actual nutrition levels or changes in wealth among the poor.[16] As society as a whole gets richer over time and the line for what counts as “poverty” goes up, measuring poverty in this way will not reveal that poor families are, in fact, materially better off (better fed, better clothed, having more access to health care, etc.) than they used to be. Berliner and Biddle do also present data on absolute changes in wealth among the poor, but here their data are misleading for a much simpler reason: they arbitrarily begin their analysis in 1977 and end it in 1988.[17] Between those two particular years, the well-being of the poor did go down a little, but restricting the analysis to this narrow period obscures a longer-term trend in the opposite direction (see below).

The Community Index measures the presence of helpful and harmful social influences in children’s lives. It includes four factors: crime victimization, drug use, religious observance, and residential mobility. Our measurement of crime victimization is the number of children ages 12 to 19 per 1,000 in the population victimized by violent crime, based on U.S. Department of Justice data.[18] Students who have suffered the trauma of victimization are likely to have more difficulty learning. Our measurement of drug use is the percentage of 12th graders who have ever used illicit drugs, based on U.S. Department of Health and Human Services data.[19] Students who use drugs will be more difficult to teach because of the harmful effects of drug use. Our measurement of religious observance is the percentage of the population attending religious services either every week or almost every week, based on National Science Foundation data.[20] When more families attend religious services, students are more exposed to social influences that will make them more teachable, such as supportive communities and positive behavioral norms. Our measurement of residential mobility is the percentage of people in the population that have changed residences in the previous year, based on U.S. Census data.[21] When families move, children are separated from their social networks and familiar surroundings, an emotional strain that reduces their teachability.

The Health Index measures the physical and mental well-being of students. It includes four factors: disabilities, mortality, low birth-weight survival, and suicide. Our measurement of disabilities is the percentage of students placed in special-education programs with disability diagnoses other than specific learning disabilities, based on U.S. Department of Education data.[22] Disabled students face a variety of challenges to learning. Our measurement of mortality is the number of children ages 14 and under per 100,000 in the population who died, based on U.S. Department of Health and Human Services data.[23] Mortality provides a broad-based indicator of the  level of physical health in the population; when children’s mortality rates go up, this indicates that child health has worsened, and children thus will have more difficulty learning. Our measurement of low birth-weight survival is the percentage of all babies born with birth weights below 2,500 grams multiplied by the percentage of all babies in that birth-weight category who do not suffer infant death, based on U.S. Department of Health and Human Services data.[24] Babies with low birth weights are more likely to develop health problems that interfere with learning; the number of low-birth-weight babies that survive would therefore have an effect on the teachability of the student population. Our measurement of mental health is the number of children ages 15 to 19 per 100,000 in the population who commit suicide, based on U.S. Department of Health and Human Services data.[25] Suicide rates provide an indicator of children’s level of mental health; when suicide rates go up, this indicates that mental health has worsened, and students thus will have more difficulty learning.

Our measurement of disabilities differs from the normal way that disabilities are measured in that we exclude specific learning disabilities. We do this to address a problem with the data: the only easily available measurement of student disabilities is the number of students assigned to special education, but there is good reason to believe that factors other than actual student disabilities are affecting these assignments. The overall percentage of students assigned to special education has grown astonishingly: from 8.3% in 1976-77, when federal programs for disabled students had just been enacted, to 13.3% in 2000-01, an increase of 60%.[26] Berliner and Biddle cite this growth as evidence that students are less teachable than they used to be, as does Rothstein.[27] But there is no plausible explanation consistent with the available data for why disabilities would have risen during this period, and there is strong evidence indicating that most of the growth in special-education assignments is attributable to factors other than real growth in disabilities.

Sheldon Berman, Perry Davis, Ann Koufman-Frederick, and David Urion claim that there has been real growth in student disabilities. They attribute this alleged growth to more frequent survival of babies with neurologically harmful birth defects, deinstitutionalization of children with severe disabilities, and increases in childhood poverty.[28] But this account is not consistent with the facts. The number of babies expected to develop retardation due to birth conditions has indeed grown. But, crucially, the total number of students classified as mentally retarded has undergone a dramatic drop—from about 961,000 in 1976-77 to about 599,000 in 2000-01.[29] Any growth in neurological disorders caused by increased numbers of surviving babies with birth defects has been more than offset by improvements in the prevention of such disorders in other areas, such as reduced exposure to lead paint. Deinstitutionalization also can’t be driving the growth of special-education enrollment, because the growth has not occurred among students diagnosed with the kind of severe disabilities that would cause a child to be institutionalized.[30] As for childhood poverty, it hasn’t actually increased. The percentage of children under six living in poverty was 18.1% in 1977 and 18.5% in 2002, a trivial difference compared with the 60% increase in special-education enrollment, especially when we consider that the material well-being of the poor underwent a small net increase over the same period.[31] What’s more, the childhood poverty rate has gone up and down while special-education diagnoses have grown smoothly and steadily.

If there is no plausible explanation for why student disabilities would have increased over the past quarter-century, how do we explain rising enrollment in special-education programs? Some of the growth may be attributable to better diagnosis of existing disabilities, but probably not much of it because very little of it has occurred in disability categories where significant improvement in diagnosis is likely to have occurred.[32] Unfortunately, the dominant cause of rising special-education enrollment appears to be perverse financial incentives from the special-education funding system. In most states, school districts receive more funding when their special-education enrollments go up. This provides districts with a financial reward for placing students in special education, a process that some educators frankly refer to as “the bounty system.” Defenders of the process claim that these funding increases are offset by increased costs. However, some of the “costs” of serving special-education students do not reflect true increases in spending but only a reclassification (from “regular education” to “special education”) of expenditures that were going to be made anyway. For this reason, the bounty system does provide positive financial incentives to place students in special education regardless of whether they are truly disabled. In a previously published study, we found that special-education enrollment grew significantly faster in the 1990s in states with bounty funding systems than in states that did not increase schools’ funding if their special-education enrollments went up. The difference was so large that 62% of the growth of special-education enrollment in bounty-system states was attributable to funding incentives. This accounts for about 390,000 extra students in special-education programs.[33]

To exclude the influence of artificial student labeling driven by perverse incentives, our measurement of student disabilities excludes students placed in the category of specific learning disabilities. Diagnosis in this category is substantially more subjective than in other categories—a student is considered to have a specific learning disability if his actual academic performance does not live up to his potential, and this discrepancy is not attributable to environmental factors such as poverty, family problems, or poor teaching. This diagnostic formula calls on schools to make subjective judgments both as to whether there is a discrepancy between a student’s potential and his performance and as to whether that discrepancy is caused by environmental factors. The inherent subjectivity of the diagnosis makes this category particularly susceptible to artificial labeling. The percentage of all students who are diagnosed with specific learning disabilities has tripled, growing from 1.8% in 1976-77 to 6.0% in 2000-01.[34] There is no logical explanation for why a real explosion of learning disabilities would have taken place during this time.

Excluding specific learning disabilities is not a perfect solution to this problem. The remaining categories are not immune from the potential for artificial labeling, and it is possible that some amount of the growth of specific learning-disability diagnoses represents real growth. Nonetheless, this solution is still a good one. We are only excluding the one category that is most susceptible to this significant problem; if there has been real growth in disabilities, it would be bizarre for that growth to occur only in one category. In particular, anything that might plausibly cause an increase in learning disabilities—some pollutant in the water, for example—would be likely to cause growth in other mental disabilities as well.

The Race Index measures the changing racial composition of the student population. Research has shown that minority students face greater disadvantages that pose special challenges for educating them. Unlike our other component indexes, the Race Index has only one factor: the percentage of the population that is non-Hispanic white, according to U.S. Census data.[35]

The Family Index measures the extent to which family structures impose educational challenges on children. It includes two factors: teenage birth and single parenthood. Our measurement of teenage birth is the percentage of all live births in which the mother is under age 18, based on U.S. Department of Health and Human Services data.[36] Teenage births reduce the teachability of the student population because children raised by teenage parents face greater difficulties in a number of ways and because teenage students who are themselves mothers will be carrying the enormous burden of motherhood while also pursuing their studies. Our measurement of single parenthood is the percentage of all children under age 18 who are not living with both parents, based on U.S. Census data.[37] Research has shown that children raised without both parents in the home face significant challenges that impede learning.

We compared changes in teachability over time with the financial inputs and educational outputs of the public school system. This allows us to test the proposition that changes in teachability explain why increased spending has not been associated with higher test scores. While the Teachability Index is not precise enough to allow for detailed statistical analysis of this question, it does give us an idea of the direction in which teachability has moved and the rough magnitude of the movement. Our measurements of academic achievement are 12th grade reading and math achievement levels on the longitudinal version of the National Assessment of Educational Progress (NAEP), also known as the Nation’s Report Card.[38] We also included data on the graduation rate, since changes in academic achievement for students who remain in the school system should be understood in the context of changes in the rate at which students leave the system. Our historical measurement of the graduation rate is the number of regular high-school diplomas awarded divided by the 17-year-old population.[39] For comparison purposes, we standardized the data on spending, achievement, and graduation rates the same way that we standardized the factors that went into the Teachability Index.[40]

Teachability by State

We also developed a version of the Teachability Index that allows us to compare teachability across states. We sought out state-by-state data for each of our sixteen indicators. Wherever possible, we used data from the year 2001; where state-by-state data for that year were not available, we used the closest available year, which was usually 2000. In some cases, we had to modify the measurement we used for a factor because state-by-state data for our preferred measurement were not available; in each case, we found a similar measurement for which data were available.

This index uses a different scale that allows us to compare states with one another rather than measuring performance over time. Whereas in the main Teachability Index zero represents the value of a factor in 1970, in the state-by-state version of the Teachability Index zero represents the average of the values for all states. First we divided the value of a given factor (mortality, preschool attendance, etc.) in each state by the average state value of that factor. In cases where an increase in the factor would lead to lower rather than higher student teachability, we did the reverse, dividing the average value by each state’s value. This gave us a standardized measurement in which each factor is expressed as the ratio of a particular state’s value to the average value. We then computed values for each of the six component indexes by taking the average value of the factors within each component index. For ease of presentation, we then divided each state’s component index value by the average of the component index values for all states, subtracted one from the result and then multiplied by 100; this gave us final values centered on zero and scaled such that the index value for a state is equal to the percentage by which it is better or worse than the average for all states. We then took the average of the six component indexes to get the Teachability Index by state.

For example, in 2001 the median family income in Virginia was $57,256, and the average state median family income was $50,304. Dividing $57,256 by $50,304 we get 1.14. Virginia gets a score of 1.25 on the poverty measurement, producing a preliminary Economics Index value of 1.19 (the average of 1.14 and 1.25). The average preliminary Economics Index score for all states was 1.04, so we divide 1.19 by 1.04 to get 1.143. Subtracting one and multiplying by 100 gives us a final Economics Index value of 14.3. This indicates that the economic well-being of students in Virginia is better than the average for all states by 14.3%. Averaging this with Virginia’s scores on the five other component indexes, we get a final Teachability Index score of -3.2, indicating that the teachability of Virginia students is less than the average for all states by 3.2%.

In the Readiness Index, our measurements for preschool attendance and native languages other than English remained unchanged.[41] Our measurement of parents’ education is the percentage of parents living with their own children under 18 who have at least a bachelor’s degree, based on U.S. Census data.[42]

In the Economics Index, our measurement of income remained unchanged.[43] Our measurement of poverty is the percentage of all families with incomes below $15,000, based on U.S. Census data.[44] Since changing standards of poverty over time are not a factor in this analysis, it is appropriate to measure poverty by a fixed benchmark.

In the Community Index, our measurement of crime victimization is the average of the annual numbers of homicide victims ages 14 to 17 per 100,000 in the population during the 1990s, based on U.S. Department of Justice data.[45] It was necessary to take the average over the decade because numerous states were missing data for the most recent years. Our measurement of drug use is the percentage of children ages 12 to 17 who used an illicit drug in the previous month, based on U.S. Department of Health and Human Services data.[46] Our measurement of religious observance is the percentage of the population attending religious services at least once per week, based on data obtained from the Pew Research Center for the People and the Press.[47] Our measurement of residential mobility is unchanged.[48]

In the Health Index and Race Index, the measurements of all indicators remained unchanged.[49] In the Family Index, the measurement of teenage birth is the number of births to mothers ages 15 to 17 per 1,000 girls ages 15 to 17 in the population, based on U.S. Department of Health and Human Services data.[50] Our measurement of single parenthood remained unchanged.[51]

Validity of the Teachability Index

To ensure that the Teachability Index truly measures student teachability, we ran five regression analyses to examine the relationship between the Teachability Index and academic outcomes in each state. Our measurements of academic outcomes were scale scores for NAEP math and reading, the percentage of students achieving the “basic” level or better in NAEP math and reading, and the high school graduation rate. We used the 2003 NAEP results because this was the first year in which all states participated in the NAEP. Ideally, we would have used 12th grade results, since these represent the final output of the education system, but 12th grade results are not available at the state level, so we used eighth-grade results instead. Our state-by-state measurement of the graduation rate uses enrollment data collected by the U.S. Department of Education, dividing the number of diplomas awarded in a given year by an estimate of the number of students who should have graduated based on enrollment data for earlier grades. We omit a more detailed description of this method, as it has been published elsewhere; for this study, we used our previously published state-by-state calculations of the 2001 graduation rate.[52] We controlled for the level of education spending in each state.[53]

The School Performance Index

Once we have a state-by-state index of teachability, it becomes possible to evaluate the educational performance of schools based on the teachability of the students they serve. We developed the School Performance Index to measure how well each state is doing at teaching its students, given the educational challenges presented by their student populations. First, using a regression analysis we calculated predicted NAEP achievement levels (specifically, the percentage of students achieving the “basic” level) in math and reading based on that state’s Teachability Index value.[54] Then we took the average of the state’s actual NAEP math and reading achievement levels and divided this by the average of the state’s predicted NAEP math and reading achievement levels. This gives us the actual level of students’ educational achievement in each state expressed as a percentage of the achievement predicted by the students’ teachability.

For example, our regression analysis predicts that, based on what the Teachability Index tells us about the teachability of students in Illinois, about 70.7% of Illinois students will achieve at the basic level in reading and about 63.9% in math on the NAEP. Illinois’ actual achievement levels on the NAEP were 77% in reading and 66% in math. Thus the state’s average actual performance was 71.5%, while its average predicted performance was about 67.3%. Dividing 71.5 by 67.3 we get a School Performance Index of 106%, meaning that Illinois students performed at 106% of the level that would be predicted based on their teachability.

We can use the School Performance Index to test the claim that school quality doesn’t make much difference because student teachability determines academic outcomes. If this claim is true, there should be a high correlation between states’ values on the Teachability Index and the School Performance Index. That is, states with highly teachable students should reliably produce high academic performance, and states whose students are difficult to educate should reliably produce low academic performance. However, if school quality makes a difference despite student teachability levels, there should be less correlation between the two indexes. We used the Pearson’s correlation method to measure the relationship between variations in the Teachability Index and variations in the School Performance Index.

This index measures the relative performance of the states, not their absolute performance. That is, it tells us which states are performing better than others and by how much, but not whether that level of performance lives up to some objective standard. Thus the School Performance Index gives us no information on the question of whether school systems are currently “good enough,” but it does allow us to compare states with one another and determine whether their various education policies make a substantial difference in outcomes.

The Effect of School Choice and Accountability on School Performance

The School Performance Index gives us information on the variation in student achievement relative to teachability among state education systems. It therefore allows us to measure the relative effectiveness of the education policies that have been adopted in various states. By examining the statistical relationship between the School Performance Index and indexes of reform policies like school choice and accountability, we can determine whether these policies are helping schools rise above the level of performance that we would expect, given the teachability of their students.

We ran two linear regressions to measure the effect of school reforms on the School Performance Index. In the first analysis, we used the 2001 edition of the Education Freedom Index, our previously published index of the availability of school choice in each state. The Education Freedom Index takes into account four types of school choice: charter schools, public school choice, subsidies for private school choice, and home schooling. We omit a description of our method for calculating it, as it has been published elsewhere.[55] In the second analysis, we developed the Accountability Index, an index that measures the strength of accountability testing in each state. We calculated this index by standardizing and then averaging the values of four existing indexes of accountability, published by: the Fordham Foundation, Education Week’s “Quality Counts,” Martin Carnoy and Susanna Loeb, and Audrey Amrein and David Berliner.[56] Assessing the strength of widely varying accountability policies requires a certain amount of subjective judgment; by averaging several existing indexes, we were able to remove much of the subjectivity involved in this process.

The School Efficiency Index

We are also able to use the state-by-state version of the Teachability Index to measure which states are getting good value for their education dollars. To calculate the School Efficiency Index, we first divided each state’s School Performance Index value by its per-pupil education spending. This gives us a percentage figure that measures how much achievement the state is buying per dollar relative to the value predicted by the teachability of its students. Then we multiplied this number by the average state per-pupil spending figure, which was $8,542. This gives us the achievement level of the state’s students expressed as a percentage of the achievement predicted by the students’ teachability and the state’s spending level.

For example, Ohio has a School Performance Index value of 107%. Dividing 107 by the state’s per-pupil spending level of $8,898, we get about 1.2%. Multiplying this by the national average spending figure of $8,542, we get a School Efficiency Index of 103%. This means that Ohio students performed at 103% of the level predicted by their teachability and the state’s spending level.

We also calculated an Adjusted School Efficiency Index to account for differences in the cost of living in each state. We obtained COLA factors for each state that allowed us to adjust for the cost of living.[57] The calculation was performed by dividing our spending figure by the state’s COLA factor and then multiplying by 100. We then recalculated the index just as we had for the main School Efficiency Index.

Results

The Teachability Index

Table 1 and Figure 1a present the Teachability Index for 1970–2001. Teachability levels remained effectively unchanged from 1970 through 1990, with the index rising no higher than 1.3% and falling no lower than -1.0%, a negligible field of movement. The student population then underwent a modest decline in teachability from 1990 through 1993, reaching its lowest index level at -4.0%. Starting in 1994, the teachability level began a gradual upward trend that has continued through at least 2001, leaving the Teachability Index at 8.7% in that year. Thus students are somewhat more teachable now than they were in 1970.

Values for the six component indexes are presented in Table 1 and Figure 1b. These allow us to account for changes over time (or lack thereof) in the Teachability Index. During the initial twenty-year period of stable teachability, gradual improvement in the Readiness Index was roughly counterbalanced by a gradual decline in the Family Index. The movement in both these indexes tapered off in the late 1970s, and both were relatively flat during the 1980s. Meanwhile, the Economics Index provided modest and inconsistent improvements in teachability that were roughly counterbalanced by the combined weight of small declines in the remaining three component indexes. The small decline in the overall Teachability Index from 1990 to 1993 resulted primarily from a resumption of downward movement in the Family Index. The periodic downward oscillation of the Economics Index also contributed. The reversal of the Teachability Index’s decline after 1994 was driven by simultaneous improvement in the three most prominent indexes: a renewal of growth in the Readiness Index, a reversal of the downward trend in the Family Index, and the periodic recovery of the Economics Index.

Figure 1c compares the movement over time of school spending, academic achievement, graduation rates, and student teachability. This allows us to test whether changes in student teachability are a plausible explanation for the failure of spending increases to produce better academic outcomes. As the figure shows, the increase in school spending in the past thirty years has been enormous compared with the very small movements in student teachability. Furthermore, the ultimate movement in teachability has been upward, which is inconsistent with teachability serving as an explanation for why vast increases in spending have not increased academic achievement.

Figures 2 through 7 provide complete breakdowns of each of the six component indexes. The Readiness Index was most prominently characterized by fairly steady growth in preschool attendance and (to a lesser extent) parents’ education, partially offset by increases in students whose native language was not English. The Economics Index saw closely parallel cyclical movements in income and poverty. The factors making up the Community Index roughly balanced one another out until the last few years, during which time a dramatic reduction in crime victimization contributed to a net improvement in the index. Improvement in physical health (as measured by mortality) was offset by a decline in mental health (as measured by suicide rates), keeping the Health Index relatively stable for most of the period observed. Mental health reversed its downward trend after 1990, contributing to improvement in the Health Index later in the observed period. The Race Index saw a modest gradual decline throughout the period. The Family Index was driven downward by deterioration of two-parent families, an effect somewhat mitigated by ongoing decline in teenage birth rates. When the decline of two-parent families tapered off in the late 1990s, this allowed the continuing improvement in teenage birthrates to pull the Family Index back upward.

Teachability by State

Teachability values for the states are provided in Table 8, and a ranking of the states in order of student teachability is provided in Figure 8. The states with the highest scores on the Teachability Index were North Dakota, Maine, New Hampshire, Vermont, and South Dakota. These states had student populations with the lowest levels of disadvantages that present obstacles to learning as well as the highest levels of advantages. The states with the lowest scores on the Teachability Index were Louisiana, Texas, Arizona, New Mexico, and the District of Columbia. Northern states tended to have high rates of student teachability, while western and southern states tended to have low rates of student teachability.

Validity of the Teachability Index

The results of our two historical analyses of the Teachability Index’s validity are provided in Table 9. All five analyses found positive and statistically significant relationships between the Teachability Index and academic outcomes. These relationships exist at an extremely high level of statistical certainty (the p-values for all five analyses rounded to zero even when read out to thousandths of a point), so we can be very certain that we are detecting relationships that are not produced by chance. This indicates that the Teachability Index does reflect real levels of student teachability.

The School Performance Index

State scores on the School Performance Index are provided in Table 10, and a ranking of states based on their school performance is provided in Figure 10. Montana led the nation with a School Performance Index of 112%, indicating that Montana students have achieved a level of academic performance equal to 112% of the level predicted by the teachability of its students. Colorado was close behind, with a score of 111%, followed by Kansas and Texas with 110%. The District of Columbia trailed the pack by a large margin, with a School Performance Index score of only 64%. The District may have students with great disadvantages, but its schools perform much worse than would be expected, even given those disadvantages. Among the states, Hawaii had the lowest performance, with an index score of 83%, followed by Mississippi at 84% and Alabama at 87%.

The variation in states’ scores on the School Performance Index suggests that schools do make a difference in spite of social problems. The view that disadvantaged students are unable to learn regardless of what schools do—a view expressed by Cohen and implicitly endorsed by many other education commentators—implies that there should be very little variation in the School Performance Index. If schools are helpless to make a difference in the face of larger social forces shaping student outcomes, every state’s index score should be close to 100%. In fact, many states produce students who perform well above the level of achievement that we might expect, based on the challenges their students face, while others fall well below expectations. North Carolina’s students rank 43rd in the nation for teachability but perform at 109% of expectations, and Texas’s students rank 48th in teachability but perform at 110% of expectations. These states have successfully overcome poor student teachability to a much larger degree than New Mexico, Louisiana, and Nevada. Meanwhile, Hawaii’s students rank 19th in the nation for teachability but its schools bring them up to only 83% of what their teachability would predict, and Maine performs at 88% of expectations with students who rank second in the nation for teachability.

These naked-eye observations are confirmed by the result of our correlation analysis. We found that the state-by-state Teachability Index and the School Performance Index are correlated at 0.067. In other words, only 6.7% of the variation in the School Performance Index is attributable to variation in the Teachability Index. This is a very low correlation that is not statistically significant. It appears that most of the variation in states’ academic performance is attributable to factors other than the teachability of their students.

The Effect of School Choice and Accountability on School Performance

The results of our analyses of the effect of school choice and accountability on the School Performance Index are provided in Table 11. Both the Education Freedom Index and the Accountability Index had positive and statistically significant relationships with school performance. This indicates that states implementing these reforms produced higher levels of student achievement relative to student teachability levels. Our results for accountability testing are also supported by the noticeably high School Performance Index scores of states that have implemented this type of reform, including Colorado (#2), Texas (#4), North Carolina (#5), Virginia (#6), Ohio (#9), New York (#11), Illinois (#12), and Massachusetts (#15). Other prominent testing states did less well, but the overall tendency seems to be positive.

The School Efficiency Index

The School Efficiency Index is provided in Table 12, and a ranking of states based on their school efficiency is provided in Figure 12. Utah was the most efficient state; although the School Performance Index shows that it reached 98% of expectations based on its students’ high teachability, it accomplished this on very low spending ($5,710 per pupil, compared with a national average of $8,542) and thus received a School Efficiency Index score of a whopping 146%, meaning that it performed at 146% of what we would expect, given its students’ teachability and its spending level. Following Utah were Idaho at 138%, Oklahoma at 137%, and Kentucky and Arkansas at 134%. At the bottom of the efficiency scale was the District of Columbia, which achieved dismal academic results (64% of expectations) while spending a stratospheric amount of money ($15,249 per pupil), earning a catastrophic School Efficiency Index score of 36%. The next lowest in efficiency were New Jersey with a score of 70%, Connecticut with 73%, and New York with 74%.

The Adjusted School Efficiency Index is provided in Table 13, and a ranking of states based on it is provided in Figure 13. Adjusting for the cost of living did not bring about large-scale changes in the results of the index. Utah remained at the top of the index despite a COLA factor indicating that its local prices were somewhat lower than the national average. Idaho, Oklahoma, and Kentucky remained near the top, though Montana managed to crawl over them from sixth place to second place. The lower end of the index remained more or less unchanged as well; Michigan moved down and New York moved up when cost of living was taken into account.

Conclusion

While not a perfect measurement of student teachability, the Teachability Index gives us a reasonably accurate picture of the advantages and disadvantages that students bring with them to school, how those challenges have changed over time, and how they vary from place to place. The index was stable for most of the past thirty years and its net movement has been upward, both of which undermine the claim that low student teachability is a valid excuse for the school system having produced no improvements in academic achievement despite large increases in spending. While defenders of the educational status quo make excuses for the system by pointing to isolated factors that have grown worse—or that they claim have grown worse—the evidence indicates that on the whole, students are easier to teach today than they have been at any time in the past thirty years.

States with low scores on the index do not inevitably produce low-performing students, and states with high scores do not inevitably produce high-performing students. Instead, the relationship between teachability and student performance varies considerably, suggesting that schools are not, in fact, helpless in the face of obstacles to student learning. Some schools rise to the challenge of teaching disadvantaged student populations while others do not. In particular, school choice and accountability testing both lead to higher student performance relative to student teachability levels.

In explaining school outcomes, education experts have long stressed school inputs—money and students’ backgrounds—often to the exclusion of other factors. These indexes suggest that what schools do makes a big difference in how much students learn, independent of inputs to the system.

 


Center for Civic Innovation.

EMAIL THIS | PRINTER FRIENDLY

EWP 6 PDF (631 kb)

WHAT THE PRESS SAID

SUMMARY:
Student “teachability”—the advantages and disadvantages that students bring to school—is often offered as an excuse for educational failure. Many claim that students are less teachable than they used to be, and that reforms cannot meaningfully improve student achievement due to problems like poverty and social dysfunction. This study measures student teachability by examining sixteen key social factors that affect learning.

TABLE OF CONTENTS:

Executive Summary

About the Authors

Acknowledgements

About Education Working Papers

Introduction

Method

Results

Conclusion

Endnotes

Appendix

Table 1: The Teachability Index

Figure 1a: The Teachability Index

Figure 1b: Component Indexes

Figure 1c: Teachability, Spending, and Achievement

Table 2: The Readiness Index

Figure 2: The Readiness Index

Table 3: The Economics Index

Figure 3: The Economics Index

Table 4: The Community Index

Figure 4: The Community Index

Table 5: The Health Index

Figure 5: The Health Index

Table 6: The Race Index

Figure 6: The Race Index

Table 7: The Family Index

Figure 7: The Family Index

Table 8: Teachability by State in 2001

Figure 8: Ranking of States by Teachability in 2001

Table 9: Validity of the Teachability Index

Table 10: The School Performance Index for 2001

Figure 10: Ranking of States by School Performance Index in 2001

Table 11: The Effect of School Choice and Accountability on School Performance

Table 12: The School Efficiency Index for 2001

Figure 12: Ranking of States by School Efficiency in 2001

Table 13: The Adjusted School Efficiency Index for 2001

Figure 13: Ranking of States by Adjusted School Efficiency in 2001

 


Home | About MI | Scholars | Publications | Books | Links | Contact MI
City Journal | CAU | CCI | CEPE | CLP | CMP | CRD | ECNY
Thank you for visiting us.
To receive a General Information Packet, please email support@manhattan-institute.org
and include your name and address in your e-mail message.
Copyright © 2009 Manhattan Institute for Policy Research, Inc. All rights reserved.
52 Vanderbilt Avenue, New York, N.Y. 10017
phone (212) 599-7000 / fax (212) 599-3494