An education study rife with flaws

(Patrick R. Gibbons/NPRI) – For decades, teacher unions and others have argued that a research study done in Tennessee in the 1980s justifies making small public-school class sizes a public policy priority — despite the millions of additional taxpayer dollars in higher costs.

Now, however, data amassed in that very same research study is coming back to undercut those same arguments.

Project STAR was a random-assignment study intended to address the question of whether children in smaller classes, everything else equal, learned more than children in larger classes.

Students were randomly assigned to three treatment groups: a small class (13-17 students), a regular class (22-25 students) and a regular class with a teacher aid. Beginning in kindergarten, student achievement was recorded for each treatment group through the third grade.

Teacher unions have celebrated the Project STAR study because it reported a statistically significant gain in student achievement for students randomly assigned to small class sizes. And thus in many states, including Nevada, the study’s results have been regularly cited in efforts to promote class-size reduction programs.

The interest of teacher-union leaders in smaller class sizes is clear: If more teachers must be hired, that greatly expands union membership, wealth and political power. Among education experts and policymakers, however, the question has remained: Are ever-smaller class sizes the best use of limited public dollars?

One reason has been that other research studies were not able to replicate the Project STAR findings. Another major problem is that most measurable effects from early-childhood education fade quickly. For preschool, most effects disappear between first and third grade, while the amount of value added in kindergarten disappears by middle school.

Recently, however, a Harvard study found evidence that long-term gains for certain Project STAR students inexplicably reappear later in adulthood. And that, while small kindergarten class sizes seem to have had a positive effect, the biggest positive effect tracks with superior kindergarten teachers.

David Leonhardt, a columnist for the New York Times, looked at the Harvard study and calculated that “a standout kindergarten teacher,” teaching a classroom of 20 students, is “worth about $320,000 a year,” as those students, when adults, will each on average earn an additional $16,000 annually.

Comparatively, an average teacher — even with a smaller class size of 13-17 students — yields only an additional earning capacity per student of $11,842.

Thus, once again, research suggests that great teachers are more effective — and a better investment — than small class sizes. While effective teachers are critically important factors in high-quality education, however, teacher quality varies widely.

The Harvard study correlated individual Project STAR participants with their later tax data. The comparison suggested that students in small class sizes and/or with good kindergarten teachers earned higher incomes later in life. However, many questions yet remain:

•The Harvard study cannot account for the effects of teacher quality in later grades and thus cannot attribute, with certainty, that it was the kindergarten teachers who were responsible for the measured effects.

•The Tennessee Project STAR study showed statistically significant gains for students in smaller class sizes than in larger class sizes, but this effect occurred only once. If small class sizes truly made a difference, the effects should have been cumulative, year after year.
Dr. Eric Hanushek of Stanford University has argued that the Project STAR study was flawed in multiple ways:

•It failed to test the achievement levels of students before the study to ensure a proper random assignment.

•It did not randomly select schools.

•It did not randomly select teachers, and

•Students who left the small-class treatment group were overwhelmingly below-average students — suggesting that the low-performing students may have been removed from small classes on purpose.

Any one of these non-random events could bias the results.

Hanushek and others suspect the one-time effect of Project STAR could have resulted from advantaged students being purposely placed into small class sizes. That is, parents lobbied to get their kids into small classes, and principals and teachers (keen to see the legislature pump more money into the system thanks to a positive study) eagerly obliged.

If true, the Project STAR study only finds that advantaged students do better in school, and the Harvard study only finds that students from advantaged families earn higher incomes. Neither result would be surprising.

Most evidence suggests that effects from preschool and kindergarten evaporate quickly. So why would the effects disappear in terms of student achievement in test scores, only to reappear later in life in terms of graduation rates and income?

As David Leonhardt and the Harvard economists acknowledge they have no explanation, perhaps they should apply Occam’s Razor — the methodological rule that the simplest answer is usually the right answer.

And that answer suggests that the STAR study was flawed from the start.

(Patrick R. Gibbons is an education policy analyst at the Nevada Policy Research Institute. For more information visit http://npri.org)