Washington — When it comes to judging teacher effectiveness, value-added models—statistical models that a number of states and districts have adopted to rate teachers based on student test scores—are too problematic to be of practical use and could unfairly hurt teachers who get assignments in struggling schools.
That was the heart of the message that one of the nation’s leading experts on educational testing and assessment delivered during a recent lecture on the pros and cons of value-added models, or VAMs.
“Teacher VAM scores should emphatically not be included as a substantial factor with a fixed weight in consequential teacher personnel decisions,” said Edward Haertel, Jacks Family Professor of Education, Emeritus, at Stanford University. “The information they provide is simply not good enough to use in that way.”
One major reason it is difficult to compare teachers fairly based on VAM scores is because of social stratification within America’s schools, Haertel said.
Some teachers, he said, may teach all low-achieving students from poor families while others may teach almost exclusively high-achieving students from affluent families.
“In the real world of schooling, students are sorted by background and achievement through patterns of residential segregation, and they may also be grouped or tracked within schools,” Haertel said, noting that achievement levels vary greatly among students from different schools. “Ignoring this fact is likely to result in penalizing teachers of low-performing students and favoring teachers of high-performing students, just because the teachers of low-performing students cannot go as fast.”
To bolster his point, Haertel noted how research has found that students from affluent families tend to make reading gains over summer vacation whereas students from low-income families tend to experience what is known as “summer loss.”
Those and other non-school factors that impact student achievement far more than the teacher in front of the class render VAMs incapable of being fair measures of teacher effectiveness, Haertel said.
“No statistical manipulation can ensure a fair comparison of teachers working in very different schools with very different students under very different conditions,” Haertel said. “We cannot do a good enough job of isolating the signal of teacher effects from the massive influence of students’ individual aptitudes, prior educational histories, out-of-school experiences, peer influences, and differential summer learning loss nor can we adequately adjust away the varying academic climates of different schools.”
Haertel made his remarks Friday at the National Press Club at ETS’s Angoff Memorial Lecture. The title of the lecture was “Inferences About Teachers Based on Student Test Scores.”
The lecture comes at a time when various efforts—including the use of value-added models—are afoot throughout the nation to evaluate teachers as well as teacher prep programs based on student achievement.
In some ways, Haertel’s lecture was a synthesis of recent research that he used to underscore the problematic nature of VAMs.
For instance, he cited a 2010 Harvard study where two math teachers of questionable competence were observed making serious mistakes in math instruction—one of them concluded that 0.28 minutes was the same as 0.28 seconds—but nevertheless achieved VAM scores in the top two quartiles.
The teacher who scored in the top quartile usually offered “only the briefest of mathematical presentations” and failed in one lesson to “teach any material,” but presumably scored high because he was teaching a classroom of accelerated students, Haertel said of the study.
“These sound like the kinds of ‘bad teachers’ we might imagine VAM scores should catch,” Haertel said, “but they both sailed right through.”
Haertel said VAMs could have a negative effect on children’s education.
“These include increased pressure to teach to the test, more competition and less cooperation among the teachers within a school, and resentment or avoidance of students who do not score well,” Haertel said.
One D.C. teacher who was in the audience said a principal at the school where he used to teach seemed to be using the VAM scores arbitrarily.
“Value added is very dangerous,” the teacher said. “I remember the day when they rolled it out. We all looked at each other like, ‘This is not good.’”
If VAMs are “put in the hands of people with a low value system,” the teacher said, they could be “very punitive.”