Today on NPR’s Morning Edition, I heard a report about how the LA Times is going to release the test scores of LA teachers on the internet so that parents can make informed decisions about their child’s teachers.
Here was the intro to that report (taken from the transcript online):
Any day now, 6,000 elementary teachers in Los Angeles will see their names published online, along with data showing how much their students improved on standardized tests.
The Los Angeles Times has promised to release the information to help parents measure teacher effectiveness. The database has sparked a national debate on how to evaluate teachers.
The Times stories use some fancy number-crunching to compare the effectiveness of teachers in the nation’s second-largest school system. It has led to an explosive controversy in the city — some argue it’s about time parents had an objective measure to compare their kids’ teachers. Still, others say, this is just a way to humiliate educators.
I think it’s more the latter than the former. But more than anything, I think all of this is just covering up larger, systemic issues in our national education system that a focus on individual teacher performance will not fix.
The numbers are created through something called “Value-Added System” (in the following quote – from Wisconsin Policy Research – it is VAM, or “Value-Added Methodologies”), a system that I really, really like – in theory:
Testing conducted via VAM describes growth in students’ test scores over a school year. Thus, value-added testing reveals what a year’s worth of learning actually achieved, regardless of whether a child passes the annual test. Because value-added tests focus on gains rather than raw scores, each student’s performance is gauged against his or her own past performance. […]
According to that same Wisconsin Policy website, here are the benefits of VAS or, as they call it, VAM:
Sanders and his colleagues have utilized the massive database assembled in Tennessee to study a number of topics related to value-added assessment, including socioeconomic variables and teacher effects on student growth. These studies consistently yield two major findings: Student socioeconomic variables are poor predictors of student success, and teachers are the most important determinant for student academic growth. In a summary of the VAM research, Sanders and Horn7 state that “Differences in teacher effectiveness [are] the dominant factor affecting student academic gain. The importance of . . . certain classroom contextual variables (e.g., class size, classroom heterogeneity) appears to be rather minor….” And again, citing a 1997 study:8 “the two most important factors impacting student gain are the differences in classroom teacher effectiveness and the prior achievement level of the students. The teacher effect is highly significant in every analysis and has a larger effect size that any other factor in twenty of the thirty analyses.” Investigators for a major research report by the RAND Corporation9 have reached similar conclusions.
Another website lays out potential MAJOR flaws in what sounds like an amazing plan:
In the real world, you can’t always assume that last year’s test scores show how smart each teacher’s students are on average. Last years scores were likely driven up or down by the quality of the teacher last year. The really confusing thing is that it’s likely that students whose test scores were unnaturally depressed by a bad teacher last year are likely to go up more this year than students whose test scores were boosted last year by a very good teacher. That’s regression toward the mean.
Here’s a flaw that I see. From a very good friend of mine (who I don’t want to name on this blog) who teaches in a low-income school, I know that in some places teachers who have good test scores are often the ones then given the much more difficult students, the ones who need extra help, extra time, extra support. That’s good. Other than the fact that I know that this puts lots of stress and burden on the best teachers and thus has the potential for running them away from the field of education, I see why an administration would want their best teachers to handle the most difficult students. But when it comes down to comparing this teacher’s effectiveness from year to year, it gets more dicey.
Let’s start by looking at VAM if we are tracking these students. Year one: they do okay with an okay teacher but not as good as their peers who either are simply better students or have better teachers. Let’s say they learned about 70% of the what you would expect to learn in a full year, their peers learning 100%. In year two, they are assigned to a better teacher. Their VA test shows that they did better in their second year than they did in their first, learning roughly 85% of what you would expect to learn in a full year, though they still may lag behind other students in their school. But from these students’ test scores, it is clear that their second-year teacher did a very, very good job, especially considering that the teacher was helping kids who had only learned 70% the year before.
Now, what does the data look like when you show it teacher by teacher, the way the LA Times is going to? In that first year, let’s say that the teacher taught honor classes with students who generally perform well and learn at a high level. They end up testing very well across the year, learning 105% of what you expect kids to learn in one year. In the second, with their new students, that teacher is only able to teach them 85% of what you expect. That would look really, really bad for the teacher. But without understanding the massive changes between the students from year to year, you couldn’t actually assess the data on the teacher.
This begs the question: which scores matter for determining the effectiveness of this teacher and what service is the LA Times doing for teachers or parents by releasing this information? If you look simply at how their students did from year to year, the effectiveness of this teacher may look like it is slowly declining. But once you take into account that the administration is giving this teacher more challenging students in order to help those students out AND you focus on the improvement of the students, you will see that the teacher was, in fact, a total rockstar. Which my friend is. So, giving parents raw data on a teacher from the last five years may not actually show how phenomenal the teacher is. But the administration may understand. And may continue to give the good teachers more challenging students, thus making their value-added test scores appear lower. That makes me extremely nervous.
I hope it’s clear that I don’t actually have a problem with VAM as a theory of measurement. I have a problem with the LA Times publishing raw data (which it’s not clear if that is what they are doing – they may also be publishing some sort of analysis with it). Even with an analysis, it’s not clear if this data will actually show parents what it is they are hoping to find.
As the NPR report made clear, there are so many variables when it comes to teaching that this may only a single measures in a system of measurements:
A number of papers over the past year have cautioned that there are just too many variables in student performance to rely heavily on student test scores.
Tim Sass of Florida State University says there are dozens of factors outside a teacher’s control that can limit improvement in test scores. For example, Sass says, what if “the principal always gives the weaker students to the new teachers, let’s say, and always favors the more senior teachers at their school?”
The L.A. Times analysis carefully tries to correct for that, and for many other variables. And, in fact, value-added analysis is a well-accepted approach that has been used for decades in education research, though not for the purpose of publicly comparing individual teachers. […]
The Los Angeles Unified School District admits it has been sitting on this data and hasn’t used it to help teachers improve. Now, Deputy Superintendent John Deasy says, the real tragedy would be if all this information were ignored. “You would never evaluate a teacher only using this metric. On the other hand, you wouldn’t evaluate a teacher completely [without] considering how students are doing over time in achievement.”
Deasy says the controversy is now spurring the district to develop a new teacher evaluation system that uses value-added data.
In the end, I think that is all a red herring. Even with a much better system of evaluating individual teachers (though it’s not clear how to do that), reports like this that focus on VAM just take our attention away from the overall educational system. I think Obama was right when he proposed a year ago that we revamp the education calendar in this country. Like most things he has brought up, I doubt anything will come of it (especially since his administration is all about focusing on the release of this teacher VAM data). But that doesn’t discount the value in the idea:
“Now, I know longer school days and school years are not wildly popular ideas,” the president said earlier this year. “Not with Malia and Sasha, not in my family, and probably not in yours. But the challenges of a new century demand more time in the classroom.”
The president, who has a sixth-grader and a third-grader, wants schools to add time to classes, to stay open late and to let kids in on weekends so they have a safe place to go.
“Our school calendar is based upon the agrarian economy and not too many of our kids are working the fields today,” Education Secretary Arne Duncan said in a recent interview with The Associated Press.
I seriously believe that changes like this to the system will matter much more in the long run than the ability of a school district (or the parents in it) to evaluate the individual teacher performances.
Education is a communal thing, something that our society admits is important yet still begrudgingly supports. And yet we still like to point our fingers at individual teachers. Yes, for each individual child, their education will be different and their individual teachers will matter in huge ways to their overall performance and their acquisition of knowledge. I don’t deny that. And I think parents need to do the best they can to try to secure those educational experiences for their children. But parents can only do so much, and some parents can do much, much, much more than other parents can.
So, when it comes time to decide how to FIRST reform the educational system in a way that will have the largest impact, I just don’t think putting all our resources and attention into individual teacher performances is it (though I hope, hope, hope that school administrators, school boards, superintendents, etc. will keep at it). Like many things, though, I am not with the majority of Americans on this:
– Americans agree that effective teachers who prepare students for successful careers and productive lives deserve to be paid based on their performance. In fact, 71 percent of Americans support moving from a one-size-fits-all approach to teacher pay to contracts based on performance, with 73 percent support basing teacher pay on student achievement (up 13 percent over 10 years).
– Americans know that conventional public schools are underperforming and they’re embracing the need for more and better school choices like never before. In a country where 79 percent of people grade public schools a C, D, or an F, charter schools have grown in popularity by nearly 20 percentage points over just five years ago, with 68 supporting charters. In addition, 60 percent of Americans would support a “large increase” in the number of charters nationwide and 65 percent would support more charters in their own community.