The Case for Looking at More Than Just Test Scores

The Atlanta Public Schools cheating scandal illustrates the peril of focusing on a single performance measure to the exclusion of all other data and should open the eyes of school leaders and policymakers to the need for a broader set of indicators of school quality.

What the school system created was a focus on test scores so intense that seating of principals at an annual meeting was based on how their schools fared, with high-scorers getting prime seating and low-scorers forced to stand in the back. The message was clear: Deliver high test scores or be humiliated. So, as Superintendent Beverly Hall collected national accolades and hundreds of thousands of dollars in performance bonuses, principals and teachers engaged in rampant cheating.
The lesson here is not that tests don’t matter. They do and they should. It’s that they should not be positioned as the one and only measure of school quality. Today, unfortunately, they often are — as a matter of federal policy.

The intentions of those who have pushed for greater accountability in our education system are good ones. But the very students who were supposed to benefit from reform efforts are now collateral damage in scandals such as Atlanta’s — with teachers erasing students’ incorrect answers and putting in the right ones rather than focusing on the much harder task of helping those students learn.

Just as in business, in which a focus on short-term stock appreciation and quarterly profits has resulted time and time again in fraud and accounting scandals, the over-emphasis on test scores — with careers, tenure decisions and compensation made or broken by the results — has led to cheating that has disgraced school systems once held up as exemplars. Sadly, when jobs or money are on the line, people will try to game the system.

One powerful way to reduce the chances of scandal and dishonesty is to create a well-rounded set of indicators of school quality that are reviewed together — to form a more complete understanding of school performance. A robust, multifaceted approach to assessment renders gaming much more difficult.

So what other data should be tapped to complement test scores? Here’s a radical idea for one additional source of data that, today, is largely ignored: students themselves.

Students have a valuable perspective on their educations, yet remarkably little attention is paid to their voices by those in power.

Four years ago, with support from the Bill & Melinda Gates Foundation, we created YouthTruth, a survey program that rigorously captures confidential student perceptions of their high school experience – and then puts their ratings in a comparative context. What we have seen is that when you ask the right questions, when students believe the process is genuine and that administrators will listen to what they have to say, and when you play back the results to the schools and the students, new insights emerge about educational quality.

Through this work, we have seen student survey data fuel dramatic change. One high school revamped its approach to grading and assignments and created new capstone programs for seniors in response to survey data suggesting that students were not being sufficiently challenged. Another school retooled its discipline process, instituting a system for adjudication that involved students themselves and resulted in a better culture for learning. A third school revamped its college advising and mentoring program in response to student feedback.

In addition to providing fodder for important real-time improvements, student survey data could help vigilant leaders and policymakers detect cheating early on. In the Gates Foundation’s Measures of Effective Teaching study, Ron Ferguson of Harvard University has demonstrated empirically that student perceptions of teacher quality and student academic achievement are correlated. So a disconnect between survey results and test scores could be a kind of early warning system, the presence of which in itself also would act as a deterrent to those who would consider cheating.
Students are just one source of potentially powerful data. For example, many districts survey parents, but decision-makers rarely pay attention to the results. Teachers themselves are another key source of data on what’s really happening within the school walls: They, too, should be surveyed regularly — in a way that protects confidentiality to ensure candor — with results making their way to district leaders and policymakers. Some districts do this today, too, but these data are often overlooked in the obsessive, legislated focus on test results.

Perceptual surveys have their limitations, of course, but offering key constituencies — starting with students — the opportunity to share their views on what’s really happening with those in power would be a significant step in the right direction. Ultimately, unless we change policy to ensure that schools and their leaders are not judged by test results alone, we’re destined to read about more scandals like the epic one in Atlanta.

Phil Buchanan is president of the Center for Effective Philanthropy.
Valerie Threlfall is director of YouthTruth, an initiative of the Center for Effective Philanthropy.
Fay Twersky is senior adviser at the Yad Hanadiv Foundation and previously served as director of Impact Planning and Improvement at the Bill & Melinda Gates Foundation.