This post is adapted from the introduction and conclusion of Experimental Conversations: Perspectives on Randomized Trials in Development Economics, a new book from Timothy Ogden published earlier this year by MIT Press.
Roughly seven years ago I embarked on a project to change people’s minds about randomized control trials (RCTs), particularly as applied to figuring out “what works” in aid, philanthropy, and development. The project was borne out of conversations I was having with economists who were part of what might be called a movement that was changing the field of development economics (and is closely tied to philanthropy): conducting field experiments to answer age-old questions about the impact of anti-poverty programs. I was fascinated with the novelty of what this group of economists was doing and enthusiastic about what they seemed to be accomplishing.
But the conversations I had with these economists about their work did not match the perceptions I saw and read among policy makers, practitioners, and commenters on development and poverty programs. It seemed to me that a much larger group of people who cared about making the world a better place were ignorant of the work these economists were doing, or misunderstood it and its motivations. I wanted to give other non-specialists a chance to hear from the economists — in their own words — to understand what drives them and the work they do.
You might expect some general conclusions would have been reached on issues of evidence-based policy, impact evaluation, and RCTs, since these issues have been hotly debated for decades already. But that’s not the case. At the American Economic Association annual meetings in January, I watched vigorous debate about the use and value of RCTs. One of the seminal studies that launched the movement is still the subject of symposia and special editions of academic journals. The use of RCTs continues to grow while no general conclusions have been reached.
When I started this project, I was skeptical of critiques of RCTs. I will confess that I generally assumed that these critiques were the result not of honest, principled thinking about the benefits, value, and limitations of RCTs but of some darker motive. Some version of the thought, “These people must have reason to hide the truth!” nestled in my mind. Thankfully, I was nudged away from this thinking by a number of the people I interviewed.
Spending seven years interviewing proponents, critics, funders, consumers, and observers of the movement led me to reach some personal conclusions quite different from the perspective I had when I started this project. Understanding and engaging with the debate on the use of RCTs in development economics has changed the way I think in both my professional and personal life. I spend more time thinking about theories of change, and I am more skeptical about all research.
One thing that became clear to me is that the arguments for and against RCTs often put too much emphasis on the particularities of methodology, which distracts from the more important disagreement behind them: different theories of change.
Arguments over theories of change — ideas about how the world changes — are hardly unique to the debates about RCTs. Indeed, it is the foundation of cause-based philanthropy: how is it that inequality arises? How can it be attacked? There has always been wide disagreement within philanthropy about theories of change, harkening back to the efforts to distinguish “charity” from “philanthropy.”
While wary of reducing the arguments to short summaries, I find it helpful in the context of the RCT movement in impact evaluation to think about the competing theories of change along three main axes:
- the value of small versus big changes;
- the value of local knowledge versus technocratic expertise; and
- the role of individuals versus institutions.
Almost all of the discussion on the proper use of RCTs and evidence from them in evaluating programs and forming policy would benefit from a greater focus on these points, rather than on methodological specifics.
Another of my personal conclusions is the need to reframe the RCT debate in order to move forward. Both sides of the debate often point to the medical field to bolster their positions. Proponents of RCTs note that RCTs are required to test the efficacy and safety of new drugs before they are approved for use, and thus social programs deserve the same scrutiny. Meanwhile, critics say that medicine has a valid theory of external validity — that we understand how bodies work and so we can be confident that what works for study subjects will work for others — which economics lacks, rendering the results of RCTs useless for guiding decisions in different contexts.
I believe the appeals to medicine should stop, because the realities of the medical field don’t support either point. The quality of medical research and practice, it turns out, is shockingly bad. A large portion of published studies do not replicate. The existence of a movement advocating for “evidence-based medicine” highlights the fact that regardless of the reliability of the evidence from medical studies, it is difficult to get doctors to change practices. It’s clear that medical research and practice should not be held up as an example to aspire to.
But that doesn’t mean there is nothing to be learned from the medical field. Recognizing that, in general, the quality of evidence is not sufficient to support truly evidence-based medicine, some members of the medical research and practice community are shifting to talk about “evidence-generating medicine.” In particular, evidence-generating medicine is about gathering more data about medical care in actual practice and patient outcomes outside of medical trials. It seems to me that a move from focusing on “evidence-based policy” to “evidence-generating policy” could short-circuit many of the less productive arguments around the use of RCTs for impact evaluation and evidence-based programs and policies.
The growing use of RCTs inside of organizations for their own learning and evaluation — combined with better ongoing data on programs, policies, and program participants in an “evidence-generating policy” framework — would materially advance both internal and external validity and improve policies and programs in small and large ways. Such an approach would value both local knowledge and technocratic expertise. It would allow individuals and institutions to learn and take action. As I survey how the field has changed since I began paying attention in 2010, I’m hopeful that this is the direction of movement. A number of RCT proponents are also at the leading edge of advocating for research quality and reproducibility efforts. Additional advocacy for data gathering outside of experiments would be a useful addition to the movement.
In the end, I remain an advocate for the use of RCTs in impact evaluations; even for the increased use of RCTs. While my enthusiasm has been tempered by wrestling with the many valid points in the critiques of RCTs, it hasn’t been extinguished. Working on this book has helped me see that my personal theory of change puts high value on small, incremental changes; a high value on technocratic expertise; and ascribes a large role to individuals within institutions.
Timothy Ogden is managing director of the Financial Access Initiative, a research center at NYU Wagner Graduate School of Public Service; executive partner of Sona Partners; and the author of Experimental Conversations: Perspectives on Randomized Trials in Development Economics. For more from the book, you can read Ogden’s interview with 2015 Nobel Prize winner Angus Deaton here, and find more resources at econ-versations.com.
Follow Ogden on Twitter at @timothyogden.
