Two interesting articles were published within a few days of one another last week on the topic of experimental methods in political science.
The first article is by Jasjeet S. Sekhon and Rocio Titiunik in the American Political Science Review, and it discusses the uses and misuses of natural experiments:
Natural experiments help to overcome some of the obstacles researchers face when making causal inferences in the social sciences. However, even when natural interventions are randomly assigned, some of the treatment–control comparisons made available by natural experiments may not be valid. We offer a framework for clarifying the issues involved, which are subtle and often overlooked. We illustrate our framework by examining four different natural experiments used in the literature. In each case, random assignment of the intervention is not sufficient to provide an unbiased estimate of the causal effect. Additional assumptions are required that are problematic. For some examples, we propose alternative research designs that avoid these conceptual difficulties.
In other words, many of the natural experiments found in the literature do not allow identifying causal effects, and the authors do a good job of providing examples of four published natural experiments whose findings they question. The findings in Sekhon and Titiunik’s article apply to some regression discontinuity designs as well.
The second article is a very short piece by James N. Druckman and Arthur Lupia in Science‘s “Perspective” section, and it describes the role of experimental methods in political science. The authors conclude that
The studies described above and others like them have transformed political science into a discipline in which experiments are increasingly seen as a preferred method of discovery and inference. Yet, important challenges persist in expanding the domain of experimental political science. One such challenge is that typical experimental subjects often lack the experience needed to act “as if ” they were professional legislators; yet, legislators themselves are often reluctant to participate in experiments as subjects. Another challenge is that politics entails not just debates about the empirical consequences of choosing one policy over another, but also disagreements over basic values. Experiments have less power to settle such questions. Nevertheless, many aspects of modern politics follow a logic that can be evaluated scientifically. Political science experiments are increasingly helping researchers and citizens around the world to better understand how humans organize themselves.
The Science article also features a podcast interview with one of the authors.