A while back we noted, among other papers in the "judging judges" genre, one by Stephen Choi (NYU), Mitu Galati (Duke), Mirya Holman (Duke/UNC), and Eric Posner (Chicago) that sought to hold claims about Justice Sotomayor’s now-infamous assertion, uttered prior to her Senate confirmation hearing, that female judges might be “better” than male judges. As noted in 2009, the Judging Women paper "estimate three measures of judicial output: opinion production, outside state citations, and co-partisan disagreements." The paper found that "male and female judges perform at about the same level."
The enterprise of empirically "judging judges," however, recently generated its own push-back. In The Costs of Judging Judges by the Numbers, the authors, Marin Levy (Duke), Kate Stith (Yale), and José Cabranes (CA2), advance some plausibly dramatic claims about the efficacy of "judging judges," at least empirically. The paper's abstract follows.
"This essay discredits current empirical models that are designed to “judge” or rank appellate judges, and then assesses the harms of propagating such models. First, the essay builds on the discussion of empirical models by arguing that (1) the judicial virtues that the legal empiricists set out to measure have little bearing on what actually makes for a good judge; and (2) even if they did, the empiricists’ chosen variables have not measured those virtues accurately. The essay then concludes that by generating unreliable claims about the relative quality of judges, these studies mislead both decision-makers and the public, degrade discussions of judging, and could, if taken seriously, detrimentally alter the behavior of judges themselves."