Friday, February 11, 2011

Proof Beyond the Flip of a Coin

In a previous post, I described the "Proof Beyond a Reasonable Doubt" as a convenient judicial myth. At the end of that post, I presented the following summary plot comparing the idealized concept against the reality of jury and judge verdicts.  I include that plot below for easy reference. Click to enlarge and clarify.
Judges and scholars, if forced to quantify the reasonable doubt threshold, tend place the threshold near 90%. While that number flies in the face of the plot above, it comports with quite a few studies attempting to quantify the threshold.

In a an earlier post, I presented evidence that jurors self-report that they place the threshold at 85%. First order, this means that if you are innocent but on trial, you stand a 15% chance of being convicted. As horrific as this is, I believe it seriously understates the problem faced by the (relatively few) innocent people facing a jury.

From my analysis of judge-jury agreement data, I calculate that an innocent person has a 28% chance of being convicted. From Professor (of Statistics) Bruce Spencer's analysis of judge-jury agreement data, he calculates that an innocent person has a 25% chance of being convicted. (Professor Spencer is careful to limit his conclusion to the sample of trials he studied.) I present comparitive summaries of our results in Judges Gone Wild: Revised, and one of the plots from that post below. Click to enlarge and clarify.

If jurors do in fact use an 85% threshold for reasonable doubt, then I am probably wrong in my calculation that 28% of innocent people are convicted (rather than acquitted) by juries. In this case, I would be pleased to be wrong. Unfortunately, I don't believe I am.

In his 1993 book Inside the Juror, Reid Hastie actually looked at three approaches for determining the reasonable doubt threshold used by jurors. The 85% threshold is based on the self-reporting approach. Hastie calls that the direct approach. I call it the least reliable approach. In this post, I'll describe the results from a the second approach, what Hastie calls the parallel approach. Quoting from Hastie, here's how the  first of the parallel approach studies worked. It's pretty clever, actually.
Two independent groups of subjects made judgments after listening to an audiotaped homicide trial; one group made a dichotomous [either-or] guilty-innocent verdict judgement and the other half of the subjects rate the probability that the defendant had committed the crime.
Assuming the two groups were valid samples of our jury pool, the researchers could determine the actual reasonable doubt threshold by comparing the guilty votes with the probability of guilt assessments. Neither group knew how the other group would vote or rate the probability of innocence.

The initial study was repeated by two other investigators. In the two follow-on studies, only one group was used. That group was asked to provide both a verdict and a probability of innocence. Below, I tabulate the results from the three studies as reported by Hastie.


Using this method of determining an actual (rather than self-reported) reasonable doubt threshold, the first-order conclusion is that jurors will find 34% of the innocent defendants guilty. That rate of innocents being convicted is greater even than the 28% rate I calculated and the 25% rate Professor Spencer calculated. Perhaps one could fault the studies for relying entirely on students, though during the self-reporting studies the students responded substantially the same as other potential jurors.

Alternatively, one might consider the possibility that juries convict somewhere between a quarter and a third of the innocent defendants who come before them. That's bad, but better than the flip of a coin.

But not much better.