Burden of Proof as a Legal Fiction: One Year Later

Originally posted on 21 November 2010.

In this post, I intend to prove to you beyond a reasonable doubt that the concept of "proof beyond a reasonable doubt" is nothing more than a legal fiction. This shouldn't take too long.

I begin by asking you to consider the burden of proof in a civil case. That burden is usually described as the "preponderance of the evidence."  It's a 50-50 type decision. If the evidence favors the plaintiff even by a  teensy weensy bit, the jury is to find in favor of the plaintiff. If, on the other hand, the evidence favors the defense even by a smidgeon, then the jury is to find in favor of the defense. It's so simple.

I present the idealized concept of preponderance of the evidence in the figure below. Click to enlarge.

Matters are a little more confusing for jurors in a criminal case. They are asked to decide if the evidence proves the State's case beyond a reasonable doubt. The jurors must figure out how heavily the evidence favors the prosecution AND they must figure out what the hell the term reasonable doubt means AND they must decide whether the weight of the evidence provided by the prosecution exceeds the reasonable doubt threshold.

Courts are of almost no help in determining the meaning of reasonable doubt. One thing they absolutely will not do is apply any numeric value to reasonable doubt. They never suggest anything such as "you must be 90% certain."

Never. Ever.

Not now. Not ever.

Instead, they give you a reasonable doubt instruction something along the line of:

The question naturally is what is a reasonable doubt. The words almost define themselves. It is a doubt based upon reason and common sense. It is a doubt that a reasonable person has after carefully weighing all of the evidence. It is a doubt which would cause a reasonable person to hesitate to act in a matter of importance in his or her personal life. Proof beyond a reasonable doubt must be proof of such a convincing character that a reasonable person would not hesitate to rely and act upon it in the most important of his own affairs.

There you go. I'm sure that clarified things.

Legal scholars, and I've been reading a lot of their scholarly work on this issue recently, recognized that if you are to quantify the number, it should be greater than 50%. That after all is the standard for civil cases, and the burden of proof in criminal cases is supposed to be substantially greater. Legal scholars tend to throw out a value of 90% as a starting point. This corresponds roughly to William Blackstone's 250 year old claim that it is "better that ten guilty persons escape than that one innocent suffer."

That traditional view is portrayed in the figure below.

Those legal scholars who believe that figure portrays reality are full of scholarly beans.

When Rita Simon and Linda Mahon asked 106 judges to put a non-legally-binding number to proof beyond a reasonable doubt, those judges gave a number (presumably on average) of 89%. Since that's almost identical to the number the full-of-beans legal scholars provided, I declare that judges too are full of judicial beans, at least with respect to this concept.

Simon and Mahon also asked 25 jurors to put a number to the concept. Those jurors put the number at 79%. That number is not far from a couple other studies I've seen where jurors put the number around 75%, assuming they are asked before they deliberate in a mock trial. If they are asked after they deliberate, the put the number closer to 55%. The suggestion is that they wanted to convict, the evidence didn't support conviction at even a 75% standard, so they reduced the standard.

Even though the students eventually put the standard just barely above a civil suit "preponderance of the evidence" standard, I declare my fellow jurors to be full of deliberative beans, at least with respect to this issue.

I'll allow the judges and jurors to make my point for me. During the NCSC judge-jury agreement study used by Bruce Spencer to establish a wrongful conviction rate around 10%, the judges and jurors were asked to provide their estimate of the strength of the evidence used to convict (or acquit) a defendant. I'll simply provide a summary plot of the results below, and move away from the keyboard a bit as you study the plot and say "What the hell?" Click to enlarge.

Okay. I'm back. The first thing the Skeptical Spouse said when I showed her this plot was "That's not right!!?"  She was pointing to the portion where the jurors voted more frequently to convict when the strength of evidence was 1 than when it was 2. I replied with something equally as scholarly, something along the lines of "Screw that. They're voting to convict a third of the time when by their own admission the evidence strongly favors the defense."

Even when the evidence favors neither the prosecution nor the defense, both judge and jury are voting two out of three times to convict. Anyone who studies the chart for more than a minute and  still believes that we presume defendants to be innocent, or that we burden the prosecution with proof beyond a reasonable doubt, is full of blog-reading beans.

This one simple plot, based directly on data provided by judges and juries, demolishes dozens of scholarly works I've read regarding quantifying reasonable doubt. This plot causes me to propose an alternative hypothesis of jury behavior. It may not be scholarly, but it better explains the data. Here goes.

Besides destroying much scholarly work and putting the lie to some of the most fundamental principles of our judicial system, the plot also tells you that those who guess and those who divide are pretty much wrong when they arrive at wrongful conviction rates of 1 or 2 or 3%. There is no way that the behavior portrayed in the chart would result in such low error rates.

I'll add one final chart, then I'll step away from the keyboard. In the chart below, I simply overlay the idealized and realized worlds of proof beyond a reasonable doubt. I'll allow the chart to speak for itself.

Wrongful Convictions From Civil Behavior

In a previous post, I described "Proof Beyond a Reasonable Doubt" as a convenient judicial myth. At the end of that post, I presented the following summary plot comparing the idealized concept against the reality of jury and judge verdicts.  I include that plot below for easy reference. Click to enlarge and clarify.

Judges and scholars, if forced to quantify the reasonable doubt threshold, tend place the threshold near 90%. In Proof Kinda Beyond a Reasonable Doubt, I reported that our jury pool self-reports a threshold of 85%. In Proof Beyond a Flip of the Coin, I report that more-subtle, less-biased surveys indicate that our jury pool places the threshold closer to 66%.

In this post, I regret to inform you that some studies place the actual threshold closer to 50%. The 50% limit is incompatible with a "proof beyond a reasonable doubt" standard. Rather, a 50% threshold is by definition a "preponderance of the evidence" standard. It's the standard jurors are instructed to use in civil cases, when John Doe accuses Joe Blow of destroying his topiary garden with a runaway snowmobile. A preponderance-of-evidence standard absolutely should not be applied when deciding matters of criminal guilty or innocence, when deciding matters of life and death.

But I'm ahead of myself. First, I should explain the studies.

The third group of studies we consider herein is based on decision theory. Decision theorists argue that jurors vote to maximize their personal satisfaction. (This is also known as minimizing their personal dissatisfaction.) Decision theorists describe such satisfaction (or dissatisfaction) as "utility." According to decision theorists, jurors intuitively weigh four utilities:

1. Utility of acquitting the innocent
2. Utility of convicting the innocent
3. Utility of acquitting the guilty
4. Utility of convicting the guilty

Decision theorists simply ask potential jurors to rate the relative utilities on a common scale, say from 1 to 100, then calculate the person's decision threshold based on a logical and relatively straightforward equation. I refer the reader to the source for the equation rather than complicate this post further by trying to describe it.

Though decision-theory surveys may sound more convoluted than the direct and parallel surveys, my experience tells me there is some merit in the concept. I have independently arrived at a hypothesis that jurors most fear putting bad people back on the street. In decision-theory speak, they assign a substantial disutility to acquitting the guilty, and apply a high utility to convicting the guilty. They show less concern about acquitting or convicting the innocent, since they tend to assume the defendant must be guilty. Certainly the prosecuting attorney and the police and good honest citizens wouldn't attempt to convict an innocent person.

If you simply ask a potential juror to state the reasonable doubt threshold they would use, they will give you number somewhere near 100%, because they know that's what the answer is supposed to be. If you conduct more subtle parallel studies of guilty / not guilty versus probability of guilt, you get closer to the actual threshold jurors use. Even in the parallel studies, however, the jurors don't really feel the gut-wrenching fear of putting a murderer, rapist, armed robber, or child molester back on the street.

Decision theory surveys attempt to penetrate into those inner feelings. While I'm confident the decision-theory surveys provide a more reasonable result than merely asking people what threshold they would use, I'm not sure if decision-theory surveys are superior to parallel surveys. My guess is that the best answer lies somewhere between the two.

In any case, I present the summary of decision theory surveys below. As usual, click on the image to enlarge and clarify it.

So after three posts on reasonable doubt threshold surveys, what have we learned? I fear we have learned why we have so many innocent people are in prison.

If we vote guilty when we are only 85% certain of guilt, or 68% certain, or possibly only 55% certain (depending on which study is more accurate), we should not be surprised that we frequently get it wrong. We should not be surprised when some of the convicted protest their innocence as we slam the prison gate shut behind them.

Nor should we be surprised as some protest their innocence while we inject lethal fluids into them through a sterilized needle.

Proof Kinda Beyond a Reasonable Doubt

In a previous post, I described the concept of "Proof Beyond a Reasonable Doubt" as a convenient judicial myth. At the end of that post, I presented the following summary plot comparing the idealized concept against the reality of jury and judge verdicts.  I include that plot below for easy reference. Click to enlarge and clarify.

Judges and scholars, if forced to quantify the reasonable doubt threshold, tend place the threshold near 90%. While that number flies in the face of the plot above, it comports with quite a few studies attempting to quantify the threshold.

In his 1993 book Inside the Juror, Reid Hastie provides a summary of studies in which groups of people were merely asked to quantify reasonable doubt after being read a standard reasonable doubt instruction. I repeat his summary in a somewhat simplified form below.

In summary, when our country's jury pool is asked to quantify the reasonable doubt standard, they claim they set the threshold at 85%. That's not much lower than the threshold typically selected by judges and scholars. However even if jurors set the standard that high (the plot at the top of this post proves they don't), it seems to me still to be a problem.

If I have a ten-sided die with sides numbered from 1 to 10, I have a 90% confidence that any single roll will result in a number greater than 1. Do I therefore have proof beyond a reasonable doubt that I will not roll a 1? 

In this simple die rolling experiment, I know that if I set 90% as my threshold for reasonable doubt, I will be wrong nearly 10% of the time, over a large sample of tests. Is that acceptable?

If instead of rolling a die, I am sitting as a juror in a criminal trial, should I vote guilty if I am 90% confident in the guilt of the defendant? If I am willing to do so, does that mean there is a 10% chance the defendant is innocent?

If all my fellow jurors are willing to do so, to vote guilty when we are 90% confident, should we expect a wrongful conviction rate near 10%, just as Spencer and Allen have calculated we do?

Burden of Proof as a Legal Fiction

In this post, I intend to prove to you beyond a reasonable doubt that the concept of "proof beyond a reasonable doubt" is nothing more than a legal fiction. This shouldn't take too long.

I begin by asking you to consider the burden of proof in a civil case. That burden is usually described as the "preponderance of the evidence."  It's a 50-50 type decision. If the evidence favors the plaintiff even by a  teensy weensy bit, the jury is to find in favor of the plaintiff. If, on the other hand, the evidence favors the defense even by a smidgeon, then the jury is to find in favor of the defense. It's so simple.

I present the idealized concept of preponderance of the evidence in the figure below. Click to enlarge.

Matters are a little more confusing for jurors in a criminal case. They are asked to decide if the evidence proves the State's case beyond a reasonable doubt. The jurors must figure out how heavily the evidence favors the prosecution AND they must figure out what the hell the term reasonable doubt means AND they must decide whether the weight of the evidence provided by the prosecution exceeds the reasonable doubt threshold.

Courts are of almost no help in determining the meaning of reasonable doubt. One thing they absolutely will not do is apply any numeric value to reasonable doubt. They never suggest anything such as "you must be 90% certain."

Never. Ever.

Not now. Not ever.

Instead, they give you a reasonable doubt instruction something along the line of:

The question naturally is what is a reasonable doubt. The words almost define themselves. It is a doubt based upon reason and common sense. It is a doubt that a reasonable person has after carefully weighing all of the evidence. It is a doubt which would cause a reasonable person to hesitate to act in a matter of importance in his or her personal life. Proof beyond a reasonable doubt must be proof of such a convincing character that a reasonable person would not hesitate to rely and act upon it in the most important of his own affairs.

There you go. I'm sure that clarified things.

Legal scholars, and I've been reading a lot of their scholarly work on this issue recently, recognized that if you are to quantify the number, it should be greater than 50%. That after all is the standard for civil cases, and the burden of proof in criminal cases is supposed to be substantially greater. Legal scholars tend to throw out a value of 90% as a starting point. This corresponds roughly to William Blackstone's 250 year old claim that it is "better that ten guilty persons escape than that one innocent suffer."

That traditional view is portrayed in the figure below.

Those legal scholars who believe that figure portrays reality are full of scholarly beans.

When Rita Simon and Linda Mahon asked 106 judges to put a non-legally-binding number to proof beyond a reasonable doubt, those judges gave a number (presumably on average) of 89%. Since that's almost identical to the number the full-of-beans legal scholars provided, I declare that judges too are full of judicial beans, at least with respect to this concept.

Simon and Mahon also asked 25 jurors to put a number to the concept. Those jurors put the number at 79%. That number is not far from a couple other studies I've seen where jurors put the number around 75%, assuming they are asked before they deliberate in a mock trial. If they are asked after they deliberate, the put the number closer to 55%. The suggestion is that they wanted to convict, the evidence didn't support conviction at even a 75% standard, so they reduced the standard.

Even though the students eventually put the standard just barely above a civil suit "preponderance of the evidence" standard, I declare my fellow jurors to be full of deliberative beans, at least with respect to this issue.

I'll allow the judges and jurors to make my point for me. During the NCSC judge-jury agreement study used by Bruce Spencer to establish a wrongful conviction rate around 10%, the judges and jurors were asked to provide their estimate of the strength of the evidence used to convict (or acquit) a defendant. I'll simply provide a summary plot of the results below, and move away from the keyboard a bit as you study the plot and say "What the hell?" Click to enlarge.

Okay. I'm back. The first thing the Skeptical Spouse said when I showed her this plot was "That's not right!!?"  She was pointing to the portion where the jurors voted more frequently to convict when the strength of evidence was 1 than when it was 2. I replied with something equally as scholarly, something along the lines of "Screw that. They're voting to convict a third of the time when by their own admission the evidence strongly favors the defense."

Even when the evidence favors neither the prosecution nor the defense, both judge and jury are voting two out of three times to convict. Anyone who studies the chart for more than a minute and  still believes that we presume defendants to be innocent, or that we burden the prosecution with proof beyond a reasonable doubt, is full of blog-reading beans.

This one simple plot, based directly on data provided by judges and juries, demolishes dozens of scholarly works I've read regarding quantifying reasonable doubt. This plot causes me to propose an alternative hypothesis of jury behavior. It may not be scholarly, but it better explains the data. Here goes.

Besides destroying much scholarly work and putting the lie to some of the most fundamental principles of our judicial system, the plot also tells you that those who guess and those who divide are pretty much wrong when they arrive at wrongful conviction rates of 1 or 2 or 3%. There is no way that the behavior portrayed in the chart would result in such low error rates.

I'll add one final chart, then I'll step away from the keyboard. In the chart below, I simply overlay the idealized and realized worlds of proof beyond a reasonable doubt. I'll allow the chart to speak for itself.