As I have mentioned many times previously, I am preparing a monograph on the rate of wrongful conviction. Each chapter will deal with one estimate of that rate, beginning with zero and ending beyond 10%. I am posting the draft chapters here, as I write them. I have so far posted the following:
Chapter 0.027: The Scalia Number
Chapter 0.5: The Huff Number
Chapter 0.8: The Prosecutor Number
Chapter 1.0: The Rosenbaum Number
Chapter 1.3: The Police Number
Chapter 1.4: The Poveda NumberChapter 1.9: The Judge Number
Chapter 2.3: The Gross Number
Chapter 3.3: The Risinger Number
Chapter 5.4: The Defense Number
Chapter 9.5: The Inmate Number
Chapter 10.1: A Skeptical Juror Number
I must confess to a grievous nomenclature error. I previously published my judge-jury agreement results in Chapter 8.4. The corresponding 8.4% wrongful conviction rate was actually a measure of wrongful convictions divided by a corresponding number of trials. I should have instead published a number that measures the number of wrongful convictions per conviction. Only that second number when multiplied by the number of people incarcerated will result in the number of people wrongfully incarcerated.
You can probably feel my pain.
To set matters right, I recalculated my numbers and below I republish my results in what is now Chapter 11.1. Though this post is similar to Chapter 8.4, it contains new information and new insight not addressed in the previous, incorrect post. For example, I compare my judge-jury agreement results with the inmate survey results, to my great surprise.
To minimize further confusion, I have removed my incorrectly labeled and incorrectly calculated Chapter 8.4. I present below the properly labeled and properly calculated chapter.
Chapter 11.1
A Skeptical Juror Number
At this point we move beyond those who guess, those who divide, and those who are behind bars. We now move into the rich and informative world of judge-jury agreement.
In 1966, Harry Kalven, Jr., and Hans Zeisel published The American Jury. Though it was not intended as such, I consider that book to be the most significant contribution to the quantification of our country’s wrongful conviction rate.
In the 1950s and 1960s, Kalven and Zeisel participated in The Chicago Jury Project, a scholarly look into the American jury system. The project was conducted by the University of Chicago Law School. Kalven and Zeisel were then and there professors of law.
As one part of that project, the two professors sent questionnaires to each U.S. trial judge they could identify as one who might preside over a criminal trial. They asked the judges to record the verdict they would have cast had their cases been bench trials. Of the 3500 judges contacted, 555 responded and cooperated by providing data on 3,576 criminal trials from all over the country.
Kalven and Zeisel presented their results for 42 different crimes in The American Jury, summarizing each crime and the overall results in a 2-by-2 table similar to the one presented below.
In 1966, Harry Kalven, Jr., and Hans Zeisel published The American Jury. Though it was not intended as such, I consider that book to be the most significant contribution to the quantification of our country’s wrongful conviction rate.
In the 1950s and 1960s, Kalven and Zeisel participated in The Chicago Jury Project, a scholarly look into the American jury system. The project was conducted by the University of Chicago Law School. Kalven and Zeisel were then and there professors of law.
As one part of that project, the two professors sent questionnaires to each U.S. trial judge they could identify as one who might preside over a criminal trial. They asked the judges to record the verdict they would have cast had their cases been bench trials. Of the 3500 judges contacted, 555 responded and cooperated by providing data on 3,576 criminal trials from all over the country.
Kalven and Zeisel presented their results for 42 different crimes in The American Jury, summarizing each crime and the overall results in a 2-by-2 table similar to the one presented below.
A lot can be learned from the simple table just provided. Juries, for example, are twice as likely to acquit as are judges. Based on the Kalven-Zeisel data, juries acquit 33% of the time (14% + 19%) while judges (the bench) acquit only 17% of the time (14% + 3%). Judges and juries are generally in agreement: they agree 78% of the time (14% + 64%) and disagree only 22% of the time (3% and 19%). They almost always disagree because the jury acquits where the judge would convict (19% out of 22%).
<<>>
An astute reader will note that the data, while interesting, are now a half-century old, and that reader may wonder whether the results are applicable today. The answer seems to be that the results are indeed still applicable. A more recent judge-jury agreement study of 290 cases, reported in 2003 by National Center for State Courts (NCSC), reveal strikingly similar results.
Though half a century separates the two studies, the overall agreement rates appear to be unchanged: 78% in the Kalven-Zeisel data and 77% in the NCSC data. Also, the tendency of the jury to acquit at nearly twice the rate as the judge is equally apparent in both studies.
While less current, the Kalven-Zeisel data have two distinct advantages. First, their study includes more than ten times as many trials as the more recent NCSC study. Studies having larger sample sizes, assuming everything else to be equal, are preferred over studies having smaller sample sizes.
The second advantage of the Kalven-Zeisel data is that they are presented on a crime-by-crime basis, and many of those crimes are supported by a statistically significant sample of trials. The Kalven-Zeisel results for aggravated assault, for example, are based on 292 trials. That is two more trials than the total number of trials for all crimes reported in the NCSC studies.
Since having the trials broken down by crime is critical to my methodology, I rely on the Kalven-Zeisel results. I’m naturally pleased to see that they still seem applicable today.
While less current, the Kalven-Zeisel data have two distinct advantages. First, their study includes more than ten times as many trials as the more recent NCSC study. Studies having larger sample sizes, assuming everything else to be equal, are preferred over studies having smaller sample sizes.
The second advantage of the Kalven-Zeisel data is that they are presented on a crime-by-crime basis, and many of those crimes are supported by a statistically significant sample of trials. The Kalven-Zeisel results for aggravated assault, for example, are based on 292 trials. That is two more trials than the total number of trials for all crimes reported in the NCSC studies.
Since having the trials broken down by crime is critical to my methodology, I rely on the Kalven-Zeisel results. I’m naturally pleased to see that they still seem applicable today.
<<>>
There are five inter-related bits of information we hope to learn from the judge-jury agreement tables. The most obvious, given the name of this monograph, is the frequency with which juries vote guilty when the defendant is factually innocent.
The second bit of information we hope to learn is similar, but it is associated with the judge rather than the jury. It is the frequency with which judges find the defendant guilty during a bench trial when the defendant is factually innocent.
Two additional bits of information of interest are the errors associated with acquitting a factually guilty person. Juries sometimes make this error, and judges sometimes make this error. Strictly speaking, they make these errors only in a mathematical sense. In our jury system, a person is to be acquitted unless the State proves guilt beyond a reasonable doubt. It is not by any means a legal error to acquit a factually guilty person if the State has failed to meet its burden of proof. Nonetheless, quantifying such “wrongful” acquittal errors is of interest in the analysis of judge-jury agreement data.
There is one additional, critical bit of information associated with the judge-jury agreement model. We must know or be able to calculate the percentage of defendants who were factually guilty.
The second bit of information we hope to learn is similar, but it is associated with the judge rather than the jury. It is the frequency with which judges find the defendant guilty during a bench trial when the defendant is factually innocent.
Two additional bits of information of interest are the errors associated with acquitting a factually guilty person. Juries sometimes make this error, and judges sometimes make this error. Strictly speaking, they make these errors only in a mathematical sense. In our jury system, a person is to be acquitted unless the State proves guilt beyond a reasonable doubt. It is not by any means a legal error to acquit a factually guilty person if the State has failed to meet its burden of proof. Nonetheless, quantifying such “wrongful” acquittal errors is of interest in the analysis of judge-jury agreement data.
There is one additional, critical bit of information associated with the judge-jury agreement model. We must know or be able to calculate the percentage of defendants who were factually guilty.
<<>>
To understand how one might calculate such information from judge-jury agreement data, consider the highly simplified judge-jury agreement table below.
Of the four possible maximally-simplified tables, this one best approximates the real world. Most defendants are factually guilty of the crime for which they are charged. Most of the time, the jury and judge agree that Guilty is the proper verdict. Most of the time, the judge and jury get it right.
Because the 100% conviction table crudely represents the basics of our judicial system, one could estimate our wrongful conviction rate based on that table and not be too far off, at least in a numeric sense. One could simply estimate our wrongful conviction rate to be 0% or nearly so, as did Judges Learned Hand and Antonin Scalia, and be within perhaps 10% percent of the correct answer. A 0% wrongful conviction rate, however, is consistent with the table only if one assumes that each and every defendant was factually guilty.
If, on the other hand, only 90% of the defendants were factually guilty, then the wrongful conviction rate would be 10%. Similarly, if only 80% of the defendants were factually guilty, then the wrongful conviction rate would be 20%.
Because the 100% conviction table crudely represents the basics of our judicial system, one could estimate our wrongful conviction rate based on that table and not be too far off, at least in a numeric sense. One could simply estimate our wrongful conviction rate to be 0% or nearly so, as did Judges Learned Hand and Antonin Scalia, and be within perhaps 10% percent of the correct answer. A 0% wrongful conviction rate, however, is consistent with the table only if one assumes that each and every defendant was factually guilty.
If, on the other hand, only 90% of the defendants were factually guilty, then the wrongful conviction rate would be 10%. Similarly, if only 80% of the defendants were factually guilty, then the wrongful conviction rate would be 20%.
There are, in fact, an infinite number of solutions that can be derived from the 100% conviction table. The ambiguity arises because we need to calculate five bits of information from the table and we have only four squares giving us information. We need more information before we can unambiguously extract the correct wrongful conviction rate encoded within the judge-jury agreement tables.
<<>>
Consider a second (more-complicated but still-simplified) judge-jury agreement table.
One mathematically-viable interpretation of the table is that 50% of the defendants were factually guilty, the judge and jury were both correct when they agreed (that would account for 25% of those who were guilty), and the jury failed to convict 25% of the time when the defendant was factually guilty (that would account for the other 25% of those who were guilty.) Also, the judge and jury must have both been always correct when they agreed on acquittal, and the judge must have been always wrong when voting to convict while the jury voted to acquit.
A second mathematically-viable (but unlikely) interpretation is that 75% of the defendants were factually guilty, that the judge and jury agreed to find them guilty 25% of the time, and that one and only one of them failed to vote guilty 50% of the time. It's a statistically unlikely scenario, but it is a valid mathematical interpretation of the data as they are presented.
Again, in this more complicated table, an infinite number of solutions are mathematically possible. Additional, quantifiable insight into the table is necessary to calculate the correct answer.
A second mathematically-viable (but unlikely) interpretation is that 75% of the defendants were factually guilty, that the judge and jury agreed to find them guilty 25% of the time, and that one and only one of them failed to vote guilty 50% of the time. It's a statistically unlikely scenario, but it is a valid mathematical interpretation of the data as they are presented.
Again, in this more complicated table, an infinite number of solutions are mathematically possible. Additional, quantifiable insight into the table is necessary to calculate the correct answer.
<<>>
"I needed a drink, I needed a lot of life insurance, I needed a vacation, I needed a home in the country. What I had was a coat, a hat and a gun." -- Philip Marlowe, Farewell, My Lovely
<<>>
I needed wrongful conviction rates, both for the jury and the judge. I needed wrongful acquittal rates, for both judge and jury. I needed to know how many of the defendants were actually guilty. What I had was a set of judge-jury agreement data.
As a data gumshoe, I have various tools of the trade to help me close the gap between the information I have and the information I want. First, I have a set of equations describing the relationship between wrongful conviction rates and judge-jury agreement data. Hopefully, my equations accurately and concisely describe with mathematical precision that which I so poorly and imprecisely described in the preceding paragraphs. Hopefully.
Second, I have a computer and I know how to use it. I programmed it to solve my rather complex sets of equations.
Third, I have a way of obtaining missing information I need but don’t have: I guess.
I prefer you believe that I merely make entirely rational assumptions, but the fact of the matter is I guess. Most of us do. It’s sometimes necessary in this business. Each of us must keep in mind, however, that our final answer can be wrong if we make badguesses assumptions. We must therefore be honest with our clients and ourselves by distinguishing between what we actually know and what we merely assume. That said, I’ll reveal my fundamental assumption here, even if you won’t understand why I had to make it, even if you won’t know how it might effect the outcome.
Here goes. I assumed that the judge and jury errors were proportional to the rate at which they disagreed on the verdicts. For those crimes where the judge and jury agreed most frequently on the verdict, I assumed they made the fewest errors. For those crimes where the judge and jury disagreed most frequently, I assumed they made the most errors.
As a data gumshoe, I have various tools of the trade to help me close the gap between the information I have and the information I want. First, I have a set of equations describing the relationship between wrongful conviction rates and judge-jury agreement data. Hopefully, my equations accurately and concisely describe with mathematical precision that which I so poorly and imprecisely described in the preceding paragraphs. Hopefully.
Second, I have a computer and I know how to use it. I programmed it to solve my rather complex sets of equations.
Third, I have a way of obtaining missing information I need but don’t have: I guess.
I prefer you believe that I merely make entirely rational assumptions, but the fact of the matter is I guess. Most of us do. It’s sometimes necessary in this business. Each of us must keep in mind, however, that our final answer can be wrong if we make bad
Here goes. I assumed that the judge and jury errors were proportional to the rate at which they disagreed on the verdicts. For those crimes where the judge and jury agreed most frequently on the verdict, I assumed they made the fewest errors. For those crimes where the judge and jury disagreed most frequently, I assumed they made the most errors.
There, I said it. I’m glad to have that off my chest.
<<>>
"No, no: I never guess. It is a shocking habit, -- destructive to the logical faculty." -- Sherlock Holmes, The Sign of the Four
<<>>
My calculated results for jury trials are presented in tabular form below. They are the result of some forty million iterations, give or take a few million, my computer needed to simultaneously solve my rather extensive set of equations.
The resulting wrongful conviction and acquittal rates appear in bold. I took them directly from my analysis. I obtained the conviction rates from the judge-jury agreement tables. I found the conviction numbers in the Sourcebook for Criminal Justice Statistics Online. All other numbers are simply calculated values.
The resulting wrongful conviction and acquittal rates appear in bold. I took them directly from my analysis. I obtained the conviction rates from the judge-jury agreement tables. I found the conviction numbers in the Sourcebook for Criminal Justice Statistics Online. All other numbers are simply calculated values.
The good news from this table is that juries are accurate 82% of the time, at least according to my analysis. (From here on, you can assign that “according to my analysis” qualifier to all remaining claims in this chapter.) And perhaps it’s good news, given the State’s burden of proof, that juries are twice as likely to free a guilty person than they are to convict an innocent person, 11.6% versus 6.5% of those tried. The bad news is that, assuming the data are generally applicable, the 9.1% wrongful conviction rate suggests we have more than 200,000 Americans wrongfully imprisoned.
You can see that the wrongful conviction rate for murder/manslaughter is 9.1%. (That is coincidentally equal to the weighted average for all crimes.) We can compare this number to the various estimates of the wrongful conviction rate derived by on dividing murder exonerations by murder convictions. Marty Rosenbaum calculated a wrongful conviction rate of only 1%. Tony Poveda’s number was slightly higher at 1.4%, and Samuel Gross’ number was higher still at 2.3%. Michael Risinger calculated a value of 3.3%, then allowed it might be as high as 5%, but no higher. By using a radically different denominator, I calculated 10.1%.
I’m naturally pleased to see that my two calculated results agree so well: 9.1% according to my judge-jury agreement calculations and 10.1% according to my exoneration-execution calculation. I therefore boldly claim that the two calculations tend to confirm one another. I note, however, that the calculations are hardly independent of one another, given that I calculated them both. That is why I publish two versions of this monograph, one of which includes an extensive addendum explaining in painful detail how I arrived at my judge-jury agreement numbers. I seek and encourage peer review.
You can see that the wrongful conviction rate for murder/manslaughter is 9.1%. (That is coincidentally equal to the weighted average for all crimes.) We can compare this number to the various estimates of the wrongful conviction rate derived by on dividing murder exonerations by murder convictions. Marty Rosenbaum calculated a wrongful conviction rate of only 1%. Tony Poveda’s number was slightly higher at 1.4%, and Samuel Gross’ number was higher still at 2.3%. Michael Risinger calculated a value of 3.3%, then allowed it might be as high as 5%, but no higher. By using a radically different denominator, I calculated 10.1%.
I’m naturally pleased to see that my two calculated results agree so well: 9.1% according to my judge-jury agreement calculations and 10.1% according to my exoneration-execution calculation. I therefore boldly claim that the two calculations tend to confirm one another. I note, however, that the calculations are hardly independent of one another, given that I calculated them both. That is why I publish two versions of this monograph, one of which includes an extensive addendum explaining in painful detail how I arrived at my judge-jury agreement numbers. I seek and encourage peer review.
<<>>
One significant (and now obvious) advantage of using the Kalven-Zeisel data is that they allow us to see how the wrongful conviction rate varies from crime to crime. The wrongful conviction rate for juries ranges from a high of 16.8% for forcible rape to a low of 4.7% for drug offenses.
The careful reader might be struck by these trends. They are similar to the trends claimed by inmates as described in Chapter 9.6. There, inmates convicted of forcible rape were most likely to claim a wrongful conviction. Those charged with possessing or selling drugs were least likely to claim a wrongful conviction.
This apparent similarity in trends prompted me to compare the two sets of results in the plot I present below. I’ll confess up front that I am amazed.
The careful reader might be struck by these trends. They are similar to the trends claimed by inmates as described in Chapter 9.6. There, inmates convicted of forcible rape were most likely to claim a wrongful conviction. Those charged with possessing or selling drugs were least likely to claim a wrongful conviction.
This apparent similarity in trends prompted me to compare the two sets of results in the plot I present below. I’ll confess up front that I am amazed.
Clearly, the wrongful conviction rates are not the same. The inmates claim rates greater than those I calculate for juries. The striking aspect is that the two rates correlate so well. If the inmates were merely making wild, unsubstantiated claims, why would their rates correlate so nicely with my calculated jury rates? Why would those inmates convicted of rape claim the highest rates, those convicted of other sex crimes claim the next highest rates, those convicted or murder the third highest rates, and so on down to those convicted of drugs who claim the lowest rates? Why does the inmate survey sequence match the jury sequence?
It seems instead as if those wrongfully charged with a crime have a relative sense of their chance of being wrongfully convicted at trial. It seems as if those wrongfully charged with rape are aware that juries are most likely to wrongfully convict them of rape, and those wrongfully charged with drugs are aware that juries are least likely to wrongfully convict them of drugs. The plot suggests that wrongfully charged defendants correctly assess their chance at trial, understand the penalties they will receive if wrongfully convicted, weigh that against the certain penalty they will receive if they plead guilty, and decide accordingly.
If the plot represents a true relationship between the behavior of juries and the expectations of those considering a plea bargain, rather than a coincidental relationship, then the plot is an amazing plot.
It seems instead as if those wrongfully charged with a crime have a relative sense of their chance of being wrongfully convicted at trial. It seems as if those wrongfully charged with rape are aware that juries are most likely to wrongfully convict them of rape, and those wrongfully charged with drugs are aware that juries are least likely to wrongfully convict them of drugs. The plot suggests that wrongfully charged defendants correctly assess their chance at trial, understand the penalties they will receive if wrongfully convicted, weigh that against the certain penalty they will receive if they plead guilty, and decide accordingly.
If the plot represents a true relationship between the behavior of juries and the expectations of those considering a plea bargain, rather than a coincidental relationship, then the plot is an amazing plot.
<<>>
My calculated results for bench trials are presented in tabular form below. The table is parallel to the jury-results table. The sources of the data are as explained previously.
Another significant (and now obvious) advantage of using the Kalven-Zeisel data is that they allow us to compare the performance of juries and judges. Judges have nearly the same accuracy rate as do jurors. Both decide nearly 82% of verdicts correctly. The small difference in accuracy rate between juries and judges is insignificant given the uncertainty of the analysis.
There is, however, a substantial difference in the fashion in which juries are judges err. While juries “wrongfully” acquit twice as many people as they wrongfully convict, judges wrongfully convict twice as many people as they “wrongfully” acquit (12.6% versus 5.9%).
Where the juries had an overall wrongful conviction rate of 11.1%, judges had an overall wrongful conviction rate of 14%.
These results are not particularly surprising, given the initial judge-jury agreement table presented in this chapter. That table, summarizing the results for the Kalven-Zeisel study, show that judges indicate they would have convicted in 83% of the trials. The jurors, on the other hand, voted to acquit in 67% of the cases.
I’ve seen a number of essays discussing the wisdom, or lack thereof, associated with choosing a bench trial rather than a jury trial. More than a few of these essays argue in favor of a bench trial. Based on my analysis of the judge-jury agreement data, I don’t see it. It seems to me to be a foolish choice, whether the defendant is factually guilty or factually innocent.
There is, however, a substantial difference in the fashion in which juries are judges err. While juries “wrongfully” acquit twice as many people as they wrongfully convict, judges wrongfully convict twice as many people as they “wrongfully” acquit (12.6% versus 5.9%).
Where the juries had an overall wrongful conviction rate of 11.1%, judges had an overall wrongful conviction rate of 14%.
These results are not particularly surprising, given the initial judge-jury agreement table presented in this chapter. That table, summarizing the results for the Kalven-Zeisel study, show that judges indicate they would have convicted in 83% of the trials. The jurors, on the other hand, voted to acquit in 67% of the cases.
I’ve seen a number of essays discussing the wisdom, or lack thereof, associated with choosing a bench trial rather than a jury trial. More than a few of these essays argue in favor of a bench trial. Based on my analysis of the judge-jury agreement data, I don’t see it. It seems to me to be a foolish choice, whether the defendant is factually guilty or factually innocent.
<<>>
I present below the error rates for jury and bench trials combined. The only new data are the Fraction Guilty numbers presented in bold. The are taken directly from my analysis of the Kalven-Zeisel judge-jury agreement data. All other numbers are taken from one of the previous two tables or are determined by calculation. The overall wrongful conviction rate for trials (judge and jury trials combined) is found to be 11.1%.
The Faction Guilty values allow us to easily calculate the number of people who were actually guilty and the number who were actually innocent during state trials in 2004, at least for the crime categories considered. If we take one more simple, distressing step, we will reach a shocking, appalling conclusion. If we divide the number of wrongful convictions by the number of people who were innocent (4,168 / 10,998), we see that an innocent person who goes to trial stands a 38% chance of being wrongfully convicted.
<<>>
Overall, 77.8% of trial defendants are guilty, assuming 2004 was a representative year. That means that 22.2% of all defendants are innocent. I declare this 22.2% number to be an error rate equivalent to the jury’s and the judge’s wrongful conviction rates. I declare this to be the police and prosecutors’ wrongful trial rate.
No person should be subjected to the legal and emotional costs of a trial, much less a 38% chance of a wrongful conviction, unless the state is confident it can prove beyond a reasonable doubt the defendant is guilty. Any acquittal therefore represents a wrongful trial error, just as any conviction of an innocent person represents a wrongful conviction error. The police and prosecutors’ wrongful trial error is nearly twice as large as the judges’ wrongful-convictions-per-trial rate (22.2% versus 12.6%), and more than three times as large as the jurors’ wrongful-convictions-per trial rate of 6.5%.
Many of you bemoan the quality of our juries. I join in your concern. We should all be aware, though, that juries protect us far better against wrongful convictions by our own government than do judges, police, and prosecutors.
No person should be subjected to the legal and emotional costs of a trial, much less a 38% chance of a wrongful conviction, unless the state is confident it can prove beyond a reasonable doubt the defendant is guilty. Any acquittal therefore represents a wrongful trial error, just as any conviction of an innocent person represents a wrongful conviction error. The police and prosecutors’ wrongful trial error is nearly twice as large as the judges’ wrongful-convictions-per-trial rate (22.2% versus 12.6%), and more than three times as large as the jurors’ wrongful-convictions-per trial rate of 6.5%.
Many of you bemoan the quality of our juries. I join in your concern. We should all be aware, though, that juries protect us far better against wrongful convictions by our own government than do judges, police, and prosecutors.
Notes:
1. Harry Kalven, Jr. was the Harry A. Bigelow Professor of Law at the University of Chicago Law School. He died in 1974.
2. Hans Zeisel was professor of Law and Sociology Emeritus at the University of Chicago. He was also a public opinion analyst, a sociologist, a statistician, and a Shakespearean scholar. He died in 1992.
3. I obtained the state court conviction data for year 2004 from Sourcebook for Criminal Justice Statistics Online, found at http://www.albany.edu/sourcebook/pdf/t546.pdf
4. I plotted many of my judge-jury agreement results against the inmate survey data. While most correlated reasonably well, no other set of judge-jury agreement results correlated with the inmate survey results as did the jurors’ wrongful conviction rate.
