Watson and the Future of Everything

The world changed Monday night. Perhaps you missed it. Perhaps you were eating dinner or watching the news. Perhaps you were taking a walk or making love. Perhaps you were watching Jeopardy and did not fully appreciate the world changed before your very eyes.

On Monday, the two best Jeopardy players in the world squared off against IBM's Watson computer, and the world changed.

Let me begin by introducing the contestants.

On your far left is Ken Jennings. Ken won 74 Jeopardy matches in a row, and earned $2,520,700 in the process. He also earned $2,000 in his 75th game when he came in second to Nancy Zerg, who came in third during her next game. Jennings then won $500,000 when he placed second in the Jeopardy Ultimate Tournament of Champions. That's a grand total of $3,022,700.

It is almost statistically impossible that Ken Jennings won that many games in a row. Prior to Jennings' streak of 74 games in a row, the longest streak was a pitiful (only by comparison) 8 games in a row by Tom Walsh. Jennings streak was more than 9 times longer than Walsh's streak. By comparison, Joe DiMaggio's 56 game hitting streak, considered by many to be the greatest feat ever in baseball, is only 1.3 times as long as Pete Rose's 44 game streak, second longest in the modern era.

On your far right is Brad Rutter. Brad Rutter has never lost a Jeopardy game to another human. When Rutter first appeared on Jeopardy in 2000, contestants were limited to five games in a row. (That rule was eliminated in 2003, one year before Ken Jennings began his remarkable run.) Rutter, however, later won each of the three Tournaments of Champions in which he was invited to play.

In the third of those tournaments, the Ultimate Tournament of Champions, Rutter beat Ken Jennings and took home $2,100,000 for his efforts. In all, Rutter has earned $3,255,102 for his initial five Jeopardy games and his three Jeopardy tournaments, making him the top Jeopardy earner of all time.

In the center is Watson, a massively parallel IBM supercomputer.

Born in 2007, Watson has (at the hands of some amazing teachers) learned to understand the subtleties of natural language. Consider Watson's rapid fire questions to some of Jeopardy's ambiguous, nuanced questions in the practice session recorded before the formal contest.

Since Jeopardy might present any of an infinite number of convoluted answers to the contestants, there was no way Watson's teachers could simply program it to select the right question from a database of question/answer pairs. They had to teach Watson to understand natural language. That was the challenge. That's why they created Watson. It was to be the first computer to understand the intricacies and subtleties of natural language well enough to compete on Jeopardy.

Watson's knowledge is kept in much the same way as Google stores all its data. When faced with a Jeopardy answer, Watson uses keywords, much as Google does, to collect a long list of possible answers from its stored information.

It is at this point, however, that IBM has gone beyond Google and put itself back into the lead as the world's foremost computing company. Watson is programmed to use its natural language skills to decide which of the many possible answers is most likely correct, and to answer only if it is confident beyond some threshold value.

On its web site, IBM provides a 22 minute video explaining how sophisticated Watson's decision-making can be. Consider the Jeopardy answer "In May 1898 Portugal celebrated the 400th anniversary of this explorer's arrival in India." The graphic below represents one possible result Watson might quickly find. Based on a keyword search, Watson finds a match for "celebrated", "may", "anniversary", "Portugal", and "India". Watson also finds that the link is associated with some guy named Gary.

As it turns out, "Who is Gary?" is unlikely to be the question Jeopardy was searching for.

The link segment reads "In May, Gary arrived in India after he celebrated his anniversary in Portugal."  As humans, we can easily see Gary is not who we are looking for, because we understand natural language. No computer, other than Watson, shares that understanding.

For Google, the Gary link would be a terrific match. For Watson, however, it would be but one of many to be rejected as improbable. Instead, Watson would home in on a Vasco de Gama link, which reads "On the 27th of May in 1498, Vasco de Gama landed in Kappad Beach."

The astonishing point here is that the Vasco de Gama link has only one matching keyword, and that keyword is the commonly used and ambiguous word "may." Holy linguistics, Batman! There are millions and millions of documents out there with the word "may" in them. Why did Watson decide to consider this one?

Watson picked up on this link because Watson has been programmed, as have we humans, to apply temporal reasoning, geospatial reasoning, and statistical paraphrasing.

While the Vasco de Gama link doesn't actually mention 1898 as being the 400th anniversary of anything, it does mention the year 1498. Watson knows that 1898 minus 1498 is equal to 400, and  therefore recognizes two very strong clues in the link, strong clues missing from the Gary link.

Furthermore, while the Vasco de Gama link doesn't actually mention India, Watson knows that Kappad Beach is in India. It knows that from a few of the other millions of links it has in its memory. So that's another big clue.

Finally, Watson understands statistical paraphrasing. It knows that "arrival in" is a good match of "landed in."

Watson combines all the information at hand and assigns a high probability that Vasco de Gama is the correct result. No other answer it considers has a higher probability. Since the Vasco de Gama probability is greater than the confidence threshold, it buzzes in and says, in the best computerized voice to date, "Who is Vasco de Gama?"

When Watson finds a reasonably probable answer, it typically buzzes in (using a mechanical finger) more quickly than its human competitors. This too is impressive, given all the mental work it has to do. I learn from the video that if Watson used my home computer as its processor, it would need several hours to decide each question. Watson, however, uses thousands of processors working at the same time, working in parallel.

So how did all this pan out in the actual game? As I said in the beginning, the world changed on Monday night. Watson finished the first game (played over two nights) with $35,734. Rutter had $10,400. Jennings had $4,800.

After three days and two games, the totals were:

Watson: $77,147

Jennings: $24,000

Rutter: $21,600

In my next post, I'll discuss what this might mean to the quarter million people we have wrongfully incarcerated in America.

Proof Beyond the Flip of a Coin

In a previous post, I described the "Proof Beyond a Reasonable Doubt" as a convenient judicial myth. At the end of that post, I presented the following summary plot comparing the idealized concept against the reality of jury and judge verdicts.  I include that plot below for easy reference. Click to enlarge and clarify.

Judges and scholars, if forced to quantify the reasonable doubt threshold, tend place the threshold near 90%. While that number flies in the face of the plot above, it comports with quite a few studies attempting to quantify the threshold.

In a an earlier post, I presented evidence that jurors self-report that they place the threshold at 85%. First order, this means that if you are innocent but on trial, you stand a 15% chance of being convicted. As horrific as this is, I believe it seriously understates the problem faced by the (relatively few) innocent people facing a jury.

From my analysis of judge-jury agreement data, I calculate that an innocent person has a 28% chance of being convicted. From Professor (of Statistics) Bruce Spencer's analysis of judge-jury agreement data, he calculates that an innocent person has a 25% chance of being convicted. (Professor Spencer is careful to limit his conclusion to the sample of trials he studied.) I present comparitive summaries of our results in Judges Gone Wild: Revised, and one of the plots from that post below. Click to enlarge and clarify.

If jurors do in fact use an 85% threshold for reasonable doubt, then I am probably wrong in my calculation that 28% of innocent people are convicted (rather than acquitted) by juries. In this case, I would be pleased to be wrong. Unfortunately, I don't believe I am.

In his 1993 book Inside the Juror, Reid Hastie actually looked at three approaches for determining the reasonable doubt threshold used by jurors. The 85% threshold is based on the self-reporting approach. Hastie calls that the direct approach. I call it the least reliable approach. In this post, I'll describe the results from a the second approach, what Hastie calls the parallel approach. Quoting from Hastie, here's how the  first of the parallel approach studies worked. It's pretty clever, actually.

Two independent groups of subjects made judgments after listening to an audiotaped homicide trial; one group made a dichotomous [either-or] guilty-innocent verdict judgement and the other half of the subjects rate the probability that the defendant had committed the crime.

Assuming the two groups were valid samples of our jury pool, the researchers could determine the actual reasonable doubt threshold by comparing the guilty votes with the probability of guilt assessments. Neither group knew how the other group would vote or rate the probability of innocence.

The initial study was repeated by two other investigators. In the two follow-on studies, only one group was used. That group was asked to provide both a verdict and a probability of innocence. Below, I tabulate the results from the three studies as reported by Hastie.

Using this method of determining an actual (rather than self-reported) reasonable doubt threshold, the first-order conclusion is that jurors will find 34% of the innocent defendants guilty. That rate of innocents being convicted is greater even than the 28% rate I calculated and the 25% rate Professor Spencer calculated. Perhaps one could fault the studies for relying entirely on students, though during the self-reporting studies the students responded substantially the same as other potential jurors.

Alternatively, one might consider the possibility that juries convict somewhere between a quarter and a third of the innocent defendants who come before them. That's bad, but better than the flip of a coin.

But not much better.

The Impending Execution of Timothy Wayne Adams

Timothy Wayne Adams is scheduled to be executed by the people of Texas on 22 Feb 2011 for the murder of his  19-month-old son, Timothy Wayne Adams, Jr., No one disputes Adams committed the murder. Those who argue against his execution ask for clemency based on the entirety of his life. From Timothy Adams: A Case for Clemency:

In 2002, Timothy Wayne Adams shot and killed his 19-month-old son, Timothy Wayne Adams, Jr. during a standoff with Houston police. After a fight with his wife escalated out-of-hand, Mr. Adams “snapped” and decided to take his own life and the life of his youngest son. Mr. Adams did not take his own life on that horrible day due to the support of his family and friends, who spoke to him over the phone and told him that his life was worth saving. One of those friends convinced him to speak to an HPD negotiator, who in turn persuaded Mr. Adams to let go of his suicidal thoughts and end the standoff. Ultimately, Mr. Adams left his apartment and surrendered peacefully to police a few hours after the ordeal began.

From the moment that Mr. Adams was taken into police custody, he has taken full responsibility for his actions. Mr. Adams realizes that it is nearly impossible for the Board, as well as any citizen in our society, to comprehend what could lead a father to kill his own son. In no way would Mr. Adams ever try to justify his actions; what he did was wrong, plain and simple.  He would take back his actions that horrible day in an instant if it were possible.

What Mr. Adams requests is that he have the opportunity to tell his life story, something that the jury did not hear at his trial. Mr. Adams’s defense counsel did not present crucial mitigating evidence to counter the prosecution’s contention that Mr. Adams was a future danger to society or to show that his life was worth saving. Consequently, the jury learned almost no information about Mr. Adams’s life and upbringing, which would have helped them determine that Mr. Adams, a deeply religious, hard-working family man, was not a future danger to society and never will be.

Lacking this mitigating evidence, it is perhaps not surprising that the jury sentenced Mr. Adams to death. But since learning additional information about Mr. Adams’s character and background, jurors Rebecca Hayes, Ngoc Duong, and Kathryn Starling have urged the Board to commute Mr. Adams’s death sentence to a life sentence. They believe that information relating to Mr. Adams’s upbringing, deep devotion to religion, and mental state would have caused them to stick with their initial inclination, which was to spare Mr. Adams and sentence him to life in prison.

With this petition, Mr. Adams seeks to show the Board that February 20, 2002 was an aberration in his life.  Before that day, Mr. Adams had never been arrested or convicted of a crime.  Since that day, he has not had a single disciplinary write-up in prison.  Mr. Adams wants to share his life story to show the Board that, before committing this crime, he was a religious, hard-working individual who suffered from extreme anxiety but who loved and provided for his family just the same. Since being incarcerated, he has had the opportunity to reflect on his actions, which has brought him closer to God and deepened his devotion to Jesus Christ. 

In telling his story, Mr. Adams wants to give his family the opportunity to speak on his behalf, something that defense counsel prevented them from doing at trial.  In this case, the defendant’s family is unfortunately also the victim’s family—Mr. Adams’s father lost his grandson, his siblings lost their nephew, and his oldest son lost his half-brother. Yet, none of these family members were able to stand up in front of the jury to describe the severe hurt and suffering they had endured as a result of Mr. Adams’s actions. Nor were they able to explain that, despite their pain, they still supported and loved Mr. Adams and did not want to lose their son, brother, and father to this tragedy as well.

Mr. Adams has filed a clemency petition with the Texas Board of Pardons and Paroles, asking them to vote to spare his life.  Nothing good will come from executing Tim and causing his family any more unimaginable pain and anguish. If ever there was a man who deserved clemency, it is Tim Adams.

I oppose the execution of people who might be factually innocent of the crime for which they are to die. I suspect that to prevent the execution of the factually innocent, we might have to ban the death penalty entirely.

With respect specifically to the execution of Timothy Wayne Adams, I stand mute.

A Clarification

A couple of readers have commented about the penalty phase of Michael Wayne Hall's trial. That is a point worthy of discussion, but I'll leave it for those who comment and those who write elsewhere.

For clarification, my focus is on actual, factual guilt or innocence. I am primarily interested in whether the person committed or participated in the crime. This is different than legal guilt or innocence.

A person may be legally innocent though he comitted or participated in the crime. The jury may have found that the state failed to prove its case beyond a reasonable doubt, or that the defendant was legally insane at the time of the crime. I don't have a fundamental objection to either situation, though I would prefer that sane, factually guilty people pay for their crime and that insane people not pay for their insanity.

More disturbingly, at least to me, a person may be legally guilty though he neither committed nor participated in the crime. The jury may have concluded that the state proved its case beyond a reasonable doubt, though the defendant was factually innocent (as in the case of Byron Case) or though no crime in fact had occurred (as in the case of Michael Ledford.) These are the cases that motivate me, and threaten to consume me.

The Impending Execution of Frank Spisak

Frank Spisak, Then

Frank G. Spisak is scheduled to be executed by the people of Ohio on 17 Feb 2011 for the murder of three people. I find no credible evidence that Spisak might be factually innocent of the crimes.

In fact, Spisak has always admitted to the killings. He has yet to deny them. His defense has always been that he is crazy. He may be right.

That's Spisak at the right, during his trial. Note the moustache. Spisak was an admirer of Adolph Hitler, a Nazi sympathizer and a neo-Nazi. He was also a gender-confused transexual and a serial killer.

From A Gender Variance Who's Who, I offer the following insight.

Frank was the son of a factory worker in Cleveland, Ohio, who moved home because too many blacks had moved into the area. He had childhood fantasies of being a woman.  He married at 22, and they had one daughter. He read constantly on Hitler and the Nazis and his wife tried to ignore the issue. He worked in various factory jobs. 

At 25 he suffered a head injury in a car accident. In 1977, he started dressing as female. As Frankie Ann, Spisak received treatment from the Gender Dysphoria clinic at Cleveland Metropolitan Hospital, lived as female full-time, changed her ID, saw a psychologist and started taking female hormones. She was also saving for surgery. She was fired from her factory job after turning up as female. Frankie Ann found employment with a maker of eyeglasses until found not to be a woman. She tried prostitution until charged with solicitation, and briefly worked as a Kelly Girl temp. When she brought a trans woman home for sex, her wife and daughter moved out. 

But in 1979 the Nazi Frank took over. He collected Nazi memorablia, and played Hitler’s speeches on his stereo. He was stockpiling guns and ammunition. He was also dating a black female prostitute. In February 1982 he found a black preacher in the next stall in the men’s toilet at the Cleveland State University Library and shot him dead. In June he shot a black man at a train station, who survived. In August he returned to to CSU and shot at a female student in the ladies room but missed. He became paranoid about a maintenance worker at CSU who might identify him, and so shot him dead, again in the men’s toilets. The next night he killed a young man waiting at a bus stop.

A week later he got drunk and shot his gun out of the window of his house. For this he was arrested, but was allowed to post bond. An anonymous phone call suggested to the police that they re-examine the guns that they had taken from Spisak’s house, and they were found to match those used in the killings.

Once arrested he admitted the murders, grew a Hitler-style mustache and carried a copy of Mein Kampf. At his 1983 trial, his attorney presented him as crazy but he was not found to be ‘legally insane’. He declared that he was under orders from God, and that Jews were to blame for his transvestite episodes, having seized control of his mind. The jury quickly found him guilty and sentenced him to death.

In jail, Spisak lobbied for a sex change, filed a lawsuit to force the state to refer to him as a woman, and appealed the death sentence. In 2006 the appeal court ruled that he had not received a fair defense and struck down his death sentence. In 2010, this was reversed.

The accounts of the crimes are consistent with those found in Spisak's appeal Spisak v. Mitchell:

On February 1, 1982, the body of the Reverend Horace T. Rickerson was discovered by a fellow student on the floor of a restroom on the Cleveland State University campus. Rickerson had been shot seven times by an assailant from a distance of more than eighteen inches. Four spent bullet casings were recovered from the scene.

On the evening of June 4, 1982, John Hardaway was shot seven times while waiting for an RTA train at the West 117th Rapid Station in Cleveland. He observed a man walking up the platform steps and had turned away when the man opened fire on him. Hardaway survived the shooting, and was later able to identify his assailant as the appellant, Frank G. Spisak Three pellets and seven shell casings were recovered from the scene.

At approximately 5:00 p.m. on August 9, 1982, Coletta Dartt, an employee of Cleveland State University, left her office to use the restroom. Upon exiting the stall, she encountered the appellant, holding a gun, who ordered her back into the stall. Instead, Dartt shoved appellant out of the way and ran down the hallway. Appellant shot at her, but missed. A pellet was later removed from a wall in the hallway. Dartt identified the appellant as her assailant.

On August 27, 1982, the body of Timothy Sheehan, an employee of Cleveland State University, was discovered in a restroom at the university by a security guard. The guard had been searching for Sheehan after his office reported that he had failed to answer his beeper page. Sheehan had been shot four times, and two pellets were retrieved from the scene.

On the morning of August 30, 1982, the body of a young student, Brian Warford, was discovered in a bus shelter on the campus of Cleveland State University. Warford died from a single gunshot wound to the head, although five spent.22 caliber casings were recovered from the scene.

On September 4, 1982, Cleveland police answered a call that a man was firing shots from a window at 1367 East 53rd Street. The police were directed to appellant's apartment and appellant, after admitting he had fired one shot, invited the officers inside. A shotgun and a .22 caliber automatic pistol were observed in the room. Appellant made a suspicious move toward the couch but was stopped by one of the officers who discovered a loaded .38 caliber handgun and a two-shot derringer under the couch cushions. Appellant was arrested for possession of unregistered handguns and discharging firearms within city limits, but was later released on bond.

The weapons, however, were confiscated. Early the next day, an anonymous caller told police that the confiscated weapons had been used in the Cleveland State University shootings. Ballistics tests confirmed the tip. A warrant was obtained, and the police returned to Spisak's apartment, confiscating several items including newspaper clippings of the homicides and Nazi-White Power paraphernalia. Appellant was later arrested, hiding in the basement of a friend's house. During a brief search of appellant's suitcase at the scene, police discovered the beeper pager belonging to Sheehan.

Appellant later admitted to shooting Rickerson for allegedly making a homosexual advance toward him; to killing Sheehan as a possible witness to the Rickerson shooting; to killing Warford while on a "hunting party" looking for a black person to kill; and finally, to shooting at Dartt and to shooting Hardaway. He also told police he had replaced the barrel of the .22 caliber handgun in order to conceal the murder weapon.

Frank Spisak, Now

I oppose the execution of people who might be factually innocent of the crime for which they are to die. I suspect that to prevent the execution of the factually innocent, we might have to ban the death penalty entirely.

With respect specifically to the execution of Frank Spisak, I stand mute.