Thomas and the Lie Detector

This post is partially based on Eliezer Yudowsky’s popular introduction to Bayes’ Theorem, which I do recommend if you want a more rigorous explanation, but is meant to illustrate a few different points along the way.

Imaging that you are a cyberpunk detective living in a dystopian version of Barcelona where one person in four is a murder. There is so much murder going on that your grizzled old chief decides to arrest people en masse and make you sort it out. In order to ease this process you’ve been provided with a state of the art lie detector, the Decto-1000. Unlike the competing brand this one is confirmed by Science!^TM to work.

You bring in your first subject and begin the interrogation.

“Are you  murderer?”
“No.”

The machine flashes LIE on its fancy screen.

Interrogation done. Throw her in the slammer and move on to the next one, right?

(No, wrong, obviously. I’d never ask a question like that when the answer is yes.)

You’re probably aware that you sometimes miss lies when people tell them. It stands to reason that the machine probably doesn’t get all of them either. Let’s go look at the box that the Decto-1000 came in.

On the cover it says “Catches 90% of Lies!”

Okay, so she’s probably a murder. Good enough. You live in a dystopia anyway.

Not so fast. That claim is pretty easy to manipulate. I, for example, have the power to catch far more than 90% of lies. Indeed my powers allow me to perform this feat across time and space without even meeting the potential liar.

Think of a lie or a falsehood then highlight this next section.

You thought of a lie.

With this method I can catch 100% of lies! Of course, I will also wrongly say that every truth I get is a lie so my technique is clearly a poor one and, yes, something of a strawman. The point is the claim on the front of the box is not nearly enough information. They are providing only a measure of “sensitivity” when we also need a measure of “specificity,” the machine’s ability to tell the difference between truths and lies. To be a bit more technical: Sensitivity measures how often you get a true positive and specificity measures how often you get a false positive.

Since sensitivity and specificity are a bit too esoteric and technical to be put on a flashy advertisement you’ll have to go digging through the manual to find this information. After going through some technicolor images of human brains being scanned by fMRI, explanations of microexpressions, and analysis of the logic behind voice stress analysis (all very impressive I assure you) we find the listing for sensitivity and specificity.

Conveniently both are exactly 90%. This means that when given a lie it calls it a lie 90% of the time and when given a truth it calls it a truth 90% of the time. As a result it is not wrong to say that the machine is correct 90% of the time. However, we will see that it is a bit misleading and accepting it without consideration can lead to make serious mistakes.

To clean up the language in this next section it is important to note that I’ll be assuming everyone denies being a murderer. In reality there are some people who genuinely confess and some people who falsely confess and this probably applies in dystopian Barcelona as well. Fortunately it doesn’t change our results if they do, it only makes the language much more confusing.

 

Back in the holding cells are 1000 people who were grabbed off the street to be interrogated. Recall that we know 250 of them are actually murders and 750 are not.

Before we look at the results try to imagine what results we will get.

Will the machine say that 250 of them are murderers? Will it say that 750 are innocent?

. . .

. . .

When we test all of them we get the following results: 300 murderers and 700 innocents.

The first thing you can see here is that if we believe the machine unconditionally we’re a good distance off from what the numbers really should be. That’s not surprising since the machine isn’t perfect.

Of the 250 murderers the machine will wrongly tell us to let 25 of them go (10%) and correctly tell us to lock up 225 of them (90%).

Of the 750 non-murderers the machine will wrongly tell us to lock up 75 of them (10%) and correctly tell us to let 675 of them (90%).

We’re seeing that the machine is right 90% of the time just like the advertisements say (900 of the 1000 answers are correct) yet I’ve already warned you not to think that way. This is where I part ways with Eliezer Yudowsky. Knowing that the machine is right 90% of the time is a useful piece of information even though we may misuse that information

Let me ask you a question: When the first woman you interviewed denied being a murder and the machine said she was lying what was the probability the machine was wrong?

Go look at the breakdown up top.

Do you see it? The machine is wrong about lies 25% of the time, not 10% like we might have thought. There are 225 people correctly accused of murder and 75 people wrongly accused of murder.

The opposite happens when the machine detects a truth. It is wrong about truths only 4% of the time. There are 675 people correctly let go and 25 wrongly let go.

It is very simple to understand what is happening here: Even in our horrible dystopia most people aren’t murderers. As a result the majority of our false positives, in absolute terms, come from non-murderers.

Now I’ll give you two situations so we can see how and why these differences matter.

The grizzled old police veteran who runs your precinct (he has a robot leg and an eyepatch) comes in and looks at all of this that we’ve gone over. He’s a savvy person and he tells you that you should just put everyone the machine tells you is a murderer in prison. After all, he points out, this will get 90% of the murderers off the streets.

Is he right?

Yes, he is. This is similar to what is known as the long run interpretation of statistics. In the long run the machine is right 90% of the time. Now perhaps a 10% error rate is unacceptably high but the logic behind the order is perfectly sound.

Next you are called into court to testify about the lie detector results for the woman you interviewed at the start of the article and found to be a murderer. The lawyer for the prosecution tells the jury that the machine only has a 10% chance of being wrong so they must convict her. When the lawyer for the defense stands up he asks you if this is true.

You must say no, that the prosecution is misunderstanding the statistics. The machine is right 90% of the time but on each trial it is either has a 75% chance of being right or 96% chance of being right. On any given trial the probability of being correct is never 90%. In the case of the accused woman there is a 25% that the machine was wrong.

This difference is why you may see people write weird things like “just because this method is right 95% of the time doesn’t mean it’s right with 95% probability” without any explanation when talking about statistics. There are varying ways of talking about probability.

 

Actually that’s what made me write this post in the first place. For some reason no one, anywhere, ever, feels the need to explain this very simple difference. The next post will explore the issue of “prior distributions” which were a critical piece of information in this post. Which is so say: “How did I know one in four people were murderers in the first place?”