Type I/II Errors

Video Transcription

This tutorial is going to teach you about the difference between a Type I error and a Type II error in a hypothesis test. Now, when you think about a hypothesis test as a decision making tool, it's possible that we could be making some errors. So, suppose we're in a clinical trial for a new drug. There are two possibilities for whether the drug is, in fact, an effective drug or not. Either it is or it's not. All right?

So now, when you use a hypothesis test as a decision-making tool, we might make a different decision. There are two possibilities for the decision we arrive at. We could go with-- we could fail to reject the null hypothesis of the drug being not effective. Or, we could reject it in favor of the alternative, the drug is effective. So, one of those two will be our conclusion.

But there's only one thing that's actually true and fact. Now, suppose these are the four different possibilities. Two of them are OK. These are correct decisions. If the drug was effective, we should reject the null hypothesis and decide that the drug is effective. And if the drug isn't effective, we should fail to reject the null hypothesis and decide that the drug isn't as effective as it would have needed to be to reject it. But if you look at these two cells here, the drug is, in fact, not effective but we decide that it is.

That is called a Type I error. The drug is not effective but we rejected the null hypothesis, anyway. Based on our data, we thought that we had enough evidence to reject the null hypothesis. But, in fact, the drug is not effective. And down here, the drug was effective but the data that we got didn't make it clear enough, and so we failed to reject the null hypothesis. This is another incorrect decision. It's called a Type II error. So, there are two different types of errors that we can make.

So, what are the consequences of each of those? So, think back to a Type I error versus a Type II error. A Type I error would have a consequence of we approve the drug and allow the public to have it, even though, in fact, it's not effective. But we're also unleashing all the potential negative side effects that this drug might have. So, there's really no upside here and some negative consequences.

In a Type II error, we would not allow the drug to go to market because we think it's not effective when, in fact, it is. And we would deny an effective drug that we didn't know was effective, based on our data. Our data made us think it wasn't, but it is, in fact, an effective drug to the public who might need it. This is another negative consequence. These errors always have negative consequences.

Now, which one are we more easily able to sleep at night with? It's probably, in this case, a Type II error. We have a hard time dealing with this idea of unleashing something that might hurt people on there, just because we think it might be effective. Typically, we need some really hard evidence. And if there's not hard evidence, we would deny the drug.

In the criminal justice system, juries are told to presume that someone is innocent until proven guilty. Meaning the null hypothesis is that the suspect is innocent, and the prosecution has to prove its case. So, what would a Type I and Type II error look like in this context? Pause the video and scribble down what you think a Type I and Type II error would look like.

What you should have come up with is a Type I error would be that the person is, in fact, innocent but they're convicted anyway. And a Type II error would be that the person is, in fact, guilty but the result of the trial is that they're acquitted. Obviously, both of these are problematic. But the criminal justice system in America puts a lot of safeguards in place to make sure that a Type I error doesn't happen very often.

In fact, the criminal justice system allows a Type II error to happen fairly frequently in order to reduce a Type I error. Because we think a Type I error is absolutely the worst thing we can do in this particular case. It's not always this way. Sometimes a Type II error is worse. It depends on the situation, and so, we have to analyze each situation to determine which one is a worse mistake to make.

So, to recap. When we talk about a hypothesis test as a decision-making tool, we might be making an error in our judgment. It's not that we made a mistake, but the result that we choose might not match what is, in fact, really the case. A Type I error is when the null hypothesis is rejected when it's true. In fact, that is the alpha level. That is the significance level that we create in a hypothesis test. And a Type II error is when the null hypothesis is not rejected. But in reality, it's actually not the case, it's false. So, it's false but we didn't reject it.

And the severity of these errors depends on the context. In both the examples that we did, a Type I error was worse. But there is conceivably some scenarios where a Type II error might be worse. So, we talked about Type I and Type II errors. Good luck and we'll see you next time.