Every time we make a decision based on a Statistic Hypothesis, there are four possible outcomes, for depending on our decision and the Null Hypothesis:
| Decision / Reality | is True | is False |
|---|---|---|
| Reject | Type I Error (false positive) | Correct decision (power) |
| Do not reject | Correct decision (confidence) | Type II Error (false negative) |
Definitions:
Type I:
Rejecting when it is actually true. Error
→ significance level.
Type II Error
Not rejecting when it is actually false. Error
.
Important Concepts
- Power of the test: , the probability of correctly rejecting a false .
- Confidence level: , the probability of correctly not rejecting a true .
Important relationships:
- and do not sum to 1. They are inversely related (generally, reducing one increases the other).
- The only way to reduce both simultaneously is to increase the sample size .
- The choice of depends on the context: which error is more costly?
Example from the document (parachutist):
- : The equipment is correctly installed (I jump).
- : The equipment is faulty (I do not jump).
- Type I error: I think the equipment is faulty, but it is actually fine → I lose the chance to jump (less severe).
- Type II error: I think the equipment is fine, but it is faulty → I jump and die (more severe). Here the Type II error is more serious, so we might tolerate a higher to keep low.