Type I and Type II Errors

tags: #hypothesis_testing

There are two possible errors that need to be taken into consideration when conducting Null-Hypothesis Significance Testing (NHST):

Type 1 Error ( $α$ )
Type 2 Error ( $β$ )

Type I Error - False Positive

Type 1 Error represents the probability of rejecting the null hypothesis when it is True.

Directly correlated to the alpha significance level:

Alpha represents the maximum allowable probability of rejecting the null when it is True.
When you set an alpha significance level, you are accepting the risk of making this error at that threshold.
For example, if $α = 0.05$ with a 95% confidence level of making the right decision, then you have a $5 %$ chance of rejecting the null hypothesis when it is in fact True.

Type II Error - False Negative

Type II Error is directly correlated with statistical power.

This represents the average or proportion of times (probability) of failing to reject the null hypothesis when it is FALSE (therefore, you are falsely accepting the null).

$β$ depends on the statistical power of the test (i.e., the probability of correctly rejecting the null when the alternative is True):

P o w e r = 1 - β

Trade-off: Type I vs Type II Error

The Type I and Type II error rates influence each other.

800

⚠ Switch to EXCALIDRAW VIEW in the MORE OPTIONS menu of this document. ⚠

Text Elements

Statistical Power = 1 - Type II Error (B)

Type 1 Error = Alpha Significance Level
Maximum allowable probability
of rejecting the null when it
is actually True (False Positive)
Statistical Power = 1 - Type II Error (B)
Type 2 Error (B)
Type 1 Error
1
alpha = 0.01
alpha = 0.05
When alpha = 0.05, we
have a confidence level of
95% of being correct.
When we decrease the alpha to 0.01,
such that we have a confidence level
of 99% of being correct, this also
increases the chance of FAILING TO
REJECT THE NULL WHEN IT IS ACTUALLY FALSE
(TYPE 2).
Requires stronger evidence
to reject the null.

This increases the probability
of not rejecting the null hypothesis
when it is True.
NOTE: Type 1 error rate is determined
by the significance level (alpha) that is
set for the test, and increasing power does
not necessarily change the alpha level.

This means there’s an important tradeoff between Type I and Type II errors; the lower the significance level, the higher the risk of type II error

Which Error is More Important?

Depending on the context - we may care about one kind of error more than than other

Example - Contagious Diseases (when cost of FN is high):

High false negatives rate (failing to detect someone who is positive) are more costly for increasing the spread of diseases.
In such cases, a HIGHER significance level (and therefore a lower Type II error rate) may be preferred, even if it means a higher risk of a false positive (Type I error).
In this case: $H_{0} : No disease$ and $H_{A} : Has Disease$ :

Type 1: Rejecting the Null when it is True (Has disease when they are NOT infected)

Type 2: Falsely Rejecting the Null when it is False (No disease when they ARE infected).

Contagious Disease Example

Higher $α$ = increased risk of False Positives (i.e., rejecting the null when it is false), therefore, detecting patients with disease even when they are NOT infected.
However, this reduces the risks of FN/Type 2 (i.e., missing infected patients), which is important for preventing spread.