Type I and Type II Errors and Decision Rules

A type I error is rejecting a true null hypothesis (probability α, the significance level), and a type II error is failing to reject a false one (probability β); the power of a test, 1 − β, is the chance of detecting a real effect, and the decision rule fixes α and then tries to make β small. In brief: in the test of H₀: μ = 100 with σ = 14 and n = 49 (standard error 2), the two-sided 5% rule rejects when x̄ is below 96.08 or above 103.92. If the true mean is 105, the type II error probability is about 0.29, so the power is about 0.71.

Every test can go wrong in two ways, and the two risks pull in opposite directions. This guide defines both errors, shows how to write a decision rule with a critical region, computes α, β and the power in worked examples, explains what raises power, and relates the errors to false positives and false negatives in diagnostic tests, with a table.

What are type I and type II errors?

There are four combinations of the truth and the decision. If H₀ is true and we reject it, that is a type I error (a false alarm), with probability α. If H₀ is false and we do not reject it, that is a type II error (a missed effect), with probability β, which depends on the true value of the parameter. The power 1 − β is the probability of rejecting a false H₀. The two other combinations (keeping a true H₀ and rejecting a false one) are correct decisions.

What is a decision rule?

A decision rule specifies the critical region, the set of sample outcomes for which H₀ is rejected, so that the probability of landing in it under H₀ equals α. For a z-test it is a statement such as “reject if |z| > 1.96” (α = 0.05, two-sided) or “reject if z > 1.645” (α = 0.05, one-sided). It can be translated into a rule for the sample mean.

Example 1: the rule in terms of x̄

H₀: μ = 100, σ = 14, n = 49, so the standard error is 2. Two-sided, α = 0.05: reject if |x̄ − 100| > 1.96 · 2 = 3.92, i.e. if x̄ < 96.08 or x̄ > 103.92. The probability of this under H₀ is 0.05 by construction, the type I error rate.

Example 2: a one-sided rule

For H₁: μ > 100 at α = 0.05: reject if x̄ > 100 + 1.645 · 2 = 103.29.

How do you compute β and the power?

Fix a specific alternative value μ₁. Then β is the probability that x̄ falls in the non-rejection region when the true mean is μ₁, computed with the normal distribution centered at μ₁ (the standard error stays 2).

Example 3: β for the two-sided rule

True μ = 105. β = P(96.08 < x̄ < 103.92) = Φ((103.92 − 105)/2) − Φ((96.08 − 105)/2) = Φ(−0.54) − Φ(−4.46) ≈ 0.2946 − 0 = 0.2946. The power is 1 − 0.2946 ≈ 0.705.

Example 4: β for the one-sided rule

With the rule x̄ > 103.29 and true μ = 105: β = Φ((103.29 − 105)/2) = Φ(−0.855) ≈ 0.196, so the power is about 0.804. The one-sided test is more powerful for the effect in the stated direction, because the whole α is spent on that side.

Example 5: the trade-off between α and β

Lowering α to 0.01 in the one-sided test moves the critical value to 100 + 2.326 · 2 = 104.65. Then β = Φ((104.65 − 105)/2) = Φ(−0.175) ≈ 0.43. Fewer false alarms, more missed effects: for a fixed sample size, α and β cannot both be made small.

What increases the power?

• A larger sample size (a smaller standard error).
• A larger true effect (μ₁ farther from μ₀).
• A larger α (at the price of more false alarms).
• Less variability in the data (a smaller σ).
• A one-sided instead of a two-sided test, when a direction is justified in advance.

Example 6: the effect of the sample size

With n = 196 the standard error is 1. The one-sided 5% rule rejects for x̄ > 101.645, and for true μ = 105, β = Φ((101.645 − 105)/1) = Φ(−3.355) ≈ 0.0004: the power is almost 1. Quadrupling the sample size halves the standard error. Planning n in advance for a desired power (often 80% or 90%) is standard practice.

How do the errors relate to medical tests?

A diagnostic test is a hypothesis test with H₀: “healthy”. A false positive is a type I error, and a false negative is a type II error. The sensitivity is the power (1 − β), and the specificity is 1 − α.

Example 7: why a positive result is not conclusive

A disease affects 1% of people. The test has the sensitivity 99% and the specificity 95% (false positive rate α = 5%). Out of 10,000 people, 100 are ill, of whom 99 test positive; of the 9,900 healthy, 495 test positive. So P(ill | positive) = 99/(99 + 495) = 1/6 ≈ 0.167, as the formula of Bayes’ theorem gives. A small α does not make a positive result reliable when the condition is rare.

Example 8: many tests

If 20 independent true null hypotheses are each tested at α = 0.05, the probability of at least one false rejection is 1 − 0.95²⁰ ≈ 0.64. This is the multiple testing problem, addressed by corrections such as Bonferroni’s (using α/20 for each test).

Table of outcomes

Where does the framework come from?

Jerzy Neyman and Egon Pearson introduced the two types of error and the idea of power in their 1933 paper, proving that for simple hypotheses the likelihood ratio test is the most powerful at a given α (the Neyman–Pearson lemma). Fisher criticized parts of this approach, preferring p-values as measures of evidence. The convention α = 0.05 comes from Fisher’s suggestions of the 1920s and is a convention, not a law of nature.

Which mistakes are common?

Confusing α with the probability that H₀ is true

α is the rate of false rejections when H₀ is true, not the probability of H₀ given a rejection.

Computing β without a specific alternative

β depends on the true parameter value. It is meaningful only for a chosen μ₁, and it grows as μ₁ approaches μ₀.

Treating “not significant” as “no effect”

A low-powered test often fails to detect real effects. Check the power and the confidence interval.

Choosing α after the data

The significance level and the decision rule must be set before looking at the results, otherwise the stated error rate is meaningless.