Law of Large Numbers and Prediction Intervals

The law of large numbers describes the stabilisation of averages under suitable assumptions, while prediction intervals describe uncertainty about future observations. In brief: many independent fair coin tosses tend to produce a head proportion near one half, but the next toss remains uncertain. Learning the average more precisely does not eliminate the variability of individual outcomes.

These ideas are related but answer different questions. One concerns the behaviour of an average as the sample grows; the other concerns a future value or future group of values. This guide explains the distinction, develops probability and interval examples and states the assumptions behind a standard normal-model prediction interval.

What does the law of large numbers say?

A standard version considers independent, identically distributed observations with a finite mean μ. Their sample average approaches μ in probability as the number of observations grows: for any fixed positive tolerance, the probability that the average differs from μ by more than that tolerance tends to zero.

For Bernoulli observations taking values 1 for success and 0 for failure, the average is the success proportion. Its limiting target is the success probability p. Under independent fair coin tosses, the observed head proportion therefore tends toward 0.5 in this probabilistic sense.

The assumptions matter. A changing mechanism, dependent observations or a distribution without the required finite mean may fall outside this basic version. There are broader laws under other conditions, but a large dataset alone does not guarantee that an average estimates a stable, meaningful target.

What does the law not guarantee?

The law does not promise exact agreement after a fixed number of trials or a steadily decreasing error after every new observation. A running proportion can move away from its target temporarily, even late in a sequence. The theorem concerns limiting probability, not a monotonic path for each finite dataset.

It also does not make an independent process compensate for earlier results. After several tails, the next head probability remains 0.5 under the independent fair-coin model. A past imbalance can become a smaller fraction of a growing total without requiring future tosses to favour the opposite result.

More observations do not remove systematic bias. If a measurement procedure is consistently shifted or a sample excludes part of the population, collecting more of the same data can estimate the wrong target very precisely. Sampling design and model stability remain important alongside sample size.

Why do averages become more stable than individual observations?

For independent identically distributed observations with finite variance σ², the sample mean has variance σ²/n and standard deviation σ/√n. The distribution of the average becomes more concentrated as n grows, even though the distribution of each individual observation stays the same.

For a fair coin indicator, σ² = 0.25. The head proportion has standard deviation 0.5/√n, giving 0.05 at n = 100 and 0.025 at n = 400. Quadrupling the sample size halves this standard deviation rather than reducing it by a factor of four.

This formula describes typical sampling variability, not a guaranteed maximum error. A realised proportion can lie farther away. A probability bound or distributional calculation is needed to attach a stated coverage level to a range, and the law of large numbers alone does not supply an exact finite-sample interval.

What is a prediction interval?

A prediction interval is a range constructed to cover a specified future observation or future statistic with a stated coverage level under a model. Its target might be one new measurement, the average of several new measurements or a future success count. The target must be named because the uncertainty differs across these questions.

A 95% prediction procedure is designed so that, across repeated data collection and future outcomes under its assumptions, the interval covers the designated future target about 95% of the time. It is not a guarantee that the next outcome lies inside, and its coverage can fail if the model is unsuitable.

A prediction interval for one observation includes both uncertainty about the estimated centre and the natural variation of the new observation. A confidence interval for a population mean addresses uncertainty about that fixed mean. The two intervals therefore answer different questions and usually have different widths.

How can a known distribution give a prediction range?

If a future measurement is modelled as normal with known mean 50 and known standard deviation 4, a central 95% range is approximately 50 ± 1.96 × 4. This gives about 42.16 to 57.84. The calculation uses the known distribution’s quantiles rather than estimating its parameters from a sample.

For a future count, use the appropriate discrete distribution instead. If a count is binomial, an interval can be chosen by summing binomial probabilities until the desired coverage is reached. Because counts take integer values, the achieved coverage may be at least the target rather than exactly equal to it.

A normal approximation to a count may be useful under suitable conditions, but it should not be presented as an exact guarantee. Parameter uncertainty also matters if the success probability was estimated rather than known. A plug-in range that ignores that uncertainty can be too narrow.

What is a prediction interval for one new normal observation?

For n independent observations from a normal population with unknown mean and variance, a standard two-sided prediction interval for one independent future observation is x̄ ± t* s√(1 + 1/n). Here s is the sample standard deviation and t* is the appropriate Student t quantile with n − 1 degrees of freedom.

The term 1/n represents uncertainty in estimating the mean, while the additional 1 represents the variability of the new observation. By comparison, a confidence interval for the mean uses x̄ ± t*s/√n under the same normal model. The prediction interval is wider because it has a different target.

For n = 25, x̄ = 50 and s = 4, a 95% interval uses t* approximately 2.064. The prediction margin is about 2.064 × 4 × √1.04 = 8.42, giving approximately 41.58 to 58.42. The corresponding mean-confidence margin is about 1.65, illustrating the substantial width difference.

Why does more data not make a single-value prediction interval vanish?

As n increases, the estimated mean becomes more precise, but a new individual observation still varies around that mean. In the normal-model prediction formula, √(1 + 1/n) approaches 1 rather than zero. The irreducible individual spread remains even with extensive information about the population centre.

Predicting an average of m independent future observations is different. Under the same normal model, its prediction standard-error factor is s√(1/n + 1/m). Averaging future values reduces their random variation, while the 1/n term still reflects uncertainty in the estimated mean from the original sample.

This distinction connects prediction with the law of large numbers. Large averages can stabilise even when single observations remain variable. Before interpreting an interval, ask whether it targets a population mean, one future observation or a future average; similar-looking formulas can represent substantially different uncertainty questions.

What assumptions and mistakes matter most?

Define the prediction target, coverage level and model before calculating. Check whether future observations are expected to follow the same distribution as the data and whether independence is plausible. The exact t-based interval above relies on normality and the stated sampling structure, not simply on having a moderately sized dataset.

Do not confuse the law of large numbers with the central limit theorem. The first concerns convergence of averages toward a mean; the second concerns an approximate distribution for appropriately standardised sums or averages under its own conditions. Neither automatically makes the raw observations normally distributed.

Finally, avoid describing a confidence interval for the mean as a range containing 95% of future observations. Report the interval’s target explicitly, retain sensible precision and treat coverage as conditional on the model. Stable averages, accurate mean estimates and reliable individual predictions are related goals, but they are not interchangeable.