Evaluating Methods and Improving Experimental Validity
Evaluating an experiment means judging how well the method and the results answer the original question, by identifying the sources of error and uncertainty, deciding which has the greatest effect, and proposing specific improvements; validity is the extent to which the experiment measures what it claims to measure. In brief: compare the result with the accepted value using the uncertainty, decide whether the difference is explained by random errors or points to systematic errors, rank the limitations by their effect on the result, and suggest improvements that deal with the most significant ones, with an explanation of how each would change the result.
The final stage of an investigation is critical, because it shows how much the conclusion can be trusted. This guide explains validity, reliability and reproducibility, how to evaluate a method and to suggest improvements, compares the common limitations with their improvements in a table, works through an evaluation of the pendulum experiment and corrects common mistakes.
What are validity, reliability and reproducibility?
- Validity: an experiment is valid if it measures what it is intended to measure, with the other variables properly controlled and no major systematic error. An experiment with uncontrolled variables is not valid, however consistent the results.
- Reliability (repeatability): the same experimenter with the same method and apparatus obtains consistent results on repeating the measurements. Reliability is linked to the random errors.
- Reproducibility: a different experimenter, apparatus or method gives results that agree. Reproducibility with a different method is a strong check for systematic errors.
How do you evaluate an experiment?
- Compare with the accepted value (or the prediction): calculate the percentage difference and decide whether the accepted value lies within the range of the result ± uncertainty. If it does, the result is consistent with it. If not, there are likely systematic errors that have not been taken into account.
- Assess the quality of the data: the scatter of the points about the line of best fit, the size of the error bars, the number of repeats, and any anomalies.
- Identify the sources of uncertainty and error: random (reading, timing, fluctuations) and systematic (calibration, zero errors, heat loss, air resistance, unmodeled effects). Distinguish the cause from its effect on the result: say whether the result is too high or too low.
- Find the largest contribution: the quantity with the largest percentage uncertainty (after allowing for powers) dominates the uncertainty of the result, so the improvements should deal with it first.
- Check the validity: were the control variables really constant? Was the range wide enough? Did the assumptions of the model hold (for example, the small-angle approximation)?
How do you improve the method?
Good improvements are specific, practical and related to the identified limitation, with the expected effect: replace the stopwatch with a light gate and a data logger (removes reaction-time error); time more oscillations (reduces the percentage uncertainty); use an instrument with finer resolution (a micrometer instead of a ruler); take more values over a wider range and repeat each (improves the line of best fit and shows anomalies); insulate or shield the apparatus (reduces heat transfer); use a different method for comparison (checks systematic errors); control the environment (temperature, drafts); and use a video analysis or motion sensor for detailed motion. “Be more careful” and “use better equipment” are too vague and do not count. Each improvement should state what is replaced by what and what changes in the result.
Limitations and improvements compared
| Limitation | Type of error | Effect on the result | Improvement |
|---|---|---|---|
| Reaction time in starting and stopping the stopwatch | Random (mostly) | Scatter in T; about 1% of T for 10 swings | Time 20 swings or use a light gate |
| Large amplitude | Systematic | T too long, so g too low | Keep the angle below 10° |
| Length measured to the wrong point | Systematic | Constant offset in L; nonzero intercept | Measure to the center of the bob; plot T² against L and use the gradient |
| String stretches or bob swings in an ellipse | Systematic | Length or period changes | Use a thin inextensible thread; release without sideways push |
| Only five lengths | Limited data | Harder to see anomalies | More lengths and repeats |
Worked example: evaluating the pendulum experiment
- Result. The gradient of T2 against L gives g = 9.8 ± 0.2 m/s2 (best-fit value 9.81 m/s2). The accepted local value, 9.81 m/s2, is within the range 9.6 to 10.0 m/s2, so the result is consistent with it.
- Data quality. The residuals from the line are at most 0.02 s2 in a range of 4 s2 (0.5 percent), and the intercept 0.005 s2 is close to zero. The data are reliable (low scatter) and no systematic offset in the length is visible.
- Largest contribution. The length is 1.000 ± 0.002 m (0.2 percent) and the period 2.01 ± 0.02 s (1 percent, which counts twice, 2 percent, in g). The timing is therefore the dominant uncertainty.
- Effect of the improvement. Timing 20 oscillations (40.2 s) with the same ±0.2 s gives 0.2 ÷ 40.2 = 0.5 percent for T, so the contribution to g halves from 2 percent to 1 percent, and the total becomes about 1.2 percent (0.2 + 1.0) instead of 2.2 percent.
- Check of validity: amplitude. The small-angle formula assumes a small amplitude. The period is longer by about θ2/16 (θ in radians): for 10° (0.175 rad) this is 0.19 percent, but for 30° it is about 1.7 percent, and the period would be 3.4 percent too long in T2 and g about 3.4 percent too small. The amplitude of 10° or less keeps this error below the other uncertainties.
- Conclusion. The experiment supports T2 ∝ L and gives g within about 2 percent. The method could be improved by light gates, more lengths and a check of the effect of amplitude and of the bob’s mass.
How did ideas about validity develop?
The importance of replication was recognized from the start of experimental science: Robert Boyle and the Royal Society (founded in 1660) required experiments to be witnessed and described in enough detail to repeat. The statistical design of experiments by Ronald Fisher in the 1920s and 1930s introduced randomization and controls to remove bias. In recent decades, the “reproducibility crisis” in several sciences has increased the attention to open data, pre-registered methods and independent replication. In physics, the agreement of independent experiments with different methods is the standard test for a new result, such as the measurement of the Planck constant by different techniques before the 2019 redefinition of the kilogram.
Which mistakes are common?
Mistake 1: Listing “human error” or “faulty equipment” as the limitation
Name the specific cause and its effect on the result (too high or too low).
Mistake 2: Suggesting improvements that do not address the main source of uncertainty
Improving the least uncertain measurement changes nothing. Find the dominant contribution first.
Mistake 3: Confusing reliability with validity
Consistent results can all be wrong because of an uncontrolled variable or a systematic error.
Mistake 4: Claiming that the result “proves” the law
An experiment can only be consistent with a law within its uncertainty. A disagreement should be explained with errors that have been identified, not just stated as “an error”.