Hypothesis Testing

State the hypotheses in terms of a parameter

Suppose the question concerns whether a population’s mean task-completion time differs from 12 minutes. Write

The null hypothesis specifies a reference value. The alternative here is two-sided, allowing either a higher or lower mean. A one-sided alternative such as $\mu>12$ asks a different question and should be justified in advance.

Hypotheses concern the population parameter, not whether the observed sample mean equals a number. The sample mean will usually differ from a reference value simply because sampling varies.

Identify the population, measurement procedure and units. “There is a difference” is too vague to determine which quantity or model is being tested.

Use a test statistic to compare signal with sampling variability

Use the synthetic summary from Confidence Intervals: $n=25$, sample mean 12.8 minutes and sample standard deviation 2.0 minutes. For the usual one-sample t procedure,

The observed mean is two estimated standard errors above the null value. Under independent normal observations and the null hypothesis, this statistic has a Student t distribution with 24 degrees of freedom.

The test uses the same basic conditions as the corresponding t confidence interval. For non-normal data the procedure may be approximate, with reliability depending on sample size, shape and other conditions. Repeated observations of the same person should not automatically be treated as independent people.

Interpret the p-value conditionally

For this two-sided test, the p-value is

It is the probability, under the specified null model, of obtaining a test statistic at least as incompatible with the null in the chosen directions as the observed one. The observed statistic 2 is included in the definition, even though an exact endpoint has probability zero in this continuous distribution.

The p-value is not the probability that $H_0$ is true. It is not the probability that the result “occurred by chance,” nor the probability that the next study will disagree. Those are different questions.

The American Statistical Association’s statement on p-values emphasises that a p-value does not measure effect size or replace scientific reasoning and transparent reporting.

Compare with a preselected threshold

If the significance level was set at $\alpha=0.05$, then $0.05694>0.05$. The conventional decision is fail to reject $H_0$ at that level. It is not “accept the null,” “prove no difference,” or “the results are identical.”

The evidence is not transformed from worthless to definitive at an arbitrary boundary. A p-value of 0.049 and one of 0.051 should not produce radically different accounts of otherwise similar data.

Here the observed difference is 0.8 minutes. The 95% t confidence interval for the mean is approximately $[11.974,13.626]$, which includes 12. That agrees with the two-sided 5% test using the same model and method. Subtracting 12 gives an interval for the mean difference of approximately $[-0.026,1.626]$ minutes.

The data leave both a very small negative difference and a larger positive difference compatible with this interval procedure. A claim of practical equivalence would require an equivalence margin and a procedure designed for that question.

Separate statistical significance from effect size

Statistical significance depends on the estimated effect, its variability, sample size and model. A tiny effect can produce a small p-value with enough information. A potentially important effect can remain imprecisely estimated in a small study.

Report the difference in meaningful units and its uncertainty. In this example, the sample difference is 0.8 minutes; a simple standardised sample difference is $0.8/2=0.4$. That number alone does not determine whether the difference matters to learners or to a decision.

A suitable report states the estimate, interval, test statistic, degrees of freedom, p-value, sampling procedure and assumptions. Avoid reducing the entire result to “significant” or “not significant.”

An exact discrete example

Suppose a hypothetical coin is tossed independently ten times and produces eight heads. Under $H_0:p=0.5$, the number of heads has a binomial distribution. For a two-sided test of departures from fairness, the symmetric outcomes at least this far from five heads are 0, 1, 2, 8, 9 and 10 heads.

This two-sided result is not the same as the one-sided upper-tail probability $56/1024=0.0546875$. The direction of the question affects the calculation.

This symmetric fair-coin case has an unambiguous matching-tail construction. For other discrete models, definitions of a two-sided exact p-value can differ; state the method rather than applying “double one tail” universally.

Eight heads are not proof of bias, but failing to reject fairness is not proof that the coin is perfectly fair. More precise estimation would require more appropriate information.

Understand errors and power

A Type I error rejects a true null hypothesis. A procedure with significance level $\alpha$ is designed to control this probability under its stated model; for discrete tests, the actual rejection probability can be below the nominal level.

A Type II error fails to reject a false null hypothesis. Power is the probability of rejecting under a specified alternative. It depends on the actual effect, variability, sample size, design and testing rule. There is no single power value without an alternative and model.

These are repeated-procedure properties. A p-value of 0.03 does not mean this particular conclusion has a 3% probability of being a Type I error.

Protect the analysis from avoidable distortions

Testing many hypotheses and reporting only small p-values creates a selection problem. If twenty independent tests each have an actual 5% false-positive rate under their nulls, the probability of at least one false positive is

The independence and common-error-rate conditions are part of that calculation. Multiple-testing methods, prespecified outcomes and full reporting address different aspects of the problem.

Also disclose data exclusions, alternative analyses, stopping decisions and departures from the planned method. Outliers should not be removed merely to cross a significance threshold. A larger sample cannot compensate for an inappropriate target population or an unacknowledged causal assumption.

Related topics

Return to Confidence Intervals to report estimates with uncertainty and Sampling Methods to examine the design. The next probability branch develops Probability, Random Variables and Probability Distributions in greater depth.