Sampling Bias

Distinguish bias from sampling variation

A random sample can differ from its population by chance. Repeated samples would produce different estimates, sometimes above and sometimes below the target. Bias concerns a systematic difference in an estimator’s long-run behaviour under the selection or measurement process.

For example, asking only library visitors about library use predictably gives more opportunity to people who visit. A larger survey at the same entrance can make the visitor estimate more precise without making it an estimate for all learners.

Statistics Canada separates sampling variability from non-sampling error. Coverage, nonresponse and measurement mechanisms belong in the evaluation of a study even when the original sample was randomly selected.

Coverage: who is missing from the frame?

Coverage problems arise when the frame omits eligible units, includes ineligible units or duplicates some units. An online survey distributed only through a particular school platform may miss people who rarely log in. A list of current club members may be unsuitable for evaluating demand among all learners.

Consider a fully specified synthetic population. There are 100 learners: 40 cycle regularly and 60 do not. The population cycling proportion is 40%. A frame containing all 40 cyclists but only 10 non-cyclists has a cycling proportion of

Even a census of that frame would describe an 80% cycling group, not the 40% target population. The discrepancy is caused by coverage, not by the arithmetic or insufficient effort within the frame.

The example does not imply that every omitted group changes every estimate. Bias depends on the relationship between coverage and the outcome. It explains why analysts need evidence about that relationship.

Voluntary response and self-selection

An open poll lets people decide whether to participate. Those with strong views, more time, better access or greater interest may respond at different rates. The observed percentage then mixes the population’s views with its response behaviour.

Suppose two fictional groups each contain 100 people. Eighty from one group respond and only 20 from the other. The respondent pool weights the first group four times as heavily, even though the population groups are equal in size. Whether that changes a particular estimate depends on what the groups would report.

A large count of responses does not reveal the selection probabilities. Statistics Canada’s discussion of non-probability sampling explains why ordinary design-based inference cannot simply be assumed for these samples.

Nonresponse after a probability sample

A probability sample establishes how people were selected, not whether they will provide data. If selected learners with long journeys are less likely to answer a travel survey, the responding sample may understate journey times.

A response rate is useful information, but not a complete bias measure. A relatively low response rate could have limited effect if response is unrelated to the estimate under an adequate model. A higher response rate could still leave a systematically different missing group. Investigate patterns and follow-up results instead of treating one percentage as a universal quality threshold.

Unit nonresponse means a selected unit supplies no response. Item nonresponse means some questions are left unanswered. State the denominator used for each statistic and do not code a missing quantity as zero unless zero is genuinely observed.

Question wording and measurement are related problems

A question such as “Do you support the school’s excellent new study programme?” encourages a favourable framing before an answer is given. An unclear reference period can also make people answer different questions: “How often do you read?” might mean yesterday, a typical week or the last year.

These are measurement or response-bias problems, not necessarily sampling bias in the narrow sense. Statistics Canada’s non-sampling-error guidance treats collection and processing problems separately from random sample selection. The distinction helps choose the right remedy: improving a sampling frame does not fix a leading question.

Pilot a neutral, specific version. For example: “During the last seven days, on how many days did you use the resource?” Include a way to report “did not use” and distinguish it from a missing answer.

Survivorship and excluded records

A report on successful programme completers can omit people who withdrew. Their outcomes may be central to evaluating the programme. Similarly, calculating average delivery time only from successfully completed deliveries can omit cancellations or failures that matter to the question.

Define the intended outcome before filtering the data. If the question genuinely concerns completers, say so. If it concerns everyone who started, keep track of all starters and account for incomplete outcomes. A tidy final file is not automatically a faithful account of the original group.

Why a larger sample does not repair the mechanism

Under many well-specified independent-sampling models, standard error shrinks approximately as $1/\sqrt n$. Increasing $n$ from 100 to 400 halves that component because

It does not halve an unknown coverage or nonresponse bias. An ordinary confidence interval describes uncertainty under its sampling model; it is not a numerical allowance for every way the collection might be wrong.

Weighting can adjust for known selection probabilities or differences described by an appropriate response model. It cannot create information about an entirely omitted group without additional assumptions or evidence. Report what the adjustment used and what it could not address.

Improve the design and narrow the claim

Practical improvements include using a more complete frame, selecting randomly within defined groups, offering accessible response modes, following up systematically and checking wording through a pilot. Protect privacy by collecting only information needed for the question and avoiding identifiable learner details in published examples.

When better collection is not possible, a narrower statement is often the honest improvement. “Among respondents to our open poll…” identifies the observed group. It is better than presenting the same percentage as a population fact with an unjustified margin of error.

Related topics

Return to Sampling Methods when revising collection. Misleading Graphs and Statistics follows these issues into public reporting, and Confidence Intervals explains exactly what an interval does and does not quantify.