Guide · Evidence Notes
Sample size, representativeness and the assumptions buyers should check
A large sample is not the same as a representative one. Before you trust a survey’s margin of error, check the assumptions behind it.
The short answer
The margin of error a survey reports assumes a random sample, a stated confidence level and an expected proportion. If the sample was not random, the margin does not apply, however large the sample. Size fixes chance error, not bias.
What sample size can and cannot do
Sample size controls random error: the variation you would expect from drawing a different random sample. It does nothing about systematic error: who was reachable, who chose to answer, and how the questions were asked.
A thousand responses from an online link shared on social media can be less representative than two hundred drawn at random from a register.
The assumptions to check
- Selection: were respondents drawn at random, or did they select themselves?
- Confidence and margin: what level was used, and for which estimate?
- Expected proportion: 50% gives the largest, safest sample; other values need a reason.
- Design effect: clustered samples need more responses than simple random ones.
- Response: who did not answer, and how might they differ?
A worked example
In our survey sample-size calculator, a staff survey in an organisation of 12,000 needs 373 completed responses for ±5% at 95% confidence, and about 622 invitations at a 60% response rate. Switch the selection method to “convenience or opt-in” and the calculator warns that the margin no longer applies: the arithmetic is the same, but the claim it supports is not.
What to ask a research supplier
Ask how respondents were selected, what the response rate was, and how the achieved sample compares with known figures for the population. A supplier who cannot answer those three questions cannot tell you how much to trust the results.
What this guide does not cover
This guide covers estimating a single proportion. Comparing groups or detecting change needs a power calculation.
Responsible for this guide
Examples in this guide are illustrative, not client results. Figures come from the free tool’s worked example; change the inputs in the tool to see your own.
Related services
The same method, delivered with you: scoped in writing, with a named principal and a fee agreed before any work begins.
Expert-interview programmes
We recruit and conduct an agreed number of genuine expert interviews in Ghana and across Africa, with informed consent, and deliver a synthesis memo that answers your questions.
See the serviceServiceSurvey and qualitative analysis
We analyse one survey wave or a set of interview transcripts with a transparent codebook, missing-data checks and reproducible outputs, so every figure and theme can be traced back to the data.
See the serviceMore from Evidence Notes
What needs to move forward?
Tell us the decision, the challenge or the opportunity. We reply with a scoped approach, a named principal and a fee before any work begins.
No charge to submit an enquiry.