The below page outlines Survation’s standard methodology used in the majority of our publicly featured work, particularly that relating to voting intentions. However, some unusual polls will use unique methodologies that differ substantially from what is listed below – it is worth checking the methodology page that will generally be attached to the top of published data tables to confirm the methodology used in any particular case. Survation takes steps to ensure that its practices are compliant with the rules of the British Polling Council and the MRS Code of Conduct.
Data Collection Methods
Survation has three main methods of data collection; online interviews, telephone interviews and face-to-face interviews. Online interviews are conducted via opt-in online consumer panels. Invitations to complete surveys are sent out to a random selection of members of the panel. Differential response rates from different demographic groups are taken into account to expand demographic groups in the sample with a low response rate through additional invites.
Telephone interviews are conducted via telephone from our in-house call centre. A combination of random numbers seeded from the complete BT directory of households in the relevant wards and, in addition, a smaller amount of targeted lifestyle data for specific younger age brackets are called, to achieve a broad sample of ages and wards. Generally only landline numbers are called – under sampling of younger respondents is corrected for through the use of targeted lifestyle data.
Face-to-face surveys use custom methodology to sample populations that have no defined demographic profile other than passing through a geographic location or locations of interest. These could for example be particular shops, railway stations or tourist sites. Interviewers will typically position themselves at a key location and approach the ‘next’ available person passing by after the completion of each interview in turn.
For a standard national poll, the population is all adults aged 18+ England, Scotland and Wales (not Northern Ireland). Bespoke online polls can also be constructed for a wide range of different characteristics such as age, gender, household composition, employment status and geographic areas down to NUTS 1 region size. For geographic areas of NUTS 2 region size or below, a poll of 1,000 respondents is usually conducted by telephone.
Data can be weighted by multiple variables to match the profile of the target population. For typical national polls we weight by age, sex, region, household income, education and, for national political polls, vote in the 2010 general election, or vote in the 2011 Scottish Parliament election (constituency) for Scotland-only polling. For constituency telephone polls, we typically weight by age, gender and electoral ward, but do not use past vote weighting; for an explanation of this, please see this note. Targets for the weighted data are derived from Office of National Statistics data, usually the 2011 Census data where available.
Margin of Error
Because only a sample of the full population is ever interviewed, results are always subject to margin of error, meaning that not all differences are statistically significant. For example, in a question where 50% (the worst case scenario as far as margin of error is concerned) give a particular answer, with a sample of 1,000 respondents it is 95% certain that the ‘true’ value will fall within the range of 3.1% from the sample result. All margin of error figures relate to the 95% confidence level for a question with 50% / 50% responses unless otherwise specified. This is the maximum margin of error for any question – for questions where the percentages are smaller than 50%, the margins of error will be correspondingly smaller.
In order to assess voting intention for an election, we first ask respondents how likely they would be to vote in the next election on a scale of 0-10. This likelihood to vote rating is then used to weight voters’ responses, such that respondents replying “10” are weighted by a factor of 1.0, whilst those responding “9” are weighted by a factor of 0.9, and so on down to responses of “0” being excluded altogether.
Respondents are then typically asked who they would be most likely to vote for if that election were tomorrow, with the responses “Labour”, “Conservative”, “Liberal Democrat” and “UKIP” prompted in a randomising order, and other parties listed if respondents select “Another Party”.
As an additional weighting step for national political polls, respondents who replied “undecided” or “refused” are then removed from the sample. “Undecided” respondents are then re-inserted into the sample based on a 0.3 factor of which party they voted for in the 2010 general election.
In all questions where the responses are a list of parties, names or statements, these will typically be displayed / read to respondents in a randomising order. The only questions which would not usually have randomising responses would be those in which there was a natural order to maintain – e.g. a scale from “strongly agree” to “strongly disagree”, a list of numbers from 0 to 10 or questions which had factual rather than opinion-related answers such as demographic information. “Other”, “Don’t know” and “Refuse” responses are not randomised but fixed at the bottom or response lists for online polls, and typically not prompted for telephone polls.
Survation is a member of the British Polling Council and abides by its rules on transparency and disclosure. This means that for any results of our polls that enter the public domain for any reason, we are obliged to publish within 48 hours the full data tables of any questions mentioned and any questions in related topics that preceded them in the questionnaire. Private polls may still be commissioned which we are not obliged to disclose so long as no information from their results enters the public domain. Please refer to the BPC website for further details.