Referendum polls – what went wrong?
09 October 2014

'Polls failed to predict such a decisive result'

That was the headline for an article by Alberto Nardelli on Saturday morning in the Guardian discussing the performance of the opinion polls following the decisive ‘No’ vote in the Scottish referendum.

So, as the dust settles on the most important vote in the recent political history of the UK, do the pollsters need to examine their methods, or, are their strong mitigating circumstances that can be used in their defence? 

Whilst the final result showed a decisive win for the ‘No’ campaign, the polls were showing a likely result too close to call, the leads shown being around, or within, the margins of error. 

Ipsos Mori and Survation came closest to ‘predicting’ the final outcome. Some see evidence of a systematic error in the methods used in estimating the forecasts, but did this referendum create challenges for the pollsters that are not normally met when measuring voting intentions in the run-up to an election? Firstly, let’s not forget that polls provide a snapshot of intention at a moment in time, rather than a prediction. 

Secondly, the voting age was lowered, bringing in a whole new generation with no history of voting behaviour or party support to help forecast their likely behaviour. Yes, this is always the case in normal polling, but the numbers were proportionately greater on this occasion. 

Thirdly, whilst many voters had already supposedly made up their minds well before the day of reckoning, the ‘not sure’ proportion remained relatively high in all the polls, but with such a high voting turn-out, many of these citizens obviously made up their minds one way or another when it really mattered. 

Fourthly, the rather belated, last minute flurry of activity, and promises for more devolved powers made by the ‘Better together’ campaign, made the closing days a highly dynamic and emotionally charged situation. 

Fifthly, there was no ‘devo-max’ option on the voting paper, but the promises made to offer more devolved powers if the vote was in favour of the continuing Union in effect added that option for those who saw that as what they would have liked to vote for, if given the choice. 

Sixthly, maybe the one poll showing for the first time a slender lead for the ‘Yes’ campaign in the closing stages also influenced voting behaviour, perhaps ensuring that the shy ‘No’ supporters, or those wavering, actually turned out on the day. 

Finally, did Gordon Brown’s eloquent speech in the run up to the day, passionately and forcefully arguing the case for retaining the Union sway those either ‘not sure’ or not fully committed to independence. There was in fact a precedent to call on – the result in Quebec, where a slender ‘Yes’ lead was overturned on the day, again perhaps influenced by the last minute promise of more devolved powers.

Whilst the main nationwide political parties, and others in the wider Europe facing strong nationalistic groups, as in Spain, breathed a sigh of relief on Friday morning, can the pollsters feel the same? A UK general election is only months away, and the promises made to Scotland are leading to calls, and further promises, for a new political constitution favouring devolved powers throughout the UK.

How will this change the political landscape that polling companies will be measuring and forecasting voting behaviour from now until next spring? The last election produced no overall winning party, and the latest polls suggest that this maybe the outcome in 2015 – although the referendum result should salvage something of the Labour support north of the border. 

However, the Scottish Nationalist Party is recording record levels of new members since the referendum. We know, for example, that pollsters faced a challenge in factoring in the impact of the first time voters, traditionally those less likely to bother voting on the day, but this could change next year.

If there is to be a second referendum on membership of the EU in the years ahead, as promised by the Conservative party, then will we see a repeat of the situation in Scotland - what influence will the performance of the polls have on methodologies for measuring referendum behaviour? I expect that the pollsters will be taking to heart the lessons learned from the recent referendum. 

The public, in addition to politicians and the media, expect the polls to provide them with an accurate view on changing opinion in the run-up to an election or referendum. In this social media era, polls no longer provide the only measure of opinion, but they do provide an estimate based on supposedly scientific principles in terms of research design. 

I just hope that the headline I’ve used for this blog is not what we see after the 2015 general election...

I welcome your thoughts and comments below.