pc28����

In the case of the most recent federal election, Canadian pollsters by and large correctly predicted that the Liberal party would win the most votes. Nearly every major polling firm underestimated support for the Conservatives and overestimated support for the NDP, particularly in Ontario. This is in keeping with past elections, wherein the Conservative vote has been consistently underestimated by a plurality of polling firms, and helps explain why a widely anticipated Liberal majority government resulted in a “majority minority” instead.

But while most polls were relatively similar in their vote share projections, even marginal differences can sometimes significantly impact outcomes, given the nature of our electoral system.

There are a variety of ways that we can evaluate individual poll performance and no single measure of polling accuracy, but one of the most widely used is root mean squared error (RMSE). This metric evaluates how far off a poll’s predictions are from the actual results by averaging the size of the errors — while giving greater weight to larger misses. In essence, RMSE captures not just whether a poll was wrong, but how wrong, and how often. The lower the error, the more accurate the prediction — a perfect poll would have an RMSE of zero.

When applying this metric to each polling firm’s final poll of the campaign, Mainstreet Research (0.27), Liaison Strategies (0.47) and the Angus Reid Institute (0.67) emerge as the front-runners in terms of the accuracy of their vote share projections. EKOS Research (2.05), Abacus Data (2.14) and Innovative Research Group (2.88) had the highest error rates. Among the poll aggregators, which produce election forecasts by modelling polling data, The Signal by Vox Pop Labs in partnership with the pc28����Star was the most accurate in terms of RMSE for both vote share and seat share projections (0.7), followed by the CBC Poll Tracker (1.15) and 338Canada (1.32).

Of course, consistency is key when assessing the reliability of Canadian pollsters. A long view provides essential context. Looking back at the four federal elections over the past decade, one observes that Nanos Research, Abacus Data and Research Co. offered the most accurate predictions in 2021. Léger, Ipsos and Abacus Data had some of the lowest error rates in 2019. Nanos Research, Forum Research and Mainstreet Research were among the top performers in 2015.

This variability in polling performance may well reflect differing methodological choices between the pollsters, but one might expect such choices to consistently favour one method (and thus one firm) over another. The fact that we don’t see such a pattern speaks to the reality that polling is a complex endeavour — increasingly so in the digital age, where modes of communication are so varied. When dealing in probabilities — to borrow a phrase from Pierre Poilievre’s concession speech on election night— “sometimes you do everything right and it doesn’t work out.”

This is not to dismiss the importance of holding pollsters to account, but rather to acknowledge that the performance of the industry in general bodes well for the reliability of public opinion research in Canada. Given the challenges ahead, it is imperative to ensure that we can make accurate inferences about what Canadians are thinking so that we can ensure that government is acting in a manner that is responsive to the will of the people.