Tuesday, March 11, 2014
When people in Maine talk about the problems with public opinion polling, they often mention the 2009 referendum about allowing same-sex couples the right to marry and how the pollsters got it wrong.
In a way, this is a fair criticism. Average polling results in 2009 (which showed about a 2 percentage point advantage toward the "no" side) were far different from the eventual outcome of the vote (in which the "yes" side won by six percentage points, successfully repealing Maine's equal marriage law).
Looking at the results in such a blunt way, however, misses the fact that some polls got the race right and ignores the importance of looking at public opinion in a nuanced way.
An understanding about what went wrong and what went right in this previous polling is especially important as Maine once again will vote on same-sex marriage this November.
First of all, we should note that only a small number of organizations conducted polls in 2009. In fact, only three groups released independent, public polls in the weeks leading up to the election.
One of these, Research 2000 later was sued by its client (the progressive political website DailyKos) for falsifying results and has been opaque in its methods. Let's set it aside.
Of the two remaining groups, one actually got the right answer in 2009: Public Policy Polling released a survey just days before the election that was correct within the margin of error of the poll.
The other firm, Maine-based Pan Atlantic SMS, got the results very wrong. In two polls conducted at the beginning and end of October, they projected margins in favor of same-sex marriage of nine and 11 percentage points, respectively. It was these two results that threw off the prediction average for the referendum and made election night such a disappointing loss for the No on 1 side.
A few differences in methods used by PPP and PA could explain some of the difference in results.
For one thing, PPP polls used pre-recorded questions in an Interactive Voice Response system, while PA used live interviewers. For a sensitive issue such as same-sex marriage, respondents who oppose marriage rights might be more likely to tell the truth to a robot than a real person.
Secondly, PPP had a much larger sample size -- nearly three times larger than PA's. That means Pan Atlantic's results were more susceptible to random statistical error.
Finally, they asked the question in different ways. While PA just used the language as it would appear on the ballot, PPP clarified that a "yes" vote would "undo the law that lets same-sex couples marry."
With a somewhat confusing ballot question ("no" meant keeping the law; "yes" meant rejecting it) and less time to study the wording while on the telephone than in the voting booth, this extra step may have helped elicit true opinion on the matter.
None of these factors can fully explain how Pan Atlantic was off by a full 17 percentage points on the referendum, but they do point us toward the need for a more comprehensive look at polling results than just taking the topline numbers as gospel. It also would be valuable to have more results from more pollsters in order to get a comprehensive view of real public opinion.
Luckily, this year, things seem to be starting off on the right foot.
It's still more than four months until the election, and we already have a number of public polls. In addition to internal results from Mainers United, we also have results on this issue from PPP, MassINC and the Maine People's Resource Center (for which I work).
The numbers so far look much better for equal marriage supporters than they did in 2009.
PPP found a 15-point margin in favor of allowing same-sex marriage in March, MPRC found an 18-point margin in April and the WBUR/MassINC poll this month found the "yes" side up by 19 points. All these are far larger margins than were seen on any poll in 2009.
PPP also has already polled twice (and will continue to check in on the race) and has been asking the question in different ways -- practices that should give us a better idea about where things actually stand.
It's too early to draw conclusions about what will happen in November, but it's not too early to examine with a critical eye these polls and the public opinion trends that eventually will determine the outcome of the race.
Mike Tipping is a political junkie. He writes the Tipping Point blog on Maine politics at DownEast.com, his own blog at MainePolitics.net and works for the Maine People's Alliance and the Maine People's Resource Center. He's @miketipping on Twitter. Email to firstname.lastname@example.org.