Every day, we see polls in the national news about the presidential race. And Utah Policy Daily also conducts regular survey research measuring public opinion about politicians and important issues in Utah.
We’re going to see a lot more polling in the big election year ahead, commissioned by the news media, candidates, and interest groups.
So how accurate is polling? Well, it’s only as accurate as the methodology used and the care and integrity of the pollster. Certainly, conducting accurate survey research is more difficult today than in past decades. But it can still be done with a high degree of accuracy.
Dan Jones & Associates, the firm that conducts survey research for Utah Policy Daily, is one of the oldest and most respected polling organizations in the country. Survey research firms don’t last very long if they aren’t careful and accurate. Clients won’t stick with firms whose research conclusions differ dramatically from actual results. Jones has been successfully conducting polls for many decades in Utah and other states.
The best recent example of excellent polling was research Jones did on the Salt Lake City mayoral race in November. That was a very difficult race to poll accurately because it was the first time mail-in balloting was used for all voters. Low-turnout municipal elections, where voters aren’t paying close attention, are notoriously difficult to predict accurately.
It was also a hard election to survey because events and opinion were changing rapidly. Challenger Jackie Biskupski won the primary election with very high numbers and was expected to waltz to a final election victory. When Jones’ polling for Utah Policy showed incumbent mayor Ralph Becker closing rapidly on Biskupski, her campaign questioned the accuracy of the research.
Just before election day, Jones’ research showed Biskupski holding a very slight lead, and that’s precisely how the election turned out. It was a remarkable example of a pollster working incredibly hard to get the right demographics, and survey those who would actually vote. Jones’ research was right on target in a highly volatile election.
Today, it’s much harder than in decades past to get people to spend time on the phone with a researcher. And with the dramatic decline in the number of people who use land lines, it’s also harder to connect with a good sample.
Some research firms use pre-recorded automated calls that ask respondents to hit numbers on their phone to respond to questions. That polling methodology isn’t considered to be as accurate as live interviews. Respondents aren’t as likely to be truthful and it’s harder to get the demographics right.
Good pollsters today use a combination of land lines, cell phones and on-line panels to obtain an accurate random sample with proper demographic characteristics.
Some people question the accuracy of polling. Here’s an analogy that shows how good polling works:
Let’s say you have a barrel with 10,000 marbles in it. 2,000 are red; 5,000 are blue; and 3,000 are green. If the marbles are stirred and mixed up so the different colored marbles are evenly distributed, it makes sense mathematically that if you pull out several handfuls of marbles, the proportion of red to blue to green will be the same as in the full barrel. You don’t have to count every marble in the barrel to show that 20 percent of the marbles are red; 50 percent are blue; and 30 percent are green. With a random sample, you can predict those percentages accurately.
However, if the different colors are not evenly distributed, you can pull out several handfuls from the barrel and you won’t get the right proportions. Your sample will be totally inaccurate if the blue marbles are all in one corner and the green are all at the bottom.
The different marble colors, in polling, represent different demographics of respondents. If a pollster gets the demographics right, survey research will be accurate. Thus, the number of conservatives, liberals, males, females, ages, religion, location, income, etc., in the sample surveyed must reflect the general population being surveyed. A randomly-drawn sample of 600 respondents, evenly distributed demographically, will accurately reflect the opinions of the general population.
Dan Jones is accurate because he works extremely hard to get the demographics and sampling correct.
Another important variable in political polling is who will actually turn out to vote. That’s a variable many pollsters have trouble with. A pollster can accurately measure support of politicians among the general public. But that doesn’t necessarily show who will win, because only the most motivated citizens may turn out to vote.
Thus, in conducting campaign research, pollsters much have techniques to screen out those who are not likely to vote.
At best, survey research is only a snapshot in time. Big events can rapidly change public opinion. Research taken months ago may not reflect opinions today.
Why do we conduct survey research? We believe it’s important for regular citizens and political leaders to understand how the population feels about important issues. However, I don’t believe politicians should make their public policy decisions based solely on what polling shows. What’s right and wrong, what one’s gut instinct says, what advice one receives, are as important, or more important, than knowing what the public thinks. But public opinion is still an important data point.
It’s worthwhile for news organizations to conduct independent survey research as a check on special interest research and polling done by candidates themselves. Candidates and interest groups regularly conduct polling, and will release results if it suits their strategy. It’s good to have independent research to keep them honest.
