Why we do polling . . . and why it is reliable

LaVarr Webb, UtahPolicy.com publisher

As readers of UtahPolicy.com might have noticed, we’re back in the business of providing regular survey research. We’re measuring sentiment on key local and national issues, along with matching up candidates and determining how much Utahns like their politicians. 

It’s important that independent survey research is conducted on current public policy issues as a check on surveys done by special interest groups and the internal polling done by candidates themselves.

We’re pleased to partner in this polling project with Y2 Analytics, a firm with a stellar reputation that holds to the highest professional standards, ethics and best practices in survey research. A new web site, www.utpoliticaltrends.com, has been created to provide public access to survey results. It also includes information about methodology, sample design, etc. Results of the first survey will be posted soon.  

High-quality, reliable survey research is very difficult to do in today’s communications environment. Accurate results are not reached by simply calling landlines and cell phones. Robocalls are problematic and on-line surveys are not reliable unless sophisticated sampling has been conducted so respondent opinions reflect the general population.

Anyone can conduct surveys using easily available apps and software. Some of these are done just for the fun of it (including UtahPolicy.com’s Insider Survey and Reader Survey). But it’s important to distinguish them from professional research intended to accurately reflect the opinions of voters, or whatever population is surveyed.

Commenters on social media frequently criticize survey research, questioning its accuracy and objectivity. They often say, “Well, no one asked me.” Or, “How can you measure the opinions of millions of people with a sample of 600?”

Polling is only as correct as the methodology used and the care and integrity of the pollster. Certainly, while conducting accurate survey research is more difficult today than in past decades, it can still be accomplished with a high degree of confidence.

Here’s an analogy I’ve used a few times that shows how good polling works:

Let’s say you have a barrel filled with 10,000 marbles. Some are red, some are blue, and some are green.  Must your count all 10,000 marbles to determine how many are red, how many are blue, and how many are green?

No, you can use a representative sample. But the trick is, you have to get the right sample. If you get the right sample, you can pull out, say, 500 marbles, divide them into colors, and accurately predict the color makeup of the entire 10,000 marble population.

Let’s suppose 2,000 of the marbles are red; 5,000 are blue; and 3,000 are green. If the marbles are carefully stirred and mixed up so the different colored marbles are evenly distributed in the barrel, it makes sense mathematically that if you pull out several handfuls of marbles, the proportion of red to blue to green will be the same as in the full barrel. You don’t have to count every marble in the barrel to show that 20 percent of the marbles are red; 50 percent are blue; and 30 percent are green. A relatively small random sample accurately represents the makeup of the 10,000 marbles.

However, if the different colors are not evenly distributed, you can pull out several handfuls from the barrel and you won’t get the right proportions. Your sample will be totally inaccurate if the blue marbles are all in one corner the green are mostly at the bottom, and red are at the top.

The different marble colors, in polling, represent different demographics of respondents. If a pollster gets the demographics right, survey research will be accurate. Thus, the number of conservatives, liberals, males, females, ages, religion, location, income, etc., in the sample surveyed must reflect the general population being surveyed. A randomly-drawn sample of 600 respondents, evenly distributed demographically, will accurately reflect the opinions of the general population.

Y2 Analytics is successful because the professionals there work incredibly hard to recruit a sample that correctly represents the universe of Utah voters.

At best, survey research is only a snapshot in time. Events can rapidly change public opinion. Research taken weeks ago may not reflect opinions today.

Why do we conduct survey research? We believe it’s valuable both for regular citizens and political leaders to understand how the population feels about important issues.

However, I don’t believe politicians should make their public policy decisions based solely on what polling shows. It is an important data point. But what’s right and wrong, what one’s gut instinct says, what advice one receives, are as important, or more important, than knowing what the public thinks. A politician who lives by the polls might die by the polls.