Monday’s Iowa caucus results once again have raised questions about the accuracy of survey research. Polling just before caucus day had Donald Trump winning. Instead, Ted Cruz won. And Marco Rubio did much better than his standing in the final polls.
So can polling be done accurately? It all depends on the validity of the methodology used by researchers. Because we conduct monthly surveys at Utah Policy Daily, we frequently receive questions about our polling methodology. Some readers question the accuracy of our polls.
We want to be completely transparent about our polling methodology, so working with Dan Jones & Associates and Cicero Group, we prepared this overview of how our survey research is conducted.
Contacting people in a randomized and representative manner to ensure survey accuracy has become increasingly difficult as people migrate to cell phones, e-mail, and other techniques for their mode of communication (and as they get bombarded with propositions for their time). Many respected national polling firms are exiting the market because it is becoming so expensive and difficult to sample correctly. This is happening simultaneously to some firms using interactive voice response (IVR or “robocalling”) as a sampling technique (which we believe is highly flawed).
Correct sampling has become as much of an art as it is a science. This is one of the reasons we rely so heavily on Dan Jones & Associates. Dr. Dan Jones has been polling throughout Utah for over 55 years. His intimate knowledge of the market, combined with his rigorous sampling techniques, is why he has been able to predict so accurately election results throughout these 55 years.
Dan Jones & Associates performs a series of measures and tests to not only assess the validity of poll responses but also ensure the results of the survey are representative of the larger population. This allows Dan Jones & Associates to generalize the findings of the poll to the general population of adults in Utah.
Because every poll requires some degree of opt-in (whether to join an online panel or participate in a phone call), DJ&A ensures the accuracy and legitimacy of its poll results by using a random and mixed methodology. This means that not only are all respondents randomly chosen, but also that DJ&A utilizes a variety of polling methods (landlines, cell phones, online, mail) to ensure a valid and representative sample.
Now to be clear, there are some cost and time restrictions. For example, for a truly random and representative sample, a random sample from a general population would be pulled. Those individuals would then be contacted by mail, telephone, and online and offered an incentive to participate. Methods would be put in place to ensure no “type” of individual systemically did not participate (to ensure accurate representation), and a very high participation rate would be required.
Time and costs don’t allow for the above approach, but DJ&A applies a very similar, rigorous sampling method that selects a random number of registered voters and then contacts them by telephone (cell and mobile) and by e-mail. Mail is not always used (although occasionally when time permits) and an incentive is rarely used due to budgetary constraints. And yet DJ&A, because of its well-respected brand ID often generates a participation level exceeding 20%.
DJ&A utilizes the registered voter list as the individuals most closely aligned with our target demographic and because it is used as an industry standard for political polls. DJ&A also has created the largest and most representative contact list with e-mail addresses of Utahns to also contact those randomly selected by e-mail. Once individuals have randomly been chosen to participate in the poll (either phone or online), DJ&A further filter individuals using a specific screener criteria to ensure it is obtaining a representative sample.
DJ&A also ensures a representative sample by analyzing the percentage of poll completes across some demographic variables (e.g., age, gender, income, geography, etc.) to confirm the sample is similar to that of the larger population. For example, when DJ&A compares the population levels in each of the Utah counties, the sample proportion for each county very nearly matches the overall population distribution of the counties.
The fact that Utah Policy and Dan Jones & Associates randomly select respondents through a mixed methodology, as well as employ many measures and analyses, increases our confidence that the poll was conducted among a highly representative sample of the state of Utah and provides us grounds to generalize the results to the larger population. More importantly, we test these methodologies and predictions annually as together Utah Policy, and Dan Jones & Associates consistently and accurately forecasts election results.
