On September 1, 2015, CNN announced that it was amending its criteria used for determining which candidates would participate in the Republican debates to be held on September 16th.
Back in May, when the network first announced the original criteria, it had said that it would use the top 10 candidates based on the average of specific polls taken between July 16th and September 10th. However, after the more recent Sept. 1 announcement, the criteria was amended so that any candidate polling in the top 10 between August 7th and September 10th will be included in the next debate. And as of last Thursday night, when the network announced the final lineup, we can see that the change in criteria resulted in one additional candidate being added—Carly Fiorina, who was the former CEO of HP. So there will be 11 candidates participating in the debate instead of the originally anticipated 10.
Why did CNN change the criteria? It said, that the original expectation was that there would be approximately 15 polls during the predetermined time period, but as the window was closing, it turned out there were only five. The limited number of polls potentially caused the sample to be less representative of recent changes in the popular opinion, as Fiorina, for example, experienced a strong surge in popularity after the first Republican debate back in August.
So just how reasonable is it to use polls to determine the debate participants? For the Fox News Network’s August 6th debate, five different polls were used. And there was some controversy about the fact that which five polls would be used was not publicly announced until after the decision was made. Because of the limit of 10 participants in the August 6th debate, the determination of that 10th place slot happened to cutoff between two candidates—John Kasich and Rick Perry—who were actually within the polling margin of error of one another.
In fact, due to his objections to polls being used this way, Lee Miringoff, who is director of the Marist Institute for Public Opinion, a polling organization, said that Marist was intentionally limiting its questioning so that its poll would be excluded from Fox’s consideration. He told McClatchy, the media conglomerate, that by using a tenth of a percentage point in the polling results, Fox was relying on a level of precision that didn’t even exist in the polling. And he further said that candidates were making public appearances and promotional advertisements timed in order to try to affect their standing specifically within Fox’s poll selection window in an attempt make the cut, meaning that polling activity, which is supposed to be measuring the results, actually ended up affecting the process.
Clearly there’s an issue. How accurate is your measurement process when it starts affecting the outcome? Statisticians, scientists, and other practitioners concern themselves with these and other questions when they’re trying to develop an accurate representation. In auditing, the practice of using a sample from a population in order to make judgments about that population as a whole is an accepted approach. As a matter of fact, auditing standard AU-C 530 specifically addresses this topic and can provide some insight into this polling problem.
Auditing Standard AU-C 530
Auditing Standard AU-C 530 says that auditors use sampling in order to draw conclusions about the population. And the standard defines sampling as the practice of selecting and evaluating less than 100 percent of a population in such a way that the conclusions reached about the selections are the same as would be reached if 100 percent of the population was tested.
But there are risks associated with sampling. Sampling risk is defined as the risk that conclusions reached about the sample are different than the ones that would be reached about the whole population. If your sample leads you to believe that you have to do more auditing procedures than really required, then that reduces the efficiency of the audit; but if the sample leads you to believe that there isn’t a concern where one actually exists, then that affects the effectiveness of your audit—a far more serious risk. Because if you think that there’s not a misstatement, but there actually is one present, you could end up giving the wrong opinion on the financial statements.
Part of effective sampling is reducing sampling risk to a tolerable level, and the best way to do that is by taking the proper considerations when planning and designing the sample. When you’re designing a sample, it’s important to clearly define its purpose—that is, what conclusions are you trying to make about the population? Are you determining if a balance is quantitatively misstated? Are you testing some other characteristic about the items?
“Variables sampling” is the name used when testing the amount of quantitative misstatement in a population; conversely, “attributes sampling” is used to draw conclusions about some characteristic of a population, like whether or not a control is functioning. So it’s important to clearly define and document the purpose of the test and make sure that the population you’re sampling from is the appropriate population for drawing those conclusions—and, of course, you have to make sure that the population is complete, that you’re sampling from the whole population.
Properly documenting the completeness of your population ahead of time can prevent headaches down the road. For example, back in May, when Fox announced the qualifying criteria for its debate, the network said that it would use only those polls “conducted by major, nationally recognized organizations that use standard methodological techniques.” But the network didn’t specify the techniques and didn’t specify which polls were qualified. This ambiguity led to some criticism later.
AU-C 530 further says that you need to determine the appropriate size of the population so that you can reduce sampling risk to an acceptable level. The more confidence needed in the sample, the larger the sample should be. Also, the expected rate of deviation or of quantitative misstatement in the population factors into the equation. In this commentary, we’ve noted that both Fox and CNN ended up using five polls to determine participants in the debates—although CNN anticipated that its sample would include 15 polls. But really these are a sample of a sample, which introduces additional risk. Each poll conducted can use different criteria. Some polls were of registered Republicans, whereas others included those who identified as Republican or those who were identified as “leaning” Republican, which is a considerably more subjective determination.
Finally, AU-C 530 says you have to make the selections in such a way that you address sampling risk as well. Remember the conclusions you draw about the sample should be the same as if you tested the entire population, so some approach is necessary that will select a variety of items that are representative of the whole population. Often a random selection process is used to accomplish this goal.
When Fox said that they were using only those polls with “standard methodological techniques,” many people interpreted this to mean that a human was making a phone call, rather than a robo-dialer or an internet questionnaire, because that traditional method is anticipated to achieve more accurate results. However with fewer and fewer people using house phones these days, and with the restrictions made in some states on calling people’s cellphones, even those standard practices are being questioned for their effectiveness.
But all of these are technical or methodological issues raised by the use of sampling. What about Miringoff’s criticism that polls simply shouldn’t be used for these purposes? Among other issues, a problem arises when polls’ timing and techniques are changing the way a candidate conducts his or her campaign. How can we really have confidence in the accuracy of our measurements when the measuring process itself changes the results?
The Heisenberg Uncertainty Principle
In science, when the measuring process changes the measurement, that’s called the “Observer Effect.” That is, by observing some phenomenon, you change it. For example, imagine that you’re baking a cake, and you want to test whether or not the cake is ready by sticking a toothpick in the middle and seeing if it comes out dry. This is actually a really common and effective test.
There’s just one problem. Each time you open up the oven to stick that toothpick into the middle of the cake, you’re letting a lot of heat out—not to mention the risk of burning your hand. So if you do this several times, you’re actually slowing the baking process, and you’re causing the cake to be baked at a slightly lower temperature, which could lead to the exterior parts becoming dried out or to under-baking the middle. Your measuring process is changing the process you’re trying to measure.
Werner Heisenberg famously described this problem in the 1920s. He wasn’t talking about baking a cake, though. He was talking about measuring properties of subatomic particles. Heisenberg said that if you wanted to know where an electron was or how fast it was moving, then you had to obtain information about that electron. And the least intrusive way of doing so was to fire a single particle of light—a photon—at that electron. The photon would bounce off of the electron and you could measure that bounce and know where the electron was or how fast it was moving. But the problem is that the more accurately you measure the location, the less accurately you measure the speed and vice-versa. Because when you fire that photon at the electron, you’re knocking the electron out of its position and changing its speed.
It turns out that what Heisenberg was describing was not only a measurement problem—the Observer Effect—but actually a property about all subatomic particles, which ended up becoming one of the most bizarre and incomprehensible concepts in quantum mechanics. Today we call it the Heisenberg Uncertainty Principle, and we know that it means that sometimes knowing more about one characteristic means knowing less about another.
Perhaps some of the criticism about polls being used to determine which candidates participate in the debates is misplaced. Because no matter what method we use to determine the participants, there will always be a risk associated with the methodology. As accountants and auditors we’re conscious of this risk, so we develop an expectation of the anticipated rate of deviation or misstatement. Maybe the problem is that we, as the consumers of journalism, are not properly understanding the risks from the outset and setting an expectation for ourselves.
Perhaps a better criticism is that we’re placing too much confidence in the media and relying too heavily on its ability to make these editorial judgments in the first place. Maybe we’re investing too much confidence in what’s on television when, in the words of Donald Trump himself, “Sometimes your best investments are the ones you don’t make.”