Poof! Profs Pop Pol Poll

Can we trust political pollsters? There is no agency, like the FDA or Moody's (trust them?), that audits polling companies to see if they are conducting polls they way they say they are. An obvious method of assessment is to compare their predictions to outcomes of actual elections, but elections occur only once in a while, and polling on, e.g., presidential approval never receives a genuine check.

The polling firm Research 2000 (Wikipedia entry), a well-respected concern, presents an interesting case study. The website Daily Kos had commissioned them to conduct a constant stream of polls. The website FiveThirtyEight, which specializes in analyzing political polls, found Research 2000's accuracy to be close to the bottom of the barrel, at which point Daily Kos severed its relationship with Research 2000. That's not the interesting part.

Mark Grebner, Michael Weissman, and Jonathan Weissman then published the paper Research 2000: Problems in plain sight on the Kos site. By looking more carefully at the Research 2000 polls, they found the polls were not random enough. For example, approval ratings were broken down into males and females. On 6/3/2010, Obama's ratings were Favorable: Men 43%, Women 59%; Unfavorable: Men 54%, Women 34%. In the Research 2000 polling, there were 778 "favorable" and 778 "unfavorable" Male vs Female breakdowns, for Obama and other people and groups, at many different time points. The problem: for all but three of the 2×778 breakdowns, the Male and Female percentages had the same parity, i.e., both were even, or both were odd. The probability of the same parity in any one breakdown, given the sample sizes, is basically fifty percent. Imagine flipping a fair coin 778 times. No way you'd get 776 heads (as for the "favorables"), or 777 heads (for the "unfavorables"). In fact, the probabilities are order-of-magnitude 10^-228 and 10^-231, respectively. Those are less than the chance the Cubs have of winning 150 World Series' in a row.

The author's detail two more anomalies, wherein the polls are not as random as they should be. Random chance that those anomalies would occur were each about 10^-16, again essentially impossible.

The overall conclusion: The results Research 2000 presented were not the raw data from independent simple random samples, as claimed. Was there some other sampling scheme? Were the data collected correctly, but then massaged heavily before presentation? Were the results made up out of whole cloth? (Which is certainly cheaper than actually phoning people.) What we do know is that Research 2000 never delivered the raw data, nor came up with an explanation for the unusual data; that Kos sued the firm, the matter seemed to be settled in 2011. In July 2012, however, a default judgement of over US$300,000 was entered against the defendant. --- Wikipedia.