N.C. Controversy Reveals Perils of Reporting on Polls
By Stuart Rothenberg
One of the growing problems with political reporting is the explosion of polls and the tendency — particularly among local TV reporters and editors, cable TV hosts and bloggers — to report all of them as if they are equally reliable and newsworthy, and to draw dramatic conclusions from small subsamples and from statistically insignificant changes.
Polls receive so much attention that they become the focus of races — even if the actual races haven’t really started. This is true right now in North Carolina and Nevada, where Sens. Richard Burr (R-N.C.) and Harry Reid (D-Nev.) look weak in early surveys even though they have not drawn heavyweight opponents.
Recently, Republicans have started complaining long and hard about polling conducted this cycle by Public Policy Polling in the Tar Heel State. They note, quite correctly, that PPP is a Democratic polling firm and that too many reporters fail to note their partisan bent. GOP insiders also complain about the firm’s sample, arguing that it often is too urban and too Democratic, and that its surveys understate Burr’s strength and his prospects for re-election.
If readers don’t know that PPP is a Democratic firm, they are reading the wrong publications. At the Rothenberg Political Report, we’ve regarded PPP as a Democratic firm, and identified it as such, since it has been around. In February, Roll Call reporter John McArdle wrote a lengthy article about PPP, calling it “a Democratic firm based in Raleigh” and referring to the company’s “controversial” methodology. National Journal’s Hotline also identifies PPP as a Democratic firm, as does the News & Observer (Raleigh).
It’s true that some newspapers don’t always note PPP’s Democratic credentials — including the Charlotte Observer, the Atlanta Journal-Constitution and the News-Topic (Lenoir, N.C.) — but that’s not PPP’s fault. Obviously, any reporter who fails to note the firm’s partisan bent is making an error, and Republicans have a legitimate gripe with them.
GOP efforts to discredit PPP because it is a Democratic firm are a different story. Yes, it’s important to note the firm’s partisan connections, and it’s not unreasonable to be wary, at least initially, of its numbers. But the fact that the polling firm works for Democrats doesn’t make its poll numbers inherently flawed.
In fact, the handful of us who have been reporting on and handicapping House and Senate races for many years tend to believe that partisan pollsters generally produce more reliable numbers than colleges and some newspapers. The key, of course, is to get them to share those numbers and to discuss them free of spin.
I personally have been slow to give PPP’s surveys a lot of credibility because of its interactive voice response methodology. Pollsters who are relatively new to political polling need to prove that they have a successful track record before they deserve to be taken seriously, and I haven’t been convinced that PPP has met that standard.
But PPP’s polling in this year’s Virginia Democratic gubernatorial primary was good, and a pollster whom I respect highly tells me that the firm’s poll numbers in North Carolina last year were good, as well. So dismissing PPP’s data out of hand seems unwise.
PPP puts its polls up on its Web site, including the demographics of each survey, so anyone who is interested can view those data and evaluate the sample. Of course, many people who talk about polls don’t pay any attention to the mechanics of polling or to individual samples, but that’s part of the broader problem that I already mentioned.
Ironically, in reporting on the North Carolina controversy recently, Politico mistakenly treated a seriously flawed Republican “poll” as if it were a legitimate public opinion survey. The Hotline made the same mistake, proving that even careful, politically astute journalists can miss things.
Politico’s article on PPP refers to Burr consultant Paul Shumaker and a “survey” conducted by his firm, Carolina Strategy Group, which appears to show the Senator somewhat better positioned for re-election than does PPP’s polling.
The problem is that many of the questions in that particular “poll” are loaded, discrediting the entire survey and making it look much worse than PPP’s approach. (Interestingly, Republican strategists aren’t complaining about it — or Republican or Republican-leaning pollsters who also produce survey results that sometimes seem mind-bogglingly outlandish.)
For example, instead of asking a straight Congressional job approval question, Shumaker’s survey asked, “Do you approve or disapprove of the job Congress is doing under the leadership of Senator Harry Reid and Speaker Nancy Pelosi?”
National surveys increasingly show Pelosi is a divisive figure, so including a mention of her in the question could well distort the results about respondents’ attitudes about Congress. It could also poison the rest of the survey.
Two questions later, respondents are asked which Senate candidate, Burr or North Carolina Secretary of State Elaine Marshall (D), would “better serve the people of North Carolina as a check and balance on the policies of President Obama.”
That question is followed immediately by a straight ballot test between Burr and Marshall. Unfortunately, the results from this question — which showed Burr ahead 52 percent to 37 percent — have been polluted by the previous questions, two of which included the “check and balance” language.
Rather than whining about PPP, Republicans might want to try to set the record straight in North Carolina by releasing their own poll by a credible, full-time polling firm. That would go a long way to helping develop a more balanced, more thoughtful narrative about the race.
This column first appeared in Roll Call on June 29, 2009. 2009 © Roll Call Inc. All rights reserved. Reprinted with permission.
