It May Look and Smell Like a Poll, but Is It?

By Stuart Rothenberg

I read “Beware of Online Polls” on the Hotline Web log only a few hours before I received a press release from Tennessee GOP Senate hopeful Ed Bryant’s campaign screaming that “Zogby Poll Confirms Bryant Strongest Candidate.” Bryant’s memo was mostly balderdash, and it’s not alone. I’m seeing other silly press releases and references to controversial polls, so I figured it’s about time to discuss online polls.

The Hotline entry, written by the always astute Chuck Todd, warned readers about the Wall Street Journal Online/Zogby International’s online polls and noted that some of the results “don’t make any sense.”

He asserted, correctly in my view, that pollsters have not yet figured out how to conduct online polls in a way that’s accurate. Like the Hotline, both Roll Call and my newsletter, the Rothenberg Political Report, don’t report online polls. Well-regarded pollsters are right to say their methodology is unproved.

Of course, we aren’t the only ones skeptical about online polls. Other media outlets, including CNN and the print edition of The Wall Street Journal, generally don’t report on online polls either. Many media outlets also ignore polls taken by automated phone systems rather than real people, because of concerns about their accuracy. Unfortunately, others in the media aren’t as discriminating.

Let’s be clear about one thing: I’m not merely defending the status quo when I question online polling. Polling is based on statistics, and sampling techniques matter if you care at all about accuracy and reliability.

There are plenty of veteran pollsters who challenge the reliability of online polls. One of them, Warren Mitofsky, who served as executive director of the CBS News election and survey unit for more than 20 years and started the CBS News/New York Times Poll in 1975, told me recently that online polls have “no scientific basis” because “their samples are people who volunteer.”

Those samples don’t represent the general population, and efforts to weight online results “are bogus,” said Mitofsky, a trustee of the National Council on Public Polls, an association of public polling organizations established more than 35 years ago.

While he acknowledges that some people defend online polling — in fact, online pollster Harris Interactive is a member of the NCPP — Mitofsky also argues that his highly skeptical views about online polling “are widely held among people with a knowledge of methodology. Those people don’t even take [online polling] seriously.”

Gallup Poll editor-in-chief Frank Newport, an old friend, told me that “Gallup does not do online polling to represent the general population.” His complaint, echoed by other pollsters who preferred not to be identified, is that online polls rely on volunteers, and that a self-selected universe is not necessarily representative of all adults or all voters — even if polling firms weight respondents on the basis of age, gender, partisanship, etc.

“People who volunteer to do polling, typically, are very different from people drawn at random,” insisted one veteran pollster whose firm does conduct online polling, but only “when we have a defined sampling frame.” A sampling frame is a type of listing from which pollsters can draw respondents in such a way as to give everyone in the universe a roughly equal chance of being selected.

So why would the WSJ’s Web site collaborate with a pollster to conduct and publicize online polling, especially when the newspaper itself collaborates with NBC News and two highly regarded pollsters, Peter Hart and Bill McInturff, to conduct and release traditional, high-quality telephone polls?

I figured I ought to ask WSJ.com about its online polls, but I was greeted coolly and put off when I called an editor at the Web site. Instead, a Dow Jones public relations person called me and insisted that I needed to submit my questions in writing. I pointed out that WSJ reporters would find that requirement unacceptable when researching a story, but he wouldn’t budge. So I submitted the questions, along with a complaint about the requirement.

Here is what I got back from a Dow Jones spokesman: “We run the Zogby Interactive polls on The Wall Street Journal Online because we believe that they provide news value for our readers, who can also review the methodology themselves on our site.”

I take that to mean that Dow Jones doesn’t care whether the data is accurate or the methodology appropriate, only that they have numbers and that makes them “news.” Thank goodness the folks at the print edition of the WSJ have more sense and journalistic integrity.

Why would a media Web site run polls that most polling experts call unreliable and unscientific? I can’t be sure about the WSJ Online, since they wouldn’t answer my questions. But, generally, I suspect that part of the answer is ignorance, and some of it is sloppiness. Most of it is simply about business — putting up “content” to draw page views.

Many reporters and editors simply know little about methodology, and I expect some of them regard talk about such things as arcane.

But I also suspect that some don’t really care about whether the numbers are entirely reliable or whether they meet some standard that they don’t understand. Media Web sites and 24-hour cable television networks consume a great deal of information, and for some, it doesn’t matter whether the data is right, only that it exists.

Fortunately, most major media Web sites, reporters and producers know that not all polls are created equal, and they take steps to assure that they are offering only reliable data. They don’t leave it to readers and viewers to analyze and evaluate survey sampling techniques.

Reporters, producers, news writers, Web loggers and everyone else who distributes news and analysis — including candidates — need to understand that not all polls are created equal, and that online polling is both extremely controversial and generally not accepted as credible by most polling experts. Hawking those polls doesn’t add to the credibility of a media outlet, or a candidate.

This column first appeared in Roll Call on April 6, 2006. Copyright 2006 © Roll Call Inc. All rights reserved. Reprinted with permission.