I Never Read ‘Informed Ballots.’ You Shouldn’t Either

By Stuart Rothenberg

Last week, The Hotline published an interesting collection of responses to the question, "Should media outlets, including The Hotline, run informed ballots of head-to-head match-ups?" "Informed ballots" are ballot tests asked after those being polled are read descriptions of candidates and/or their accomplishments and positions.The question arose after I e-mailed Hotline editor Chuck Todd and suggested that he stop reporting those data, even if campaigns supply the full wording of their survey questions.

I realize that people generally want more information rather than less, no matter how misleading and useless it is. But I rarely, if ever, look at an "informed ballot," no matter where it appears. When it is released, it’s used primarily to raise funds to try to jump-start a campaign.

First, let me be clear that I don’t dispute the value of message testing to campaigns, including well-constructed informational questions followed by a second or third ballot test. Certainly, it matters to campaigns whether voters prefer one policy position to another, or certain credentials to others. But campaigns rarely, if ever, release this information.

What I don’t find useful are allegedly unbiased paragraphs that describe two candidates and are followed by a ballot test that purports to give us answers about who will win. Usually, the results do little more than raise more questions.

They often have so much information in them that it’s impossible to know exactly what quality or qualities respondents are reacting to. The informed ballot is an invitation to what psychologists and marketing professionals call the "halo effect." People focus on one characteristic or quality and generalize about it.

I expressed my concern about the informed ballot after seeing a recent Lake Snell Perry Mermin/Decision Research poll memo for Mayor Michael Coleman of Columbus, Ohio, some of which was published in The Hotline.

It included an informed ballot as proof that Coleman is "in prime position" to defeat Rep. Ted Strickland for the Ohio Democratic nomination for governor. Respondents were given profiles of both candidates and asked to choose which profile they preferred.

Each man was described by at least a dozen bits of information. The question about Coleman devoted 91 words to his description and positive attributes and accomplishments, compared to 26 words of criticism. Strickland, on the other hand, received only 70 words of positive comment and 29 words of criticism.

Twenty-one words in Strickland’s description were about his "strong values," "steelworker" father and active church life, comments that may not have excited those being polled. The Congressman was associated with only one policy or accomplishment, fighting hard "to protect American industry and jobs from foreign competition."

Coleman’s description included no family references or general qualities. Instead, aside from his race, he was described entirely by his performance as mayor. He "brought people together to get things done," "helped create jobs" and balanced budgets.

Couldn’t the difference in the descriptions account for the results of the informed ballot? Did voters focus on one characteristic in evaluating the two candidates? Is all the "negative" information really negative? For example, the phrase "big city mayor" is used as a negative attribute of Coleman, but couldn’t its use explain why Coleman "surges to a whopping 37-point lead" in the Cleveland media market after his description?

Even assuming that I put much confidence into a question based on a Democratic subsample of 260 primary voters - and remember that the 6.2 percent margin of error doesn’t apply to the margin between the two candidates but to every subsample percentage - I’d need to know more before I could conclude that Coleman is "well positioned" to be elected governor.

What is his standing in various media markets? How much money can he raise? Are there things in his background that were not included in the informational paragraphs that might prove to be a problem for him?

Although I have no evidence, I’m convinced that pollsters have a better handle than I do as to which words evoke what reactions. I suspect that they often, though not always, create these questions to get a specific answer, particularly when they want to woo a candidate into a race or release the data for fundraising.

But my concern goes further. It tests qualities and accomplishments on paper, while ignoring both the dynamic of a campaign and the personalities of the candidates.

Surveys can accurately record voters’ positions on issues, their views of the incumbent’s job performance and their desire for change. But voters cannot measure the appeal of candidates by reading a few sentences about them - yet that’s what most elections are about.

I continue to regard polls as crucial to campaigns and to my own assessments. But I’m increasingly worried about question order, sampling techniques and attempts to manipulate those of us who analyze and handicap campaigns. I’m generally more wary about polls these days, and I wish everyone else was, too.

All polls are not created equal, and all poll questions are not of equal value. The "informed ballot" result is so dependent on question wording and order, and so vulnerable to misuse, that I have no confidence in it. I’ll look elsewhere for clues about races.

This column first appeared in Roll Call on October 6, 2005. Copyright 2005 © Roll Call Inc. All rights reserved. Reprinted with permission.