A Polling Primer with ABC News…

By CJP | March 27, 2008

Welcome to the CJP interview with Gary Langer, ABC News' Polling Director. We've done this one before, but with polls being used today to tell candidates to stay in, get out, change positions and disown their pastors, we felt we could all use another does of Gary's guidance on polls. Which ones are right? Should we believe them? What about cell phones? Gary was kind enough to take us through the art and science of polling for the 2008 elections. ..

CJP: You’re the polling Director for ABC News, what does that mean?

GL: I cover the beat of public opinion for the news division. That works in three ways.

We do our own polls, we look at other poll results to see to what extent it can inform us and make us smarter, and we evaluate other research that’s presented to us to see if it’s of good enough quality for our air.

CJP: I have found that there are different kinds of organizations doing the polling, in terms of news organizations, universities, partisan pollsters or professional pollsters. It seems like everybody’s getting in on the race. For people who are seeing something that says “A poll said this today,” how should they evaluate the source of the polling?

GL: Before ABC News will use any polling data, we require a detailed methodology of the poll, the complete questionnaire including all

questions asked, the overall results for each question. These are the areas where polls can go off the rails, either through bias or flawed methodology, leading or stacked or loaded questions, or through misanalysis or cherry picking of the data.

We have to keep in mind that there is an enormous amount of survey research before us, some of which is very high quality, and a lot of which is pure junk, produced not to assess and understand public attitudes, but to promote someone’s product, agenda or point of view. As consumers, it is essential to consider the source.

CJP: So for consumers, what are the questions we need to ask ourselves about a poll, and what are the ideal answers?

GL: A very fundamental question is who paid for it?

CJP: How would you find that out?

GL: If it’s not reported, I wouldn’t give that poll any regard whatsoever. Don’t even let it enter your consciousness. It is very rare that any organization would produce and circulate a poll that found results counter to its purpose in life. The vast majority of commercially sponsored surveys, whether they are coming from political groups, interest groups, corporate or other business interest, are intentionally biased. They’ve been produced to promote a product or an agenda, or just for publicity. A basic rule I would give you is that a privately sponsored survey that came from a sponsor with an interest in the outcome should not be taken seriously.

CJP: Are there any great brands in polling, in addition of course to ABC News, and any bad brands in polling- names we should and should not trust?

GL: News organizations do polls as part of their responsibility to independently and accurately report the news.

That’s why we go to the time and expense of doing news polls. It’s not like polls wouldn’t exist, but they would all be done by somebody with an interest in the outcome. Polls produced by independent organizations are qualitatively different than data produced by sponsored research.

I also think that news polls need to be held to the same level of scrutiny as we provide. We put our detailed methodologies, along with complete surveys, up on our website, and I would ask the same of any other survey.

CJP: In terms of scrutiny, if someone opens up the paper and sees a news poll that gives the sample size, the dates it was conducted and the margin of error, what should they be looking for to have confidence in those results?

GL: Let me first say that the news media have long indulged themselves in the lazy luxury of being both data hungry and math-phobic, and it is unacceptable. You can pick up almost any newspaper and see unreliable, invalid, misleading, misrepresented survey research reported as if it were the real thing. It’s a real failing of the news media and one they need to address, and some are.

I’d like to be able to say you could pick up the paper and that every survey you see is reliable. Far too often that is not the case, so I would have to say: wisely select your media source. You could even ask the editor of ombudsman what they do to hold and enforce standards for the polls and surveys are reported in their paper. And if they have no standards, maybe you want to pick another paper.

CJP: So what are good standards for a poll?

GL: You want to start with a probability sample, a random sample, not a self-selected sample. This is done in the U.S. through random digit-dialing for telephone interviews. Internet samples do not rise to the level of reliability we need in our survey research.

CJP: In political polls, I see the terms “likely voters” and “registered voters”. What difference does it make?

GL: You can poll a general population; you can poll people who tell you they are registered to vote, and people who tell you they are likely to vote. The greater distance from an election, the more these groups all look pretty much the same.

As you get closer to an election, they can start to look different and you want to see the difference reflected in the reporting. Also, you very much want to know how a “likely voter” is defined or the size of the group. I’ve seen polls in Iowa that say they are among likely voters but they cover very large segments of the Iowa population, when caucus goers are a very small portion of the population. It’s much more expensive to drill down to voters who are actually likely to attend a caucus.

CJP: How does ABC News differentiate? How do you know if they’re likely to attend?

GL: We ask them.

CJP: Oh, that makes sense.

GL: There are other ways, but asking them is a good place to start.

CJP: Two complaints I hear a lot from people are, one, the question about more and more people having only cell phones, when random dialing does not include cell phones.

GL: The number of cell-only households in the country is about 14 percent, and it’s of interest and concern to us. But it’s also entirely possible to include cell-only respondents by drawing a separate survey of cell-only individuals and interviewing them and including them. There’s work being done to see if that’s necessary.

Non-coverage is only an issue if the group is large enough to make a difference in your overall results and if the diversity of attitudes and opinions within those groups on the measures that you care about are sufficiently different. In other words, if a person in a cell-phone-only household has pretty much the same views as everyone else, then their exclusion doesn’t make any difference. All the research that’s been done to date indicates that this is the case. But it’s something that we’re continuing to track, and if and when necessary, we will make changes.

CJP: I also hear from people who say that they’ve never been contacted by a pollster, so how can this be real?

GL: That’s human nature. The old pollsters’ joke goes that if you don’t believe in random sampling, then the next time the doctor wants to take a blood test, have him take it all. The point being that a tiny drop of blood drawn from your body is enough to portray or understand the components of all of the blood in your body. Your red cell count, your white cell count, the cholesterol, you name it. And it’s enough regardless of the size of your body, no matter if you are a white mouse or a man or a women or someone the size of Pluto.

That is random sampling and it works with people the same way it works with blood.

CJP: Even though people aren’t as homogeneous as blood?

GL: Blood is less homogeneous than you think.

CJP: I actually don’t anything about blood.

GL: If you’d like, you can use minestrone soup as a metaphor.

But the real objection from people who say, "How can my opinion be expressed in a survey in which I was not personally included?" is a personal objection rather than a sampling objection. It sounds wrong, almost un-American. But the basic thing to understand is that all individuals reach their opinions through completely personal and idiosyncratic paths. Nonetheless, we all arrive via those paths to the same point of decision, where you either support candidate A, candidate B, or are undecided.

CJP: As Iowa and New Hampshire approach, we’re hearing that the national polls don’t matter- it’s the early states that matter.

GL: If you care more about who is ahead in a state, you should look at a good, quality poll from that state. If you care more about how our nation is coming together to evaluate the candidates and the issues as we move toward a decision, then national polls capture that very well.

CJP: I also hear guidance that if you average the results of several polls then you’ll have an indication of who’s really up and who’s really down. Is that good advice?

GL: I don’t think it’s good advice because it ignores the issue of quality in polls. So you may be averaging surveys that are well done and surveys that are poorly done, which s not a good idea. We do need to find our way away from this false notion of pinpoint accuracy in polling. A poll is an estimate. This is not laser surgery we’re doing on an eyeball here.

CJP: For somebody who is seeing information about a poll, how would you quantify a reliable size for a poll?

GL: It makes no difference. You can have a very large number of interviews in an Internet click-in survey that should be absolutely meaningless. You want a representative sample, not to get hung up on sample size.

A probability sample in a good, quality representative public opinion poll of 100 cases is a sufficient size to start analyzing subgroups. The margin of sampling error is about 10 points in a group of 100 people. The biggest danger in polling is not size, but over-hyped, over-analyzed data.

CJP: So your biggest piece of advice for news consumers is to consider the source.

GL: Absolutely.

Topics: CJP Interviews, Voting

4 Responses to “A Polling Primer with ABC News…”

1. Mike Says:
   March 27th, 2008 at 5:25 pm

   I find it a complete boggling of the mind that Gary Langer believes he is non-partisan and neutral in his polling for ABC.There is not one person in this world that is completely unbiased in there opinions,outlook,preferences,political persuasion,etc.For this website to not challenge the validity of Gary’s polling prefrences is interesting.Is it bias?

2. admin Says:
   March 27th, 2008 at 5:38 pm

   HI Mike,

   Thanks for your comments. Are there instances of bias you’re thinking of for ABC? He was as critical of his own newsroom as any others, but I’d be interested to know more about why you think he’s partisan or not neutral….

   As for bias of this website, we really don’t have a dog in the fight.

3. Lynette Says:
   April 4th, 2008 at 9:27 pm

   Senator Hillary Clinton is receiving calls from Senator Patrick Leahy, Senator Chris Dodd and others to drop out of the Presidential Primary even though the difference between the two candidates in both popular vote and pledged delegates is minimal and the fact that neither candidate can win the nomination without the Super Delegates. The question no one is asking is, if the fortunes of the two candidates were reversed, would there be calls for Senator Obama to withdraw? Frankly, I doubt it. The DNC would be afraid of alienating the loyal and highly vocal African American base. More than likely any attempt to push Senator Obama out of the race would be met with charges of racism.
   Women comprise fifty-one percent of the population and an even greater percentage of the voting electorate. Yet, women are vastly underrepresented in all three branches of government, from the congress to the cabinet, and there has never been a female President or Vice President. On a personal level, I am reminded daily of the subtle yet significant government sponsored sexism that permeates my life. Not a single piece of paper currency has a picture of a woman on it and the vast majority of pictures on stamps are of men. Pick up any newspaper and compare the number of photos of men vs. women, and you’ll be astonished by the results. From Wall Street to the White House, we are in many ways still a nation of “men and girls.” Women are undoubtedly still the second sex.
   Hillary’s candidacy is significant to me as a woman. She knows what it is to be a woman, to be marginalized and sidelined. I want the Obama supporters who are trying to push Hillary out of the race to know to me a woman in the White House is CHANGE, REAL CHANGE and the most significant political event of my lifetime. I want the woman and girls of America to see a woman in the White House. I am a registered democrat, but if Hillary is pushed out of the race, I will not vote and I urge other women to do the same. Why should I be loyal to a party where a group of men try to shame the first female candidate for the Presidency out of the race?

4. Walter Garet Says:
   May 5th, 2008 at 1:57 pm

   Lynette, I agree with you. I believe Hillary Clinton is best suited to be our candidate to represent us in the Presidential race. I also would like to ask our Democratic Leaders, why they don’t count our votes from Michigan and Florida. I vote for Hillary Clinton and yet they act like it doesn’t count. Well I’d like to tell them if my vote don’t count they are not going to get it either.

Comments