The Crisis in Polling
Lawrence R. Jacobs
Associate Professor
University of Minnesota
(612) 625-3384
Robert Y. Shapiro
Professor
Columbia University
Public opinion polling is in crisis. Many in Washington mistrust polling results as unreliable. Return to the 1996 presidential election. CNN and other media reported wild swings in President Clinton's lead over Bob Dole, and Clinton's actual margin of victory was narrower than CNN and other media polls had predicted. An ABC survey last fall found that half of Americans did not believe the election polls. Even pollsters were at each other's necks, with Everett Carll Ladd charging in the Wall Street Journal that election polling in 1996 was "worse" than even the infamous debacle in 1948 when pre-election surveys erroneously picked Thomas Dewey over Harry Truman.
The crisis in polling goes deeper than one election. The fundamental concern is that polling has lured politicians into "pandering" to public opinion when making significant policy decisions. The cost has been an abdication of leadership and of the will to use personal judgement to promote the national interest. President Clinton is often cited as prime evidence of a pandering politician.
Polling is in crisis but not because they are either unreliable or induce pandering. Research by ourselves and others lead us to conclude that serious polls are generally reliable and that today's politicians do not pander. Skepticism toward polling is understandable but the blame has been misplaced.
The problem is not polls themselves. Rather, the problem is how they have been used. Today's crisis has been caused by journalists who distort poll results and by media pollsters who grant the White House and other prominent political activists privileged access, information and influence. What we need is a new code of conduct for political activists, journalists, and media pollsters.
The Myth of Pandering
That today's politicians pander is nearly universally accepted but there is no systematic evidence to support it. We set out -- with support from the Pew Charitable Trusts -- to investigate pandering by interviewing 114 people who advised or worked in the executive and legislative branches from 1993 to 1996. The persistent theme from our interviews was that public opinion and polls were not decisive influences on decision making. For instance, when senior congressional staff were asked whether opinion influenced decisions, three quarters spontaneously challenged the premise of the question and volunteered that public opinion was of no use. The consistent message was that the policy positions of members were guided by their personal beliefs, ideology, and judgments about policy and the effectiveness of government and business. As one Republican staffer explained: "On policy, beliefs are more important than public opinion."
Officials in Congress and the Administration pointed to a number of reasons for discounting public opinion and polls. Nearly everyone we interviewed emphasized their distrust of polling results. A Republican staffer expressed a pervasive judgment: "we discount any polling -- it all depends on how questions are asked, who is asked. Who designs [a poll] can make it say what they want." Moreover, congressional and administration officials explained -- as one Republican staffer put it -- that public opinion and polls were "not at all useful in day to day policy making." They were either not specific enough or avoided the tough tradeoffs that faced policy makers. Finally, a few simply argued that public opinion was an inappropriate consideration in formulating policy, pointing to their responsibility to use their judgement.
Obviously, those we interviewed may simply have offered the expected answers to politically loaded questions. The finding that personal philosophy and beliefs (not public opinion) drove policy is so at odds with conventional wisdom that it may seem unbelievable. Indeed, we would not trust the interviews if they were the only evidence. But, they are not.
Several bodies of research corroborate the pattern in our interviews. First, research on the relationship between individual members of Congress and their constituencies indicates that public opinion has a modest and highly contingent impact on members' voting decisions and electoral prospects. Part of the explanation for the muted impact of constituents is that voters give substantial weight to candidates' personal characteristics and party in their voting decisions, and know little about (and often do not act on) members' policy positions or records.
Another body of research compares overall government policy with reliable polling results. Several different methods of comparing polls and policy reveal a similar pattern since 1981: a decline in responsiveness to public opinion (not an increase as the pandering myth supposes). Compared to the period before 1981, policy is less in agreement with the preferences of the majority of Americans, and policy changes less often in response to shifts in public opinion. The exception to this overall pattern are the isolated periods just before elections (such as the fall of 1996) when popular legislation is passed.
The overall picture seems clear: government responsiveness to public opinion is muted or declining. There is no systematic evidence that politicians persistently pander when making important decisions.
But, if public opinion does not appear to drive policy decisions, why do political leaders stock a veritable warehouse of information on public opinion? According to interviews and other evidence, the explanation for the apparent disregarding of public opinion amidst studious monitoring is that the Clinton administration and congressional leaders used information about public opinion to identify the language, arguments, and symbols for presenting already decided policies in the most appealing manner. Contrary to the notion that pandering politicians obsequiously follow public opinion, administration and congressional officials repeatedly emphasized that polls and focus groups were a "tool to shape our message," "educate the citizens" and "learn what messages we need to get across."
Understandable Reasons for Mistrusting Polls
The pervasive theme in our interviews -- that polls are not trusted by policy makers to reach policy decisions -- is a shrill wake-up call to pollsters and journalists who rely on opinion surveys. The harsh truth is that public opinion research lacks credibility in Washington as a tool for making policy.
The harsher truth is that the low credibility of polling is understandable. But, the source of the skepticism is misplaced. One of the real culprits are the journalists who report on polls.
We studied the polls conducted during the 1996 presidential election that tracked the standing of Clinton and Robert Dole -- the so-called "horse race" polls. These polls actually agreed far more than either journalists or the general public realized; the variations in Clinton's lead were normal and were readily explained by survey research methodology or changes in the campaign itself.
The problem with horse race polling in the 1996 election was the media's interpretation and reporting of them. A careful analysis of media coverage revealed that journalists -- in their scramble to package a drama-filled story of a changing race -- oversold, misrepresented, and misused polls.
We found two recurring themes. First, journalists were preoccupied with using polls to discuss the horse race, latching on to any sign of Dole closing in on the President. The fixation with horse race polls soaked up time from exploring public attitudes toward policy issues and other politically relevant matters, for which there were ample and easily accessible opinion surveys. Second, journalists largely limited their coverage to reporting internal polls (i.e. those commissioned by the news organization itself). CNN rarely reported polls by the New York Times or other organizations. The result was that audiences were denied the opportunity to weigh an internal poll against comparable surveys from competing sources; lost was the ability to reach more fully informed and balanced conclusions.
The Corrosive Ties that Bind Pollsters and Politicians
Confidence in polling as a trusted arbiter of public thinking has also been harmed by the regular contacts that media pollsters maintain with political activists. The lurking suspicion in Washington is that unfavorable media polls are the product of media pollsters being "reached" by their political opponents. It was precisely the suspicion of political favoritism that prompted House Speaker Newt Gingrich's outburst against the New York Times in October 1995 for devoting its lead column -- on the morning of a critical House vote -- to its poll that found public opposition to Republican proposals. Gingrich accused the Times of using "deliberately rigged questions that are totally phony" to further a campaign of "disinformation."
The danger of media pollsters' close political ties is most vividly illustrated by President Richard Nixon's secret ongoing relationship with the Gallup organization and Louis Harris, who began his national career as John Kennedy's pollster. Our recent article in Political Science Quarterly documented that Nixon and his senior aides secured three attractive advantages from their relationship with Harris and Gallup. First, the pollsters provided the Nixon White House with access to results well before they were published; this allowed the White House to plan elaborate strategies to promote favorable results or to deflect unfavorable ones. Second, the White House influenced the preparation of favorable questions. Nixon's aides "planted" the topics that Gallup and Harris selected for surveys as well as question wording.
Perhaps the most chilling evidence is that the Nixon White House used its close contacts with Gallup and Harris to attempt to influence -- with some real success -- the reporting of poll results. The most fundamental issue in polling is whether we can trust the numbers that are published. At least one well-documented case indicates that Harris actually changed the results he published in response to White House pressure. Tampering with published polls is akin to stuffing the ballot box.
Harris's and Gallup's relationship with Nixon violated longstanding professional norms to assure the reliability of results, to report full information, and to maintain distance from politicians.
Pollsters for the media continue to this day to maintain contacts with political activists. Harris and Gallup defend these contacts. Richard Morin, who directs polling for the Washington Post, confirmed in a column on our research that presidential aides continue to hound media pollsters to complain about question wording, the "spin" in recent news reports, and unreleased data that they heard about from a journalist. Speaker Gingrich's charge of politically inspired "misinformation" against the New York Times is an understandable reaction to the all-too cozy ties that bind media pollsters and political activists.
New Code of Conduct
The links between pollsters, journalists and leading political activists are largely hidden from public scrutiny. The first job at hand is to bring the hidden world of political polling into public view and to identify the problems that have created the crisis of confidence that now exists.
The second job is to specify a code of conduct by journalists, media pollsters, and politicians. First, journalists must do a better job of reporting a fuller and more accurate package of polling information. What are needed are "pollwatches" by journalists and others that would sound the alarms when distorted media coverage of polls appear. The pollwatch would rely on comparisons of polls by diverse organizations to evaluate the accuracy of media reports on polls. The confusion over polling in the 1996 campaign can be avoided in the future through greater care in reporting.
Further, media pollsters must keep the White House and other political organizations at a distance if they are to safeguard the credibility of published surveys. Being a detached scorekeeper requires minimizing contact with the political figures being evaluated. Even if political pressure is successfully resisted, contacts with presidents and others create an unmistakable and corrosive appearance that can only tarnish the faith of the public and political activists in pollsters as impartial and objective.
Finally, politicians need to look inward and consider whether their use of polls and focus groups is guided toward responsible education or self-serving manipulation of public opinion. Polls can make a useful contribution to identifying strongly felt public wishes and areas of misunderstanding. But, using them to sow confusion and deliberately mislead breeds confusion and, ultimately, political deadlock.
Curriculum Vitae / Bio Selected Articles and Reports / Recent Media References to Research
|