What to conclude about public opinion research post-US election
The apparent failure of political polling to accurately predict the stunning outcome of the 2016 US election is raising fresh questions about the validity of this kind of research. There are many reasons why it is becoming more difficult to measure voter intentions and rely on modelling to predict election outcomes.
But there are other, arguably more valuable, purposes to survey research. Reprinted here is an op-ed published by the Institute's Executive Director Keith Neuman, that speaks to this issue.
The polling that matters comes after the election
(Globe and Mail, June 17, 2014)
Another provincial election has come and gone in Canada, and once again pre-election polling is coming under scrutiny for not getting it right. What used to be a predictable track record of accuracy in measuring voter support leading up to election day has now become anything but predictable or seemingly accurate. As the pollsters correctly tell us, trends have conspired to make the science of voter prediction an increasing challenge, stemming from the growing difficulty in engaging representative samples of the public, diminishing resources available to pay for quality research, declining party loyalty (what McGill professor Stuart Soroka characterizes as “flexible partisanship”), and declining voter turnout which makes it difficult to know which survey respondents will act on their stated preference.
This state of affairs prompts renewed questions about the value of pre-election polling, and whether it has a future in its present form, or at all. The tone of this discussion is often one of betrayal, as if suddenly we can no longer trust our weather forecasters to tell us if it will rain or shine the next day. This debate about pre-election polling will not be resolved anytime soon, but there are two important lessons that can be drawn from it.
The first lesson is to place more realistic expectations around what pre-election polling can and cannot do for us. The focus of such polls is to accurately track the horse race voter preferences of leading parties and make a prediction of how voters will cast their ballots on election day. This exercise will never be a true science because predicting behaviours from attitudes is inherently imprecise given the many factors that influence what people do. The true purpose of published election polling is media attention: For media companies to generate inexpensive headlines and attract audiences, and for pollsters to boost their profile for marketing purposes (and for this they invest their own money since media no longer can afford to pay for polling).
The most important truth about pre-election polling is that it offers very short term value. Each poll is important until the next one is published, and once the vote is in they all become irrelevant except for academics and research methodologists. Pre-election polls have no lasting value to governments, governance or society; like a horse race, once the winner has been announced it’s all done and everyone moves on. In the end, the significance of polling is largely in how it feeds the media’s need for a narrative to make elections newsworthy.
The second important lesson is to pay more attention to the polling that does not take place during elections and which is not fixated on horse race winners and losers. This includes the surveys that measure citizens’ opinions, perceptions, priorities and expectations about the issues that matter to them and to the broader society (e.g., the economy, health care, taxation, transit, immigration). Unlike horse race polls, this type of public opinion research can tell us something meaningful about the state of our society, where it is heading (through trend analysis) and what sorts of policies and initiatives the public might be prepared to support.
Public affairs research is less encumbered with the problems confronting election polling: Rather than trying to predict voting behavior precisely to determine election outcomes, it focuses on attitudes and the population segments holding different viewpoints on issues.
Election outcomes make a definitive statement about what voters want, but they do not tell us why voters made the choices they did. Pundits, parties and pollsters are now interpreting why a plurality of Ontarians chose Kathleen Wynne’s Liberals over the opposition parties, but most of it is speculation and much of it coloured by a partisan view. Were the swing voters attracted to the Liberal leader and/or her platform, or repulsed by the Tory plan for smaller government? On this question the pre-election polls cannot tell us with any certainty, and the vote itself is mute. Exit polling can play an important role in helping to answer this question, but this type of research is rarely done in Canada, due to its cost and media disinterest once the vote is in.
Public opinion research focusing on issues does not get the spotlight of election polling, but it is being done and deserves more serious attention. In the US, the respected Pew Research Center publishes a steady stream of high quality research that paints an invaluable portrait of the American people and how it is evolving over time. In Canada, there is much less of this research but it is there for anyone choosing to look. A handful of leading Canadian research companies conduct this type of public affairs research, some of which is proprietary (intended only for the eyes of paying clients) and some published for all to learn from. In 2006, Michael Adams founded the Environics Institute for Survey Research as the first non-profit entity in Canada dedicated to conducting and publishing independent research on public policy and social trends, with a mission to do for Canada what the Pew Center is doing for the US. This type of research has lasting value for all sectors of society who want to know who we are as a country and a people, without the sometimes distorted lens of media hype and partisan agendas.
In the absence of a more serious investment by media to strengthen the quality and rigor of pre-election polling, it is wise for them to demonstrate greater restraint in what they publish and for the rest of us to consume it with more realistic expectations. In the end, it would be more productive to direct our attention to the polls conducted outside of election periods on the broad questions that really matter.
Conservatives and the 'values' thing The next leader must embrace the base — without scaring off everyone else
Theh following commentary was published by Michael Adams in the July 15, 2016 edition of iPolitics >>>>>>>>>>>>>>>>>
Even as most of us are glued to America's rancorous election, some Canadians--notably committed Conservatives and New Democrats--are charged with choosing leaders whose ideas and personal identities will rally current supporters and even attract some new ones. Few would disagree that last fall’s election was about values and leadership. And it will be values and leadership that determine who will lead the two parties currently in the midst of leadership contests, and who will lead the country when the Liberals conclude their current mandate.
In the old days, partisan divides in Canada were said to be about the three Rs: Religion (Catholic/Protestant), Race (French/English), and Region (West/Centre/East). Economic interests that fell outside those categories, like union membership, also mattered. Today most of these past drivers of party affiliation are either irrelevant or sporadic in their influence. Contemporary political divides have more to do with personal values than traditional group identities or our positions relative to Marx’s means of production.
To understand the social values of Canadians, Environics has conducted annual surveys of people aged 15 and over since 1983. Earlier this year we surveyed over 4,000 Canadians, tracking 74 social values that illuminate our motivations and mindsets as they relate to our roles as citizens, consumers, workers, family members, and spiritual beings.
The data shed interesting light on supporters of Canadian political parties. Although over the years we have come to expect certain patterns to recur in partisans’ values, this year we were amazed at just how closely the values of Liberal and Conservative party supporters lined up with the positions and sensibilities their parties expressed during the fall election campaign.
Liberal supporters score high on values associated with diversity: multiculturalism, flexible definition of the family, and social learning (the idea that we’re enriched by contact with people different from ourselves). These values are accompanied by a strong sense of national pride. In many societies, strong patriotism goes hand in hand with xenophobia: “I love my country, and don’t want Others to ruin it.” For Canadian Liberals, the combination is quite the opposite: “I love my country because different kinds of people can coexist peacefully here.” Justin Trudeau’s Liberals embody these values strongly.
But Liberals’ affinity with their party’s current image extends farther. Liberal voters also scored high on nearly all the values associated with personal style, novelty, and originality. Although there is nothing novel about the Liberal party itself, a big part of its leader’s appeal was a sense of generational change and youthful flair. The images of Justin Trudeau sporting colourful socks with a sober suit, doing yoga stunts, and even posing for selfies might seem superficial to his critics, but these playful, spontaneous gestures resonate with Liberal voters who say they strive for such moments of fun and authentic self-expression in their own lives.
The Conservatives, currently being ably represented by interim leader Rona Ambrose, are the party most likely to dislodge the Liberals if any party does at the end of their current mandate. Their challenge is to find a leader who embodies Conservative values as effortlessly as Justin Trudeau seems to embody Liberal ones. The task is not altogether straightforward. Conservatives must find a way to hit the “refresh” button, presenting a new face and approach, without alienating voters who (arguably by definition) have little appetite for change.
Consider the example of the value Traditional Family, which boils down to a belief that a “real” family is a married mom and dad with kids. Traditional Family is the single strongest value among those who voted Conservative in the last election--meaning not that it is their top priority as a group, but that it’s the one that most sharply distinguishes them from the national average. That said, while the other parties remain much more accepting of same-sex marriage overall, Conservatives on average have moved more than anyone else toward acceptance of same-sex marriage over the past decade. This helps to explain the party’s official acceptance of such marriages at their recent convention.
A second tricky value for Conservatives to navigate will be Cultural Assimilation, the second strongest Conservative value. This value is the opposite of Multiculturalism, and registers a belief that it is the duty of immigrants to adopt Canadian customs and values, leaving behind the customs and values of their countries of origin. One of the great achievements of the Harper government was its success in attracting immigrant voters. Their strong disavowal of anti-immigrant messages yielded rewards at the ballot box. When Harper’s team changed course, most notoriously through Kellie Leitch and Chris Alexander’s “Barbaric Cultural Practices Hotline,” they suffered. (Hotline sting?) The hotline episode gives a hint of the Conservatives’ dilemma on this file. They cannot alienate the foreign-born population that represents more than a fifth of Canadians, including many voters favourably disposed to both fiscally and socially conservative ideas. Nor can they alienate the portions of their base who are driving the high scores on Cultural Assimilation and who would be drawn to--if not a Canadian Trump then perhaps a Canadian Cameron or Sarkozy.
Conservatives tend to stand out in their support for traditional social structures: religion, father-led families, and hierarchical organizational models. Conservative MPs’ recent efforts to block the introduction of gender-neutral language into the national anthem is a smart way to channel supporters’ sentiments, combining a belief in both traditional patriarchal authority and a desire to simply leave existing rituals well enough alone. For them, the fact that something is traditional, regardless of the content of the tradition, holds value in itself. Conservatives also stand out in their fear of violence; they are more uneasy than average about the threat of violence in the world, including in their own neighbourhoods at night. Finally, Conservatives disproportionately believe in virtues like duty and work ethic: they believe people must shoulder their responsibilities with stoicism, not indulge themselves.
After a decade of his leadership, most Canadians and many Conservatives were ready to turn the page on Stephen Harper. But whatever false notes he hit, the former PM did a good job of embodying Conservative ideas and, importantly, conservative sensibilities. He didn’t pretend to be fun. He worked hard and, except for a rare turn at the piano, met public life with dutiful seriousness. He did nothing if not lead an orderly, hierarchical team governed by extreme loyalty and deference. He was admiring of all manner of traditional institutions and symbols , from the military to the monarchy.
The fact that the core values that most differentiate Liberals and Conservatives revolve around orientation to the family and social diversity is both fascinating and meaningful. We are not talking here about the usual fodder for our day to day policy debates: Medicare, infrastructure, carbon pricing, equalization payments. Instead, values data reveal divergent orientations to our most fundamental institution, the family, and to the accommodation of diversity as expressed in culture and sexual orientation. In the data’s portrait of Liberals, who have been the primary custodians of the progressive values of the country over the past 50 years (often nudged along by the NDP), you see a continuing openness to social change: support for the equality of women and those of various sexual orientations and gender identities, and acceptance--even embrace--of immigration and ethno-cultural diversity.
As the Conservative party selects its next leader, it will need to find someone who can speak to Canadians who drive their party’s high scores on Traditional Family and Cultural Assimilation without alienating the young, urban, highly educated voters whose social and political clout can only be expected to grow. As for tone, for the time being Canadians, unlike our American and European cousins, seem to insist on civility and cooperation. Perhaps the next Conservative leader will tackle the next election by fighting sunshine with sunshine, and finding a way to celebrate Canada Day as enthusiastically as Remembrance Day.
Michael Adams is Founder and President of the Environics Institute for Survey Research.
Muslims and diversity in western democracies: Lessons from Canada
Commentary by Michael Adams, Founder and President of the Environics Institute
Sadiq Khan, London’s recently elected mayor and a Muslim, predicted that Hilary Clinton will “trounce” Donald Trump in the U.S. general election in November.
Khan’s Conservative opponent Zac Goldsmith was not as outrageous as Trump, but Goldsmith did try to stoke nativist fears to win votes. The strategy backfired and helped fuel Khan’s decisive victory. Khan remarked that Londoners’ “rejection of the politics of fear” should hearten Hillary Clinton.
Another political parable that might hearten Clinton is the outcome of last fall’s federal election in Canada. After years of being notably un-xenophobic and indeed savvy about reaching out to religious and cultural minority groups, Canada’s Conservatives, in power for a decade and seeking ways to energize a weary base, tried some divisive tactics. Their campaign’s most notorious element was the announcement of a “barbaric cultural practices hotline” through which non-barbaric citizens could snitch on their neighbours. Voters rejected the Conservatives’ gambit. Justin Trudeau’s Liberals received a mandate much larger than had seemed possible just a few weeks before the election.
Some Americans who think of Canada as a progressive hinterland might imagine this election outcome was inevitable, but the race was a nail-biter. Initially, stoking fear (in both explicit and implicit ways) about Muslims looked like it was helping the Conservatives in the polls. America is not the only society where a segment of the population feels a murky but potent anxiety about a combination of immigration, demographic change, and violent extremism by those claiming to represent Islam. Does the Canadian election hold any lessons for U.S. Democrats who are fighting an especially potent brew of fear and anger, and an opponent who seems willing to do anything to exploit these?
One lesson of the Canadian election is that even when polls suggest the public is sold on an issue--sympathetic to a discriminatory policy, for instance--those who make an eloquent case for the larger landscape of inclusive values can still prevail.
A Conservative proposal that received a great deal of attention during the last election was that Muslim women who wear niqabs (headscarves that expose the eyes but conceal the rest of the face) should be required to remove them while being sworn in as citizens. From the outset, it was clear that the Charter of Rights and Freedoms guaranteed the women’s right to wear their religious garments; the Conservative government fought this plainly losing legal battle in order to be seen to be fighting it. And most Canadians agreed with the government’s position, constitution or no.
But despite the popularity of the unconstitutional anti-niqab policy, many analysts agreed that the niqab debate (combined with the ill-fated hotline) helped to turn the tide of the election not toward but away from the Conservatives.
Most Canadians are positive about immigration and proud of their multicultural society. Some argue that the story we tell ourselves about diversity and inclusion papers over ongoing racism. There is certainly evidence that such racism exists. Nevertheless, Canada is an extremely diverse society and that diversity has become deeply ingrained in the story we tell about ourselves, our equivalent of America’s rags-to-riches Horatio Alger myth.
Polls suggested that a strong majority of Canadians supported the Conservative niqab policy, but at the ballot box nearly two-thirds of voters (63.5%) chose parties that explicitly opposed it. The proposal was an odd combination: popular but ultimately toxic.
The country seemed to say, “That’s just not us.” Of course, the niqab issue was one among many in the campaign, but it featured in most post-mortems of the election, including those carried out by Conservatives who regretted their party’s tactics. In other words, even many members of the party that proposed the plan concluded after the intensity of the campaign had subsided, “That’s just not us.”
There are plenty of differences between Canada and the United States, and our former PM, Stephen Harper, who attempted the fateful xenophobic gambit, could hardly bear less resemblance to Donald Trump. But Americans, even if polls suggest they like some of Trump’s proposals that shock their fellow citizens, may be won over by words and actions that call them back to a more expansive national story as Canadians were. In Canada, deeper values of pride in multiculturalism and inclusiveness trumped the (significant) short-term appetite for a discriminatory policy.
Hillary Clinton has already positioned herself as the candidate of Americans’ better angels. Having declared that she is not a natural politician, perhaps she will not be able to summon a speech on fear in the register of President Obama’s riveting lecture on race in 2008. But she might. Or perhaps she will, as our eventual prime minister Justin Trudeau did, simply embody the qualities that set her apart from her opponent. Some of Justin Trudeau’s strongest weapons were smiles, open ears, and personal warmth that all seemed genuine. This might sound like a standard glad-handing-politician tool kit, but in contrast to his main opponents, two older men with divergent politics but personas that tended equally toward anger, an easy smile was powerful. Trudeau came across as liking Canadians--all Canadians--and liking Canada. The morning after the election, the images on the front pages of the papers were of Trudeau in the Montreal Metro, posing for selfies with young women in hijabs. It was just an image, but it connected to a larger story that most Canadians recognized and liked.
After an inflamed, sometimes alarming election year, perhaps November will find Americans coming back to the more idealistic and inclusive angels of their nature, looking at pictures and headlines in which they recognize themselves and like what they see.
How well do web surveys represent the public?
Arguably the most significant development in survey research over the past decade has been the rapid emergence of web-based online research methods. This trend has been driven primarily by the spread of online access and activity across the population; today roughly nine in ten citizens are connected to the Internet, and a majority do so everyday.
Online research is not without its limitations (click here for a cogent review), chief among them is reaching representative samples of the target population. Unlike telephone-based surveys (in which every individual with a telephone can in theory be reached), online surveys of the general population rely on panels of individuals who are recruited through various means (in most cases through popular websites). This means that the composition of such panels is not representative of the full population, and moreover it is difficult to determine clearly who is left out. This is a major limitation for research that aims to extrapolate findings to the broad population, and the reason why online methods are not considered acceptable for many research applications that require accurate population estimates.
The Pew Research Center has just published a helpful new analysis to answer the question of how online and offline populations are different, both in their composition and how they answer survey questions. They accomplished this through their unique American Trends Panel, which was created to accurately mirror the US adult population.
What did they find? The Pew analysis compared the results of online and offline samples across 406 survey items across a broad range of topics, and in most cases the differences were “quite small.” But there were some notable differences, most notably on questions related the Internet and technology, and to a lesser extent on political knowledge and financial circumstances. Perhaps more important are the differences that emerged within certain subgroups in which the online and offline profiles were different. The one in five Americans who were surveyed offline were more likely to be 65 years and older, Black, rural, Protestant, and have lower levels of education and income. And within these groups, the opinions of online and offline samples are noticeably different (that is, older Americans surveyed online are distinct from their counterparts who participated offline). This means that online surveys will not effectively represent the perspectives of these segments of the population, even if the correct proportion of them are included in the survey sample.
The Pew analysis concludes that the overall coverage error (or bias) is modest in scope, but clearly present. While such coverage error doesn’t challenge the legitimacy of online research, it poses a substantive weakness for studies that seek to produce valid population estimates. This will be most problematic for research topics pertinent to broad public interest and especially ones that have implications for vulnerable segments of the population (e.g., social programs, access to health care).
This valuable work focuses on the US population, but similar findings would likely apply in Canada. What Canada lacks is an organization like the Pew Research Center with the mandate and resources to undertake comparable work in this country.
Katrina 10 years later – How is New Orleans doing?
Ten years ago this month Hurricane Katrina came ashore at New Orleans and devastated parts of the city, with aftereffects taking an even greater toll on the community in terms of dislocation, government mismanagement, racism and post-traumatic stress. Reconstruction of the infrastructure has been underway and can be easily measured but what is less clear is how well residents are coping with the aftermath. This is where social research is needed, and the Kaiser Family Foundation has taken important leadership in launching a sustained study of residents post-Katrina that has involved city-wide surveys in 2006, 2008, 2010 and now most recently in 2015 (10 years after the tragedy).
The latest survey was conducted June 2 – July 5, 2015, among 1,517 randomly selected adults ages 18 and older residing in Orleans Parish, Louisiana (the city of New Orleans). Computer-assisted interviews conducted via landline telephone (705) and cell phone (812) were carried out in English and Spanish.
The survey results show remarkable progress New Orleans has made in the 10 years since Katrina, as well as the stark challenges that remain. But not surprisingly, the biggest challenges were not brought about by the storm itself but how it exacerbated the vast difference in the living circumstances of the city’s African American and white residents. This gap is reflected not only in the rates at which African Americans and whites report ongoing financial problems and a lack of neighborhood services, but also in their feelings about how far New Orleans has come in its recovery, and their views of the city as a good place for young people.
While a majority of both blacks and whites remain optimistic about New Orleans’ future, a third of African Americans and nearly half of young adults are considering moving away. This is a telling and troublng indicator for the future of this unique American city.
Survey research establishes a beachhead in Cuba
Survey research has spread to virtually every corner of the populated world in recent years, extending even into conflict-ridden countries like Iraq and Afghanistan. The list of countries where such research is not possible is very small and recently shrunk by one with the elimination of Cuba. Earlier this year, the Washington Post reported on what is likely the first ever independently commissioned public opinion survey of the Cuban people living in Cuba. The survey was conducted by Miami-based Bendixen & Amandi International (a Miami-based research and communications firm) on behalf of the media company Univision Noticias and Fusion. The research was conducted without authorization of the Cuban government, and it is interesting to speculate whether the government was aware that it was taking place.
The survey was conducted in-person (the standard method for most non-OECD countries) with a representative sample of 1,200 Cuban adults over a 10 day period in March 2015. Interviews were conducted by Cuban residents trained by the research firm, and the interviews were recorded on handheld electronic devices so that responses could be sent electronically to a server outside the country. As reported, the demographic profile of the survey participants closely reflects the known distribution of the population by race, gender, and religion, while somewhat underepresenting older generations.
What makes this research so important is that it provides a unique empirically-based glimpse into the minds of Cubans, that is likely to offer a more accurate picture than the anecdotal and politically-tinged assumptions.
The survey tells an interesting story. On the one hand, there is a positive story in that most Cubans express optimism about their future and that of their family, and say they are satisfied their education and health care systems. And nearly all Cubans agree that normalization with the USA (now finally about to happen) is good for Cuba, and most have a positive view of US President Barack Obama. On the other hand, there are areas of clear discontent: especially with the state-run economy, but also with the political system. Only one in five feel comfortable expressing themselves freely in public, and more than half say they would like to leave and go live in another country (the USA being the top choice of most, followed by European countries and Canada).
Also of interest are the media habits of Cubans. Radio is the principal source of news and entertainment, with only one in six (16%) reporting access to the Internet (and mostly outside of the home). Among those with such access, four in ten use social media (with Facebook being the most popular) and it is used primarily to communicate with others outside the country (no one in the sample said they use social media only to connect with other Cubans in the country).
Given the politics and culture of the country, did Cubans feel it was safe to provide honest answers to the survey questions? There is no way to determine this, but it is likely that some of the respondents declined to answer certain questions even if they had an opinion. About one in five declined to answer questions about whether the country should have more political parties, if the US is a friend of Cuba and whether they hold a positive or negative opinion of the Catholic Church in Cuba. This limitation notwithstanding, this survey represents a landmark first picture of public opinion in Cuba, and could well serve as a precedent for future research in the near future.
NYT on election polling – does it have a future?
Political polls have been part of the fabric of elections in democracies since the 1970s and for most of this period the biggest ongoing controversy has been whether polling results influence voters, and if so whether this is a good thing or not.
Today, political polls face a much more fundamental challenge, resulting from their uneven performance in predicting election outcomes in the past few years. There is a growing chorus of skepticism, if not dismissiveness, about whether polling is still valid. Polling has now become a target from many quarters, many of which are people who do not adequately understand how polling is done or how it should be used.
The most valuable perspective comes from those who are actively involved in the practice of survey research. The most recent published piece – and among the best so far – appeared in the June 21-2015 edition of The New York Times, by Cliff Zukin, a professor of public policy and political science at Rutgers University and one of the leading experts in this field.
Professor Zukin does not pull punches in what his profession is facing: “Election polling is in near crisis, and we pollsters know.” And he then proceeds to outline the specific trends that are making election polling increasingly unreliable (growth of cell phones leading to less representative samples, decline in people’s willingness to answer surveys leading to lower response rates).
His conclusion is stark:
“So what is the solution for election polling? There isn’t one. Our old paradigm has broken down, and we haven’t figured out how to replace it. . . Polls and pollsters are going to be less reliable. We may not even know when we’re off base. What this means for 2016 is anybody’s guess.“
This is hardly welcome news for those who look to political polls as a gauge of voter behaviour, and this state of affairs is as much an issue in Canada (and for our own upcoming federal election in October). But the profession is also actively working on new ways of conducting surveys that will work in today’s globalized online world. This will be a turbulent but exciting process to watch, so stay tuned.
Polling methods are under the gun, but remain essential to democracy
It is by painfully clear that public opinion surveys and polls of general populations are getting more difficult to do effectively, even as new technologies have arrived to make them cheaper and easier to do (e.g., SurveyMonkey). Recent election polling in the UK, Poland and Israel proved well off the mark, as just the most recent examples of how challenging it is becoming to take an accurate reading of voter intentions.
The latest voice to weigh in on this issue is Nate Silver, creator of the well-known data aggregation website FiveThirtyEight.com, which pioneered new methods for aggregating poll results to generate election outcomes (with much success in recent US election cycles). Some might assume that this type of big data methodology is a rival, and possibly a replacement, for traditional survey methods. But in fact the aggregation methods rely on polling data (and lots of it) to work. Nate Silver is not a pollster, but he is well positioned to comment on the current landscape and he has done so in a recent post on his website entitled: Polling is getting harder, but it’s a vital check on power.”
Silver’s piece is well worth reading, and the following excerpts in particular:
“So if the polls fared poorly, does that mean you should have listened to the pundits after all? Not really: In these elections, the speculation among media insiders was usually no better than the polls and was often worse. Almost no one, save perhaps Mick Jagger, assigned much of a chance to the Conservatives’ big win in the U.K. last month, with some betting shops offering odds of 25-to-1 against a Conservative majority.”
Polls are also essential to understanding public opinion on a host of issues that people never get a chance to vote upon. How do Americans feel about higher taxes on the rich? The Keystone XL pipeline? Abortion? Capital punishment? Obamacare?
Left to their own devices, politicians are not particularly good at estimating prevailing public opinion. Neither, for the most part, are journalists. One reason that news organizations like The New York Times and (FiveThirtyEight partner) ABC News continue to conduct polls — at great expense and at a time when their newsrooms are under budgetary pressure — is as a corrective to inaccurate or anecdotal representations of public opinion made by reporters based mostly in New York and Washington. Polling isn’t a contrast to “traditional” reporting. When done properly, it’s among the most rigorous types of reporting, consisting of hundreds or thousands of interviews with statistically representative members of a particular community.”
Crimeans weigh in on annexation, one year later
It has now been a year since the Russian annexation of Crimea, which contravened international law and raised international tensions between East and West rarely seen since the end of the Cold War. The Russian take-over of this region from Ukraine was widely viewed in western democracies as a belligerent political move carried out by military might against the wishes of many in the local population. But recent public opinion research reveals a different picture, in which a strong majority of Crimean residents approve of the annexation and believe it has been a positive change for their region.
The research comes from a recent public opinion study commissioned by openDemocracy, and conducted in December 2014 by the Moscow-based Levada Center. Unlike many surveys reported in western media, this survey was in-depth (conducted by telephone in Russian), encompassing about 150 questions covering a range of topics about identity, politics, media consumption and general issues facing the region.
Results from the survey show that a strong majority of Crimeans approve of the Russian annexation, with 84 percent of Russian and Ukrainian ethnic groups saying it was “absolutely the right decision”, with Ukrainian sentiment only modestly lower than that of Russians. The small minority of ethnic Tatars (who make up an estimated 12% of the population) is split between approval and disapproval, with 20 percent saying the annexation was “absolutely” right.
Consistent with these results is the fact that few Crimeans consider themselves to be “European”, in contrast to sentiments in other parts of Ukraine. And a clear majority (85%) expressed the view that Crimea is now moving in the right direction, in contrast to previous polling (e.g., 6% indicated this view in a 2009 survey by the International Republican Institute). Ethnic Tatars are largely divided on these questions.
The research shows that the vast majority of Crimeans of Russian and Ukrainian identity approve of the recent annexation of their region, with the Tatar minority divided. Does popular will trump international law? openDemocracy describes this as an example of an act that is “illegal but legitimate”, the same words used almost 20 years ago to describe NATO’s intervention that led to Kosovo’s separation from Serbia.
Is this survey legitimate, in terms of accurately portraying the opinions of the Crimean population? openDemocracy describes the Levada Center as having a reputation for “integrity, professionalism, and independence”, and there was no indication of government interference with the survey fieldwork. There is no other way to validate the results other than to await further research that might be undertaken by other organizations based outside of Russia.
Apart from providing important insights about the public mood in Crimea, this research provides a current and compelling example of the valuable role that survey research can play in international affairs. This type of research may not contribute directly to resolving political disputes, but it does provide necessary empirical evidence to settle key questions about public opinion that would otherwise be a battle of anecdotes and political spin.
Research Industry wrestles Margin of Error monkey
Almost any time you read about a public opinion poll you will see a sentence, usually at the end, stating a “margin of error” percentage, “plus or minus.” Those who know something about research will understand this to be a statistical measure of the representativeness of the sample of survey respondents recruited from the broader population under study. Those who know how survey research is done are likely aware that there is growing controversy about the use of this statistic as an indicator of survey quality.
The issues boils down to the following: The margin of error statistic applies to probability samples, such as those historically used for telephone surveys where every household is theoretically available to be sampled for a given survey. Such samples are increasingly difficult (and costly) to generate , and most surveys today are conducted through online methods that rely on non-probability samples. And yet the research industry and its clients continue to rely heavily on margin of error as the sine qua non indicator of survey accuracy.
The issue is well known within the research community but not discussed, until now. Annie Petit of Peanut Labs recently hosted a well attended webinar on this topic. The webinar featured four senior level researchers from the market research industry who discussed the relevance and applicability of margin of error in today’s world. The discussion is a bit dry, and may be difficult to follow for those who lack a basic understanding of margin of error and how it works since the webinar is aimed at industry practitioners. But the discussion is well worth listening to for those involved in survey research as a practitioner, client, journalist, or anyone who wants to understand better about how surveys are done.
There is no surprise in the fact that the four panellists all agreed that margin of error statistics are no longer as relevant in survey research today, and that they may do more harm than good in providing irrelevant and possibly misleading information about survey data quality. So why does the practice persist? In part this is because there are no other metrics of survey quality that offer the conciseness and face validity of margin of error. There are numerous sources of error that can affect the accuracy of survey results, and these are complicated if not impossible to measure.
The most revealing insight to come out of the webinar is how research companies are stuck with an irrelevant metric of survey quality because their clients demand it. Several panellists noted how they write survey reports that include a margin of error and then state it does not actually apply to the results (as a way to appease clients who insist the statistic is included). One panellist commented that to stop quoting margin of error even when it does not apply could well risk to the loss of valued clients. What this reveals is an underlying conflict between the science and business aspects of market research in today’s world. Commercial and media clients need data to drive or justify decisions, and they need to show their data is sound. Margin of error has been cast in the role of providing that seal of approval, and the inconvenient truth behind the science is easily ignored.
Not all survey research is conducted for business, and it would be illuminating to also hear the perspective on margin of error from practitioners in government, university and non-profit settings who are focused more on sound data than business confidence. Perhaps this will be the topic of a future webinar.
You can listen to the Peanut Labs webinar in its entirety here.