Insights with Impact Blog
Out and About: NEMRA Conference Goes Back to Basics with a Focus on Questions
June 8, 2017
At the New England Market Research Association’s (NEMRA) semi-annual conference, the Anderson Robbins Research team had the pleasure of connecting with, and learning from, our peers in the market research industry. All of the sessions were thought-provoking and centered around the theme of “questioning”.
Asking the right, well-developed questions, both before beginning a study and during the surveying phase, are crucial elements for successful marketing research projects. And so, it’s no surprise that these topics were touched upon in almost every session. Below are a few of the key take-aways on these important topics:
Asking the right questions before starting a project.
Many of the panelists stressed that the key to turning research into actionable results is to truly understand the research question you’re aiming to answer before you start a project. And this information is often learned by asking questions with key stakeholders, such as “what is the business decision you are going to make with this data?” Trying to glean valuable insights from your data without knowing this information, becomes much more difficult. This is second nature at Anderson Robbins.
Asking the right questions in a survey.
Just as important as asking good questions before starting a project, is the importance of asking quality questions during the surveying phase. As our first presenter, Ted Pulsifer said, “the data you get is only as good as the instrument [questionnaire] they [the respondents] are sent to.” This first session, on the importance of writing a good questionnaire, set the tone for the subsequent sessions and the topic was touched upon by many researchers throughout their presentations. Although we as researchers may think that we are always writing good questionnaires, it is easy to overlook some of the elements which are important to create an intuitive, thoughtful, and engaging questionnaire. David Harris explained four characteristics which are important for questions to embody. Questions should be:
1. Clear – Since you, as researchers, are not present to answer respondents’ questions as they are taking the survey, the questions need to be worded as clearly and unambiguously as possible.
2. Answerable – It’s very difficult to accurately answer a question for which you know nothing about. Ensuring that the respondent knows the information / topic matter that you are inquiring about will help to ensure that you get accurate data.
3. Easy – Tying back to the Clear point above, poorly worded and confusing questions are hard for respondents to answer, and additionally, they detract from having an enjoyable survey taking experience. Personalizing the question by wording it to ask about ‘you,’ (the respondent), is another way to make questions easier to understand and more enjoyable to answer. Additionally, personalization adds to the richness and depth of the given responses. As Frank Kelly demonstrated in his presentation, respondents were much more likely to give in-depth and thoughtful answers when the question was personalized, compared to when it was not.
4. Unbiased – Ensuring that your questions are not biased in any way is important to guarantee that your data is accurate and uninfluenced. The presenters raised caution around the use of Agree / Disagree scales as they inherently carry a positive bias. As David Harris mentioned, “people have more trouble disagreeing than agreeing”, and satisficing occurs all too regularly.Beyond the power of scales to influence respondents, Dr. Ronald Shapiro demonstrated the power of suggestivity, through question wording. In an engaging demonstration, Dr. Ronald Shapiro showed a volunteer a colorful birthday-themed bandana and then placed it over her eyes. Dr. Ronald Shapiro then asked the volunteer to tell him the color of the balloons on the bandana. At first the volunteer said that she did not remember any balloons, but when pushed and asked how many there were, without hesitation she said, “more than one and less than five”. Although it was a birthday-themed bandana and it could conceivably have balloons on it, there were none. This clearly demonstrates the power of questions to influence participants’ responses.
As researchers, it’s important for us all to consider the impact that questions have on our projects. The questions we ask and the questionnaires that we use to gather data are part of our professional discipline and should be treated as communication – as they are so important for the quality of our research.
Alternative Ways to Get Below Top-of-Mind
May 5, 2017
Frustrated by employees’ inability to develop creative ideas for advertising campaigns, industry executive Alex Faickney Osborn began developing methods for creative problem-solving in 1939. He began hosting group-thinking sessions and discovered a significant improvement in the quality and quantity of ideas produced by employees.
Much like Osborn, professional moderators today have techniques to help get their respondents thinking more deeply. In focus groups sessions, your respondents are your data. What they say is crucial. But from time to time, traditional questions and probes can sometimes fail to get deep enough into respondents’ perceptions, opinions, beliefs, attitudes – or POBAs. This is because sometimes, respondents do not have a way to verbally access their POBAs.
Here are a few of my favorite alternative ways to dig deeper during a focus group session:
• Deserted Island. Describe the scenario to your respondents (have them visualize or use pictures) – Imagine, you’re being marooned by pirates to live on a deserted island, and you can only bring 10 items. What items would you bring with you, what items would you leave on the shoreline in the event you are rescued, what items would you drop into the ocean to never see again? This exercise allows the respondents to compartmentalize items/attributes (or whatever is being tested) by importance and value to them.
• Bullseye. Print-out a bulls-eye, or draw one on a large piece of paper. Stick it on the wall or an easel, and provide respondents with sticker dots. Ask a question or read a statement, then have respondents place a sticker on the bulls-eye in regards to how they feel about the statement/question or how best the statement/question describes an item, new product, etc. Allow for discussion time after. This technique will also help moderators in reporting phase to more accurately describe how a question/statement resonated with respondents.
• Happy-Crappy Scale. As moderators, we want to stay away from quantitative methods in our qualitative work. Enter, the Happy-Crappy Scale. Based on the traditional Likert Scale, the Happy-Crappy Scale shows a sad-feeling face at the bottom, and a happy face at the top, with a line connecting the two. Respondents can react to a statement, question, new product, etc. by placing a sticker dot anywhere on the scale. Allow for group discussion after. The purpose of keeping the middle of the scale blank is so that respondents don’t have to feel corralled into “feeling” a certain way. There can be many feelings between crappy and happy, and so we want to represent that in the scale.
Be sure to Follow @ARResearch for more insights into the market research industry as well as our latest news.
January 24, 2017
Anderson Robbins is like a unicorn. It’s the mythological company you’ve always wanted to work for but thought didn’t exist. Let me tell you a little bit about why you want to join our team.
We work with some of the biggest names in business and politics and we help uncover the information they need to shape the world around us. As a part of our team, you will act as an investigative reporter discovering the most interesting data and insights from real people: Asking the right questions that drive decisions. You feel smarter when you leave work each day and you are a lot of fun at cocktail parties or playing Jeopardy on the couch.
We are looking to fill the positions of Senior Analyst; Analyst; Director and fulltime paid Intern. Check out our careers page for more details.
December 29, 2016
Making a promise just to break it after a few weeks, never seemed like much fun. But some people actually keep their New Year Resolutions. One example: Famed Facebook CEO Mark Zuckerberg. Last year, he committed to reading a new book every other week and he chronicled it on his social media platform. He also resolved to build himself a personal assistant using artificial intelligence, which he recently showed off at a Facebook Town Hall. Let’s get real. I’m not sure coding robots is achievable for me in 2017. But standing on the precipice of a new year, I asked my always motivating co-workers at Anderson Robbins Research what their resolutions are for 2017 to try and inspire my own goals. Here’s what they had to say:
Chris Anderson, President
• One thing I am committed to in 2017, is being greener. I want this to be the year that I finally get solar panels on my house.
• In 2016, my company worked very hard to pass Question 4 in Massachusetts. I am committed to making sure the will of the voters is heard, and the measure is implemented properly in the next year.
• The political shock waves that are still reverberating around the country inspire my next resolution. I am going to remind clients and media that low probability outcomes do happen.
• Finally, on a lighter note, in 2017 I want to win at least one Pokemon battle against each of my sons. After all, low probability outcomes do happen.
Jennifer Robbins, CEO
I’m always up for a challenge. In 2016, I chaired the March of Dimes Falmouth Road Race Team. We conquered 7.2 miles in the heat and raised thousands of dollars to make sure every baby has a fighting chance. I also decided to run my first half-marathon. While I have more races on the books for 2017, I am also going to take on my biggest challenge yet. In 2017 I am starting to write a book on the fascinating work I do everyday at Anderson Robbins Research. Stay tuned for updates on my progress.
Lauren Manganello, Director
Working fulltime and being a mom of three, I’d like to try and create more life balance in 2017. This includes practicing yoga to provide mindfulness, exercising to provide better heath and reading to expand my knowledge.
Andrew Schwartz, Director
Generally, I’m not one for resolutions, but this year I have a few goals that jump out:
• Get married. Since I’m engaged, this should be both self-explanatory and remarkably easy.
• Hike more. A major part of the reason I moved out to Denver is to enjoy the great outdoors. That was pretty challenging this fall due to the election, but it’s the best way I know to recharge.
• Apply my professional skills to local community organizations. Whether this is volunteering or bringing on new clients, I’ve got this base of knowledge and skills to help organizations advocate for a cause and communicate effectively. There are lots of local opportunities to do so – it just requires me getting out into the community and doing it. Especially important to me following the election.
Henry Kanter, Director of Operations
I’m trying to live more in the moment, not spend all my time thinking about the next moment or the moment that is five moments away. When I’m on walks, I literally stop and smell the roses. So many little things come up in my daily life that I just push past but now I’m taking a moment to check them out. I am also hoping to have the opportunity to move to somewhere closer to the ocean with my wife Cindy.
Julia Valletta, Analyst
My New Year Resolution came early! In 2017, I hoped to be able to moderate my very first in-person focus group for Anderson Robbins Research. I got that chance this past Monday in Plymouth, MN with our client, Cargill – the largest privately held corporation in the US in terms of revenue. Despite a few nervous jitters, it went smoothly… I did, after all, learn from the best – my boss, Jen Robbins.
Lauren Coates, Analyst
My New Year’s resolution for 2017 is to become more involved with charity work. Chris and Jen have been great examples of giving back to the community through their work with Read to a Child and March of Dimes.
Hannah Jaeger, Intern Extraordinaire
I’m looking forward to getting back into the habit of running, and plan to run a 5K every month. I ran my first half marathon in September and found that the benefits were beyond just the physical. When I run regularly, I find that my mind is clearer, and I am more focused and productive in my personal life as well as in my career. In fact, during my half-marathon training I received my internship position at ARR, and the race was during my first week of work! Consistent goals allow me to be the best version of myself.
As for me, there are so many things I’d like to work on 2017. I have been told by peers in the past that I thrive on a “frantic Francy” pace. Being super busy (and stressed) makes me most productive. I want to break that habit because it isn’t sustainable in the long run. My motto for 2017 will be to breathe more and talk less. I hope it will make me – and those who have to listen to me – even happier in the new year.
An Inside Straight
November 15, 2016
The national polls were right. On average, national polls found Hillary Clinton up by 3 points leading up to Election Day. She appears to have won the popular vote by about 1 point.
National polls have averaged a 2-point differential from the final result over the past 12 presidential races, so the 2016 polling was about as accurate as usual.
Based largely on eight national polls I conducted in partnership with Fox News over the fall, I thought Hillary Clinton would win the national vote by 2 to 4 points – enough that an Electoral College upset would have been out of reach.
But when I revisited our pre-election polling, the story of a potentially much closer race was right in front of me. Clinton’s support had been flat and her edge ranged from just 1 to 3 points, with the exception of right after the release of the Access Hollywood video when Trump’s support briefly slipped 4 points.
So why was there such widespread surprise at a Trump victory when Clinton’s edge in polling was within the margin of error all fall?
The main reason for the great surprise was that a lower probability outcome prevailed. Donald Trump drew the inside straight many said he would need in order to prevail. Before Trump’s election, 3 of 44 presidents lost the popular vote but won in the Electoral College. The odds of drawing an inside straight are 4 in 47.
But what happened that led to this lower probability outcome?
Late deciding voters broke disproportionately for Trump.
Undecided voters and those supporting third-party candidates in our polling tended to view both Clinton and Trump unfavorably, while also thinking Clinton had the judgment and qualifications and Trump did not. I thought these voters were unlikely to break disproportionately for Trump at the end, but they did. According to exit polls, voters who were unfavorable toward both candidates voted for Trump by 20 points (49% to 29%).
Nationally, voters who decided in the final week broke for Trump, 47% to 42%. But the late break was even more decisive in Wisconsin (59% to 30%) and Pennsylvania (54% to 37%), where Trump achieved his biggest upsets. As it turned out, late-deciding voters were more interested in “draining the swamp” than in traditional qualities associated with competence.
There was a lower and harder ceiling on Clinton’s support than many thought.
Clinton never trailed in any of the eight polls we conducted over the fall, and I assumed this meant she had a higher ceiling of support than Trump. In hindsight, I should have questioned this assumption when Clinton’s support stayed stuck at 45% after the Access Hollywood video release. When voters didn’t move toward her after the release of a video that would have been fatal to any past candidate, why would they on Election Day?
There was no Rust Belt firewall.
Either there were some significant polling misses in Michigan, Pennsylvania and Wisconsin, or there was a huge late break for Trump. Either way, polling in these states contributed to a belief that Clinton had a Rust Belt firewall that would save her in the Electoral College, even if she lost the popular vote.
It is now clear that polling (both nationally and especially in the upper Midwest) was close enough that we should have spent more time exploring and articulating alternate outcomes rather than arguing for the data and logic that supported the higher probability outcome. But at the same time, we should recognize that when a lower probability outcome becomes the result, this isn’t necessarily a failure of polling or modeling. It might just be a lower probability outcome.
Inside straights happen, and Donald Trump just drew one.