Is Your Online Data Quality Strategy Sustainable?… Continuing the Conversation
by Brad Franz
Corporate researchers and insights users are consistently juggling the impact of budgets, timing, and access to quality respondents as they design research initiatives. Added to these challenges is ensuring the quality of insights that companies depend on to grow.
To help shed light on these important topics and offer actionable, tangible solutions to short-term pressures and long-term challenges, myself, Jason Thomas of Symmetric, and Arianne Larimer of PureSpectrum have met for ongoing conversations around industry challenges. As colleagues for over a decade, we see ourselves as a team of sampling experts and thought leaders from across the industry offering distinct perspectives on how a variety of purchasing considerations can directly impact the quality and reliability of research.
The series of collaborative conversations began last year at the Insights Association’s 2022 Corporate Researchers Conference, when Burke hosted a panel discussion on data quality. Below, we continue our conversation; we share firsthand experiences on how buying considerations can impact data collection deliverables, and explore considerations and offer recommendations regarding the importance of online data collection design and its impact on data quality.
What is one thing that can be done to improve quality and make sample more sustainable for the future?
Recognize the Nuances of Quality
BRAD: As researchers, we care so much about the objectives and outcomes during the design phase that we can often lose sight of the human being on the other end of the survey. Quality means more than how many “dupes” or “frauds” are flagged and removed before the survey starts. How respondents interact and engage with survey opportunities has a direct line to how attentive they are and how reliable their data will be.
Understand the Business of Sample
JASON: We need to better understand and recognize the business model behind panel builds and respondent sourcing – including administrative costs, respondent incentives, and margins. Panel companies have had to adapt and reposition their business models to account for competitive prices and higher demand. High-quality sources require larger investments, and those are reflected in the CPI. Price does not guarantee quality, but sample buyers and insights users need to better understand the impact data collection costs have on the supply and the respondent’s experience.
Visibility in a Cloudy Data Collection Industry
ARIANNE: Technological advancements, specifically the marketplace model, have enabled buyers to interact with multiple providers in one ecosystem. Sustainable data quality is not just about protecting against fraud; it also includes mitigating potential source bias that can occur when working with only traditional panels or single sources. We can improve sample quality and sustainability by talking transparently about why visibility of source composition and pricing structures matters and using this knowledge to then enhance buyer decision-making power.
Given the pressures of faster, cheaper, better, what impact has modern data collection had on source transparency, and what are the quality concerns?
Understand How the Sausage is Made
JASON: Data collection providers are not only in competition with each other, but they also work together… and are dependent upon each other to help support modern demand. Reselling is extremely common. Unless you understand who is sourcing your projects, it can be challenging to truly understand the impact of sources on quality. Blending sourcing is a great approach to addressing this bias, but you need to know what is proprietary and what’s not. To me, proprietary means I build it, I recruit it, I maintain it. I own the relationship.
The Word “Proprietary” Has Lost All Meaning
BRAD: That’s a bit dramatic, but some providers will resell sample and repackage it as proprietary just because they’ve databased and stored some distinguishing metrics on a specific respondent. “Proprietary” should mean a preexisting relationship between the provider and member, one where the member has opted in to be part of that community. Be open and build trust with providers. Make sure you understand how they define proprietary and how that meets your expectations. Accountability works both ways. Understanding the source and its consistency over time means differentiating between sample noise and real market changes.
You Need to Decide How Much You Care
ARIANNE: Similar to the food industry or fast fashion, there are choices to be made regarding sustainability and its impact on the industry as a whole. Marketplaces built with transparency in mind provide choice and accountability. It can be easy to choose the path of fast and cheap, and there are times when that is exactly what research calls for. But, there are times when a collection strategy requires a different approach, and an understanding of a respondent’s path from their living room to the survey is critical to the reliability of the data. Or maybe it’s a mix. Understanding the choice and the impact is why it matters so much.
How do the sourcing considerations impact inclusivity and accessibility?
Accessing Fresh Respondents
ARIANNE: Combining traditional panel resources and online publishers can provide more opportunities for respondents who would historically be left out of survey opportunities. Creating blends and filters on the front end can increase access to broader audiences. It’s also critically important to look at survey design. Diversity and inclusion require considerations around how you write survey questions, so respondents feel included. Are you asking and framing questions in an inclusive way? Are you asking questions in a way that enhances the respondent experience?
Right Respondent, Right Opportunity
BRAD: The key to understanding inclusivity and accessibility requires knowledge and control of how respondents are being sourced for your study and how the study itself can impact their performance. It requires an understanding of their motivation. Consider, for example, a respondent who was intercepted from a website offering a survey opportunity as an alternative to a traditional paywall. Their motivation is to return to the original site – the survey is a means to an end, and data reliability could be impacted. This is particularly impactful when considering length of interview and survey complexity.
Cast a Wide Net!
JASON: Consider incorporating a wide range of specialty partners and methodologies into your research plan. Research doesn’t have to live solely online. In addition to online, CATI/phone interviewing or mail research are still valuable tools in capturing and validating respondents. This is especially important when surveying B2B or niche consumer audiences.
From your perspective, what one question should the audience ask of companies like yours as they consider their sampling options?
Focus on Data Quality, and Commit to It
JASON: Talk to your providers about what they are doing on the front end to protect and enhance the data quality. As researchers, it’s traditionally your responsibility to write a good survey and write a good screener to make sure the right respondents are qualifying. Add multiple measures throughout the screener and survey to detect various forms of fraud. Ask multiple response questions and disqualify if too many choices are selected; add in red herring questions. Add open-end questions to every screener and survey as the open end is the most powerful question, while acknowledging it must remain easily reviewable. We need to work together (client, research company and sample provider) to properly battle this.
Sampling Techniques x Respondent Experience = Data Quality
BRAD: Quality is everyone’s responsibility, not just the sample providers. Leveraging best practice sampling techniques while understanding how respondents engage is critical to understanding quality at a holistic level. The respondent experience is too often overlooked when determining the level quality and reliability in our research.
Learn from Today’s Conversation
ARIANNE: The questions and topics discussed on this panel are an excellent starting point for any researcher who cares about data quality and its long-term sustainability. Use these topics and this information as conversation starters with your research agencies and sample providers. A better understanding of supply and quality approaches will help shape your collection strategy.
The conversation will continue in the Spring as this group leads a roundtable discussion on Creating a Sustainable Online Data Quality Strategy at SampleCon, the research industry’s only conference solely dedicated to data collection and sampling. Stay tuned to BeyondMeasure for future conversations as Burke continues to champion efforts around data collection quality and adapting to industry trends and challenges.
“The Sample Buyer”
Brad is the Director of Online Quality and Sourcing at Burke, Inc. He works directly with Burke’s clients and a network of online providers to create and maintain best practices for online data collection.
“The Sample Marketplace”
Arianne is a VP at PureSpectrum. She leads a sales and account management team that provides innovative sourcing options to her clients.
“The Sample Provider”
Jason is the CEO of Symmetric, where he oversees the creation and management of proprietary online panels and communities.
Interested in reading more? Check out Brad’s other articles:
SampleCon 2022 | Enhancing the Respondent Experience
Become a Smarter Sample Buyer: A Retrospect 5 Years Later.
Become a Smarter Sample Buyer – Essential Questions to Ask Your Providers
As always, you can follow Burke, Inc. on our LinkedIn, Twitter, Facebook and Instagram pages.
Source: Feature Image – ©Rawpixel.com – stock.adobe.com
BEYONDMEASURE POSTS:
