Why you need more than surveys
Talking to the people you serve provides invaluable insights. You need additional data to fully understand how well you’re achieving your outcomes.
When you need something, where do you turn? If you’re like me, you either go to people you know for a recommendation, or you go to the computer…where you can get recommendations from thousands of strangers.
I’d argue that in some ways, the proliferation of online reviews has undercut our willingness to simply stop at the first restaurant we see and give it a try for ourselves, but that’s a post for another time.
The reality is that we increasingly rely on the digital word of others to make decisions about where we invest our resources. As a result, business leaders seek out any form of testimonial, review, or love note they can paste on the internet, in as many places as they have the glue.
Unfortunately, unscrupulous businesses are willing to manipulate the system to ensure that five-star reviews light up their products. Third-party sellers have been gaming Amazon’s reviews section for years by sending free things to real people in exchange for glowing write-ups — even if that luscious pillow case falls apart after the second trip through the washing machine.
Amazon continues to try and fight off the fake review machine. The company’s latest actions include lawsuits and something called the “frequently returned item” badge.
In the name of giving more power to customers, Amazon evaluates return rates and applies badges using the same methodology whether the product is sold by Amazon or a third-party vendor (according to the company).
But my point isn’t about reviews. Combining quantitative data with qualitative data made me think about outcomes measurement.
One of the best ways to understand how well you’re achieving your outcomes is to talk to the people you serve. You’ll find the insights you receive through quality surveys and in-depth conversations will help you better understand your people and the value you deliver. When done well, you will know if you’re achieving your outcomes or missing the mark.
Qualitative data can also produce muddled insights. We often hear from the most enthusiastic on either end of the spectrum but not the people in between. Sometimes we simply can’t get responses because we’re all being asked to complete a tremendous number of surveys — from the quality of each and every Starbucks order to reviewing every single app we open. Damn that *please review our app* pop-up.
Poorly worded questions can also produce distorted results. Worse, carefully orchestrated conversations to make the customer uncomfortable saying anything bad and data skewed to sound better in promotional materials make that information useless when it comes to monitoring how well your actions deliver value.
You need to balance your human input loops and the qualitative data they provide with quantitative data, in particular data that illuminates behavior1.
In the case of Amazon, they’re monitoring returns. Their customers read reviews to ensure they don’t have to return the item. While someone may say this is the most amazing air fryer known to man and smash that five-star icon, people actually returning the item remains totally rooted in reality. Most importantly, sellers cannot scam this data point because Amazon handles returns.
Take a look at your own business. Your outcomes define the specific things you need to help the people you serve do or achieve to realize the ultimate value defined by your purpose. Are there actions they take that would give you insight into how well you’re achieving your outcomes?
To give an example, perhaps you’re a publisher. You can (and should) ask your readers what they value and why they value it. You can (and should) ask questions that help you assess if they’re able to do or achieve the things outlined in your outcomes.
You should also monitor how many people actually read what you publish. Moreover, reading 5% of an article is different than reading it all the way through, which is different than engaging with the article, which is different still than going off and sharing it with others. And to go further — depending on your purpose and outcomes — applying what they learned is another thing entirely.
Which data points you choose to focus on will vary depending on your specific outcomes. It will also vary depending on the difficulty (and potentially the ethics) of capturing the data you want to monitor.
In all cases, you want the information you choose to monitor to be as focused and relevant as possible. While it can be tempting to horde data, carefully select which things you monitor and ensure you know the point of doing so.
But don’t stop at the survey.
[1] You can take this to a creepy place. Don’t. Start by looking at innocuous data. Keep data anonymous when you can. Tell the people you serve how you’re using any data you collect. Give them the ability to opt out.
.
As a writer of a young substack, I found this very helpful advice.