Which comes first, good data or data literacy? I think it’s a chicken or the egg kind of problem. 

To build a good data culture you need to build a flywheel: Good data leads to exciting insights, which leads to people becoming data curious and eventually more data literate, where they understand the difference between reliable and unreliable data. In turn, teams that are data curious and data literate want good data, so they invest in making sure their data is good. Data curious teams are willing and excited to measure the success of their product releases, build and maintain the tracking plan, implement proper analytics events, and review the insights that help them make their next decision.

One of my favorite data leaders is Maura Church, Director of Data Science at Patreon. It didn’t take long for us to discover how many shared beliefs we have, so, naturally, she was one of the first people I spoke with on The Right Track. It’s long overdue that I synthesize some of the insights from my conversation with her into a blog post. Read on for her takeaways about why people don’t trust data and how to overcome that knee-jerk reaction and become a more data-literate organization.

Why People Don’t Trust Data

One of the most significant barriers to data literacy is that people mistrust the data. In our chat, Maura broke down a few reasons for why people don’t trust data, so we could dive into some of the things we can do to help the organization to overcome it:

1. People don't know what normal looks like for the business

It isn’t easy to interpret and trust data if you don’t know what normal looks like. The retention charts and conversion rates you use as a guiding light for your business are for example often cohort-specific, and are meant as input into a specific business strategy. As a result, they may look quite different from what you’d expect. If you’re new to the business, seeing something like that can easily make you doubt the data. This challenge is one of the reasons Maura personally onboards every new team member at Patreon with a data walkthrough (huge kudos and shout out for that; more on that below).

How to fix it:

Onboard every new team member into your company’s key metrics. This will kickstart every team member’s journey into seeing data – and yourself – as an approachable ally.

2. The data isn't what they want

It’s easy to dismiss data when it looks different from what we expected. The term confirmation bias describes the human tendency to seek out and interpret information in a way that confirms what you already believe to be accurate. This bias works inversely, too: if your data presents a truth different from what you think (or what you want to be true), you might be inclined to doubt the data. I’m guilty of this myself.

How to fix it:

A great way to help people understand the importance of healthy scrutiny and how our biases work can be to do a session where people guess the impact of an A/B test before you reveal it. Listen to examples from Gustaf Alströmer at Airbnb and Nick Threapleton at CultureAmp.

3. Lack of trust between the data analyst and the data consumer

If people don’t feel they can trust and approach the analyst, they won’t engage in a meaningful dialog about data findings, particularly if the data is different from what they expected. So whether you’re a data leader, data product manager, a data scientist, or a data analyst, invest in building relationships of trust with your internal customers.

How to fix it:

See it as part of your role as a data practitioner to build trust and relationships. Get excited when people ask questions. Embrace confusion and curiosity. Offer help. Be friendly.

4. Data quality is bad

It’s a significant project to get your data to the place where even you, the expert, will trust it – let alone someone who doesn’t know the ins and outs of it. Lack of data quality is a twofold problem: Lack of data reliability, and lack of data relevance. Both need to be addressed with both culture and tools.

How to fix it:

Whatever you do, don’t try to fix all your data at once. Start small. Start with one metric and two to three analytics events. Start with a single product team. Use that success story to get more of your team curious about data and interested in learning how it affects the products they’re building. That’s how you get more people excited to use data and produce better data. Remember, data literacy and data quality is a team effort. 

How to Increase Data Literacy and Data Trust

I always advocate for processes like the Purpose Meeting to bring stakeholders together and align on data needs before you generate the data. This increases your team’s data literacy and the probability of producing relevant raw data that you can turn into actionable insights. Use the output of the Purpose Meeting to maintain a tracking plan – be it a spreadsheet, JSON schema, or the Avo tracking plan. Agreeing on what to implement – and why – increases the probability that the data will be well implemented. 

As your organization matures, you’ll find the growing need to add some data governance, including to automate data validation for your analytics events. You can build validation on a JSON schema and add the analytics event quality control to the release QA process, or adopt tools for tracking observability and data validations like Avo’s Inspector, Avo Functions, and Avo’s in-app debugger. Without effective solutions for your tracking plan, analytics implementation in code, and data governance, it’s simply too easy to mess up your data.

Maura shared a few of the tactics and best practices she and her team at Patreon apply to generally increase trust in data:

Focus on key metrics to get right

Identify which of your metrics are the most essential for your business and have the most significant impact on the company if the data behind those metrics is flawed. Maura advises focusing on those key metrics that you need to get 100% right, 100% of the time, and start there. Make sure those systems are well-documented, reliable, and robust and have alerting and error reporting. Then branch out to the other areas of the product.

Here be dragons

Your data will never be perfect. While you want a couple of key metrics that are “100% right, 100% of the time”, you will always have messy data. You won’t get everything 100% right. One of the things Maura and the Patreon data team do to build trust is they make sure people know when there will be dragons in data sets. Doing this will create the feeling among your co-workers that they’re working with a data science team that “knows where the potholes are”, which in turn increases trust in the good data that you should be able to trust.

Multiple angles on data are useful, but align on the main definition

Businesses evolve, and so do your metric definitions and so does your tracking plan. Along the way, you’ll create a lot of charts and dashboards that grow outdated. You might even end up with twenty different retention definitions. Maura provided a great example of one of Patreon’s key metrics, Total Membership Volume (TMV), of which they have all sorts of flavors; TMV for podcastors, TMV retention, TMV from creators in Germany. These different flavors correspond to different ways to understand the business. And it’s helpful to look at alternative angles, but it’s important for your team to know the primary definition. As the aphorism goes, “All models are wrong, but some are useful.” 

One of my favorite nuggets from the episode was Maura sharing a question to be expected from savvy employees:

“Okay, these are our key performance indicators today as a business, but what did they use to be? What did we grow out of? And what does that tell us about where Patreon has come as a business over the past seven years?”

Have regular cleanups and Audits

As a follow-up to the above, it’s crucial to build a single source of truth for what metrics and data each team should be using. As Maura said, if you open up a Revenue Reporting folder and find seven different revenue metrics for a single month, that’s how people lose confidence. 

How they combat it: They put in the work to maintain a source of truth for what each team should be looking at. Including regular cleanups:

Cleanups and audits are not always fun, but they are crucial to keeping your data trust up to par. Maura suggests spending a few hours on monthly, quarterly, or biannual cleanups — with some pizza and soda — to identify which parts of the data are no longer relevant and make changes accordingly.

Maintaining quality data and building a culture of data literacy in your company is a team effort. It doesn’t scale to have that being managed fully by the data team. But as a data team, you should lead by example, and then work with the “tripod”—the PM, the engineering manager, and the designer—so they gain ownership of this empowering source of truth for their team.

—

You’ve just read through part 1 in the unpacking of the incredible advice and insights Maura had to offer. Stay tuned for part 2, about how to structure your organization for data literacy and how to make data more accessible to more people in your organization. In the meantime, listen to Maura Church's podcast episode for more insights on how Patreon builds a data literate organization. And if you’d like to learn how we’ve contributed to maintaining Patreon’s data quality, try Avo today, schedule office hours with us and join our community.‍