Chad Sanderson’s Post

The vast majority of data in Snowflake/Databricks/BigQuery is barely used. Stop wasting time thinking about governance, data quality, and ownership of data that no one cares about. The amount of data that IS useful follows a Pareto distribution: Not only is it important, but it is exponentially more important. This type of data is used everywhere from accounting pipelines to machine learning models. An example of this at Subway was franchisee data: Store locations, foot traffic, opening hours, and sales data taken from POS systems were a part of most core dashboards, models, and shareholder reports. That meant the data coming from restaurants should have been pristine, up-to-date, and enforced with rigorous quality. This...was not always the case. If data teams want to make an outsized impact, start by driving governance and quality for the pipelines that have tangible value to the business. Too often platform teams take a horizontal approach but end up serving all their customers poorly, instead of knocking a few use cases out of the park, generating goodwill, and expanding to the long tail of use cases over time. Data Engineers: If you want to get ahead in your career - SHRINK your surface area, don't grow it. It might seem counter-intuitive, but hyper-focusing on fewer valuable things is far better for your career than doing more work that is less impactful. Good luck!

Some people believe that Gen AI will automate the work of data analysts. That might be true. However, it's only going to happen when the work of data analysts doesn't involve figuring out which data is trustworthy, whether the semantic meaning is clear and accurate, why the same column is present in 4 different databases each with different numbers, what it means semantically, how data has changed over time and why, wading through all the gotchas and layers of filters in SQL in legacy data systems some of which are coming from 3rd party services, going back and forth with engineering because there's no documentation, figuring out to do when the data changes or events are dropped, then wrapping all that context in a pretty bow and communicating it to stakeholders. Something I've heard before from Gen AI advocates is: "If an LLM had all the context on what the data means, then they could do this work!" Yes...but where does that all that information come from? Who is consistently maintaining the semantic meaning of the data? How does this happen when teams have dozens of databases and microservices, 100s of events, and thousands of tables in analytical databases? Simply getting this data into a reasonable state for LLMs would be a multi-year effort, and even then, it would be immediately out of date. It would require wrangling business teams, engineers, and analysts and changing the culture so that everyone participated in constant data documentation, data management, and ownership. Writing SQL and creating charts is the easiest 10% of an analyst's job. The other 90% is a thankless grind chopping through an ever-growing jungle of data debt. Unless that problem is solved, anything AI can do is just put lipstick on a pig. Good luck

I'm excited to announce that I'll teach my first-ever live course: "Data Infrastructure for AI: Building a Next-Generation Data Platform." Coming this spring, the course will focus on the ways data engineering, data platform, and data science teams can prepare their data infrastructure for AI at scale: - Proactively resolving data debt - Choosing your modern data infrastructure stack - Driving producer-level ownership and accountability - Scaling AI Data Products - Tracking and Maintaining AI Model Quality - OKRs for Federated Data Governance & Management - Implementing Data Contracts - and more... You can register for the course here: https://lnkd.in/dhknYk6w If you're interested in the live training, please sign up above and provide the most ideal class hours for you. That will help me decide on times and dates which are still TBD at the moment. Thank you, and looking forward to seeing everyone live!

Data Contracts are data agreements that are enforceable programmatically. They are an application of governance-as-code. Here's an example below! There are 3 critical aspects to a data contract that teams can enforce in a straightforward preventative manner: 1. Schema: Column names, data types & variables 2. Business Logic: NULL values, expected data ranges, etc 3. SLAs: Frequency of data refreshes Where each component of the contract is enforced depends on the complexity of the solution that has been implemented and the technology stack of the organization. What's important is that each constraint in the contract can be at minimal alerting upon, and ideally enforced programmatically. Data can be checked and measured against the contract upon arrival, or in the best case scenario contracts can be validated in CI/CD as integration tests, preventing data generating code assets from ever being deployed into production. Here are a few of the essential additional parameters of a contract: ID: Each contract should have a unique identifier so that it can be referenced by tests in the CI/CD pipeline or elsewhere Resource Name: Contracts should be 1-to-1 with data resources. I prefer to start implementation on upstream source data sets like databases, protobuf/avro files, JSON schemas, or unstructured event schemas. Anything with a schema can be covered by a contract! The spec version: All contracts should be version-controlled and ultimately stored in Git. If any consumers wish to modify their contract, it can be done through a PR. This prevents many contracts from being applied to the same data resource. Name: The semantic object the data contract describes. In the example below, we have a contract on the vehicle status of each bus in Seattle's bus fleet. Namespace: The data domain. The combination of Name and Namespace prevents duplicate contracts from being created, and each can be categorized/stored according to their Namespace. Documentation: In my opinion, documentation should be a required field for all data contracts. Consumers should not get the benefit of data protection, without putting in the work to describe what data they expect and how it's being used. (Also, this allows semantic level checks to be surfaced qualitatively) Owner: The owner is responsible for the contract. Contracts should be initially defined by data consumers who have some expectation of their data, but ultimately should be owned by data producers once the cost of managing many downstream dependencies becomes significant. Once the foundations are in place, contracts can be extended to include PII, compliance, and other forms of data governance. These checks should be done as upstream/left as possible, ideally informing data producers when violations are about to occur. Data Contracts are absolutely ESSENTIAL to scale data governance and data quality to the software engineering organization. Good luck!

Very excited to announce that I'm rolling out a new email series on Data Contracts and Applying Data Quality at scale! The email series will arrive in your inbox every two weeks, written by myself and 🎯 Mark Freeman II. Here is a quick overview of the first four articles in the series if you sign up: 📧 Part 1/4 - Why the data industry needs data contracts today 📧 Part 2/4 - What is the data contract workflow and its components 📧 Part 3/4 - Where do data contracts fit within the data lifecycle 📧 Part 4/4 - How to onboard your organization onto data contracts Throughout the rest of the series we'll get into use cases and product examples of data contracts in action, companies and data leaders learning from data contract implementations, and data we've collected on the maturity cuves of rolling out data management across the data supply chain. If you're interested, simply go to gable[.]ai and enter your email where it says "join our product waitlist" and you will automatically start receiving these blogs. If there are any other optics you'd be interested in me covering during the series please let me know!

The first two chapters of my book "Data Contracts: Developing Production-Grade Data Pipelines at Scale" are available for early release! - Chapter 1: Why The Industry Needs Data Contracts - Chapter 2: Data Quality Isn’t About Pristine Data 🎯 Mark Freeman II and I have been working hard on the title which will be the definitive 350-page guide on the what, why, and how of data contracts including implementation examples, where they fit within the Modern (and not so modern) Data Stack, and the future world they unlock for Heads of Data, Data Producers, and Data Consumers. Right now, you can download the book's first two chapters through the O'Reilly website. If you are interested in being an early technical reviewer, please comment below and I'd love to loop you in once we're ready for feedback. Thank you for all the support so far!

A letter from an OpenAI employee confirms what we already knew: Data is the only competitive advantage in AI. “…trained on the same dataset for long enough, pretty much every model with enough weights and training time converges to the same point…Model behavior is not determined by architecture, hyperparameters or model choices. It’s determined by your dataset, nothing else.” The only advantage companies have is their data and its quality. This means that quality, governance, and compliance is going to become most essential at the source! Hang in there folks - I think we are on the verge on a change in how data infrastructure is perceived. Good luck!

Some companies think that implementing an analytical database, ELT tool, and BI product means they are 'done' with data. This is kind of like saying that roads and cars exist - but no maps, speed limits, seatbelts, headlights, or gas stations. It's like saying that raw materials exist, but the meat is shipped at room temperature, 50% of the people who eat the seafood get sick, and there are no standards for compliance or health. It would be like saying that chefs CAN cook their food and make it available to sell, but all of it is laid out in the middle road covered in mud. There are no restaurants, no prices, the menu doesn't make sense and is always out of date, the customers are always complaining, and nobody cares. Storage and ETL are the BEGINNING of the journey to data products. It's not the end. Teams should understand that to truly drive value from an infrastructure, they need to be in for the long haul and not a quick buck. Good luck

When I was an analyst, I blamed data engineers for low-quality data. When I led data engineering teams, I blamed software engineers. When I was in charge of the whole system, I realized data quality was everyone's responsibility. No one person or group is 'in charge' of data quality. We all have a part to play. Helping teams collaborate with each other, generating awareness of how data is used, and providing infrastructure that facilitates change management is 10x more valuable than trying to point fingers for failures in communication. Good luck!

The software model of highly decoupled services doesn’t work for data. Data requires a supply-chain oriented approach where producers are aware of their dependencies, and consumers provide active feedback on their expectations and use cases. Imagine the chaos in a supply chain if a supplier of beef simply dumped raw meat into a warehouse for anyone to consume regardless of its quality. The meat would have very different use cases- a quick service restaurant might turn the beef into a patty and use it in hamburgers, while another might make dried dog treats. Each use case has a radically different set of compliance requirements, governance models, and so on. In the decoupled model, the restaurant owners would be applying quality constraints at the point of retail. This would be extremely tedious and time consuming. Every waking moment would be spent validating that the meat received met expectations because the consequences of getting it wrong are severe (people getting sick, lawsuits, store closures) Sound familiar? This is exactly what data scientists and data engineers do today. Reports show that over 70% of a data developers time is spent on validation, meaning only a fraction of data teams most valuable resources are focused on producing ROI for the company. Applying quality close to the consumer alone has exponentially mcreasing costs over time and cannot scale. If AI is truly the future, then data producers must become an active participant in the data supply chain and not merely a decoupled data provider. Good luck!