Why do you need data reliability engineers?

I was asked once after a meetup how data reliability engineers (DREs) differ from site reliability engineers (SREs). To non-data engineers, this is a hard question. At the time all I said was, “Listen to this podcast because SLOs for data are super important.”After a couple of months of studying and working as a DRE, I have a much better answer. The book “Foundations for Architecting Data Solutions” highlights three kinds of data projects:

• Datastores for customer-facing applications (Software Data stack)
• Data infrastructure/platforms (Enterprise Data stack)
• Data pipelines for analytics and machine learning (Modern Data stack)

Each of these projects requires a specific type of data expertise to build, manage, monitor, and scale. We do not expect a specific engineer to have expertise in all these areas, so different roles have evolved to allow engineers to become practitioners for each unique set of needs. Let’s explore the data needs of each project and the types of engineers that satisfy the needs.

Software Data

In the first kind of data project, data stores are used by customer-facing software applications. These are your common web-scale databases(MySQL Postgres) or document stores(MongoDB). They store the data needed for the software application to perform its job. SREs or IT/Operations professionals manage the data needs for standard software projects. SREs are concerned with uptime, availability, scalability, and security around these data systems. SREs have expertise in many DevOps practices that give them a specific skill set needed to manage both the infrastructure and the software needed for production software services and their datastores. SREs have to stay current on technology like service mesh, operators, Infrastructure as code, CICD, etc., that enable them to effectively manage the reliability of the software application needs– like a database.

Enterprise Data

The second example is an enterprise-scale distributed database or data warehouse that often has many shards and replicas that have to be managed and reconciled. This type of project is traditionally managed by Database Administrators (DBAs). These DBAs, sometimes called Database Reliability Engineers, are experts in data storage systems. They can shard, partition, replicate, drop, backup, and perform many more types of database operations to keep Big Datastores up in enterprise settings. DBAs have specialized skill sets that enable them to manage large databases and data warehouses on an enormous scale. Part of the DBA skill set is not just knowledge of the SQL language, but also in the many types of databases, database procedures, and custom database paradigms (MSSQL, Postgres, Oracle, MySQL, etc.).

Modern Data

This leaves us with data pipelines. Modern data needs require the constant moving and manipulating of data to support and empower machine learning and analytics loads in modern companies. What do we do for projects where data is not stored but is constantly on the move: being transformed, being inferenced, or being analyzed? This is where data reliability engineers (DRE) come in.

DREs have specialized skills, not just in data ingress and egress, but also in data transformations, data storage, data management, data governance, and data analysis. Engineering of data and pipeline for machine learning, analytics, and reporting is currently undergoing a major shift toward automation, monitoring, and engineering. DREs are not just munging data around, but are actively putting tools in the hands of data scientists and analysts to manage their own models, statistics, and analyses to deliver rapid new customers and executives. They additionally have to understand the contracts to the customers by managing uptime, latency, and availability of their data. DREs have to stay up to date on current and ever-changing pipeline technologies and paradigms, DAGengines, data mesh, stream engines, and messaging platforms.

Modern organizations have incredibly diverse data needs. They need engineers with all types of specialized skills. Don’t play catch-up with your data organizations. As data products and services become more central to the success of a company, DREs will become more essential to the stability of the data projects.