The Role of Custom Tooling in Data Engineering: Leveraging Build Versus Buy Strategies

In the realm of Data Engineering, the trend toward out-of-the-box tools over building in-house software tools has always puzzled me. It is never a cost conversation like in software engineering (will it cost more to build and maintain our service or purchase a tool) but instead adopting out-of-the-box solutions seems like the only option unless you are at Google size. This blog delves into the strategic considerations behind building custom tools versus opting for pre-existing solutions.

The Imperative of Custom Tooling

Data engineering mandates robust solutions that are also adaptable to evolving requirements. But instead of building small right-size software solutions to meet the needs of newer or small data organizations, we look for tools that can help us when we hit the “Big Data scale”. But often instead of the right-sized software solutions, we choose a large tool like Apache Spark. Why? I think either a lack of software engineering experience or a lack of diverse best practices for small and medium-sized Data organizations vs. Big Data organizations. Right-sized software tools crafted using lower-level languages like Go, Rust, or even Java offer the potential to solve problems quickly. This enables speed and adaptability while exploring new data use cases or products

Custom, right-sized software tools can seamlessly integrate with various data sources, APIs pub/subs, and data stores…