Unit testing dbt fashions has all the time been probably the most vital lacking items of the dbt ecosystem. This text proposes a brand new unit testing method that depends on requirements and dbt finest practices
Ever since dbt launched software program engineering finest practices to the realm of knowledge engineering, its functionalities and the ecosystem round it have stored increasing to cowl but extra areas of the information transformation house.
Nevertheless, one important piece of the “information engineering with software program engineering finest practices” puzzle stays elusive and an unsolved drawback: unit testing.
Justifying the significance of unit exams, why they’re vital for any line of code earlier than it may be known as “production-ready”, and why they’re totally different from dbt Exams or information high quality exams is one thing that has already been brilliantly tackled and explained. But when we wished to summarize their significance in a one-minute elevator pitch, it’d be the next:
In information engineering there are typically two totally different parts that we wish to take a look at: the information and our code — dbt Exams (and different information high quality techniques/instruments) permit us to check the information, whereas unit exams permit us to check our code.
With the above in thoughts, it’s solely pure that there have been a number of initiatives by the neighborhood to reinforce dbt with an open-source unit testing functionality (like Equal Consultants’ dbt Unit Testing package or GoDataDriven’s dbt-focused Pytest plugin). Nevertheless, these packages stay restricted in functionalities and have a steep studying curve.
This text introduces a unique method that’s a lot easier but extra elegant, counting on requirements and dbt finest practices to implement a scalable and dependable unit testing course of.
Earlier than diving into the method, let’s first outline the extent at which we wish to run our unit exams. The query to reply…