Often seen as an afterthought, but actually essential for every data project: maintaining a data catalogue. In a data catalogue, you keep metadata about all the data you use (or want to use). Think of the meaning of a table, field or dataset, but also the sensitivity of the information or examples of queries. Another important aspect to track is lineage: the origin and usage of a table or dataset. Why this is so important, you will read in this article.
# 1: Discoverability
The most obvious reason is discoverability. A good data catalogue has a strong search function. And that search function must be able to handle queries in natural language. After all, not everyone uses the same terms or words.
Important tip: test the search function before purchasing data catalogue software.
# 2: Reusability
In the catalogue, you not only track names, definitions and privacy aspects, but also examples of queries and use cases in which you use the data. This helps you and your colleagues avoid reinventing the wheel time and again. Moreover, more experienced colleagues can store higher-quality queries, so that junior analysts can learn from them.
Tip 2: ensure you save query examples and provide as much explanation as possible. Consider questions like: Why do you join tables in a particular way? Which filters do you apply? How do you transform the data?
# 3: Prevent breaking your own and colleagues' work
A data catalogue without lineage is like a car without fuel: worthless. So make sure you track lineage properly - preferably automatically, of course.
But what exactly is lineage? In short: the origin of datasets. A dataset does not simply appear out of thin air. By recording where datasets come from - in the case of linked datasets - you can always trace back to the original. You also keep track of any intermediate transformations (think of joins with other datasets, cleansing, aggregation and so on). This gives you a complete overview of provenance.
…but it does not stop there. Also ensure you properly register the actual usage of your datasets. In which dashboards, for which models? Who uses the data? This prevents a colleague turning up at your desk angry because you broke their dashboard (because you thought 'datum' looked nicer than 'date' and renamed it - or because you suddenly appended the time to the date and the model could not handle it).
Tip 3: without lineage, a data catalogue is useless. If you must choose between a data catalogue or a lineage catalogue, the latter is more useful.
# 4: Ready for the (AI) future
AI models, such as ChatGPT, are here to stay and they are becoming more capable at an almost frightening pace. Many analysts and programmers use chat models daily to speed up their work or solve problems. An AI model is certainly not omniscient: it does not know what you have stored in your data warehouse. You can provide database schemas and this certainly helps, but how can an AI model warn you if you are using sensitive data in a query? Or how does it make connections between datasets that are not explicitly linked?
Enter: the data catalogue. Especially if that catalogue has built-in AI features so you can find information even more easily and have queries generated.
Want to see how we approach this at Datalab? Watch the video below of our internal chatbot, Henk, helping a (fictitious) healthcare institution find data.