A blog about working as a trainee data engineer at Datalab from an unrelated background.
Update after temporary contract ends
It catches me a bit by surprise: suddenly it is the end of August. This means the past seven months have flown by, and my temporary contract at Datalab has ended. A good moment to look back at what I have learned over the past months.
Unlocking APIs and building DAGs
In my previous blog I wrote about struggling with other people's code and maintaining overview with complex problems. Besides building a financial dashboard, I was also allowed to unlock my first API - namely Google Analytics.
By now I have come a good deal further. For our retail client by-bar I have unlocked an entire series of APIs, namely: Google Analytics, Google Ads, Google Search Console, Google Spreadsheets, KNMI, CBS, Bing, Pinterest and Meta. These APIs all work in a similar way. First you must create credentials that have the correct permissions to retrieve a company's data via the API. Then you use these credentials in a Directed Acyclic Graph - a step-by-step plan that divides a large task into smaller tasks. You need to take quite a few things into account, such as:
- What data exactly do you want to retrieve?
- How do you request this data from the API?
- How is the data then delivered by the API?
- Does the data still need to be processed before you can store it?
- Where do you then store the data?
- In what structure do you store the data?
- How often must the DAG retrieve data?
- How do you deal with unexpected results from the API or failed tasks?
- How do you ensure that incomplete data is supplemented later?
During this process I can always turn to my colleagues Aniek and Koen. However, some data sources are new to them as well. Then I really do need to trawl through the documentation myself to learn how I can unlock the correct data. This process can - with a more complex source such as Meta - really take quite some time. Partly because along the way I keep discovering metrics that are also worth unlocking. I write the code mostly independently; occasionally I receive a comment that helps me reuse code better and keep things organised. All in all an enjoyable and intensive puzzle, but naturally not without the necessary frustrations.
Dashboards
In R Shiny I am now building fairly complex dashboards. Compared to my previous blog I have made substantial progress. Gradually, more mental space is emerging to concentrate on writing good code, rather than merely writing code that does not catch fire when I turn my head. Debugging my code is getting faster and I work more purposefully. It is increasingly less 'just test it, maybe it will work by itself' and increasingly more 'how am I going to set this up so that it works'. I also notice that the moments I get stuck now occur increasingly later in a project.
It is great fun to occasionally stand still and appreciate the wealth of knowledge you have gained. Recently I had to puzzle for an hour to transform a dataframe in the right way, and when it finally succeeded, I realised that a year ago this would have been too complex altogether. The more I discover about data engineering, the more conscious I become that I actually still know very little. Fortunately, Jeroen and Harmen indicate they are satisfied with my progress, and with a permanent contract I get to learn new things for quite a while yet!
I will regularly blog about my experiences as a trainee at Datalab. In this way I want to involve my readership in my work as a data engineer at Datalab.