A blog about working as a trainee data engineer at Datalab from an unrelated background.
Update: two months later
Two months ago I wrote a first blog about my development from ecologist to data engineer. Time for an update: how have the past months been for me?
My first assignment: an internal financial dashboard
The underlying data for this dashboard consists of financial transactions from Datalab over the past years. A chart of accounts is essentially a way of dividing transactions into categories. There is quite a lot to choose from when it comes to chart of accounts schemes. To get the right data, with the scheme of our preference, I had to link several tables together with PostgreSQL. That may sound simple but it actually made this assignment quite complex straight away. Also because the column names are very abstract. This makes it quite challenging to maintain an overview in a sea of abstract names and interrelationships.
Building the dashboard would take place in R Shiny. A number of colleagues had already developed similar dashboards so I could 'look over their shoulder'. This brought several complications however. Their assignments were considerably more complex. How do you then filter the code for aspects that are relevant versus those that are not relevant for your assignment? You quickly notice that you first need to fully understand that code before you can determine whether it is relevant. With the result that you spend four hours straight trying to decipher code. You go so deep that you increasingly lose the bigger picture…
Therefore tip #1: don't try to figure everything out first, but only what is strictly necessary. Then get started writing your own code. Along the way it gradually becomes more concrete. And it also becomes clear which things you do understand, and which things are still unclear.
Using other people's code is simply difficult too, especially when that example is many times more complex than what you are trying to create yourself. So I also had to learn to let go of other people's code and start with a clean slate. Which is much easier said than done. Especially when you need to do something complex that you have never done before. The advantage is that the questions you have are easier to answer. Moreover, you can implement small pieces of simple code much more easily than large chunks full of dependencies.
Tip #2: search thoroughly, but don't sit too long with your questions. If you have been struggling for an hour, it really is time to ask a question.
Looking back, I can summarise the first months in the following sentences: 'What does Datalab expect of me and what do I expect of myself?' I often waited too long before asking a question. Of course because I was secretly afraid that it was expected I should know the answer myself. Because I wanted to show that I have perseverance and that I can figure things out independently - but the assignment was often just a bit too complex for that. Then it is nice to hear from your colleagues and manager that struggle and frustration are sometimes just part of it, but that you are on the right track. Fortunately I do notice that through all the obstacles during this period, I have already learned a great deal!
Although the dashboard is not quite finished yet, I am now allowed to continue with a new assignment. For this assignment I am unlocking data from the Google Analytics API with Python. Then I build a DAG in Apache Airflow so that the data is unlocked daily and stored directly in the client's database. I am pleased to have some variety again and notice that this assignment connects better to things I already know.
I will regularly blog about my experiences as a trainee at Datalab. In this way I want to involve my readership in my work as a data engineer at Datalab.