Organisations are increasingly using data for better insight and process optimisation. It is important to realise that the volumes and sources can be very diverse. After all, data is generated and stored in a variety of technologies and systems. To make this data useful, 'data pipelines' are developed - enabling you to systematically and reliably unlock sources. By Koen, data engineer at Datalab
By 'data pipeline' we mean a collection of steps - or tasks - that are executed in a logical order, resulting in a usable dataset. This dataset can then be deployed for various purposes such as analytics and machine learning. Developing and managing pipelines is, in practice, quite challenging. This is where Apache Airflow comes in. Airflow is an 'orchestrator tool': a utility that drives and connects different systems. You can compare Airflow to a spider in a web - it captures different systems and connects them through pipelines.
Airflow was developed internally by Airbnb but was released as an open-source project in 2015 and is today maintained and improved by developers worldwide. Airflow was primarily designed for building, managing and monitoring pipelines. It therefore includes a scheduler that activates all pipelines at the right moments and a comprehensive graphical user interface for precise monitoring and management. Airflow itself is written in Python, and building pipelines is also done using this programming language. This offers tremendous flexibility in defining the various steps, as you can write them yourself using relatively simple code.
Airflow was developed because the company wanted more control over the numerous batch processing* flows within the organisation.
*) Batch processing means processing batches of data at fixed intervals.
For this reason, Datalab has embraced it as one of our most important tools. We also use it for our ETL processes - Extract, Transform and Load. In a standard ETL process, we unlock one or more data sources, transform and validate the data, and store it in a data lake, database, or both. This way, we bring data from different sources together in one place for our clients in a format suitable for their data products.
Airflow is not limited to ETL processes. You can use it for many other purposes: periodically training and keeping machine learning models up to date, exchanging information between systems/applications such as structurally updating a CRM system with accounting software, or managing secondary business processes.