Working with data also means thinking about how to manage it. Broadly speaking, there are two choices: a data lake or a data warehouse. In this blog, we explain what these terms mean, what the differences are, and which type suits your organisation.
The benefit of central data storage
Business information from various sources such as Exact Online, a procurement system, Google Analytics from the webshop and so on, gains added value when data is combined and then thoroughly analysed. By managing data centrally, data can be combined so that new information emerges. For example, comparing revenue figures against click behaviour of visitors in the webshop. There are two methods for managing data centrally: using a data lake or a data warehouse.
Data lake: how it works and its advantages and disadvantages
A data lake is simply a central location where data is stored in a largely unstructured manner. Related chunks of information are stored unprocessed in containers - for example, a copy of all records from a procurement system, issued quotations, 'raw' film materials, or various Word documents. There are multiple data lake providers such as Snowflake, Amazon Data Lake and Azure Blob Storage. Each provider has its own capabilities. Azure Blob Storage, for instance, offers not only cloud storage but also the ability to search and analyse data using the Data Lake Analytics application. In this way, data can be used for analytical purposes.
Advantages of data lake systems include the pricing, the speed and the ease with which a data lake can be set up. Moreover, there is no need to make selections in advance because all data is retained. However, therein lies the risk that a data lake becomes a data swamp - through the large and unstructured volume of data. Try finding your way through that. And importantly, security policy is difficult to arrange because data is stored in an unstructured manner, making it impossible to regulate access per level - it is all or nothing. The analyst therefore also becomes responsible for structuring, which is not always a role that fits well.
Data warehouse: methodology and its strengths
A data warehouse closely resembles a data lake but with data - under the direction of a data engineer - stored in a structured manner. You can choose between a centralisation model and a hub model. With the centralisation model, local copies are made of only the relevant information. This model fits well with the logic of ETL: Extract, Transform and Load. In other words: extracting from source files, transforming into a fixed structure, and storing in a data warehouse. With a hub model, the information is not stored; instead, various data formats are translated into one uniform structure using software. The data remains in the source files.
The greatest advantage of a data warehouse is its structured setup, which enables security policy to be arranged down to row-and-column-level permissions. Moreover, data analysts can work more easily when data has already been structured to a certain degree. Structuring is generally done by a data engineer who must also be able to create data models. In short, a data warehouse requires specific skills and takes more time to set up.
Which suits your organisation?
Only a few organisations have sufficient use for just a data lake - for example, media companies with large volumes of raw footage. For the vast majority of organisations, a data warehouse is preferred. Not only because data is stored in a structured manner and security policy can be arranged in greater detail, but also because analysts can then primarily focus on performing in-depth analyses. When a data lake is combined with a data warehouse, it offers even more advantages. The data lake then serves as the primary source for the data warehouse. Moreover, the data lake functions as a secured backup of all data, since no selections are made in advance for a data lake. Access is limited to an engineer. Data analysts use the data warehouse for analyses.
The greatest advantage of the data lake/data warehouse combination is that a single version of the truth emerges: all information is stored once in a properly structured manner in the data warehouse.
It is important to think carefully about what suits your organisation. A good understanding of the underlying techniques is then required. Datalab has extensive experience and more than sufficient knowledge to guide your organisation towards a structured way of working with data. Do not hesitate to contact us - we are happy to inform you about our offering without obligation.