1
Introduction to Data Lakes: Definitions and Discussions

As stated by Power [POW 08, POW 14], a new component of information systems is emerging when considering data-driven decision support systems. This is the case because enhancing the value of data requires that information systems contain a new data-driven component, instead of an information-driven component1. This new component is precisely what is called data lake.

In this chapter, we first briefly review existing work on data lakes and then introduce a global architecture for information systems in which data lakes appear as a new additional component, when compared to existing systems.

1.1. Introduction to data lakes

The interest in the emerging concept of data lake is increasing, as shown in Figure 1.1, which depicts the number of times the expression "data lake" has been searched for during the last five years on Google. One of the earliest research works on the topic of data lakes was published in 2015 by Fang [FAN 15].

The term data lake was first introduced in 2010 by James Dixon, a Penthao CTO, in a blog [DIX 10]. In this seminal work, Dixon expected that data lakes would be huge sets of row data, structured or not, which users could access for sampling, mining or analytical purposes.

Figure 1.1. Queries about "data lake" on Google

In 2014, Gartner [GAR 14] considered that the concept of data lake was nothing but a new way of storing data at low cost. However, a few years later, this claim was changed2, based on the fact that data lakes have been considered valuable in many companies [MAR 16a]. Consequently, Gartner now considers that the concept of data lake is like a graal in information management, when it comes to innovating through the value of data.

In the following, we review the industrial and academic literature about data lakes, aiming to better understand the emergence of this concept. Note that this review should not be considered as an exhaustive, state of the art of the topic, due to the recent increase in published papers about data lakes.

1.2. Literature review and discussion

In [FAN 15], which is considered one of the earliest academic papers about data lakes, the author lists the following characteristics:

• - storing data, in their native form, at low cost. Low cost is achieved because (1) data servers are cheap (typically based on the standard X86 technology) and (2) no data transformation, cleaning and preparation is required (thus avoiding very costly steps);
• - storing various types of data, such as blobs, data from relational DBMSs, semi-structured data or multimedia data;
• - transforming the data only on exploitation. This makes it possible to reduce the cost of data modeling and integrating, as done in standard data warehouse design. This feature is known as the schema-on-read approach;
• - requiring specific analysis tools to use the data. This is required because data lakes store row data;
• - allowing for identifying or eliminating data;
• - providing users with information on data provenance, such as the data source, the history of changes or data versioning.

According to Fang [FAN 15], no particular architecture characterizes data lakes and creating a data lake is closely related to the settlement of an Apache Hadoop environment. Moreover, in this same work, the author anticipates the decline of decision-making systems, in favor of data lakes stored in a cloud environment.

As emphasized in [MAD 17], considering data lakes as outlined in [FAN 15] leads to the following four limitations:

1. 1) only Apache Hadoop technology is considered;
2. 2) criteria for preventing the movement of the data are not taken into account;
3. 3) data governance is decoupled from data lakes;
4. 4) data lakes are seen as data warehouse "killers".

In 2016, Bill Inmon published a book on a data lake architecture [INM 16] in which the issue of storing useless or impossible to use data is addressed. More precisely, in this book, Bill Inmon advocates that the data lake architecture should evolve towards information systems, so as to avoid storing only row data, but also "prepared" data, through a process such as ETL (Extract-Transform-Load) that is widely used in data warehouses. We also stress that, in this book, the use of metadata and the specific profile of data lake users (namely that of data scientists) are emphasized. It is proposed the data is organized according to three types, namely analog data, application data and textual data. However, the issue of how to store the data is not addressed.

In [RUS 17], Russom first mentioned the limitations of Apache Hadoop technology as being the only possible environment of data lakes, which explains why Russom's proposal is based on a hybrid technology, i.e. not only on Apache Hadoop technology but also on relational database technology. Therefore, similar to data warehouses, a few years after Fang's proposal [FAN 15], data lakes are now becoming multi-platform and hybrid software components.

The work in [SUR 16] considers the problems of data lineage and traceability before their transformation in the data lake. The authors propose a baseline architecture that can take these features into account in the context of huge volumes of data, and they assess their proposal through a prototype, based on Apache Hadoop tools, such as Hadoop HDFS, Spark and Storm. This architecture is shown in Figure 1.3, from which it can be seen that elements of the IBM architecture (as introduced in [IBM 14] and shown in Figure 1.2) are present.

In [ALR 15], the authors introduced what they call personal data lake, as a means to query and analyze personal data. To this end, the considered option is to store the data in a single place so as to optimize data management and security. This work thus addresses the problem of data confidentiality, a crucial issue with regard to the General Data Protection Regulation3.

Figure 1.2. Baseline architecture of a data lake as proposed by IBM [IBM 14]. For a color version of this figure, see www.iste.co.uk/laurent/data.zip

In [MIL 16], the authors referred to the three Vs cited by Gartner [GAR 11] (Volume, Variety, Velocity), considered the additional V (Veracity) introduced by IBM and proposed three more Vs, namely Variability, Value and Visibility. In this context, the authors of [MIL 16] stated that the data lake should be part of IT systems, and then studied the three standard modes for data acquisition, namely batch pseudo real time, real time (or streaming) and hybrid. However, the same authors did not study the impact of these different modes on the data lake architecture. In this work, a data lake is seen as a data pool, gathering historical data along with new data produced by some pseudo real-time processes, in a single place and without specific schema, as long as data is not queried. A catalog containing data lineage is thus necessary in this context.

The most successful work about data lake architecture, components and positioning is presented in [IBM 14], because the emphasis is on data governance and more specifically on the metadata catalog. In [IBM 14], the authors highlighted, in this respect, that the metadata catalog is a major component of data lakes that prevents them from being transformed into data "swamps". This explains why metadata and their catalog currently motivate important research efforts, some of which are mentioned as follows:

• - in [NOG 18a], the authors presented an approach to data vault (an approach to data modeling for storing historical data coming from different sources) for storing data lake metadata;
• - in [TER 15], the importance of metadata as a key challenge is emphasized. It is then proposed that semantic information obtained from domain ontologies and vocabularies be part of metadata, in addition to traditional data structure descriptions;
• - in [HAI 16], the authors proposed an approach for handling metadata called Constance. This approach focuses on discovering and summarizing structural metadata and their annotation using semantic information;
• - in [ANS 18], the author introduced a semantic profiling approach to data lakes to prevent them from being transformed into "data swamps". To this end, it is shown that the semantic web provides improvements to data usability and the detection of integrated data in a data lake.

Regarding data storage, in [IBM 14], it is argued that the exclusive use of Apache Hadoop is now migrating to hybrid approaches for data storage (in particular using relational or NoSQL techniques, in addition to Apache Hadoop), and also for platforms (considering different servers either locally present or in the cloud). As mentioned earlier, these changes were first noted in [RUS 17].

An attempt to unify these different approaches to data lakes can be found in [MAD 17] as the following definition:

A data lake is a collection of data such that:

• - the data have no fixed schema;
• - all data formats should be possible;
• - the data have not been transformed;
• - the data are conceptually present in one single place, but...