Your Next Gen Data Architecture: Data Lakes

Software Engineering as a field thrives on ever-changing keywords. In the last decade it was "big data" and "noSQL". Every engineer was a big data engineer and every one was working on noSQL. Similarly, now the world is crazy about Data Science and Machine Learning. Half of my connections on LinkedIn have become "data scientists" overnight. Somewhere in between there was another keyword which made a splash (no pun intended!) in the data warehousing world. That keyword was "Data Lakes". In this post, we will talk about what a data lake is, what are the reasons to have a data lake and what are the things you should keep in mind while designing a data lake for your data architecture.

Why Data Lakes?

The two biggest reasons to have a data lake in your DWH architecture are: scalability and support for unstructured data.
A data lake allows your data to grow, enormously. Without any major change in the storage strategy. In this age of big data, you can never really predict the size of your data. So, you need to have a place which can grow by orders of magnitude, if needed. A data lake fits that bill, perfectly.
A data lake also allows you to have disparate data sources and unstructured/ semi-structured data. This helps in having a loosely coupled processing architecture, without having to worry about supporting all previously unknown structured data. You can have a query engine now, knowing fully well that tomorrow you might have to get a new engine to process just one particular data source which is not present today. Having a lake to store such new types of data gives you a freedom for the future.

Design Principles

Design for scale: Your data lake will grow, and it will grow fast. Make sure that you have rules and policies to control who is pushing data into your lake. Your data governance rules should control when and how data gets loaded.
Modularity and extensibility: Your data lake should allow your consumer teams to plug and play. Your lake should not be restrictive for which technology can be used by your downstream teams.
(Big) Data lake: Unless you have very strong reasons, go for an HDFS based data lake. Also, in almost all cases, having a Hive metastore is a good idea. This will allow you to have a lot of choices with Hive, Presto, Spark, Impala, Kafka etc. Your storage might be S3 or any other cloud, but have HDFS and a Hive metastore. Hive’s catalog service allows you to maintain your metastore as a database and query it. You can host this metastore on any RDBMS (like MySQL, AWS RDS etc.).
Raw Data: Store all the raw data in your lake. You don’t realize it now but you will need it. This way, your lake becomes the single source of truth for all your lineage analysis. This looks like a waste of storage till the day you need to trace the source of bad data in your production pipelines.
All Inclusive Lake: Force your teams to adapt to the new architecture. A data lake is only fruitful if you have all your data platforms contributing. If half of your data is not in your lake then you have the same problem which plagued most data warehouses - siloed data stores. A data lake strategy is an all-in strategy, make sure that you have your management buy-in for the same.
Invest in data discovery and self service: As your lake grows in size, you will start facing problems in discovering what data resides where. You will have trouble in understanding the structure of data and location of files. So, start early in investing for a self service architecture. You will need to invest into a data discovery mechanism which can be used by the consumers of the lake to figure things out by themselves.
Out of all the keywords floating around in the engineering communities, data lake is one of those which sticks around for a long time. There is very clear merit in the idea of lakes in the world of big data. But, whether the implementation is successful or not depends completely on the strategy. In this post, I have tried to summarize the key principles which needs to be kept in mind. Do share your feedback in the comments below.

Comments

Post a Comment

Popular posts from this blog

Uber Data Model

Image
In this post I will try to come up with a data model which can serve the requirements of ride sharing companies like Uber, Lyft, Ola etc. We will approach the problem as an interview and see how we can come up with a feasible data model by answering important questions. Important Entities The first step towards building a data model is to identify important actors/ entities involved in the process. In our case, if we think about our interaction with taxi apps, we can identify important entities involved. The user (i.e. Rider) is one such entity, so is the Driver/ Partner . Once we open the app, we try to book a trip by finding a suitable taxi/ cab from a particular location to another . After the trip gets finished, the app collects the payment and we are done . Ideally, the flow continues to reviews/ ratings, helpcenter in case of issues etc. but for this post we will only consider scenarios till the ride gets finished. So, to summarize, we have the following key entities...
Read more

Data Engineer Interview Questions: SQL

Image
In this post, I will try to share some actual questions asked by top companies for Data Engineer positions. A lot of these companies will cover data modelling as one of the rounds and will use the data model for the next round based on SQL queries. Q1: Find the number of drivers available for rides in any area at any given point of time. Q2: Do you consider Driver and Rider as separate entities? Why or why not? Q3: Give me all passenger names who used the app for only airport rides. Q4: How will you decide where to apply surge pricing? Q5: How will you calculate wait times for rides? Q6: A driver can ride multiple cars, how will you find out who is driving which car at any moment? Q7: Find out Rank without using any function. Q8: How will you delete duplicates from a table? Q9: How will you find percentile? Q10:  You have 3 tables, user_dim (user_id, account_id), account_dim (account_id, paying_customer), and dload_facts (date, user_id, and downloads), fin...
Read more

Cracking Data Engineering Interviews

Of late, a lot of people have asked me for tips on how to crack Data Engineering interviews at FAANG (Facebook, Amazon, Apple, Netflix, Google) or similar companies. I have not worked at all of these companies so I can't share tips which will necessarily apply for all of them but I will share tips which can be generalized for most of the big companies. First of all, the field of Data Engineering has expanded a lot in the last few years and has become one of the core functions of any big technology company. The obvious reason for this expansion is the amount of data being generated by devices and data-centric economy of the internet age. Each company is focussed on making the best use of data owned by them by making data driven decisions. If you compare this to the Data Engineering roles which used to exist a decade back, you will see a huge change. In the past, Data Engineering was invariably focussed on Databases and SQL. Even now, these two form some part of most Data Engin...
Read more

Hive Challenges: Bucketing, Bloom Filters and More

In the second post of this series, we will learn about few more aspects of table design in Hive. Now that you know about partitioning challenges , you will be able to appreciate these features which will help you to further tune your Hive tables. Bucketing In addition to Partitioning the tables, you can enable another layer of bucketing of data based on some attribute value by using the Clustering method. Clustering can be used with partitioned or non-partitioned hive tables. One particular use case where Clustering becomes useful when your partitions might have unequal number of records (e.g. users logging in per country, US partition might be a lot bigger than New Zealand). In such cases, you can define the number of buckets and the clustered by field (like user Id), so that all the buckets have equal records. In partitioning each partition gets a directory while in Clustering, each bucket gets a file. Records with the same bucketed column will always be stored in the same bucke...
Read more

UBER Data Architecture

Image
In this post, I will try to cover the data architecture Uber has built to support their big data applications. This should be applicable for other ride hailing apps as well. There are multiple modules at play here so I will try to give a brief overview along with a detailed discussion on the data architecture.  An aeroplane view of the problem tells us that effectively we are trying to solve a demand vs. supply problem. All the Drivers active at any point of time constitute the supply while all the cab requesters (Riders) form the demand. What Uber tries to do is to have the best matching between demand and supply "at any point at any moment". I have highlighted the location and temporal conditions as they are critical for the success of what Uber tries to do. Let's get started to see how Uber comes around these challenges by creating a scalable data architecture. Every Driver active on Uber keeps sending his location data to the server (e.g. every 5 seconds). T...
Read more