Big Data Things

This is the second post in this series, I will walk you through the steps to run Hadoop on your puny little Macbook. If you ever had a hesitation in learning about Hadoop services because of a lack of a "place to run code", you need to find another excuse for not learning Hadoop after this post. I will be using Homebrew for installing a lot of things in this series, it makes your life a lot simpler.  Pre-requisites You need to have Java installed. And you need to know how to set up your bash_profile. If you don't know how to do it: Creating your bash_profile file   Installation The way Homebrew works is that it will pick the latest available stable build and install it. But, in our case, we need our Hadoop installation to support other services like Hive and Presto so we need to go for a stable build of Hadoop, which in our case is 2.7.6. For installing a previous version, you need to get the checksum of the older version and add that to the Hadoop f...

When starting to learn big data, the biggest problem most people face is the lack of a place to try things out. This becomes a hindrance in learning and eventually people lose hope and put an end to the learning attempts. So, I have decided to document the steps needed to get started on this journey.  In this series of posts, I will walk you through the steps needed to setup your machine (a Macbook in my case) with tools needed for learning big data technologies. I will start with a simple post about setting up the bash profile for your Macbook. Then I will install Hadoop services and move on to install Hive and Presto using the same Hadoop namenode. The inspiration here is a set of wonderful posts from Keith, but unfortunately not all the steps work anymore. So, I will be documenting all the steps which worked for me (as of June 2019). You can follow all these posts here: Setting your Macbook to run Apache Hadoop Setting your Macbook to run Apache Hive Setting your Macbo...

In this series of articles, I will talk about the challenges while working with Hive tables. Like with any other Apache toolkit, there are a lot of configurations to play around with and you need to know a few of them. Luckily, for partitioning there are a lot of concepts which are similar to the RDBMS world. So, if you have some idea about table partitioning in databases, this should not be very complex for you. Q: How to decide number of partitions for a very large table? There are multiple aspects to look out for, Cardinality:  Find out the cardinality of important attributes to find out how many distinct values the attribute holds. If you are making any attribute as a partition, it should have as few distinct values as possible( though few hundred partitions for a table are also fine). Sorting:  If you can compromise on the write speeds, sort the data while writing. This allows formats like ORC to capture the starting and ending values in the metadata of fi...

Of late, a lot of people have asked me for tips on how to crack Data Engineering interviews at FAANG (Facebook, Amazon, Apple, Netflix, Google) or similar companies. I have not worked at all of these companies so I can't share tips which will necessarily apply for all of them but I will share tips which can be generalized for most of the big companies. First of all, the field of Data Engineering has expanded a lot in the last few years and has become one of the core functions of any big technology company. The obvious reason for this expansion is the amount of data being generated by devices and data-centric economy of the internet age. Each company is focussed on making the best use of data owned by them by making data driven decisions. If you compare this to the Data Engineering roles which used to exist a decade back, you will see a huge change. In the past, Data Engineering was invariably focussed on Databases and SQL. Even now, these two form some part of most Data Engin...

In this post, I will cover the core concepts which govern the execution model of Spark. I will talk about the different components, how they interact with each other and what happens when you fire a query. I will also take few examples to illustrate how Spark configs change these behaviours. Let's talk about the architecture first. Each Spark application (instance of SparkContext) starts with a Driver program. The Driver talks to the Cluster Manager( YARN, Mesos, Kubernetes etc.) to demand the resources it needs. Executors (being one such resource) are processes which run computations and stored data for applications. Each application gets its own executor processes on Worker Nodes. Effectively, it means that data can't be shared across different applications without writing to a storage layer. Also, executors typically run for the lifetime of a Spark application and run multiple tasks over its lifetime (parallelly and sequentially). What happens when you submit ...

In this post I will try to come up with a data model which can serve the requirements of ride sharing companies like Uber, Lyft, Ola etc. We will approach the problem as an interview and see how we can come up with a feasible data model by answering important questions. Important Entities The first step towards building a data model is to identify important actors/ entities involved in the process. In our case, if we think about our interaction with taxi apps, we can identify important entities involved. The user (i.e. Rider) is one such entity, so is the Driver/ Partner . Once we open the app, we try to book a trip by finding a suitable taxi/ cab from a particular location to another . After the trip gets finished, the app collects the payment and we are done . Ideally, the flow continues to reviews/ ratings, helpcenter in case of issues etc. but for this post we will only consider scenarios till the ride gets finished. So, to summarize, we have the following key entities...

In this post, I will try to share some actual questions asked by top companies for Data Engineer positions. A lot of these companies will cover data modelling as one of the rounds and will use the data model for the next round based on SQL queries. Q1: Find the number of drivers available for rides in any area at any given point of time. Q2: Do you consider Driver and Rider as separate entities? Why or why not? Q3: Give me all passenger names who used the app for only airport rides. Q4: How will you decide where to apply surge pricing? Q5: How will you calculate wait times for rides? Q6: A driver can ride multiple cars, how will you find out who is driving which car at any moment? Q7: Find out Rank without using any function. Q8: How will you delete duplicates from a table? Q9: How will you find percentile? Q10:  You have 3 tables, user_dim (user_id, account_id), account_dim (account_id, paying_customer), and dload_facts (date, user_id, and downloads), fin...

In this post, I will try to cover the data architecture Uber has built to support their big data applications. This should be applicable for other ride hailing apps as well. There are multiple modules at play here so I will try to give a brief overview along with a detailed discussion on the data architecture.  An aeroplane view of the problem tells us that effectively we are trying to solve a demand vs. supply problem. All the Drivers active at any point of time constitute the supply while all the cab requesters (Riders) form the demand. What Uber tries to do is to have the best matching between demand and supply "at any point at any moment". I have highlighted the location and temporal conditions as they are critical for the success of what Uber tries to do. Let's get started to see how Uber comes around these challenges by creating a scalable data architecture. Every Driver active on Uber keeps sending his location data to the server (e.g. every 5 seconds). T...