Cracking Data Engineering Interviews

Of late, a lot of people have asked me for tips on how to crack Data Engineering interviews at FAANG (Facebook, Amazon, Apple, Netflix, Google) or similar companies. I have not worked at all of these companies so I can't share tips which will necessarily apply for all of them but I will share tips which can be generalized for most of the big companies.

First of all, the field of Data Engineering has expanded a lot in the last few years and has become one of the core functions of any big technology company. The obvious reason for this expansion is the amount of data being generated by devices and data-centric economy of the internet age. Each company is focussed on making the best use of data owned by them by making data driven decisions. If you compare this to the Data Engineering roles which used to exist a decade back, you will see a huge change. In the past, Data Engineering was invariably focussed on Databases and SQL. Even now, these two form some part of most Data Engineering stacks but things have grown a lot towards the NoSQL world. Another major change happening right about now is the push towards real time analytics via Streaming applications. Combine all these together and Data Engineering becomes a quite daunting field with ever changing stacks and ever moving goal posts. In this post, I will try to break it down into pieces which can be grouped together for preparations. These are based on hundreds of interviews I have taken and loops I have been part of over the years working as a Data Engineer. I will also try to add useful resources which can be used for your preparation.

Most of the big companies will try to evaluate a Data Engineer( mid-senior level) candidate for the following skills:

Coding
NoSQL Stack
Databases + SQL
ETL - Batch + Streaming
Data Architecture and Design

I will briefly describe each of these skillsets and how companies evaluate candidates for these skills. 

Coding

Most technology companies will want to evaluate a candidate on the coding skills. Depending on the exact role/ org/ company, it might be an intense coding session or just a verbal interview to check the coding concepts. Most popular languages in the field are Python, Java and Scala. So, brush up your coding skills using online resources like Leetcode, HackerRank etc. You can also practice on online compilers like,

Python Resources:

NoSQL Stack

Most of the companies now have so much data that Hadoop and its assorted list of Apache toolset has become omnipresent. Though each company has its own technology stack, most are a combination of few popular ones. For storage, most places use HDFS (or some variant) because of the level of support for different technologies. For the processing part of the stack, Spark has become the most popular framework. A lot of places are still using Hive for legacy support. For the query engines, HiveQL is still very popular, even though a lot of people are moving to Presto/ SparkSQL etc. Though most big companies use the Apache version of these technologies, there are a lot of managed solutions in place as well. AWS has EMR which gives you a managed Hadoop cluster and services, DataBricks, HortonWorks etc. have similar offerings.

It is daunting to be a master of all these technologies. The key here is to understand what to use when and why. E.g. You might get asked about the strengths of Presto against Spark and when to use which one in real life scenarios. Refer to some of the posts below for more use cases, examples and exercises:

 Setup Hadoop on your machine
Setup Hive on your machine
Setup Presto on your machine
Hive vs Spark vs Presto
Sample NoSQL Scenarios

Databases + SQL

Even though there have been a lot of advancements in Data Engineering world, it is a fact that a lot of analysis happens over SQL. Which means, a good Data Engineer should know key fundamentals of databases (partitioning, indexes, High Availability, Disaster Recovery etc.). Also, you should be good in SQL. Most of these companies will have one interview to evaluate your SQL skills so make sure that you practice a lot.

 Sample SQL Interview

 SQL Practice Resources:
https://www.w3resource.com/mysql-exercises/

ETL/ Data Pipelines

Playing with tons of data means that Data Engineers spend a lot of time building (and maintaining) Data Pipelines to move data around systems. A lot of people I have interviewed mistake this to just tools ("I know Talend/ Informatica/ <random tool here>"). That is not an entirely convincing way to learn about data pipelines. Engineers at big Tech companies are supposed to keep moving to better systems all the time. Which means that knowledge of specific tools become irrelevant very quickly. Rather, a better way is to understand different ways data can be moved around from/ to different types of systems. E.g. An AWS centric team might rely heavily on S3 as the storage layer and access this layer using AWS SDKs for Spark while just using Python to load data into Redshift from S3. Similarly, another team might be using Airflow to build and orchestrate their Data Pipelines to read/ write data into their HDFS layer. 

The key here is to understand the generic approach for each type of systems. It is preferable to gain expertise of a few of these (e.g. Airflow, Oozie, Talend etc.). 

Another big push that is transforming the field is the move towards real time analytics using Streams. Streaming brings in a whole new level of complexity as most of the times batch designs will not work for streams. You will need to understand how Lambda architectures work to support both Streams and Batches for huge volumes of data. Also, Streaming apps need another specialized set of databases designed for fast reads (e.g. Cassandra, DynamoDB etc.). These databases allow dimension lookups at mind-numbing speeds to allow for the streams to not get delayed because of these lookups. 

Data Architecture

This section acts as the glue for all the things above. You need to know what fits where and how. Most companies will have this round for experienced Data Engineers to evaluate the candidate's understanding of system design. Some companies will give you an open ended problem/ App and ask you to design the Data Platform for the situation (e.g. Design the data platform for Twitter/ Uber/ Youtube etc.). Sometimes, this might go in the old-school direction of Star/ Snowflake modelling for a data warehouse. This may cover various aspects like file formats (ORC/ Parquet/ Avro etc.), partitioning schemes, columnar vs rowstore storage, performance benchmarking processes etc.
Sample Architecture

Resources:
Plumbers of Data Science

Uber's blog:

AWS blog


Comments

  1. StephanicaMarch 15, 2021 at 10:13 PM

    Thank you so much for this nice information. Hope so many people will get aware of this and useful as well.

    Indium Software

    ReplyDelete
    Replies
    1. AnonymousApril 7, 2022 at 4:13 AM

      Cracking Data Engineering Interviews >>>>> Download Now

      >>>>> Download Full

      Cracking Data Engineering Interviews >>>>> Download LINK

      >>>>> Download Now

      Cracking Data Engineering Interviews >>>>> Download Full

      >>>>> Download LINK 84

      Delete
  2. JamesOctober 1, 2021 at 11:50 PM

    Very Nice, Thanks for sharing such an informative Article, It was great reading this article. I would like to know more about it

    data engineering course

    ReplyDelete
  3. iTelenetFebruary 8, 2022 at 9:17 AM

    Those who are looking for a data engineering solution in the UK must connect with the team of iTelenet. We have a team of experts who offer the right kind of services to deliver data engineering services.

    ReplyDelete
  4. ERP Training NoidaFebruary 25, 2022 at 3:51 AM

    This comment has been removed by the author.

    ReplyDelete
  5. Eva BrielleAugust 19, 2022 at 4:39 AM

    Thank you for sharing valuable content.
    Data Engineering Training

    ReplyDelete

Post a Comment

Popular posts from this blog

Uber Data Model

Image
In this post I will try to come up with a data model which can serve the requirements of ride sharing companies like Uber, Lyft, Ola etc. We will approach the problem as an interview and see how we can come up with a feasible data model by answering important questions. Important Entities The first step towards building a data model is to identify important actors/ entities involved in the process. In our case, if we think about our interaction with taxi apps, we can identify important entities involved. The user (i.e. Rider) is one such entity, so is the Driver/ Partner . Once we open the app, we try to book a trip by finding a suitable taxi/ cab from a particular location to another . After the trip gets finished, the app collects the payment and we are done . Ideally, the flow continues to reviews/ ratings, helpcenter in case of issues etc. but for this post we will only consider scenarios till the ride gets finished. So, to summarize, we have the following key entities;
Read more

Data Engineer Interview Questions: SQL

Image
In this post, I will try to share some actual questions asked by top companies for Data Engineer positions. A lot of these companies will cover data modelling as one of the rounds and will use the data model for the next round based on SQL queries. Q1: Find the number of drivers available for rides in any area at any given point of time. Q2: Do you consider Driver and Rider as separate entities? Why or why not? Q3: Give me all passenger names who used the app for only airport rides. Q4: How will you decide where to apply surge pricing? Q5: How will you calculate wait times for rides? Q6: A driver can ride multiple cars, how will you find out who is driving which car at any moment? Q7: Find out Rank without using any function. Q8: How will you delete duplicates from a table? Q9: How will you find percentile? Q10:  You have 3 tables, user_dim (user_id, account_id), account_dim (account_id, paying_customer), and dload_facts (date, user_id, and downloads), find the ave
Read more

Hive Challenges: Bucketing, Bloom Filters and More

In the second post of this series, we will learn about few more aspects of table design in Hive. Now that you know about partitioning challenges , you will be able to appreciate these features which will help you to further tune your Hive tables. Bucketing In addition to Partitioning the tables, you can enable another layer of bucketing of data based on some attribute value by using the Clustering method. Clustering can be used with partitioned or non-partitioned hive tables. One particular use case where Clustering becomes useful when your partitions might have unequal number of records (e.g. users logging in per country, US partition might be a lot bigger than New Zealand). In such cases, you can define the number of buckets and the clustered by field (like user Id), so that all the buckets have equal records. In partitioning each partition gets a directory while in Clustering, each bucket gets a file. Records with the same bucketed column will always be stored in the same bucke
Read more

UBER Data Architecture

Image
In this post, I will try to cover the data architecture Uber has built to support their big data applications. This should be applicable for other ride hailing apps as well. There are multiple modules at play here so I will try to give a brief overview along with a detailed discussion on the data architecture.  An aeroplane view of the problem tells us that effectively we are trying to solve a demand vs. supply problem. All the Drivers active at any point of time constitute the supply while all the cab requesters (Riders) form the demand. What Uber tries to do is to have the best matching between demand and supply "at any point at any moment". I have highlighted the location and temporal conditions as they are critical for the success of what Uber tries to do. Let's get started to see how Uber comes around these challenges by creating a scalable data architecture. Every Driver active on Uber keeps sending his location data to the server (e.g. every 5 seconds). T
Read more