Big Data Things

This is the fourth post in this series, geared up to making the reader self-sufficient for learning big data applications. In the previous posts of the series, we have installed Hadoop, Hive on the same namenode and MySQL metastore for Hive. In this post, we will build on the same setup and install Presto to use the same HDFS and Hive metastore. Let's get started then. Prerequisites Setup your machine for learning big data Setup Apache Hadoop on your machine Setup Apache Hive on your machine Installing Presto Again, we will use Homebrew to install Presto on our machine. Run brew install presto on your terminal. Configuring Presto You will need to edit following files ( at location etc/ ): node.properties config.properties log.properties jvm.config Export Environment Variables Add the following to your .bash_profile file and restart terminal (or source it): export PRESTO_VERSION=0.221 export PRESTO_HOME=/usr/local/Cellar/presto/0.221/l...

This is the third post of this series where we are setting up our machines to get started with learning big data. In the first post, we have installed and configured Hadoop on our Macbook and now we will install Apache Hive and use the same Hadoop cluster to save data in HDFS. Let's get started then. Prerequisites 1. Setup your machine for learning big data 2. Setup Apache Hadoop on your machine Installation We will be installing Hive 1.2.2, so we need to update the brew formula (as we did for Hadoop in the previous article). Use brew edit hive to open the formula and edit it to have these values: url " https://www.apache.org/dyn/closer.cgi?path=hive/hive-1.2.2/apache-hive-1.2.2-bin.tar.gz " sha256 "763b246a1a1ceeb815493d1e5e1d71836b0c5b9be1c4cd9c8d685565113771d1" Once you have updated the formula, install hive using brew install hive Modify Hive Configurations First things first, we need to edit the .bash_profile fil...

This is the second post in this series, I will walk you through the steps to run Hadoop on your puny little Macbook. If you ever had a hesitation in learning about Hadoop services because of a lack of a "place to run code", you need to find another excuse for not learning Hadoop after this post. I will be using Homebrew for installing a lot of things in this series, it makes your life a lot simpler.  Pre-requisites You need to have Java installed. And you need to know how to set up your bash_profile. If you don't know how to do it: Creating your bash_profile file   Installation The way Homebrew works is that it will pick the latest available stable build and install it. But, in our case, we need our Hadoop installation to support other services like Hive and Presto so we need to go for a stable build of Hadoop, which in our case is 2.7.6. For installing a previous version, you need to get the checksum of the older version and add that to the Hadoop f...

When starting to learn big data, the biggest problem most people face is the lack of a place to try things out. This becomes a hindrance in learning and eventually people lose hope and put an end to the learning attempts. So, I have decided to document the steps needed to get started on this journey.  In this series of posts, I will walk you through the steps needed to setup your machine (a Macbook in my case) with tools needed for learning big data technologies. I will start with a simple post about setting up the bash profile for your Macbook. Then I will install Hadoop services and move on to install Hive and Presto using the same Hadoop namenode. The inspiration here is a set of wonderful posts from Keith, but unfortunately not all the steps work anymore. So, I will be documenting all the steps which worked for me (as of June 2019). You can follow all these posts here: Setting your Macbook to run Apache Hadoop Setting your Macbook to run Apache Hive Setting your Macbo...