Amaltas

Anay Tamhankar

Apache Airflow

Apache Airflow

Background: Data Warehouse and Data Lakes Data Warehouse Data Warehouse is a type of data management system that is designed to enable and support business intelligence (BI) applications and analytics. The data in a data warehouse contains large amounts of historical data, is structured and the system is used to

Hive Overview

Hive Overview

Apache Hive is a framework for data warehousing on top of Hadoop. Hive was created at Facebook and Hive grew from a need to to manage and learn from huge amounts of data that Facebook was generating every day. Facebook chose to built it on top of Hadoop for storage

HBase Overview

HBase Overview

Introduction HBase is a distributed column oriented database built over HDFS. Although the terminology in HBase is of a "table","row","column" etc... HBase should not be compared with a Relational Database Managment System like Oracle or MySQL. It is easiest to think of HBase as a persistent Hashtable or

Map Reduce and HDFS

Map Reduce and HDFS

Background Map Reduce framework and HDFS go hand-in-hand. The large datasets on which analysis is to be performed usually are saved in HDFS cluster. Each Data Node in the HDFS cluster is also a compute resource and is capable of executing a Map Reduce Job managed by YARN (Yet Another

Explaining Map Reduce

Explaining Map Reduce

Hadoop Map Reduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data sets) in parallel, on large clusters of commodity hardware (thousands of nodes) in a reliable and fault tolerant manner. A Map Reduce Job splits the input data-set into independent chunks which

Introduction to Apache Kafka

Introduction to Apache Kafka

Apache Kafka is general purpose Publish / Subscribe messaging system. It can also be described as a distributed streaming platform. Comparing Kafka to other modern messaging systems, Kakfa runs as a modern distirbuted system that runs as a cluster. Also Kakfa is a true storage system built to store data as

Stream Processing

Stream Processing

Stream processing is a key requirement In big data applications, stream processing is a key requirement. As soon as an application computes anything of value the organization will want to compute the result immediately and continuously, unlike in case of batch processing, where computations happen only after a certain batch