Amaltas

hadoop

Map Reduce and HDFS

Map Reduce and HDFS

Background Map Reduce framework and HDFS go hand-in-hand. The large datasets on which analysis is to be performed usually are saved in HDFS cluster. Each Data Node in the HDFS cluster is also a compute resource and is capable of executing a Map Reduce Job managed by YARN (Yet Another

Explaining Map Reduce

Explaining Map Reduce

Hadoop Map Reduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data sets) in parallel, on large clusters of commodity hardware (thousands of nodes) in a reliable and fault tolerant manner. A Map Reduce Job splits the input data-set into independent chunks which

HDFS Java API

HDFS Java API

Starting Point There are two important Java classes which are the starting point for using the Java API. Upto Hadoop Version 1.x org.apache.hadoop.fs.FileSystem Hadoop Version 2.x and later org.apache.hadoop.fs.FileContext FileContext class has better handling of multiple fileystems (A single FileContext

HDFS Architecture

HDFS Architecture

HDFS Architecture Diagram Data Replication How does HDFS manage to provide data reliability and high fault tolerance when it uses commodity (and therefore cheap and prone to failure) hardware ? HDFS does it using a very simple idea. 1. Break a large file into series of chunks or blocks (each block

Hadoop Distributed File System (HDFS)

Hadoop Distributed File System (HDFS)

What is HDFS ? HDFS is a distributed file system designed to run on commodity hardware. HDFS is highly fault tolerant and provides high throughput access to "application data". It is suitable for applications which have large datasets. Distributed File Systems are file systems that manage storage across a network of

Introduction to Hadoop

Introduction to Hadoop

What is Hadoop ? Hadoop is a collection of open source software utilities that facilitate solving big data problems using a network of computers. Hadoop comprises of a software framework for distributed storage (HDFS) and a software framework for query and analysis (Map Reduce). Design Principal 💡Hardware failure are common occurrences