Hadoop Technologies
Download as ppt, pdf7 likes5,779 views
The document discusses various Hadoop technologies including HDFS, MapReduce, Pig/Hive, HBase, Flume, Oozie, Zookeeper, and HBase. HDFS provides reliable storage across multiple machines by replicating data on different nodes. MapReduce is a framework for processing large datasets in parallel. Pig and Hive provide high-level languages for analyzing data stored in Hadoop. Flume collects log data as it is generated. Oozie manages Hadoop jobs. Zookeeper allows distributed coordination. HBase provides a fault-tolerant way to store large amounts of sparse data.
Hadoop Technologies
- 2. Hadoop Technologies Google Google File System MapReduce Sawzall BigTable Google Open Source HDFS Hadoop MapReduce Pig / Hive HBase Open source communities,Yahoo, facebook, Cloudera, twitter and LinkedIn
- 3. Other Players Amazon File System : Amazon S3 Instances : Amazon EC2 Cluster Platform : Hadoop Microsoft Dryad (Distributed Runtime) DryadLINQ (High level lang)
- 4. HDFS Storage : Large files stored across multiple machines Reliability : Data replicated across multiple hosts Replication : Default replication value 3. Data is stored on three nodes: two on the same rack, and one on a different rack
- 5. MapReduce Framework inspired by map, reduce functional contructs in LISP Classic paper : http://labs.google.com/papers/mapreduce.html Hadoop Map Reduce Pig / Hive
- 6. Apache Pig Pig Latin : High level language Pig Engine : Compiles Pig code to Map Reduce Nested Data Model (Atom, Tuple, Bag) UDFs are first class citizens
- 7. Apache Pig good_urls = FILTER urls BY pagerank > 0.2; groups = GROUP good_urls BY category; big_groups = FILTER groups BY COUNT(good_urls)>106 ; output = FOREACH big_groups GENERATE category, AVG(good_urls.pagerank)
- 8. Flume Log Collection Flume is a distributed, reliable, and available service for efficiently moving large amounts of data as the data is produced. MapQuest Log Processing Example 100's of prod servers, 5 Log processing machines, 1 Netezza Data Warehouse
- 10. Other Technologies Oozie ¨C Yahoo!¡¯s workflow engine for Hadoop Open-source workflow solution to manage and coordinate jobs running on Hadoop, including HDFS, Pig and MapReduce. Zookeeper ¨C ZooKeeper allows distributed processes to coordinate with each other through a shared hierarchal name space of data registers (znodes), much like a file system. HBase ¨C HBase is an open source, non-relational, distributed database modeled after Google's BigTable and runs on top of HDFS. It provides a fault-tolerant way of storing large quantities of sparse data. Tables in HBase can serve as the input and output for MapReduce jobs run in Hadoop. (Powerset) Hive ¨C SQL like interface (Jeff Hammerbacher)