Analytics 3
Download as pptx, pdf0 likes238 views
The presentation covers the analytics landscape, focusing on big data and Hadoop architecture. Key components discussed include HDFS, MapReduce, YARN, Spark, Hive, HBase, Flume, Mahout, and Sqoop, highlighting their roles in data processing and management. It emphasizes the efficiency and capabilities of each component, especially in relation to machine learning and data querying.
Analytics 3
- 1. ITS ALL ABOUT DATA & ANALYTICS Session - 3
- 2. Topics covered in this Presentation Analytics Landscape BigData Hadoop BigData Landscape
- 5. Big Data
- 8. Hadoop Architecture Hadoop Common- A set of common libraries and utilities used by other Hadoop modules. HDFS-The default storage layer for Hadoop. MapReduce- Executes a wide range of analytic functions by analysing datasets in parallel before reducing the results.The Map job distributes a query to different nodes, and the Reduce gathers the results and resolves them into a single value. YARN- Present in version 2.0 onwards,YARN is the cluster management layer of Hadoop. Prior to 2.0, MapReduce was responsible for cluster management as well as processing.The inclusion ofYARN means you can run multiple applications in Hadoop (so youre no longer limited to MapReduce), which all share common cluster management.
- 9. Hadoop Architecture Spark- Used on top of HDFS, Spark promises speeds up to 100 times faster than the two- step MapReduce function in certain applications. Allows data to loaded in-memory and queried repeatedly, making it particularly apt for machine learning algorithms Hive- Originally developed by Facebook, Hive is a data warehouse infrastructure built on top of Hadoop. Hive provides a simple, SQL-like language called HiveQL, whilst maintaining full support for MapReduce. This means SQL programmers with little former experience with Hadoop can use the system easier, and provides better integration with certain analytics packages like Tableau. Hive also provides indexes, making querying faster. HBase- Is a NoSQL columnar database which is designed to run on top of HDFS. It is modelled after Googles BigTable and written in Java. It was designed to provide BigTable- like capabilities to Hadoop, such as the columnar data storage model and storage for sparse data. Flume- Flume collects (typically log) data from agents which it then aggregates and moves into Hadoop. In essence, Flume is what takes the data from the source (say a server or mobile device) and delivers it to Hadoop. Mahout- Mahout is a machine learning library. It collects key algorithms for clustering, classification and collaborative filtering and implements them on top of distributed data systems, like MapReduce. Mahout primarily set out to collect algorithms for implementation on the MapReduce model, but has begun implementing on other systems which were more efficient for data mining, such as Spark. Sqoop- Sqoop is a tool which aids in transitioning data from other database systems (such as relational databases) into Hadoop.
- 17. In God,WeTrust... All Others must bring the "Data" Srikanth Ayithy about.me/srikanthayithy