Introduce to spark
0 likes38 views
Spark is a big data processing tool built with Scala that runs on the Java Virtual Machine (JVM). It is up to 100 times faster than Hadoop for iterative jobs because it keeps intermediate data in memory rather than writing to disk. Spark uses Resilient Distributed Datasets (RDDs) that allow data to be kept in memory across transformations and actions. RDDs also maintain lineage graphs to allow recovery from failures.
![Error Fixing
RDD Lineage
RDD1 RDD2
Transformation Action
[ 7, 10 ]
[ 2, 3 ] [ ?, ? ]f(x) = x2
+1
13
RDD2RDD1
2
[ 7, 10 ]
Fix !
1](https://image.slidesharecdn.com/introducetospark-171127064142/85/Introduce-to-spark-13-320.jpg)
Introduce to spark
- 1. A big data processing tool built with Scala and runs on JVM Introduce to Spark ADB 2017 Yen Hao Huang 1
- 2. Big Data 4Vs Volume/Variety/Velocity/Veracity Due to the rise of Big Data, faster tools are required for processing data. 2
- 3. Hadoop 3
- 4. Hadoop A platform to store and process large scale data Features Scalable Economical : many cheap servers Flexible : schema-less Reliable : replicas 4
- 5. Hadoop MapReduce Map - Divide job to multiple tiny tasks and distribute to servers Reduce - Summary the results from those servers 5
- 7. File I/O - write the middle process data to disk Hadoop - Bottleneck Iteration Iteration Read Read WriteWrite 7
- 8. Spark 8
- 10. RDD (Resilient Distributed Dataset) Write the middle process data to memory 10 - 100 times faster than hadoop Iteration Iteration Read Memory Read WriteMemory Write RDD 10
- 11. Spark Features Speed Ease of use : ScalaPythonJavaR Supports hadoop : HDFSMapReduce Accessibility : runs on many platforms 11
- 12. RDD Features Computations Transformation - Lazy compute Action - Execute the computations Persistence - Keep RDD in ram/ disk Transformation RDD OutputAction 12
- 13. Error Fixing RDD Lineage RDD1 RDD2 Transformation Action [ 7, 10 ] [ 2, 3 ] [ ?, ? ]f(x) = x2 +1 13 RDD2RDD1 2 [ 7, 10 ] Fix ! 1