端端舝

/ 20
Spark GraphFrames 及旦旦丟
申永弘犯奈正窒樓㮼墿嬡
2016爛3堎23゜耋哱詈LT撬曰

/ 20
赻撩畿賡
? 樓㮼墿嬡
? 2014爛4堎‵ DMM.com仿示
? Hadoop價屜�慷
? Spark MLlib, GraphX, spark.ml, GraphFrames 毛蚚中凶伊戊丟件玉嶱逃
? 疑五卅晟惤
? SQL
? Cypher
2

/ 20
GraphFrames午反ˋ
? GraphFrames
? http://graphframes.github.io/
? 煦汃弘仿白�I燴及凶戶及 Apache Spark 由永弗奈斥
? Spark GraphX 午 DataFrames (SparkSQL) 及緙磁
? Databricks互2016爛3堎3゜卞伉伉奈旦
3

/ 20
卅兮GraphFrames井ˋ
4
汜宴俶ㄗ�I燴及𤩸五支允今ㄘ
旦弗奈仿申伉氾奴
GraphFrames
↗ ��及覜砑匹允
弘仿白DB/弘仿白�I燴炵庨ⅲ

/ 20
GraphFrames及丟伉永玄
? 詢伊奶乩及API
? 杅俴及�扴匹弘仿白及煦汃�I燴互灍政匹五月
? 弘仿白犯奈正及�慷互⺍眢
? RDB支DataFrames卅升及氾奈皮伙倛宒及犯奈正井日
忒幏卞弘仿白�婖及犯奈正毛釬傖匹五月
? 皮伙奈左奈扑乓件ㄐ
5
https://www.google.co.jp/search?q=graphframes&ie=utf-8&oe=utf-8&hl=ja (2016.3.23政婓)

/ 20
GraphFrames毛�允
? Spark午肮�﹜Scala, Java, Python, R砃仃及API毛妏蚚褫夔
? Spark Shell 匹奶件正仿弁氾奴皮卞�允
? Spark 1.4眕奻卞�𡛟
? DataFrames及瞳萸毛魂井允卅日郔陔唳毛芢𠾴
6
# spark 毛母它件伕奈玉
$ wget http://ftp.jaist.ac.jp/pub/apache/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.6.tgz
$ tar xzvf spark-1.6.0-bin-hadoop2.6.tgz
# graphframes由永弗奈斥毛硌隅仄化spark-shell毛お�
$ spark-1.6.0-bin-hadoop2.6/bin/spark-shell --packages graphframes:graphframes:0.1.0-spark1.6

/ 20
GraphFrames 每弘仿白及釬傖
7
// graphframes由永弗奈斥及奶件禾奈玄
scala> import org.graphframes._
import org.graphframes._
// Vertexㄗ�萸ㄘ午卅月DataFrame毛釬傖
scala> val v = sqlContext.createDataFrame(List(
| (0L, "user", "u1"),
| (1L, "user", "u2"),
| (2L, "item", "i1"),
| (3L, "item", "i2"),
| (4L, "item", "i3"),
| (5L, "item", "i4")
| )).toDF("id", "type", "name")
v: org.apache.spark.sql.DataFrame = [id: bigint, type: string, name: string]
u1
u2
交奈扒
i1
i2
i3
i4
失奶氾丞

/ 20
GraphFrames 每弘仿白及釬傖
8
// Edgeㄗ煘ㄘ午卅月DataFrame毛釬傖
scala> val e = sqlContext.createDataFrame(List(
| (0L, 2L, "purchase"),
| (1L, 5L, "purchase")
| )).toDF("src", "dst", "type")
e: org.apache.spark.sql.DataFrame = [src: bigint, dst: bigint, type: string]
// GraphFrame毛釬傖
scala> val g = GraphFrame(v, e)
g: org.graphframes.GraphFrame = GraphFrame(v:[id: bigint, attr: string, gender: string],
e:[src: bigint, dst: bigint, relationship: string])
u1
u2
i1
i2
i3
i4
�⻌伕弘

/ 20
GraphFrames 每失奶氾丞伊戊丟件玉及灍俴瞰
9
// 伊戊丟件玉失奶氾丞及�中磁歹六瞰
scala> g.find(
| " (a)-[]->(x); (b)-[]->(x);" +
| " (b)-[]->(y); !(a)-[]->(y)"
| ).groupBy(
| "a.name", "y.name"
| ).count().show()
+----+----+-----+
|name|name|count|
+----+----+-----+
| u1| i4| 2|
| u2| i1| 2|
+----+----+-----+
u1
u2
i1
i2
i3
i4
僕籵及妀ⅲ毛
�⻌仄凶交奈扒
引分�⻌仄化中卅中失奶氾丞毛伊戊丟件玉
(b)
(y)
(a)
(x)

/ 20
GraphFrames 每 PageRank及灍俴瞰
11
// PageRank毛�呾
scala> val pr = g.pageRank.resetProbability(0.1).tol(0.01).run()
// PageRank及旦戊失毛桶尨
scala> pr.vertices.show()
+---+-------+--------+
| id|v_attr1|pagerank|
+---+-------+--------+
| 0| root| 0.55|
| 1| node-1| 0.1|
| 2| node-2| 0.1|
| 3| node-3| 0.1|
| 4| node-4| 0.1|
| 5| node-5| 0.1|
+---+-------+--------+
0
1
2
3 4
5
0.1 0.1
0.1
0.1 0.1
0.55

/ 20
a
GraphFrames 每郔傻擒褩毛�呾
13
// 允屯化及交奈扒井日交奈扒 ※a§ 引匹及郔傻擒褩毛�呾
scala> val d1 = friends.shortestPaths.landmarks(Seq("a")).run()
// 磐彆毛桶尨
scala> d1.show()
+---+-------+---+-----------+
| id| name|age| distances|
+---+-------+---+-----------+
| f| Fanny| 36| Map()|
| g| Gabby| 60| Map()|
| a| Alice| 34|Map(a -> 0)|
| b| Bob| 36| Map()|
| c|Charlie| 30| Map()|
| d| David| 29|Map(a -> 1)|
| e| Esther| 32|Map(a -> 2)|
+---+-------+---+-----------+
a
b
c
de
f
a -> 0
g
a -> 2 a -> 1

/ 20
a
c
GraphFrames 每郔傻擒褩毛�呾
14
// 允屯化及交奈扒井日交奈扒 ※a§, ※c§ 引匹及郔傻擒褩毛�呾
scala> val d2 = friends.shortestPaths.landmarks(Seq("a", "c")).run()
// 磐彆毛桶尨
scala> d2.show()
+---+-------+---+-------------------+
| id| name|age| distances|
+---+-------+---+-------------------+
| f| Fanny| 36| Map(c -> 1)|
| g| Gabby| 60| Map()|
| a| Alice| 34|Map(a -> 0, c -> 2)|
| b| Bob| 36| Map(c -> 1)|
| c|Charlie| 30| Map(c -> 0)|
| d| David| 29|Map(a -> 1, c -> 3)|
| e| Esther| 32|Map(a -> 2, c -> 2)|
+---+-------+---+-------------------+
a
b
c
de
f
g
a -> 0
c -> 2
a -> 2
c -> 2
a -> 1
c -> 3
c -> 0
c -> 1
c -> 1

/ 20
a
b
c
d
GraphFrames 每郔傻磊繚及抻坰
15
// 交奈扒 ※d§井日※c§ 尺及郔傻磊繚毛抻坰
scala> val path = friends.bfs.fromExpr("id = 'd'").toExpr("id = 'c'").run()
// 磐彆毛桶尨
scala> path.show()
+------------+------------+------------+
| from| e0| v1|
+------------+------------+------------+
|[d,David,29]|[d,a,friend]|[a,Alice,34]|
+------------+------------+------------+
+------------+----------+------------+--------------+
| e1| v2| e2| to|
+------------+----------+------------+--------------+
|[a,b,friend]|[b,Bob,36]|[b,c,follow]|[c,Charlie,30]|
+------------+----------+------------+--------------+
a
b
c
de
f
g
Alice, 34
Bob, 36
Charlie, 30
David, 29
friend
follow
friend

/ 20
GraphFrames 每公及坻及辻夔
? GraphFrames User Guide
16
http://graphframes.github.io/user-guide.html

/ 20
GraphFrames 每交奈旦弗奈旦
? On-Time Flight Performance with GraphFrames for Apache Spark
17
https://databricks.com/blog/2016/03/16/on-time-flight-performance-with-spark-graphframes.html

/ 20
GraphFrames vs. Neo4j
18
竘蚚ㄩhttp://www.slideshare.net/SparkSummit/graphframes-graph-queries-in-spark-sql-by-ankur-dave

/ 20
GraphFrames ℅ Spark 2.0
19
竘蚚ㄩ http://www.slideshare.net/databricks/2016-spark-summit-east-keynote-matei-zaharia

/ 20
GraphFrames引午戶
? 詢伊奶乩及煦汃弘仿白�I燴API
? 詢中汜宴俶
? 詢厒卅煦汃弘仿白�I燴
? 踏堎伉伉奈旦今木凶壬井曰
? 引分引分辻夔支ロ�反屾卅中
? 踏摽及逃桯支魂蚚卞ヽ渾
20

端端舝

Spark GraphFrames 及旦旦丟

More Related Content

Spark GraphFrames 及旦旦丟