DMM.com ¥é¥Ü¤Ï¤Ê¤¼Spark¤ò’ñÓä·¤¿¤Î¤«£¿ ¥ì¥³¥á¥ó¥É¥¨¥ó¥¸¥óé_°k¤ÎÑY‚Ȥò¤ªÔ’¤·¤Þ¤¹
?
9 likes?2,347 views
This document discusses DMM's use of Apache Spark for real-time recommendations. It covers how Spark is used for tracking APIs, Hive integration, item-to-item and user-to-item recommendations using Spark MLlib ALS, connecting to databases using Sqoop, and deploying and executing recommendation APIs on Jenkins with BuildFlow. Tips are provided on using dataframes/datasets, optimizing memory, and the top 5 mistakes to avoid in Spark applications.
![API
item2item(id: ItemId): List[ItemId]
user2item(id: UserId): List[ItemId]
popular : List[ItemId]](https://image.slidesharecdn.com/hikalabo-160330112223/85/DMM-com-Spark-24-320.jpg)
![[ ]](https://image.slidesharecdn.com/hikalabo-160330112223/85/DMM-com-Spark-31-320.jpg)
DMM.com ¥é¥Ü¤Ï¤Ê¤¼Spark¤ò’ñÓä·¤¿¤Î¤«£¿ ¥ì¥³¥á¥ó¥É¥¨¥ó¥¸¥óé_°k¤ÎÑY‚Ȥò¤ªÔ’¤·¤Þ¤¹
- 2. 2015/4 - DMM labo API
- 4. DMM
- 5. DMM
- 8. SPARK UC Berkekey Apache Scala, Python, Java, SQL, R API
- 13. Hadoop
- 14. item to item user to item popular
- 15. 1. (Tracking API) 2. (Hive on Spark) 3. (Spark) 4. (Sqoop) 5. API(Play)
- 19. val itemToItems = userProducts.join(userProducts).filter { case (user, ((item1, keyword1, score1), (item2, keyword2, score2))) => item1 }.map { case (user, ((item1, keyword1, score1), (item2, keyword2, score2))) => ((item }.reduceByKey(_ + _).mapValues(math.sqrt(_)).map { case ((item1, keyword1, item2), score) => ((item1, keyword1), (item2, score)) }.groupByKey().mapValues(_.toList.sortBy(_._2).reverse.take(config.numDisplayIt case ((item1, keyword1), items) => items.size >= config.numDisplayItems }.cache()
- 21. val model = ALS.train(ratings.map(_._1), config.alsRank, config.alsNumIterations, config.alsLambda) val predictions = model.predict(candidates).groupBy(_.user).map { case (user, ratings) => (user, ratings.toList.sortBy(_.rating) .reverse.take(config.numDisplayItems)) }.cache()
- 23. Sqoop MariaDB
- 24. API item2item(id: ItemId): List[ItemId] user2item(id: UserId): List[ItemId] popular : List[ItemId]
- 25. DEPLOY AND EXECUTE Jenkins + Build Pipeline + BuildFlow
- 26. (2015/09) Jenkins + Build Pipeline + BuildFlow Job Script + Git Hive Spark Sqoop Recommend API(Node.js) MariaDB(Galera Cluster)
- 27. Jenkins + Build Pipeline + BuildFlow Job Script + Management API Hive on Spark Spark Sqoop Recommend API(Play) MariaDB(Galera Cluster)
- 28. Management API File Hive on Spark Hive 3 Play Spark, Hive UDF Util
- 30. AB PDCA
- 31. [ ]
- 32. 701
- 33. 75 % ¡ü 97% ¡ü
- 34. TIPS use dataframes or datasets hive
- 35. executor memoryOverhead cheat sheet Top 5 Mistakes to Avoid When Writing Apache Spark Applications
- 38. DATAFRAMES DATASETS (1.3 - ) Dataframes (1.6 - ) Datasets Project Tungsten(1.5 - )
- 40. Realtime Recommend Dataframes & Datasets Graphframes