際際滷

際際滷Share a Scribd company logo
Spark x Solrによるリアルタイムト
レンド渇竃の醤悶議返隈
wびzみ喘
書恬った
R&Dち`む: tanaka.yuichi
徭失B初とか
署蛮狼?Mみzみ狼?コミュニティサ`ビス?ゲ`ム?レ御吉の光システムをUてDMMへ。
DMM.comラボにおいて、碧システムの泡仟、レコメンドエンジンの仟_kなど、児
P室gの_kを毅輝しております。
サ`バ`サイド、フロントエンド、ミドルウェア、インフラとI囃わず、戻宛からB?g
廾まで嫌レくやらせてもらっています。その麿OSSのcontributor吉もやっております。
幄塀氏芙 DMM.comラボ
CTO片
CTO片ってなんだ
仟しい室gの函りzみ
冩梢_kとか
室gサポ`トとか
翌試咾箸
どんな並やってるのかの
函りMみのコンセプト
ユ`ザ`の佩咼蹈阿簓牝轡禰`タ、ソ`シャルデ`タからニ`ズ
にマッチした斌瞳?レ御?コンテンツの戻工を佩う
咾衂呂欧臨邊鷸?パイの恷寄晒?Lれた俶勣の条り軟こし
寄トデ`タの盾裂?C亠僥によるデ`タを圷にしたアプロ`チ
念粁
ミッション
やる並
リアルタイムトレント?渇竃敬ひ?zみ喘
リアルタイムレコメンドのおさらい
リアルタイムレコメンドのおさらい
書指麼につかったもの
? Apache Spark
? SparkStreaming
? GraphX
? SparkMLlib
? Apache Solr
Sparkの供、辺h苧
Apache Spark?
互堀に寄トデ`タI尖を佩うエンジン
SparkCoreについてのh苧
Cluster Manager
Yarn
Mesos
Data Source
Stream
HDFS
SparkRDDsのh苧
SparkではRDDsというg了でデ`タをQいます。
val test = sc.textFile("/tmp/sample.txt")
これRDD
test.count このRDDにして
count
take
collect
map
filter
といったI尖を佩います
SparkDStreamのh苧
RDDsのまとまりをDStreamとしてQいます
RDDと貌たv方を隔っていて、和のようなStreamデ`タを
n昼で蛍けてバッチI尖をする碧Mみ
SparkStreamingのWindowOperationh苧
Tweet10
Tweet9
Tweet8
Tweet7
Tweet6
Tweet5
Tweet4
Tweet3
Tweet2
Tweet1
rg
DStream
1昼
DStream
2昼
DStream
3昼
DStream
4昼
DStream
まとめ
Sparkで}になりやすい何蛍
1. 頼畠に蛍柊I尖させるためのハ`ドルが互い┘〒`キテクト議に
1. HDFS、YARNのB?O協がKわってる並念戻
2. Sparkのrequirementだけじゃなくてト`タルで深える駅勣がある
3. メモリR孫个い垢
2. プログラムが恬り仭い
1. Standalonモ`ドだと蛍柊I尖にしたときける並が謹い
2. yarn-clientモ`ドでもdriverをclusterにしたときにけたりする
3. じゃぁ、蛍柊でやれば措いかと冱うと、蛍柊するとデバッグがyしい
SparkStreamingで}になりやすい何蛍
1. 余嶄で峭める圭隈がない
1. 宥械のSparkプログラムとCluster貧での嘛が`う
2. graceful-stopはプログラム箸g廾しないとダメ
3. Streamingの來|貧、デ`タロストの貨を深えないとダメ
4. プログラムの餓し紋えの圭隈を深えないとダメ
5. どうやって\喘するかはn}が表eみ
參貧
Ad

Recommended

ソニ`のディ`プラ`ニングツ`ルで酒gエッジコンピュ`ティング
ソニ`のディ`プラ`ニングツ`ルで酒gエッジコンピュ`ティング
Ryohei Kamiya
?
及匯指 AI Code Review
及匯指 AI Code Review
Ryohei Kamiya
?
禽馨馨に秘芙してみた
禽馨馨に秘芙してみた
DMM.com
?
マイクロサ`ヒ?ス扮旗の強鮫塘佚児檎顎恢霞〜乙看=±
マイクロサ`ヒ?ス扮旗の強鮫塘佚児檎顎恢霞〜乙看=±
DMM.com
?
禽珂珂.界看馨におけるヒ?ック?テ?`タI尖のための皆河晦試喘宝
禽珂珂.界看馨におけるヒ?ック?テ?`タI尖のための皆河晦試喘宝
DMM.com
?
芙坪鬚韻冒軌ゲ`ム恬ってみた
芙坪鬚韻冒軌ゲ`ム恬ってみた
DMM.com
?
20151014 spark茶氏a怎Y創
20151014 spark茶氏a怎Y創
DMM.com
?
It manager in フィリピン
It manager in フィリピン
DMM.com
?
茶氏_タ`ミナルコマント?秘薦儻堀晒_20250620. pptx. .
茶氏_タ`ミナルコマント?秘薦儻堀晒_20250620. pptx. .
iPride Co., Ltd.
?
Protect Your IoT Data with UbiBot's Private Platform.pptx
Protect Your IoT Data with UbiBot's Private Platform.pptx
ユビボット 幄塀氏芙
?
粥鴛室宝慌嗤氏2025-06-05喝禽艶艶沿檎艶壊艶温姻界鞄の尖盾とg樹.沿糸韓
粥鴛室宝慌嗤氏2025-06-05喝禽艶艶沿檎艶壊艶温姻界鞄の尖盾とg樹.沿糸韓
Takuma Oda
?
弼について.pptx .
弼について.pptx .
iPride Co., Ltd.
?
OWASP ASVS5.0 overview 20240607_owaspnagoya
OWASP ASVS5.0 overview 20240607_owaspnagoya
OWASP Nagoya
?
Forguncy 10 u瞳古勣Y創 - ノ`コ`ドWebアプリ_kプラットフォ`ム
Forguncy 10 u瞳古勣Y創 - ノ`コ`ドWebアプリ_kプラットフォ`ム
フォ`ガンシ`
?
PGConf.dev 2025 歌紗レポ`ト (JPUGt氏龝Oセミナ`2025 k燕Y創)
PGConf.dev 2025 歌紗レポ`ト (JPUGt氏龝Oセミナ`2025 k燕Y創)
NTT DATA Technology & Innovation
?
Vibe Codingを兵めよう ?Cursorを箭に、ノ`コ`ドでのプログラミング悶Y?
Vibe Codingを兵めよう ?Cursorを箭に、ノ`コ`ドでのプログラミング悶Y?
iPride Co., Ltd.
?
2024 Trend Updates: What Really Works In SEO & Content Marketing
2024 Trend Updates: What Really Works In SEO & Content Marketing
Search Engine Journal
?
Storytelling For The Web: Integrate Storytelling in your Design Process
Storytelling For The Web: Integrate Storytelling in your Design Process
Chiara Aliotta
?
Artificial Intelligence, Data and Competition C SCHREPEL C June 2024 OECD dis...
Artificial Intelligence, Data and Competition C SCHREPEL C June 2024 OECD dis...
OECD Directorate for Financial and Enterprise Affairs
?
How to Leverage AI to Boost Employee Wellness - Lydia Di Francesco - SocialHR...
How to Leverage AI to Boost Employee Wellness - Lydia Di Francesco - SocialHR...
SocialHRCamp
?
2024 State of Marketing Report C by Hubspot
2024 State of Marketing Report C by Hubspot
Marius Sescu
?
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
Expeed Software
?
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
Pixeldarts
?
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
ThinkNow
?
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
marketingartwork
?
Skeleton Culture Code
Skeleton Culture Code
Skeleton Technologies
?
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
Neil Kimberley
?
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
contently
?
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
Albert Qian
?
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
Kurio // The Social Media Age(ncy)
?

More Related Content

Recently uploaded (8)

茶氏_タ`ミナルコマント?秘薦儻堀晒_20250620. pptx. .
茶氏_タ`ミナルコマント?秘薦儻堀晒_20250620. pptx. .
iPride Co., Ltd.
?
Protect Your IoT Data with UbiBot's Private Platform.pptx
Protect Your IoT Data with UbiBot's Private Platform.pptx
ユビボット 幄塀氏芙
?
粥鴛室宝慌嗤氏2025-06-05喝禽艶艶沿檎艶壊艶温姻界鞄の尖盾とg樹.沿糸韓
粥鴛室宝慌嗤氏2025-06-05喝禽艶艶沿檎艶壊艶温姻界鞄の尖盾とg樹.沿糸韓
Takuma Oda
?
弼について.pptx .
弼について.pptx .
iPride Co., Ltd.
?
OWASP ASVS5.0 overview 20240607_owaspnagoya
OWASP ASVS5.0 overview 20240607_owaspnagoya
OWASP Nagoya
?
Forguncy 10 u瞳古勣Y創 - ノ`コ`ドWebアプリ_kプラットフォ`ム
Forguncy 10 u瞳古勣Y創 - ノ`コ`ドWebアプリ_kプラットフォ`ム
フォ`ガンシ`
?
PGConf.dev 2025 歌紗レポ`ト (JPUGt氏龝Oセミナ`2025 k燕Y創)
PGConf.dev 2025 歌紗レポ`ト (JPUGt氏龝Oセミナ`2025 k燕Y創)
NTT DATA Technology & Innovation
?
Vibe Codingを兵めよう ?Cursorを箭に、ノ`コ`ドでのプログラミング悶Y?
Vibe Codingを兵めよう ?Cursorを箭に、ノ`コ`ドでのプログラミング悶Y?
iPride Co., Ltd.
?
茶氏_タ`ミナルコマント?秘薦儻堀晒_20250620. pptx. .
茶氏_タ`ミナルコマント?秘薦儻堀晒_20250620. pptx. .
iPride Co., Ltd.
?
Protect Your IoT Data with UbiBot's Private Platform.pptx
Protect Your IoT Data with UbiBot's Private Platform.pptx
ユビボット 幄塀氏芙
?
粥鴛室宝慌嗤氏2025-06-05喝禽艶艶沿檎艶壊艶温姻界鞄の尖盾とg樹.沿糸韓
粥鴛室宝慌嗤氏2025-06-05喝禽艶艶沿檎艶壊艶温姻界鞄の尖盾とg樹.沿糸韓
Takuma Oda
?
OWASP ASVS5.0 overview 20240607_owaspnagoya
OWASP ASVS5.0 overview 20240607_owaspnagoya
OWASP Nagoya
?
Forguncy 10 u瞳古勣Y創 - ノ`コ`ドWebアプリ_kプラットフォ`ム
Forguncy 10 u瞳古勣Y創 - ノ`コ`ドWebアプリ_kプラットフォ`ム
フォ`ガンシ`
?
PGConf.dev 2025 歌紗レポ`ト (JPUGt氏龝Oセミナ`2025 k燕Y創)
PGConf.dev 2025 歌紗レポ`ト (JPUGt氏龝Oセミナ`2025 k燕Y創)
NTT DATA Technology & Innovation
?
Vibe Codingを兵めよう ?Cursorを箭に、ノ`コ`ドでのプログラミング悶Y?
Vibe Codingを兵めよう ?Cursorを箭に、ノ`コ`ドでのプログラミング悶Y?
iPride Co., Ltd.
?

Featured (20)

2024 Trend Updates: What Really Works In SEO & Content Marketing
2024 Trend Updates: What Really Works In SEO & Content Marketing
Search Engine Journal
?
Storytelling For The Web: Integrate Storytelling in your Design Process
Storytelling For The Web: Integrate Storytelling in your Design Process
Chiara Aliotta
?
Artificial Intelligence, Data and Competition C SCHREPEL C June 2024 OECD dis...
Artificial Intelligence, Data and Competition C SCHREPEL C June 2024 OECD dis...
OECD Directorate for Financial and Enterprise Affairs
?
How to Leverage AI to Boost Employee Wellness - Lydia Di Francesco - SocialHR...
How to Leverage AI to Boost Employee Wellness - Lydia Di Francesco - SocialHR...
SocialHRCamp
?
2024 State of Marketing Report C by Hubspot
2024 State of Marketing Report C by Hubspot
Marius Sescu
?
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
Expeed Software
?
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
Pixeldarts
?
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
ThinkNow
?
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
marketingartwork
?
Skeleton Culture Code
Skeleton Culture Code
Skeleton Technologies
?
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
Neil Kimberley
?
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
contently
?
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
Albert Qian
?
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
Kurio // The Social Media Age(ncy)
?
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
Search Engine Journal
?
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
SpeakerHub
?
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
Clark Boyd
?
Getting into the tech field. what next
Getting into the tech field. what next
Tessa Mero
?
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Lily Ray
?
How to have difficult conversations
How to have difficult conversations
Rajiv Jayarajah, MAppComm, ACC
?
2024 Trend Updates: What Really Works In SEO & Content Marketing
2024 Trend Updates: What Really Works In SEO & Content Marketing
Search Engine Journal
?
Storytelling For The Web: Integrate Storytelling in your Design Process
Storytelling For The Web: Integrate Storytelling in your Design Process
Chiara Aliotta
?
How to Leverage AI to Boost Employee Wellness - Lydia Di Francesco - SocialHR...
How to Leverage AI to Boost Employee Wellness - Lydia Di Francesco - SocialHR...
SocialHRCamp
?
2024 State of Marketing Report C by Hubspot
2024 State of Marketing Report C by Hubspot
Marius Sescu
?
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
Expeed Software
?
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
Pixeldarts
?
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
ThinkNow
?
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
marketingartwork
?
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
Neil Kimberley
?
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
contently
?
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
Albert Qian
?
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
Search Engine Journal
?
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
SpeakerHub
?
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
Clark Boyd
?
Getting into the tech field. what next
Getting into the tech field. what next
Tessa Mero
?
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Lily Ray
?
Ad

リアルタイムトレント?渇竃敬ひ?zみ喘

Editor's Notes

  • #5: ☆橿忖何蛍はXくh苧
  • #9: ☆ここで馨温沿姻艶糸顎界艶との離いにも冱式しておく