Călin Andrei Burloiu - Connecting Hadoop with Couchbase: Engineering for performance
•
0 likes•654 views
We needed a bridge between the real-time tier, where we used Couchbase, and the batch tier, built on Hadoop. For lack of something better, we built our own: Couchdoop – an open-source Hadoop connector for Couchbase. Our presentation will discuss best practices on how to create a Hadoop connector for a NoSQL database. We will talk about the challenges we encountered while developing Couchdoop and share how we tuned it for performance. Together with Bigstep we worked on performance benchmarks for our technology, which show how much throughput that can be squeezed from a Hadoop connector.
![{
“user”: “Rudy”,
“recommendations”: [
[“Ibanez Acoustic Guitar”,
450],
[“Guitar Tuner”, 120],
[“Sound Mixer”, 30]
]
}
E
X
P
O
R
T
Exporting Data
Couchdoop
Machine
Learning
Recommenda0ons
Hadoop](https://image.slidesharecdn.com/o3qwrp8bsboumb95pmmm-signature-b74b6da01f3d030d5fe8fcd10c4c17f2b0e8c801933c00bd363e808c05fe50eb-poli-141003122243-phpapp01/85/Calin-Andrei-Burloiu-Connecting-Hadoop-with-Couchbase-Engineering-for-performance-17-320.jpg)
![{
“user”: “Rudy”,
“recommendations”: [
[“Ibanez Acoustic Guitar”,
450],
[“Guitar Tuner”, 120],
[“Sound Mixer”, 30]
]
}
U
p
d
a
t
e
Updating Data
Couchdoop
Machine
Learning
Recommenda0ons
Hadoop](https://image.slidesharecdn.com/o3qwrp8bsboumb95pmmm-signature-b74b6da01f3d030d5fe8fcd10c4c17f2b0e8c801933c00bd363e808c05fe50eb-poli-141003122243-phpapp01/85/Calin-Andrei-Burloiu-Connecting-Hadoop-with-Couchbase-Engineering-for-performance-18-320.jpg)
Călin Andrei Burloiu - Connecting Hadoop with Couchbase: Engineering for performance
- 11. Two-tier Architecture Real-time Tier (Couchbase) • Detects user intent • Gives next best recommendation or deal Data Bridge (Couchdoop) Batch Tier (Hadoop) • Recommends products User events Recommendations
- 16. Importing Data { “user”: “Rudy”, “action”: “view”, “product”: “Fender Guitar” } { “user”: “Rudy”, “action”: “click”, “product”: “Guitar Amplifier” } { “user”: “Emma”, “action”: “buy”, “product”: “Blue Skirt” } Couchdoop Machine Learning Recommenda0ons Hadoop IMPORT HDFS
- 17. { “user”: “Rudy”, “recommendations”: [ [“Ibanez Acoustic Guitar”, 450], [“Guitar Tuner”, 120], [“Sound Mixer”, 30] ] } E X P O R T Exporting Data Couchdoop Machine Learning Recommenda0ons Hadoop
- 18. { “user”: “Rudy”, “recommendations”: [ [“Ibanez Acoustic Guitar”, 450], [“Guitar Tuner”, 120], [“Sound Mixer”, 30] ] } U p d a t e Updating Data Couchdoop Machine Learning Recommenda0ons Hadoop