�ݺ�ߣ

Introduction to Data Pipeline at Kakao

Table of Contents
1. What is a Data Pipeline?
2. Architecture
3. Future Work

+ A??? ?????
IT?
Ryan? ??? 7
?? ??(A)? ???.
+A??? ?? ???? ?
?? 7, ??, ????.
+A??? ???? ???? ?:
? ??? 6:4?.
Definition

???(OLAP, Data Warehouse)???(OLTP, Data Hub)
Ryan? ?? ? ????
A
Ryan?
??? ?? ? ???? ??? ??????
IT
Ryan? ??? ?? ? ????? ?? ??
?? ?????? ??? 7, ??, ??, ??,
V20
A? ? ????? ??? ????
30? 90%
IT ????? ??? ? ?? ?? ?? ????
A??.
Ryan? ?? ? ??? ??? ??
?? B
Ryan? ?? ??? ? ?? ?????
?? ?? ???? C
Ryan? ???? ?? ?? ??? D
A? ? ???? ?? ?? ?? ???
B, C
??? 7? ?? ?? ?????? N?, Ryan.
A??? ? ??
? ?? ????? ??? ?????? N?,
Ryan.
Usecase ???

- ???? ??? event??
- ?? ?? 40~50? ?? ??? event??.
- 10? ?? Data pipeline? output? ??.
- ??? ??? ?? ??? ?? ???(meta)? ????? ??, ?? ??, ??? event? join.
- Event?? join??? ?? meta?? 2~3?(Avg).
- ?? ????? ??? ?? ?? ??? ??? ?? ??? 70??.
- ?? data source? ??? ? ?? ??? ??? ??, scalable? ??? ??.
- ?? ????? ???? event? ?? ??? ????? ??, 50? x 30 = 1500??.
- ?? ?? ???? ?? ?? ?? ????? ??? 500??.
- 6???? ????? ???
- ???? ????? ???? ??? data? ??? storage? ????? ??? storage?
???? ??? ???.
- ????? ??? data? ?? ???? unified, scalable? ??? API? ???? ?.
- ?? ?? ??? ??? ??? ??? ??? ?? ??.
Technical Challenges

- Open source? ??? ???? ? ??? ?? ????.
- ?????? ??? ??? ?? ???, ? ???? ????? ??? ????? ???
????? ??.
- ??? ???? ????, ?? ???? ???? iteration? ??? ?? ?. (?? << ???).
- ?? ???? ?? ????.
- ??? ?? ? ?? ??
- ??? ?? Endpoint?? ? ???, ??.
- Data? ??, ??, API? ?? ??? ??.
- ?? ???? ????? ???, ?? ??? ??? ??? ??? ? ??? ??? ???
??? ???.
- ??? ??? ???? ??? ??? ??.
- ???? ???? ????? ???
- ??? ????? ?? ?? ?? -> ??? ?? -> AB??? -> ???? -> ?? ?? ???
???? Iteration.
- Iteration ???? ??? ???? ?????? ?? ??? ??? ?
- Data >> Opinion
Non-Technical Challenges

- ? ??? ???? ??
- ???? ???? ???? ????
- ??? platform ??? ??? ? ??? ??(?? ?????? ??? ??, ??? ??? ?)
- ?? ??? ???
- ????? ??? ???? ?????
- ????? ? ???? ??? ? ?? Input/Outpu format? ?????
- Event?? ?? data?? join???? ?? view? ?? ? ? ??? ?? ?? ??.
- ??? ?? ??? ???!
- ???? ??.
- ?? ???? ?? ???? ???? ??? ??. ?? ??? ?? ??????
- ??? ? ?? Data Product/API? ???? ??? ??.
- ???? ? ????
- ??? ???? ??? ? ??????
- ???? ?? ?????
- ??? raw data? ?????.
- ????? operation? ?? ?? ??? ??? ???? ?? ???, ??? ?? ????
??.
- ??? ?? kafka????, spark ????, ? ??? ????? ??.
Personal Note.

[236] ????????????????????????? ????????

?? ??
?? ?
API??
1. ??? ??
a. ?????? ??? ??? ???? ??.
b. ??? format? ????, ???? ??? endpoint? ??? ?? ??.
c. ??? ??? ?? ??? ??? ??.
2. ??
a. ??? ???, ????, ??? ?????? ?? Fancy? ??.
b. ??? ??? ????, ??? ??.
3. ?? ? API??
a. ??? ???? ????? ??? ??? ? ?? product??? ????, API? ??.
b. ???? ??, adhoc?? ?? ?? ??? ??.
Physical ?? ??: ?? + ?? + ?? + API??

?? ??
?? ?
API??
1. ?? Layer ?? ??
Client
Javascript
IOS
app
Android
app
Web
app
REST
endpoint
Kafka
log_event_sync()
log_event_async()
S2Graph
log_event_sync()
log_event_async()
- ??? service?? Data Pipeline? REST API? ?? data? ingest.
- ? ??? input? ??? ??.
- Synchronous: ingest??? ??? ??? Storage? persist? ??? ?????.
- ?? ??? ?? ???? ???? ??
- Asynchronous: ingest??? ??? ?? ??, asynchronous?? event? persist?.
- ?? activity ???? ???? ??
- Apache Kafka? ??? ??.
- Kafka ?? != Data hub. Kafka ?? != data pipeline.

?? ??
?? ?
API??
1. ?? Layer Solution: S2Event
Client
Javascript
IOS
app
Android
app
Web
app
REST
Endpoint
Kafka
log_event_sync()
log_event_async()
S2Graph
log_event_sync()
log_event_async()
S2Event: REST Endpoint
- Kafka broker? down??? ?? event?? local file? buffering ??? Fallback??.
- Multiple Kafka cluster? publish?? Fanout ??.
- Admin? ?? rule? ???? data? ?? kafka cluster?? publish?? ?? ??.
- Event ????? Metrics ??.
- ?? ???? ?? ???(click??, ??? ??)?? ?? in/out???.
- Synchronous Input
- S2Graph? CRUD
- Asynchronous Input
- Kafka? Publish

?? ??
?? ?
API??
1. ?? Layer Solution: S2Event
Client
Javascript
IOS
app
Android
app
Web
app
REST
Endpoint
Kafka
log_event_sync()
log_event_async()
S2Graph
log_event_sync()
log_event_async()
- ???? ?? Edge(??, ???, ?? ??? ??, extra data) format.
timestamp operation from to label props
now insert Ryan ?? A ??? ??? {��Tags��: [��???7��, ��??"]}
Edge Format: https://steamshon.gitbooks.io/s2graph-book/content/manage_edges.html

2. ?? Layer ?? ??
- Event?? ??? operation?? ???? ??? Event? ??.
- ?? ??? ?? Real-time, Batch ? ??? ??.
- Batch
- ?? ?? ??? ??? ???? ????? ?? ????.
- ??? 70? ??, ?? 500? ?? ???? ????
- ?? ???? ?? similarity matrix?? ???? ??? ??.
- ???? ?? ??? ????? ???? API? ??? ??? ???, ??? ?? ???,
???? ???? ???? ??, ?? Read throughput, latency? ??? ??? ?.
- Real-time
- ??? stream? ??? ETL? ??.
- Input event? ?? ???? ? ???? ?? 2~3? ??.
- event? ????? topology?? 10??? ?? ??.
?? ?? ?? ? API??
Computing Engine
Spark
Streaming MLlib
S2Lambda Bulk Loader
Batch

2. ?? Layer Solution: S2Lambda(Real-time) ?? ??
?? ?
API??
Computing Engine
Spark
Streaming MLlib
Batch
S2Lambda: ETL environment.
- AWS Lambda? ??? ??? spark streaming?? ??.
- Event?? ??? ? event handler(scala function)? ????.
- Source -> Parser -> Flow -> Sink? topology? ??? event handler?? ??? ??.
- Spark Streaming Job? ?? ??? topology?? event?? ??.
- Marathon? ?? Streaming Job HA??.

2. ?? Layer Solution: S2Lambda, Why?
- ???? ?? Spark Streaming job? ????.
- Event handle logic? runtime? tightly coupling.
- Not Composable!
- ????? resource??.
- Streaming Job?? executor? ??? ??.
- Fragmentation!
- ETL process? test case?? ? validation?? ???.
- ???? ??? ?.
- Data? ingest, transform, join, sink? ??? flow? ?? ??? ???.
- IntelliJ?? Spark Streaming Job ?? ?? ???? ???.
- Event??? ??? function??, ?? ??? ?? engine? ??.
- Event processing framework? ???? ?? ??? ?? ?? ?? ?.
- ?? ??? Join.
- ?? ??? Join?? ?? ? ?? cache?? ???(Local ?? >> Remote??).
- ??? JVM??? local cache(RocksDB) instance?? ??.
?? ??
?? ?
API??
Computing Engine
Spark
Streaming MLlib
Batch

2. ?? Layer Solution: S2Lambda ?? ??
?? ?
API??
Computing Engine
Spark
Streaming MLlib
Batch
//Flow example. parse external api route and
//Get data through HTTP Get then merge with input event.
new EtlFlow {
var baseUrl = ""
def init(options: String) = {
baseUrl = (Json.parse(options) "external.api.route").as[String]
}
override def apply: Apply = {
case (event @ StringInput(s), ctx) =>
val tokens = s.split("t")
val articleId = tokens(3)
val uri = baseUrl + articleId
ctx.httpClient.url(uri).get().map { response =>
Seq(InputWithResult(event, response.json, response.status))
}
}
}

2. ?? Layer Solution: S2Lambda ?? ??
?? ?
API??
Computing Engine
Spark
Streaming MLlib
Batch
//Sink Example. parse kafka topic from option parameter and publish to kafka.
new EtlFlow {
var topics = Array.empty[String]
def init(options: String) = {
topics = (Json.parse(options) "kafka.topics").as[String].split(",").map(_.trim)
}
def apply: Apply = {
case (input, ctx) =>
ctx.kafkaProducerOpt.foreach { producer =>
topics.foreach { topic =>
producer.send(new ProducerRecord[String, String](topic, null, input.toString)
}
}
Future.successful(Nil)
}
}

S2Lambda Admin Console
- Write event handler as
scala code.
- Check if the code
compiles.
- Check if the code runs
correctly with an
example input.

2. ?? Layer Solution: S2Lambda, ?? ?? ??
?? ?
API??
Computing Engine
Spark
Streaming MLlib
Batch
- ??? ?? ?? ?? ? ??.
- ?? ?? ?? Event Handler???, ?? ? ?? Parameter? ????? ??/??? ?? ??.
- Event Handler??? Composable!
- Event Handler? Validation?? ??? + ???? DB? ??.
- ?? ???? ??? ?? ???? ??? sample input?? ??? validation?? ? ??.
- ?? ???? test case? ??? ??.
- ?? ???? ??? DB??? ???.
= ??/?? ?? ?? + ??? ??(JVM, classpath, compiler)

2. ?? Layer Solution: S2Loader ?? ??
?? ?
API??
Computing Engine
Spark
Streaming MLlib
Batch
- Production HBase????? ??, ??? ??
?????? Spark Job? ?? ??? HFile???
??? ??.
- HFileOutputFormat2
- ??? HFile? Production HBase ????? Distcp
?, HBase? completebulkload ?? ????
HBase? ???.
Put? ?? ???? ??? ?? Region??? ???
??.
- Memstore??
- WAL??
- ?? compaction
- Region?? GC
Bulk Load? read throughput, latency? ?? ?? ??.
http://blog.cloudera.com/blog/2013/09/how-to-use-hba
se-bulk-loading-and-why/

3. ?? ? API?? ?? ??
?? ??
?? ?
API??
- OLTP
- ?? ???? ????, ?? ???? ??? local graph? traverse?? ???.
- ??? ??? ??. Data Hub? ??.
- ex) Ryan? ? ??? ??? ??.
- ex) A??? ??? ???.
- ex) Ryan? ???? ?? ? ??.
- OLAP
- ???? ???? ?? ?? graph? ?? scan, aggregation? ?? ???.
- ??? ??? ??. Data Warehouse? ??.
- ex) A??? ? ????? ?? ? ??.
- ex) ??? 7??? ?? ????? ???? ???? ???? ?? ???.
Data Hub
(S2Graph)
Data Warehouse
(Druid, Hive)

3. ?? ? API?? Solution: S2Graph(Data Hub)
?? ??
?? ?
API?? Data Hub
(S2Graph)
Data Warehouse
(Druid, Hive)
- Data Hub: Apache S2Graph(incubating), Scalable distributed OLTP Graph Database on HBase.
- OLTP Query? ?? ?? ???. ??, ?? ???? ???? ??? ??? ??.
- Asynchronous Input type??? ??? Synchronous? ???? ???.
- Vertex/Edge? ?? CRUD? ????, ??? Edge?? ?? BFS search traverse? ???.
- ????? ??? cache layer?? ?? ?? ??? ?? ??? ??.
- API? ?? Asynchronous.

3. ?? ? API?? Solution: S2Graph(Data Hub)
?? ??
?? ?
API?? Data Hub
(S2Graph)

3. ?? ? API?? Solution: Druid(Warehouse)
?? ??
?? ?
API?? Data Warehouse
(Druid, Hive)
- Data Warehouse: Druid, Hive
- Realtime Slice & Dice with Druid.
- ??? fix?? ?? ????? interactive?? ??? ???? ??? ??(explore)
- Fix? ??? Hive + Jenkins? ??.
- Dashboard, Report?? ??.
- Long term ???? ?? ???? ??? ??? ????? ??.
- ?? ?? ????? ?? S2Graph(Data Hub)? Bulk Load?? ?????? ???
???? ??.

3. ?? ? API?? Solution: Druid(Warehouse)
?? ??
?? ?
API?? Data Warehouse
(Druid, Hive)

3. ?? ? API??: Summary
?? ??
?? ?
API?? Data Hub
(S2Graph)
Data Warehouse
(Druid, Hive)
- OLTP
- S2Graph Query REST API? ?? Service??? ??? ???? Query.
- ??? Service server?? ?? ??.
- ~70K QPS, ~100ms latency.
- Druid? ??? ????? ???? Druid to S2Graph ETL? ?? S2Graph? ??.
- ???? ?? ?? ?? ?? ??? TopK.
- ??? ?? ?? ?? ?? TopK.
- ?? ??? ??? ?? ?? ??.
- OLAP
- Druid
data? visualize??? Pivot??? UI?? ???? ??? ? ?????? interactive?? slice &
dice?? ? ?? ?? ??.
- ??? ???? ??? ?? ?? ????? ???? admin?? druid REST Query
API? ?? ??.
- 1~10 QPS, ~ 10s latency.

- ?? ???? ???? abstract layer(Property Graph Model)? ????, ???? API(Graph Query)
? ??? ??, service?? ?? API??? ?? ? ??? ??.
- ?? ??? getEdges, getVertices ??? API??? ?? ???? ????? ??.
- ??? Vertex, Edge??, ??? getEdges, getVertices.
- ?? ?? ?? ??, Kafka cluster, spark, HBase?? ?? ?? ??, ?? cost? ?? ?.
- ???? ?? ??? scalable? architecture? ????, ? ?? ???? ??? multi tenancy
? ??? ???? ??? ?? ? ? ?? ??.
- ?? ?? ?? ?? ?? ??? ??? ???? ???? ???? ? ?? ??(??)? ? ? ??
??? ???.
- A/B??? ? Data? ??? ?? ???? ??.
= ?? ??? + ?? ??? ?? + ???? ?? + ?? ?? ?? ?? + ??? ?? ????
Data Pipeline? ?? ?? ?? ???.

- Apache S2Graph(incubating)
- ?? ASF incubating??, ?? ? ??? open source? ?? source? ??? ???.
- ? ?? ??? v0.1.0 release vote?? pass.
- ?? ???
- Apache Tinkerpop initial??.
- OLAP layer? ??.
- ?? Storage ??(Redis, RocksDB, Postgresql, Mysql, Cassandra, ��)
- S2Lambda open source(Apache License V2)
- ??, ???, ??.
- ???? ?? ?? ???? ?? ? ?? ????��(?? ??)??? ???.
- ??? stream ?? framework? ?? ? ? ?? ??.
- ?? ??? ?? ?? ??? ??? ????, ???? ??? ?? Pipeline? ?? ?? ????
??? ????, ?? ???? ?? ?? ???? ?? ??? ?????? ?? ??.
? ??.

�ݺ�ߣ

[236] ????????????????????????? ????????

Recommended

More Related Content

What's hot (20)

Similar to [236] ????????????????????????? ???????? (20)

More from NAVER D2 (20)

[236] ????????????????????????? ????????