際際滷

際際滷Share a Scribd company logo

Data in Motion Tour Seoul 2024 - Roadmap Demo

confluent
confluent

DIMT Seoul 2024 - Roadmap_Demo

1 of 30
Con鍖uent 豕 朱 ろ碁Μ覦 一危 危殊 蟲豢 襦襷 + 磯 譯狩 Team-Lead, Solutions Engineering Con鍖uent
豢 襦襷旧 覲蟆暑 朱襦 蠍一 瑚 蠍磯レ 覲蟆暑 朱, 螳 螻給讌 蟇磯 螻給讌 給. 覲 覲企 蠍磯レ 螻牛蟆る 曙 覃 螻螳 螳ロ 蠍磯レ 蠍磯朱 蟲襷るゼ 蟆一伎 . 讌 豐 蠍讌.
語 覦 襦襷 螳 Kora襯 ろ碁Μ覦 一危 Flink襯 ろ碁 豌襴 + 磯 Stream Governance襯 一危 蟲豢 + 磯 Table鍖ow 觜 + 磯 Q&A
る る誤 伎 ... Kora Engine朱 Kafka襯 觜襯願, 所, 觜 朱 襷れ給! 一危 ろ碁Μ覦 Kafka襯 伎 讌 弰朱 讌給. Con鍖uent 一危 ろ碁Μ覦 弰 一危 朱 一危 螳豺襦 伎.
Con鍖uent 郁屋 80螳 伎 蟯襴 貉るロ Custom Connectors 一危 覈磯 牛 一危 螻 Schema Registry 99.99% SLA ろ碁 郁屋 襦語 蟯襴 讌 12螳螳 Con鍖uent 企殊危 一危 ろ碁Μ覦 弰 蠏殊, , ｸ レ 螻螳 Apache Kafka 蠍磯レ 螳螻 ロ覃伎 TCO襯 豕 60%蟾讌 螳. 企ろ 覦 豢 壱殊伎 企ろ Apache Kafka 螳 螳 SLA レ 99.99% KORA ENGINE 企殊磯襯 蟲豢 Apache Kafka速 讌 蠍襦覯 Schema Linking 殊企 ろ語 一危 Con鍖uent CLI Con鍖uent Platform 一危 ろ碁 豌襴 豪 企欧碁ゼ 觜襯 襦觜 Apache Flink速 Con鍖uent Cloud + 螻 郁屋 貉るロ一 螳
譯殊 轟: Con鍖uent 襦襷 譟一 覦 讌ロ 一危 讌 STREAM CONNECT PROCESS GOVERN Con鍖uent Terraform Provider 一危 Cluster Linking 一危 覦 讌 讌 Zookeeper襯 KRaft Controller襦 蟲豌 Con鍖uent Platform Stream Governance Con鍖uent Connect 蟯襴 貉るロ 誤启Μ Con鍖uent Platform Flink + Flink襯 Gen AI レ煙 Flink API 企殊伎誤 豸 譴 誤 一危 覿襯, 豈 覦 襦朱 Vector DB/Vector Search 碁 + + FedRAMP 語 螻螳 Apache Kafka 蠍磯レ 螳螻 ロ覃伎 TCO襯 豕 60%蟾讌 螳. Freight 企ろ TableFlow
蟯襴 (Govern)
"Con鍖uent Stream Governance 蟲一 れ螳朱 讌企 一危(Data in Motion) 蠍一 譴豢 蟆所 蟲豢 譴 蟆. Stream Catalog 覦 Stream Lineage 觜 蠍磯レ 牛 螳 譴 一危磯ゼ る 襦 覲企 螳螻 螳 蟆." 一危 ろ碁Μ覦 狩 蟯襴 蟇磯 蟲 Stream Quality Stream Catalog Stream Lineage `
覲願 危ｴ覲願鍵 蟲豢 1 誤 覲願鍵 ′語 豌 2 3 4 Con鍖uent Cloud 螻螳 一危 語 れ螳 一危 蟆, 覦 蟲豢 給 New
"schemaType": "AVRO", "schema": "{ "type": "record", "name":"CustomerRecord", "fields":[{ "name":"customer_ssn", "type":"string"] }]}" "metadata": { "tags": { "**.ssn": [ "PII"] }, "properties": { "owner": "Bob Jones", "email": "bob@acme.com" } } "ruleSet": { "domainRules": [{ "name": "checkSsnLen", "kind": "CONDITION", "type": "CEL", "mode": "WRITE", "expr": "size(message.customer_ssn) == 9", "onFailure": "DLQ" }] } New 一危 螻 蟲譟 Schema 覓願屋 譟郁唄 Data Quality Rules Metadata 蠏, KV New 豈 譴 誤 讌 襷願係伎 蠏豺 New { "schemaType": "...", "schema": "...", "metadata": { "tags": {...}, "properties": {...} }, "ruleSet": { "domainRules": [...] } } Data Quality Rules CP 覦 CC 覦 蟇語 一危 螻 襴 覦 New New
螳ｸ蟆 覲伎 螻 覩手 一危磯ゼ 3(Con鍖uent ) 襦覿 覲危 覩手 一危一 ′語 蟠 轟 螳語蟆 朱 覿 Infosec/蠏 譴 蟲 蟆 豢譟 Schema Registry Producer Kafka 讌蠍 Kafka 讌蠍 Consumer Kafka 6. 誤 螳 覃讌 9. 誤 螳 覃讌 伎 7. ろる 覦 豈 螳語り鍵 8. 襷 覦 豈 覦 3. ろる 覦 豈 螳語り鍵 4. 襷 覦 豈 覦 KMS 10. 企 1. ろる 覦 豈 2. 襷ろ れ ′語 蟠 覿 5. 誤 企殊伎誤 豸 譴 誤 New
ろる 蟾 * UI, CLI API襯 牛 蠏 蠍一ヾ ろる * 螻螳 Con鍖uent 襷ろ れ ′語 蟠 覿 螳 誤 豈 (*) Kafka Serializer襦 螻螳 襷ろ 蟲 覦 1 2 3 { ... { "name": "customerName", "type": "string", "confluent.tags": [ "PII", "OII"] } ] } 企殊伎誤 (AWS ): rule.executors=aws rule.executors.aws.class=io.con鍖uent .kafka.schemaregistry.encryption.aws .AwsFieldEncryptionExecutor Serializer: KafkaAvroSerializer KafkaProtobufSerializer KafkaJsonSchemaSerializer { ... "ruleSet": { "domainRules": [ { "name": "encryptPII", "type": "ENCRYPT", "tags": ["PII"], "params": { "encrypt.kek.name": "mykek" } } ] } } , 企殊伎誤 豸 譴 誤 New
Import a new KEK
Add encryption rule
蟆, 譟壱 覦
蟠 螻 蟆 豌
豌襴 (Process)
Blob ろ襴讌 一危磯伎 一危 危一 一危磯伎 SaaS 讌 螳 讌ъ 覦 一危 危殊 手 螻 螳ロ 一危 豕 襴 企 譴 一危 ろ碁殊 豌襴 ろ 螳レ, 豢る 覦 企 蠏豪
蟯襴 伎 覿 讀 覯襴 SaaS 蠍磯 蟆曙 牛 Flink 襴貅伎 所 螳覦 給. レ レ 螻殊 襦觜 螳 覲旧″ 襦 蟲 襷豢 朱 豢螳 螳ロ. 蠍磯 螻手 襦 蠍覿 襦觜 誤手 襴れ 伎襷 觜 讌覿 蠍 . 讌 誤一 一危 語企 覯 API 誤壱伎るゼ 螳豢 豕 襦 讌 弰 蟲豢. 螳 蟆所骸 磯ジ 豌襴 "Apache Flink Apache Kafka襯 蟷 覃 一危磯ゼ 螻 譴覲 れ伎ろ碁 豌襴襯 狩 給. Flink Kafka襯 蟯襴 觜る 螻牛覃 覲旧″ 誤 蟯襴 ろ碁 豌襴螳 螳ロ覩襦 觜讀るゼ 谿覲 れ螳 ろ碁Μ覦 襴貅伎螻 危殊 蟲豢 讌譴 給." 覈 蠏覈 螻焔レ ろ碁 豌襴 讌
C 螳蟆>讌(螳蟆) D 螳蟆<讌(螳蟆) C price>lag(price) B price<lag(price) A 觚 覦 蠍郁 覦 蟇磯 蠍 れ 螳 讌 覦螻 一危 蠏碁９ 覦 碁蟆 危 襴貅伎 螳覦: 誤, 觚襷, 語 OVER 讌螻 MATCH_RECOGNIZE襯 襷れ広 企欧 蠍磯 襴貅伎 伎 語 企欧語
れ 覲企 讌 一危瑚 襴貅伎 豕 蟆郁骸 螻: 蟲豌危 覲願鍵 螳 覿 貎朱Μ 螻 A $15 B $2 C $15 螻 A, +$10 螻 B, +$12 螻 C, +$5 螻 B, -$10 螻 C, +$10 螻 A, -$5 Account A, +$10 螳 れ螳 覿 れ螳 一危 ろ碁殊 覿 譴 觜讀 語危
れ 螳豺 覿螻 襦 ろ碁 : 一危 覲 ろ碁Μ覦 譟一, 譟一, 譟壱 譟一, 覯 譟一 貎朱Μ 覿, れ 企ろ 貎朱Μ 22 t1, 21.5 USD t3, 55 EUR t5, 35.3 EUR t0, EUR:USD=1.00 t2, EUR:USD=1.05 t4: EUR:USD=1.10 t1, 21.5 USD t3, 57.75 USD t5, 38.83 USD 譯朱 ろ碁Μ覦 一危 危殊 ろ碁Μ覦 一危 危殊語 蟲豢 れ螳 蟆一 覲 螻
Data in Motion Tour Seoul 2024 - Roadmap Demo
INSERT INTO enriched_reviews SELECT id, review, invoke_openai(prompt,review) as score FROM product_reviews; K N B Kate 4螳 豕 蟆一伎給. Nikola 1 讌 . 危る 譬 . Brian 3 襷 譬! 覦 ! K N B Kate 4 螳 This was the worst decision ever. Nikola 1 Not bad. Could have been cheaper. Brian 3 Amazing! Game Changer! 襦 "れ review ろ碁ゼ 1螻 5 豌襦 襯 襷り襖譴. 1 覿願 5螳 蠍伎. 襷 覦伎" 一危 ろ碁Μ覦 燕 AI襦 Flink SQL 覦襦 れ螳 一危 ろ碁 螳
Table鍖ow Vision (ICEBERG) ロ
STREAM 讌 譴觜 Parquet朱 覲 ろる 讌 覲 豢 Data quality rules 豺危襦蠏語 覃一危 蠍壱 讌 襦 る 覦 螻 企 觜讀る 蠏豺 覦 朱Μ CDC 蟲豌危 譴覲 蟇 磯 觚襦讀() 企 螳豌 S3 GCS ABS
ろ碁 + 讌 + 譴觜 螻 碁 豺危襦蠏 覃一危一 讌 ′語 覦襦 螳ロ Iceberg 企 Con鍖uent Cloud Convert to parquet Schema evolution 襷ろ/覲 CDC materialization 豢 一危 讌/蠏豺 (SR) 覃一危磯ゼ 豺危襦蠏語 蠍壱 螻 讌 Bronze Silver Gold
Data in Motion Tour Seoul 2024 - Roadmap Demo
PROCESS GOVERN STREAM 螳 伎願 襦 牛 一危 ろ碁Μ覦 貉覯る 一危 伎伎 企伎覃, 一危磯 Kora Iceberg襦 企. 一危 企 蟆 危一れ 讀螳 蟲譟磯ゼ 螻牛. Con鍖uent Flink 讌 Kafka Iceberg 一危磯ゼ 牛.
Data in Motion Tour Seoul 2024 - Roadmap Demo

More Related Content

Data in Motion Tour Seoul 2024 - Roadmap Demo

  • 1. Con鍖uent 豕 朱 ろ碁Μ覦 一危 危殊 蟲豢 襦襷 + 磯 譯狩 Team-Lead, Solutions Engineering Con鍖uent
  • 2. 豢 襦襷旧 覲蟆暑 朱襦 蠍一 瑚 蠍磯レ 覲蟆暑 朱, 螳 螻給讌 蟇磯 螻給讌 給. 覲 覲企 蠍磯レ 螻牛蟆る 曙 覃 螻螳 螳ロ 蠍磯レ 蠍磯朱 蟲襷るゼ 蟆一伎 . 讌 豐 蠍讌.
  • 3. 語 覦 襦襷 螳 Kora襯 ろ碁Μ覦 一危 Flink襯 ろ碁 豌襴 + 磯 Stream Governance襯 一危 蟲豢 + 磯 Table鍖ow 觜 + 磯 Q&A
  • 4. る る誤 伎 ... Kora Engine朱 Kafka襯 觜襯願, 所, 觜 朱 襷れ給! 一危 ろ碁Μ覦 Kafka襯 伎 讌 弰朱 讌給. Con鍖uent 一危 ろ碁Μ覦 弰 一危 朱 一危 螳豺襦 伎.
  • 5. Con鍖uent 郁屋 80螳 伎 蟯襴 貉るロ Custom Connectors 一危 覈磯 牛 一危 螻 Schema Registry 99.99% SLA ろ碁 郁屋 襦語 蟯襴 讌 12螳螳 Con鍖uent 企殊危 一危 ろ碁Μ覦 弰 蠏殊, , ｸ レ 螻螳 Apache Kafka 蠍磯レ 螳螻 ロ覃伎 TCO襯 豕 60%蟾讌 螳. 企ろ 覦 豢 壱殊伎 企ろ Apache Kafka 螳 螳 SLA レ 99.99% KORA ENGINE 企殊磯襯 蟲豢 Apache Kafka速 讌 蠍襦覯 Schema Linking 殊企 ろ語 一危 Con鍖uent CLI Con鍖uent Platform 一危 ろ碁 豌襴 豪 企欧碁ゼ 觜襯 襦觜 Apache Flink速 Con鍖uent Cloud + 螻 郁屋 貉るロ一 螳
  • 6. 譯殊 轟: Con鍖uent 襦襷 譟一 覦 讌ロ 一危 讌 STREAM CONNECT PROCESS GOVERN Con鍖uent Terraform Provider 一危 Cluster Linking 一危 覦 讌 讌 Zookeeper襯 KRaft Controller襦 蟲豌 Con鍖uent Platform Stream Governance Con鍖uent Connect 蟯襴 貉るロ 誤启Μ Con鍖uent Platform Flink + Flink襯 Gen AI レ煙 Flink API 企殊伎誤 豸 譴 誤 一危 覿襯, 豈 覦 襦朱 Vector DB/Vector Search 碁 + + FedRAMP 語 螻螳 Apache Kafka 蠍磯レ 螳螻 ロ覃伎 TCO襯 豕 60%蟾讌 螳. Freight 企ろ TableFlow
  • 8. "Con鍖uent Stream Governance 蟲一 れ螳朱 讌企 一危(Data in Motion) 蠍一 譴豢 蟆所 蟲豢 譴 蟆. Stream Catalog 覦 Stream Lineage 觜 蠍磯レ 牛 螳 譴 一危磯ゼ る 襦 覲企 螳螻 螳 蟆." 一危 ろ碁Μ覦 狩 蟯襴 蟇磯 蟲 Stream Quality Stream Catalog Stream Lineage `
  • 9. 覲願 危ｴ覲願鍵 蟲豢 1 誤 覲願鍵 ′語 豌 2 3 4 Con鍖uent Cloud 螻螳 一危 語 れ螳 一危 蟆, 覦 蟲豢 給 New
  • 10. "schemaType": "AVRO", "schema": "{ "type": "record", "name":"CustomerRecord", "fields":[{ "name":"customer_ssn", "type":"string"] }]}" "metadata": { "tags": { "**.ssn": [ "PII"] }, "properties": { "owner": "Bob Jones", "email": "bob@acme.com" } } "ruleSet": { "domainRules": [{ "name": "checkSsnLen", "kind": "CONDITION", "type": "CEL", "mode": "WRITE", "expr": "size(message.customer_ssn) == 9", "onFailure": "DLQ" }] } New 一危 螻 蟲譟 Schema 覓願屋 譟郁唄 Data Quality Rules Metadata 蠏, KV New 豈 譴 誤 讌 襷願係伎 蠏豺 New { "schemaType": "...", "schema": "...", "metadata": { "tags": {...}, "properties": {...} }, "ruleSet": { "domainRules": [...] } } Data Quality Rules CP 覦 CC 覦 蟇語 一危 螻 襴 覦 New New
  • 11. 螳ｸ蟆 覲伎 螻 覩手 一危磯ゼ 3(Con鍖uent ) 襦覿 覲危 覩手 一危一 ′語 蟠 轟 螳語蟆 朱 覿 Infosec/蠏 譴 蟲 蟆 豢譟 Schema Registry Producer Kafka 讌蠍 Kafka 讌蠍 Consumer Kafka 6. 誤 螳 覃讌 9. 誤 螳 覃讌 伎 7. ろる 覦 豈 螳語り鍵 8. 襷 覦 豈 覦 3. ろる 覦 豈 螳語り鍵 4. 襷 覦 豈 覦 KMS 10. 企 1. ろる 覦 豈 2. 襷ろ れ ′語 蟠 覿 5. 誤 企殊伎誤 豸 譴 誤 New
  • 12. ろる 蟾 * UI, CLI API襯 牛 蠏 蠍一ヾ ろる * 螻螳 Con鍖uent 襷ろ れ ′語 蟠 覿 螳 誤 豈 (*) Kafka Serializer襦 螻螳 襷ろ 蟲 覦 1 2 3 { ... { "name": "customerName", "type": "string", "confluent.tags": [ "PII", "OII"] } ] } 企殊伎誤 (AWS ): rule.executors=aws rule.executors.aws.class=io.con鍖uent .kafka.schemaregistry.encryption.aws .AwsFieldEncryptionExecutor Serializer: KafkaAvroSerializer KafkaProtobufSerializer KafkaJsonSchemaSerializer { ... "ruleSet": { "domainRules": [ { "name": "encryptPII", "type": "ENCRYPT", "tags": ["PII"], "params": { "encrypt.kek.name": "mykek" } } ] } } , 企殊伎誤 豸 譴 誤 New
  • 18. Blob ろ襴讌 一危磯伎 一危 危一 一危磯伎 SaaS 讌 螳 讌ъ 覦 一危 危殊 手 螻 螳ロ 一危 豕 襴 企 譴 一危 ろ碁殊 豌襴 ろ 螳レ, 豢る 覦 企 蠏豪
  • 19. 蟯襴 伎 覿 讀 覯襴 SaaS 蠍磯 蟆曙 牛 Flink 襴貅伎 所 螳覦 給. レ レ 螻殊 襦觜 螳 覲旧″ 襦 蟲 襷豢 朱 豢螳 螳ロ. 蠍磯 螻手 襦 蠍覿 襦觜 誤手 襴れ 伎襷 觜 讌覿 蠍 . 讌 誤一 一危 語企 覯 API 誤壱伎るゼ 螳豢 豕 襦 讌 弰 蟲豢. 螳 蟆所骸 磯ジ 豌襴 "Apache Flink Apache Kafka襯 蟷 覃 一危磯ゼ 螻 譴覲 れ伎ろ碁 豌襴襯 狩 給. Flink Kafka襯 蟯襴 觜る 螻牛覃 覲旧″ 誤 蟯襴 ろ碁 豌襴螳 螳ロ覩襦 觜讀るゼ 谿覲 れ螳 ろ碁Μ覦 襴貅伎螻 危殊 蟲豢 讌譴 給." 覈 蠏覈 螻焔レ ろ碁 豌襴 讌
  • 20. C 螳蟆>讌(螳蟆) D 螳蟆<讌(螳蟆) C price>lag(price) B price<lag(price) A 觚 覦 蠍郁 覦 蟇磯 蠍 れ 螳 讌 覦螻 一危 蠏碁９ 覦 碁蟆 危 襴貅伎 螳覦: 誤, 觚襷, 語 OVER 讌螻 MATCH_RECOGNIZE襯 襷れ広 企欧 蠍磯 襴貅伎 伎 語 企欧語
  • 21. れ 覲企 讌 一危瑚 襴貅伎 豕 蟆郁骸 螻: 蟲豌危 覲願鍵 螳 覿 貎朱Μ 螻 A $15 B $2 C $15 螻 A, +$10 螻 B, +$12 螻 C, +$5 螻 B, -$10 螻 C, +$10 螻 A, -$5 Account A, +$10 螳 れ螳 覿 れ螳 一危 ろ碁殊 覿 譴 觜讀 語危
  • 22. れ 螳豺 覿螻 襦 ろ碁 : 一危 覲 ろ碁Μ覦 譟一, 譟一, 譟壱 譟一, 覯 譟一 貎朱Μ 覿, れ 企ろ 貎朱Μ 22 t1, 21.5 USD t3, 55 EUR t5, 35.3 EUR t0, EUR:USD=1.00 t2, EUR:USD=1.05 t4: EUR:USD=1.10 t1, 21.5 USD t3, 57.75 USD t5, 38.83 USD 譯朱 ろ碁Μ覦 一危 危殊 ろ碁Μ覦 一危 危殊語 蟲豢 れ螳 蟆一 覲 螻
  • 24. INSERT INTO enriched_reviews SELECT id, review, invoke_openai(prompt,review) as score FROM product_reviews; K N B Kate 4螳 豕 蟆一伎給. Nikola 1 讌 . 危る 譬 . Brian 3 襷 譬! 覦 ! K N B Kate 4 螳 This was the worst decision ever. Nikola 1 Not bad. Could have been cheaper. Brian 3 Amazing! Game Changer! 襦 "れ review ろ碁ゼ 1螻 5 豌襦 襯 襷り襖譴. 1 覿願 5螳 蠍伎. 襷 覦伎" 一危 ろ碁Μ覦 燕 AI襦 Flink SQL 覦襦 れ螳 一危 ろ碁 螳
  • 26. STREAM 讌 譴觜 Parquet朱 覲 ろる 讌 覲 豢 Data quality rules 豺危襦蠏語 覃一危 蠍壱 讌 襦 る 覦 螻 企 觜讀る 蠏豺 覦 朱Μ CDC 蟲豌危 譴覲 蟇 磯 觚襦讀() 企 螳豌 S3 GCS ABS
  • 27. ろ碁 + 讌 + 譴觜 螻 碁 豺危襦蠏 覃一危一 讌 ′語 覦襦 螳ロ Iceberg 企 Con鍖uent Cloud Convert to parquet Schema evolution 襷ろ/覲 CDC materialization 豢 一危 讌/蠏豺 (SR) 覃一危磯ゼ 豺危襦蠏語 蠍壱 螻 讌 Bronze Silver Gold
  • 29. PROCESS GOVERN STREAM 螳 伎願 襦 牛 一危 ろ碁Μ覦 貉覯る 一危 伎伎 企伎覃, 一危磯 Kora Iceberg襦 企. 一危 企 蟆 危一れ 讀螳 蟲譟磯ゼ 螻牛. Con鍖uent Flink 讌 Kafka Iceberg 一危磯ゼ 牛.