How we built an event-time merge of two kafka-streams with spark-streaming

Download as pptx, pdf

1 like2,580 views

The document outlines the construction of an event-time merge of two Kafka streams using Spark Streaming by a team from Otto GmbH & Co. KG, addressing challenges like merging messages by key, managing timeouts, and handling variable request rates. Key solutions involve using the 'updatestatebykey' function for message merging and implementing a custom Kafka stream to handle different request rates while ensuring at-least-once semantics. Lessons learned highlight the system's performance and scalability, as well as complexities in the driver/executor model and limitations around checkpointing.

Software

How We Built an Event-Time
Merge of Two Kafka-Streams
with Spark Streaming
Ralf Sigmund, Sebastian Schröder
Otto GmbH & Co. KG

Agenda
• Who are we?
• Our Setup
• The Problem
•Our Approach
• Requirements
• Advanced Requirements
• Lessons learned

Who are we?
• Ralf
–Technical Designer Team Tracking
–Cycling, Surfing
–Twitter: @sistar_hh
• Sebastian
–Developer Team Tracking
–Taekwondo, Cycling
–Twitter: @Sebasti0n

Who are we working for?
• 2nd largest ecommerce company in Germany
• more than 1 million visitors every day
• our blog: https://dev.otto.de
• our jobs: www.otto.de/jobs

Our Setup
otto.de
Varnish Reverse
Proxy
Tracking-Server
Vertical 1Vertical 0
Vertical n
Service 0
Service 1
Service n

The Problem
otto.de
Varnish Reverse
Proxy
Vertical 1Vertical 0
Vertical n
Service 0
Service 1
Service n
Has a limit on the
size of messages
Needs to send
large tracking
messages
Tracking-Server

Our Approach
otto.de
Varnish Reverse
Proxy
Tracking-Server
Vertical 1Vertical 0
Vertical n
S0
S1
Sn
Spark
Service

Messages with the same key are
merged
t: 1
id: 1
t: 3
id: 2
t: 2
id: 3
t: 0
id: 2
t: 4
id: 0
t: 2
id: 3
t: 1
id: 1
t: 0
id: 2
t: 3
id: 2
t: 2
id: 3
t: 2
id: 3 t: 4
id: 0

Messages have to be timed out
t: 1
id: 1
t: 3
id: 2
t: 2
id: 3
t: 0
id: 2
t: 4
id: 0
t: 2
id: 3
t: 1
id: 1
t: 0
id: 2
t: 3
id: 2
t: 2
id: 3
t: 2
id: 3 t: 4
id: 0
max(business event timestamp)
time-out after 3 sec

Event Streams might get stuck
t: 1
id: 1
t: 3
id: 2
t: 2
id: 3
t: 0
id: 2
t: 4
id: 0
t: 2
id: 3
t: 1
id: 1
t: 0
id: 2
t: 3
id: 2
t: 2
id: 3
t: 2
id: 3 t: 4
id: 0
clock A
clock B
clock: min(clock A, clock B)

There might be no messages
t: 1
id: 1
t: 4
id: 2
t: 2
id: 3
t: 5
heart
beat
t: 1
id: 1
t: 4
id: 2
t: 2
id: 3
time-out after 3 sec
t: 5
heart
beat
clock: min(clock A, clock B) : 4

Requirements Summary
• Messages with the same key are merged
• Messages are timed out by the combined event
time of both source topics
• Timed out messages are sorted by event time

Merge messages with the same Id
•UpdateStateByKey, which is defined on pairwise
RDDs
•It applies a function to all new messages and
existing state for a given key
•Returns a StateDStream with the resulting
messages

UpdateStateByKey
1
2
3
1
2
3
1
2
3
UpdateFunction:
(Seq[InputMessage], Option[MergedMessage]) =>
Option[MergedMessage]

Flag And Remove Timed Out Msgs
1
MM
TO
2
2
3
MM
MM
TO
MM
TO
2
3
1 MM
1 MM
3
needs global
Business
Clock
UpdateFunction

Messages are timed out by event time
•We need access to all messages from the
current micro batch and the history (state) =>
Custom StateDStream
•Compute the maximum event time of both topics
and supply it to the updateStateByKey function

topicBtopicA Effect of different request rates in
source topics
10K
5K
0
10K
5K
0
catching up
with 5K per
microbatch
data read too
early

Handle different request rates in topics
•Stop reading from one topic if the event time of
other topic is too far behind
•Store latest event time of each topic and
partition on driver
•Use custom KafkaDStream with clamp method
which uses the event time

Handle different request rates in topics

Guarantee at least once semantics
o: 1
t: a
o: 7
t: a
o: 2
t: a
o: 5
t: b
o: 9
t: b
t: 6
t: b
Current State Sink Topic
a-offset: 1
b-offset: 5
a-offset: 1
b-offset: 5
a-offset: 1
b-offset: 5
On Restart/Recovery
Retrieve last message
Timeout
Spark Merge Service

Lessons learned
• Excellent Performance and Scalability
• Extensible via Custom RDDs and Extension to
DirectKafka
•No event time windows in Spark Streaming
See: https://github.com/apache/spark/pull/2633
• Checkpointing cannot be used when deploying new
artifact versions
• Driver/executor model is powerful, but also complex

THANK YOU.
Twitter:
@sistar_hh
@Sebasti0n
Blog: dev.otto.de
Jobs: otto.de/jobs

The old process for the We Want You Back email campaign involved running over 1500 lines of hardcoded SQL scripts against the production database over 5 days each month. This process was risky, time-consuming, and costly. The new process uses SSIS packages and stored procedures to automate data extraction, transformation, and loading into a separate email database in just 3 hours each month. It reduces costs by over 90% and eliminates points of failure from the old process.

Processing messages in a sqs with lambda functionSubhamay Bhattacharyya

��

This document outlines a personal project focused on processing SQS messages with AWS Lambda and handling failed messages using a dead letter queue. The project involves creating a complete stack utilizing CloudFormation, which includes a main SQS, a dead letter queue, and a Lambda function with specific execution roles. Additionally, it provides a link to the CloudFormation template and describes the process of sending messages to SQS, monitoring with CloudWatch, and cleaning up resources after testing.

Observer, a "real life" time series applicationKévin LOVATO

��

The document provides an overview of the Observer time-series application developed by Kévin Lovato, detailing its architecture, key features, and CQL schema for data management. It discusses various components like metrics aggregation, alerting mechanisms, schema best practices, and performance optimization strategies. Additionally, it outlines future plans including transitioning to SSDs and using different data storage configurations.

Infrastructure as code with troposphere on aws in 5 minMathias Mårtens

��

The document discusses using Troposphere to implement infrastructure as code on AWS. Some key points: Troposphere allows describing AWS resources like CloudFormation templates in Python code for easier version control and code reviews compared to JSON/YAML templates. It generates CloudFormation templates without requiring provisioning. Troposphere supports features like extended template classes, cross-stack references, and tooling to help with conformity, naming, tagging, and other aspects. It can help decouple dependencies and make infrastructure migration and blue/green deployments easier. The document emphasizes testing infrastructure code for reliability, and building infrastructure that is designed to handle failures and change over time.

Should you read Kafka as a stream or in batch? Should you even care? | Ido Na...HostedbyConfluent

��

This document discusses whether it is better to process data using a stream or batch approach. It describes how one company evolved their data pipeline from a micro-batch streaming process to a batch approach. The streaming process was very expensive, costing $400,000 per year to run. It also had issues with wasted resources during idle times, slow processing during bursts of data, and long recovery times from outages. The company rearchitected the process to use discrete time windows run in isolated batch jobs. This new batch approach reduced costs by 60% to $160,000 per year and improved processing efficiency and outage recovery times.

Building your own Distributed System The easy way - Cassandra Summit EU 2014Kévin LOVATO

��

This document presents an overview of building a distributed system using Cassandra, focusing on service bus architecture and subscription management. It discusses the design requirements for directory servers and the management of subscriptions, including static and dynamic methods, as well as timestamps for handling out-of-order messages. Key takeaways emphasize the importance of schema design, unique timestamp generation, and considerations of data types across programming languages.

Distributed Kafka Architecture Taboola ScaleApache Kafka TLV

��

The document discusses Taboola's scalable software infrastructure accommodating over 2.4 billion page views per day, 400,000 HTTP requests per second, and 1.5 billion unique monthly users. It highlights the architecture's principles, like stateless applications, real-time data processing, and the use of Kafka for message distribution, while ensuring data accuracy and service continuity even during traffic spikes. Investing in best-of-breed technologies and custom development enables efficient handling of significant data volume and agile development cycles.

I Heart Log: Real-time Data and Apache KafkaJay Kreps

��

The document discusses Apache Kafka, a distributed event streaming platform, emphasizing its characteristics such as high throughput, scalability, and low-latency performance. It highlights its implementation at LinkedIn, detailing the substantial volume of log data managed and the integration with various data systems. Stream processing and data integration using Kafka are also presented as critical use cases.

Spark Summit EU talk by Sebastian Schroeder and Ralf SigmundSpark Summit

��

Otto needed to merge large tracking messages from two Kafka streams with different throughput into a single stream while preserving event timestamps and ensuring at least once delivery. They used Spark Streaming to build a custom event-time merge that used UpdateStateByKey to merge messages, tracked event times to timeout messages, handled uneven throughput, and stored offsets to guarantee at least once delivery. The solution provided excellent performance and scalability while being extensible through custom RDDs.

NoLambda: Combining Streaming, Ad-Hoc, Machine Learning and Batch AnalysisHelena Edelson

��

The document discusses Tuplejump's integration of streaming, ad-hoc, machine learning, and batch analytics through their technologies, particularly in the context of using Apache Spark and Cassandra. It highlights the challenges of building scalable, fault-tolerant systems capable of handling massive data and emphasizes the advantages of their Filodb database for optimizing analytics and storage solutions. Key use cases include real-time processing and the benefits of a unified streaming architecture over traditional Lambda architectures, addressing issues of performance, complexity, and data consistency.

Leveraging Azure Databricks to minimize time to insight by combining Batch an...Microsoft Tech Community

��

The document outlines a collaborative analytics platform based on Apache Spark optimized for Azure, specifically Databricks, featuring streamlined workflows for data scientists and analysts. It emphasizes integration with various Azure services and capabilities for real-time data processing, including event-time processing and exactly-once guarantees. The platform's advanced functionalities include support for complex queries, stateful processing, and built-in support for diverse data formats and sources.

Unbounded bounded-data-strangeloop-2016-monal-daxiniMonal Daxini

��

The document is a comprehensive overview of unbounded data processing systems, emphasizing the distinctions between streaming and batch processing, as well as the key trade-offs in choosing processing engines based on latency, accuracy, and cost. It details various concepts such as event time, windowing, watermarks, and handling of late data, while presenting Netflix's Keystone stream processing architecture and its evolution over time. Additionally, it highlights future plans for Stream Processing as a Service (SPaaS) and ongoing improvements in system performance.

Spark Streaming and IoT by Mike FreedmanSpark Summit

��

This document discusses using Spark Streaming for IoT applications and the challenges involved. It notes that while Spark simplifies programming across different processing intervals from batch to stream, programming models alone are not sufficient as IoT data streams can have varying rates and delays. It proposes a unified data infrastructure with abstractions like data series that support joining real-time and historical data while handling delays transparently. It also suggests approaches for Spark Streaming to better support processing many independent low-volume IoT streams concurrently and improving resource utilization for such applications. Finally, it introduces the Device-Model-Infra framework for addressing these IoT analytics challenges through combined programming models and data abstractions.

A Practical Approach to Building a Streaming Processing Pipeline for an Onlin...Databricks

��

The document discusses the implementation of a streaming processing pipeline for Yelp's online advertising platform using Apache Spark. It covers state management techniques, particularly focusing on handling real-time metrics and applying complex business rules through functions like updateStateByKey. The text highlights challenges with existing APIs and suggests potential improvements in Spark for managing state in a scalable manner, while also addressing aspects of checkpointing and data recovery.

Building Scalable Data Pipelines - 2016 DataPalooza SeattleEvan Chan

��

The document discusses the construction of scalable data pipelines, highlighting the importance of real-time insights and personalized data processing in various applications such as financial trading and smart city management. It emphasizes technologies like Apache Kafka, Spark, and Cassandra for handling data streams and event storage efficiently, while also outlining the benefits of unified systems over traditional ETL processes. The author, Evan Chan, advocates for a flexible and resilient architecture to support fast data processing and analytics.

Have Your Cake and Eat It Too -- Further Dispelling the Myths of the Lambda A...C4Media

��

The document presents a discussion on the lambda architecture and its comparison with streaming architectures, highlighting the importance of strong consistency and correct data processing. It covers various approaches to handling time in data streams, including time-agnostic processing, approximations, and different windowing techniques. The presentation emphasizes that while lambda is useful, streaming offers better low-latency and correct results along with tools for effective time reasoning.

Confluent-Ably-AWS-ID-2023 - G�ݺ�ߣ.pptxAhmed791434

��

This document provides an agenda and overview for a Confluent, Ably, and AWS Immersion Day event. The agenda includes an introduction, presentations on Confluent and Ably, a workshop setup, lunch, and a workshop session. The document then introduces Ahmed Zamzam from Confluent and Roman De Oliveira from Ably as the day's hosts. It provides overview information on Confluent Cloud, the benefits of real-time data, a typical real-time data pipeline, the Kora engine, and how Confluent integrates with AWS.

Learn Apache Kafka Online | Comprehensive Kafka Course & TrainingAccentfuture

��

An Introduction to time series with Team ApachePatrick McFadin

��

The document presents an agenda, focusing on Apache Kafka, Apache Spark, and Apache Cassandra, highlighting hands-on activities and theoretical concepts relevant to data processing. It discusses time series data and analysis, architectures, and deployment including Kafka's message ordering and durability features. Additionally, it introduces Spark's resilient distributed datasets and its streaming capabilities, alongside Cassandra's basic architecture, data replication, and high availability.

Streaming AnalyticsNeera Agarwal

��

Streaming analytics provides real-time processing of continuous data streams. It contrasts with batch processing which operates on bounded datasets. Streaming analytics is used for applications like clickstream analysis, fraud detection, and IoT. Key concepts include event time windows, exactly-once processing, and state management. LinkedIn's streaming platform standardizes profile data in real-time using techniques like stream-table joins, broadcast joins, and reprocessing prior data. Popular open source streaming systems include Kafka Streams, Spark Streaming, Flink, and Storm.

Streaming analytics better than batch – when and why by Dawid Wysakowicz and ...Big Data Spain

��

The document discusses the advantages of streaming analytics over batch processing, highlighting use cases such as user session analytics at Spotify. It explains various architectures including batch, micro-batch, and streaming-first, emphasizing the efficiency and low latency of modern stream processing engines. Additionally, it outlines the capabilities of technologies like Apache Flink, Spark, and Kafka in handling real-time data processing and analytics.

Real-Time Data Pipelines Made Easy with Structured Streaming in Apache Spark.pdfnilanjan172nsvian

��

Streaming analytics better than batch when and why - (Big Data Tech 2017)GetInData

��

The document discusses the advantages of streaming analytics over batch processing, particularly in user session analytics scenarios like those at Spotify. It elaborates on architectural approaches for processing user data in real-time and highlights the challenges of late data and session handling, especially using Apache Flink. Various streaming technologies are analyzed, including their features and limitations, providing guidance on when to use batch processing versus streaming analytics.

Deep dive into stateful stream processing in structured streaming by Tathaga...Databricks

��

The document provides an in-depth exploration of stateful stream processing within Spark's Structured Streaming framework, presented at the Spark Summit Europe 2017. It covers the architecture, including sources, sinks, transformations, and checkpointing, while explaining concepts such as watermarking for managing late data, streaming joins, and custom stateful operations using functions like mapGroupsWithState. The talk also discusses performance monitoring and debugging strategies for stateful queries in real-time processing applications.

Why And When Should We Consider Stream Processing In Our Solutions Teqnation ...Soroosh Khodami

��

The presentation by Soroosh Khodami at #teqnation2021 focuses on stream processing, discussing its significance, frameworks, and practical implementation. It covers essential concepts, patterns, and use cases across various industries while highlighting the benefits and drawbacks of adopting stream processing solutions. The talk aims to guide software developers and architects on when and how to integrate stream processing in their projects.

Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...Databricks

��

Learn Apache Kafka Online | Comprehensive Kafka Course & TrainingAccentfuture

��

Unified Stream & Batch Processing with Apache Flink (Hadoop Summit Dublin 2016)ucelebi

��

Apache Flink is an open-source stream processing framework that offers low latency, high throughput, and robust event time handling. It enables efficient batch and stream processing while ensuring state and fault tolerance, capable of handling both stateless and stateful operations with precise time semantics. Recent developments include enhancements in performance, complex event processing, and upcoming features like SQL support and dynamic scaling.

Decipher SEO Solutions for your startup needs.mathai2

��

A solution deck that gives you an idea of how you can use Decipher SEO to target keywords, build authority and generate high ranking content. With features like images to product you can create a E-commerce pipeline that is optimized to help your store rank. With integrations with shopify, woocommerce and wordpress theres a seamless way get your content to your website or storefront. View more at decipherseo.com

Streamlining CI/CD with FME Flow: A Practical GuideSafe Software

��

More Related Content

Similar to How we built an event-time merge of two kafka-streams with spark-streaming (20)

Spark Summit EU talk by Sebastian Schroeder and Ralf SigmundSpark Summit

��

NoLambda: Combining Streaming, Ad-Hoc, Machine Learning and Batch AnalysisHelena Edelson

��

Leveraging Azure Databricks to minimize time to insight by combining Batch an...Microsoft Tech Community

��

Unbounded bounded-data-strangeloop-2016-monal-daxiniMonal Daxini

��

Spark Streaming and IoT by Mike FreedmanSpark Summit

��

A Practical Approach to Building a Streaming Processing Pipeline for an Onlin...Databricks

��

Building Scalable Data Pipelines - 2016 DataPalooza SeattleEvan Chan

��

Have Your Cake and Eat It Too -- Further Dispelling the Myths of the Lambda A...C4Media

��

Confluent-Ably-AWS-ID-2023 - G�ݺ�ߣ.pptxAhmed791434

��

Learn Apache Kafka Online | Comprehensive Kafka Course & TrainingAccentfuture

��

An Introduction to time series with Team ApachePatrick McFadin

��

Streaming AnalyticsNeera Agarwal

��

Streaming analytics better than batch – when and why by Dawid Wysakowicz and ...Big Data Spain

��

Real-Time Data Pipelines Made Easy with Structured Streaming in Apache Spark.pdfnilanjan172nsvian

��

Streaming analytics better than batch when and why - (Big Data Tech 2017)GetInData

��

Deep dive into stateful stream processing in structured streaming by Tathaga...Databricks

��

Why And When Should We Consider Stream Processing In Our Solutions Teqnation ...Soroosh Khodami

��

Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...Databricks

��

Learn Apache Kafka Online | Comprehensive Kafka Course & TrainingAccentfuture

��

Unified Stream & Batch Processing with Apache Flink (Hadoop Summit Dublin 2016)ucelebi

��

Spark Summit EU talk by Sebastian Schroeder and Ralf SigmundSpark Summit

��

NoLambda: Combining Streaming, Ad-Hoc, Machine Learning and Batch AnalysisHelena Edelson

��

Leveraging Azure Databricks to minimize time to insight by combining Batch an...Microsoft Tech Community

��

Unbounded bounded-data-strangeloop-2016-monal-daxiniMonal Daxini

��

Spark Streaming and IoT by Mike FreedmanSpark Summit

��

A Practical Approach to Building a Streaming Processing Pipeline for an Onlin...Databricks

��

Building Scalable Data Pipelines - 2016 DataPalooza SeattleEvan Chan

��

Have Your Cake and Eat It Too -- Further Dispelling the Myths of the Lambda A...C4Media

��

Confluent-Ably-AWS-ID-2023 - G�ݺ�ߣ.pptxAhmed791434

��

Learn Apache Kafka Online | Comprehensive Kafka Course & TrainingAccentfuture

��

An Introduction to time series with Team ApachePatrick McFadin

��

Streaming AnalyticsNeera Agarwal

��

Streaming analytics better than batch – when and why by Dawid Wysakowicz and ...Big Data Spain

��

Real-Time Data Pipelines Made Easy with Structured Streaming in Apache Spark.pdfnilanjan172nsvian

��

Streaming analytics better than batch when and why - (Big Data Tech 2017)GetInData

��

Deep dive into stateful stream processing in structured streaming by Tathaga...Databricks

��

Why And When Should We Consider Stream Processing In Our Solutions Teqnation ...Soroosh Khodami

��

Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...Databricks

��

Learn Apache Kafka Online | Comprehensive Kafka Course & TrainingAccentfuture

��

Unified Stream & Batch Processing with Apache Flink (Hadoop Summit Dublin 2016)ucelebi

��

Recently uploaded (20)

Decipher SEO Solutions for your startup needs.mathai2

��

Streamlining CI/CD with FME Flow: A Practical GuideSafe Software

��

AI for PV: Development and Governance for a Regulated IndustryBiologit

��

Best Software Development at Best Pricessoftechies7

��

Welcome to Softechies Solutions—your ultimate resource for all technology and software needs. We specialize in delivering expert solutions and guidance to help you tackle everyday tech challenges. We’ve got you covered, from software troubleshooting and website development to digital marketing. Whether you need technology upgrades, mobile app development, cloud-managed services, or IT consulting, Softechies Solutions is here to enhance your digital experience. https://softechies.net/

Why Edge Computing Matters in Mobile Application Tech.pdfIMG Global Infotech

��

Azure AI Foundry: The AI app and agent factoryMaxim Salnikov

��

Discover how Azure AI Foundry is transforming the way organizations innovate and operate. This session explores the pivotal role of AI in reshaping business processes, enhancing employee experiences, and redefining customer engagement. Attendees will gain insights into the strategic value of AI investments, including the measurable returns and the growing momentum behind generative AI experimentation. The talk introduces the emerging paradigm of agentic AI - intelligent agents that are revolutionizing how we build, manage, and interact with software. Through real-world examples, the session demonstrates how Azure AI Foundry empowers teams to design, customize, and scale AI applications and agents, unlocking new levels of efficiency and creativity across industries.

CodeCleaner: Mitigating Data Contamination for LLM Benchmarkingarabelatso

��

declaration of Variables and constants.pptxmeemee7378

��

Complete WordPress Programming Guidance BookShabista Imam

��

From Data Preparation to Inference: How Alluxio Speeds Up AIAlluxio, Inc.

��

Alluxio Webinar June 17, 2025 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: Jingwen Ouyang (Sr. Product Manager @ Alluxio) In this talk, Jingwen Ouyang, Senior Product Manager at Alluxio, will share how Alluxio make it easy to share and manage data from any storage to any compute engine in any environment with high performance and low cost for your model training, model inference, and model distribution workload.

HYBRIDIZATION OF ALKANES AND ALKENES ...karishmaduhijod1

��

Sysinfo OST to PST Converter InfographicSysInfo Tools

��

Test Case Design Techniques – Practical Examples & Best Practices in Software...Muhammad Fahad Bashir

��

This presentation was part of the SQA & PM Bootcamp, where I served as a trainer. It focuses on effective test case design techniques, blending theoretical knowledge with practical application, especially useful for manual testers and QA beginners. Video Lecture Recording : https://www.facebook.com/share/v/1Z44DiXN5v/ What’s Covered: 🧾 Definition and Purpose of Test Case Design 🚀 Importance of Structured Test Design 🧠 Test Case Design Techniques Overview: ✅ Black Box Techniques ✅ White Box Techniques ✅ Experience-Based Techniques 📝 Manual Test Case Design in Detail 🔐 Real-World Example: Login Page Test Case 🧩 Black Box Methods Explained with Examples: Equivalence Partitioning (EP) Boundary Value Analysis (BVA) Difference Between EP & BVA 📊 Decision Table Testing 💡 Experience-Based Testing: Error Guessing, Exploratory Testing 📂 White Box Techniques (Introductory Overview) ➕ Much More to Help You Build Strong Manual Testing Skills This resource is especially helpful for students, aspiring QA professionals, and junior testers looking to master test design fundamentals with clarity and practical insight.

Sap basis role in public cloud in s/4hana.pptxhtmlprogrammer987

��

Modern Platform Engineering with Choreo - The AI-Native Internal Developer Pl...WSO2

��

Building and operating internal platforms has become increasingly complex — sprawling toolchains, rising costs, and fragmented developer experiences are slowing teams down. It’s time for a new approach. This slide deck explores modern platform engineering with Choreo, WSO2’s AI-native Internal Developer Platform as a Service. Discover how you can streamline platform operations, empower your developers, and drive faster innovation — all while reducing complexity and cost. Learn more: https://wso2.com/choreo/platform-engineering/

Which Hiring Management Tools Offer the Best ROI?HireME

��

Humans vs AI Call Agents - Qcall.ai's Special ReportUdit Goenka

��

Humans vs Agentic AI Cost Comparison: Cut 85% Support Costs TL;DR: Agentic AI crushes human costs in L1/L2 support with 85% savings, 300% efficiency gains, and 90%+ success rates. Your CFO just asked: "Why spend $2.3M annually on L1 support when competitors automated 80% for under $200K?" The humans vs agentic AI cost debate isn't theoretical - it's happening now. Real Cost Breakdown: - India L1/L2 Agent: ₹6,64,379 ($8,000) annually (full cost) - US L1/L2 Agent: $70,000 annually (loaded cost) - Qcall.ai Agentic AI: ₹6/min ($0.07/min) = 85% savings Game-Changing Results: - 47 human agents → 8 agents + AI system - Operating costs: $2.8M → $420K (85% reduction) - Customer satisfaction: 78% → 91% - First call resolution: 72% → 94% - Languages: 2-3 → 15+ fluently Why Agentic AI Wins: - Human Limitations: 8hrs/day, 1-2 languages, 30-45% attrition, mood-dependent, 6-12 weeks training - Qcall.ai Advantages: 24/7 availability, 97% humanized voice, <60 second response, intelligent qualification, seamless handoff Industry Impact: - E-commerce: 82% cost reduction - Healthcare: 90% appointment automation - SaaS: 23% satisfaction improvement - Financial: 95% inquiry automation Implementation: - Setup: 2-5 days vs 6-12 weeks hiring - ROI: Break-even in 3 months - Scaling: Instant global expansion Competitive Gap: - Early Adopters: 85% cost advantage, superior metrics, faster expansion Late Adopters: Higher costs, limited scalability, compounding disadvantage Similar automation patterns emerge across industries. Marketing teams use autoposting.ai for social media automation - same human-AI collaboration model driving L1/L2 transformation. "We reduced support costs by 78% while improving satisfaction by 31%. ROI was immediate and keeps compounding." - COO, Global SaaS Company Getting Started: Calculate current L1/L2 costs Assess repetitive tasks Request Qcall.ai demo Plan 20% pilot Bottom Line: Companies avoiding this shift hemorrhage millions while competitors scale globally with fraction of costs. The question isn't whether agentic AI will replace routine tasks - it's whether you'll implement before competitors gain insurmountable advantage. Ready to transform operations? This reveals complete cost analysis, implementation strategy, and competitive advantages.

CodeCleaner: Mitigating Data Contamination for LLM Benchmarkingarabelatso

��

Building Geospatial Data Warehouse for GIS by GIS with FMESafe Software

��

Data warehouses are often considered the backbone of data-driven decision-making. However, traditional implementations can struggle to meet the iterative and dynamic demands of GIS operations, particularly for Iterative Cross-Division Data Integration (ICDI) processes. At WVDOT, we faced this challenge head-on by designing a Geospatial Data Warehouse (GDW) tailored to the needs of GIS. In this talk, we’ll share how WVDOT developed a metadata-driven, SESSI-based (Simplicity, Expandability, Spatiotemporality, Scalability, Inclusivity) GDW that transforms how we manage and leverage spatial data. Key highlights include: - Simplifying data management with a lean schema structure. - Supporting ad hoc source data variations using JSON for flexibility. - Scaling for performance with RDBMS partitioning and materialized views. - Empowering users with FME Flow-based self-service metadata tools.We’ll showcase real-world use cases, such as pavement condition data and LRS-based reporting, and discuss how this approach is elevating GIS as an integrative framework across divisions. Attendees will walk away with actionable insights on designing GIS-centric data systems that combine the best of GIS and data warehouse methodologies.

Simplify Insurance Regulations with Compliance Management SoftwareInsurance Tech Services

��

Decipher SEO Solutions for your startup needs.mathai2

��

Streamlining CI/CD with FME Flow: A Practical GuideSafe Software

��

AI for PV: Development and Governance for a Regulated IndustryBiologit

��

Best Software Development at Best Pricessoftechies7

��

Why Edge Computing Matters in Mobile Application Tech.pdfIMG Global Infotech

��

Azure AI Foundry: The AI app and agent factoryMaxim Salnikov

��

CodeCleaner: Mitigating Data Contamination for LLM Benchmarkingarabelatso

��

declaration of Variables and constants.pptxmeemee7378

��

Complete WordPress Programming Guidance BookShabista Imam

��

From Data Preparation to Inference: How Alluxio Speeds Up AIAlluxio, Inc.

��

HYBRIDIZATION OF ALKANES AND ALKENES ...karishmaduhijod1

��

Sysinfo OST to PST Converter InfographicSysInfo Tools

��

Test Case Design Techniques – Practical Examples & Best Practices in Software...Muhammad Fahad Bashir

��

Sap basis role in public cloud in s/4hana.pptxhtmlprogrammer987

��

Modern Platform Engineering with Choreo - The AI-Native Internal Developer Pl...WSO2

��

Which Hiring Management Tools Offer the Best ROI?HireME

��

Humans vs AI Call Agents - Qcall.ai's Special ReportUdit Goenka

��

CodeCleaner: Mitigating Data Contamination for LLM Benchmarkingarabelatso

��

Building Geospatial Data Warehouse for GIS by GIS with FMESafe Software

��

Simplify Insurance Regulations with Compliance Management SoftwareInsurance Tech Services

��

How we built an event-time merge of two kafka-streams with spark-streaming

1. How We Built an Event-Time Merge of Two Kafka-Streams with Spark Streaming Ralf Sigmund, Sebastian Schröder Otto GmbH & Co. KG

2. Agenda • Who are we? • Our Setup • The Problem •Our Approach • Requirements • Advanced Requirements • Lessons learned

3. Who are we? • Ralf –Technical Designer Team Tracking –Cycling, Surfing –Twitter: @sistar_hh • Sebastian –Developer Team Tracking –Taekwondo, Cycling –Twitter: @Sebasti0n

4. Who are we working for? • 2nd largest ecommerce company in Germany • more than 1 million visitors every day • our blog: https://dev.otto.de • our jobs: www.otto.de/jobs

5. Our Setup otto.de Varnish Reverse Proxy Tracking-Server Vertical 1Vertical 0 Vertical n Service 0 Service 1 Service n

6. The Problem otto.de Varnish Reverse Proxy Vertical 1Vertical 0 Vertical n Service 0 Service 1 Service n Has a limit on the size of messages Needs to send large tracking messages Tracking-Server

7. Our Approach otto.de Varnish Reverse Proxy Tracking-Server Vertical 1Vertical 0 Vertical n S0 S1 Sn Spark Service

8. Requirements

9. Messages with the same key are merged t: 1 id: 1 t: 3 id: 2 t: 2 id: 3 t: 0 id: 2 t: 4 id: 0 t: 2 id: 3 t: 1 id: 1 t: 0 id: 2 t: 3 id: 2 t: 2 id: 3 t: 2 id: 3 t: 4 id: 0

10. Messages have to be timed out t: 1 id: 1 t: 3 id: 2 t: 2 id: 3 t: 0 id: 2 t: 4 id: 0 t: 2 id: 3 t: 1 id: 1 t: 0 id: 2 t: 3 id: 2 t: 2 id: 3 t: 2 id: 3 t: 4 id: 0 max(business event timestamp) time-out after 3 sec

11. Event Streams might get stuck t: 1 id: 1 t: 3 id: 2 t: 2 id: 3 t: 0 id: 2 t: 4 id: 0 t: 2 id: 3 t: 1 id: 1 t: 0 id: 2 t: 3 id: 2 t: 2 id: 3 t: 2 id: 3 t: 4 id: 0 clock A clock B clock: min(clock A, clock B)

12. There might be no messages t: 1 id: 1 t: 4 id: 2 t: 2 id: 3 t: 5 heart beat t: 1 id: 1 t: 4 id: 2 t: 2 id: 3 time-out after 3 sec t: 5 heart beat clock: min(clock A, clock B) : 4

13. Requirements Summary • Messages with the same key are merged • Messages are timed out by the combined event time of both source topics • Timed out messages are sorted by event time

14. Solution

15. Merge messages with the same Id •UpdateStateByKey, which is defined on pairwise RDDs •It applies a function to all new messages and existing state for a given key •Returns a StateDStream with the resulting messages

16. UpdateStateByKey 1 2 3 1 2 3 1 2 3 UpdateFunction: (Seq[InputMessage], Option[MergedMessage]) => Option[MergedMessage]

17. Merge messages with the same Id

18. Merge messages with the same Id

19. Flag And Remove Timed Out Msgs 1 MM TO 2 2 3 MM MM TO MM TO 2 3 1 MM 1 MM 3 needs global Business Clock UpdateFunction

20. Messages are timed out by event time •We need access to all messages from the current micro batch and the history (state) => Custom StateDStream •Compute the maximum event time of both topics and supply it to the updateStateByKey function

21. Messages are timed out by event time

22. Messages are timed out by event time

23. Messages are timed out by event time

24. Messages are timed out by event time

25. Advanced Requirements

26. topicBtopicA Effect of different request rates in source topics 10K 5K 0 10K 5K 0 catching up with 5K per microbatch data read too early

27. Handle different request rates in topics •Stop reading from one topic if the event time of other topic is too far behind •Store latest event time of each topic and partition on driver •Use custom KafkaDStream with clamp method which uses the event time

28. Handle different request rates in topics

29. Handle different request rates in topics

30. Handle different request rates in topics

31. Handle different request rates in topics

32. Guarantee at least once semantics o: 1 t: a o: 7 t: a o: 2 t: a o: 5 t: b o: 9 t: b t: 6 t: b Current State Sink Topic a-offset: 1 b-offset: 5 a-offset: 1 b-offset: 5 a-offset: 1 b-offset: 5 On Restart/Recovery Retrieve last message Timeout Spark Merge Service

33. Guarantee at least once semantics

34. Guarantee at least once semantics

35. Everything put together

36. Lessons learned • Excellent Performance and Scalability • Extensible via Custom RDDs and Extension to DirectKafka •No event time windows in Spark Streaming See: https://github.com/apache/spark/pull/2633 • Checkpointing cannot be used when deploying new artifact versions • Driver/executor model is powerful, but also complex

37. THANK YOU. Twitter: @sistar_hh @Sebasti0n Blog: dev.otto.de Jobs: otto.de/jobs

�ݺ�ߣ

How we built an event-time merge of two kafka-streams with spark-streaming

Recommended

More Related Content

Similar to How we built an event-time merge of two kafka-streams with spark-streaming (20)

Recently uploaded (20)

How we built an event-time merge of two kafka-streams with spark-streaming