Getting started with streaming analytics: Setting up a pipeline

Aug 24, 20200 likes177 views

In this session I will show you how to create a simple streaming analytics pipeline, first using open source tools and developing locally, then moving to a VM, then moving to fully managed AWS services. The session will serve as an introduction to some details of Apache Kafka, Apache Flink, ElasticSearch, Amazon Managed Streaming for Kafka, Kinesis Data Analytics, and Amazon ElasticSearch. It will be an almost slideless presentation, as I will spent most of the time at the command line and the IDE.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Part 2 of 3: Setting up a pipeline
Getting started with streaming analytics
Javier Ramirez
AWS Developer Advocate
@supercoco9

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Agenda
Building and running a basic Apache Kafka + Apache
Flink pipeline locally
Deploying to Amazon MSK + Kinesis Data Analytics
Adding aggregations and using Elasticsearch and
Kibana for the dashboards
Replacing our Kafka input by Kinesis Data Streams

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Ingestion/in-stream storage: Apache Kafka
A distributed streaming platform
Concepts:
Producers
Topics
Brokers
Consumers

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Stream Processing: Apache Flink
Stateful computation over Data Streams
Concepts:
Job Manager/Workers
Source
DataStream
Transforms/Operators
TableAPI/SQL
Sinks

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Dashboard: Elasticsearch with Kibana
Elasticsearch is a distributed JSON-based search and
analytics engine. Kibana gives shape to your data
https://www.elastic.co/kibana
Wikimedia has a live
interactive dashboard
powered by Kibana at
https://wikimedia.biterg.io/
Concepts:
Master Node
Data Nodes
Shard
Index

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Amazon Kinesis Data Streams
• Easy administration and low cost
• Real-time, elastic performance
• Secure, durable storage
• Available to multiple real-time analytics applications

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Amazon Kinesis Data Streams
Time-based
seek
• Data streams are made of Shards
• Each Shard ingests data up to 1MB/sec,
and up to 1000 TPS
• Each Shard emits up to 2 MB/sec
• All data is stored for 24 hours – 7 days
• Scale Kinesis data streams by splitting or
merging Shards
• Replay data inside of 24Hr -7days
Window

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Getting Started
https://ci.apache.org/projects/flink/flink-docs-stable/
Apache Flink official docs
https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
Elasticsearch official docs
https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html
Getting started with Apache Kafka/Amazon MSK
https://aws.amazon.com/kinesis/
Amazon Kinesis Services for streaming data
https://aws.amazon.com/elasticsearch-service/
Amazon ElasticSearch Service
https://kafka.apache.org/documentation/
Apache Kafka official docs

Data comes in different shapes and sizes, at different speeds, and with different processing needs. AWS offers managed databases covering SQL, NoSQL, in-memory, graphs, time series, and ledger databases. Learn which are the main use cases for each of those, which ones offer autoscaling or serverless capabilities, how you can start from scratch or via database migration services, and why Amazon Aurora is the fastest-growing service in the history of AWS.

Getting started with streaming analytics: streaming basics (1 of 3)javier ramirez

��

In this webinar we explain which are some of the problems of streaming analytics, and why they are different to batch/big data analytics. Then we go into introducing some basic streaming concepts, like event queues, event processors, event vs processing time, and delivery guarantees. We end this first part of the series presenting a few of the most common open source components for streaming (Kafka, Spark, Flink, Cassandra, or ElasticSearch) and we mention the different options you have to run them on AWS.

Open Distro for ElasticSearch and how Grimoire is using it. Madrid DevOps Oct...javier ramirez

��

Bitergia uses Open Distro for Elasticsearch to provide analytics on software development projects. Open Distro is an Apache 2.0-licensed distribution of Elasticsearch that includes enterprise-grade security, alerting, SQL support, and performance analysis. It can be deployed flexibly on Docker or RPM/Debian and is easy to get started with. Bitergia analyzes activity, community, and performance metrics of projects to help organizations understand and improve their software development.

Building a Modern Data Platform in the Cloud. AWS Initiate Portugaljavier ramirez

��

The document discusses the evolution of data platforms and analytics over time from the pre-2009 era to present day. It covers major technologies and approaches that have been used at different stages, including Hadoop, NoSQL databases, Spark, data lakes, and cloud-based solutions. It also outlines some of the common challenges encountered with on-premises and open source implementations and how AWS services aim to simplify building a modern data platform in the cloud.

Consulta cualquier fuente de datos usando SQL con Amazon Athena y sus consult...javier ramirez

��

This document discusses Amazon Athena and how it allows users to query data stored in Amazon S3 using standard SQL. It provides examples of common use cases for Athena including ad-hoc queries, using it for software as a service applications, and for extract, transform, load workflows. It also introduces Athena's new federated query capability which allows querying multiple data sources without needing to move data into a single location. The document demonstrates how to build a custom data source connector for Athena to query external sources.

Analyzing your web and application logs with the Amazon Elasticsearch Service...javier ramirez

��

This presentation shows how you can use the Amazon Elasticsearch Service in general, and in particular I showcase how you can host a static website, use CloudFront as a global CDN, and process HTTP logs using serverless functions with Lambda to ingest into ElasticSearch. I finish by creating a Kibana dashboard. Live demo was done in Stockholm and Oslo, and screenshots are included. Also included the URL for you to build the demo by yourself

Cloud Computing - How AWS can help your businessAmazon Web Services LATAM

��

AWS controls over 50% of the cloud computing market and offers a broad range of services to help businesses innovate. The document discusses how AWS provides agility, economy, and elasticity to users, allowing them to focus on their business while AWS manages the infrastructure. It also highlights examples of large customers like NASA that have found AWS to be more secure than their own data centers.

Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...AWS Summits

��

AWS provides multiple ways to ingest and process real-time data generated from sources such as Edge device, logs, websites, mobile apps, IoT devices and more. In this session we will compare the different tools and technologies and share best practices for when to use what. The session will cover: Apache Kafka, Kinesis Data Streams/Firehose, MSK (Managed Kafka), Kinesis Data Analytics for SQL and Java (Flink), Apache Spark and more.

Serverless in Big DataEric Johnson

��

The document discusses serverless technologies for big data processing, including ingestion using Amazon Kinesis Video Streams, Data Streams, and Data Firehose. It covers real-time processing with Kinesis Data Streams and Lambda, real-time analytics using Kinesis Data Analytics, and post processing options like Amazon S3, DynamoDB, Athena and AWS Glue. The role of serverless computing in simplifying and scaling big data workloads is a key focus.

Keynote: Customer Journey with Streaming Data on AWS - Rahul Pathak, AWSFlink Forward

��

AWS has offered streaming data services for over 5 years, starting with Kinesis Data Streams. Now customers use streaming across most AWS services, including two that support Apache Flink - EMR and Kinesis Data Analytics. The keynote described how customer use of streaming data on AWS has evolved from early customers processing tens of millions of events daily to now processing trillions of events, and how Flink usage has grown both internally at AWS and for customers.

AWS Data Immersion Webinar Week - Entenda como ampliar suas possibilidades de...Amazon Web Services LATAM

��

This document discusses expanding a data lake on AWS. It covers topics like what a data lake on AWS is, the importance of storage choices, data sources and ingestion options on AWS. Specific ingestion options covered include Amazon S3, streams using Kinesis and Kinesis Firehose, logs using CloudWatch and Kinesis Agent, and databases using Database Migration Service. The document provides diagrams and examples of ingesting from various sources into a data lake on AWS for storage and analytics.

Cutting to the chase for Machine Learning Analytics Ecosystem & AWS Lake Form...AWS Riyadh User Group

��

AWS Lake Formation Deep DiveCobus Bernard

��

This document provides an overview of AWS Lake Formation and related services for building a secure data lake. It discusses how Lake Formation provides a centralized management layer for data ingestion, cleaning, security and access. It also describes how Lake Formation integrates with services like AWS Glue, Amazon S3 and ML transforms to simplify and automate many data lake tasks. Finally, it provides an example workflow for using Lake Formation to deduplicate data from various sources and grant secure access for analysis.

AWS 2019 Taipei Summit - Building Serverless Analytics Platform on AWSSteven Hsieh

��

This document discusses building serverless analytics solutions on AWS. It describes how serverless analytics can provide on-demand analytics on data lakes with no infrastructure to manage. Key services mentioned include Amazon S3 for storage, AWS Glue for ETL and data cataloging, Amazon Athena for interactive queries, and Amazon QuickSight for visualization. The document provides examples of using these services together for automated reporting, query monitoring with workgroups, and embedding dashboards in applications.

Wild Rydes with Big Data/Kinesis focus: AWS Serverless WorkshopAWS Germany

��

This is a hands-on workshop where every participant will not only learn how to architect and implement a serverless application on Amazon Web Services using nothing but serverless resources for all layers in theory, but actually do it in practice, with all the necessary support from the speakers. Serverless computing allows you to build and run applications and services without thinking about servers. Serverless applications don't require you to provision, scale, and manage any servers. You can build them for nearly any type of application or backend service, and everything required to run and scale your application with high availability is handled for you. Building serverless applications means that developers can focus on their core product instead of worrying about managing and operating servers or runtimes. This reduced overhead lets developers reclaim time and energy that can be spent on developing great products which scale and that are reliable.

Value of Data Beyond Analytics by Darin BriskmanSameer Kenkare

��

The document discusses analytics capabilities provided by Amazon Web Services (AWS). It describes how AWS offers a variety of services for building data lakes, loading and querying data, and performing analytics. These services include Amazon S3, Amazon Redshift, Amazon Athena, Amazon EMR, and Amazon QuickSight. It also provides examples of how customers like Epic Games and a large media company use these AWS analytics services.

Building Data Lakes and Analytics on AWS. IPExpo Manchester.javier ramirez

��

Over 90% of today's data was generated in the last 2 years, and the rate of data growth isn't slowing down. In this session, we'll step through the challenges and best practices on how to capture all the data that is being generated, understand what data you have, and start driving insights and even predict the future using purpose built AWS Services. We'll frame the session around common pitfalls of building Data Lakes and how to successful drive analytics and insights from the data. This session will focus on the architecture patterns bringing together key AWS Services and rather than a deep dive on any single service. We'll show how services such as Amazon S3, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR, Amazon Kinesis, and even Amazon Machine Learning services are put together to build a successful data lake for various role including both data scientists and business users.

AWS Floor28 - WildRydes Serverless Data Processsing workshop (Ver2)Adir Sharabi

��

This document provides an overview of a serverless data processing workshop hosted by AWS. The workshop will guide participants in building out infrastructure to collect and analyze real-time streaming data from unicorn sensors. This will involve setting up data streams from the sensors to stream, aggregate, and process the data. The aggregated data will then be written to DynamoDB and sent to S3 for long-term storage and querying in Athena. Services used will include Kinesis, Lambda, DynamoDB, S3, Firehose and Athena.

[AWS Media Symposium 2019] Perfecting the Media Experience with AWS - Bhavik ...Amazon Web Services Korea

��

This document provides an overview of a presentation about using AWS for media workloads. Some key points: - AWS can help media companies meet rising consumer expectations more efficiently through an end-to-end cloud solution. - AWS offers a wide range of services to address the complexity of media workloads across content creation, supply chain management, distribution and analytics. - Migrating media workloads to AWS provides benefits like infinite scalability, visibility, and not having to maintain physical infrastructure. - Examples are given of customers like Netflix, Discovery and Turner using AWS for tasks like content delivery, broadcast playout and machine learning. - Solutions and services discussed include AWS Elemental, Amazon S3

How Disney+ uses fast data ubiquity to improve the customer experience Martin Zapletal

��

Disney+ uses Amazon Kinesis to drive real-time actions like providing title recommendations for customers, sending events across microservices, and delivering logs for operational analytics to improve the customer experience. In this session, you learn how Disney+ built real-time data-driven capabilities on a unified streaming platform. This platform ingests billions of events per hour in Amazon Kinesis Data Streams, processes and analyzes that data in Amazon Kinesis Data Analytics for Apache Flink, and uses Amazon Kinesis Data Firehose to deliver data to destinations without servers or code. Hear how these services helped Disney+ scale its viewing experience to tens of millions of customers with the required quality and reliability. Learn more about re:Invent 2020 at http://bit.ly/3c4NSdY

The Future of Fast Databases: Lessons from a Decade of QuestDBjavier ramirez

��

Over the last decade, QuestDB has been at the forefront of handling time series data with a focus on speed and efficiency. In this talk, I’ll share practical insights from our experience serving thousands of users, highlighting what we’ve learned about building and maintaining a fast database that can ingest millions of events per second. QuestDB, an open-source time series database, has traditionally relied on a custom-built, non-standard data storage format designed for performance. As we move forward, we’re actively developing its architecture to support open formats like Apache Parquet and Arrow, reflecting a broader industry shift. I’ll discuss the engineering challenges we’ve faced during this transition, the new possibilities it creates, and why these changes are crucial for the evolving database landscape. Through live demos, I’ll showcase QuestDB’s performance in real-time data ingestion and queries, and demonstrate some of the features enabled by these new formats.

Cómo hemos implementado semántica de "Exactly Once" en nuestra base de datos ...javier ramirez

��

Los sistemas distribuidos son difíciles. Los sistemas distribuidos de alto rendimiento, más. Latencias de red, mensajes sin confirmación de recibo, reinicios de servidores, fallos de hardware, bugs en el software, releases problemáticas, timeouts... hay un montón de motivos por los que es muy difícil saber si un mensaje que has enviado se ha recibido y procesado correctamente en destino. Así que para asegurar mandas el mensaje otra vez.. y otra... y cruzas los dedos para que el sistema del otro lado tenga tolerancia a los duplicados. QuestDB es una base de datos open source diseñada para alto rendimiento. Nos queríamos asegurar de poder ofrecer garantías de "exactly once", deduplicando mensajes en tiempo de ingestión. En esta charla, te cuento cómo diseñamos e implementamos la palabra clave DEDUP en QuestDB, permitiendo deduplicar y además permitiendo Upserts en datos en tiempo real, añadiendo solo un 8% de tiempo de proceso, incluso en flujos con millones de inserciones por segundo. Además, explicaré nuestra arquitectura de log de escrituras (WAL) paralelo y multithread. Por supuesto, todo esto te lo cuento con demos, para que veas cómo funciona en la práctica.

How We Added Replication to QuestDB - JonTheBeachjavier ramirez

��

Building a database that can beat industry benchmarks is hard work, and we had to use every trick in the book to keep as close to the hardware as possible. In doing so, we initially decided QuestDB would scale only vertically, on a single instance. A few years later, data replication —for horizontally scaling reads and for high availability— became one of the most demanded features, especially for enterprise and cloud environments. So, we rolled up our sleeves and made it happen. Today, QuestDB supports an unbounded number of geographically distributed read-replicas without slowing down reads on the primary node, which can ingest data at over 4 million rows per second. In this talk, I will tell you about the technical decisions we made, and their trade offs. You'll learn how we had to revamp the whole ingestion layer, and how we actually made the primary faster than before when we added multi-threaded Write Ahead Logs to deal with data replication. I'll also discuss how we are leveraging object storage as a central part of the process. And of course, I'll show you a live demo of high-performance multi-region replication in action.

The Building Blocks of QuestDB, a Time Series Databasejavier ramirez

��

Talk Delivered at Valencia Codes Meetup 2024-06. Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds. It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.

¿Se puede vivir del open source? T3chfestjavier ramirez

��

Hubo un tiempo en el que casi cualquier componente de software requería pagar una licencia. Afortunadamente, hoy en día gracias al software libre y de código abierto, se puede desarrollar prácticamente cualquier aplicación usando componentes gratuitos. Pero, si el software es gratis, ¿Quién lo desarrolla? ¿Trabaja la comunidad de software libre de forma altruista? ¿Se puede desarrollar software libre de forma profesional? De hecho, hay quien dice que el código abierto tal y como lo conocimos ya no existe, y que lo que hay hoy en día es otra cosa. En esta charla hablaré de cómo se puede monetizar el código libre, y de algunos posibles conflictos que puedes encontrarte en el camino. Además, te contaré cómo hacemos desde QuestDB para desarrollar una base de datos de código abierto y mantener un equipo estable viviendo de ello. Comentaré también algunas situaciones problemáticas a las que proyectos muy destacados se han enfrentado, o que se enfrentan a día de hoy.

QuestDB: The building blocks of a fast open-source time-series databasejavier ramirez

��

(talk delivered at OSA CON 23) Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds. It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will learn how it deals with data ingestion, and which SQL extensions it implements for working with time-series efficiently. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or data deduplication.

Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...javier ramirez

��

QuestDB es una base de datos open source de alto rendimiento. Mucha gente nos comentaba que les gustaría usarla como servicio, sin tener que gestionar las máquinas. Así que nos pusimos manos a la obra para desarrollar una solución que nos permitiese lanzar instancias de QuestDB con provisionado, monitorización, seguridad o actualizaciones totalmente gestionadas. Unos cuantos clusters de Kubernetes más tarde, conseguimos lanzar nuestra oferta de QuestDB Cloud. Esta charla es la historia de cómo llegamos ahí. Hablaré de herramientas como Calico, Karpenter, CoreDNS, Telegraf, Prometheus, Loki o Grafana, pero también de retos como autenticación, facturación, multi-nube, o de a qué tienes que decir que no para poder sobrevivir en la nube.

Ingesting Over Four Million Rows Per Second With QuestDB Timeseries Database ...javier ramirez

��

How would you build a database to support sustained ingestion of several hundreds of thousands rows per second while running near real-time queries on top? In this session I will go over some of the technical decisions and trade-offs we applied when building QuestDB, an open source time-series database developed mainly in JAVA, and how we can achieve over four million row writes per second on a single instance without blocking or slowing down the reads. There will be code and demos, of course. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.

Deduplicating and analysing time-series data with Apache Beam and QuestDBjavier ramirez

��

Time series data pipelines tend to prioritise speed and freshness over completeness and integrity. In such scenarios, it is very common to ingest duplicate data, which may be fine for many analytical use cases, but is very inconvenient for others. There are many open source databases built specifically for the speed and query semantics of time series, and most of them lack automatic deduplication of events in near real-time. One such database is QuestDB, which requires a manual batch process to deduplicate ingested data. In this talk, we will see how we can successfully use Apache Beam to deduplicate streaming time series, which can then be analysed by a time series database.

More Related Content

Similar to Getting started with streaming analytics: Setting up a pipeline (13)

Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...AWS Summits

��

Serverless in Big DataEric Johnson

��

Keynote: Customer Journey with Streaming Data on AWS - Rahul Pathak, AWSFlink Forward

��

AWS Data Immersion Webinar Week - Entenda como ampliar suas possibilidades de...Amazon Web Services LATAM

��

Cutting to the chase for Machine Learning Analytics Ecosystem & AWS Lake Form...AWS Riyadh User Group

��

AWS Lake Formation Deep DiveCobus Bernard

��

AWS 2019 Taipei Summit - Building Serverless Analytics Platform on AWSSteven Hsieh

��

Wild Rydes with Big Data/Kinesis focus: AWS Serverless WorkshopAWS Germany

��

Value of Data Beyond Analytics by Darin BriskmanSameer Kenkare

��

Building Data Lakes and Analytics on AWS. IPExpo Manchester.javier ramirez

��

AWS Floor28 - WildRydes Serverless Data Processsing workshop (Ver2)Adir Sharabi

��

[AWS Media Symposium 2019] Perfecting the Media Experience with AWS - Bhavik ...Amazon Web Services Korea

��

How Disney+ uses fast data ubiquity to improve the customer experience Martin Zapletal

��

Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...AWS Summits

��

Serverless in Big DataEric Johnson

��

Keynote: Customer Journey with Streaming Data on AWS - Rahul Pathak, AWSFlink Forward

��

AWS Data Immersion Webinar Week - Entenda como ampliar suas possibilidades de...Amazon Web Services LATAM

��

Cutting to the chase for Machine Learning Analytics Ecosystem & AWS Lake Form...AWS Riyadh User Group

��

AWS Lake Formation Deep DiveCobus Bernard

��

AWS 2019 Taipei Summit - Building Serverless Analytics Platform on AWSSteven Hsieh

��

Wild Rydes with Big Data/Kinesis focus: AWS Serverless WorkshopAWS Germany

��

Value of Data Beyond Analytics by Darin BriskmanSameer Kenkare

��

Building Data Lakes and Analytics on AWS. IPExpo Manchester.javier ramirez

��

AWS Floor28 - WildRydes Serverless Data Processsing workshop (Ver2)Adir Sharabi

��

[AWS Media Symposium 2019] Perfecting the Media Experience with AWS - Bhavik ...Amazon Web Services Korea

��

How Disney+ uses fast data ubiquity to improve the customer experience Martin Zapletal

��

More from javier ramirez (20)

The Future of Fast Databases: Lessons from a Decade of QuestDBjavier ramirez

��

Cómo hemos implementado semántica de "Exactly Once" en nuestra base de datos ...javier ramirez

��

How We Added Replication to QuestDB - JonTheBeachjavier ramirez

��

The Building Blocks of QuestDB, a Time Series Databasejavier ramirez

��

¿Se puede vivir del open source? T3chfestjavier ramirez

��

QuestDB: The building blocks of a fast open-source time-series databasejavier ramirez

��

Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...javier ramirez

��

Ingesting Over Four Million Rows Per Second With QuestDB Timeseries Database ...javier ramirez

��

Deduplicating and analysing time-series data with Apache Beam and QuestDBjavier ramirez

��

Your Database Cannot Do this (well)javier ramirez

��

Relational databases were created a long time ago for a simpler world. Even if they are still awesome tools for generic workloads, there are some things they cannot do well. In this session I will speak about purpose-built databases that you can use for specific business scenarios. We will see the type of queries you can run on a Graph database, a Document Database, and a Time-Series database. We will then see how a relational database could also be used for the same use cases, just in a much more complex way.

Your Timestamps Deserve Better than a Generic Databasejavier ramirez

��

This document discusses the challenges of working with timestamped data in databases and introduces QuestDB as a time-series database designed to address these challenges. It highlights QuestDB's high performance for ingesting and querying large volumes of timestamped data. It also demonstrates several time-series focused query patterns in QuestDB like time range queries, sampling, filling missing data, retrieving the latest value, and approximate joins between tables. Finally, it outlines some areas QuestDB is exploring to further improve performance.

Cómo se diseña una base de datos que pueda ingerir más de cuatro millones de ...javier ramirez

��

En esta sesión voy a contar las decisiones técnicas que tomamos al desarrollar QuestDB, una base de datos Open Source para series temporales compatible con Postgres, y cómo conseguimos escribir más de cuatro millones de filas por segundo sin bloquear o enlentecer las consultas. Hablaré de cosas como (zero) Garbage Collection, vectorización de instrucciones usando SIMD, reescribir en lugar de reutilizar para arañar microsegundos, aprovecharse de los avances en procesadores, discos duros y sistemas operativos, como por ejemplo el soporte de io_uring, o del balance entre experiencia de usuario y rendimiento cuando se plantean nuevas funcionalidades.

QuestDB-Community-Call-20220728javier ramirez

��

Processing and analysing streaming data with Python. Pycon Italy 2022javier ramirez

��

QuestDB: ingesting a million time series per second on a single instance. Big...javier ramirez

��

Servicios e infraestructura de AWS y la próxima región en Aragónjavier ramirez

��

Primeros pasos en desarrollo serverlessjavier ramirez

��

¿Qué es eso del desarrollo sin servidores? ¿Qué lenguajes puedo utilizar? ¿Cómo hago cosas como autenticación, o guardar en base de datos, o enviar notificaciones? ¿Esto escala? A todas estas preguntas, y a alguna más, intentaré dar respuesta en esta sesión, donde haré una pequeña demo de montar una app muy sencilla y desplegarla en la nube sin preocuparnos de gestionar infraestructura. Charla realizada por primera vez para AlcarriaConf 2021

How AWS is reinventing the cloudjavier ramirez

��

Analitica de datos en tiempo real con Apache Flink y Apache BEAMjavier ramirez

��

This document summarizes a presentation about real-time data analytics with Apache Flink and Apache BEAM. It discusses possible real-time and batch processing systems using AWS services, challenges of streaming systems including state management, and demos of analyzing user clickstreams and taxi trips with Apache Flink, Kafka, and Elasticsearch. It also covers advantages of Apache BEAM including a unified batch and streaming API that can run on different frameworks like Flink, benefits of native support for Java, Python, and Go, and how it allows mixing languages in pipelines.

Getting started with streaming analyticsjavier ramirez

��

The Future of Fast Databases: Lessons from a Decade of QuestDBjavier ramirez

��

Cómo hemos implementado semántica de "Exactly Once" en nuestra base de datos ...javier ramirez

��

How We Added Replication to QuestDB - JonTheBeachjavier ramirez

��

The Building Blocks of QuestDB, a Time Series Databasejavier ramirez

��

¿Se puede vivir del open source? T3chfestjavier ramirez

��

QuestDB: The building blocks of a fast open-source time-series databasejavier ramirez

��

Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...javier ramirez

��

Ingesting Over Four Million Rows Per Second With QuestDB Timeseries Database ...javier ramirez

��

Deduplicating and analysing time-series data with Apache Beam and QuestDBjavier ramirez

��

Your Database Cannot Do this (well)javier ramirez

��

Your Timestamps Deserve Better than a Generic Databasejavier ramirez

��

Cómo se diseña una base de datos que pueda ingerir más de cuatro millones de ...javier ramirez

��

QuestDB-Community-Call-20220728javier ramirez

��

Processing and analysing streaming data with Python. Pycon Italy 2022javier ramirez

��

QuestDB: ingesting a million time series per second on a single instance. Big...javier ramirez

��

Servicios e infraestructura de AWS y la próxima región en Aragónjavier ramirez

��

Primeros pasos en desarrollo serverlessjavier ramirez

��

How AWS is reinventing the cloudjavier ramirez

��

Analitica de datos en tiempo real con Apache Flink y Apache BEAMjavier ramirez

��

Getting started with streaming analyticsjavier ramirez

��

Recently uploaded (20)

Big-O notations, Algorithm and complexity analaysisdrsomya2019

��

Chapter-4-Plane-Wave-Propagation-pdf.pdfShamsAli42

��

Satisfaction_Framework_Presentation.pptxnagom47355

��

Exploratory data analysis (EDA) is used by data scientists to analyze and inv...jimmy841199

��

Model Context Protocol - path to LLM standartizationAntons Kranga

��

DII-WS Training Manual with Links_V2.pdfcoolprince739

��

GLOBAL-GOALS-LOCAL-ACTIONS-The-SDG-Journey-from-Vision-to-Reality.pptxKunalBhadana3

��

PostGIS Workshop: a comprehensive tutorial.pptLonJames2

��

networkmonitoringtools-200615094423.pptxkelvinzallan5

��

ARCH 2025: New Mexico Respite Provider RegistryAllen Shaw

��

AI-Powered Contact Centre Virtual Assistant DSSrinivasan N

��

ENG8-Q4-MOD2.pdfajxnjdabajbadjbiadbiwdhiwdhwdhiwdshekainahrosej

��

Database's & presentation's for beginnerschubzja07

��

Information Security Management-Planning 1.pptxFrancisFayiah

��

diagram ANN of factor and responses.pptxEdunjobiTunde1

��

Control_Chart_Presentation copy for Business process and planning.pptxPranavRaut36

��

STS-PRELIM-2025.pptxtyyfddjugggfssghghihfTristanEvasco

��

Chat Bots - An Analytical study including Indian playersDR. Ram Kumar Pathak

��

Mastering Data Science with Tutort Academyyashikanigam1

��

## **Mastering Data Science with Tutort Academy: Your Ultimate Guide** ### **Introduction** Data Science is transforming industries by enabling data-driven decision-making. Mastering this field requires a structured learning path, practical exposure, and expert guidance. Tutort Academy provides a comprehensive platform for professionals looking to build expertise in Data Science. --- ## **Why Choose Data Science as a Career?** - **High Demand:** Companies worldwide are seeking skilled Data Scientists. - **Lucrative Salaries:** Competitive pay scales make this field highly attractive. - **Diverse Applications:** Used in finance, healthcare, e-commerce, and more. - **Innovation-Driven:** Constant advancements make it an exciting domain. --- ## **How Tutort Academy Helps You Master Data Science** ### **1. Comprehensive Curriculum** Tutort Academy offers a structured syllabus covering: - **Python & R for Data Science** - **Machine Learning & Deep Learning** - **Big Data Technologies** - **Natural Language Processing (NLP)** - **Data Visualization & Business Intelligence** - **Cloud Computing for Data Science** ### **2. Hands-on Learning Approach** - **Real-World Projects:** Work on datasets from different domains. - **Live Coding Sessions:** Learn by implementing concepts in real-time. - **Industry Case Studies:** Understand how top companies use Data Science. ### **3. Mentorship from Experts** - **Guidance from Industry Leaders** - **Career Coaching & Resume Building** - **Mock Interviews & Job Assistance** ### **4. Flexible Learning for Professionals** - **Best DSA Course Online:** Strengthen your problem-solving skills. - **System Design Course Online:** Master scalable system architectures. - **Live Courses for Professionals:** Balance learning with a full-time job. --- ## **Key Topics Covered in Tutort Academy’s Data Science Program** ### **1. Programming for Data Science** - Python, SQL, and R - Data Structures & Algorithms (DSA) - System Design & Optimization ### **2. Data Wrangling & Analysis** - Handling Missing Data - Data Cleaning Techniques - Feature Engineering ### **3. Statistics & Probability** - Descriptive & Inferential Statistics - Hypothesis Testing - Probability Distributions ### **4. Machine Learning & AI** - Supervised & Unsupervised Learning - Model Evaluation & Optimization - Deep Learning with TensorFlow & PyTorch ### **5. Big Data & Cloud Technologies** - Hadoop, Spark, and AWS for Data Science - Data Pipelines & ETL Processes ### **6. Data Visualization & Storytelling** - Tools like Tableau, Power BI, and Matplotlib - Creating Impactful Business Reports ### **7. Business Intelligence & Decision Making** - How data drives strategic business choices - Case Studies from Leading Organizations --- ## **Mastering Data Science: A Step-by-Step Plan** ### **Step 1: Learn the Fundamentals** Start with **Python for Data Science, Statistics, and Linear Algebra.** Understanding these basics is crucial for advanced t

537116365-Domain-6-Presentation-New.pptxPorshaAbril1

��

Big-O notations, Algorithm and complexity analaysisdrsomya2019

��

Chapter-4-Plane-Wave-Propagation-pdf.pdfShamsAli42

��

Satisfaction_Framework_Presentation.pptxnagom47355

��

Exploratory data analysis (EDA) is used by data scientists to analyze and inv...jimmy841199

��

Model Context Protocol - path to LLM standartizationAntons Kranga

��

DII-WS Training Manual with Links_V2.pdfcoolprince739

��

GLOBAL-GOALS-LOCAL-ACTIONS-The-SDG-Journey-from-Vision-to-Reality.pptxKunalBhadana3

��

PostGIS Workshop: a comprehensive tutorial.pptLonJames2

��

networkmonitoringtools-200615094423.pptxkelvinzallan5

��

ARCH 2025: New Mexico Respite Provider RegistryAllen Shaw

��

AI-Powered Contact Centre Virtual Assistant DSSrinivasan N

��

ENG8-Q4-MOD2.pdfajxnjdabajbadjbiadbiwdhiwdhwdhiwdshekainahrosej

��

Database's & presentation's for beginnerschubzja07

��

Information Security Management-Planning 1.pptxFrancisFayiah

��

diagram ANN of factor and responses.pptxEdunjobiTunde1

��

Control_Chart_Presentation copy for Business process and planning.pptxPranavRaut36

��

STS-PRELIM-2025.pptxtyyfddjugggfssghghihfTristanEvasco

��

Chat Bots - An Analytical study including Indian playersDR. Ram Kumar Pathak

��

Mastering Data Science with Tutort Academyyashikanigam1

��

537116365-Domain-6-Presentation-New.pptxPorshaAbril1

��

Getting started with streaming analytics: Setting up a pipeline

1. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Part 2 of 3: Setting up a pipeline Getting started with streaming analytics Javier Ramirez AWS Developer Advocate @supercoco9

2. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Agenda Building and running a basic Apache Kafka + Apache Flink pipeline locally Deploying to Amazon MSK + Kinesis Data Analytics Adding aggregations and using Elasticsearch and Kibana for the dashboards Replacing our Kafka input by Kinesis Data Streams

3. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Ingestion/in-stream storage: Apache Kafka A distributed streaming platform Concepts: Producers Topics Brokers Consumers

4. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Stream Processing: Apache Flink Stateful computation over Data Streams Concepts: Job Manager/Workers Source DataStream Transforms/Operators TableAPI/SQL Sinks

6. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Dashboard: Elasticsearch with Kibana Elasticsearch is a distributed JSON-based search and analytics engine. Kibana gives shape to your data https://www.elastic.co/kibana Wikimedia has a live interactive dashboard powered by Kibana at https://wikimedia.biterg.io/ Concepts: Master Node Data Nodes Shard Index

7. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Amazon Kinesis Data Streams • Easy administration and low cost • Real-time, elastic performance • Secure, durable storage • Available to multiple real-time analytics applications

8. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Amazon Kinesis Data Streams Time-based seek • Data streams are made of Shards • Each Shard ingests data up to 1MB/sec, and up to 1000 TPS • Each Shard emits up to 2 MB/sec • All data is stored for 24 hours – 7 days • Scale Kinesis data streams by splitting or merging Shards • Replay data inside of 24Hr -7days Window

9. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Getting Started https://ci.apache.org/projects/flink/flink-docs-stable/ Apache Flink official docs https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html Elasticsearch official docs https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html Getting started with Apache Kafka/Amazon MSK https://aws.amazon.com/kinesis/ Amazon Kinesis Services for streaming data https://aws.amazon.com/elasticsearch-service/ Amazon ElasticSearch Service https://kafka.apache.org/documentation/ Apache Kafka official docs

10. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential ThanksJavier Ramirez AWS Developer Advocate @supercoco9

�ݺ�ߣ

Getting started with streaming analytics: Setting up a pipeline

Recommended

More Related Content

Similar to Getting started with streaming analytics: Setting up a pipeline (13)

More from javier ramirez (20)

Recently uploaded (20)

Getting started with streaming analytics: Setting up a pipeline