際際滷shows by User: sap1ens / http://www.slideshare.net/images/logo.gif 際際滷shows by User: sap1ens / Fri, 06 Oct 2023 08:48:15 GMT 際際滷Share feed for 際際滷shows by User: sap1ens Dynamic Change Data Capture with Flink CDC and Consistent Hashing /slideshow/dynamic-change-data-capture-with-flink-cdc-and-consistent-hashing/261826138 current2023dynamicchangedatacapturewithflinkcdcandconsistenthashing-231006084815-2354f87d
Change Data Capture (CDC) is a popular technique for extracting data from databases in realtime. However, many CDC deployments are static: e.g. a single connector is configured to ingest data for one or several tables. At Goldsky, we needed a way to configure CDC for a large Postgres database dynamically: the list of tables to ingest is driven by customer-facing features and is constantly changing. We started using Flink CDC connectors built on top of the Debezium project, but we immediately faced many challenges caused mainly by the lack of incremental snapshotting. But even after implementing incremental snapshotting ourselves, we still faced an issue around using replication slots in Postgres: we couldn't use a single connector to ingest all tables (it's just too much data), and we couldn't create a new connector for every new set of tables (we'd quickly run out of replication slots). So we needed to find a way to maintain a fixed number of replication slots for a dynamic list of tables. In the end, we chose a consistent hashing algorithm to distribute the list of tables across multiple Flink jobs. The jobs also required some customizations to support the incremental snapshotting semantics from Flink CDC. We learned a lot about Debezium, Flink CDC and Postgres replication, and we're excited to share our learnings with the community!]]>
Change Data Capture (CDC) is a popular technique for extracting data from databases in realtime. However, many CDC deployments are static: e.g. a single connector is configured to ingest data for one or several tables. At Goldsky, we needed a way to configure CDC for a large Postgres database dynamically: the list of tables to ingest is driven by customer-facing features and is constantly changing. We started using Flink CDC connectors built on top of the Debezium project, but we immediately faced many challenges caused mainly by the lack of incremental snapshotting. But even after implementing incremental snapshotting ourselves, we still faced an issue around using replication slots in Postgres: we couldn't use a single connector to ingest all tables (it's just too much data), and we couldn't create a new connector for every new set of tables (we'd quickly run out of replication slots). So we needed to find a way to maintain a fixed number of replication slots for a dynamic list of tables. In the end, we chose a consistent hashing algorithm to distribute the list of tables across multiple Flink jobs. The jobs also required some customizations to support the incremental snapshotting semantics from Flink CDC. We learned a lot about Debezium, Flink CDC and Postgres replication, and we're excited to share our learnings with the community!]]> Fri, 06 Oct 2023 08:48:15 GMT /slideshow/dynamic-change-data-capture-with-flink-cdc-and-consistent-hashing/261826138 sap1ens@slideshare.net(sap1ens) Dynamic Change Data Capture with Flink CDC and Consistent Hashing sap1ens Change Data Capture (CDC) is a popular technique for extracting data from databases in realtime. However, many CDC deployments are static: e.g. a single connector is configured to ingest data for one or several tables. At Goldsky, we needed a way to configure CDC for a large Postgres database dynamically: the list of tables to ingest is driven by customer-facing features and is constantly changing. We started using Flink CDC connectors built on top of the Debezium project, but we immediately faced many challenges caused mainly by the lack of incremental snapshotting. But even after implementing incremental snapshotting ourselves, we still faced an issue around using replication slots in Postgres: we couldn't use a single connector to ingest all tables (it's just too much data), and we couldn't create a new connector for every new set of tables (we'd quickly run out of replication slots). So we needed to find a way to maintain a fixed number of replication slots for a dynamic list of tables. In the end, we chose a consistent hashing algorithm to distribute the list of tables across multiple Flink jobs. The jobs also required some customizations to support the incremental snapshotting semantics from Flink CDC. We learned a lot about Debezium, Flink CDC and Postgres replication, and we're excited to share our learnings with the community! <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/current2023dynamicchangedatacapturewithflinkcdcandconsistenthashing-231006084815-2354f87d-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> Change Data Capture (CDC) is a popular technique for extracting data from databases in realtime. However, many CDC deployments are static: e.g. a single connector is configured to ingest data for one or several tables. At Goldsky, we needed a way to configure CDC for a large Postgres database dynamically: the list of tables to ingest is driven by customer-facing features and is constantly changing. We started using Flink CDC connectors built on top of the Debezium project, but we immediately faced many challenges caused mainly by the lack of incremental snapshotting. But even after implementing incremental snapshotting ourselves, we still faced an issue around using replication slots in Postgres: we couldn&#39;t use a single connector to ingest all tables (it&#39;s just too much data), and we couldn&#39;t create a new connector for every new set of tables (we&#39;d quickly run out of replication slots). So we needed to find a way to maintain a fixed number of replication slots for a dynamic list of tables. In the end, we chose a consistent hashing algorithm to distribute the list of tables across multiple Flink jobs. The jobs also required some customizations to support the incremental snapshotting semantics from Flink CDC. We learned a lot about Debezium, Flink CDC and Postgres replication, and we&#39;re excited to share our learnings with the community!
Dynamic Change Data Capture with Flink CDC and Consistent Hashing from Yaroslav Tkachenko
]]> 85 0 https://cdn.slidesharecdn.com/ss_thumbnails/current2023dynamicchangedatacapturewithflinkcdcandconsistenthashing-231006084815-2354f87d-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 Streaming SQL for Data Engineers: The Next Big Thing? /slideshow/streaming-sql-for-data-engineers-the-next-big-thing/253353261 yaroslav-tkachenko04-10-2022v2-221005042546-d6aeb5b0
SQL is the lingua franca of data analysis, but should we use it more as data engineers? Modern tools like dbt make it easier to express transformations in SQL, but streaming is more complicated than batch. Streaming pipelines usually require higher SLAs and many CI/CD and observability practices, so data engineers prefer to use familiar languages like Python, Java and Scala along with many useful frameworks and libraries. Can SQL replace that? I was very skeptical when I first heard the idea of using SQL for writing somewhat complex stream-processing data application a few years ago. How do you unit test it? How do you version it? Over the years, Spark SQL streaming, Flink SQL, ksqlDB and similar tools have matured, now they easily support complex stateful transformations. However, developer experience is still questionable: it's easy to write a SQL statement, but how do you maintain it over the years as a long-running application? In this presentation, I hope to share the discoveries I made over the years in this area, as well as working practices and patterns I've seen.]]>
SQL is the lingua franca of data analysis, but should we use it more as data engineers? Modern tools like dbt make it easier to express transformations in SQL, but streaming is more complicated than batch. Streaming pipelines usually require higher SLAs and many CI/CD and observability practices, so data engineers prefer to use familiar languages like Python, Java and Scala along with many useful frameworks and libraries. Can SQL replace that? I was very skeptical when I first heard the idea of using SQL for writing somewhat complex stream-processing data application a few years ago. How do you unit test it? How do you version it? Over the years, Spark SQL streaming, Flink SQL, ksqlDB and similar tools have matured, now they easily support complex stateful transformations. However, developer experience is still questionable: it's easy to write a SQL statement, but how do you maintain it over the years as a long-running application? In this presentation, I hope to share the discoveries I made over the years in this area, as well as working practices and patterns I've seen.]]> Wed, 05 Oct 2022 04:25:45 GMT /slideshow/streaming-sql-for-data-engineers-the-next-big-thing/253353261 sap1ens@slideshare.net(sap1ens) Streaming SQL for Data Engineers: The Next Big Thing? sap1ens SQL is the lingua franca of data analysis, but should we use it more as data engineers? Modern tools like dbt make it easier to express transformations in SQL, but streaming is more complicated than batch. Streaming pipelines usually require higher SLAs and many CI/CD and observability practices, so data engineers prefer to use familiar languages like Python, Java and Scala along with many useful frameworks and libraries. Can SQL replace that? I was very skeptical when I first heard the idea of using SQL for writing somewhat complex stream-processing data application a few years ago. How do you unit test it? How do you version it? Over the years, Spark SQL streaming, Flink SQL, ksqlDB and similar tools have matured, now they easily support complex stateful transformations. However, developer experience is still questionable: it's easy to write a SQL statement, but how do you maintain it over the years as a long-running application? In this presentation, I hope to share the discoveries I made over the years in this area, as well as working practices and patterns I've seen. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/yaroslav-tkachenko04-10-2022v2-221005042546-d6aeb5b0-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> SQL is the lingua franca of data analysis, but should we use it more as data engineers? Modern tools like dbt make it easier to express transformations in SQL, but streaming is more complicated than batch. Streaming pipelines usually require higher SLAs and many CI/CD and observability practices, so data engineers prefer to use familiar languages like Python, Java and Scala along with many useful frameworks and libraries. Can SQL replace that? I was very skeptical when I first heard the idea of using SQL for writing somewhat complex stream-processing data application a few years ago. How do you unit test it? How do you version it? Over the years, Spark SQL streaming, Flink SQL, ksqlDB and similar tools have matured, now they easily support complex stateful transformations. However, developer experience is still questionable: it&#39;s easy to write a SQL statement, but how do you maintain it over the years as a long-running application? In this presentation, I hope to share the discoveries I made over the years in this area, as well as working practices and patterns I&#39;ve seen.
Streaming SQL for Data Engineers: The Next Big Thing? from Yaroslav Tkachenko
]]> 233 0 https://cdn.slidesharecdn.com/ss_thumbnails/yaroslav-tkachenko04-10-2022v2-221005042546-d6aeb5b0-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 Apache Flink Adoption at Shopify /slideshow/apache-flink-adoption-at-shopify/251420147 apacheflinkadoptionatshopify-220324210259
At the beginning of 2021, Shopify Data Platform decided to adopt Apache Flink to enable modern stateful stream-processing. Shopify had a lot of experience with other streaming technologies, but Flink was a great fit due to its state management primitives. After about six months, Shopify now has a flourishing ecosystem of tools, tens of prototypes from many teams across the company and a few large use-cases in production. Yaroslav will share a story about not just building a single data pipeline but building a sustainable ecosystem. You can learn about how they planned their platform roadmap, the tools and libraries Shopify built, the decision to fork Flink, and how Shopify partnered with other teams and drove the adoption of streaming at the company. ]]>
At the beginning of 2021, Shopify Data Platform decided to adopt Apache Flink to enable modern stateful stream-processing. Shopify had a lot of experience with other streaming technologies, but Flink was a great fit due to its state management primitives. After about six months, Shopify now has a flourishing ecosystem of tools, tens of prototypes from many teams across the company and a few large use-cases in production. Yaroslav will share a story about not just building a single data pipeline but building a sustainable ecosystem. You can learn about how they planned their platform roadmap, the tools and libraries Shopify built, the decision to fork Flink, and how Shopify partnered with other teams and drove the adoption of streaming at the company. ]]> Thu, 24 Mar 2022 21:02:59 GMT /slideshow/apache-flink-adoption-at-shopify/251420147 sap1ens@slideshare.net(sap1ens) Apache Flink Adoption at Shopify sap1ens At the beginning of 2021, Shopify Data Platform decided to adopt Apache Flink to enable modern stateful stream-processing. Shopify had a lot of experience with other streaming technologies, but Flink was a great fit due to its state management primitives. After about six months, Shopify now has a flourishing ecosystem of tools, tens of prototypes from many teams across the company and a few large use-cases in production. Yaroslav will share a story about not just building a single data pipeline but building a sustainable ecosystem. You can learn about how they planned their platform roadmap, the tools and libraries Shopify built, the decision to fork Flink, and how Shopify partnered with other teams and drove the adoption of streaming at the company. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/apacheflinkadoptionatshopify-220324210259-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> At the beginning of 2021, Shopify Data Platform decided to adopt Apache Flink to enable modern stateful stream-processing. Shopify had a lot of experience with other streaming technologies, but Flink was a great fit due to its state management primitives. After about six months, Shopify now has a flourishing ecosystem of tools, tens of prototypes from many teams across the company and a few large use-cases in production. Yaroslav will share a story about not just building a single data pipeline but building a sustainable ecosystem. You can learn about how they planned their platform roadmap, the tools and libraries Shopify built, the decision to fork Flink, and how Shopify partnered with other teams and drove the adoption of streaming at the company.
Apache Flink Adoption at Shopify from Yaroslav Tkachenko
]]> 1180 0 https://cdn.slidesharecdn.com/ss_thumbnails/apacheflinkadoptionatshopify-220324210259-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 Storing State Forever: Why It Can Be Good For Your Analytics /slideshow/storing-state-forever-why-it-can-be-good-for-your-analytics/250563042 storingstateforeverwhyitcanbegoodforyouranalytics-211031032330
State is an essential part of the modern streaming pipelines: it enables a variety of foundational capabilities like windowing, aggregation, enrichment, etc. But usually, the state is either transient, so we only keep it until the window is closed, or it's fairly small and doesn't grow much. But what if we treat the state differently? The keyed state in Flink can be scaled vertically and horizontally, it's reliable and fault-tolerant... so is scaling a stateful Flink application that different from scaling any data store like Kafka or MySQL? At Shopify, we've worked on a massive analytical data pipeline that's needed to support complex streaming joins and correctly handle arbitrarily late-arriving data. We came up with an idea to never clear state and support joins this way. We've made a successful proof of concept, ingested all historical transactional Shopify data and ended up storing more than 10 TB of Flink state. In the end, it allowed us to achieve 100% data correctness.]]>
State is an essential part of the modern streaming pipelines: it enables a variety of foundational capabilities like windowing, aggregation, enrichment, etc. But usually, the state is either transient, so we only keep it until the window is closed, or it's fairly small and doesn't grow much. But what if we treat the state differently? The keyed state in Flink can be scaled vertically and horizontally, it's reliable and fault-tolerant... so is scaling a stateful Flink application that different from scaling any data store like Kafka or MySQL? At Shopify, we've worked on a massive analytical data pipeline that's needed to support complex streaming joins and correctly handle arbitrarily late-arriving data. We came up with an idea to never clear state and support joins this way. We've made a successful proof of concept, ingested all historical transactional Shopify data and ended up storing more than 10 TB of Flink state. In the end, it allowed us to achieve 100% data correctness.]]> Sun, 31 Oct 2021 03:23:30 GMT /slideshow/storing-state-forever-why-it-can-be-good-for-your-analytics/250563042 sap1ens@slideshare.net(sap1ens) Storing State Forever: Why It Can Be Good For Your Analytics sap1ens State is an essential part of the modern streaming pipelines: it enables a variety of foundational capabilities like windowing, aggregation, enrichment, etc. But usually, the state is either transient, so we only keep it until the window is closed, or it's fairly small and doesn't grow much. But what if we treat the state differently? The keyed state in Flink can be scaled vertically and horizontally, it's reliable and fault-tolerant... so is scaling a stateful Flink application that different from scaling any data store like Kafka or MySQL? At Shopify, we've worked on a massive analytical data pipeline that's needed to support complex streaming joins and correctly handle arbitrarily late-arriving data. We came up with an idea to never clear state and support joins this way. We've made a successful proof of concept, ingested all historical transactional Shopify data and ended up storing more than 10 TB of Flink state. In the end, it allowed us to achieve 100% data correctness. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/storingstateforeverwhyitcanbegoodforyouranalytics-211031032330-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> State is an essential part of the modern streaming pipelines: it enables a variety of foundational capabilities like windowing, aggregation, enrichment, etc. But usually, the state is either transient, so we only keep it until the window is closed, or it&#39;s fairly small and doesn&#39;t grow much. But what if we treat the state differently? The keyed state in Flink can be scaled vertically and horizontally, it&#39;s reliable and fault-tolerant... so is scaling a stateful Flink application that different from scaling any data store like Kafka or MySQL? At Shopify, we&#39;ve worked on a massive analytical data pipeline that&#39;s needed to support complex streaming joins and correctly handle arbitrarily late-arriving data. We came up with an idea to never clear state and support joins this way. We&#39;ve made a successful proof of concept, ingested all historical transactional Shopify data and ended up storing more than 10 TB of Flink state. In the end, it allowed us to achieve 100% data correctness.
Storing State Forever: Why It Can Be Good For Your Analytics from Yaroslav Tkachenko
]]> 552 0 https://cdn.slidesharecdn.com/ss_thumbnails/storingstateforeverwhyitcanbegoodforyouranalytics-211031032330-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 It's Time To Stop Using Lambda Architecture /slideshow/its-time-to-stop-using-lambda-architecture/249384670 itstimetostopusinglambdaarchitecture-210616212016
Lambda Architecture has been a common way to build data pipelines for a long time, despite difficulties in maintaining two complex systems. An alternative, Kappa Architecture, was proposed in 2014, but many companies are still reluctant to switch to Kappa. And there is a reason for that: even though Kappa generally provides a simpler design and similar or lower latency, there are a lot of practical challenges in areas like exactly-once delivery, late-arriving data, historical backfill and reprocessing. In this talk, I want to show how you can solve those challenges by embracing Apache Kafka as a foundation of your data pipeline and leveraging modern stream-processing frameworks like Apache Flink.]]>
Lambda Architecture has been a common way to build data pipelines for a long time, despite difficulties in maintaining two complex systems. An alternative, Kappa Architecture, was proposed in 2014, but many companies are still reluctant to switch to Kappa. And there is a reason for that: even though Kappa generally provides a simpler design and similar or lower latency, there are a lot of practical challenges in areas like exactly-once delivery, late-arriving data, historical backfill and reprocessing. In this talk, I want to show how you can solve those challenges by embracing Apache Kafka as a foundation of your data pipeline and leveraging modern stream-processing frameworks like Apache Flink.]]> Wed, 16 Jun 2021 21:20:16 GMT /slideshow/its-time-to-stop-using-lambda-architecture/249384670 sap1ens@slideshare.net(sap1ens) It's Time To Stop Using Lambda Architecture sap1ens Lambda Architecture has been a common way to build data pipelines for a long time, despite difficulties in maintaining two complex systems. An alternative, Kappa Architecture, was proposed in 2014, but many companies are still reluctant to switch to Kappa. And there is a reason for that: even though Kappa generally provides a simpler design and similar or lower latency, there are a lot of practical challenges in areas like exactly-once delivery, late-arriving data, historical backfill and reprocessing. In this talk, I want to show how you can solve those challenges by embracing Apache Kafka as a foundation of your data pipeline and leveraging modern stream-processing frameworks like Apache Flink. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/itstimetostopusinglambdaarchitecture-210616212016-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> Lambda Architecture has been a common way to build data pipelines for a long time, despite difficulties in maintaining two complex systems. An alternative, Kappa Architecture, was proposed in 2014, but many companies are still reluctant to switch to Kappa. And there is a reason for that: even though Kappa generally provides a simpler design and similar or lower latency, there are a lot of practical challenges in areas like exactly-once delivery, late-arriving data, historical backfill and reprocessing. In this talk, I want to show how you can solve those challenges by embracing Apache Kafka as a foundation of your data pipeline and leveraging modern stream-processing frameworks like Apache Flink.
It's Time To Stop Using Lambda Architecture from Yaroslav Tkachenko
]]> 246 0 https://cdn.slidesharecdn.com/ss_thumbnails/itstimetostopusinglambdaarchitecture-210616212016-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streaming /sap1ens/bravo-six-going-realtime-transitioning-activision-data-pipeline-to-streaming bravosixgoingrealtime-200818053422
Activision Data team has been running a data pipeline for a variety of Activision games for many years. Historically we used a mix of micro-batch microservices coupled with classic Big Data tools like Hadoop and Hive for ETL. As a result, it could take up to 4-6 hours for data to be available to the end customers. In the last few years, the adoption of data in the organization skyrocketed. We needed to de-legacy our data pipeline and provide near-realtime access to data in order to improve reporting, gather insights faster, power web and mobile applications. I want to tell a story about heavily leveraging Kafka Streams and Kafka Connect to reduce the end latency to minutes, at the same time making the pipeline easier and cheaper to run. We were able to successfully validate the new data pipeline by launching two massive games just 4 weeks apart.]]>
Activision Data team has been running a data pipeline for a variety of Activision games for many years. Historically we used a mix of micro-batch microservices coupled with classic Big Data tools like Hadoop and Hive for ETL. As a result, it could take up to 4-6 hours for data to be available to the end customers. In the last few years, the adoption of data in the organization skyrocketed. We needed to de-legacy our data pipeline and provide near-realtime access to data in order to improve reporting, gather insights faster, power web and mobile applications. I want to tell a story about heavily leveraging Kafka Streams and Kafka Connect to reduce the end latency to minutes, at the same time making the pipeline easier and cheaper to run. We were able to successfully validate the new data pipeline by launching two massive games just 4 weeks apart.]]> Tue, 18 Aug 2020 05:34:22 GMT /sap1ens/bravo-six-going-realtime-transitioning-activision-data-pipeline-to-streaming sap1ens@slideshare.net(sap1ens) Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streaming sap1ens Activision Data team has been running a data pipeline for a variety of Activision games for many years. Historically we used a mix of micro-batch microservices coupled with classic Big Data tools like Hadoop and Hive for ETL. As a result, it could take up to 4-6 hours for data to be available to the end customers. In the last few years, the adoption of data in the organization skyrocketed. We needed to de-legacy our data pipeline and provide near-realtime access to data in order to improve reporting, gather insights faster, power web and mobile applications. I want to tell a story about heavily leveraging Kafka Streams and Kafka Connect to reduce the end latency to minutes, at the same time making the pipeline easier and cheaper to run. We were able to successfully validate the new data pipeline by launching two massive games just 4 weeks apart. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/bravosixgoingrealtime-200818053422-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> Activision Data team has been running a data pipeline for a variety of Activision games for many years. Historically we used a mix of micro-batch microservices coupled with classic Big Data tools like Hadoop and Hive for ETL. As a result, it could take up to 4-6 hours for data to be available to the end customers. In the last few years, the adoption of data in the organization skyrocketed. We needed to de-legacy our data pipeline and provide near-realtime access to data in order to improve reporting, gather insights faster, power web and mobile applications. I want to tell a story about heavily leveraging Kafka Streams and Kafka Connect to reduce the end latency to minutes, at the same time making the pipeline easier and cheaper to run. We were able to successfully validate the new data pipeline by launching two massive games just 4 weeks apart.
Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streaming from Yaroslav Tkachenko
]]> 557 0 https://cdn.slidesharecdn.com/ss_thumbnails/bravosixgoingrealtime-200818053422-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 Apache Kafka: New Features That You Might Not Know About /slideshow/apache-kafka-new-features-that-you-might-not-know-about/130768742 apachekafkanewfeaturesthatyoumightnotknowabout-190206174355
In the last two years Apache Kafka rapidly introduced new versions, going from 0.10.x to 2.x. It can be hard to keep up with all the updates and a lot of companies still run 0.10.x clusters (or even older ones). Join this session to learn new exciting features in Kafka introduced in 0.11, 1.0, 1.1 and 2.0 versions including, but not limited to, the new protocol and message headers, transactional support and exactly-only delivery semantics, as well as controller changes that make it possible to shutdown even large clusters in seconds.]]>
In the last two years Apache Kafka rapidly introduced new versions, going from 0.10.x to 2.x. It can be hard to keep up with all the updates and a lot of companies still run 0.10.x clusters (or even older ones). Join this session to learn new exciting features in Kafka introduced in 0.11, 1.0, 1.1 and 2.0 versions including, but not limited to, the new protocol and message headers, transactional support and exactly-only delivery semantics, as well as controller changes that make it possible to shutdown even large clusters in seconds.]]> Wed, 06 Feb 2019 17:43:55 GMT /slideshow/apache-kafka-new-features-that-you-might-not-know-about/130768742 sap1ens@slideshare.net(sap1ens) Apache Kafka: New Features That You Might Not Know About sap1ens In the last two years Apache Kafka rapidly introduced new versions, going from 0.10.x to 2.x. It can be hard to keep up with all the updates and a lot of companies still run 0.10.x clusters (or even older ones). Join this session to learn new exciting features in Kafka introduced in 0.11, 1.0, 1.1 and 2.0 versions including, but not limited to, the new protocol and message headers, transactional support and exactly-only delivery semantics, as well as controller changes that make it possible to shutdown even large clusters in seconds. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/apachekafkanewfeaturesthatyoumightnotknowabout-190206174355-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> In the last two years Apache Kafka rapidly introduced new versions, going from 0.10.x to 2.x. It can be hard to keep up with all the updates and a lot of companies still run 0.10.x clusters (or even older ones). Join this session to learn new exciting features in Kafka introduced in 0.11, 1.0, 1.1 and 2.0 versions including, but not limited to, the new protocol and message headers, transactional support and exactly-only delivery semantics, as well as controller changes that make it possible to shutdown even large clusters in seconds.
Apache Kafka: New Features That You Might Not Know About from Yaroslav Tkachenko
]]> 4915 0 https://cdn.slidesharecdn.com/ss_thumbnails/apachekafkanewfeaturesthatyoumightnotknowabout-190206174355-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned /slideshow/building-scalable-and-extendable-data-pipeline-for-call-of-duty-games-lessons-learned/112349165 buildingscalableandextendabledatapipelineforcallofdutygameslessonslearned1-180831033005
What can be easier than building a data pipeline nowadays? You add a few Apache Kafka clusters, some way to ingest data (probably over HTTP), design a way to route your data streams, add a few stream processors and consumers, integrate with a data warehouse... wait, it does start to look like A LOT of things, doesn't it? And you probably want to make it highly scalable and available in the end, correct? We've been developing a data pipeline in Demonware/Activision for a while. We learned how to scale it not only in terms of messages/sec it can handle, but also in terms of supporting more games and more use-cases. In this presentation you'll hear about the lessons we learned, including (but not limited to): - Message schemas - Apache Kafka organization and tuning - Topics naming conventions, structure and routing - Reliable and scalable producers and ingestion layer - Stream processing]]>
What can be easier than building a data pipeline nowadays? You add a few Apache Kafka clusters, some way to ingest data (probably over HTTP), design a way to route your data streams, add a few stream processors and consumers, integrate with a data warehouse... wait, it does start to look like A LOT of things, doesn't it? And you probably want to make it highly scalable and available in the end, correct? We've been developing a data pipeline in Demonware/Activision for a while. We learned how to scale it not only in terms of messages/sec it can handle, but also in terms of supporting more games and more use-cases. In this presentation you'll hear about the lessons we learned, including (but not limited to): - Message schemas - Apache Kafka organization and tuning - Topics naming conventions, structure and routing - Reliable and scalable producers and ingestion layer - Stream processing]]> Fri, 31 Aug 2018 03:30:05 GMT /slideshow/building-scalable-and-extendable-data-pipeline-for-call-of-duty-games-lessons-learned/112349165 sap1ens@slideshare.net(sap1ens) Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned sap1ens What can be easier than building a data pipeline nowadays? You add a few Apache Kafka clusters, some way to ingest data (probably over HTTP), design a way to route your data streams, add a few stream processors and consumers, integrate with a data warehouse... wait, it does start to look like A LOT of things, doesn't it? And you probably want to make it highly scalable and available in the end, correct? We've been developing a data pipeline in Demonware/Activision for a while. We learned how to scale it not only in terms of messages/sec it can handle, but also in terms of supporting more games and more use-cases. In this presentation you'll hear about the lessons we learned, including (but not limited to): - Message schemas - Apache Kafka organization and tuning - Topics naming conventions, structure and routing - Reliable and scalable producers and ingestion layer - Stream processing <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/buildingscalableandextendabledatapipelineforcallofdutygameslessonslearned1-180831033005-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> What can be easier than building a data pipeline nowadays? You add a few Apache Kafka clusters, some way to ingest data (probably over HTTP), design a way to route your data streams, add a few stream processors and consumers, integrate with a data warehouse... wait, it does start to look like A LOT of things, doesn&#39;t it? And you probably want to make it highly scalable and available in the end, correct? We&#39;ve been developing a data pipeline in Demonware/Activision for a while. We learned how to scale it not only in terms of messages/sec it can handle, but also in terms of supporting more games and more use-cases. In this presentation you&#39;ll hear about the lessons we learned, including (but not limited to): - Message schemas - Apache Kafka organization and tuning - Topics naming conventions, structure and routing - Reliable and scalable producers and ingestion layer - Stream processing
Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned from Yaroslav Tkachenko
]]> 1881 0 https://cdn.slidesharecdn.com/ss_thumbnails/buildingscalableandextendabledatapipelineforcallofdutygameslessonslearned1-180831033005-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 Designing Scalable and Extendable Data Pipeline for Call Of Duty Games /slideshow/designing-scalable-and-extendable-data-pipeline-for-call-of-duty-games/99972986 designing-scalable-and-extendable-data-pipeline-for-call-of-duty-games-180601190833
Designing Scalable and Extendable Data Pipeline for Call Of Duty Games.]]>
Designing Scalable and Extendable Data Pipeline for Call Of Duty Games.]]> Fri, 01 Jun 2018 19:08:33 GMT /slideshow/designing-scalable-and-extendable-data-pipeline-for-call-of-duty-games/99972986 sap1ens@slideshare.net(sap1ens) Designing Scalable and Extendable Data Pipeline for Call Of Duty Games sap1ens Designing Scalable and Extendable Data Pipeline for Call Of Duty Games. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/designing-scalable-and-extendable-data-pipeline-for-call-of-duty-games-180601190833-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> Designing Scalable and Extendable Data Pipeline for Call Of Duty Games.
Designing Scalable and Extendable Data Pipeline for Call Of Duty Games from Yaroslav Tkachenko
]]> 1068 0 https://cdn.slidesharecdn.com/ss_thumbnails/designing-scalable-and-extendable-data-pipeline-for-call-of-duty-games-180601190833-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 10 tips for making Bash a sane programming language /slideshow/10-tips-for-making-bash-a-sane-programming-language/94643137 bash-180422023229
Do you sometimes feel like long Bash scripts look like a mix of dark magic and rocket science? Me too! But Ive finally managed to understand a set of reasonable guidelines to make my interaction with Bash not only effective, but also enjoyable! Now you can learn the best 10 Bash scripting tips too.]]>
Do you sometimes feel like long Bash scripts look like a mix of dark magic and rocket science? Me too! But Ive finally managed to understand a set of reasonable guidelines to make my interaction with Bash not only effective, but also enjoyable! Now you can learn the best 10 Bash scripting tips too.]]> Sun, 22 Apr 2018 02:32:29 GMT /slideshow/10-tips-for-making-bash-a-sane-programming-language/94643137 sap1ens@slideshare.net(sap1ens) 10 tips for making Bash a sane programming language sap1ens Do you sometimes feel like long Bash scripts look like a mix of dark magic and rocket science? Me too! But Ive finally managed to understand a set of reasonable guidelines to make my interaction with Bash not only effective, but also enjoyable! Now you can learn the best 10 Bash scripting tips too. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/bash-180422023229-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> Do you sometimes feel like long Bash scripts look like a mix of dark magic and rocket science? Me too! But Ive finally managed to understand a set of reasonable guidelines to make my interaction with Bash not only effective, but also enjoyable! Now you can learn the best 10 Bash scripting tips too.
10 tips for making Bash a sane programming language from Yaroslav Tkachenko
]]> 778 0 https://cdn.slidesharecdn.com/ss_thumbnails/bash-180422023229-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 Actors or Not: Async Event Architectures /slideshow/actors-or-not-async-event-architectures/89955066 actorsornotasynceventarchitectures-180307195147
With more and more companies adopting microservices and service-oriented architectures, it becomes clear that the HTTP/RPC synchronous communication (while great) is not always the best option for every use case. In this presentation, I discuss two approaches to an asynchronous event-based architecture. The first is a "classic" style protocol (Python services driven by callbacks with decorators communicating using a messaging layer) that we've been implementing at Demonware (Activision) for Call of Duty back-end services. The second is an actor-based approach (Scala/Akka based microservices communicating using a messaging layer and a centralized router) in place at Bench Accounting. Both systems, while event based, take different approaches to building asynchronous, reactive applications. This talk explores the benefits, challenges, and lessons learned architecting both Actor and Non-Actor systems.]]>
With more and more companies adopting microservices and service-oriented architectures, it becomes clear that the HTTP/RPC synchronous communication (while great) is not always the best option for every use case. In this presentation, I discuss two approaches to an asynchronous event-based architecture. The first is a "classic" style protocol (Python services driven by callbacks with decorators communicating using a messaging layer) that we've been implementing at Demonware (Activision) for Call of Duty back-end services. The second is an actor-based approach (Scala/Akka based microservices communicating using a messaging layer and a centralized router) in place at Bench Accounting. Both systems, while event based, take different approaches to building asynchronous, reactive applications. This talk explores the benefits, challenges, and lessons learned architecting both Actor and Non-Actor systems.]]> Wed, 07 Mar 2018 19:51:47 GMT /slideshow/actors-or-not-async-event-architectures/89955066 sap1ens@slideshare.net(sap1ens) Actors or Not: Async Event Architectures sap1ens With more and more companies adopting microservices and service-oriented architectures, it becomes clear that the HTTP/RPC synchronous communication (while great) is not always the best option for every use case. In this presentation, I discuss two approaches to an asynchronous event-based architecture. The first is a "classic" style protocol (Python services driven by callbacks with decorators communicating using a messaging layer) that we've been implementing at Demonware (Activision) for Call of Duty back-end services. The second is an actor-based approach (Scala/Akka based microservices communicating using a messaging layer and a centralized router) in place at Bench Accounting. Both systems, while event based, take different approaches to building asynchronous, reactive applications. This talk explores the benefits, challenges, and lessons learned architecting both Actor and Non-Actor systems. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/actorsornotasynceventarchitectures-180307195147-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> With more and more companies adopting microservices and service-oriented architectures, it becomes clear that the HTTP/RPC synchronous communication (while great) is not always the best option for every use case. In this presentation, I discuss two approaches to an asynchronous event-based architecture. The first is a &quot;classic&quot; style protocol (Python services driven by callbacks with decorators communicating using a messaging layer) that we&#39;ve been implementing at Demonware (Activision) for Call of Duty back-end services. The second is an actor-based approach (Scala/Akka based microservices communicating using a messaging layer and a centralized router) in place at Bench Accounting. Both systems, while event based, take different approaches to building asynchronous, reactive applications. This talk explores the benefits, challenges, and lessons learned architecting both Actor and Non-Actor systems.
Actors or Not: Async Event Architectures from Yaroslav Tkachenko
]]> 1833 0 https://cdn.slidesharecdn.com/ss_thumbnails/actorsornotasynceventarchitectures-180307195147-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 Kafka Streams: the easiest way to start with stream processing /slideshow/kafka-streams-the-easiest-way-to-start-with-stream-processing/86801946 kafkastreamstheeasiestwaytostartwithstreamprocessing-180128061206
Stream processing is getting more & more important in our data-centric systems. In the world of Big Data, batch processing is not enough anymore - everyone needs interactive, real-time analytics for making critical business decisions, as well as providing great features to the customers. There are many stream processing frameworks available nowadays, but the cost of provisioning infrastructure and maintaining distributed computations is usually very high. Sometimes you just have to satisfy some specific requirements, like using HDFS or YARN. Apache Kafka is de facto a standard for building data pipelines. Kafka Streams is a lightweight library (available since 0.10) that uses powerful Kafka abstractions internally and doesn't require any complex setup or special infrastructure - you just deploy it like any other regular application. In this session I want to talk about the goals behind stream processing, basic techniques and some best practices. Then I'm going to explain main fundamental concepts behind Kafka and explore Kafka Streams syntax and streaming features. By the end of the session you'll be able to write stream processing applications in your domain, especially if you already use Kafka as your data pipeline.]]>
Stream processing is getting more & more important in our data-centric systems. In the world of Big Data, batch processing is not enough anymore - everyone needs interactive, real-time analytics for making critical business decisions, as well as providing great features to the customers. There are many stream processing frameworks available nowadays, but the cost of provisioning infrastructure and maintaining distributed computations is usually very high. Sometimes you just have to satisfy some specific requirements, like using HDFS or YARN. Apache Kafka is de facto a standard for building data pipelines. Kafka Streams is a lightweight library (available since 0.10) that uses powerful Kafka abstractions internally and doesn't require any complex setup or special infrastructure - you just deploy it like any other regular application. In this session I want to talk about the goals behind stream processing, basic techniques and some best practices. Then I'm going to explain main fundamental concepts behind Kafka and explore Kafka Streams syntax and streaming features. By the end of the session you'll be able to write stream processing applications in your domain, especially if you already use Kafka as your data pipeline.]]> Sun, 28 Jan 2018 06:12:05 GMT /slideshow/kafka-streams-the-easiest-way-to-start-with-stream-processing/86801946 sap1ens@slideshare.net(sap1ens) Kafka Streams: the easiest way to start with stream processing sap1ens Stream processing is getting more & more important in our data-centric systems. In the world of Big Data, batch processing is not enough anymore - everyone needs interactive, real-time analytics for making critical business decisions, as well as providing great features to the customers. There are many stream processing frameworks available nowadays, but the cost of provisioning infrastructure and maintaining distributed computations is usually very high. Sometimes you just have to satisfy some specific requirements, like using HDFS or YARN. Apache Kafka is de facto a standard for building data pipelines. Kafka Streams is a lightweight library (available since 0.10) that uses powerful Kafka abstractions internally and doesn't require any complex setup or special infrastructure - you just deploy it like any other regular application. In this session I want to talk about the goals behind stream processing, basic techniques and some best practices. Then I'm going to explain main fundamental concepts behind Kafka and explore Kafka Streams syntax and streaming features. By the end of the session you'll be able to write stream processing applications in your domain, especially if you already use Kafka as your data pipeline. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/kafkastreamstheeasiestwaytostartwithstreamprocessing-180128061206-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> Stream processing is getting more &amp; more important in our data-centric systems. In the world of Big Data, batch processing is not enough anymore - everyone needs interactive, real-time analytics for making critical business decisions, as well as providing great features to the customers. There are many stream processing frameworks available nowadays, but the cost of provisioning infrastructure and maintaining distributed computations is usually very high. Sometimes you just have to satisfy some specific requirements, like using HDFS or YARN. Apache Kafka is de facto a standard for building data pipelines. Kafka Streams is a lightweight library (available since 0.10) that uses powerful Kafka abstractions internally and doesn&#39;t require any complex setup or special infrastructure - you just deploy it like any other regular application. In this session I want to talk about the goals behind stream processing, basic techniques and some best practices. Then I&#39;m going to explain main fundamental concepts behind Kafka and explore Kafka Streams syntax and streaming features. By the end of the session you&#39;ll be able to write stream processing applications in your domain, especially if you already use Kafka as your data pipeline.
Kafka Streams: the easiest way to start with stream processing from Yaroslav Tkachenko
]]> 6578 0 https://cdn.slidesharecdn.com/ss_thumbnails/kafkastreamstheeasiestwaytostartwithstreamprocessing-180128061206-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 Building Stateful Microservices With Akka /slideshow/building-stateful-microservices-with-akka/81117250 buildingstatefulmicroserviceswithakka-171023200657
Although most microservices are stateless - they delegate things like persistence and consistency to a database or external storage. But sometimes you benefit when you keep the state inside the application. In this talk Im going to discuss why you want to build stateful microservices and design choices to make. Ill use Akka framework and explain tools like Akka Clustering and Akka Persistence in depth and show a few practical examples.]]>
Although most microservices are stateless - they delegate things like persistence and consistency to a database or external storage. But sometimes you benefit when you keep the state inside the application. In this talk Im going to discuss why you want to build stateful microservices and design choices to make. Ill use Akka framework and explain tools like Akka Clustering and Akka Persistence in depth and show a few practical examples.]]> Mon, 23 Oct 2017 20:06:56 GMT /slideshow/building-stateful-microservices-with-akka/81117250 sap1ens@slideshare.net(sap1ens) Building Stateful Microservices With Akka sap1ens Although most microservices are stateless - they delegate things like persistence and consistency to a database or external storage. But sometimes you benefit when you keep the state inside the application. In this talk Im going to discuss why you want to build stateful microservices and design choices to make. Ill use Akka framework and explain tools like Akka Clustering and Akka Persistence in depth and show a few practical examples. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/buildingstatefulmicroserviceswithakka-171023200657-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> Although most microservices are stateless - they delegate things like persistence and consistency to a database or external storage. But sometimes you benefit when you keep the state inside the application. In this talk Im going to discuss why you want to build stateful microservices and design choices to make. Ill use Akka framework and explain tools like Akka Clustering and Akka Persistence in depth and show a few practical examples.
Building Stateful Microservices With Akka from Yaroslav Tkachenko
]]> 1948 0 https://cdn.slidesharecdn.com/ss_thumbnails/buildingstatefulmicroserviceswithakka-171023200657-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 Querying Data Pipeline with AWS Athena /slideshow/querying-data-pipeline-with-aws-athena/80241590 queryingdatapipelinewithawsathena-170928051304
Data Pipeline team at Demonware (Activision) has to deal with routing large amounts of data from various sources to many destinations every day. Our team always wanted to be able to query processed data for debugging and analytical purposes, but creating large data warehouses was never our priority, since it usually happens downstream. AWS Athena is completely serverless query service that doesn't require any infrastructure setup or complex provisioning. We just needed to save some of our data streams to AWS S3 and define a schema. Just a few simple steps, but in the end we were able to write complex SQL queries against gigabytes of data and get results in seconds. In this presentation I want to show multiple ways to stream your data to AWS S3, explain some underlying tech, show how to define a schema and finally share some of the best practices we applied.]]>
Data Pipeline team at Demonware (Activision) has to deal with routing large amounts of data from various sources to many destinations every day. Our team always wanted to be able to query processed data for debugging and analytical purposes, but creating large data warehouses was never our priority, since it usually happens downstream. AWS Athena is completely serverless query service that doesn't require any infrastructure setup or complex provisioning. We just needed to save some of our data streams to AWS S3 and define a schema. Just a few simple steps, but in the end we were able to write complex SQL queries against gigabytes of data and get results in seconds. In this presentation I want to show multiple ways to stream your data to AWS S3, explain some underlying tech, show how to define a schema and finally share some of the best practices we applied.]]> Thu, 28 Sep 2017 05:13:04 GMT /slideshow/querying-data-pipeline-with-aws-athena/80241590 sap1ens@slideshare.net(sap1ens) Querying Data Pipeline with AWS Athena sap1ens Data Pipeline team at Demonware (Activision) has to deal with routing large amounts of data from various sources to many destinations every day. Our team always wanted to be able to query processed data for debugging and analytical purposes, but creating large data warehouses was never our priority, since it usually happens downstream. AWS Athena is completely serverless query service that doesn't require any infrastructure setup or complex provisioning. We just needed to save some of our data streams to AWS S3 and define a schema. Just a few simple steps, but in the end we were able to write complex SQL queries against gigabytes of data and get results in seconds. In this presentation I want to show multiple ways to stream your data to AWS S3, explain some underlying tech, show how to define a schema and finally share some of the best practices we applied. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/queryingdatapipelinewithawsathena-170928051304-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> Data Pipeline team at Demonware (Activision) has to deal with routing large amounts of data from various sources to many destinations every day. Our team always wanted to be able to query processed data for debugging and analytical purposes, but creating large data warehouses was never our priority, since it usually happens downstream. AWS Athena is completely serverless query service that doesn&#39;t require any infrastructure setup or complex provisioning. We just needed to save some of our data streams to AWS S3 and define a schema. Just a few simple steps, but in the end we were able to write complex SQL queries against gigabytes of data and get results in seconds. In this presentation I want to show multiple ways to stream your data to AWS S3, explain some underlying tech, show how to define a schema and finally share some of the best practices we applied.
Querying Data Pipeline with AWS Athena from Yaroslav Tkachenko
]]> 3217 0 https://cdn.slidesharecdn.com/ss_thumbnails/queryingdatapipelinewithawsathena-170928051304-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 Akka Microservices Architecture And Design /sap1ens/akka-microservices-architecture-and-design akkamicroservices-170722205448
Nowadays Akka is a popular choice for building distributed systems - there are a lot of case studies and successful examples in the industry. But it still can be hard to switch to actor-based systems, because most of the tutorials and documentation don't show the way to assemble a real application using actors, especially in microservices environment. Actor is a powerful abstraction in the message-driven environments, but it can be challenging to use familiar patterns and methodologies. At the same time, message-driven nature of actors is the biggest advantage that can be used for Reactive systems and microservices. I want to share my experience and show how Domain-Driven Design and Enterprise Integration Patterns can be leveraged to design and build fine-grained microservices with synchronous and asynchronous communication. I'll focus on the core Akka functionality, but also explain how advanced features like Akka Persistence and Akka Cluster Sharding can be used together for achieving incredible results.]]>
Nowadays Akka is a popular choice for building distributed systems - there are a lot of case studies and successful examples in the industry. But it still can be hard to switch to actor-based systems, because most of the tutorials and documentation don't show the way to assemble a real application using actors, especially in microservices environment. Actor is a powerful abstraction in the message-driven environments, but it can be challenging to use familiar patterns and methodologies. At the same time, message-driven nature of actors is the biggest advantage that can be used for Reactive systems and microservices. I want to share my experience and show how Domain-Driven Design and Enterprise Integration Patterns can be leveraged to design and build fine-grained microservices with synchronous and asynchronous communication. I'll focus on the core Akka functionality, but also explain how advanced features like Akka Persistence and Akka Cluster Sharding can be used together for achieving incredible results.]]> Sat, 22 Jul 2017 20:54:48 GMT /sap1ens/akka-microservices-architecture-and-design sap1ens@slideshare.net(sap1ens) Akka Microservices Architecture And Design sap1ens Nowadays Akka is a popular choice for building distributed systems - there are a lot of case studies and successful examples in the industry. But it still can be hard to switch to actor-based systems, because most of the tutorials and documentation don't show the way to assemble a real application using actors, especially in microservices environment. Actor is a powerful abstraction in the message-driven environments, but it can be challenging to use familiar patterns and methodologies. At the same time, message-driven nature of actors is the biggest advantage that can be used for Reactive systems and microservices. I want to share my experience and show how Domain-Driven Design and Enterprise Integration Patterns can be leveraged to design and build fine-grained microservices with synchronous and asynchronous communication. I'll focus on the core Akka functionality, but also explain how advanced features like Akka Persistence and Akka Cluster Sharding can be used together for achieving incredible results. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/akkamicroservices-170722205448-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> Nowadays Akka is a popular choice for building distributed systems - there are a lot of case studies and successful examples in the industry. But it still can be hard to switch to actor-based systems, because most of the tutorials and documentation don&#39;t show the way to assemble a real application using actors, especially in microservices environment. Actor is a powerful abstraction in the message-driven environments, but it can be challenging to use familiar patterns and methodologies. At the same time, message-driven nature of actors is the biggest advantage that can be used for Reactive systems and microservices. I want to share my experience and show how Domain-Driven Design and Enterprise Integration Patterns can be leveraged to design and build fine-grained microservices with synchronous and asynchronous communication. I&#39;ll focus on the core Akka functionality, but also explain how advanced features like Akka Persistence and Akka Cluster Sharding can be used together for achieving incredible results.
Akka Microservices Architecture And Design from Yaroslav Tkachenko
]]> 4600 0 https://cdn.slidesharecdn.com/ss_thumbnails/akkamicroservices-170722205448-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 Why Actor-Based Systems Are The Best For Microservices /slideshow/why-actorbased-systems-are-the-best-for-microservices-75503998/75503998 yaroslavtkachenko-170428142948
Nowadays microservices are widely adopted in the industry, but we're still trying to understand the best practices for building and maintaining successful systems. HTTP/RPC synchronous communication is very popular in microservices, but it has a lot of challenges like service discovery, retries, back-pressure, etc. From the other side, microservices can be build to communicate asynchronously using messages. This approach also has its pros and cons, but I'm confident that most of the businesses can benefit from it. Also, messages are natural way to exchange data in actor-based systems, so it should be possible to leverage actors for building great microservices environments. I want to share my experience and show how Enterprise Integration Patterns can be used to design and build fine-grained microservices with asynchronous communication using actors.]]>
Nowadays microservices are widely adopted in the industry, but we're still trying to understand the best practices for building and maintaining successful systems. HTTP/RPC synchronous communication is very popular in microservices, but it has a lot of challenges like service discovery, retries, back-pressure, etc. From the other side, microservices can be build to communicate asynchronously using messages. This approach also has its pros and cons, but I'm confident that most of the businesses can benefit from it. Also, messages are natural way to exchange data in actor-based systems, so it should be possible to leverage actors for building great microservices environments. I want to share my experience and show how Enterprise Integration Patterns can be used to design and build fine-grained microservices with asynchronous communication using actors.]]> Fri, 28 Apr 2017 14:29:47 GMT /slideshow/why-actorbased-systems-are-the-best-for-microservices-75503998/75503998 sap1ens@slideshare.net(sap1ens) Why Actor-Based Systems Are The Best For Microservices sap1ens Nowadays microservices are widely adopted in the industry, but we're still trying to understand the best practices for building and maintaining successful systems. HTTP/RPC synchronous communication is very popular in microservices, but it has a lot of challenges like service discovery, retries, back-pressure, etc. From the other side, microservices can be build to communicate asynchronously using messages. This approach also has its pros and cons, but I'm confident that most of the businesses can benefit from it. Also, messages are natural way to exchange data in actor-based systems, so it should be possible to leverage actors for building great microservices environments. I want to share my experience and show how Enterprise Integration Patterns can be used to design and build fine-grained microservices with asynchronous communication using actors. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/yaroslavtkachenko-170428142948-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> Nowadays microservices are widely adopted in the industry, but we&#39;re still trying to understand the best practices for building and maintaining successful systems. HTTP/RPC synchronous communication is very popular in microservices, but it has a lot of challenges like service discovery, retries, back-pressure, etc. From the other side, microservices can be build to communicate asynchronously using messages. This approach also has its pros and cons, but I&#39;m confident that most of the businesses can benefit from it. Also, messages are natural way to exchange data in actor-based systems, so it should be possible to leverage actors for building great microservices environments. I want to share my experience and show how Enterprise Integration Patterns can be used to design and build fine-grained microservices with asynchronous communication using actors.
Why Actor-Based Systems Are The Best For Microservices from Yaroslav Tkachenko
]]> 946 0 https://cdn.slidesharecdn.com/ss_thumbnails/yaroslavtkachenko-170428142948-thumbnail.jpg?width=120&height=120&fit=bounds presentation 000000 http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 Why actor-based systems are the best for microservices /slideshow/why-actorbased-systems-are-the-best-for-microservices/62501495 actor-based-microservices-160528230602
I explain why it's important to have asynchronous messaging as your main internal communication protocol for microservices and how Actors can fit it. ]]>
I explain why it's important to have asynchronous messaging as your main internal communication protocol for microservices and how Actors can fit it. ]]> Sat, 28 May 2016 23:06:02 GMT /slideshow/why-actorbased-systems-are-the-best-for-microservices/62501495 sap1ens@slideshare.net(sap1ens) Why actor-based systems are the best for microservices sap1ens I explain why it's important to have asynchronous messaging as your main internal communication protocol for microservices and how Actors can fit it. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/actor-based-microservices-160528230602-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> I explain why it&#39;s important to have asynchronous messaging as your main internal communication protocol for microservices and how Actors can fit it.
Why actor-based systems are the best for microservices from Yaroslav Tkachenko
]]> 4343 0 https://cdn.slidesharecdn.com/ss_thumbnails/actor-based-microservices-160528230602-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 Building Eventing Systems for Microservice Architecture /slideshow/building-eventing-systems-for-microservice-architecture/55051061 eventing-service-151112182846-lva1-app6892
In Bench Accounting we heavily use various events as first class citizens: notifications, in-app TODO lists (and messaging solution in future) rely on the eventing framework we built. Recently weve migrated our old legacy eventing system to the new framework with a focus on microservices architecture. Weve chosen event sourcing approach as well as tools like Akka, Camel, ActiveMQ, Slick and Postgres (JSONB). In this presentation I would like to share high-level overview of the system, implementation details and challenges weve faced.]]>
In Bench Accounting we heavily use various events as first class citizens: notifications, in-app TODO lists (and messaging solution in future) rely on the eventing framework we built. Recently weve migrated our old legacy eventing system to the new framework with a focus on microservices architecture. Weve chosen event sourcing approach as well as tools like Akka, Camel, ActiveMQ, Slick and Postgres (JSONB). In this presentation I would like to share high-level overview of the system, implementation details and challenges weve faced.]]> Thu, 12 Nov 2015 18:28:46 GMT /slideshow/building-eventing-systems-for-microservice-architecture/55051061 sap1ens@slideshare.net(sap1ens) Building Eventing Systems for Microservice Architecture sap1ens In Bench Accounting we heavily use various events as first class citizens: notifications, in-app TODO lists (and messaging solution in future) rely on the eventing framework we built. Recently weve migrated our old legacy eventing system to the new framework with a focus on microservices architecture. Weve chosen event sourcing approach as well as tools like Akka, Camel, ActiveMQ, Slick and Postgres (JSONB). In this presentation I would like to share high-level overview of the system, implementation details and challenges weve faced. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/eventing-service-151112182846-lva1-app6892-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br> In Bench Accounting we heavily use various events as first class citizens: notifications, in-app TODO lists (and messaging solution in future) rely on the eventing framework we built. Recently weve migrated our old legacy eventing system to the new framework with a focus on microservices architecture. Weve chosen event sourcing approach as well as tools like Akka, Camel, ActiveMQ, Slick and Postgres (JSONB). In this presentation I would like to share high-level overview of the system, implementation details and challenges weve faced.
Building Eventing Systems for Microservice Architecture from Yaroslav Tkachenko
]]> 3664 0 https://cdn.slidesharecdn.com/ss_thumbnails/eventing-service-151112182846-lva1-app6892-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 舒 亳 弍亠亰弍仂仍亠亰仆亠仆仆舒 舒亰舒弍仂从舒 从仍亳亠仆从仂亶 舒亳 于亠弍-仗亳仍仂亢亠仆亳亶 /slideshow/sap1ens/13082275 sap1ens-120526000536-phpapp02
]]>
]]> Sat, 26 May 2012 00:05:34 GMT /slideshow/sap1ens/13082275 sap1ens@slideshare.net(sap1ens) 舒 亳 弍亠亰弍仂仍亠亰仆亠仆仆舒 舒亰舒弍仂从舒 从仍亳亠仆从仂亶 舒亳 于亠弍-仗亳仍仂亢亠仆亳亶 sap1ens <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/sap1ens-120526000536-phpapp02-thumbnail.jpg?width=120&amp;height=120&amp;fit=bounds" /><br>
舒 亳 弍亠亰弍仂仍亠亰仆亠仆仆舒 舒亰舒弍仂从舒 从仍亳亠仆从仂亶 舒亳 于亠弍-仗亳仍仂亢亠仆亳亶 from Yaroslav Tkachenko
]]> 802 2 https://cdn.slidesharecdn.com/ss_thumbnails/sap1ens-120526000536-phpapp02-thumbnail.jpg?width=120&height=120&fit=bounds presentation White http://activitystrea.ms/schema/1.0/post http://activitystrea.ms/schema/1.0/posted 0 https://cdn.slidesharecdn.com/profile-photo-sap1ens-48x48.jpg?cb=1699640642 Experienced software engineer and architect focused on data streaming, databases, data platforms and distributed systems. sap1ens.com https://cdn.slidesharecdn.com/ss_thumbnails/current2023dynamicchangedatacapturewithflinkcdcandconsistenthashing-231006084815-2354f87d-thumbnail.jpg?width=320&height=320&fit=bounds slideshow/dynamic-change-data-capture-with-flink-cdc-and-consistent-hashing/261826138 Dynamic Change Data Ca... https://cdn.slidesharecdn.com/ss_thumbnails/yaroslav-tkachenko04-10-2022v2-221005042546-d6aeb5b0-thumbnail.jpg?width=320&height=320&fit=bounds slideshow/streaming-sql-for-data-engineers-the-next-big-thing/253353261 Streaming SQL for Data... https://cdn.slidesharecdn.com/ss_thumbnails/apacheflinkadoptionatshopify-220324210259-thumbnail.jpg?width=320&height=320&fit=bounds slideshow/apache-flink-adoption-at-shopify/251420147 Apache Flink Adoption ...