�ݺ�ߣshows by User: shiv4289

�ݺ�ߣshows by User: shiv4289 / http://www.slideshare.net/images/logo.gif �ݺ�ߣshows by User: shiv4289 / Tue, 09 Jul 2024 04:44:16 GMT �ݺ�ߣShare feed for �ݺ�ߣshows by User: shiv4289 Batch to near-realtime: inspired by a real production incident /slideshow/batch-to-near-realtime-inspired-by-a-real-production-incident/270137775 batch-to-realtime-platformatory-meetup-6-july-2024-240709044416-b215b7e9
This slide deck was used for the platformatory streams meetup in Bengaluru on July 7, 2024. This is a real world account from an Apache Druid cluster in production. A story of 48 hours of debugging, learning and understanding batch vs stream better, filing a couple of issues in Druid open source projects and finally a stable production pipeline again thanks to the Druid community. We will discuss what parts of your design could be impacted, how you should change the related systems so the cascading failures don’t bring down your complete production availability. As an example, we will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how to mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our Grafana dashboards. Finally we will list all the leanings and how we made sure we never repeat the same mistakes in production systems.]]>
This slide deck was used for the platformatory streams meetup in Bengaluru on July 7, 2024. This is a real world account from an Apache Druid cluster in production. A story of 48 hours of debugging, learning and understanding batch vs stream better, filing a couple of issues in Druid open source projects and finally a stable production pipeline again thanks to the Druid community. We will discuss what parts of your design could be impacted, how you should change the related systems so the cascading failures don’t bring down your complete production availability. As an example, we will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how to mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our Grafana dashboards. Finally we will list all the leanings and how we made sure we never repeat the same mistakes in production systems.]]> Tue, 09 Jul 2024 04:44:16 GMT /slideshow/batch-to-near-realtime-inspired-by-a-real-production-incident/270137775 shiv4289@slideshare.net(shiv4289) Batch to near-realtime: inspired by a real production incident shiv4289 This slide deck was used for the platformatory streams meetup in Bengaluru on July 7, 2024. This is a real world account from an Apache Druid cluster in production. A story of 48 hours of debugging, learning and understanding batch vs stream better, filing a couple of issues in Druid open source projects and finally a stable production pipeline again thanks to the Druid community. We will discuss what parts of your design could be impacted, how you should change the related systems so the cascading failures don’t bring down your complete production availability. As an example, we will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how to mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our Grafana dashboards. Finally we will list all the leanings and how we made sure we never repeat the same mistakes in production systems. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/batch-to-realtime-platformatory-meetup-6-july-2024-240709044416-b215b7e9-thumbnail.jpg?width=120&height=120&fit=bounds" /> This slide deck was used for the platformatory streams meetup in Bengaluru on July 7, 2024. This is a real world account from an Apache Druid cluster in production. A story of 48 hours of debugging, learning and understanding batch vs stream better, filing a couple of issues in Druid open source projects and finally a stable production pipeline again thanks to the Druid community. We will discuss what parts of your design could be impacted, how you should change the related systems so the cascading failures don’t bring down your complete production availability. As an example, we will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how to mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our Grafana dashboards. Finally we will list all the leanings and how we made sure we never repeat the same mistakes in production systems.

Batch to near-realtime: inspired by a real production incident from Shivji Kumar Jha

]]> 16 0 https://cdn.slidesharecdn.com/ss_thumbnails/batch-to-realtime-platformatory-meetup-6-july-2024-240709044416-b215b7e9-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 Navigating Transactions: ACID Complexity in Modern Databases /slideshow/navigating-transactions-acid-complexity-in-modern-databases/266088421 transactions1-240203033846-28a1a2e2
Transactions are anything but straightforward, with each database vendor offering its unique interpretation of the term. By scrutinising the internal architectures of these databases, engineers can gain valuable insights, enabling them to write more stable applications.This talk explores the intricacies of transactions, focusing on modern databases. Delving into distributed transactions, we discuss network challenges and cloud deployments in the contemporary era. The session provides a concise examination of the internal architectures of cloud-scale, multi-tenant databases such as Spanner, DynamoDB, and Amazon Aurora.]]>
Transactions are anything but straightforward, with each database vendor offering its unique interpretation of the term. By scrutinising the internal architectures of these databases, engineers can gain valuable insights, enabling them to write more stable applications.This talk explores the intricacies of transactions, focusing on modern databases. Delving into distributed transactions, we discuss network challenges and cloud deployments in the contemporary era. The session provides a concise examination of the internal architectures of cloud-scale, multi-tenant databases such as Spanner, DynamoDB, and Amazon Aurora.]]> Sat, 03 Feb 2024 03:38:46 GMT /slideshow/navigating-transactions-acid-complexity-in-modern-databases/266088421 shiv4289@slideshare.net(shiv4289) Navigating Transactions: ACID Complexity in Modern Databases shiv4289 Transactions are anything but straightforward, with each database vendor offering its unique interpretation of the term. By scrutinising the internal architectures of these databases, engineers can gain valuable insights, enabling them to write more stable applications.This talk explores the intricacies of transactions, focusing on modern databases. Delving into distributed transactions, we discuss network challenges and cloud deployments in the contemporary era. The session provides a concise examination of the internal architectures of cloud-scale, multi-tenant databases such as Spanner, DynamoDB, and Amazon Aurora. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/transactions1-240203033846-28a1a2e2-thumbnail.jpg?width=120&height=120&fit=bounds" /> Transactions are anything but straightforward, with each database vendor offering its unique interpretation of the term. By scrutinising the internal architectures of these databases, engineers can gain valuable insights, enabling them to write more stable applications.This talk explores the intricacies of transactions, focusing on modern databases. Delving into distributed transactions, we discuss network challenges and cloud deployments in the contemporary era. The session provides a concise examination of the internal architectures of cloud-scale, multi-tenant databases such as Spanner, DynamoDB, and Amazon Aurora.

Navigating Transactions: ACID Complexity in Modern Databases from Shivji Kumar Jha

]]> 110 0 https://cdn.slidesharecdn.com/ss_thumbnails/transactions1-240203033846-28a1a2e2-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 Druid Summit 2023 : Changing Druid Ingestion from 3 hours to 5 minutes /shiv4289/druid-summit-2023-changing-druid-ingestion-from-3-hours-to-5-minutes druidsummit2023slides-231123173714-c20c2b42
This is a real world account from a Druid cluster in production. A story of 48 hours of debugging, learning and understanding Druid better, filing a couple of issues in Druid github and finally a stable production pipeline again thanks to the Druid community. We will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how we mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our grafana dashboards.]]>
This is a real world account from a Druid cluster in production. A story of 48 hours of debugging, learning and understanding Druid better, filing a couple of issues in Druid github and finally a stable production pipeline again thanks to the Druid community. We will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how we mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our grafana dashboards.]]> Thu, 23 Nov 2023 17:37:14 GMT /shiv4289/druid-summit-2023-changing-druid-ingestion-from-3-hours-to-5-minutes shiv4289@slideshare.net(shiv4289) Druid Summit 2023 : Changing Druid Ingestion from 3 hours to 5 minutes shiv4289 This is a real world account from a Druid cluster in production. A story of 48 hours of debugging, learning and understanding Druid better, filing a couple of issues in Druid github and finally a stable production pipeline again thanks to the Druid community. We will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how we mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our grafana dashboards. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/druidsummit2023slides-231123173714-c20c2b42-thumbnail.jpg?width=120&height=120&fit=bounds" /> This is a real world account from a Druid cluster in production. A story of 48 hours of debugging, learning and understanding Druid better, filing a couple of issues in Druid github and finally a stable production pipeline again thanks to the Druid community. We will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how we mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our grafana dashboards.

Druid Summit 2023 : Changing Druid Ingestion from 3 hours to 5 minutes from Shivji Kumar Jha

]]> 39 0 https://cdn.slidesharecdn.com/ss_thumbnails/druidsummit2023slides-231123173714-c20c2b42-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 osi-oss-dbs.pptx /slideshow/osiossdbspptx/262528618 osi-oss-dbs-231020041218-71843c61
In this slide deck, we go exploring the database landscape today and the common lego blocks that are used to build these different falvours of databses. We will dive through internals of a database, explore some choices and towards the end also explore some real world database architectures in view of the concepts (legos) we explored earlier.]]>
In this slide deck, we go exploring the database landscape today and the common lego blocks that are used to build these different falvours of databses. We will dive through internals of a database, explore some choices and towards the end also explore some real world database architectures in view of the concepts (legos) we explored earlier.]]> Fri, 20 Oct 2023 04:12:18 GMT /slideshow/osiossdbspptx/262528618 shiv4289@slideshare.net(shiv4289) osi-oss-dbs.pptx shiv4289 In this slide deck, we go exploring the database landscape today and the common lego blocks that are used to build these different falvours of databses. We will dive through internals of a database, explore some choices and towards the end also explore some real world database architectures in view of the concepts (legos) we explored earlier. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/osi-oss-dbs-231020041218-71843c61-thumbnail.jpg?width=120&height=120&fit=bounds" /> In this slide deck, we go exploring the database landscape today and the common lego blocks that are used to build these different falvours of databses. We will dive through internals of a database, explore some choices and towards the end also explore some real world database architectures in view of the concepts (legos) we explored earlier.

osi-oss-dbs.pptx from Shivji Kumar Jha

]]> 81 0 https://cdn.slidesharecdn.com/ss_thumbnails/osi-oss-dbs-231020041218-71843c61-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 pulsar-platformatory-meetup-2.pptx /slideshow/pulsarplatformatorymeetup2pptx/260199528 pulsar-platformatory-meetup-2-230826122721-9161f6ff
This is a deep tech presentation on what Apache Pulsar ( a great choice for streaming & messaging) does internally to store a message in single digit milliseconds at high throughput. We will look at the internal data structures, read and write paths. This will give you a great perspective on what and how of designing your apps right to make the best use of Apache Pulsar (or any other streaming framework).]]>
This is a deep tech presentation on what Apache Pulsar ( a great choice for streaming & messaging) does internally to store a message in single digit milliseconds at high throughput. We will look at the internal data structures, read and write paths. This will give you a great perspective on what and how of designing your apps right to make the best use of Apache Pulsar (or any other streaming framework).]]> Sat, 26 Aug 2023 12:27:21 GMT /slideshow/pulsarplatformatorymeetup2pptx/260199528 shiv4289@slideshare.net(shiv4289) pulsar-platformatory-meetup-2.pptx shiv4289 This is a deep tech presentation on what Apache Pulsar ( a great choice for streaming & messaging) does internally to store a message in single digit milliseconds at high throughput. We will look at the internal data structures, read and write paths. This will give you a great perspective on what and how of designing your apps right to make the best use of Apache Pulsar (or any other streaming framework). <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/pulsar-platformatory-meetup-2-230826122721-9161f6ff-thumbnail.jpg?width=120&height=120&fit=bounds" /> This is a deep tech presentation on what Apache Pulsar ( a great choice for streaming & messaging) does internally to store a message in single digit milliseconds at high throughput. We will look at the internal data structures, read and write paths. This will give you a great perspective on what and how of designing your apps right to make the best use of Apache Pulsar (or any other streaming framework).

pulsar-platformatory-meetup-2.pptx from Shivji Kumar Jha

]]> 68 0 https://cdn.slidesharecdn.com/ss_thumbnails/pulsar-platformatory-meetup-2-230826122721-9161f6ff-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 Pulsar Summit Asia 2022 - Streaming wars and How Apache Pulsar is acing the battle /shiv4289/pulsar-summit-asia-2022-streaming-wars-and-how-apache-pulsar-is-acing-the-battle pulsarsummitasia2022-shivandsachi-221120044823-a4a3e46a
This presentation will cover why we prefer Apache Pulsar over other streaming solutions. Given the streaming requirements of near-realtime action, scalability, high availability, disaster recovery, load balancing, low cost of operations, multi-tenancy and flexibility to fit a variety of use cases, we have run kafka, kinesis and NATS Jetstream across different use cases. And we chose Apache Pulsar as our platform of choice for cloud-native messaging. This talk presents the operational challenges we have faced running Pulsar for over 4 years and how Pulsar fit into different use cases given its multi-tenancy and configurability. We will also talk about how we have aced these challenges to stick to pulsar and even moved application from other messaging solutions to Pulsar. We will end with the challenges and learnings on moving to Pulsar from Kafka and Kinesis. After this session, you will learn more on common messaging requirements, why you should also choose Apache Pulsar as your platform of choice and how you can safely transition to Pulsar if you have been running other messaging solutions.]]>
This presentation will cover why we prefer Apache Pulsar over other streaming solutions. Given the streaming requirements of near-realtime action, scalability, high availability, disaster recovery, load balancing, low cost of operations, multi-tenancy and flexibility to fit a variety of use cases, we have run kafka, kinesis and NATS Jetstream across different use cases. And we chose Apache Pulsar as our platform of choice for cloud-native messaging. This talk presents the operational challenges we have faced running Pulsar for over 4 years and how Pulsar fit into different use cases given its multi-tenancy and configurability. We will also talk about how we have aced these challenges to stick to pulsar and even moved application from other messaging solutions to Pulsar. We will end with the challenges and learnings on moving to Pulsar from Kafka and Kinesis. After this session, you will learn more on common messaging requirements, why you should also choose Apache Pulsar as your platform of choice and how you can safely transition to Pulsar if you have been running other messaging solutions.]]> Sun, 20 Nov 2022 04:48:23 GMT /shiv4289/pulsar-summit-asia-2022-streaming-wars-and-how-apache-pulsar-is-acing-the-battle shiv4289@slideshare.net(shiv4289) Pulsar Summit Asia 2022 - Streaming wars and How Apache Pulsar is acing the battle shiv4289 This presentation will cover why we prefer Apache Pulsar over other streaming solutions. Given the streaming requirements of near-realtime action, scalability, high availability, disaster recovery, load balancing, low cost of operations, multi-tenancy and flexibility to fit a variety of use cases, we have run kafka, kinesis and NATS Jetstream across different use cases. And we chose Apache Pulsar as our platform of choice for cloud-native messaging. This talk presents the operational challenges we have faced running Pulsar for over 4 years and how Pulsar fit into different use cases given its multi-tenancy and configurability. We will also talk about how we have aced these challenges to stick to pulsar and even moved application from other messaging solutions to Pulsar. We will end with the challenges and learnings on moving to Pulsar from Kafka and Kinesis. After this session, you will learn more on common messaging requirements, why you should also choose Apache Pulsar as your platform of choice and how you can safely transition to Pulsar if you have been running other messaging solutions. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/pulsarsummitasia2022-shivandsachi-221120044823-a4a3e46a-thumbnail.jpg?width=120&height=120&fit=bounds" /> This presentation will cover why we prefer Apache Pulsar over other streaming solutions. Given the streaming requirements of near-realtime action, scalability, high availability, disaster recovery, load balancing, low cost of operations, multi-tenancy and flexibility to fit a variety of use cases, we have run kafka, kinesis and NATS Jetstream across different use cases. And we chose Apache Pulsar as our platform of choice for cloud-native messaging. This talk presents the operational challenges we have faced running Pulsar for over 4 years and how Pulsar fit into different use cases given its multi-tenancy and configurability. We will also talk about how we have aced these challenges to stick to pulsar and even moved application from other messaging solutions to Pulsar. We will end with the challenges and learnings on moving to Pulsar from Kafka and Kinesis. After this session, you will learn more on common messaging requirements, why you should also choose Apache Pulsar as your platform of choice and how you can safely transition to Pulsar if you have been running other messaging solutions.

Pulsar Summit Asia 2022 - Streaming wars and How Apache Pulsar is acing the battle from Shivji Kumar Jha

]]> 61 0 https://cdn.slidesharecdn.com/ss_thumbnails/pulsarsummitasia2022-shivandsachi-221120044823-a4a3e46a-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 Pulsar Summit Asia 2022 - Keeping on top of hybrid cloud usage with Pulsar /slideshow/pulsar-summit-asia-2022-keeping-on-top-of-hybrid-cloud-usage-with-pulsar/254306976 pulsarsummitasia2022shivtarun-221118164951-98a7d7e1
This presentation will cover how we force controls on an application over a hybrid cloud infrastructure built from a combination of different clouds that could include private and public clouds. For instance, you could deploy your microservice in AWS but use BigTable as your data store. Every cloud or on-premise infrastructure provider provides monitoring, alerting, metering, audit trail etc. In a hybrid cloud use case, the IT team needs a single view of the usage across the cloud providers. Such a platform needs to combine the data sourcing of these utilities from different infrastructure providers, parse them into a common format and build an integrated data sink. Adding to it the challenge of each data source evolving its data formats, volume, velocity, throughput, latency etc. You have a challenging task to understand data from varied sources and present it in one view. We will present an architecture that has been battle-tested in production for over five years. The components include Pulsar, Flink, PostgreSQL, Redis, Neo4J DB, rule/ML engine etc., to name a few technologies. After this presentation, you will learn more about 1. Combining infrastructure from multiple clouds and on-premise providers to build your application. 2. Appreciate the need for lambda architecture. 3. How to stream ever-evolving multi-schema data using pulsar 4. How to write custom rules over a stream analytics framework to make your application. ]]>
This presentation will cover how we force controls on an application over a hybrid cloud infrastructure built from a combination of different clouds that could include private and public clouds. For instance, you could deploy your microservice in AWS but use BigTable as your data store. Every cloud or on-premise infrastructure provider provides monitoring, alerting, metering, audit trail etc. In a hybrid cloud use case, the IT team needs a single view of the usage across the cloud providers. Such a platform needs to combine the data sourcing of these utilities from different infrastructure providers, parse them into a common format and build an integrated data sink. Adding to it the challenge of each data source evolving its data formats, volume, velocity, throughput, latency etc. You have a challenging task to understand data from varied sources and present it in one view. We will present an architecture that has been battle-tested in production for over five years. The components include Pulsar, Flink, PostgreSQL, Redis, Neo4J DB, rule/ML engine etc., to name a few technologies. After this presentation, you will learn more about 1. Combining infrastructure from multiple clouds and on-premise providers to build your application. 2. Appreciate the need for lambda architecture. 3. How to stream ever-evolving multi-schema data using pulsar 4. How to write custom rules over a stream analytics framework to make your application. ]]> Fri, 18 Nov 2022 16:49:51 GMT /slideshow/pulsar-summit-asia-2022-keeping-on-top-of-hybrid-cloud-usage-with-pulsar/254306976 shiv4289@slideshare.net(shiv4289) Pulsar Summit Asia 2022 - Keeping on top of hybrid cloud usage with Pulsar shiv4289 This presentation will cover how we force controls on an application over a hybrid cloud infrastructure built from a combination of different clouds that could include private and public clouds. For instance, you could deploy your microservice in AWS but use BigTable as your data store. Every cloud or on-premise infrastructure provider provides monitoring, alerting, metering, audit trail etc. In a hybrid cloud use case, the IT team needs a single view of the usage across the cloud providers. Such a platform needs to combine the data sourcing of these utilities from different infrastructure providers, parse them into a common format and build an integrated data sink. Adding to it the challenge of each data source evolving its data formats, volume, velocity, throughput, latency etc. You have a challenging task to understand data from varied sources and present it in one view. We will present an architecture that has been battle-tested in production for over five years. The components include Pulsar, Flink, PostgreSQL, Redis, Neo4J DB, rule/ML engine etc., to name a few technologies. After this presentation, you will learn more about 1. Combining infrastructure from multiple clouds and on-premise providers to build your application. 2. Appreciate the need for lambda architecture. 3. How to stream ever-evolving multi-schema data using pulsar 4. How to write custom rules over a stream analytics framework to make your application. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/pulsarsummitasia2022shivtarun-221118164951-98a7d7e1-thumbnail.jpg?width=120&height=120&fit=bounds" /> This presentation will cover how we force controls on an application over a hybrid cloud infrastructure built from a combination of different clouds that could include private and public clouds. For instance, you could deploy your microservice in AWS but use BigTable as your data store. Every cloud or on-premise infrastructure provider provides monitoring, alerting, metering, audit trail etc. In a hybrid cloud use case, the IT team needs a single view of the usage across the cloud providers. Such a platform needs to combine the data sourcing of these utilities from different infrastructure providers, parse them into a common format and build an integrated data sink. Adding to it the challenge of each data source evolving its data formats, volume, velocity, throughput, latency etc. You have a challenging task to understand data from varied sources and present it in one view. We will present an architecture that has been battle-tested in production for over five years. The components include Pulsar, Flink, PostgreSQL, Redis, Neo4J DB, rule/ML engine etc., to name a few technologies. After this presentation, you will learn more about 1. Combining infrastructure from multiple clouds and on-premise providers to build your application. 2. Appreciate the need for lambda architecture. 3. How to stream ever-evolving multi-schema data using pulsar 4. How to write custom rules over a stream analytics framework to make your application.

Pulsar Summit Asia 2022 - Keeping on top of hybrid cloud usage with Pulsar from Shivji Kumar Jha

]]> 24 0 https://cdn.slidesharecdn.com/ss_thumbnails/pulsarsummitasia2022shivtarun-221118164951-98a7d7e1-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 Pulsar summit asia 2021: Designing Pulsar for Isolation /slideshow/pulsar-summit-asia-2021-designing-pulsar-for-isolation/251001086 pulsarsummitasia2021isolation1-220115181745
In this slide deck, we will go over the configurations knobs and all the virtual isolation features to run multiple clients in their assigned resource space without interfering with one another. At nutanix we run a single physical pulsar cluster shared by multiple use cases and applications. The biggest challenge in such a setup is an unintended denial of service for all client just because one of the clients breached their quotas and exceeded load expectations. In traditional distributed apps, one would do that with multiple physical clusters which is easier on day 1 but the operational complexity on day 2 or 3 can very quickly overwhelm a small team. Adding to its multi-tenant architecture, Pulsar provides a lot of features to restrict usage from different clients ranging from quotas, ttls, retention with simple configurations to more advanced features in namespace isolation, failure domains, anti affinity for namespaces, bookie groups and affinity between those etc.]]>
In this slide deck, we will go over the configurations knobs and all the virtual isolation features to run multiple clients in their assigned resource space without interfering with one another. At nutanix we run a single physical pulsar cluster shared by multiple use cases and applications. The biggest challenge in such a setup is an unintended denial of service for all client just because one of the clients breached their quotas and exceeded load expectations. In traditional distributed apps, one would do that with multiple physical clusters which is easier on day 1 but the operational complexity on day 2 or 3 can very quickly overwhelm a small team. Adding to its multi-tenant architecture, Pulsar provides a lot of features to restrict usage from different clients ranging from quotas, ttls, retention with simple configurations to more advanced features in namespace isolation, failure domains, anti affinity for namespaces, bookie groups and affinity between those etc.]]> Sat, 15 Jan 2022 18:17:45 GMT /slideshow/pulsar-summit-asia-2021-designing-pulsar-for-isolation/251001086 shiv4289@slideshare.net(shiv4289) Pulsar summit asia 2021: Designing Pulsar for Isolation shiv4289 In this slide deck, we will go over the configurations knobs and all the virtual isolation features to run multiple clients in their assigned resource space without interfering with one another. At nutanix we run a single physical pulsar cluster shared by multiple use cases and applications. The biggest challenge in such a setup is an unintended denial of service for all client just because one of the clients breached their quotas and exceeded load expectations. In traditional distributed apps, one would do that with multiple physical clusters which is easier on day 1 but the operational complexity on day 2 or 3 can very quickly overwhelm a small team. Adding to its multi-tenant architecture, Pulsar provides a lot of features to restrict usage from different clients ranging from quotas, ttls, retention with simple configurations to more advanced features in namespace isolation, failure domains, anti affinity for namespaces, bookie groups and affinity between those etc. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/pulsarsummitasia2021isolation1-220115181745-thumbnail.jpg?width=120&height=120&fit=bounds" /> In this slide deck, we will go over the configurations knobs and all the virtual isolation features to run multiple clients in their assigned resource space without interfering with one another. At nutanix we run a single physical pulsar cluster shared by multiple use cases and applications. The biggest challenge in such a setup is an unintended denial of service for all client just because one of the clients breached their quotas and exceeded load expectations. In traditional distributed apps, one would do that with multiple physical clusters which is easier on day 1 but the operational complexity on day 2 or 3 can very quickly overwhelm a small team. Adding to its multi-tenant architecture, Pulsar provides a lot of features to restrict usage from different clients ranging from quotas, ttls, retention with simple configurations to more advanced features in namespace isolation, failure domains, anti affinity for namespaces, bookie groups and affinity between those etc.

Pulsar summit asia 2021: Designing Pulsar for Isolation from Shivji Kumar Jha

]]> 186 0 https://cdn.slidesharecdn.com/ss_thumbnails/pulsarsummitasia2021isolation1-220115181745-thumbnail.jpg?width=120&height=120&fit=bounds presentation 000000 http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 Event sourcing Live 2021: Streaming App Changes to Event Store /slideshow/event-sourcing-live2021streaming-app-changes-to-event-store/250396591 eventsourcing-live-2021-streaming-app-changes-211008033143
This deck was used for the this talk at EventSourcing Live 2021 : https://lnkd.in/gbpshVA5 In the slides, we will go over identifying, capturing and delivering app changes to event stores. The event store can then be used as a data warehouse, data lake or a lakehouse. We will go over different ways to capture change data and deliver to an event store and the pros /cons of each.]]>
This deck was used for the this talk at EventSourcing Live 2021 : https://lnkd.in/gbpshVA5 In the slides, we will go over identifying, capturing and delivering app changes to event stores. The event store can then be used as a data warehouse, data lake or a lakehouse. We will go over different ways to capture change data and deliver to an event store and the pros /cons of each.]]> Fri, 08 Oct 2021 03:31:43 GMT /slideshow/event-sourcing-live2021streaming-app-changes-to-event-store/250396591 shiv4289@slideshare.net(shiv4289) Event sourcing Live 2021: Streaming App Changes to Event Store shiv4289 This deck was used for the this talk at EventSourcing Live 2021 : https://lnkd.in/gbpshVA5 In the slides, we will go over identifying, capturing and delivering app changes to event stores. The event store can then be used as a data warehouse, data lake or a lakehouse. We will go over different ways to capture change data and deliver to an event store and the pros /cons of each. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/eventsourcing-live-2021-streaming-app-changes-211008033143-thumbnail.jpg?width=120&height=120&fit=bounds" /> This deck was used for the this talk at EventSourcing Live 2021 : https://lnkd.in/gbpshVA5 In the slides, we will go over identifying, capturing and delivering app changes to event stores. The event store can then be used as a data warehouse, data lake or a lakehouse. We will go over different ways to capture change data and deliver to an event store and the pros /cons of each.

Event sourcing Live 2021: Streaming App Changes to Event Store from Shivji Kumar Jha

]]> 107 0 https://cdn.slidesharecdn.com/ss_thumbnails/eventsourcing-live-2021-streaming-app-changes-211008033143-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 Apache Con 2021 Structured Data Streaming /slideshow/apache-con-2021-structured-data-streaming/249940935 apachecon2021-schema-210808054846
Type safety is extremely important in any application built around a stream / queue. Type definition and evolution can either be built in the application or relied upon the data layer to support it out of the box allowing the application to only concentrate on business logic, not how of data store and evolution. It is this property of the good old relational databases (among others) that make them a favourite among all the modern NoSQL databases. Modern software architectures requires asynchronous communication (via stream / queue). While the data store and query design changes with asynchronous communication, type safety is still equally important. In this slide deck, used for Apache Con 2021 talk, we will go over ways in which one can force structure (schema) over the streaming data. As an example, we will talk about Apache Pulsar. Apache pulsar offers server as well as client side support for the structured streaming. We have been using pulsar for asynchronous communication among microservices in our nutanix beam and flow security central apps for over 1.5 years in production. This deck presents the technical details on what is schema, how to represent schema, what is available in the apache pulsar server and client side, how we have used pulsar’s schema support to build our use cases and our learnings from them.]]>
Type safety is extremely important in any application built around a stream / queue. Type definition and evolution can either be built in the application or relied upon the data layer to support it out of the box allowing the application to only concentrate on business logic, not how of data store and evolution. It is this property of the good old relational databases (among others) that make them a favourite among all the modern NoSQL databases. Modern software architectures requires asynchronous communication (via stream / queue). While the data store and query design changes with asynchronous communication, type safety is still equally important. In this slide deck, used for Apache Con 2021 talk, we will go over ways in which one can force structure (schema) over the streaming data. As an example, we will talk about Apache Pulsar. Apache pulsar offers server as well as client side support for the structured streaming. We have been using pulsar for asynchronous communication among microservices in our nutanix beam and flow security central apps for over 1.5 years in production. This deck presents the technical details on what is schema, how to represent schema, what is available in the apache pulsar server and client side, how we have used pulsar’s schema support to build our use cases and our learnings from them.]]> Sun, 08 Aug 2021 05:48:45 GMT /slideshow/apache-con-2021-structured-data-streaming/249940935 shiv4289@slideshare.net(shiv4289) Apache Con 2021 Structured Data Streaming shiv4289 Type safety is extremely important in any application built around a stream / queue. Type definition and evolution can either be built in the application or relied upon the data layer to support it out of the box allowing the application to only concentrate on business logic, not how of data store and evolution. It is this property of the good old relational databases (among others) that make them a favourite among all the modern NoSQL databases. Modern software architectures requires asynchronous communication (via stream / queue). While the data store and query design changes with asynchronous communication, type safety is still equally important. In this slide deck, used for Apache Con 2021 talk, we will go over ways in which one can force structure (schema) over the streaming data. As an example, we will talk about Apache Pulsar. Apache pulsar offers server as well as client side support for the structured streaming. We have been using pulsar for asynchronous communication among microservices in our nutanix beam and flow security central apps for over 1.5 years in production. This deck presents the technical details on what is schema, how to represent schema, what is available in the apache pulsar server and client side, how we have used pulsar’s schema support to build our use cases and our learnings from them. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/apachecon2021-schema-210808054846-thumbnail.jpg?width=120&height=120&fit=bounds" /> Type safety is extremely important in any application built around a stream / queue. Type definition and evolution can either be built in the application or relied upon the data layer to support it out of the box allowing the application to only concentrate on business logic, not how of data store and evolution. It is this property of the good old relational databases (among others) that make them a favourite among all the modern NoSQL databases. Modern software architectures requires asynchronous communication (via stream / queue). While the data store and query design changes with asynchronous communication, type safety is still equally important. In this slide deck, used for Apache Con 2021 talk, we will go over ways in which one can force structure (schema) over the streaming data. As an example, we will talk about Apache Pulsar. Apache pulsar offers server as well as client side support for the structured streaming. We have been using pulsar for asynchronous communication among microservices in our nutanix beam and flow security central apps for over 1.5 years in production. This deck presents the technical details on what is schema, how to represent schema, what is available in the apache pulsar server and client side, how we have used pulsar’s schema support to build our use cases and our learnings from them.

Apache Con 2021 Structured Data Streaming from Shivji Kumar Jha

]]> 334 0 https://cdn.slidesharecdn.com/ss_thumbnails/apachecon2021-schema-210808054846-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 Apache Con 2021 : Apache Bookkeeper Key Value Store and use cases /slideshow/apache-con-2021-apache-bookkeeper-key-value-store-and-use-cases/249929975 apachecon2021-bookie-usecases-210806055136
In order to leverage the best performance characters of your data or stream backend, it is important to understand the nitty gritty details of how your backend store and compute works, how data is stored, how is it indexed and how the read path is. Understanding this empowers you to design your use case solutioning so as to make the best use of resources at hand as well as get the optimum amount of consistency, availability, latency and throughput for a given amount of resources at hand. With this underlying philosophy, in this slide deck, we will get to the bottom of storage tier of pulsar (apache bookkeeper), the barebones of the bookkeeper storage semantics, how it is used in different use cases ( even other than pulsar), understand the object models of storage in pulsar, different kinds of data structures and algorithms pulsar uses therein and how that maps to the semantics of the storage class shipped with pulsar by default. Oh yes, you can change the storage backend too with some additional code! The focus will be more on storage backend so as to not keep this tailored to pulsar specifically but to be able to apply it different data stores or streams.]]>
In order to leverage the best performance characters of your data or stream backend, it is important to understand the nitty gritty details of how your backend store and compute works, how data is stored, how is it indexed and how the read path is. Understanding this empowers you to design your use case solutioning so as to make the best use of resources at hand as well as get the optimum amount of consistency, availability, latency and throughput for a given amount of resources at hand. With this underlying philosophy, in this slide deck, we will get to the bottom of storage tier of pulsar (apache bookkeeper), the barebones of the bookkeeper storage semantics, how it is used in different use cases ( even other than pulsar), understand the object models of storage in pulsar, different kinds of data structures and algorithms pulsar uses therein and how that maps to the semantics of the storage class shipped with pulsar by default. Oh yes, you can change the storage backend too with some additional code! The focus will be more on storage backend so as to not keep this tailored to pulsar specifically but to be able to apply it different data stores or streams.]]> Fri, 06 Aug 2021 05:51:35 GMT /slideshow/apache-con-2021-apache-bookkeeper-key-value-store-and-use-cases/249929975 shiv4289@slideshare.net(shiv4289) Apache Con 2021 : Apache Bookkeeper Key Value Store and use cases shiv4289 In order to leverage the best performance characters of your data or stream backend, it is important to understand the nitty gritty details of how your backend store and compute works, how data is stored, how is it indexed and how the read path is. Understanding this empowers you to design your use case solutioning so as to make the best use of resources at hand as well as get the optimum amount of consistency, availability, latency and throughput for a given amount of resources at hand. With this underlying philosophy, in this slide deck, we will get to the bottom of storage tier of pulsar (apache bookkeeper), the barebones of the bookkeeper storage semantics, how it is used in different use cases ( even other than pulsar), understand the object models of storage in pulsar, different kinds of data structures and algorithms pulsar uses therein and how that maps to the semantics of the storage class shipped with pulsar by default. Oh yes, you can change the storage backend too with some additional code! The focus will be more on storage backend so as to not keep this tailored to pulsar specifically but to be able to apply it different data stores or streams. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/apachecon2021-bookie-usecases-210806055136-thumbnail.jpg?width=120&height=120&fit=bounds" /> In order to leverage the best performance characters of your data or stream backend, it is important to understand the nitty gritty details of how your backend store and compute works, how data is stored, how is it indexed and how the read path is. Understanding this empowers you to design your use case solutioning so as to make the best use of resources at hand as well as get the optimum amount of consistency, availability, latency and throughput for a given amount of resources at hand. With this underlying philosophy, in this slide deck, we will get to the bottom of storage tier of pulsar (apache bookkeeper), the barebones of the bookkeeper storage semantics, how it is used in different use cases ( even other than pulsar), understand the object models of storage in pulsar, different kinds of data structures and algorithms pulsar uses therein and how that maps to the semantics of the storage class shipped with pulsar by default. Oh yes, you can change the storage backend too with some additional code! The focus will be more on storage backend so as to not keep this tailored to pulsar specifically but to be able to apply it different data stores or streams.

Apache Con 2021 : Apache Bookkeeper Key Value Store and use cases from Shivji Kumar Jha

]]> 397 0 https://cdn.slidesharecdn.com/ss_thumbnails/apachecon2021-bookie-usecases-210806055136-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 How pulsar stores data at Pulsar-na-summit-2021.pptx (1) /slideshow/how-pulsar-stores-data-at-pulsarnasummit2021pptx-1/249397876 howpulsarstoresdata-shivji-pulsar-na-summit-2021-210617152227
In order to leverage the best performance characters of your stream backend, it is important to understand the nitty gritty details of how pulsar stores your data. Understanding this empowers you to design your use case solutioning so as to make the best use of resources at hand as well as get the optimum amount of consistency, availability, latency and throughput for a given amount of resources at hand. With this underlying philosophy, in this talk, we will get to the bottom of storage tier of pulsar (apache bookkeeper), the barebones of the bookkeeper storage semantics, how it is used in different use cases ( even other than pulsar), understand the object models of storage in pulsar, different kinds of data structures and algorithms pulsar uses therein and how that maps to the semantics of the storage class shipped with pulsar by default. Oh yes, you can change the storage backend too with some additional code! This session will empower you with the right background to map your data right with pulsar.]]>
In order to leverage the best performance characters of your stream backend, it is important to understand the nitty gritty details of how pulsar stores your data. Understanding this empowers you to design your use case solutioning so as to make the best use of resources at hand as well as get the optimum amount of consistency, availability, latency and throughput for a given amount of resources at hand. With this underlying philosophy, in this talk, we will get to the bottom of storage tier of pulsar (apache bookkeeper), the barebones of the bookkeeper storage semantics, how it is used in different use cases ( even other than pulsar), understand the object models of storage in pulsar, different kinds of data structures and algorithms pulsar uses therein and how that maps to the semantics of the storage class shipped with pulsar by default. Oh yes, you can change the storage backend too with some additional code! This session will empower you with the right background to map your data right with pulsar.]]> Thu, 17 Jun 2021 15:22:27 GMT /slideshow/how-pulsar-stores-data-at-pulsarnasummit2021pptx-1/249397876 shiv4289@slideshare.net(shiv4289) How pulsar stores data at Pulsar-na-summit-2021.pptx (1) shiv4289 In order to leverage the best performance characters of your stream backend, it is important to understand the nitty gritty details of how pulsar stores your data. Understanding this empowers you to design your use case solutioning so as to make the best use of resources at hand as well as get the optimum amount of consistency, availability, latency and throughput for a given amount of resources at hand. With this underlying philosophy, in this talk, we will get to the bottom of storage tier of pulsar (apache bookkeeper), the barebones of the bookkeeper storage semantics, how it is used in different use cases ( even other than pulsar), understand the object models of storage in pulsar, different kinds of data structures and algorithms pulsar uses therein and how that maps to the semantics of the storage class shipped with pulsar by default. Oh yes, you can change the storage backend too with some additional code! This session will empower you with the right background to map your data right with pulsar. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/howpulsarstoresdata-shivji-pulsar-na-summit-2021-210617152227-thumbnail.jpg?width=120&height=120&fit=bounds" /> In order to leverage the best performance characters of your stream backend, it is important to understand the nitty gritty details of how pulsar stores your data. Understanding this empowers you to design your use case solutioning so as to make the best use of resources at hand as well as get the optimum amount of consistency, availability, latency and throughput for a given amount of resources at hand. With this underlying philosophy, in this talk, we will get to the bottom of storage tier of pulsar (apache bookkeeper), the barebones of the bookkeeper storage semantics, how it is used in different use cases ( even other than pulsar), understand the object models of storage in pulsar, different kinds of data structures and algorithms pulsar uses therein and how that maps to the semantics of the storage class shipped with pulsar by default. Oh yes, you can change the storage backend too with some additional code! This session will empower you with the right background to map your data right with pulsar.

How pulsar stores data at Pulsar-na-summit-2021.pptx (1) from Shivji Kumar Jha

]]> 153 1 https://cdn.slidesharecdn.com/ss_thumbnails/howpulsarstoresdata-shivji-pulsar-na-summit-2021-210617152227-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 Pulsar Summit Asia - Structured Data Stream with Apache Pulsar /slideshow/pulsar-summit-asia-structured-data-stream-with-apache-pulsar/239634593 pulsarasiasummit-structureddatastreamwithpulsar-201201042440
Type safety is extremely important in any application built around a message bus like Pulsar. Type definition and evolution can either be built in the application or relied upon the data layer to support it out of the box allowing the application to only concentrate on business logic, not how of data store and evolution. Apache pulsar offers server as well as client side support for the structured streaming. We have been using pulsar for asynchronous communication among microservices in our nutanix beam app for over an year in production. This talk presents the technical details on what is available in the apache pulsar server and client side, how we have used pulsar's schema support to build our use cases and our learnings from them.]]>
Type safety is extremely important in any application built around a message bus like Pulsar. Type definition and evolution can either be built in the application or relied upon the data layer to support it out of the box allowing the application to only concentrate on business logic, not how of data store and evolution. Apache pulsar offers server as well as client side support for the structured streaming. We have been using pulsar for asynchronous communication among microservices in our nutanix beam app for over an year in production. This talk presents the technical details on what is available in the apache pulsar server and client side, how we have used pulsar's schema support to build our use cases and our learnings from them.]]> Tue, 01 Dec 2020 04:24:40 GMT /slideshow/pulsar-summit-asia-structured-data-stream-with-apache-pulsar/239634593 shiv4289@slideshare.net(shiv4289) Pulsar Summit Asia - Structured Data Stream with Apache Pulsar shiv4289 Type safety is extremely important in any application built around a message bus like Pulsar. Type definition and evolution can either be built in the application or relied upon the data layer to support it out of the box allowing the application to only concentrate on business logic, not how of data store and evolution. Apache pulsar offers server as well as client side support for the structured streaming. We have been using pulsar for asynchronous communication among microservices in our nutanix beam app for over an year in production. This talk presents the technical details on what is available in the apache pulsar server and client side, how we have used pulsar's schema support to build our use cases and our learnings from them. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/pulsarasiasummit-structureddatastreamwithpulsar-201201042440-thumbnail.jpg?width=120&height=120&fit=bounds" /> Type safety is extremely important in any application built around a message bus like Pulsar. Type definition and evolution can either be built in the application or relied upon the data layer to support it out of the box allowing the application to only concentrate on business logic, not how of data store and evolution. Apache pulsar offers server as well as client side support for the structured streaming. We have been using pulsar for asynchronous communication among microservices in our nutanix beam app for over an year in production. This talk presents the technical details on what is available in the apache pulsar server and client side, how we have used pulsar's schema support to build our use cases and our learnings from them.

Pulsar Summit Asia - Structured Data Stream with Apache Pulsar from Shivji Kumar Jha

]]> 318 1 https://cdn.slidesharecdn.com/ss_thumbnails/pulsarasiasummit-structureddatastreamwithpulsar-201201042440-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 Pulsar Summit Asia - Running a secure pulsar cluster /slideshow/pulsar-summit-asia-running-a-secure-pulsar-cluster/239567184 pulsarasiasummit-runningasecurepulsarcluster-201128085707
A Pulsar instance consists of one or more Pulsar clusters. Clusters, in turn, consist of a broker cluster, a bookkeeper cluster and a zookeeper cluster. While this provides a modular and flexible design, there is a lot of bytes flowing over the network. In an application that is paranoid about security that is a lot of connections, both intra cluster and inter-cluster, to secure. At nutanix we have deploy pulsar in a secure environment and run it through a lot of security audits for infrastructure certifications. In this talks, we will go over how we set up authentication and authorization on all the network communications to make our pulsar deployment secure.]]>
A Pulsar instance consists of one or more Pulsar clusters. Clusters, in turn, consist of a broker cluster, a bookkeeper cluster and a zookeeper cluster. While this provides a modular and flexible design, there is a lot of bytes flowing over the network. In an application that is paranoid about security that is a lot of connections, both intra cluster and inter-cluster, to secure. At nutanix we have deploy pulsar in a secure environment and run it through a lot of security audits for infrastructure certifications. In this talks, we will go over how we set up authentication and authorization on all the network communications to make our pulsar deployment secure.]]> Sat, 28 Nov 2020 08:57:07 GMT /slideshow/pulsar-summit-asia-running-a-secure-pulsar-cluster/239567184 shiv4289@slideshare.net(shiv4289) Pulsar Summit Asia - Running a secure pulsar cluster shiv4289 A Pulsar instance consists of one or more Pulsar clusters. Clusters, in turn, consist of a broker cluster, a bookkeeper cluster and a zookeeper cluster. While this provides a modular and flexible design, there is a lot of bytes flowing over the network. In an application that is paranoid about security that is a lot of connections, both intra cluster and inter-cluster, to secure. At nutanix we have deploy pulsar in a secure environment and run it through a lot of security audits for infrastructure certifications. In this talks, we will go over how we set up authentication and authorization on all the network communications to make our pulsar deployment secure. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/pulsarasiasummit-runningasecurepulsarcluster-201128085707-thumbnail.jpg?width=120&height=120&fit=bounds" /> A Pulsar instance consists of one or more Pulsar clusters. Clusters, in turn, consist of a broker cluster, a bookkeeper cluster and a zookeeper cluster. While this provides a modular and flexible design, there is a lot of bytes flowing over the network. In an application that is paranoid about security that is a lot of connections, both intra cluster and inter-cluster, to secure. At nutanix we have deploy pulsar in a secure environment and run it through a lot of security audits for infrastructure certifications. In this talks, we will go over how we set up authentication and authorization on all the network communications to make our pulsar deployment secure.

Pulsar Summit Asia - Running a secure pulsar cluster from Shivji Kumar Jha

]]> 586 1 https://cdn.slidesharecdn.com/ss_thumbnails/pulsarasiasummit-runningasecurepulsarcluster-201128085707-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 lessons from managing a pulsar cluster /slideshow/lessons-from-managing-a-pulsar-cluster/234617108 webinar-lessonsfrommanagingapulsarclusternutanix1-200527054903
Having used apache pulsar in production for an year for our pub sub use cases such as stream analytics, event sourcing etc, this slide deck presents the lesson learned per se understanding the architecture, tuning the cluster, managing to keep it highly available and fault tolerant and much more. While the slides are presented in terms of apache pulsar, a lot of the concepts can be easily extended to a lot of distributed systems. The views here are my own and do not represent the view of nutanix corporation.]]>
Having used apache pulsar in production for an year for our pub sub use cases such as stream analytics, event sourcing etc, this slide deck presents the lesson learned per se understanding the architecture, tuning the cluster, managing to keep it highly available and fault tolerant and much more. While the slides are presented in terms of apache pulsar, a lot of the concepts can be easily extended to a lot of distributed systems. The views here are my own and do not represent the view of nutanix corporation.]]> Wed, 27 May 2020 05:49:03 GMT /slideshow/lessons-from-managing-a-pulsar-cluster/234617108 shiv4289@slideshare.net(shiv4289) lessons from managing a pulsar cluster shiv4289 Having used apache pulsar in production for an year for our pub sub use cases such as stream analytics, event sourcing etc, this slide deck presents the lesson learned per se understanding the architecture, tuning the cluster, managing to keep it highly available and fault tolerant and much more. While the slides are presented in terms of apache pulsar, a lot of the concepts can be easily extended to a lot of distributed systems. The views here are my own and do not represent the view of nutanix corporation. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/webinar-lessonsfrommanagingapulsarclusternutanix1-200527054903-thumbnail.jpg?width=120&height=120&fit=bounds" /> Having used apache pulsar in production for an year for our pub sub use cases such as stream analytics, event sourcing etc, this slide deck presents the lesson learned per se understanding the architecture, tuning the cluster, managing to keep it highly available and fault tolerant and much more. While the slides are presented in terms of apache pulsar, a lot of the concepts can be easily extended to a lot of distributed systems. The views here are my own and do not represent the view of nutanix corporation.

lessons from managing a pulsar cluster from Shivji Kumar Jha

]]> 484 0 https://cdn.slidesharecdn.com/ss_thumbnails/webinar-lessonsfrommanagingapulsarclusternutanix1-200527054903-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 FOSSASIA 2015: MySQL Group Replication /shiv4289/fossasia-2015-mysql-group-replication fossasiagroupreplication-150314120214-conversion-gate01
This slide was used to introduce group replication at http://fossasia.org/. The document has following contents: 1) Starts with asynchronous and semi-synchronous protocols supported by MySQL replication and goes ahead to show how group replication fits into the whole high availability offering by MySQL. 2) Shows a step-by-step process from a user perspective as to how a transaction is executed in the group. 3) Shows the building blocks making the layered architecture of Group replication plugin and what the roles of these building blocks are. 4) Where (and where not) to use group replication.]]>
This slide was used to introduce group replication at http://fossasia.org/. The document has following contents: 1) Starts with asynchronous and semi-synchronous protocols supported by MySQL replication and goes ahead to show how group replication fits into the whole high availability offering by MySQL. 2) Shows a step-by-step process from a user perspective as to how a transaction is executed in the group. 3) Shows the building blocks making the layered architecture of Group replication plugin and what the roles of these building blocks are. 4) Where (and where not) to use group replication.]]> Sat, 14 Mar 2015 12:02:14 GMT /shiv4289/fossasia-2015-mysql-group-replication shiv4289@slideshare.net(shiv4289) FOSSASIA 2015: MySQL Group Replication shiv4289 This slide was used to introduce group replication at http://fossasia.org/. The document has following contents: 1) Starts with asynchronous and semi-synchronous protocols supported by MySQL replication and goes ahead to show how group replication fits into the whole high availability offering by MySQL. 2) Shows a step-by-step process from a user perspective as to how a transaction is executed in the group. 3) Shows the building blocks making the layered architecture of Group replication plugin and what the roles of these building blocks are. 4) Where (and where not) to use group replication. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/fossasiagroupreplication-150314120214-conversion-gate01-thumbnail.jpg?width=120&height=120&fit=bounds" /> This slide was used to introduce group replication at http://fossasia.org/. The document has following contents: 1) Starts with asynchronous and semi-synchronous protocols supported by MySQL replication and goes ahead to show how group replication fits into the whole high availability offering by MySQL. 2) Shows a step-by-step process from a user perspective as to how a transaction is executed in the group. 3) Shows the building blocks making the layered architecture of Group replication plugin and what the roles of these building blocks are. 4) Where (and where not) to use group replication.

FOSSASIA 2015: MySQL Group Replication from Shivji Kumar Jha

]]> 3497 5 https://cdn.slidesharecdn.com/ss_thumbnails/fossasiagroupreplication-150314120214-conversion-gate01-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 MySQL High Availability with Replication New Features /slideshow/mysql-high-availability-with-replication-new-features/41292951 osi2014-mysqlreplicationnewfeatures-141108082713-conversion-gate01
The session was presented at open source India 2014 (http://osidays.com/osidays/) by Shivji (me) and Manish Kumar. It talks of the new features in MySQL-5.7 Replication. It covered work on 1) performance enhancements in MySQL Replication 2) Usability improvements 3) More flexibility to provide more options to our users so they can chose what is best for their application. 4) Semisynchronous and MySQL Group Replication At then end, there are a lot of links to the blogs written on these features by the MySQL Replication engineers.]]>
The session was presented at open source India 2014 (http://osidays.com/osidays/) by Shivji (me) and Manish Kumar. It talks of the new features in MySQL-5.7 Replication. It covered work on 1) performance enhancements in MySQL Replication 2) Usability improvements 3) More flexibility to provide more options to our users so they can chose what is best for their application. 4) Semisynchronous and MySQL Group Replication At then end, there are a lot of links to the blogs written on these features by the MySQL Replication engineers.]]> Sat, 08 Nov 2014 08:27:13 GMT /slideshow/mysql-high-availability-with-replication-new-features/41292951 shiv4289@slideshare.net(shiv4289) MySQL High Availability with Replication New Features shiv4289 The session was presented at open source India 2014 (http://osidays.com/osidays/) by Shivji (me) and Manish Kumar. It talks of the new features in MySQL-5.7 Replication. It covered work on 1) performance enhancements in MySQL Replication 2) Usability improvements 3) More flexibility to provide more options to our users so they can chose what is best for their application. 4) Semisynchronous and MySQL Group Replication At then end, there are a lot of links to the blogs written on these features by the MySQL Replication engineers. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/osi2014-mysqlreplicationnewfeatures-141108082713-conversion-gate01-thumbnail.jpg?width=120&height=120&fit=bounds" /> The session was presented at open source India 2014 (http://osidays.com/osidays/) by Shivji (me) and Manish Kumar. It talks of the new features in MySQL-5.7 Replication. It covered work on 1) performance enhancements in MySQL Replication 2) Usability improvements 3) More flexibility to provide more options to our users so they can chose what is best for their application. 4) Semisynchronous and MySQL Group Replication At then end, there are a lot of links to the blogs written on these features by the MySQL Replication engineers.

MySQL High Availability with Replication New Features from Shivji Kumar Jha

]]> 1789 2 https://cdn.slidesharecdn.com/ss_thumbnails/osi2014-mysqlreplicationnewfeatures-141108082713-conversion-gate01-thumbnail.jpg?width=120&height=120&fit=bounds presentation 000000 http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 MySQL Developer Day conference: MySQL Replication and Scalability /slideshow/my-sql-replicationscalability/33456748 mysqlreplicationscalability-140412223120-phpapp01
The slide deck contains the latest developments in MySQL Replication. It covers: - An introduction to MySQL Replication - Scaling with Multi-threaded slaves - Data aggregation with Multi-source replication - Lossless failover with semi-synchronous replication - Replication Monitoring made easier]]>
The slide deck contains the latest developments in MySQL Replication. It covers: - An introduction to MySQL Replication - Scaling with Multi-threaded slaves - Data aggregation with Multi-source replication - Lossless failover with semi-synchronous replication - Replication Monitoring made easier]]> Sat, 12 Apr 2014 22:31:20 GMT /slideshow/my-sql-replicationscalability/33456748 shiv4289@slideshare.net(shiv4289) MySQL Developer Day conference: MySQL Replication and Scalability shiv4289 The slide deck contains the latest developments in MySQL Replication. It covers: - An introduction to MySQL Replication - Scaling with Multi-threaded slaves - Data aggregation with Multi-source replication - Lossless failover with semi-synchronous replication - Replication Monitoring made easier <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/mysqlreplicationscalability-140412223120-phpapp01-thumbnail.jpg?width=120&height=120&fit=bounds" /> The slide deck contains the latest developments in MySQL Replication. It covers: - An introduction to MySQL Replication - Scaling with Multi-threaded slaves - Data aggregation with Multi-source replication - Lossless failover with semi-synchronous replication - Replication Monitoring made easier

MySQL Developer Day conference: MySQL Replication and Scalability from Shivji Kumar Jha

]]> 2735 3 https://cdn.slidesharecdn.com/ss_thumbnails/mysqlreplicationscalability-140412223120-phpapp01-thumbnail.jpg?width=120&height=120&fit=bounds presentation White http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 MySQL User Camp: MySQL Cluster /slideshow/mysql-user-camp-mysql-cluster/32651537 cluster-140324013913-phpapp01
Priyanka, a MySQL cluster developer, presented MySQL cluster in the MySQL User camp. The slide deck contains an introduction to the cluster module- the architecture, auto-sharding, failover etc in the cluster module.]]>
Priyanka, a MySQL cluster developer, presented MySQL cluster in the MySQL User camp. The slide deck contains an introduction to the cluster module- the architecture, auto-sharding, failover etc in the cluster module.]]> Mon, 24 Mar 2014 01:39:12 GMT /slideshow/mysql-user-camp-mysql-cluster/32651537 shiv4289@slideshare.net(shiv4289) MySQL User Camp: MySQL Cluster shiv4289 Priyanka, a MySQL cluster developer, presented MySQL cluster in the MySQL User camp. The slide deck contains an introduction to the cluster module- the architecture, auto-sharding, failover etc in the cluster module. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/cluster-140324013913-phpapp01-thumbnail.jpg?width=120&height=120&fit=bounds" /> Priyanka, a MySQL cluster developer, presented MySQL cluster in the MySQL User camp. The slide deck contains an introduction to the cluster module- the architecture, auto-sharding, failover etc in the cluster module.

MySQL User Camp: MySQL Cluster from Shivji Kumar Jha

]]> 1642 3 https://cdn.slidesharecdn.com/ss_thumbnails/cluster-140324013913-phpapp01-thumbnail.jpg?width=120&height=120&fit=bounds presentation Black http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 MySQL User Camp: GTIDs /shiv4289/mysql-user-camp-gtids gtidusercampslideshare-140311224213-phpapp02
The slde contains an introduction to the global transaction identifiers(GTIDs) in MySQL Replication. The new protocol at re-connect, skipping transactions with GTIDS, replication filters, purging logs, backup/restore ets are covered here.]]>
The slde contains an introduction to the global transaction identifiers(GTIDs) in MySQL Replication. The new protocol at re-connect, skipping transactions with GTIDS, replication filters, purging logs, backup/restore ets are covered here.]]> Tue, 11 Mar 2014 22:42:13 GMT /shiv4289/mysql-user-camp-gtids shiv4289@slideshare.net(shiv4289) MySQL User Camp: GTIDs shiv4289 The slde contains an introduction to the global transaction identifiers(GTIDs) in MySQL Replication. The new protocol at re-connect, skipping transactions with GTIDS, replication filters, purging logs, backup/restore ets are covered here. <img style="border:1px solid #C3E6D8;float:right;" alt="" src="https://cdn.slidesharecdn.com/ss_thumbnails/gtidusercampslideshare-140311224213-phpapp02-thumbnail.jpg?width=120&height=120&fit=bounds" /> The slde contains an introduction to the global transaction identifiers(GTIDs) in MySQL Replication. The new protocol at re-connect, skipping transactions with GTIDS, replication filters, purging logs, backup/restore ets are covered here.

MySQL User Camp: GTIDs from Shivji Kumar Jha

]]> 2299 2 https://cdn.slidesharecdn.com/ss_thumbnails/gtidusercampslideshare-140311224213-phpapp02-thumbnail.jpg?width=120&height=120&fit=bounds presentation 000000 http://activitystrea.ms/schema/1.0/post

http://activitystrea.ms/schema/1.0/posted

0 https://cdn.slidesharecdn.com/profile-photo-shiv4289-48x48.jpg?cb=1719912401 Shiv is a senior software developer at Nutanix and works for the cloud platform team helping Nutanix customers minimise cloud costs and security risks for hybrid cloud usage. Shiv loves spending time on data stores (databases, streams, analytics etc) and has contributed to MySQL and pulsar codebases. Shiv is an avid reader (tech, fiction, economics etc) and is always looking at ways to simplify software architectures. www.shivjijha.com/ https://cdn.slidesharecdn.com/ss_thumbnails/batch-to-realtime-platformatory-meetup-6-july-2024-240709044416-b215b7e9-thumbnail.jpg?width=320&height=320&fit=bounds slideshow/batch-to-near-realtime-inspired-by-a-real-production-incident/270137775 Batch to near-realtime... https://cdn.slidesharecdn.com/ss_thumbnails/transactions1-240203033846-28a1a2e2-thumbnail.jpg?width=320&height=320&fit=bounds slideshow/navigating-transactions-acid-complexity-in-modern-databases/266088421 Navigating Transaction... https://cdn.slidesharecdn.com/ss_thumbnails/druidsummit2023slides-231123173714-c20c2b42-thumbnail.jpg?width=320&height=320&fit=bounds shiv4289/druid-summit-2023-changing-druid-ingestion-from-3-hours-to-5-minutes Druid Summit 2023 : Ch...