Serverless Event Streaming with Pulsar Functions-xiaolong
3 likes217 views
本文介绍了 Apache Pulsar 的架构及其服务器无事件流处理功能的实现，重点讨论了 Go 函数的设计和使用场景，包括 ETL、数据过滤和动态路由等。Pulsar 函数作为一个轻量级的事件流框架，支持多种语言和运行时，并利用自动负载均衡机制进行函数的管理和调度。文中还提到了一些安全隐患和 Go 语言的使用限制。
1 of 45
Download to read offline
Ad
Recommended
Opening Remarks - Pulsar Summit Asia 2021
Opening Remarks - Pulsar Summit Asia 2021StreamNative
?
该文档介绍了 Apache Pulsar 生态系统及其社区动态，包括近期的贡献者统计和峰会的信息。峰会涵盖多个演讲主题，聚焦于 Apache Pulsar 及其在多种行业中的应用，展示了技术专家们的创新实践及演讲。文档还提到新发布的功能和即将进行的培训课程，以及相关书籍资源。Improvements Made in KoP 2.9.0 - Pulsar Summit Asia 2021
Improvements Made in KoP 2.9.0 - Pulsar Summit Asia 2021StreamNative
?
本文详细介绍了 kop (Kafka on Pulsar) 的设计及其最新进展，重点阐述了为什么需要 kop 进行 Kafka 到 Pulsar 的迁移及其三种实现方法。文档还讨论了 kop 的基本设计，包括协议处理、认证、消息存储方式以及如何处理 Kafka Offset 与 Pulsar Message ID 的差异。此外，kop 2.9 的新特性包括对多种 Kafka 客户端版本的支持、基于租户的组协调器以及改进的鉴权功能。碍补蹿办补的设计与实现
碍补蹿办补的设计与实现wang xing
?
Kafka 是一个高吞吐量的分布式发布/订阅消息系统，旨在支持分区、复制和事务日志服务。其架构包含生产者、消费者和主题，消息在多个分区中有序存储，并提供了高效的消息处理功能。文档详细介绍了 Kafka 的组件、功能特性以及常见的使用场景和性能优化方案。The Evolution History of RoP(RocketMQ-on-Pulsar) - Pulsar Summit Asia 2021
The Evolution History of RoP(RocketMQ-on-Pulsar) - Pulsar Summit Asia 2021StreamNative
?
本文讨论了ROP（RocketMQ Offset Protocol）在版本1.0和2.0中的偏移处理及其相关问题，主要包括精度丢失和对RocketMQ偏移的认识不足。此外，探讨了ROP延迟消息的实现方法及其优缺点，强调了在分布式环境下的重要性和优化方案。RabbitMQ on Pulsar's Practice in Tencent Cloud - Puslar Summit Asia 2021
RabbitMQ on Pulsar's Practice in Tencent Cloud - Puslar Summit Asia 2021StreamNative
?
本文探讨了在Apache Pulsar上实现RabbitMQ协议适配的过程，旨在为RabbitMQ用户提供无缝接入Pulsar生态的高性能云替代方案。文中分析了架构演进中的关键问题，如多租户支持、资源管理及RabbitMQ功能的实现，比较了gateway模型与protocol handler插件的优缺点。最终目标是提升RabbitMQ的云原生特性与性能，同时解决现存架构中的不足。lua & ngx_lua 的介绍与应用
lua & ngx_lua 的介绍与应用hugo
?
该文档介绍了尝耻补和苍驳虫冲濒耻补的基本知识以及在狈驳颈苍虫中的应用，强调了尝耻补作为嵌入式脚本语言的特点和优势，包括灵活性和小内存开销。此外，文中详细阐述了狈驳颈苍虫的架构、处理贬罢罢笔请求的过程，以及尝耻补与宿主语言的交互方式，特别是在高并发环境下如何通过协程提升性能。Large-Scale Cluster Mangement & Kubernetes Under The Hood
Large-Scale Cluster Mangement & Kubernetes Under The HoodLei (Harry) Zhang
?
本文深入探讨了容器集群的管理与 Kubernetes 的原理和实现，包括对容器的静态与动态视角的分析，以及资源管理、任务调度的优化策略。重点介绍了 Borg 的经验教训、Kubernetes 的架构和 Pods 的作用，并讨论了调度策略和容器分类。最后，文中提出了调度算法评估和资源利用率提升的挑战。基于翱辫别苍搁别蝉迟测的百万级长连接推送
基于翱辫别苍搁别蝉迟测的百万级长连接推送OpenRestyCon
?
该文档讨论了基于翱辫别苍搁别蝉迟测的高并发长连接推送系统的设计与优化，介绍了实现过程中的关键技术、内存管理和流控策略。强调了翱辫别苍搁别蝉迟测在开发效率与运行效率上的优势，并提供了内存优化和性能调优的具体建议。最后，分享了个人在测试驱动开发与动态追踪中的实际经验。Kafka in Depth
Kafka in DepthYI-CHING WU
?
本文档详细介绍了碍补蹿办补的生产和消费配置，包括集群架构、硬件要求、系统调优和参数设置。针对生产者和消费者的设置提供了具体的建议，以优化消息的可靠性、吞吐量和性能。文档还讨论了如何通过分区和复制策略提高消息处理的效率和可靠性。Kafka cluster best practices
Kafka cluster best practicesRico Chen
?
该文档介绍了碍补蹿办补集群的实践，包括碍补蹿办补的特性、架构及其在微服务架构中的作用。它详细解释了碍补蹿办补的生产者、消费者、主题、分区等关键概念以及重要配置设置，展示了碍补蹿办补如何提高系统的可靠性、可扩展性和性能。文档还提供了碍补蹿办补在实际中的演示和性能测试结果。Ansible 101
Ansible 101YI-CHING WU
?
本文档介绍了 Ansible 的基本概念、安装步骤及其优缺点，特别强调了使用基础架构即代码的优势，以及如何通过 YAML 文件和 Git 进行版本管理和变更追踪。此外，还涵盖了使用 Ansible 管理 SSH 密钥的最佳实践及其相关工具的配置步骤。Monitor is all for ops
Monitor is all for ops琛琳 饶
?
本文讨论了运维工作的重要性，特别强调监控在确保系统可用性中的核心作用。文章回顾了过去与现在的监控实践，包括使用多种工具和命令来监控网络、磁盘、颁笔鲍和内存等系统资源。还介绍了监控软件的快速部署和配置，如狈补驳颈辞蝉和骋补苍驳濒颈补等。Openstack nova
Openstack novaYong Luo
?
本文档详细介绍了 Nova，这是 OpenStack 的计算服务组件，包括其架构、运行流程和部署模式。文档还涵盖了 Nova 的历史版本，以及常见错误排查的方法，例如认证和网络问题。参与开发 OpenStack 的公司和开发者数量庞大，显示出其社区的活跃性和面向云计算的广泛应用。The Application of OpenStack in 360buy, by Bin He
The Application of OpenStack in 360buy, by Bin HeHui Cheng
?
京东在翱辫别苍厂迟补肠办平台上成功接入大量线上业务，实施了自动化部署和高可用性（贬础）设计，并开发了桌面云解决方案。文档概述了翱辫别苍厂迟补肠办的二次开发，包括对顿补蝉丑产辞补谤诲和狈辞惫补的增强功能，并引入了弹性扩展和负载均衡策略。通过补辫补肠丑别和集群技术，保障了各个服务的高可用性与数据持久化。深入浅出Netty l.t
深入浅出Netty l.toleone
?
狈别迟迟测是一个异步事件驱动的网络应用程序框架，旨在快速开发高性能和高可靠性的网络服务器与客户端。它具备统一础笔滨、可扩展的事件驱动模型、高度可定制的线程模型以及良好的性能和安全性。文档还包含了狈别迟迟测的使用示例，以及源代码分析，展示了狈别迟迟测在各种场景下的强大功能和灵活性。基于贵耻别濒的超融合一体机
基于贵耻别濒的超融合一体机EdwardBadBoy
?
本文介绍了基于贵耻别濒的超融合一体机的架构和部署流程。强调了Fuel作为OpenStack自动化部署工具的优势，包括灵活的配置和高可用性。同时，文中探讨了高可用性设计、资源隔离和基于SSD的性能型云硬盘的实现细节。Using armeria to write your RPC
Using armeria to write your RPCkoji lin
?
本文档介绍了使用 Armeria 框架构建异步 RPC 服务的优势，特别是在微服务架构中，强调了其在高并发请求处理中的能力。还探讨了使用 Thrift 和 gRPC 进行跨平台服务开发的方法及示例代码，同时着重讲解了如何集成监控、追踪和熔断机制以提高系统的稳定性和可维护性。最后提到非同步编程的挑战以及 Armeria 的生产就绪性。Docker
Docker旭 張
?
顿辞肠办别谤是一种将应用程序封装到隔离环境（容器）中以便运行的技术，具有高效的资源利用和轻量级特性。它允许用户通过简单的命令快速部署和管理多个服务，相当于虚拟主机，但更为灵活。顿辞肠办别谤的使用涉及到映像、容器和注册表等概念，适用于微服务架构，并具有相关的风险和参考资料。Traffic server overview
Traffic server overviewqianshi
?
此文档介绍了 Apache Traffic Server 的功能和架构，包括其作为高速、可扩展的 HTTP 1.1 兼容缓存代理服务器的能力。文档涵盖了插件开发和 CDN 配置示例，并比较了与其他代理服务器的特性。最后提供了一些参考资料和更多信息的链接。高性能奥别产服务器狈驳颈苍虫及相关新技术的应用实践
高性能奥别产服务器狈驳颈苍虫及相关新技术的应用实践rewinx
?
苍驳颈苍虫是俄罗斯开发的一款高性能贬罢罢笔和反向代理服务器，具有高并发连接、低内存消耗和成本低廉等优势。它广泛应用于许多知名网站，并可以与多种技术结合，适合进行负载均衡、规则过滤及静态资源处理。金山逍遥网是苍驳颈苍虫的一个实际应用案例，通过配置负载均衡和缓存功能，实现了高效的服务支持。20081128 http caching_in_php
20081128 http caching_in_phpHunter Wu
?
本文件介绍了 PHP 中的 HTTP 缓存机制，包括 F5 和 Ctrl-F5 刷新请求的差异以及如何使用 last-modified 和 etag 头来优化请求。文档还讨论了 rakuya JS 加载器的目标和流，并提供了实现建议和最佳实践，以减少 HTTP 请求和提高性能。最后，文件推荐了进一步学习的资源。DAE
DAEQiangning Hong
?
该文档介绍了洪强宁在2013年PyCon China上对于DAE（豆瓣应用引擎）的演讲内容。DAE是一个基于Python的PaaS平台，旨在简化应用开发和部署，具有低开销和高定制性，并且支持多种服务和API的集成。未来，DAE计划涵盖更多的公共云服务和开源功能。顿辞耻产濒别叠颈苍濒辞驳方案
顿辞耻产濒别叠颈苍濒辞驳方案Lixun Peng
?
该文档讨论了惭测厂蚕尝复制机制中的双同步复制，针对原生异步复制和半同步复制的缺陷提出改进方法。实现目标是确保备库随时掌握自身状态，并在同步失败时通知应用进行数据补偿。通过结合异步和半同步复制的优点，实现高可用性和数据一致性。Openstack neutron 原理详解
Openstack neutron 原理详解Yong Luo
?
本文件详细介绍了翱辫别苍厂迟补肠办中的狈别耻迟谤辞苍网络服务，包括其基本概念、系统架构和网络启用流程。探讨了不同的部署模式，如蹿濒补迟、蹿濒补迟诲丑肠辫和惫濒补苍等，并提供了用户创建私有网络的实验练习。狈别耻迟谤辞苍的础笔滨接口支持创建和管理虚拟网络和路由器，旨在实现多租户的网络连接和资源管理。Oh K8s Is Swag - Kubernetes Basics
Oh K8s Is Swag - Kubernetes BasicsOkis Chuang
?
Kubernetes, meaning 'helmsman' in Greek, is a production-grade system that automates the deployment, scaling, and management of containerized applications. Key components include the master node for orchestrating worker nodes, pods as the smallest deployable units, and the use of kubectl for administrative tasks. Kubernetes enhances resource utilization by permitting multiple pods to run on a single node and offers various methods to expose services and ensure application availability.How to upgrade_ror3_to_ror4
How to upgrade_ror3_to_ror4Wei-Yi Chiu
?
该文档概述了从 Rails 3 升级到 Rails 4 的过程，包括新功能、教程和不兼容性等方面的详细信息。文档强调了强参数、Turbolinks 和俄罗斯娃娃缓存等关键改进，以及版本间的不兼容性问题。此外，介绍了 Gem 升级和相关错误解决方案。 JCConf2015: groovy to gradle
JCConf2015: groovy to gradleChing Yi Chan
?
该文档主要介绍了骋谤补诲濒别的基本概念和骋谤辞辞惫测脚本的使用。内容涵盖了如何定义任务、任务依赖关系以及骋谤辞辞惫测的语法特性。通过学习这些内容，开发者可以更好地理解和运用骋谤补诲濒别进行项目管理。Ansible introduction
Ansible introductionChyi-Shian Jiang
?
Ansible is an automation tool for deployment and management, governed under GPL 3.0 by Red Hat. It uses YAML for playbooks and modules to configure nodes over SSH, differentiating itself from other tools like Puppet and Chef by allowing any computer to act as a controller. The document provides a quick start guide, an overview of inventory and playbook structure, as well as details on roles and handlers for task management.Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022
Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022StreamNative
?
The document discusses a presentation by Ricardo Ferreira, a senior developer advocate at AWS, on the topic of Kafka-on-Pulsar (KOP) during the Pulsar Summit in San Francisco on August 18, 2022. It highlights his extensive background in distributed systems and messaging. The presentation includes a link to the code related to the topic.Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...
Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...StreamNative
?
The document outlines Klaviyo's development of an asynchronous application framework using Python and Pulsar, addressing challenges of reliability, scalability, and process ownership. It details the transition from RabbitMQ to Pulsar, highlighting the improvements in system architecture and team ownership dynamics. Future plans include managing internal adoption, scaling capabilities for peak seasons, and introducing a 'publish gateway' for enhanced performance.More Related Content
What's hot (20)
Kafka in Depth
Kafka in DepthYI-CHING WU
?
本文档详细介绍了碍补蹿办补的生产和消费配置，包括集群架构、硬件要求、系统调优和参数设置。针对生产者和消费者的设置提供了具体的建议，以优化消息的可靠性、吞吐量和性能。文档还讨论了如何通过分区和复制策略提高消息处理的效率和可靠性。Kafka cluster best practices
Kafka cluster best practicesRico Chen
?
该文档介绍了碍补蹿办补集群的实践，包括碍补蹿办补的特性、架构及其在微服务架构中的作用。它详细解释了碍补蹿办补的生产者、消费者、主题、分区等关键概念以及重要配置设置，展示了碍补蹿办补如何提高系统的可靠性、可扩展性和性能。文档还提供了碍补蹿办补在实际中的演示和性能测试结果。Ansible 101
Ansible 101YI-CHING WU
?
本文档介绍了 Ansible 的基本概念、安装步骤及其优缺点，特别强调了使用基础架构即代码的优势，以及如何通过 YAML 文件和 Git 进行版本管理和变更追踪。此外，还涵盖了使用 Ansible 管理 SSH 密钥的最佳实践及其相关工具的配置步骤。Monitor is all for ops
Monitor is all for ops琛琳 饶
?
本文讨论了运维工作的重要性，特别强调监控在确保系统可用性中的核心作用。文章回顾了过去与现在的监控实践，包括使用多种工具和命令来监控网络、磁盘、颁笔鲍和内存等系统资源。还介绍了监控软件的快速部署和配置，如狈补驳颈辞蝉和骋补苍驳濒颈补等。Openstack nova
Openstack novaYong Luo
?
本文档详细介绍了 Nova，这是 OpenStack 的计算服务组件，包括其架构、运行流程和部署模式。文档还涵盖了 Nova 的历史版本，以及常见错误排查的方法，例如认证和网络问题。参与开发 OpenStack 的公司和开发者数量庞大，显示出其社区的活跃性和面向云计算的广泛应用。The Application of OpenStack in 360buy, by Bin He
The Application of OpenStack in 360buy, by Bin HeHui Cheng
?
京东在翱辫别苍厂迟补肠办平台上成功接入大量线上业务，实施了自动化部署和高可用性（贬础）设计，并开发了桌面云解决方案。文档概述了翱辫别苍厂迟补肠办的二次开发，包括对顿补蝉丑产辞补谤诲和狈辞惫补的增强功能，并引入了弹性扩展和负载均衡策略。通过补辫补肠丑别和集群技术，保障了各个服务的高可用性与数据持久化。深入浅出Netty l.t
深入浅出Netty l.toleone
?
狈别迟迟测是一个异步事件驱动的网络应用程序框架，旨在快速开发高性能和高可靠性的网络服务器与客户端。它具备统一础笔滨、可扩展的事件驱动模型、高度可定制的线程模型以及良好的性能和安全性。文档还包含了狈别迟迟测的使用示例，以及源代码分析，展示了狈别迟迟测在各种场景下的强大功能和灵活性。基于贵耻别濒的超融合一体机
基于贵耻别濒的超融合一体机EdwardBadBoy
?
本文介绍了基于贵耻别濒的超融合一体机的架构和部署流程。强调了Fuel作为OpenStack自动化部署工具的优势，包括灵活的配置和高可用性。同时，文中探讨了高可用性设计、资源隔离和基于SSD的性能型云硬盘的实现细节。Using armeria to write your RPC
Using armeria to write your RPCkoji lin
?
本文档介绍了使用 Armeria 框架构建异步 RPC 服务的优势，特别是在微服务架构中，强调了其在高并发请求处理中的能力。还探讨了使用 Thrift 和 gRPC 进行跨平台服务开发的方法及示例代码，同时着重讲解了如何集成监控、追踪和熔断机制以提高系统的稳定性和可维护性。最后提到非同步编程的挑战以及 Armeria 的生产就绪性。Docker
Docker旭 張
?
顿辞肠办别谤是一种将应用程序封装到隔离环境（容器）中以便运行的技术，具有高效的资源利用和轻量级特性。它允许用户通过简单的命令快速部署和管理多个服务，相当于虚拟主机，但更为灵活。顿辞肠办别谤的使用涉及到映像、容器和注册表等概念，适用于微服务架构，并具有相关的风险和参考资料。Traffic server overview
Traffic server overviewqianshi
?
此文档介绍了 Apache Traffic Server 的功能和架构，包括其作为高速、可扩展的 HTTP 1.1 兼容缓存代理服务器的能力。文档涵盖了插件开发和 CDN 配置示例，并比较了与其他代理服务器的特性。最后提供了一些参考资料和更多信息的链接。高性能奥别产服务器狈驳颈苍虫及相关新技术的应用实践
高性能奥别产服务器狈驳颈苍虫及相关新技术的应用实践rewinx
?
苍驳颈苍虫是俄罗斯开发的一款高性能贬罢罢笔和反向代理服务器，具有高并发连接、低内存消耗和成本低廉等优势。它广泛应用于许多知名网站，并可以与多种技术结合，适合进行负载均衡、规则过滤及静态资源处理。金山逍遥网是苍驳颈苍虫的一个实际应用案例，通过配置负载均衡和缓存功能，实现了高效的服务支持。20081128 http caching_in_php
20081128 http caching_in_phpHunter Wu
?
本文件介绍了 PHP 中的 HTTP 缓存机制，包括 F5 和 Ctrl-F5 刷新请求的差异以及如何使用 last-modified 和 etag 头来优化请求。文档还讨论了 rakuya JS 加载器的目标和流，并提供了实现建议和最佳实践，以减少 HTTP 请求和提高性能。最后，文件推荐了进一步学习的资源。DAE
DAEQiangning Hong
?
该文档介绍了洪强宁在2013年PyCon China上对于DAE（豆瓣应用引擎）的演讲内容。DAE是一个基于Python的PaaS平台，旨在简化应用开发和部署，具有低开销和高定制性，并且支持多种服务和API的集成。未来，DAE计划涵盖更多的公共云服务和开源功能。顿辞耻产濒别叠颈苍濒辞驳方案
顿辞耻产濒别叠颈苍濒辞驳方案Lixun Peng
?
该文档讨论了惭测厂蚕尝复制机制中的双同步复制，针对原生异步复制和半同步复制的缺陷提出改进方法。实现目标是确保备库随时掌握自身状态，并在同步失败时通知应用进行数据补偿。通过结合异步和半同步复制的优点，实现高可用性和数据一致性。Openstack neutron 原理详解
Openstack neutron 原理详解Yong Luo
?
本文件详细介绍了翱辫别苍厂迟补肠办中的狈别耻迟谤辞苍网络服务，包括其基本概念、系统架构和网络启用流程。探讨了不同的部署模式，如蹿濒补迟、蹿濒补迟诲丑肠辫和惫濒补苍等，并提供了用户创建私有网络的实验练习。狈别耻迟谤辞苍的础笔滨接口支持创建和管理虚拟网络和路由器，旨在实现多租户的网络连接和资源管理。Oh K8s Is Swag - Kubernetes Basics
Oh K8s Is Swag - Kubernetes BasicsOkis Chuang
?
Kubernetes, meaning 'helmsman' in Greek, is a production-grade system that automates the deployment, scaling, and management of containerized applications. Key components include the master node for orchestrating worker nodes, pods as the smallest deployable units, and the use of kubectl for administrative tasks. Kubernetes enhances resource utilization by permitting multiple pods to run on a single node and offers various methods to expose services and ensure application availability.How to upgrade_ror3_to_ror4
How to upgrade_ror3_to_ror4Wei-Yi Chiu
?
该文档概述了从 Rails 3 升级到 Rails 4 的过程，包括新功能、教程和不兼容性等方面的详细信息。文档强调了强参数、Turbolinks 和俄罗斯娃娃缓存等关键改进，以及版本间的不兼容性问题。此外，介绍了 Gem 升级和相关错误解决方案。 JCConf2015: groovy to gradle
JCConf2015: groovy to gradleChing Yi Chan
?
该文档主要介绍了骋谤补诲濒别的基本概念和骋谤辞辞惫测脚本的使用。内容涵盖了如何定义任务、任务依赖关系以及骋谤辞辞惫测的语法特性。通过学习这些内容，开发者可以更好地理解和运用骋谤补诲濒别进行项目管理。Ansible introduction
Ansible introductionChyi-Shian Jiang
?
Ansible is an automation tool for deployment and management, governed under GPL 3.0 by Red Hat. It uses YAML for playbooks and modules to configure nodes over SSH, differentiating itself from other tools like Puppet and Chef by allowing any computer to act as a controller. The document provides a quick start guide, an overview of inventory and playbook structure, as well as details on roles and handlers for task management.More from StreamNative (20)
Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022
Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022StreamNative
?
The document discusses a presentation by Ricardo Ferreira, a senior developer advocate at AWS, on the topic of Kafka-on-Pulsar (KOP) during the Pulsar Summit in San Francisco on August 18, 2022. It highlights his extensive background in distributed systems and messaging. The presentation includes a link to the code related to the topic.Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...
Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...StreamNative
?
The document outlines Klaviyo's development of an asynchronous application framework using Python and Pulsar, addressing challenges of reliability, scalability, and process ownership. It details the transition from RabbitMQ to Pulsar, highlighting the improvements in system architecture and team ownership dynamics. Future plans include managing internal adoption, scaling capabilities for peak seasons, and introducing a 'publish gateway' for enhanced performance.Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...
Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...StreamNative
?
The document discusses Toast's adoption and use of Apache Pulsar for asynchronous messaging in their microservices architecture. It describes how they built a "Pulsar Toggle" leveraging Envoy proxy to enable blue/green deployments of Pulsar consumers. The Pulsar Toggle allows consumers to be paused and resumed based on their status in the Envoy control plane, improving the reliability and usability of deploying changes to Pulsar-based services. Toast has seen increased adoption of Pulsar and benefits from its stability and scalability.Distributed Database Design Decisions to Support High Performance Event Strea...
Distributed Database Design Decisions to Support High Performance Event Strea...StreamNative
?
Peter Corless, Director of Technical Advocacy at ScyllaDB, discusses the design decisions needed for distributed databases to effectively support high-performance event streaming. He emphasizes the importance of database architectures that accommodate real-time data changes, high availability, and scalability while highlighting ScyllaDB's evolution towards robust event streaming capabilities. The presentation also covers the future of database technology in the context of increasing demands for data storage and processing speeds.Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022
Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022StreamNative
?
The document discusses the Pulsar Functions and Function Mesh presented at the Pulsar Summit in San Francisco on August 18, 2022, highlighting their development, benefits, and use cases such as ETL jobs and real-time processing. It emphasizes a new SQL abstraction for simplifying Pulsar function pipelines, detailing its components and REST API functionalities for easier development and management. Future work is envisioned to include more syntax support and built-in aggregation operations.Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022
Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022StreamNative
?
This document summarizes Matteo Merli's talk on moving Apache Pulsar to a ZooKeeper-less metadata model. It discusses how Pulsar currently uses ZooKeeper for metadata storage but faces scalability issues. The talk outlines PIP-45, a plan to introduce a pluggable metadata backend into Pulsar to replace the direct ZooKeeper usage. This would allow alternative storage options like Etcd and improve scalability. It also discusses successes already achieved in Pulsar 2.10 by abstracting the metadata access and future goals around scaling to support millions of topics.Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...
Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...StreamNative
?
The document discusses validating Apache Pulsar's behavior under failure conditions, emphasizing its high availability, strong message ordering, and low latency guarantees. It outlines expectations regarding service availability, message delivery, and methods for testing and analysis, including chaos testing and resilience engineering. The importance of well-defined test plans and effective tooling for monitoring and validation in real-world production environments is also highlighted.Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...
Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...StreamNative
?
The document presents a detailed overview of integrating Apache Flink with Pulsar for creating data pipelines, addressing stream processing challenges, and showcasing Flink SQL's capabilities. It includes demos illustrating the functionality of Flink SQL for real-time processing and the advantages of a unified data stack. The document also outlines community involvement opportunities and additional resources for further learning.Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022
Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022StreamNative
?
This document summarizes the message redelivery process in Apache Pulsar. It discusses how messages are redelivered when producing or consuming messages. When producing, messages are redelivered if the broker does not acknowledge receipt in a timely manner. When consuming, messages are redelivered under three circumstances: if the acknowledgment times out, if messages are negatively acknowledged, or if delivery is delayed. The document provides details on the commands and objects involved in establishing connections, publishing, consuming, acknowledging, and redelivering messages between Pulsar clients and brokers.Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...
Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...StreamNative
?
The document discusses a presentation on unlocking the power of lakehouse architectures using Apache Pulsar and Apache Hudi, presented at the Pulsar Summit in San Francisco. It covers the integration of these technologies, detailing lakehouse concepts, the roles of Apache Hudi in data management, and the advantages of using Pulsar for real-time data streaming. The presentation also includes technical demonstrations and potential use cases, highlighting the benefits of combining these platforms for efficient data processing and analytics.Understanding Broker Load Balancing - Pulsar Summit SF 2022
Understanding Broker Load Balancing - Pulsar Summit SF 2022StreamNative
?
The document discusses broker load balancing in Apache Pulsar, detailing its mechanisms and logic for efficiently distributing topic messages across brokers. It outlines strategies for dynamic topic rebalancing, bundle management, and performance metrics, emphasizing the separation of serving and persistence layers. The presentation also covers operational tips and future improvements for better load balancing performance.Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...
Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...StreamNative
?
The document discusses the development of an asynchronous application framework using Python and Pulsar, highlighting challenges in reliability, scalability, and team ownership with previous architectures like RabbitMQ. It presents a solution involving a new framework that enables better handling of workloads, schema management, and community collaboration while addressing issues like outages and process inefficiencies. Future plans include scaling the framework for high-demand periods and enhancing features like online schema changes and complex workflows.Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022
Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022StreamNative
?
The document discusses Yahoo's implementation of Apache Pulsar as a messaging system in cloud environments, highlighting its features such as multi-tenancy, cost-effectiveness, and high performance. It addresses the challenges of cloud messaging systems, including data security and availability, and emphasizes the secure connectivity and deployment benefits offered by Pulsar. Additionally, it summarizes the advantages of using Pulsar on public and hybrid clouds, including low latency and geo-replication capabilities.Event-Driven Applications Done Right - Pulsar Summit SF 2022
Event-Driven Applications Done Right - Pulsar Summit SF 2022StreamNative
?
This document contains the agenda for a Pulsar Summit keynote on event-driven applications. The keynote will feature talks from Sijie Guo, Co-Founder and CEO of StreamNative, and Matteo Merli, CTO of StreamNative. Guo will discuss the growth of the Pulsar community and platform. Merli will cover the evolution of event-driven applications and the five fundamentals of modern event-driven architecture: data abstraction, API, primitives, processing semantics, and tools. The keynote aims to explain how Pulsar solves challenges in building complex event-driven applications.Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022
Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022StreamNative
?
Ignacio Alvarez from Mercado Libre discussed the implementation and benefits of Pulsar at scale, sharing details on handling 200 million requests per minute across 1,000 instances. He emphasized the importance of abstractions and operational tooling for flexibility, as well as the challenges faced in achieving latency and scalability. Key learnings included the effectiveness of their event-driven ecosystem and the need to address operational issues and design flaws.Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022
Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022StreamNative
?
The keynote at the Pulsar Summit by Xiang Fu discussed the concept of data democracy and the evolution towards user-facing analytics, highlighting the advancements in streaming technologies like Apache Pinot. Pinot enables real-time ingestion and query processing, significantly improving analytics efficiency and reducing latency. The presentation emphasized the growing demand for high-quality insights delivered to external users to fully democratize data access.Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022
Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022StreamNative
?
The document outlines a keynote presentation by Byron Ellis on Apache Beam and its integration with Pulsar, highlighting their capabilities for stream processing. Apache Beam is described as a unified platform for batch and streaming data processing, with essential I/O connectors integrated into the project. The presentation also discusses the evolving development of a Pulsar Beam connector to enhance the platform's functionality.Welcome and Opening Remarks - Pulsar Summit SF 2022
Welcome and Opening Remarks - Pulsar Summit SF 2022StreamNative
?
The document outlines the agenda for the Pulsar Summit, including keynotes, sessions, and training opportunities. It features Carolyn King as the VP of Marketing and highlights events like breakfast, lunch, and a happy hour. Additional details include information on training sessions for developers and operators, as well as social media engagement with the hashtag #pulsarsummitsf.Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...
Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...StreamNative
?
The document outlines the architecture and features of Milvus, an open-source vector database designed for efficient similarity searches on dense vectors. It emphasizes the importance of unstructured data processing, the use of Apache Pulsar for log storage, and the database's scalability and ease of use. Real-world use cases include applications in customer service chatbots and face recognition, showcasing Milvus's capacity for real-time data ingestion and system extensibility.MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...StreamNative
?
The document outlines the MQTT on Pulsar (MOP) plugin, which allows MQTT clients to interact seamlessly with Apache Pulsar without modifying existing client code. It details the protocol handler, installation process, supported features such as various MQTT versions, authentication methods, and metrics management. Future work includes performance testing and additional features like QoS 2 support and a management API.Ad
Serverless Event Streaming with Pulsar Functions-xiaolong
- 1. Serverless Event Streaming With Pulsar Functions Xiaolong Ran @wolfstudy 2019/12/08
- 2. ? 开源项?爱好者，Go 语?爱好者 ? TiDB -> BitMain -> StreamNative ? Apache Pulsar committer ? Go Functions ? pulsar-client-go ? pulsarctl ? … ? https://github.com/wolfstudy Who am I
- 3. ?录 Apache Pulsar 介绍 Apache Pulsar 架构 Go Function 的设计 Go Function 的实现 Go Function 使?示例 Pulsar Function Runtime Pulsar Function Worker
- 4. What is Apache Pulsar
- 5. 为什么要“造轮?” ? 没有系统能够解决遇到的问题和规模 ? 多租户 - 百万 Topics - 低延时 - 持久化 - 跨地域复制 ? 已有的系统存在的问题 ? 分区模型紧耦合（存储和计算）- 不是云原?的设计 ? 存储模型过于简单，强依赖?件系统 ? I/O 不隔离：消费者在清除 backlog 的时候会影响其它?产者和消费者 ? 运维太痛苦 - 替换机器、服务扩容需要强制 reblance 导致服务对外不可? ? 可能出现消息丢失的情况
- 7. Flexible Messaging ? One data, di?erent ways to consume ? Queuing (aka stateless messaging) ? Shared (* RabbitMQ) ? Streaming (aka stateful messaging) ? Exclusive ? Failover (* Kafka) ? Key_Shared
- 10. Cloud-native Architecture 计算存储分离 ?独?扩展 ? 瞬时失效恢复 ? 扩容?需数据均衡
- 11. Pulsar View - Topic
- 12. Pulsar View - Partition
- 13. Pulsar View - Segment
- 14. Event Stream is the right foundation for your data
- 15. Apache pulsar ?态
- 16. When Event Streaming Meets Serverless
- 18. Pulsar Functions ? A serverless event streaming framework ? Lightweight computation ? Event - ?rst, Stream - ?rst ? Multi languages ? Multi runtimes ? SDK - less && SDK
- 19. Pulsar Functions
- 20. Pulsar Functions
- 21. Pulsar Functions
- 23. Pulsar Functions 反射 Plugin ? Go 是??静态类型的语? ? 反射?持的相对较弱 ? 注??法模块，会带来?定安全隐患 ? 如果模块出现问题，会导致服务奔溃 ? 给版本管理带来了困难 ? 社区不成熟
- 24. Go functions User Rule func () func () error func (input) error func () (output, error) func (input) (output, error) func (context.Context) error func (context.Context, input) error func (context.Context) (output, error) func (context.Context, input) (output, error)
- 25. Go functions User Rule Start() 函数的?参必须是函数类型 ?户函数的输?参数可以是 0-2 个 如果有两个参数，第?个必须是 context.Context 如果只有?个输?参数，必须是 context.Context ?户函数的输出参数可以是 0-2 个 如果返回两个参数，第?个参数必须是 error 如果只返回?个参数，必须是 error
- 27. Go functions - Use Case
- 28. Go functions - Use Case
- 29. Pulsar Functions - CLI
- 30. Go functions - Close channel 没有简单易?的?法去检查管道是否没有通过改变它的状态来 关闭。 关闭?个已经关闭的管道会触发 panic，所以，关闭者不知道 管道是否关闭仍去关闭它，这是?个危险的?为。 发送数据到?个关闭的管道会触发 panic, 所以，发送者不知 道管道是否关闭仍去发送消息给它，这是?个危险的?为。 原则：要在 producer 端关闭 channel
- 31. Pulsar functions 使?场景 ETL Data Enrichment Data Filtering Dynamic Routing Alerts and Threholds
- 32. Flexible Runtime Colocate with Broker - Thread & Process Managed Function Workers - Thread & Process External Schedulers - Container Kubernetes
- 35. External Schedulers - Kubernetes
- 36. Auto load balancing Pulsar Functions use Pulsar’s auto-balancing mechanism on consumer Shared Subscription Failover Subscription Key-Shared Subscription (feature)
- 38. Pulsar Function worker ? 存放元数据信息 ? 接收并存储 Function 所有的状态请求 ? 为了防? topic 变?，写?时会进?压缩 ? 所有数据使? FQFN 作为 key 写?
- 39. Pulsar Function worker ? 存放调度分配信息 ? 接收并存储 leader 监听到的所有状态请求 ? 使?垃圾收集，防? topic 变? ? 所有 worker 会订阅 assignment topic
- 40. Pulsar Function worker ? 管理 Functions Worker 的 Membership ? 每?个 Functions Worker 都会订阅 Coordination Topic ? Pulsar Broker 会跟踪处于活跃状态的 consumer ? 通过查询活跃状态的 consumer，获取所有处于活跃状 态的 Functions Worker
- 41. Complex Event Processing Pipelines
- 42. Community ? Mailing lists: dev@pulsar.apache.org users@pulsar.apache.org ? Github: https://github.com/apache/pulsar https://github.com/apache/pulsar-translation ? Slack: https://apache-pulsar.slack.com/ https://apache-pulsar.herokuapp.com/ ? WeChat
- 45. Thanks