Quilt - Distributed Load Simulation from AWS
0 likes222 views
The document details the speaker's experience in load testing and resiliency as an engineering manager at BlueJeans, emphasizing the importance of automating load simulation and performance testing to achieve high availability. It outlines the challenges of testing large distributed systems and the tools and frameworks developed for effective monitoring and analysis. Key takeaways include the necessity of automation, scalability through cloud resources, and a vendor-agnostic approach to infrastructure.
1 of 19
Download to read offline
Ad
Recommended
Scaling on AWS for the First 10 Million Users at Websummit Dublin
Scaling on AWS for the First 10 Million Users at Websummit DublinIan Massingham
?
The presentation by Ian Massingham at the 2014 Web Summit discusses strategies for scaling applications on AWS, emphasizing the importance of utilizing various AWS services effectively. It outlines a phased approach to scaling, starting from a single EC2 instance to load balancing across multiple instances with redundancy and failover capabilities by employing database solutions like Amazon RDS and DynamoDB. Key concepts include auto-scaling, monitoring, and optimization to handle increasing user demand efficiently.AWS Pop-up Loft Berlin: Cache is King - Running Lean Architectures: Optimizin...
AWS Pop-up Loft Berlin: Cache is King - Running Lean Architectures: Optimizin...AWS Germany
?
The document emphasizes the importance of caching in scaling applications and improving performance while reducing costs, highlighting various types of caching methods such as browser, edge, web, and application caching. It provides insights into effective caching strategies, metrics for evaluating cache performance, and recommendations for implementation, particularly using services like Amazon CloudFront and Elasticache. Additionally, the document illustrates practical experimentation demonstrating significant improvements in response times when employing caching techniques.Performance testing in scope of migration to cloud by Serghei Radov
Performance testing in scope of migration to cloud by Serghei RadovValeriia Maliarenko
?
This document discusses performance testing considerations for migrating an application to the cloud. It covers cloud computing principles like multi-tenancy and horizontal scalability. Challenges like over-provisioning and network issues are addressed. Effective provisioning using predictive auto-scaling is recommended. Tools for monitoring, load testing, and analyzing results are presented, including New Relic, DataDog, Flood.io, and JMeter. The document emphasizes defining acceptance criteria, workload characterization, and iterating on tests to analyze and scale resources. Costs of various performance testing tools on cloud providers are compared.Things You MUST Know Before Deploying OpenStack: Bruno Lago, Catalyst IT
Things You MUST Know Before Deploying OpenStack: Bruno Lago, Catalyst ITOpenStack
?
Bruno Lago's presentation outlines key considerations for deploying OpenStack, including costs estimated at $150k for a production cluster and recommendations for hardware setup using open-source switches. It emphasizes the importance of proper node segmentation, automated testing, and avoiding common deployment mistakes like running unmerged patches. The document also compares OpenStack's pricing with major cloud providers, demonstrating potential savings.Rails Conf Europe 2007 Notes
Rails Conf Europe 2007 NotesRoss Lawley
?
This document summarizes notes from several talks and discussions about scaling Ruby on Rails applications. Some key points include:
- To scale a Rails app, focus on load balancing and separating application tiers (front-end servers, application servers, databases). Use caching, database replication, and queuing systems.
- Monitoring tools like Munin and Nagios are important for tracking performance bottlenecks. Metrics like requests per second help determine capacity.
- Scaling the database is critical; techniques include sharding, replication, and optimizing queries with indexing. Caching data in Memcached also helps scale performance.
- Virtualization allows consolidating servers. Software load balancers like Nginx can routePerformance on a budget
Performance on a budgetDimitry Ushakov
?
This document summarizes a presentation on performance optimization on a budget. It discusses measuring and improving performance at the front-end through asset optimization, latency reduction, and client-side rendering. It also discusses measuring and optimizing performance at the backend through caching, databases, and server-side architecture. The document lists several free and paid tools for profiling, testing, and analyzing performance. It concludes with best practices for performance including establishing goals, architecture, testing, and an SDLC approach.AWS (Hadoop) Meetup 30.04.09
AWS (Hadoop) Meetup 30.04.09Chris Purrington
?
The document discusses a London user group meeting held on April 30, 2009, covering topics such as auto-scaling with Amazon EC2, performance improvements using NGINX and Memcached, and persistent Django on AWS. It includes insights into scalable hosting solutions, parallel computing with Hadoop, and applications of data processing at companies like Facebook and Yahoo. The meeting highlights advancements in cloud computing and practical implementations for developers and network administrators.Clouds: All fluff and no substance?
Clouds: All fluff and no substance?Guy Coates
?
The document discusses the use of cloud computing at the Sanger Institute, particularly for large-scale genomic research and data management. It explains the improvements in sequencing technology, cost efficiency of cloud services compared to traditional hardware, and the challenges of data transfer and infrastructure setup. The potential for utilizing cloud resources to enhance genomic data analysis and address increased demand for genomic annotation is emphasized, alongside the need for optimizing data accessibility and compute capabilities.PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...
PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...Chris Fregly
?
The document outlines the integration of Kubernetes and Istio for high-performance model serving, highlighting the capabilities of Pipeline.ai in deploying and managing TensorFlow models in production environments. It covers various functionalities such as continuous model training, A/B testing, and the importance of real-time predictions, along with detailed instructions on deploying and optimizing machine learning models. Additionally, it emphasizes open-source collaboration, encouraging community engagement through GitHub.Netflix keystone streaming data pipeline @scale in the cloud-dbtb-2016
Netflix keystone streaming data pipeline @scale in the cloud-dbtb-2016Monal Daxini
?
The document discusses Netflix's scalable streaming data pipeline, focusing on the transition from older technologies to a new event processing infrastructure called Keystone. It highlights the pipeline's capabilities, including high event ingestion rates, reliability, and the use of Kafka for event routing, as well as the automation and self-service tools implemented for operational efficiency. Future directions include support for various stream processing engines and advanced metrics and monitoring systems.Flexible compute
Flexible computePeter Clapham
?
The document outlines the development and implementation of a flexible compute platform at Sanger, discussing the transition from traditional HPC to a hybrid cloud architecture using OpenStack. It highlights the infrastructure specifications, resource management strategies, and dynamic workflows aimed at improving application development and scaling services efficiently. The document also addresses the need for collaboration in cloud application development, mentioning early adopters and future opportunities for adaptation and feature requests.Sanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticiansPeter Clapham
?
The document outlines Sanger's development and deployment of a flexible compute platform utilizing OpenStack, detailing its architecture, components, and the transition from existing systems. It emphasizes the benefits of a private cloud for resource management, scalability, and application development, while also discussing the planned enhancements and migration strategies. The initiative aims to enable dynamic workflows, improve collaboration, and provide a robust infrastructure for computational needs in genomics research.Large-scaled Deploy Over 100 Servers in 3 Minutes
Large-scaled Deploy Over 100 Servers in 3 MinutesHiroshi SHIBATA
?
Large-scaled Deploy Over 100 Servers in 3 Minutes
Deployment strategy for next generation involves creating OS images using Packer and cloud-init that allow deploying over 100 servers within 3 minutes through automation. The strategy uses Puppet for configuration management and builds minimal and application-specific images to reduce bootstrap time. All deployment operations are implemented through a CLI tool for rapid and automated scaling.OpenNebulaConf 2016 - Measuring and tuning VM performance by Boyan Krosnov, S...
OpenNebulaConf 2016 - Measuring and tuning VM performance by Boyan Krosnov, S...OpenNebula Project
?
The document discusses strategies for measuring and optimizing VM performance in cloud environments, focusing on key metrics like application latency and throughput. It emphasizes best practices for hardware, network, and storage configuration to enhance performance while minimizing costs. The author encourages skepticism towards synthetic workloads and highlights the importance of real-world testing to ensure accurate performance assessment.Azug - successfully breeding rabits
Azug - successfully breeding rabitsYves Goeleven
?
This document discusses successfully breeding rabbits in the cloud. It describes rabbits as small public apps that want to live outside and scale quickly. It discusses how the cloud provides rabbits with global reach, fast time to market, performance, scalability, availability and cost efficiency. The document emphasizes that the weakest link limits overall scalability and that caches, queues and separating concerns can help address this. It stresses understanding cloud capacity and efficiently using resources to keep costs low. Finally, it discusses how rabbits can achieve self-service, marketing, support, education and testing at scale.Scalling Rails: The Journey to 200M Notifications
Scalling Rails: The Journey to 200M NotificationsGustavo Araujo
?
Talk presented at Tropical On Rails.DevOps, CLI, APIs, Oh My! Security Gone Agile
DevOps, CLI, APIs, Oh My! Security Gone AgileMatt Tesauro
?
The document discusses the integration of DevOps practices with security measures in a rapidly changing software development landscape. It emphasizes the need for automated testing, continuous integration, and collaboration between development and operations teams to secure applications, infrastructure, and code. Key takeaways include the importance of automation, workflow standardization, and adapting to ongoing changes in software methodologies.Load testing and performance tracing
Load testing and performance tracingHans H?chtl
?
The document discusses load testing and performance optimization techniques for applications, emphasizing the importance of profiling, identifying bottlenecks, and using tools like Apache JMeter and BlazeMeter. It provides insights into optimizing PHP applications, utilizing various tools for memory and performance profiling, and improving database and frontend performance. The author, Hans H?chtl, also shares contact information and additional resources for further reading.Ensuring Performance in a Fast-Paced Environment (CMG 2014)
Ensuring Performance in a Fast-Paced Environment (CMG 2014)Martin Spier
?
The document presents an overview of performance engineering practices at Netflix, emphasizing their culture of freedom and responsibility, independent development teams, and agile methodologies. Key strategies include auto-scaling, canary analysis, and large-scale performance testing to ensure high availability and reliability in their streaming service. Additionally, it highlights the use of advanced tools like automated canary analysis to optimize deployment and performance metrics monitoring.Infrastructure Considerations : Design : "webops"
Infrastructure Considerations : Design : "webops"Piyush Kumar
?
This document provides recommendations for preparing infrastructure for the future. It suggests choosing a long-term supported Linux distribution like RHEL, CentOS, or Ubuntu LTS. Infrastructure should be designed to be scalable, robust, manageable, resilient, and cost-effective. Services should be split across redundant servers and high availability/failover implemented. Automation, configuration management, and continuous integration/deployment are also recommended. Comprehensive monitoring of infrastructure and applications is important for capacity planning and issue detection. The document provides additional tips regarding caching, security, change management, backups, and ongoing re-architecting.3.2 Streaming and Messaging
3.2 Streaming and MessagingÕñ¶« Áõ
?
The document presents a discussion on real-time statistics leveraging Apache Storm and RocketMQ, highlighting their integration within the streaming ecosystem and best practices. It covers challenges in stateful statistics, event counting, and performance tuning related to distributed systems. Additionally, it provides insights into using RocketMQ with various streaming frameworks, alongside suggestions for improving performance and handling common issues.Introduction openstack-meetup-nov-28
Introduction openstack-meetup-nov-28Sadique Puthen
?
Red Hat Openstack and Ceph Meetup, Pune | 28th NOV 2015
Sadique Puthen, Principal Technical Support Engineer at Red Hat, Inc., gave an introduction to Red Hat Openstack (RDO) and its components. He discussed how Openstack provides infrastructure services like compute (Nova), storage (Cinder, Swift), networking (Neutron), and database (Trove) as a service. He also covered Openstack deployment options like Packstack, TripleO, and Ironic for bare metal provisioning. The meetup aimed to introduce Openstack components and services and their role in providing infrastructure as a service through a cloud platform.Sanger OpenStack presentation March 2017
Sanger OpenStack presentation March 2017Dave Holland
?
The Sanger Institute's transition to OpenStack aimed to enhance data handling and computing efficiency, while addressing shortcomings of their previous LSF system. They outlined their journey from initial training to production systems, discussed hardware choices, and detailed numerous challenges encountered, including networking and deployment issues. Future directions emphasize expanding Ceph storage capabilities and exploring advanced OpenStack features like bare-metal deployment and instance pricing.A real-life account of moving 100% to a public cloud
A real-life account of moving 100% to a public cloudJulien SIMON
?
This document summarizes Viadeo's experience moving their entire infrastructure to AWS. It describes their motivation for moving to the cloud to focus on their product instead of managing hardware. It outlines their step-by-step process, including automating infrastructure with CloudFormation, baking AMIs, and continuous integration/delivery. It discusses lessons learned around networking, databases, and involving stakeholders in the transition. Currently, their staging and production environments run in AWS across multiple regions, with future plans to migrate more services.introduction to node.js
introduction to node.jsorkaplan
?
This document provides an introduction to Node.js, a framework for building scalable server-side applications with asynchronous JavaScript. It discusses what Node.js is, how it uses non-blocking I/O and events to avoid wasting CPU cycles, and how external Node modules help create a full JavaScript stack. Examples are given of using Node modules like Express for building RESTful APIs and Socket.IO for implementing real-time features like chat. Best practices, limitations, debugging techniques and references are also covered.PipelineAI + AWS SageMaker + Distributed TensorFlow + AI Model Training and S...
PipelineAI + AWS SageMaker + Distributed TensorFlow + AI Model Training and S...Chris Fregly
?
Pipeline.AI is a platform for deploying and optimizing machine learning models at scale. It allows users to package models with their runtime dependencies, perform load testing and optimizations, deploy models to production safely using techniques like canary deployments, and monitor models both offline and online. The platform aims to enable live, continuous model training directly in production environments.Optimizing, Profiling, and Deploying High Performance Spark ML and TensorFlow AI
Optimizing, Profiling, and Deploying High Performance Spark ML and TensorFlow AIData Con LA
?
Pipeline.ai specializes in high-performance model training and serving using GPUs and various cloud platforms. The platform focuses on model serving to provide continuous insights in live production environments, enabling real-time predictions and optimizations. It supports deploying models with a strong emphasis on maintaining runtime consistency across various stages of model deployment.Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And...
Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And...Lightbend
?
The document discusses the architecture and benefits of using Akka Streams and Kafka for efficient web crawling. It highlights the reactive manifesto principles of responsiveness, resilience, and elasticity while detailing the system's design for handling high-volume URL processing. Key features include asynchronous processing, back-pressure management, and modular components to ensure scalability and performance without compromising resource efficiency.Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdfcaoyixuan2019
?
A presentation at Internetware 2025.AI VIDEO MAGAZINE - June 2025 - r/aivideo
AI VIDEO MAGAZINE - June 2025 - r/aivideo1pcity Studios, Inc
?
AI VIDEO MAGAZINE - r/aivideo community newsletter ¨C Exclusive Tutorials: How to make an AI VIDEO from scratch, PLUS: How to make AI MUSIC, Hottest ai videos of 2025, Exclusive Interviews, New Tools, Previews, and MORE - JUNE 2025 ISSUE -More Related Content
Similar to Quilt - Distributed Load Simulation from AWS (20)
PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...
PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...Chris Fregly
?
The document outlines the integration of Kubernetes and Istio for high-performance model serving, highlighting the capabilities of Pipeline.ai in deploying and managing TensorFlow models in production environments. It covers various functionalities such as continuous model training, A/B testing, and the importance of real-time predictions, along with detailed instructions on deploying and optimizing machine learning models. Additionally, it emphasizes open-source collaboration, encouraging community engagement through GitHub.Netflix keystone streaming data pipeline @scale in the cloud-dbtb-2016
Netflix keystone streaming data pipeline @scale in the cloud-dbtb-2016Monal Daxini
?
The document discusses Netflix's scalable streaming data pipeline, focusing on the transition from older technologies to a new event processing infrastructure called Keystone. It highlights the pipeline's capabilities, including high event ingestion rates, reliability, and the use of Kafka for event routing, as well as the automation and self-service tools implemented for operational efficiency. Future directions include support for various stream processing engines and advanced metrics and monitoring systems.Flexible compute
Flexible computePeter Clapham
?
The document outlines the development and implementation of a flexible compute platform at Sanger, discussing the transition from traditional HPC to a hybrid cloud architecture using OpenStack. It highlights the infrastructure specifications, resource management strategies, and dynamic workflows aimed at improving application development and scaling services efficiently. The document also addresses the need for collaboration in cloud application development, mentioning early adopters and future opportunities for adaptation and feature requests.Sanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticiansPeter Clapham
?
The document outlines Sanger's development and deployment of a flexible compute platform utilizing OpenStack, detailing its architecture, components, and the transition from existing systems. It emphasizes the benefits of a private cloud for resource management, scalability, and application development, while also discussing the planned enhancements and migration strategies. The initiative aims to enable dynamic workflows, improve collaboration, and provide a robust infrastructure for computational needs in genomics research.Large-scaled Deploy Over 100 Servers in 3 Minutes
Large-scaled Deploy Over 100 Servers in 3 MinutesHiroshi SHIBATA
?
Large-scaled Deploy Over 100 Servers in 3 Minutes
Deployment strategy for next generation involves creating OS images using Packer and cloud-init that allow deploying over 100 servers within 3 minutes through automation. The strategy uses Puppet for configuration management and builds minimal and application-specific images to reduce bootstrap time. All deployment operations are implemented through a CLI tool for rapid and automated scaling.OpenNebulaConf 2016 - Measuring and tuning VM performance by Boyan Krosnov, S...
OpenNebulaConf 2016 - Measuring and tuning VM performance by Boyan Krosnov, S...OpenNebula Project
?
The document discusses strategies for measuring and optimizing VM performance in cloud environments, focusing on key metrics like application latency and throughput. It emphasizes best practices for hardware, network, and storage configuration to enhance performance while minimizing costs. The author encourages skepticism towards synthetic workloads and highlights the importance of real-world testing to ensure accurate performance assessment.Azug - successfully breeding rabits
Azug - successfully breeding rabitsYves Goeleven
?
This document discusses successfully breeding rabbits in the cloud. It describes rabbits as small public apps that want to live outside and scale quickly. It discusses how the cloud provides rabbits with global reach, fast time to market, performance, scalability, availability and cost efficiency. The document emphasizes that the weakest link limits overall scalability and that caches, queues and separating concerns can help address this. It stresses understanding cloud capacity and efficiently using resources to keep costs low. Finally, it discusses how rabbits can achieve self-service, marketing, support, education and testing at scale.Scalling Rails: The Journey to 200M Notifications
Scalling Rails: The Journey to 200M NotificationsGustavo Araujo
?
Talk presented at Tropical On Rails.DevOps, CLI, APIs, Oh My! Security Gone Agile
DevOps, CLI, APIs, Oh My! Security Gone AgileMatt Tesauro
?
The document discusses the integration of DevOps practices with security measures in a rapidly changing software development landscape. It emphasizes the need for automated testing, continuous integration, and collaboration between development and operations teams to secure applications, infrastructure, and code. Key takeaways include the importance of automation, workflow standardization, and adapting to ongoing changes in software methodologies.Load testing and performance tracing
Load testing and performance tracingHans H?chtl
?
The document discusses load testing and performance optimization techniques for applications, emphasizing the importance of profiling, identifying bottlenecks, and using tools like Apache JMeter and BlazeMeter. It provides insights into optimizing PHP applications, utilizing various tools for memory and performance profiling, and improving database and frontend performance. The author, Hans H?chtl, also shares contact information and additional resources for further reading.Ensuring Performance in a Fast-Paced Environment (CMG 2014)
Ensuring Performance in a Fast-Paced Environment (CMG 2014)Martin Spier
?
The document presents an overview of performance engineering practices at Netflix, emphasizing their culture of freedom and responsibility, independent development teams, and agile methodologies. Key strategies include auto-scaling, canary analysis, and large-scale performance testing to ensure high availability and reliability in their streaming service. Additionally, it highlights the use of advanced tools like automated canary analysis to optimize deployment and performance metrics monitoring.Infrastructure Considerations : Design : "webops"
Infrastructure Considerations : Design : "webops"Piyush Kumar
?
This document provides recommendations for preparing infrastructure for the future. It suggests choosing a long-term supported Linux distribution like RHEL, CentOS, or Ubuntu LTS. Infrastructure should be designed to be scalable, robust, manageable, resilient, and cost-effective. Services should be split across redundant servers and high availability/failover implemented. Automation, configuration management, and continuous integration/deployment are also recommended. Comprehensive monitoring of infrastructure and applications is important for capacity planning and issue detection. The document provides additional tips regarding caching, security, change management, backups, and ongoing re-architecting.3.2 Streaming and Messaging
3.2 Streaming and MessagingÕñ¶« Áõ
?
The document presents a discussion on real-time statistics leveraging Apache Storm and RocketMQ, highlighting their integration within the streaming ecosystem and best practices. It covers challenges in stateful statistics, event counting, and performance tuning related to distributed systems. Additionally, it provides insights into using RocketMQ with various streaming frameworks, alongside suggestions for improving performance and handling common issues.Introduction openstack-meetup-nov-28
Introduction openstack-meetup-nov-28Sadique Puthen
?
Red Hat Openstack and Ceph Meetup, Pune | 28th NOV 2015
Sadique Puthen, Principal Technical Support Engineer at Red Hat, Inc., gave an introduction to Red Hat Openstack (RDO) and its components. He discussed how Openstack provides infrastructure services like compute (Nova), storage (Cinder, Swift), networking (Neutron), and database (Trove) as a service. He also covered Openstack deployment options like Packstack, TripleO, and Ironic for bare metal provisioning. The meetup aimed to introduce Openstack components and services and their role in providing infrastructure as a service through a cloud platform.Sanger OpenStack presentation March 2017
Sanger OpenStack presentation March 2017Dave Holland
?
The Sanger Institute's transition to OpenStack aimed to enhance data handling and computing efficiency, while addressing shortcomings of their previous LSF system. They outlined their journey from initial training to production systems, discussed hardware choices, and detailed numerous challenges encountered, including networking and deployment issues. Future directions emphasize expanding Ceph storage capabilities and exploring advanced OpenStack features like bare-metal deployment and instance pricing.A real-life account of moving 100% to a public cloud
A real-life account of moving 100% to a public cloudJulien SIMON
?
This document summarizes Viadeo's experience moving their entire infrastructure to AWS. It describes their motivation for moving to the cloud to focus on their product instead of managing hardware. It outlines their step-by-step process, including automating infrastructure with CloudFormation, baking AMIs, and continuous integration/delivery. It discusses lessons learned around networking, databases, and involving stakeholders in the transition. Currently, their staging and production environments run in AWS across multiple regions, with future plans to migrate more services.introduction to node.js
introduction to node.jsorkaplan
?
This document provides an introduction to Node.js, a framework for building scalable server-side applications with asynchronous JavaScript. It discusses what Node.js is, how it uses non-blocking I/O and events to avoid wasting CPU cycles, and how external Node modules help create a full JavaScript stack. Examples are given of using Node modules like Express for building RESTful APIs and Socket.IO for implementing real-time features like chat. Best practices, limitations, debugging techniques and references are also covered.PipelineAI + AWS SageMaker + Distributed TensorFlow + AI Model Training and S...
PipelineAI + AWS SageMaker + Distributed TensorFlow + AI Model Training and S...Chris Fregly
?
Pipeline.AI is a platform for deploying and optimizing machine learning models at scale. It allows users to package models with their runtime dependencies, perform load testing and optimizations, deploy models to production safely using techniques like canary deployments, and monitor models both offline and online. The platform aims to enable live, continuous model training directly in production environments.Optimizing, Profiling, and Deploying High Performance Spark ML and TensorFlow AI
Optimizing, Profiling, and Deploying High Performance Spark ML and TensorFlow AIData Con LA
?
Pipeline.ai specializes in high-performance model training and serving using GPUs and various cloud platforms. The platform focuses on model serving to provide continuous insights in live production environments, enabling real-time predictions and optimizations. It supports deploying models with a strong emphasis on maintaining runtime consistency across various stages of model deployment.Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And...
Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And...Lightbend
?
The document discusses the architecture and benefits of using Akka Streams and Kafka for efficient web crawling. It highlights the reactive manifesto principles of responsiveness, resilience, and elasticity while detailing the system's design for handling high-volume URL processing. Key features include asynchronous processing, back-pressure management, and modular components to ensure scalability and performance without compromising resource efficiency.Recently uploaded (20)
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdfcaoyixuan2019
?
A presentation at Internetware 2025.AI VIDEO MAGAZINE - June 2025 - r/aivideo
AI VIDEO MAGAZINE - June 2025 - r/aivideo1pcity Studios, Inc
?
AI VIDEO MAGAZINE - r/aivideo community newsletter ¨C Exclusive Tutorials: How to make an AI VIDEO from scratch, PLUS: How to make AI MUSIC, Hottest ai videos of 2025, Exclusive Interviews, New Tools, Previews, and MORE - JUNE 2025 ISSUE -Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Coordinated Disclosure for ML - What's Different and What's the Same.pdfPriyanka Aash
?
Coordinated Disclosure for ML - What's Different and What's the SameUsing the SQLExecutor for Data Quality Management: aka One man's love for the...
Using the SQLExecutor for Data Quality Management: aka One man's love for the...Safe Software
?
The SQLExecutor is one of FME¡¯s most powerful and flexible transformers. Pivvot maintains a robust internal metadata hierarchy used to support ingestion and curation of thousands of external data sources that must be managed for quality before entering our platform. By using the SQLExecutor, Pivvot can efficiently detect problems and perform analysis before data is extracted from our staging environment, removing the need for rollbacks or cycles waisted on a failed job. This presentation will walk through three distinct examples of how Pivvot uses the SQLExecutor to engage its metadata hierarchy and integrate with its Data Quality Management workflows efficiently and within the source postgres database. Spatial Validation ¨CValidating spatial prerequisites before entering a production environment. Reference Data Validation - Dynamically validate domain-ed columns across any table and multiple columns per table. Practical De-duplication - Removing identical or near-identical well point locations from two distinct source datasets in the same table.¡°MPU+: A Transformative Solution for Next-Gen AI at the Edge,¡± a Presentation...
¡°MPU+: A Transformative Solution for Next-Gen AI at the Edge,¡± a Presentation...Edge AI and Vision Alliance
?
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2025/06/mpu-a-transformative-solution-for-next-gen-ai-at-the-edge-a-presentation-from-fotonation/
Petronel Bigioi, CEO of FotoNation, presents the ¡°MPU+: A Transformative Solution for Next-Gen AI at the Edge¡± tutorial at the May 2025 Embedded Vision Summit.
In this talk, Bigioi introduces MPU+, a novel programmable, customizable low-power platform for real-time, localized intelligence at the edge. The platform includes an AI-augmented image signal processor that enables leading image and video quality.
In addition, it integrates ultra-low-power object and motion detection capabilities to enable always-on computer vision. A programmable neural processor provides flexibility to efficiently implement new neural networks. And additional specialized engines facilitate image stabilization and audio enhancements.WebdriverIO & JavaScript: The Perfect Duo for Web Automation
WebdriverIO & JavaScript: The Perfect Duo for Web Automationdigitaljignect
?
In today¡¯s dynamic digital landscape, ensuring the quality and dependability of web applications is essential. While Selenium has been a longstanding solution for automating browser tasks, the integration of WebdriverIO (WDIO) with Selenium and JavaScript marks a significant advancement in automation testing. WDIO enhances the testing process by offering a robust interface that improves test creation, execution, and management. This amalgamation capitalizes on the strengths of both tools, leveraging Selenium¡¯s broad browser support and WDIO¡¯s modern, efficient approach to test automation. As automation testing becomes increasingly vital for faster development cycles and superior software releases, WDIO emerges as a versatile framework, particularly potent when paired with JavaScript, making it a preferred choice for contemporary testing teams.OpenPOWER Foundation & Open-Source Core Innovations
OpenPOWER Foundation & Open-Source Core InnovationsIBM
?
penPOWER offers a fully open, royalty-free CPU architecture for custom chip design.
It enables both lightweight FPGA cores (like Microwatt) and high-performance processors (like POWER10).
Developers have full access to source code, specs, and tools for end-to-end chip creation.
It supports AI, HPC, cloud, and embedded workloads with proven performance.
Backed by a global community, it fosters innovation, education, and collaboration."Database isolation: how we deal with hundreds of direct connections to the d...
"Database isolation: how we deal with hundreds of direct connections to the d...Fwdays
?
What can go wrong if you allow each service to access the database directly? In a startup, this seems like a quick and easy solution, but as the system scales, problems appear that no one could have guessed.
In my talk, I'll share Solidgate's experience in transforming its architecture: from the chaos of direct connections to a service-based data access model. I will talk about the transition stages, bottlenecks, and how isolation affected infrastructure support. I will honestly show what worked and what didn't. In short, we will analyze the controversy of this talk.UserCon Belgium: Honey, VMware increased my bill
UserCon Belgium: Honey, VMware increased my billstijn40
?
VMware¡¯s pricing changes have forced organizations to rethink their datacenter cost management strategies. While FinOps is commonly associated with cloud environments, the FinOps Foundation has recently expanded its framework to include Scopes¡ªand Datacenter is now officially part of the equation. In this session, we¡¯ll map the FinOps Framework to a VMware-based datacenter, focusing on cost visibility, optimization, and automation. You¡¯ll learn how to track costs more effectively, rightsize workloads, optimize licensing, and drive efficiency¡ªall without migrating to the cloud. We¡¯ll also explore how to align IT teams, finance, and leadership around cost-aware decision-making for on-prem environments. If your VMware bill keeps increasing and you need a new approach to cost management, this session is for you!The Future of Technology: 2025-2125 by Saikat Basu.pdf
The Future of Technology: 2025-2125 by Saikat Basu.pdfSaikat Basu
?
A peek into the next 100 years of technology. From Generative AI to Global AI networks to Martian Colonisation to Interstellar exploration to Industrial Nanotechnology to Artificial Consciousness, this is a journey you don't want to miss. Which ones excite you the most? Which ones are you apprehensive about? Feel free to comment! Let the conversation begin!Connecting Data and Intelligence: The Role of FME in Machine Learning
Connecting Data and Intelligence: The Role of FME in Machine LearningSafe Software
?
In this presentation, we want to explore powerful data integration and preparation for Machine Learning. FME is known for its ability to manipulate and transform geospatial data, connecting diverse data sources into efficient and automated workflows. By integrating FME with Machine Learning techniques, it is possible to transform raw data into valuable insights faster and more accurately, enabling intelligent analysis and data-driven decision making.Quantum AI: Where Impossible Becomes Probable
Quantum AI: Where Impossible Becomes ProbableSaikat Basu
?
Imagine combining the "brains" of Artificial Intelligence (AI) with the "super muscles" of Quantum Computing. That's Quantum AI!
It's a new field that uses the mind-bending rules of quantum physics to make AI even more powerful.Cracking the Code - Unveiling Synergies Between Open Source Security and AI.pdf
Cracking the Code - Unveiling Synergies Between Open Source Security and AI.pdfPriyanka Aash
?
Cracking the Code - Unveiling Synergies Between Open Source Security and AIOh, the Possibilities - Balancing Innovation and Risk with Generative AI.pdf
Oh, the Possibilities - Balancing Innovation and Risk with Generative AI.pdfPriyanka Aash
?
Oh, the Possibilities - Balancing Innovation and Risk with Generative AIGenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdf
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdfPriyanka Aash
?
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now¡°MPU+: A Transformative Solution for Next-Gen AI at the Edge,¡± a Presentation...
¡°MPU+: A Transformative Solution for Next-Gen AI at the Edge,¡± a Presentation...Edge AI and Vision Alliance
?
Ad
Quilt - Distributed Load Simulation from AWS
- 1. Quilt distributed load test tooling and infrastructure
- 2. About me ? Engineering Manager (Performance and Resiliency Test) @ BlueJeans ? had seen some of the worst outages! ? Had simulated some real failures! ? Now we have 99.95% availability! ? Have team of 10 Engineers ? Interests ? Java, Ruby, Zk, Cassandra, Couchbase, RMQ, Jenkins, CI/CD ? AWS, RS, Google compute engine ? Design and develop tools ? Simulate failures ? Develop android apps! ? Startups ? ~15 years ? Developer, QA, Solution Architect, Devops ? 3 years @bluejeans ? 6.5 years @apigee (xml parsing engine to 4G gateway) ? 2 years @ Bea Systems ? 3 years as owner, dev, customer management etc
- 3. Outages
- 4. Why Outages ? Performance issues ? Natural load increase ¨C poor software design, poor sizing, mc@bjn ? Load spikes ¨C Xiaomi@fk, sip-dos-attack@bjn ? Cascading failures ? Hardware failure ? Network switch failure ? Software crash ? 3rd party app failure ¨C CB, ZK, RMQ, C* ? Memory leak ? Network blips/outage ? between the datacenters ? Inside the datacenter!
- 5. What can we do ? ? Performance ? Load simulation ? Monitoring systems ? Comparing against benchmarks ? Size the environment ? Resilience ? Simulate failures ? Evaluate the impact AUTOMATE, AUTOMATE, AUTOMATE!!!
- 6. Load simulation - challenge ? Large-scale heterogeneous distributed system ? Real-time video/audio mixing ? Heavy RTP data transfer ? Multiple protocols ¨C sip, h323, webrtc, hls, http, websocket ? Geo-located (5-6 regions) & partitions ? AWS + Datacenter hosting ¨C Scaling needs advance notice for DC! ? Inter-region zookeeper lookup
- 7. Environment Say, 20,000 concurrent EPs at peak load
- 8. It¡¯s complex! Say, 20,000 concurrent EPs at peak load 1 EP (with the least quality) 150 kbps for video 80 kbps for audio 20,000 EPs Rx ¨C 230 * 20K = 4 gbps Tx = 4 gbps *apigee office network is 40 mbps CPU 1 m3.xlarge supports 30 EP simulators ~700 m3.xlarge instances for 20K We generate terabytes of logs for each run!
- 9. EP simulators ? Sipp ¨C open source ? Callgen ¨C in-house developed h323 simulator ? Webrtc ¨C runs on headless browser with selenium ? Proprietary (mobile and thick clients) ¨C in-house developed simulator
- 10. The tool ? Controller ? Distribution logic ¨C EP types, meeting ids, client instances ? Ruby ? Clients ? Simulate EPs ? Http api calls - Ruby ? Websocket ¨C sockjs on node.js ? RPC using DRuby
- 11. The tool
- 12. Cost! ? The test is in-evitable, no major releases go without that ? Solution - optimal usage of aws instances ? All automated ? Bring up instances when needed ? Setup ¨C checkout and copy latest builds ¡°concurrently¡± of the tool to the instances ? Run the tests with real time monitoring ? Copy the logs to S3 ¡°concurrently¡± ? Bring down the instances ? Analyze, debug etc ¨C offline activity
- 13. The setup ? All automated ? Bring up instances when needed ? Setup ¨C checkout latest build of the tool to the instances ? Run the tests with real time monitoring ? Bring down the instances ? Analyze, debug etc ¨C offline activity
- 14. Our tool stack ? Quilt ¨C Setup Infra and simulate distributed load. We just talked about it ? Analyzer ¨C post-test analysis by collecting metrics from various sources such as sensu, atop, new-relic. Graphs generated with high-charts. ? Scoreboard ¨C Real time monitoring ? Catapult ¨C UI around Quilt to enable Devs to do the test ? Goblin (being open-sourced and presented in root-conf) ¨C resiliency testing framework and utils ? Scout ¨C the agent who resides in the RMZ of system under test ? Rain ¨C the new load generation framework in node.js for testing Bluejeans Primetime. Scales to 100s of thousands!
- 18. Key takeaways ? Peaceful sleep ¨C No way out, need load and resiliency testing! ? Automate - Design the Tools & Infra properly ? Scale ¨C Generating more load is just adding more aws instances ? Extensibility ¨C adding a new endpoint type is quick ? Automated analysis and reporting ¨C sensu, atop, new-relic ? Vendor agnostic - aws/rackspace/GCE ? Cost optimization - Use sdk to dynamically launch instances only when needed ¨C save cost
- 19. Thank you Stay connected : Ajith Jose BlueJeans Network https://in.linkedin.com/in/ajithvj