Elasticsearch in Production

May 8, 20141 like1,575 views

This document summarizes best practices for running Elasticsearch in production environments. It covers anti-patterns to avoid like arbitrary keys and heavy updating. It discusses resource usage and distributed problems like memory usage, out of memory errors, and network glitches. It also provides guidance on security practices, client concerns, and strategies for changing clusters.

Elasticsearch in production
!
Alex Brasetvik
alex@found.no
@alexbrasetvik

Who?
Co-founder of Found AS
8+ years search, 3+ Elasticsearch
Herding hundreds of Elasticsearch clusters

Agenda
? Anti-patterns
? Memory / Resource Usage
? Distributed problems
? Security
? Client concerns
? Changing a cluster

found.no/foundation
Elasticsearch in Production Elasticsearch as a NoSQL Database
Intro to Function Scoring All About Analyzers
Securing your Elasticsearch Cluster

Snapshot / Restore
Circuit breakers
Document values
Aggregations
Distributed percolation
Suggesters
��

Arbitrary Keys
? ��Schema Free��
? One ?eld per value
? Ever-growing cluster state
acls:
1234: READ
42: WRITE

Heavy Updating
? Update = Delete + Reindex
? Be careful with counters

$Slow queries ? WHERE foo ILIKE ��%bar%�� ? {��query_string��: {��query��: ��foo:*bar*��}}$

Arbitrary searches
query:
filtered:
filter:
term:
user_id: 42
query:
[user��s query here]

Memory
? Field caches
? Filter caches
? Page caches
? Aggregations
? Index building

Page Cache
? Keeping index pages in memory
? Can��t have too much
? Outgrow: Gradual slowdown

Heap Space
? Memory used by Elasticsearch process
? Field / Filter caches
? Aggregations

OutOfMemoryError
Woah there
I ate all the memories
Your cluster may or may not work any more

OutOfMemory
? Growing too big
? Selecting too big timespan in Kibana
? Document ingestion peak

Preventing OOMs
? Have enough memory :-)
? Understand your search��s memory pro?le
? Bulk / Circuit breaker settings
? Monitoring
? Document values

$"my_field": { "type": "string", "fielddata": { "format": "doc_values" } }$

Sizing
? Test, don��t guess
? Start big, scale down
? Index, search, monitor

? Tie-breaker can be a cheap master-node
? Applies to data centers / availability zones too

Jepsen
? Kyle Kingsbury��s series on distributed systems
? Distributed systems are hard
? aphyr.com

Security
? ��Not my job!�� C Elasticsearch
? That��s ?ne!

Dynamic Scripts
!
? Scoring
? Aggregations
? Updating

Dynamic Scripts
Runtime.getRuntime().exec(��)

Dynamic Scripts
Runtime.getRuntime().exec(��)
<script src=/slideshow/elasticsearch-in-production-34431652/34431652/��http:/127.0.0.1:9200/_search?callback=capture&��

Security
!
? Disable dynamic scripts
? Mind index patterns
? Even then, don��t accept arbitrary requests

Client Concerns
? Connection pools
? Idempotent requests
? Have sane syncing/indexing strategies

Cluster changes
? Make new nodes join existing cluster
? No rolling restarts
? Easy rollback if things go bad

Cluster changes
? Test ?rst
? Mind recover_*-settings

Multi-Cluster Work?ows
? Snapshot/Restore
? Operations across clusters
? Swap clusters!
? Works well with good syncing strategy

Misc
? Same JVM
? ulimits
? Unicast
? SSD? noop-scheduler

Video available at http://www.youtube.com/watch?v=gkdfNl0WL-A Original slides at http://presentations.found.no/berlin-buzzwords-2013/ This talk covers some of the lessons we've learned from securing and herding hundreds of Elasticsearch clusters. It is applicable whether you operate Elasticsearch in your own infrastructure, in the cloud, or if you're a developer who wants a better understanding of Elasticsearch's various failure modes. Elasticsearch easily lets you develop amazing things, and it has gone to great lengths to make Lucene's features readily available in a distributed setting. However, when it comes to running Elasticsearch in production, you still have a fairly complicated system on your hands: a system with high expectations on network stability, a huge appetite for memory, and a system that assumes all users are trustworthy. Instead of delving deeply into a few specifics, we give a brief overview of problems you are likely to run into and suggested solutions to these problems. We cover topics that are applicable to both developers and users with Elasticsearch clusters of every shape and size �C with an emphasis on resiliency and security. Basic familiarity with Elasticsearch is assumed.

Elasticsearch in Production (London version)foundsearch

Elasticsearch in production Boston Meetup October 2014beiske

Elasticsearch easily lets you develop amazing things, and it has gone to great lengths to make Lucene's features readily available in a distributed setting. However, when it comes to running Elasticsearch in production, you still have a fairly complicated system on your hands: a system with high demands on network stability, a huge appetite for memory, and a system that assumes all users are trustworthy. This talk will cover some of the lessons we've learned from securing and herding hundreds of Elasticsearch clusters.

Cross-Cluster and Cross-Datacenter Elasticsearch Replication at sahibinden.comErtu? Karamatl?

This document discusses replicating Elasticsearch clusters across multiple datacenters at sahibinden.com to prevent downtime, enable backups and testing, and provide low latency access. It describes three options for replication: synchronous writes, asynchronous writes, and a hybrid approach. Sahibinden.com uses Apache Kafka and Zookeeper for asynchronous cross-cluster and cross-datacenter replication between four Elasticsearch clusters in two datacenters. Performance is maintained using multiple replication threads. A sync script coordinates the replication process.

Elastic Stack RoadmapImma Valls Bernaus

chOpaal -- Senior Project Presentationasimfayaz

This document discusses strategies for building a scalable social networking engine. It describes previous approaches that failed, such as polling databases, SQL query caching, and SQL table caching. These caused performance issues as user and data size grew. The document then introduces Memcached as a high-performance distributed caching system that will allow the social network to scale in a write-through, non-blocking manner. It emphasizes that Memcached uses memory caching which is faster than database lookups and helps optimize the architecture.

An��lisis del roadmap del Elastic StackElasticsearch

Using ElasticSearch as a fast, flexible, and scalable solution to search occu...kristgen

Elasticsearch is an open source search engine that provides fast, flexible, and scalable search of occurrence records and checklists. It allows adding and querying data through a REST API or Java API. Data can be imported from databases or other sources using rivers. Mappings customize indexing and querying. Elasticsearch has been used at Canadensys to index vascular plant names with filters for autocompletion, genus filtering, and epithet hierarchy. It is also used at GBIF France to search biodiversity data from MongoDB with filters and calculate statistics with facets.

Small intro to Big DataSoftwareMill

Pig, Hive, Flink, Kafka, Zeppelin... if you now wonder if someone just tried to offend you or are those just Pokemon names, then this talk is just for you! Big Data is everywhere and new tools for it are released almost at the speed of new JavaScript frameworks. During this entry level presentation I will walk though the challenges which Big Data presents, reflect how big is big and introduce currently most fancy and popular (mostly open source) tools. I'll try to spark off interest in Big Data by showing application areas and by throwing ideas where you can later dive into.

Cassandra - how to fail?SoftwareMill

Elatic{on}'16 recapShravanthi Denthumdas

A Developer Overview of RedisYuriy Guts

Redis for .NET DevelopersYuriy Guts

Redis is an open-source, in-memory data structure store that allows for atomic operations on data structures like strings, hashes, lists, sets, sorted sets, and bitmaps. It supports common operations like push, pop, and get and can be used as a cache, for session storage, or to compute statistics. StackExchange uses Redis with two servers each with 96GB RAM handling over 60k requests per second on average.

Cassandra Day SV 2014: Beyond Read-Modify-Write with Apache CassandraDataStax Academy

This document discusses strategies for updating data in Apache Cassandra beyond using read-modify-write operations. It describes how eventual consistency allows safe updates without locking by propagating changes asynchronously. It also covers Cassandra features like collections, lightweight transactions, and content-addressable storage that provide flexible data models for modern web-scale applications while avoiding the need for read-modify-write in many cases.

High Performance Mysqlliufabin 66688

The document discusses various topics related to optimizing performance in MySQL databases, including: 1. Choosing optimal data types can improve performance by reducing storage size and simplifying queries. Smaller and simpler types are generally better. 2. Indexing is important for performance. The document discusses different index types like B-Tree indexes and their uses. Indexes should be chosen based on the types of queries. 3. Benchmarking and profiling are important to identify bottlenecks. Benchmarking tests overall performance while profiling identifies specific costly operations. The results can help optimize the database, queries, or hardware configuration.

Configuring elasticsearch for performance and scaleBharvi Dixit

Why Elastic? @?50th Vinitaly 2016Christoph Wurm

Elastic Stack IntroductionVikram Shinde

Leveraging chaos mesh in Astra Serverless testingPierre Laporte

Azure Large Scale Deployments - Tales from the TrenchesAaron Saikovski

The document discusses best practices for deploying large scale Azure environments based on lessons learned. It recommends using subscriptions per environment, tagging resources for organization and cost management, planning storage accounts and networking carefully, automating deployments through ARM templates and tools, and monitoring deployments through OMS with alerts for issues. Key tips include using Azure AD accounts, billing alerts, automated shutdown of untagged resources, encryption on provisioning, unique storage account prefixes, avoiding NSGs at NIC level, updated tools, resource locks, and awareness of OMS indexing intervals.

My First 90 days with VitessMorgan Tocker

This document summarizes the author's first 90 days of experience with Vitess, an open source database proxy. It provides an overview of Vitess, including that it sits between applications and MySQL to provide routing, query consolidation, and other features. It also discusses Vitess terminology, questions about MySQL compatibility, consistency models, and other quirks and features. The document concludes with a discussion of the best use cases for Vitess and areas where it could be improved.

How to run Elasticsearch on Azure in just a few minutesChristoph Wurm

Collo -02 , en?? ?

The document provides instructions for setting up a sample that moves data from Redis to MySQL using the Collo library. It involves 6 steps: 1) downloading samples, 2) setting up a database and table in MySQL, 3) inserting sample data into Redis, 4) configuring repository settings, 5) setting up a job to read from Redis and write to MySQL, and 6) running Collo to test the job. The document also suggests making a new sample job for moving data from HTTP posts to MySQL as a next step.

NOSQL - not only sqlSergey Shishkin

Building a Dev/Test Cloud with Apache CloudStackke4qqq

RedisDiego Pacheco

Redis is an open-source, in-memory data structure store used for caching, session storage, queues, and analytics. It supports common data structures like strings, hashes, and sets. Redis commands operate on these data structures, providing features like keys with time-to-live and user-defined data structures through modules. Redis 5.x introduced modules to extend Redis capabilities with new data structures in a low-latency manner. The document discusses using Redis from Java applications through libraries like Lettuce and Spring Data, and provides exercises to practice Redis commands and build services using Redis data structures.

PENXY - Redis in Azuremourhoon

Living with SQL and NoSQL at craigslist, a Pragmatic ApproachJeremy Zawodny

From the 2012 Percona Live MySQL Conference in Santa Clara, CA. Craigslist uses a variety of data storage systems in its backend systems: in-memory, SQL, and NoSQL. This talk is an overview of how craigslist works with a focus on the data storage and management choices that were made in each of its major subsystems. These include MySQL, memcached, Redis, MongoDB, Sphinx, and the filesystem. Special attention will be paid to the benefits and tradeoffs associated with choosing from the various popular data storage systems, including long-term viability, support, and ease of integration.

Elasticsearch in production New York Meetup at Twitter October 2014beiske

Modernizing WordPress Search with ElasticsearchTaylor Lovett

More Related Content

What's hot (20)

Small intro to Big DataSoftwareMill

Cassandra - how to fail?SoftwareMill

Elatic{on}'16 recapShravanthi Denthumdas

A Developer Overview of RedisYuriy Guts

Redis for .NET DevelopersYuriy Guts

Cassandra Day SV 2014: Beyond Read-Modify-Write with Apache CassandraDataStax Academy

High Performance Mysqlliufabin 66688

Configuring elasticsearch for performance and scaleBharvi Dixit

Why Elastic? @?50th Vinitaly 2016Christoph Wurm

Elastic Stack IntroductionVikram Shinde

Leveraging chaos mesh in Astra Serverless testingPierre Laporte

Azure Large Scale Deployments - Tales from the TrenchesAaron Saikovski

My First 90 days with VitessMorgan Tocker

How to run Elasticsearch on Azure in just a few minutesChristoph Wurm

Collo -02 , en?? ?

NOSQL - not only sqlSergey Shishkin

Building a Dev/Test Cloud with Apache CloudStackke4qqq

RedisDiego Pacheco

PENXY - Redis in Azuremourhoon

Living with SQL and NoSQL at craigslist, a Pragmatic ApproachJeremy Zawodny

Small intro to Big DataSoftwareMill

Cassandra - how to fail?SoftwareMill

Elatic{on}'16 recapShravanthi Denthumdas

A Developer Overview of RedisYuriy Guts

Redis for .NET DevelopersYuriy Guts

Cassandra Day SV 2014: Beyond Read-Modify-Write with Apache CassandraDataStax Academy

High Performance Mysqlliufabin 66688

Configuring elasticsearch for performance and scaleBharvi Dixit

Why Elastic? @?50th Vinitaly 2016Christoph Wurm

Elastic Stack IntroductionVikram Shinde

Leveraging chaos mesh in Astra Serverless testingPierre Laporte

Azure Large Scale Deployments - Tales from the TrenchesAaron Saikovski

My First 90 days with VitessMorgan Tocker

How to run Elasticsearch on Azure in just a few minutesChristoph Wurm

Collo -02 , en?? ?

NOSQL - not only sqlSergey Shishkin

Building a Dev/Test Cloud with Apache CloudStackke4qqq

RedisDiego Pacheco

PENXY - Redis in Azuremourhoon

Living with SQL and NoSQL at craigslist, a Pragmatic ApproachJeremy Zawodny

Similar to Elasticsearch in Production (20)

Elasticsearch in production New York Meetup at Twitter October 2014beiske

Modernizing WordPress Search with ElasticsearchTaylor Lovett

Log Analytics with Amazon Elasticsearch Service - September Webinar SeriesAmazon Web Services

Elasticsearch is a popular open-source search and analytics engine used for log analytics. With Amazon Elasticsearch Service, you can easily run Elasticsearch on AWS. In this webinar, we will provide an overview of Amazon Elasticsearch Service and demo how to set up and configure an Amazon Elasticsearch domain for the log analytics use case. Learning Objectives: '- Understand Amazon Elasticsearch Service use cases and key features - Learn how to secure your Amazon Elasticsearch cluster for access from Kibana and other plug-ins - Learn best practices for scaling, monitoring, and troubleshooting Amazon Elasticsearch domains

Microservices, Continuous Delivery, and Elasticsearch at Capital OneNoriaki Tatsumi

ElasticSearch: Distributed Multitenant NoSQL Datastore and Search EngineDaniel N

Elk presentation1#3uzzal basak

The document describes the ELK stack, which consists of three open source projects - Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a data processing pipeline that ingests data from multiple sources and sends it to Elasticsearch. Kibana lets users visualize data from Elasticsearch with charts and graphs. Beats ship data from their sources to Logstash or Elasticsearch. The Elastic Stack is the evolved version of the ELK stack.

Elasticsearch Introduction at BigData meetupEric Rodriguez (Hiring in Lex)

Real time analytics using Hadoop and ElasticsearchAbhishek Andhavarapu

This document discusses using Hadoop and Elasticsearch for real-time analytics. It provides an overview of Elasticsearch, including how it is document-oriented, schema-free, distributed and fast. It also demonstrates indexing, retrieving, updating and deleting documents from Elasticsearch. The demo portion involves extracting data from a SQL database using Hive, transforming it with Hadoop/Hive, and loading it into Elasticsearch to run queries. Lessons learned focus on concurrency, filtering, field data caching and JVM memory usage.

Making (Almost) Any Database Faster and Cheaper with CachingAmazon Web Services

Redis is an in-memory database that can be used for caching to improve database performance. Amazon ElastiCache provides a fully managed Redis service on AWS. Using ElastiCache for caching provides benefits like 34% greater throughput, automatic operations management, high availability, and reliability compared to self-managed Redis. ElastiCache supports Redis data types and clustering to enable horizontal scaling for large datasets and high throughput workloads.

Getting started with Laravel & ElasticsearchPeter Steenbergen

Cassandraexsuns

Dev nexus 2017Roy Russo

Elasticsearch is a distributed, RESTful search and analytics engine that can be used for processing big data with Apache Spark. It allows ingesting large volumes of data in near real-time for search, analytics, and machine learning applications like feature generation. Elasticsearch is schema-free, supports dynamic queries, and integrates with Spark, making it a good fit for ingesting streaming data from Spark jobs. It must be deployed with consideration for fast reads, writes, and dynamic querying to support large-scale predictive analytics workloads.

AWS October Webinar Series - Introducing Amazon Elasticsearch ServiceAmazon Web Services

Running Elasticsearch often requires specialized expertise and significant resources to operate and manage infrastructure and Elasticsearch software. Amazon Elasticsearch Service makes it easy to deploy, operate, and scale Elasticsearch in AWS. In this webinar, we will walk through how to launch a fully functional Amazon Elasticsearch domain, load your data, and analyze it using the built-in Kibana integration. We will also cover the CloudWatch Logs integration, which enables you to have your log data, such as VPC logs, automatically loaded into your Amazon Elasticsearch domain for analysis and exploration.

(BDT209) Launch: Amazon Elasticsearch For Real-Time Data AnalyticsAmazon Web Services

Organizations are collecting an ever-increasing amount of data from numerous sources such as log systems, click streams, and connected devices. Launched in 2009, Elasticsearch ��an open-source analytics and search engine�� has emerged as a popular tool for real-time analytics and visualization of data. Some of the most common use cases include risk assessment, error detection, and sentiment analysis. However, as data volumes and applications grow, managing Elasticsearch clusters can consume significant IT resources while adding little or no differentiated value to the organization. Amazon Elasticsearch Service (Amazon ES) is a managed service that makes it easy to deploy, operate, and scale Elasticsearch clusters in the AWS Cloud. Amazon ES offers the benefits of a managed service, including cluster provisioning, easy configuration, replication for high availability, scaling options, data durability, security, and node monitoring. This session presents a technical deep dive on Amazon ES. Attendees learn: Common challenges with real-time data analytics and visualization and how to address them; the benefits, reference architecture, and best practices for using Amazon ES; and data ingestion options with Amazon DynamoDB, AWS Lambda, and Amazon Kinesis.

Devnexus 2018Roy Russo

Elasticsearch is a distributed, RESTful search and analytics engine that can be used for processing big data with Apache Spark. Data is ingested from Spark into Elasticsearch for features generation and predictive modeling. Elasticsearch allows for fast reads and writes of large volumes of time-series and other data through its use of inverted indexes and dynamic mapping. It is deployed on AWS for its elastic scalability, high availability, and integration with Spark via fast queries. Ongoing maintenance includes archiving old data, partitioning indices, and reindexing large datasets.

NoSQL - Cassandra & MongoDB.pptxNaveen Kumar

This document provides an overview of NoSQL databases Cassandra and MongoDB. It begins with an introduction to RDBMS and discusses the need for NoSQL databases in terms of handling big data. Key concepts covered include the CAP theorem, data models of Cassandra and MongoDB, replication, and automatic failover. The document concludes by emphasizing the usefulness of NoSQL for availability and processing unstructured data at scale.

Using SparkML to Power a DSaaS (Data Science as a Service): Spark Summit East...Spark Summit

The document discusses Sparkle, a solution built by Comcast to address challenges in processing massive amounts of data and enabling data science workflows at scale. Sparkle is a centralized processing system with SQL and machine learning capabilities that is highly scalable and accessible via a REST API. It is used by Comcast to power various use cases including churn modeling, price elasticity analysis, and direct mail campaign optimization.

ElasticSearch AJUG 2013Roy Russo

This document provides an introduction to Elasticsearch, covering the basics, concepts, data structure, inverted index, REST API, bulk API, percolator, Java integration, and topics not covered. It discusses how Elasticsearch is a document-oriented search engine that allows indexing and searching of JSON documents without schemas. Documents are distributed across shards and replicas for horizontal scaling and high availability. The REST API and query DSL allow full-text search and filtering of documents.

How ElasticSearch lives in my DevOps life��

ElasticSearch is a flexible and powerful open source, distributed real-time search and analytics engine for the cloud. It is JSON-oriented, uses a RESTful API, and has a schema-free design. Logstash is a tool for collecting, parsing, and storing logs and events in ElasticSearch for later use and analysis. It has many input, filter, and output plugins to collect data from various sources, parse it, and send it to destinations like ElasticSearch. Kibana works with ElasticSearch to visualize and explore stored logs and data.

Elasticsearch in production New York Meetup at Twitter October 2014beiske

Modernizing WordPress Search with ElasticsearchTaylor Lovett

Log Analytics with Amazon Elasticsearch Service - September Webinar SeriesAmazon Web Services

Microservices, Continuous Delivery, and Elasticsearch at Capital OneNoriaki Tatsumi

ElasticSearch: Distributed Multitenant NoSQL Datastore and Search EngineDaniel N

Elk presentation1#3uzzal basak

Elasticsearch Introduction at BigData meetupEric Rodriguez (Hiring in Lex)

Real time analytics using Hadoop and ElasticsearchAbhishek Andhavarapu

Making (Almost) Any Database Faster and Cheaper with CachingAmazon Web Services

Getting started with Laravel & ElasticsearchPeter Steenbergen

Cassandraexsuns

Dev nexus 2017Roy Russo

AWS October Webinar Series - Introducing Amazon Elasticsearch ServiceAmazon Web Services

(BDT209) Launch: Amazon Elasticsearch For Real-Time Data AnalyticsAmazon Web Services

Devnexus 2018Roy Russo

NoSQL - Cassandra & MongoDB.pptxNaveen Kumar

Using SparkML to Power a DSaaS (Data Science as a Service): Spark Summit East...Spark Summit

ElasticSearch AJUG 2013Roy Russo

How ElasticSearch lives in my DevOps life��

Recently uploaded (20)

UNIT 1FUNDAMENTALS OF OPERATING SYSTEMS.pptxKesavanT10

Sorting Algorithms: Bubble Sort, Selection Sort,Dr. Madhuri Jawale

Taykon-Kalite belgeleriTAYKON

decarbonization steel industry rev1.pptxgonzalezolabarriaped

Common Network Architecture:X.25 Networks, Ethernet (Standard and Fast): fram...SnehPrasad2

Air pollution is contamination of the indoor or outdoor environment by any ch...dhanashree78

Air pollution is contamination of the indoor or outdoor environment by any chemical, physical or biological agent that modifies the natural characteristics of the atmosphere. Household combustion devices, motor vehicles, industrial facilities and forest fires are common sources of air pollution. Pollutants of major public health concern include particulate matter, carbon monoxide, ozone, nitrogen dioxide and sulfur dioxide. Outdoor and indoor air pollution cause respiratory and other diseases and are important sources of morbidity and mortality. WHO data show that almost all of the global population (99%) breathe air that exceeds WHO guideline limits and contains high levels of pollutants, with low- and middle-income countries suffering from the highest exposures. Air quality is closely linked to the earth��s climate and ecosystems globally. Many of the drivers of air pollution (i.e. combustion of fossil fuels) are also sources of greenhouse gas emissions. Policies to reduce air pollution, therefore, offer a win-win strategy for both climate and health, lowering the burden of disease attributable to air pollution, as well as contributing to the near- and long-term mitigation of climate change.

Dijkstra Shortest Path Algorithm in Network.pptRAJASEKARAN G

US Patented ReGenX Generator, ReGen-X Quatum Motor EV Regenerative Accelerati...Thane Heins NOBEL PRIZE WINNING ENERGY RESEARCHER

Preface: The ReGenX Generator innovation operates with a US Patented Frequency Dependent Load Current Delay which delays the creation and storage of created Electromagnetic Field Energy around the exterior of the generator coil. The result is the created and Time Delayed Electromagnetic Field Energy performs any magnitude of Positive Electro-Mechanical Work at infinite efficiency on the generator's Rotating Magnetic Field, increasing its Kinetic Energy and increasing the Kinetic Energy of an EV or ICE Vehicle to any magnitude without requiring any Externally Supplied Input Energy. In Electricity Generation applications the ReGenX Generator innovation now allows all electricity to be generated at infinite efficiency requiring zero Input Energy, zero Input Energy Cost, while producing zero Greenhouse Gas Emissions, zero Air Pollution and zero Nuclear Waste during the Electricity Generation Phase. In Electric Motor operation the ReGen-X Quantum Motor now allows any magnitude of Work to be performed with zero Electric Input Energy. Demonstration Protocol: The demonstration protocol involves three prototypes; 1. Protytpe #1, demonstrates the ReGenX Generator's Load Current Time Delay when compared to the instantaneous Load Current Sine Wave for a Conventional Generator Coil. 2. In the Conventional Faraday Generator operation the created Electromagnetic Field Energy performs Negative Work at infinite efficiency and it reduces the Kinetic Energy of the system. 3. The Magnitude of the Negative Work / System Kinetic Energy Reduction (in Joules) is equal to the Magnitude of the created Electromagnetic Field Energy (also in Joules). 4. When the Conventional Faraday Generator is placed On-Load, Negative Work is performed and the speed of the system decreases according to Lenz's Law of Induction. 5. In order to maintain the System Speed and the Electric Power magnitude to the Loads, additional Input Power must be supplied to the Prime Mover and additional Mechanical Input Power must be supplied to the Generator's Drive Shaft. 6. For example, if 100 Watts of Electric Power is delivered to the Load by the Faraday Generator, an additional >100 Watts of Mechanical Input Power must be supplied to the Generator's Drive Shaft by the Prime Mover. 7. If 1 MW of Electric Power is delivered to the Load by the Faraday Generator, an additional >1 MW Watts of Mechanical Input Power must be supplied to the Generator's Drive Shaft by the Prime Mover. 8. Generally speaking the ratio is 2 Watts of Mechanical Input Power to every 1 Watt of Electric Output Power generated. 9. The increase in Drive Shaft Mechanical Input Power is provided by the Prime Mover and the Input Energy Source which powers the Prime Mover. 10. In the Heins ReGenX Generator operation the created and Time Delayed Electromagnetic Field Energy performs Positive Work at infinite efficiency and it increases the Kinetic Energy of the system.

Indian Soil Classification System in Geotechnical EngineeringRajani Vyawahare

This PowerPoint presentation provides a comprehensive overview of the Indian Soil Classification System, widely used in geotechnical engineering for identifying and categorizing soils based on their properties. It covers essential aspects such as particle size distribution, sieve analysis, and Atterberg consistency limits, which play a crucial role in determining soil behavior for construction and foundation design. The presentation explains the classification of soil based on particle size, including gravel, sand, silt, and clay, and details the sieve analysis experiment used to determine grain size distribution. Additionally, it explores the Atterberg consistency limits, such as the liquid limit, plastic limit, and shrinkage limit, along with a plasticity chart to assess soil plasticity and its impact on engineering applications. Furthermore, it discusses the Indian Standard Soil Classification (IS 1498:1970) and its significance in construction, along with a comparison to the Unified Soil Classification System (USCS). With detailed explanations, graphs, charts, and practical applications, this presentation serves as a valuable resource for students, civil engineers, and researchers in the field of geotechnical engineering.

Introduction to Safety, Health & Environmentssuserc606c7

1-H2020-NEREID_RM3_T3.1Nanoscale-FET.pptDepartmentofElectric4

ESIT135 Problem Solving Using Python Notes of Unit-3prasadmutkule1

Closure Properties of Regular Languages.pptsvsivagami

GM Meeting 070225 TO 130225 for 2024.pptxcrdslalcomumbai

Biases, our brain and software developmentMatias Iacono

GREEN BULIDING PPT FOR THE REFRENACE.PPTkamalkeerthan61

eng funda notes.pdfdddddddddddddddddddddddaayushkumarsinghec22

TASK-DECOMPOSITION BASED ANOMALY DETECTION OF MASSIVE AND HIGH-VOLATILITY SES...samueljackson3773

Von karman Equation full derivation .pdfEr. Gurmeet Singh

PPt physics -GD.pptx gd topic for physics btechkavyamittal2201735

UNIT 1FUNDAMENTALS OF OPERATING SYSTEMS.pptxKesavanT10

Sorting Algorithms: Bubble Sort, Selection Sort,Dr. Madhuri Jawale

Taykon-Kalite belgeleriTAYKON

decarbonization steel industry rev1.pptxgonzalezolabarriaped

Common Network Architecture:X.25 Networks, Ethernet (Standard and Fast): fram...SnehPrasad2

Air pollution is contamination of the indoor or outdoor environment by any ch...dhanashree78

Dijkstra Shortest Path Algorithm in Network.pptRAJASEKARAN G

US Patented ReGenX Generator, ReGen-X Quatum Motor EV Regenerative Accelerati...Thane Heins NOBEL PRIZE WINNING ENERGY RESEARCHER

Indian Soil Classification System in Geotechnical EngineeringRajani Vyawahare

Introduction to Safety, Health & Environmentssuserc606c7

1-H2020-NEREID_RM3_T3.1Nanoscale-FET.pptDepartmentofElectric4

ESIT135 Problem Solving Using Python Notes of Unit-3prasadmutkule1

Closure Properties of Regular Languages.pptsvsivagami

GM Meeting 070225 TO 130225 for 2024.pptxcrdslalcomumbai

Biases, our brain and software developmentMatias Iacono

GREEN BULIDING PPT FOR THE REFRENACE.PPTkamalkeerthan61

eng funda notes.pdfdddddddddddddddddddddddaayushkumarsinghec22

TASK-DECOMPOSITION BASED ANOMALY DETECTION OF MASSIVE AND HIGH-VOLATILITY SES...samueljackson3773

Von karman Equation full derivation .pdfEr. Gurmeet Singh

PPt physics -GD.pptx gd topic for physics btechkavyamittal2201735

Elasticsearch in Production

1. Elasticsearch in production ! Alex Brasetvik alex@found.no @alexbrasetvik

2. Elasticsearch in production ! Alex Brasetvik alex@found.no @alexbrasetvik

3. Who? Co-founder of Found AS 8+ years search, 3+ Elasticsearch Herding hundreds of Elasticsearch clusters

4. Agenda

5. Agenda ? Anti-patterns ? Memory / Resource Usage ? Distributed problems ? Security ? Client concerns ? Changing a cluster

6. found.no/foundation Elasticsearch in Production Elasticsearch as a NoSQL Database Intro to Function Scoring All About Analyzers Securing your Elasticsearch Cluster

11. Snapshot / Restore Circuit breakers Document values Aggregations Distributed percolation Suggesters ��

12. Anti-Patterns

13. Arbitrary Keys ? ��Schema Free�� ? One ?eld per value ? Ever-growing cluster state acls: 1234: READ 42: WRITE

14. Heavy Updating ? Update = Delete + Reindex ? Be careful with counters

15. Slow queries ? WHERE foo ILIKE ��%bar%�� ? {��query_string��: {��query��: ��foo:*bar*��}}

16. Arbitrary searches query: filtered: filter: term: user_id: 42 query: [user��s query here]

18. Time Bomb

19. Memory

20. Memory ? Field caches ? Filter caches ? Page caches ? Aggregations ? Index building

21. Page Cache ? Keeping index pages in memory ? Can��t have too much ? Outgrow: Gradual slowdown

22. Heap Space ? Memory used by Elasticsearch process ? Field / Filter caches ? Aggregations

23. Time Bomb

24. Time Bomb

25. OutOfMemoryError Woah there I ate all the memories Your cluster may or may not work any more

26. OutOfMemory ? Growing too big ? Selecting too big timespan in Kibana ? Document ingestion peak

27. Preventing OOMs ? Have enough memory :-) ? Understand your search��s memory pro?le ? Bulk / Circuit breaker settings ? Monitoring ? Document values

28. Marvel ( /_stats )

31. Document Values

32. "my_field": { "type": "string", "fielddata": { "format": "doc_values" } }

33. Sizing

34. Sizing ? Test, don��t guess ? Start big, scale down ? Index, search, monitor

38. Glitch Meltdown

40. Glitch Meltdown

43. ? Tie-breaker can be a cheap master-node ? Applies to data centers / availability zones too

44. Data-only nodes Master-only nodes

46. Jepsen

47. Jepsen ? Kyle Kingsbury��s series on distributed systems ? Distributed systems are hard ? aphyr.com

48. Security

49. Security ? ��Not my job!�� C Elasticsearch ? That��s ?ne!

50. Dynamic Scripts ! ? Scoring ? Aggregations ? Updating

51. Dynamic Scripts Runtime.getRuntime().exec(��)

52. Dynamic Scripts Runtime.getRuntime().exec(��) <script src=/slideshow/elasticsearch-in-production-34431652/34431652/��http:/127.0.0.1:9200/_search?callback=capture&��

53. Security ! ? Disable dynamic scripts ? Mind index patterns ? Even then, don��t accept arbitrary requests

54. Client Concerns

55. Client Concerns ? Connection pools ? Idempotent requests ? Have sane syncing/indexing strategies

57. # BOOM !

58. Cluster changes

59. Cluster changes ? Make new nodes join existing cluster ? No rolling restarts ? Easy rollback if things go bad

60. v1.0.0 v1.0.1

61. Cluster changes ? Test ?rst ? Mind recover_*-settings

62. Multi-Cluster Work?ows ? Snapshot/Restore ? Operations across clusters ? Swap clusters! ? Works well with good syncing strategy

63. Misc ? Same JVM ? ulimits ? Unicast ? SSD? noop-scheduler

64. ? @foundsays

�ݺ�ߣ

Elasticsearch in Production

Recommended

More Related Content

What's hot (20)

Similar to Elasticsearch in Production (20)

Recently uploaded (20)

Elasticsearch in Production