Log Management Systems

Jan 20, 2017Download as ODP, PDF1 like582 views

This document compares message and metric management solutions like Fluentd and Logstash. It discusses how these solutions can collect, store, visualize, and alert on log and metric data from heterogeneous environments. While commercial solutions like Splunk are very expensive, open source solutions like Fluentd, Logstash, Elasticsearch, and Kibana provide similar functionality through various "bricks" or components at no cost. The document analyzes key differences between Fluentd and Logstash, such as their configuration, buffering approaches, high availability features, and plugin ecosystems.

Log Management Systems
A comparison of message and metric
management solutions
Presenter: Mehdi Hamidi
( @eXtrem0us )

What is LOG?
●Combination of Time, Tag and Message
●Indicates State of Applications (?)
●Human and Machine Readable Messages (?)

Level of logs in syslog standard
●From Debug to Panic
●rsyslog, syslog, syslog-ng
●/var/log/syslog /var/log/rsyslog

Importance of logs
●Companies and Businesses
●Even Personal Use!
(Twitter, Sensors,... )

Importance of Logging Systems
WHAT Actually We NEED?
Collect
Messages
Metrics
●Store
●Visualize
●Alert

Importance of Logging Systems
Heterogeneous Environment
Write our own script for each type of log (?)
Not in an enterprise environment with lots of
devices and services!
●Technical Fragility and dependency to
Individuals
●Strong Dependency to knowledge about
underlying process

Commercial Solutions
Splunk
(500M/Day is Free, then: 5,000,000 $)
●Nagios
Everything is restricted to Nagios Concept
No separation between metrics and messages
No stylish diagrams (in free solution)
Problems in cloud infrastructure
No realtime monitoring
No manipulating messages
(1,995 $ for commercial solution)
●Online Services

Good logging system Specifications
●Have a common interface
●Decouple data sources from data outputs
Prevent mentioned dependencies
No effect of adding new data source/output
Reliability
Persistent Buffering
●Extensibility
High Availability
Load Balancing
Robustness

Lots of OpenSource Bricks (OSB!)
Logging Systems:
● Fluentd
● LogStash
● GrayLog
● Logalice
● Rsyslog
● Scribe
Message Stores:
● ElasticSeach
● Hadoop
● MongoDB
● File
● RDBMS
● Redis
● ...
Visualization
(Dashboards):
● Kibana
● Grafana
● Gaylog-WebUI
● PacketBeat
● Chronograph
● ...
Metric Stores:
● InfluxDB
● Prometheus
● Graphite
● ...
Alerting:
● Kapacitor
● Skyline
● Oculus
● Cabot

Log Nature
Semistructured or Unstructured
Generated Massively
More Written and less Read
(That's why we use NoSQL)

Overview
Fluentd:
Written in Cruby
Used in Google Cloud Platform and Kubernetes
Maintained by Tresure Data
●Logstash:
Written in Jruby
Used in ELK Stack
Maintained by Elastic Co.
➢Both use their own RubyGems Repo
➢Out of the box nature, less dependencies

Configuration
Fluentd:
Each Input is tagged
Logs are routed by tags
Logstash:
All inputs are Gathered and Scattered
Conditional Outputs, No tags

Transport and Buffering
Fluentd: built-in
LogStash: bundled Redis
version 5.3: persistent buffering

Full Buffer or Output Exception occurrence
Fluentd:
Exception: streaming
Block input plugin: batch
Drop oldest chunk: monitoring
LogStash:
Retry
Discard
Dead Letter Queuing

Memory
Fluentd: 40 M
Logstash: 120 M
(in big clusters matter)

Forwarders
●Fluentd:
 Fluentbit (Written in C)
 Fluentd-Forwarder (Written in Go)
 (all in one)
●LogStash:
 Filebeat
 Metricbeat
 Packetbeat
 Winlogbeat
(beat family: separated component for each
purpose)

Community and Support
Fluentd: Poor
Japanese Blogs
Google Group
Logstash: Rich
Documents
Blogs
IRC
Meetups and Certs

Plugins
Fluentd Plugins Verified
Input/Output 554 44
Filter 90 8
Parser 30 2
Formatter 6 0
Obsolete 8 0
Plugins are maintained more by other people.

Plugins
All Plugins are in a Single GitHub Repo.
LogStash Plugins
Input 52
Filter 46
Output 55

Thanks You!
ahamidi66@gmail.com
@eXtrem0us

Resources
http://logz.io/blog/fluentd-logstash
http://docs.fluentd.org/articles/buffer-plugin-overvie
https://prometheus.io/docs/introduction/comparison
http://logz.io/blog/elk-stack-5-0
https://www.youtube.com/watch?v=1ye0-sityBw
https://www.youtube.com/watch?v=0lAHrspviIs&list=PL62pIycqXx-TPwtk4JDd0wMuFAyP0gU1y
https://www.youtube.com/watch?v=mfb0R7azKZc
https://www.youtube.com/watch?v=_BAWi9Zhmic

Log management involves collecting logs from various sources, normalizing the data into a readable format, and using log intelligence and monitoring tools to detect threats, enable incident response and forensic investigations, and ensure regulatory compliance. It provides a centralized way to search logs, correlate events, and generate reports which can help security teams more efficiently investigate issues compared to traditional methods of reviewing raw logs. Challenges include a lack of standard log formats and capturing all activity, but the market for log management software is large and growing due to its importance for compliance needs.

Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...Anton Chuvakin

��

Logging is essential for security, operations, and compliance. However, common mistakes in log management include not logging at all, not reviewing logs, retaining logs for too short a time, prioritizing log collection, ignoring application logs, and only searching for known bad events. Effective log management requires collecting all relevant logs and retaining them for appropriate time periods according to a well-defined strategy.

Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinAnton Chuvakin

��

Title: Enterprise Logging and Log Management: Hot TopicsDate & Time: Thursday, April 1, 2010, 11:00am Eastern��Capturing log information is critical to IT organizations for many reasons, including for security incident detection and response, and for compliance with numerous regulations and standards. Join one of the foremost experts on log management, Dr. Anton Chuvakin, as we discuss enterprise logging challenges and issues.

Choosing Your Log Management Approach: Buy, Build or OutsourceAnton Chuvakin

��

NIST 800-92 Log Management Guide in the Real WorldAnton Chuvakin

��

Anton's Log Management 'Worst Practices'Anton Chuvakin

��

Qradar as a SOC coreMona Arkhipova

��

Svetlana Arkhipova presents on QIWI's experience using Qradar as the security operations center (SOC) core. The QIWI SOC processes over 11,000 events per second from over 2,800 log sources and 700,000 network flows per minute. The Qradar deployment includes 45 virtual and 2 physical servers to handle the large volumes of log data. Key challenges include collecting logs from various sources like Windows, databases, and custom in-house applications in a standardized way and tuning the SIEM for internal security scanners. The QIWI SOC also works to automate compliance with standards like PCI, SOX, and national security regulations.

Présentation ELK/SIEM et démo WazuhAurélie Henriot

��

"Grand Challenges" of Log ManagementAnton Chuvakin

��

Information Security: Advanced SIEM TechniquesReliaQuest

��

Joe Parltow, CISO, ReliaQuest (www.reliaquest.com) -We’ve all heard it before; SIEM is dead, defense is boring, logs suck, etc. The fact is having total visibility into what’s happening on your network is absolutely necessary and keeps you from having to answer questions like “How did you not know we were compromised for the past 6 months!” This talk focuses on advanced tips and tricks you can implement with your SIEM to give you better visibility into all areas of your environment. Also includes top secret, 1337 (ok maybe just average) code snippets.

EventLog Analyzer - Product overviewManageEngine EventLog Analyzer

��

A quick overview of MangeEngine EventLog Analyzer, the most cost-effective Log Management, Compliance Reporting software for Security Information and Event Management (SIEM). Using this Log Analyzer software, organizations can automate the entire process of managing terabytes of machine generated logs by collecting, analyzing, searching, reporting, and archiving from one central location. This event log analyzer software helps to mitigate security threats, archive data for conducting log forensics analysis, root cause analysis & more at http://www.manageengine.com/products/eventlog/

Implementing and Running SIEM: Approaches and LessonsAnton Chuvakin

��

This document provides an outline and overview of implementing and running a SIEM (Security Information and Event Management) system. It discusses different approaches such as building, buying, or outsourcing a SIEM. It analyzes the choices and risks/advantages of each approach. The document also details common "worst practices" seen in SIEM and log management projects and provides lessons learned from case studies of projects that did not go well.

RuSIEM overview (english version)Olesya Shelestova

��

This document summarizes a SIEM product called RuSIEM. It describes RuSIEM's team and technology, how the product works, its components, data scaling abilities, and performance capabilities. The document also outlines how RuSIEM differs from other SIEM solutions and provides details on installations, correlations, receiving and sending events, analytics, and the product's current status and 2017 roadmap.

Large enterprise SIEM: get ready for oversizeMona Arkhipova

��

This document discusses large enterprise security information and event management (SIEM) systems. It begins by distinguishing SIEM from simple log collection and systems monitoring. It then discusses the IBM qRadar SIEM platform and some of its architecture and performance challenges. The remainder of the document addresses challenges around log collection from various sources like Windows, Unix, databases and custom applications. It provides best practices guides and discusses normalization, indexing and storage of large volumes of log data. Specific metrics are given for the large QIWI SIEM installation handling millions of events per day. The document concludes by discussing automation of security monitoring and response.

Security Information and Event Management (SIEM)k33a

��

This document provides an overview of security information and event management (SIEM). It defines SIEM as software and services that combine security information management (SIM) and security event management (SEM). The key objectives of SIEM are to identify threats and breaches, collect audit logs for security and compliance, and conduct investigations. SIEM solutions centralize log collection, correlate events in real-time, generate reports, and provide log retention, forensics and compliance reporting capabilities. The document discusses typical SIEM features, architecture, deployment options, and reasons for SIEM implementation failures.

Tips on SIEM Ops 2015Anton Chuvakin

��

Dr. Anton Chuvakin provides an overview of SIEM architecture and operational processes. He notes that while a SIEM tool can be purchased, developing a full security monitoring capability requires growing people and maturing processes over time. The document outlines key aspects of deploying, running, and evolving a SIEM program, including common pitfalls to avoid, such as failing to define an initial scope or assuming the SIEM will run itself. It emphasizes taking an "output-driven" approach focused on solving security problems.

From SIEM to Business processesOlesya Shelestova

��

This document discusses RuSIEM Analytics, a product that provides log management, security information and event management, and real-time analytics capabilities. It aims to automate business processes, detect security incidents, analyze business metrics, and provide a single interface for employees. The product is already in use by many enterprise customers. It collects data from various sources, normalizes it, stores it for analysis, and ensures continuous data collection. It also provides security incident detection and prevention, reporting, and compliance functions. Real-time analytics are performed to detect incidents, establish baselines, and analyze multiple algorithms. The solution has various applications for IT, security, business units, and other teams.

SIEM Primer:Anton Chuvakin

��

SIEM stands for Security Information and Event Management. It involves collecting, aggregating, normalizing and retaining logs and other security-related data from across an organization. SIEM performs analysis on this data through correlation, prioritization and notification/alerting. It also provides reporting and workflow capabilities for security teams. While SIEM promises improved security through these functions, it requires careful planning, scoping, requirements development and ongoing focus to avoid failures and ensure value.

Chaos monitoringMona Arkhipova

��

The document discusses the challenges of implementing security practices at fintech companies that blend aspects of both IT and banking. It notes that while banks have strict regulations and processes, many fintech startups initially neglect security. The document advocates for approaches like identifying and examining systems and traffic, continuous monitoring, and having response plans to balance security and business needs in a rapidly changing environment that combines enterprise and custom technologies.

Understanding the Event Logchuckbt

��

Log Standards & Future Trends by Dr. Anton ChuvakinAnton Chuvakin

��

This document discusses log standards and future trends in logging. It outlines the current state of log chaos, with no standard formats, schemas, taxonomies, or transport mechanisms. This results in logs being difficult to analyze and make sense of. The document then discusses past attempts at log standards that failed and proposes that a common event expression standard could help bring order to the chaos by establishing common log syntax, taxonomy, transport, and recommendations. This would allow for improved log management, correlation, and security analysis capabilities. It also notes that growing log volumes will continue to pose challenges and will need to be addressed.

Positive Hack Days 7 - Ransomware forensiсsMona Arkhipova

��

You Can't Correlate what you don't have - ArcSight Protect 2011Scott Carlson

��

Security Incident Log Review Checklist by Dr Anton Chuvakin and Lenny ZeltserAnton Chuvakin

��

Security Ops for large and small companiesMona Arkhipova

��

This document provides an overview of security operations (SOC) for companies of all sizes. It discusses the basic questions companies should ask regarding their security needs, how to approach monitoring both business and system metrics, and an in-depth look at monitoring and establishing an internal SOC. It then discusses solutions for security operations, both free and enterprise options, and things to consider if solutions are not working as expected or fail. The document concludes with embedding security into IT operations, discussing overlapping processes, compliance standards, typical incident response plans, and automating security processes. It also covers different approaches to security testing programs.

Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinAnton Chuvakin

��

End-User Case Study: Five Best and Five Worst Practices for SIEM Implementing SIEM sounds straightforward, but reality sometimes begs to differ. In this session, Dr. Anton Chuvakin will share the five best and worst practices for implementing SIEM as part of security monitoring and intelligence. Understanding how to avoid pitfalls and create a successful SIEM implementation will help maximize security and compliance value, and avoid costly obstacles, inefficiencies, and risks

Nagios Conference 2012 - Jared Bird - Providing Value Throughout the Organiza...Nagios

��

This document discusses how Nagios monitoring software can provide value across different departments in an organization. It describes how Nagios can help infrastructure teams monitor availability, capacity, and proper functioning. It also explains how Nagios can help security teams by detecting threats like default configurations, website defacement, missing patches, and unauthorized access. Finally, it outlines how Nagios supports compliance with regulations like PCI, SOX, and HIPAA by monitoring technical controls and security requirements.

Risks vs real lifeMona Arkhipova

��

This document discusses how traditional approaches to information security risk management may not fully account for real-life risks. It provides examples of how risks can arise from unexpected places, like shared physical access to offices, insecure internal systems due to lack of segmentation, reuse of passwords in test environments, lack of oversight of third-party services, failure to patch legacy systems, poor code quality leading to stability issues, and insecure employee devices and actions. The document argues that a comprehensive security program must anticipate risks from all parts of an organization's systems and operations, not just external threats.

Logging for ContainersEduardo Silva Pereira

��

Monitoring.pptxShadi Akil

��

Fluentd is used to collect and forward logs from microservices to Elasticsearch. Fluentd integrates with applications using its tail input plugin to read logs without saving files. Logs are forwarded and structured in Elasticsearch. Kibana dashboards provide visualization and querying of logs indexed in Elasticsearch to monitor application status, errors, and performance across all microservices. This solves the problems of losing logs during deployments, saving files on containers, and gaining insights without complex logging.

More Related Content

What's hot (20)

"Grand Challenges" of Log ManagementAnton Chuvakin

��

Information Security: Advanced SIEM TechniquesReliaQuest

��

EventLog Analyzer - Product overviewManageEngine EventLog Analyzer

��

Implementing and Running SIEM: Approaches and LessonsAnton Chuvakin

��

RuSIEM overview (english version)Olesya Shelestova

��

Large enterprise SIEM: get ready for oversizeMona Arkhipova

��

Security Information and Event Management (SIEM)k33a

��

Tips on SIEM Ops 2015Anton Chuvakin

��

From SIEM to Business processesOlesya Shelestova

��

SIEM Primer:Anton Chuvakin

��

Chaos monitoringMona Arkhipova

��

Understanding the Event Logchuckbt

��

Log Standards & Future Trends by Dr. Anton ChuvakinAnton Chuvakin

��

Positive Hack Days 7 - Ransomware forensiсsMona Arkhipova

��

You Can't Correlate what you don't have - ArcSight Protect 2011Scott Carlson

��

Security Incident Log Review Checklist by Dr Anton Chuvakin and Lenny ZeltserAnton Chuvakin

��

Security Ops for large and small companiesMona Arkhipova

��

Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinAnton Chuvakin

��

Nagios Conference 2012 - Jared Bird - Providing Value Throughout the Organiza...Nagios

��

Risks vs real lifeMona Arkhipova

��

"Grand Challenges" of Log ManagementAnton Chuvakin

��

Information Security: Advanced SIEM TechniquesReliaQuest

��

EventLog Analyzer - Product overviewManageEngine EventLog Analyzer

��

Implementing and Running SIEM: Approaches and LessonsAnton Chuvakin

��

RuSIEM overview (english version)Olesya Shelestova

��

Large enterprise SIEM: get ready for oversizeMona Arkhipova

��

Security Information and Event Management (SIEM)k33a

��

Tips on SIEM Ops 2015Anton Chuvakin

��

From SIEM to Business processesOlesya Shelestova

��

SIEM Primer:Anton Chuvakin

��

Chaos monitoringMona Arkhipova

��

Understanding the Event Logchuckbt

��

Log Standards & Future Trends by Dr. Anton ChuvakinAnton Chuvakin

��

Positive Hack Days 7 - Ransomware forensiсsMona Arkhipova

��

You Can't Correlate what you don't have - ArcSight Protect 2011Scott Carlson

��

Security Incident Log Review Checklist by Dr Anton Chuvakin and Lenny ZeltserAnton Chuvakin

��

Security Ops for large and small companiesMona Arkhipova

��

Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinAnton Chuvakin

��

Nagios Conference 2012 - Jared Bird - Providing Value Throughout the Organiza...Nagios

��

Risks vs real lifeMona Arkhipova

��

Similar to Log Management Systems (20)

Logging for ContainersEduardo Silva Pereira

��

Monitoring.pptxShadi Akil

��

Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...Hernan Costante

��

Nowadays in an increasingly more complex and dynamic network its not enough to be a regex ninja and storing only the logs you think you might need. From network traffic to custom logs you won't know which logs will be crucial to stop the next attacker, and if you are not planning to spend a half of your security budget in a commercial solution we will show you a way to building you own SIEM with open source. The talk will go from how to build a powerful logging environment for your organization to scaling on the cloud and storing everything forever. We will walk through how to build such a system with open source solutions as Elasticsearch and Hadoop, and creating your own custom monitoring rules to monitor everything you need. The talk will also include how to secure the environment and allow restricted access to other teams as well as avoiding common pitfalls and ensuring compliance standards.

Kibana+ElasticSearch+LogStash to handle Log messages on Prod serversHYS Enterprise

��

The document provides an overview of logging and the Elastic Stack (Elasticsearch, Logstash, Kibana) for log analysis. It discusses what logs are, popular logging processors like Loggly and Splunk, and the key components of the Elastic Stack - Elasticsearch for storage, Logstash for processing, and Kibana for visualization. It also gives a live demo example of using Logstash to ingest application logs and explore them in Kibana.

Fluent Bit: Log Forwarding at ScaleEduardo Silva Pereira

��

This document summarizes a presentation about log forwarding at scale. It discusses how logging works internally and requires understanding the logging pipeline of parsing, filtering, buffering and routing logs. It then introduces Fluent Bit as a lightweight log forwarder that can be used to cheaply forward logs from edge nodes to log aggregators in a scalable way, especially in cloud native environments like Kubernetes. Hands-on demos show how Fluent Bit can parse and add metadata to Kubernetes logs.

Big data Argentina meetup 2020-09: Intro to presto on dockerFederico Palladoro

��

Containers and LoggingEduardo Silva Pereira

��

Log forwarding at ScaleEduardo Silva Pereira

��

Turbo charge your logsJeremy Cook

��

This document discusses logging best practices and tools for log centralization and analysis. It recommends: 1. Using Monolog for PHP logging to centralize logs from application code in a standardized format. Monolog supports multiple handlers for storage and alerts. 2. Leveraging Rsyslog to also centrally collect logs from software, services and the operating system. Rsyslog can scale across servers. 3. Ingesting logs into Logstash for filtering, parsing and forwarding. Logstash supports many input sources and output targets. 4. Storing logs long-term in Graylog2 for powerful searching, analytics, dashboards and alerting. Graylog2 is highly scalable and easy to use.

Centralized logging system using mongoDBVivek Parihar

��

This talk will cover the need of a centralized logging system, showcasing the architecture of the system. Also, I talk about how we ended up building this centralized logging system, What was the need for such a system, what problems we faced, how MongoDB fits into this and what others can learn from this. I also covered some how can we use mongoDB to make our logging system for realtime analytics and alerting system . The major use case of this system it to keeping track of meaningful events. This could be -: 1. How many users registered ? 2. How many registrations fails ? 3. Most occurred errors while doing something. 4. Realtime Analytics and Alerts 5. Identifying the possible threats.

Serverless for High Performance ComputingLuciano Mammino

��

Serverless is great for web applications and APIs, but this does not mean it cannot be used successfully for other use cases. In this talk, we will discuss a successful application of serverless in the field of High Performance Computing. Specifically we will discuss how Lambda, Fargate, Kinesis and other serverless technologies are being used to run sophisticated financial models at one of the major reinsurance companies in the World. We we learn about the architecture, the tradeoffs, some challenges and some unresolved pain points. Most importantly, we'll find out if serverless can be a great fit for HPC and if we can finally stop managing those boring EC2 instances!

Distributed real time stream processing- why and howPetr Zapletal

��

In this talk you will discover various state-of-the-art open-source distributed streaming frameworks, their similarities and differences, implementation trade-offs, their intended use-cases, and how to choose between them. Petr will focus on the popular frameworks, including Spark Streaming, Storm, Samza and Flink. You will also explore theoretical introduction, common pitfalls, popular architectures, and much more. The demand for stream processing is increasing. Immense amounts of data has to be processed fast from a rapidly growing set of disparate data sources. This pushes the limits of traditional data processing infrastructures. These stream-based applications, include trading, social networks, the Internet of Things, and system monitoring, are becoming more and more important. A number of powerful, easy-to-use open source platforms have emerged to address this. Petr's goal is to provide a comprehensive overview of modern streaming solutions and to help fellow developers with picking the best possible solution for their particular use-case. Join this talk if you are thinking about, implementing, or have already deployed a streaming solution.

linux_internals_2.3 (1).pdf àaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaYasaswiniChintamalla1

��

Building scalable and language independent java services using apache thriftTalentica Software

��

Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with ...Miguel Pérez Colino

��

Node.js Web Apps @ ebay scaleDmytro Semenov

��

Logging, Serilog, Structured Logging, SeqDoruk Uluçay

��

This document discusses logging best practices and compares different .NET logging frameworks like log4net, NLog, and Serilog. It recommends using Serilog for its simpler API and support for structured logging. Structured logging stores log data in JSON format for easier parsing and querying. Seq is introduced as a log management platform that supports structured logs and has tools for filtering, analyzing, and alerting on logs. The document demonstrates logging an object to Seq using Serilog's structured logging features.

FluentD vs. LogstashAll Things Open

��

Presented by: Justin Reock Presented at the All Things Open 2021 Raleigh, NC, USA Raleigh Convention Center Abstract: In our FluentD vs. Logstash comparison blog we talked about the importance of easily capturing, parsing, and visualizing log data at enterprise scale.��We looked at the approaches FluentD and Logstash take to accomplish these tasks and defined particular areas of complexity and challenge that users face.��Participants in this demo-driven webinar will watch as a system is configured for log analysis using both approaches, highlighting the strengths and weaknesses of each technology in the process. Companion Blog:��https://www.openlogic.com/blog/fluentd-vs-logstash

Fedora Developer's Conference 2014 TalkRainer Gerhards

��

Turbo charge your logsJeremy Cook

��

This document discusses ways to optimize logging by centralizing and proactively using log data. It recommends using Monolog to log from application code in a standardized format. Rsyslog can then collect logs centrally from applications and systems. Logstash can further process logs with filters and output them to destinations like Elasticsearch. Graylog2 provides a web interface for powerful log searching, analytics, and alerting. Centralizing, standardizing, and proactively analyzing logs with these open source tools allows for improved monitoring and troubleshooting.

Logging for ContainersEduardo Silva Pereira

��

Monitoring.pptxShadi Akil

��

Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...Hernan Costante

��

Kibana+ElasticSearch+LogStash to handle Log messages on Prod serversHYS Enterprise

��

Fluent Bit: Log Forwarding at ScaleEduardo Silva Pereira

��

Big data Argentina meetup 2020-09: Intro to presto on dockerFederico Palladoro

��

Containers and LoggingEduardo Silva Pereira

��

Log forwarding at ScaleEduardo Silva Pereira

��

Turbo charge your logsJeremy Cook

��

Centralized logging system using mongoDBVivek Parihar

��

Serverless for High Performance ComputingLuciano Mammino

��

Distributed real time stream processing- why and howPetr Zapletal

��

linux_internals_2.3 (1).pdf àaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaYasaswiniChintamalla1

��

Building scalable and language independent java services using apache thriftTalentica Software

��

Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with ...Miguel Pérez Colino

��

Node.js Web Apps @ ebay scaleDmytro Semenov

��

Logging, Serilog, Structured Logging, SeqDoruk Uluçay

��

FluentD vs. LogstashAll Things Open

��

Fedora Developer's Conference 2014 TalkRainer Gerhards

��

Turbo charge your logsJeremy Cook

��

Recently uploaded (20)

Test slides produced by Presentie.ai to test usabiltyOtto Wijnen

��

Aquatic Mamalsdddddddddddddcholadeck.pptxsarveshsinghbhati

��

Front-End Specialization Certification: A Journey with Style!ShubhamSharma2528

��

Book Presentation - Hacking Home Devices I: PoCs & Hacks Just for FunGerard Fuguet

��

The official presentation of my first book: “Hacking Home Devices I: PoCs & Hacks Just for Fun” Published under 0xWord editorial. Was presented in Barcelona, in the library of my hometown (Sant Just Desvern). The event was scheduled on 2025-03-12 and started at 19:00h About the book: This book explores creative and surprising ways to “hack” home devices for fun, with proofs of concepts (PoCs) and ingenious hacks. It's a practical guide for hackers and enthusiasts, curious about computing (IT) who want to experiment with home technology in a safe and entertaining way. The town council made it public in the following links (under Catalan language): https://web.archive.org/web/20250226080308/https://biblioteca.santjust.net/presentacio-del-llibre-hacking-home-devices-de-gerard-fuguet/ https://www.radiodesvern.com/news/presentacio-a-can-ginestar-del-llibre-sobre-hacks-domestics-creatius-de-gerard-fuguet The book (under Spanish language) is available online at: https://0xword.com/es/libros/238-hacking-home-devices-i-pocs-hacks-just-for-fun.html

Regional Economic Performance and Prospects: Progress Towards a Resilient FutureCaribbean Development Bank

��

2025-03-23 FATC 04 Mary, Martha, Lazarus & Mary Magdalene (shared slides).pptxDale Wells

��

SMALL-BUSINESS: the impacts of small business enterprisesestenzomaria08

��

Comprehensive Hematology Overview: Blood Physiology, Disorders, and Diagnosticsrdiasolatorio

��

QUIZ-4-INTERTEXTUALITY.pdf2024- 2025by linhsBabyJaneGPillado

��

Leeds Play Community - Play Explorer Kithappyasabean

��

Leeds Play Community - Play Explorer Kit Training  An introduction to the Leeds Play Community and how to use the Leeds Play Explorer cards. RESOURCES: Raising the Nation Play Commission interim report, Feb 2025 https://cdn.prod.website-files.com/659fd56cbd8d3f4a80aaac76/67b5f05190b914fe09e20a6f_decc46b350746fcccde8c6539bd44f3d_%5BFINAL%20-%20Digital%5D%20State%20of%20Play%20%E2%80%94%20Raising%20the%20Nation%20Play%20Commission.pdf Leeds Play Sufficiency Study www.cfl.leeds.gov.uk/play/play-sufficiency Play England website www.playengland.org.uk/ Child Friendly Leeds website www.cfl.leeds.gov.uk/ Playing Out website (Playstreet resources) www.playingout.net/ Playful Anywhere Website www.playfulanywhere.fun/

The Dubai Street - Latest UAE News & Gulf NewsThe Dubai Street

��

Video Skills for Authors: Mastering Your On-Camera PresencePaula Rizzo

��

Hosted by Jane Friedman - March 21, 2025 Video can connect you with readers like no other medium, and leveraging video helps you expand your reach. Join Paula Rizzo, Emmy Award-winning TV producer and bestselling author, in this free session where you'll learn how to create captivating videos, excel in media interviews, and harness the power of video to boost your book's visibility and success. Learn from a seasoned professional and take your video skills to the next level. We'll cover: — Crafting Engaging Videos: produce videos that capture attention and hold it, making your message memorable. — Having On-Camera Confidence: deliver compelling and persuasive media interviews that resonate with your audience in podcasts, on television, radio and anywhere online. — Creating Newsworthy Content: Structure your stories to appeal to both the media and your readers. — Production Best Practices: Get the inside scoop on what works (and what doesn't) in video production. — Leveraging Video for Marketing: Drive book sales, attract media attention, and grow your audience. paularizzo.com/10q-Jane - for free download "10 Questions Every Author Needs to Answer In the Media"

Alternative-toothpaste: Coconut oil and baking sodaestenzomaria08

��

Rebirth: Innovate, Transform and Thrive for a Resilient FutureCaribbean Development Bank

��

adgcraft-com-blog-esg-communication-digital-age-pr-strategy-....pptadgcraft23

��

As businesses place a stronger emphasis on transparency and accountability, Environmental, Social, and Governance (ESG) communications have become a key driver of trust and reputation. Stakeholders, from investors to consumers, expect organizations to not only make ESG commitments but also communicate them effectively. How a company shares its sustainability efforts, ethical practices, and social impact can shape public perception and influence long-term success.

Cause and Effect PowerPoint PresentationTamaraCarey1

��

Time management tools for presentations/quiz/discussionsadebayoekosanmi

��

HARD PALATE & SOFT PALATE.pptx anatomy 1 st yearmuhdhashim332657

��

Building Resilience to Climate Change and Disasters in the CaribbeanCaribbean Development Bank

��

Comment materi PPG dari lembar kerja LK 1agustinusgudipun

��

Test slides produced by Presentie.ai to test usabiltyOtto Wijnen

��

Aquatic Mamalsdddddddddddddcholadeck.pptxsarveshsinghbhati

��

Front-End Specialization Certification: A Journey with Style!ShubhamSharma2528

��

Book Presentation - Hacking Home Devices I: PoCs & Hacks Just for FunGerard Fuguet

��

Regional Economic Performance and Prospects: Progress Towards a Resilient FutureCaribbean Development Bank

��

2025-03-23 FATC 04 Mary, Martha, Lazarus & Mary Magdalene (shared slides).pptxDale Wells

��

SMALL-BUSINESS: the impacts of small business enterprisesestenzomaria08

��

Comprehensive Hematology Overview: Blood Physiology, Disorders, and Diagnosticsrdiasolatorio

��

QUIZ-4-INTERTEXTUALITY.pdf2024- 2025by linhsBabyJaneGPillado

��

Leeds Play Community - Play Explorer Kithappyasabean

��

The Dubai Street - Latest UAE News & Gulf NewsThe Dubai Street

��

Video Skills for Authors: Mastering Your On-Camera PresencePaula Rizzo

��

Alternative-toothpaste: Coconut oil and baking sodaestenzomaria08

��

Rebirth: Innovate, Transform and Thrive for a Resilient FutureCaribbean Development Bank

��

adgcraft-com-blog-esg-communication-digital-age-pr-strategy-....pptadgcraft23

��

Cause and Effect PowerPoint PresentationTamaraCarey1

��

Time management tools for presentations/quiz/discussionsadebayoekosanmi

��

HARD PALATE & SOFT PALATE.pptx anatomy 1 st yearmuhdhashim332657

��

Building Resilience to Climate Change and Disasters in the CaribbeanCaribbean Development Bank

��

Comment materi PPG dari lembar kerja LK 1agustinusgudipun

��

Log Management Systems

1. Log Management Systems A comparison of message and metric management solutions Presenter: Mehdi Hamidi ( @eXtrem0us )

2. Introduction

4. What is LOG? ●Combination of Time, Tag and Message ●Indicates State of Applications (?) ●Human and Machine Readable Messages (?)

7. Level of logs in syslog standard ●From Debug to Panic ●rsyslog, syslog, syslog-ng ●/var/log/syslog /var/log/rsyslog

8. Importance of logs ●Companies and Businesses ●Even Personal Use! (Twitter, Sensors,... )

9. LogAnalyzer: a simple solution :)

10. LogAnalyzer: a simple solution :)

11. Importance of Logging Systems WHAT Actually We NEED? Collect Messages Metrics ●Store ●Visualize ●Alert

12. Importance of Logging Systems Heterogeneous Environment Write our own script for each type of log (?) Not in an enterprise environment with lots of devices and services! ●Technical Fragility and dependency to Individuals ●Strong Dependency to knowledge about underlying process

15. Commercial Solutions Splunk (500M/Day is Free, then: 5,000,000 $) ●Nagios Everything is restricted to Nagios Concept No separation between metrics and messages No stylish diagrams (in free solution) Problems in cloud infrastructure No realtime monitoring No manipulating messages (1,995 $ for commercial solution) ●Online Services

16. Good logging system Specifications ●Have a common interface ●Decouple data sources from data outputs Prevent mentioned dependencies No effect of adding new data source/output Reliability Persistent Buffering ●Extensibility High Availability Load Balancing Robustness

17. Lots of OpenSource Bricks (OSB!) Logging Systems: ● Fluentd ● LogStash ● GrayLog ● Logalice ● Rsyslog ● Scribe Message Stores: ● ElasticSeach ● Hadoop ● MongoDB ● File ● RDBMS ● Redis ● ... Visualization (Dashboards): ● Kibana ● Grafana ● Gaylog-WebUI ● PacketBeat ● Chronograph ● ... Metric Stores: ● InfluxDB ● Prometheus ● Graphite ● ... Alerting: ● Kapacitor ● Skyline ● Oculus ● Cabot

18. Log Nature Semistructured or Unstructured Generated Massively More Written and less Read (That's why we use NoSQL)

19. Popular Stacks (metrics): TICK Stack

20. Popular Stacks (Messages) : ELK Stack

23. Fluentd VS LogStash

24. Overview Fluentd: Written in Cruby Used in Google Cloud Platform and Kubernetes Maintained by Tresure Data ●Logstash: Written in Jruby Used in ELK Stack Maintained by Elastic Co. ➢Both use their own RubyGems Repo ➢Out of the box nature, less dependencies

25. Configuration Fluentd: Each Input is tagged Logs are routed by tags Logstash: All inputs are Gathered and Scattered Conditional Outputs, No tags

26. Configuration

27. Transport and Buffering Fluentd: built-in LogStash: bundled Redis version 5.3: persistent buffering

28. Full Buffer or Output Exception occurrence Fluentd: Exception: streaming Block input plugin: batch Drop oldest chunk: monitoring LogStash: Retry Discard Dead Letter Queuing

29. High Availability and Load balancing

30. High Availability and Load balancing

31. High Availability and Load balancing

32. High Availability and Load balancing

33. Memory Fluentd: 40 M Logstash: 120 M (in big clusters matter)

34. Forwarders ●Fluentd:  Fluentbit (Written in C)  Fluentd-Forwarder (Written in Go)  (all in one) ●LogStash:  Filebeat  Metricbeat  Packetbeat  Winlogbeat (beat family: separated component for each purpose)

35. Community and Support Fluentd: Poor Japanese Blogs Google Group Logstash: Rich Documents Blogs IRC Meetups and Certs

36. Plugins Fluentd Plugins Verified Input/Output 554 44 Filter 90 8 Parser 30 2 Formatter 6 0 Obsolete 8 0 Plugins are maintained more by other people.

37. Plugins All Plugins are in a Single GitHub Repo. LogStash Plugins Input 52 Filter 46 Output 55

38. Questions?

39. Thanks You! ahamidi66@gmail.com @eXtrem0us

40. Resources http://logz.io/blog/fluentd-logstash http://docs.fluentd.org/articles/buffer-plugin-overvie https://prometheus.io/docs/introduction/comparison http://logz.io/blog/elk-stack-5-0 https://www.youtube.com/watch?v=1ye0-sityBw https://www.youtube.com/watch?v=0lAHrspviIs&list=PL62pIycqXx-TPwtk4JDd0wMuFAyP0gU1y https://www.youtube.com/watch?v=mfb0R7azKZc https://www.youtube.com/watch?v=_BAWi9Zhmic

�ݺ�ߣ

Log Management Systems

Recommended

More Related Content

What's hot (20)

Similar to Log Management Systems (20)

Recently uploaded (20)

Log Management Systems