Elasticsearch(java) fluentbit(c++) fluentd(ruby) kibana(javascript)
Download as PPTX, PDF
0 likes242 views
亠仄 仆舒仄 仄仂亞 舒从舒亰舒 access 仍仂亞亳 于亠弍亠于亠舒? 仂亳从 舒仆仂仄舒仍亳亶 亳 仂从仍仂仆亠仆亳亶 仂 仆仂仄? 从亟舒 仆舒亳 仗仂仍亰仂于舒亠仍亳? 仂仂亟? 弌舒仆舒? 弌舒亶? 弌从仂仍从仂 亰舒仗仂仂于 亞亠仆亠亳ﾑ 仂弍仂? 仂亞亟舒 于 仗仂仍亠亟仆亳亶 舒亰 从 仆舒仄 仗亳仂亟亳仍 仗仂亳从仂于亳从 亟仍 亳仆亟亠从舒亳亳? 舒从舒 亟亳仆舒仄亳从舒 仗仂 仂亳弍从舒仄 亳 舒仆亳舒仄 仂于ﾑ亳仄 仆舒 舒亶亠?
![丼仂 舒从仂亠 仍仂亞亳?
log_format main '$scheme://$host $remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" "$cookie_sid" $request_time';](https://image.slidesharecdn.com/elasticsearchjavafluentbitc-fluentdrubykibanajavascript-171206122709/85/Elasticsearch-java-fluentbit-c-fluentd-ruby-kibana-javascript-6-320.jpg)
![舒 仗亳仄亠 match
<match geoip.es.*.*>
buffer_chunk_limit 100m
buffer_queue_limit 100
@type geoip
geoip_lookup_key remote
<record>
country ${country_code["remote"]}
city ${city["remote"]}
</record>
remove_tag_prefix geoip.
</match>](https://image.slidesharecdn.com/elasticsearchjavafluentbitc-fluentdrubykibanajavascript-171206122709/85/Elasticsearch-java-fluentbit-c-fluentd-ruby-kibana-javascript-20-320.jpg)
![<match es.*.**>
@type elasticsearch_dynamic
buffer_type memory
buffer_chunk_limit 100m
buffer_queue_limit 256
host 127.0.0.1
port 9200
resurrect_after 5
reconnect_on_error true
logstash_format true
logstash_prefix ${tag_parts[1]}
include_tag_key true
tag_key @log_name
time_key_format %Y-%m-%dT%H:%M:%S.%N%z
flush_interval 30s
</match>](https://image.slidesharecdn.com/elasticsearchjavafluentbitc-fluentdrubykibanajavascript-171206122709/85/Elasticsearch-java-fluentbit-c-fluentd-ruby-kibana-javascript-21-320.jpg)
![亳仄亠 filter
<filter geoip.**.*>
@type record_transformer
enable_ruby
auto_typecast true
<record>
hostname ${hostname}
speed_kbps ${record["size"] * 8 / (record["resptime"] + 0.001) / 1000 }
</record>
</filter>](https://image.slidesharecdn.com/elasticsearchjavafluentbitc-fluentdrubykibanajavascript-171206122709/85/Elasticsearch-java-fluentbit-c-fluentd-ruby-kibana-javascript-22-320.jpg)
Elasticsearch(java) fluentbit(c++) fluentd(ruby) kibana(javascript)
- 1. Elasticsearch(java) Fluentbit -> Fluentd(ruby) Kibana(javascript) 弌亳亠仄舒 弍仂舒 亳 于亳亰舒仍亳亰舒亳亳 仍仂亞仂于 Alexander Sigachev for developers-day.ru
- 2. 从亟舒 舒 仆仂亞亳? 舒亠亠 从仂仍亳亠于仂 亠于亠仂于 仍仂亢仆磳 亰舒亟舒 仍仂从舒仍亳亰舒亳亳 仗仂弍仍亠仄 仂仗仍仆亠 仄亳从仂亠于亳 于亠仍亳亳于舒ﾑ 从仂仍亳亠于仂 log 舒亶仍仂于 舒亰仆亠 亠于亳 亳仄亠ﾑ 舒亰仆亶 仂仄舒 仍仂亞仂于
- 3. 舒从 弍仍仂 舒仆亠? nginx -> access.log -> logrotate -> access.log.1.gz grep 亟仍 仗仂亳从舒 仆舒仍亳亳从舒 zcat | cat | sort | uniq -c | sort -nr
- 4. 仂亳从 亳仆仄亠仆仂于 rsyslog 弌于仂亶 于亠仍仂亳仗亠亟 - 仆舒 mysql / other db 舒弍亳舒ﾑ舒 仗仂仗仍仆仂 ELK (elasticsearch+logstash+kibana) loghouse - 亠亠仆亳亠 仆舒 弍舒亰亠 clickhouse (1 仆仂磡 2017)
- 5. Open source data collection engine logreplica 2011-06-29 (http://dklab.ru) logstash fluend heka
- 6. 丼仂 舒从仂亠 仍仂亞亳? log_format main '$scheme://$host $remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for" "$cookie_sid" $request_time';
- 7. log_format main_json '{ "time": "$time_iso8601", ' '"scheme": "$scheme", ' '"domain": "$host", ' '"remote_addr": "$remote_addr", ' '"body_bytes_sent": $body_bytes_sent, ' '"status": $status, ' '"request": "$request", ' '"uri": "$uri", ' '"botflag": $botflag, ' '"http_user_agent": "$http_user_agent" }';
- 9. 舒 于弍仂 - fluend http://docs.fluentd.org/ 仂仆 仆舒 ruby +1 于 从舒仄 丕仗仂仄亳仆舒仍 于 亟仂从仍舒亟亠 仆舒 HighLoad (https://goo.gl/JB5fys)
- 12. 仍舒亞亳仆 于于仂亟舒 亟舒仆仆: in_tail - 亠仆亳亠 access.log nginx in_forward - 仗亳亠仄 仍仂亞仂于 仂 亟亞亳 舒亞亠仆仂于 亟仍 亠仆舒仍仆仂亶 仂弍舒弍仂从亳 in_http - http 亳仆亠亠亶 亠亠 仄仆仂亞仂 于舒亳舒仆仂于 ...
- 13. Buffer Pugins buf_memory buf_file
- 14. Filter plugins filter_record_transformer filter_grep filter_parser filter_stdout
- 15. Parser plugins parser_regexp parser_apache2 parser_apache_error parser_nginx parser_syslog parser_ltsv parser_csv parser_tsv parser_json parser_multiline parser_none
- 16. Output plugins out_file out_forward out_secure_forward out_exec out_exec_filter out_copy out_roundrobin out_s3 out_splunk out_kafka out_mongofs
- 17. IN_HTTP <source> @type http port 8888 bind 0.0.0.0 body_size_limit 32m keepalive_timeout 10s </source>
- 18. IN_TAIL <source> @type tail path /var/log/nginx/access.log pos_file /var/log/td-agent/nginx-access.log.pos tag nginx.access format nginx </source>
- 19. IN_FORWARD <source> @type forward port 24224 bind 0.0.0.0 </source>
- 20. 舒 仗亳仄亠 match <match geoip.es.*.*> buffer_chunk_limit 100m buffer_queue_limit 100 @type geoip geoip_lookup_key remote <record> country ${country_code["remote"]} city ${city["remote"]} </record> remove_tag_prefix geoip. </match>
- 21. <match es.*.**> @type elasticsearch_dynamic buffer_type memory buffer_chunk_limit 100m buffer_queue_limit 256 host 127.0.0.1 port 9200 resurrect_after 5 reconnect_on_error true logstash_format true logstash_prefix ${tag_parts[1]} include_tag_key true tag_key @log_name time_key_format %Y-%m-%dT%H:%M:%S.%N%z flush_interval 30s </match>
- 22. 亳仄亠 filter <filter geoip.**.*> @type record_transformer enable_ruby auto_typecast true <record> hostname ${hostname} speed_kbps ${record["size"] * 8 / (record["resptime"] + 0.001) / 1000 } </record> </filter>
- 23. 弌亠仄舒 source - tail filter - record match - forward source - tail filter - record match - forward source - forward match - geo match - elasctic Elasticsearch kibana
- 24. 亳仄亠 亞舒亳从仂于
- 25. nginx access log
- 27. 亠: 仍 亠舒 亳仗仂仍亰仂于舒仍 Cloud Hosting https://goo.gl/KfytA8 2Gb+2G(swap) 1core - 1000rps RAM + CPU 8Gb 2core - 2000rps RAM + CPU = ~ 31Gb 亰舒 从亳 16Gb 4core - 3000rps xxxx
- 28. 舒从 弍仂 仗仂仗仂弍仂于舒? 亠亠仄 亠于亠 2Gb 1Core ~ 20Gb (350弍/仄亠) docker + docker-compose 舒舒亳于舒亠仄 td-agent (仂仆 亢亠 fluentd) 舒仍舒亢亟舒亠仄 亠亰仍舒仂仄 丕舒仆仂于从舒 elasticsearch + kibana curl -L https://goo.gl/usglpi | sh - 弌舒 从舒从 弍仂 舒仆仂于亳 efk https://goo.gl/6vgdAX
- 29. 亞舒仆亳亠仆亳 亟仂仗舒 舒亳舒仆 仂亞舒仆亳亠仆亳 亟仂仗舒: X-Pack - 仗仍舒仆亶 仗仍舒亞亳仆 仂 elastc.io (1 仄亠 demo + 亠 弍舒亰仂于舒 仍亳亠仆亰亳) nginx base auth
- 30. 仂弍仍亠仄 fluentd - ruby 亞舒仆亳亠仆亳亠 从仂仂亳 亠仆亳 仂弍亳亶 亳亰 仍仂亞舒 仂亠弍仍磳仄亠 亠 亠亠仆亳亠 fluent-bit - C (仍亠亞从仂于亠仆亶 舒亞亠仆) 仂仗亳亳 仂亞舒仆亳亠仆亳 从仂仂亳 亠仆亳 亟舒仆仆 仂亠弍仍磳 仄亠仆亠 丕
- 32. 仂仗仂? 仍亠从舒仆亟 弌亳亞舒亠于 asigatchov@gmail.com 仆于亠仆仂 (亳仆亠仆舒舒 -> 仗亠亟仍仂亢亠仆亳亠 仂 舒弍仂亠) 舒仆舒仍 于 亠仍亠亞舒仄仄亠 - https://t.me/orelrb