酷鉛顎艶稼岳糸がよくわからなかった三
?
40 likes?11,129 views
The document discusses the Fluentd logging system. It includes an explanation of how Fluentd buffers and queues log data before outputting it. The emit method is used to add log data to a chunk, and if the chunk limit is reached a new chunk is created and added to the queue. Once data is in the queue it can be output by the configured plugin.
![Fluent::BasicBu?er#emit
def
?emit(key,
?data,
?chain)
?
?key
?=
?key.to_s
?
?top
?=
?(@map[key]
?||=
?new_chunk(key))
?
?if
?top.size
?+
?data.bytesize
?<=
?@buffer_chunk_limit
?
?
?
?chain.next
?
?
?
?top
?<<
?data
?
?
?
?return
?false
?
?elsif
?@queue.size
?>=
?@buffer_queue_limit
?
?
?
?raise
?BufferQueueLimitError,
?
?
?
?
?
?
?
?
?
?"queue
?size
?exceeds
?limit"
?
?end
?
?nc
?=
?new_chunk(key)
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-28-320.jpg)
![Fluent::BasicBu?er#emit
def
?emit(key,
?data,
?chain)
?
?...
?
?top
?=
?(@map[key]
?||=
?new_chunk(key))
?
?if
?top.size
?+
?data.bytesize
?<=
?@buffer_chunk_limit
...
?
?
?
?top
?<<
?data
?
?
?
?return
?false
top=@map[key]
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-29-320.jpg)
![Fluent::BasicBu?er#emit
def
?emit(key,
?data,
?chain)
?
?...
?
?top
?=
?(@map[key]
?||=
?new_chunk(key))
?
?if
?top.size
?+
?data.bytesize
?<=
?@buffer_chunk_limit
...
?
?
?
?top
?<<
?data
?
?
?
?return
?false
top=@map[key]
emit(key,data)
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-30-320.jpg)
![Fluent::BasicBu?er#emit
def
?emit(key,
?data,
?chain)
?
?...
?
?top
?=
?(@map[key]
?||=
?new_chunk(key))
?
?if
?top.size
?+
?data.bytesize
?<=
?@buffer_chunk_limit
...
?
?
?
?top
?<<
?data
?
?
?
?return
?false
top=@map[key]
emit(key,data)
○chunk
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-31-320.jpg)
![Fluent::BasicBu?er#emit
def
?emit(key,
?data,
?chain)
?
?...
?
?top
?=
?(@map[key]
?||=
?new_chunk(key))
?
?if
?top.size
?+
?data.bytesize
?<=
?@buffer_chunk_limit
...
?
?
?
?top
?<<
?data
?
?
?
?return
?false
top=@map[key]
emit(key,data)
data
○chunk
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-32-320.jpg)
![Fluent::BasicBu?er#emit
def
?emit(key,
?data,
?chain)
?
?...
?
?top
?=
?(@map[key]
?||=
?new_chunk(key))
?
?if
?top.size
?+
?data.bytesize
?<=
?@buffer_chunk_limit
...
?
?
?
?top
?<<
?data
?
?
?
?return
?false
top=@map[key]
emit(key,data)
data
emit(key,data)
○chunk
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-33-320.jpg)
![Fluent::BasicBu?er#emit
def
?emit(key,
?data,
?chain)
?
?...
?
?top
?=
?(@map[key]
?||=
?new_chunk(key))
?
?if
?top.size
?+
?data.bytesize
?<=
?@buffer_chunk_limit
...
?
?
?
?top
?<<
?data
?
?
?
?return
?false
top=@map[key]
emit(key,data)
data
data
emit(key,data)
○chunk
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-34-320.jpg)
![Fluent::BasicBu?er#emit
def
?emit(key,
?data,
?chain)
?
?...
?
?top
?=
?(@map[key]
?||=
?new_chunk(key))
?
?if
?top.size
?+
?data.bytesize
?<=
?@buffer_chunk_limit
...
?
?
?
?top
?<<
?data
?
?
?
?return
?false
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data)
○chunk
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-35-320.jpg)
![Fluent::BasicBu?er#emit
?
?end
?
?nc
?=
?new_chunk(key)
?
?ok
?=
?false
?
?begin
?
?
?
?nc
?<<
?data
?
?
?
?chain.next
?
?
?
?flush_trigger
?=
?false
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data) nc
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-36-320.jpg)
![Fluent::BasicBu?er#emit
?
?end
?
?nc
?=
?new_chunk(key)
?
?ok
?=
?false
?
?begin
?
?
?
?nc
?<<
?data
?
?
?
?chain.next
?
?
?
?flush_trigger
?=
?false
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data) nc
data
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-37-320.jpg)
![Fluent::BasicBu?er#emit
?
?
?
?flush_trigger
?=
?false
?
?
?
?@queue.synchronize
?{
?
?
?
?
?
?enqueue(top)
?
?
?
?
?
?flush_trigger
?=
?@queue.empty?
?
?
?
?
?
?@queue
?<<
?top
?
?
?
?
?
?@map[key]
?=
?nc
?
?
?
?}
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data) nc
data
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-38-320.jpg)
![Fluent::BasicBu?er#emit
?
?
?
?flush_trigger
?=
?false
?
?
?
?@queue.synchronize
?{
?
?
?
?
?
?enqueue(top)
?
?
?
?
?
?flush_trigger
?=
?@queue.empty?
?
?
?
?
?
?@queue
?<<
?top
?
?
?
?
?
?@map[key]
?=
?nc
?
?
?
?}
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data) nc
data
@queue
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-39-320.jpg)
![Fluent::BasicBu?er#emit
?
?
?
?flush_trigger
?=
?false
?
?
?
?@queue.synchronize
?{
?
?
?
?
?
?enqueue(top)
?
?
?
?
?
?flush_trigger
?=
?@queue.empty?
?
?
?
?
?
?@queue
?<<
?top
?
?
?
?
?
?@map[key]
?=
?nc
?
?
?
?}
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data) nc
data
@queue
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-40-320.jpg)
![Fluent::BasicBu?er#emit
?
?
?
?flush_trigger
?=
?false
?
?
?
?@queue.synchronize
?{
?
?
?
?
?
?enqueue(top)
?
?
?
?
?
?flush_trigger
?=
?@queue.empty?
?
?
?
?
?
?@queue
?<<
?top
?
?
?
?
?
?@map[key]
?=
?nc
?
?
?
?}
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data)
data
@queue
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-41-320.jpg)
![Fluent::BasicBu?er#emit
?
?
?
?flush_trigger
?=
?false
?
?
?
?@queue.synchronize
?{
?
?
?
?
?
?enqueue(top)
?
?
?
?
?
?flush_trigger
?=
?@queue.empty?
?
?
?
?
?
?@queue
?<<
?top
?
?
?
?
?
?@map[key]
?=
?nc
?
?
?
?}
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data)
data
@queue
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-42-320.jpg)
![Fluent::BasicBu?er#push
def
?push(key)
?
?top
?=
?@map[key]
?
?if
?!top
?||
?top.empty?
?
?
?
?return
?false
?
?end
?
?@queue.synchronize
?do
?
?
?
?enqueue(top)
?
?
?
?@queue
?<<
?top
?
?
?
?@map.delete(key)
?
?end
?
?return
?true
end
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-44-320.jpg)
![Fluent::BasicBu?er#push
?
?@queue.synchronize
?do
?
?
?
?enqueue(top)
?
?
?
?@queue
?<<
?top
?
?
?
?@map.delete(key)
?
?end
?
?return
?true
end
top=@map[key]
data
data
data
@queue
Sunday, June 2, 13](https://image.slidesharecdn.com/fluentd-130602023832-phpapp01/85/Fluentd-45-320.jpg)
酷鉛顎艶稼岳糸がよくわからなかった三
- 3. Sunday, June 2, 13
- 4. Sunday, June 2, 13
- 5. Ruby on Rails Sunday, June 2, 13
- 6. Ruby on Rails Capistrano Sunday, June 2, 13
- 7. Ruby on Rails Capistrano Fluentd Sunday, June 2, 13
- 8. fluentd Sunday, June 2, 13
- 9. Front Access App. Access Activity - Sign-up, Sign-in, bookmark, ... PHP/Rails error JS(client-side) error MySQL slow queries Solr slow requests IRC Sunday, June 2, 13
- 11. fluentd Sunday, June 2, 13
- 13. Input Plugin Bu?er Plugin Output Plugin Input data (tag, data) Output data Fluentd Basic architecture Sunday, June 2, 13
- 14. Configure Sunday, June 2, 13
- 15. <source> type tail path /tmp/test.log tag test.log format json </source> <match test.log> type mongo host localhost database test collection log bu?er_chunk_limit 2m bu?er_queue_limit 256 flush_interval 1s </match> Sunday, June 2, 13
- 16. なんか咾 Sunday, June 2, 13
- 17. <source> type tail path /tmp/test.log tag test.log format json </source> <match test.log> type mongo host localhost database test collection log bu?er_chunk_limit 2m bu?er_queue_limit 256 flush_interval 1s </match> Sunday, June 2, 13
- 18. <source> type tail path /tmp/test.log tag test.log format json </source> <match test.log> type mongo host localhost database test collection log bu?er_chunk_limit 2m bu?er_queue_limit 256 flush_interval 1s </match> Sunday, June 2, 13
- 19. ようわからん Sunday, June 2, 13
- 20. Sunday, June 2, 13
- 21. ようわからん Sunday, June 2, 13
- 22. | Sunday, June 2, 13
- 23. y叟業 哂Z < Ruby Sunday, June 2, 13
- 24. y叟業 Ruby < 哂Z Sunday, June 2, 13
- 28. Fluent::BasicBu?er#emit def ?emit(key, ?data, ?chain) ? ?key ?= ?key.to_s ? ?top ?= ?(@map[key] ?||= ?new_chunk(key)) ? ?if ?top.size ?+ ?data.bytesize ?<= ?@buffer_chunk_limit ? ? ? ?chain.next ? ? ? ?top ?<< ?data ? ? ? ?return ?false ? ?elsif ?@queue.size ?>= ?@buffer_queue_limit ? ? ? ?raise ?BufferQueueLimitError, ? ? ? ? ? ? ? ? ? ?"queue ?size ?exceeds ?limit" ? ?end ? ?nc ?= ?new_chunk(key) Sunday, June 2, 13
- 29. Fluent::BasicBu?er#emit def ?emit(key, ?data, ?chain) ? ?... ? ?top ?= ?(@map[key] ?||= ?new_chunk(key)) ? ?if ?top.size ?+ ?data.bytesize ?<= ?@buffer_chunk_limit ... ? ? ? ?top ?<< ?data ? ? ? ?return ?false top=@map[key] Sunday, June 2, 13
- 30. Fluent::BasicBu?er#emit def ?emit(key, ?data, ?chain) ? ?... ? ?top ?= ?(@map[key] ?||= ?new_chunk(key)) ? ?if ?top.size ?+ ?data.bytesize ?<= ?@buffer_chunk_limit ... ? ? ? ?top ?<< ?data ? ? ? ?return ?false top=@map[key] emit(key,data) Sunday, June 2, 13
- 31. Fluent::BasicBu?er#emit def ?emit(key, ?data, ?chain) ? ?... ? ?top ?= ?(@map[key] ?||= ?new_chunk(key)) ? ?if ?top.size ?+ ?data.bytesize ?<= ?@buffer_chunk_limit ... ? ? ? ?top ?<< ?data ? ? ? ?return ?false top=@map[key] emit(key,data) ○chunk Sunday, June 2, 13
- 32. Fluent::BasicBu?er#emit def ?emit(key, ?data, ?chain) ? ?... ? ?top ?= ?(@map[key] ?||= ?new_chunk(key)) ? ?if ?top.size ?+ ?data.bytesize ?<= ?@buffer_chunk_limit ... ? ? ? ?top ?<< ?data ? ? ? ?return ?false top=@map[key] emit(key,data) data ○chunk Sunday, June 2, 13
- 33. Fluent::BasicBu?er#emit def ?emit(key, ?data, ?chain) ? ?... ? ?top ?= ?(@map[key] ?||= ?new_chunk(key)) ? ?if ?top.size ?+ ?data.bytesize ?<= ?@buffer_chunk_limit ... ? ? ? ?top ?<< ?data ? ? ? ?return ?false top=@map[key] emit(key,data) data emit(key,data) ○chunk Sunday, June 2, 13
- 34. Fluent::BasicBu?er#emit def ?emit(key, ?data, ?chain) ? ?... ? ?top ?= ?(@map[key] ?||= ?new_chunk(key)) ? ?if ?top.size ?+ ?data.bytesize ?<= ?@buffer_chunk_limit ... ? ? ? ?top ?<< ?data ? ? ? ?return ?false top=@map[key] emit(key,data) data data emit(key,data) ○chunk Sunday, June 2, 13
- 35. Fluent::BasicBu?er#emit def ?emit(key, ?data, ?chain) ? ?... ? ?top ?= ?(@map[key] ?||= ?new_chunk(key)) ? ?if ?top.size ?+ ?data.bytesize ?<= ?@buffer_chunk_limit ... ? ? ? ?top ?<< ?data ? ? ? ?return ?false top=@map[key] emit(key,data) data data emit(key,data) emit(key,data) ○chunk Sunday, June 2, 13
- 36. Fluent::BasicBu?er#emit ? ?end ? ?nc ?= ?new_chunk(key) ? ?ok ?= ?false ? ?begin ? ? ? ?nc ?<< ?data ? ? ? ?chain.next ? ? ? ?flush_trigger ?= ?false top=@map[key] emit(key,data) data data emit(key,data) emit(key,data) nc Sunday, June 2, 13
- 37. Fluent::BasicBu?er#emit ? ?end ? ?nc ?= ?new_chunk(key) ? ?ok ?= ?false ? ?begin ? ? ? ?nc ?<< ?data ? ? ? ?chain.next ? ? ? ?flush_trigger ?= ?false top=@map[key] emit(key,data) data data emit(key,data) emit(key,data) nc data Sunday, June 2, 13
- 38. Fluent::BasicBu?er#emit ? ? ? ?flush_trigger ?= ?false ? ? ? ?@queue.synchronize ?{ ? ? ? ? ? ?enqueue(top) ? ? ? ? ? ?flush_trigger ?= ?@queue.empty? ? ? ? ? ? ?@queue ?<< ?top ? ? ? ? ? ?@map[key] ?= ?nc ? ? ? ?} top=@map[key] emit(key,data) data data emit(key,data) emit(key,data) nc data Sunday, June 2, 13
- 39. Fluent::BasicBu?er#emit ? ? ? ?flush_trigger ?= ?false ? ? ? ?@queue.synchronize ?{ ? ? ? ? ? ?enqueue(top) ? ? ? ? ? ?flush_trigger ?= ?@queue.empty? ? ? ? ? ? ?@queue ?<< ?top ? ? ? ? ? ?@map[key] ?= ?nc ? ? ? ?} top=@map[key] emit(key,data) data data emit(key,data) emit(key,data) nc data @queue Sunday, June 2, 13
- 40. Fluent::BasicBu?er#emit ? ? ? ?flush_trigger ?= ?false ? ? ? ?@queue.synchronize ?{ ? ? ? ? ? ?enqueue(top) ? ? ? ? ? ?flush_trigger ?= ?@queue.empty? ? ? ? ? ? ?@queue ?<< ?top ? ? ? ? ? ?@map[key] ?= ?nc ? ? ? ?} top=@map[key] emit(key,data) data data emit(key,data) emit(key,data) nc data @queue Sunday, June 2, 13
- 41. Fluent::BasicBu?er#emit ? ? ? ?flush_trigger ?= ?false ? ? ? ?@queue.synchronize ?{ ? ? ? ? ? ?enqueue(top) ? ? ? ? ? ?flush_trigger ?= ?@queue.empty? ? ? ? ? ? ?@queue ?<< ?top ? ? ? ? ? ?@map[key] ?= ?nc ? ? ? ?} top=@map[key] emit(key,data) data data emit(key,data) emit(key,data) data @queue Sunday, June 2, 13
- 42. Fluent::BasicBu?er#emit ? ? ? ?flush_trigger ?= ?false ? ? ? ?@queue.synchronize ?{ ? ? ? ? ? ?enqueue(top) ? ? ? ? ? ?flush_trigger ?= ?@queue.empty? ? ? ? ? ? ?@queue ?<< ?top ? ? ? ? ? ?@map[key] ?= ?nc ? ? ? ?} top=@map[key] emit(key,data) data data emit(key,data) emit(key,data) data @queue Sunday, June 2, 13
- 44. Fluent::BasicBu?er#push def ?push(key) ? ?top ?= ?@map[key] ? ?if ?!top ?|| ?top.empty? ? ? ? ?return ?false ? ?end ? ?@queue.synchronize ?do ? ? ? ?enqueue(top) ? ? ? ?@queue ?<< ?top ? ? ? ?@map.delete(key) ? ?end ? ?return ?true end Sunday, June 2, 13
- 45. Fluent::BasicBu?er#push ? ?@queue.synchronize ?do ? ? ? ?enqueue(top) ? ? ? ?@queue ?<< ?top ? ? ? ?@map.delete(key) ? ?end ? ?return ?true end top=@map[key] data data data @queue Sunday, June 2, 13
- 46. Fluent::BasicBu?er#push ? ?@queue.synchronize ?do ? ? ? ?enqueue(top) ? ? ? ?@queue ?<< ?top ? ? ? ?@map.delete(key) ? ?end ? ?return ?true end data data data @queue Sunday, June 2, 13
- 48. Fluent::BasicBu?er#pop def ?pop(out) ? ?chunk ?= ?@queue.first ? ?if ?!chunk.empty? ? ? ? ?write_chunk(chunk, ?out) ? ?end ? ?@queue.delete_if ?{|c| ? ? ? ?c.object_id ?== ?chunk.object_id ? ?} ? ?chunk.purge ? ?return ?!@queue.empty? end Sunday, June 2, 13
- 49. Fluent::BasicBu?er#pop data data data @queue def ?pop(out) ? ?chunk ?= ?@queue.first ? ?if ?!chunk.empty? ? ? ? ?write_chunk(chunk, ?out) ? ?end ? ?@queue.delete_if ?{|c| ? Sunday, June 2, 13
- 50. Fluent::BasicBu?er#pop data data data @queue def ?pop(out) ? ?chunk ?= ?@queue.first ? ?if ?!chunk.empty? ? ? ? ?write_chunk(chunk, ?out) ? ?end ? ?@queue.delete_if ?{|c| ? Sunday, June 2, 13
- 52. Configure parameters Sunday, June 2, 13
- 53. <source> type tail path /tmp/test.log tag test.log format json </source> <match test.log> type mongo host localhost database test collection log bu?er_chunk_limit 2m bu?er_queue_limit 256 flush_interval 1s </match> Sunday, June 2, 13
- 57. まとめ Sunday, June 2, 13