Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)
?Download as PPTX, PDF?
5 likes?3,380 views
Fluentd is an open source data collector to simplify and scale log management. In this talk, I gave a brief overview of Fluentd, focusing on the why: Why should you care about logging? Why do you want a unified logging layer? Why Fluentd?
![127.0.0.1 - - [05/Feb/2012:17:11:55
+0000] "GET / HTTP/1.1" 200 140 "-"
"Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/535.19 (KHTML, like Gecko)
Chrome/18.0.1025.5 Safari/535.19"](https://image.slidesharecdn.com/sflogging2014july-140716145103-phpapp01/85/Fluentd-Towards-Unified-Logging-SF-Logging-Meetup-Jult-2014-21-320.jpg)
![[¡°05/Feb/2012:17:11:55¡±,¡°web.access¡±,{
"host": "127.0.0.1",
"user": "-",
"method": "GET",
"path": "/",
"code": "200",
"size": "140",
"referer": "-",
"agent": ¡°Mozilla/5.0 (Windows¡"
}]](https://image.slidesharecdn.com/sflogging2014july-140716145103-phpapp01/85/Fluentd-Towards-Unified-Logging-SF-Logging-Meetup-Jult-2014-26-320.jpg)
Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)
- 1. Fluentd: Towards Unified Logging Layer Kiyoto Tamura @kiyototamura Treasure Data, Inc.
- 3. ? Not an ops engineer myself ? ¡°marketing guy¡± ? My laptop isn¡¯t very hipster Why you shouldn¡¯t listen to me
- 4. ? Fluentd maintainer and community manager ? Former programmer ? data nerd Why you should listen to me
- 5. why Fluentd?
- 6. I was a data scientist before it was cool to be one.
- 7. http://cacm.acm.org/blogs/blog-cacm/169199-data-science-workflow- overview-and-challenges/fulltext
- 8. Acquire Data (or so you think) WUT!? Invalid UTF8? Fix the encoding issue¡ Yell at the engineers Some columns are missing!? Run the script¡DIVISION BY ZERO!!!
- 9. ±á³¾³¾¡
- 14. outage.needs? :logs => [¡°no¡±, ¡°²õ³ó¾±³Ù¡±±Õ
- 19. overview
- 21. 127.0.0.1 - - [05/Feb/2012:17:11:55 +0000] "GET / HTTP/1.1" 200 140 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19"
- 22. { "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": ¡°Mozilla/5.0 (Windows¡" }
- 24. Parse as JSON!
- 25. ?
- 26. [¡°05/Feb/2012:17:11:55¡±,¡°web.access¡±,{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": ¡°Mozilla/5.0 (Windows¡" }]
- 28. Fluentd Design Patterns (credit: @tagomoris¡¯s talk at RedDotRubyConf)
- 29. Real World Use Cases
- 31. ? Automate monitoring! ? App and System metrics ? JSON everywhere
- 33. ? 2000+ node ? ~1B events/day ? Forwarder- Aggregator
- 34. Demo
- 35. Fluentd Treasure Data Twitter API #NED #ARG Postgres + d3.js Collect Raw Tweets Tokenize + Count serve + visualize
- 36. { ¡°install¡±: ¡°gem install fluentd¡±, ¡°website¡±: ¡°www.fluentd.org¡±, ¡°github¡± : ¡°fluent/fluentd¡±, ¡°twitter¡±: ¡°@fluentd¡± }