ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo

AWS??? ?????????? Daily Report ???????? : ?? ???? ???? ????

?
?
Changje Jeong
Changje Jeong

15.04.21 ????? ??? AWS Global Summit Seoul?? ??? "Amazon AWS ? ???? ?? ??, ????? ? ???? ????" ???? ???????.

1 of 47
AWS? ???? Daily Report ??? ?? ???? ???? ???? 1 ????? ??? changje@devsisters.com
AWS? ???? Daily Report ??? 2 ??? ?? ? 7,000? ?? ???? ? ?? 1,000? DAU ? ??, ??, ?? ? 10??? ???? 1? ? 38?? Top10 ????
AWS? ???? Daily Report ??? 3 200~400GB 100TB ???? ?? ??? 1??? ???? ? ??? ??
AWS? ???? Daily Report ??? 0. ?? ?? 4 Amazon S3 Game Servers Redis Elastic Search Kibana . . . Logstash . . .
AWS? ???? Daily Report ??? 5 ???? ???? ?? ??? ??? ??? ??..? 200~400GB 100TB ???? ?? ??? 1??? ???? ? ??? ??
AWS? ???? Daily Report ??? 6 ??? ?????? ???! ???? ???? ?? ??? ??? ??? ??..? 200~400GB 100TB ???? ?? ??? 1??? ???? ? ??? ??
AWS? ???? Daily Report ??? 1. ?? ??? ?? ???? 7 BREADCRUMB
AWS? ???? Daily Report ??? 1. ??????? ???? - ?? ? ??? 1? ? ??? ????! ¡ú ???? ????? ?? 8
AWS? ???? Daily Report ??? - ?? ? ??? 1? ? ??? ????! ¡ú ???? ????? ?? - ?? ?? ? ?? ??? ??? ???! ¡ú Action? MemberID? Index 9 1. ??????? ????
AWS? ???? Daily Report ??? 1. ??????? ???? - ?? ? ??? 1? ? ??? ????! ¡ú ???? ????? ?? - ?? ?? ? ?? ??? ??? ???! ¡ú Action? MemberID? Index - ?? ??????? ???! ¡ú ?? ???? ????! 10
AWS? ???? Daily Report ??? 1. ?? ??? 11 Logstash Amazon S3 20??? ??? 2015-04-21T00.01.txt 2015-04-21T00.21.txt 2015-04-21T00.41.txt 2015-04-21T01.01.txt 2015-04-21T01.21.txt 2015-04-21T01.41.txt ¡­ 2015-04-21T22.41.txt 2015-04-21T23.01.txt 2015-04-21T23.21.txt 2015-04-21T23.41.txt 2015-04-22T00.01.txt
AWS? ???? Daily Report ??? 1. ?? ??? 12 Logstash Amazon S3 Amazon S3 2015-04-21T00.01.txt 2015-04-21T00.21.txt 2015-04-21T00.41.txt 2015-04-21T01.01.txt 2015-04-21T01.21.txt 2015-04-21T01.41.txt ¡­ 2015-04-21T22.41.txt 2015-04-21T23.01.txt 2015-04-21T23.21.txt 2015-04-21T23.41.txt 2015-04-22T00.01.txt 00.01.json 00.01.txt.sz 00.21.json 00.21.txt.sz ¡­ 23.41.json 23.41.txt.sz 00.01.json 00.01.txt.sz 20??? ??? ??? ??
AWS? ???? Daily Report ??? 1. ?? ??? 13 Logstash Amazon S3 Amazon S3 Amazon S3 ?? ?? 2015-04-21T00.01.txt 2015-04-21T00.21.txt 2015-04-21T00.41.txt 2015-04-21T01.01.txt 2015-04-21T01.21.txt 2015-04-21T01.41.txt ¡­ 2015-04-21T22.41.txt 2015-04-21T23.01.txt 2015-04-21T23.21.txt 2015-04-21T23.41.txt 2015-04-22T00.01.txt index.json gamelog.sz 00.01.json 00.01.txt.sz 00.21.json 00.21.txt.sz ¡­ 23.41.json 23.41.txt.sz 00.01.json 00.01.txt.sz ??? ?? 20??? ???
AWS? ???? Daily Report ??? 1. ?? ??? 14 gamelog.sz ? ??? ?? ? Action, MemberID? ?? ? ?? ?? ? ?? ??? ??? ??? ?? ? ??? ? ?? ???? ????? ??? ? ??? MemberID? ??? ?? ?? ?? ? Block ?? Snappy ?? ? Networt Latency ¡ý ? Snappy : CPU ??? ¡ý ? Splittable (Snappy Framing Format)
AWS? ???? Daily Report ??? 1. ?? ??? 15 ¡­}, { "key_?rst": ¡°<Action> <Reversed MemberID> <Time> <Hash>¡±, "key_last": ¡°<Action> <Reversed MemberID> <Time> <Hash>¡±, "offset": 0, "length": <Length> }, ¡­ ? gamelog? ???? ?? Metadata index.json
AWS? ???? Daily Report ??? 1. ?? ??? 16 [{ "key_?rst": ¡°member/connect 00001¡±, "key_last": ¡°member/connect 000024¡±, "offset": 0, ¡°length": 31243 }, { "key_?rst": ¡°member/connect 000025¡±, "key_last": ¡°member/disconnect 0003¡±, "offset": 31244, "length": 42347 }, { "key_?rst": ¡°member/disconnect 00031¡±, "key_last": ¡°game/startGame 00012¡±, "offset": 73591, "length": 49323 }] index.jsongamelog.sz member/connect member/connect member/connect member/disconnect game/startGame member/disconnect 10000 240000 250000 3000 13000 00012
AWS? ???? Daily Report ??? BREADCRUMB 1. ?? ?? 17 Amazon S3 EMR index.json gamelog.sz output by 2015-04-21T0 0.01.txt 00.01.json 00.01.txt.sz https://github.com/twitter/scalding
AWS? ???? Daily Report ??? 1. ?? - ActiveUser 18https://github.com/twitter/scalding class ActiveUserCount(args : Args) extends Job(args) {? CookierunLog("member/connect", args("sampling"),? args("timeStart"), args("timeEnd"),? Tuple1("memberSeq"))? .groupBy(¡®memberSeq){ _.take(1) }? .groupAll{ _.size }? .write(Csv(args("output")))? } 1/1000 ???? 1??? ?? ???? 15?
AWS? ???? Daily Report ??? 19 ?? ???? ???... ??? ???... ??? ???? ?????? ???? ??...
AWS? ???? Daily Report ??? 20 ?? ???? ???... ??? ???... ??? ???? ?????? ???? ??... ???? ??? ????!
AWS? ???? Daily Report ??? 2. ???? ??? ???? 21
AWS? ???? Daily Report ??? ? ?? ?? ?? ??? ???? ???! ? ???? ?? ??? ?? ???! ? ??? ???? ?? ???! 22 2. ???? ??? ???? - ??
AWS? ???? Daily Report ??? BREADCRUMB 2-1. ?? ?? ?? ??? ???? ???! 23 Amazon S3 EMR Powered by LogsOutput
AWS? ???? Daily Report ??? BREADCRUMB 2-1. ?? ?? ?? ??? ???? ???! 24 Amazon S3 EMR SES boto Powered by LogsOutput Result cron Send via E-Mail Docker RDS INSERT / SELECT
AWS? ???? Daily Report ??? 2-1. ?? ?? ?? ??? ???? ???! 25
AWS? ???? Daily Report ??? 2-1. ?? ?? ?? ??? ???? ???! 26 Upload JAR S3 JAR BREADCRUMB
AWS? ???? Daily Report ??? 2-1. ?? ?? ?? ??? ???? ???! 27 EMR run_job?ow BREADCRUMB S3 JAR
AWS? ???? Daily Report ??? 2-1. ?? ?? ?? ??? ???? ???! 28 EMR BREADCRUMB BREADCRUMB S3 add_job?ow_steps Logs Output JAR
AWS? ???? Daily Report ??? BREADCRUMB 2-1. ?? ?? ?? ??? ???? ???! 29 Amazon S3 EMR SES boto Powered by LogsOutput Result cron Send via E-Mail Docker RDS INSERT / SELECT
AWS? ???? Daily Report ??? 2-1. ?? ?? ?? ??? ???? ???! 30
AWS? ???? Daily Report ??? EMR - ?? ?? ????? ??? ????? ????? ?? - Spot Instance? ???? ??? ??? ????! - S3? Direct Access ?? HDFS? ??? ???? ??? ?? 31 Amazon S3 - ??? ???? ?? ??? ??? - ??? ? ?? ???? - Offset-Range? ???? ???? ??? ??? ???? ?? ??
AWS? ???? Daily Report ??? 2. ???? ??? ???? - ?? ? ?? ?? ?? ??? ???? ???! ???? ?? ??? ?????! ? ??? ???? ?? ???! 32
AWS? ???? Daily Report ??? 2-2. ???? ?? ??? ?????! 33 Amazon S3 EMR SES boto Powered by LogsOutput Result Send via E-Mail RDS INSERT / SELECT HTTPS Web cron Docker SELECT BREADCRUMB
AWS? ???? Daily Report ??? 2-2. ???? ?? ??? ?????! 34
AWS? ???? Daily Report ??? 2. ???? ??? ???? - ?? ? ?? ?? ?? ??? ???? ???! ? ???? ?? ??? ?? ???! ?? ???? ?? ???! 35
AWS? ???? Daily Report ??? 36 2-3. ??? ?? ???? ?? ???! ?? ??? = ??? ? ???
AWS? ???? Daily Report ??? 37 2-3. ??? ?? ???? ?? ???! ?? ??? = ??? ? ??? ?? ?? ?? ????? ??..?
AWS? ???? Daily Report ??? 38 2-3. ??? ?? ???? ?? ???! ?? ??? = ??? ? ??? ?? ?? ?? ????? ??..?
AWS? ???? Daily Report ??? 39 2-3. ??? ?? ???? ?? ???! View? ????! ?????? ?? ?? ?
AWS? ???? Daily Report ??? 40 View 2-3. ??? ?? ???? ?? ???! Documents??? MapReduce? ?? ??? ??? ???? ???? ?? Mongo? Aggregate, MapReduce? ??
AWS? ???? Daily Report ??? 41 View 2-3. ??? ?? ???? ?? ???! ?? ??? ???? ?? ??? ??? ???? ??! MapReduce[ ??? ??, ??? ??, ??? ?? ] ??? ?? : 3 ??? ?? : 2 ??? ?? : 2 ??? ?? : 1 [ ??? ??, ??? ??, ??? ?? ] [ ??? ??, ??? ??, ??? ?? ] Documents??? MapReduce? ?? ??? ??? ???? ???? ?? Mongo? Aggregate, MapReduce? ??
AWS? ???? Daily Report ??? 2. ???? ??? ???? - ??! 42 Amazon S3 EMR SES boto Powered by LogsOutput Result View Query Send via E-Mail View RDS INSERT / SELECT HTTPS Web cron Docker BREADCRUMB SELECT
AWS? ???? Daily Report ??? 2. ???? ??? ???? - ??! ? ?? ?? ??? ?? ??? ???? ???! ? ???? ?? ??? ?????! ? ??? ???? ?? ???! 43
AWS? ???? Daily Report ??? 3. ?? - ??????? Breadcrumb ? ??? ??? ???? ???? ?? ??? ? Scalding ???? ?? ?? ?? ? ???? ???? ?????? ??? ?? ???? ?? ???? ?? ? ????? ??? ???? ??? ?? ¡ú ?? ??? ??? ??? ??? ?? ?? 44
AWS? ???? Daily Report ??? 3. ?? - Daily Report Pancake ? ??? ?? ???? ????? ??? ? ??? ???? ??? ??? ??? ????? ?? ? ?? ??? ????? ??? ?? ? ?? ?? ?? ? ??? ??? ??? ????? ?? ? Breadcrumb? ??? ????? ??? ??? ?? ? 45
AWS? ???? Daily Report ??? 46 http://www.devsisters.com/jobs/ ??? ??? ??? ??? ????
AWS? ???? Daily Report ??? 4747 ????? ??? changje@devsisters.com

More Related Content

AWS??? ?????????? Daily Report ???????? : ?? ???? ???? ????

  • 1. AWS? ???? Daily Report ??? ?? ???? ???? ???? 1 ????? ??? changje@devsisters.com
  • 2. AWS? ???? Daily Report ??? 2 ??? ?? ? 7,000? ?? ???? ? ?? 1,000? DAU ? ??, ??, ?? ? 10??? ???? 1? ? 38?? Top10 ????
  • 3. AWS? ???? Daily Report ??? 3 200~400GB 100TB ???? ?? ??? 1??? ???? ? ??? ??
  • 4. AWS? ???? Daily Report ??? 0. ?? ?? 4 Amazon S3 Game Servers Redis Elastic Search Kibana . . . Logstash . . .
  • 5. AWS? ???? Daily Report ??? 5 ???? ???? ?? ??? ??? ??? ??..? 200~400GB 100TB ???? ?? ??? 1??? ???? ? ??? ??
  • 6. AWS? ???? Daily Report ??? 6 ??? ?????? ???! ???? ???? ?? ??? ??? ??? ??..? 200~400GB 100TB ???? ?? ??? 1??? ???? ? ??? ??
  • 7. AWS? ???? Daily Report ??? 1. ?? ??? ?? ???? 7 BREADCRUMB
  • 8. AWS? ???? Daily Report ??? 1. ??????? ???? - ?? ? ??? 1? ? ??? ????! ¡ú ???? ????? ?? 8
  • 9. AWS? ???? Daily Report ??? - ?? ? ??? 1? ? ??? ????! ¡ú ???? ????? ?? - ?? ?? ? ?? ??? ??? ???! ¡ú Action? MemberID? Index 9 1. ??????? ????
  • 10. AWS? ???? Daily Report ??? 1. ??????? ???? - ?? ? ??? 1? ? ??? ????! ¡ú ???? ????? ?? - ?? ?? ? ?? ??? ??? ???! ¡ú Action? MemberID? Index - ?? ??????? ???! ¡ú ?? ???? ????! 10
  • 11. AWS? ???? Daily Report ??? 1. ?? ??? 11 Logstash Amazon S3 20??? ??? 2015-04-21T00.01.txt 2015-04-21T00.21.txt 2015-04-21T00.41.txt 2015-04-21T01.01.txt 2015-04-21T01.21.txt 2015-04-21T01.41.txt ¡­ 2015-04-21T22.41.txt 2015-04-21T23.01.txt 2015-04-21T23.21.txt 2015-04-21T23.41.txt 2015-04-22T00.01.txt
  • 12. AWS? ???? Daily Report ??? 1. ?? ??? 12 Logstash Amazon S3 Amazon S3 2015-04-21T00.01.txt 2015-04-21T00.21.txt 2015-04-21T00.41.txt 2015-04-21T01.01.txt 2015-04-21T01.21.txt 2015-04-21T01.41.txt ¡­ 2015-04-21T22.41.txt 2015-04-21T23.01.txt 2015-04-21T23.21.txt 2015-04-21T23.41.txt 2015-04-22T00.01.txt 00.01.json 00.01.txt.sz 00.21.json 00.21.txt.sz ¡­ 23.41.json 23.41.txt.sz 00.01.json 00.01.txt.sz 20??? ??? ??? ??
  • 13. AWS? ???? Daily Report ??? 1. ?? ??? 13 Logstash Amazon S3 Amazon S3 Amazon S3 ?? ?? 2015-04-21T00.01.txt 2015-04-21T00.21.txt 2015-04-21T00.41.txt 2015-04-21T01.01.txt 2015-04-21T01.21.txt 2015-04-21T01.41.txt ¡­ 2015-04-21T22.41.txt 2015-04-21T23.01.txt 2015-04-21T23.21.txt 2015-04-21T23.41.txt 2015-04-22T00.01.txt index.json gamelog.sz 00.01.json 00.01.txt.sz 00.21.json 00.21.txt.sz ¡­ 23.41.json 23.41.txt.sz 00.01.json 00.01.txt.sz ??? ?? 20??? ???
  • 14. AWS? ???? Daily Report ??? 1. ?? ??? 14 gamelog.sz ? ??? ?? ? Action, MemberID? ?? ? ?? ?? ? ?? ??? ??? ??? ?? ? ??? ? ?? ???? ????? ??? ? ??? MemberID? ??? ?? ?? ?? ? Block ?? Snappy ?? ? Networt Latency ¡ý ? Snappy : CPU ??? ¡ý ? Splittable (Snappy Framing Format)
  • 15. AWS? ???? Daily Report ??? 1. ?? ??? 15 ¡­}, { "key_?rst": ¡°<Action> <Reversed MemberID> <Time> <Hash>¡±, "key_last": ¡°<Action> <Reversed MemberID> <Time> <Hash>¡±, "offset": 0, "length": <Length> }, ¡­ ? gamelog? ???? ?? Metadata index.json
  • 16. AWS? ???? Daily Report ??? 1. ?? ??? 16 [{ "key_?rst": ¡°member/connect 00001¡±, "key_last": ¡°member/connect 000024¡±, "offset": 0, ¡°length": 31243 }, { "key_?rst": ¡°member/connect 000025¡±, "key_last": ¡°member/disconnect 0003¡±, "offset": 31244, "length": 42347 }, { "key_?rst": ¡°member/disconnect 00031¡±, "key_last": ¡°game/startGame 00012¡±, "offset": 73591, "length": 49323 }] index.jsongamelog.sz member/connect member/connect member/connect member/disconnect game/startGame member/disconnect 10000 240000 250000 3000 13000 00012
  • 17. AWS? ???? Daily Report ??? BREADCRUMB 1. ?? ?? 17 Amazon S3 EMR index.json gamelog.sz output by 2015-04-21T0 0.01.txt 00.01.json 00.01.txt.sz https://github.com/twitter/scalding
  • 18. AWS? ???? Daily Report ??? 1. ?? - ActiveUser 18https://github.com/twitter/scalding class ActiveUserCount(args : Args) extends Job(args) {? CookierunLog("member/connect", args("sampling"),? args("timeStart"), args("timeEnd"),? Tuple1("memberSeq"))? .groupBy(¡®memberSeq){ _.take(1) }? .groupAll{ _.size }? .write(Csv(args("output")))? } 1/1000 ???? 1??? ?? ???? 15?
  • 19. AWS? ???? Daily Report ??? 19 ?? ???? ???... ??? ???... ??? ???? ?????? ???? ??...
  • 20. AWS? ???? Daily Report ??? 20 ?? ???? ???... ??? ???... ??? ???? ?????? ???? ??... ???? ??? ????!
  • 21. AWS? ???? Daily Report ??? 2. ???? ??? ???? 21
  • 22. AWS? ???? Daily Report ??? ? ?? ?? ?? ??? ???? ???! ? ???? ?? ??? ?? ???! ? ??? ???? ?? ???! 22 2. ???? ??? ???? - ??
  • 23. AWS? ???? Daily Report ??? BREADCRUMB 2-1. ?? ?? ?? ??? ???? ???! 23 Amazon S3 EMR Powered by LogsOutput
  • 24. AWS? ???? Daily Report ??? BREADCRUMB 2-1. ?? ?? ?? ??? ???? ???! 24 Amazon S3 EMR SES boto Powered by LogsOutput Result cron Send via E-Mail Docker RDS INSERT / SELECT
  • 25. AWS? ???? Daily Report ??? 2-1. ?? ?? ?? ??? ???? ???! 25
  • 26. AWS? ???? Daily Report ??? 2-1. ?? ?? ?? ??? ???? ???! 26 Upload JAR S3 JAR BREADCRUMB
  • 27. AWS? ???? Daily Report ??? 2-1. ?? ?? ?? ??? ???? ???! 27 EMR run_job?ow BREADCRUMB S3 JAR
  • 28. AWS? ???? Daily Report ??? 2-1. ?? ?? ?? ??? ???? ???! 28 EMR BREADCRUMB BREADCRUMB S3 add_job?ow_steps Logs Output JAR
  • 29. AWS? ???? Daily Report ??? BREADCRUMB 2-1. ?? ?? ?? ??? ???? ???! 29 Amazon S3 EMR SES boto Powered by LogsOutput Result cron Send via E-Mail Docker RDS INSERT / SELECT
  • 30. AWS? ???? Daily Report ??? 2-1. ?? ?? ?? ??? ???? ???! 30
  • 31. AWS? ???? Daily Report ??? EMR - ?? ?? ????? ??? ????? ????? ?? - Spot Instance? ???? ??? ??? ????! - S3? Direct Access ?? HDFS? ??? ???? ??? ?? 31 Amazon S3 - ??? ???? ?? ??? ??? - ??? ? ?? ???? - Offset-Range? ???? ???? ??? ??? ???? ?? ??
  • 32. AWS? ???? Daily Report ??? 2. ???? ??? ???? - ?? ? ?? ?? ?? ??? ???? ???! ???? ?? ??? ?????! ? ??? ???? ?? ???! 32
  • 33. AWS? ???? Daily Report ??? 2-2. ???? ?? ??? ?????! 33 Amazon S3 EMR SES boto Powered by LogsOutput Result Send via E-Mail RDS INSERT / SELECT HTTPS Web cron Docker SELECT BREADCRUMB
  • 34. AWS? ???? Daily Report ??? 2-2. ???? ?? ??? ?????! 34
  • 35. AWS? ???? Daily Report ??? 2. ???? ??? ???? - ?? ? ?? ?? ?? ??? ???? ???! ? ???? ?? ??? ?? ???! ?? ???? ?? ???! 35
  • 36. AWS? ???? Daily Report ??? 36 2-3. ??? ?? ???? ?? ???! ?? ??? = ??? ? ???
  • 37. AWS? ???? Daily Report ??? 37 2-3. ??? ?? ???? ?? ???! ?? ??? = ??? ? ??? ?? ?? ?? ????? ??..?
  • 38. AWS? ???? Daily Report ??? 38 2-3. ??? ?? ???? ?? ???! ?? ??? = ??? ? ??? ?? ?? ?? ????? ??..?
  • 39. AWS? ???? Daily Report ??? 39 2-3. ??? ?? ???? ?? ???! View? ????! ?????? ?? ?? ?
  • 40. AWS? ???? Daily Report ??? 40 View 2-3. ??? ?? ???? ?? ???! Documents??? MapReduce? ?? ??? ??? ???? ???? ?? Mongo? Aggregate, MapReduce? ??
  • 41. AWS? ???? Daily Report ??? 41 View 2-3. ??? ?? ???? ?? ???! ?? ??? ???? ?? ??? ??? ???? ??! MapReduce[ ??? ??, ??? ??, ??? ?? ] ??? ?? : 3 ??? ?? : 2 ??? ?? : 2 ??? ?? : 1 [ ??? ??, ??? ??, ??? ?? ] [ ??? ??, ??? ??, ??? ?? ] Documents??? MapReduce? ?? ??? ??? ???? ???? ?? Mongo? Aggregate, MapReduce? ??
  • 42. AWS? ???? Daily Report ??? 2. ???? ??? ???? - ??! 42 Amazon S3 EMR SES boto Powered by LogsOutput Result View Query Send via E-Mail View RDS INSERT / SELECT HTTPS Web cron Docker BREADCRUMB SELECT
  • 43. AWS? ???? Daily Report ??? 2. ???? ??? ???? - ??! ? ?? ?? ??? ?? ??? ???? ???! ? ???? ?? ??? ?????! ? ??? ???? ?? ???! 43
  • 44. AWS? ???? Daily Report ??? 3. ?? - ??????? Breadcrumb ? ??? ??? ???? ???? ?? ??? ? Scalding ???? ?? ?? ?? ? ???? ???? ?????? ??? ?? ???? ?? ???? ?? ? ????? ??? ???? ??? ?? ¡ú ?? ??? ??? ??? ??? ?? ?? 44
  • 45. AWS? ???? Daily Report ??? 3. ?? - Daily Report Pancake ? ??? ?? ???? ????? ??? ? ??? ???? ??? ??? ??? ????? ?? ? ?? ??? ????? ??? ?? ? ?? ?? ?? ? ??? ??? ??? ????? ?? ? Breadcrumb? ??? ????? ??? ??? ?? ? 45
  • 46. AWS? ???? Daily Report ??? 46 http://www.devsisters.com/jobs/ ??? ??? ??? ??? ????
  • 47. AWS? ???? Daily Report ??? 4747 ????? ??? changje@devsisters.com