????? ?? AWS ????? ???? ?? ?? :: ??? :: AWS Summit Seoul 2016
24 likes7,369 views
5? 17? ??COEX?? ?? AWS Summit Seoul 2016?? ??? ???? ?????? ???? "????? ?? AWS ????? ???? ?? ??" ???????.
![?????? ??
???? ??? ??
?? ??? ?? ? ???
???? ???[ ]](https://image.slidesharecdn.com/aws-160519074731/85/AWS-AWS-Summit-Seoul-2016-47-320.jpg)
![?? Solution? ???????
? ???? ??
? ??? ?? Script? ????? ??? ??
? Algorithm? 70%? Query? ??
? ??? ??? ?? Log? ???? S3? ??
? Redshift? ????? ??
[ ]?????? ???? ???? ???
????? ??? SQL?? ???
????? ???](https://image.slidesharecdn.com/aws-160519074731/85/AWS-AWS-Summit-Seoul-2016-51-320.jpg)
![[AWS Builders] AWS? ???? ???? ???](https://cdn.slidesharecdn.com/ss_thumbnails/awsbuildersaws101webinarcloudcomputingchoelkang-190305081301-thumbnail.jpg?width=560&fit=bounds)
![[AWS Migration Workshop] ??????? AWS? ??? ?????? ??](https://cdn.slidesharecdn.com/ss_thumbnails/fy19q2awsmigrationworkshop-yoojeongchoi-190619044742-thumbnail.jpg?width=560&fit=bounds)
More Related Content
What's hot (20)
Similar to ????? ?? AWS ????? ???? ?? ?? :: ??? :: AWS Summit Seoul 2016 (20)
More from Amazon Web Services Korea (20)
![[D3T1S01] Gen AI? ?? Amazon Aurora ?? ?? ??](https://cdn.slidesharecdn.com/ss_thumbnails/d3t1s01genaiamazonaurora-240702042912-516e67f4-thumbnail.jpg?width=560&fit=bounds)
![[D3T1S06] Neptune Analytics with Vector Similarity Search](https://cdn.slidesharecdn.com/ss_thumbnails/d3t1s06neptuneanalyticsvectorsilimliaritysearch-240702042912-94c41309-thumbnail.jpg?width=560&fit=bounds)
![[D3T1S03] Amazon DynamoDB design puzzlers](https://cdn.slidesharecdn.com/ss_thumbnails/d3t1s03amazondynamodbdesignpuzzlers-240702042912-ad6df881-thumbnail.jpg?width=560&fit=bounds)
![[D3T1S04] Aurora PostgreSQL performance monitoring and troubleshooting by use...](https://cdn.slidesharecdn.com/ss_thumbnails/d3t1s04aurorapostgresqlperformancemonitoringandtroubleshooting-240702042912-5df626e3-thumbnail.jpg?width=560&fit=bounds)
![[D3T1S07] AWS S3 - ???? ???? ?????? ????](https://cdn.slidesharecdn.com/ss_thumbnails/d3t1s07-240702042911-cb134cd6-thumbnail.jpg?width=560&fit=bounds)
![[D3T1S05] Aurora ?? ?? ????? ???? ??? ?? ??? ?? ????](https://cdn.slidesharecdn.com/ss_thumbnails/d3t1s05aurora-240702042911-c7f3f22d-thumbnail.jpg?width=560&fit=bounds)
![[D3T1S02] Aurora Limitless Database Introduction](https://cdn.slidesharecdn.com/ss_thumbnails/d3t1s02auroralimitlessdatabaseintroduction-240702042911-cb5552b7-thumbnail.jpg?width=560&fit=bounds)
![[D3T2S01] Amazon Aurora MySQL ??? ?? ????? ? Amazon B/G Deployments ??](https://cdn.slidesharecdn.com/ss_thumbnails/d3t2s01amazonaurorabluegreendeployment-240702042226-3ae36566-thumbnail.jpg?width=560&fit=bounds)
![[D3T2S03] Data&AI Roadshow 2024 - Amazon DocumentDB ??](https://cdn.slidesharecdn.com/ss_thumbnails/d3t2s03documentdbhandson-240702042224-047bbc2c-thumbnail.jpg?width=560&fit=bounds)
![[Keynote] ???? AWS ?????? ???? - ???: ???, Korea Database SA Manager, WWSO, A...](https://cdn.slidesharecdn.com/ss_thumbnails/d3s01aws-230704014400-3eeae447-thumbnail.jpg?width=560&fit=bounds)
????? ?? AWS ????? ???? ?? ?? :: ??? :: AWS Summit Seoul 2016
- 1. ? 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ??? | ???? ???? 2016? 5? 17? ????? ?? AWS ????? ???? ?? ??
- 2. ?? ′ ???? ?? & ??? ?? ′ ?? ???? ′ ?? ??? ???? ??? ′ ?? ′ ???? ′ ?? ??(MangoPlate) ′ ??? ??
- 5. Amazon Glacier S3 DynamoDB RDS EMR Amazon Redshift Data Pipeline Amazon Kinesis CloudSearch Kinesis-enabled app Lambda ML SQS ElastiCache DynamoDB Streams ???? ???
- 6. ?~? ?? ????? ???? ?? ??? ???? ???? ?? ????
- 7. ???? ?? ＾??? ??￣? ?? ? Data ★ Store ★ Process ★ Answers ??? ??? ??? ?? ? Data structure, latency, throughput, access patterns ?? ???? ???? ?? ? Immutable (append-only) log, batch/speed/serving layer AWS ??? ??? ?? ? No/low admin ??? ?? ?? ? Big data 』 Big cost
- 8. ???? ??? ????´ ?? ?? ?? ?? / ??? ?? ??(??) ??? ??
- 9. ??
- 10. ???? ?? ???? ? Database reads & writes (OLTP) ? Cache ?? ? Logs ? Streams ?? ? Log files (/var/log) ? Log collectors & frameworks ??? ? Log records ? Sensors & IoT data A iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Amazon Glacier Amazon ElastiCache SearchSQLNoSQLCacheFileStorage Transactional Data File Data Stream Data Mobile Apps Search Data Database File Storage Search ?? ?? LoggingIoTApplications Stream Storage
- 11. ??
- 12. ??? ???? A iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon ElastiCache SearchSQLNoSQLCacheStreamStorageFileStorage Transactional Data File Data Stream Data Mobile Apps Search Data Database File Storage Search ?? ?? LoggingIoTApplications ┨
- 13. ??? ???? ??? AWS ??? ??? ? Amazon Kinesis ★ streams ? Amazon DynamoDB Streams ★ table + streams ? Amazon SQS ★ queue ? Amazon SNS ★ pub/sub Do-It-Yourself ? Apache Kafka ★ stream
- 14. ???? ???? ?? ???? ?? ??? ???? ?? ???? ?? ?? ???? ???? ???? ?? 4 4 3 3 2 2 1 1 4 3 2 1 4 3 2 1 4 3 2 1 4 3 2 1 4 4 3 3 2 2 1 1 Shard 1 / Partition 1 Shard 2 / Partition 2 Consumer 1 Count of Red = 4 Count of Violet = 4 Consumer 2 Count of Blue = 4 Count of Green = 4 DynamoDB Stream Kinesis Stream Kafka Topic ??? ????
- 15. Queues & Pub/Sub ? ? ??? ? ???/???? ?? ? ???? ?? ? ??? ???? ?? ? No ??? ?? ? No ??? ?? for Amazon SQS ? Amazon SNS ? ??? ? ?? ??(Lambda) ??? ?? ?? ? No ???? ???? Consumers Producers Producers Amazon SNS Amazon SQS queue topic function ? AWS Lambda Amazon SQS queue Subscriber
- 16. Amazon Kinesis DynamoDB Streams Amazon SQS Amazon SNS Kafka Managed Yes Yes Yes No Ordering Yes Yes No Yes Delivery at-least-once exactly-once at-least-once at-least-once Lifetime 7 days 24 hours 14 days Configurable Replication 3 AZ 3 AZ 3 AZ Configurable Throughput No Limit No Limit No Limit ~ Nodes Parallel Clients Yes Yes No (SQS) Yes MapReduce Yes Yes No Yes Record size 1MB 400KB 256KB Configurable Cost Low Higher(table cost) Low-Medium Low (+admin) ?? ??? ????? ???? ????
- 17. ?? ???? Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon ElastiCache SearchSQLNoSQLCacheStreamStorageFileStorage Transactional Data File Data Stream Data Search Data Database Search ?? A iOS Android Web Apps Logstash Mobile Apps ?? LoggingIoTApplications ┨
- 18. ? Amazon S3? ????? ???? ? ????? ???? ????? ??(Spark, Hive, Presto, etc.) ? ????? ?? ??? ????? ??? (HDFS? ??) ? Amazon EC2 ?? ????? ???? ?? ???? ?? ?? ? ??? ???? ?? ??(Spark, Hive, Presto) ????? ??? ?? ? ???? ?? ??? ? 99.999999999%? ???? ?? ?? ? ? ??? C AZ ?? ?? ? ????? ??? ??-???? (Standard, IA, Amazon Glacier) ? ?? C SSL, client/server-side encryption at rest ? ??? ? ?? ?? ??? C ? ??? ?? ??
- 19. ? Amazon S3? ????? ???? ? ????? ???? ????? ??(Spark, Hive, Presto, etc.) ? ????? ?? ??? ????? ??? (HDFS? ??) ? Amazon EC2 ?? ????? ???? ?? ???? ?? ?? ? ??? ???? ?? ??(Spark, Hive, Presto) ????? ??? ?? ? ???? ?? ??? ? 99.999999999%? ???? ?? ?? ? ? ??? C AZ ?? ?? ? ????? ??? ??-???? (Standard, IA, Amazon Glacier) ? ?? C SSL, client/server-side encryption at rest ? ??? ? ?? ?? ??? C ? ??? ?? ??
- 20. ? ?? ?? ????(hot) ???? HDFS ?? ? ?? ???? ???? Amazon S3 Standard ?? ? ??? ???? ???? Amazon S3 Standard C IA ?? ? ?? ???? ??(cold) ???? Amazon Glacier ???? ???? S3? HDFS, Amazon Glacier? ??´
- 21. ?????? + ?? ?? A iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon ElastiCache SearchSQLNoSQLCacheStreamStorageFileStorage Transactional Data File Data Stream Data Mobile Apps Search Data ?? ?? LoggingIoTApplications ┨
- 22. Database + Search Tier ?????? + ?? ?? Anti-pattern
- 23. Data Tier Search Amazon Elasticsearch Service Amazon CloudSearch Cache Redis Memcached SQL Amazon Aurora MySQL MariaDB PostgreSQL Oracle SQL Server NoSQL Cassandra Amazon DynamoDB HBase MongoDB Database + Search Tier ?? ?? C ??? ?? ??? ?? ??
- 24. ???? ?
- 25. ??? ??? ?? ?? ?? ?? What to use? Put/Get (Key, Value) Cache, NoSQL Simple relationships ★ 1:N, M:N NoSQL Cross table joins, transaction, SQL SQL Faceting, Search Search ??? ?? What to use? Fixed schema SQL, NoSQL Schema-free (JSON) NoSQL, Search (Key, Value) Cache, NoSQL
- 26. ??? / ?? ???
- 27. Hot Warm Cold ??? ?? MBCGB GBCTB PB ??? ?? BCKB KBCMB KBCTB ???? ms ms, sec min, hrs ??? LowCHigh High Very High ?? ?? Very High High Low ??/GB $$-$ $-?? ? Hot Data Warm Data Cold Data ??? / ?? ??: Hot, Warm, Cold
- 28. Cache SQL Request Rate High Low Cost/GB High Low Latency Low High Data Volume Low High Glacier Structure NoSQL Hot Data Warm Data Cold Data Low High Search
- 29. Amazon ElastiCache Amazon DynamoDB Amazon Aurora Amazon Elasticsearch Amazon EMR (HDFS) Amazon S3 Amazon Glacier Average latency ms ms ms, sec ms,sec sec,min,hrs ms,sec,min (~ size) hrs Data volume GB GBCTBs (no limit) GBCTB (64 TB Max) GBCTB GBCPB (~nodes) MBCPB (no limit) GBCPB (no limit) Item size B-KB KB (400 KB max) KB (64 KB) KB (1 MB max) MB-GB KB-GB (5 TB max) GB (40 TB max) Request rate High - Very High Very High (no limit) High High Low C Very High Low C Very High (no limit) Very Low Storage cost GB/month $$ ?? ?? ?? ? ? ?/10 Durability Low - Moderate Very High Very High High High Very High Very High Hot Data Warm Data Cold Data Hot Data Warm Data Cold Data ?? ??? ???? ???
- 30. ??
- 31. ??A iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon Redshift Impala Pig Amazon ML Amazon Kinesis AWS Lambda AmazonElasticMapReduce Amazon ElastiCache SearchSQLNoSQLCache StreamProcessingBatchInteractive Logging StreamStorage IoTApplications FileStorage Hot Cold War m Hot Hot ML Transactional Data File Data Stream Data Mobile Apps Search Data ?? ?? ?? ┨ ┨ Streaming
- 32. ?? / ?? ???? ?? ??? ??? ??? ????, ??? ????, ??? ????? ???? ?? ???? ???? ??, ??, ??, ??? ??? ?? ???? ?? ??? ???? ★ ??? ??(Interactive Analytics) ??/??/?? ??? ★ ?? ??(Batch Analytics) ??/?? ??, 1 ? ?? ★ ??? ??(Real-time Analytics) ?? ??, ?? ?? ★ ?? ??(Machine learning)
- 33. ??? ?? ???? ??? (warm/cold) ??? ???? ? ? ?? ?: ?? ??? ????
- 34. ?? ?? ???? ??? (warm/cold) ??? ???? ? ? ?? ? ?? ?? ?: ??, ??, ?? ??? ??
- 35. ??? ?? ?? ??? Hot ? ???? ?? ??? ???? ?? ?? ?? (? ??? ?? ? ?) ??? (???) ? ??? ????? ??? ??? ?? ? ?: ??/?? ?? ? ??? (???? ??) ? ??? ???? ???? ??? ?? ? ??? ?? ? ?: 1 ? ??
- 36. ?? ??? ?? ?? ?? ??(ML)? ????? ????? ????? ?? ?? ??? ? ?? ??? ?? ?? ?? ????: ?? ?? ○ ＾teach￣ program - Classification ○ Is this transaction fraud? (Yes/No) - Regression ○ Customer Life-time value? ?? ?? ○ let it learn by itself - Clustering ○ Market Segmentation
- 37. ?? ?? ? Mahout, Spark ML, Amazon ML ??? ?? ? Amazon Redshift, Presto, Impala, Spark ?? ?? ? MapReduce, Hive, Pig, Spark ??? ?? ? Micro-batch: Spark Streaming, KCL, Hive, Pig ? Real-time: Storm, AWS Lambda, KCL Amazon Redshift Impala Pig Amazon Machine Learning Amazon Kinesis AWS Lambda AmazonElasticMapReduce StreamProcessingBatchInteractiveML ?? Streaming ?? ??? ?????
- 38. Spark Streaming Apache Storm Amazon Kinesis Client Library AWS Lambda Amazon EMR (Hive, Pig) Scale / Throughput ~ Nodes ~ Nodes ~ Nodes Automatic ~ Nodes Batch or Real- time Real-time Real-time Real-time Real-time Batch Manageability Yes (Amazon EMR) Do it yourself Amazon EC2 + Auto Scaling AWS managed Yes (Amazon EMR) Fault Tolerance Single AZ Configurable Multi-AZ Multi-AZ Single AZ Programming languages Java, Python, Scala Any language via Thrift Java, via MultiLangDaemon ( .Net, Python, Ruby, Node.js) Node.js,Java, Python Hive, Pig, Streaming languages High ?? ??? ?? ??? ???? ????
- 39. Amazon Redshift Impala Presto Spark Hive Query Latency Low Low Low Low Medium (Tez) C High (MapReduce) Durability High High High High High Data Volume 1.6 PB Max ~Nodes ~Nodes ~Nodes ~Nodes Managed Yes Yes (EMR) Yes (EMR) Yes (EMR) Yes (EMR) Storage Native HDFS / S3A* HDFS / S3 HDFS / S3 HDFS / S3 SQL Compatibility High Medium High Low (SparkSQL) Medium (HQL) HighMedium ?? ??? ?? ??? ???? ????
- 41. ?? / ???
- 42. ?? ?? ? ??? IDE ?????? & API Consume Analysis&VisualizationNotebooks Predictions Apps & APIs IDE ?? ?? ??ETL Business users Data Scientist, Developers Amazon QuickSight ??
- 43. ?? ????
- 44. ?? ?? ?? ?? A iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon Redshift Impala Pig Amazon ML Amazon Kinesis AWS Lambda AmazonElasticMapReduce Amazon ElastiCache SearchSQLNoSQLCache StreamProcessingBatchInteractive Logging StreamStorage IoTApplications FileStorage Analysis&Visualization Hot Cold Warm Hot Slow Hot ML Fast Fast Transactional Data File Data Stream Data Notebooks Predictions Apps & APIs Mobile Apps IDE Search Data ETL Streaming Amazon QuickSight
- 45. ? 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ??? | CTO 2016? 5? 17? Customer Story MangoPlate
- 46. Redshift ??? ?????? ?? ? ?? ???
- 47. ?????? ?? ???? ??? ?? ?? ??? ?? ? ??? ???? ???[ ]
- 48. ??????? ?? ??????: 250?+ MAU : 180?+ ?? ????: 2000?+ ??
- 49. ???????? ?? ?? ?? ? ?? ?? ? Recommendation Engine ? Restaurant to Restaurant Similarity ?? ? User to User Similarity ?? ? User to Restaurant Similarity ?? ? Fraud Detection ? Fake review/user Identification ? User/Review/Picture Scoring ? Restaurant Rating ? User Behavior ?? ? Web/App user mapping ? Retention queries ? User segmentation/testing
- 50. ??? ??? ?? Pain Point? ? ??? ??? ?? ?? ????? ?? ??? ?? ?? ?? ? ?? ? Rating ???? ???? ?? Query ? ???? ? ???? ?? ???? ?? ??? ??
- 51. ?? Solution? ??????? ? ???? ?? ? ??? ?? Script? ????? ??? ?? ? Algorithm? 70%? Query? ?? ? ??? ??? ?? Log? ???? S3? ?? ? Redshift? ????? ?? [ ]?????? ???? ???? ??? ????? ??? SQL?? ??? ????? ???
- 52. Redshift? ?? ? ?? Petabyte ???? Scale ?? ? ?? ?? ?? ? ??? ?? ? dc1.large? ?? ? 20???? ?? ?? ? ?? SQL ?? ? ??? Analytics Function ??
- 53. ???? AWS ? Redshift?? ?? ???? ??? ? VM instances ? MySQL ? Redis AWS Seoul Region ? EC2 ? RDS ? Elastic Cache ? VPC ? Route53 ? S3 ? SNS ? SQS ? Redshift Migration Consulting & Technical Support AWS Tokyo Region ? S3 ? SNS ? SQS ? Redshift + 2015 2016
- 54. ??????? Architecture ??? ??? ?? ???? ??? AWS S3 AWS Redshift . . . copy table SQL DB records to raw file AWS EC2 Analytics Visualization & Consume Collection Store AWS RDS AWS EC2 AWS EC2 AWS RDS
- 55. ??? ??????? ? Algorithm queries ? Restaurant Similarity: 600 ? > 80 ? (7.5?) ? Restaurant/User Recommendation: 720 ? > 80 ? (9?) ? Retention queries ? Base Table: 1200 ? > 60 ? (20?) ? Main: 2400 ? > 200 ? (12?) ?? ?? ??
- 56. ??? ??????? ? Analytic function(window function) ? ?? ? median, dense_rank ? ntile, stddev_samp/stddev_pop ? JSON function?? ???? ?? ????? ?? ? Json_extract_path_text ? json_extract_array_element_text ?? Query?? ???
- 57. Managed Service by ????? ! https://www.mangoplate.com/career
- 58. ??? ??
- 59. ??? ?? ′ ??? ??(Interactive Analytics) : ??? ???? ′ ?? ??(Batch Analytics) : ??/??/?? ??? ′ ??? ??(Real-time Analytics) : ??/?? ?? 1 ? ?? ′ ?? ??(Machine learning) : ?? ??, ?? ??
- 60. ?? ?? ???? ??? ???? Store Process Store Process process store ＾??? ??￣ ??
- 61. ??? ?? ??????(?? ???)? ??? ??? ????? ??/?? ?? process store Amazon Kinesis AWS Lambda Amazon S3 Amazon DynamoDB Amazon Kinesis S3 Connector
- 62. Amazon Kinesis AWS Lambda Amazon S3 Amazon DynamoDB Hive SparkStorm Amazon Kinesis S3 Connector process store ?? ?????(KCL, Storm, Hive, Spark, etc.)? ??? ??? ????? ?? ??
- 63. Spark Streaming Apache Storm AWS Lambda KCL Amazon Redshift Spark Impala Presto Hive Amazon Redshift Hive Spark Presto Impala Amazon Kinesis Apache Kafka Amazon DynamoDB Amazon S3data Hot Cold ??? ?? ?????? Low High Answers Amazon EMR (HDFS) Hive Native KCL AWS Lambda Batch ??? ?? vs. ?? ????
- 64. ??? ?? Producer Apache Kafka KCL AWS Lambda Spark Streaming Apache Storm Amazon SNS Amazon ML Notifications Amazon ElastiCache (Redis) Amazon DynamoDB Amazon RDS Amazon ES Alert App state Real-time Prediction KPI process store DynamoDB Streams Amazon Kinesis
- 65. ??? & ?? ?? Producer Amazon S3 Amazon EMR Hive Pig Spark Amazon ML process store Consume Amazon Redshift Amazon EMR Presto Impala Spark ?? ??? Batch Prediction Real-time Prediction
- 66. Batch Layer Amazon Kinesis DATA process store Amazon Kinesis S3 Connector Amazon S3 A p p l i c a t i o n s Amazon Redshift Amazon EMR Presto Hive Pig Spark ANSWER Speed Layer ANSWER Serving Layer Amazon ElastiCache Amazon DynamoDB Amazon RDS Amazon ES ANSWER Amazon ML KCL AWS Lambda Spark Streaming Storm ?? ????
- 67. ?? ＾??? ??￣? ?? ? Data ★ Store ★ Process ★ Answers ??? ??? ??? ?? ? Data structure, latency, throughput, access patterns ?? ???? ???? ?? ? Immutable (append-only) log, batch/speed/serving layer AWS ??? ??? ?? ? No/low admin ??? ?? ?? ? Big data 』 Big cost
- 68. ???? ???? ?????! https://www.awssummit.co.kr ??? ???? ?????, ?? ?? ??? ?????, ??? ???? ????. #AWSSummit ????? ?? ???? ???? ?? ??? ?????. ?? ?? ? ?? ???? AWS Korea ?? ?? ??? ? ??? ?????.