AWS Segment XO Group Joint webinar
0 likes457 views
This document summarizes a presentation about loading and analyzing behavioral data in Amazon Redshift. It includes the following: - An overview of the Amazon Redshift architecture including leader nodes, compute nodes, columnar storage, data compression, and parallel query execution. - A demonstration of Segment SQL which allows querying user, page, and event data from Segment in an Amazon Redshift database. - Three case studies from XO Group on how they used Segment SQL and Amazon Redshift to perform membership analysis, analyze share functionality, and gain insights for version support.
AWS Segment XO Group Joint webinar
- 1. Loading and Analyzing Behavioral Data in Amazon Redshift PresentedbySegment,AWS&XOGroup Inc. March3,2015
- 3. Today¡¯s agenda ?? AmazonRedshiftArchitecture ?? SegmentSQLQuickDemo ?? XOGroup¡¯sStory
- 6. Amazon Redshift is Easy to Use ?? Provisioninminutes ?? Monitorqueryperformance ?? Pointandclickresize ?? Builtinsecurity ?? Automaticbackups
- 7. Amazon Redshift Architecture ?? LeaderNode ¨C? SQLendpoint ¨C? Storesmetadata ¨C? Coordinatesqueryexecution ?? ComputeNodes ¨C? Local,columnarstorage ¨C? Executequeriesinparallel ¨C? Load,backup,restoreviaAmazonS3 ¨C? ParallelloadfromAmazonDynamoDB,Amazon EMR,AmazonS3,HDFS/SSH ?? Twohardwareplatforms ¨C? Optimizedfordataprocessing ¨C? DW1:HDD;scalefrom2TBto1.6PB ¨C? DW2:SSD;scalefrom160GBto256TB 10 GigE (HPC) Ingestion Backup Restore SQL Clients/BI Tools 128GB RAM 16TB disk 16 cores Amazon S3 JDBC/ODBC 128GB RAM 16TB disk 16 coresCompute Node 128GB RAM 16TB disk 16 coresCompute Node 128GB RAM 16TB disk 16 coresCompute Node Leader Node
- 8. ?? Columnstorage ?? Datacompression ?? Zonemaps ?? Direct-attachedstorage ?? Withrowstorageyoudo unnecessaryI/O ?? Togettotalamount,youhavetoread everything ID Age State Amount 123 20 CA 500 345 25 WA 250 678 40 FL 125 957 37 WA 375 Amazon Redshift Dramatically Reduces I/O
- 9. ?? Withcolumnstorage,youonly readthedatayouneed ID Age State Amount 123 20 CA 500 345 25 WA 250 678 40 FL 125 957 37 WA 375 ?? Columnstorage ?? Datacompression ?? Zonemaps ?? Direct-attachedstorage Amazon Redshift Dramatically Reduces I/O
- 10. analyze compression listing; Table | Column | Encoding ---------+----------------+---------- listing | listid | delta listing | sellerid | delta32k listing | eventid | delta32k listing | dateid | bytedict listing | numtickets | bytedict listing | priceperticket | delta32k listing | totalprice | mostly32 listing | listtime | raw ?? Columnstorage ?? Datacompression ?? Zonemaps ?? Direct-attachedstorage ?? COPYcompressesautomatically ?? Youcananalyzeandoverride ?? Moreperformance,lesscost Amazon Redshift Dramatically Reduces I/O
- 11. ?? Columnstorage ?? Datacompression ?? Zonemaps ?? Direct-attachedstorage 10 | 13 | 14 | 26 |¡ ¡ | 100 | 245 | 324 375 | 393 | 417¡ ¡ 512 | 549 | 623 637 | 712 | 809 ¡ ¡ | 834 | 921 | 959 10 324 375 623 637 959 ?? Tracktheminimumandmaximum valueforeachblock ?? Skipoverblocksthatdon¡¯tcontain relevantdata Amazon Redshift Dramatically Reduces I/O
- 12. ?? Columnstorage ?? Datacompression ?? Zonemaps ?? Direct-attachedstorage 128 GB RAM 16 cores 16 TB disk DW.HS1.8XL: ?? >2GB/sscanrate ?? Optimizedfordataprocessing ?? Highdiskdensity 16 GB RAM 2 cores 2 TB disk DW.HS1.XL: Amazon Redshift Dramatically Reduces I/O
- 13. ?? Query ?? Load ?? Backup/Restore ?? Resize Amazon Redshift Parallelizes and Distributes Everything
- 14. Amazon S3/DynamoDB 128GB RAM 16TB disk 16 coresCompute Node 128GB RAM 16TB disk 16 coresCompute Node 128GB RAM 16TB disk 16 coresCompute Node ?? Query ?? Load ?? Backup/Restore ?? Resize ?? ParallelloadfromAmazonDynamoDB,AmazonEMR, AmazonS3,HDFS/SSH ?? Kinesisintegration ?? Dataautomaticallydistributedandsortedaccordingto DDL ?? Scaleslinearlywithnumber ofnodes Amazon Redshift Parallelizes and Distributes Everything
- 15. Amazon S3 128GB RAM 16TB disk 16 coresCompute Node 128GB RAM 16TB disk 16 coresCompute Node 128GB RAM 16TB disk 16 coresCompute Node ?? Query ?? Load ?? Backup/Restore ?? Resize ?? BackupstoAmazonS3areautomatic,continuous andincremental ?? Backupyourclustertoasecondregion ?? Configurablesystemsnapshotretentionperiod;take usersnapshotson-demand ?? Streamingrestoresenableyoutoresumequerying faster Amazon Redshift Parallelizes and Distributes Everything
- 16. SQL Clients/BI Tools 128GB RAM 48TB disk 16 cores Comput e Node 128GB RAM 48TB disk 16 cores Comput e Node 128GB RAM 48TB disk 16 cores Comput e Node 128GB RAM 48TB disk 16 cores Leader Node 128GB RAM 48TB disk 16 cores Comput e Node 128GB RAM 48TB disk 16 cores Comput e Node 128GB RAM 48TB disk 16 cores Comput e Node 128GB RAM 48TB disk 16 cores Comput e Node 128GB RAM 48TB disk 16 cores Leader Node ?? Query ?? Load ?? Backup/Restore ?? Resize ?? Add/removenodesorchangenodetypewhile remainingonline ?? Provisionanewclusterandcopydatainparallelfrom nodetonode ?? OnlychargedforsourceclusteruntilSQLendpoint hasautomaticallybeenswitchedoverviaDNS Amazon Redshift Parallelizes and Distributes Everything
- 17. ?? SSLtosecuredataintransit ?? Encryptiontosecuredataatrest ¨C? AES-256;hardwareaccelerated ¨C? AllblocksondisksandinAmazonS3 encrypted ¨C? HSM/CloudHSM ?? Nodirectaccesstocomputenodes ?? AmazonVPCsupport 10 GigE (HPC) Ingestion Backup Restore SQL Clients/BI Tools 128GB RAM 16TB disk 16 cores 128GB RAM 16TB disk 16 cores 128GB RAM 16TB disk 16 cores 128GB RAM 16TB disk 16 cores Amazon S3 / Amazon DynamoDB Customer VPC Internal Security Group JDBC/ODBC Leader Node Compute Node Compute Node Compute Node Amazon Redshift Has Security Built In
- 18. Segment SQL
- 21. analytics.track({ 'Added Product' , { name: 'Monopoly: 3rd Edition', price: 18.99, quantity: 3 }); 1 2 3 4 5 ReplaceallyourtagsandtrackingwithasingleAPI.
- 27. XO Group Inc.
- 28. XO Group Inc.
- 30. XO Group Inc. + Segment Individualproductteams wantedisolatedaccessto theirownanalytics. Segment+Mixpanel+ Customer.io+Optimizely+ Uservoice
- 31. XO Group Inc. + Segment Stillneededasolutionto connectSegmentdatafrom multipleproductsand platformsintoasingleview. SegmentSQL+ ModeAnalytics
- 33. Membership Analysis Existingdatabasedidn¡¯tgiveany contextintohowauserbecamea memberintheapp. Couldn¡¯tanalyzewhichfeatures weredrivingmembership upgrades,marketingattribution, orcross-productpollination. SELECT userID, source, reason FROM xogrp.signed_up 1 2 3 4
- 34. Isolated Views
- 35. Membership Analysis App A App B
- 37. Share Functionality Facebook&Twitterdrovethe mostinboundtraffic. So,weguessedthey¡¯dalsobethe mostpopularsharingoptions. SELECT title, shareOption, quantity FROM xogrp.shared_article 1 2 3 4
- 39. More considerate ¡°Share¡± options SMSadressfrom desktopormobileweb browser. Oneclickemailyourselfthe detailsofavenue.