Big Data on AWS
3 likes382 views
This document provides an introduction to various AWS big data services including Kinesis, Kinesis Firehose, Kinesis Analytics, SQS, IoT, Data Pipeline, DynamoDB, EMR, Lambda, Redshift, and QuickSight. It describes the core concepts and components of each service. For services like Kinesis, Kinesis Analytics, and EMR it provides details on architecture, terminology, pricing models, and best practices. The document aims to give readers an overview of the AWS big data landscape and how the different services can be used together in a big data context.
1 of 84
Downloaded 13 times
![AWS KINESIS ANALYTICS
STREAMING SQL
Tumbling Window
[...] GROUP BY
FLOOR((SOURCE_SQL_STREAM_001.ROWTIME TIMESTAMP
1970-01-01 00:00:00) SECOND / 10 TO SECOND)
Sliding Window
SELECT AVG(change) OVER W1 as avg_change
FROM "SOURCE_SQL_STREAM_001"
WINDOW W1 AS (PARTITION BY ticker_symbol RANGE INTERVAL
'10' SECOND PRECEDING)](https://image.slidesharecdn.com/awsbigdatastack-170612103317/85/Big-Data-on-AWS-26-320.jpg)
Ad
Recommended
Aws Kinesis
Aws KinesisSzilveszter Moln叩r
油
This document provides an overview of AWS Kinesis and its components for streaming data. It discusses Kinesis Streams for ingesting and processing streaming data at scale. Kinesis Streams uses shards to provide throughput capacity. To ingest 10,000 records per second of 512 byte size each would require a Kinesis stream configured with 10 shards. Kinesis Firehose is for delivering streaming data to destinations like S3 or Redshift. Kinesis Analytics allows running SQL queries on streaming data and processing it in real-time. Big data on aws
Big data on awsSerkan zal
油
The Ankara Cloud Meetup in July 2016 covered Big Data on AWS, including insights into its concepts, storage, analytics, processing, and data flow capabilities. Key AWS services like S3, DynamoDB, Redshift, and EMR were discussed alongside scalability features and practical demonstrations for storing, searching, processing, and analyzing data. The session highlighted AWSs managed services for big data solutions, showcasing how to effectively leverage them for various use cases.20141021 AWS Cloud Taekwon - Big Data on AWS
20141021 AWS Cloud Taekwon - Big Data on AWSAmazon Web Services Korea
油
This document discusses big data solutions on AWS. It covers data generation, collection and storage using services like S3, DynamoDB and Kinesis. It also discusses analytics and computation using EMR, Redshift and real-time analytics with Kinesis. Finally it discusses collaboration and sharing insights using visualization tools and BI tools. Real-world examples of using these AWS services for big data are provided.Amazon Elastic Map Reduce - Ian Meyers
Amazon Elastic Map Reduce - Ian Meyershuguk
油
The document discusses Amazon Web Services (AWS) and its cloud computing offerings, focusing on the advantages of utility computing, scalability, and cost management. It details various services including Elastic MapReduce, Simple Storage Service, and databases like RDS and DynamoDB, along with their performance metrics and use cases. Additionally, the document highlights the benefits of using AWS for IT infrastructure, emphasizing agility, reduced capital expenditure, and global reach.Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)Emprovise
油
The document discusses various AWS services and their functionalities, including Amazon S3 for storage and parallelization, DynamoDB for data modeling, Amazon Redshift for data warehousing, and Lambda functions for serverless computing. It highlights best practices for performance optimization, architecture design, and data handling, as well as new features like zero-ETL integrations and event-driven architectures. Overall, it serves as a comprehensive guide for leveraging AWS services for scalable and efficient data management and processing.AWS Innovate: Build a Data Lake on AWS- Johnathon Meichtry
AWS Innovate: Build a Data Lake on AWS- Johnathon MeichtryAmazon Web Services Korea
油
This document discusses building a data lake on AWS. It describes using Amazon S3 for storage of structured, semi-structured, and unstructured data at scale. Amazon Kinesis is used for streaming ingest of data. A metadata catalogue using Amazon DynamoDB and AWS Lambda allows for data discovery and governance. IAM policies control access and encryption using AWS KMS provides security. APIs built using Amazon API Gateway provide programmatic access to the data lake resources.Get Value from Your Data
Get Value from Your DataDanilo Poccia
油
The document provides an extensive overview of AWS services like Amazon EMR, Amazon Redshift, Amazon Kinesis, and Amazon QuickSight, emphasizing their capabilities in data processing, analytics, and machine learning. It discusses the architecture, cost-saving tips, and how these services interact with structured and unstructured data. Additionally, it outlines the importance of real-time data and analytics for better business insights.Get Value From Your Data
Get Value From Your DataDanilo Poccia
油
The document provides an overview of various AWS services related to data analytics, including Amazon EMR for big data processing, Amazon Redshift for data warehousing, and Amazon Kinesis for real-time streaming data. It highlights the importance of choosing right data architectures and technologies to optimize costs, performance, and scalability. The document emphasizes the benefits of Amazon's services, such as pay-per-use pricing, ease of integration, and managing data effectively using cloud solutions.Data Analytics on AWS
Data Analytics on AWSDanilo Poccia
油
The document discusses data analytics on AWS. It describes how AWS services like Amazon S3, DynamoDB, Redshift, EMR, and Kinesis can be used to generate, store, analyze and share data at scale. It provides examples of how companies are using these services for tasks like processing millions of records per second and adding thousands of new records daily. The document emphasizes that AWS allows users to remove constraints on data analytics by providing elastic, scalable infrastructure without upfront costs.Big Data Day LA 2015 - The AWS Big Data Platform by Michael Limcaco of Amazon
Big Data Day LA 2015 - The AWS Big Data Platform by Michael Limcaco of AmazonData Con LA
油
The document outlines Amazon Web Services' (AWS) comprehensive big data platform, detailing its services for data collection, storage, processing, and analysis. It highlights the scalability, flexibility, and various applications of tools such as Amazon S3, Amazon Kinesis, and Amazon Redshift, which support real-time processing and machine learning capabilities. The global infrastructure and architecture enables businesses to efficiently manage vast amounts of data in different regions, ensuring high availability and security.AWS Floor 28 - Building Data lake on AWS
AWS Floor 28 - Building Data lake on AWSAdir Sharabi
油
The document presents an agenda for various technical sessions within the context of building data lakes on AWS, highlighting key topics, speakers, and the importance of managing diverse data sources. It discusses challenges in data analytics, the evolution of data lakes compared to traditional methods, and the integration of AWS services like Amazon S3 and Amazon Redshift for efficient data processing and storage. Additionally, it emphasizes the significance of a centralized data catalog and event-driven transformations for effective data management in modern analytics.Em tempo real: Ingest達o, processamento e analise de dados
Em tempo real: Ingest達o, processamento e analise de dadosAmazon Web Services LATAM
油
The document presents an overview of real-time data ingestion, processing, and analysis using Amazon Web Services (AWS) technology. It highlights various AWS services such as Amazon Kinesis, AWS Lambda, and Amazon Redshift, detailing their functionalities for capturing, analyzing, and storing streaming data efficiently and cost-effectively. The architecture and use cases illustrate how these services support applications requiring real-time analytics and machine learning capabilities.Builders' Day - Building Data Lakes for Analytics On AWS LC
Builders' Day - Building Data Lakes for Analytics On AWS LCAmazon Web Services LATAM
油
This document discusses big data analytics and architectural principles for building big data solutions. It covers collecting and storing data from various sources, processing and analyzing data using services like Amazon Kinesis, Redshift, EMR and Athena, and choosing the right tools based on factors like data structure, access patterns, and latency requirements. Key principles emphasized include building decoupled systems, leveraging managed services, using event-driven architectures, and focusing on cost efficiency.Make your data fly - Building data platform in AWS
Make your data fly - Building data platform in AWSKimmo Kantoj辰rvi
油
This document summarizes a presentation on building a data platform in AWS. It discusses the architectural evolution from on-premise data warehouses to cloud-based data lakes and platforms. It provides examples of using AWS services like EMR, Redshift, Airflow and visualization tools. It also covers best practices for data modeling, performance optimization, security and DevOps approaches.Aws architecture problems while being fancy
Aws architecture problems while being fancyGoran Kopevski
油
This document discusses various AWS services, including their benefits, common design patterns, and challenges. It provides an overview of services like EC2, S3, DynamoDB, API Gateway, Lambda, SQS, and Elasticsearch. While AWS makes infrastructure management easier, the document notes challenges like limited functionality of some newer services, inconsistent availability across regions, complexity of DynamoDB queries, and limitations of services like SQS message size. Custom Lambda functions are highlighted as a way to customize AWS architectures.谿場螳 AWS 碁碁(蟲襦,螳,蟲) - AWS 蠍磯 觜一危 覦覯 (蟾狩 襭讀 ろ)
谿場螳 AWS 碁碁(蟲襦,螳,蟲) - AWS 蠍磯 觜一危 覦覯 (蟾狩 襭讀 ろ)Amazon Web Services Korea
油
The document discusses AWS big data services and tools. It provides an overview of AWS building blocks for big data like Amazon S3, Kinesis, DynamoDB, Redshift and EMR. It covers topics like log data collection and storage using Kinesis, data analytics using services like Redshift and EMR, and collaboration and sharing of data. Generation, collection and storage of data, analytics and computation, and collaboration and sharing are highlighted as key aspects of a big data platform on AWS.Data Analysis - Journey Through the Cloud
Data Analysis - Journey Through the CloudIan Massingham
油
This document provides an overview of big data analytics options on AWS, including Amazon Redshift, Amazon Kinesis, Amazon Elastic MapReduce, Amazon DynamoDB, and applications running on Amazon EC2. It describes ideal usage patterns, performance, cost models, and scalability for each option. It also presents two example scenarios: an enterprise data warehouse using Redshift and capturing and analyzing sensor data using Kinesis and EMR. Additional resources are provided to help readers learn more about big data analytics on AWS.Big Data Architecture and Design Patterns
Big Data Architecture and Design PatternsJohn Yeung
油
The document presents an overview of big data architecture and design patterns using AWS services. It outlines key components such as data collection, storage, processing, and analysis, emphasizing the importance of using the right tools and architectural principles. The session includes practical demonstrations and highlights various AWS products for managing big data challenges effectively.AWS Cloud Experience CA: Bases de Datos en AWS: distintas necesidades, distin...
AWS Cloud Experience CA: Bases de Datos en AWS: distintas necesidades, distin...Amazon Web Services LATAM
油
This document discusses using purpose-built databases for applications in the cloud. It begins by introducing Daniel, the CIO of an online bookstore who is looking to improve his company's systems. The document then covers various database types and how they are suited to different data and usage patterns. It discusses Amazon's managed database services including DynamoDB, ElastiCache, Neptune, RDS, and Elasticsearch. Examples are given of companies like Airbnb using different databases for different purposes. The document advocates choosing the right database for the job. It profiles an example retail application that could use various Amazon databases.Architecting Data Lakes on AWS
Architecting Data Lakes on AWSSajith Appukuttan
油
This document discusses HiFX helping Malayala Manorama, one of the largest media conglomerates in India, build a data lake on AWS. Malayala Manorama faced challenges with data silos, lack of access to data for analysis, and poor data management. HiFX designed a solution connecting multiple data sources and repositories into a unified data pipeline on AWS. Raw data is stored in the data lake on S3 for low-cost storage. Spark on EMR is used for processing, with results stored in Redshift, Druid and DynamoDB. A custom app called Lens provides business users visualizations and insights. The benefits include improved user experience, campaign management, and faster product decisions based on data insightsBig data and Analytics on AWS
Big data and Analytics on AWS2nd Watch
油
The document provides an overview of big data concepts and Amazon Web Services (AWS) products for big data and analytics. It describes challenges of big data including unpredictable resource demand and job orchestration complexities. It then summarizes AWS products for data collection, storage, processing, analytics and machine learning. Specific examples are given using AWS services like Redshift, EMR, Kinesis and DynamoDB for scenarios like data warehousing, real-time streaming and Hadoop workloads. Core principles and common challenges of big data implementations on AWS are also outlined.2017 AWS DB Day | Amazon DynamoDB 觜, 螳 覦 蠏 蠍磯 螳
2017 AWS DB Day | Amazon DynamoDB 觜, 螳 覦 蠏 蠍磯 螳Amazon Web Services Korea
油
Amazon DynamoDB is a fully managed NoSQL database service that offers fast, scalable, and secure data handling, suitable for various data types and large volumes. It features high availability through multi-AZ replication, consistent performance, and options like DynamoDB Accelerator (DAX) for microsecond response times. Key benefits include ease of management, robust security, and flexible cost structures, making it ideal for applications with diverse and demanding data needs.Introduction to AWS Big Data
Introduction to AWS Big Data Omid Vahdaty
油
The document provides a comprehensive overview of AWS big data services, including the architecture of data storage, processing, and visualization solutions such as EMR, Redshift, DynamoDB, Kinesis, and associated tools. It covers key concepts such as ingestion types, technology recommendations, best practices, security measures, cost optimization, and various service comparisons for big data analytics and manipulation. Additionally, it discusses the roles of libraries and frameworks like Spark, Hive, and others in facilitating data operations in the AWS environment.Building Data Analytics pipelines in the cloud using serverless technology
Building Data Analytics pipelines in the cloud using serverless technologyDomino Data Lab
油
The document discusses building serverless data pipelines in the cloud, emphasizing serverless architectures that utilize third-party services for managing resources. It details various AWS serverless services like Lambda, Kinesis, and DynamoDB, highlighting their features, advantages, and use cases in big data analytics. Additionally, it introduces cloud assessment tools from Linux Academy to enhance AWS skills with tailored learning paths and real-time analytics.Cutting to the chase for Machine Learning Analytics Ecosystem & AWS Lake Form...
Cutting to the chase for Machine Learning Analytics Ecosystem & AWS Lake Form...AWS Riyadh User Group
油
The document discusses AWS Lake Formation, designed to simplify building and managing data lakes by automating data ingestion, cleansing, and cataloging processes. It provides benefits such as enhanced security, access control, and seamless integration with other AWS services, enabling users to derive insights from structured and unstructured data efficiently. The document also highlights customer enthusiasm for AWS Lake Formation's ability to reduce operational burdens and ensure compliance.Re:Invent 2018 Database Announcements
Re:Invent 2018 Database AnnouncementsSteven Ensslen
油
Steven Ensslen of Clarity Cloudworks provides an overview of several new and updated AWS database services:
1. DynamoDB now supports transactions, making it suitable for more complex workloads.
2. DynamoDB has a new on-demand pricing model that saves the effort of capacity planning and the cost of excess capacity, but is more expensive per request.
3. Aurora can now be set up across multiple regions for disaster recovery and high availability purposes.
4. RDS now supports running on VMWare hardware within a customer's data center for outsourced database administration and bursting to AWS.
5. Several new database preview services are introduced: MSK for Apache KafkaAWS Analytics Immersion Day - Build BI System from Scratch (Day1, Day2 Full V...
AWS Analytics Immersion Day - Build BI System from Scratch (Day1, Day2 Full V...Sungmin Kim
油
The document discusses the architecture and implementation of a Business Intelligence (BI) system using AWS services such as Amazon QuickSight, Kinesis Data Firehose, and Amazon S3. It emphasizes key considerations for data management including scalability, cost optimization, and data processing efficiency, while also outlining best practices for building decoupled systems and leveraging serverless solutions. Additionally, the document touches on advanced analytics capabilities such as machine learning and real-time data processing for insights and recommendations.多Qui辿n es Amazon Web Services?
多Qui辿n es Amazon Web Services?Software Guru
油
The document outlines AWS's growth and extensive capabilities, highlighting a $14 billion revenue run rate with 47% year-over-year growth as of Q4 2016. Key features include a robust cloud infrastructure, real-time data processing capabilities through Amazon Kinesis, and various managed services for big data analytics. It emphasizes AWS's flexibility, low cost, and scalability in handling vast amounts of data for diverse business needs.AWS VPC, ELB, Route53 and CloudFront
AWS VPC, ELB, Route53 and CloudFrontSzilveszter Moln叩r
油
This document discusses VPC, ELB, Route53, and CloudFront in AWS. It describes that VPC allows users to create their own private networks and manage subnets, ACLs, and elastic IPs. ELB distributes traffic to multiple EC2 instances and handles SSL termination and health checks. Route53 provides DNS services with options for geography-based routing, load balancing, and integration with ELB health checks. CloudFront is a CDN that provides low latency access to static content through its global edge locations. The document poses the question of how these AWS services work together.Introduction 2 to aws and storage options
Introduction 2 to aws and storage optionsSzilveszter Moln叩r
油
The document provides an overview of Amazon Web Services (AWS) storage options. It discusses Elastic Block Store (EBS) which provides block-level storage volumes for EC2 instances; instance store volumes which are temporary storage attached to an EC2 host; Amazon Simple Storage Service (S3) which provides scalable object storage; and Amazon Glacier which provides low-cost archival storage. For each service, it outlines their key features and characteristics as well as ideal and non-ideal usage patterns. The document concludes with a brief recap of the main differences between EBS, S3, and Glacier storage services.More Related Content
Similar to Big Data on AWS (20)
Data Analytics on AWS
Data Analytics on AWSDanilo Poccia
油
The document discusses data analytics on AWS. It describes how AWS services like Amazon S3, DynamoDB, Redshift, EMR, and Kinesis can be used to generate, store, analyze and share data at scale. It provides examples of how companies are using these services for tasks like processing millions of records per second and adding thousands of new records daily. The document emphasizes that AWS allows users to remove constraints on data analytics by providing elastic, scalable infrastructure without upfront costs.Big Data Day LA 2015 - The AWS Big Data Platform by Michael Limcaco of Amazon
Big Data Day LA 2015 - The AWS Big Data Platform by Michael Limcaco of AmazonData Con LA
油
The document outlines Amazon Web Services' (AWS) comprehensive big data platform, detailing its services for data collection, storage, processing, and analysis. It highlights the scalability, flexibility, and various applications of tools such as Amazon S3, Amazon Kinesis, and Amazon Redshift, which support real-time processing and machine learning capabilities. The global infrastructure and architecture enables businesses to efficiently manage vast amounts of data in different regions, ensuring high availability and security.AWS Floor 28 - Building Data lake on AWS
AWS Floor 28 - Building Data lake on AWSAdir Sharabi
油
The document presents an agenda for various technical sessions within the context of building data lakes on AWS, highlighting key topics, speakers, and the importance of managing diverse data sources. It discusses challenges in data analytics, the evolution of data lakes compared to traditional methods, and the integration of AWS services like Amazon S3 and Amazon Redshift for efficient data processing and storage. Additionally, it emphasizes the significance of a centralized data catalog and event-driven transformations for effective data management in modern analytics.Em tempo real: Ingest達o, processamento e analise de dados
Em tempo real: Ingest達o, processamento e analise de dadosAmazon Web Services LATAM
油
The document presents an overview of real-time data ingestion, processing, and analysis using Amazon Web Services (AWS) technology. It highlights various AWS services such as Amazon Kinesis, AWS Lambda, and Amazon Redshift, detailing their functionalities for capturing, analyzing, and storing streaming data efficiently and cost-effectively. The architecture and use cases illustrate how these services support applications requiring real-time analytics and machine learning capabilities.Builders' Day - Building Data Lakes for Analytics On AWS LC
Builders' Day - Building Data Lakes for Analytics On AWS LCAmazon Web Services LATAM
油
This document discusses big data analytics and architectural principles for building big data solutions. It covers collecting and storing data from various sources, processing and analyzing data using services like Amazon Kinesis, Redshift, EMR and Athena, and choosing the right tools based on factors like data structure, access patterns, and latency requirements. Key principles emphasized include building decoupled systems, leveraging managed services, using event-driven architectures, and focusing on cost efficiency.Make your data fly - Building data platform in AWS
Make your data fly - Building data platform in AWSKimmo Kantoj辰rvi
油
This document summarizes a presentation on building a data platform in AWS. It discusses the architectural evolution from on-premise data warehouses to cloud-based data lakes and platforms. It provides examples of using AWS services like EMR, Redshift, Airflow and visualization tools. It also covers best practices for data modeling, performance optimization, security and DevOps approaches.Aws architecture problems while being fancy
Aws architecture problems while being fancyGoran Kopevski
油
This document discusses various AWS services, including their benefits, common design patterns, and challenges. It provides an overview of services like EC2, S3, DynamoDB, API Gateway, Lambda, SQS, and Elasticsearch. While AWS makes infrastructure management easier, the document notes challenges like limited functionality of some newer services, inconsistent availability across regions, complexity of DynamoDB queries, and limitations of services like SQS message size. Custom Lambda functions are highlighted as a way to customize AWS architectures.谿場螳 AWS 碁碁(蟲襦,螳,蟲) - AWS 蠍磯 觜一危 覦覯 (蟾狩 襭讀 ろ)
谿場螳 AWS 碁碁(蟲襦,螳,蟲) - AWS 蠍磯 觜一危 覦覯 (蟾狩 襭讀 ろ)Amazon Web Services Korea
油
The document discusses AWS big data services and tools. It provides an overview of AWS building blocks for big data like Amazon S3, Kinesis, DynamoDB, Redshift and EMR. It covers topics like log data collection and storage using Kinesis, data analytics using services like Redshift and EMR, and collaboration and sharing of data. Generation, collection and storage of data, analytics and computation, and collaboration and sharing are highlighted as key aspects of a big data platform on AWS.Data Analysis - Journey Through the Cloud
Data Analysis - Journey Through the CloudIan Massingham
油
This document provides an overview of big data analytics options on AWS, including Amazon Redshift, Amazon Kinesis, Amazon Elastic MapReduce, Amazon DynamoDB, and applications running on Amazon EC2. It describes ideal usage patterns, performance, cost models, and scalability for each option. It also presents two example scenarios: an enterprise data warehouse using Redshift and capturing and analyzing sensor data using Kinesis and EMR. Additional resources are provided to help readers learn more about big data analytics on AWS.Big Data Architecture and Design Patterns
Big Data Architecture and Design PatternsJohn Yeung
油
The document presents an overview of big data architecture and design patterns using AWS services. It outlines key components such as data collection, storage, processing, and analysis, emphasizing the importance of using the right tools and architectural principles. The session includes practical demonstrations and highlights various AWS products for managing big data challenges effectively.AWS Cloud Experience CA: Bases de Datos en AWS: distintas necesidades, distin...
AWS Cloud Experience CA: Bases de Datos en AWS: distintas necesidades, distin...Amazon Web Services LATAM
油
This document discusses using purpose-built databases for applications in the cloud. It begins by introducing Daniel, the CIO of an online bookstore who is looking to improve his company's systems. The document then covers various database types and how they are suited to different data and usage patterns. It discusses Amazon's managed database services including DynamoDB, ElastiCache, Neptune, RDS, and Elasticsearch. Examples are given of companies like Airbnb using different databases for different purposes. The document advocates choosing the right database for the job. It profiles an example retail application that could use various Amazon databases.Architecting Data Lakes on AWS
Architecting Data Lakes on AWSSajith Appukuttan
油
This document discusses HiFX helping Malayala Manorama, one of the largest media conglomerates in India, build a data lake on AWS. Malayala Manorama faced challenges with data silos, lack of access to data for analysis, and poor data management. HiFX designed a solution connecting multiple data sources and repositories into a unified data pipeline on AWS. Raw data is stored in the data lake on S3 for low-cost storage. Spark on EMR is used for processing, with results stored in Redshift, Druid and DynamoDB. A custom app called Lens provides business users visualizations and insights. The benefits include improved user experience, campaign management, and faster product decisions based on data insightsBig data and Analytics on AWS
Big data and Analytics on AWS2nd Watch
油
The document provides an overview of big data concepts and Amazon Web Services (AWS) products for big data and analytics. It describes challenges of big data including unpredictable resource demand and job orchestration complexities. It then summarizes AWS products for data collection, storage, processing, analytics and machine learning. Specific examples are given using AWS services like Redshift, EMR, Kinesis and DynamoDB for scenarios like data warehousing, real-time streaming and Hadoop workloads. Core principles and common challenges of big data implementations on AWS are also outlined.2017 AWS DB Day | Amazon DynamoDB 觜, 螳 覦 蠏 蠍磯 螳
2017 AWS DB Day | Amazon DynamoDB 觜, 螳 覦 蠏 蠍磯 螳Amazon Web Services Korea
油
Amazon DynamoDB is a fully managed NoSQL database service that offers fast, scalable, and secure data handling, suitable for various data types and large volumes. It features high availability through multi-AZ replication, consistent performance, and options like DynamoDB Accelerator (DAX) for microsecond response times. Key benefits include ease of management, robust security, and flexible cost structures, making it ideal for applications with diverse and demanding data needs.Introduction to AWS Big Data
Introduction to AWS Big Data Omid Vahdaty
油
The document provides a comprehensive overview of AWS big data services, including the architecture of data storage, processing, and visualization solutions such as EMR, Redshift, DynamoDB, Kinesis, and associated tools. It covers key concepts such as ingestion types, technology recommendations, best practices, security measures, cost optimization, and various service comparisons for big data analytics and manipulation. Additionally, it discusses the roles of libraries and frameworks like Spark, Hive, and others in facilitating data operations in the AWS environment.Building Data Analytics pipelines in the cloud using serverless technology
Building Data Analytics pipelines in the cloud using serverless technologyDomino Data Lab
油
The document discusses building serverless data pipelines in the cloud, emphasizing serverless architectures that utilize third-party services for managing resources. It details various AWS serverless services like Lambda, Kinesis, and DynamoDB, highlighting their features, advantages, and use cases in big data analytics. Additionally, it introduces cloud assessment tools from Linux Academy to enhance AWS skills with tailored learning paths and real-time analytics.Cutting to the chase for Machine Learning Analytics Ecosystem & AWS Lake Form...
Cutting to the chase for Machine Learning Analytics Ecosystem & AWS Lake Form...AWS Riyadh User Group
油
The document discusses AWS Lake Formation, designed to simplify building and managing data lakes by automating data ingestion, cleansing, and cataloging processes. It provides benefits such as enhanced security, access control, and seamless integration with other AWS services, enabling users to derive insights from structured and unstructured data efficiently. The document also highlights customer enthusiasm for AWS Lake Formation's ability to reduce operational burdens and ensure compliance.Re:Invent 2018 Database Announcements
Re:Invent 2018 Database AnnouncementsSteven Ensslen
油
Steven Ensslen of Clarity Cloudworks provides an overview of several new and updated AWS database services:
1. DynamoDB now supports transactions, making it suitable for more complex workloads.
2. DynamoDB has a new on-demand pricing model that saves the effort of capacity planning and the cost of excess capacity, but is more expensive per request.
3. Aurora can now be set up across multiple regions for disaster recovery and high availability purposes.
4. RDS now supports running on VMWare hardware within a customer's data center for outsourced database administration and bursting to AWS.
5. Several new database preview services are introduced: MSK for Apache KafkaAWS Analytics Immersion Day - Build BI System from Scratch (Day1, Day2 Full V...
AWS Analytics Immersion Day - Build BI System from Scratch (Day1, Day2 Full V...Sungmin Kim
油
The document discusses the architecture and implementation of a Business Intelligence (BI) system using AWS services such as Amazon QuickSight, Kinesis Data Firehose, and Amazon S3. It emphasizes key considerations for data management including scalability, cost optimization, and data processing efficiency, while also outlining best practices for building decoupled systems and leveraging serverless solutions. Additionally, the document touches on advanced analytics capabilities such as machine learning and real-time data processing for insights and recommendations.多Qui辿n es Amazon Web Services?
多Qui辿n es Amazon Web Services?Software Guru
油
The document outlines AWS's growth and extensive capabilities, highlighting a $14 billion revenue run rate with 47% year-over-year growth as of Q4 2016. Key features include a robust cloud infrastructure, real-time data processing capabilities through Amazon Kinesis, and various managed services for big data analytics. It emphasizes AWS's flexibility, low cost, and scalability in handling vast amounts of data for diverse business needs.AWS Cloud Experience CA: Bases de Datos en AWS: distintas necesidades, distin...
AWS Cloud Experience CA: Bases de Datos en AWS: distintas necesidades, distin...Amazon Web Services LATAM
油
Cutting to the chase for Machine Learning Analytics Ecosystem & AWS Lake Form...
Cutting to the chase for Machine Learning Analytics Ecosystem & AWS Lake Form...AWS Riyadh User Group
油
More from Szilveszter Moln叩r (7)
AWS VPC, ELB, Route53 and CloudFront
AWS VPC, ELB, Route53 and CloudFrontSzilveszter Moln叩r
油
This document discusses VPC, ELB, Route53, and CloudFront in AWS. It describes that VPC allows users to create their own private networks and manage subnets, ACLs, and elastic IPs. ELB distributes traffic to multiple EC2 instances and handles SSL termination and health checks. Route53 provides DNS services with options for geography-based routing, load balancing, and integration with ELB health checks. CloudFront is a CDN that provides low latency access to static content through its global edge locations. The document poses the question of how these AWS services work together.Introduction 2 to aws and storage options
Introduction 2 to aws and storage optionsSzilveszter Moln叩r
油
The document provides an overview of Amazon Web Services (AWS) storage options. It discusses Elastic Block Store (EBS) which provides block-level storage volumes for EC2 instances; instance store volumes which are temporary storage attached to an EC2 host; Amazon Simple Storage Service (S3) which provides scalable object storage; and Amazon Glacier which provides low-cost archival storage. For each service, it outlines their key features and characteristics as well as ideal and non-ideal usage patterns. The document concludes with a brief recap of the main differences between EBS, S3, and Glacier storage services.Introduction to AWS
Introduction to AWSSzilveszter Moln叩r
油
This document provides an introduction and overview of Amazon Web Services (AWS). It discusses what the cloud is and how AWS infrastructure works with regions and availability zones. It then focuses on explaining Amazon Elastic Compute Cloud (EC2), describing what EC2 is, the terminology used, how resources are shared, and the differences between local disks vs Elastic Block Store storage. The document concludes with a thank you and mentions future sessions will provide more in-depth coverage of specific AWS services.Lambda architecture
Lambda architectureSzilveszter Moln叩r
油
The Lambda Architecture is a data processing architecture designed to handle large volumes of data by separating the data flow into batch, serving and speed layers. The batch layer computes views over all available data but has high latency. The serving layer serves queries using pre-computed batch views but cannot answer queries in real-time. The speed layer computes real-time views incrementally from new data and answers queries with low latency. Together these layers are able to provide robust, scalable and low-latency query capabilities over massive datasets.Introduction to Regression Analysis
Introduction to Regression AnalysisSzilveszter Moln叩r
油
This document introduces regression analysis through an example of analyzing the relationship between hotel prices and distance from the city center. It shows histograms and scatterplots of the hotel data, revealing a negative relationship between price and distance. Loess and linear regression models are fit to quantify this relationship, with the linear regression indicating that for every 1km further from the city center, hotel prices decrease by an average of $2.60. The document explains that linear regression finds the best fitting straight line to describe how the mean or expected value of the target variable (price) changes with the predictor variable (distance).2015 Mar 12 - IoT Conference, Moscow - Beyond Fitness Trackers: Discovering H...
2015 Mar 12 - IoT Conference, Moscow - Beyond Fitness Trackers: Discovering H...Szilveszter Moln叩r
油
The document discusses the significance of wearable technology, which enhances quality of life through devices worn on the body. It outlines the growth of the wearable device market and highlights various types of wearables, their history, and the importance of data in healthcare. Additionally, it mentions developer resources for creating compatible applications.MongoDB + Node.JS + EPAM ROAD
MongoDB + Node.JS + EPAM ROADSzilveszter Moln叩r
油
The document discusses the development of a backend and iOS client using MongoDB, Node.js, and the EPAM Road framework. It covers key topics such as architecture, server setup, and the use of Express for routing and Mongoose for data modeling. Additionally, it provides demo code snippets and resources for further exploration.Ad
Recently uploaded (20)
Folding Cheat Sheet # 9 - List Unfolding as the Computational Dual of ...
Folding Cheat Sheet # 9 - List Unfolding as the Computational Dual of ...Philip Schwarz
油
For newer version see https://fpilluminated.org/deck/264
In this deck we look at the following:
* How unfolding lists is the computational dual of folding lists
* Different variants of the function for unfolding lists
* How they relate to the iterate functionOpenChain Webinar - AboutCode - Practical Compliance in One Stack Licensing...
OpenChain Webinar - AboutCode - Practical Compliance in One Stack Licensing...Shane Coughlan
油
OpenChain Webinar - AboutCode - Practical Compliance in One Stack Licensing, Vulnerabilities, and MoreBest MLM Compensation Plans for Network Marketing Success in 2025
Best MLM Compensation Plans for Network Marketing Success in 2025LETSCMS Pvt. Ltd.
油
Discover the top MLM compensation plans including Unilevel, Binary, Matrix, Board, and Australian Plans. Learn how to choose the best plan for your business growth with expert insights from MLM Trees. Explore hybrid models, payout strategies, and earning potential.
Learn more: https://www.mlmtrees.com/mlm-plans/
Decipher SEO Solutions for your startup needs.
Decipher SEO Solutions for your startup needs.mathai2
油
A solution deck that gives you an idea of how you can use Decipher SEO to target keywords, build authority and generate high ranking content.
With features like images to product you can create a E-commerce pipeline that is optimized to help your store rank.
With integrations with shopify, woocommerce and wordpress theres a seamless way get your content to your website or storefront.
View more at decipherseo.comBest Practice for LLM Serving in the Cloud
Best Practice for LLM Serving in the CloudAlluxio, Inc.
油
Alluxio Webinar
June 17, 2025
For more Alluxio Events: https://www.alluxio.io/events/
Speaker:
Nilesh Agarwal (Co-founder & CTO @ Inferless)
Nilesh Agarwal, co-founder & CTO at Inferless, shares insights on accelerating LLM inference in the cloud using Alluxio, tackling key bottlenecks like slow model weight loading from S3 and lengthy container startup time. Inferless uses Alluxio as a three-tier cache system that dramatically cuts model load time by 10x.declaration of Variables and constants.pptx
declaration of Variables and constants.pptxmeemee7378
油
Rules of declaration of variables and constantsFoundations of Marketo Engage - Programs, Campaigns & Beyond - June 2025
Foundations of Marketo Engage - Programs, Campaigns & Beyond - June 2025BradBedford3
油
Join us for an exciting introductory session on, Foundations of Marketo Engage: Programs, Campaigns & Beyond. Ideal for new and early-stage users looking to build confidence and capability in Marketo Engage.
Our speakers for this session will be:
AJ Navarro
Marketing Operations Manager - Sprout Social
Bobby Coppola
Marketing Manager - Blue Yonder
This event will guide you through the essential steps to build strong, scalable Programs, Channels, Tags, Smart Campaigns, and Reporting Basics in Marketo Engage.
Attendees will gain practical knowledge on how to structure and launch marketing initiatives, leverage templates for efficiency, and implement best practices for campaign execution and measurement. You'll leave with actionable insights you can immediately apply to your own Marketo instance to improve organization, automation, and reporting. Marketo Engage.
Learn how to create impactful programs and understand the best practices for seamless campaign management and effective reporting.
Dont miss out on this opportunity to gain hands-on guidance from experienced practitioners and to connect with peers who are also new to the platform. Secure your spot today and start building your path to becoming a Marketo pro!
Please feel free to invite colleagues who you think would also benefit from this session.Download Adobe Illustrator Crack free for Windows 2025?
Download Adobe Illustrator Crack free for Windows 2025?grete1122g
油
COPY & PASTE LINK https://click4pc.com/after-verification-click-go-to-download-page/
Adobe Illustrator CC Crack will make your illustration level upgrade by moving toward real modification purpose using adobe pro tools there ..Azure AI Foundry: The AI app and agent factory
Azure AI Foundry: The AI app and agent factoryMaxim Salnikov
油
Discover how Azure AI Foundry is transforming the way organizations innovate and operate. This session explores the pivotal role of AI in reshaping business processes, enhancing employee experiences, and redefining customer engagement. Attendees will gain insights into the strategic value of AI investments, including the measurable returns and the growing momentum behind generative AI experimentation.
The talk introduces the emerging paradigm of agentic AI - intelligent agents that are revolutionizing how we build, manage, and interact with software. Through real-world examples, the session demonstrates how Azure AI Foundry empowers teams to design, customize, and scale AI applications and agents, unlocking new levels of efficiency and creativity across industries.arctitecture application system design os dsa
arctitecture application system design os dsaza241967
油
wow amazing see and enjoy for this the best pdf slideshow you could see random shit is what i do best HYBRIDIZATION OF ALKANES AND ALKENES ...
HYBRIDIZATION OF ALKANES AND ALKENES ...karishmaduhijod1
油
Hybridization happens when atomic orbitals mix to form a new atomic orbital. CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
CodeCleaner: Mitigating Data Contamination for LLM Benchmarkingarabelatso
油
The pdf version of "CodeCleaner: Mitigating Data Contamination for LLM Benchmarking" presented in Internetware 2025.On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?
On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?Hassan Abid
油
As mobile hardware becomes more powerful, the promise of running advanced AI directly on-device is closer than ever. With Googles latest on-device model Gemini Nano, accessible through the new ML Kit GenAI APIs and AI Edge SDK, alongside the open-source Gemma-3n models, developers can now integrate lightweight, multimodal intelligence that works even without an internet connection. But does this mean we no longer need cloud-based AI? This session explores the practical trade-offs between on-device and cloud AI for mobile apps.Sysinfo OST to PST Converter Infographic
Sysinfo OST to PST Converter InfographicSysInfo Tools
油
The SysInfo OST to PST Converter is the most secure software to perform the OST to PST conversion. It converts OST files into various formats, including MBOX, EML, EMLX, HTML, and CSV, with 100% accuracy. Along with that, it imports OST files into different email clients like Office 365 and G Suite.
visit: https://www.sysinfotools.com/ost-to-pst-converter.phpDigital Transformation: Automating the Placement of Medical Interns
Digital Transformation: Automating the Placement of Medical InternsSafe Software
油
Discover how the Health Service Executive (HSE) in Ireland has leveraged FME to implement an automated solution for its Order of Merit process, which assigns medical interns to their preferred placements based on applications and academic performance. This presentation will showcase how FME validates applicant data, including placement preferences and academic results, ensuring both accuracy and eligibility. It will also explore how FME automates the matching process, efficiently assigning placements to interns and distributing offer letters. Beyond this, FME actively monitors responses and dynamically reallocates placements in line with the order of merit when offers are accepted or declined. The solution also features a self-service portal (FME Flow Gallery App), enabling administrators to manage the entire process seamlessly, make adjustments, generate reports, and maintain audit logs. Additionally, the system upholds strict data security and governance, utilising FME to encrypt data both at rest and in transit.Y - Recursion The Hard Way GopherCon EU 2025
Y - Recursion The Hard Way GopherCon EU 2025Eleanor McHugh
油
In most modern programming languages, including Go, it's possible for a function to call itself recursively. It's such a mainstream technique that we've come to take it for granted.
But what if functions couldn't call themselves?
There's a branch of mathematics called Combinatorics which answers just these kinds of questions, and this presentation takes these ideas and phrases them in Go to teach a deeper view of computation through the medium of code.
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
CodeCleaner: Mitigating Data Contamination for LLM Benchmarkingarabelatso
油
The pdf-version of "CodeCleaner: Mitigating Data Contamination for LLM Benchmarking" presented in Internetware 2025.Streamlining CI/CD with FME Flow: A Practical Guide
Streamlining CI/CD with FME Flow: A Practical GuideSafe Software
油
Join us as we explore how to deploy a fault-tolerant FME Flow installation using FME Flows IaC templates alongside Terraform, AWS, and GitHub in a CI/CD workflow. This session will also cover how to leverage FMEs CI/CD capabilities for FME Flow upgrades and FME object migration between different environments, ensuring seamless automation and scalability.Ad
Big Data on AWS
- 1. BIG DATA WITH AWS INTRO TO Szilveszter Molnar, Senior Pre-Sales Engineer @moszinet
- 2. GENERIC AWS
- 4. REGIONS AND AVAILABILITY ZONES
- 5. GENERIC AWS Regions Availability Zones Edge Locations
- 6. GENERIC AWS Amazon EC2 Elastic Compute Cloud Amazon S3 Simple Storage Service (Volume Types) (Pricing Model) IAM Identity And Access Management
- 8. GLACIER
- 9. GLACIER CONCEPTS Keep all your data at a much lower cost Move automatically from S3 to Glacier Compliance requirements to keep your data Vault Lock (based on IAM policies)
- 11. KINESIS
- 12. AWS KINESIS WHAT IS KINESIS? Platform for streaming data on AWS "sometimes TBs per hour"
- 13. AWS KINESIS KINESIS COMPONENTS Firehose AnalyticsStreams
- 14. KINESIS STREAMS
- 16. AWS KINESIS STREAMS CONCEPTS Streams - ordered sequence of data records Data record - Sequence Number, Partition Key, Data Blob 1MB max Retention period - 24h - 7d Producers, Consumers Shards
- 17. AWS KINESIS STREAMS KINESIS STREAMS - SHARDS Fixed unit of capacity Read 5 transaction / sec 2MB / sec Write 1000 records / sec 1MB / sec
- 18. KINESIS FIREHOSE
- 20. AWS KINESIS FIREHOSE CONCEPTS Firehose delivery stream record - 1MB max data producer buffer size, buffer interval
- 21. AWS KINESIS FIREHOSE DATA DELIVERY KINESIS STREAM S3 Redshift Elasticsearch
- 25. AWS KINESIS ANALYTICS TERMINOLOGY Input Application Code In-App-Streams Pumps Streaming SQL Output
- 26. AWS KINESIS ANALYTICS STREAMING SQL Tumbling Window [...] GROUP BY FLOOR((SOURCE_SQL_STREAM_001.ROWTIME TIMESTAMP 1970-01-01 00:00:00) SECOND / 10 TO SECOND) Sliding Window SELECT AVG(change) OVER W1 as avg_change FROM "SOURCE_SQL_STREAM_001" WINDOW W1 AS (PARTITION BY ticker_symbol RANGE INTERVAL '10' SECOND PRECEDING)
- 27. AWS KINESIS ANALYTICS STREAMING SQL - TUMBLING WINDOW
- 29. SQS
- 30. SQS CONCEPTS Simple Queue Service Send, Store, Retrieve messages between applications Acts as a buffer At least once delivery FIFO is also supported
- 31. SQS CONCEPTS Queues are created in regions 14 days retention no message priority is supported (2 queues)
- 32. SQS EXAMPLE WEB SERVER REQUEST QUEUE RESPONSE QUEUE PROCESSOR NODES
- 34. IOT
- 35. IOT CONCEPTS Managed Cloud Platform from Internet of Things Billions of devices, trillions of messages Messages can be routed to other AWS services and other devices
- 36. IOT IOT AND BIG DATA IoT devices produce data Analyze streaming data real time Process and store data Don't worry about capacity, scaling, infrastructure
- 37. IOT ARCHITECTURE GENERIC IOT THING AWS IOT Elasticsearch Kinesis Firehose Kinesis Streams DynamoDB Machine Learning CloudWatch S3 SQS, SNS
- 38. IOT ARCHITECTURE AWS IOT SDK AUTHENTICATION AUTHORIZATION DEVICE GATEWAY RULE ENGINE DEVICE SHADOWS DEVICE REGISTRY
- 40. DATA PIPELINE
- 41. DATA PIPELINE TERMINOLOGY A web service to process and move data between AWS compute and storage services or on-premise data sources ETL Work鍖ow Runs on an EC2 instance or EMR cluster that are provisioned automatically
- 42. DATA PIPELINE
- 43. DATA PIPELINE ARCHITECTURE Data Nodes - destination for your data Activity Hive Pig Shell script etc Preconditions Schedules
- 45. DYNAMO DB
- 46. DYNAMO DB CONCEPTS Fully managed NoSQL database No visible "servers" Single digit latency Document & Key-Value models No storage limitations Runs on SSD
- 47. DYNAMO DB CONCEPTS Collection of Tables Performance is set on the tables Write Capacity Units - count of 1KB blocks Read Capacity Units - count of 4KB blocks Eventually Consistent by default (data in 3 regions) Strong consistent reads supported
- 48. DYNAMO DB CONCEPTS Schemaless Contains Items (rows) and Attributes (elements) Special Attributes Partition Key Sort Key
- 49. DYNAMO DB DATA TYPES String Number Binary Bool Null Document (List/Map) Set
- 50. DYNAMO DB DYNAMO DB IN THE AWS ECOSYSTEM On EMR Dynamo DB is integrated with Hive Copy data to/from S3 with Data Pipeline Lambda can be used as triggers Move data into Redshift with the COPY command
- 52. EMR
- 53. EMR CONCEPTS Managed Cluster Platform Hadoop, Spark, Presto, HBase, etc... Master, Core and Task Nodes Single AZ concept
- 54. EMR CONCEPTS
- 55. EMR CONCEPTS
- 56. EMR CORE NODE Like a slave node Runs tasks HDFS Data node, Node Manager, Application Master
- 57. EMR TASK NODE Like a slave node No HDFS Can be added/remove from a running cluster Extra compute capacity
- 58. EMR STORAGE OPTIONS Instance Store ephemeral - deleted when instance terminates/lost use when High I/O performance is necessary EBS for HDFS EMRFS
- 59. EMR EMRFS Implementation of HDFS Wrapper over S3 Resize, terminate clusters without loosing data Multiple clusters can point to the same data in S3 EMRFS & HDFS S3DistCp
- 60. EMR EMRFS - CONSISTENT VIEWS S3 Read After Write consistent for new data Eventual consistent for overwrite & delete Solve it with Consistent Views switch
- 61. EMR INSTANCE TYPES Guidelines by Amazon Map Reduce Batch Oriented - M3 & M4 instance types Machine Learning P2, C3, C4 Spark R3, R4 Large Instance store for HDFS I2, D2
- 63. LAMBDA
- 64. LAMBDA CONCEPTS Function as a Service Billed for the ms of use Isolated & Stateless Persistence - your responsibility Event-driven
- 65. LAMBDA LIMITS /tmp space - 512MB threads - 1,024 max execution duration - 300 seconds
- 67. LAMBDA EXAMPLES S3 LAMBDA FILE WITH WATERMARK new 鍖le Add Watermark DYNAMO DB LAMBDA change data EMAIL WITH REPORT
- 69. REDSHIFT
- 70. REDSHIFT CONCEPTS Fully managed petabyte scale data warehouse MPP (Massively Parallel Processing database) OLAP & BI applications ANSI SQL compatible Column oriented Single AZ Backups to S3
- 71. REDSHIFT ARCHITECTURE LEADER NODE COMPUTE NODE COMPUTE NODE COMPUTE NODE
- 72. REDSHIFT LEADER NODE SQL Endpoint Coordinates parallel query execution Stores metadata
- 73. REDSHIFT COMPUTE NODES Execute queries in parallel Scale out/in, up/down
- 74. REDSHIFT COLUMNAR DATABASE Stores data on disk grouped by columns, not rows Row based storage Column based storage
- 75. REDSHIFT BENEFITS Queries use fewer columns - less disk I/O Data aggregation (eg. SUM(sales)) Compression
- 77. QUICKSIGHT
- 78. QUICKSIGHT CONCEPTS Cloud based analytics service Build visualizations Ad-hoc analysis Connect to AWS services, EC2 or on-premises DB
- 79. QUICKSIGHT SUPPORTED DATA SOURCES Redshift Aurora Athena RDS (Maria DB, SQL Server, MySQL, PostgreSQL) Files Salesforce
- 80. QUICKSIGHT
- 82. AWS BIG DATA READING MATERIALS Kinesis Kinesis Firehose Transformation with Lambda Implementing producers with Amazon Kinesis Producer Library EMR Best practices for Amazon EMR (2013) Lambda Big Data processing with serverless MapReduce Redshift Redshift Table Design Optimizing for Star Schemas and Interleaved Sorting
- 83. AWS BIG DATA HOMEWORK - OPTIONAL 1. Create a transient EMR cluster that automatically starts your application (Hive, Pig, Spark, etc...) 2. Create a Redshift cluster, experiment with the COPY command (bulk load rows) vs INSERT statement 100 rows at least Note: Make sure you do not forget to delete the created resources ...
- 84. THANK YOU