狠狠撸

1
Introduction to Big Data
Dr. Amira Abdelatey

2
How to manage very large amounts of data and extract
value and knowledge from them

3
Introduction
? Big data refers to extremely large and diverse collections
of structured, unstructured, and semi-structured data that
continues to grow exponentially over time. These datasets
are so huge and complex in volume, velocity, and
variety, that traditional data management systems cannot
store, process, and analyze them.
? The amount and availability of data is growing rapidly,
spurred on by digital technology advancements, such as
connectivity, mobility, the Internet of Things (IoT), and
artificial intelligence (AI).
? Big data tools are emerging to help companies collect,
process, and analyze data at the speed needed to gain the
most value from it.

4
Introduction
What is Big Data?
What makes data, “Big” Data?

5
Big Data Definition
? No single standard definition…
“Big Data” is data whose scale, diversity,
and complexity require new architecture,
techniques, algorithms, and analytics to
manage it and extract value and hidden
knowledge from it…

6
Characteristics of Big Data:
1-Volume (Scale)
Exponential increase in
collected/generated data
? Data Volume
? 44x increase from 2009 2020
? From 0.8 zettabytes to 35zb
? Data volume is increasing exponentially

7
2-Variety (Complexity)
? Different types of data
? Relational Data (Tables/Transaction/Legacy Data)
? Text Data (Web)
? Semi-structured Data (XML)
? Graph Data
? Social Network, Semantic Web (RDF), …
? Streaming Data (Stream vs static)
? You can only scan the data once
? A single application can be generating/collecting many
types of data
? Big Public Data (online, weather, finance, etc)
To extract knowledge? all these types of data need to linked together

8
3-Velocity (Speed)
? Data is begin generated fast and need to be processed fast
? Online Data Analytics
? Late decisions ? missing opportunities
? Examples
? E-Promotions: Based on your current location, your purchase history, what you
like ? send promotions right now for store next to you
? Healthcare monitoring: sensors monitoring your activities and body ? any
abnormal measurements require immediate reaction

9
Real-time/Fast Data
Social media and networks
(all of us are generating data)
Scientific instruments
(collecting all sorts of data)
Mobile devices
(tracking all objects all the time)
Sensor technology and
networks
(measuring all kinds of data)
? The progress and innovation is no longer hindered by the ability to collect data
? But, by the ability to manage, analyze, summarize, visualize, and discover knowledge
from the collected data in a timely manner and in a scalable fashion

10
Real-Time Analytics/Decision Requirement
Customer
Influence
Behavior
Product
Recommendations
that are Relevant
& Compelling
Friend Invitations
to join a
Game or Activity
that expands
business
Preventing Fraud
as it is Occurring
& preventing more
proactively
Learning why Customers
Switch to competitors
and their offers; in
time to Counter
Improving the
Marketing
Effectiveness of a
Promotion while it
is still in Play

12
Harnessing ??????? Big Data
? OLTP: Online Transaction Processing (DBMSs)
? OLAP: Online Analytical Processing (Data Warehousing)
? RTAP: Real-Time Analytics Processing (Big Data Architecture & technology)

13
The Model Has Changed…
? The Model of Generating/Consuming Data has Changed
Old Model: Few companies are generating data, all others are consuming data
New Model: all of us are generating data, and all of us are consuming
data

14
What’s driving Big Data
- Ad-hoc querying and reporting
- Data mining techniques
- Structured data, typical sources
- Small to mid-size datasets
- Optimizations and predictive analytics
- Complex statistical analysis
- All types of data, and many sources
- Very large datasets
- More of a real-time

15
Cloud Computing
? The cloud is a distributed collection of servers that host software
and infrastructure, and it is accessed over the Internet
? IT resources provided as a service
? Compute, storage, databases, queues
? Clouds leverage economies of scale of commodity ?????? hardware
? Cheap storage, high bandwidth networks & multicore processors
? Geographically distributed data centers
? Offerings from Microsoft, Amazon, Google, …

17
Benefits
? Cost & management
? Economies of scale, “out-sourced” resource management
? Reduced Time to deployment
? Ease of assembly, works “out of the box”
? Scaling
? On demand provisioning, co-locate data and compute
? Reliability
? Massive, redundant, shared resources
? Sustainability
? Hardware not owned

18
Types of Cloud Computing
? Public Cloud: Computing infrastructure is hosted at the
vendor’s premises.
? Private Cloud: Computing architecture is dedicated to the
customer and is not shared with other organizations.
? Hybrid Cloud: Organizations host some critical, secure
applications in private clouds. The not so critical applications
are hosted in the public cloud
? Cloud bursting: the organization uses its own infrastructure for normal
usage, but cloud is used for peak loads.
? Community Cloud

19
Classification of Cloud Computing
based on Service Provided
? Infrastructure as a service (IaaS)
? Offering hardware related services using the principles of cloud computing. These could
include storage services (database or disk storage) or virtual servers.
? Amazon EC2, Amazon S3, Rackspace Cloud Servers and Flexiscale.
? Platform as a Service (PaaS)
? Offering a development platform on the cloud.
? Google’s Application Engine, Microsofts Azure.
? Software as a service (SaaS)
? Including a complete software offering on the cloud. Users
can access a software application hosted by the cloud vendor
on pay-per-use basis. This is a well-established sector.
? Salesforce.coms’ offering in the online Customer Relationship
Management (CRM) space, Googles gmail and Microsofts
hotmail, Google docs.

20
Topics overview
Section
? Postgres with Python
? ETL with python
? Framework for big data (Python spark)
? Data modelling
? Data warehouse
? Dimensional data modeling
? ETL
? The power of spark
? Big data processing pipeline
? Data wrangling with spark
? Natural Language Processing
? Association rule mining

狠狠撸

Lecture 1-big data engineering (Introduction).pdf

Recommended

More Related Content

Similar to Lecture 1-big data engineering (Introduction).pdf (20)

Recently uploaded (20)

Lecture 1-big data engineering (Introduction).pdf