Bigdata analytics
Download as PPTX, PDF
5 likes1,476 views
Big data refers to extremely large datasets that are difficult to store, manage and analyze using traditional database tools. The volume of data in the world is growing exponentially and is projected to reach 35 zettabytes by 2020. Big data is important because it enables new forms of innovation and competition through deeper insights. Visual analytics is the use of interactive visual interfaces to synthesize and analyze large, complex datasets. It draws upon techniques from fields like statistics, machine learning and information visualization. Challenges include developing intelligent algorithms to combine human and machine analysis of large datasets and enabling the visualization of intermediate results from large-scale simulations and computations.
![GapMinder [Demo]](https://image.slidesharecdn.com/bigdataanalytics-150101073112-conversion-gate02/85/Bigdata-analytics-27-320.jpg)
![Smart Money Map [Demo]](https://image.slidesharecdn.com/bigdataanalytics-150101073112-conversion-gate02/85/Bigdata-analytics-28-320.jpg)
More Related Content
What's hot (18)
Similar to Bigdata analytics (20)
Recently uploaded (20)
Bigdata analytics
- 1. Big Data & Analytics Keshav Tripathy, Bharti Consulting Inc.
- 2. Outline Big Data Gartner Hype Cycle 2012 Large scale data processing Visual Analytics Chances and Challenges Discussions
- 3. Big Data V3 Volume鐚Gigabyte(109), Terabyte(1012), Petabyte(1015), Exabyte(1018), Zettabytes(1021) Variety: Structured,semi-structured, unstructured; Text, image, audio, video, record Velocity(Dynamic, sometimes time-varying) Big Data refers to datasets that grow so large that it is difficult to capture, store, manage, share, analyze and visualize with the typical database software tools.
- 4. Numbers How many data in the world? 800 Terabytes, 2000 160 Exabytes, 2006 500 Exabytes(Internet), 2009 2.7 Zettabytes, 2012 35 Zettabytes by 2020 How many data generated ONE day? 7 TB, Twitter 10 TB, Facebook Big data: The next frontier for innovation, competition, and productivity McKinsey Global Institute 2011
- 5. Why Is Big Data Important?
- 6. Gartner Hype Cycle 2012
- 7. Large Scale Visual Analytics Definition: Visual analytics is the science of analytical reasoning facilitated by interactive visual interfaces. People use visual analytics tools and techniques to Synthesize information and derive insight from massive, dynamic, ambiguous, and often conflicting data Detect the expected and discover the unexpected Provide timely, defensible, and understandable assessments Communicate assessment effectively for action.
- 8. Inforviz Reference Model to Visual Analytics
- 9. Applications Terrorism and Responses Multimedia Visual Analytics Situation Surveillance and Awareness in Investigative Analysis Disease visual analytics for Disease outbreak Prediction Financial Visual Analytics Cybersecurity Visual Analytics Visual Analytics for Investigative Analysis on Text Documents
- 10. Techniques and Technologies A wide variety of techniques and technologies has been developed and adapted for Data aggregation Data manipulation Data analysis Data visualization These techniques and technologies draw from several fields including Statistics Computer science Applied mathematics Economics.
- 11. Techniques and Applications Statistics: A/B testing(split testing/bucket testing ),Spatial analysis , Predictive modeling :Regression Machine Learning Unsupervised learning: cluster analysis Supervised learning: classification, support vector machines(SVM), ensemble learning Association rule learning Data Mining and Pattern Recognition: neural network, classification, clustering Natural language processing(NLP): Sentiment analysis Dimension Reduction: PCA, MDS, SVD Data fusion and data integration鐚 Visual Word Time series analysis: Combination of statistics and signal processing Simulation: Monte Carlo simulations, MRF Optimization: Genetic algorithms Visualization: Scientific Viz, Inforviz, Visual Analtytics
- 12. Technologies Database and Data warehouse Google File System and MapReduce: Big Table Hadoop: HBase and MapReduce, open source Apache project Cassandra: An open source (free) DBMS, originally developed at Facebook and now an Apache Software foundation project. Data warehouse: ETL (extract, transform, and load) tools and business intelligence tools. Business intelligence (BI): data warehouse, reporting, real-time management dashboards Cloud computing: Services, SOA, etc. Metadata: XML Stream processing R, SAS and SPSS Visualization:Tag cloud,Clustergram,History flow, Themeriver, Treemap
- 13. Origin of Information Visualization
- 14. InforViz Techniques Scatterplot and Scatterplot Matrix Hierarchies Visualization:Node-Link Diagrams, Sunburst,Treemap, Circle- packing layouts Network Visualization:Force-Directed Layout,Arc Diagrams,Matrix Views Multidimensional Visualization/Parallel Coordinates Stacked Graphs Flow Maps
- 15. Scatterplot and Scatterplot Matrix
- 18. Network Visualization Force-Directed Layout Arc Diagrams Matrix Views
- 20. Stacked Graphs
- 21. Flow Maps
- 22. Examples
- 24. Fraud Detection of Bank Wire Transactions
- 26. A classical VA tool
- 27. GapMinder [Demo]
- 28. Smart Money Map [Demo]
- 29. A recent project
- 30. Chances and Challenges The basic techniques for large scale simulation and computing are ready However, large and time-consuming computing tasks need steering or visualize the intermediate computing results. Most simulation and computing tasks have to tune hundreds of parameters. Smart/intelligent data mining/data processing algorithms are ready However, most data mining algorithms have high computational complexity: N2 rather than Nlog(N), or N How to combine automatic computing(machine) and high-level intelligence to gain insight(Human), and involve human in the computing?
- 31. Recent Research Topics Unified Visual Analytics by Heterogeneous Data Sources(esp. Text) Structured and semi-structured data fusion framework Data indexing and similarity rank Visual analytics for high-dimensional heterogeneous data Domain Risk Management and Preventive Control by Sensor Data Collection and Data Mining Sensor techniques Data Warehouse Coordinated Views integrate visual analytic techniques Parallel/Distributed Computing Steering by Parameter Optimization and Visualization Parameter tuning and computing optimization Intermediate results visualization and task steering Markov Chain Monte Carlo(MCMC) Simulation