effective data analysis with R
?Download as PPTX, PDF?
2 likes?249 views
This document provides an overview of effective data analysis using R. It discusses common challenges with data preparation and introduces the TTVM process for data analysis, which stands for Tidy, Transform, Visualize, Model, and Interpret. The document explains why R is a useful tool for data analysis due to its packages for data access, cleaning, analysis, and reporting. It also emphasizes that most of the work in data analysis involves cleaning and preparing the data before analyzing or modeling can begin.
effective data analysis with R
- 1. Effective Data Analysis A Comprehensive Workflow with TTVM Process & R 呂奕 1
- 2. About These slides provide introductory material to help improve skills for manipulation data, efficiently modeling, and getting insights through such process. The tool used in this slide is "R", which is a popular open-source software, not only as a statistical software but a programming language. 2
- 3. 你是否有以下困擾？ ? 不知道什麼資料可以被使用 ? 拿到的資料跟天書一樣無法整理 ? 終於下定決心要整理資料時不知道從哪裡開始 ? 火眼金睛的整理方式無法確保是否遺漏或手殘 ? 一旦資料出錯就無法回頭，資料夾一堆"xxx_backup"檔案 ? 終於整理完後可以分析的方法非常有限 ? 下次遇到同樣的東西，又要從頭再來 3
- 4. 你是否有以下困擾？ ? 別人看不懂你的處理方法，協同工作很困難 ? 決定方法後要動手做很痛苦，開始從厚厚一疊參考資料翻找 ? 畫圖很痛苦 ? 做模型很痛苦 ? 一大堆模型不知道怎麼解釋和選擇 4
- 6. Data analysis is the process by which data becomes understanding, knowledge and insight 6
- 7. Why R? 7
- 9. Why R? ? It is free ? It has a comprehensive set of packages ? Data access ? Data cleaning ? Analysis ? Data reporting ? It has one of the best development environments - RStudio http://www.rstudio.com/ ? It has an amazing ecosystem of developers ? Packages are easy to install and "play nicely together" 9
- 11. TTVM Process 11
- 12. Before thinking outside the box We have to look inside the black box and figure out how it works. Not until we understand the mechanism of (quantitative) data analysis do we really master the (quantitative) analysis skill. 12
- 13. 13 Tidy Acquiring Data Transform Visualize Model Interpret Modified from Hadley Wickham
- 14. 14 used to be… Computation time >> Cognition time
- 15. 15https://www.flickr.com/photos/mutsmuts/4695658106 should be… Cognition time ? Computation time
- 16. Tidy Data 16
- 17. 17
- 19. Data analysis It is often said that 80% of data analysis is spent on the process of cleaning and preparing the data (Dasu and Johnson 2003). 19
- 20. Defining tidy data Like families, tidy datasets are all alike but every messy dataset is messy in its own way. 1. Each variable forms a column. 2. Each observation forms a row. 3. Each type of observational unit forms a table. This is Codd's 3rd normal form (Codd 1990) 20
- 21. Messy dataset 21 religion <$10k $10-20k $20-30k $30-40k $40-50k $50-75k Agnostic 27 34 60 81 76 137 Atheist 12 27 37 52 35 70 Buddhist 27 21 30 34 33 58 Catholic 418 617 732 670 638 1116 Don't know/refused 15 14 15 11 10 35 Evangelical Prot 575 869 1064 982 881 1486 Hindu 1 9 7 9 11 34 Historically Black Prot 228 244 236 238 197 223 Jehovah's Witness 20 27 24 24 21 30 Jewish 19 19 25 25 30 95
- 22. Tidy data 22 religion income freq Agnostic <$10k 27 Agnostic $10-20k 34 Agnostic $20-30k 60 Agnostic $30-40k 81 Agnostic $40-50k 76 Agnostic $50-75k 137 Agnostic $75-100k 122 Agnostic $100-150k 109 Agnostic >150k 84 Agnostic Don't know/refused 96
- 23. If you can get things done at a time, then don't spend dozens! 23
- 24. 24
- 26. Split, Apply, Combine 26 name n Al 2 Bo 4 Bo 0 Bo 5 Ed 5 Ed 10 name n Al 2 name n Bo 4 Bo 0 Bo 5 name n Ed 5 Ed 10 total 2 total 9 total 15 name n Al 2 Bo 9 Ed 15
- 27. Simple and comprehensible code makes your work replicable and easy to debug 最重要的是：永遠不需要再砍掉重練，錯了可以重來 27
- 28. Re-level 28
- 29. Recode 29
- 31. Visualization 31
- 33. 跑表跑到眼花撩亂 33
- 34. 用之嫷缴 34
- 37. Modeling 37
- 38. Flexible / Learning by doing 38
- 39. 很多別人佛心寫好的套件 39
- 41. Data Product 41
- 43. 淺談 Big data 先別管big不big了，你知道分析方法有甚麼不同嗎？ → 事實上沒什麼不同：假設、驗證、預測(學習) 而且只是你的硬碟還裝得下的檔案，基本上都不算big 43
- 44. 淺談 Big data big data 不是什麼新概念，就是一個莫名近年來莫名在炒的話題。 big data確實是管理問題，很多公司還把它當data mining，跑跑公司的 交易資料，甚至根本不信data這套的還更多。太多的unstructured data 和machine data沒利用到了，甚至是open structured data也根本沒在用。 再來是公司data-driven decision 做到什麼程度，只有行銷做一做嗎？策 略性的去累積你的data，以及訓練你的model、data automation的程度， 都會變成贏過對手的競爭優勢。透過結合Big data+ machine learning + cloud打造出來的應用會大大替代過去的各種商業模式。 44http://www.bnext.com.tw/article/view/id/34692