Data science in action
Download as pptx, pdf2 likes546 views
The document presents an overview of various applications of text mining and machine learning, including analysis of Dutch soap operas, restaurant reviews, and image data. It discusses techniques such as term-document matrices, clustering, and predictive modeling to extract insights from large datasets. Additionally, it highlights specific case studies including the 'IKEA Billy Index' and facial outlier detection.
Data science in action
- 1. Copyright 息 2012, SAS Institute Inc. All rights reserv ed. GOEDE TIJDEN SLECHTE TIJDEN, RESTAURANT REVIEWS, BRAD PITT AND THE IKEA BILLY INDEX Longhow Lam Freelance Data Scientist https://www.linkedin.com/in/longhowlam https://longhowlam.wordpress.com @longhowlam
- 2. Data Science in Action AGENDA TEXT MINING AND MACHINE LEARNING SOME CRAZY EXAMPLES Goede tijden Slechte tijden IENS Restaurant Reviews Who looks like Brad Pitt? The IKEA Billy Index
- 3. Text mining and Machine Learning
- 4. Text mining: simple example Doc 1 I walked accross the street in Amsterdam, 1057DK, with my bike Doc 2 She didnt walk but cycled with her blue biike, //bitly.com/sdrtw Doc 3 My bicycle is broken, what a piece of junk, @#$%$@! Terms Doc 1 Doc 2 Doc 3 +Bicycle (noun) 1 1 1 Cycling (verb) 0 1 0 Blue (adjective) 0 1 0 Amsterdam (location) 1 0 0 +Walk (verb) 1 1 0 Street (noun) 1 0 0 Broken (adjective) 0 0 1 Piece of junk (noun) 0 0 1 1057DK (postal code) 1 0 0 //bitly.com/sdrtw 0 1 0 TERM DOCUMENT MATRIX: A Every text document is a (very) long string (with many zeros!) Data mining techniques are applied to this matrix A
- 5. Data Science in Action TEXT MINING PREDICT OR CLUSTER Combine texts and normal data to predict behaviour (churn / fraude) Use machine learning to train a learner f to predict the TARGET Automatically create topics / clusters in huge piles of documents Apply cluster techniques to divide documents into topic Topic 1 Topic 2 Topic 3
- 6. Data Science in Action MACHINE LEARNING SOME ALGORITHMS Predict Trees Random Forests Cluster K-means Hierarchical clustering DBSCAN Lineair regression f y = f(x) = a0 + a1x1 + a2x2+anxn Neural networks y = f(g(h(x)))
- 8. Data Science in Action GTST ANALYSIS TEXT ANALYTICS Business pain Looking at GTST (Dutch soap): what the hack is this all about? Are there trends in the series, is it not all the same? Approach Take the 5000 summaries and apply text mining in SAS
- 9. Data Science in Action GTST ANALYSIS RESULTS Main topics in 5000 episodes
- 10. Data Science in Action GTST ANALYSIS DISTANCES BETWEEN TOPICS
- 11. Data Science in Action GTST ANALYSIS ZOOMING IN ON A TOPIC
- 12. Data Science in Action GTST ANALYSIS ZOOMING IN ON A TOPIC Sub-topics of main topic: topic 16 (Ludo, Isabelle, Martine, Janine) Harmsen feeling lonely. Plan by Jack, dangerous Writing a farewell letter Panic, fear, Questions about giving kid assignment Getting money back, paying IMPORTANT: Business validation! I asked my wife, she used to be a loyal GTST watcher
- 13. Data Science in Action GTST ANALYSIS TREND RESULTS Trends over time with SAS text profile feature
- 14. Data Science in Action GTST ANALYSIS TRENDS OVER TIME
- 15. Data Science in Action GTST ANALYSIS SIMILARITY OF EPISODES THROUGH THE YEARS
- 16. Data Science in Action Can you shake hands with your neighbor? A LITTLE STATISTICAL EXPERIMENT Two statistics that I like to share:
- 17. Data Science in Action Can you shake hands with your neighbor? A LITTLE STATISTICAL EXPERIMENT 50.1% of people dont wash their hands after visiting the toilet
- 18. Data Science in Action Can you shake hands with your neighbor? A LITTLE STATISTICAL EXPERIMENT 50.1% of people dont wash their hands after visiting the toilet 84.6% of all statistics are just made up on the spot !!
- 20. Data Science in Action IENS RESTAURANT PATH ANALYTICS Business pain I have eaten Chinese, where should I go next?. Approach Look at what others do, IENS restaurant reviewers!
- 21. Data Science in Action A FEW FACTS IENS DATA (TRADITIONAL BI) Most occurring restaurant name (39 times) Among dutch restaurant (6 times) % Sustainable kitchens Biological (67%) French (58%) Fish (44%) Vegetarian (39%) Chinese (3%) 700 reviews on a normal Saturday Valentine 2015 1200 reviews (1.7 times) 23 times 12 times
- 22. Data Science in Action IENS RESTAURANT PATH ANALYSIS: GENERATED PATHS
- 23. Data Science in Action IENS REVIEWS CAN SENTIMENT BE PREDICTED? Translate the reviews into a term document matrix Apply machine learning to predict scores Why would you do this?
- 24. Data Science in Action IENS REVIEWS CAN I PREDICT THE SENTIMENT?
- 25. Data Science in Action IENS REVIEWS PREDICT THE EAT SCORE Neural (2 X 20) R2 of 0.65 Linear reg model R2 of 0.56
- 26. Data Science in Action Predicted review score vs. Given review score IENS REVIEWS PREDICTION THE EAT SCORE
- 27. Data Science in Action IENS REVIEWS SENTIMENT ANALYSIS / PREDICTIVE MODELING
- 29. Data Science in Action OUTLIERS IN FACES DATA MINING & MACHINE LEARNING Business pain Tell me: Who has a strange face at SAS Netherlands? Approach Take SAS photos and translate to data and apply machine learning
- 30. Data Science in Action OUTLIERS IN FACES DATA MINING & MACHINE LEARNING
- 31. Data Science in Action STRANGE FACE DETECTION COMBO OF OPEN API & SAS Use Face++ to do facial landmarking (no deep learning!!) Import all landmarks in SAS as an ABT Now you can solve some funny business issues with machine learning: Which persons are look-alikes? Hierarchical clustering Are there any accountmanagers? Predictive modeling / machine learning Who is the Brad Pitt at SAS? Nearest Neighbor Funny faces Anomaly / outlier detection
- 32. Data Science in Action STRANGE FACE DETECTION HIERARCHICAL CLUSTERING
- 33. Data Science in Action STRANGE FACE DETECTION BRAD PITT LOOK-A-LIKES
- 34. Data Science in Action STRANGE FACE DETECTION OUTLIER DETECTION
- 36. Data Science in Action IKEA WEBSITE KEEP TRACK OF BILLY STOCK Define the IKEA Billy Index as the change in stock over time
- 37. Data Science in Action IKEA WEBSITE THE IKEA BILLY INDEX
- 38. Data Science in Action THE BILLY INDEX SOME STATISTICS
- 39. Data Science in Action Every extra unit increase in wind speed results in 19 less Billys sold
- 40. Copyright 息 2012, SAS Institute Inc. All rights reserv ed. Thanks for your attention, QUESTIONS? Freelance Data Scientist, Ik sta open om eens een kop koffie te drinken https://www.linkedin.com/in/longhowlam https://longhowlam.wordpress.com/ @longhowlam