Data science in action
Download as PPTX, PDF2 likes542 views
slides from my presentation at data science in action meetup 16-Oct-2017 https://www.meetup.com/Data-Science-in-Action/
![Big Data [sorry] & Data Science: What Does a Data Scientist Do?](https://cdn.slidesharecdn.com/ss_thumbnails/dslatcloudmsevent20130125-130126065651-phpapp01-thumbnail.jpg?width=560&fit=bounds)
More Related Content
Similar to Data science in action (20)
More from Longhow Lam (13)
Recently uploaded (20)
Data science in action
- 1. Copyright ? 2012, SAS Institute Inc. All rights reserv ed. GOEDE TIJDEN SLECHTE TIJDEN, RESTAURANT REVIEWS, BRAD PITT AND THE IKEA BILLY INDEX Longhow Lam – Freelance Data Scientist https://www.linkedin.com/in/longhowlam https://longhowlam.wordpress.com @longhowlam
- 2. Data Science in Action AGENDA ? TEXT MINING AND MACHINE LEARNING ? SOME CRAZY EXAMPLES ? Goede tijden Slechte tijden ? IENS Restaurant Reviews ? Who looks like Brad Pitt? ? The IKEA Billy Index
- 3. Text mining and Machine Learning
- 4. Text mining: simple example Doc 1 “I walked accross the street in Amsterdam, 1057DK, with my bike” Doc 2 “She didn’t walk but cycled with her blue biike, //bitly.com/sdrtw” Doc 3 “My bicycle is broken, what a piece of junk, @#$%$@!” Terms Doc 1 Doc 2 Doc 3 +Bicycle (noun) 1 1 1 Cycling (verb) 0 1 0 Blue (adjective) 0 1 0 Amsterdam (location) 1 0 0 +Walk (verb) 1 1 0 Street (noun) 1 0 0 Broken (adjective) 0 0 1 Piece of junk (noun) 0 0 1 1057DK (postal code) 1 0 0 //bitly.com/sdrtw 0 1 0 TERM DOCUMENT MATRIX: A ? Every text document is a (very) long string (with many zeros!) ? Data mining techniques are applied to this matrix A
- 5. Data Science in Action TEXT MINING PREDICT OR CLUSTER Combine texts and “normal data” to predict behaviour (churn / fraude) Use machine learning to train a learner f to predict the TARGET Automatically create topics / clusters in huge piles of documents Apply cluster techniques to divide documents into topic Topic 1 Topic 2 Topic 3
- 6. Data Science in Action MACHINE LEARNING SOME ALGORITHMS Predict Trees Random Forests Cluster K-means Hierarchical clustering DBSCAN Lineair regression f y = f(x) = a0 + a1x1 + a2x2+…anxn Neural networks y = f(g(h(x)))
- 8. Data Science in Action GTST ANALYSIS TEXT ANALYTICS Business pain Looking at GTST (Dutch soap): what the hack is this all about? Are there trends in the series, is it not all the same? Approach Take the 5000 summaries and apply text mining in SAS
- 9. Data Science in Action GTST ANALYSIS RESULTS Main topics in 5000 episodes
- 10. Data Science in Action GTST ANALYSIS DISTANCES BETWEEN TOPICS
- 11. Data Science in Action GTST ANALYSIS ZOOMING IN ON A TOPIC
- 12. Data Science in Action GTST ANALYSIS ZOOMING IN ON A TOPIC Sub-topics of main topic: topic 16 (Ludo, Isabelle, Martine, Janine) ? Harmsen feeling lonely. ? Plan by Jack, dangerous ? Writing a farewell letter ? Panic, fear, ? Questions about giving kid assignment ? Getting money back, paying IMPORTANT: Business validation! I asked my wife, she used to be a loyal GTST watcher
- 13. Data Science in Action GTST ANALYSIS TREND RESULTS Trends over time with SAS text profile feature
- 14. Data Science in Action GTST ANALYSIS TRENDS OVER TIME
- 15. Data Science in Action GTST ANALYSIS SIMILARITY OF EPISODES THROUGH THE YEARS
- 16. Data Science in Action Can you shake hands with your neighbor? A LITTLE STATISTICAL EXPERIMENT Two statistics that I like to share:
- 17. Data Science in Action Can you shake hands with your neighbor? A LITTLE STATISTICAL EXPERIMENT 50.1% of people don’t wash their hands after visiting the toilet
- 18. Data Science in Action Can you shake hands with your neighbor? A LITTLE STATISTICAL EXPERIMENT 50.1% of people don’t wash their hands after visiting the toilet 84.6% of all statistics are just made up on the spot !!
- 20. Data Science in Action IENS RESTAURANT PATH ANALYTICS Business pain I have eaten Chinese, where should I go next?. Approach Look at what others do, IENS restaurant reviewers!
- 21. Data Science in Action A FEW FACTS… IENS DATA (TRADITIONAL BI) Most occurring restaurant name (39 times) Among “dutch” restaurant (6 times) % Sustainable kitchens Biological (67%) French (58%) Fish (44%) Vegetarian (39%) … … … Chinese (3%) 700 reviews on a “normal” Saturday Valentine 2015 1200 reviews (1.7 times) 23 times 12 times
- 22. Data Science in Action IENS RESTAURANT PATH ANALYSIS: GENERATED PATHS
- 23. Data Science in Action IENS REVIEWS CAN SENTIMENT BE PREDICTED? ? Translate the reviews into a term document matrix ? Apply machine learning to predict scores ? Why would you do this?
- 24. Data Science in Action IENS REVIEWS CAN I PREDICT THE SENTIMENT?
- 25. Data Science in Action IENS REVIEWS PREDICT THE ‘EAT’ SCORE Neural (2 X 20) R2 of 0.65 Linear reg model R2 of 0.56
- 26. Data Science in Action Predicted review score vs. Given review score IENS REVIEWS PREDICTION THE ‘EAT’ SCORE
- 27. Data Science in Action IENS REVIEWS SENTIMENT ANALYSIS / PREDICTIVE MODELING
- 29. Data Science in Action OUTLIERS IN FACES DATA MINING & MACHINE LEARNING Business pain Tell me: Who has a strange face at SAS Netherlands? Approach Take SAS photos and translate to data and apply machine learning
- 30. Data Science in Action OUTLIERS IN FACES DATA MINING & MACHINE LEARNING
- 31. Data Science in Action STRANGE FACE DETECTION COMBO OF OPEN API & SAS ? Use Face++ to do facial landmarking (no deep learning!!) ? Import all landmarks in SAS as an ABT Now you can solve some funny business issues with machine learning: ? Which persons are look-alikes? Hierarchical clustering ? Are there any accountmanagers? Predictive modeling / machine learning ? Who is the Brad Pitt at SAS? Nearest Neighbor ? Funny faces Anomaly / outlier detection
- 32. Data Science in Action STRANGE FACE DETECTION HIERARCHICAL CLUSTERING
- 33. Data Science in Action STRANGE FACE DETECTION BRAD PITT LOOK-A-LIKES…
- 34. Data Science in Action STRANGE FACE DETECTION OUTLIER DETECTION
- 36. Data Science in Action IKEA WEBSITE KEEP TRACK OF BILLY STOCK Define the IKEA Billy Index as the change in stock over time
- 37. Data Science in Action IKEA WEBSITE THE IKEA BILLY INDEX
- 38. Data Science in Action THE BILLY INDEX SOME STATISTICS
- 39. Data Science in Action Every extra unit increase in wind speed results in 19 less Billy’s sold ?
- 40. Copyright ? 2012, SAS Institute Inc. All rights reserv ed. Thanks for your attention, QUESTIONS? Freelance Data Scientist, Ik sta open om eens een kop koffie te drinken https://www.linkedin.com/in/longhowlam https://longhowlam.wordpress.com/ @longhowlam