Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems
2 likes509 views
Surprising progress has been made on Machine Learning algorithms and infrastructures recently. These techniques are being used to solve a lot of previously un-solvable problems. However, not all problems are directly ML-applicable. Some need to be divided and transformed properly before the powerful ML technique can be applied to it. In this session, Ill talk about what kinds of black magic we applied to the data problems we are facing daily at Mattermark, so that we can solve them in graceful and scalable way using Machine Learning.
Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems
- 1. Black Magic How to apply ML to real-world problems
- 2. It is great tool for some purposes ML is (Magic) Hammer
- 3. If all you have is a hammer, everything looks like a nail
- 5. I am Evion Kim Lead Machine Learning Engineer @ Mattermark Senior Software Engineer /Data Scientist @ Linkedin M.S. , Computer Science @ Stanford University B.S., Computer Science @ KAIST Hello!
- 6. Todays Talk Machine Learning - the concept Mattermark? Funding Extraction Problem @ Mattermark & Some Magic Spells
- 7. Not about # Deep academic technical knowledge about ML algorithms # Data Infrastructure About # How to transform real- world problem into ML- problem # Tips and tricks on Machine Learning based problem solving This talk is...
- 8. Machine Learning The powerful hammer 1
- 10. def traditional(x): return x*(x+1) Traditional way y = x * (x+1)
- 11. 2 = 6 3 = 12 4 = 20 5 = 30 6 = 42 ML Way y = x * (x+1) Model
- 14. It is not SKYNET at least not yet.
- 15. It is tool that can be used for some problems
- 16. What is (just quick advertisement) 2
- 22. Case Study: Funding Extraction -And Dark Magic spells we learned 3
- 23. Small(er) company # Much smaller training data points # Very high precision requirement. Big(ger) company # Millions of Millions of training data points # Precision requirement: not that high ML @ Big(ger) vs. Small(er)
- 24. Whats the bottleneck? # Scalability or Accuracy? # Precision or Recall? # Engineering or Machine Learning? Spell 1: Know your Enemy
- 25. ~$156 BillionTotal VC funding in year 2015 ~8,532VC funding events in year 2015
- 26. Problem to solve
- 27. Divide big chunky problem into smaller ML-solvable problems. Spell 2: Slice and Dice
- 29. Classify Funding Article TF-IDF + SVM Classifier NO YES
- 30. Analyze and understand the problem space you are working on. Spell 3: Understand your Domain
- 31. Amount/Series/Investors ...has closed a $3.5m Series A funding round led by Inter Capital, ... Investors Intel Capital led the round with participation from other investors that included Horizons Ventures Amount/ Series ...has raised $3.5 million in Series A Funding Funding Sentences Patterns
- 32. Classify Funding Sentences Word2Vec + Semantic Role Labeling (SRL) + Gradient Boosting Classsifier
- 33. Regex Parsing + Named Entity Recognition Extract Funding Entities
- 34. Spell 4: Probabilistic Train and use the probabilistic models helps a lot sometimes.
- 35. Whats the probability of these extracted information to be correct? Confidence Scoring 0~1 probability score
- 36. Spell 5: Human + Machine Let some part of the job get the help from mighty human-being
- 38. ~$156 BillionTotal VC funding in year 2015 ~8,532VC funding events in year 2015
- 39. Spell 5: Human + MachineSpell 4: ProbabilisticSpell 3: Understand your domain Spell 2: Slice and DiceSpell 1: Know your enemyML is powerful Hammer Summary
- 40. We are hiring!
- 42. Thanks! Any questions? You can find me at: in/evionkim twitter@evion12 evion12@gmail.com