Machine-Learning-Overview a statistical approach
Download as PPTX, PDF0 likes39 views
This document provides an overview of machine learning concepts including what machine learning is, common machine learning tasks like fraud detection and recommendation engines, and different machine learning techniques like supervised and unsupervised learning. It discusses neural networks and deep learning, and explains the machine learning process from data acquisition to model deployment. It also covers important concepts for evaluating machine learning models like overfitting, accuracy, recall, precision, F1 score, confusion matrices, and regression metrics like mean absolute error, mean squared error and root mean squared error.
More Related Content
Similar to Machine-Learning-Overview a statistical approach (20)
Recently uploaded (20)
Machine-Learning-Overview a statistical approach
- 2. Machine Learning Before we jump into Neural Networks, Tensorflow, Keras API etc its a good idea to understand a few fundamental ideas regarding machine learning. In this section well cover some important theory and concepts surrounding machine learning.
- 3. Machine Learning Section Overview: What is Machine Learning? What is Deep Learning? Difference between Supervised and Unsupervised Learning Supervised Learning Process Evaluating performance Overfitting
- 4. What is Machine Learning? Machine learning is a method of data analysis that automates analytical model building. Using algorithms that iteratively learn from data, machine learning allows computers to find hidden insights without being explicitly programmed where to look.
- 5. What is it used for? Fraud detection. Web search results. Real-time ads on web pages Credit scoring. Prediction of equipment failures. New pricing models. Network intrusion detection. Recommendation Engines Customer Segmentation Text Sentiment Analysis Customer Churn Pattern and image recognition. Email spam filtering.
- 6. What are Neural Networks? Neural Networks are a way of modeling biological neuron systems mathematically. These networks can then be used to solve tasks that many other types of algorithms can not (e.g. image classification) Deep Learning simply refers to neural networks with more than one hidden layer.
- 7. Machine Learning There are different types of machine learning we will focus on during the next sections of the course: Supervised Learning Unsupervised Learning
- 8. Machine Learning Machine Learning Automated analytical models. Neural Networks A type of machine learning architecture modeled after biological neurons. Deep Learning A neural network with more than one hidden layer.
- 9. Machine Learning Lets begin by learning about one of the most common machine learning tasks- Supervised Learning!
- 11. Supervised Learning Supervised learning algorithms are trained using labeled examples, such as an input where the desired output is known. For example, a segment of text could have a category label, such as: Spam vs. Legitimate Email Positive vs. Negative Movie Review
- 12. Supervised Learning The network receives a set of inputs along with the corresponding correct outputs, and the algorithm learns by comparing its actual output with correct outputs to find errors. It then modifies the model accordingly.
- 13. Supervised Learning Supervised learning is commonly used in applications where historical data predicts likely future events.
- 14. Machine Learning Process Data Acquisition Data Cleaning Test Data Model Training & Building Model Testing Model Deployment
- 15. Machine Learning Process Data Acquisition Get your data! Customers, Sensors, etc...
- 16. Machine Learning Process Data Acquisition Data Cleaning Clean and format your data (using Pandas)
- 19. Machine Learning Process Data Acquisition Data Cleaning Test Data Model Training & Building Model Testing
- 20. Machine Learning Process Data Acquisition Data Cleaning Test Data Model Training & Building Model Testing Adjust Model Parameters
- 21. Machine Learning Process Data Acquisition Data Cleaning Test Data Model Training & Building Model Testing Model Deployment
- 22. Supervised Learning What we just showed is a simplified approach to supervised learning, it contains an issue! Is it fair to use our single split of the data to evaluate our models performance? After all, we were given the chance to update the model parameters again and again.
- 23. Supervised Learning To fix this issue, data is often split into 3 sets Training Data Used to train model parameters Validation Data Used to determine what model hyperparameters to adjust Test Data Used to get some final performance metric
- 24. Supervised Learning This means after we see the results on the final test set we dont get to go back and adjust any model parameters! This final measure is what we label the true performance of the model to be.
- 25. Supervised Learning In this course, in general we will simplify our data by using a simple train/test split. We will simply train and then evaluate on a test set (leaving the option to students to go back and adjust parameters). After going through the course, you will be able to easily perform another split to get 3 data sets if you desire.
- 27. Machine Learning Now that we understand the full process for supervised learning, lets touch upon the important topics of overfitting and underfitting.
- 28. Machine Learning Overfitting The model fits too much to the noise from the data. This often results in low error on training sets but high error on test/validation sets.
- 35. Machine Learning Underfitting Model does not capture the underlying trend of the data and does not fit the data well enough. Low variance but high bias. Underfitting is often a result of an excessively simple model.
- 38. Machine Learning This data was easy to visualize, but how can we see underfitting and overfitting when dealing with multi dimensional data sets? First lets imagine we trained a model and then measured its error over training time.
- 39. Machine Learning Good Model Error Training Time
- 40. Machine Learning Good Model Error Epochs
- 41. Machine Learning Bad Model Error Epochs
- 42. Machine Learning When thinking about overfitting and underfitting we want to keep in mind the relationship of model performance on the training set versus the test/validation set.
- 43. Machine Learning Lets imagine we split our data into a training set and a test set
- 44. Machine Learning We first see performance on the training set Epochs Error
- 45. Machine Learning Next we check performance on the test set Epochs Error
- 46. Machine Learning Ideally the model would perform well on both, with similar behavior. Epochs Error
- 47. Machine Learning But what happens if we overfit on the training data? That means we would perform poorly on new test data! Epochs Error
- 48. Machine Learning But what happens if we overfit on the training data? That means we would perform poorly on new test data! Epochs Error
- 49. Machine Learning This is a good indication of training too much on the training data, you should look for the point to cut off training time! Epochs Error
- 50. Machine Learning Well check on this idea again when we actually begin creating models! For now just be aware of this possible issue!
- 52. Model Evaluation We just learned that after our machine learning process is complete, we will use performance metrics to evaluate how our model did. Lets discuss classification metrics in more detail!
- 53. Model Evaluation The key classification metrics we need to understand are: Accuracy Recall Precision F1 -Score
- 54. Model Evaluation But first, we should understand the reasoning behind these metrics and how they will actually work in the real world!
- 55. Model Evaluation Typically in any classification task your model can only achieve two results: Either your model was correct in its prediction. Or your model was incorrect in its prediction.
- 56. Model Evaluation Fortunately incorrect vs correct expands to situations where you have multiple classes. For the purposes of explaining the metrics, lets imagine a binary classification situation, where we only have two available classes.
- 57. Model Evaluation In our example, we will attempt to predict if an image is a dog or a cat. Since this is supervised learning, we will first fit/train a model on training data, then test the model on testing data. Once we have the models predictions from the X_test data, we compare it to the true y values (the correct labels).
- 59. Model Evaluation Test Image from X_test TRAINED MODEL
- 60. Model Evaluation Test Image from X_test Correct Label from y_test DOG TRAINED MODEL
- 61. Model Evaluation Test Image from X_test Correct Label from y_test DOG TRAINED MODEL DOG Prediction on Test Image
- 62. Model Evaluation Test Image from X_test Correct Label from y_test DOG TRAINED MODEL DOG Prediction on Test Image DOG == DOG ? Compare Prediction to Correct Label
- 63. Model Evaluation Test Image from X_test Correct Label from y_test DOG TRAINED MODEL CAT Prediction on Test Image DOG == CAT ? Compare Prediction to Correct Label
- 64. Model Evaluation We repeat this process for all the images in our X test data. At the end we will have a count of correct matches and a count of incorrect matches. The key realization we need to make, is that in the real world, not all incorrect or correct matches hold equal value!
- 65. Model Evaluation Also in the real world, a single metric wont tell the complete story! To understand all of this, lets bring back the 4 metrics we mentioned and see how they are calculated. We could organize our predicted values compared to the real values in a confusion matrix.
- 66. Model Evaluation Accuracy Accuracy in classification problems is the number of correct predictions made by the model divided by the total number of predictions.
- 67. Model Evaluation Accuracy For example, if the X_test set was 1 00 images and our model correctly predicted 80 images, then we have 80/1 00. 0.8 or 80% accuracy.
- 68. Model Evaluation Accuracy Accuracy is useful when target classes are well balanced In our example, we would have roughly the same amount of cat images as we have dog images.
- 69. Model Evaluation Accuracy Accuracy is not a good choice with unbalanced classes! Imagine we had 99 images of dogs and 1 image of a cat. If our model was simply a line that always predicted dog we would get 99% accuracy!
- 70. Model Evaluation Accuracy Imagine we had 99 images of dogs and 1 image of a cat. If our model was simply a line that always predicted dog we would get 99% accuracy! In this situation well want to understand recall and precision
- 71. Model Evaluation Recall Ability of a model to find all the relevant cases within a dataset. The precise definition of recall is the number of true positives divided by the number of true positives plus the number of false negatives.
- 72. Model Evaluation Precision Ability of a classification model to identify only the relevant data points. Precision is defined as the number of true positives divided by the number of true positives plus the number of false positives.
- 73. Model Evaluation Recall and Precision Often you have a trade-off between Recall and Precision. While recall expresses the ability to find all relevant instances in a dataset, precision expresses the proportion of the data points our model says was relevant actually were relevant.
- 74. Model Evaluation F1 -Score In cases where we want to find an optimal blend of precision and recall we can combine the two metrics using what is called the F1 score.
- 75. Model Evaluation F1 -Score The F1 score is the harmonic mean of precision and recall taking both metrics into account in the following equation:
- 76. Model Evaluation F1 -Score We use the harmonic mean instead of a simple average because it punishes extreme values. A classifier with a precision of 1 .0 and a recall of 0.0 has a simple average of 0.5 but an F1 score of 0.
- 77. Model Evaluation We can also view all correctly classified versus incorrectly classified images in the form of a confusion matrix.
- 80. Math & Statistics Domain Knowledge Machine Learning Software Research DS Model Evaluation The main point to remember with the confusion matrix and the various calculated metrics is that they are all fundamentally ways of comparing the predicted values versus the true values. What constitutes good metrics, will really depend on the specific situation!
- 81. Math & Statistics Domain Knowledge Machine Learning Software Research DS Model Evaluation Still confused on the confusion matrix? No problem! Check out the Wikipedia page for it, it has a really good diagram with all the formulas for all the metrics. Throughout the training, well usually just print out metrics (e.g. accuracy).
- 82. Math & Statistics Domain Knowledge Machine Learning Software Research DS Model Evaluation Lets think back on this idea of: What is a good enough accuracy? This all depends on the context of the situation! Did you create a model to predict presence of a disease? Is the disease presence well balanced in the general population? (Probably not!)
- 83. Math & Statistics Domain Knowledge Machine Learning Software Research DS Model Evaluation Often models are used as quick diagnostic tests to have before having a more invasive test (e.g. getting urine test before getting a biopsy) We also need to consider what is at stake!
- 84. Math & Statistics Domain Knowledge Machine Learning Software Research DS Model Evaluation Often we have a precision/recall trade off, We need to decide if the model will should focus on fixing False Positives vs. False Negatives. In disease diagnosis, it is probably better to go in the direction of False positives, so we make sure we correctly classify as many cases of disease as possible!
- 85. Math & Statistics Domain Knowledge Machine Learning Software Research DS Model Evaluation All of this is to say, machine learning is not performed in a vacuum, but instead a collaborative process where we should consult with experts in the domain (e.g. medical doctors)
- 87. Evaluating Regression Lets take a moment now to discuss evaluating Regression Models Regression is a task when a model attempts to predict continuous values (unlike categorical values, which is classification)
- 88. Evaluating Regression You may have heard of some evaluation metrics like accuracy or recall. These sort of metrics arent useful for regression problems, we need metrics designed for continuous values!
- 89. Evaluating Regression For example, attempting to predict the price of a house given its features is a regression task. Attempting to predict the country a house is in given its features would be a classification task.
- 90. Evaluating Regression Lets discuss some of the most common evaluation metrics for regression: Mean Absolute Error Mean Squared Error Root Mean Square Error
- 91. Evaluating Regression Mean Absolute Error (MAE) This is the mean of the absolute value of errors. Easy to understand
- 92. Evaluating Regression MAE wont punish large errors however.
- 93. Evaluating Regression MAE wont punish large errors however.
- 94. Evaluating Regression We want our error metrics to account for these!
- 95. Evaluating Regression Mean Squared Error (MSE) This is the mean of the squared errors. Larger errors are noted more than with MAE, making MSE more popular.
- 96. Evaluating Regression Root Mean Square Error (RMSE) This is the root of the mean of the squared errors. Most popular (has same units as y)
- 97. Machine Learning Most common question from students: Is this value of RMSE good? Context is everything! A RMSE of $1 0 is fantastic for predicting the price of a house, but horrible for predicting the price of a candy bar!
- 98. Machine Learning Compare your error metric to the average value of the label in your data set to try to get an intuition of its overall performance. Domain knowledge also plays an important role here!
- 99. Machine Learning Context of importance is also necessary to consider. We may create a model to predict how much medication to give, in which case small fluctuations in RMSE may actually be very significant.
- 100. Evaluating Regression You should now feel comfortable with the various methods of evaluating a regression task.
- 102. Machine Learning Weve covered supervised learning, where the label was known due to historical labeled data. But what happens when we dont have historical labels?
- 103. Machine Learning There are certain tasks that fall under unsupervised learning: Clustering Anomaly Detection Dimensionality Reduction
- 104. Machine Learning Clustering Grouping together unlabeled data points into categories/clusters Data points are assigned to a cluster based on similarity
- 105. Machine Learning Anomaly Detection Attempts to detect outliers in a dataset For example, fraudulent transactions on a credit card.
- 106. Machine Learning Dimensionality Reduction Data processing techniques that reduces the number of features in a data set, either for compression, or to better understand underlying trends within a data set.
- 107. Machine Learning Unsupervised Learning Its important to note, these are situations where we dont have the correct answer for historical data! Which means evaluation is much harder and more nuanced!
- 108. Unsupervised Process Data Acquisition Data Cleaning Test Data Model Training & Building Transformation Model Deployment
- 109. Machine Learning Later on in the course, well explore unsupervised learning processes with specialized neural network structures, such as autoencoders.