Web mining of drug reviews for market analysis
Download as PPTX, PDF0 likes53 views
This document outlines a project to analyze drug reviews from WebMD to gain market insights. It discusses gathering review data from WebMD, cleaning the data, performing exploratory data analysis on the ratings and reviews. Natural language processing techniques like TF-IDF vectorization and word clouds are used to analyze common words. Classification algorithms are tested to predict satisfaction ratings from reviews. Topic modeling with LDA identifies common topics. Comparative analyses of side effects for different drugs and age groups are also performed. The conclusion notes that while WebMD reviews provide consumer sentiment, they may not be ideal for drug companies due to potential biases in reviews.
More Related Content
Similar to Web mining of drug reviews for market analysis (20)
Recently uploaded (20)
Web mining of drug reviews for market analysis
- 1. Web Mining of Drug Reviews for Market Analysis Ajinkya Ingle Rohan Waghere Priyanka Bhandari Gaurav Kshirsagar
- 2. Agenda 1. Introduction 2. Gathering data 3. Cleaning data 4. Exploratory data analysis 5. Reviews analysis 6. NLP analysis 7. Classification Algorithms 8. LDA Topic Modelling 9. Conclusion
- 3. Introduction The US has the largest pharmaceutical market in the world with a value of $339 billion USD. US prescription drug spending is expected to reach as high as $610 billion by 2021. Pfizer alone spent 7.6 billion dollars on R&D in financial year 2017. The growth is expected to accelerate in coming years. While these drugs are prescribed for their therapeutic properties, their use may result in unintended or adverse effects. There is a need for people to know the quality of drugs in this overcrowded market.
- 4. Gathering Data - Data Source We chose WebMD as our primary source to gather data. It is considered a legitimate source of information for all sorts of drugs
- 5. Gathering Data Searched for top 5 drugs for a given condition. Parsed the data from each of the drugs page. Gathered reviews and ratings from the drug review page. Stored the obtained results in a csv file.
- 6. Gathering Data We parsed all the pages to acquire all the reviews. Age, Gender, Ratings and Comments were collected in a dataframe. Beautifulsoup was used to parse the data from webpages. Ratings Comments/Reviews Age and Gender
- 7. Gathering Data - Final output The data had a lot of noise and repetitive terms. The age and gender had to be extracted from the Reviewer Details column
- 8. Cleaning Data Got rid of the repetitive terms Created separate columns for Age and Gender
- 9. Cleaning Data Got rid of rows with NA values or null. Removed Punctuations Categorized age groups Assigned genders to respective users.
- 10. Exploratory Data Analysis Stats of the sample dataset
- 11. EDA: Gender Not a balanced gender distribution Female: 67% Male: 33%
- 12. EDA: Age Use increased with the increase in age 54.7% of the analgesics users were between age group 45 to 64 13.1% of analgesics users were above the age group 65
- 13. EDA: Effectiveness Rating 26.6% of Analgesics were had the highest Effectiveness Ratings While comparable percentage of ratings were rated as the second highest
- 14. EDA: Satisfaction Ratings 32.2% of reviews had the lowest Satisfaction Ratings 27.1% of reviews had the Highest Satisfaction Rating
- 15. EDA:Comparing two or more features Satisfaction Rating vs Ease of Use Rating Ease of Using the Analgesics doesnt assure the satisfaction of the customer
- 16. Effectiveness Rating EDA:Comparing two or more features Male customers in 19-24 age category have higher effectiveness rating Whereas the opposite is true in 25-34 age category Drug has different effects in males and females in different age groups Satisfaction Rating
- 17. Reviews Analysis Most of the long reviews come from people who rated 1, 4 or 5. Maximum reviews do not contain more than 50 words. We also calculated the review length to rating correlation to be 0.068
- 18. NLP - TFIDF vectorization We used TF-IDF vectorization to identify the important words in the documents. Extracted top 5 most important words in each document by its tfidf weight Gives a basic ideas of what people are talking about more frequently Reveals side effects experienced by people
- 19. NLP - WordClouds WordCloud for female patients Migraine, headache, fever are some of the most cited problems that can be observed WordCloud for male patients Sluggish , headache, fever are some of the most cited problems that can be observed
- 20. Classification - Using Tfidf vectors and sklearn Classifying Satisfaction Rating based on comments Created 3 buckets of reviews Comments Cleaned Comments Adjective Comments Gives a broader picture of variation in the accuracy of the models
- 21. Classification - Confusion Matrices The model struggles with 1 star ratings classification . Max accuracy we obtained was of linear svm i.e. 52% We try a different approach to classify
- 22. Classification based on extracted features We estrated additional features like number of words, chars,..etc. Applied classification algorithms using these features. Compared the results with previous method. New Features
- 23. Correlation of features This is a correlation plot where we can see satisfaction rating which is our chosen variable contrasted with other variables to see how they impact. We see effectiveness rating and satisfaction rating have 0.86 correlation which depicts a high correlation.
- 24. Age-wise ratings distribution Also for example I have shown satisfaction rating distribution for every age interval and we can thus say that age group 45- 54 and 55-64 were highly satisfied with their feedback of prescribed medicines.
- 25. Comparison of Different Classification Algorithms Classifying Satisfaction Rating based on extracted features Demonstrating different accuracies for our decision variable in contrast to other variables: KNN=40% Naive Bayes=59% Logistic Regression=60.5%
- 26. Comparison of Different Classification Algorithms SVM=43.5% Random Forest=61% Neural Networks=59% Thus we can say Logistic regression outperforms other algorithms in our scenario.
- 27. Parameters Tuning We used GridSearch parameters tuning to improve accuracy in MLP classifier But no significant performance improvement was observed. Average precision of 50% was obtained. Neural Networks gets better with more data.
- 28. LDA - Topic Modeling (Without TFIDF) Top 7 topics and their most frequent words. Headache and addictive are some of the perceived side-effects
- 29. LDA - Topic Modeling (With TFIDF) 1st topic mentions the positive words, it means it works for a fair amount of customers However, 2nd Topic shows the side effects like Dizziness and Itching Also, People who need more potent drugs use Vicodin instead of Tramadol Fibromyalgia is widespread muscle pain in back and shoulder. That one of the reason patients are prescribed Tramadol to reduce pain
- 30. Sentiment Analysis More than half of the comments are observed to be subjective. Therefore the reliability reduced significantly
- 31. Analysis of Drug Side-effects - Comparative Study OxyContinOxyCodone We have tried to analyze the side-effects of drugs and how they vary based on the age groups and drug.
- 32. Analysis of Drug Side-effects - Comparative Study Methadone Morphine
- 33. Tramadol Oral All Drugs Analysis of Drug Side-effects - Comparative Study This data can also be used by doctors, so that while prescribing these medicines he can also prescribe other medicines to reduce the side-effects.
- 34. Conclusion WebMD reviews are not the ideal place for drug companies to look for insights since people are inherently biased either along a positive side or a negative one. However, it is a good source to know the customers sentiments, the drug side-effects and study competitors drugs. Headaches: When Is It an Emergency? The first page contains no hard facts you have to click and thereby drive up the sites lucrative click-throughs but instead quickly transforms visitors from Web users with headaches to hard-core migraineurs and drug. - Virginia Hefferman (The New York Times)
- 35. Thank You !!
Editor's Notes
- #11: Knowledge derived from Analysis: Exploratory and Confirmatory nature Detection of Mistakes Checking for assumptions Determining relationships between explanatory variables Assessing the direction and rough size of relationships between exploratory and outcome variables Helps in preliminary selection of appropriate models of the relationship between outcome and other explanatory variables
- #21: Source : https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html