Mzi pakdd2019 v2
Download as PPTX, PDF
0 likes58 views
This document presents a new ensemble learning method called Explanation-Based Combination Method (EBCM) that considers the reasoning behind predictions of base classifiers. EBCM extracts explanations for predictions using feature importance, measures consistency among explanations, and combines predictions based on probability and consistency. It was evaluated on 16 sentiment analysis datasets and outperformed baseline combination methods like majority voting and averaging. The contributions are incorporating explanations into ensemble learning, the EBCM method, measures of explanation consistency, and comprehensive evaluation on real-world data.
Mzi pakdd2019 v2
- 1. Semantic Explanations in Ensemble Learning Md Zahidul Islam, Jixue Liu, Lin Liu, Jiuyong Li, Wei Kang University of South Australia (UniSA) Adelaide, South Australia
- 2. Presentation outline Background Motivation Problem definition Explanation-Based Combination Method (EBCM) Experimental results
- 3. Background An ensemble classifier achieves better result than a single classifier. Two main issues related to ensemble classifiers: 1. How to train multiple classifiers? 2. How to combine the predictions of multiple classifiers? our focus
- 4. Prediction combination methods The most common one is majority voting (MV) Others include: Arithmetic functions Averaging Summation Maximum/Minimum Weighted methods Weighted averaging Weighted majority voting Weight assignment strategies:
- 5. Motivation Existing methods relies on the majority of base classi鍖ers. Main idea: the class predicted by most classi鍖ers is most likely to be the true class when the majority predict the incorrect class combined prediction is incorrect To improve: assign weights Weights are computed once and the same weights are applied to all instances in reality: better predictions for a certain type of data even though its overall accuracy is poor.
- 6. Motivation Group decision making: Con鍖icting decisions often lead to errors. Decisions (by the majority) lacking consistency in the reasons, less reliable than the minoritys decision with higher consistency. For human decision makers, we ask not only for decision, but also the reason. Example: Paper review process: 03/04 reviewers and 01 meta reviewer The decisions are made by the meta reviewer considering other scores and explanations from the reviewers. We cannot solely depend on the decisions of the majority, further explanations are needed to make reasonable decisions.
- 7. Problem definition Objective: to develop a new combination method which will consider the reasoning behind the predictions. Notations: A set of classifiers A set of target classes Given , derives a probability for and makes prediction , set of classi鍖ers supporting The combined predication: where is a combination measure such that
- 8. Problem definition For our purpose, should consider two factors: the probabilities of the class (we represent that as ). the consistency of the explanations to the prediction made by the base classi鍖ers in (we represent that as ). Now, can be obtained by: Our problem can be divided into the two following sub problems: 1. How to extract the explanations? 2. How to measure the consistency among the explanations?
- 9. Explanation-based Combination Method (EBCM) Three steps: for each , compute the aggregated probabilities extract the explanations for the predictions compute the consistency score among the explanations
- 10. Explanation-based Combination Method (EBCM) Explanation extraction: Classifiers learn different weights for the features (training phase). We extract such features as the explanations i.e. features in are most weighted by to predict the class We use LIME (a tool presented by Ribeiro et al. KDD 16)
- 11. Consistency measurement Intuition: the classifiers predicting the same class with similar explanations should be weighted more (larger value for ). predicting the same class based on different explanations, downweight their contributions (smaller value for ). Two measures Cohesion-based Measure (C) Cohesion and Separation-based Measure (CS)
- 12. Consistency measurement Cohesion-based Measure (C) based on the closeness of the explanations within the same cluster. the closer, the higher the consistency. the consistency is measured by the similarities among the explanations (cohesion).
- 13. Consistency measurement Cohesion and Separation-based Measure (CS) (Similar to silhouette coefficient from clustering) assigns a score to each point indicating how well it suits in the corresponding cluster. Interpretation of a score: a large score (closer to 1) indicates that the point is well clustered 0 means that it is on the border of two clusters a negative score means that better suited in a different cluster. a fitness score describes how an explanation fits w.r.t others.
- 14. Cohesion and Separation-based Measure (CS) The fitness function is derived from its average distance within its own cluster average distance to other clusters The consistency score is the average of the fitness scores.
- 16. Experimental settings Sentiment analysis of 16 real-world social media datasets (user reviews and tweets). Five base classifiers - NB, DT, k-NN, SVM, and LR Five baseline combination methods Majority vote Averaging Weighted averaging Maximum Sum Evaluation metrics Accuracy (10 fold cross validation) Statistical analysis Friedman test followed by post-hoc Bonferroni- Dunn test
- 20. Contributions The main contributions of our work are: The idea of including semantic explanations in ensemble learning. EBCM for incorporating semantic explanations in ensemble learning. Two measures for determining the most consistent predictions. A comprehensive evaluation of EBCM on 16 real-world social media datasets.
- 21. Thank you Any questions? For future query: http://nugget.unisa.edu.au/ictproject/dag/dagwp/ Data Analytics Group @ UniSA