Area Under the Precision-Recall Curve: Point Estimates and Confidence Intervals
|
|
|
- Edwina Tyler
- 6 years ago
- Views:
Transcription
1 Area Under the Precision-Recall Curve: Point Estimates and Confidence Intervals Kendrick Boyd 1 Kevin H. Eng 2 C. David Page 1 1 University of Wisconsin-Madison, Madison, WI 2 Roswell Park Cancer Institute, Buffalo, NY September 26, 2013 Boyd et al. (ECML 2013) Area under PR Curve September 26,
2 Binary Classification Evaluation Receiver-operating characteristic (ROC) curves Preferred over accuracy alone [Provost et al., 1998] Insensitive to skew (π = proportion of positives) Area under ROC curve (AUCROC) TPR FPR Boyd et al. (ECML 2013) Area under PR Curve September 26,
3 Binary Classification Evaluation Receiver-operating characteristic (ROC) curves Preferred over accuracy alone [Provost et al., 1998] Insensitive to skew (π = proportion of positives) Area under ROC curve (AUCROC) Precision-recall (PR) curves Alternative to ROC curves when π near 0 [Davis and Goadrich, 2006; Goadrich et al., 2006] Sensitive to skew Area under PR curve (AUCPR) TPR precision FPR recall Boyd et al. (ECML 2013) Area under PR Curve September 26,
4 The Difficulty Given precision Do Estimate area: AUCPR = 0.5 Obtain confidence interval: [0.4, 0.6] recall Boyd et al. (ECML 2013) Area under PR Curve September 26,
5 Empirical PR Points 5 positives (n) 15 negatives (m) π = 0.25 score true label pos 0.95 neg 0.90 neg 0.85 pos 0.80 pos 0.75 neg 0.70 neg 0.65 neg Boyd et al. (ECML 2013) Area under PR Curve September 26,
6 Empirical PR Points 5 positives (n) 15 negatives (m) π = 0.25 score true label pos 0.95 neg 0.90 neg 0.85 pos 0.80 pos 0.75 neg 0.70 neg 0.65 neg precision Recall = 1 5 Precision = recall Boyd et al. (ECML 2013) Area under PR Curve September 26,
7 Empirical PR Points 5 positives (n) 15 negatives (m) π = 0.25 score true label pos 0.95 neg 0.90 neg 0.85 pos 0.80 pos 0.75 neg 0.70 neg 0.65 neg precision Recall = 1 5 Precision = recall Boyd et al. (ECML 2013) Area under PR Curve September 26,
8 Empirical PR Points 5 positives (n) 15 negatives (m) π = 0.25 score true label pos 0.95 neg 0.90 neg 0.85 pos 0.80 pos 0.75 neg 0.70 neg 0.65 neg precision Recall = 1 5 Precision = recall Boyd et al. (ECML 2013) Area under PR Curve September 26,
9 Empirical PR Points 5 positives (n) 15 negatives (m) π = 0.25 score true label pos 0.95 neg 0.90 neg 0.85 pos 0.80 pos 0.75 neg 0.70 neg 0.65 neg precision Recall = 2 5 Precision = recall Boyd et al. (ECML 2013) Area under PR Curve September 26,
10 Empirical PR Points 5 positives (n) 15 negatives (m) π = 0.25 score true label pos 0.95 neg 0.90 neg 0.85 pos 0.80 pos 0.75 neg 0.70 neg 0.65 neg precision Recall = 3 5 Precision = recall Boyd et al. (ECML 2013) Area under PR Curve September 26,
11 Empirical PR Points 5 positives (n) 15 negatives (m) π = 0.25 score true label pos 0.95 neg 0.90 neg 0.85 pos 0.80 pos 0.75 neg 0.70 neg 0.65 neg precision Recall = 3 5 Precision = recall Boyd et al. (ECML 2013) Area under PR Curve September 26,
12 Empirical PR Points 5 positives (n) 15 negatives (m) π = 0.25 score true label pos 0.95 neg 0.90 neg 0.85 pos 0.80 pos 0.75 neg 0.70 neg 0.65 neg precision recall Boyd et al. (ECML 2013) Area under PR Curve September 26,
13 AUCPR Estimators Many existing methods to estimate AUCPR Boyd et al. (ECML 2013) Area under PR Curve September 26,
14 AUCPR Estimators Many existing methods to estimate AUCPR 0.75 precision lower trap recall [Abeel et al., 2009] Boyd et al. (ECML 2013) Area under PR Curve September 26,
15 AUCPR Estimators Many existing methods to estimate AUCPR 0.75 precision lower trap upper trap recall [Abeel et al., 2009; Davis and Goadrich, 2006] Boyd et al. (ECML 2013) Area under PR Curve September 26,
16 AUCPR Estimators Many existing methods to estimate AUCPR 0.75 precision lower trap upper trap avg prec recall [Manning et al., 2008] Boyd et al. (ECML 2013) Area under PR Curve September 26,
17 AUCPR Estimators Many existing methods to estimate AUCPR precision lower trap upper trap avg prec interp conv recall [Davis and Goadrich, 2006] Boyd et al. (ECML 2013) Area under PR Curve September 26,
18 AUCPR Estimators Many existing methods to estimate AUCPR 0.75 lower trap upper trap precision avg prec interp conv interp max interp med interp mean recall Boyd et al. (ECML 2013) Area under PR Curve September 26,
19 AUCPR Estimators Many existing methods to estimate AUCPR precision recall lower trap upper trap avg prec interp conv interp max interp med interp mean binormal Boyd et al. (ECML 2013) Area under PR Curve September 26,
20 Estimator Evaluation Goals Estimator Desiderata Unbiased: average estimate is equal to true AUCPR Boyd et al. (ECML 2013) Area under PR Curve September 26,
21 Estimator Evaluation Goals Estimator Desiderata Unbiased: average estimate is equal to true AUCPR Robust to different output distributions Boyd et al. (ECML 2013) Area under PR Curve September 26,
22 Estimator Evaluation Goals Estimator Desiderata Unbiased: average estimate is equal to true AUCPR Robust to different output distributions Robust to various skews (π) and data sizes (n + m) Boyd et al. (ECML 2013) Area under PR Curve September 26,
23 Simulation Setup Assume example scores are drawn from known distributions Similar to binormal analysis on ROC curves [Pepe, 2004; Bamber, 1975] Allows calculation of true PR curve and AUCPR Boyd et al. (ECML 2013) Area under PR Curve September 26,
24 Simulation Setup Assume example scores are drawn from known distributions Similar to binormal analysis on ROC curves [Pepe, 2004; Bamber, 1975] Allows calculation of true PR curve and AUCPR Analyzed distributions Binormal density precision Binormal score recall negatives N(0, 1) positives N(1, 1) negatives positives Sample Curve True Curve Boyd et al. (ECML 2013) Area under PR Curve September 26,
25 Simulation Setup Assume example scores are drawn from known distributions Similar to binormal analysis on ROC curves [Pepe, 2004; Bamber, 1975] Allows calculation of true PR curve and AUCPR Analyzed distributions Binormal Bibeta density precision Bibeta score recall negatives Beta(2, 5) positives Beta(5, 2) negatives positives Sample Curve True Curve Boyd et al. (ECML 2013) Area under PR Curve September 26,
26 Simulation Setup Assume example scores are drawn from known distributions Similar to binormal analysis on ROC curves [Pepe, 2004; Bamber, 1975] Allows calculation of true PR curve and AUCPR Analyzed distributions Binormal Bibeta Offset uniform density precision Offset Uniform score recall negatives U(0, 1) positives U(0.5, 1.5) negatives positives Sample Curve True Curve Boyd et al. (ECML 2013) Area under PR Curve September 26,
27 Simulation Setup Assume example scores are drawn from known distributions Similar to binormal analysis on ROC curves [Pepe, 2004; Bamber, 1975] Allows calculation of true PR curve and AUCPR Analyzed distributions Binormal Bibeta Offset uniform Additional parameters # of examples (n + m) skew (π = 0.1) Boyd et al. (ECML 2013) Area under PR Curve September 26,
28 AUCPR Estimator Results 1.4 binormal 1.06 bibeta bias ratio (n=10) offset uniform 1000 (n=100) (n=1000) n + m interp conv upper trap interp max avg prec lower trap interp mean interp med binormal Boyd et al. (ECML 2013) Area under PR Curve September 26,
29 AUCPR Estimator Results 1.4 binormal 1.06 bibeta bias ratio (n=10) offset uniform 1000 (n=100) (n=1000) n + m interp conv upper trap interp max avg prec lower trap interp mean interp med binormal Boyd et al. (ECML 2013) Area under PR Curve September 26,
30 AUCPR Confidence Intervals Definition A 1 α confidence interval is an interval that contains the true value with probability at least (1 α). [Wasserman, 2004] Boyd et al. (ECML 2013) Area under PR Curve September 26,
31 AUCPR Confidence Intervals Definition A 1 α confidence interval is an interval that contains the true value with probability at least (1 α). [Wasserman, 2004] Empirical Cross-validation: compute interval using mean and variance of K estimates from K-fold cross-validation Boyd et al. (ECML 2013) Area under PR Curve September 26,
32 AUCPR Confidence Intervals Definition A 1 α confidence interval is an interval that contains the true value with probability at least (1 α). [Wasserman, 2004] Empirical Cross-validation: compute interval using mean and variance of K estimates from K-fold cross-validation Bootstrap: choose interval that contains (1 α)% of empirical distribution of AUCPR estimates Boyd et al. (ECML 2013) Area under PR Curve September 26,
33 AUCPR Confidence Intervals Definition A 1 α confidence interval is an interval that contains the true value with probability at least (1 α). [Wasserman, 2004] Empirical Cross-validation: compute interval using mean and variance of K estimates from K-fold cross-validation Bootstrap: choose interval that contains (1 α)% of empirical distribution of AUCPR estimates Parametric Binomial: ˆθ ± Φ 1 α/2 ˆθ(1 ˆθ) n ˆθ is the estimated AUCPR Boyd et al. (ECML 2013) Area under PR Curve September 26,
34 AUCPR Confidence Intervals Definition A 1 α confidence interval is an interval that contains the true value with probability at least (1 α). [Wasserman, 2004] Empirical Cross-validation: compute interval using mean and variance of K estimates from K-fold cross-validation Bootstrap: choose interval that contains (1 α)% of empirical distribution of AUCPR estimates Parametric Binomial: ˆθ ± Φ ˆθ(1 ˆθ) 1 α/2 n ˆθ is the[ estimated AUCPR ] ˆη Φ(1 α/2) ˆτ e e Logit:, ˆη+Φ(1 α/2) ˆτ 1+e ˆη Φ(1 α/2) ˆτ 1+e ˆη+Φ(1 α/2) ˆτ ˆη = log ˆθ, ˆτ = (nˆθ(1 1/2 ˆθ)) 1 ˆθ Boyd et al. (ECML 2013) Area under PR Curve September 26,
35 Confidence Interval Evaluation Goals Confidence Interval Desiderata Valid - at least (1 α)% coverage Boyd et al. (ECML 2013) Area under PR Curve September 26,
36 Confidence Interval Evaluation Goals Confidence Interval Desiderata Valid - at least (1 α)% coverage Prefer narrower (but still valid) Boyd et al. (ECML 2013) Area under PR Curve September 26,
37 Confidence Interval Evaluation Goals Confidence Interval Desiderata Valid - at least (1 α)% coverage Prefer narrower (but still valid) Robust to different output distributions Boyd et al. (ECML 2013) Area under PR Curve September 26,
38 Confidence Interval Evaluation Goals Confidence Interval Desiderata Valid - at least (1 α)% coverage Prefer narrower (but still valid) Robust to different output distributions Robust to various skews (π) and data sizes (n + m) Boyd et al. (ECML 2013) Area under PR Curve September 26,
39 AUCPR Confidence Interval Results binormal bibeta offset uniform 0.95 coverage (n=20) 1000 (n=100) (n=500) (n=20) 1000 (n=100) n+m (n=500) (n=20) 1000 (n=100) 5000 (n=500) estimator lower trap interval cross validation Boyd et al. (ECML 2013) Area under PR Curve September 26,
40 AUCPR Confidence Interval Results binormal bibeta offset uniform 0.95 coverage (n=20) 1000 (n=100) (n=500) (n=20) 1000 (n=100) n+m (n=500) (n=20) 1000 (n=100) 5000 (n=500) estimator avg prec lower trap interp med interval cross validation Boyd et al. (ECML 2013) Area under PR Curve September 26,
41 AUCPR Confidence Interval Results coverage binormal bibeta offset uniform (n=20) 1000 (n=100) (n=500) (n=20) 1000 (n=100) n+m (n=500) (n=20) 1000 (n=100) 5000 (n=500) estimator avg prec lower trap interp med interval logit binomial bootstrap cross validation Boyd et al. (ECML 2013) Area under PR Curve September 26,
42 Conclusion Choice of AUCPR estimator and confidence interval is important Particularly for small data sets Boyd et al. (ECML 2013) Area under PR Curve September 26,
43 Conclusion Choice of AUCPR estimator and confidence interval is important Particularly for small data sets Recommended estimators Lower trapezoid Average precision Interpolated median Boyd et al. (ECML 2013) Area under PR Curve September 26,
44 Conclusion Choice of AUCPR estimator and confidence interval is important Particularly for small data sets Recommended estimators Lower trapezoid Average precision Interpolated median Recommended confidence intervals Binomial Logit Boyd et al. (ECML 2013) Area under PR Curve September 26,
45 Conclusion Choice of AUCPR estimator and confidence interval is important Particularly for small data sets Recommended estimators Lower trapezoid Average precision Interpolated median Recommended confidence intervals Binomial Logit What about cross-validation and bootstrap? Converge to proper coverage, but from below Problematic for small data sets and low numbers of positive examples Boyd et al. (ECML 2013) Area under PR Curve September 26,
46 Thank You Questions? Acknowledgments NIGMS grant R01GM NLM grant R01LM UW Carbone Cancer Center ICTR NIH NCA TS grant UL1TR CIBM Training Program grant 5T15LM Roswell Park Cancer Institute NCI grant P30 CA Boyd et al. (ECML 2013) Area under PR Curve September 26,
47 AUCPR Estimators Results by π binormal bibeta bias ratio 2 offset uniform 0.8 interp conv lower trap 0.75 upper trap interp mean (n=10) 0.1 (n=100) 0.5 (n=500) π interp max avg prec interp med binormal Boyd et al. (ECML 2013) Area under PR Curve September 26,
48 AUCPR Confidence Interval Widths coverage binormal bibeta offset uniform width ratio estimator avg prec lower trap interp med interval logit binomial bootstrap cross validation Boyd et al. (ECML 2013) Area under PR Curve September 26,
49 AUCPR Confidence Interval Locations bias ratio binormal bibeta offset uniform (n=20) 1000 (n=100) 5000 (n=500) 200 (n=20) (n=100) (n=500) n+m 200 (n=20) 1000 (n=100) 5000 (n=500) estimator avg prec lower trap interp med interval binomial bootstrap cross validation Boyd et al. (ECML 2013) Area under PR Curve September 26,
50 References I Thomas Abeel, Yves Van de Peer, and Yvan Saeys. Toward a gold standard for promoter prediction evaluation. Bioinformatics, 25(12):i313 i320, Donald Bamber. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology, 12(4): , Jesse Davis and Mark Goadrich. The relationship between precision-recall and ROC curves. In Proceedings of the 23rd International Conference on Machine learning, ICML 06, pages , New York, NY, USA, ACM. Mark Goadrich, Louis Oliphant, and Jude Shavlik. Gleaner: Creating ensembles of first-order clauses to improve recall-precision curves. Machine Learning, 64: , Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to Information Retrieval. Cambridge University Press, New York, NY, USA, Margaret Sullivan Pepe. The statistical evaluation of medical tests for classification and prediction. Oxford University Press, USA, Boyd et al. (ECML 2013) Area under PR Curve September 26,
51 References II Foster J Provost, Tom Fawcett, and Ron Kohavi. The case against accuracy estimation for comparing induction algorithms. In ICML, volume 98, pages , Larry Wasserman. All of statistics: A concise course in statistical inference. Springer Verlag, Boyd et al. (ECML 2013) Area under PR Curve September 26,
Unachievable Region in Precision-Recall Space and Its Effect on Empirical Evaluation
Unachievable Region in Precision-Recall Space and Its Effect on Empirical Evaluation Kendrick Boyd Vitor Santos Costa Jesse Davis David Page May 30, 2012 Abstract Precision-recall (PR) curves and the areas
Pointwise Exact Bootstrap Distributions of Cost Curves
Pointwise Exact Bootstrap Distributions of Cost Curves Charles Dugas and David Gadoury University of Montréal 25th ICML Helsinki July 2008 Dugas, Gadoury (U Montréal) Cost curves July 8, 2008 1 / 24 Outline
Hypothesis Evaluation
Hypothesis Evaluation Machine Learning Hamid Beigy Sharif University of Technology Fall 1395 Hamid Beigy (Sharif University of Technology) Hypothesis Evaluation Fall 1395 1 / 31 Table of contents 1 Introduction
Performance evaluation of binary classifiers
Performance evaluation of binary classifiers Kevin P. Murphy Last updated October 10, 2007 1 ROC curves We frequently design systems to detect events of interest, such as diseases in patients, faces in
Classification Ensemble That Maximizes the Area Under Receiver Operating Characteristic Curve (AUC)
Classification Ensemble That Maximizes the Area Under Receiver Operating Characteristic Curve (AUC) Eunsik Park 1 and Y-c Ivan Chang 2 1 Chonnam National University, Gwangju, Korea 2 Academia Sinica, Taipei,
Data Privacy in Biomedicine. Lecture 11b: Performance Measures for System Evaluation
Data Privacy in Biomedicine Lecture 11b: Performance Measures for System Evaluation Bradley Malin, PhD (b.malin@vanderbilt.edu) Professor of Biomedical Informatics, Biostatistics, & Computer Science Vanderbilt
EXAM IN STATISTICAL MACHINE LEARNING STATISTISK MASKININLÄRNING
EXAM IN STATISTICAL MACHINE LEARNING STATISTISK MASKININLÄRNING DATE AND TIME: June 9, 2018, 09.00 14.00 RESPONSIBLE TEACHER: Andreas Svensson NUMBER OF PROBLEMS: 5 AIDING MATERIAL: Calculator, mathematical
Stephen Scott.
1 / 35 (Adapted from Ethem Alpaydin and Tom Mitchell) sscott@cse.unl.edu In Homework 1, you are (supposedly) 1 Choosing a data set 2 Extracting a test set of size > 30 3 Building a tree on the training
Estimating Optimum Linear Combination of Multiple Correlated Diagnostic Tests at a Fixed Specificity with Receiver Operating Characteristic Curves
Journal of Data Science 6(2008), 1-13 Estimating Optimum Linear Combination of Multiple Correlated Diagnostic Tests at a Fixed Specificity with Receiver Operating Characteristic Curves Feng Gao 1, Chengjie
SUPERVISED LEARNING: INTRODUCTION TO CLASSIFICATION
SUPERVISED LEARNING: INTRODUCTION TO CLASSIFICATION 1 Outline Basic terminology Features Training and validation Model selection Error and loss measures Statistical comparison Evaluation measures 2 Terminology
Data Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA
SUPPORT VECTOR REGRESSION WITH A GENERALIZED QUADRATIC LOSS
SUPPORT VECTOR REGRESSION WITH A GENERALIZED QUADRATIC LOSS Filippo Portera and Alessandro Sperduti Dipartimento di Matematica Pura ed Applicata Universit a di Padova, Padova, Italy {portera,sperduti}@math.unipd.it
Learning Classification with Auxiliary Probabilistic Information Quang Nguyen Hamed Valizadegan Milos Hauskrecht
Learning Classification with Auxiliary Probabilistic Information Quang Nguyen Hamed Valizadegan Milos Hauskrecht Computer Science Department University of Pittsburgh Outline Introduction Learning with
CSC314 / CSC763 Introduction to Machine Learning
CSC314 / CSC763 Introduction to Machine Learning COMSATS Institute of Information Technology Dr. Adeel Nawab More on Evaluating Hypotheses/Learning Algorithms Lecture Outline: Review of Confidence Intervals
How do we compare the relative performance among competing models?
How do we compare the relative performance among competing models? 1 Comparing Data Mining Methods Frequent problem: we want to know which of the two learning techniques is better How to reliably say Model
Performance Evaluation and Comparison
Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Introduction 2 Cross Validation and Resampling 3 Interval Estimation
Performance Evaluation
Performance Evaluation David S. Rosenberg Bloomberg ML EDU October 26, 2017 David S. Rosenberg (Bloomberg ML EDU) October 26, 2017 1 / 36 Baseline Models David S. Rosenberg (Bloomberg ML EDU) October 26,
MINIMUM EXPECTED RISK PROBABILITY ESTIMATES FOR NONPARAMETRIC NEIGHBORHOOD CLASSIFIERS. Maya Gupta, Luca Cazzanti, and Santosh Srivastava
MINIMUM EXPECTED RISK PROBABILITY ESTIMATES FOR NONPARAMETRIC NEIGHBORHOOD CLASSIFIERS Maya Gupta, Luca Cazzanti, and Santosh Srivastava University of Washington Dept. of Electrical Engineering Seattle,
Machine Learning for natural language processing
Machine Learning for natural language processing Classification: Naive Bayes Laura Kallmeyer Heinrich-Heine-Universität Düsseldorf Summer 2016 1 / 20 Introduction Classification = supervised method for
Distribution-free ROC Analysis Using Binary Regression Techniques
Distribution-free Analysis Using Binary Techniques Todd A. Alonzo and Margaret S. Pepe As interpreted by: Andrew J. Spieker University of Washington Dept. of Biostatistics Introductory Talk No, not that!
Data Mining: Concepts and Techniques. (3 rd ed.) Chapter 8. Chapter 8. Classification: Basic Concepts
Data Mining: Concepts and Techniques (3 rd ed.) Chapter 8 1 Chapter 8. Classification: Basic Concepts Classification: Basic Concepts Decision Tree Induction Bayes Classification Methods Rule-Based Classification
Appendix 1 PROOFS 1.1 PROOF OF LEMMA 1
Appendix 1 PROOFS In this section, we provide proofs for all the lemmas and theorems in the main paper. We always assume that a class-conditional extension of the Classification Noise Process (CNP) (Angluin
Evaluation & Credibility Issues
Evaluation & Credibility Issues What measure should we use? accuracy might not be enough. How reliable are the predicted results? How much should we believe in what was learned? Error on the training data
Regression III Lecture 1: Preliminary
Regression III Lecture 1: Preliminary Dave Armstrong University of Western Ontario Department of Political Science Department of Statistics and Actuarial Science (by courtesy) e: dave.armstrong@uwo.ca
Non-Parametric Estimation of ROC Curves in the Absence of a Gold Standard
UW Biostatistics Working Paper Series 7-30-2004 Non-Parametric Estimation of ROC Curves in the Absence of a Gold Standard Xiao-Hua Zhou University of Washington, azhou@u.washington.edu Pete Castelluccio
AN EVALUATION OF PARAMETRIC AND NONPARAMETRIC VARIANCE ESTIMATORS IN COMPLETELY RANDOMIZED EXPERIMENTS. Stanley A. Lubanski. and. Peter M.
AN EVALUATION OF PARAMETRIC AND NONPARAMETRIC VARIANCE ESTIMATORS IN COMPLETELY RANDOMIZED EXPERIMENTS by Stanley A. Lubanski and Peter M. Steiner UNIVERSITY OF WISCONSIN-MADISON 018 Background To make
Finite Population Correction Methods
Finite Population Correction Methods Moses Obiri May 5, 2017 Contents 1 Introduction 1 2 Normal-based Confidence Interval 2 3 Bootstrap Confidence Interval 3 4 Finite Population Bootstrap Sampling 5 4.1
Cost-based classifier evaluation for imbalanced problems
Cost-based classifier evaluation for imbalanced problems Thomas Landgrebe, Pavel Paclík, David M.J. Tax, Serguei Verzakov, and Robert P.W. Duin Elect. Eng., Maths and Comp. Sc., Delft University of Technology,
Boosting SVM Classifiers with Logistic Regression
Boosting SVM Classifiers with Logistic Regression Yuan-chin Ivan Chang Institute of Statistical Science Academia Sinica Taipei, Taiwan 11529, ROC ycchang@sinica.edu.tw Editor: Abstract The support vector
Concentration-based Delta Check for Laboratory Error Detection
Northeastern University Department of Electrical and Computer Engineering Concentration-based Delta Check for Laboratory Error Detection Biomedical Signal Processing, Imaging, Reasoning, and Learning (BSPIRAL)
Machine Learning Linear Classification. Prof. Matteo Matteucci
Machine Learning Linear Classification Prof. Matteo Matteucci Recall from the first lecture 2 X R p Regression Y R Continuous Output X R p Y {Ω 0, Ω 1,, Ω K } Classification Discrete Output X R p Y (X)
A noninformative Bayesian approach to domain estimation
A noninformative Bayesian approach to domain estimation Glen Meeden School of Statistics University of Minnesota Minneapolis, MN 55455 glen@stat.umn.edu August 2002 Revised July 2003 To appear in Journal
Cross Validation & Ensembling
Cross Validation & Ensembling Shan-Hung Wu shwu@cs.nthu.edu.tw Department of Computer Science, National Tsing Hua University, Taiwan Machine Learning Shan-Hung Wu (CS, NTHU) CV & Ensembling Machine Learning
Methods and Criteria for Model Selection. CS57300 Data Mining Fall Instructor: Bruno Ribeiro
Methods and Criteria for Model Selection CS57300 Data Mining Fall 2016 Instructor: Bruno Ribeiro Goal } Introduce classifier evaluation criteria } Introduce Bias x Variance duality } Model Assessment }
PDEEC Machine Learning 2016/17
PDEEC Machine Learning 2016/17 Lecture - Model assessment, selection and Ensemble Jaime S. Cardoso jaime.cardoso@inesctec.pt INESC TEC and Faculdade Engenharia, Universidade do Porto Nov. 07, 2017 1 /
Some Assorted Formulae. Some confidence intervals: σ n. x ± z α/2. x ± t n 1;α/2 n. ˆp(1 ˆp) ˆp ± z α/2 n. χ 2 n 1;1 α/2. n 1;α/2
STA 248 H1S MIDTERM TEST February 26, 2008 SURNAME: SOLUTIONS GIVEN NAME: STUDENT NUMBER: INSTRUCTIONS: Time: 1 hour and 50 minutes Aids allowed: calculator Tables of the standard normal, t and chi-square
Classifier performance evaluation
Classifier performance evaluation Václav Hlaváč Czech Technical University in Prague Czech Institute of Informatics, Robotics and Cybernetics 166 36 Prague 6, Jugoslávských partyzánu 1580/3, Czech Republic
Introduction and Descriptive Statistics p. 1 Introduction to Statistics p. 3 Statistics, Science, and Observations p. 5 Populations and Samples p.
Preface p. xi Introduction and Descriptive Statistics p. 1 Introduction to Statistics p. 3 Statistics, Science, and Observations p. 5 Populations and Samples p. 6 The Scientific Method and the Design of
An Introduction to Statistical Machine Learning - Theoretical Aspects -
An Introduction to Statistical Machine Learning - Theoretical Aspects - Samy Bengio bengio@idiap.ch Dalle Molle Institute for Perceptual Artificial Intelligence (IDIAP) CP 592, rue du Simplon 4 1920 Martigny,
Estimating the accuracy of a hypothesis Setting. Assume a binary classification setting
Estimating the accuracy of a hypothesis Setting Assume a binary classification setting Assume input/output pairs (x, y) are sampled from an unknown probability distribution D = p(x, y) Train a binary classifier
ROC Curve Analysis for Randomly Selected Patients
INIAN INSIUE OF MANAGEMEN AHMEABA INIA ROC Curve Analysis for Randomly Selected Patients athagata Bandyopadhyay Sumanta Adhya Apratim Guha W.P. No. 015-07-0 July 015 he main objective of the working paper
A Simulation Comparison Study for Estimating the Process Capability Index C pm with Asymmetric Tolerances
Available online at ijims.ms.tku.edu.tw/list.asp International Journal of Information and Management Sciences 20 (2009), 243-253 A Simulation Comparison Study for Estimating the Process Capability Index
The Normal Distribution. Chapter 6
+ The Normal Distribution Chapter 6 + Applications of the Normal Distribution Section 6-2 + The Standard Normal Distribution and Practical Applications! We can convert any variable that in normally distributed
CptS 570 Machine Learning School of EECS Washington State University. CptS Machine Learning 1
CptS 570 Machine Learning School of EECS Washington State University CptS 570 - Machine Learning 1 IEEE Expert, October 1996 CptS 570 - Machine Learning 2 Given sample S from all possible examples D Learner
Subject CS1 Actuarial Statistics 1 Core Principles
Institute of Actuaries of India Subject CS1 Actuarial Statistics 1 Core Principles For 2019 Examinations Aim The aim of the Actuarial Statistics 1 subject is to provide a grounding in mathematical and
Diagnostics. Gad Kimmel
Diagnostics Gad Kimmel Outline Introduction. Bootstrap method. Cross validation. ROC plot. Introduction Motivation Estimating properties of an estimator. Given data samples say the average. x 1, x 2,...,
Ensemble Methods. NLP ML Web! Fall 2013! Andrew Rosenberg! TA/Grader: David Guy Brizan
Ensemble Methods NLP ML Web! Fall 2013! Andrew Rosenberg! TA/Grader: David Guy Brizan How do you make a decision? What do you want for lunch today?! What did you have last night?! What are your favorite
Contents. Acknowledgments. xix
Table of Preface Acknowledgments page xv xix 1 Introduction 1 The Role of the Computer in Data Analysis 1 Statistics: Descriptive and Inferential 2 Variables and Constants 3 The Measurement of Variables
Linear and Logistic Regression. Dr. Xiaowei Huang
Linear and Logistic Regression Dr. Xiaowei Huang https://cgi.csc.liv.ac.uk/~xiaowei/ Up to now, Two Classical Machine Learning Algorithms Decision tree learning K-nearest neighbor Model Evaluation Metrics
A Large Deviation Bound for the Area Under the ROC Curve
A Large Deviation Bound for the Area Under the ROC Curve Shivani Agarwal, Thore Graepel, Ralf Herbrich and Dan Roth Dept. of Computer Science University of Illinois Urbana, IL 680, USA {sagarwal,danr}@cs.uiuc.edu
Sampling Strategies to Evaluate the Performance of Unknown Predictors
Sampling Strategies to Evaluate the Performance of Unknown Predictors Hamed Valizadegan Saeed Amizadeh Milos Hauskrecht Abstract The focus of this paper is on how to select a small sample of examples for
Lecture 4 Discriminant Analysis, k-nearest Neighbors
Lecture 4 Discriminant Analysis, k-nearest Neighbors Fredrik Lindsten Division of Systems and Control Department of Information Technology Uppsala University. Email: fredrik.lindsten@it.uu.se fredrik.lindsten@it.uu.se
VC dimension, Model Selection and Performance Assessment for SVM and Other Machine Learning Algorithms
03/Feb/2010 VC dimension, Model Selection and Performance Assessment for SVM and Other Machine Learning Algorithms Presented by Andriy Temko Department of Electrical and Electronic Engineering Page 2 of
Nonparametric and Semiparametric Methods in Medical Diagnostics
Nonparametric and Semiparametric Methods in Medical Diagnostics by Eunhee Kim A dissertation submitted to the faculty of the University of North Carolina at Chapel Hill in partial fulfillment of the requirements
Distribution-Free Distribution Regression
Distribution-Free Distribution Regression Barnabás Póczos, Alessandro Rinaldo, Aarti Singh and Larry Wasserman AISTATS 2013 Presented by Esther Salazar Duke University February 28, 2014 E. Salazar (Reading
SYSTEMATIC CONSTRUCTION OF ANOMALY DETECTION BENCHMARKS FROM REAL DATA. Outlier Detection And Description Workshop 2013
SYSTEMATIC CONSTRUCTION OF ANOMALY DETECTION BENCHMARKS FROM REAL DATA Outlier Detection And Description Workshop 2013 Authors Andrew Emmott emmott@eecs.oregonstate.edu Thomas Dietterich tgd@eecs.oregonstate.edu
Economics 573 Problem Set 4 Fall 2002 Due: 20 September
Economics 573 Problem Set 4 Fall 2002 Due: 20 September 1. Ten students selected at random have the following "final averages" in physics and economics. Students 1 2 3 4 5 6 7 8 9 10 Physics 66 70 50 80
Machine Learning for natural language processing
Machine Learning for natural language processing Classification: k nearest neighbors Laura Kallmeyer Heinrich-Heine-Universität Düsseldorf Summer 2016 1 / 28 Introduction Classification = supervised method
Model Accuracy Measures
Model Accuracy Measures Master in Bioinformatics UPF 2017-2018 Eduardo Eyras Computational Genomics Pompeu Fabra University - ICREA Barcelona, Spain Variables What we can measure (attributes) Hypotheses
Empirical Risk Minimization, Model Selection, and Model Assessment
Empirical Risk Minimization, Model Selection, and Model Assessment CS6780 Advanced Machine Learning Spring 2015 Thorsten Joachims Cornell University Reading: Murphy 5.7-5.7.2.4, 6.5-6.5.3.1 Dietterich,
Module 22: Bayesian Methods Lecture 9 A: Default prior selection
Module 22: Bayesian Methods Lecture 9 A: Default prior selection Peter Hoff Departments of Statistics and Biostatistics University of Washington Outline Jeffreys prior Unit information priors Empirical
Odds ratio estimation in Bernoulli smoothing spline analysis-ofvariance
The Statistician (1997) 46, No. 1, pp. 49 56 Odds ratio estimation in Bernoulli smoothing spline analysis-ofvariance models By YUEDONG WANG{ University of Michigan, Ann Arbor, USA [Received June 1995.
Better Bootstrap Confidence Intervals
by Bradley Efron University of Washington, Department of Statistics April 12, 2012 An example Suppose we wish to make inference on some parameter θ T (F ) (e.g. θ = E F X ), based on data We might suppose
Resampling Methods CAPT David Ruth, USN
Resampling Methods CAPT David Ruth, USN Mathematics Department, United States Naval Academy Science of Test Workshop 05 April 2017 Outline Overview of resampling methods Bootstrapping Cross-validation
Introduction to Supervised Learning. Performance Evaluation
Introduction to Supervised Learning Performance Evaluation Marcelo S. Lauretto Escola de Artes, Ciências e Humanidades, Universidade de São Paulo marcelolauretto@usp.br Lima - Peru Performance Evaluation
Z score indicates how far a raw score deviates from the sample mean in SD units. score Mean % Lower Bound
1 EDUR 8131 Chat 3 Notes 2 Normal Distribution and Standard Scores Questions Standard Scores: Z score Z = (X M) / SD Z = deviation score divided by standard deviation Z score indicates how far a raw score
Class Prior Estimation from Positive and Unlabeled Data
IEICE Transactions on Information and Systems, vol.e97-d, no.5, pp.1358 1362, 2014. 1 Class Prior Estimation from Positive and Unlabeled Data Marthinus Christoffel du Plessis Tokyo Institute of Technology,
Performance Evaluation
Statistical Data Mining and Machine Learning Hilary Term 2016 Dino Sejdinovic Department of Statistics Oxford Slides and other materials available at: http://www.stats.ox.ac.uk/~sejdinov/sdmml Example:
Algorithm-Independent Learning Issues
Algorithm-Independent Learning Issues Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Spring 2007 c 2007, Selim Aksoy Introduction We have seen many learning
Statistical and Learning Techniques in Computer Vision Lecture 2: Maximum Likelihood and Bayesian Estimation Jens Rittscher and Chuck Stewart
Statistical and Learning Techniques in Computer Vision Lecture 2: Maximum Likelihood and Bayesian Estimation Jens Rittscher and Chuck Stewart 1 Motivation and Problem In Lecture 1 we briefly saw how histograms
AUC Maximizing Support Vector Learning
Maximizing Support Vector Learning Ulf Brefeld brefeld@informatik.hu-berlin.de Tobias Scheffer scheffer@informatik.hu-berlin.de Humboldt-Universität zu Berlin, Department of Computer Science, Unter den
Machine Learning Ensemble Learning I Hamid R. Rabiee Jafar Muhammadi, Alireza Ghasemi Spring /
Machine Learning Ensemble Learning I Hamid R. Rabiee Jafar Muhammadi, Alireza Ghasemi Spring 2015 http://ce.sharif.edu/courses/93-94/2/ce717-1 / Agenda Combining Classifiers Empirical view Theoretical
Estimating False Negatives for Classification Problems with Cluster Structure
Estimating False Negatives for Classification Problems with Cluster Structure György J. Simon Vipin Kumar Zhi-Li Zhang Department of Computer Science, University of Minnesota {gsimon,kumar,zhzhang}@cs.umn.edu
Statistical Debugging with Latent Topic Models
Statistical Debugging with Latent Topic Models David Andrzejewski, Anne Mulhern, Ben Liblit, Xiaojin Zhu Department of Computer Sciences University of Wisconsin Madison European Conference on Machine Learning,
Smart Home Health Analytics Information Systems University of Maryland Baltimore County
Smart Home Health Analytics Information Systems University of Maryland Baltimore County 1 IEEE Expert, October 1996 2 Given sample S from all possible examples D Learner L learns hypothesis h based on
Confidence Intervals for the Area under the ROC Curve
Confidence Intervals for the Area under the ROC Curve Corinna Cortes Google Research 1440 Broadway New York, NY 10018 corinna@google.com Mehryar Mohri Courant Institute, NYU 719 Broadway New York, NY 10003
ECE521 week 3: 23/26 January 2017
ECE521 week 3: 23/26 January 2017 Outline Probabilistic interpretation of linear regression - Maximum likelihood estimation (MLE) - Maximum a posteriori (MAP) estimation Bias-variance trade-off Linear
Performance Evaluation and Hypothesis Testing
Performance Evaluation and Hypothesis Testing 1 Motivation Evaluating the performance of learning systems is important because: Learning systems are usually designed to predict the class of future unlabeled
Applied Machine Learning Annalisa Marsico
Applied Machine Learning Annalisa Marsico OWL RNA Bionformatics group Max Planck Institute for Molecular Genetics Free University of Berlin 22 April, SoSe 2015 Goals Feature Selection rather than Feature
Classification of Publications Based on Statistical Analysis of Scientific Terms Distributions
AUSTRIAN JOURNAL OF STATISTICS Volume 37 (2008), Number 1, 109 118 Classification of Publications Based on Statistical Analysis of Scientific Terms Distributions Vaidas Balys and Rimantas Rudzkis Institute
ABC random forest for parameter estimation. Jean-Michel Marin
ABC random forest for parameter estimation Jean-Michel Marin Université de Montpellier Institut Montpelliérain Alexander Grothendieck (IMAG) Institut de Biologie Computationnelle (IBC) Labex Numev! joint
A plug-in approach to maximising precision at the top and recall at the top
A plug-in approach to maximising precision at the top and recall at the top Dirk Tasche arxiv:1804.03077v1 [stat.ml] 9 Apr 2018 For information retrieval and binary classification, we show that precision
Generalization Bounds for the Area Under the ROC Curve
Generalization Bounds for the Area Under the ROC Curve Shivani Agarwal Thore Graepel Sariel Har-Peled Ralf Herbrich Dan Roth April 3, 005 Abstract We study generalization properties of the area under the
Review of Statistics 101
Review of Statistics 101 We review some important themes from the course 1. Introduction Statistics- Set of methods for collecting/analyzing data (the art and science of learning from data). Provides methods
Regularization. CSCE 970 Lecture 3: Regularization. Stephen Scott and Vinod Variyam. Introduction. Outline
Other Measures 1 / 52 sscott@cse.unl.edu learning can generally be distilled to an optimization problem Choose a classifier (function, hypothesis) from a set of functions that minimizes an objective function
Introduction to Probability and Statistics (Continued)
Introduction to Probability and Statistics (Continued) Prof. icholas Zabaras Center for Informatics and Computational Science https://cics.nd.edu/ University of otre Dame otre Dame, Indiana, USA Email:
Association studies and regression
Association studies and regression CM226: Machine Learning for Bioinformatics. Fall 2016 Sriram Sankararaman Acknowledgments: Fei Sha, Ameet Talwalkar Association studies and regression 1 / 104 Administration
Statistical Analysis of List Experiments
Statistical Analysis of List Experiments Graeme Blair Kosuke Imai Princeton University December 17, 2010 Blair and Imai (Princeton) List Experiments Political Methodology Seminar 1 / 32 Motivation Surveys
Tuning as Linear Regression
Tuning as Linear Regression Marzieh Bazrafshan, Tagyoung Chung and Daniel Gildea Department of Computer Science University of Rochester Rochester, NY 14627 Abstract We propose a tuning method for statistical
Class 4: Classification. Quaid Morris February 11 th, 2011 ML4Bio
Class 4: Classification Quaid Morris February 11 th, 211 ML4Bio Overview Basic concepts in classification: overfitting, cross-validation, evaluation. Linear Discriminant Analysis and Quadratic Discriminant
Confidence Intervals for a Ratio of Binomial Proportions Based on Unbiased Estimators
Proceedings of The 6th Sino-International Symposium Date published: October 3, 2009 on Probability, Statistics, and Quantitative Management pp. 2-5 Conference held on May 30, 2009 at Fo Guang Univ., Taiwan,
Data Mining und Maschinelles Lernen
Data Mining und Maschinelles Lernen Ensemble Methods Bias-Variance Trade-off Basic Idea of Ensembles Bagging Basic Algorithm Bagging with Costs Randomization Random Forests Boosting Stacking Error-Correcting
A multiple testing procedure for input variable selection in neural networks
A multiple testing procedure for input variable selection in neural networks MicheleLaRoccaandCiraPerna Department of Economics and Statistics - University of Salerno Via Ponte Don Melillo, 84084, Fisciano
P leiades: Subspace Clustering and Evaluation
P leiades: Subspace Clustering and Evaluation Ira Assent, Emmanuel Müller, Ralph Krieger, Timm Jansen, and Thomas Seidl Data management and exploration group, RWTH Aachen University, Germany {assent,mueller,krieger,jansen,seidl}@cs.rwth-aachen.de
Estimation of the Conditional Variance in Paired Experiments
Estimation of the Conditional Variance in Paired Experiments Alberto Abadie & Guido W. Imbens Harvard University and BER June 008 Abstract In paired randomized experiments units are grouped in pairs, often
Lehmann Family of ROC Curves
Memorial Sloan-Kettering Cancer Center From the SelectedWorks of Mithat Gönen May, 2007 Lehmann Family of ROC Curves Mithat Gonen, Memorial Sloan-Kettering Cancer Center Glenn Heller, Memorial Sloan-Kettering
Anomaly Detection. Jing Gao. SUNY Buffalo
Anomaly Detection Jing Gao SUNY Buffalo 1 Anomaly Detection Anomalies the set of objects are considerably dissimilar from the remainder of the data occur relatively infrequently when they do occur, their
JACKKNIFE EMPIRICAL LIKELIHOOD BASED CONFIDENCE INTERVALS FOR PARTIAL AREAS UNDER ROC CURVES
Statistica Sinica 22 (202), 457-477 doi:http://dx.doi.org/0.5705/ss.20.088 JACKKNIFE EMPIRICAL LIKELIHOOD BASED CONFIDENCE INTERVALS FOR PARTIAL AREAS UNDER ROC CURVES Gianfranco Adimari and Monica Chiogna
A Deep Interpretation of Classifier Chains
A Deep Interpretation of Classifier Chains Jesse Read and Jaakko Holmén http://users.ics.aalto.fi/{jesse,jhollmen}/ Aalto University School of Science, Department of Information and Computer Science and
Accouncements. You should turn in a PDF and a python file(s) Figure for problem 9 should be in the PDF
Accouncements You should turn in a PDF and a python file(s) Figure for problem 9 should be in the PDF Please do not zip these files and submit (unless there are >5 files) 1 Bayesian Methods Machine Learning