Computing and using the deviance with classification trees
|
|
|
- Leslie West
- 5 years ago
- Views:
Transcription
1 Computing and using the deviance with classification trees Gilbert Ritschard Dept of Econometrics, University of Geneva Compstat, Rome, August 2006 Outline 1 Introduction 2 Motivation 3 Deviance for Trees 4 Outcome for the mobility tree example 5 Computational Issues 6 Women s labour participation example 7 Conclusion COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 1
2 1 Introduction About classification trees Descriptive non classificatory usages Measuring the quality of the tree (with the deviance) Computational issues COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 2
3 Principle of tree induction Goal: Find a partition of data such that the distribution of the outcome variable differs as much as possible from one leaf to the other. How: Proceeds by successively splitting nodes. Starting with root node, seek attribute that generates the best split according to a given criterion. Repeat operation at each new node until some stopping criterion, a minimal node size for instance, is met.. " 2 2 Main algorithms: CHAID (Kass, 1980), significance of Chi-Squares CART (Breiman et al., 1984), Gini index, binary trees C4.5 (Quinlan, 1993), gain ratio COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 3
4 2 Motivation In social sciences, induced trees are most often used for descriptive (non classificatory) aims. Examples: Mobility trees between social statuses of sons, fathers and grandfathers (data from act of marriage in the 19th century Geneva) (Ritschard and Oris, 2005) Goal: How do the statuses of the father and grandfather affect the chances of the groom to be in a lower, medium or high position? Determinants of women s labor participation (Swiss census data) (Losa et al., 2006) Goal: How do age, number of children, education, etc. affect the chances of the woman to work at full time, long part time, short part time or not to work at all? COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 4
5 Mobility tree Statuses defined from profession mentioned in marriage acts. Acts for all men having a name beginning with a B. For 572 cases, was possible to match with data from father s marriage social mobility over 3 generations Father s marriage Son s marriage Grand-father s status M 1 Father s M 2 Father s M 3 status status Son s status Groom s status (3 values) is response variable. Predictors are birthplace and statuses of father and grandfather. Method: CHAID (sig 5%, minimal child node size = 15, parent node = 30) COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 5
6 Mobility tree. Son s Status: Low (workers and craftmen), Clock Maker, High!! "! " " ". = JD A H I? I J= JI I = H 4 J % 0 EC D +?, A? A = I M % $ $ & $ $ $! ' " '! " "! & $! %! %! &! % * EH JD F =? A = JD A H I? I J= J * EH JD F =? A / A H = + 0 A EC D >. 8, / - / -? EJO., 1 JD. H+ 0 JD A H 0 EC D +? M, A? A = I / - / -? EJO. / A H = + 0 JD. H+ 0 A EC D >. 8, $ $ %!!! ' $ ' & " $ $ " & '!! ' " % '!! &! $ $ ' % &. = JD A H I? I J= JD EI = H * EH JD F =? A * EH JD F =? A 0 EC D +? M / - 8, 6 1., / -? EJO / A H = A EC D >. 1 / - / -? EJO 1 / A H = + 0 A EC D >. 8,., JD A H % $ $ & & $! "! & % " $ %! $ % ' ' $ $ $ $ & ' % 0 EC D, A? A = I = JD A H I? I J= J +? M 0 EC D +? M. = JD A H I? I J= JD EI = H " $ " & ' $ "! "! % " ' "! & % " ' " $! $ COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 6
7 Validating Tree in a Non-classificatory Setting Trees are usually validated with the classification error rate (on test data or through cross-validation) Claim : Classification error rate not suited for non classificatory purposes Example: Split into two groups with distribution 10% and 45% 90% 55% Distributions clearly different (valuable knowledge) Split does not improve the error rate (assuming majority rule). Our suggestion (Ritschard and Zighed, 2003): Use the deviance for measuring the descriptive power of a tree. COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 7
8 3 Deviance for Trees. " & " " & ' % & ' % & D(m 0 m) D(m) Root Node Induced Tree Saturated Tree Independence Leaf Table Target Table D(m 0 ) COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 8
9 Target and Predicted Tables Predicted Table ˆT % %! "! & " & ' 5 6 " & % % & Target Table T & " " & ' % & ˆT = T = COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 9
10 Deviance: Formal Definition T = (n ij ) r c target table: r rows = categories of the outcome variable c columns = different profiles in terms of the predictors ˆT = (ˆn ij ) r c table predicted from the tree Total of each column (profile) distributed according to the distribution in the leaf to which the profile belongs r D(m) = 2 i=1 c n ij ln j=1 ( ˆnij n ij ) Under regularity conditions (Bishop et al., 1975): D(m) χ 2 with d = (r 1)(c q) degrees of freedom (see Ritschard and Zighed, 2003) D(m 2 m 1 ) = D(m 2 ) D(m 1 ) χ 2 with d 2 d 1 degrees of freedom if m 2 restricted version of m 1 COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 10
11 Deviance based indicators BIC: deviance penalized for complexity (nbr of parameters) BIC= D(m) d ln(n)+constant pseudo R 2 McFadden R 2 = 1 D(m)/D(m 0 ), Nagelkerke R 2 = 1 exp{ ( 2 n D(m0 ) D(m) ) } 1 exp{ 2 n D(m 0)} Theil s u (proportion of reduction of Shannon s entropy) u = D(m 0 m) 2 i n i. ln(n i. /n) Evolves quadratically between independence and full association u represents position between the 2 extremes. COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 11
12 4 Outcome for the mobility tree example Error rate: 42.4%, (55.6% at the root node; 10 folds CV: 51.4% error) Goodness of fit Tree m D(m) df sig BIC AIC Theil u Indep Level Level Level Fitted Saturated COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 12
13 Between level deviance improvement D(row model) D(column model) Level 1 Level 2 Level 3 Fitted Saturated Indep (6 df) (14 df) (20 df) (24 df) (324 df) Level (8 df) (14 df) (18 df) (318 df) Level (6 df) (10 df) (310 df) Level (4 df) (304 df) Fitted (300 df) significant at 1%, at 5%, at 10% COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 13
14 Between level BIC variation BIC(row model) BIC(column model) Level 1 Level 2 Level 3 Fitted Saturated Indep Level Level Level Fitted From the BIC standpoint, Level 1 and Level 2 models look the most interesting. COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 14
15 5 Computational Issues 1. Softwares for growing trees do not provide the deviance nor easily usable information for computing the target and predicted tables Solution: look at LR statistics for cross tables. 2. Number of possible profiles (columns) may become excessively large. V May be as large as ν=1 with c ν the number of values of the v-th predictor c ν Solution: partial deviance (distance to a smaller arbitrary target table.) COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 15
16 Deviance and Likelihood Ratio Chi-squares D(m 0 m) = LR Chi-square statistic for testing independence on Leaf Table (crosstabulation of response variable with leaf variable). D(m 0 ) = LR Chi-square statistic for testing independence on Target Table (crosstabulation of response variable with profile variable). These statistics can easily be computed with most statistical package (SPSS, SAS,...) Deviance of Tree m is just their difference D(m) = D(m 0 ) D(m 0 m) Need just to retrieve for each case: - leaf number - profile number COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 16
17 Partial deviance D(m m T ) Arbitrary r c target table T defined from the c profiles in terms of the mere predictors and value groupings retained by the induced tree. Due to arbitrariness of T - Deviance D(m T ) is no longer distance to true target. - Pseudo R 2 s based on D(m T ) are irrelevant. Differences of deviances between nested trees are independent of the target. For example: D(m 0 m) = D(m 0 ) D(m) = D(m 0 m T ) D(m m T ) measures the gain over the root node (as the classical Chi-square used with logistic regression). BIC and u can still be used. COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 17
18 8 Fabio B. Losa*, Pau Origoni*, Gilbert Ritschard** 6 Women s labour participation example Tree for participation of divorced or single mothers, French speaking region. divorced, single Mothers age of last-born child 0-13 years age of last-born child 14 years and older other prof. groups profession of education, health,... professions of education, health,... other prof. groups low education medium, high education age years age years 1 child 2 or more children low, medium education high education other households couple with children age years age years FIG. 4. Tree for participation of divorced or single mothers, French speaking region COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 18
19 Quality of the trees q c p n D(m 0 m) d sig. CHI CHF CHG BIC(m 0, m) BIC(m T, m) u Theil u CHI CHF CHG COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 19
20 7 Conclusion Summary: Deviance may be used with trees. Deviance and differences in deviances useful for evaluating the descriptive power of trees. Deviance based measures, such as BIC and Theil s u, also useful. Computation issues: solutions exist. Further issues for descriptive trees: Using BIC as tree growing criterion. Evaluating the stability of induced trees (Dannegger, 2000). COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 20
21 THANK YOU COMPSTAT06 toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 21
22 References Bishop, Y. M. M., S. E. Fienberg, and P. W. Holland (1975). Discrete Multivariate Analysis. Cambridge MA: MIT Press. Breiman, L., J. H. Friedman, R. A. Olshen, and C. J. Stone (1984). Classification And Regression Trees. New York: Chapman and Hall. Dannegger, F. (2000). Tree stability diagnostics and some remedies for instability. Statistics In Medicine 19(4), Kass, G. V. (1980). An exploratory technique for investigating large quantities of categorical data. Applied Statistics 29(2), Losa, F. B., P. Origoni, and G. Ritschard (2006). Experiences from a socio-economic application of induction trees. In N. Lavrač, L. Todorovski, and K. P. Jantke (Eds.), Discovery Science, 9th International Conference, DS 2006, Barcelona, October 7-10, 2006, Proceedings, Volume LNAI 4265, pp Berlin Heidelberg: Springer. Quinlan, J. R. (1993). C4.5: Programs for Machine Learning. San Mateo: Morgan Kaufmann. Ritschard, G. and M. Oris (2005). Life course data in demography and social sciences: Statistical and data mining approaches. In P. Ghisletta, J.-M. Le Goff, R. Levy, D. Spini, and E. Widmer (Eds.), Towards an Interdisciplinary Perspective on the Life Course, Advancements in Life Course Research, Vol. 10, pp Amsterdam: Elsevier. References toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 22
23 Ritschard, G. and D. A. Zighed (2003). Goodness-of-fit measures for induction trees. In N. Zhong, Z. Ras, S. Tsumo, and E. Suzuki (Eds.), Foundations of Intelligent Systems, ISMIS03, Volume LNAI 2871, pp Berlin: Springer. References toc Intro Motiv MobTr Dev Ex1 Comp Ex2 Conc 8/9/2006gr 23
Growing a Large Tree
STAT 5703 Fall, 2004 Data Mining Methodology I Decision Tree I Growing a Large Tree Contents 1 A Single Split 2 1.1 Node Impurity.................................. 2 1.2 Computation of i(t)................................
day month year documentname/initials 1
ECE471-571 Pattern Recognition Lecture 13 Decision Tree Hairong Qi, Gonzalez Family Professor Electrical Engineering and Computer Science University of Tennessee, Knoxville http://www.eecs.utk.edu/faculty/qi
Decision Trees. CS57300 Data Mining Fall Instructor: Bruno Ribeiro
Decision Trees CS57300 Data Mining Fall 2016 Instructor: Bruno Ribeiro Goal } Classification without Models Well, partially without a model } Today: Decision Trees 2015 Bruno Ribeiro 2 3 Why Trees? } interpretable/intuitive,
Data Mining. CS57300 Purdue University. Bruno Ribeiro. February 8, 2018
Data Mining CS57300 Purdue University Bruno Ribeiro February 8, 2018 Decision trees Why Trees? interpretable/intuitive, popular in medical applications because they mimic the way a doctor thinks model
Classification Using Decision Trees
Classification Using Decision Trees 1. Introduction Data mining term is mainly used for the specific set of six activities namely Classification, Estimation, Prediction, Affinity grouping or Association
ONE MORE TIME ABOUT R 2 MEASURES OF FIT IN LOGISTIC REGRESSION
ONE MORE TIME ABOUT R 2 MEASURES OF FIT IN LOGISTIC REGRESSION Ernest S. Shtatland, Ken Kleinman, Emily M. Cain Harvard Medical School, Harvard Pilgrim Health Care, Boston, MA ABSTRACT In logistic regression,
Induction of Decision Trees
Induction of Decision Trees Peter Waiganjo Wagacha This notes are for ICS320 Foundations of Learning and Adaptive Systems Institute of Computer Science University of Nairobi PO Box 30197, 00200 Nairobi.
Statistical Consulting Topics Classification and Regression Trees (CART)
Statistical Consulting Topics Classification and Regression Trees (CART) Suppose the main goal in a data analysis is the prediction of a categorical variable outcome. Such as in the examples below. Given
Rule Generation using Decision Trees
Rule Generation using Decision Trees Dr. Rajni Jain 1. Introduction A DT is a classification scheme which generates a tree and a set of rules, representing the model of different classes, from a given
Data Mining. 3.6 Regression Analysis. Fall Instructor: Dr. Masoud Yaghini. Numeric Prediction
Data Mining 3.6 Regression Analysis Fall 2008 Instructor: Dr. Masoud Yaghini Outline Introduction Straight-Line Linear Regression Multiple Linear Regression Other Regression Models References Introduction
Generalization to Multi-Class and Continuous Responses. STA Data Mining I
Generalization to Multi-Class and Continuous Responses STA 5703 - Data Mining I 1. Categorical Responses (a) Splitting Criterion Outline Goodness-of-split Criterion Chi-square Tests and Twoing Rule (b)
Data Mining Classification: Basic Concepts and Techniques. Lecture Notes for Chapter 3. Introduction to Data Mining, 2nd Edition
Data Mining Classification: Basic Concepts and Techniques Lecture Notes for Chapter 3 by Tan, Steinbach, Karpatne, Kumar 1 Classification: Definition Given a collection of records (training set ) Each
Class Notes: Week 8. Probit versus Logit Link Functions and Count Data
Ronald Heck Class Notes: Week 8 1 Class Notes: Week 8 Probit versus Logit Link Functions and Count Data This week we ll take up a couple of issues. The first is working with a probit link function. While
Logistic Regression. Continued Psy 524 Ainsworth
Logistic Regression Continued Psy 524 Ainsworth Equations Regression Equation Y e = 1 + A+ B X + B X + B X 1 1 2 2 3 3 i A+ B X + B X + B X e 1 1 2 2 3 3 Equations The linear part of the logistic regression
REGRESSION TREE CREDIBILITY MODEL
LIQUN DIAO AND CHENGGUO WENG Department of Statistics and Actuarial Science, University of Waterloo Advances in Predictive Analytics Conference, Waterloo, Ontario Dec 1, 2017 Overview Statistical }{{ Method
ARBEITSPAPIERE ZUR MATHEMATISCHEN WIRTSCHAFTSFORSCHUNG
ARBEITSPAPIERE ZUR MATHEMATISCHEN WIRTSCHAFTSFORSCHUNG Some Remarks about the Usage of Asymmetric Correlation Measurements for the Induction of Decision Trees Andreas Hilbert Heft 180/2002 Institut für
BINARY TREE-STRUCTURED PARTITION AND CLASSIFICATION SCHEMES
BINARY TREE-STRUCTURED PARTITION AND CLASSIFICATION SCHEMES DAVID MCDIARMID Abstract Binary tree-structured partition and classification schemes are a class of nonparametric tree-based approaches to classification
Influence measures for CART
Jean-Michel Poggi Orsay, Paris Sud & Paris Descartes Joint work with Avner Bar-Hen Servane Gey (MAP5, Paris Descartes ) CART CART Classification And Regression Trees, Breiman et al. (1984) Learning set
Dyadic Classification Trees via Structural Risk Minimization
Dyadic Classification Trees via Structural Risk Minimization Clayton Scott and Robert Nowak Department of Electrical and Computer Engineering Rice University Houston, TX 77005 cscott,nowak @rice.edu Abstract
Comparison of Shannon, Renyi and Tsallis Entropy used in Decision Trees
Comparison of Shannon, Renyi and Tsallis Entropy used in Decision Trees Tomasz Maszczyk and W lodzis law Duch Department of Informatics, Nicolaus Copernicus University Grudzi adzka 5, 87-100 Toruń, Poland
Log-linear multidimensional Rasch model for capture-recapture
Log-linear multidimensional Rasch model for capture-recapture Elvira Pelle, University of Milano-Bicocca, e.pelle@campus.unimib.it David J. Hessen, Utrecht University, D.J.Hessen@uu.nl Peter G.M. Van der
Decision Trees (Cont.)
Decision Trees (Cont.) R&N Chapter 18.2,18.3 Side example with discrete (categorical) attributes: Predicting age (3 values: less than 30, 30-45, more than 45 yrs old) from census data. Attributes (split
CS145: INTRODUCTION TO DATA MINING
CS145: INTRODUCTION TO DATA MINING 4: Vector Data: Decision Tree Instructor: Yizhou Sun yzsun@cs.ucla.edu October 10, 2017 Methods to Learn Vector Data Set Data Sequence Data Text Data Classification Clustering
Introduction to Machine Learning CMU-10701
Introduction to Machine Learning CMU-10701 23. Decision Trees Barnabás Póczos Contents Decision Trees: Definition + Motivation Algorithm for Learning Decision Trees Entropy, Mutual Information, Information
arxiv: v1 [cs.ds] 28 Sep 2018
![arxiv: v1 [cs.ds] 28 Sep 2018](/thumbs/87/96221057.jpg "arxiv: v1 [cs.ds] 28 Sep 2018")
Minimization of Gini impurity via connections with the k-means problem arxiv:1810.00029v1 [cs.ds] 28 Sep 2018 Eduardo Laber PUC-Rio, Brazil laber@inf.puc-rio.br October 2, 2018 Abstract Lucas Murtinho
An asymmetric entropy measure for decision trees
An asymmetric entropy measure for decision trees Simon Marcellin Laboratoire ERIC Université Lumière Lyon 2 5 av. Pierre Mendès-France 69676 BRON Cedex France simon.marcellin@univ-lyon2.fr Djamel A. Zighed
MULTINOMIAL LOGISTIC REGRESSION
MULTINOMIAL LOGISTIC REGRESSION Model graphically: Variable Y is a dependent variable, variables X, Z, W are called regressors. Multinomial logistic regression is a generalization of the binary logistic
Kernel Density Estimation
Kernel Density Estimation If Y {1,..., K} and g k denotes the density for the conditional distribution of X given Y = k the Bayes classifier is f (x) = argmax π k g k (x) k If ĝ k for k = 1,..., K are
A Quality Index for Decision Tree Pruning
A Quality Index for Decision Tree Pruning D. Fournier and B. Crémilleux GREYC, CNRS - UMR 6072 Université de Caen F-14032 Caen Cédex France Tel : +33 (0)2.31.56.73.79 Fax : +33 (0)2.31.56.73.30 fournier,cremilleux
CS 6375 Machine Learning
CS 6375 Machine Learning Decision Trees Instructor: Yang Liu 1 Supervised Classifier X 1 X 2. X M Ref class label 2 1 Three variables: Attribute 1: Hair = {blond, dark} Attribute 2: Height = {tall, short}
Analysis of data in square contingency tables
Analysis of data in square contingency tables Iva Pecáková Let s suppose two dependent samples: the response of the nth subject in the second sample relates to the response of the nth subject in the first
Outline. Clustering. Capturing Unobserved Heterogeneity in the Austrian Labor Market Using Finite Mixtures of Markov Chain Models
Capturing Unobserved Heterogeneity in the Austrian Labor Market Using Finite Mixtures of Markov Chain Models Collaboration with Rudolf Winter-Ebmer, Department of Economics, Johannes Kepler University
Machine Learning & Data Mining
Group M L D Machine Learning M & Data Mining Chapter 7 Decision Trees Xin-Shun Xu @ SDU School of Computer Science and Technology, Shandong University Top 10 Algorithm in DM #1: C4.5 #2: K-Means #3: SVM
A note on R 2 measures for Poisson and logistic regression models when both models are applicable
Journal of Clinical Epidemiology 54 (001) 99 103 A note on R measures for oisson and logistic regression models when both models are applicable Martina Mittlböck, Harald Heinzl* Department of Medical Computer
LISA Short Course Series Generalized Linear Models (GLMs) & Categorical Data Analysis (CDA) in R. Liang (Sally) Shan Nov. 4, 2014
LISA Short Course Series Generalized Linear Models (GLMs) & Categorical Data Analysis (CDA) in R Liang (Sally) Shan Nov. 4, 2014 L Laboratory for Interdisciplinary Statistical Analysis LISA helps VT researchers
Addressing Corner Solution Effect for Child Mortality Status Measure: An Application of Tobit Model
Addressing Corner Solution Effect for Child Mortality Status Measure: An Application of Tobit Model Hafiz M. Muddasar Jamil Shera & Irum Sajjad Dar College of Statistics and Actuarial Sciences, University
Longitudinal Data Analysis of Health Outcomes
Longitudinal Data Analysis of Health Outcomes Longitudinal Data Analysis Workshop Running Example: Days 2 and 3 University of Georgia: Institute for Interdisciplinary Research in Education and Human Development
CS6220: DATA MINING TECHNIQUES
CS6220: DATA MINING TECHNIQUES Matrix Data: Prediction Instructor: Yizhou Sun yzsun@ccs.neu.edu September 14, 2014 Today s Schedule Course Project Introduction Linear Regression Model Decision Tree 2 Methods
Generalized Linear Models
Generalized Linear Models Lecture 3. Hypothesis testing. Goodness of Fit. Model diagnostics GLM (Spring, 2018) Lecture 3 1 / 34 Models Let M(X r ) be a model with design matrix X r (with r columns) r n
Topic 21 Goodness of Fit
Topic 21 Goodness of Fit Contingency Tables 1 / 11 Introduction Two-way Table Smoking Habits The Hypothesis The Test Statistic Degrees of Freedom Outline 2 / 11 Introduction Contingency tables, also known
Machine Learning and Data Mining. Decision Trees. Prof. Alexander Ihler
+ Machine Learning and Data Mining Decision Trees Prof. Alexander Ihler Decision trees Func-onal form f(x;µ): nested if-then-else statements Discrete features: fully expressive (any func-on) Structure:
One-Way ANOVA. Some examples of when ANOVA would be appropriate include:
One-Way ANOVA 1. Purpose Analysis of variance (ANOVA) is used when one wishes to determine whether two or more groups (e.g., classes A, B, and C) differ on some outcome of interest (e.g., an achievement
Model Based Statistics in Biology. Part V. The Generalized Linear Model. Chapter 16 Introduction
Model Based Statistics in Biology. Part V. The Generalized Linear Model. Chapter 16 Introduction ReCap. Parts I IV. The General Linear Model Part V. The Generalized Linear Model 16 Introduction 16.1 Analysis
A Metric Approach to Building Decision Trees based on Goodman-Kruskal Association Index
A Metric Approach to Building Decision Trees based on Goodman-Kruskal Association Index Dan A. Simovici and Szymon Jaroszewicz University of Massachusetts at Boston, Department of Computer Science, Boston,
Chapter 14 Combining Models
Chapter 14 Combining Models T-61.62 Special Course II: Pattern Recognition and Machine Learning Spring 27 Laboratory of Computer and Information Science TKK April 3th 27 Outline Independent Mixing Coefficients
CS6220: DATA MINING TECHNIQUES
CS6220: DATA MINING TECHNIQUES Matrix Data: Prediction Instructor: Yizhou Sun yzsun@ccs.neu.edu September 21, 2015 Announcements TA Monisha s office hour has changed to Thursdays 10-12pm, 462WVH (the same
Decision Tree Learning
Decision Tree Learning Goals for the lecture you should understand the following concepts the decision tree representation the standard top-down approach to learning a tree Occam s razor entropy and information
Binary Logistic Regression
The coefficients of the multiple regression model are estimated using sample data with k independent variables Estimated (or predicted) value of Y Estimated intercept Estimated slope coefficients Ŷ = b
Adaptive Crowdsourcing via EM with Prior
Adaptive Crowdsourcing via EM with Prior Peter Maginnis and Tanmay Gupta May, 205 In this work, we make two primary contributions: derivation of the EM update for the shifted and rescaled beta prior and
Lab 3: Two levels Poisson models (taken from Multilevel and Longitudinal Modeling Using Stata, p )
Lab 3: Two levels Poisson models (taken from Multilevel and Longitudinal Modeling Using Stata, p. 376-390) BIO656 2009 Goal: To see if a major health-care reform which took place in 1997 in Germany was
Informal Definition: Telling things apart
9. Decision Trees Informal Definition: Telling things apart 2 Nominal data No numeric feature vector Just a list or properties: Banana: longish, yellow Apple: round, medium sized, different colors like
Stat/F&W Ecol/Hort 572 Review Points Ané, Spring 2010
1 Linear models Y = Xβ + ɛ with ɛ N (0, σ 2 e) or Y N (Xβ, σ 2 e) where the model matrix X contains the information on predictors and β includes all coefficients (intercept, slope(s) etc.). 1. Number of
UNIVERSITY OF TORONTO. Faculty of Arts and Science APRIL 2010 EXAMINATIONS STA 303 H1S / STA 1002 HS. Duration - 3 hours. Aids Allowed: Calculator
UNIVERSITY OF TORONTO Faculty of Arts and Science APRIL 2010 EXAMINATIONS STA 303 H1S / STA 1002 HS Duration - 3 hours Aids Allowed: Calculator LAST NAME: FIRST NAME: STUDENT NUMBER: There are 27 pages
SOS3003 Applied data analysis for social science Lecture note Erling Berge Department of sociology and political science NTNU.
SOS3003 Applied data analysis for social science Lecture note 08-00 Erling Berge Department of sociology and political science NTNU Erling Berge 00 Literature Logistic regression II Hamilton Ch 7 p7-4
Simultaneous row and column partitioning: The scope of a heuristic approach
Simultaneous row and column partitioning: The scope of a heuristic approach Gilbert Ritschard 1 and Djamel A. Zighed 2 1 Dept of Econometrics, University of Geneva, CH-1211 Geneva 4, Switzerland ritschard@themes.unige.ch
Logistic Regression. Interpretation of linear regression. Other types of outcomes. 0-1 response variable: Wound infection. Usual linear regression
Logistic Regression Usual linear regression (repetition) y i = b 0 + b 1 x 1i + b 2 x 2i + e i, e i N(0,σ 2 ) or: y i N(b 0 + b 1 x 1i + b 2 x 2i,σ 2 ) Example (DGA, p. 336): E(PEmax) = 47.355 + 1.024
Classification: Decision Trees
Classification: Decision Trees Outline Top-Down Decision Tree Construction Choosing the Splitting Attribute Information Gain and Gain Ratio 2 DECISION TREE An internal node is a test on an attribute. A
Variable Selection and Sensitivity Analysis via Dynamic Trees with an application to Computer Code Performance Tuning
Variable Selection and Sensitivity Analysis via Dynamic Trees with an application to Computer Code Performance Tuning Robert B. Gramacy University of Chicago Booth School of Business faculty.chicagobooth.edu/robert.gramacy
Fuzzy Cognitive Maps Learning through Swarm Intelligence
Fuzzy Cognitive Maps Learning through Swarm Intelligence E.I. Papageorgiou,3, K.E. Parsopoulos 2,3, P.P. Groumpos,3, and M.N. Vrahatis 2,3 Department of Electrical and Computer Engineering, University
EPSY 905: Fundamentals of Multivariate Modeling Online Lecture #7
Introduction to Generalized Univariate Models: Models for Binary Outcomes EPSY 905: Fundamentals of Multivariate Modeling Online Lecture #7 EPSY 905: Intro to Generalized In This Lecture A short review
Model Estimation Example
Ronald H. Heck 1 EDEP 606: Multivariate Methods (S2013) April 7, 2013 Model Estimation Example As we have moved through the course this semester, we have encountered the concept of model estimation. Discussions
Statistics and learning: Big Data
Statistics and learning: Big Data Learning Decision Trees and an Introduction to Boosting Sébastien Gadat Toulouse School of Economics February 2017 S. Gadat (TSE) SAD 2013 1 / 30 Keywords Decision trees
EDF 7405 Advanced Quantitative Methods in Educational Research. Data are available on IQ of the child and seven potential predictors.
EDF 7405 Advanced Quantitative Methods in Educational Research Data are available on IQ of the child and seven potential predictors. Four are medical variables available at the birth of the child: Birthweight
Holdout and Cross-Validation Methods Overfitting Avoidance
Holdout and Cross-Validation Methods Overfitting Avoidance Decision Trees Reduce error pruning Cost-complexity pruning Neural Networks Early stopping Adjusting Regularizers via Cross-Validation Nearest
CLUe Training An Introduction to Machine Learning in R with an example from handwritten digit recognition
CLUe Training An Introduction to Machine Learning in R with an example from handwritten digit recognition Ad Feelders Universiteit Utrecht Department of Information and Computing Sciences Algorithmic Data
Related Concepts: Lecture 9 SEM, Statistical Modeling, AI, and Data Mining. I. Terminology of SEM
Lecture 9 SEM, Statistical Modeling, AI, and Data Mining I. Terminology of SEM Related Concepts: Causal Modeling Path Analysis Structural Equation Modeling Latent variables (Factors measurable, but thru
Quantitative Analysis and Empirical Methods
Hypothesis testing Sciences Po, Paris, CEE / LIEPP Introduction Hypotheses Procedure of hypothesis testing Two-tailed and one-tailed tests Statistical tests with categorical variables A hypothesis A testable
High-dimensional regression modeling
High-dimensional regression modeling David Causeur Department of Statistics and Computer Science Agrocampus Ouest IRMAR CNRS UMR 6625 http://www.agrocampus-ouest.fr/math/causeur/ Course objectives Making
Machine Learning for OR & FE
Machine Learning for OR & FE Regression II: Regularization and Shrinkage Methods Martin Haugh Department of Industrial Engineering and Operations Research Columbia University Email: martin.b.haugh@gmail.com
Decision Trees. Common applications: Health diagnosis systems Bank credit analysis
Decision Trees Rodrigo Fernandes de Mello Invited Professor at Télécom ParisTech Associate Professor at Universidade de São Paulo, ICMC, Brazil http://www.icmc.usp.br/~mello mello@icmc.usp.br Decision
Time-Invariant Predictors in Longitudinal Models
Time-Invariant Predictors in Longitudinal Models Today s Class (or 3): Summary of steps in building unconditional models for time What happens to missing predictors Effects of time-invariant predictors
Bias Correction in Classification Tree Construction ICML 2001
Bias Correction in Classification Tree Construction ICML 21 Alin Dobra Johannes Gehrke Department of Computer Science Cornell University December 15, 21 Classification Tree Construction Outlook Temp. Humidity
Generalized Linear Models (GLZ)
Generalized Linear Models (GLZ) Generalized Linear Models (GLZ) are an extension of the linear modeling process that allows models to be fit to data that follow probability distributions other than the
Decision Trees Part 1. Rao Vemuri University of California, Davis
Decision Trees Part 1 Rao Vemuri University of California, Davis Overview What is a Decision Tree Sample Decision Trees How to Construct a Decision Tree Problems with Decision Trees Classification Vs Regression
Log-linear Models for Contingency Tables
Log-linear Models for Contingency Tables Statistics 149 Spring 2006 Copyright 2006 by Mark E. Irwin Log-linear Models for Two-way Contingency Tables Example: Business Administration Majors and Gender A
Standard Errors & Confidence Intervals. N(0, I( β) 1 ), I( β) = [ 2 l(β, φ; y) β i β β= β j
Standard Errors & Confidence Intervals β β asy N(0, I( β) 1 ), where I( β) = [ 2 l(β, φ; y) ] β i β β= β j We can obtain asymptotic 100(1 α)% confidence intervals for β j using: β j ± Z 1 α/2 se( β j )
From statistics to data science. BAE 815 (Fall 2017) Dr. Zifei Liu
From statistics to data science BAE 815 (Fall 2017) Dr. Zifei Liu Zifeiliu@ksu.edu Why? How? What? How much? How many? Individual facts (quantities, characters, or symbols) The Data-Information-Knowledge-Wisdom
Factor Analysis & Structural Equation Models. CS185 Human Computer Interaction
Factor Analysis & Structural Equation Models CS185 Human Computer Interaction MoodPlay Recommender (Andjelkovic et al, UMAP 2016) Online system available here: http://ugallery.pythonanywhere.com/ 2 3 Structural
Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation
Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation Lecture Notes for Chapter 4 Part I Introduction to Data Mining by Tan, Steinbach, Kumar Adapted by Qiang Yang (2010) Tan,Steinbach,
Lecture 7: DecisionTrees
Lecture 7: DecisionTrees What are decision trees? Brief interlude on information theory Decision tree construction Overfitting avoidance Regression trees COMP-652, Lecture 7 - September 28, 2009 1 Recall:
Decision Tree. Decision Tree Learning. c4.5. Example
Decision ree Decision ree Learning s of systems that learn decision trees: c4., CLS, IDR, ASSISA, ID, CAR, ID. Suitable problems: Instances are described by attribute-value couples he target function has
Aijun An and Nick Cercone. Department of Computer Science, University of Waterloo. methods in a context of learning classication rules.
Discretization of Continuous Attributes for Learning Classication Rules Aijun An and Nick Cercone Department of Computer Science, University of Waterloo Waterloo, Ontario N2L 3G1 Canada Abstract. We present
ESP 178 Applied Research Methods. 2/23: Quantitative Analysis
ESP 178 Applied Research Methods 2/23: Quantitative Analysis Data Preparation Data coding create codebook that defines each variable, its response scale, how it was coded Data entry for mail surveys and
Machine Learning 3. week
Machine Learning 3. week Entropy Decision Trees ID3 C4.5 Classification and Regression Trees (CART) 1 What is Decision Tree As a short description, decision tree is a data classification procedure which
Development of a Data Mining Methodology using Robust Design
Development of a Data Mining Methodology using Robust Design Sangmun Shin, Myeonggil Choi, Youngsun Choi, Guo Yi Department of System Management Engineering, Inje University Gimhae, Kyung-Nam 61-749 South
Text Mining. Dr. Yanjun Li. Associate Professor. Department of Computer and Information Sciences Fordham University
Text Mining Dr. Yanjun Li Associate Professor Department of Computer and Information Sciences Fordham University Outline Introduction: Data Mining Part One: Text Mining Part Two: Preprocessing Text Data
Directed acyclic graphs and the use of linear mixed models
Directed acyclic graphs and the use of linear mixed models Siem H. Heisterkamp 1,2 1 Groningen Bioinformatics Centre, University of Groningen 2 Biostatistics and Research Decision Sciences (BARDS), MSD,
Knowledge Discovery and Data Mining
Knowledge Discovery and Data Mining Lecture 06 - Regression & Decision Trees Tom Kelsey School of Computer Science University of St Andrews http://tom.home.cs.st-andrews.ac.uk twk@st-andrews.ac.uk Tom
INFORMATION AS A UNIFYING MEASURE OF FIT IN SAS STATISTICAL MODELING PROCEDURES
INFORMATION AS A UNIFYING MEASURE OF FIT IN SAS STATISTICAL MODELING PROCEDURES Ernest S. Shtatland, PhD Mary B. Barton, MD, MPP Harvard Medical School, Harvard Pilgrim Health Care, Boston, MA ABSTRACT
Faculty of Health Sciences. Regression models. Counts, Poisson regression, Lene Theil Skovgaard. Dept. of Biostatistics
Faculty of Health Sciences Regression models Counts, Poisson regression, 27-5-2013 Lene Theil Skovgaard Dept. of Biostatistics 1 / 36 Count outcome PKA & LTS, Sect. 7.2 Poisson regression The Binomial
Data analysis strategies for high dimensional social science data M3 Conference May 2013
Data analysis strategies for high dimensional social science data M3 Conference May 2013 W. Holmes Finch, Maria Hernández Finch, David E. McIntosh, & Lauren E. Moss Ball State University High dimensional
Classification using stochastic ensembles
July 31, 2014 Topics Introduction Topics Classification Application and classfication Classification and Regression Trees Stochastic ensemble methods Our application: USAID Poverty Assessment Tools Topics
ML techniques. symbolic techniques different types of representation value attribute representation representation of the first order
MACHINE LEARNING Definition 1: Learning is constructing or modifying representations of what is being experienced [Michalski 1986], p. 10 Definition 2: Learning denotes changes in the system That are adaptive
Regression tree methods for subgroup identification I
Regression tree methods for subgroup identification I Xu He Academy of Mathematics and Systems Science, Chinese Academy of Sciences March 25, 2014 Xu He (AMSS, CAS) March 25, 2014 1 / 34 Outline The problem
JEROME H. FRIEDMAN Department of Statistics and Stanford Linear Accelerator Center, Stanford University, Stanford, CA
1 SEPARATING SIGNAL FROM BACKGROUND USING ENSEMBLES OF RULES JEROME H. FRIEDMAN Department of Statistics and Stanford Linear Accelerator Center, Stanford University, Stanford, CA 94305 E-mail: jhf@stanford.edu
ABHELSINKI UNIVERSITY OF TECHNOLOGY
Cross-Validation, Information Criteria, Expected Utilities and the Effective Number of Parameters Aki Vehtari and Jouko Lampinen Laboratory of Computational Engineering Introduction Expected utility -
Research Methodology: Tools
MSc Business Administration Research Methodology: Tools Applied Data Analysis (with SPSS) Lecture 05: Contingency Analysis March 2014 Prof. Dr. Jürg Schwarz Lic. phil. Heidi Bruderer Enzler Contents Slide
Section on Survey Research Methods JSM 2010 STATISTICAL GRAPHICS OF PEARSON RESIDUALS IN SURVEY LOGISTIC REGRESSION DIAGNOSIS
STATISTICAL GRAPHICS OF PEARSON RESIDUALS IN SURVEY LOGISTIC REGRESSION DIAGNOSIS Stanley Weng, National Agricultural Statistics Service, U.S. Department of Agriculture 3251 Old Lee Hwy, Fairfax, VA 22030,
1. BINARY LOGISTIC REGRESSION
1. BINARY LOGISTIC REGRESSION The Model We are modelling two-valued variable Y. Model s scheme Variable Y is the dependent variable, X, Z, W are independent variables (regressors). Typically Y values are
Supervised Learning! Algorithm Implementations! Inferring Rudimentary Rules and Decision Trees!
Supervised Learning! Algorithm Implementations! Inferring Rudimentary Rules and Decision Trees! Summary! Input Knowledge representation! Preparing data for learning! Input: Concept, Instances, Attributes"