CART: Classification and Regression Trees

Transcription

1 CART: Claifiation and Regreion Tree Günther Sawitzki Otober, 04 Literature [] L. Breiman, J. H. Friedman, R. A. Olhen, and C. J. Stone. Claifiation and Regreion Tree. Wadworth, Belmont, CA., 984. [] J. H. Friedman. Loal learning baed on reurie oering. Tehnial report, Department of Statiti, Stanford Unierity, 996. [3] J. H. Friedman. On bia, ariane, 0/ lo, and the ure-of-dimenionality. Data Min. Knowl. Dio., ():55 77, 997. [4] T. Hatie, R. Tibhirani, and J. H. Friedman. The Element of Statitial Learning. Springer Serie in Statiti. Springer, nd edition, 009. [5] B.D. Ripley. Pattern reognition and neural network. Cambridge Unierity Pre, 996. Günther Sawitzki: CART Otober, 04 Günther Sawitzki: CART Otober, Coariable X with alue in X, for example X = R p. Repone Y with alue in Y, for example Y = R. and find model funtion m from loal model funtion m l : Elementary ae: m l ontant on X l. Example (regreion): m l = Y (x) x Xl Example (laifiation): m l = majority{y (x) x Xl } Günther Sawitzki: CART Otober, 04 3 Günther Sawitzki: CART Otober, Coariable X with alue in X, for example X = R p. Repone Y with alue in Y, for example Y = R. and find model funtion m from loal model funtion m l : Elementary ae: m l ontant on X l. Coariable X with alue in X, for example X = R p. Repone Y with alue in Y, for example Y = R. and find model funtion m from loal model funtion m l : Elementary ae: m l ontant on X l. Example (regreion): m l = Y (x) x Xl Example (laifiation): m l = majority{y (x) x Xl } Example (regreion): m l = Y (x) x Xl Example (laifiation): m l = majority{y (x) x Xl } Günther Sawitzki: CART Otober, 04 5 Coariable X with alue in X, for example X = R p. Repone Y with alue in Y, for example Y = R. and find model funtion m from loal model funtion m l : 7 Build partition by reurie plitting: X 0 = X Günther Sawitzki: CART Otober, 04 6 Coariable X with alue in X, for example X = R p. Repone Y with alue in Y, for example Y = R. and find model funtion m from loal model funtion m l : 8 Build partition by reurie plitting: X 0 = X for ome l for ome l Günther Sawitzki: CART Otober, 04 7 Günther Sawitzki: CART Otober, 04 8

2 9 0 Coariable X with alue in X, for example X = R p. Repone Y with alue in Y, for example Y = R. View on CART Reurie plitting ha two iew: iew Deiion tree and find model funtion m from loal model funtion m l : Build partition by reurie plitting: X 0 = X petal width irginia etoa eriolor irginia etoa Petal.Length <.45 Petal.Width <.75 Petal.Length < 4.95 eriolor irginia irginia Günther Sawitzki: CART Otober, 04 9 for ome l petal length We will ue the tree terminology node, edge here. Günther Sawitzki: CART Otober, 04 0 Elementary CART Only one ariable ued per plit. Continuou ariable: for ome l = {x X l : X k x 0 } {x X l : X k > x 0 } Fator: repet order, if appliable. Günther Sawitzki: CART Otober, 04 3 Günther Sawitzki: CART Otober, 04 4 Elementary CART a an optimiation problem For all urrent node X l and all ariable X k, find optimal plit point x 0. Selet optimal node and ariable. Need a quality meaure. For now, think of regreion a goodne of fit laifiation a milaifiation rate predition a predition error Elementary CART a an optimiation problem For all urrent node X l and all ariable X k, find optimal plit point x 0. Selet optimal node and ariable. Need a quality meaure. For now, think of regreion a goodne of fit laifiation a milaifiation rate predition a predition error Günther Sawitzki: CART Otober, 04 3 I 5 Günther Sawitzki: CART Otober, 04 4 II 6 Oberation X l with alue in X, for example X = R p. Clae Z l with alue,, K. Idea: reurie partition of X baed on a parameter X l, uh that Z l i homogeneou on the ell of the partition. Definition For a probability meaure P on,, K, let p k = P(k). A purity index i a funtion P Φ(P) uh that Φ ha exatly one maximum at P = ( K,, K ) Φ ha minima only at the meaure P = (, 0,, 0), P = (0,,, 0),..., P = (0, 0,, ) Φ i ymmetri, i.e. Φ(p,, p K ) = Φ(p π(),, p π(k) ) for any permutation π of,, K. Günther Sawitzki: CART Otober, 04 5 Günther Sawitzki: CART Otober, 04 6

3 III 7 8 Example (Shannon-Information) Φ(P) = K k= p klogp k Example (Gini Index of Dierity) Φ(P) = k k p kp k For X 0 X, let P X0 (k) = P(Z = k X X ). Let P X0 = #l:z l=k&x l X #l:x l X denote the orreponding empirial ditribution. The purity index for a partition i the weighted um P(X l )Φ(P Xl ). The purity index of a tree i the weighted um P(X l )Φ(P Xl ) l l:x l terminal node Günther Sawitzki: CART Otober, Günther Sawitzki: CART Otober, A imple regreion tree For all urrent node X l and all ariable X k, find optimal plit point x 0. Selet optimal node and ariable. Quality meaure: um of reidual quare. Note: thi i additie. So for eah node, we an ole the optimiation problem loally, and then elet the node giing the maximum redution in the um of reidual quare. ToDo Example A imple regreion tree For all urrent node X l and all ariable X k, find optimal plit point x 0. Selet optimal node and ariable. Quality meaure: um of reidual quare. Note: thi i additie. So for eah node, we an ole the optimiation problem loally, and then elet the node giing the maximum redution in the um of reidual quare. ToDo Example Günther Sawitzki: CART Otober, 04 9 Günther Sawitzki: CART Otober, 04 0 A imple regreion tree For all urrent node X l and eah ariable X k, find optimal plit point x 0. Selet optimal node and ariable. Quality meaure: e.g. um of reidual quare. Problem Degenerate to point et: need topping and pruning. A imple regreion tree For all urrent node X l and eah ariable X k, find optimal plit point x 0. Selet optimal node and ariable. Quality meaure: e.g. um of reidual quare. Problem Degenerate to point et: need topping and pruning. Data-baed um of reidual quare i biaed (oer-optimiti). Realiti quality meaure i needed. Data-baed um of reidual quare i biaed (oer-optimiti). Realiti quality meaure i needed. Günther Sawitzki: CART Otober, 04 A imple regreion tree 3 For all urrent node X l and eah ariable X k, find optimal plit point x 0. Selet optimal node and ariable. Quality meaure: e.g. um of reidual quare. Problem Degenerate to point et: need topping and pruning. Günther Sawitzki: CART Otober, 04 4 Reurie partitioning degenerate to point et: need topping and pruning. Early hope: Speify a topping trategy to aoid degenerate plit. Data-baed um of reidual quare i biaed (oer-optimiti). Realiti quality meaure i needed. Günther Sawitzki: CART Otober, 04 3 Günther Sawitzki: CART Otober, 04 4

4 5 6 Ad ho Do not plit a node if it ontain at mot n 0 oberation for ome hoen n 0 or Y i ontant or X i ontant n 0 = 5 or n 0 = 0, ay. Aoid degeneration, but till leae large ariane Still tree may need ome redution (pruning) in an additional tep. (To ome... ) Ad ho Do not plit a node if it ontain at mot n 0 oberation for ome hoen n 0 or Y i ontant or X i ontant n 0 = 5 or n 0 = 0, ay. Aoid degeneration, but till leae large ariane Still tree may need ome redution (pruning) in an additional tep. (To ome... ) Günther Sawitzki: CART Otober, Günther Sawitzki: CART Otober, Ad ho Do not plit a node if it ontain at mot n 0 oberation for ome hoen n 0 or Y i ontant or X i ontant n 0 = 5 or n 0 = 0, ay. Simple CART i a greedy algorithm, and only look at one tep. To improe algorithm, a multi tep iew i neeary. Complexity? There are aymptotially about.5 L ubtree of a balaned binary tree with L leae. P. Feigin, aording to [] p. 84. Aoid degeneration, but till leae large ariane Still tree may need ome redution (pruning) in an additional tep. (To ome... ) Günther Sawitzki: CART Otober, Günther Sawitzki: CART Otober, Simple CART i a greedy algorithm, and only look at one tep. To improe algorithm, a multi tep iew i neeary. Complexity? There are aymptotially about.5 L ubtree of a balaned binary tree with L leae. P. Feigin, aording to [] p. 84. Simple CART i a greedy algorithm, and only look at one tep. To improe algorithm, a multi tep iew i neeary. Complexity? There are aymptotially about.5 L ubtree of a balaned binary tree with L leae. P. Feigin, aording to [] p. 84. Günther Sawitzki: CART Otober, Günther Sawitzki: CART Otober, Exhautie earh oon not feaible. Bold idea: Build one large tree until it hit a topping barrier. Ue thi tree a an exploration of the data and ut down unneeary branhe. Heuriti to find andidate for good tree: Start with a large tree. Find a good ubtree with ame root t 0. Gien T 0, a rooted tree i a tree with ame root a T 0. Günther Sawitzki: CART Otober, 04 3 Günther Sawitzki: CART Otober, 04 3

5 33 34 An attempt to formaliation: For a tree T, define ome rik etimate R(T ), e.g. the milaifiation rate uing the data. Thi i too optimiti, due to a tendeny to oer-fitting. Add a omplexity penalty R α (T ) := R(T ) + α omplexity(t ) for ome free parameter α, where α = 0 mean: no penalty. omplexity(t ) = # of node of T, or omplexity(t ) = # of leae of T, or... For a binary tree: # of node of T = # of leae of T. For now: omplexity(t ) = # of leae of T = T. An attempt to formaliation: For a tree T, define ome rik etimate R(T ), e.g. the milaifiation rate uing the data. Thi i too optimiti, due to a tendeny to oer-fitting. Add a omplexity penalty R α (T ) := R(T ) + α omplexity(t ) for ome free parameter α, where α = 0 mean: no penalty. omplexity(t ) = # of node of T, or omplexity(t ) = # of leae of T, or... For a binary tree: # of node of T = # of leae of T. For now: omplexity(t ) = # of leae of T = T. Günther Sawitzki: CART Otober, Günther Sawitzki: CART Otober, An attempt to formaliation: For a tree T, define ome rik etimate R(T ), e.g. the milaifiation rate uing the data. Thi i too optimiti, due to a tendeny to oer-fitting. Add a omplexity penalty R α (T ) := R(T ) + α omplexity(t ) for ome free parameter α, where α = 0 mean: no penalty. omplexity(t ) = # of node of T, or omplexity(t ) = # of leae of T, or... For a binary tree: # of node of T = # of leae of T. For now: omplexity(t ) = # of leae of T = T. Start with a large tree T 0. For eah α, find ome rooted ubtree T T 0 minimiing R α (T ) := R(T ) + α T Günther Sawitzki: CART Otober, Günther Sawitzki: CART Otober, T T 0 i minimiing R α (T ) := R(T ) + α T T T 0 i minimiing R α (T ) := R(T ) + α T iff T = argmin T =L T 0 R α (T ) := R(T ) + α T iff T = argmin T =L T 0 R α (T ) := R(T ) + α T So we an build a equene T L = T 0, T L,..., T = t 0 of tree, minimiing R α for ome α 0 = 0, α L,..., α. R α i ontant on the interal and T l i minimier on the interal. There an be eeral tree with the ame alue of R α (T ), and minimiing tree need not be neted. So we an build a equene T L = T 0, T L,..., T = t 0 of tree, minimiing R α for ome α 0 = 0, α L,..., α. R α i ontant on the interal and T l i minimier on the interal. There an be eeral tree with the ame alue of R α (T ), and minimiing tree need not be neted. Günther Sawitzki: CART Otober, Günther Sawitzki: CART Otober, Remark It i poible to build a equene of neted minimiing tree from T 0 to {t 0 }. If there i a tree whih i a ubtree of all tree optimiing R α (T ), it will be denoted by T (α). Lemma If T 0 i not triial, T (α) exit. For a non-triial tree T and a non-terminal node t, let T t be the ubtree rooted at t. Definition T i pruned at t if T t i replaed by {t}. Let Remark g(t, T ) > α R α (t) > R α (T t ) g(t, T ) := R(t) R(T t) T t {t} Günther Sawitzki: CART Otober, Günther Sawitzki: CART Otober, 04 40

6 4 4 Günther Sawitzki: CART Otober, Günther Sawitzki: CART Otober, Günther Sawitzki: CART Otober, Tet et enario 45 Degenerate to point et: need topping and pruning. Split data et into training et and ealuation et. Tree ontrution ue only training et. Quality i judged by applying regreion tree to ealuation et. RSS t = n eal (Y i n Ŷi) eal Sum of term with idential ditribution. Standard deiation i= ( ) ( ( 4 ( ) )) ) (E Y Ŷ E Y Ŷ neal Günther Sawitzki: CART Otober, Tet et enario 46 RSS t = n eal (Y i n Ŷi) eal Sum of term with idential ditribution. Standard deiation Etimated by i= ( ) ( ( 4 ( ) )) ) (E Y Ŷ E Y Ŷ neal ( neal n eal n eal i= (Y i Ŷi) 4 RSSt ) Günther Sawitzki: CART Otober, Cro alidation enario 47 Diide the oberation V in group of nearly equal ize V,..., V k. For etimation, ue the omplete data et. Define the k-fold ro-alidation etimate of the quality index a the aerage R CV = k Rj t k where Rj t i the tet ample etimate, uing V \ V j a a training ample, and V j a an ealuation ample. j= Speial ae: k = n i the leae-one-out etimate. Günther Sawitzki: CART Otober, Cro alidation enario 48 Diide the oberation V in group of nearly equal ize V,..., V k. For etimation, ue the omplete data et. Define the k-fold ro-alidation etimate of the quality index a the aerage R CV = k Rj t k where Rj t i the tet ample etimate, uing V \ V j a a training ample, and V j a an ealuation ample. j= Speial ae: k = n i the leae-one-out etimate. Günther Sawitzki: CART Otober, Günther Sawitzki: CART Otober, 04 48

7 Cro alidation enario 49 Diide the oberation V in group of nearly equal ize V,..., V k. For etimation, ue the omplete data et. Define the k-fold ro-alidation etimate of the quality index a the aerage R CV = k Rj t k where Rj t i the tet ample etimate, uing V \ V j a a training ample, and V j a an ealuation ample. j= Speial ae: k = n i the leae-one-out etimate. Cro alidation enario 50 Diide the oberation V in group of nearly equal ize V,..., V k. For etimation, ue the omplete data et. Define the k-fold ro-alidation etimate of the quality index a the aerage R CV = k Rj t k where Rj t i the tet ample etimate, uing V \ V j a a training ample, and V j a an ealuation ample. j= Speial ae: k = n i the leae-one-out etimate. Günther Sawitzki: CART Otober, Günther Sawitzki: CART Otober, Günther Sawitzki: CART Otober, 04 5 Günther Sawitzki: CART Otober, Referene I Leo Breiman, Jerome H. Friedman, Rihard A. Olhen, and Charle J. Stone. Claifiation and Regreion Tree. Wadworth, Belmont, CA., 984. Jerome H. Friedman. Loal learning baed on reurie oering. Tehnial report, Department of Statiti, Stanford Unierity, 996. Jerome H. Friedman. On bia, ariane, 0/ lo, and the ure-of-dimenionality. Data Min. Knowl. Dio., ():55 77, 997. Günther Sawitzki: CART Otober, Günther Sawitzki: CART Otober, Referene II Index I Treor Hatie, Robert Tibhirani, and Jerome H. Friedman. The Element of Statitial Learning. Springer Serie in Statiti. Springer, nd edition, 009. Brian D. Ripley. Pattern reognition and neural network. Cambridge Unierity Pre, Cambridge, 996. bagging, 57 booting, 57 random foret, 57 Günther Sawitzki: CART Otober, Günther Sawitzki: CART Otober, 04 56

8 57 Günther Sawitzki: CART Otober, 04 57