Artfcal Itellgece Learg of decso trees Peter Atal atal@mt.bme.hu A.I. November 21, 2016 1
Problem: decde whether to wat for a table at a restaurat, based o the followg attrbutes: 1. Alterate: s there a alteratve restaurat earby? 2. Bar: s there a comfortable bar area to wat? 3. Fr/Sat: s today Frday or Saturday? 4. Hugry: are we hugry? 5. Patros: umber of eole the restaurat (Noe, Some, Full) 6. Prce: rce rage ($, $$, $$$) 7. Rag: s t rag outsde? 8. Reservato: have we made a reservato? 9. Tye: kd of restaurat (Frech, Itala, Tha, Burger) 10. WatEstmate: estmated watg tme (0-10, 10-30, 30-60, >60)
Examles descrbed by attrbute values (Boolea, dscrete, cotuous) E.g., stuatos where I wll/wo't wat for a table: Classfcato of examles s ostve (T) or egatve (F)
Oe ossble reresetato for hyotheses E.g., here s the true tree for decdg whether to wat:
Decso trees ca exress ay fucto of the ut attrbutes. E.g., for Boolea fuctos, truth table row ath to leaf: Trvally, there s a cosstet decso tree for ay trag set wth oe ath to leaf for each examle (uless f odetermstc x) but t robably wo't geeralze to ew examles Prefer to fd more comact decso trees
How may dstct decso trees wth Boolea attrbutes? = umber of Boolea fuctos = umber of dstct truth tables wth 2 rows = 2 2 E.g., wth 6 Boolea attrbutes, there are 18,446,744,073,709,551,616 trees
How may dstct decso trees wth Boolea attrbutes? = umber of Boolea fuctos = umber of dstct truth tables wth 2 rows = 2 2 E.g., wth 6 Boolea attrbutes, there are 18,446,744,073,709,551,616 trees How may urely cojuctve hyotheses (e.g., Hugry Ra)? Each attrbute ca be (ostve), (egatve), or out 3 dstct cojuctve hyotheses More exressve hyothess sace creases chace that target fucto ca be exressed creases umber of hyotheses cosstet wth trag set may get worse redctos
Total error I ractce, the target tycally s ot sde the hyothess sace: the total real error ca be decomosed to bas + varace bas : exected error/modellg error varace : estmato/emrcal selecto error For a gve samle sze the error s decomosed: Modelg error Statstcal error (Model selecto error) Total error Model comlexty
Am: fd a small tree cosstet wth the trag examles Idea: (recursvely) choose "most sgfcat" attrbute as root of (sub)tree
Idea: a good attrbute slts the examles to subsets that are (deally) "all ostve" or "all egatve" Patros? s a better choce
To mlemet Choose-Attrbute the DTL algorthm Iformato Cotet (Etroy): I(P(v 1 ),, P(v )) = Σ =1 -P(v ) log 2 P(v ) For a trag set cotag ostve examles ad egatve examles: I(, ) log 2 log 2
A chose attrbute A dvdes the trag set E to subsets E 1,, E v accordg to ther values for A, where A has v dstct values. Iformato Ga (IG) or reducto etroy from the attrbute test: Choose the attrbute wth the largest IG v I A remader 1 ), ( ) ( ) ( ), ( ) ( A remader I A IG
For the trag set, = = 6, I(6/12, 6/12) = 1 bt Cosder the attrbutes Patros ad Tye (ad others too): 2 4 6 2 4 IG( Patros) 1[ I(0,1) I(1,0) I(, )].0541bts 12 12 12 6 6 2 1 1 2 1 1 4 2 2 4 2 IG( Tye) 1[ I(, ) I(, ) I(, ) I( 12 2 2 12 2 2 12 4 4 12 4 2, )] 4 0 bts Patros has the hghest IG of all attrbutes ad so s chose by the DTL algorthm as the root
Decso tree leared from the 12 examles: Substatally smler tha true tree---a more comlex hyothess s t justfed by small amout of data
abset Bleedg weak strog Oset Regularty P(D Bleedg=strog) Oset=early Oset=late regular P(D a,e) Mutato P(D w,r) h.wld mutated P(D a,l,h.w.) P(D a,l,m) rregular Mutato h.wld mutated P(D w,,h.w.) P(D w,,m) Decso tree: Each teral ode rereset a (uvarate) test, the leafs cotas the codtoal robabltes gve the values alog the ath. Decso grah: If codtos are equvalet, the subtrees ca be merged. E.g. If (Bleedg=abset,Oset=late) ~ (Bleedg=weak,Regularty=rreg)