4/18/2005. Statistical Learning Theory

Transcription

1 Statistical Leaning Theoy

2 Statistical Leaning Theoy A model of supevised leaning consists of: a Envionment - Supplying a vecto x with a fixed but unknown pdf F x (x b Teache. It povides a desied esponse d fo evey x accoding to a conditional pdf ( x d. These ae elated by d = f ( x v F x

3 Statistical Leaning Theoy v is a noise tem. c Leaning machine. It is capable of implementing a set of I/O mapping functions: y = F( x w whee y is the actual esponse and w is a set of fee paametes (weights selected fom the paamete (weight space Wˆ.

4 Statistical Leaning Theoy The supevised leaning poblem is that of selecting the paticula F( x w that appoximates d in an optimum fashion. The selection itself is based on a set of iid taining samples: Each sample is dawn fom Tˆ with a joint pdf ˆ = = T {( x N i d i } i 1 F x d ( x d

5 Statistical Leaning Theoy Supevised leaning depends on the following: Do the taining examples {( x d } i i contain enough infomation to constuct a LM capable of good genealization? To answe we will see this poblem as an appoximation poblem. We wish to find the function F( x w which is the best possible appoximation to f (x.

6 Statistical Leaning Theoy Let 2 L( d F( x w = ( d F( x w denote a measue of the discepancy between a d coesponding to a vecto and the actual esponse poduced by x F( x w The expected value of the loss is defined by the isk functional R( w = L( d F( x w df ( x d x D

7 Statistical Leaning Theoy The isk functional may be easily undestood fom the finite appoximation = R ( w L( x d P( x d i whee ( x i d i denotes the pobability of dawing the i-th sample. P i i i i

8 Pinciple of Empiical Risk Minimization Instead of using R(w we use an empiical measue: 1 ( RE w = L( d i F( xi w N i This measue diffes fom R(w in two desiable ways: a It does not depend on the unknown pdf ( x explicitly. F x D d

9 Pinciple of Empiical Risk Minimization b In theoy it can be minimized with espect to w Let w E and F( x we denote the weight vecto and the mapping that minimize R E (w Also let w 0 and F( x w0 denote the analogues fo R(w Both and w coespond to the space Wˆ. w 0 E

10 Pinciple of Empiical Risk Minimization We must now conside unde which conditions is close to F( x we F( x w0 as measued by the mismatch between R E (w and R(w.

11 Pinciple of Empiical Risk Minimization 1. In place of R(w constuct R E 1 ( w = N i L( d F( x w on the basis of the taining set of iid samples x i d i = 1... N ( i i i

12 Pinciple of Empiical Risk Minimization 2. R( w E conveges in pobability to the minimum possible values of R(w as N povided that R E (w conveges unifomly to R(w. 3. Unifom convegence as pe is necessay and sufficient fo consistency of the PERM. P( sup R( w R ( w > w W E 0

13 The Vapnik Chevonenkis Dimension The theoy of unifom convegence of R E (w to R(w includes ates of convegence based on a paamete called the VC dimension. It is a measue of the capacity o expessive powe of the family of classification functions ealized by the leaning machine.

14 The Vapnik Chevonenkis Dimension To descibe the concept of VC dimension let us conside a binay patten classification poblem fo which the desied esponse is d {01}. A dichotomy is a classification function. Let denote the set of dichotomies implemented by a leaning machine: Fˆ = { ( : ˆ m F x w w W F : R Wˆ {01}} Fˆ

15 The Vapnik Chevonenkis Dimension Let Lˆ denote the set of N points in the m- dimensional space Xˆ of input vectos: Lˆ = { x Xˆ i ; i = Lˆ 1... N} A dichotomy patitions into two disjoint sets ˆL 0 and ˆL 1 such that 0 fo x Lˆ 0 F( x w = 1 fo x Lˆ 1

16 The Vapnik Chevonenkis Dimension Let F ˆ ( Lˆ denote the numbe of distinct dichotomies implemented by the L.M. Let F ˆ ( l denote the maximum F ˆ ( Lˆ ove all Lˆ with L ˆ = l. L Lˆ is shatteed by Fˆ if. That is F L ˆ ˆ ( ˆ = 2 if all the possible dichotomies of Lˆ can be induced by functions in Fˆ.

17 The Vapnik Chevonenkis Dimension In the figue we illustate a two-dimensional space consisting of 4 points (x 1...x 4. The decision boundaies of F 0 and F 1 coespond to the classes 0 and 1 being tue. F 0 induces the dichotomy:

18 The Vapnik Chevonenkis ˆ { ˆ [ ] ˆ D0 = L0 = x1 x2 x4 L1 = [ x 3 ]} While F 1 induces ˆ { ˆ [ ] ˆ D1 = L0 = x1 x2 L1 = [ x3 x 4 ]} Dimension with the set Lˆ consisting of fou points the cadinality Lˆ = 4 Hence 4 ˆ ( Lˆ = 2 = 16 F

19 The Vapnik Chevonenkis Dimension We now fomally define the VC dimension as: The VC dimension of an ensemble of dichotomies Fˆ is the cadinality of the lagest set Lˆ that is shatteed by Fˆ.

20 The Vapnik Chevonenkis Dimension In moe familia tems the VC dimension of the set of classification functions { F( x w : w Wˆ } is the maximum numbe of taining examples that can be leaned by the machine without eo fo all possible labelings of the classification functions.

21 Impotance of the VC Dimension Roughly speaking the numbe of examples needed to lean a class of inteest eliably is popotional to the VC dimension. In some cases the VC dimension is detemined by the fee paametes of a Neual Netwok. In this egad the following two esults ae of inteest.

22 Impotance of the VC Dimension Nˆ 1. Let denote an abitay feedfowad netwok built up fom neuons with a theshold activation function: ϕ( v = 1 0 Nˆ fo fo v < 0 the VC dimension of is O(W logw whee W is the total numbe of fee paametes in the netwok. v 0

23 Impotance of the VC Dimension Nˆ 2. Let denote a multilaye feedfowad netwok whose neuons use a sigmoid activation function ϕ( v 1 = 1 + e the VC dimension is O(W 2 whee W is the numbe of fee paametes in the netwok. v

24 Impotance of the VC Dimension In the case of binay patten classification the loss function has only two possible values: L[( d F( x w] = 0 if F( x w = 1 othewise The isk functional R( w and the empiical isk functional R emp ( w assume the following intepetations: d

25 Impotance of the VC Dimension R( w is the pobability of classification eo denoted by P( w. R emp ( w is the taining eo denoted by v( w. Then (Haykin p.98: P(sup P( w v( w > 0 as N

26 Impotance of the VC Dimension The notion of VC povides a bound on the ate of unifom convegence. Fo the set of classification functions with VC dimension h the following inequality holds: P(sup P( w v( w > < 2eN h exp( 2 N (vc.1 whee N is the size of the taining sample. In othe wods a finite VC dimension is a necessay and sufficient condition fo unifom convegence of the pinciple of empiical isk minimization. h

27 Impotance of the VC dimension The facto 2eN / h in (vc.1 epesents a bound on the gowth function F ˆ ( l fo the family of functions ˆ F = { F( x w; w Wˆ } fo l h 1 Povided that this function does not gow too fast the ight hand side will go to zeo as N goes to infinity. This equiement is satisfied if the VC dimension is finite. ( h

28 Impotance of the VC Dimension Thus a finite VC dimension is a necessay and sufficient condition fo unifom convegence of the pinciple of empiical isk minimization. Let α denote the pobability of occuence of the event sup P( w v( w α using the pevious bound (vc.1 we find α = 2eN h exp( 2 N h (vc.2

29 Impotance of the VC Dimension ( N h Let 0 α denote the special value of that satisfies (vc.2. Then we obtain (Haykin 99: h 2N 1 0 ( N h α = log 1 logα N + h N We efe to as the confidence inteval. 0

30 Impotance of the VC Dimension We may also wite whee ( ( ( 1 v h N w v w P α = ( ( 1 1 ( 2 ( α α α h N w v h N v h N

31 Impotance of the VC Dimension Conclusions: Fo a small taining eo (close to zeo: 3. Fo a lage taining eo (close to unity: ( ( ( 1 v h N w v w P α + ( 4 ( ( 0 α h N w v w P + ( ( ( 0 α h N w v w P +

32 Stuctual Risk Minimization The taining eo is the fequency of eos made duing the taining session fo some machine with weight vecto w duing the taining session. The genealization eo is the fequency of eos made by the machine when it is tested with examples not seen befoe. Let this two eos to be denoted with (w and (w. v tain v gene

33 Stuctual Risk Minimization Let h be the VC dimension of a family of classification functions F( x w ; w Wˆ with espect to the input space Xˆ The genealization eo v gene (w is lowe than the guaanteed isk defined by the sum of competing tems v w = v ( w + ( N h α v whee the confidence inteval is defined as befoe. { } guaant ( tain 1 tain ( N h α v 1 tain

34 Stuctual Risk Minimization Fo a fixed numbe of taining samples N the taining eo deceases monotonically as the capacity o h is inceased wheeas the confidence inteval inceases monotonically. + + = ( ( 1 1 ( 2 ( α α α h N w v h N v h N tain

35 Stuctual Risk Minimization The taining eo is the fequency of eos made duing the taining session fo some machine with weight vecto w duing the taining session. The genealization eo is the fequency of eos made by the machine when it is tested with examples not seen befoe. Let this two eos to be denoted with (w and (w. v tain v gene

36 Stuctual Risk Minimization The taining eo is the fequency of eos made duing the taining session fo some machine with weight vecto w duing the taining session. The genealization eo is the fequency of eos made by the machine when it is tested with examples not seen befoe. Let this two eos to be denoted with (w and (w. v tain v gene

37 Stuctual Risk Minimization The challenge in solving a supevised leaning poblem lies in ealizing the best genealization pefomance by matching the machine capacity to the available amount of taining data fo the poblem at hand. The method of stuctual isk minimization povides an inductive pocedue to achieve this goal by making the VC dimension of the leaning machine a contol vaiable.

38 Stuctual Risk Minimization Conside an ensemble of patten classifies and define a nested stuctue of n such machines such that we have coespondingly the VC dimensions of the individual patten classifies satisfy which implies that the VC dimension of each classifie is finite (see next figue { F( x w : w Wˆ } Fˆ k = { F( x w; w Wˆ k } k = 1... Fˆ Fˆ h 1 ˆ F n h... 2 n h n

39 Illustation of elationship between taining eo confidence inteval and guaanteed isk

40 Stuctual Risk Minimization Then: a The empiical isk (taining eo of each classifie is minimized b The patten classifie ˆF * with the smallest guaanteed isk is identified; this paticula machine povides the best compomise between the taining eo (quality of appoximation and the confidence inteval (complexity of the appoximation function.

41 Stuctual Risk Minimization Ou goal is to find a netwok stuctue such that deceasing the VC dimension occus at the expense of the smallest possible incease in taining eo. We achieve this fo example vaying h by vaying the numbe of hidden neuons. We evaluate the ensemble of fully connected multilaye feedfowad netwoks in which the numbe of neuons in one of the hidden layes is inceased in a monotonic fashion.

42 Stuctual Risk Minimization The pinciple of SRM states that the best netwok in this ensemble is the one fo which the guaanteed isk is the minimum.