1.1. Example: Polynomial Curve Fitting 4 1. INTRODUCTION

Transcription

1 4. INTRODUCTION Figure.2 Plo of a raining daa se of N = poins, shown as blue circles, each comprising an observaion of he inpu variable along wih he corresponding arge variable. The green curve shows he funcion sin(2π) used o generae he daa. Our goal is o predic he value of for some new value of, wihou knowledge of he green curve. deailed reamen lies beyond he scope of his book. Alhough each of hese asks needs is own ools and echniques, many of he key ideas ha underpin hem are common o all such problems. One of he main goals of his chaper is o inroduce, in a relaively informal way, several of he mos imporan of hese conceps and o illusrae hem using simple eamples. Laer in he book we shall see hese same ideas re-emerge in he cone of more sophisicaed models ha are applicable o real-world paern recogniion applicaions. This chaper also provides a self-conained inroducion o hree imporan ools ha will be used hroughou he book, namely probabiliy heory, decision heory, and informaion heory. Alhough hese migh sound like dauning opics, hey are in fac sraighforward, and a clear undersanding of hem is essenial if machine learning echniques are o be used o bes effec in pracical applicaions... Eample: Polynomial Curve Fiing We begin by inroducing a simple regression problem, which we shall use as a running eample hroughou his chaper o moivae a number of key conceps. Suppose we observe a real-valued inpu variable and we wish o use his observaion o predic he value of a real-valued arge variable. For he presen purposes, i is insrucive o consider an arificial eample using synheically generaed daa because we hen know he precise process ha generaed he daa for comparison agains any learned model. The daa for his eample is generaed from he funcion sin(2π) wih random noise included in he arge values, as described in deail in Appendi A. Now suppose ha we are given a raining se comprising N observaions of, wrien (,..., N ) T, ogeher wih corresponding observaions of he values of, denoed (,..., N ) T. Figure.2 shows a plo of a raining se comprising N = daa poins. The inpu daa se in Figure.2 was generaed by choosing values of n, for n =,...,N, spaced uniformly in range [, ], and he arge daa se was obained by firs compuing he corresponding values of he funcion

2 .. Eample: Polynomial Curve Fiing 5 sin(2π) and hen adding a small level of random noise having a Gaussian disribuion (he Gaussian disribuion is discussed in Secion.2.4) o each such poin in order o obain he corresponding value n. By generaing daa in his way, we are capuring a propery of many real daa ses, namely ha hey possess an underlying regulariy, which we wish o learn, bu ha individual observaions are corruped by random noise. This noise migh arise from inrinsically sochasic (i.e. random) processes such as radioacive decay bu more ypically is due o here being sources of variabiliy ha are hemselves unobserved. Our goal is o eploi his raining se in order o make predicions of he value of he arge variable for some new value of he inpu variable. As we shall see laer, his involves implicily rying o discover he underlying funcion sin(2π). This is inrinsically a difficul problem as we have o generalize from a finie daa se. Furhermore he observed daa are corruped wih noise, and so for a given here is uncerainy as o he appropriae value for. Probabiliy heory, discussed in Secion.2, provides a framework for epressing such uncerainy in a precise and quaniaive manner, and decision heory, discussed in Secion.5, allows us o eploi his probabilisic represenaion in order o make predicions ha are opimal according o appropriae crieria. For he momen, however, we shall proceed raher informally and consider a simple approach based on curve fiing. In paricular, we shall fi he daa using a polynomial funcion of he form y(, w) =w + w + w w M M = M w j j (.) where M is he order of he polynomial, and j denoes raised o he power of j. The polynomial coefficiens w,...,w M are collecively denoed by he vecor w. Noe ha, alhough he polynomial funcion y(, w) is a nonlinear funcion of, i is a linear funcion of he coefficiens w. Funcions, such as he polynomial, which are linear in he unknown parameers have imporan properies and are called linear models and will be discussed eensively in Chapers 3 and 4. The values of he coefficiens will be deermined by fiing he polynomial o he raining daa. This can be done by minimizing an error funcion ha measures he misfi beween he funcion y(, w), for any given value of w, and he raining se daa poins. One simple choice of error funcion, which is widely used, is given by he sum of he squares of he errors beween he predicions y( n, w) for each daa poin n and he corresponding arge values n, so ha we minimize E(w) = 2 j= N {y( n, w) n } 2 (.2) n= where he facor of /2 is included for laer convenience. We shall discuss he moivaion for his choice of error funcion laer in his chaper. For he momen we simply noe ha i is a nonnegaive quaniy ha would be zero if, and only if, he

3 6. INTRODUCTION Figure.3 The error funcion (.2) corresponds o (one half of) he sum of he squares of he displacemens (shown by he verical green bars) of each daa poin from he funcion y(, w). n y( n, w) n funcion y(, w) were o pass eacly hrough each raining daa poin. The geomerical inerpreaion of he sum-of-squares error funcion is illusraed in Figure.3. Eercise. We can solve he curve fiing problem by choosing he value of w for which E(w) is as small as possible. Because he error funcion is a quadraic funcion of he coefficiens w, is derivaives wih respec o he coefficiens will be linear in he elemens of w, and so he minimizaion of he error funcion has a unique soluion, denoed by w, which can be found in closed form. The resuling polynomial is given by he funcion y(, w ). There remains he problem of choosing he order M of he polynomial, and as we shall see his will urn ou o be an eample of an imporan concep called model comparison or model selecion. In Figure.4, we show four eamples of he resuls of fiing polynomials having orders M =,, 3, and9 o he daa se shown in Figure.2. We noice ha he consan (M = ) and firs order (M = ) polynomials give raher poor fis o he daa and consequenly raher poor represenaions of he funcion sin(2π). The hird order (M =3) polynomial seems o give he bes fi o he funcion sin(2π) of he eamples shown in Figure.4. When we go o a much higher order polynomial (M =9), we obain an ecellen fi o he raining daa. In fac, he polynomial passes eacly hrough each daa poin and E(w )=. However, he fied curve oscillaes wildly and gives a very poor represenaion of he funcion sin(2π). This laer behaviour is known as over-fiing. As we have noed earlier, he goal is o achieve good generalizaion by making accurae predicions for new daa. We can obain some quaniaive insigh ino he dependence of he generalizaion performance on M by considering a separae es se comprising daa poins generaed using eacly he same procedure used o generae he raining se poins bu wih new choices for he random noise values included in he arge values. For each choice of M, we can hen evaluae he residual value of E(w ) given by (.2) for he raining daa, and we can also evaluae E(w ) for he es daa se. I is someimes more convenien o use he roo-mean-square

4 .. Eample: Polynomial Curve Fiing 7 M = M = M =3 M =9 Figure.4 Figure.2. Plos of polynomials having various orders M, shown as red curves, fied o he daa se shown in (RMS) error defined by E RMS = 2E(w )/N (.3) in which he division by N allows us o compare differen sizes of daa ses on an equal fooing, and he square roo ensures ha E RMS is measured on he same scale (and in he same unis) as he arge variable. Graphs of he raining and es se RMS errors are shown, for various values of M, in Figure.5. The es se error is a measure of how well we are doing in predicing he values of for new daa observaions of. We noe from Figure.5 ha small values of M give relaively large values of he es se error, and his can be aribued o he fac ha he corresponding polynomials are raher infleible and are incapable of capuring he oscillaions in he funcion sin(2π). Values of M in he range 3 M 8 give small values for he es se error, and hese also give reasonable represenaions of he generaing funcion sin(2π), as can be seen, for he case of M =3, from Figure.4.

5 8. INTRODUCTION Figure.5 Graphs of he roo-mean-square error, defined by (.3), evaluaed on he raining se and on an independen es se for various values of M. Training Tes ERMS.5 3 M 6 9 For M =9, he raining se error goes o zero, as we migh epec because his polynomial conains degrees of freedom corresponding o he coefficiens w,...,w 9, and so can be uned eacly o he daa poins in he raining se. However, he es se error has become very large and, as we saw in Figure.4, he corresponding funcion y(, w ) ehibis wild oscillaions. This may seem paradoical because a polynomial of given order conains all lower order polynomials as special cases. The M =9polynomial is herefore capable of generaing resuls a leas as good as he M =3polynomial. Furhermore, we migh suppose ha he bes predicor of new daa would be he funcion sin(2π) from which he daa was generaed (and we shall see laer ha his is indeed he case). We know ha a power series epansion of he funcion sin(2π) conains erms of all orders, so we migh epec ha resuls should improve monoonically as we increase M. We can gain some insigh ino he problem by eamining he values of he coefficiens w obained from polynomials of various order, as shown in Table.. We see ha, as M increases, he magniude of he coefficiens ypically ges larger. In paricular for he M =9polynomial, he coefficiens have become finely uned o he daa by developing large posiive and negaive values so ha he correspond- Table. Table of he coefficiens w for polynomials of various order. Observe how he ypical magniude of he coefficiens increases dramaically as he order of he polynomial increases. M = M = M =6 M =9 w w w w w w w w w w

6 .. Eample: Polynomial Curve Fiing 9 N = 5 N = Figure.6 Plos of he soluions obained by minimizing he sum-of-squares error funcion using he M =9 polynomial for N =5daa poins (lef plo) and N = daa poins (righ plo). We see ha increasing he size of he daa se reduces he over-fiing problem. Secion 3.4 ing polynomial funcion maches each of he daa poins eacly, bu beween daa poins (paricularly near he ends of he range) he funcion ehibis he large oscillaions observed in Figure.4. Inuiively, wha is happening is ha he more fleible polynomials wih larger values of M are becoming increasingly uned o he random noise on he arge values. I is also ineresing o eamine he behaviour of a given model as he size of he daa se is varied, as shown in Figure.6. We see ha, for a given model compleiy, he over-fiing problem become less severe as he size of he daa se increases. Anoher way o say his is ha he larger he daa se, he more comple (in oher words more fleible) he model ha we can afford o fi o he daa. One rough heurisic ha is someimes advocaed is ha he number of daa poins should be no less han some muliple (say 5 or ) of he number of adapive parameers in he model. However, as we shall see in Chaper 3, he number of parameers is no necessarily he mos appropriae measure of model compleiy. Also, here is somehing raher unsaisfying abou having o limi he number of parameers in a model according o he size of he available raining se. I would seem more reasonable o choose he compleiy of he model according o he compleiy of he problem being solved. We shall see ha he leas squares approach o finding he model parameers represens a specific case of maimum likelihood (discussed in Secion.2.5), and ha he over-fiing problem can be undersood as a general propery of maimum likelihood. By adoping a Bayesian approach, he over-fiing problem can be avoided. We shall see ha here is no difficuly from a Bayesian perspecive in employing models for which he number of parameers grealy eceeds he number of daa poins. Indeed, in a Bayesian model he effecive number of parameers adaps auomaically o he size of he daa se. For he momen, however, i is insrucive o coninue wih he curren approach and o consider how in pracice we can apply i o daa ses of limied size where we

7 . INTRODUCTION ln λ = 8 ln λ = Figure.7 Plos of M =9polynomials fied o he daa se shown in Figure.2 using he regularized error funcion (.4) for wo values of he regularizaion parameer λ corresponding o ln λ = 8 and ln λ =. The case of no regularizer, i.e., λ =, corresponding o ln λ =, is shown a he boom righ of Figure.4. Eercise.2 may wish o use relaively comple and fleible models. One echnique ha is ofen used o conrol he over-fiing phenomenon in such cases is ha of regularizaion, which involves adding a penaly erm o he error funcion (.2) in order o discourage he coefficiens from reaching large values. The simples such penaly erm akes he form of a sum of squares of all of he coefficiens, leading o a modified error funcion of he form Ẽ(w) = N {y( n, w) n } 2 + λ 2 2 w 2 (.4) n= where w 2 w T w = w 2 + w wm 2, and he coefficien λ governs he relaive imporance of he regularizaion erm compared wih he sum-of-squares error erm. Noe ha ofen he coefficien w is omied from he regularizer because is inclusion causes he resuls o depend on he choice of origin for he arge variable (Hasie e al., 2), or i may be included bu wih is own regularizaion coefficien (we shall discuss his opic in more deail in Secion 5.5.). Again, he error funcion in (.4) can be minimized eacly in closed form. Techniques such as his are known in he saisics lieraure as shrinkage mehods because hey reduce he value of he coefficiens. The paricular case of a quadraic regularizer is called ridge regression (Hoerl and Kennard, 97). In he cone of neural neworks, his approach is known as weigh decay. Figure.7 shows he resuls of fiing he polynomial of order M =9o he same daa se as before bu now using he regularized error funcion given by (.4). We see ha, for a value of ln λ = 8, he over-fiing has been suppressed and we now obain a much closer represenaion of he underlying funcion sin(2π). If, however, we use oo large a value for λ hen we again obain a poor fi, as shown in Figure.7 for ln λ =. The corresponding coefficiens from he fied polynomials are given in Table.2, showing ha regularizaion has he desired effec of reducing

8 .. Eample: Polynomial Curve Fiing Table.2 Table of he coefficiens w for M = 9 polynomials wih various values for he regularizaion parameer λ. Noe ha ln λ = corresponds o a model wih no regularizaion, i.e., o he graph a he boom righ in Figure.4. We see ha, as he value of λ increases, he ypical magniude of he coefficiens ges smaller. ln λ = ln λ = 8 ln λ = w w w w w w w w w w Secion.3 he magniude of he coefficiens. The impac of he regularizaion erm on he generalizaion error can be seen by ploing he value of he RMS error (.3) for boh raining and es ses agains ln λ, as shown in Figure.8. We see ha in effec λ now conrols he effecive compleiy of he model and hence deermines he degree of over-fiing. The issue of model compleiy is an imporan one and will be discussed a lengh in Secion.3. Here we simply noe ha, if we were rying o solve a pracical applicaion using his approach of minimizing an error funcion, we would have o find a way o deermine a suiable value for he model compleiy. The resuls above sugges a simple way of achieving his, namely by aking he available daa and pariioning i ino a raining se, used o deermine he coefficiens w, and a separae validaion se, also called a hold-ou se, used o opimize he model compleiy (eiher M or λ). In many cases, however, his will prove o be oo waseful of valuable raining daa, and we have o seek more sophisicaed approaches. So far our discussion of polynomial curve fiing has appealed largely o inuiion. We now seek a more principled approach o solving problems in paern recogniion by urning o a discussion of probabiliy heory. As well as providing he foundaion for nearly all of he subsequen developmens in his book, i will also Figure.8 Graph of he roo-mean-square error (.3) versus ln λ for he M =9 polynomial. Training Tes ERMS ln λ 25 2

9 2. INTRODUCTION give us some imporan insighs ino he conceps we have inroduced in he cone of polynomial curve fiing and will allow us o eend hese o more comple siuaions..2. Probabiliy Theory A key concep in he field of paern recogniion is ha of uncerainy. I arises boh hrough noise on measuremens, as well as hrough he finie size of daa ses. Probabiliy heory provides a consisen framework for he quanificaion and manipulaion of uncerainy and forms one of he cenral foundaions for paern recogniion. When combined wih decision heory, discussed in Secion.5, i allows us o make opimal predicions given all he informaion available o us, even hough ha informaion may be incomplee or ambiguous. We will inroduce he basic conceps of probabiliy heory by considering a simple eample. Imagine we have wo boes, one red and one blue, and in he red bo we have 2 apples and 6 oranges, and in he blue bo we have 3 apples and orange. This is illusraed in Figure.9. Now suppose we randomly pick one of he boes and from ha bo we randomly selec an iem of frui, and having observed which sor of frui i is we replace i in he bo from which i came. We could imagine repeaing his process many imes. Le us suppose ha in so doing we pick he red bo 4% of he ime and we pick he blue bo 6% of he ime, and ha when we remove an iem of frui from a bo we are equally likely o selec any of he pieces of frui in he bo. In his eample, he ideniy of he bo ha will be chosen is a random variable, which we shall denoe by B. This random variable can ake one of wo possible values, namely r (corresponding o he red bo) or b (corresponding o he blue bo). Similarly, he ideniy of he frui is also a random variable and will be denoed by F. I can ake eiher of he values a (for apple) or o (for orange). To begin wih, we shall define he probabiliy of an even o be he fracion of imes ha even occurs ou of he oal number of rials, in he limi ha he oal number of rials goes o infiniy. Thus he probabiliy of selecing he red bo is 4/ Figure.9 We use a simple eample of wo coloured boes each conaining frui (apples shown in green and oranges shown in orange) o inroduce he basic ideas of probabiliy.