3-2-1 ANN Architecture

Transcription

1 ARTIFICIAL NEURAL NETWORKS (ANNs) Profssor Tom Fomby Dpartmnt of Economics Soutrn Mtodist Univrsity Marc 008 Artificial Nural Ntworks (raftr ANNs) can b usd for itr prdiction or classification problms. ANNs ar basd on rprsntations of nural activity in t brain. T most popular dsign for ANNs is t so-calld multilayr fd-forward ntwork. Suc ntworks av an input layr, an output layr, on or mor iddn layrs. T following arcitctural diagram rprsnts a 3-- prdiction ANN. T first numbr, 3, rprsnts t numbr of inputs in t input layr; t scond numbr,, rprsnts t numbr of nurons or nods,, in t iddn layr, t last numbr,, rprsnts t numbr of nods in t output layr. 3-- ANN Arcitctur INPUT HIDDEN OUTPUT X w w 0 w w w Φ w Φ0 X w 3 w w 0 w Φ Φ w 3 X 3 X, X, X 3 = tr inputs., = iddn nod in iddn layr Φ = output. w 0, w 0, w Φ0 = bias wigts, otrwis w s first subscript = iddn nod numbr w s scond subscript = input numbr A total of wigts to b dtrmind.

2 In t abov diagram x, x, x 3 rprsnt t tr inputs, t two iddn nods ar rprsntd by, t output nod is rprsntd by. Morovr t wigts conncting t tr inputs to t first iddn layr ar rprsntd by w, w, w3 wr t first subscript,, rprsnts linkag to t first nod of t iddn layr, wras t scond subscript rprsnts t input tat t wigt is associatd wit. Similarly, t wigts conncting t tr inputs to t scond nod of t iddn layr ar rprsntd by w, w, w 3. In going from t iddn layr to t output layr t wigts for t two iddn nods ar rprsntd by w w. In addition to ts conncting wigts, tis ANN also as bias wigts w 0, w 0, w Φ0. In ANNs t wigts ar applid vis-à-vis so-calld squasing or transfr functions, say f (). Popular squasing functions includ t logistic function, t arc tangnt function, t linar function. In XLMINER prdiction problms t iddn layr squasing function is t logistic function, wras t squasing function for t output layr(s) is t linar squasing function. Tis is bcaus in prdiction problms t output variabl is an intrval variabl can potntially rang in valu from to. In contrast, for classification problms, t output squasing function(s) is t logistic function bcaus, in classification problms, t output variabls ar of t binary form. To dmonstrat t stimatd form of a simpl 3-- ANN w considr t Boston Housing data. T output variabl is MEDV, t mdian valu of oms in t givn Boston ousing district, wil t input variabls ar RM, AGE, DIS. T ANN output for a 60% training data st wit t inputs bing normalid (mor about tis latr) is rproducd blow: Intr-layr connctions wigts Hiddn Layr # Input Layr RM AGE DIS Bias Nod Nod # Nod # Hiddn Layr # Output Layr Nod # Nod # Bias Nod Output Nod T input wigts going into t iddn nods ar as follows: w , w , w , w , w , w T iddn nod wigts 3 3 going into t output layr ar w w In addition

3 to ts wigts you av t bias nod wigts w w associatd wit, rspctivly, t first scond iddn layr nods wil t bias nod wigt for going from t iddn layr to t output layr is w Matmatically tn, t iddn nods ar rprsntd by wr t squasing function is t logistic function w0 wx wx w3x3 = RM AGE DIS w0 wx wx w3x3 = RM AGE DIS. Givn ts iddn nod valus w can gt t output nod by using t linar squasing function rsulting in w To scor t validation data st w fd t input valus of ac validation cas into t iddn nod formulas tn gt t output scor by using t output formula immdiatly abov. Tis modl is igly nonlinar in t wigts tat must b stimatd from t training data st, unfortunatly, a convntional mtod lik last squars is inappropriat. Instad a frquntly usd mtod for dtrmining t wigts of tis modl is t Back Propagation mtod wic w will discuss subsquntly. But bfor w do, lt s considr t two-iddn-layr ANN 3--- wic is rprsntd diagrammatically blow. 3

4 3--- ANN Arcitctur INPUT FIRST HIDDEN SECOND HIDDEN OUTPUT X w, w, 0 w, 0 w, w, w, w, w Φ w Φ0 X w, w,3 w,3 w, 0 w, w, 0 w, w Φ Φ X 3 X, X, X 3 = tr inputs, = iddn nod of First iddn layr, = iddn nod of Scond iddn layr w ij, k, = i, j iddn nod wigt associatd wit k-input (or nod in layr) w, 0, w, 0, w, 0, w, 0, w Φ0 = bias wigts Tn t two nods in t first iddn layr ar rprsntd by wr w,0 w, x w, x w,3 x3 w,0 w,x w,x w,3x3. T two nods of t scond iddn layr ar rprsntd by 4

5 wr w,0 w, w, w,0 w, w,. Finally t output layr is givn by w. 0 For practic, using t abov quations, you sould try to writ out t following 3--- ANN modl stimatd from t Boston Housing data st: Intr-layr connctions wigts Hiddn Layr # Input Layr RM AGE DIS Bias Nod Nod # Nod # Hiddn Layr # Hiddn Layr # Nod # Nod # Bias Nod Nod # Nod # Hiddn Layr # Output Layr Nod # Nod # Bias Nod Output Nod ANN Classification Modls T output layr of classification ANN modls as as many output nods as tr ar classification lvls. XLMINER only ls binary classification problms - on output nod for t succss () on output nod for t failur (0). Blow w rport a 3-- Classification Modl basd on t Boston Housing data using t binary classification variabl CAT.MEDV. T squasing function for going from t input layr to t output layr is t logistic function wil t squasing function for t output layrs is also t logistic function. Notic blow t output layr wigts for t 5

6 two classs ar ssntially qual in magnitud but opposit in sign wic guarants tat Pr(output = ) = Pr(output = 0) as on would dsir of binary probability outcoms. Intr-layr connctions wigts Input Layr Hiddn Layr # RM AGE DIS Bias Nod Nod # Nod # Hiddn Layr # Output Layr Nod # Nod # Bias Nod Tis ANN Classification modl is writtn matmatically as follows: T iddn nods in t singl iddn layr ar wr w0 wx wx w3x3 = RM AGE DIS w0 wx wx w3x3 = RM AGE DIS. Givn ts iddn nod valus w can gt t binary output nods using t following logistic squasing function rsulting in u 0 u 0 wr u,0,,

7 u 0 0,0 0, 0, Tn to scor tis modl in t sns of obtaining a confusion tabl on as to cos a cutoff probability for t succss class (=). Obviously t cutoff probability is a tuning paramtr in t ANN Classification Modl. Tat is, t confusion tabls for t ANN Classification modls ar dpndnt on t coic of cutoff probability. Normaliation of Input Data t Back Propagation Mtod It is oftn rcommndd tat t inputs to an ANN b normalid bfor training. By normaliation w man t following. Lt X dnot on of t inputs to t ANN. * T normalid valu of tis input, X, is dfind as follows: X * X X X X min, max min wr t minimum maximum valus of X ar rprsntd by X min X max, rspctivly. Tis normaliation convrts t original X valu to a normalid valu tat rsids in t [0,} intrval. It in turn lps t back propagation mtod bttr dtrmin t wigts of t ANN. On of t most popular mtods for dtrmining t wigts of ANN modls is t so-calld back propagation mtod. As t titl implis t rrors of t ANN ar calculatd from t output layr back troug t iddn layrs of t modl. Givn an ANN structur, t back propagation mtod starts out wit rom draws on t wigts nar ro. Tn t initial obsrvation of t training data st is run troug t ntwork, givn a prdiction problm, t rror is dtrmind as rr = ( y yˆ ) wr y is t first training valu of t output variabl ŷ is t ANN prdictd valu using t initially drawn wigts. Using tis rror, t connction bias wigts ar updatd by a fraction of t output rror. Givn t updatd wigts, t scond obsrvation of t training data st is fd troug t ntwork an rror is again dtrmind givn t scond ralid valu of t output variabl. Tis rror is tn usd to updat t wigts again wit ac likwis itration troug t training data st lading to, in gnral, a squnc of smallr smallr rrors, tus, smallr smallr rvisions of t wigts until on mor training obsrvation lads to a minimal rvision in t wigts. At tis point t final wigts of t ANN ar dtrmind t back propagation procss stops. Tn t rsulting ANN modl can b usd to scor additional data sts for validation tsting purposs. * X 7

8 Avoiding Ovr-Training of ANN Modls Picking t Rigt Arcitctur Of cours, t training data st fit can b continually improvd by making t structur of t ANN mor mor complx to t point of fitting t training data st prfctly. But tis would rsult in t fitting of not only t signal in t data but t nois as wll. Tis is, of cours, calld ovr-training t ANN modl. Unfortunatly, suc ovr-traind modls ar quit likly to prform poorly on an indpndnt data st. On way to prvnt t ovr-training of an ANN is to try svral diffrnt arcitcturs of incrasing complxity tn coos t arcitctur tat provids t bst accuracy wn scoring t validation tst data sts. Tat is, t training data st is usd to dtrmin t wigts of t compting ANN arcitcturs vis-à-vis back propagation wil t validation data st is usd to dtrmin t winning arcitctur, t winning arcitctur producing t bst validation data st scors. 8