A Negative Log Likelihood Function-Based Nonlinear Neural Network Approach

Size: px

Start display at page:

Download "A Negative Log Likelihood Function-Based Nonlinear Neural Network Approach"

Laurence Grant
5 years ago
Views:

1 A Negaive Log Likelihood Funcion-Based Nonlinea Neual Newok Appoach Ponip Dechpichai,* and Pamela Davy School of Mahemaics and Applied Saisics Univesiy of Wollongong, Wollongong NSW 5, AUSTRALIA * Coesponding auho: pd5@uow.edu.au Absac The mos commonly used objecive funcion in Aificial Neual Newoks (ANNs) is he sum of squaed eos. This equies he age and foecased oupu veco o have he same dimension. In he conex of nonlinea financial ime seies, boh condiional mean and vaiance (volailiy) end o evolve ove ime. I is heefoe of inees o conside neual newoks wih wo-dimensional oupu even hough he age daa ae one-dimensional. The idea of he back-popagaion algoihm can be exended o his siuaion. Fo example, he negaive log-likelihood based on a paameic saisical model is a possible alenaive o he adiional leas squaes objecive. I has been found ha he RMSPE fo he mean is smalle fo of he developed neual newok (LLNN) han adiional neual newok (NN) in he majoiy cases. LLNN also povides compaable pefomance o NN fo he eal daa se (Index of Sock Exchange of Thailand). KEY WORDS: Aificial neual newok, log-likelihood, condiional mean, condiional heeoscedasiciy. Inoducion Financial daa, especially sock euns, may no always confom o a linea model. Howeve, he majoiy of echniques developed fo modeling financial ime seies aim o deec linea sucue. Nonlinea behavio in financial ime seies modeling can ypically be classified ino wo goups including nonlineaiy in mean and nonlineaiy in vaiance (Tsay, 00). Supisingly, he exising nonlinea models developed appea o meely have a single focus. Even hough, hee migh exis a few models ha conside boh aeas of he nonlinea behavio in financial ime seies, hese models seem o pefom sequenially. Addiionally, i can be found ha mos nonlinea models developed in he saisical lieaue focus on he condiional mean (Tong, 990), while hose in he financial lieaue focus on he condiional vaiance. Thee is, heefoe, a eseach oppouniy o develop a new echnique o enable modeling of nonlinea financial ime seies in boh he condiional mean and he condiional vaiance simulaneously. This eseach will be caied ou using he advanages of a nonlinea neual newok, ained by a log-likelihood based eo funcion o poduce wo oupus simulaneously o epesen mean and vaiance. Lieaue This secion povides a summay of nonlinea models and neual newoks. Some noaion used lae in his pape will also be defined hee. In mos financial sudies, euns ae analyzed ahe han diecly obseved pices because of a scale-fee summay of he invesmen oppouniy and aacive saisical popeies. Le be a sock eun a ime, and le F be he condiioning infomaion se available a ime, on which foecass ae based. 443

2 Nonlinea models Conside he model Dechpichai and Davy. ASIMMOD007, Chiang Mai, Thailand = g(f ) + h(f ) ε [] whee g (.) and h (.) ae well-defined funcions wih h (.) > 0, and ε is he cuen shock wih E( ε ) = 0 and Va( ε ) =. Geneally, F denoes he collecion of elemens in {,,...} and ε, ε,...}. { The condiional mean and vaiance of given F ae in Eq.[] and Eq.[3] especively. μ = ( F ) g(f ) [] E σ = Va( F ) h(f ) [3] can be egaded as nonlinea in mean if g (.) is nonlinea, and nonlinea in vaiance if h (.) is imevaian. This leads o a naual (alhough no muually exclusive) classificaion in nonlinea financial ime seies. Nonlineaiy in vaiance Volailiy is a measuemen of flucuaion, he chaaceisic o fall o ise shaply in pice ove a peiod. Mahemaically, volailiy is ofen calculaed using sandad deviaion. Volailiy models of asse euns ae geneally efeed o as condiional heeoscedasic models, which ae nonlinea in vaiance because hei condiional vaiance ( σ ) evolves ove ime. The GARCH model, poposed by Bolleslev (986) is a genealizaion of he ARCH model (Engle, 98). The GARCH(p,q) model is shown in Eq.[4] (p = 0 coesponds o ARCH(q)): p i= i i q j= a j j σ = α + β σ + γ [4] whee a = σ ε and ε is a sequence of independen and idenically disibued (iid) andom vaiables wih mean zeo and vaiance, α > 0, β 0 γ 0 and max(p,q) i= ( β + γ ) Nonlineaiy in mean i j < i i Tadiional auoegessive ime seies ae based on linea elaionships beween he expeced values of fuue obsevaions condiional upon pas obsevaions. Howeve, many echniques have been ecenly poposed o model nonlineaiy, including Aificial Neual Newoks (ANNs) (Donaldson and Kamsa, 996; Omonei and Neuneie, 996; Qi, 999; Schienkopf, e al. 000). Alhough ANNs ae usually descibed as having he objecive of fiing age values ahe han esimaing means, minimizing expeced squaed eo is heoeically equivalen o finding a condiional mean. Implicily, he use of unweighed leas squaes is equivalen o an assumpion of consan vaiance. So ANN s wih a leas squaes objecive (as descibed in he nex secion) implicily fi models which ae nonlinea in mean bu no in vaiance. 444

Dechpichai and Davy. ASIMMOD007, Chiang Mai, Thailand Figue : A feed-fowad neual newok (Secion ). Figue : Feed-fowad neual newok wih wo oupus and single age vaiable (Secion 3).

The compaaive advanage of ANNs ove moe convenional nonlinea financial ime seies models is ha hey can appoximae any nonlinea funcion o an abiay degee of accuacy wih a suiable numbe of hidden unis

3 Dechpichai and Davy. ASIMMOD007, Chiang Mai, Thailand Figue : A feed-fowad neual newok (Secion ). Figue : Feed-fowad neual newok wih wo oupus and single age vaiable (Secion 3). Aificial Neual Newoks: ANNs ANNs ae a class of genealized nonlinea nonpaameic model inspied by sudies of he bain and neve sysem. The compaaive advanage of ANNs ove moe convenional nonlinea financial ime seies models is ha hey can appoximae any nonlinea funcion o an abiay degee of accuacy wih a suiable numbe of hidden unis hough he composiion of a newok of elaively simple funcions. The feed-fowad mulilayeed pecepon achiecue poposed hee is a neual newok pocess fom one laye o he nex using an "acivaion funcion". Gaphically, a feed-fowad neual newok can be epesened as in Figue, which shows a simple achiecue wih one hidden laye and a single oupu node. The inpu laye can be epesened by a veco X = (x, x,..., x )', he hidden laye can be epesened by a veco H = (h, h,..., h )', and is he oupu. The value h of he m j node in he hidden laye is defined as Eq.[5] n h j n + w x ), j j j 0 j = ij i i= h = F (w,,..., m [5] whee w 0 j is called a bias, w is he weigh feeding fom he i h inpu uni o he ij F j (.) is an acivaion funcion. Similaly, he hidden nodes ae elaed o he oupu laye as follows: h j hidden uni, and ˆ = G o (b 0o m + b h ) j= jo j [6] whee ˆ is he oupu, m is numbe of hidden unis, b is called a bias, b 0o jo is he weigh feeding h fom he j hidden uni o he oupu uni and G o (.) is an acivaion funcion. Subsiuing Eq.[5] ino Eq.[6], he oupu funcion can be ewien as n ( w + w x ) f (X, ) 0 j = θ ij i m ˆ = G b b F o + 0o jo j [7] j= i= whee θ = ( b,...,b, w,..., w,..., w,..., w )' is he veco of newok weighs. Eq.[7] is efeed o 0 m 0 n 0m nm as a semi-paameic funcion due o he fac ha only is funcional fom is known, bu he numbe of unis and hei biases and weighs ae unknown. The oupu laye can be exended fom a single oupu o muli-oupus by allowing o o ange fom o k. F and G can ake seveal acivaion funcional foms, such as sigmoid, hypebolic angen, exponenial, and linea funcion. The mos widely used esimaion mehod, o so-called leaning ule, of ANNs is Backpopagaion leaning. Back-popagaion is a ecusive gadien descen mehod ha minimizes he sum of squaed eos of he sysem by moving down he gadien of he eo cuve. Moe specifically, he newok weigh veco θ is chosen o minimize he eo funcion E as shown in Eq.[0] fo a single oupu: 445

4 Dechpichai and Davy. ASIMMOD007, Chiang Mai, Thailand N E = ( ˆ ) [8] N = whee N is he sample size, is acual oupu value, and ˆ is he calculaed oupu value as in Eq.[7]. The back-popagaion leaning algoihm woks as he name suggess. Tha is, he eo is popagaed back hough he newok and weighs ae adjused o make he eo smalle, afe popagaing inpu hough he newok. The basic concep of back-popagaion is he epeaed applicaion of he chain ule o compue he influence of each weigh in he newok upon he eofuncion ( E ), saing wih he oupu laye fis as in Eq.[], and hen exending backwads o he pevious layes. ( ˆ) b jo ( ˆ) = ˆ ˆ z z b jo [9] whee z is he weighed sum of he inpus of he oupu neuon. When he paial deivaive fo each weigh is known, he aim of minimizing he eo-funcion is achieved by pefoming a simple gadien descen. Thee ae many kinds of Adapive leaning algoihms in back-popagaion leaning. Howeve, he Resilien Back-Popagaion algoihm (Rpop) poposed by Riedmille and Baun (993) has been employed in his pape. Eo funcion Modificaion The adiional appoach fo aining a neual newok equies he same dimension fo acual and age oupus. Wih he objecive of simulaneous pedicion of condiional mean and condiional vaiance, i is of inees o conside neual newoks wih wo-dimensional oupu even hough he age daa ae one-dimensional (Figue ). The log-likelihood funcion ( LL ) of a wo-paamee nomal pobabiliy densiy funcion shown in Eq.[0] povides a link beween wo oupus ( μ ˆ, σˆ ) and a single age (). LL = ln ( ) μˆ ˆ ˆ σ πσ e = ln( πσˆ ) μˆ σˆ [0] The aveage negaive log likelihood can be used o define he eo-funcion in Eq.[], o be minimized wih espec o he veco of newok weighs θ. N E = LL( ; μˆ, σˆ ) [] N = The acivaion funcion used fo he vaiance oupu mus be chosen o guaanee non-negaive values. Moeove, esimaed vaiances close o zeo will poduce singulaiies in he eo funcion. This is a possible cause of esimaed vaiances diffeing fom acual vaiance, esuling in ove-fiing. A minimum vaiance paamee ( σ ), called acuiy, will be chosen because of no compleely pecise 0 measuemen and will be imposed ino he acivaion funcion fo he vaiance oupu. G (z) + o z = σ0 e [] Gadien descen on an eo Because back-popagaion leaning is used in esimaion, deivaives of he new eo-funcion as Eq.[] ae equied wih espec o he wo desied oupus, he condiional mean and he condiional vaiance: 446

5 Dechpichai and Davy. ASIMMOD007, Chiang Mai, Thailand LL μˆ = μˆ σˆ [3] LL = σˆ σˆ ( μˆ ) σˆ 4 [4] These deivaives ae used fo he fis em in chain ule as Eq [9] fo calculaion weighs of back-popagaion. Expeimenal esuls This secion povides he deails of daa se used in analyzing he pefomance of echniques. The pefomance of each echnique will be also specified in cases of simulaed and eal daa. Daa ses Simulaion daa ae used in his pape because he ue volailiies ae needed in he evaluaion sep. Vaiances have been simulaed o be ime dependen as he heeoscedasic chaaceisics of sock euns. ~ N( μ, σ ) [5] whee μ ~ N( μ,) is an AR()-pocess and σ ~ α + βσ + η, α + βσ + γσ + η o α + βσ + γ[( μ ) / σ ] + η especively, whee α = 0., β = 0.5, γ = 0. and η ~ N(0,). Moeove, in ode o peven simulaed daa having uni oo, he auoegessive paamee has been se 0.95 in AR()-pocess. Theefoe, hee ae hee daa ses, called SIMR, SIMR, SIMR3 especively, each of hem has 00 obsevaions. They have been analyzed by boh seleced cuen echnique and new echnique hough ou his pape. Some basic saisics of simulaed daa ae summaized in Table. Table : Basic saisics of he hee ses of simulaed daa Saisics SIMR SIMR SIMR3 μ σ μ σ μ σ Mean S.D Table : Basic saisics of he hee ses of simulaed daa classified by ole of daa se Daa Se Sais- SIMR SIMR SIMR3 ics μ σ μ σ μ σ Taining Se Mean S.D Validaion Se Mean S.D Tes Se Mean S.D

6 Dechpichai and Davy. ASIMMOD007, Chiang Mai, Thailand Nomally, daa ses have been paiioned ino wo disjoin pas: Taining daa and Tes daa fo evaluaion puposes. Howeve, in ode o peven ove-fiing, he hid Validaion se has been used o esimae newok pefomance duing aining as equied by he sopping cieia and neve used fo weigh adjusmen. The daa, heefoe, has been paiioned ino Taining (50%), Validaion (5%), and Tes se (5%) (Pechel, 998). Some basic saisics of simulaed μ,σ and classified by ole of daa se have been epoed in Table. Simulaion esuls The Taining Se is egaded as In-sample daa because i is used o obain he bes weighs, while he Tes se (Ou-sample Se) is used fo evaluaing he model. The same newok achiecue and iniial weighs ae used in all models. Thee ae wo echniques used o compae pefomance in modeling. The fis echnique is he oiginal neual newok, which uses sum of squaed eos o be eo-funcion, so called NN. The second echnique is he new echnique, adoping log-likelihood of nomal pobabiliy densiy funcion o be eo-funcion, so called LLNN. The same acivaion funcion, sigmoid funcion has been used in all layes, excep in he las laye whee a linea funcion is used fo he condiional mean oupu and an exponenial funcion wih added acuiy, as in equaion [], is used in he condiional vaiance oupu o guaanee non-negaive values. The achiecue used houghou his expeimen (boh simulaed and eal daa) is wo hidden layes wih five nodes each and hee inpus: lags and of eun and sandad deviaion of pevious euns. The Roo Mean Squae Pedicion Eo (RMSPE) in Eq [6], whee foecased value especively, has been used o evaluae pefomance. Δ and ˆΔ ae acual and N RMSPE = ( Δ Δˆ ) [6] N = Table 3: Roo Mean Squae Pedicion Eo classified in Taining and Tes Se by simulaed daa ses. Taining Se Tes Se Mehod SIMR SIMR SIMR3 SIMR SIMR SIMR3 μ σ μ σ μ σ μ σ μ σ μ σ NN LLNN Eos of each mehod fo mean and vaiance pedicion wihin he Taining and Tes Se have been epoed in Table 3. The RMSPE fo he mean is smalle fo of LLNN han NN in all cases excep in SIMR3 (Taining Se). Thee is no RMSPE calculaion fo vaiance in he case of NN, as his model only povides mean pedicions. Applicaion o eal daa (NN) m(llnn) (NN) m(llnn) a) Taining Se b) Tesing Se Figue 3: Acual and foecased euns of Closed Index (monhly) fom NN and LLNN. 448

7 Dechpichai and Davy. ASIMMOD007, Chiang Mai, Thailand RMSPE, measue fo pefomance evaluaion equies acual and foecased values fo calculaion. Howeve, when he models have been applied o eal daa, having only acual eun in eal daa, his measue could no diecly done; heefoe, hee is inees o seach how o evaluae model s pefomance. Daily and monhly Sock Index daa fom Sock Exchange of Thailand (SET) fom 00() o 005(5) has been employed in his pape. Reuns ae calculaed fom he fis diffeence of log of Closed Index. The plos (Figue 3) show ha LLNN has compaable pefomance o NN m m /00 04/00 05/00 06/00 07/00 08/00 09/00 0/00 /00 /00 0/00 0/00 03/00 04/00 05/00 06/00 07/00 08/00 09/00 0/00 /00 /00 0/003 0/003 03/003 05/004 06/004 07/004 08/004 09/004 0/004 /004 /004 0/005 0/005 03/005 04/005 05/005 a) Taining Se b) Tesing Se Figue 4: Foecased mean wih foecased sandad deviaion fom he LLNN. Alhough LLNN s mean pedicion is smoohe and slighly sensiive han NN, i has been compensaed by poviding ohe useful infomaion (volailiy) fo ading in sock make (Figue 4). The plos show acual eun supeimposed on he pediced mean plus/minus sandad deviaion. On mos of he occasions when hee is a lage discepancy beween he pediced mean and he acual eun, he pediced volailiy also ends o be lage. This foecased condiional vaiance (volailiy) povided by LLNN could be of poenial benefi fo he pupose of isk managemen and developmen of ading saegies. Summay and discussion In his eseach, a neual newok, based on he log likelihood of nomal pobabiliy densiy funcion (LLNN), has been developed and hen applied o daa aificially simulaed o mimic key chaaceisics of financial ime seies. The esuls wee compaed o hose obained fom a adiional neual newok (NN). The LLNN allows diffeen dimensions of acual oupu and foecased oupu. Oveall, he LLNN model povides slighly bee pefomance han NN in pedicing he condiional mean fo he simulaed daa, and compaable pefomance fo he eal daa se. Howeve, i has he addiional benefi of esimaing ime dependen volailiy. A pomising idea fo fuue eseach is o include ecuen sucues ino he model ahe han feed-fowad achiecue. I would also be ineesing o compae pofiabiliy calculaed fom ading saegies based on models fied o eal financial daa. Refeences Bolleslev, T Genealized auoegessive condiional heeoskedasiciy. Jounal of Economeics 3: Engle, R.F. 98. Auoegessive condiional heeoscedasiciy wih esimaes of he vaiance of Unied Kingdom inflaion. Economeica 50:987. Donaldson, R.G., and Kamsa, M Foecas combining wih neual newoks. Jounal of Foecasing 5:49-6. Omonei, D., and Neuneie, R Expeimens in pedicing he Geman sock index DAX wih densiy esimaing neual newoks.p In: Poceedings of he IEEE/IAFE 996 Confeence on Compuaional Inelligence in Financial Engineeing (CIFE). 449

8 Dechpichai and Davy. ASIMMOD007, Chiang Mai, Thailand Pechel, L Ealy Sopping - Bu When? p In: O, G.B., and Mülle, K-R. (Eds.). Neual Newok: Tick of he Tade. Vol. 54 of lecue noes in compue science. Spinge- Velag. Qi, M Nonlinea pedicabiliy of sock euns using financial and economic vaiables. Jounal of Business & Economics Saisics 7: Riedmille, M., and Baun, H A diec adapive mehod fo fase back popagaion leaning: The pop algoihm. p In: Poceedings of he IEEE Inenaional Confeence on Neual Newoks. Schienkopf, C., Doffne, G., and Dockne, E. J Foecasing ime-dependen condiional densiies: A semi-nonpaameic neual newok appoach. Jounal of Foecasing 9:355. Tong, H Non-Linea Time Seies: A Dynamical Sysem Appoach. Oxfod. Oxfod Univesiy Pess. Tsay, R.S. 00. Analysis of Financial Time Seies. Canada. Wiley. 450

Low-complexity Algorithms for MIMO Multiplexing Systems

Low-complexity Algorithms for MIMO Multiplexing Systems Low-complexiy Algoihms fo MIMO Muliplexing Sysems Ouline Inoducion QRD-M M algoihm Algoihm I: : o educe he numbe of suviving pahs. Algoihm II: : o educe he numbe of candidaes fo each ansmied signal. :