Linear Inferential Modeling: Theoretical Perspectives, Extensions, and Comparative Analysis

Transcription

1 Intellgent Control nd Automton, 2012, 3, Publshed Onlne November 2012 ( Lner Inferentl Modelng: heoretcl Perspectves, Extensons, nd Comprtve Anlyss Muddu Mdkyru 1, Mohmed N. Nounou 1*, Hzem N. Nounou 2 1 Chemcl Engneerng Progrm, exs A & M Unversty, Doh, Qtr 2 Electrcl nd Computer Engneerng Progrm, exs A & M Unversty, Doh, Qtr Eml: * mohmed.nounou@qtr.tmu.edu Receved July 26, 2012; revsed August 26, 2012; ccepted September 4, 2012 ABSRAC Inferentl models re wdely used n the chemcl ndustry to nfer key process vrbles, whch re chllengng or expensve to mesure, from other more esly mesured vrbles. he m of ths pper s three-fold: to present theoretcl revew of some of the well known lner nferentl modelng technques, to enhnce the predctve blty of the regulrzed cnoncl correlton nlyss (RCCA) method, nd fnlly to compre the performnces of these technques nd hghlght some of the prctcl ssues tht cn ffect ther predctve bltes. he nferentl modelng technques consdered n ths study nclude full rnk modelng technques, such s ordnry lest squre (OLS) regresson nd rdge regresson (RR), nd ltent vrble regresson (LVR) technques, such s prncpl component regresson (PCR), prtl lest squres (PLS) regresson, nd regulrzed cnoncl correlton nlyss (RCCA). he theoretcl nlyss shows tht the lodng vectors used n LVR modelng cn be computed by solvng egenvlue problems. Also, for the RCCA method, we show tht by optmzng the regulrzton prmeter, n mprovement n predcton ccurcy cn be cheved over other modelng technques. o llustrte the performnces of ll nferentl modelng technques, comprtve nlyss ws performed through two smulted exmples, one usng synthetc dt nd the other usng smulted dstllton column dt. All technques re optmzed nd compred by computng the cross vldton men squre error usng unseen testng dt. he results of ths comprtve nlyss show tht sclng the dt helps mprove the performnces of ll modelng technques, nd tht the LVR technques outperform the full rnk ones. One reson for ths dvntge s tht the LVR technques mprove the condtonng of the model by dscrdng the ltent vrbles (or prncpl components) wth smll egenvlues, whch lso reduce the effect of the nose on the model predcton. he results lso show tht PCR nd PLS hve comprble performnces, nd tht RCCA cn provde n dvntge by optmzng ts regulrzton prmeter. Keywords: Inferentl Modelng; Ltent Vrble Regresson; Regulrzed Cnoncl Correlton Anlyss; Dstllton Columns 1. Introducton Models ply n mportnt role n vrous process opertons, such s process control, montorng, nd optmzton. In process control, where mesurng the controlled vrble (s) s dffcult, t s usully reled on nferentl models tht cn estmte the controlled vrble (s) from other more esly mesured vrbles. For exmple, the control of dstllton column compostons requres the vlblty of nferentl models tht cn ccurtely predct the compostons from other vrbles, such s temperture nd pressure t dfferent trys of the column. hese nferentl models re expected to provde ccurte predctons of the output vrbles over wde rnge of opertng condtons. However, constructng such nfer- * Correspondng uthor. entl models s usully ssocted wth mny chllenges, whch nclude ccountng for the presence of mesurement nose n the dt nd delng wth collnerty or redundncy mong the vrbles. Collnerty s common n nferentl models snce they usully nvolve lrge number of vrbles. he presence of collnerty ncreses the vrnce of the estmted model prmeters, nd thus degrdes the predcton ccurcy of the estmted models. Over-prmeterzed models cn ft the orgnl dt well, but they usully led to poor predctons [1]. One smple pproch for delng wth ths problem s to select subset of ndependent vrbles to be used n the model [2-4]. Other modelng technques tht del wth collnerty cn be dvded nto two mn ctegores: full rnk models nd reduced rnk models (or ltent vrble regresson models). Full rnk models utlze regulrzton to mprove Copyrght 2012 ScRes.

2 M. MADAKYARU E AL. 377 the condtonng of the nput covrnce mtrx [5-7], nd nclude Rdge Regresson (RR). RR reduces the vrtons n the model prmeters by mposng penlty on the L 2 norm of ther estmted vlues. RR lso hs Byesn nterpretton, where the estmted model prmeters re obtned by mxmzng posteror densty n whch the pror densty functon s zero men Gussn dstrbuton [8]. Ltent vrble regresson (LVR) models, on the other hnd, del wth collnerty by trnsformng the vrbles so tht most of the dt nformton s cptured n smller number of vrbles tht re used to construct the model. In other words, LVR models perform regresson on smll number of ltent vrbles tht re lner combntons of the orgnl vrbles. hs generlly results n well-condtoned models nd good predctons [9]. LVR model estmton technques nclude prncpl component regresson (PCR) [5,10], prtl lest squres (PLS) [5,11,12], nd regulrzed cnoncl correlton nlyss (RCCA) [13-16]. PCR s performed n two mn step: trnsform the nput vrbles usng prncpl component nlyss (PCA), nd then construct smple model reltng the nput to the trnsformed nputs (prncpl components) usng ordnry lest squres (OLS). hus, PCR completely gnores the output(s) when determnng the prncpl components. Prtl lest squres (PLS), on the other hnd, trnsforms the vrbles tkng the nput-output reltonshp nto ccount by mxmzng the covrnce between the trnsformed nputs nd outputs vrbles. herefore, PLS hs been wdely used n prctce, such s n the chemcl ndustry to estmte dstllton column compostons [1,17-19]. Other LVR model estmton methods nclude regulrzed cnoncl correlton nlyss (RCCA). RCCA s n extenson of nother estmton technque clled cnoncl correlton nlyss (CCA), whch determnes the trnsformed nput vrbles by mxmzng the correlton between the trnsformed nputs nd the output(s) [13,20]. hus, CCA lso tkes the nput-output reltonshp nto ccount when trnsformng the vrbles. CCA, however, requres computng the nverses of the nput covrnce mtrx. hus, n the cse of collnerty mong the vrbles, regulrzton of these mtrces s performed to enhnce the condtonng of the estmted model, whch s referred to s regulrzed CCA (RCCA). Snce the covrnce nd correlton of the trnsformed vrbles re relted, RCCA reduces to PLS under certn ssumptons. here re three mn objectves n ths pper. he frst objectve s to theoretclly revew the formultons nd the underlyng ssumptons of some of the nferentl model estmton technques, whch nclude OLS, RR, PCR, PLS, nd RCCA. hs theoretcl revew wll shed some lght on the smlrtes nd dfferences mong these modelng technques. he second objectve s to enhnce the predcton blty of LVR nferentl models by optmzng the regulrzton prmeter of the RCCA modelng method. he thrd objectve of ths pper s to compre the performnces of these technques through two smulted exmples, one usng synthetc dt nd the other usng smulted dstllton column dt. hs comprtve ntlyss lso provdes some nsght bout some of the prctcl ssues nvolved n constructng nferentl models. he remnder of ths pper s orgnzed s follows. In Secton 2, problem sttement s presented followed by theoretcl revew of the vrous nferentl model estmton technques n Secton 3. hs theoretcl dscusson ncludes full rnk models (such s OLS nd RR) nd ltent vrble regresson models (such s PCR, PLS, d RCCA). hs dscusson presented n extenson to optmze the RCCA to enhnce ts predcton blty. hen, n Secton 4, the vrous modelng technques re compred through two smulted exmples, one nvolvng synthetc dt nd the other nvolvng dstllton column dt. Fnlly, some concludng remrks re presented n Secton Problem Sttement hs work ddresses the problem of developng lner nferentl models tht cn be used to estmte or nfer key process vrbles tht re not esly mesured from other more esly mesured vrbles. All vrbles, nputs nd outputs, re ssumed to be contmnted wth ddtve zeros men Gussn nose. Also, t s ssumed tht there exsts strong collnerty mong the vrbles. hus, gven mesurements of the nput nd output dt, t s desred to construct lner model of the form, y Xb ε, (1) nm n1 where, X s the nput mtrx, y s the m1 output vector, b s the unknown model prmen1 ter vector, nd ε s the model error, respectvely. Severl estmton technques hve been developed to solve ths modelng problem; some of the full rnk models nd ltent vrble regresson models re descrbed n the followng secton. In ths pper, however, we seek to revew the formultons of these nferentl modelng methods, present n extenson of the RCCA method for enhnced predcton, nd provde some nsght bout the performnces of these technques long wth dscsson of some of the prctcl spects nvolved n nferentl modelng. 3. heoretcl Formultons of Lner Inferentl Models In ths secton, theoretcl perspectve of some of the Copyrght 2012 ScRes.

3 378 M. MADAKYARU E AL. full rnk nd ltent vrble regresson model estmton technques s presented. Full rnk modelng technques nclude ordnry lest squre (OLS) regresson nd rdge regresson (RR); whle ltent vrble regresson technques nclude prncpl component regresson (PCR), prtl lest squres (PLS), nd regulrzed cnoncl correlton nlyss (RCCA). he objectve behnd ths theoretcl presentton of the vrous nferentl modelng technques s to provde some nsght bout the smlrtes nd dfferences between these technques through ther formultons nd the ssumptons mde by ech technque Full Rnk Models Ordnry Lest Squres (OLS) Ordnry lest squre regresson s one of the most populr model estmton technques, n whch the model prmeters re estmted by mnmzng the L 2 norm of the resdul error or the sum of resdul squre error [5,10]. herefore, the model prmeter vector s estmted by solvng the followng optmzton problem: 2 b ˆ rg mn Xb y (2) b whch hs the followng closed form soluton for the prmeter vector ˆb : 1. 2, b ˆ X X X y (3) Note tht the OLS soluton (3) requres nvertng the mtrx X X. herefore, when X X s close to sngulrty (due to collnerty mong the nput vrbles), the vrnce of estmted prmeter vector ˆb ncreses, whch lso ncreses the uncertnty bout ts estmton. One wy to del wth ths collnerty problem s through regulrzton of the estmted prmeters s performed n rdge regresson (RR), whch s descrbed next Rdge Regresson (RR) o reduce the uncertnty bout the estmted model prmeters, RR not only mnmzes the L 2 norm of the model predcton error (s n OLS), but lso the L 2 norm of the estmted prmeters themselves [6]. hus, RR cn be formulted s follows: bˆ rg mn Xb y (4) b 2 b 2, 2 2 whch hs the followng closed form soluton: 1 b ˆ X X I X y, (5) mm where λ s postve constnt, nd I s the dentty mtrx. It cn be seen from Equton (5) tht ddng λi to the mtrx X X before nvertng mproves the condtonng of the estmton problem. he L 2 regulr- zton of the model prmeters n RR mkes t n effectve mens to cheve numercl stblty n fndng the soluton nd lso to mprove the predctve performnce of the estmted nferentl model Ltent Vrble Regresson (LVR) Models Delng wth the lrge number of hghly correlted mesured vrbles nvolved n nferentl models s one of the key ssues tht ffect ther estmton nd predctve bltes. It s known tht over-prmeterzed models cn ft the orgnl dt well, but they usully led to poor predctons. Multvrte sttstcl projecton methods such s PCR, PLS, nd RCCA cn be utlzed to del wth ths ssue by performng regresson on smller number of trnsformed vrbles, clled ltent vrbles (or prncpl components), whch re lner combntons of the orgnl vrbles. hs pproch, whch s clled ltent vrble regresson (LVR), generlly results n well-condtoned prmeter estmtes nd good model predctons [9]. In the subsequent secton, the problem formultons nd soluton technques for PCR, PLS, nd RCCA re presented. However, before we ntroduce these methods, let s ntroduce some defntons. Let the mtrx D be defned s the ugmented scled nput nd output dt,.e., D. Note tht sclng the dt s performed by mkng ech vrble (nput nd output) zero men wth unt vrnce. hen, the covrnce of D cn be defned s follows [16]: C E DD E X X EX y y X Ey y E (6) E C C CyX Cyy where the mtrces C, C, C yx nd C yy re of dmensons m m, m 1, m1, nd 1 1, respectvely. Snce the ltent vrble model wll be developed usng trnsformed vrbles, let s defne the trnsformed nputs s follows: z X, (7) where z s the ltent nput vrble 1,, m, th nd s the nput lodng vector, whch s of dmenson m 1. th Prncpl Component Regresson (PCR) PCR ccounts for collnerty n the nput vrbles by Copyrght 2012 ScRes.

4 M. MADAKYARU E AL. 379 reducng ther dmenson usng prncpl component nlyss (PCA), whch utlzes sngulr vlue decomposton (SVD) to compute the ltent vrbles or prncpl components. hen, t constructs smple lner model between the ltent vrbles nd the output usng ordnry lest squre (OLS) regresson [5,10]. herefore, PCR cn be formulted s two consecutve estmton problems. Frst, the lodng vectors re estmted by mxmzng the vrnce of the estmted prncpl components s follows: ˆ rg mx vr z 1,, m (8) st.. 1; z X whch, snce the dt re men centered, cn lso be expressed n terms of the nput covrnce mtrx C s follows: ˆ rg mx st.. 1. C 1,, m he soluton of the optmzton problem (9) cn be obtned usng the method of Lgrngn multpler, whch results n the followng egenvlue problem (see proof n Appendx A): C ˆ ˆ, (10) whch mens tht the estmted lodng vectors re the egenvectors of the mtrx C. Secondly, fter the prncpl components (PCs) re estmted, subset (or ll) of these PCs (whch correspond to the lrgest egenvlues) re used to construct smple lner model, tht reltes these PCs to the output, usng OLS. Let the subset of PCs used to construct the model be defned s Z z1 zp, where p m, then the model reltng these PCs to the output cn be estmted s follows: ˆ rg m n Zβ y whch hs the followng soluton, Z Z 1 ˆ. 2 2 (9) (11) β Z y (12) Note tht f ll the estmted prncpl components re used n constructng the nferentl model (.e., p m), then PCR reduces to OLS. Note lso tht ll prncpl components n PCR re estmted t the sme tme (usng Equton (10)) nd wthout tkng the model output nto ccount. Other methods tht consder the nput-output reltonshp nto consderton when estmtng the prncpl components nclude prtl lest squres (PLS) nd regulrzed cnoncl correlton nlyss (RCCA), whch re presented next Prtl Lest Squre (PLS) PLS computes the nput lodng vectors,, by mxmzng the covrnce between the estmted ltent vrble z ˆ nd model output, y,.e., [20,21]: ˆ rg mx cov z, y (13) st.. 1; z X where, 1,, p, p m. Snce z X nd the dt re men centered, equton (13) cn lso be expressed n terms of the covrnce mtrx C s follows: ˆ rg mx C (14) st.. 1. he soluton of the optmzton problem 14 cn be obtned usng the method of Lgrngn multpler, whch leds to the followng egenvlue problem (see proof n Appendx B): C C ˆ ˆ (15) 2 yx whch mens tht the estmted lodng vectors re the egenvectors of the mtrx CCyX. Note tht PLS utlzes n tertve lgorthm [20,22] to estmte the ltent vrbles used n the model, where one ltent vrble or prncpl component s dded tertvely to the model. After the ncluson of ltent vrble, the nput nd output resduls re computed nd the process s repeted usng the resdul dt untl cross vldton error crteron s mnmzed [5,10,22,23] Regulrzed Cnoncl Correlton Anlyss (RCCA) RCCA s n extenson of method clled cnoncl correlton nlyss (CCA), whch ws frst proposed n [13]. CCA reduces the dmenson of the model nput spce by explotng the correlton mong the nput nd output vrbles. he ssumpton behnd CCA s tht the nput nd output dt contn some jont nformton tht cn be represented by the correlton between these vrbles. hus, CCA computes the model lodng vectors by mxmzng the correlton between the estmted prncpl components nd the model output [13-16],.e., ˆ rg mx corr z, y (16) st.. z X where, 1,, p, p m. Snce the correlton between two vrbles s the covrnce dvded by the product of the vrnces of the ndvdul vrbles, Equton (16) cn be wrtten n terms of the covrnce between z nd y subject to the followng two ddtonl con- strnts: C ˆ ˆ 1 nd C yy 1. hus, the CCA formulton cn be expressed s follows, Copyrght 2012 ScRes.

5 380 M. MADAKYARU E AL. ˆ =rg mx cov z, y (17) 0, the RCCA soluton (Equton (21)) reduces to the CCA soluton (Equton (19)). On the other hnd, Note tht the constrnt C yy =1 s omtted from when 1, the RCCA soluton (Equton (21)) reduces Equton (17) becuse t s stsfed by sclng the dt to to the PLS soluton (Equton (15)) snce C yy s sclr. hve zero men nd unt vrnce s descrbed n Secton 3.2. Snce the dt re men centered, Equton (17) Optmzng the RCCA Regulrzton cn be wrtten n terms of the covrnce mtrx C s Prmeter follows: he bove dscusson shows tht dependng on the vlue ˆ rg mx cov z, y st.. z X ; C 1 2 yx 1. C C C ˆ ˆ (19) (18) he soluton of the optmzton problem (18) cn be obtned usng the method of Lgrngn multpler, whch leds to the followng egenvlue problem (see proof n Appendx C): whch mens tht the estmted lodng vector s the egenvector of the mtrx CCC yx 1. Equton (19) shows tht CCA requres nvertng the mtrx C to obtn the lodng vector,. In the cse of collnerty n the model nput spce, the mtrx C becomes nerly sngulr, whch results n poor estmton of the lodng vectors, nd thus poor model. herefore, regulrzed verson of CCA (clled RCCA) hs been developed n [20] to ccount for ths drwbck of CCA. he formulton of RCCA cn be expressed s follows: ˆ rg mx st.. 1 C C I C C ˆ ˆ C I yx (21) whch mens tht the estmted lodng vectors re the egenvectors of the mtrx 1 1 C I C CyX. (20) he soluton of Equton (20) cn be obtned usng the method of Lgrngn multpler, whch leds to the followng egenvlue problem (see proof n Appendx D): Note from Equton (21) tht RCCA dels wth possble collnerty n the model nput spce by nvertng weghted sum of the mtrx C nd the dentty mtrx,.e., 1 1 C I, nsted of nvertng the mtrx C tself. However, ths requres knowledge of the weghtng or regulrzeton prmeter. We know, however, tht when of, where 0 1, RCCA provdes soluton tht converges to CCA or PLS t the two end ponts, 0 or 1, respectvely. he uthors n [20] showed tht RCCA cn provde better results thn PLS for some ntermedte vlues of between 0 nd 1. hs observton motvted us to enhnce the predcton blty of RCCA even further by optmzng ts regulrzton prmeter. o do tht, n ths secton, we propose the followng nested optmzton problem to solve for the optmum vlue of : y yˆ y yˆ ˆ rg mn st.. yˆ RCCA model predcton. (22) he nner loop of the optmzton problem shown n Equton (22) solves for the RCCA model predcton gven the vlue of the regulrzton prmeter, nd the outer loop selects the vlue of tht provdes the lest cross vldton men squre error usng unseen testng dt. he dvntges of optmzng the regulrzton prmeter n RCCA wll be demonstrted through smulted exmples n Secton 4. Note tht RCCA solves for the ltent vrble regresson model n n tertve fshon smlr to PLS, where one ltent vrbles s estmted n ech terton [20]. hen, the contrbutons of the ltent vrble nd ts correspondng model predcton re subtrcted from the nput nd output dt, nd the process s repeted usng the resdul dt untl n optmum number of prncpl components or ltent vrbles re used ccordng to some cross vldton error crteron. More detls bout the selecton of optmum number of prncpl components re provded through the llustrtve exmples n the next secton, whch wll provde some nsght bout the reltve performnces of the vrous nferentl modelng methods nd some of the prctcl ssues ssocted wth mplementng these methods. 4. Illustrtve Exmples In ths secton, the performnces of the nferentl modelng technques descrbed n Secton 3 nd the dvntges of optmzng the regulrzton prmeters n RCCA re llustrted through two smulted exmples. In the frst exmple, models reltng ten nputs nd one output of synthetc dt re estmted nd compred usng the vrous model estmton technques. In the sec- Copyrght 2012 ScRes.

6 M. MADAKYARU E AL. 381 ond exmple, on the other hnd, nferentl models predctng dstllton column composton re estmted from mesurements of other vrbles, such s temperture, flow rtes, nd reflux. In both exmples, the estmted models re optmzed nd compred usng cross vldton, by mnmzng the output predcton men squre error (MSE) usng unseen testng dt s follow, n 1 MSE y k yˆ k (23) y k nd ŷ k n k 1 2 where re the mesured nd predcted outputs t tme step k, nd n s the totl number testng mesurements. Also, the number of retned ltent vrbles (or prncpl components) by the vrous LVR modelng technques (PCR, PLS, nd RCCA) s optmzed usng cross vldton. Fnlly, the dt (nputs nd output) re scled (by subtrctng the men nd dvdng by the stndrd devton) before constructng the models to enhnce ther predcton bltes. More detls bout the dvntges of dt sclng re presented n Secton Exmple 1: Inferentl Modelng of Synthetc Dt In ths exmple, the performnces of the vrous nferentl modelng technques re compred by modelng synthetc dt consstng of ten nput vrbles nd one output Dt Generton he dt re generted s follows. he frst two nput vrbles re block nd hevy-sne sgnls, nd the other nput vrbles re computed s lner combntons of the frst two nputs s follows: x3 x ; 1 x2 x4 0.3x1 0.7 x2; x5 0.3x3 0.2 x4; x6 2.2x11.7 x3; x7 2.1x6 1.2 x5; x8 1.4x2 1.2 x7; x9 1.3x2 2.1 x1; x10 1.3x6 2.3 x9; whch mens tht the nput mtrx X s of rnk 2. hen, the output s computed s weghed sum of ll nputs s follows: where, 10 y b x (24) 1 b 0.07, 0.03, 0.05, 0.04, 0.02, 1.1, 0.04, 0.02, 0.01, 0.03 for 1,,10. he totl number of generted dt smples s 128. All vrbles, nputs nd output, re ssumed to be nose-free, whch re then contmnted wth ddtve zero men Gussn nose. Dfferent levels of nose, whch correspond to sgnl-to-nose rtos (SNR) of 10, 20, nd 50, re used to llustrte the performnces of the vrous methods t vrous nose contrbutons. he SNR s defned s the vrnce of the nose-free dt dvded by the vrnce of the contmntng nose. A smple of the output dt, where SNR = 20 s shown n Fgure 1. Fgure 1. A smple output dt set used n the synthetc exmple for the cse where SNR = 20 (sold lne: nose-free dt; dots: nosy dt). Copyrght 2012 ScRes.

7 382 M. MADAKYARU E AL Smulton Results he smulted dt re splt nto two sets: trnng nd testng. he trnng dt re used to estmte nferentl models usng the vrous modelng methods, nd the testng dt re used to compute the model predcton MSE (s shown n Equton (24)) usng unseen dt. o mke sttstclly vld conclusons bout the performnces of the vrous modelng technques, Monte Crlo smulton of 1000 relztons s performed nd the results re shown n ble 1 nd Fgure 2. hese results show tht the performnce of RR s better thn tht of OLS, nd tht the performnces of the LVR modelng technques (PCR, PLS, nd RCCA) clerly outperform the performnces of the full rnk models (OLS nd RR). hs s, n prt, due to the fct tht n LVR modelng, porton of the nose n the nput vrbles s removed wth the neglected prncpl components, whch enhnces the model predcton. hs s not the cse n full rnk models (OLS nd RR) where ll nputs re used to predct the model output. he results lso show tht the performnces of PCR nd PLS re comprble. hese results gree wth those reported n the lterture [24,25], where the number of prncpl components s freely optmzed for ech model usng cross vldton nd the models predctons re compred usng unseen testng dt. he optmum numbers of prncpl components used by the vrous LVR models for the cse where SNR=20 re shown n Fgures 3(), (c) nd (e), whch show tht the optmum number of prncpl components used n PCR s usully more thn wht s used n PLS nd RCCA to cheve comprble predcton ccurcy. he results n ble 1 nd Fgure 2 lso show tht RCCA provdes slght dvntge over PCR nd PLS when the optmum vlue of the regulrzton prmeter s used. he vlue of s optmzed usng cross vldton s shown n the RCCA problem formulton gven n Equton (22). he optmzton of for one relzton s shown n Fgure 4, n whch s optmzed by mnmzng the cross vldton MSE of the estmted RCCA model wth respect to the testng dt. Note lso from Fgures 3(), (c) nd (e), whch compre the number of prncpl components used by the vrous ble 1. Comprson between the predcton MSE s obtned by the vrous modelng methods wth respect to the nose-free testng dt. Model ype SNR = 10 SNR = 20 SNR = 50 RCCA PLS PCR RR OLS Fgure 2. Hstogrms comprng the predcton MSE s for the vrous modelng technques nd t dfferent sgnl-tonose rtos. modelng methods, tht RCCA s cpble of provdng ths mprovement usng smller number of prncpl components thn PCR nd PLS Effect of Sclng the Dt on the Predctons nd Dmensons of Estmted Models As mentoned erler, scled nput nd output dt re used n ths exmple to estmte the vrous nferentl models. o llustrte the dvntges of sclng the dt (over usng the rw dt), the predcton nd the number of prncpl components used (n the cse of LVR models) re compred for the vrous model estmton technques. o do tht, Monte Crlo smulton of 1000 relztons s performed to conduct ths comprson, nd the results re shown n Fgures 3 nd 5. Fgure 5, whch compres the MSE for the vrous modelng technques usng scled nd rw dt, shows cler dvntge for dt sclng on the models predctve bltes. Copyrght 2012 ScRes.

8 M. MADAKYARU E AL. 383 () (b) (c) (d) (e) Fgure 3. Hstogrms comprng the optmum number of prncpl components used by the vrous modelng technques for scled nd rw dt for the cse where SNR = 20. (f) MSE Optml vlue Regulrzton prmeter n RCCA (τ ) Fgure 4. Optmzton of the RCCA regulrzton prmeter usng cross vldton wth respect to the testng dt. Fgure 3, on the other hnd, whch compres the effect of sclng on the optmum number of prncpl components (for PCR, PLS, nd RCCA), shows tht when scled dt re used, smller numbers of PCs re needed for ll model estmton technques, nd tht RCCA uses the lest number of PCs mong ll technques Exmple 2: Inferentl Modelng of Dstllton Column Compostons In ths exmple, the vrous modelng technques re compred when they re used to model the dstllte nd bottom strem compostons of dstllton column from other esly mesured vrbles. Copyrght 2012 ScRes.

9 384 M. MADAKYARU E AL Process Descrpton he column used n ths exmple, whch s smulted usng Aspen Plus, conssts of 32 theoretcl stges (ncludng the reboler nd totl condenser). he feed strem, whch s bnry mxture of propne nd sobutene, enters the column t stge 16 s sturted lqud. he feed strem hs flow rte of 1 kmol/s, temperture of 322 K, nd propne composton of 0.4. he nomnl stedy stte opertng condtons of the column re presented n the ble Dt Generton he dt used n ths modelng problem re generted by perturbng the flow rtes of the feed nd the reflux strems from ther nomnl opertng condtons. Frst, step chnges of mgntudes ±2% n the feed flow rte round ts nomnl condton re ntroduced, nd n ech cse, the process s llowed to settle to new stedy stte. After ttnng the nomnl condtons gn, smlr step chnges of ±2% n the reflux flow rte round ts nomnl condton re ntroduced. hese perturbtons re used to generte trnng nd testng dt (ech consstng of 64 dt ponts) to be used n developng the vrous models. hese perturbtons (for the trnng nd testng dt sets) re shown n Fgures 6(e)-(h). In ths smulted modelng problem, the nput vrbles consst of ten tempertures t dfferent trys of the column, n ddton to the flow rtes of the feed nd reflux strems. he output vrbles, on the other hnd, re the compostons of the lght component (propne) n the Fgure 5. Comprson between the predcton MSE s of the vrous modelng technques usng scled nd rw dt for the cse where SNR = 20. ble 2. Stedy stte opertng condtons of the dstllton column. Process Vrble Vlue Process Vrble Vlue Feed F 1 kg mole/sec P P 322 K x D P P Reboler Drum z F 0.4 B kg mole/sec Reflux Drum Q Wtts D kg mole/sec 366 K 325 K P P Reflux kg/sec x B 0.01 Copyrght 2012 ScRes.

10 M. MADAKYARU E AL. 385 () (b) (c) (d) (e) (f) (g) Fgure 6. Smple dt sets showng the chnges n the feed nd reflux flow rtes nd the resultng dynmc chnges n the dstllte nd bottom strem compostons; For the composton dt sold lne: nose-free dt, dots: nosy dt, SNR = 20. (h) dstllte nd bottom strems (.e., x D nd x B, respectvely). he dynmc temperture nd composton dt generted usng the Aspen smultor (due to the perturbtons n the feed nd reflux flow rtes) re ssumed to be nose-free, whch re then contmnted wth zeros men Gussn nose. o ssess the robustness of the vrous modelng technques to dfferent nose contrbutons, dfferent levels of nose (whch correspond to sgnl-to-nose rtos of 10, 20 nd 50) re used. Smple trnng nd testng dt sets showng the effect of the perturbtons on the column compostons re shown n Fgures 6()-(d) for the cse where the sgnl-to-nose rto s Smulton Results he smulted dstllton column dt (trnng dt nd testng) used n ths exmple re scled s dscussed n Exmple 1. he trnng dt set re used to estmte the model, whle the testng dt re used to optmze nd vldte the qulty of the estmted models. As performed n exmple 1, the number of prncpl components (n the cse of LVR technques,.e., PCR, PLS, nd RCCA) nd other prmeters (such s the regulrzton prmeters,.e., λ n RR or n RCAA) re determned by mnmzng the cross vldton MSE for the unseen testng dt. o obtn sttstclly vld conclusons bout the performnces of the vrous modelng technques, Monte Crlo smulton of 1000 relztons s performed, nd the results re presented n Fgure 7 nd ble 3. hese results show tht, n generl, the LVR modelng methods (PCR, PLS, nd RCCA) outperform the full rnk methods (OLS nd RR). he results lso show tht the performnces of PCR nd PLS re comprble, nd tht by optmzng ts regulrzton prmeter, RCCA cn provde n mprovement over these technques. he vlue of s optmzed usng cross vldton s shown n the RCCA problem formulton gven n Equton (22). he optmzton of for one relzton for the output x D s shown n Fgure 8. Fnlly, the results show tht the predcton bltes of ll modelng technques degrde for lrger nose contents,.e., for smller sgnl-to-nose rtos. he results obtned n ths dstllton column exmple gree wth the results obtned n Exmple Concluson Inferentl models re very commonly used n prctce to estmte vrbles whch re dffcult to mesure from other eser-to-mesure vrbles. hs pper presents theoretcl revew, n extenson to optmze RCCA for Copyrght 2012 ScRes.

11 386 M. MADAKYARU E AL. Fgure 7. Hstogrms chrt comprng the predcton MSE s of the vrous modelng technques usng the dstllton column dt. ble 3. Comprson between the predcton MSE s (wth respect to the nose-free testng dt) of the dstllte nd bottom strem compostons for the vrous modelng technques nd t dfferent sgnl-to-nose rtos. Model ype output- x D output- x B SNR = 10 ( 10 6 ) SNR = 20 ( 10 6 ) SNR = 50 ( 10 6 ) SNR = 10 ( 10 6 ) SNR = 20 ( 10 7 ) SNR = 50 ( 10 7 ) RCCA PLS PCR RR OLS enhnced predcton, s well s comprtve nlyss for vrous nferentl modelng technques, whch nclude ordnry lest squre (OLS) regresson, rdge regresson (RR), prncpl component regresson (PCR), prtl lest squre (PLS), nd regulrzed cnoncl correlton nlyss (RCCA). he theoretcl revew shows tht the lodng vectors used n LVR modelng cn be computed by solvng egenvlue problems. For RCCA, t s shown tht t cn be optmzed (to provde enhnced predcton blty) by optmzng ts regulrzton prmeter, whch cn be performed by solvng nested optmzton problem. he vrous nferentl modelng technques re compred through two exmples, one usng synthetc dt nd the other usng smulted dstllton column dt, where the dstllte nd bottom strem compostons re estmted usng other esly mesured vrbles. Both exmples show tht the ltent vrble regresson ( LVR) te chnques (.e., PCR, PLS, nd RCCA) outperform the full rnk technques (.e., OLS nd RR). hs s due to ther blty to mprove the condtonng of the model by neglectng prncpl components wth smll egenvlues, nd thus reducng the effect of nose on the model predcton. he obtned results lso show tht the performnces of PCR nd PLS re comprble when the number of prncpl components used re freely optmzed usng cross vldton. Fnlly, t s shown tht by optmzng ts regulrzton prmeter, RCCA cn provde n mprovement (n terms of ts predcton MSE) over PCR nd PLS usng smller number of prncpl components. 6. Acknowledgements hs work ws mde possble by NPRP grnt NPRP Copyrght 2012 ScRes.

12 M. MADAKYARU E AL. 387 MSE Optml vlue Regulrzton prmeter n RCCA (τ ) Fgure 8. Optmzton of the RCCA regulrzton prmeter usng cross vldton wth respect to the testng dt from the Qtr Ntonl Reserch Fund ( member of Qtr Foundton). he sttements mde heren re solely the responsblty of the uthors. REFERENCES [1] J. V. Krest,. E. Mrln nd J. F. McGregor, Devel- opment of Inferentl Process Models Usng PLS, Computers & Chemcl Engneerng, Vol. 18, No. 7, 1994, pp do: / (93)e0006-u [2] R. Weber nd C. B. Broslow, he Use of Secondry Mesurement to Improve Control, AIChE Journl, Vol. 18, No. 3, 1972, pp do: /c [3] B. Joseph nd C. B. Broslow, Inferentl Control Processes, AIChE Journl, Vol. 24, No. 3, 1978, pp do: /c [4] M. Morr nd G. Stephnopoulos, Optml Selecton of Secondry Mesurements wthn the Frmework of Stte Estmtonn the Presence of Persstent Unknown Dsturbnces, AIChE Journl, Vol. 26, No. 2, 1980, pp do: /c [5] I. Frnk nd J. Fredmn, A Sttstcl Vew of Some Chemometrc Regresson ools, echnometrcs, Vol. 35, No. 2, 1993, pp do: / [6] A. Hoerl nd R. Kennrd, Rdge Regresson Bsed Estmton for Nonorthogonl Problems, echnometrcs, Vol. 8, 1970, pp [7] J. McGregor,. Kourt nd J. Krest, Multvrte Identfcton: A Study of Severl Methods, IFAC ADCHEM Proceedngs, oulouse, Vol. 4, 1991, pp [8] M. N. Nounou, Delng wth Collnerty n Fr Models Usng Byesn Shrnkge, Industrl nd Engneerng Chemstry Reserch, Vol. 45, 2006, pp do: /e048897m [9] B. R. Kowlsk nd M. B. Sesholtz, Recent Developments n Multvrte Clbrton, Journl of Chemometrcs, Vol. 5, 1991, pp do: /cem [10] M. Stone nd R. J. Brooks, Contnuum Regresson: Cross-Vldted Sequentlly Constructed Predcton Embrcng Ordnrylest Squres, Prtl Lest Squres nd Prncpl Components Regresson, Journl of the Royl Sttstcl Socety B, Vol. 52, No. 2, 1990, pp [11] S. Wold, Soft Modelng: he Bsc Desgn nd Some Extensons, Systems under Indrect Observtons, Elsever, Amsterdm, [12] E. Mlthouse, A. mhne nd R. Mh, Non-Lner Prtl Lest Squres, Computers nd Chemcl Engneerng, Vol. 21, 1997, pp do: /s (96) [13] H. Hotellng, Reltons between wo Sets of Vrbles, Bometrk, Vol. 28, 1936, pp [14] F. R. Bch nd M. I. Jordn, Kernel Independent Component Anlyss, Journl of Mchne Lernng Reserch, Vol. 3, No. 1, 2002, pp [15] S. S. D. R. Hrdoon nd J. Shwetylor, Cnoncl Correlton Anlyss: An Overvew wth Applcton to Lernng Methods, Neurl Computton, Vol. 16, No. 12, 2004, pp do: / [16] M. Borg,. Lndelus nd H. Knutsson, A Unfed Approch to PCA, PLS, MLR nd CCA, echncl Report, echncl Report, Lnkopng Unversty, [17]. Mejdell nd S. Skogestd, Estmton of Dstllton Copyrght 2012 ScRes.

13 388 M. MADAKYARU E AL. Compostons from Multple emperture Mesurements Usng Prtl Lest Squres Regresson, Industrl & Engneerng Chemstry Reserch, Vol. 30, 1991, pp do: /e [18] M. kno, K. Myzk, S. Hsebe nd I. Hshmoto, Inferentl Control System of dstllton Compostons Usng Dynmcprtl Lest Squres Regresson, Journl of Process Control, Vol. 10, No. 2, 2000, pp do: /s (99)00027-x [19]. Mejdell nd S. Skogestd, Composton Estmtor n Plot-Plnt Dstllton Column, Industrl & Engneerng Chemstry Reserch, Vol. 30, 1991, pp do: /e [20] Y. Hroyuk, Y. B. Hdek, F. C. E. O. Hromu nd F. Hdek, Cnoncl Correlton Anlyss for Multvrte Regresson nd Its Applcton to Metbolc Fngerprnt- ng, Bochemcl Engneerng Journl, Vol. 40, No. 2, 2008, pp [21] R. Rospl nd N. Krmer, Overvew nd Recent Advnces n Prtl Lest Squres. Subspce, Ltent Structure nd Feture Selecton echnques, Lecture Notes n Computer Scence, Vol. 3940, 2006, pp do: / _2 [22] P. Geld nd B. R. Kowlsk, Prtl Lest Squre Regresson: A utorl, Anlytc Chmc Act, Vol. 185, No. 1, 1986, pp do: / (86) [23] S. Wold, Cross-Vldtory Estmton of the Number of Components n Fctor nd Prncpl Components Models, echnometrcs, Vol. 20, No. 4, 1978, p do: / [24] O. Yeny nd A. Gokts, A Comprson of Prtl Lest Squres Regresson wth Other Predcton Methods, Hcettepe Journl of Mthemtcs nd Sttstcs, Vol. 31, 2002, pp [25] P. D. Wentzell nd L. V. Montoto, Comprson of Prncpl Components Regresson nd Prtl Lest Squre Regresson through Generc Smultons of Complex Mxtures, Chemometrcs nd Intellgent Lbortory Systems, Vol. 65, 2003, pp do: /s (02) Appendx A. Determnng the Lodng Vectors Usng PCR Strtng wth the optmzton problem shown n Equton (9),.e., ˆ =rg mx C =1,, m (A.1) st.. 1, the Lgrngn functon for ths optmzton problem cn be wrtten s: L, C 1. (A.2) kng the prtl dervtve of L wth respect to nd equtng t to 0, we get, L 2C 2 0 (A.3 whch gves the followng egenvlue problem: C ˆ ˆ (A.4).e., the lodng vectors used n PCR re the egenvectors of the covrnce mtrx C. Appendx B. Determnng the Lodng Vectors Usng PLS Strtng wth the optmzton problem shown n Equton (14),.e., ˆ rg mx C st.. 1. (B.1) the Lgrngn functon cn be wrtten s follows: L, C 1. (B.2) kng the prtl dervtve of L nd equtng t to 0 we get, L C 2 0 whch gves the followng egenvlue problem, ˆ C wth respect to (B.3) (B.4) where, 2. Multplyng Equton (B.4) by ˆ nd enforcng the constrnt ( ˆ ˆ 1), we get, ˆ ˆ C ˆ (B.5) kng the trnspose of Equton (B.5), we get, C ˆ. (B.6) yx Combng Equtons (B.4) nd (B.6), we get the followng egenvlue problem: yx ˆ 2 ˆ. (B.7) C C Appendx C. Determnng the Lodng Vectors Usng CCA Strtng wth the optmzton problem shown n Equton (18),.e., ˆ rg mx C st.. C 1 (C.1) Copyrght 2012 ScRes.

14 M. MADAKYARU E AL. 389 the Lgrngn functon cn be wrtten s: L C 1 C. (C.2), kng the prtl dervtve of L wth respect to nd equtng t to 0 we get, L C 2 C whch gves the followng soluton, 1 ˆ 0, (C.3) C C (C.4) where 2. Multplyng Equton (C.4) by C ˆ nd enforcng the constrnt (.e., ˆ ˆ 1 ), we ge t, C ˆ C ˆ C ˆ. (C.5) kng the trnspose of Equton (C.5), we get, C ˆ. (C.6) yx Combng Equtons (C.4) nd (C. 6), we get the followng egenvlue problem: 1 2 C C C ˆ. (C.7) yx ˆ Appendx D. Determnng the Lodng Vectors Usng RCCA Strtng wth the optmzton problem shown n Equton (20),.e., ˆ rg mx C st.. 1 C I 1 (D.1) the Lgrngn multpler functon cn be wrtten s follows: L, C 1 1 C I. (D.2) kng the prtl dervtve of L wth respect to nd equtng t to 0, we get, L C 2 1C I 0, (D.3) whch gves the followng soluton: 1 C ˆ I C (D.4) 1 where 2. Multplyng Equton (D.4) by (.e., nd enforcng the constrnt ˆ 1 C I ˆ 1 ˆ C I 1), we get: C ˆ. (D.5) kng the trnspose of Equton (D.5), we get,. (D.6) CyX ˆ Com bnng Equtons (D.4) nd (D.6), we get the followng egenvlue problem: 1 2 ˆ ˆ C I CCyX 1. (D.7) Copyrght 2012 ScRes.