DIMENSION REDUCTION FOR POWER SYSTEM MODELING USING PCA METHODS CONSIDERING INCOMPLETE DATA READINGS

Transcription

1 Michiga Techological Uiversity Digital Michiga Tech Dissertatios, Master's Theses ad Master's Reports - Ope Dissertatios, Master's Theses ad Master's Reports 3 DIMENSION REDUCTION FOR POWER SYSTEM MODELING USING PCA METHODS CONSIDERING INCOMPLETE DATA READINGS Tig Zhao Michiga Techological Uiversity Copyright 3 Tig Zhao Recommeded Citatio Zhao, Tig, "DIMENSION REDUCTION FOR POWER SYSTEM MODELING USING PCA METHODS CONSIDERING INCOMPLETE DATA READINGS", Master's report, Michiga Techological Uiversity, 3. Follow this ad additioal wors at: Part of the Mathematics Commos

2 DIMENSION REDUCTION FOR POWER SYSTEM MODELING USING PCA METHODS CONSIDERING INCOMPLETE DATA READINGS By Tig Zhao A REPORT Submitted i partial fulfillmet of the requiremets for the degree of MASTER OF SCIENCE I Mathematical Scieces MICHIGAN TECHNOLOGICAL UNIVERSITY 3 3 Tig Zhao

3 This report has bee approved i partial fulfillmet of the requiremets for the Degree of MASTER OF SCIENCE i Mathematical Scieces. Departmet of Mathematical Scieces Report Advisor: Jiapig Dog Committee Member: Thomas D. Drummer Committee Member: Chee-Wooi Te Departmet Chair: Mar Gocebach

4 . Abstract Pricipal Compoet Aalysis (PCA) is a popular method for dimesio reductio that ca be used i may fields icludig data compressio, image processig, exploratory data aalysis, etc. However, traditioal PCA method has several drawbacs, sice the traditioal PCA method is ot efficiet for dealig with high dimesioal data ad caot be effectively applied to compute accurate eough pricipal compoets whe hadlig relatively large portio of missig data. I this report, we propose to use EM-PCA method for dimesio reductio of power system measuremet with missig data, ad provide a comparative study of traditioal PCA ad EM-PCA methods. Our extesive experimetal results show that EM-PCA method is more effective ad more accurate for dimesio reductio of power system measuremet data tha traditioal PCA method whe dealig with large portio of missig data set.. Itroductio The most importat property of PCA is that it ca achieve the optimal results i terms of mea squared error (MSE) via performig liear trasformatio of high dimesioal vectors. As a result, a ew set of much lower dimesioal vectors ca be obtaied, while the origial data set ca be recostructed approximately. The objective of this project is to reduce the cost i buildig respose surface performace models for power system cotrol ad optimizatio. The dimesio reductio techiques of parameter space have bee applied by itegrated circuit modelig researchers i the past decade, where PCA plays a importat role [7]. It has bee show that high-dimesioal circuit parameters will first be reduced by dimesio reductio techiques, such as PCA method. The the top few pricipal compoets (liear combiatios of the origial parameter set) will be used for buildig the quadratic (secod order) respose surface models for circuit performaces. I [7],

5 researchers are worig o dimesio reductio methods for itegrated circuit applicatios, whereas i our project dimesio reductio for power distributio system modelig is cocered. However, traditioal PCA method has several drawbacs. Oe of them is that the traditioal PCA method is ot suitable for dealig with high dimesioal data, sice computig the sample covariace required by PCA method ca be very costly. Cosequetly, it is desirable to avoid computig sample covariace for high dimesioal data set. Aother drawbac is that it is ot obvious how to compute the pricipal compoets properly with missig data. To address these drawbacs of traditioal PCA method, a variety of the alterative PCA-lie dimesio reductio methods have bee proposed i may research fields. The goal of such research is to improve classical PCA method such that the computatioal efficiecy ca be improved ad missig data i the high dimesioal data set ca be also hadled. I the followig sectios, we will itroduce alterative PCA-based dimesio reductio methods ad also demostrate experimetal results o realistic high dimesioal power system measuremet data set. 3. Classic Pricipal Compoet Aalysis We will start with a simple example for itroducig traditioal PCA method. Cosider usig a straight lie to best represet scattered poits i two-dimesios (with X-X coordiates), which is show i the followig figure. The ey is to fid a ew coordiate (X ), such that data variace observed alog X is maximized. Similarly, for m-dimesioal data set with m orthogoal coordiates, it is usually very importat to fid a much fewer ew coordiates such that the maximum variace of the origial data set ca be well preserved. The classical PCA method ca be applied to solve the above problems by usig Least Squared Error miimizatio method.

6 Figure. The geeral derivatio of PCA is i terms of a stadardized liear projectio that maximizes the variace i the projected space [3]. For observed m-dimesioal vectors ( samples) X= [ xx... x] T, PCA ca fid the top-oe pricipal compoet (axe) that maximize the data variace preserved from the origial data set, which ca be achieved by fidig a m-dimesio vector ( x ) such that ( ) = () = J x x x is the smallest, where J( x ) is Mea Squared Error criterio fuctio. Let So we have: [4] x = m = () t ( ) = ( ) ( ) = ( ) ( ) + = = = = J x x m x m x m x m x m x m x m x m (3) = = = + where x m is idepedet of = x. So if x = m, J( x ) will be the smallest. Suppose that we wat to fid a coordiate X' ad the goal is still to miimize the squared-error. Let every x has a correspodig elemet ' x i the X'. If we move it i the vector e directio, the [4] 3

7 x = m+ ae (4) ' where a is a scalar. Now we redefie the squared-error criterio fuctio as follows: ' = = + = = J( a,..., a, e) ( x x) (( m ae) x) = ( ae ( x m)) = t a e ae ( x m) x m = = = (5) = + where e is uit vector ad e =, hece ' t = = (6) J( a,..., a, e) = a ae( x m) Solvig partial differetial for a, we ca get t a = e ( x m) (7) It meas if we have already ow vector e, ay vector x projected to the lie X ca be computed by taig the ier product with e t. The we ca obtai the ew ' coordiate after liear trasformatio ( x = a ). Hece, we ca rewrite J () e, such that [4] t = = = = = J ( e) = a e a + x m = [ e ( x m)] + x m t t t e ( x m)( x m) e x m e S e x m = = = (8) = + = + where t t S = e ( x m)( x m) e, is called a scatter matrix. Usig method of = Lagrage Multipliers, we will obtai vector e so that t e Se is maximized. Let t t u = e Se λ( e e ) (9) ad obtai the partial differetial for vector e, () 4

8 Let it equal, so Se = λe () This is a classic problem that is related to eigedecompositio ad the vector e is exactly oe of the eigevectors. Suppose we eed to reduce m-dimesioal vectors to -dimesioal vectors, we will select the top eigevectors with their correspodig eigevalues ad project every x i oto them, the we ca get x = [ x x... x ], such that [4] ' ' ' ' i i i i x = e ( x m) () ' T ij j i where m is the data sample mea. Therefore, the classic PCA algorithm ca reduce dimesio successfully ad mae data aalysis ad system modelig easier. 4. EM-PCA Algorithm I this sectio, we will itroduce the expectatio maximizatio (EM) algorithm for pricipal compoet aalysis of data set [][5][8]. The EM-PCA algorithm ca hadle high dimesioal data more efficietly tha traditioal PCA method sice it does't eed to calculate the sample covariace matrix explicitly. 4. Probabilistic Model of PCA Pricipal compoet aalysis ca be used as a limitatio case of liear Gaussia models. Liear Gaussia model is to assume y as a liear trasformatio of some dimesioal latet variable x plus additive Gaussia oise. Let G be the m* matrix, ad v is the m -dimesioal error vector (with covariace matrix R), so the geeral model ca be writte as y = Gx + v (3) where x ~ N(, I ), the error v ~ N(, R ). So the y is a correspodig 5

9 Gaussia-distributio vector for the observatios T y ~ N(, GG + R) (4) We ow the probabilistic model is the followig accordig to [5]: T px ( ) = ( ) exp( xx) (5) m/ π For the case of error R = σ, the probability distributio over y space for a give I x is m/ p( y x) = ( πσ ) exp( y Gx ) (6) σ So, the distributio of y is [5] m/ / T p( y) p( y x) p( x) d x ( π ) = = W exp[( ) y W y] (7) where W is a m mmatrix ad W I GG T = σ +. pxpy ( ) ( x) Usig Bayes' rule, px ( y) =, the latet variables x give the observed p( y) y ca be calculated as follows: [5] m/ / T T p( x y) = ( π) σ M exp[ ( x M G m) σ M( x M G y)] (8) where M is a matrix ad T M = ( σ I+ GG). Therefore, accordig to [5], the log fuctio of y is N L = p y = m + W + tr W S (9) N l[ ( )] [ l( π ) l ( )] = where S is the sample covariace matrix of the observatios { y }, N T yy N = S=. Estimates for G ad σ ca be obtaied by iterative maximizatio of L usig the EM algorithm. 4. The EM-PCA Algorithm 6

10 We use the EM algorithm to obtai the parameter G ad σ i the probabilistic model. We will use the latet variables { x } to be the missig data ad cosider the complete data for composig the observatios together with these latet variables. The complete-data log-lielihood ca be derived as follows [5]: N L = l{ p( y, x )} () C = Usig formula (5) (6),we ca obtai [5] m/ y Gx / (, ) ( ) exp( T p y x = πσ )( π ) exp{ x } x () σ I the E-step, we tae the expectatio of L C with respect to the distributio (,, ) px y Gσ : [5] () where < >=, T x M G y T T < xx >= σ M + < x >< x >, T M σ I GG = +. I the M-Step, < > is maximized with respect to G ad LC σ givig the ew parameter estimates [5] σ T T ew = ( < > )( < > ) (3) G y x xx N T T T T ew = y < x > Gew y + tr < xx > GewGew Nm = [ ( )] (4) These equatios are iterated i sequece util the algorithm is cosidered to have coverged [5]. 5. PCA with Missig Data I the above sectios, we itroduced a traditioal PCA algorithm ad described a existig EM algorithm for calculatig pricipal compoets. I the ext sectio, we will discuss PCA methods for data sets with missig poits. I geeral, the 7

11 real-world data is ot always complete. If the umber of missig data poits is very small compared to the full data set, we ca directly use the sample mea to substitute the missig data poits. O the other had, if the umber of missig data poits is relatively large, the followig EM formulatio of PCA is applied for hadlig missig values [6]. 5. EM-PCA with Missig Data We cosider the same problem whe the data matrix Y has missig values ad suppose that values are missig radomly. For example, Y matrix ca be writte i the followig form: Figure y y y3 Y = y y y4 y 5 y3 y34 y35 where every idicates the missig value. For such id of data set where relatively large portio of data is missig, the EM-PCA with missig data ca be used [6]. Now, our goal is to fid the parameters G ad σ that maximize the lielihood of observed data (vectors y is partially observed). We use EM algorithm that estimates i the E-step the missig values (the vector x ad the missig parts of the y which is deoted by y h ). I the M-step, we fix these estimates, ad maximize the expected joit log-lielihood of x ad y. We assume that the distributio over x ad yh factors so that we get the lower-boud log-lielihood as follows [6]: log p( y) = log M [ q( x) + q( yh)] + Eq[log p( x) + log p( y x)] (5) We will maximize this equatio with respect to the distributio of q i the E-step, ad respect to the parameters i the M-step. E-step: From the above we fid the optimal distributios q as [6] 8

12 qy qx y x N y Gx I (6) ( h) exp ( )log( h ) ~ ( h; h, σ ) q x p x y q y y x N x MG y M (7) T ( ) ( )exp ( h)log( h ) ~ ( ; σ, ) where y is the mea of qy ( h) for the missig data, y for the observed part, ad x is the mea of qx. ( ) M-step: Based o equatio (8), it follows as [6] N = ND T E [log p( x ) + log p( y x )] = log σ ( y G x tr( GMG )) q σ Dh σ old σ N x tr( M ) (8) where D h is the umber of missig data poits, σ old is the curret value for σ that was used i the E-step to compute the q. Maximizig the equatio (3) over G adσ, we ca obtai [6] G XY NM XX T T ew = ( + ) (9) σ = [ Ntr ( GMG ) + y Gx + D σ ] (3) T h old Nm where X,Y are the matrices that collect all x, y as colums, separately. 6. Experimetal Results As metioed at the begiig of this project report, our goal is to perform dimesio reductios o the measured data set, ad subsequetly build distributio performace models that will be utilized by MTU's power distributio cotrol ceter to predict future electricity usages i December such that resources ca be optimized accordigly. The parameter dimesio reductio methods demostrated i this wor are implemeted i Matlab programmig laguage, ad will sigificatly reduce the performace model characterizatio cost, thereby allowig for real time processig of huge amout of data obtaied by the-state-of-art smart meters. 9

13 We demostrate the results obtaied by traditioal PCA ad EM-PCA methods for aalyzig power distributio data sets i EERC buildig of MTU durig last December. Three smart meters have collected the data sets, ad each of them records a measuremet every te miutes. It should be oted that the meter readigs are just iput parameters for buildig the respose surface models that will predict activities happeed iside a buildig, though the detailed model extractio procedures have ot started yet. We use the meter readig of each hour as a variable, sice the evets durig two cosecutive hours are typically ot strogly correlated. For istace, typical udergraduate courses will ot last loger tha 5 miutes. So we cosider the sum of meter readigs i every hour as a variable, so we have totally 4*3=7 variables. Cosequetly, for 3 days i December, we have a data set with 7*3=3 measuremets. We also wat to emphasize that the meter readigs are ot always idepedet. Sice some facilities i the EERC buildig are usig more tha oe power supply sources, multiple meter readigs will be chagig if such facilities are started. For istace, air heater will be started with the fa for air circulatio at the same time. If they are usig two separate power sources, the readigs of two meters will be simultaeously affected. Table :Example of the origial data set Timestamp Tred Flags Status Value ::8 AM // EST 35.3 :: AM // EST 45.3 :: AM // EST 47 :3:8 AM // EST 34.3 :4: AM // EST 33.3 :5: AM // EST 35.4 ::7 AM // EST 33.8 :: AM // EST 37. :: AM // EST 44. :3:7 AM // EST 3 :4: AM // EST 33.3 :5: AM // EST 33.5 ::4 AM // EST 3.4 :: AM // EST 45.5 :: AM // EST 33

14 We also wat to ote that this is ot a traditioal time series problem, sice the meter measuremets of oe time poit (hour) is ot liely to be iflueced by the previous may time poits (hours) measuremets. The missig data poits aalyzed i our problem are typically due to the istability of smart meter devices maufactured by differet vedors. Although the meters i the EERC buildig do ot show obvious istabilities, the meters operatig i extreme coditios (extremely high temperature eviromet, extreme humidity coditios, etc) ca produce icorrect/missig results. Sice this research project is targetig geeral power systems modelig that ca ivolve smart meters operatig uder ay eviromets, differet missig data rates (% to 5%) are cosidered.. Results obtaied by usig traditioal PCA ad EM-PCA with full data set have bee demostrated as follows. Figure 3.Sigular values after performig traditioal PCA with full data set.5 x 6 Sigular values Parameters Table. Top 7 Sigular values Sigular value

15 It is observed that eepig oly top three priciple compoets will be sufficiet sice they ca already explai more tha 99% variability of the origial data set. Table 3. Top 3 Priciple Compoet Mappig Vectors (PCMVs) PCMV PCMV PCMV

16

17

18 Comparig pricipal compoet mappig vectors (PCMVs) obtaied by traditioal PCA ad EM-PCA with full data set, we obtaied the followig results. If the scatter plot shows that all dots fall o the lie y=x, the elemets iside the two vectors are perfectly matchig with each other. Figure 4. PCMV.5. EM-PCA Traditioal PCA Figure 5. PCMV.4. EM-PCA Traditioal PCA. Results of the pricipal compoet mappig vectors obtaied by usig traditioal PCA (that replaces missig data by mea values) ad EM-PCA with partial data have bee demostrated as follows. We cosider various missig data rates that rage from % to 5%. () Whe missig data rate is %, results are show below. a) Error of fidig the PCMV Figure 6 Traditioal PCA: Rage (.5%-.5%) EM-PCA: Rage (.%-.6%) 5

19 Traditioal PCA ---- EM-PCA (b) Error of fidig the PCMV: Figure 7 Traditioal PCA: Rage (4%-8%) EM-PCA: Rage (.5%-.5%) Traditioal PCA ---- EM-PCA (c) Error of fidig the PCMV3: 6

20 Figure 8 Traditioal PCA: Rage (5%-%) EM-PCA: Rage (%-5%) Traditioal PCA ---- EM-PCA () Whe missig data rate is %, results are show below. a) Error of fidig the PCMV: Figure 9 Traditioal PCA: Rage (%-4%) EM-PCA: Rage (.%-.5%) Traditioal PCA ---- EM-PCA

21 b) Error of fidig the PCMV: Figure Traditioal PCA: Rage (%-5%) EM-PCA: Rage (%-4%) Traditioal PCA ---- EM-PCA c) Error of fidig the PCMV3: Figure Traditioal PCA: Rage (5%-5%) EM-PCA: Rage (%-4%) Traditioal PCA ---- EM-PCA

22 (3) Whe missig data rate is 5%, results are show below. a) Error of fidig the PCMV: Figure Traditioal PCA: Rage (5%-%) EM-PCA: Rage (%-.5%) Traditioal PCA ---- EM-PCA b) Error of fidig the PCMV: Figure 3 Traditioal PCA: Rage (5%-3%) EM-PCA: Rage (.5%-%) Traditioal PCA ---- EM-PCA 9

23 c) Error of fidig the PCMV3: Figure 4 Traditioal PCA: Rage (5%-4%) EM-PCA: Rage (5%-%) Traditioal PCA ---- EM-PCA Sice ier products of the three priciple compoet mappig vectors obtaied by usig the traditioal PCA with full data set ad the EM-PCA with partial data (the traditioal PCA with partial data set) ca idicate the quality of the computed priciples, we demostrate the followig ier products results. Table 4. PCMV % missig data % missig data 5% missig data Traditioal PCA EM-PCA Table 5. PCMV % missig data % missig data 5% missig data Traditioal PCA EM-PCA

24 Table 6. PCMV 3 % missig data % missig data 5% missig data Traditioal PCA EM-PCA Ier product of two uit-legth vectors equals to cosie value of their agels. v w ( v w = v w cosθ cosθ =, where θ is the agel of vector v ad v w w). Obviously, the EM-PCA method is cosistetly much better tha traditio PCA method i hadlig missig data, sice the ier product values obtaied by EM-PCA are closer to, idicatig a almost perfectly matchig result with the true priciple compoets obtaied by traditioal PCA with full data set. 7. Coclusio I this wor, we have preseted the fudametals of traditioal PCA method ad EM-PCA method, ad their applicatios i hadlig missig data. By aalyzig power distributio data sets i EERC buildig of MTU durig last December, we have performed dimesio reductios o the measured data set for future research o power distributio system modelig by usig the traditioal PCA ad EM-PCA methods. I the last, we coclude that the EM-PCA method is cosistetly more effective ad accurate i fidig pricipal compoets tha traditio PCA method whe missig data set has to be cosidered.

25 Refereces [] T. Wiberg. Computatio of pricipal compoets whe data is missig. I Proc. Secod Symp. Computatioal Statistics, pages 9-36, 976 [] Sam Roweis. EM algorithm for PCA ad SPCA. Neural Iformatio Processig Systems,997 [3] H. Hotellig. Aalysis of a complex of statistical variables ito pricipal compoets. Joural of Educatioal Psychology,993. [4] Richard O.duda, Peter E. hart, David G.Stor. Patter Classificatio [5] Tippig M E, Bishop C M. Probabilistic priciple compoet aalysis. Joural of the Royal Statistical Society (3):6-6 [6] Jaob Verbee, Note o Probabilistic PCA with Missig Values. [7] Xi Li, Jiayog Le, Lawrece T. Pileggi: Statistical Performace Modelig ad Optimizatio. Foudatios ad Treds i Electroic Desig Automatio (4) (6). [8] Tippig M E, Bishop C M. Mixture of Probabilistic Priciple Compoet Aalyses. Neural Computatio, 999, ():