Out of sample extensions of PCA, kernel PCA, and MDS

Transcription

1 Out of sample extesos of PCA, erel PCA, ad MDS Math 85 Project, Fall 05 Wlso A. Florero-Salas Da L

2 ABLE OF CONENS. Itroducto.... Prcpal Compoet Aalyss (PCA).... he out of sample exteso of PCA he out of sample exteso of PCA DEMO Kerel PCA he out of sample exteso of Kerel PCA he out of sample exteso of KPCA (demo) Multdmesoal Scalg (MDS) he out of sample exteso of MDS he out of sample exteso of MDS (DEMO) he out of sample exteso of MDS (DEMO) Cocluso Appedx Refereces... 5 Page of 5

3 . INRODUCION Classfcato s the problem of categorzg a ew observato based o a set of observatos called the trag set, whose membershp s already ow. Classfcato problems usually volve the trag of a model, usg the trag set, to later mae predctos or classfy ew observatos to oe of the ow categores. I recet years the collecto of huge amouts of data has bee eased by mprovemets techology, ad t s ow commo to have observatos wth thousads, f ot mllos of features. Eve though there has bee a great jump techology, may moder computers are stll ot able to effcetly hadle observatos wth very large umber of features, whch some cases maes model trag ufeasble. However, may cases, t s stll possble to tra a model wth a subset of the features, or a trasformato of the feature space to a smaller space whch feature selecto s possble. hs leads us to the dea of Dmesoalty Reducto. Dmesoalty reducto (DR) s the process of reducg the umber of varables uder cosderato for the purpose of feature selecto or feature extracto. o ths ed, dmesoalty reducto allows the modeler to tra models usg less varables, ad some cases, obta a vsualzato of the data set two or three dmesos. hree commo DR techques ow the lterature are Prcpal Compoet Aalyss (PCA), Kerel PCA, ad Classcal Multdmesoal Scalg (MDS). o perform the correspodg trasformato each of these methods use the etre data set. he questo s ow: If ew data becomes avalable, how ca these ew observatos be corporated the ew feature space? I some cases redog the DR s eough, but that s ot our preset cocer. I other cases the data set may so large that retrag s o loger feasble. I ths cotext, we eed a way to corporate these ew observatos the ew feature space, wthout retrag ad f possble, recyclg formato already obtaed from the frst tme we performed dmesoalty reducto. hs dea of brgg ew observatos to the ew feature space s ow the lterature as out-of-sample-extesos, whch wll be the focus of ths paper. I the followg we brefly revew PCA, erel PCA, ad MDS before cosderg ther correspodg out-ofsample extesos.. PRINCIPAL COMPONEN ANALYSIS (PCA) Prcpal Compoet Aalyss (PCA) s a lear DR feature extracto tool. PCA attempts to fd a lear subspace of lower dmeso tha the orgal feature space, where the ew features have the largest varace [B006]. Oe way to derve the prcpal d compoets of a data set { x } s by maxmzg the trace of the covarace matrx of data pots { y }, gve by S : ( y y)( y y) Y, where y y ad where we assume that there s a V such that y. A equvalet way to derve the prcpal compoets s by solvg the problem of fdg the best -dmesoal subspaces ( le ) V x d that mmze the orthogoal dstaces. We tae ths approach here. Cocretely, let { x }, where,,, to solve [S003] where PS () s the projecto oto the subspace S. Let S m x PS ( x ) S d m represet a fxed pot ad B. he tas s d a orthoormal bass of. If x m B s a parametrc equato for the plae, the P ( x ) m BB ( x m). It ca be show that the above mmzato problem s equvalet to solvg S he reader s referred to the refereces for addtoal detals. Page of 5

4 m X XBB B F hs mmum s acheved whe Frobeus orm, ad X B Vd X ad XBB m x. Here X, where the colums correspod to the frst. he above the ca be summarzed the followg theorem U V heorem: he projecto of X oto the best-ft -plae s X U V s the best ra- approxmato to X colums of V uder the the SVD of X. I other words he ew coordates wth respect to the bass Vd XV.e., the rows of d U are called the prcpal compoets. hs theorem allows us to easly fd the frst prcpal compoets of X usg the followg algorthm Algorthm a: Prcpal Compoet Aalyss (PCA) X [ x, x,, x ] Iput: Data set Ouput: op prcpal compoets. Ceter the data:. Perform SVD o: x x x for all X U V XV U 3. Retur the rows of: d. HE OU OF SAMPLE EXENSION OF PCA I the prevous secto, we obtaed the matrx Vd whch mapped pots d x to pots pots are cetered to beg wth, the P ( x ) V V x or matrx form P ( X ) XV V subspace wth dmeso X x x x [,,, ], we smply cosder S d d XV S, where the subspace S d S d d y va a lear map. If the. o obta the pots the was costructed usg the data set. If a ew data set becomes avalable, how ca PCA be exteded to ths ew data set? We ow llustrate ths exteso wth fgure, Secto.3. Assume we have a ew data set. Accordg to the Algorthm a, we ceter the data. he usg ths cetered data we costruct the le S ad map these pots to the le. If a ew data set avalable, the we ca map t to the le usg the matrx Vd Z m becomes. However, because the orgal data set has bee cetered, the data pots ad the le (subspace) are a ew set of axs. o brg the ew data set Z to the curret set of axs, t must be cetered exactly the same way X was cetered. Fally, we may project the cetered data set Z summarzed Algorthm b ad a vsualzato Secto.3: va the matrx V d [G966]. hs s I ths paper, gve a matrx AM N, we defe AM to be the matrx oly cludg the frst colums. We wll also ths otato to deote ts dmesos. Page 3 of 5

5 Algorthm b: Out of sample (PCA) Iput: New data set Z [ z, z,, z ] m, Vd Ouput: op prcpal compoets. Ceter the data:. Retur the rows of: z z x for all ZVd Both Algorthms a ad b wrtte as a fucto MALAB ca be foud Appedx HE OU OF SAMPLE EXENSION OF PCA DEMO Fgure : (L) Orgal data set (M) cetered data alog wth best-ft le (R) curret projecto space ad ew ucetered data (red pots). Fgure : (L) New data pots brought to curret axs by ceterg (R) ew pots projected oto the curret space. It s worth comparg the out-of-ft exteso wth a retraed model that uses the etre data set. Notce the dfferece. Page 4 of 5

6 3. KERNEL PCA PCA s a lear method that caot properly hadle olear data. If the data s olear, the ma dea of Kerel PCA s to use a map () that taes each data vector appled [W0]. Cocretely, let data pots that x to a vector x ( x ) d a hgher dmesoal space (called feature space) where PCA ca be be gve, ad suppose : d D, where D d. Assume further ( x ) 0, meag that the feature vectors have zero mea. Defe : ( ) ( ) ( ) D x x x, ad cosder the SVD VD U V. he applyg PCA to va Algorthm the ew coordates wth respect to the bass D DD V U. Usually the are gve by the rows of D decomposto explctly. o remedate ths defe ( x, x ) : ( x ) ( x ) ad cosder ( x ) j j are uow ad t s ot possble to wor out the ( x ) ( x ) ( x, x j ) : K. he matrx K s called the Kerel matrx, whch uder the proper mappg whch s postve sem-defte. If the data s ot cetered the feature space, t ca be show that by cosderg ( x ) ( x ) ( x ), costructg we ca obta smlar equatos by replacg K D by K aalyss, we have we obta the matrx 3 () K K K K K, for whch prevous formulas or formulas that follow 4 [S998]. Proceedg wth our K U V U V U U so that KU D DD D DD D D U, whch s a egevalue problem. I other words, by solvg for the egevalues ad egevectors of K, we are able to obta the matrces eeded the SVD of. Note that f we cosder VD D DDVD D we obta VD VD D. Ad for the purpose of prcpal compoet extracto, we choose vectors so that vv, whch leads to uu / ( K) ( ), for,, the u by /. hs s equvalet to cosderg K U so that u. KPCA s summarzed the followg Algorthm a. Algorthm a: Kerel PCA Iput: Data set X [ x, x,, x ] Ouput: op prcpal compoets. Costruct the erel matrx K ( x, x j). Ceter K, va K K K K K 3. Solve the egevalue problem: KU U 4. Retur the rows of U u u where we chose u /, where U because we wat to scale, 3 Here defe [] 4 Place a tlde o all varables, ad the results are smlar Page 5 of 5

7 3. HE OU OF SAMPLE EXENSION OF KERNEL PCA o obta a out-of-sample exteso we proceed as before: ceter the ew data wth respect to the trag set, ad the apply the projecto matrx. Here the trasformato VD s explctly uow, but we ca use Kerels to our advatage le the prevous secto. Assume that s the ew data set. Defe z z z ceterg, so defe Z zz z m : ( ) ( ) ( ) md. hs set may eed Z ( z ) ( z ) ( x ) to obta the matrx : ( ) ( ) ( ) Z z z z m Applyg VD ZVD Z VD Z VD cosder ZVD Z VD Z VD, where we are assumg Note that. Let KZ : ( z ) ( x j ) V K U K U Z D Z Z m gves cotas cetered data. Otherwse K Z was costructed usg cetered data the feature space, whch s explctly uow; however, t s possble to wrte the erel terms of the orgal erels as he out of sample KPCA s summarzed Algorthm b. K K K K K Z Z m Z m Algorthm b: out of sample Kerel PCA Z zz z m, Iput: New data set U, Ouput: op prcpal compoets for ew data. Costruct the erel matrx KZ ( z, x j ). Ceter K Z, va 5 K K K K K Z Z m Z m 3. Retur the rows of: KZU KZ u u m, the Both Algorthms a ad b wrtte as a fucto MALAB ca be foud Appedx Here m m, where m [S998] Page 6 of 5

8 3.3 HE OU OF SAMPLE EXENSION OF KPCA (DEMO) Fgure 3. Left: Data set orgal dmesos. Gree pots correspod to out-of-sample observatos. Mddle & Rght: out-of-sample extesos. Gree observatos are mapped correctly to ther correspodg clusters. Fgure 4: A D perspectve of the above plot. 4. MULIDIMENSIONAL SCALING (MDS) Multdmesoal scalg (MDS) vsualzes a set of hgh dmesoal pots lower dmesos (usually two or three) based o ther parwse dstaces [Y985]. he problem MDS solves s to map the orgal data to lower dmesos whle preservg parwse dstaces. I other words, two pots that have large dstace rema far apart the reduced dmeso ad those pots that are close by shall be mapped close to each other. Mathematcally: Gve a set of pots ad ther parwse dstaces, fd pots { } y such that y y j dj, j s mmzed. o solve ths problem cosder the proxmty matrx D [ d j ]. he proxmty matrx s varat to chage locato ad rotato, ad we ca obta a uque soluto provded that we assume y 0. If we cosder the equalty y y j dj, t ca be show that D YY, where [ ] Y y y y D s the cetered proxmty matrx 6. o explctly fd the / Y U d j ad y we use the fact that D s utarly dagoalzable ad obta, where D U U. he above approach has bee see before: From data pots{ x }, create a 6 Some propertes of D clude: () D D () D [ y y j], ad (3) D 0 & D 0. Page 7 of 5

9 eghborhood or smlarty matrx D, ceter ths matrx f eeded, ad the solve a egevalue problem [B003;pg]. Cocretely, f we let S Dj j be the th row sum of D, the ceterg s doe va D D S S S j j j ad the embeddg correspods to the rows of Y u u. [B003; pg]. hs s summarzed Algorthm 3a: Algorthm 3a: MDS Iput: Date set [,,, ] (or sp to step ) X x x x Y y y y Ouput: Embeddg. Costruct the parwse squared dstace matrx D [ d j ] D. Ceter, va Dj Dj S S j S 3. Dagoalze D U U 4. Retur the rows of / U u u 4. HE OU OF SAMPLE EXENSION OF MDS Oe of the facts to cosder the out of sample exteso of MDS s to ote that smultaeously embeddg m ew objects s ot equvalet to dvdually embeddg the same objects oe at a tme, as dvdual embeddgs does ot attempt to approxmate dssmlartes betwee pars of ew objects [P008; pg 3]. For smplcty let m s straghtforward. Suppose that d deotes the squared dssmlarty of the ew object x d [ d d d ], where d x x. Costruct a ew matrx sample exteso, let us troduce otato ad a few deftos. m. he exteso for from the orgal objects. I other words, D d A d 0. Before we formalze the out of Defto: Gve m w, we say that,, m y y s w cetered f ad oly f m j wy j j 0 Defto: For m w such that w m 0, defe w w m m w( C) I C I mw mw Notce that m gves the same matrx that was used the ceterg of D Algorthm 3a. Moreover, uder approprate (D) choces of w, the matrx w( C ) has specal propertes that allow t to be factored for the purpose of MDS [P008], exactly as we dd wth the matrx D. he usefuless of ( ) wll be show the dscusso that follows, but before we start our aalyss, let w C Page 8 of 5

10 us revst the questo of why applyg MDS to A does ot solve the out-of-sample exteso problem. he reaso s due to the fact that applyg MDS to D etals approxmatg er products orgal pots. O the other had, f we let ( ) (A), computed wth respect to the cetrod of all must preserve the orgal ceterg. o do ths, let to fd a yx m (D), whch are computed wth respect to the cetrod of the ( ) [ ], applyg MDS to A etals approxmatg er products w [ 0] pots [P008;pg5]. hus to solve the out-of-sample problem we ad costruct that correspods to the out of sample exteso. Let the orgal pots ad costruct a ew matrx as follows: Y* [ Y y x ] (D) b w( A) : B b. he goal s Y y y y represet the embeddg of ( ). he the problem reduces to approxmatg B m B Y Y m b y y y y y x x x x * * y x If the term y y s dropped, the objectve becomes covex wth soluto x x Y Y yx approxmato 7 of a out-of-sample exteso correspodg to a ew ad uow data pot x Y b, where y x d x. If Y y Y Y b. here s a strog relatoshp betwee PCA ad CMDS [G966] whch t ca be show that X X yx X b ad that ( ) b x ( ) x x x x x so that / yx U b represets a va Y has full ra, the y x s a soluto to [00, pg 3-4]. I other words, to obta a out-sample-exteso for CMDS, we would frst have to compute PCA 8 o the orgal data set X, whch s uow by assumpto, ad thus ot very useful. o overcome ths dffcult, we ca explctly compute b ad wrte t terms of ow values. Frst, by defto, D b B I ( ) w A I w( ) b A w A Aw w Aw ( ) ( ) ( ) ( ) D d 0 D d D d 0 D d d 0 0 d 0 d d D d D d D D D D d 0 D d d d D D Sce we are oly terested b, ether tag the last colum or row of B, we have b d d D D 7 o obta the optmal soluto, the above olear optmzato problem must be solved umercally. 8 MDS ad PCA attempt to fd the most accurate data represetato a lower dmesoal space. MDS preserves the most smlartes of the orgal data set, whle PCA reduces dmeso by preservg most of the covarace of data. Page 9 of 5

11 he out-out-of-sample exteso for the case approxmate case s foud Appedx 6.3 m s summarzed Algorthm 3b, ad ts correspodg mplemetato for the Algorthm 3b: out of sample MDS (m = ) U / d Iput: Data set,,from MDS &, the square dssmlarty betwee ew object ad orgal objects Ouput: out of sample y x. Costruct vector b, b d d D D. For approxmate out-of-sample exteso (colum) vector retur y U b / x 3. For optmal out-of-sample exteso retur costruct Y* [ Y y ] x, retur y x that mmzes D B : b m B Y Y y x * * b, ad 9 Notce that the objectve fucto for the optmal soluto s a fourth degree polyomal, whch ca be solved usg gradet umercal methods [008;pg0] 4.3 HE OU OF SAMPLE EXENSION OF MDS (DEMO) Fgure 5: Map of Chese ctes based o ther parwse dstaces. op left: MDS appled to the etre data except Lhasa. Bottom left: out-of-sample exteso for Lhasa. op Rght ad Bottom: he same aalyss ad comparso doe for Hohot. 9 Here d d j j Page 0 of 5

12 Fgure 6: MDS o the seed data set 0. Left: MDS appled to the etre data set. Red crosses correspod to pots that wll be traced the out-of-sample exteso. Rght: MDS o a subset of the data set. Gree pots correspod to the out-of-sample extesos. 4.4 HE OU OF SAMPLE EXENSION OF MDS (DEMO) I secto 4.3, we provded the out-of-sample Algorthm for MDS for the case A m m. For the case m, aalogously defe to be the squared dssmlartes betwee all m objects. Frst, costruct the matrx ( ) m A that ( ) cossts of the MDS embeddg of all m orgal pots, ad let objects. As before, let m Z zz z m be the matrx cotag the o fd the out-of-sample exteso for these ew m (D) B YZ w( A) : B BYZ BZZ Z, ad obta the optmal Z m Y y y y represet the embeddg of the w [ 0 0] -dmesoal embeddg of m ew objects. m objects let, costruct the matrx [P008;pg] by solvg the optmzato problem: Y m B Y Z m B YZ B ZZ m Z m YZ ZZ Z 5. CONCLUSION I ths paper we dscussed the out-of-sample extesos for PCA, Kerel PCA, ad MDS. All three extesos share a ceterg step whch ca be doe easly the PCA case, or eeded to be appled to the Kerel matrx or dssmlarty matrx for Kerel PCA ad MDS, respectvely. o fd the out-of-sample exteso for PCA, ceterg ad mappg to the best-ft le s eough. As for Kerel PCA, a addtoal Kerel matrx, betwee the trag ad ew data eeds to be created, ad ceterg s also mportat before mag trasformatos usg the prevously bult matrces. For the thrd method, MDS, the costructo of matrces volvg trag ad ew data sets, ad ceterg s also requred. Ule the prevous two methods, exact aalytc solutos caot be foud, but rather a approxmate (based o PCA) or a optmal (by solvg a optmzato problem) soluto ca be computed. We have also provded some examples whch we show that out-of-sample extesos are ot equvalet to a embeddg or trasformato of the etre data set, especally methods that are sestve to outlers. 0 Ca be dowloaded at : Page of 5

13 6. APPENDIX Appedx 6. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%% Prcpal Compoet Aalyss (PCA) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Xtr = trag data set. Each row s a observato. % Xtst = ew observatos (out of sample) % = # of compoets to eep. fucto [XtrPCA,V,XtstPCA] = PCA(Xtr,Xtst,) meaxtr = mea(xtr); % ceter the data X_tlde = Xtr - repmat(meaxtr, sze(xtr,), ); [U,S,V] = svds(x_tlde,); XtrPCA = U*S; V = V(:,:); % projecto matrx % out of sample exteso XtstPCA = (Xtst-repmat(meaXtr, sze(xtst,), ))*V; Page of 5

14 Appedx 6. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%% Kerel PCA %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Xtr = data whch each row s a observato. % Xtst = ew observatos (out of sample) % = reduced dmeso; var = sgma^ % Here we use the Gaussa Kerel. Code ca easly be modfed for % other erels. fucto [XtrKPCA,XtstKPCA] = KPCA(Xtr,Xtst,,var) % Calculate parwse dstace matrces Dtr = pdst(xtr,xtr); % Costructg Kerel matrx usg Gaussa Kerel K = K(Dtr, var); % ceterg = sze(xtr,); Kc = K - K*oes(,)/ - oes(,)*k/ + oes(,)*k*oes(,)/(^); % Obta the evectors of K_tlde that correspod to the largest evalues. % hose evectors are the data pots already projected oto the respectve % prcpal compoets. [U,S] = eg(kc,'vector'); [~,dx] = sort(s,'desced'); S = S(dx); S = dag(s); U = U(:,dx); S = abs(s(:,:)); XtrKPCA = U(:,:)*sqrt(S); % out of sample exteso of KPCA Dtst = pdst(xtst,xtr); Kz = K(Dtst, var); % ceterg m = sze(xtst,); Kzc = Kz - oes(m,)*k/ - Kz*oes(,)/ + oes(m,)*k*oes(,)/(^); %XtstKPCA = Kzc*U(:,:)*v(sqrt(S)); XtstKPCA = bsxfu(@rdvde,kyc*xtrkpca,dag(s)'); ed %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Gram matrx fucto %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% fucto K = K(D, var) % (x,xj) = exp(-0.5* x-xj ^/var_j) K = exp(bsxfu(@rdvde, -0.5*D.^, var)); ed Page 3 of 5

15 Appedx 6.3 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%% Math 85 Project Fucto: MDS %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %INPU: X = dstace matrx, assumed to be symmetrc. % = target dmeso; Y = ew (lower) coordates dmeso % d = dstace of ew pot wth orgal pots as a row vector. fucto [Y, y_x, stress] = mds(x,d,) = sze(x,); D = X.^; mead = mea(d,); mmead = mea(mead); %D_tlde = 0.5(-D + oes(,)*d/ + D*oes(,)/ - oes(,)*d*oes(,) D_tlde = 0.5*(repmat(meaD',, ) + repmat(mead,, ) - D - mmead); % Costructg matrx Y [U,S] = eg(d_tlde,'vector'); [~,dx] = sort(s,'desced'); S = S(dx); S = abs(dag(s)); U = U(:,dx); Y = U(:,:)*sqrt(S(:,:)); % Computg stress: stress = sqrt(sum(sum(x,))/(^*l_dotdot)); stress = sqrt(*sum((squareform(x) - pdst(y)).^)/mmead)/; % Out-of-sample exteso d = d'.^; % b = d - oes(,)*d/ - D*oes(,)/ + oes(,)*d*oes(,)/^; b = -0.5*(d - mea(d) - mea(d,) + mmead); y_x = b'*y/s(:,:); % (sqrt(s)u'b)'; retur as row vector Page 4 of 5

16 7. REFERENCES [A003] Aderso, M.J ad Robso, J. Geeralzed dscrmat aalyss based o dstaces. Australa & New Zealad Joural of Statstcs, 45:30 38, 003 [B997] Borg, I., & Groee, P. (997). Moder multdmesoal scalg: theory ad appl- catos. New Yor: Sprger. [B003] Y. Bego, J.-F. Paemot, ad P. Vcet. Out-of-sample extesos for LLE, Isomap, MDS, egemaps, ad spectral clusterg. echcal Report 38, D epartemet d Iformatque et Recherche Op eratoelle, Uverst e de Motr eal, Motr eal, Qu ebec, Caada, July 003. [B006] Bshop, Chrstopher M. (ed.). Patter Recogto ad Mache Learg. Sprger, Cambrdge, U.K.,006. [G996] J. C. Gower. Some dstace propertes of latet root ad vector methods multvarate aalyss. Bometra, 53:35 338, 966. [S998]. Schölopf, B., Smola, A. ad Müller, K.-R. Nolear compoet aalyss as a erel egevalue problem. Neural Computato, 998 [S999]. Scholopf, B., Smola, A.,ad Muller, K.-R. (999). Kerel prcpal compoet aalyss. I B.Scholopf, C. J. C. Burges, ad A. J. Smola, edtors, Advaces Kerel Methods SV Learg, pages MI Press, Cambrdge, MA [S003]. Shles, J. A utoral o Prcpal Compoet Aalyss: Dervato, Dscusso, ad Sgular Value Decomposto [008] M. W. rosset ad C. E. Prebe. he out-of-sample problem for classcal multdmesoal scalg. Computatoal Statstcs ad Data Aalyss, 5: , Jue 008 [00] M. W. rosset ad M. ag. he out-of-sample problem for classcal multdmesoal scalg: Addedum, November 00 [W0]. Q. Wag, Kerel Prcpal Compoet Aalyss ad ts Applcatos Face Recogto ad Actve Shape Models. CoRR, 0. [Y985] Forrest W. Youg, Uversty of North Carola Kotz-Johso (Ed.) Ecyclopeda of Statstcal Sceces, Volume 5, Copyrght (c) 985 by Joh Wley & Sos, Ic Page 5 of 5