Johann Wolfgang Goethe-Universität Frankfurt am Main

Transcription

1 Fachberech Informatk Johann Wolfgang Goethe-Unverstät Frankfurt am Man 7KH3ULQFLSDO,QGHSHQGHQW &RPSRQHQWVRI,PDJHV %$UOW5%UDXVH )DFKEHUHLFK,QIRUPDWLN 5REHUWD\HU6WUD H )UDQNIXUWDPDLQ

2 Abstract Classcally, encodng of mages by only a few, mportant components s done by the Prncpal Component Analyss (PCA). Recently, a data analyss tool called Independent Component Analyss (ICA) for the separaton of ndependent nfluences n sgnals has found strong nterest n the neural network communty. Ths approach has also been appled to mages. Whereas the approach assumes contnuous source channels mxed up to the same number of channels by a mxng matrx, we assume that mages are composed by only a few mage prmtves. Ths means that for mages we have less sources than pxels. Addtonally, n order to reduce unmportant nformaton, we am only for the most mportant source patterns wth the hghest occurrence probabltes or bggest nformaton called Prncpal Independent Components (PIC). For the example of a synthetc pcture composed by characters ths dea gves us the most mportant ones. Nevertheless, for natural mages where no a-pror probabltes can be computed ths does not lead to an acceptable reproducton error. Combnng the tradtonal prncpal component crtera of PCA wth the ndependence property of ICA we obtan a better encodng. It turns out that ths defnton of PIC mplements the classcal demand of Shannon s rate dstorton theory. Keywords: Prncpal Component Analyss PCA, Independent Component Analyss ICA, Prncpal Independent Component Analyss PICA, Rate Dstorton Theory 1 Introducton One of the most nterestng and ambtous propertes of artfcal neural networks s grounded n the actve nformaton processng of real world data: the unsupervsed analyss of sgnals. 1.1 Prncpal components and PCA An nterestng approach has been developed throughout the recent years: the lnear transformaton of the nput space to the base of prncpal components whch mnmzes the mean squared error when droppng some of the transformed channels. Ths transformaton called Prncpal Component Analyss (PCA) and obtaned by algnng the base vectors to the drectons of maxmal varance, s dentcal to a dscrete Karhunen-Loève or Hotellng transformaton. Here, we decompose the n sgnals (x 1,,x n ) T x by a lnear transform y = Wx wth y = (y 1,,y n ) T (1) such that a subset y = (y 1,,y m ) T of m < n components used wth the matrx W m 1 (consstng of m columns of W 1 ) to reconstruct the orgnal sgnals by x = W m 1 y obtans the smallest mean squared error (x x ) = mn n the reconstructon process. It s well known that ths s the case for the projectons of the nput x on the m egenvectors wth the bggest egenvalues λ 1,,λ m of the covarance matrx C xx = (x x )(x x ) T Thus, the varance of a component y s gven by λ = (y y ) = σ, and the rows of W meet the condtons for orthonormalty w T w = 1 and w T w j = for j () We see that the whole sgnal s decomposed by a nonscalng lnear transformaton nto dfferent drectons w. To obtan the smallest error of reconstructon, we use the drectons wth the bggest varances. So, the components (and the correspondng drectons or base vectors) are ordered accordng to a crteron. The selected m ones are called the prncpal components. Many neural networks have already been proposed whch let ther assocated weght vectors converge to the base of prncpal components, the egenvectors of the nput covarance matrx, by proper learnng rules, see e.g. [OJA9], [BRA93a]. For mages, the search for the prncpal components (called "transform mage codng") can be organzed as a local process. Thus, a whole pcture can be encoded n parallel by many neurons on a sensory plane wth local nteractons (e.g. lateral nhbton), usng only the selforganzed prncpal components [BRA96] obtaned by analog crcuts [BRA94]. 1. Independent components and ICA The approach of PCA s only optmal for the performance measure of the mean squared error and assumes no specfc nformaton about the hgher order statstcs of the observed sgnals. If we want to maxmze other measures of nformaton processng, for nstance the nformaton capacty of the encodng coeffcents (.e. the output sgnals of the transformng system), we have to obtan other propertes. Here, the mutual nformaton H(y 1 ;y ; ;y n ) between the output channels s a good measure for an effcent output codng. The output nformaton H(y 1,y ) of two channels y 1 and y H(y 1,y ) = H(y 1 ) + H(y ) H(y 1 ;y )

3 becomes maxmal f for constant channel nformaton H(y ) the mutual nformaton becomes mnmal. Ths s the case f H(y 1,y ) = H(y 1 ) + H(y ) whch means for the probablty densty functons (pdf) p(y 1,y ) = p(y 1 )p(y ) Thus, the demand for mnmal transnformaton s dentcal wth the demand for ndependent channel pdf ("factoral code"). For n channels ths means p(x) = p(x 1 )p(x ) p(x n ) (3) Let us assume that all observed sgnals x = (x 1,,x n ) T are derved from a lnear mxture of n unknown ndependent source sgnals s = (s 1,,s n ) T wth an unknown mxng matrx M wth rows m x = Ms, x = m s (4) How can the orgnal source sgnals be reconsttuted? Another lnear transformaton wth a matrx B y = Bx = BMs (5) mght obtan the sources f y = s BM = I (6) the demxng matrx B becomes the nverse of M. The problem of fndng the demxng matrx s known as the problem of "blnd separaton of sources" or "Independent Component Analyss" (ICA) and s a fast growng topc n neural network research, see e.g. [ACY96], [BUR9],[COM94], [DEO96], [HYO96]. The ndependent sgnals are obtaned by usng objectve functons (called contrast functons [COM94]). One of them s the demand for mnmal transnformaton between the sgnals and can be used to obtan learnng rules for the unknown base vectors of the nverse transformaton B of ICA, see [ACY96]. There are several condtons nvolved n the demxng process n order to get the source sgnals (see [COM94]): The mxng matrx M must be regular to have the nverse B=M 1 to exst wth Bx = BMs = M 1 Ms = s. Ths means that we have to have the same number n of sources as of observed mxtures. The source s determned regardless of the order (ndex) of the channels n s. Ths s due to the fact that the crucal condton for ndependence, the factorzaton p(s) = p(s 1 )p(s ) p(s n ) of the probablty dstrbuton functon (pdf) by the margnal pdfs, s stll vald for p(s) = p(s 1 )p(s n ) p(s ) or any other permutaton of the ndces. In eq.(4), the same mxture x s produced f we scale a source s by a factor c and the correspondng column M of M by a factor 1/c. Thus, wthout further knowledge, we cannot determne the scale of the source sgnals: the ICA s an "ll-posed problem". For two Gaussan sources s 1 and s a smple decorrelaton procedure (PCA) gves us ndependent sources. Nevertheless, t s well known that the PCA decorrelaton s done by an orthogonal matrx composed by the egenvectors of C xx, see eq.(). Snce we assume M to be generally not orthogonal (.e. t does perform more than a rotaton), we cannot demx the sgnals just by a rotaton: the demxng s not correct. The operaton of separatng the sgnals nto s 1 and s s not unque; wthout any further nformaton the ambguty for Gaussan sgnals cannot be resolved. For addtonal Gaussan sources, ths problem aggravates. Ths means for successful demxng at most one source can have a pdf wth Gaussan characterstc. Thus, we cannot expect to recover the exact source sgnals s but only ther scaled and permutated versons y = DPs wth a dagonal scalng matrx D and a permutaton matrx P. Ths relaxes the condtons on the demxng matrx B n eq.(5) to BM = DP (7) Here, B s n general not equal to M 1 although n the followng we stll call B "the nverse matrx of M" and y "the source sgnals". In order to enable a soluton t s convenent to assume that the recovered source sgnals y have unt varance σ ² snce D s unknown. Furthermore we assume that the y are centered,.e. y. Ths requres the demxng process to center the observed sgnals x as well for ther average x mght be non-zero. Consequently, we get the relaton y = B (x- x ) = BM (s- s ) = DP (s- s ) (8) The standard ICA procedure conssts manly of the followng stages (shown n Fg.1). M x- x W PCA W ICA s mx x center x whten v ndep. y B Fg.1 The processng stages n ICA The observed sgnals x are dmnshed by ther frst and second moments: They are centered, decorrelated and whtened to unt varance by a lnear transform wth a matrx W PCA, and then separated by ther hgher moments n the last stage by another lnear transform W ICA. The latter whch uses the preprocessed nput s often referred as "the ICA matrx". So far we have 3

4 y = B (x- x ) = W ICA W PCA (x x ) (9) wth B = W ICA W PCA. If we use a PCA process for the decorrelaton process n W PCA we also can addtonally scale the rows w of W PCA whch are the egenvectors of C xx by ther egenvalues w w λ ½ such that w = λ 1 Ths normalzes the varance of v because we have v = (w T x ) = w T x x T w = w T C xx w = w T w λ = 1 The whtenng process gves us an advantage: For whtened, decorrelated nput of vv T = I the ICA matrx W ICA s orthogonal,.e. just a rotaton of the base of the nput space. Ths can be easly shown: Wth v W PCA x and the assumptons of centered and ndependent sources havng unt varance (.e. y and yy T = I), we get I = yy T = W ICA vv T W ICA T = W ICA W ICA T Thus, the nverse matrx W ICA 1 s dentcal to the transposed matrx W ICA T whch mples that W ICA has to be orthogonal. The classcal ICA encodng system above can be traned usng separate layers of neural networks. The frst stage s obtaned by learnng the expectaton value as an offset n order to center the nput: x (t+1) = x (t) + 1/t (x(t) x (t)) For the second stage standard PCA learnng rule can be used, see e.g. [OJA9], coupled by a rescalng descrbed above. Otherwse, specal whtenng learnng rules can be used, see [SIL91],[PLUM93],[BRA98]. For the thrd stage, the ICA layer, one of the ICA learnng rules may be taken, e.g. [HYO96]. Now, for encodng pctures by a decomposton wth the most mportant, ndependent components we wll run nto trouble. Let us assume that we have just 4 ndependent vsual objects on a pcture of = pxels. Certanly, we want to obtan a sgnfcant smaller number of outputs to descrbe the pcture than But f we use less neurons for data compresson, ths becomes n conflct wth the demand of the same number for sources and mxtures, the frst condton for ICA cted above. What can we do? One common soluton, taken n [BES96] and [OLS96a,b] s to cut the mages nto smaller patches, say 1 1=144 pxels, present many patches of many mages (preferably natural scenes) and then make an ICA of the 144 channels. Ths gves us 144 ndependent "base pctures". Nevertheless, not all ICA components are equally mportant. Some of them are just spurous patterns wth a low occurrence probablty. Snce we want to obtan a stable code whch covers most of the nput data, we am for the m ICA components wth the hghest occurrence probablty. Here, we encounter a serous problem: how can we order the components, e.g. by an occurrence probablty, whch the ICA model so far dd not provde? In standard ICA applcatons, all (tme seres) channels are always present,.e. equally probable. However, ths s not the case for real world objects. In order to cover ths aspect also, we have to develop a new mage model whch s composed by sgnals and events. An event-orented mage model Let us model the mages as a superposton of many small, ndependent mage patches, just lke a sngle neuron of the retna sees the world by a very restrcted focus. Our task conssts now of fndng the most probable ones..1 Image event prmtves, sgnals, and ICA As an ntroductory example, let us consder as nput events several pctures composed of four pxels. The four sample pctures are shown n Fg.. The black pxels are coded as 1, the whte ones as +1 and the gray ones as zero. M 1 =(1,-1,,) T M =(,1,1,1) T M 3 =(-1,1,1,) T M 4 =(,-1,-1,) T Fg. The four sample pctures In the followng state-tme dagram (Fg.3) four events are presented ndependently. Here, each event s denoted by two states, present (on) or not present (off). The tme order of the ndependent events s assumed to be random. tme step on off pcture Fg.3 The state-tme dagram of nput events t

5 Each event ω manfests tself on all four pxels or four channels. Assgnng a sgnal vector s ω to the event ω ="pcture appears" we note the events by the vectors 1 1 s ω1 =, s ω =, s ω3 =, s ω4 = 1 1 The pcture tself can be descrbed by the nfluence of the event on the pxels. Formally, we can wrte ths as a lnear mxture performed accordng to eq.(4) by the mxng matrx 1 M = (M 1, M, M 3, M 4 ) = The superposton of the nfluences can be observed at each pxel as the tme seres of superposed sgnals. In Fg.4 the ntensty of all four pxels s shown for the ntroductory example. tme step pxels pxel value Fg.4 The tme seres of the pxel channels In Fg.5 the correspondng mages are shown. x 1 =(1,-1,,) T x =(,,,1) T x 3 =(,,1,) T x 4 =(,-1,-1,) T x 5 =(,1,1,1) T x 6 =(,-1,,) T -1 Fg.5 The sx sample pctures Snce we assume the four events to be ndependent, we can see our task as not only separatng the four channels of the source sgnal s from the lnear mxture x wthout t any knowledge about the mxture matrx M, but also to deduce the occurrence probabltes P(ω ) for the ndependent events ω.. Orderng the Independent Components To ntroduce the man dea for computng the probabltes for the prncple ndependent components, we notce that the source sgnals are defned as s = 1 for ω for ω Thus, we have as the average source sgnal s s = P(s =1) 1+ P(s =) = P(ω ) (1) The varance σ s of the source sgnal s s σ s = (s s ) = s s s + s = s s = P(s =1) 1+ P(s =) s = s s = s (1 s ) (11) Suppose that we have already computed the demxng matrx B satsfyng eq.(8). The recovered source sgnals y are derved from the centered source sgnals s by scalng and permutaton wth a matrx A BM = DP. As stated n secton Independent components and ICA t s mpossble to determne the permutaton matrx P so we assume P I and A D. For one component y we get y = a (s s ) (1) where a denotes the correspondng dagonal, non-zero, coeffcent of A. Snce y s centered and has unt varance σ y the followng relaton holds: 1 = σ y = (y ) = (a (s s )) = a σ s = a s (1 s ) (13) The average s of the source sgnals s transformed by the mxng matrx to the observed average sgnal x = M s (14) and by the demxng matrx B to the average transform output y = B x = BM s = A s (15) Note that here y s obvously non-zero snce we omtted the centerng stage. Therefore we have y = a s (16) Combnng eqs.(13) and (16) gves us the relaton between the observed, non-centered output and the needed occurrence probabltes 5

6 or 1 = ( y / s ) s (1 s ) P(ω ) = s = y / (1+ y ) (17) By ths we have a measure to order the obtaned ICA components accordng to ther assocated occurrence probabltes P(ω 1 ) > P(ω ) > P(ω m ). Snce the most probable events should not be neglected at all they are the most mportant ones. There s also an correspondence to the average nformaton of each component. Wth the defnton of the average Shannon nformaton 3 Smulatons and results In ths secton we want to vsualze our theoretcal results of the prevous secton and show the valdty of our mage model. 3.1 Recoverng the occurrence probabltes of events For the start we want to show that t s possble to obtan the occurrence probabltes of ndependent events. For ths purpose we use very basc mage events. We chose 16 letters A P, represented by a very coarse matrx of 8x8 pxels, see Fg.7. H(y) = α Ω P ( α ) log( P( α)) (18) and settng the state space to Ω {ω, ω } we obtan the margnal nformaton for one recovered source y H(y )= P(ω ) log(p(ω )) (1 P(ω )) log(1 P(ω )) (19) By assgnng an order to the components accordng to ther nformaton we defne wth H 1 H H m another order. How s ths order related to the prevous crteron of maxmal occurrence probablty? In Fg.6 the nformaton of one component s shown as a functon of ts probablty. Fg.7 The mage encodng of the events For each one of 496 tranng patterns, a random lnear combnaton of the letters was computed and presented to a network of 16 neurons. In Fg.8 ffteen nput sample pctures out of the 496 are shown.,7 H,6,5,4,3,,1 p p,5,5,75 1 Fg.6 The nformaton of one component as functon of ts occurrence probablty Snce nformaton s a convex functon, probablty and nformaton are both monotoncally ncreasng up to the local maxmum whch s located at or H (p)/ p = log(p ) + log(1 p ) = p =.5 Thus f we order the components n ths range accordng to j P(ω ) p P(ω j ) p H H j () we get the desred decreasng entropy order stated above. Fg.8 Sample nput pctures of mxed events The nput events are transformed to decorrelated components by a PCA stage. Intally, we used the full alphabet, but after the PCA stage some components wth zero egenvalues were observed. Ths means that some letters of the alphabet can be decomposed by a lnear combnaton of others. To obtan really ndependent sources we chose the subset of 16 letters shown n Fg.7. The egenmages formed n the PCA stage,.e. the rows of matrx W PCA, correspond to the decorrelated components found by the PCA stage and are shown n Fg.9. Fg.9 The egenmages of the nput pctures 6

7 Here, we observed a near-gaussan probablty dstrbuton of the sgnal values, see Fg.1. 6 w.pca Fg.11 The nverse source mages obtaned after the ICA stage w.pca w.pca w.pca Fg.1 The probablty dstrbuton of four mage sgnals obtaned after the PCA and whtenng stages To obtan the hstograms the 496 samples were quantfed nto 56 ntervals on the horzontal axs. After the ICA stage, we recovered the source sgnals. Snce we want to concentrate on the topc of prncple components we do not descrbe the algorthms used to obtan the PCA and ICA n detal. Nevertheless, t should be mentoned that the statstcal nature of the source sgnals presented a severe problem for some algorthms. For the concrete events of ths secton, we have exclusvely bmodal source dstrbutons wth negatve kurtoss, see Fg.13. Our smulatons showed that some of the algorthms had problems wth bmodal mages,.e. negatve kurtoss [BES96], and some wth the natural mages of postve kurtoss [ACY96]; they dd not converge for these mxtures. In order to obtan the desred results, we used versons of the algorthms descrbed n [HYO96]. The nverse of the resultng matrx B s the mxng matrx M, contanng the letters. The mages correspondng to the B matrx are shown n Fg.11, the nverted B matrx gves us the reconstructed source mages n Fg.1. Fg.1 The source mages obtaned after the ICA stage We see that nether the ntal order nor the sgn of the sources were preserved. The occurrence probablty dstrbuton of four components s shown n Fg ICA ICA ICA ICA Fg.13 The probablty dstrbuton of four mage sgnals obtaned after the ICA stage Ideally, the peaks seen n Fg.13 are just spkes wth zero varance. Thus, the functon values n a small nterval around each local average can be summed up n the center of the assocated nterval, and set to zero afterwards. Ths knd of quantzaton should gve us a better estmate of the orgnal probablty dstrbuton. 7

8 The ntal and estmated occurrence probabltes of the source letters are lsted n table 1. The error s due to the mperfectly learned ICA stage. source probablty error letter used observed D,715,73 -,17 F,696,73 -,36 I,743,695,49 B,69,673,19 G,577,68 -,51 M,64,618,6 L,5,534 -,14 O,538,53,6 C,43,484 -,61 A,49,466,7 J,444,463 -,19 H,75,396 -,11 E,454,36,9 N,48,34,66 K,415,3,9 P,341,31,31 be encoded wth a smaller number of bts. Now, for further consderatons, let us change to natural mages. 3. Reconstructng natural mages Image encodng by very few number of coeffcents s stll a demandng task and has a lot of applcatons. Perhaps, by usng the ICA approach, we mght obtan an encodng wth a fewer number of components. For ths purpose, let us regard the ndependent components of natural mages. The method to obtan these components s smlar to the one n conventonal transform codng: the whole mage s splt nto submages contanng n pxels, and each submage s used as one tranng sample. In our smulatons the pcture called Cactus (Fg.14) was dvded nto 4543 submages (sze: 8 8=64 pxels) whch were randomly chosen as tranng samples. Table 1 The source letters, ther assocated and ther recovered occurrence probabltes Now, our ntal goal s stll the effcent encodng of the mage sgnals. Ths s obtaned by reducng the margnal entropy of the channels. Table shows the approxmated average nformaton, the entropy, of the frst four channels before and after the ICA stage (calculated from the probablty dstrbuton n Fg.1 and Fg.13). component observed entropy component observed entropy orgnal entropy w.pca ICA1 ('J') w.pca 7.48 ICA ('K') w.pca3 7.3 ICA3 ('F') w.pca ICA4 ('M') Table The margnal entropy of four channels (n bts) Obvously, mnmzng the mutual nformaton dramatcally reduces the sngle channel nformaton. Snce the probablty dstrbutons of the ICA components are slghtly blurred ther margnal entropy s stll hgher than the orgnal entropy accordng to eq.(19). However, by applyng a rgorous quantzaton strategy we should be able to acheve further reducton as stated above. In lnear mage codng and restoraton, we know that by defnton the prncpal decorrelated components obtaned after the PCA stage yeld the mnmal mean squared error (MSE). Thus, we cannot expect that the prncpal ndependent components wll gve us a smaller MSE. Nevertheless, what we can attend s that they can Fg.14 The tranng pcture Cactus Frst, we centered and decorrelated the 64 components of the submage ensemble. The obtaned PCA egenmages are shown n Fg.15 (page 9). After ths, the components are transformed lnearly. The transform coeffcents are updated by an teratve ICA learnng algorthm gvng us the matrx W ICA used n eq.(9). The columns of matrx B are shown as mages n Fg.16. The nverse of B s the mxng matrx M. The columns of ths matrx are the source mages, shown n Fg.17. The source mages obtaned are very smlar to those already known n the lterature, see e.g. [BES96], [OLS96b]. 8

9 Fg.15 The PCA egenmages of Cactus Fg.16 The base ICA mages of Cactus Now, what are the most mportant events? Here, the measured probablty dstrbutons of the sources were not bmodal. Ths excluded our event model of secton Recoverng the occurrence probabltes of events for calculatng the occurrence probabltes and therefore prevents an order of mportance of the sources for analyzng the observed stuatons by mportant events. Nevertheless, we stll can use the margnal nformaton to compute the order of the components nstead. Interestngly, the ntal order gven by the ICA algorthm s characterzed by ncreasng entropy. Ths s due to the goal of our (sequental) ICA algorthm whch tres to mnmze the margnal entropy for the frst component by choosng the ICA component whch dffers the most from a Gaussan dstrbuton,.e. whch has the smallest avalable entropy. To answer the basc queston f there are prncpal ndependent components whch contan consderably more or less average nformaton than others we calculated the margnal entropy of all components the same way as n secton Recoverng the occurrence probabltes of events. The cumulated margnal entropy of the frst k whtened PCA components (n order of decreasng egenvalues) and ICA components (n order of ncreasng entropy) s shown n Fg cumulated entropy [bts] w.pca ICA k Fg.18 The cumulated margnal entropy of the frst k whtened PCA components (dotted lne) and ICA components Fg.17 The source mages of Cactus The dfference between the two cumulaton functons can hardly be seen: the margnal entropy of the ICA components s just slghtly smaller than the one of the whtened PCA components. Furthermore, the cumulated entropy of both the PCA and the ICA grows approxmately proportonal. Ths means that especally all the ICA components of the mage have nearly the same nformaton; there are no components whch dffer much from the others. If not n occurrence probablty or average nformaton, are there ICA components whch dffer n mportance? Are there some whch are more mportant than the others so we have to concentrate on them? 9

10 3.3 Component orderng by nformaton One crteron for mportance s the qualty of the mage reconstructed by the remanng components. In Fg.19 a cutout of the orgnal mage Cactus s shown. In both cases, the reconstructon qualty s not acceptable, especally, when compared wth the reconstructon result of the frst 16 PCA components, shown n Fg.. Fg.19 The cutout of the mage Cactus The cutout, reconstructed by the 16 ICA components wth the smallest average nformaton, and by the 16 ICA components wth the bggest average nformaton, can be seen n Fg. and Fg.1. Fg. The reconstructon by the frst 16 PCA components It seems that the pure nformaton crteron s not approprate for mage reconstructon. In contrast to ths, the PCA transform seems to gve better results. Reconstructng the mage by ts frst k components and comparng t wth the orgnal one gves us the average error for neglectng the n k components. Certanly, by usng the k egenmages of the PCA stage wth the bggest egenvalues, the mean squared error MSE s mnmzed because the PCA operaton s defned to obtan the smallest possble MSE. Are there prncpal ICA components whch also mnmze the error? Let us compare the MSE contrbuton by the PCA components by those by the ICA. In Fg.3, ths s shown for the mage Cactus. Obvously, usng the components wth the bggest entropy does decrease the MSE sgnfcantly faster than usng the ones wth the smallest entropy. Certanly, the smallest MSE s produced usng the PCA components (dotted lne). Fg. The reconstructon by the frst 16 ICA components 7 6 MSE ICA (ncreasng entropy) ICA (decreasng entropy) 1 PCA k Fg.3 Decreasng the MSE by addng components Fg.1 The reconstructon by the last 16 ICA components 1

11 3.4 Component orderng by vrtual varance How can we further mprove the performance of the selected ICA components? The PCA sortng crteron s the decreasng value of the egenvalues. Snce we know that the egenvalue λ s λ = σ = var(y ) equal to the varance of the component, we mght order the ICA components also approprate to ther varance. Here, we encounter a problem: the ICA transform s such that all varances of the components are made equal. How to select the ones wth the bggest varance? Inspectng the transform closer we notce that the output varances are equal, but not the length of the correspondng bass vectors w of the ICA transform (rows of matrx W). To compare t to the PCA transform whch has unt length bass vectors, we have to normalze the ICA bass vectors. Thus, we mght defne a vrtual varance of a component by w var* (y ) var w var(y ) 1 x = = w w (1) Orderng the ICA components by ths crteron, we obtan a better MSE-adapted reconstructon whle preservng the performance of the cumulated entropy. In Fg.4 the best ICA orderng of Fg.3 s compared to the vrtual varance orderng MSE PCA ICA (decreasng entropy) ICA (decreasng vrtual varance) Fg.4 The MSE of the ICA orderng by vrtual varance To obtan an mpresson of the reconstructon qualty, we present the reconstructed mage cutout of Cactus by usng the ICA components wth the bggest vrtual varance n Fg.5. Clearly, ths orderng performs better than the two prevous ones, but t s stll nferor to the classcal PCA approach. k Fg.5 The reconstructed mage cutout Now, wthout usng any other mage reconstructon qualty measure (lke, for nstance, the psychophysologcal approach, see e.g. [CHR9]), we ask: what can the ICA approach do for encodng and reconstructng mages when the mnmal MSE of the reconstructon s gven by the number of PCA components? 3.5 Prncpal ndependent components and rate dstorton theory When we reduce the number of components n the transform approach for encodng mages we reduce the full space of mage components (dmensons) to a subspace. The subspace of the ICA components s characterzed by ts nformaton content whereas the subspace of the PCA components s characterzed by ts low MSE reconstructon error. Now, f we cannot replace the prncpal components of PCA for obtanng a small MSE, what about reducng ther encodng nformaton by ICA? Ths dea can be performed n two ways: Get the frst k PCA components wth an acceptable MSE. Then, by an ICA transform, we wll get the same number of encodng coeffcents but wth less nformaton,.e. less encodng bts. For the same amount of encodng nformaton as the k PCA components take, we can also get p more ICA transformed PCA components. Snce these p+k base vectors of the ICA transform span the same space as the p+k PCA components, the resultng mage qualty wll be enhanced lke addng p more PCA components. Thus our approach, startng wth the search for ndependent mage prmtves, leads us to the error-bounded maxmal nformaton for each channel. Ths s not new: the approach of maxmzng the nformaton for a tme step n a channel when an upper bound for the error (more general: for a dstorton measure) exsts or, vce versa, to mnmze the error for a channel wth a constant nformaton per tme step s classcally known as the rate dstorton theory [SHA49] and has a broad range of applcatons n the classcal telecommuncaton area. 11

12 The frst one of the deas above can be expanded f we order the k ICA components accordng to ther decreasng vrtual varance and encode only the frst k' < k components wth low addtonal reconstructon error. Ths results n a further reducton of the number of encodng bts. To valdate the latter dea we computed the ICA components of the frst k PCA components (Fg.15) for k = 16,,1. In Fg.6a,b the ICA base vectors and mages can be seen for k = 17. Note that they are dfferent to those obtaned n Fg.16 because the data space s also dfferent. a) b) Fg.6 The 17 ICA base vectors and 17 mages Then the cumulated entropy was calculated and compared to the cumulated entropy of the frst k whtened PCA components. We found that for the same nformaton rate at most one addtonal ICA component can be encoded wth an error reducton of 5%. An example for 17 ICA components s shown n Fg.7: the reconstructed mage s slghtly better than the one of Fg.. Untl now we estmated the overall encodng amount by calculatng the margnal entropy of the components wthout consderng effcent quantzaton technques. In the next secton we shall take a closer look at ths task. 3.6 Robust encodng of natural mages wth prncpal ndependent components Suppose we have an mage decomposed nto submages whch we want to encode as effcent as possble (see secton Independent components). Snce we are dealng wth dgtzed mages the n components (pxels) x of an arbtrary submage x = (x 1,,x n ) T are dscrete,.e. each x stores one of N dfferent values. Thus there s a number N n of dfferent mage patches or "mage states" that can be assgned to x. Obvously, a lot of these mage patches are unlkely to occur n natural mage data (e.g. very nosy structures) whle others are qute smlar (dfferng n only a few pxels): we assume that we have to encode only a small number L ε <<N n of "necessary" states of x whch are suffcent to descrbe natural mages at an acceptable error ε. L ε s called the error-bounded descrptonal complexty of the submages [BRA93b]. The man dea of transform codng s to derve an optmzed error-bounded representaton y = (y 1,,y n ) T of x accordng to the mage statstcs,.e. y has to encode the L ε necessary states of x as effcent as possble. Consequently, we demand the relaton L ε Q < N n () where Q denotes the number of dfferent values that can be assgned to a component y 1. The determnaton of the Q at a gven error ε s a non-trval task whch wll not be addressed n ths paper. Instead, from an opposte pont of vew, we ask for the reconstructon error ε at gven numbers Q,.e. at a gven quantzaton of the y. In the prevous secton we used the (vrtual) varance of a component y to decde whether ts quantzaton number was set to Q =56 or to Q =1. But varance can tell us even more about "mportance": n case of the PCA or the DCT (Dscrete Cosne Transform) t s well-known that decreasng the quantzaton number Q (.e. the resoluton) of a component y wth low varance reduces the overall encodng amount wthout affectng the reconstructon error perceved by the human vsual system. Ths s why PCA or DCT components wth lower varance are encoded at coarser resoluton, and the same should hold for ICA. To prove the dea we used the k = 16,,1 ICA and PCA components of the prevous secton. The ICA com- Fg.7 The reconstrucon by 17 ICA components 1 Note that the margnal entropy of a component y wll not ncrease f Q s decreased; furthermore, y wll be set to a constant value (e.g. zero) f Q =1. 1

13 ponents were scaled wth the recprocal norm of the assocated base vectors to set ther former unt varance to the vrtual varance n order to be comparable to the PCA components. Snce the coeffcents of both the PCA and the scaled ICA lay wthn an nterval I = [I mn,i max ] R we unformly dvded I nto Q subntervals I q of same length; the quantzaton was done by assgnng each (PCA or ICA) coeffcent c I q the arthmetcal mean of I q. After ths procedure me made the followng observatons: The boundares I mn and I max of I were gven by the smallest and the bggest coeffcent of the PCA component wth hghest varance. The components y wth low varance were encoded wth lower relatve resoluton than the components wth hgh varance because the length of the quantzaton ntervals were not adapted to the range of the y. We computed both the MSE and the cumulated entropy for dfferent k and Q. Fg.8 shows the resultng MSE as a functon of the entropy MSE Q=3 Q=64 Q=18 Q=19 Q= cumulated entropy PCA (k=16) ICA (k=16) PCA (k=) ICA (k=) Fg.8 The MSE as a functon of the cumulated entropy at dfferent quantzaton levels Q For both the PCA and the ICA the functonal dependency of reconstructon error and cumulated entropy s approxmately the same f k s equal. As n secton Prncpal ndependent components and rate dstorton theory, for the same amount of cumulated entropy t s possble to encode about one ICA component more than PCA components snce the margnal entropy of the ICA components s lower. Note that for k=16 components and quantzaton level Q=56 the MSE and the entropy are lower f we use more components (k=) at lower resoluton (Q=64). Accordng to ths observaton we may state that "varety" s more mportant than "accuracy",.e. to reduce the reconstructon error we should encode more components nstead of ncreasng the quantzaton resoluton. The systematc nvestgaton of ths behavor s subject to future research. 4 Dscusson In ths paper we showed that the concept of ndependent components known n Prncpal Component Analyss (PCA) can be enlarged to cover also the occurrence probabltes and the nformaton content of an Independent Component Analyss (ICA). Whereas the ICA approach assumes contnuous source channels mxed up to the same number of channels by a mxng matrx, we appled the ICA to mages assumng that they are composed by only a few mage prmtves. Certanly, the components wth the hghest probablty are also the ones whch should not be neglected. As shown n secton 3., ths corresponds roughly to the mean squared error nduced by neglectng the components, but s not dentcal to t. Theses components can be termed the Prncpal Independent Components PIC. For dstnctve mages, e.g. characters ths dea gves us the most mportant ones. Nevertheless, for natural mages we have no a-pror probabltes. Usng the ICA components wth most of the nformaton dd not lead to an acceptable reproducton error. The stuaton changed when we appled the ICA transform to the frst prncpal PCA components whch resulted n a compact and robust encodng. Ths approach combnes the tradtonal prncpal component crtera of PCA wth the ndependence property of ICA. It turned out that ths defnton of PIC mplements the classcal demand of the rate dstorton theory of Shannon. 5 References [ACY96] S.Amar, A.Cchock, H.Yang: A New Learnng Algorthm for Blnd Sgnal Separaton; Advances n Neural Informaton Processng Systems 8, Touretzky, Mozer, Hasselmo (Eds.), pp , MIT Press (1996) and avalable by [BES96] A.J.Bell, T.J.Sejnowsk: Edges are the `ndependent components' of natural scenes; Int. Conf. Advances n Neural Informaton Processng Systems NIPS 96, MIT press (1996). [BRA93a] R.Brause: A Symmetrcal Lateral Inhbted Network for PCA and Feature Decorrelaton; Proc. Int. Conf. Art. Neural Networks ICANN-93, pp , Sprnger Verlag (1993) [BRA93b] R.Brause: The Error-Bounded Descrptonal Complexty of Approxmaton Networks; Neural Networks, Vol.6, pp (1993) 13

14 [BRA94] R.Brause, A VLSI-Desgn of the Mnmum Entropy Neuron; J. Delgado-Fras, W. Moore (Eds.): VLSI for Artfcal Intellgence and Neural Networks, pp.53 6, Plenum Press (1994) [BRA96] R.Brause: Sensor Encodng Usng Lateral Inhbted, Self-organzed Cellular Neural Networks; Neural Networks, Vol.9, No.1, pp.99 1, (1996) [BRA98] R.Brause, M.Rppl: Nose Suppressng Sensor Encodng and Neural Sgnal Orthonormalzaton; accepted by IEEE Trans. on Neural Networks [BUR9] G.Burel: Blnd Separaton of Sources: A Nonlnear Neural Algorthm; Neural Networks, Vol. 5, pp (199) [COM94] P.Comon: Independent Component Analyss a new concept?; Sgnal Processng, Vol.36, pp (1994) [CHR9] B.Chtprasert, K.Rao: Human Vsual Weghted Progressve Image Transmsson; IEEE Trans.Comm., Vol.38, No.7, pp (199) [DEO96] G.Deco, D.Obradovc: An Informaton- Theoretc Approach to Neural Computng; Sprnger Verlag (1996) [HYO96] A.Hyvärnen, E.Oja: Independent Component Analyss by General Non-lnear Hebban-lke Rules; Helsnk Unversty of Technology, Dep. of Comp. Sc., Report A41 (1996) also avalable by [OJA9] E.Oja: Prncpal components, mnor components, and lnear neural networks; Neural Networks, Vol.5, pp (199) [OLS96a] B.A.Olshausen, D.J.Feld: Emergence of smple-cell receptve feld propertes by learnng a sparse code for natural mages; Nature 381, pp (1996) [OLS96b] B.A.Olshausen, D.J.Feld: Natural Image Statstcs and Effcent Codng; Network: Computaton n Neural Systems, No. 7, pp (1996) [PLUM93] M.Plumbley: Effcent Informaton Transfer and Ant-Hebban Neural Networks; Neural Networks, Vol.6, pp (1993) [SIL91] F.Slva, L.Almeda: A dstrbuted soluton for data orthonormalzaton; T.Kohonen et. al. (Eds.): Artfcal Neural Networks, Elsever Sc. Publ. (1991) [SHA49] C.E.Shannon, W.Weaver: The Mathematcal Theory of Informaton; Unversty of Illnos Press, Urbana (1949) 14