On the Contractive Nature of Autoencoders: Application to Missing Sensor Restoration

Transcription

1 On the Contactive Natue of Autoencodes: Application to Missing Senso Restoation Benjain B. Thopson, Robet J. Maks II, and Mohaed A. El-Shakawi Coputational Intelligence Applications (CIA) Laboatoy, Depatent of Electical Engineeing, Univesity of Washington, Seattle, WA Abstact The neual netwok autoencode is a useful tool fo the estoation of issing sensos when enough known sensos with soe elation to those issing ae available. Though the idea of a contaction apping, this pape povides soe insight into the convegence of seveal iteative ethods of senso estoation using the autoencode to soe unique answe given a specific opeating point (i.e., the known senso values), egadless of how the issing senso values ae initialized I. Intoduction Pevious wok has established the ability of the autoassociative neual netwok encode (o siply autoencode ) to aid in the estoation of senso data which ay be issing o coupt, given soe sot of coelation between the nueic outputs of the vaious sensos in a syste. [1], [2] Naayanan et al. [1] descibe a ethod by which the issing senso data ay be econstucted using an iteative appoach; in this pape we show that, unde a set of conditions elating to the specific paaetes of the neual netwok, we can povide a sufficient condition fo the convegence of the iteative appoach to senso estoation. We appoach this though the idea of a contaction apping. Moeove, we will show copelling evidence that thee exists a unique point of convegence fo a fully tained autoencode given an opeating point defined by the set of known sensos, and that this convegence point should be eached egadless of how the issing sensos ae initialized. II. Contaction A contactive apping is defined [3],[4] as a apping O:X X on a coplete etic space (X, d) in which, fo any x and y in that space: ( Ox, Oy) k d( x, y) d 0 k < 1 (1) o, oe clealy, a contaction apping is one in which the output distance between two points is less than the input distance. Now let us look at this popety in R 1, whee ou etic is siply the Euclidean no, and O is siply soe functional apping f(x): f ( x) f ( y) k x y (2) Now, suppose we eplace y with x+dx, yielding, with soe ino eaangeent: f ( x) f ( x + dx) dx k the liit of which as dx 0 gives us, as a less stict equieent fo contaction ( x) df dx < 1 This idea is deonstated clealy in Figue 1. It would then be sufficient to show that, fo soe function f: R 1 R 1, the deivative of f is less than unity fo all x. Conside the Banach Fixed-Point Theoe: If f is a contactive apping, then thee exists a unique fixed point x 0 fo which f(x 0 ) = x 0. Moeove, the sequence {x n }, fo which any eleent x n+1 = f(x n ), conveges, and that convegent point is x o. With this theoe, it becoes uch cleae how any contactive tendency of the autoencode can help us show whethe o not the senso estoation pocess will convege to soe unique value. III. Missing Senso Restoation with Autoencodes Thee ae thee ethods that we will exaine fo the estoation of issing sensos. The fist is a siple application of altenating pojections onto convex sets (POCS). The second and the thid both eploy seach (3) (4) /03/$ IEEE 3011

2 f(x) d(ov,oy) d(ow,oz) d(v,y) d(w,z) Figue 1 - Fo a function whose deivative is less than unity, the input distance of two points will always be geate than the output distance (the distances pojected onto the hoizontal and vetical axes, espectively). Fo a deivative geate than one, we achieve expansion athe than contaction. Note that points v and y exist whee df(x)/dx <1, and w and z exist whee df(x)/dx >1. x v y w z W 1, W 2 W 3, x,..., { x x,..., x, x, x x } T 1, 2 M k1 k 2 kk (5) whee M is the nube of issing sensos, K is the nube of known sensos, and of couse K+M = N. Likewise, on the outputs, we have: x xˆ, xˆ,..., xˆ, xˆ, xˆ,..., xˆ (6) { } T ˆ 1 2 M k1 k 2 Figue 2 shows a single-hidden-laye autoencode with the appopiate labels. Given that the encode x is tained as a feedfowad ultilayeed pecepton (MLP) [3] we also have the following eleents: W 1,k is the atix of weights whose (i, j) th eleent is the weight connecting the i th known senso value to the j th neuon in the fist hidden laye; W 1, is the coesponding atix fo the issing sensos, b l is the vecto of bias weights fo the l th laye; W 2 is the weight atix connecting the fist hidden laye to the second; and finally, W 3,k and W 3, ae the countepats to W 1,k and W 1, on the output. Note that these can easily be extended fo an encode with oe than a single hidden laye. kk A. POCS x k b 1 W 1,k b 2 b 3 W 3,k Figue 2 - a diaga of a geneic 2-laye autoencode, with appopiate labels, as descibed in the text. techniques; the fist involves siply iniizing the eo between the issing senso inputs and outputs on the autoencode, while the second looks at the eo between the entie input patten and output patten (both issing and known sensos) to achieve a final answe. The eits of each ae discussed below. Fist, howeve, a few definitions ae in ode. The input to the neual netwok is copised of two concatenated vectos whose total diension is N, the input diension (and necessaily the output diension as well). The fist vecto, x k, can be thought of as the opeating point of the estoation pocess, and is defined as the set of known senso values fo a given input patten. The second vecto is then x, the set of issing senso values. Without loss of geneality, let us foulate the input as soe vecto x: As descibed by Naayanan et al. [1], a staightfowad ethod fo issing senso estoation using a tained autoencode is the use of POCS [5] to achieve a convegent value. Unde the assuption of convexity, ou two sets ae then a) the space defined as the output of the autoencode, and b) the set of all input pattens to the neual netwok containing x k, the known sensos, and an abitay x. While the second set is definitely convex, the fist equies the assuption of convexity. By choosing soe initial x to ceate an input vecto x, we then obtain xˆ, the output of the autoencode. This coesponds to a pojection onto the fist set, the opeato fo which we will denote P 1. We then change the outputs xˆ to x k k to pefo the pojection onto the fist set, denoted as P 2. If we altenate between these pojections, unde the assuption of convexity, the seies will convege to an answe epesenting the intesection of the two sets. Thus, a single iteation of this pocess is defined as the successive application of P 1 and then P 2. B. Unconstained Seach Because of the potential lack of convexity and othe pefoance issues, we ae otivated to find a bette ethod fo discoveing the tue point of convegence. In this case, ou iteative opeato O u is siply a single iteation of any seach algoith which seeks to iniize the eo between the issing senso values and the 3012

3 Figue 4 - a histoga of the k-values based on the taining data. Note the dynaic ange of the plot is [0.966, 1.015]. outputs of the autoencode coesponding to those issing senso values, o: ag in x xˆ (7) x This ethod allows fo geate efineent of the issing senso values ove the POCS ethod descibed above; oeove, it should be noted that, if the assuption of convexity wee tue, O u and O p would convege to the sae value, assuing the two sets intesect. C. Constained Seach A notable shotcoing of both POCS and the unconstained seach is that neithe one uses the infoation contained in x ˆ k to bette efine the final answe. Thus, we define a thid opeato, O c, which is siila to O u except that it coesponds to a seach algoith which seeks to iniize the entie output eo of the autoencode; naely: ag in x xˆ (8) x ecalling that x is a vecto coposed of x and x k. This way, we actually ensue a soothe atch between the input and the output, which can help eliinate spuious answes that, while iniizing the eo between consecutive iteations on x, tend not to ake sense in the context of the known sensos. IV. Analysis Results Figue 3 - a histoga of the k-values associated with ou autoencode as a whole, fo a andoly geneated data set. The scale of the x-axis is fo 0.2 to 1. Note that the lagest tail value is actually less than unity. x be contactive. Recall that, by definition, a pefectly tained autoencode is one fo which we have the following elationship: O ( x NN ) = x x C (9) whee O NN is the neual netwok teated as an opeato, and C is the set of all taining data. Thus, except in the case of an autoencode tained on a single patten, a pefectly tained autoencode guaantees that the Banach Fixed- Point Theoe cannot hold, and thus the opeato is not contactive. An opeato is nonexpansive in (10) when, instead of k<1, we allow k 1. A convex set othogonal pojection opeato is nonexpansive [6] so this is a esult to be expected fo the autoencode. If O is nonexpansive, the opeation x n+1 = O(x n ) will convege to a fixed point. This point, howeve, is not unique and is dependent on initialization. A. Contaction of the Entie Autoencode While we have shown that the autoencode itself is neithe stictly contactive no nonexpansive, it is infoative to see how closely it appoaches these conditions. As descibed in (3), thee is a k-value associated with a set of two inputs and thei coesponding outputs. If, fo a vey lage set of input pais, we can show that that k-value is less than o equal to one, then we have justification fo teating the opeato as nealy nonexpansive We exaine the k-values fo a specific exaple of a tained autoencode. Fo the puposes of this 3013

4 Figue 5 - aveage deivat ive of ou oveall estoat ion opeato fo a single issing senso using andolyinit ialized opeat ing point s x k pape, we have tained an autoencode on Mackey-Glass chaos, defined by the nonlinea diffeence equation [7]: x [] t [ t τ ] [ t τ ] n A θ x = + 1 n n θ + x ( B) x[] t (10) whee A, B, n, θ, and τ ae defined paaetes, along with soe x[0] value. We geneate a data set using this function, and tain a autoencode using input pattens taken as consecutive 40-point windows of the data set. All data is noalized to the inteval [0,1] befoe taining. The autoencode thus tained, we then geneate a lage (on the ode of 10 5 pattens) set of andoly geneated input vectos (fo a unifo distibution on [0,1]). We then select, at ando, two diffeent vectos fo this set as vectos x and y as pe equation (3). Fo this, we can calculate a coesponding k-value. With a sufficiently lage nube of these k-values, we can ceate estiate the pobability density function of k, to exaine how it behaves, paticulaly aound the value of 1. Figue 3 shows the esult of this expeient. Clealy we have that, fo a andoly geneated data set, we neve even appoach the liits of being contactive; that is, ou autoencode behaves statistically as though it wee in fact contactive. The lagest k-value it achieves in this siulation, in fact, is , well below the theshold beyond which it would no longe be contactive. While this deonstates the behavio of the autoencode towads andoly geneated data, we next pefo a oe inteesting expeient. Given that the autoencode is tained such that the output ios the input as closely as possible, we would expect the k-values fo the actual taining data to be vey nea 1 fo each taining patten (ecall that, fo a pefectly tained Figue 6 - sae as Fig. 5 with taining data used instead of andolyinitialized x k autoencode, k would be exactly unity fo evey single one). Thus, we have otivation to epeat the above expeient, eplacing the andoly-geneated data with the taining data itself. We see the esult of this expeient in Figue 4. Fo this histoga, we have poof that ou initial conjectue holds tue even fo this ipefect autoencode due to the k-values above 1, the opeato is not stictly nonexpansive. Howeve, it would clealy be fai to say that, fo the evidence pesented in this figue, ou opeato is nealy nonexpansive, since k neve deviates fo unity by oe than B. Contaction of Subsets of the Autoencode At this point, we then want to show that, while the autoencode as a whole is neithe stictly contactive no nonexapnsive, the autoencode at soe opeating point ay be contactive as it opeates on a subset of the input vecto; naely, x. At this point, it is useful to wite out the functional fo of the neual netwok as an opeato. Let us foulate this fo a two-hidden laye neual netwok as descibed in Figue 2, although it can easily be genealized fo geate o fewe diensions: f xˆ ( x xk ) = σ W3, σ( W2 σ( = ( 1, x xˆ k, W ) + b 2 ) 3 ) + W (11) 1, k x k + b1 + b whee σ is a vecto opeato that iposes a sigoid nonlineaity on each eleent of the applied vecto, and all othe paaetes ae as descibed above. This function 3014

5 (a) (c) (e) (g) (i) Figue 7 - histogas of k-values vaious cobinations of issing sensos. Figs. (a)-(j) coespond to 1, 5, 9, 13, 17, 21, 25, 29, 33, and 37 issing sensos, espectively. The specific issing sensos wee chosen at aondo, and the opeating point, selected fo the taining data, was the sae fo each case. epesents the functional fo of ou opeato O 1, as descibed above. Likewise, we can define out opeato O 2 as: g xˆ ( xˆ, x k ) = = T xˆ + B x k x k whee T is an N N atix in which: (12) 1 i = j M T i, j = (13) 0 else and B is an N K atix defined as: 0M K B = (14) I K Thus, cobining equations (11) and (12) yields ou opeato O p : O x, x k = T f x, x k + B x (15) p ( ) ( ) k With this, we have a faewok fo which we can exaine the contactiveness of the entie pocess. Specifically, we can look at the case in which only one senso is issing. (b) (d) (f) (h) (j) If this is the case, then x is a scala, and equation (4) can be applied. Explicitly calculating the deivative of O p is a cubesoe task, paticulaly if x is not scala. Fo the puposes of this pape, we again apply a andoly initialized siulation to show that the deivative tends to be less than one fo vaious x k opeating points. We pefo this expeient using the sae Mackey-Glass autoencode as used above. Fist, we exaine all foty senso values by andoly geneating x k (as a vecto of unifo ando vaiables on [0,1]) and calculating the coesponding deivatives. Figue 5 shows the ovelaid plot of the deivatives fo each of the foty sensos. Each cuve epesents an aveage ove ultiple ealizations of x k. The axiu standad deviation at any point fo any of the sets of cuves was as sall as , giving us a geat deal of confidence that, fo ando x k, and a single issing senso, we will always convege to a unique answe, since the less-than-unity deivative iplies contactiveness of the sub-opeato as it acts aound a fixed point. Next, we pefo the sae expeient, again eplacing the andoly initialized potion (in this case, the value of the fixed point x k ) with the actual taining data. Figue 6 displays the esults clealy, in a fo identical to Figue 5. In this case, the axiu standad deviation fo any value of x ove any set of the cuves was , which gives us even geate confidence of ou conclusion. Copaing Fig. 5 to Fig. 6, we see that they ae alost copletely indistinguishable. No diffeence is gaphically discenable. This gives us substantial eason to believe that the deivative is lagely insensitive to the actual value of the opeating point (as long as the opeating point is within the unit-cube in K diensions which is easonable since it is possible to define the valid ange of sensos to be within that liit). Finally, we pefo an expeient to deonstate the contactive chaacteistics of situations in which oe than a single senso ae issing. Fo this, we calculate a seies of k-values as above, the exception being that soe fixed-point x k is chosen, and the eaining sensos x ae andoly initialized as above. We then pefo this fo a vaiety of issing-senso configuations (obviously, all the possible peutations would take a pohibitive aount of tie to calculate even fo a elatively sall autoencode, and even oe so fo ou situation using the Mackey-Glass autoencode). Figue 7 displays these esults, fo 10 diffeent cases coesponding to 1, 5, 9, 13, 17, 21, 25, 29, 33, and 37 issing sensos. The specific sensos in each case wee selected at ando fo the 40 possible sensos. We note that, in evey single plot, we ae well below the unity theshold equied fo contaction. Moeove, it is inteesting to note that the uppe liit of the k-value sees to appoach unity gadually as the nube of issing 3015

6 sensos inceases (iplying that the opeation is oe contactive fo fewe issing sensos).. V. Conclusions [7] Glass, L. and M. C. Mackey, Fo Clocks to Chaos, The Rhyths of Life, Pinceton Univesity Pess, Pinceton, NJ, By deonstating the contactive natue of the autoencode as a ethod fo estoing issing sensos, we have given copelling evidence that such iteative pocedues will, fo the case exained, convege to a unique answe dependent only on the neual netwok autoencode itself, and the opeating point (the known senso values) about which the pocess is ipleented. We have shown that the autoencode itself is nealy nonexpansive to ost types of data, the aginal exception being the taining data itself. Finally, we have povided eason to believe that, the fewe sensos that ae issing, the oe likely the autoencode-ethod of estoing issing sensos is to have such a unique value of convegence. VI. Refeences [1] Naayanan, S., R.J. Maks II, J. L. Vian, J.J. Choi, M.A. El-Shakawi & B. B. Thopson, "Set Constaint Discovey: Missing Senso Data Restoation Using Auto-Associative Regession Machines", Poceedings of the 2002 Intenational Joint Confeence on Neual Netwoks, 2002 IEEE Wold Congess on Coputational Intelligence, May12-17, 2002, Honolulu, pp [2] Reed, R. D. and R.J. Maks II, Neual Sithing: Supevised Leaning in Feedfowad Atificial Neual Netwoks, MIT Pess, Cabidge, MA, [3] Naylo, A. W., and G. R. Sell Linea Opeato Theoy in Engineeing and Science, Spinge, New Yok City, NY, [4] Luenbege, D. G., Optiization by Vecto Space Methods, John Wiley & Sons, Apil 1997 [5] Maks, R.J. II, "Altenating Pojections onto Convex Sets", in Deconvolution of Iages and Specta, edited by Pete A. Jansson, (Acadeic Pess, San Diego, 1997), pp [6] Goldbug, M.H. and R.J. Maks II, "Signal synthesis in the pesence of an inconsistent set of constaints", IEEE Tansactions on Cicuits and Systes, vol. CAS-32 pp (1985). 3016