Optimal Kernels for Unsupervised Learning

Transcription

1 Optimal Kernels for Unsupervised Learning Sepp Hochreiter and Klaus Obermaer Bernstein Center for Computational Neuroscience and echnische Universität Berlin 587 Berlin, German Abstract We investigate the optimal kernel for sample-based model selection in unsupervised learning if maimum likelihood approaches are intractable. Given a set of training data and a set of data generated b the model, two kernel densit estimators are constructed. A model is selected through gradient descent w.r.t. the model parameters on the integrated squared difference between the densit estimators. Firstl we prove that convergence is optimal, i.e. that the cost function has onl one global minimum w.r.t. the locations of the model samples, if and onl if the kernel in the reparametried cost function is a Coulomb kernel. As a consequence, Gaussian kernels commonl used for densit estimators are suboptimal. Secondl we show that the absolute value of the difference between model and reference densit convergences at least with /t. Finall, we appl the new methods to distribution free ICA and to nonlinear ICA. I. INRODUCION Unsupervised learning methods are often based on the so-called generative model approach. In this approach, one usuall considers a parameteried famil of probabilit distributions for the observable data. Model selection is tpicall performed using the likelihood of the training data or the Baes posterior as a selection criterion. Eamples for generative approaches are abundant, ranging from factor analsis [6], ICA [5], miture models [8], and Boltmann machines [3]. Here we consider classes of generative models, where a set of hidden causes is responsible for the generation of an observation (Fig. ). he hidden causes assume causes ( ) Fig.. = f (, w) generation inference he generative model framework. ( ) values according to a probabilit distribution (). Ever cause vector is then transformed via a nonlinear function f(, w), parameteried b a weight vector w, to generate an observation. For the following we denote the distribution of the generated b (). If a model has been selected, then its most common application is inference, i.e. the reconstruction of the source values which have generated an observation. In order to unambiguousl infer the source values, however, the inverse function f (, w) must eist and must be computable. Model selection is often performed using a maimum likelihood method. In order to select the optimal set of parameters, the likelihood of an observation, () = d δ ( f(, w)) (), () is calculated and the likelihood of the full set { i }, i =,..., N of, L = N i= ( i, is maimied. If an inverse function eists and can be analticall calculated, then the integral in eq. () can be evaluated and one obtains () = df d ( f () ). () he straightforward application of the maimum likelihood (ML) method therefore requires the knowledge of the inverse function f (, w). Since the inverse function is also necessar for inference one ma argue that it ma be more adequate to parameterie the inverse function f rather than the function f which describes the data generation process. his is, for eample, done in man ICA applications. However, there eist problems (i) for which the inverse function does not eist (man-to-one mappings, e.g. in the case of incomplete measurements) or (ii) for which prior knowledge eists onl for the generation process. For those cases, ML methods either fail or ma be computationall intractable. here is another potential pitfall for the ML methods. Even if the inverse function is given, evaluation of eq. () requires knowledge of the cause densities (). If the are not known or onl partiall known one has to resort to approimations (as in ICA). In order to overcome above mentioned problems we suggest to use a sample based method for model selection. Let us first consider the case that the source densities are known but that the generative function f cannot easil be inverted. In this case we suggest to generate a sample {} of causes, and to adjust the parameters w of the generative function f such that the location of the corresponding sample {} of aligns with the observed data as good as possible. For the

2 case that the source densities are onl partiall known, but an inverse function f can be constructed, we suggest to generate a set {} of b application of the inverse function f to the set {}. he parameters of the inverse function must then be adjusted in a wa, that the set of aligns with a set of reference as good as possible. hese two scenarios are depicted in Fig..! = reference model = f (,w) target model = f (,v) target model data! = Fig.. Sample based methods for model selection. op: he source densities are given, but f cannot be inverted. Model samples are generated through cause samples. Bottom: f can be computed and used for inference, but is not full known. But what is the optimal wa of doing the alignment? Here we suggest to endow the data points of the two sets with positive and negative electric charges, and to use a learning dnamics driven b Coulomb forces to move the generated data points to their correct position. Note, that movement is not free but constrained b the underling changes in the model parameters w and v. We show that this procedure corresponds to the minimiation of the quadratic difference between two kernel densit estimators for the densities () and (), which are constructed from the sample locations. We then prove, that the choice of Coulomb s law is optimal in the sense, that the quadratic difference has onl one global minimum w.r.t. the locations of the model samples. We finall show that the method provides ecellent results when applied to ICA problems. Note that due to above mentioned optimalit properties the Coulomb interaction is wa superior compared to interactions derived from a standard Gaussian kernel. II. COS FUNCIONS AND OIMIZAION Let us consider two sets of samples i, i =...N, drawn from p (.), and i, i =...N, drawn from p (.). We construct kernel densit estimators (KDE) ˆp and ˆp using a kernel k d (.,.), because we assume that the true distributions are unknown or cannot be evaluated. Model selection, i.e. the selection of the parameters w, is then performed minimiing the integrated squared difference (ISD) F between both estimators: F(k d ) = F (ˆp (.; k d ), ˆp (.; k d )) = Φ (a; k d )da, (3) where Φ(a; k d ) := ˆp (a; k d ) ˆp (a; k d ) = (4) N ( k d a, i ) N i= N ( k d a, i ). N In the following, we will call Φ the potential function. If F = then the estimate of the model output distribution is equal to the estimate of the reference distribution, and our goal of learning is reached. Minimiation of eq. (3), however, requires the evaluation of an integral which could be computationall epensive. We, therefore, define another kernel k(.,.), k (a, b) = k d (a, c)k d (b, c)dc, (5) i= for which we obtain a simpler epression F(k) = F(k d ), F(k) = F (ˆp (.; k), ˆp (.; k)) := (6) N Φ( i ; k) N Φ( i ; k) = N N i= i= N N N k ( i, j) N N k ( i, j) + N i= j= N i= j= N N k ( i, j). N i= j= In the following we will call F(k) the energ function. We now define the positive (semi)definiteness of a kernel: Definition : A kernel k : R is called positive semidefinite, if for all N N and,..., N the matri K : K ij = k( i, j ) is positive semidefinite. If k is positive semidefinite then F, and we obtain the following theorem: heorem (Equivalence of energ and ISD): Suppose the data is contained in a subset R d. Let k d, k : R be kernels for which (*) k (a, b) = k d (a, c)k d (b, c)dc. hen the equalit F(k d ) = F(k) holds if (A) k d given: (*) converges. (B) k given: is compact; k is smmetric, continuous, and positive semidefinite. roof (sketch). (A) is straightforward and for (B) we use Mercers theorem: a, b there eists an epansion k(a, b) = n= λ n e n (a) e n (b), n : λ n, for which convergence is absolute and uniform. λ n and e n are the eigenvalues and eigenfunctions of the k-induced Hilbert- Schmidt operator. We define a, b : k d (a, b) := n= (λ n) e n (a) e n (b). k d L ( ) because k induces a trace class (nuclear) operator with trace n= λ n = k(a, a)da (cf. [9], p. 67). heorem offers a big advantage. It sas that for ever smmetric, continuous, and positive semidefinite kernel k(.,.)

3 there eist a kernel k d for the densit estimate. herefore, it suffices to select k(.,.), and there is no need for performing the integration in eq. (3). But what kernel k(.,.) should be selected? We will provide an answer to this question in Section III. he cost function F can be minimied b gradient descent. We obtain w = ǫ w F = ǫ N ( ) i E ( i), (7) N w i= where ǫ is the learning rate, i / w is the Jacobian of i = f( i ; w), and if = /N E( i ) = = /N iφ ( i) (Φ(.) Φ (.; k)). We will call E(a) := a Φ (a) the field at a. III. OIMAL KERNELS AND CONVERGENCE ROERIES In order to perform the analsis of the learning rule we consider the continuous case, i.e. the case where the number of samples goes to infinit. Let ρ(a) := p (a) p (a) be the difference between the distributions p (.) and p (.) from which the samples are drawn. hen the potential and the energ is given b Φ(a) := ρ(b) k (a, b) db, F (ρ) := ρ(a) Φ(a) da = ρ(a) ρ(b) k (a, b) db da. We now consider a simpler optimiation problem than stated in eq. (7). Let us consider the optimiation of F as a function of the sample locations under the assumption, that samples can move freel and are not constrained b the underling model f(.; w). Using the continuit equation [7] ρ = (ρ v) (8) for particle densities, with particles moving with velocit v = a F = sign(ρ(a)) E, we obtain ρ(a) = sign(ρ(a)) (ρ(a) E(a)) = (9) ( ρ(a) E(a)). Let ρ = ma a ρ(a) be the maimum norm. In order to anale the convergence properties we define: Definition (Uniform learning convergence): Learning converges uniforml if ρ (t) U(t) for t, where U is a positive strictl monotonous decreasing function of time t with lim t U(t) =. At the global maimum a ma of ρ : a ρ(a ma ) = and eq. () reduces to ρ(a ma ) = ρ(a ma ) (E(a ma )). () Uniform learning convergence requires that at the global maimum a ma sign( ρ(a ma )) = sign(ρ(a ma )) and, therefore, sign( (E(a ma ))) = sign(ρ(a ma )). he net theorem characteries the kernels for which uniform learning convergence is obtained. heorem (oisson Equation): Assume that the kernel k(a, b) : R is continuousl differentiable, smmetric, and positive definite and that a k (a, b) L ( ). Assume further that forces are smmetric: a k(a, b) = b k(a, b). If uniform convergence holds for each ρ then k must be of the following form: k can be partitioned into kernels k = l k U(λ l ), where the U(λ l ) form a partition of and the k U(λl ) obe the following Dirichlet problems on U(λ l ) (oisson equation): a ( k U(λ l )(a, b)) = λ l δ U(λl )(a b), () where δ U(λl ) is the delta function restricted to U(λ l ) and λ l. roof. ijcnnsupplementar.pdf provides the proof. he most important outcome of heorem is that uniform convergence implies (under weak assumption on the kernel k) that k must obe eq. (). Other kernels do not allow uniform convergence for arbitrar ρ. If a kernel is chosen, which does not fulfill eq. (), additional local optima ma be introduced, and gradient based optimiation methods lead to inferior optimiation results. Clearl, those kernels should be avoided. If a proper kernel is chosen it follows from uniform convergence, that the cost function F has onl one global minimum w.r.t. the particle locations. All local optima of the cost function F w.r.t. the model parameters w are then onl a propert of the model class f(.; w). We now solve the Dirichlet problem eq. (). For simplicit we set U(λ l ) = U(λ) =. In order to obtain an unique solution we set k(a, b) = for a, where is the boundar. Let S (S R ) denote the surface area of the d- dimensional sphere S R at with radius R. hen the following corollar holds: Corollar (Coulomb Kernel): If () U(λ) =, () for all a the kernel is k(a, b) =, (3) is smooth, and (4) is simple connected then there eists an unique solution k for the Dirichlet problem eq. (). For = R d this solution is given b { S (S ) ln( a b ) d = k(a, b) = λ S(S ) (d ) a b d >. d roof (sketch). he corollar follows from heorem and the properties of the Dirichlet boundar problem. k U(λl ) is up to a constant factor the Green s function of the Laplace operator. he constraint is simple connected assures that for an unbounded we obtain = R d, otherwise, we can enclose a region of R d \ b a curve in through. he kernel k is is the basis of electrostatic and gives rise to forces between charged particles which obe Coulomb s law.

4 herefore we will call k a Coulomb kernel. he net theorem addresses the speed of learning convergence, still under the assumption that there are no constraints on the motion of data points (which is true for models which are sufficientl comple). Remember, that the goal of learning was to push ρ, the difference between the model output and the reference distribution, towards ero. heorem 3: For the Coulomb kernel defined above the following equation holds (t denotes the time starting at t = ): ρ (t) =. () λ t + ( ρ ()) roof (sketch). At the etremal points a: ρ(a) = ρ(a) (E(a)) = λ ρ(a) = λ ρ(a). his differential equation finishes the proof. he Coulomb kernel and the kernel k R possess a weak singularit and are not positive definite. A positive definite kernel, however, can be constructed if a b is replaced b a b + ǫ, where ǫ is a smoothing parameter. his kernels are called lummer kernels k. he are widel used in computational phsics, but have recentl also been introduced as a( useful kernel for support vector learning [4]. Because k i, i) ( does not depend on i, i.e. ik i, i) =, the learning dnamics does hardl change for small ǫ. IV. EXERIMENS: INDEENDEN COMONEN ANALYSIS Here we appl our new sample-based method to independent component analsis (ICA [5], [], []) in a framework depicted in Fig., bottom. ICA is a method that builds a representation of the observed data in which the statistical dependence between the components is minimal. ICA methods assume that the observed data have been generated b a linear miing process of source signals which are assumed to be statisticall independent from one another. he source signals should then be recovered b the ICA method. Linear ICA approaches estimate the inverse of the miing matri where an independence criterion serves as objective. Standard ICA algorithms rel on certain properties of the source densities, e.g. that the are unimodal, super-gaussian or that the have mean ero. Our approach, however, generalies these ICA approaches because it is distribution independent. It thus etends the application of ICA methods to a broader range of real world problems. In the new ICA method we repeat following steps: (.) Compute model output from. (.) Draw the target source samples. (3.) Compute electric field E. (4.) Use field E and eq. (7) to compute w. (5.) update the weights. Step. draws a sample from a distribution where the components are statisticall independent. his reference (target) distribution is constructed to be the product of the marginal distributions of the model s causes. In our numerical simulations we randoml recombine components of to generate samples from the reference distribution, i.e. each component of was obtained from an independentl, randoml chosen. Because the choice of one component of is independent from the choice of the other components we generate a proper reference distribution. A. Sub-Gaussian Source Distribution Standard independent component analsis methods work well for super-gaussian (peak) source distributions. Our distribution free algorithm, however, is also suited for sub- Gaussian source distributions, like the multimodal source distributions used in two eperiments of this section. ) 3-D Sub-Gaussian : he source distributions are N(.4,.5) + N(.8,.5), 3 N(.5,.5) + 3 N(.5,.5), 3 3 N(.8,.5) + 3 N(.4,.5) + 3 N(.,.5). hese source distribution are mied through a linear, randoml generated miing matri (matri entries are from a uniform distribution on [-,]). We then trained a linear demiing model with fied eamples and a learning rate of. for epochs. Figure 3 shows the, mitures, and recovered. he demiing result was almost perfect which is indicated b the product of the miing matri with the demiing matri which is close to a identit matri subject to permutation and scaling. Sources Mitures Recovered Sources Fig. 3. Demiing a 3-D miture of sub-gaussian distributions. Sources (first row), mitures (second row), and recovered (third row) are projected on a -D plane for visualiation. he fied demiing matri multiplied with the miing matri gives: ) 4-D Sub-Gaussian : In another eperiment we mied multimodal normal distributions with super- Gaussians. he demiing model and the parameters of the learning rule were eactl as in the previous eperiment. he are: N(.4,.) + N(.8,.), 3 N(.4,.) + 3 N(.3,.), 3 = sign( ) 4; N(, ); 4 = 3; N(, ).

5 he miing matri multiplied b the demiing matri (left) is almost a permutation matri. Standard ICA algorithms fail at this ICA task due to the multimodal source densities. B. Nonlinear Miing and De-miing Sources he miing functions f to f 3 are highl nonlinear: f () = log (3 + )( ), f () = ( + ep ( )) ( ), f 3 () = ( 5 + 3) ( ). For demiing we used a sigmoid 3-laered neural network with 5 hidden units. We trained epochs with a learning rate of.. he results depicted in Figure 4 are good given the fact that nonlinear ICA ma not have a unique solution, and the results are much better than results obtained b simple linear models if the independence is measured b the entrop. he improved performance compared to the linear model results from large weights in the nonlinear neural network, which produce useful nonlinearities for approimating the inverse miing function. Mitures ACKNOWLEDGMENS his work was funded in part b BMBF project no Recovered Sources Fig. 4. Demiing a 3-D nonlinear superimposure of 3. he figure shows projected (top row), mitures (center row), and recovered (bottom row). o demonstrate that our approach also works for nonlinear miing problems we applied it to a 3-D miture task. Here our goal was to etract the. he are N(.4,.) + N(.8,.); N(.,.), 3 3 N(.5,.) + 3 N(.5,.). REFERENCES [] A. J. Bell and. J. Sejnowski. An information-maimiation approach to blind separation and blind deconvolution. Neural Computation, 7(6):9 59, 995. [] J.-F. Cardoso and A. Souloumiac. Blind beamforming for non Gaussian signals. IEE roceedings-f, 4(6):36 37, 993. [3] G. E. Hinton and. E. Sejnowski. Learning and relearning in Boltmann machines. In arallel Distributed rocessing, volume, pages MI ress, Cambridge, MA, 986. [4] S. Hochreiter, M. C. Moer, and K. Obermaer. Coulomb classifiers: Generaliing support vector machines via an analog to electrostatic sstems. In S. Beckers, S. hrun, and K. Obermaer, editors, Advances in Neural Information rocessing Sstems 5, pages MI ress, Cambridge, MA, 3. [5] A. Hvärinen. Surve on independent component analsis. Neural Computing Surves, :94 8, 999. [6] K. G. Jöreskog. Some contributions to maimum likelihood factor analsis. schometrika, 3:443 48, 967. [7] M. Schwart. rinciples of Electrodnamics. Dover ublications, NY, 987. Republication of McGraw-Hill Book 97. [8] D. M. itterington, A. F. M. Smith, and U. E. Makov. Statistical Analsis of Finite Miture Distributions. Wile, 985. [9] D. Werner. Funktionalanalsis. Springer-Verlag, Berlin, 3. edition,.