1 Introduction Consider the following: given a cost function J (w) for the parameter vector w = [w1 w2 w n ] T, maximize J (w) (1) such that jjwjj = C
|
|
- Solomon Singleton
- 5 years ago
- Views:
Transcription
1 On Gradient Adaptation With Unit-Norm Constraints Scott C. Douglas 1, Shun-ichi Amari 2, and S.-Y. Kung 3 1 Department of Electrical Engineering, Southern Methodist University Dallas, Texas USA 2 Laboratory for Information Synthesis, RIKEN Brain Science Institute Wako-shi, Saitama JAPAN 3 Department of Electrical Engineering, Princeton University Princeton, New Jersey USA Abstract{ In this correspondence, we describe gradient-based adaptive algorithms within parameter spaces that are specied by jjwjj = 1, where jj jj is any vector norm. We provide several algorithm forms and relate them to true gradient procedures via their geometric structures. We also give algorithms that mitigate an inherent numerical instability for L2-norm-constrained optimization tasks. Simulations showing the performance of the techniques for independent component analysis are provided. submitted to IEEE TRANSACTIONS ON SIGNAL PROCESSING SP Paper No Revised August 29, 1999 Please address correspondence to: Scott C. Douglas, Department of Electrical Engineering, School of Engineering and Applied Science, Southern Methodist University, P.O. Box , Dallas, TX USA. Voice: (214) FAX: (214) Electronic mail address: douglas@seas.smu.edu. World Wide Web URL: 0 This work was supported in part by the Oce of Research and Development under Contract No. 98F Permission of the IEEE to publish this abstract separately is granted.
2 1 Introduction Consider the following: given a cost function J (w) for the parameter vector w = [w1 w2 w n ] T, maximize J (w) (1) such that jjwjj = C; (2) where jjwjj is any vector norm and C is a positive constant. This problem forms the basis for many useful tasks in communications, control, numerical analysis, signal processing, and statistics [1]{[18]. Note that (2) imposes a geometric structure to the parameter space. Consider jjwjj p = nx! 1=p jw i j p ; (3) i=1 the L p norm of w, where 1 p 1. When p = 2, the parameter vectors satisfying (2) form an n-dimensional hypersphere of radius C. When p = 1, (2) denes an n-dimensional hyperpolyhedron with vertices Ce i = [0 0 C 0 0] T. When p = 1, (2) denes an n-dimensional hypercube. This paper considers gradient-based iterative algorithms for solving (1){(2). To our knowledge, a general comparison of such approaches has not been provided in the signal processing literature. In the case of the L2-norm parameter constraint, we also investigate the numerical issues surrounding these gradient methods and describe self-stabilized methods that implicitly maintain (2) without periodic renormalization, additional penalty terms, and costly divides or square roots. An application to minimum-kurtosis independent component analysis (ICA) is provided. 2 General Forms of Gradient Algorithms Gradient algorithms for (1){(2) have dierent forms depending on how the constraint is imposed. We give general algorithm forms in this section. Without loss of generality, set C = 1, and (w) g(k) = (k) = @w n T w=w(k) be the scaled gradient of J (w) in Euclidean space evaluated at w = w(k), where (k) is a positive step size parameter. We also dene h g (k) = I? v(k)vt (k) jjv(k)jj 2 2! g(k) and w=w(k) ; (5) where jjg(k)jj2 < 1. Geometrically, h g (k) is tangent to the surface jjwjj = 1 at w = w(k) and is called the tangent gradient of J (w) in the constraint space. In dierential geometry, the set of all such vectors is known as the tangent space of the surface jjwjj = 1 at w(k) [17]. 1
3 Lagrange Multiplier Method [1]. Dene the augmented cost function bj (w) = J (w) + jjwjj; (6) where the Lagrange multiplier is chosen to satisfy jjwjj = 1. Then, one update for w(k) is b w(k + 1) = w(k) + J (w(k)) = w(k) + g(k) + (k)v(k); where (k) = (k). In this case, the sequence (k) should satisfy lim k!1 jjw(k)jj = 1. If jjw(k)jj = 1 is imposed at each k and if such a solution exists, (k) satises jjw(k) + g(k) + (k)v(k)jj = 1: (8) Consider the case where jjwjj = jjwjj2 is the L2 norm. Then, v(k) = w(k) in (5), and (7) is w(k + 1) = (k)w(k) + g(k); (9) where (k) = 1 + (k) for convenience. The value of (k) satisfying jjw(k + 1)jj2 = 1 is (k) = q 1? jjg(k)jj [wt (k)g(k)] 2? w T (k)g(k): (10) In this algorithm, w(k) is rotated to w(k + 1) in the direction of h g (k), by an angle (k), where the form of (k) is given in the rst entry of Table 1. Coecient Normalization Method [3, 4]. The well-known gradient update w(k + 1) = w(k) + g(k) (11) performs unconstrained maximization of J (w). The coecient normalization method employs a two-step update, to maintain jjw(k)jj = 1 at each iteration. w(k + 1) = w(k) + g(k) (12) w(k + 1) = w(k + 1) jjw(k + 1)jj ; (13) Equation (13) normalizes the length of w(k + 1) without changing the direction of w(k + 1) in n-dimensional space. To reduce computational complexity, one can often employ (12) with w(k) = w(k) for several iterations and invoke (13) infrequently. In addition, this method can be employed for any (k) and choice of norm, although a small value of (k) is usually required for stochastic gradient implementations. If jjwjj = jjwjj2 is chosen, (12){(13) can be written compactly as w(k + 1) = w(k) + g(k) q 1 + 2w T (k)g(k) + jjg(k)jj 2 : (14) This update is also in the form of a rotation of w(k) in the direction of h g (k), where the angle of rotation is listed in the second entry of Table 1. 2
4 Tangent Gradient Method [2]. The constraint jjwjj = 1 restricts the space of parameter vectors to those that lie on the surface of an n-dimensional geometric object (e.g., a hypersphere, hyperpolyhedron, hypercube, etc.). Can the direction of g(k) be modied so that its integrated value lies on the constraint surface? The following theorem yields one possible solution to this problem, the proof of which appears in [18]. Theorem 1: with nite L2-norm. Then, Let jjwjj denote any dierentiable vector norm, and let g(t) be any vector function dw(t) dt = h g (t) = I? v(t)v(t)t jjv(t)jj 2 2! g(t) (15) with jjw(0)jj = 1 denes a vector function w(t) that satises jjw(t)jj = 1 for all t 0. To obtain a useful algorithm from (15), substitute time dierences for time dierentials to obtain w(k + 1) = w(k) + h g (k) = w(k) + g(k)? v(k) vt (k)g(k) jjv(k)jj 2 : (16) This update does not guarantee that jjw(k)jj = 1 for all k, however, and in fact jjw(k + 1)jj jjw(k)jj (17) at each iteration if (16) is used for any valid vector norm [18]. Even so, updating w(k) using h g (k) instead of g(k) as in (11) largely decreases the rate at which jjw(k)jj deviates from jjw(k)jj = 1. To stabilize this update, one still needs to infrequently normalize the length of w(k) via (13). In the case of the L2 norm, this algorithm is w(k + 1) = w(k) + g(k)? w(k) wt (k)g(k) jjw(k)jj 2 2 ; (18) which is also the natural gradient algorithm on the unit hypersphere [17]. A calculation shows that jjw(k + 1)jj 2 2 = jjw(k)jj jjh g (k)jj 2 2 ; (19) and since jjh g (k)jj2 is of O((k)), the growth of jjw(k)jj 2 2 is linear in 2 (k). If the length of w(k) is normalized at each iteration, this update is also in the form of a rotation in the direction of h g (k), and the angle of rotation is listed in the third entry of Table 1. True Gradient Method [7, 16]. true gradient adaptation procedure for (1){(2): i) Calculate the tangent gradient h g (k) in (5). Each of the previous algorithms approximates the following ii) Move a distance jjh g (k)jj2 along a geodesic of the constraint surface in the direction of h g (k). 3
5 A geodesic is a curve on the constraint surface that connects two arbitrary points by an arc of shortest length. Implementing this procedure for a given norm constraint requires knowledge of the equations of motion on the constraint surface. When jjw(k)jj2 = 1 is imposed, updating w(k) amounts to rotating w(k) by an angle (k). For any unit vector u(k) that is perpendicular to w(k), the update w(k + 1) = cos((k))w(k) + sin((k))u(k) (20) rotates w(k + 1) by an angle (k) in the direction of u(k). For gradient adaptation, we choose u(k) = h g(k) (k) = 1 (k) g(k)? w(k)w T (k)g(k) ; (21) where the form of (k) = jjh g (k)jj2 is given in the last entry of Table 1. Note that when (k) is small, tan((k)) sin((k)) (k), yielding similar angles of rotation for all four methods. 3 Implementation Issues for L 2 -Norm Constrained Methods We consider the computational complexities and numerical stabilities of L2-norm constrained gradient approaches in this section, as the L2-norm constraint is the most popular for practical applications. To illustrate the salient issues involved in algorithm design, we shall focus on J (w) = 1 p Efjy(k)jp g; (22) as an instantaneous cost function, where y(k) = w T (k)x(k), x(k) is a discrete-time vector random process, and p is a positive integer not equal to 2. This cost function arises in certain formulations of independent component analysis (ICA), blind source separation, and blind deconvolution [13]{[15]. In this case, g(k) = (k)jy(k)j p?2 y(k)x(k) and w T (k)g(k) = (k)jy(k)j p ; (23) such that the sign of w T (k)g(k) does not change for all k. The rst four rows of Table 2 list the complexities of each of the algorithms in (9){(10), (14), (18), and (20){(21) according to the number and type of operations required, neglecting those operations needed for calculating g(k). The Lagrange, coecient normalization, and tangent gradient methods are the simplest, whereas the true gradient method is signicantly more complicated. All of these methods require operations other than multiply/adds to implement, making them more dicult to implement on real-time signal processing devices that are optimized for multiply/add calculations. Note that the structure of g(k) can often be exploited to further reduce each algorithm's complexity, e.g. by using (23) to compute w T (k)g(k) in the case of (22). 4
6 We now consider simplications that yield similar-behaving algorithms with reduced complexities. Since all four algorithms have equivalent behavior up to O( 2 (k)), we only consider modications of one approach the tangent gradient method in (18). The modied versions are Modication #1 [11]: w(k + 1) = w(k) + jjw(k)jj 2 2g(k)? w(k)w T (k)g(k); (24) Modication #2 [6]: w(k + 1) = w(k) + g(k)? w(k)w T (k)g(k) (25) Modication #3 [12, 15]: w(k + 1) = w(k) + jjw(k)jj 4 2g(k)? w(k)w T (k)g(k): (26) If jjw(k)jj 2 2 = 1, all three modied methods have similar behaviors to that of (18); however, the numerical properties of the algorithms in the radial dimension associated with the length of w(k) are quite dierent, as we shall show. For simplicity of discussion, we dene c(k) = jjw(k)jj 2 2? 1: (27) If c(k) experiences unmitigated growth, the associated algorithm is numerically-unstable. Numerical Stability of (24). " jjw(k + 1)jj 2 2 = Pre-multiplying both sides of (24) by their transposes yields 1 + jjw(k)jj 2 2g T (k) I? w(k)wt (k) jjw(k)jj 2 2! g(k) # jjw(k)jj 2 2 : (28) The matrix in large parentheses on the RHS of (28) is a projection matrix. Hence, so long as g(k) is not collinear with w(k) and jjg(k)jj2 > 0, then jjw(k + 1)jj2 > jjw(k)jj2 if jjw(0)jj2 = 1, i.e. numerical instability. Furthermore, since jjw(k)jj 2 2 appears in the factor in brackets on the RHS of (28), (24) causes accelerated growth in jjw(k)jj 2 independently of the form of J (w). Numerical Stability of (25). one from both sides, and rearranging terms, we obtain h i c(k + 1) = 1? 2w T (k)g(k) c(k) + Pre-multiplying both sides of (25) by their transposes, subtracting g(k)? w(k)w T (k)g(k) 2 2 : (29) If jjw(0)jj2 = 1, then we have c(k) > 0 for all k if g(k) 6= w(0)w T (0)g(k). Since the second term on the RHS of (29) is non-negative, we require w T (k)g(k) > 0 (30) for the numerical stability of (25). Note that (30) can often be veried for a particular J (w). For example, if (22) is chosen with a positive sign, then (30) is always true, making this algorithm appropriate for constrained maximization of p?1 Efjy(k)j p g. Conversely, the algorithm fails when p?1 Efjy(k)j p g is being minimized for all jjwjj2 = 1. This result is what justies f(23),(25)g for use in maximum-kurtosis ICA and principal component analysis tasks, and it also explains why this algorithm fails for minimum-kurtosis ICA and minor component analysis tasks [12, 15]. 5
7 Numerical Stability of (26). an update for c(k) as h c(k + 1) = 1 + 2jjw(k)jj 2 i 2w T (k)g(k) c(k) + In this case, if Performing a similar analysis of (26) as used previously, we obtain jjw(k)jj 4 2 g(k)? w(k)w T (k)g(k) 2 2 : (31) w T (k)g(k) < 0; (32) then (26) is numerically-stable. For minimum-kurtosis ICA (p = 4) tasks, choosing g(k) =?(k)jy(k)j p?2 y(k)x(k) causes this algorithm to perform in a stable fashion. Other Approaches. Of the three methods in (24){(26), (25) is the simplest, requiring 2n multiply/adds at each iteration. When w T (k)g(k) < 0, however, this method is numerically-unstable. As an alternative to (26), one can monitor the stability behavior of (25) and rescue the system from instability just prior to its occurence. It can be shown from (29) that jjw(k)jj 2 > 2 is an indicator of the onset of sudden divergence of (25) [18]. Moreover, simulations of (25) in several situations indicate that jjw(k)jj 2 2 > 2 provides a reliable divergence indicator when wt (k)g(k) < 0. One can therefore monitor the value of jjw(k)jj 2 2, and renormalize w(k) to unit length when jjw(k)jj 2 2 > C, where 1 < C < 2. Simulations indicate that values of C in the range 1:1 C 1:5 often yield good performance for typical problems and choices of (k). In addition, since jjw(k)jj 2 2 tends to grow slowly for small values of (k), the test jjw(k)jj 2 > C need only be performed at every Lth iteration where L 1, such that the complexity associated with this rescue method can be minimized. Further details regarding this approach, along with alternative reduced parameterization gradient methods for (1){(2), can be found in [18]. 4 Simulations We now explore the behaviors of the algorithms in (18), (25), and (26) in a single-component ICA task via MATLAB simulations. Let x(k) = As(k); (33) where A is an (n n) constant mixing matrix and s(k) = [s1(k) s n (k)] T contains unobservable independent components. If AA T = I, then maximizing J (w) =?0:25Efjy(k)j 4 g for y(k) = w T (k)x(k) and jjw(k)jj2 = 1 results in a negative-kurtosis signal in y(k) [14, 15]. one can guarantee AA T In practice, = I by prewhitening the signal measurements using simple adaptive procedures [9, 19]. We can then choose g(k) =?(k)jy(k)j 2 y(k)x(k) for any of the previouslydiscussed algorithms to obtain a candidate algorithm for this task. 6
8 For our simulations, we generate s(k) = [s1(k) s2(k) s3(k)] T, where s1(k) is an i.i.d. binaryf1g-distributed signal and s2(k) and s3(k) are i.i.d. Laplacian-distributed signals with p.d.f. p s (s) = e?p 2jsj = p 2. The mixing matrix A is chosen to be the eigenvector matrix of R xx = :9 0:4 0:7 0:4 0:3 0:5 0:7 0:5 1: ; (34) such that measurement prewhitening is not needed. For each algorithm, one hundred simulations have been run, and the average values of the performance factors (k) = jjc 1 (k)jj 2 2 =jjc 2(k)jj 2 2 and (k) = [jjw(k)jj 2 2? 1] 2 (35) have been computed, where c i (k) = E T i w(k) and E 1 and E 2 contain the two- and one-dimensional subspaces corresponding to the signal directions of the Laplacian and binary sources. Figure 1(a) shows the evolution of (k) for the tangent gradient method in (18), the simplied method in (25) with stabilization, and the self-normalized method in (26) for this task, where (k) = 0:001, L = 20, and C = 1:1. All three algorithms are successful at extracting the binary source from the linear mixture. Figure 1(b) shows the average evolution of (k) for the three methods, where the unmitigated growth in jjw(k)jj2 for the tangent gradient method is clearly evident. In contrast, the stabilization procedure used for (25) maintains this algorithm's stable behavior, and (26) performs in a stable, self-normalizing manner without such intervention. 5 Conclusions In this paper, we have presented an overview of algorithms that adjust a parameter vector to minimize or maximize a chosen cost function under a unit-norm parameter vector constraint. Particular attention has been paid both to methods that guarantee the unit-norm constraint at each iteration and to methods that maintain jjw(k)jj2 1 over time. Simulations verify the useful behavior of the schemes for independent component analysis. Some extensions of these results to multiple dimensions can be found in [12]. References [1] R. Courant and D. Hilbert, Methods of Mathematical Physics, vol. I (New York: Interscience, 1953). [2] T.P. Krasulina, \Method of stochastic approximation in the determination of the largest eigenvalue of the mathematical expectation of random matrices," Automat. Remote Contr., vol. 2, pp , [3] N.L. Owsley, \Adaptive data orthonormalization," Proc. IEEE Int. Conf. Acoust., Speech, Signal Processing, Tulsa, OK, pp , Apr
9 [4] P.A. Thompson, \An adaptive spectral analysis technique for unbiased frequency estimation in the presence of white noise," Proc. 13th Asilomar Conf. Circ., Syst., Comput., Pacic Grove, CA, pp , Nov [5] V.U. Reddy, B. Egardt, and T. Kailath, \Least squares type algorithm for adaptive implementation of Pisarenko's harmonic retrieval method," IEEE Trans. Acoust., Speech, Signal Processing, vol. 30, pp , June [6] E. Oja, \A simplied neural model as a principal component analyzer," J. Math. Biol., vol. 15, pp , [7] D.R. Fuhrmann and B. Liu, \An iterative algorithm for locating the minimal eigenvector of a symmetric matrix," Proc. IEEE Int. Conf. Acoust., Speech, Signal Processing, Dallas, TX, pp , [8] L. Wang and J. Karhunen, \A unied natural bigradient algorithm for robust PCA and MCA," Int. J. Neural Syst., vol. 7, no. 1, pp , Mar [9] K.I. Diamantaras and S.-Y. Kung, Principal Component Neural Networks: Theory and Applications (New York: Wiley, 1996). [10] V. Solo and X. Kong, \Performance analysis of adaptive eigenanalysis algorithms," IEEE Trans. Signal Processing, vol. 46, pp , Mar [11] T.-P. Chen, S. Amari, and Q. Lin, \A unied algorithm for principal and minor components extraction," Neural Networks, vol. 11, pp , Apr [12] S.C. Douglas, S.-Y. Kung, and S. Amari, \A self-stabilized minor subspace rule," IEEE Signal Processing Lett., vol. 5, pp , Dec [13] P. Comon, \Independent component analysis: A new concept?" Signal Processing, vol. 36, no. 3, pp , Apr [14] A. Hyvarinen and E. Oja, \Independent component analysis by general nonlinear Hebbian-like learning rules," Signal Processing, vol. 64, no. 3, pp , Feb [15] S.C. Douglas and S.-Y. Kung, \KuicNet algorithms for blind deconvolution," Proc. IEEE Workshop Neural Networks Signal Processing, Cambridge, UK, pp. 3-12, Aug [16] S.T. Smith, Geometric Optimization Methods for Adaptive Filtering, Ph.D. thesis, Harvard Univ., Cambridge, MA, [17] S.C. Douglas and S. Amari, \Natural gradient adaptation," in Unsupervised Adaptive Filtering, Vol. I: Blind Source Separation, S. Haykin, ed. (New York: Wiley, 1999), Chap. 2. [18] S.C. Douglas, S. Amari, and S.-Y. Kung, \Gradient adaptation with unit-norm constraints," Tech. Rep. EE , Dept. of Electrical Engineering, Southern Methodist University, Dallas, TX, Feb [19] S.C. Douglas and A. Cichocki, \Neural networks for blind decorrelation of signals," IEEE Trans. Signal Processing, vol. 45, pp , Nov
10 List of Tables Table 1: Angles of rotation for L2-norm constrained adaptation. Table 2: Summary of algorithms for L2-norm constrained adaptation. List of Figures Figure 1: Average performances of the various algorithms in the minimum-kurtosis ICA task: (a) evolutions of (k), and (b) evolutions of (k). 9
11 Algorithm Angle q of Rotation Lagrange sin((k)) = jjg(k)jj 2? [wt (k)g(k)] s 2 jjg(k)jj 2 Normalized sin((k)) =? (wt (k)g(k)) 2 q1 + 2w T (k)g(k) + jjg(k)jj 2 Tangent tan((k)) = jjg(k)jj 2? [wt (k)g(k)] q 2 True Gradient (k) = jjg(k)jj 2? [wt (k)g(k)] 2 Table 1: Angles of rotation for L2-norm constrained adaptation. Algorithm Complexity Stability Stabilization w new + p cos() Behavior Method p w new = w + g; = 1? jjgjj [wt g] 2? w T g w w + g new = p 1 + 2w T g + jjgjj 2 2 w g new = 1? wt jjwjj 2 2 3n + 1 3n n 2n? w + g 3n 3n? w new = cos w + sin h g; 5n 4n? h g = g? ww T g; = jjh g jj 2 w? new = 1? w T g w + jjwjj 2 2g 4n 3n? w new =? 1? w T g w + g 2n 2n w new =? 1? w T g w + jjwjj 4 2g 4n + 1 3n? Stable; jjwjj 2 = 1 Stable; jjwjj 2 = 1 Slow growth Set jjwjj 2 = 1 in jjwjj 2 infrequently Stable; jjwjj 2 = 1 Accelerated Set jjwjj 2 = 1 growth in jjwjj 2 periodically jjwjj 2 1 if w T g > 0 jjwjj 2 1 if w T g < 0 If w T g < 0, set jjwjj 2 = 1 when jjwjj 2 2 > C, 1:1 C 1:5 Table 2: Summary of algorithms for L2-norm constrained adaptation. 10
12 (a) 10 5 rho(k) 10 0 Eqn. (18) Eqn. (25) w/rescue Eqn. (26) (b) 10 2 Eqn. (18) Eqn. (25) w/rescue Eqn. (26) eta(k) number of iterations Figure 1: Average performances of the various algorithms in the minimum-kurtosis ICA task: (a) evolutions of (k), and (b) evolutions of (k). 11
ON SOME EXTENSIONS OF THE NATURAL GRADIENT ALGORITHM. Brain Science Institute, RIKEN, Wako-shi, Saitama , Japan
ON SOME EXTENSIONS OF THE NATURAL GRADIENT ALGORITHM Pando Georgiev a, Andrzej Cichocki b and Shun-ichi Amari c Brain Science Institute, RIKEN, Wako-shi, Saitama 351-01, Japan a On leave from the Sofia
More informationMULTICHANNEL BLIND SEPARATION AND. Scott C. Douglas 1, Andrzej Cichocki 2, and Shun-ichi Amari 2
MULTICHANNEL BLIND SEPARATION AND DECONVOLUTION OF SOURCES WITH ARBITRARY DISTRIBUTIONS Scott C. Douglas 1, Andrzej Cichoci, and Shun-ichi Amari 1 Department of Electrical Engineering, University of Utah
More information1 Introduction Independent component analysis (ICA) [10] is a statistical technique whose main applications are blind source separation, blind deconvo
The Fixed-Point Algorithm and Maximum Likelihood Estimation for Independent Component Analysis Aapo Hyvarinen Helsinki University of Technology Laboratory of Computer and Information Science P.O.Box 5400,
More informationORIENTED PCA AND BLIND SIGNAL SEPARATION
ORIENTED PCA AND BLIND SIGNAL SEPARATION K. I. Diamantaras Department of Informatics TEI of Thessaloniki Sindos 54101, Greece kdiamant@it.teithe.gr Th. Papadimitriou Department of Int. Economic Relat.
More informationPerformance Comparison of Two Implementations of the Leaky. LMS Adaptive Filter. Scott C. Douglas. University of Utah. Salt Lake City, Utah 84112
Performance Comparison of Two Implementations of the Leaky LMS Adaptive Filter Scott C. Douglas Department of Electrical Engineering University of Utah Salt Lake City, Utah 8411 Abstract{ The leaky LMS
More informationNatural Gradient Learning for Over- and Under-Complete Bases in ICA
NOTE Communicated by Jean-François Cardoso Natural Gradient Learning for Over- and Under-Complete Bases in ICA Shun-ichi Amari RIKEN Brain Science Institute, Wako-shi, Hirosawa, Saitama 351-01, Japan Independent
More informationx 1 (t) Spectrogram t s
A METHOD OF ICA IN TIME-FREQUENCY DOMAIN Shiro Ikeda PRESTO, JST Hirosawa 2-, Wako, 35-98, Japan Shiro.Ikeda@brain.riken.go.jp Noboru Murata RIKEN BSI Hirosawa 2-, Wako, 35-98, Japan Noboru.Murata@brain.riken.go.jp
More informationRecursive Generalized Eigendecomposition for Independent Component Analysis
Recursive Generalized Eigendecomposition for Independent Component Analysis Umut Ozertem 1, Deniz Erdogmus 1,, ian Lan 1 CSEE Department, OGI, Oregon Health & Science University, Portland, OR, USA. {ozertemu,deniz}@csee.ogi.edu
More informationNon-Euclidean Independent Component Analysis and Oja's Learning
Non-Euclidean Independent Component Analysis and Oja's Learning M. Lange 1, M. Biehl 2, and T. Villmann 1 1- University of Appl. Sciences Mittweida - Dept. of Mathematics Mittweida, Saxonia - Germany 2-
More informationA Cross-Associative Neural Network for SVD of Nonsquared Data Matrix in Signal Processing
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 5, SEPTEMBER 2001 1215 A Cross-Associative Neural Network for SVD of Nonsquared Data Matrix in Signal Processing Da-Zheng Feng, Zheng Bao, Xian-Da Zhang
More information1 Introduction Blind source separation (BSS) is a fundamental problem which is encountered in a variety of signal processing problems where multiple s
Blind Separation of Nonstationary Sources in Noisy Mixtures Seungjin CHOI x1 and Andrzej CICHOCKI y x Department of Electrical Engineering Chungbuk National University 48 Kaeshin-dong, Cheongju Chungbuk
More informationRobust extraction of specific signals with temporal structure
Robust extraction of specific signals with temporal structure Zhi-Lin Zhang, Zhang Yi Computational Intelligence Laboratory, School of Computer Science and Engineering, University of Electronic Science
More informationOptimum Sampling Vectors for Wiener Filter Noise Reduction
58 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 1, JANUARY 2002 Optimum Sampling Vectors for Wiener Filter Noise Reduction Yukihiko Yamashita, Member, IEEE Absact Sampling is a very important and
More informationTo appear in Proceedings of the ICA'99, Aussois, France, A 2 R mn is an unknown mixture matrix of full rank, v(t) is the vector of noises. The
To appear in Proceedings of the ICA'99, Aussois, France, 1999 1 NATURAL GRADIENT APPROACH TO BLIND SEPARATION OF OVER- AND UNDER-COMPLETE MIXTURES L.-Q. Zhang, S. Amari and A. Cichocki Brain-style Information
More informationPOLYNOMIAL SINGULAR VALUES FOR NUMBER OF WIDEBAND SOURCES ESTIMATION AND PRINCIPAL COMPONENT ANALYSIS
POLYNOMIAL SINGULAR VALUES FOR NUMBER OF WIDEBAND SOURCES ESTIMATION AND PRINCIPAL COMPONENT ANALYSIS Russell H. Lambert RF and Advanced Mixed Signal Unit Broadcom Pasadena, CA USA russ@broadcom.com Marcel
More informationConstrained Projection Approximation Algorithms for Principal Component Analysis
Constrained Projection Approximation Algorithms for Principal Component Analysis Seungjin Choi, Jong-Hoon Ahn, Andrzej Cichocki Department of Computer Science, Pohang University of Science and Technology,
More informationBlind Extraction of Singularly Mixed Source Signals
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL 11, NO 6, NOVEMBER 2000 1413 Blind Extraction of Singularly Mixed Source Signals Yuanqing Li, Jun Wang, Senior Member, IEEE, and Jacek M Zurada, Fellow, IEEE Abstract
More informationAverage Reward Parameters
Simulation-Based Optimization of Markov Reward Processes: Implementation Issues Peter Marbach 2 John N. Tsitsiklis 3 Abstract We consider discrete time, nite state space Markov reward processes which depend
More informationA Robust PCA by LMSER Learning with Iterative Error. Bai-ling Zhang Irwin King Lei Xu.
A Robust PCA by LMSER Learning with Iterative Error Reinforcement y Bai-ling Zhang Irwin King Lei Xu blzhang@cs.cuhk.hk king@cs.cuhk.hk lxu@cs.cuhk.hk Department of Computer Science The Chinese University
More informationA Subspace Approach to Estimation of. Measurements 1. Carlos E. Davila. Electrical Engineering Department, Southern Methodist University
EDICS category SP 1 A Subspace Approach to Estimation of Autoregressive Parameters From Noisy Measurements 1 Carlos E Davila Electrical Engineering Department, Southern Methodist University Dallas, Texas
More informationA NATURAL GRADIENT CONVOLUTIVE BLIND SOURCE SEPARATION ALGORITHM FOR SPEECH MIXTURES
A NATURAL GRADIENT CONVOLUTIVE BLIND SOURCE SEPARATION ALGORITHM FOR SPEECH MIXTURES Xiaoan Sun and Scott C. Douglas Department of Electrical Engineering Southern Methodist University Dallas, Texas 75275
More informationCONTROL SYSTEMS ANALYSIS VIA BLIND SOURCE DECONVOLUTION. Kenji Sugimoto and Yoshito Kikkawa
CONTROL SYSTEMS ANALYSIS VIA LIND SOURCE DECONVOLUTION Kenji Sugimoto and Yoshito Kikkawa Nara Institute of Science and Technology Graduate School of Information Science 896-5 Takayama-cho, Ikoma-city,
More informationTWO METHODS FOR ESTIMATING OVERCOMPLETE INDEPENDENT COMPONENT BASES. Mika Inki and Aapo Hyvärinen
TWO METHODS FOR ESTIMATING OVERCOMPLETE INDEPENDENT COMPONENT BASES Mika Inki and Aapo Hyvärinen Neural Networks Research Centre Helsinki University of Technology P.O. Box 54, FIN-215 HUT, Finland ABSTRACT
More informationA METHOD OF ICA IN TIME-FREQUENCY DOMAIN
A METHOD OF ICA IN TIME-FREQUENCY DOMAIN Shiro Ikeda PRESTO, JST Hirosawa 2-, Wako, 35-98, Japan Shiro.Ikeda@brain.riken.go.jp Noboru Murata RIKEN BSI Hirosawa 2-, Wako, 35-98, Japan Noboru.Murata@brain.riken.go.jp
More informationAlgorithms for Nonnegative Independent Component Analysis
Algorithms for Nonnegative Independent Component Analysis Mark D Plumbley IEEE Transactions on Neural Networks, 4(3), 534-543, May 2003 Abstract We consider the task of solving the independent component
More informationOn Mean Curvature Diusion in Nonlinear Image Filtering. Adel I. El-Fallah and Gary E. Ford. University of California, Davis. Davis, CA
On Mean Curvature Diusion in Nonlinear Image Filtering Adel I. El-Fallah and Gary E. Ford CIPIC, Center for Image Processing and Integrated Computing University of California, Davis Davis, CA 95616 Abstract
More informationA Coupled Helmholtz Machine for PCA
A Coupled Helmholtz Machine for PCA Seungjin Choi Department of Computer Science Pohang University of Science and Technology San 3 Hyoja-dong, Nam-gu Pohang 79-784, Korea seungjin@postech.ac.kr August
More informationICA [6] ICA) [7, 8] ICA ICA ICA [9, 10] J-F. Cardoso. [13] Matlab ICA. Comon[3], Amari & Cardoso[4] ICA ICA
16 1 (Independent Component Analysis: ICA) 198 9 ICA ICA ICA 1 ICA 198 Jutten Herault Comon[3], Amari & Cardoso[4] ICA Comon (PCA) projection persuit projection persuit ICA ICA ICA 1 [1] [] ICA ICA EEG
More informationOne-unit Learning Rules for Independent Component Analysis
One-unit Learning Rules for Independent Component Analysis Aapo Hyvarinen and Erkki Oja Helsinki University of Technology Laboratory of Computer and Information Science Rakentajanaukio 2 C, FIN-02150 Espoo,
More informationSPARSE REPRESENTATION AND BLIND DECONVOLUTION OF DYNAMICAL SYSTEMS. Liqing Zhang and Andrzej Cichocki
SPARSE REPRESENTATON AND BLND DECONVOLUTON OF DYNAMCAL SYSTEMS Liqing Zhang and Andrzej Cichocki Lab for Advanced Brain Signal Processing RKEN Brain Science nstitute Wako shi, Saitama, 351-198, apan zha,cia
More informationCovariance and Correlation Matrix
Covariance and Correlation Matrix Given sample {x n } N 1, where x Rd, x n = x 1n x 2n. x dn sample mean x = 1 N N n=1 x n, and entries of sample mean are x i = 1 N N n=1 x in sample covariance matrix
More informationSubmitted to Electronics Letters. Indexing terms: Signal Processing, Adaptive Filters. The Combined LMS/F Algorithm Shao-Jen Lim and John G. Harris Co
Submitted to Electronics Letters. Indexing terms: Signal Processing, Adaptive Filters. The Combined LMS/F Algorithm Shao-Jen Lim and John G. Harris Computational Neuro-Engineering Laboratory University
More informationAPPLICATION OF RECURRENT NEURAL NETWORK USING MATLAB SIMULINK IN MEDICINE
ITALIAN JOURNAL OF PURE AND APPLIED MATHEMATICS N. 39 2018 (23 30) 23 APPLICATION OF RECURRENT NEURAL NETWORK USING MATLAB SIMULINK IN MEDICINE Raja Das Madhu Sudan Reddy VIT Unversity Vellore, Tamil Nadu
More informationFEATURE EXTRACTION USING SUPERVISED INDEPENDENT COMPONENT ANALYSIS BY MAXIMIZING CLASS DISTANCE
FEATURE EXTRACTION USING SUPERVISED INDEPENDENT COMPONENT ANALYSIS BY MAXIMIZING CLASS DISTANCE Yoshinori Sakaguchi*, Seiichi Ozawa*, and Manabu Kotani** *Graduate School of Science and Technology, Kobe
More informationIntroduction to Independent Component Analysis. Jingmei Lu and Xixi Lu. Abstract
Final Project 2//25 Introduction to Independent Component Analysis Abstract Independent Component Analysis (ICA) can be used to solve blind signal separation problem. In this article, we introduce definition
More informationEquivalence Probability and Sparsity of Two Sparse Solutions in Sparse Representation
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 19, NO. 12, DECEMBER 2008 2009 Equivalence Probability and Sparsity of Two Sparse Solutions in Sparse Representation Yuanqing Li, Member, IEEE, Andrzej Cichocki,
More informationLinear Prediction Based Blind Source Extraction Algorithms in Practical Applications
Linear Prediction Based Blind Source Extraction Algorithms in Practical Applications Zhi-Lin Zhang 1,2 and Liqing Zhang 1 1 Department of Computer Science and Engineering, Shanghai Jiao Tong University,
More informationLearning with Ensembles: How. over-tting can be useful. Anders Krogh Copenhagen, Denmark. Abstract
Published in: Advances in Neural Information Processing Systems 8, D S Touretzky, M C Mozer, and M E Hasselmo (eds.), MIT Press, Cambridge, MA, pages 190-196, 1996. Learning with Ensembles: How over-tting
More informationLinear Regression and Its Applications
Linear Regression and Its Applications Predrag Radivojac October 13, 2014 Given a data set D = {(x i, y i )} n the objective is to learn the relationship between features and the target. We usually start
More informationUndercomplete Independent Component. Analysis for Signal Separation and. Dimension Reduction. Category: Algorithms and Architectures.
Undercomplete Independent Component Analysis for Signal Separation and Dimension Reduction John Porrill and James V Stone Psychology Department, Sheeld University, Sheeld, S10 2UR, England. Tel: 0114 222
More information= w 2. w 1. B j. A j. C + j1j2
Local Minima and Plateaus in Multilayer Neural Networks Kenji Fukumizu and Shun-ichi Amari Brain Science Institute, RIKEN Hirosawa 2-, Wako, Saitama 35-098, Japan E-mail: ffuku, amarig@brain.riken.go.jp
More informationMassoud BABAIE-ZADEH. Blind Source Separation (BSS) and Independent Componen Analysis (ICA) p.1/39
Blind Source Separation (BSS) and Independent Componen Analysis (ICA) Massoud BABAIE-ZADEH Blind Source Separation (BSS) and Independent Componen Analysis (ICA) p.1/39 Outline Part I Part II Introduction
More informationFundamentals of Principal Component Analysis (PCA), Independent Component Analysis (ICA), and Independent Vector Analysis (IVA)
Fundamentals of Principal Component Analysis (PCA),, and Independent Vector Analysis (IVA) Dr Mohsen Naqvi Lecturer in Signal and Information Processing, School of Electrical and Electronic Engineering,
More informationBlind Source Separation with a Time-Varying Mixing Matrix
Blind Source Separation with a Time-Varying Mixing Matrix Marcus R DeYoung and Brian L Evans Department of Electrical and Computer Engineering The University of Texas at Austin 1 University Station, Austin,
More informationEstimation of linear non-gaussian acyclic models for latent factors
Estimation of linear non-gaussian acyclic models for latent factors Shohei Shimizu a Patrik O. Hoyer b Aapo Hyvärinen b,c a The Institute of Scientific and Industrial Research, Osaka University Mihogaoka
More informationBLIND SEPARATION OF INSTANTANEOUS MIXTURES OF NON STATIONARY SOURCES
BLIND SEPARATION OF INSTANTANEOUS MIXTURES OF NON STATIONARY SOURCES Dinh-Tuan Pham Laboratoire de Modélisation et Calcul URA 397, CNRS/UJF/INPG BP 53X, 38041 Grenoble cédex, France Dinh-Tuan.Pham@imag.fr
More informationIndependent Component Analysis and Its Application on Accelerator Physics
Independent Component Analysis and Its Application on Accelerator Physics Xiaoying Pang LA-UR-12-20069 ICA and PCA Similarities: Blind source separation method (BSS) no model Observed signals are linear
More informationAn Improved Cumulant Based Method for Independent Component Analysis
An Improved Cumulant Based Method for Independent Component Analysis Tobias Blaschke and Laurenz Wiskott Institute for Theoretical Biology Humboldt University Berlin Invalidenstraße 43 D - 0 5 Berlin Germany
More informationBlind Deconvolution via Maximum Kurtosis Adaptive Filtering
Blind Deconvolution via Maximum Kurtosis Adaptive Filtering Deborah Pereg Doron Benzvi The Jerusalem College of Engineering Jerusalem, Israel doronb@jce.ac.il, deborahpe@post.jce.ac.il ABSTRACT In this
More informationMODELS OF LEARNING AND THE POLAR DECOMPOSITION OF BOUNDED LINEAR OPERATORS
Eighth Mississippi State - UAB Conference on Differential Equations and Computational Simulations. Electronic Journal of Differential Equations, Conf. 19 (2010), pp. 31 36. ISSN: 1072-6691. URL: http://ejde.math.txstate.edu
More informationDictionary Learning for L1-Exact Sparse Coding
Dictionary Learning for L1-Exact Sparse Coding Mar D. Plumbley Department of Electronic Engineering, Queen Mary University of London, Mile End Road, London E1 4NS, United Kingdom. Email: mar.plumbley@elec.qmul.ac.u
More informationStatistical Pattern Recognition
Statistical Pattern Recognition Feature Extraction Hamid R. Rabiee Jafar Muhammadi, Alireza Ghasemi, Payam Siyari Spring 2014 http://ce.sharif.edu/courses/92-93/2/ce725-2/ Agenda Dimensionality Reduction
More informationwhere A 2 IR m n is the mixing matrix, s(t) is the n-dimensional source vector (n» m), and v(t) is additive white noise that is statistically independ
BLIND SEPARATION OF NONSTATIONARY AND TEMPORALLY CORRELATED SOURCES FROM NOISY MIXTURES Seungjin CHOI x and Andrzej CICHOCKI y x Department of Electrical Engineering Chungbuk National University, KOREA
More informationComparative Performance Analysis of Three Algorithms for Principal Component Analysis
84 R. LANDQVIST, A. MOHAMMED, COMPARATIVE PERFORMANCE ANALYSIS OF THR ALGORITHMS Comparative Performance Analysis of Three Algorithms for Principal Component Analysis Ronnie LANDQVIST, Abbas MOHAMMED Dept.
More informationHOPFIELD neural networks (HNNs) are a class of nonlinear
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: EXPRESS BRIEFS, VOL. 52, NO. 4, APRIL 2005 213 Stochastic Noise Process Enhancement of Hopfield Neural Networks Vladimir Pavlović, Member, IEEE, Dan Schonfeld,
More informationShort-Time ICA for Blind Separation of Noisy Speech
Short-Time ICA for Blind Separation of Noisy Speech Jing Zhang, P.C. Ching Department of Electronic Engineering The Chinese University of Hong Kong, Hong Kong jzhang@ee.cuhk.edu.hk, pcching@ee.cuhk.edu.hk
More informationNew Lyapunov Krasovskii functionals for stability of linear retarded and neutral type systems
Systems & Control Letters 43 (21 39 319 www.elsevier.com/locate/sysconle New Lyapunov Krasovskii functionals for stability of linear retarded and neutral type systems E. Fridman Department of Electrical
More informationMaximum variance formulation
12.1. Principal Component Analysis 561 Figure 12.2 Principal component analysis seeks a space of lower dimensionality, known as the principal subspace and denoted by the magenta line, such that the orthogonal
More informationCS281 Section 4: Factor Analysis and PCA
CS81 Section 4: Factor Analysis and PCA Scott Linderman At this point we have seen a variety of machine learning models, with a particular emphasis on models for supervised learning. In particular, we
More informationRemaining energy on log scale Number of linear PCA components
NONLINEAR INDEPENDENT COMPONENT ANALYSIS USING ENSEMBLE LEARNING: EXPERIMENTS AND DISCUSSION Harri Lappalainen, Xavier Giannakopoulos, Antti Honkela, and Juha Karhunen Helsinki University of Technology,
More informationNON-NEGATIVE SPARSE CODING
NON-NEGATIVE SPARSE CODING Patrik O. Hoyer Neural Networks Research Centre Helsinki University of Technology P.O. Box 9800, FIN-02015 HUT, Finland patrik.hoyer@hut.fi To appear in: Neural Networks for
More informationBlind deconvolution of dynamical systems using a balanced parameterized state space approach
University of Wollongong Research Online Faculty of Informatics - Papers (Archive) Faculty of Engineering and Information Sciences 2003 Blind deconvolution of dynamical systems using a balanced parameterized
More informationSUBSPACE TRACKING FOR SIGNAL PROCESSING
CHAPTER 42 SUBSPACE TRACKING FOR SIGNAL PROCESSING 42.1 INTRODUCTION Research in subspace and component-based techniques were originated in Statistics in the middle of the last century through the problem
More informationIndependent Component Analysis
Independent Component Analysis James V. Stone November 4, 24 Sheffield University, Sheffield, UK Keywords: independent component analysis, independence, blind source separation, projection pursuit, complexity
More informationOn Information Maximization and Blind Signal Deconvolution
On Information Maximization and Blind Signal Deconvolution A Röbel Technical University of Berlin, Institute of Communication Sciences email: roebel@kgwtu-berlinde Abstract: In the following paper we investigate
More informationPost-Nonlinear Blind Extraction in the Presence of Ill-Conditioned Mixing Wai Yie Leong and Danilo P. Mandic, Senior Member, IEEE
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I: REGULAR PAPERS, VOL. 55, NO. 9, OCTOBER 2008 2631 Post-Nonlinear Blind Extraction in the Presence of Ill-Conditioned Mixing Wai Yie Leong and Danilo P. Mandic,
More informationNon-Negative Matrix Factorization And Its Application to Audio. Tuomas Virtanen Tampere University of Technology
Non-Negative Matrix Factorization And Its Application to Audio Tuomas Virtanen Tampere University of Technology tuomas.virtanen@tut.fi 2 Contents Introduction to audio signals Spectrogram representation
More informationThe Rationale for Second Level Adaptation
The Rationale for Second Level Adaptation Kumpati S. Narendra, Yu Wang and Wei Chen Center for Systems Science, Yale University arxiv:1510.04989v1 [cs.sy] 16 Oct 2015 Abstract Recently, a new approach
More informationADAPTIVE LATERAL INHIBITION FOR NON-NEGATIVE ICA. Mark Plumbley
Submitteed to the International Conference on Independent Component Analysis and Blind Signal Separation (ICA2) ADAPTIVE LATERAL INHIBITION FOR NON-NEGATIVE ICA Mark Plumbley Audio & Music Lab Department
More informationx x2 H11 H y y2 H22 H
.5.5 5 5.5.5 5 5 2 3 4 5 6 7 H 2 3 4 5 6 7 H2 5 5 x 5 5 x2.5.5.5.5.5.5.5.5 2 3 4 5 6 7 H2 2 3 4 5 6 7 H22.5.5 y.5.5 y2 Figure : Mixing coef.: H [k], H 2 [k], H 2 [k], H 22 [k]. Figure 3: Observations and
More informationChapter 6 Effective Blind Source Separation Based on the Adam Algorithm
Chapter 6 Effective Blind Source Separation Based on the Adam Algorithm Michele Scarpiniti, Simone Scardapane, Danilo Comminiello, Raffaele Parisi and Aurelio Uncini Abstract In this paper, we derive a
More informationOptimal Mean-Square Noise Benefits in Quantizer-Array Linear Estimation Ashok Patel and Bart Kosko
IEEE SIGNAL PROCESSING LETTERS, VOL. 17, NO. 12, DECEMBER 2010 1005 Optimal Mean-Square Noise Benefits in Quantizer-Array Linear Estimation Ashok Patel and Bart Kosko Abstract A new theorem shows that
More informationcomputation of the algorithms it is useful to introduce some sort of mapping that reduces the dimension of the data set before applying signal process
Optimal Dimension Reduction for Array Processing { Generalized Soren Anderson y and Arye Nehorai Department of Electrical Engineering Yale University New Haven, CT 06520 EDICS Category: 3.6, 3.8. Abstract
More informationBLIND SEPARATION USING ABSOLUTE MOMENTS BASED ADAPTIVE ESTIMATING FUNCTION. Juha Karvanen and Visa Koivunen
BLIND SEPARATION USING ABSOLUTE MOMENTS BASED ADAPTIVE ESTIMATING UNCTION Juha Karvanen and Visa Koivunen Signal Processing Laboratory Helsinki University of Technology P.O. Box 3, IN-215 HUT, inland Tel.
More informationConstrained Independent Component Analysis
Constrained Independent Component Analysis Wei Lu and Jagath C. Rajapakse School of Computer Engineering Nanyang Technological University, Singapore 639798 email: asjagath@ntu.edu.sg Abstract The paper
More informationMULTICHANNEL SIGNAL PROCESSING USING SPATIAL RANK COVARIANCE MATRICES
MULTICHANNEL SIGNAL PROCESSING USING SPATIAL RANK COVARIANCE MATRICES S. Visuri 1 H. Oja V. Koivunen 1 1 Signal Processing Lab. Dept. of Statistics Tampere Univ. of Technology University of Jyväskylä P.O.
More informationMODELLING OF FLEXIBLE MECHANICAL SYSTEMS THROUGH APPROXIMATED EIGENFUNCTIONS L. Menini A. Tornambe L. Zaccarian Dip. Informatica, Sistemi e Produzione
MODELLING OF FLEXIBLE MECHANICAL SYSTEMS THROUGH APPROXIMATED EIGENFUNCTIONS L. Menini A. Tornambe L. Zaccarian Dip. Informatica, Sistemi e Produzione, Univ. di Roma Tor Vergata, via di Tor Vergata 11,
More informationControlling Hopf Bifurcations: Discrete-Time Systems
Discrete Dynamics in Nature and Society, Vol. 5, pp. 29-33 Reprints available directly from the publisher Photocopying permitted by license only (C) 2000 OPA (Overseas Publishers Association) N.V. Published
More informationPCA & ICA. CE-717: Machine Learning Sharif University of Technology Spring Soleymani
PCA & ICA CE-717: Machine Learning Sharif University of Technology Spring 2015 Soleymani Dimensionality Reduction: Feature Selection vs. Feature Extraction Feature selection Select a subset of a given
More information1. Introduction Let the least value of an objective function F (x), x2r n, be required, where F (x) can be calculated for any vector of variables x2r
DAMTP 2002/NA08 Least Frobenius norm updating of quadratic models that satisfy interpolation conditions 1 M.J.D. Powell Abstract: Quadratic models of objective functions are highly useful in many optimization
More informationPolynomial functions over nite commutative rings
Polynomial functions over nite commutative rings Balázs Bulyovszky a, Gábor Horváth a, a Institute of Mathematics, University of Debrecen, Pf. 400, Debrecen, 4002, Hungary Abstract We prove a necessary
More informationbelow, kernel PCA Eigenvectors, and linear combinations thereof. For the cases where the pre-image does exist, we can provide a means of constructing
Kernel PCA Pattern Reconstruction via Approximate Pre-Images Bernhard Scholkopf, Sebastian Mika, Alex Smola, Gunnar Ratsch, & Klaus-Robert Muller GMD FIRST, Rudower Chaussee 5, 12489 Berlin, Germany fbs,
More informationLie Groups for 2D and 3D Transformations
Lie Groups for 2D and 3D Transformations Ethan Eade Updated May 20, 2017 * 1 Introduction This document derives useful formulae for working with the Lie groups that represent transformations in 2D and
More information(z) UN-1(z) PN-1(z) Pn-1(z) - z k. 30(z) 30(z) (a) (b) (c) x y n ...
Minimum Memory Implementations of the Lifting Scheme Christos Chrysas Antonio Ortega HewlettPackard Laboratories Integrated Media Systems Center 151 Page Mill Road, Bldg.3U3 University of Southern California
More informationARTEFACT DETECTION IN ASTROPHYSICAL IMAGE DATA USING INDEPENDENT COMPONENT ANALYSIS. Maria Funaro, Erkki Oja, and Harri Valpola
ARTEFACT DETECTION IN ASTROPHYSICAL IMAGE DATA USING INDEPENDENT COMPONENT ANALYSIS Maria Funaro, Erkki Oja, and Harri Valpola Neural Networks Research Centre, Helsinki University of Technology P.O.Box
More informationH 1 optimisation. Is hoped that the practical advantages of receding horizon control might be combined with the robustness advantages of H 1 control.
A game theoretic approach to moving horizon control Sanjay Lall and Keith Glover Abstract A control law is constructed for a linear time varying system by solving a two player zero sum dierential game
More informationEigenface-based facial recognition
Eigenface-based facial recognition Dimitri PISSARENKO December 1, 2002 1 General This document is based upon Turk and Pentland (1991b), Turk and Pentland (1991a) and Smith (2002). 2 How does it work? The
More informationLeast Squares Optimization
Least Squares Optimization The following is a brief review of least squares optimization and constrained optimization techniques. Broadly, these techniques can be used in data analysis and visualization
More informationLinearly-solvable Markov decision problems
Advances in Neural Information Processing Systems 2 Linearly-solvable Markov decision problems Emanuel Todorov Department of Cognitive Science University of California San Diego todorov@cogsci.ucsd.edu
More informationNeural Learning in Structured Parameter Spaces Natural Riemannian Gradient
Neural Learning in Structured Parameter Spaces Natural Riemannian Gradient Shun-ichi Amari RIKEN Frontier Research Program, RIKEN, Hirosawa 2-1, Wako-shi 351-01, Japan amari@zoo.riken.go.jp Abstract The
More informationIterative face image feature extraction with Generalized Hebbian Algorithm and a Sanger-like BCM rule
Iterative face image feature extraction with Generalized Hebbian Algorithm and a Sanger-like BCM rule Clayton Aldern (Clayton_Aldern@brown.edu) Tyler Benster (Tyler_Benster@brown.edu) Carl Olsson (Carl_Olsson@brown.edu)
More informationUsing Kernel PCA for Initialisation of Variational Bayesian Nonlinear Blind Source Separation Method
Using Kernel PCA for Initialisation of Variational Bayesian Nonlinear Blind Source Separation Method Antti Honkela 1, Stefan Harmeling 2, Leo Lundqvist 1, and Harri Valpola 1 1 Helsinki University of Technology,
More informationIndependent Component Analysis and Its Applications. By Qing Xue, 10/15/2004
Independent Component Analysis and Its Applications By Qing Xue, 10/15/2004 Outline Motivation of ICA Applications of ICA Principles of ICA estimation Algorithms for ICA Extensions of basic ICA framework
More informationProblem Set 9 Due: In class Tuesday, Nov. 27 Late papers will be accepted until 12:00 on Thursday (at the beginning of class).
Math 3, Fall Jerry L. Kazdan Problem Set 9 Due In class Tuesday, Nov. 7 Late papers will be accepted until on Thursday (at the beginning of class).. Suppose that is an eigenvalue of an n n matrix A and
More informationOle Christensen 3. October 20, Abstract. We point out some connections between the existing theories for
Frames and pseudo-inverses. Ole Christensen 3 October 20, 1994 Abstract We point out some connections between the existing theories for frames and pseudo-inverses. In particular, using the pseudo-inverse
More informationNMF WITH SPECTRAL AND TEMPORAL CONTINUITY CRITERIA FOR MONAURAL SOUND SOURCE SEPARATION. Julian M. Becker, Christian Sohn and Christian Rohlfing
NMF WITH SPECTRAL AND TEMPORAL CONTINUITY CRITERIA FOR MONAURAL SOUND SOURCE SEPARATION Julian M. ecker, Christian Sohn Christian Rohlfing Institut für Nachrichtentechnik RWTH Aachen University D-52056
More informationCitation Ieee Signal Processing Letters, 2001, v. 8 n. 6, p
Title Multiplierless perfect reconstruction modulated filter banks with sum-of-powers-of-two coefficients Author(s) Chan, SC; Liu, W; Ho, KL Citation Ieee Signal Processing Letters, 2001, v. 8 n. 6, p.
More informationThe model reduction algorithm proposed is based on an iterative two-step LMI scheme. The convergence of the algorithm is not analyzed but examples sho
Model Reduction from an H 1 /LMI perspective A. Helmersson Department of Electrical Engineering Linkoping University S-581 8 Linkoping, Sweden tel: +6 1 816 fax: +6 1 86 email: andersh@isy.liu.se September
More informationMultiple Similarities Based Kernel Subspace Learning for Image Classification
Multiple Similarities Based Kernel Subspace Learning for Image Classification Wang Yan, Qingshan Liu, Hanqing Lu, and Songde Ma National Laboratory of Pattern Recognition, Institute of Automation, Chinese
More information