1 Introduction Consider the following: given a cost function J (w) for the parameter vector w = [w1 w2 w n ] T, maximize J (w) (1) such that jjwjj = C

Size: px
Start display at page:

Download "1 Introduction Consider the following: given a cost function J (w) for the parameter vector w = [w1 w2 w n ] T, maximize J (w) (1) such that jjwjj = C"

Transcription

1 On Gradient Adaptation With Unit-Norm Constraints Scott C. Douglas 1, Shun-ichi Amari 2, and S.-Y. Kung 3 1 Department of Electrical Engineering, Southern Methodist University Dallas, Texas USA 2 Laboratory for Information Synthesis, RIKEN Brain Science Institute Wako-shi, Saitama JAPAN 3 Department of Electrical Engineering, Princeton University Princeton, New Jersey USA Abstract{ In this correspondence, we describe gradient-based adaptive algorithms within parameter spaces that are specied by jjwjj = 1, where jj jj is any vector norm. We provide several algorithm forms and relate them to true gradient procedures via their geometric structures. We also give algorithms that mitigate an inherent numerical instability for L2-norm-constrained optimization tasks. Simulations showing the performance of the techniques for independent component analysis are provided. submitted to IEEE TRANSACTIONS ON SIGNAL PROCESSING SP Paper No Revised August 29, 1999 Please address correspondence to: Scott C. Douglas, Department of Electrical Engineering, School of Engineering and Applied Science, Southern Methodist University, P.O. Box , Dallas, TX USA. Voice: (214) FAX: (214) Electronic mail address: douglas@seas.smu.edu. World Wide Web URL: 0 This work was supported in part by the Oce of Research and Development under Contract No. 98F Permission of the IEEE to publish this abstract separately is granted.

2 1 Introduction Consider the following: given a cost function J (w) for the parameter vector w = [w1 w2 w n ] T, maximize J (w) (1) such that jjwjj = C; (2) where jjwjj is any vector norm and C is a positive constant. This problem forms the basis for many useful tasks in communications, control, numerical analysis, signal processing, and statistics [1]{[18]. Note that (2) imposes a geometric structure to the parameter space. Consider jjwjj p = nx! 1=p jw i j p ; (3) i=1 the L p norm of w, where 1 p 1. When p = 2, the parameter vectors satisfying (2) form an n-dimensional hypersphere of radius C. When p = 1, (2) denes an n-dimensional hyperpolyhedron with vertices Ce i = [0 0 C 0 0] T. When p = 1, (2) denes an n-dimensional hypercube. This paper considers gradient-based iterative algorithms for solving (1){(2). To our knowledge, a general comparison of such approaches has not been provided in the signal processing literature. In the case of the L2-norm parameter constraint, we also investigate the numerical issues surrounding these gradient methods and describe self-stabilized methods that implicitly maintain (2) without periodic renormalization, additional penalty terms, and costly divides or square roots. An application to minimum-kurtosis independent component analysis (ICA) is provided. 2 General Forms of Gradient Algorithms Gradient algorithms for (1){(2) have dierent forms depending on how the constraint is imposed. We give general algorithm forms in this section. Without loss of generality, set C = 1, and (w) g(k) = (k) = @w n T w=w(k) be the scaled gradient of J (w) in Euclidean space evaluated at w = w(k), where (k) is a positive step size parameter. We also dene h g (k) = I? v(k)vt (k) jjv(k)jj 2 2! g(k) and w=w(k) ; (5) where jjg(k)jj2 < 1. Geometrically, h g (k) is tangent to the surface jjwjj = 1 at w = w(k) and is called the tangent gradient of J (w) in the constraint space. In dierential geometry, the set of all such vectors is known as the tangent space of the surface jjwjj = 1 at w(k) [17]. 1

3 Lagrange Multiplier Method [1]. Dene the augmented cost function bj (w) = J (w) + jjwjj; (6) where the Lagrange multiplier is chosen to satisfy jjwjj = 1. Then, one update for w(k) is b w(k + 1) = w(k) + J (w(k)) = w(k) + g(k) + (k)v(k); where (k) = (k). In this case, the sequence (k) should satisfy lim k!1 jjw(k)jj = 1. If jjw(k)jj = 1 is imposed at each k and if such a solution exists, (k) satises jjw(k) + g(k) + (k)v(k)jj = 1: (8) Consider the case where jjwjj = jjwjj2 is the L2 norm. Then, v(k) = w(k) in (5), and (7) is w(k + 1) = (k)w(k) + g(k); (9) where (k) = 1 + (k) for convenience. The value of (k) satisfying jjw(k + 1)jj2 = 1 is (k) = q 1? jjg(k)jj [wt (k)g(k)] 2? w T (k)g(k): (10) In this algorithm, w(k) is rotated to w(k + 1) in the direction of h g (k), by an angle (k), where the form of (k) is given in the rst entry of Table 1. Coecient Normalization Method [3, 4]. The well-known gradient update w(k + 1) = w(k) + g(k) (11) performs unconstrained maximization of J (w). The coecient normalization method employs a two-step update, to maintain jjw(k)jj = 1 at each iteration. w(k + 1) = w(k) + g(k) (12) w(k + 1) = w(k + 1) jjw(k + 1)jj ; (13) Equation (13) normalizes the length of w(k + 1) without changing the direction of w(k + 1) in n-dimensional space. To reduce computational complexity, one can often employ (12) with w(k) = w(k) for several iterations and invoke (13) infrequently. In addition, this method can be employed for any (k) and choice of norm, although a small value of (k) is usually required for stochastic gradient implementations. If jjwjj = jjwjj2 is chosen, (12){(13) can be written compactly as w(k + 1) = w(k) + g(k) q 1 + 2w T (k)g(k) + jjg(k)jj 2 : (14) This update is also in the form of a rotation of w(k) in the direction of h g (k), where the angle of rotation is listed in the second entry of Table 1. 2

4 Tangent Gradient Method [2]. The constraint jjwjj = 1 restricts the space of parameter vectors to those that lie on the surface of an n-dimensional geometric object (e.g., a hypersphere, hyperpolyhedron, hypercube, etc.). Can the direction of g(k) be modied so that its integrated value lies on the constraint surface? The following theorem yields one possible solution to this problem, the proof of which appears in [18]. Theorem 1: with nite L2-norm. Then, Let jjwjj denote any dierentiable vector norm, and let g(t) be any vector function dw(t) dt = h g (t) = I? v(t)v(t)t jjv(t)jj 2 2! g(t) (15) with jjw(0)jj = 1 denes a vector function w(t) that satises jjw(t)jj = 1 for all t 0. To obtain a useful algorithm from (15), substitute time dierences for time dierentials to obtain w(k + 1) = w(k) + h g (k) = w(k) + g(k)? v(k) vt (k)g(k) jjv(k)jj 2 : (16) This update does not guarantee that jjw(k)jj = 1 for all k, however, and in fact jjw(k + 1)jj jjw(k)jj (17) at each iteration if (16) is used for any valid vector norm [18]. Even so, updating w(k) using h g (k) instead of g(k) as in (11) largely decreases the rate at which jjw(k)jj deviates from jjw(k)jj = 1. To stabilize this update, one still needs to infrequently normalize the length of w(k) via (13). In the case of the L2 norm, this algorithm is w(k + 1) = w(k) + g(k)? w(k) wt (k)g(k) jjw(k)jj 2 2 ; (18) which is also the natural gradient algorithm on the unit hypersphere [17]. A calculation shows that jjw(k + 1)jj 2 2 = jjw(k)jj jjh g (k)jj 2 2 ; (19) and since jjh g (k)jj2 is of O((k)), the growth of jjw(k)jj 2 2 is linear in 2 (k). If the length of w(k) is normalized at each iteration, this update is also in the form of a rotation in the direction of h g (k), and the angle of rotation is listed in the third entry of Table 1. True Gradient Method [7, 16]. true gradient adaptation procedure for (1){(2): i) Calculate the tangent gradient h g (k) in (5). Each of the previous algorithms approximates the following ii) Move a distance jjh g (k)jj2 along a geodesic of the constraint surface in the direction of h g (k). 3

5 A geodesic is a curve on the constraint surface that connects two arbitrary points by an arc of shortest length. Implementing this procedure for a given norm constraint requires knowledge of the equations of motion on the constraint surface. When jjw(k)jj2 = 1 is imposed, updating w(k) amounts to rotating w(k) by an angle (k). For any unit vector u(k) that is perpendicular to w(k), the update w(k + 1) = cos((k))w(k) + sin((k))u(k) (20) rotates w(k + 1) by an angle (k) in the direction of u(k). For gradient adaptation, we choose u(k) = h g(k) (k) = 1 (k) g(k)? w(k)w T (k)g(k) ; (21) where the form of (k) = jjh g (k)jj2 is given in the last entry of Table 1. Note that when (k) is small, tan((k)) sin((k)) (k), yielding similar angles of rotation for all four methods. 3 Implementation Issues for L 2 -Norm Constrained Methods We consider the computational complexities and numerical stabilities of L2-norm constrained gradient approaches in this section, as the L2-norm constraint is the most popular for practical applications. To illustrate the salient issues involved in algorithm design, we shall focus on J (w) = 1 p Efjy(k)jp g; (22) as an instantaneous cost function, where y(k) = w T (k)x(k), x(k) is a discrete-time vector random process, and p is a positive integer not equal to 2. This cost function arises in certain formulations of independent component analysis (ICA), blind source separation, and blind deconvolution [13]{[15]. In this case, g(k) = (k)jy(k)j p?2 y(k)x(k) and w T (k)g(k) = (k)jy(k)j p ; (23) such that the sign of w T (k)g(k) does not change for all k. The rst four rows of Table 2 list the complexities of each of the algorithms in (9){(10), (14), (18), and (20){(21) according to the number and type of operations required, neglecting those operations needed for calculating g(k). The Lagrange, coecient normalization, and tangent gradient methods are the simplest, whereas the true gradient method is signicantly more complicated. All of these methods require operations other than multiply/adds to implement, making them more dicult to implement on real-time signal processing devices that are optimized for multiply/add calculations. Note that the structure of g(k) can often be exploited to further reduce each algorithm's complexity, e.g. by using (23) to compute w T (k)g(k) in the case of (22). 4

6 We now consider simplications that yield similar-behaving algorithms with reduced complexities. Since all four algorithms have equivalent behavior up to O( 2 (k)), we only consider modications of one approach the tangent gradient method in (18). The modied versions are Modication #1 [11]: w(k + 1) = w(k) + jjw(k)jj 2 2g(k)? w(k)w T (k)g(k); (24) Modication #2 [6]: w(k + 1) = w(k) + g(k)? w(k)w T (k)g(k) (25) Modication #3 [12, 15]: w(k + 1) = w(k) + jjw(k)jj 4 2g(k)? w(k)w T (k)g(k): (26) If jjw(k)jj 2 2 = 1, all three modied methods have similar behaviors to that of (18); however, the numerical properties of the algorithms in the radial dimension associated with the length of w(k) are quite dierent, as we shall show. For simplicity of discussion, we dene c(k) = jjw(k)jj 2 2? 1: (27) If c(k) experiences unmitigated growth, the associated algorithm is numerically-unstable. Numerical Stability of (24). " jjw(k + 1)jj 2 2 = Pre-multiplying both sides of (24) by their transposes yields 1 + jjw(k)jj 2 2g T (k) I? w(k)wt (k) jjw(k)jj 2 2! g(k) # jjw(k)jj 2 2 : (28) The matrix in large parentheses on the RHS of (28) is a projection matrix. Hence, so long as g(k) is not collinear with w(k) and jjg(k)jj2 > 0, then jjw(k + 1)jj2 > jjw(k)jj2 if jjw(0)jj2 = 1, i.e. numerical instability. Furthermore, since jjw(k)jj 2 2 appears in the factor in brackets on the RHS of (28), (24) causes accelerated growth in jjw(k)jj 2 independently of the form of J (w). Numerical Stability of (25). one from both sides, and rearranging terms, we obtain h i c(k + 1) = 1? 2w T (k)g(k) c(k) + Pre-multiplying both sides of (25) by their transposes, subtracting g(k)? w(k)w T (k)g(k) 2 2 : (29) If jjw(0)jj2 = 1, then we have c(k) > 0 for all k if g(k) 6= w(0)w T (0)g(k). Since the second term on the RHS of (29) is non-negative, we require w T (k)g(k) > 0 (30) for the numerical stability of (25). Note that (30) can often be veried for a particular J (w). For example, if (22) is chosen with a positive sign, then (30) is always true, making this algorithm appropriate for constrained maximization of p?1 Efjy(k)j p g. Conversely, the algorithm fails when p?1 Efjy(k)j p g is being minimized for all jjwjj2 = 1. This result is what justies f(23),(25)g for use in maximum-kurtosis ICA and principal component analysis tasks, and it also explains why this algorithm fails for minimum-kurtosis ICA and minor component analysis tasks [12, 15]. 5

7 Numerical Stability of (26). an update for c(k) as h c(k + 1) = 1 + 2jjw(k)jj 2 i 2w T (k)g(k) c(k) + In this case, if Performing a similar analysis of (26) as used previously, we obtain jjw(k)jj 4 2 g(k)? w(k)w T (k)g(k) 2 2 : (31) w T (k)g(k) < 0; (32) then (26) is numerically-stable. For minimum-kurtosis ICA (p = 4) tasks, choosing g(k) =?(k)jy(k)j p?2 y(k)x(k) causes this algorithm to perform in a stable fashion. Other Approaches. Of the three methods in (24){(26), (25) is the simplest, requiring 2n multiply/adds at each iteration. When w T (k)g(k) < 0, however, this method is numerically-unstable. As an alternative to (26), one can monitor the stability behavior of (25) and rescue the system from instability just prior to its occurence. It can be shown from (29) that jjw(k)jj 2 > 2 is an indicator of the onset of sudden divergence of (25) [18]. Moreover, simulations of (25) in several situations indicate that jjw(k)jj 2 2 > 2 provides a reliable divergence indicator when wt (k)g(k) < 0. One can therefore monitor the value of jjw(k)jj 2 2, and renormalize w(k) to unit length when jjw(k)jj 2 2 > C, where 1 < C < 2. Simulations indicate that values of C in the range 1:1 C 1:5 often yield good performance for typical problems and choices of (k). In addition, since jjw(k)jj 2 2 tends to grow slowly for small values of (k), the test jjw(k)jj 2 > C need only be performed at every Lth iteration where L 1, such that the complexity associated with this rescue method can be minimized. Further details regarding this approach, along with alternative reduced parameterization gradient methods for (1){(2), can be found in [18]. 4 Simulations We now explore the behaviors of the algorithms in (18), (25), and (26) in a single-component ICA task via MATLAB simulations. Let x(k) = As(k); (33) where A is an (n n) constant mixing matrix and s(k) = [s1(k) s n (k)] T contains unobservable independent components. If AA T = I, then maximizing J (w) =?0:25Efjy(k)j 4 g for y(k) = w T (k)x(k) and jjw(k)jj2 = 1 results in a negative-kurtosis signal in y(k) [14, 15]. one can guarantee AA T In practice, = I by prewhitening the signal measurements using simple adaptive procedures [9, 19]. We can then choose g(k) =?(k)jy(k)j 2 y(k)x(k) for any of the previouslydiscussed algorithms to obtain a candidate algorithm for this task. 6

8 For our simulations, we generate s(k) = [s1(k) s2(k) s3(k)] T, where s1(k) is an i.i.d. binaryf1g-distributed signal and s2(k) and s3(k) are i.i.d. Laplacian-distributed signals with p.d.f. p s (s) = e?p 2jsj = p 2. The mixing matrix A is chosen to be the eigenvector matrix of R xx = :9 0:4 0:7 0:4 0:3 0:5 0:7 0:5 1: ; (34) such that measurement prewhitening is not needed. For each algorithm, one hundred simulations have been run, and the average values of the performance factors (k) = jjc 1 (k)jj 2 2 =jjc 2(k)jj 2 2 and (k) = [jjw(k)jj 2 2? 1] 2 (35) have been computed, where c i (k) = E T i w(k) and E 1 and E 2 contain the two- and one-dimensional subspaces corresponding to the signal directions of the Laplacian and binary sources. Figure 1(a) shows the evolution of (k) for the tangent gradient method in (18), the simplied method in (25) with stabilization, and the self-normalized method in (26) for this task, where (k) = 0:001, L = 20, and C = 1:1. All three algorithms are successful at extracting the binary source from the linear mixture. Figure 1(b) shows the average evolution of (k) for the three methods, where the unmitigated growth in jjw(k)jj2 for the tangent gradient method is clearly evident. In contrast, the stabilization procedure used for (25) maintains this algorithm's stable behavior, and (26) performs in a stable, self-normalizing manner without such intervention. 5 Conclusions In this paper, we have presented an overview of algorithms that adjust a parameter vector to minimize or maximize a chosen cost function under a unit-norm parameter vector constraint. Particular attention has been paid both to methods that guarantee the unit-norm constraint at each iteration and to methods that maintain jjw(k)jj2 1 over time. Simulations verify the useful behavior of the schemes for independent component analysis. Some extensions of these results to multiple dimensions can be found in [12]. References [1] R. Courant and D. Hilbert, Methods of Mathematical Physics, vol. I (New York: Interscience, 1953). [2] T.P. Krasulina, \Method of stochastic approximation in the determination of the largest eigenvalue of the mathematical expectation of random matrices," Automat. Remote Contr., vol. 2, pp , [3] N.L. Owsley, \Adaptive data orthonormalization," Proc. IEEE Int. Conf. Acoust., Speech, Signal Processing, Tulsa, OK, pp , Apr

9 [4] P.A. Thompson, \An adaptive spectral analysis technique for unbiased frequency estimation in the presence of white noise," Proc. 13th Asilomar Conf. Circ., Syst., Comput., Pacic Grove, CA, pp , Nov [5] V.U. Reddy, B. Egardt, and T. Kailath, \Least squares type algorithm for adaptive implementation of Pisarenko's harmonic retrieval method," IEEE Trans. Acoust., Speech, Signal Processing, vol. 30, pp , June [6] E. Oja, \A simplied neural model as a principal component analyzer," J. Math. Biol., vol. 15, pp , [7] D.R. Fuhrmann and B. Liu, \An iterative algorithm for locating the minimal eigenvector of a symmetric matrix," Proc. IEEE Int. Conf. Acoust., Speech, Signal Processing, Dallas, TX, pp , [8] L. Wang and J. Karhunen, \A unied natural bigradient algorithm for robust PCA and MCA," Int. J. Neural Syst., vol. 7, no. 1, pp , Mar [9] K.I. Diamantaras and S.-Y. Kung, Principal Component Neural Networks: Theory and Applications (New York: Wiley, 1996). [10] V. Solo and X. Kong, \Performance analysis of adaptive eigenanalysis algorithms," IEEE Trans. Signal Processing, vol. 46, pp , Mar [11] T.-P. Chen, S. Amari, and Q. Lin, \A unied algorithm for principal and minor components extraction," Neural Networks, vol. 11, pp , Apr [12] S.C. Douglas, S.-Y. Kung, and S. Amari, \A self-stabilized minor subspace rule," IEEE Signal Processing Lett., vol. 5, pp , Dec [13] P. Comon, \Independent component analysis: A new concept?" Signal Processing, vol. 36, no. 3, pp , Apr [14] A. Hyvarinen and E. Oja, \Independent component analysis by general nonlinear Hebbian-like learning rules," Signal Processing, vol. 64, no. 3, pp , Feb [15] S.C. Douglas and S.-Y. Kung, \KuicNet algorithms for blind deconvolution," Proc. IEEE Workshop Neural Networks Signal Processing, Cambridge, UK, pp. 3-12, Aug [16] S.T. Smith, Geometric Optimization Methods for Adaptive Filtering, Ph.D. thesis, Harvard Univ., Cambridge, MA, [17] S.C. Douglas and S. Amari, \Natural gradient adaptation," in Unsupervised Adaptive Filtering, Vol. I: Blind Source Separation, S. Haykin, ed. (New York: Wiley, 1999), Chap. 2. [18] S.C. Douglas, S. Amari, and S.-Y. Kung, \Gradient adaptation with unit-norm constraints," Tech. Rep. EE , Dept. of Electrical Engineering, Southern Methodist University, Dallas, TX, Feb [19] S.C. Douglas and A. Cichocki, \Neural networks for blind decorrelation of signals," IEEE Trans. Signal Processing, vol. 45, pp , Nov

10 List of Tables Table 1: Angles of rotation for L2-norm constrained adaptation. Table 2: Summary of algorithms for L2-norm constrained adaptation. List of Figures Figure 1: Average performances of the various algorithms in the minimum-kurtosis ICA task: (a) evolutions of (k), and (b) evolutions of (k). 9

11 Algorithm Angle q of Rotation Lagrange sin((k)) = jjg(k)jj 2? [wt (k)g(k)] s 2 jjg(k)jj 2 Normalized sin((k)) =? (wt (k)g(k)) 2 q1 + 2w T (k)g(k) + jjg(k)jj 2 Tangent tan((k)) = jjg(k)jj 2? [wt (k)g(k)] q 2 True Gradient (k) = jjg(k)jj 2? [wt (k)g(k)] 2 Table 1: Angles of rotation for L2-norm constrained adaptation. Algorithm Complexity Stability Stabilization w new + p cos() Behavior Method p w new = w + g; = 1? jjgjj [wt g] 2? w T g w w + g new = p 1 + 2w T g + jjgjj 2 2 w g new = 1? wt jjwjj 2 2 3n + 1 3n n 2n? w + g 3n 3n? w new = cos w + sin h g; 5n 4n? h g = g? ww T g; = jjh g jj 2 w? new = 1? w T g w + jjwjj 2 2g 4n 3n? w new =? 1? w T g w + g 2n 2n w new =? 1? w T g w + jjwjj 4 2g 4n + 1 3n? Stable; jjwjj 2 = 1 Stable; jjwjj 2 = 1 Slow growth Set jjwjj 2 = 1 in jjwjj 2 infrequently Stable; jjwjj 2 = 1 Accelerated Set jjwjj 2 = 1 growth in jjwjj 2 periodically jjwjj 2 1 if w T g > 0 jjwjj 2 1 if w T g < 0 If w T g < 0, set jjwjj 2 = 1 when jjwjj 2 2 > C, 1:1 C 1:5 Table 2: Summary of algorithms for L2-norm constrained adaptation. 10

12 (a) 10 5 rho(k) 10 0 Eqn. (18) Eqn. (25) w/rescue Eqn. (26) (b) 10 2 Eqn. (18) Eqn. (25) w/rescue Eqn. (26) eta(k) number of iterations Figure 1: Average performances of the various algorithms in the minimum-kurtosis ICA task: (a) evolutions of (k), and (b) evolutions of (k). 11

ON SOME EXTENSIONS OF THE NATURAL GRADIENT ALGORITHM. Brain Science Institute, RIKEN, Wako-shi, Saitama , Japan

ON SOME EXTENSIONS OF THE NATURAL GRADIENT ALGORITHM. Brain Science Institute, RIKEN, Wako-shi, Saitama , Japan ON SOME EXTENSIONS OF THE NATURAL GRADIENT ALGORITHM Pando Georgiev a, Andrzej Cichocki b and Shun-ichi Amari c Brain Science Institute, RIKEN, Wako-shi, Saitama 351-01, Japan a On leave from the Sofia

More information

MULTICHANNEL BLIND SEPARATION AND. Scott C. Douglas 1, Andrzej Cichocki 2, and Shun-ichi Amari 2

MULTICHANNEL BLIND SEPARATION AND. Scott C. Douglas 1, Andrzej Cichocki 2, and Shun-ichi Amari 2 MULTICHANNEL BLIND SEPARATION AND DECONVOLUTION OF SOURCES WITH ARBITRARY DISTRIBUTIONS Scott C. Douglas 1, Andrzej Cichoci, and Shun-ichi Amari 1 Department of Electrical Engineering, University of Utah

More information

1 Introduction Independent component analysis (ICA) [10] is a statistical technique whose main applications are blind source separation, blind deconvo

1 Introduction Independent component analysis (ICA) [10] is a statistical technique whose main applications are blind source separation, blind deconvo The Fixed-Point Algorithm and Maximum Likelihood Estimation for Independent Component Analysis Aapo Hyvarinen Helsinki University of Technology Laboratory of Computer and Information Science P.O.Box 5400,

More information

ORIENTED PCA AND BLIND SIGNAL SEPARATION

ORIENTED PCA AND BLIND SIGNAL SEPARATION ORIENTED PCA AND BLIND SIGNAL SEPARATION K. I. Diamantaras Department of Informatics TEI of Thessaloniki Sindos 54101, Greece kdiamant@it.teithe.gr Th. Papadimitriou Department of Int. Economic Relat.

More information

Performance Comparison of Two Implementations of the Leaky. LMS Adaptive Filter. Scott C. Douglas. University of Utah. Salt Lake City, Utah 84112

Performance Comparison of Two Implementations of the Leaky. LMS Adaptive Filter. Scott C. Douglas. University of Utah. Salt Lake City, Utah 84112 Performance Comparison of Two Implementations of the Leaky LMS Adaptive Filter Scott C. Douglas Department of Electrical Engineering University of Utah Salt Lake City, Utah 8411 Abstract{ The leaky LMS

More information

Natural Gradient Learning for Over- and Under-Complete Bases in ICA

Natural Gradient Learning for Over- and Under-Complete Bases in ICA NOTE Communicated by Jean-François Cardoso Natural Gradient Learning for Over- and Under-Complete Bases in ICA Shun-ichi Amari RIKEN Brain Science Institute, Wako-shi, Hirosawa, Saitama 351-01, Japan Independent

More information

x 1 (t) Spectrogram t s

x 1 (t) Spectrogram t s A METHOD OF ICA IN TIME-FREQUENCY DOMAIN Shiro Ikeda PRESTO, JST Hirosawa 2-, Wako, 35-98, Japan Shiro.Ikeda@brain.riken.go.jp Noboru Murata RIKEN BSI Hirosawa 2-, Wako, 35-98, Japan Noboru.Murata@brain.riken.go.jp

More information

Recursive Generalized Eigendecomposition for Independent Component Analysis

Recursive Generalized Eigendecomposition for Independent Component Analysis Recursive Generalized Eigendecomposition for Independent Component Analysis Umut Ozertem 1, Deniz Erdogmus 1,, ian Lan 1 CSEE Department, OGI, Oregon Health & Science University, Portland, OR, USA. {ozertemu,deniz}@csee.ogi.edu

More information

Non-Euclidean Independent Component Analysis and Oja's Learning

Non-Euclidean Independent Component Analysis and Oja's Learning Non-Euclidean Independent Component Analysis and Oja's Learning M. Lange 1, M. Biehl 2, and T. Villmann 1 1- University of Appl. Sciences Mittweida - Dept. of Mathematics Mittweida, Saxonia - Germany 2-

More information

A Cross-Associative Neural Network for SVD of Nonsquared Data Matrix in Signal Processing

A Cross-Associative Neural Network for SVD of Nonsquared Data Matrix in Signal Processing IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 5, SEPTEMBER 2001 1215 A Cross-Associative Neural Network for SVD of Nonsquared Data Matrix in Signal Processing Da-Zheng Feng, Zheng Bao, Xian-Da Zhang

More information

1 Introduction Blind source separation (BSS) is a fundamental problem which is encountered in a variety of signal processing problems where multiple s

1 Introduction Blind source separation (BSS) is a fundamental problem which is encountered in a variety of signal processing problems where multiple s Blind Separation of Nonstationary Sources in Noisy Mixtures Seungjin CHOI x1 and Andrzej CICHOCKI y x Department of Electrical Engineering Chungbuk National University 48 Kaeshin-dong, Cheongju Chungbuk

More information

Robust extraction of specific signals with temporal structure

Robust extraction of specific signals with temporal structure Robust extraction of specific signals with temporal structure Zhi-Lin Zhang, Zhang Yi Computational Intelligence Laboratory, School of Computer Science and Engineering, University of Electronic Science

More information

Optimum Sampling Vectors for Wiener Filter Noise Reduction

Optimum Sampling Vectors for Wiener Filter Noise Reduction 58 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 1, JANUARY 2002 Optimum Sampling Vectors for Wiener Filter Noise Reduction Yukihiko Yamashita, Member, IEEE Absact Sampling is a very important and

More information

To appear in Proceedings of the ICA'99, Aussois, France, A 2 R mn is an unknown mixture matrix of full rank, v(t) is the vector of noises. The

To appear in Proceedings of the ICA'99, Aussois, France, A 2 R mn is an unknown mixture matrix of full rank, v(t) is the vector of noises. The To appear in Proceedings of the ICA'99, Aussois, France, 1999 1 NATURAL GRADIENT APPROACH TO BLIND SEPARATION OF OVER- AND UNDER-COMPLETE MIXTURES L.-Q. Zhang, S. Amari and A. Cichocki Brain-style Information

More information

POLYNOMIAL SINGULAR VALUES FOR NUMBER OF WIDEBAND SOURCES ESTIMATION AND PRINCIPAL COMPONENT ANALYSIS

POLYNOMIAL SINGULAR VALUES FOR NUMBER OF WIDEBAND SOURCES ESTIMATION AND PRINCIPAL COMPONENT ANALYSIS POLYNOMIAL SINGULAR VALUES FOR NUMBER OF WIDEBAND SOURCES ESTIMATION AND PRINCIPAL COMPONENT ANALYSIS Russell H. Lambert RF and Advanced Mixed Signal Unit Broadcom Pasadena, CA USA russ@broadcom.com Marcel

More information

Constrained Projection Approximation Algorithms for Principal Component Analysis

Constrained Projection Approximation Algorithms for Principal Component Analysis Constrained Projection Approximation Algorithms for Principal Component Analysis Seungjin Choi, Jong-Hoon Ahn, Andrzej Cichocki Department of Computer Science, Pohang University of Science and Technology,

More information

Blind Extraction of Singularly Mixed Source Signals

Blind Extraction of Singularly Mixed Source Signals IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL 11, NO 6, NOVEMBER 2000 1413 Blind Extraction of Singularly Mixed Source Signals Yuanqing Li, Jun Wang, Senior Member, IEEE, and Jacek M Zurada, Fellow, IEEE Abstract

More information

Average Reward Parameters

Average Reward Parameters Simulation-Based Optimization of Markov Reward Processes: Implementation Issues Peter Marbach 2 John N. Tsitsiklis 3 Abstract We consider discrete time, nite state space Markov reward processes which depend

More information

A Robust PCA by LMSER Learning with Iterative Error. Bai-ling Zhang Irwin King Lei Xu.

A Robust PCA by LMSER Learning with Iterative Error. Bai-ling Zhang Irwin King Lei Xu. A Robust PCA by LMSER Learning with Iterative Error Reinforcement y Bai-ling Zhang Irwin King Lei Xu blzhang@cs.cuhk.hk king@cs.cuhk.hk lxu@cs.cuhk.hk Department of Computer Science The Chinese University

More information

A Subspace Approach to Estimation of. Measurements 1. Carlos E. Davila. Electrical Engineering Department, Southern Methodist University

A Subspace Approach to Estimation of. Measurements 1. Carlos E. Davila. Electrical Engineering Department, Southern Methodist University EDICS category SP 1 A Subspace Approach to Estimation of Autoregressive Parameters From Noisy Measurements 1 Carlos E Davila Electrical Engineering Department, Southern Methodist University Dallas, Texas

More information

A NATURAL GRADIENT CONVOLUTIVE BLIND SOURCE SEPARATION ALGORITHM FOR SPEECH MIXTURES

A NATURAL GRADIENT CONVOLUTIVE BLIND SOURCE SEPARATION ALGORITHM FOR SPEECH MIXTURES A NATURAL GRADIENT CONVOLUTIVE BLIND SOURCE SEPARATION ALGORITHM FOR SPEECH MIXTURES Xiaoan Sun and Scott C. Douglas Department of Electrical Engineering Southern Methodist University Dallas, Texas 75275

More information

CONTROL SYSTEMS ANALYSIS VIA BLIND SOURCE DECONVOLUTION. Kenji Sugimoto and Yoshito Kikkawa

CONTROL SYSTEMS ANALYSIS VIA BLIND SOURCE DECONVOLUTION. Kenji Sugimoto and Yoshito Kikkawa CONTROL SYSTEMS ANALYSIS VIA LIND SOURCE DECONVOLUTION Kenji Sugimoto and Yoshito Kikkawa Nara Institute of Science and Technology Graduate School of Information Science 896-5 Takayama-cho, Ikoma-city,

More information

TWO METHODS FOR ESTIMATING OVERCOMPLETE INDEPENDENT COMPONENT BASES. Mika Inki and Aapo Hyvärinen

TWO METHODS FOR ESTIMATING OVERCOMPLETE INDEPENDENT COMPONENT BASES. Mika Inki and Aapo Hyvärinen TWO METHODS FOR ESTIMATING OVERCOMPLETE INDEPENDENT COMPONENT BASES Mika Inki and Aapo Hyvärinen Neural Networks Research Centre Helsinki University of Technology P.O. Box 54, FIN-215 HUT, Finland ABSTRACT

More information

A METHOD OF ICA IN TIME-FREQUENCY DOMAIN

A METHOD OF ICA IN TIME-FREQUENCY DOMAIN A METHOD OF ICA IN TIME-FREQUENCY DOMAIN Shiro Ikeda PRESTO, JST Hirosawa 2-, Wako, 35-98, Japan Shiro.Ikeda@brain.riken.go.jp Noboru Murata RIKEN BSI Hirosawa 2-, Wako, 35-98, Japan Noboru.Murata@brain.riken.go.jp

More information

Algorithms for Nonnegative Independent Component Analysis

Algorithms for Nonnegative Independent Component Analysis Algorithms for Nonnegative Independent Component Analysis Mark D Plumbley IEEE Transactions on Neural Networks, 4(3), 534-543, May 2003 Abstract We consider the task of solving the independent component

More information

On Mean Curvature Diusion in Nonlinear Image Filtering. Adel I. El-Fallah and Gary E. Ford. University of California, Davis. Davis, CA

On Mean Curvature Diusion in Nonlinear Image Filtering. Adel I. El-Fallah and Gary E. Ford. University of California, Davis. Davis, CA On Mean Curvature Diusion in Nonlinear Image Filtering Adel I. El-Fallah and Gary E. Ford CIPIC, Center for Image Processing and Integrated Computing University of California, Davis Davis, CA 95616 Abstract

More information

A Coupled Helmholtz Machine for PCA

A Coupled Helmholtz Machine for PCA A Coupled Helmholtz Machine for PCA Seungjin Choi Department of Computer Science Pohang University of Science and Technology San 3 Hyoja-dong, Nam-gu Pohang 79-784, Korea seungjin@postech.ac.kr August

More information

ICA [6] ICA) [7, 8] ICA ICA ICA [9, 10] J-F. Cardoso. [13] Matlab ICA. Comon[3], Amari & Cardoso[4] ICA ICA

ICA [6] ICA) [7, 8] ICA ICA ICA [9, 10] J-F. Cardoso. [13] Matlab ICA. Comon[3], Amari & Cardoso[4] ICA ICA 16 1 (Independent Component Analysis: ICA) 198 9 ICA ICA ICA 1 ICA 198 Jutten Herault Comon[3], Amari & Cardoso[4] ICA Comon (PCA) projection persuit projection persuit ICA ICA ICA 1 [1] [] ICA ICA EEG

More information

One-unit Learning Rules for Independent Component Analysis

One-unit Learning Rules for Independent Component Analysis One-unit Learning Rules for Independent Component Analysis Aapo Hyvarinen and Erkki Oja Helsinki University of Technology Laboratory of Computer and Information Science Rakentajanaukio 2 C, FIN-02150 Espoo,

More information

SPARSE REPRESENTATION AND BLIND DECONVOLUTION OF DYNAMICAL SYSTEMS. Liqing Zhang and Andrzej Cichocki

SPARSE REPRESENTATION AND BLIND DECONVOLUTION OF DYNAMICAL SYSTEMS. Liqing Zhang and Andrzej Cichocki SPARSE REPRESENTATON AND BLND DECONVOLUTON OF DYNAMCAL SYSTEMS Liqing Zhang and Andrzej Cichocki Lab for Advanced Brain Signal Processing RKEN Brain Science nstitute Wako shi, Saitama, 351-198, apan zha,cia

More information

Covariance and Correlation Matrix

Covariance and Correlation Matrix Covariance and Correlation Matrix Given sample {x n } N 1, where x Rd, x n = x 1n x 2n. x dn sample mean x = 1 N N n=1 x n, and entries of sample mean are x i = 1 N N n=1 x in sample covariance matrix

More information

Submitted to Electronics Letters. Indexing terms: Signal Processing, Adaptive Filters. The Combined LMS/F Algorithm Shao-Jen Lim and John G. Harris Co

Submitted to Electronics Letters. Indexing terms: Signal Processing, Adaptive Filters. The Combined LMS/F Algorithm Shao-Jen Lim and John G. Harris Co Submitted to Electronics Letters. Indexing terms: Signal Processing, Adaptive Filters. The Combined LMS/F Algorithm Shao-Jen Lim and John G. Harris Computational Neuro-Engineering Laboratory University

More information

APPLICATION OF RECURRENT NEURAL NETWORK USING MATLAB SIMULINK IN MEDICINE

APPLICATION OF RECURRENT NEURAL NETWORK USING MATLAB SIMULINK IN MEDICINE ITALIAN JOURNAL OF PURE AND APPLIED MATHEMATICS N. 39 2018 (23 30) 23 APPLICATION OF RECURRENT NEURAL NETWORK USING MATLAB SIMULINK IN MEDICINE Raja Das Madhu Sudan Reddy VIT Unversity Vellore, Tamil Nadu

More information

FEATURE EXTRACTION USING SUPERVISED INDEPENDENT COMPONENT ANALYSIS BY MAXIMIZING CLASS DISTANCE

FEATURE EXTRACTION USING SUPERVISED INDEPENDENT COMPONENT ANALYSIS BY MAXIMIZING CLASS DISTANCE FEATURE EXTRACTION USING SUPERVISED INDEPENDENT COMPONENT ANALYSIS BY MAXIMIZING CLASS DISTANCE Yoshinori Sakaguchi*, Seiichi Ozawa*, and Manabu Kotani** *Graduate School of Science and Technology, Kobe

More information

Introduction to Independent Component Analysis. Jingmei Lu and Xixi Lu. Abstract

Introduction to Independent Component Analysis. Jingmei Lu and Xixi Lu. Abstract Final Project 2//25 Introduction to Independent Component Analysis Abstract Independent Component Analysis (ICA) can be used to solve blind signal separation problem. In this article, we introduce definition

More information

Equivalence Probability and Sparsity of Two Sparse Solutions in Sparse Representation

Equivalence Probability and Sparsity of Two Sparse Solutions in Sparse Representation IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 19, NO. 12, DECEMBER 2008 2009 Equivalence Probability and Sparsity of Two Sparse Solutions in Sparse Representation Yuanqing Li, Member, IEEE, Andrzej Cichocki,

More information

Linear Prediction Based Blind Source Extraction Algorithms in Practical Applications

Linear Prediction Based Blind Source Extraction Algorithms in Practical Applications Linear Prediction Based Blind Source Extraction Algorithms in Practical Applications Zhi-Lin Zhang 1,2 and Liqing Zhang 1 1 Department of Computer Science and Engineering, Shanghai Jiao Tong University,

More information

Learning with Ensembles: How. over-tting can be useful. Anders Krogh Copenhagen, Denmark. Abstract

Learning with Ensembles: How. over-tting can be useful. Anders Krogh Copenhagen, Denmark. Abstract Published in: Advances in Neural Information Processing Systems 8, D S Touretzky, M C Mozer, and M E Hasselmo (eds.), MIT Press, Cambridge, MA, pages 190-196, 1996. Learning with Ensembles: How over-tting

More information

Linear Regression and Its Applications

Linear Regression and Its Applications Linear Regression and Its Applications Predrag Radivojac October 13, 2014 Given a data set D = {(x i, y i )} n the objective is to learn the relationship between features and the target. We usually start

More information

Undercomplete Independent Component. Analysis for Signal Separation and. Dimension Reduction. Category: Algorithms and Architectures.

Undercomplete Independent Component. Analysis for Signal Separation and. Dimension Reduction. Category: Algorithms and Architectures. Undercomplete Independent Component Analysis for Signal Separation and Dimension Reduction John Porrill and James V Stone Psychology Department, Sheeld University, Sheeld, S10 2UR, England. Tel: 0114 222

More information

= w 2. w 1. B j. A j. C + j1j2

= w 2. w 1. B j. A j. C + j1j2 Local Minima and Plateaus in Multilayer Neural Networks Kenji Fukumizu and Shun-ichi Amari Brain Science Institute, RIKEN Hirosawa 2-, Wako, Saitama 35-098, Japan E-mail: ffuku, amarig@brain.riken.go.jp

More information

Massoud BABAIE-ZADEH. Blind Source Separation (BSS) and Independent Componen Analysis (ICA) p.1/39

Massoud BABAIE-ZADEH. Blind Source Separation (BSS) and Independent Componen Analysis (ICA) p.1/39 Blind Source Separation (BSS) and Independent Componen Analysis (ICA) Massoud BABAIE-ZADEH Blind Source Separation (BSS) and Independent Componen Analysis (ICA) p.1/39 Outline Part I Part II Introduction

More information

Fundamentals of Principal Component Analysis (PCA), Independent Component Analysis (ICA), and Independent Vector Analysis (IVA)

Fundamentals of Principal Component Analysis (PCA), Independent Component Analysis (ICA), and Independent Vector Analysis (IVA) Fundamentals of Principal Component Analysis (PCA),, and Independent Vector Analysis (IVA) Dr Mohsen Naqvi Lecturer in Signal and Information Processing, School of Electrical and Electronic Engineering,

More information

Blind Source Separation with a Time-Varying Mixing Matrix

Blind Source Separation with a Time-Varying Mixing Matrix Blind Source Separation with a Time-Varying Mixing Matrix Marcus R DeYoung and Brian L Evans Department of Electrical and Computer Engineering The University of Texas at Austin 1 University Station, Austin,

More information

Estimation of linear non-gaussian acyclic models for latent factors

Estimation of linear non-gaussian acyclic models for latent factors Estimation of linear non-gaussian acyclic models for latent factors Shohei Shimizu a Patrik O. Hoyer b Aapo Hyvärinen b,c a The Institute of Scientific and Industrial Research, Osaka University Mihogaoka

More information

BLIND SEPARATION OF INSTANTANEOUS MIXTURES OF NON STATIONARY SOURCES

BLIND SEPARATION OF INSTANTANEOUS MIXTURES OF NON STATIONARY SOURCES BLIND SEPARATION OF INSTANTANEOUS MIXTURES OF NON STATIONARY SOURCES Dinh-Tuan Pham Laboratoire de Modélisation et Calcul URA 397, CNRS/UJF/INPG BP 53X, 38041 Grenoble cédex, France Dinh-Tuan.Pham@imag.fr

More information

Independent Component Analysis and Its Application on Accelerator Physics

Independent Component Analysis and Its Application on Accelerator Physics Independent Component Analysis and Its Application on Accelerator Physics Xiaoying Pang LA-UR-12-20069 ICA and PCA Similarities: Blind source separation method (BSS) no model Observed signals are linear

More information

An Improved Cumulant Based Method for Independent Component Analysis

An Improved Cumulant Based Method for Independent Component Analysis An Improved Cumulant Based Method for Independent Component Analysis Tobias Blaschke and Laurenz Wiskott Institute for Theoretical Biology Humboldt University Berlin Invalidenstraße 43 D - 0 5 Berlin Germany

More information

Blind Deconvolution via Maximum Kurtosis Adaptive Filtering

Blind Deconvolution via Maximum Kurtosis Adaptive Filtering Blind Deconvolution via Maximum Kurtosis Adaptive Filtering Deborah Pereg Doron Benzvi The Jerusalem College of Engineering Jerusalem, Israel doronb@jce.ac.il, deborahpe@post.jce.ac.il ABSTRACT In this

More information

MODELS OF LEARNING AND THE POLAR DECOMPOSITION OF BOUNDED LINEAR OPERATORS

MODELS OF LEARNING AND THE POLAR DECOMPOSITION OF BOUNDED LINEAR OPERATORS Eighth Mississippi State - UAB Conference on Differential Equations and Computational Simulations. Electronic Journal of Differential Equations, Conf. 19 (2010), pp. 31 36. ISSN: 1072-6691. URL: http://ejde.math.txstate.edu

More information

Dictionary Learning for L1-Exact Sparse Coding

Dictionary Learning for L1-Exact Sparse Coding Dictionary Learning for L1-Exact Sparse Coding Mar D. Plumbley Department of Electronic Engineering, Queen Mary University of London, Mile End Road, London E1 4NS, United Kingdom. Email: mar.plumbley@elec.qmul.ac.u

More information

Statistical Pattern Recognition

Statistical Pattern Recognition Statistical Pattern Recognition Feature Extraction Hamid R. Rabiee Jafar Muhammadi, Alireza Ghasemi, Payam Siyari Spring 2014 http://ce.sharif.edu/courses/92-93/2/ce725-2/ Agenda Dimensionality Reduction

More information

where A 2 IR m n is the mixing matrix, s(t) is the n-dimensional source vector (n» m), and v(t) is additive white noise that is statistically independ

where A 2 IR m n is the mixing matrix, s(t) is the n-dimensional source vector (n» m), and v(t) is additive white noise that is statistically independ BLIND SEPARATION OF NONSTATIONARY AND TEMPORALLY CORRELATED SOURCES FROM NOISY MIXTURES Seungjin CHOI x and Andrzej CICHOCKI y x Department of Electrical Engineering Chungbuk National University, KOREA

More information

Comparative Performance Analysis of Three Algorithms for Principal Component Analysis

Comparative Performance Analysis of Three Algorithms for Principal Component Analysis 84 R. LANDQVIST, A. MOHAMMED, COMPARATIVE PERFORMANCE ANALYSIS OF THR ALGORITHMS Comparative Performance Analysis of Three Algorithms for Principal Component Analysis Ronnie LANDQVIST, Abbas MOHAMMED Dept.

More information

HOPFIELD neural networks (HNNs) are a class of nonlinear

HOPFIELD neural networks (HNNs) are a class of nonlinear IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: EXPRESS BRIEFS, VOL. 52, NO. 4, APRIL 2005 213 Stochastic Noise Process Enhancement of Hopfield Neural Networks Vladimir Pavlović, Member, IEEE, Dan Schonfeld,

More information

Short-Time ICA for Blind Separation of Noisy Speech

Short-Time ICA for Blind Separation of Noisy Speech Short-Time ICA for Blind Separation of Noisy Speech Jing Zhang, P.C. Ching Department of Electronic Engineering The Chinese University of Hong Kong, Hong Kong jzhang@ee.cuhk.edu.hk, pcching@ee.cuhk.edu.hk

More information

New Lyapunov Krasovskii functionals for stability of linear retarded and neutral type systems

New Lyapunov Krasovskii functionals for stability of linear retarded and neutral type systems Systems & Control Letters 43 (21 39 319 www.elsevier.com/locate/sysconle New Lyapunov Krasovskii functionals for stability of linear retarded and neutral type systems E. Fridman Department of Electrical

More information

Maximum variance formulation

Maximum variance formulation 12.1. Principal Component Analysis 561 Figure 12.2 Principal component analysis seeks a space of lower dimensionality, known as the principal subspace and denoted by the magenta line, such that the orthogonal

More information

CS281 Section 4: Factor Analysis and PCA

CS281 Section 4: Factor Analysis and PCA CS81 Section 4: Factor Analysis and PCA Scott Linderman At this point we have seen a variety of machine learning models, with a particular emphasis on models for supervised learning. In particular, we

More information

Remaining energy on log scale Number of linear PCA components

Remaining energy on log scale Number of linear PCA components NONLINEAR INDEPENDENT COMPONENT ANALYSIS USING ENSEMBLE LEARNING: EXPERIMENTS AND DISCUSSION Harri Lappalainen, Xavier Giannakopoulos, Antti Honkela, and Juha Karhunen Helsinki University of Technology,

More information

NON-NEGATIVE SPARSE CODING

NON-NEGATIVE SPARSE CODING NON-NEGATIVE SPARSE CODING Patrik O. Hoyer Neural Networks Research Centre Helsinki University of Technology P.O. Box 9800, FIN-02015 HUT, Finland patrik.hoyer@hut.fi To appear in: Neural Networks for

More information

Blind deconvolution of dynamical systems using a balanced parameterized state space approach

Blind deconvolution of dynamical systems using a balanced parameterized state space approach University of Wollongong Research Online Faculty of Informatics - Papers (Archive) Faculty of Engineering and Information Sciences 2003 Blind deconvolution of dynamical systems using a balanced parameterized

More information

SUBSPACE TRACKING FOR SIGNAL PROCESSING

SUBSPACE TRACKING FOR SIGNAL PROCESSING CHAPTER 42 SUBSPACE TRACKING FOR SIGNAL PROCESSING 42.1 INTRODUCTION Research in subspace and component-based techniques were originated in Statistics in the middle of the last century through the problem

More information

Independent Component Analysis

Independent Component Analysis Independent Component Analysis James V. Stone November 4, 24 Sheffield University, Sheffield, UK Keywords: independent component analysis, independence, blind source separation, projection pursuit, complexity

More information

On Information Maximization and Blind Signal Deconvolution

On Information Maximization and Blind Signal Deconvolution On Information Maximization and Blind Signal Deconvolution A Röbel Technical University of Berlin, Institute of Communication Sciences email: roebel@kgwtu-berlinde Abstract: In the following paper we investigate

More information

Post-Nonlinear Blind Extraction in the Presence of Ill-Conditioned Mixing Wai Yie Leong and Danilo P. Mandic, Senior Member, IEEE

Post-Nonlinear Blind Extraction in the Presence of Ill-Conditioned Mixing Wai Yie Leong and Danilo P. Mandic, Senior Member, IEEE IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I: REGULAR PAPERS, VOL. 55, NO. 9, OCTOBER 2008 2631 Post-Nonlinear Blind Extraction in the Presence of Ill-Conditioned Mixing Wai Yie Leong and Danilo P. Mandic,

More information

Non-Negative Matrix Factorization And Its Application to Audio. Tuomas Virtanen Tampere University of Technology

Non-Negative Matrix Factorization And Its Application to Audio. Tuomas Virtanen Tampere University of Technology Non-Negative Matrix Factorization And Its Application to Audio Tuomas Virtanen Tampere University of Technology tuomas.virtanen@tut.fi 2 Contents Introduction to audio signals Spectrogram representation

More information

The Rationale for Second Level Adaptation

The Rationale for Second Level Adaptation The Rationale for Second Level Adaptation Kumpati S. Narendra, Yu Wang and Wei Chen Center for Systems Science, Yale University arxiv:1510.04989v1 [cs.sy] 16 Oct 2015 Abstract Recently, a new approach

More information

ADAPTIVE LATERAL INHIBITION FOR NON-NEGATIVE ICA. Mark Plumbley

ADAPTIVE LATERAL INHIBITION FOR NON-NEGATIVE ICA. Mark Plumbley Submitteed to the International Conference on Independent Component Analysis and Blind Signal Separation (ICA2) ADAPTIVE LATERAL INHIBITION FOR NON-NEGATIVE ICA Mark Plumbley Audio & Music Lab Department

More information

x x2 H11 H y y2 H22 H

x x2 H11 H y y2 H22 H .5.5 5 5.5.5 5 5 2 3 4 5 6 7 H 2 3 4 5 6 7 H2 5 5 x 5 5 x2.5.5.5.5.5.5.5.5 2 3 4 5 6 7 H2 2 3 4 5 6 7 H22.5.5 y.5.5 y2 Figure : Mixing coef.: H [k], H 2 [k], H 2 [k], H 22 [k]. Figure 3: Observations and

More information

Chapter 6 Effective Blind Source Separation Based on the Adam Algorithm

Chapter 6 Effective Blind Source Separation Based on the Adam Algorithm Chapter 6 Effective Blind Source Separation Based on the Adam Algorithm Michele Scarpiniti, Simone Scardapane, Danilo Comminiello, Raffaele Parisi and Aurelio Uncini Abstract In this paper, we derive a

More information

Optimal Mean-Square Noise Benefits in Quantizer-Array Linear Estimation Ashok Patel and Bart Kosko

Optimal Mean-Square Noise Benefits in Quantizer-Array Linear Estimation Ashok Patel and Bart Kosko IEEE SIGNAL PROCESSING LETTERS, VOL. 17, NO. 12, DECEMBER 2010 1005 Optimal Mean-Square Noise Benefits in Quantizer-Array Linear Estimation Ashok Patel and Bart Kosko Abstract A new theorem shows that

More information

computation of the algorithms it is useful to introduce some sort of mapping that reduces the dimension of the data set before applying signal process

computation of the algorithms it is useful to introduce some sort of mapping that reduces the dimension of the data set before applying signal process Optimal Dimension Reduction for Array Processing { Generalized Soren Anderson y and Arye Nehorai Department of Electrical Engineering Yale University New Haven, CT 06520 EDICS Category: 3.6, 3.8. Abstract

More information

BLIND SEPARATION USING ABSOLUTE MOMENTS BASED ADAPTIVE ESTIMATING FUNCTION. Juha Karvanen and Visa Koivunen

BLIND SEPARATION USING ABSOLUTE MOMENTS BASED ADAPTIVE ESTIMATING FUNCTION. Juha Karvanen and Visa Koivunen BLIND SEPARATION USING ABSOLUTE MOMENTS BASED ADAPTIVE ESTIMATING UNCTION Juha Karvanen and Visa Koivunen Signal Processing Laboratory Helsinki University of Technology P.O. Box 3, IN-215 HUT, inland Tel.

More information

Constrained Independent Component Analysis

Constrained Independent Component Analysis Constrained Independent Component Analysis Wei Lu and Jagath C. Rajapakse School of Computer Engineering Nanyang Technological University, Singapore 639798 email: asjagath@ntu.edu.sg Abstract The paper

More information

MULTICHANNEL SIGNAL PROCESSING USING SPATIAL RANK COVARIANCE MATRICES

MULTICHANNEL SIGNAL PROCESSING USING SPATIAL RANK COVARIANCE MATRICES MULTICHANNEL SIGNAL PROCESSING USING SPATIAL RANK COVARIANCE MATRICES S. Visuri 1 H. Oja V. Koivunen 1 1 Signal Processing Lab. Dept. of Statistics Tampere Univ. of Technology University of Jyväskylä P.O.

More information

MODELLING OF FLEXIBLE MECHANICAL SYSTEMS THROUGH APPROXIMATED EIGENFUNCTIONS L. Menini A. Tornambe L. Zaccarian Dip. Informatica, Sistemi e Produzione

MODELLING OF FLEXIBLE MECHANICAL SYSTEMS THROUGH APPROXIMATED EIGENFUNCTIONS L. Menini A. Tornambe L. Zaccarian Dip. Informatica, Sistemi e Produzione MODELLING OF FLEXIBLE MECHANICAL SYSTEMS THROUGH APPROXIMATED EIGENFUNCTIONS L. Menini A. Tornambe L. Zaccarian Dip. Informatica, Sistemi e Produzione, Univ. di Roma Tor Vergata, via di Tor Vergata 11,

More information

Controlling Hopf Bifurcations: Discrete-Time Systems

Controlling Hopf Bifurcations: Discrete-Time Systems Discrete Dynamics in Nature and Society, Vol. 5, pp. 29-33 Reprints available directly from the publisher Photocopying permitted by license only (C) 2000 OPA (Overseas Publishers Association) N.V. Published

More information

PCA & ICA. CE-717: Machine Learning Sharif University of Technology Spring Soleymani

PCA & ICA. CE-717: Machine Learning Sharif University of Technology Spring Soleymani PCA & ICA CE-717: Machine Learning Sharif University of Technology Spring 2015 Soleymani Dimensionality Reduction: Feature Selection vs. Feature Extraction Feature selection Select a subset of a given

More information

1. Introduction Let the least value of an objective function F (x), x2r n, be required, where F (x) can be calculated for any vector of variables x2r

1. Introduction Let the least value of an objective function F (x), x2r n, be required, where F (x) can be calculated for any vector of variables x2r DAMTP 2002/NA08 Least Frobenius norm updating of quadratic models that satisfy interpolation conditions 1 M.J.D. Powell Abstract: Quadratic models of objective functions are highly useful in many optimization

More information

Polynomial functions over nite commutative rings

Polynomial functions over nite commutative rings Polynomial functions over nite commutative rings Balázs Bulyovszky a, Gábor Horváth a, a Institute of Mathematics, University of Debrecen, Pf. 400, Debrecen, 4002, Hungary Abstract We prove a necessary

More information

below, kernel PCA Eigenvectors, and linear combinations thereof. For the cases where the pre-image does exist, we can provide a means of constructing

below, kernel PCA Eigenvectors, and linear combinations thereof. For the cases where the pre-image does exist, we can provide a means of constructing Kernel PCA Pattern Reconstruction via Approximate Pre-Images Bernhard Scholkopf, Sebastian Mika, Alex Smola, Gunnar Ratsch, & Klaus-Robert Muller GMD FIRST, Rudower Chaussee 5, 12489 Berlin, Germany fbs,

More information

Lie Groups for 2D and 3D Transformations

Lie Groups for 2D and 3D Transformations Lie Groups for 2D and 3D Transformations Ethan Eade Updated May 20, 2017 * 1 Introduction This document derives useful formulae for working with the Lie groups that represent transformations in 2D and

More information

(z) UN-1(z) PN-1(z) Pn-1(z) - z k. 30(z) 30(z) (a) (b) (c) x y n ...

(z) UN-1(z) PN-1(z) Pn-1(z) - z k. 30(z) 30(z) (a) (b) (c) x y n ... Minimum Memory Implementations of the Lifting Scheme Christos Chrysas Antonio Ortega HewlettPackard Laboratories Integrated Media Systems Center 151 Page Mill Road, Bldg.3U3 University of Southern California

More information

ARTEFACT DETECTION IN ASTROPHYSICAL IMAGE DATA USING INDEPENDENT COMPONENT ANALYSIS. Maria Funaro, Erkki Oja, and Harri Valpola

ARTEFACT DETECTION IN ASTROPHYSICAL IMAGE DATA USING INDEPENDENT COMPONENT ANALYSIS. Maria Funaro, Erkki Oja, and Harri Valpola ARTEFACT DETECTION IN ASTROPHYSICAL IMAGE DATA USING INDEPENDENT COMPONENT ANALYSIS Maria Funaro, Erkki Oja, and Harri Valpola Neural Networks Research Centre, Helsinki University of Technology P.O.Box

More information

H 1 optimisation. Is hoped that the practical advantages of receding horizon control might be combined with the robustness advantages of H 1 control.

H 1 optimisation. Is hoped that the practical advantages of receding horizon control might be combined with the robustness advantages of H 1 control. A game theoretic approach to moving horizon control Sanjay Lall and Keith Glover Abstract A control law is constructed for a linear time varying system by solving a two player zero sum dierential game

More information

Eigenface-based facial recognition

Eigenface-based facial recognition Eigenface-based facial recognition Dimitri PISSARENKO December 1, 2002 1 General This document is based upon Turk and Pentland (1991b), Turk and Pentland (1991a) and Smith (2002). 2 How does it work? The

More information

Least Squares Optimization

Least Squares Optimization Least Squares Optimization The following is a brief review of least squares optimization and constrained optimization techniques. Broadly, these techniques can be used in data analysis and visualization

More information

Linearly-solvable Markov decision problems

Linearly-solvable Markov decision problems Advances in Neural Information Processing Systems 2 Linearly-solvable Markov decision problems Emanuel Todorov Department of Cognitive Science University of California San Diego todorov@cogsci.ucsd.edu

More information

Neural Learning in Structured Parameter Spaces Natural Riemannian Gradient

Neural Learning in Structured Parameter Spaces Natural Riemannian Gradient Neural Learning in Structured Parameter Spaces Natural Riemannian Gradient Shun-ichi Amari RIKEN Frontier Research Program, RIKEN, Hirosawa 2-1, Wako-shi 351-01, Japan amari@zoo.riken.go.jp Abstract The

More information

Iterative face image feature extraction with Generalized Hebbian Algorithm and a Sanger-like BCM rule

Iterative face image feature extraction with Generalized Hebbian Algorithm and a Sanger-like BCM rule Iterative face image feature extraction with Generalized Hebbian Algorithm and a Sanger-like BCM rule Clayton Aldern (Clayton_Aldern@brown.edu) Tyler Benster (Tyler_Benster@brown.edu) Carl Olsson (Carl_Olsson@brown.edu)

More information

Using Kernel PCA for Initialisation of Variational Bayesian Nonlinear Blind Source Separation Method

Using Kernel PCA for Initialisation of Variational Bayesian Nonlinear Blind Source Separation Method Using Kernel PCA for Initialisation of Variational Bayesian Nonlinear Blind Source Separation Method Antti Honkela 1, Stefan Harmeling 2, Leo Lundqvist 1, and Harri Valpola 1 1 Helsinki University of Technology,

More information

Independent Component Analysis and Its Applications. By Qing Xue, 10/15/2004

Independent Component Analysis and Its Applications. By Qing Xue, 10/15/2004 Independent Component Analysis and Its Applications By Qing Xue, 10/15/2004 Outline Motivation of ICA Applications of ICA Principles of ICA estimation Algorithms for ICA Extensions of basic ICA framework

More information

Problem Set 9 Due: In class Tuesday, Nov. 27 Late papers will be accepted until 12:00 on Thursday (at the beginning of class).

Problem Set 9 Due: In class Tuesday, Nov. 27 Late papers will be accepted until 12:00 on Thursday (at the beginning of class). Math 3, Fall Jerry L. Kazdan Problem Set 9 Due In class Tuesday, Nov. 7 Late papers will be accepted until on Thursday (at the beginning of class).. Suppose that is an eigenvalue of an n n matrix A and

More information

Ole Christensen 3. October 20, Abstract. We point out some connections between the existing theories for

Ole Christensen 3. October 20, Abstract. We point out some connections between the existing theories for Frames and pseudo-inverses. Ole Christensen 3 October 20, 1994 Abstract We point out some connections between the existing theories for frames and pseudo-inverses. In particular, using the pseudo-inverse

More information

NMF WITH SPECTRAL AND TEMPORAL CONTINUITY CRITERIA FOR MONAURAL SOUND SOURCE SEPARATION. Julian M. Becker, Christian Sohn and Christian Rohlfing

NMF WITH SPECTRAL AND TEMPORAL CONTINUITY CRITERIA FOR MONAURAL SOUND SOURCE SEPARATION. Julian M. Becker, Christian Sohn and Christian Rohlfing NMF WITH SPECTRAL AND TEMPORAL CONTINUITY CRITERIA FOR MONAURAL SOUND SOURCE SEPARATION Julian M. ecker, Christian Sohn Christian Rohlfing Institut für Nachrichtentechnik RWTH Aachen University D-52056

More information

Citation Ieee Signal Processing Letters, 2001, v. 8 n. 6, p

Citation Ieee Signal Processing Letters, 2001, v. 8 n. 6, p Title Multiplierless perfect reconstruction modulated filter banks with sum-of-powers-of-two coefficients Author(s) Chan, SC; Liu, W; Ho, KL Citation Ieee Signal Processing Letters, 2001, v. 8 n. 6, p.

More information

The model reduction algorithm proposed is based on an iterative two-step LMI scheme. The convergence of the algorithm is not analyzed but examples sho

The model reduction algorithm proposed is based on an iterative two-step LMI scheme. The convergence of the algorithm is not analyzed but examples sho Model Reduction from an H 1 /LMI perspective A. Helmersson Department of Electrical Engineering Linkoping University S-581 8 Linkoping, Sweden tel: +6 1 816 fax: +6 1 86 email: andersh@isy.liu.se September

More information

Multiple Similarities Based Kernel Subspace Learning for Image Classification

Multiple Similarities Based Kernel Subspace Learning for Image Classification Multiple Similarities Based Kernel Subspace Learning for Image Classification Wang Yan, Qingshan Liu, Hanqing Lu, and Songde Ma National Laboratory of Pattern Recognition, Institute of Automation, Chinese

More information