Optimum Sampling Vectors for Wiener Filter Noise Reduction

Transcription

1 58 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 1, JANUARY 2002 Optimum Sampling Vectors for Wiener Filter Noise Reduction Yukihiko Yamashita, Member, IEEE Absact Sampling is a very important and basic technique for signal processing. In the case that noise is added to a signal in the sampling process, we may use a reconsuction and noise reduction filter such as the Wiener filter. The Wiener filter provides a restored signal of which mean square error is minimized. However, the mean square error by the Wiener filter depends on the sampling vectors. We may have a freedom to consuct sampling vectors. In this paper, we provide optimum sampling vectors under the condition that the Wiener filter is used for noise reduction for two cases wherein the noise is added before/after sampling. The sampling vectors provided in this paper may not be practical since they are very complicated. However, the minimum mean square error, which we provide theoretically, can be used for evaluating other sampling vectors. We provide all proofs of the theorems and lemmas. Furthermore, by experimental results, we show their advantages. Index Terms Relative Karhunen Loève ansform, sampling, Wiener filter. I. INTRODUCTION SAMPLING is a very important and basic technique in order to input a signal into a computer for signal processing [1] [7]. Furthermore, sampling theories can be applied for data compression and pattern recognition by using them for feature exaction. In this paper, we consider the case wherein the sampled data are obtained by the inner product between a signal and the sampling vectors. Many sampling processes can be expressed by this model. When the dimension of the original signal space is more than the number of sampling points, we may not reconsuct the original signal from the sampled data. Furthermore, the sampled data may be degraded with noise. In those cases, we often use a reconsuction and noise reduction filter such as the Wiener filter (WF) [11], [17]. The WF provides a restored signal of which mean square error is minimized. However, the mean square error by the WF depends on the sampling vectors. We may have a freedom to consuct sampling vectors. It brings us a problem wherein sampling vectors are the optimum. When the signal contains no noise, it is clear that the optimum sampling vectors are given by the Karhunen Loève ansform (KLT) [9] [12]. When the signal contains noise, we provided sampling vectors Manuscript received November 30, 1999; revised September 28, The associate editor coordinating the review of this paper and approving it for publication was Prof. Gregori Vazquez. The author is with the Department of International Development Engineering, Tokyo Institute of Technology, Tokyo, Japan ( yamasita@ide.titech.ac.jp). Publisher Item Identifier S X(02) that minimize the mean square error between the original signal and the signal restored by the WF under the condition that the noise is uniform and uncorrelated [8]. In this paper, we provide optimum sampling vectors for the following two cases without such a resiction. The one case is that the noise is added before sampling. In this case, the optimum sampling vectors are given by using the relative KLT (RKLT) [13]. The RKLT is an extension of the KLT for the case that a noise is added to the signal. [Hua and Liu extended the RKLT to the general KLT (GKLT) [14].] The other case is that the noise is added after sampling. The results in [8] are given as the corollaries of theorems in this paper. The sampling vectors provided in this paper may not be practical since they are very complicated. However, the minimum mean square error, which we provide theoretically, can be used for evaluating other sampling vectors. By comparing with the theoretical minimum value, we can know whether the sampling vectors can be improved or not in the sense of mean square error. Usually, the meaning of sampling is to obtain discrete data from a continuous signal. Although our theorems are for discrete discrete sampling, they can be used for the following cases. When the support of a signal is bounded or we can assume such a case approximately, by making the signal vector with the coefficients of the Fourier expansion of the signal with a sufficiently large finite dimension, we can sufficiently approximate the continuous case since usually, the high-frequency components of the signal are small, and the results of the theorems in this paper mainly depend on the large eigenvalues of the correlation maix of signals. Furthermore, the combination of simple sampling with high sampling rate and complicated subsampling by digital signal processing can provide better sampled data than direct sampling. The subsampling is discrete discrete sampling. In the field of principal component analysis (PCA) neural networks, Diamantaras [15], [16] has provided an equivalent theorem of Theorem 2 in this paper. However, the proof is not complete since the method of Lagrange s multiplier not with inequalities but only with equations were used for inequality conditions. In order to prove the theorem, the convexity of the evaluation function should be considered in detail. Therefore, we provide the sict proof of the theorem in this paper. Since, in [8], neither experiments for the case that the noise is added before sampling nor the proofs of theorems and lemmas were provided, we demonsate the advantages of the optimum sampling vectors for both cases against the periodic sampling by experimental results, and we show all proofs in this paper X/02$ IEEE

2 YAMASHITA: OPTIMUM SAMPLING VECTORS FOR WIENER FILTER NOISE REDUCTION 59 A. Mathematical Preliminaries The following notations and terminologies are used in this paper. Let be an -dimensional Euclidean space. Let and be the inner product and the norm in, respectively. Let be the ensemble average for a stochastic signal in. Let and be the range and the null space of a maix, respectively. Let, rank, and be the anspose, the rank, and the ace of a maix, respectively. For any maix, there exists a unique maix [18], [19] such that,,, and. The maix is called the Moore Penrose generalized inverse of. An -maix is said to be a non-negative definite maix if and only if for every.an -maix is said to be a positive definite maix if and only if for every. They are denoted by and, respectively. For any symmeic non-negative definite maix, there exists a unique symmeic non-negative definite maix such that. We explain about the RKLT. The KLT provides the best approximation for a stochastic signal under the condition that its rank is fixed. However, when noise is added to the signal, it is not optimum in general. Let be a ansform maix. Let be a noise. Since an approximation of is given as with, the RKLT of rank (or not greater that ) is defined as a maix that minimizes under the condition that rank (or rank ). We assume that and are uncorrelated. We provide a solution of the RKLT not greater than in the following Proposition 1, whose form is slightly different from the original one in [13]. That form is simpler than the original form for numerical calculation. We show the proof of Proposition 1 in the Appendix. Let and be correlation maices with respect to signal and noise ensembles that are defined as (1) II. OPTIMUM SAMPLING Let be a set of sampling vectors. We assume that a sampled value is given as the inner product between a signal and a sampling vector. Let be the th element of a vector. Let be the sampled data. Let be a restoration maix. In this paper, we consider the criterion minimizing It is an optimum restoration problem to minimize with respect to. On the other hand, it is an optimum sampling problem to minimize with respect to. Let be a natural basis in. It follows that Then, the sampling maix is given as When we neglect the effect of noise, the sampled vector given as However, it often happens that the sampled vector is degraded with noise. We discuss two cases for the noise. One case is that the noise is added before sampling and the other is after. When, it is called subsampling. When, it is called oversampling. They are different problems. When noise is added after sampling, we provide unified theorems for both cases. When noise is added before sampling, it is no use to oversample a signal. A. Optimum Sampling when Noise Is Added Before Sampling In the case that noise is added before sampling, the sampled data is given as (6) (7) (8) is (9) (10) respectively. We define a maix as (2) (3) (4) This is equivalent to In this case, (6) is given by (11) (12) Let and be eigenvalues and a set of corresponding eigenvectors of, respectively. We chose as an orthonormal basis Proposition 1: An RKLT of rank not greater than is given as (5) We assume that and are uncorrelated. When we fix and minimize (12) with respect to, we obtain a WF from this criterion. A WF is given as (13) However, we do not use (13) to solve this problem for the proof. We can solve this problem by using Proposition 1. We use the same notation as Proposition 1.

3 60 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 1, JANUARY 2002 Theorem 1: When noise is added before sampling, optimum sampling vectors are given as Theorem 2: When noise is added after sampling, optimum sampling vectors are given as A restoration maix is given as (14) (22) (15) where The minimum value of is given as (23) (16) From this criterion, we can know that in (15) is a WF when is given by (14). Note that (14) is a sufficient condition. For example, are also the optimum sampling vectors with constants. It is no use to set since rank. It means that it is impossible to reduce noise by oversampling when noise is added before sampling. B. Optimum Sampling when Noise Is Added After Sampling When noise is added after sampling, the sampled data is given as with the maximum integer subject to The minimum value of is given as (24) (25) (26) (27) This is equivalent to (17) (18) In order to decide the, we have to scan and find the maximum subject to (25) and (26) since also depends on. From (22), we have Criterion (6) is given as (19) In this model, since the larger the norms of are, the smaller the effect of noise is, we have to normalize them. Here, we normalize the total power of sampling vectors, that is (20) with. In this subsection, we assume that and. When we fix, the solution of minimizing (19) with respect to is given by a WF [11], [17]. In this case, it is described as (21) Then, is minimized for subject to (8) and (20). Let and be eigenvalues and a set of corresponding eigenvectors of. Let and be those of. We choose and as orthonormal bases. Let be the smaller value of or. In this case, we have the following theorem for optimum sampling vectors. (28) Then, since (24) yields that corresponds to, we can consider that a component of which signal variance is large has to be ansformed to a component of which noise variance is small for optimum sampling. In the field of PCA neural networks, Diamantaras [15], [16] has provided an equivalent theorem to Theorem 2. However, the proof is not complete since the method of Lagrange s multiplier not with inequalities but only with equations were used for inequality conditions. In order to prove the theorem, the convexity of the evaluation function should be considered in detail. Furthermore, the proof of PCA inoduced in [15] is not complete, as Ogawa pointed out [12]. It uses the greedy method, which does not guarantee the global minimization. Let be a unit maix. When the noise is uniform and uncorrelated, that is,, since for all and every orthonormal basis in is a set of eigenvectors of, we have the following corollary. Corollary 1: When, optimum sampling vectors are given as (29)

4 YAMASHITA: OPTIMUM SAMPLING VECTORS FOR WIENER FILTER NOISE REDUCTION 61 Fig. 1. Mean square error when noise is added before sampling. where Theorem 3: When,, optimum sampling vectors are given as (30) with any orthonormal system and the maximum integer under the condition that the content of the square root in the (30) is not negative. The minimum value of is given as (31) We consider the case wherein and.we can provide the solution when the resiction is not (20) but that all are the same: Let be the Walsh basis in.wehave and the th element of is 1 or. (32) (33) (34) with an integer. and the minimum value of are the same as those in Corollary 1. III. EXPERIMENTAL RESULTS We compare the mean square errors between the original and the restored signals by the optimum and by the periodic samplings. We set the dimension of the original space to and the number of sampling vectors to,2,4,8,16,32, 64, 128, and 256. This experiment includes both subsamplings and oversamplings. When noise is added before sampling, we are resicted to for the optimum sampling. In the case of the periodic sampling and, the data are sampled several times at the same point. However, the sampled values at the same point are different in general when noise is added after sampling. We assume that for a real number, the correlation maix of signal is given as (35)

5 62 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 1, JANUARY 2002 Fig. 2. Optimum sampling and restoration vectors when noise is added before sampling ( =0:95, =0:2). Fig. 3. Mean square error when noise is added after sampling. where is the -element of. For a real number, is the largest integer not less than. The periodic sampling vectors are given as (36) (else) when,or (37) (else) when. The optimum sampling vectors are provided by Theorem 1 when noise is added before sampling and by Theorem 3 when noise is added after sampling. In Theorem 3, we set as. Then, the norm of sampling vectors is fixed to one. Fig. 1 illusates the mean square error versus the number of sampling vectors when noise is added before sampling. From Fig. 1, we can see the advantage of the optimum sampling when. When the correlations of signals are large, the advantage is also large. Fig. 2 illusates the sampling vectors and the corresponding restoration vectors when noise is added before sampling. We can see the vectors are something like a sinusoidal functions.

6 YAMASHITA: OPTIMUM SAMPLING VECTORS FOR WIENER FILTER NOISE REDUCTION 63 Fig. 4. Dimension K of the subspace spanned by sampling vectors when noise is added after sampling. Fig. 5. Optimum sampling and restoration vectors when noise is added after sampling. ( =0:95, =0:2). Fig. 3 illusates the mean square error versus the number of sampling vectors when noise is added after sampling. From Fig. 2, we can also see the advantage of the optimum sampling. Fig. 4 illusates the dimension of subspace spanned by versus when noise is added after sampling. For the optimum sampling, this value coincides with in the Theorems 2 and 3. By the periodic sampling, this value coincides with. The maximum dimension of the subspace spanned by vectors is. From Fig. 4, we can see that there exists the case that. The reason is that in the case, we can reduce the mean square error caused by noise when the dimension is decreased. When noise is added before sampling, this dimension is equal to in order to minimize the mean square error. Fig. 5 illusates the sampling vectors and the corresponding restoration vectors when noise is added after sampling. We can see the vectors are very complicated. IV. CONCLUSION We have explained the RKLT. By using the RKLT, we provided an optimum sampling vector that minimizes the mean square error between the original signal and the restored signal

7 64 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 1, JANUARY 2002 when noise is added before sampling. We also provided optimum sampling vectors when noise is added after sampling. By experimental results, we showed their advantages. Since the theorems we provided are for discrete discrete sampling, we should extend them to continuous-discrete sampling. APPENDIX PROOFS A. Proof of Proposition 1 The original form of the RKLT is given as follows. We define a maix as (A.1) Let be a set of vectors of a singular value decomposition (SVD) of such that (A.2) where, and both and are orthonormal bases in. An RKLT of rank not greater than is given as (A.3) Since is a set of vectors of a SVD of,wehave (A.4) (A.5) Equations (A.4) and (A.5) yield that (A.6) Equation (A.6) yields that is the eigenvector corresponding to the eigenvalue of. Equations (A.3) and (A.5) yield that Lemma 1: For in (8), we have (A.8) (A.9) Lemma 2: When is a Wiener filter, in (19) is given as (A.10) Lemma 3: Let be an maix such that. Under this condition, if minimizes in (A.10), we have with a Lagrange s multiplier. Lemma 4: For symmeic maices and,if and is a symmeic maix, we have (A.11) (A.12) (A.13) From Lemma 1, we consider minimizing among maices such that. From Lemmas 2 and 3, its solution has to satisfy (A.11). By multiplying to (A.11) from the right-hand side, since is a symmeic maix, Lemma 4 yields that and are commutative. Then, we can assume that eigenvectors of are given by without loss of generality. Since is a basis, we can expand as (A.14) with a set of vectors. Since is a set of eigenvectors of and an orthonormal basis, we have (A.15) when. From (A.15), we can assume that (A.7) with an orthonormal basis.wehave (A.16) and a set of real numbers B. Proof of Theorem 1 Since rank, Proposition 1 yields that the criterion is minimum if and only if is an RKLT not greater than. The minimum value of is easily obtained from [13]. This completes the proof. C. Proof of Theorem 2 For the proof of Theorem 2, we provide the following lemmas. Their proofs are also given in this Appendix. We define real numbers as (A.17) (A.18) Since is a positive definite maix,. Equations (A.10), (A.14), and (A.16) (A.18) yield that (A.19)

8 YAMASHITA: OPTIMUM SAMPLING VECTORS FOR WIENER FILTER NOISE REDUCTION 65 and Lemma 7: Let (, )be integers such that if, and (A.29) The problem that we have to solve is minimizing under that condition that (A.20) in (A.19) (A.21) Then, minimizes under the conditions (A.18) and (A.28) if (A.30) From Lemma 6, (A.28) holds. This yields that the numerator of the first term in (A.27) is minimum if and only if (A.30) holds. On the other hand, since, (A.18) and (A.28) yield that the denominator of the first term in (A.27) is maximum if (A.31) with respect to and, where are given in (A.18). First, we fix and minimize in (A.19) with respect to. We define a functional (A.22) From the two results and (A.18), when is minimum, we have (A.32) Then, (A.16), (A.25), and (A.26) yield that (A.33) with a Lagrange s multiplier. If minimize,wehave Equation (A.23) yields that or (A.23) (A.24) We return to the problem where are not fixed. We then have the following lemmas. Lemma 5: If and minimize,we can assume without loss of generality that there exists an integer, and we have (A.34) (A.35) for every. Let be the maximum integer under (25) and (26). Let be an integer such that. Since are monotone increasing, the minimum for coincides with the minimum for under the condition that for. Then, the minimum for is not less than the minimum for. Since,wehave. This completes the proof. D. Proof of Theorem 3 Briefly, we let (A.25) (A.26) (A.27) and we use the notations in the proof of Theorem 2. Since, (34) and (A.20) yield that has to satisfy for every. Lemma 6: If and minimize,we have (A.28) (A.36)

9 66 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 1, JANUARY 2002 Equation (34) yields that By substituting (21) to (A.39), the Sherman Morrison Woodbury formula (the maix inversion lemma) [20] yields that (A.40) Let. It follows that (A.37) G. Proof of Lemma 3 When minimizes under the condition that, the variation of (A.41) (A.38) with respect to has to be 0 for any small with a Lagrange multiplier [17]. Neglecting not less than the second-degree terms of in,wehave so that is an orthonormal system. Equations (A.36) and (A.37) and Corollary 1 yield that are the optimum sampling vectors. E. Proof of Lemma 1 Equation (8) yields that This completes the proof. Equation (A.39) yields that This completes the proof. H. Proof of Lemma 4 Since,, and are symmeical maices (A.42) holds. By multiplying from both sides, we have (A.43) F. Proof of Lemma 2 Since we assume that and are uncorrelated, (2), (3), and (19) yield that It follows that so that we have (A.44) (A.45) Equation (A.45) yields that (A.39) Since,wehave. (A.46)

10 YAMASHITA: OPTIMUM SAMPLING VECTORS FOR WIENER FILTER NOISE REDUCTION 67 I. Proof of Lemma 5 We assume that and are the solutions of this problem. Suppose that and for an integer. We define real numbers and vectors as Suppose that for an integer. Then, there exists an integer such that. We define as (A.53) (else) (else) (A.47) (A.48) and let. Since and satisfy the conditions of the problem, and also satisfy them. When, using and is less than using and. From this conadiction, we can assume that there exists an integer such that if and only if. (When, we can also assume it without loss of generality.) Then, (A.19) and (A.21) yield (A.25) (A.27). The condition for is clear from (A.26) and rank. J. Proof of Lemma 6 We assume that and are the solutions of this problem. Let. is an orthogonal projection maix. Suppose that. Since is a symmeic maix, we have so that there exists an integer such that. For a real number, we define vectors as Let. From (A.19) and (A.21), it is clear that, by using, can be smaller than that by using. This conadiction yields that for every. Since if, the lemma is proved. K. Proof of Lemma 7 We assume that minimize under (A.28). For an integer, we assume that for every for the mathematical induction. Suppose that. From (A.28) and this assumption, we can consider that are included in the subspace spanned by. It is clear that is not an eigenvector of. However, if for every, is an eigenvector of. Therefore, there exists an integer such that. For a real number, we define as Let.Wehave (else). (A.54) It is clear that (A.49) (A.50) (A.55) (A.56) for all so that satisfy (A.28). Since, there exists such that Let. Since is a non-negative definite maix, yields that and. Then, there exists such that and for all. From (A.19) and (A.21),, by using, can be smaller than by using. This conadiction yields that. Therefore, we assume that the range of coincides with the subspace spanned by eigenvectors of, where if. Let be real numbers such that (A.51) Since and both and are orthonormal bases of the range of,wehave (A.52) Since is upwards convex, (A.28) yields that (A.57) (A.58) Equation (A.58) yields that. We can prove this lemma by using the mathematical induction. REFERENCES [1] C. E. Shannon, Communications in the presence of noise, Proc. IRE, vol. 37, pp , Jan [2] H. P. Kramer, A generalized sampling theorem, J. Math. Phys., vol. 38, no. 1, pp , Apr [3] A. Papoulis, Error analysis in sampling theory, Proc. IEEE, vol. 54, no. 7, pp , July [4] A. Jerry, The Shannon sampling theorem-its various extensions and applications: A tutorial review, Proc. IEEE, vol. 65, pp , Nov [5] H. Ogawa, A generalized sampling theorem, Trans. IEICE, Part A, vol. J71-A, no. 2, pp , Feb

11 68 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 1, JANUARY 2002 [6] A. I. Zayed, Advances in Shannon s Sampling Theory. New York: CRC, [7] J. R. Higgins, Sampling Theory in Fourier and Signal Analysis Foundations. Oxford, U.K.: Clarendon, [8] Y. Yamashita and H. Ogawa, Relative Karhunen-Loève ansform and optimum sampling, in Proc. Forth Int. Conf. Optim.: Techn. Appl., vol. 2, July 1 3, 1998, pp [9] P. A. Devijver and J. Kittler, Pattern Recognition: A Statistical Approach. Englewood Cliffs, NJ: Prentice-Hall, [10] E. Oja, Subspace Methods of Pattern Recognition. Letchworth, Hertfordshire, U.K.: Research Studies, [11] H. Ogawa and E. Oja, Projection filter, Wiener filter and Karhunen- Loève subspaces in digital image restoration, J. Math. Anal. Appl., vol. 114, no. 1, pp , Feb [12] H. Ogawa, Karhunen-Loève subspace, in Proc. 11th IAPR Int. Conf. Patt. Recogn., vol. 2, The Hague, The Netherlands, Aug. Sept. 30 3, 1992, pp [13] Y. Yamashita and H. Ogawa, Relative Karhunen-Loève ansform, IEEE Trans. Signal Processing, vol. 44, pp , Feb [14] Y. Hua and W. Q. Liu, Generalized Karhunen-Loève ansform, IEEE Signal Processing Lett., vol. 5, pp , June [15] K. I. Diamantaras and S. Y. Kung, Principal Component Neural Networks: Theory and Applications. New York: Wiley, [16] K. I. Diamantaras, K. Hornik, and M. G. Sintzis, Optimum linear compression under unreliable representation and robust PCA neural models, IEEE Trans. Neural Networks, vol. 10, Sept [17] D. G. Luenberger, Optimization by Vector Space Methods. New York: Wiley, [18] A. Albert, Regression and the Moore-Penrose Pseudo-Inverse. London, U.K.: Academic, [19] A. Ben-Israel and T. N. E. Greville, Generalized Inverses: Theory and Applications. New York: Wiley, [20] W. W. Hager, Updating the inverse of a maix, SIAM Rev., vol. 31, no. 2, pp , June Yukihiko Yamashita (M 94) was born in 1960 in Kanagawa, Japan. He received the B.E., the M.E., and the Dr. Eng. degrees from Tokyo Institute of Technology, Tokyo, Japan, in 1983, 1985, and 1993, respectively. From 1985 to 1988, he was with the Japan Atomic Energy Research Institute. From 1988 to 1989, he was with the ISAC Corporation. In 1989, he joined the faculty of the Tokyo Institute of Technology, where he is now an associated professor with the Graduate School of Science and Engineering. His research interests include pattern recognition and image processing. Dr. Yamashita received a Paper Award in 1993 from the Institute of Eleconics, Information, and Communication Engineers of Japan (IEICE). He is a member of the Institute of Eleconics, Information, and Communication Engineers of Japan, Information Processing Society of Japan, and the Audio Visual Information Research Group of Japan.