ON OPTIMIZATION OF THE MEASUREMENT MATRIX FOR COMPRESSIVE SENSING

Transcription

1 8th Euroean Signal Processing Conference (EUSIPCO-2) Aalborg, Denmark, August 23-27, 2 ON OPTIMIZATION OF THE MEASUREMENT MATRIX FOR COMPRESSIVE SENSING Vahid Abolghasemi, Saideh Ferdowsi, Bahador Makkiabadi,2, and Saeid Sanei Centre of Digital Signal Processing, School of Engineering, Cardiff University, Cardiff, CF24 3AA, UK 2 Electrical Engineering Deartment, Islamic Azad University, Ashtian Branch, Ashtian, IRAN {abolghasemiv, ferdowsis, makkiabadib, saneis}@cf.ac.uk ABSTRACT In this aer the roblem of Comressive Sensing (CS) is addressed. The focus is on estimating a roer measurement matrix for comressive samling of signals. The fact that a small mutual coherence between the measurement matrix and the reresenting matrix is a requirement for achieving a successful CS is now well known. Therefore, designing measurement matrices with smaller coherence is desired. In this aer a gradient descent method is roosed to otimize the measurement matrix. The roosed algorithm is designed to minimize the mutual coherence which is described as absolute off-diagonal elements of the corresonding Gram matrix. The otimization is mainly alied to random Gaussian matrices which is common in CS. An exted aroach is also resented for sarse signals with resect to redundant dictionaries. Our exeriments yield romising results and show higher reconstruction quality of the roosed method comared to those of both unotimized case and revious methods.. INTRODUCTION Comressive Sensing (CS) [] [2] is one of the recent interesting fields in signal and image rocessing communities. It has been utilized in many different alications from biomedical signal and image rocessing [3] to communication [4] and astronomy [5]. The core idea in CS is a novel samling technique which under certain conditions can lead to a smaller rate comared to conventional Shannon s samling rate. The key requirement for achieving a successful CS is comressibility or more recisely sarsity of the inut signal. A sarse signal has a small number of active (nonzero) comonents comared to its total length. This roerty can either exist in the samling domain of the signal or with resect to other basis such as Fourier, wavelet, curvelet or any other basis. A CS scenario mainly consists of two crucial arts; encoding (samling) and decoding (recovery). We formally exlain both arts in this section, but the focus in this work is on the first art where we try to imrove the sensing rocess which consequently affects the reconstruction erformance. Let x R n be a sarse signal, meaning to have at most s n nonzero elements. We now want to find linear measurements termed as y = Φx, where s < < n, and Φ R n is the measurement matrix. Obviously, = s is the best ossible choice. However, since we are not aware of the locations and the values of the nonzero elements when reconstructing x from y, a larger is needed to guarantee the recovery of x [6]. In addition, the structure of Φ is not arbitrary, and should be adoted based uon some secific rules. For instance, it has been roved [6] that for a Φ with i.i.d Gaussian entries with zero mean and variance / the bound C s log(n/) is achievable and can guarantee the exact reconstruction of x with overwhelming robability. The same condition can be alied to binary matrices with indeent entries taking values ±/. As another examle, we imly Fourier ensembles which are obtained by selecting renormalized rows from the n n discrete Fourier transform, with the bound C s (logn) 6. Now, assume that x is not sarse in the resent form (e.g. time domain), but it can be sarsely reresented in another basis (e.g. Fourier, wavelet, etc.). Mathematically seaking, we name α the sarse version of x satisfying x = Ψα, where Ψ R n m is called the reresenting (or sarsifying) dictionary (or basis). Ψ is called a comlete dictionary if n = m, otherwise n < m and we call it overcomlete (or redundant). For consistency in notations we always consider comlete dictionaries thorough the aer, unless it is stated. The extension to overcomlete dictionaries is easy though. In order to achieve a successful CS, we must choose a Φ having least ossible coherence with Ψ. In other words, a D := ΦΨ is desired to have columns with small correlations [7]. This roerty is termed as mutual coherence, and usually denoted by µ. More details about µ will be discussed later in this aer. The mutual coherence also affects the bound for such that C µ 2 s (logn) 6 [6]. Interestingly, it is shown that random ensembles are largely incoherent with any fixed basis [7]. This is a useful roerty which allows us to nonadatively choose a random Φ for any tye of signal. More details and roofs in this context can be found in [6] and the references therein. The second crucial job in CS is reconstruction, which can be described as solving the well known underdetermind roblem with sarsity constraint, min α s.t. y = ΦΨα Dα () where k denotes the l k -norm in general, and here, l -norm gives the suort of α, which is actually the level of sarsity. Although this roblem is sometimes described in other forms, the main issue is to find the sarsest ossible α satisfying y = Dα. Unfortunately, the roblem () is nonconvex in general, and the solution needs a combinatorial search among all ossible sarse α, which is not feasible. However, there are some greedy methods trying to iteratively solve (). The family of Matching Pursuit (MP) [8] methods such as Orthogonal Matching Pursuit (OMP), stagewise OMP [9], and Iterative Hard Thresholding (IHT) [] mainly attemt C is a comutable constant. EURASIP, 2 ISSN

2 to solve () by selecting the vectors of D which are mostly correlated with y. Thanks to the greedy nature of these algorithms; they are fast. In contrast, there have been roosed some otimization-based methods, mainly achievable by linear rogramming, called Basis Pursuit (PB) [] which attemt to recover α by converting () into a convex roblem which relaxes the l -norm to an l -norm roblem: min α s.t. y = ΦΨα Dα (2) Although random matrices are suitable choices for Φ, it has been recently shown that otimizing Φ with the hoe of reducing the mutual coherence can imrove the erformance [2 5]. Elad [2] attemts to iteratively decrease the average mutual coherence using a shrinkage oeration followed by a Singular Value Decomosition (SVD) ste. Duarte-Carvajalino et al. [3] take the advantage of an eigenvalue decomosition rocess followed by a KSVD-based algorithm [3] (see also [6]) to otimize Φ and train Ψ, resectively. Overally, the results of the current methods show enhancement in terms of both reconstruction error and comression rate. That motivates us to work more on otimizing the measurement matrix to obtain better results and also attemt to make the otimization rocess more efficient, secially for large-scale signals. In this aer we roose a gradient-based otimization aroach to decrease the mutual coherence between the measurement matrix and the reresenting matrix. This can be achieved by minimizing the absolute off-diagonal elements of the corresonding Gram matrix G = D T D, where D is the column-normalized version of D, and (.) T denotes the transosition. Our idea is to aroximate G with an identity matrix using a gradient descent method. In the next section we formally exress the sensing roblem and the required mathematics related to otimization of the measurement matrix. Then, in section 3, the gradientbased otimization method is described in details. In section 4, the simulations are given to examine the roosed method from ractical ersectives. Finally, the aer is concluded in section PROBLEM FORMULATION Similar to the notations in the revious section, we consider the signal x to be sarse with cardinality s over the dictionary Ψ R n m. Consider the noiseless case, we are going to take < n linear measurements which based on CS rules can be done by multilying a Φ with random i.i.d Gaussian entries such that y = ΦΨα 2. However, cannot exceed the bounds mentioned in the revious section, which highly des on the coherence between Φ and Ψ and the level of sarsity s. One of the suitable ways to measure the coherence between Φ and Ψ is to look at the columns of D, instead. As D = ΦΨ, the mutual coherence (which is desired to be minimized) can be defined as the maximum absolute value and normalized inner roduct between all columns in D which can be described as follows [], { } d T i µ(d) = max d j i j, i, j m d i d j (3) 2 Note that in cases where x is sarse in the current domain, we ignore Ψ and consider x = α and Φ = D without loss of generality. Another suitable way to describe µ, esecially for the urose of this aer, is to comute the Gram matrix G = D T D, where D is column-normalized version of D. We then define µ mx (which is equal to µ(d)), as the maximum absolute off-diagonal elements in G, µ mx = max g i j. (4) i j, i, j m Moreover, the average absolute off-diagonal elements in G is another useful measure defined as µ av = i j g i j m(m ). (5) Mainly, there are two imortant reasons why we are interested in matrices with small coherence and these motivate us to otimize Φ with the aim of decreasing the coherence. Suose that the following inequality holds: α < ( + ) (6) 2 µ(d) then, α is necessarily the sarsest reconstructed signal when y and D are known, and both BP and OMP are guaranteed to succeed [2] [7] [8]. This imlies that as long as µ is very small, a successful reconstruction is achievable for a wider range of sarsity degree. Another key notion here is considering the Restricted Isometry Proerty (RIP) [6] [], which imlies that for a roer isometry constant, RIP ensures any subset of columns in D with cardinality less than sarsity level s, is nearly orthogonal. This results in a better CS behavior and guarantees the identifiability of the original signal by both OMP and BP [3]. 3. PROPOSED APPROACH So far, it has been realized that a low coherence between columns of D is a desired roerty in CS framework. In section 2, the mathematical exression of coherence led to comuting the Gram matrix and from that oint we concluded that small absolute off-diagonal elements in D is desired. Let us now consider the ideal case, where minimum ossible coherence (µ mx = µ av = ) occurs. This situation will give us G = I mm, where I mm is identity matrix with indicated dimensions (note that D is column-normalized). Unfortunately, this may only occur when m, which is meaningless in CS. However, we might be able to introduce a measurement matrix leading to a G as close as ossible to identity, even if < m. One way to rovide such a matrix is to solve the following roblem, Ĝ = argmin G G I 2 (7) where. is defined as the maximum absolute off-diagonal elements of G. However, we refer to use the Frobenius norm denoted as. F, which has the advantages of simlifying the minimization roblem, and also articiating not only the maximum absolute off-diagonal, but all off-diagonal elements of G in the minimization rocess. Therefor, minimizing (7) with Frobenius norm will effectively, but not directly, minimize µ mx and µ av. In order to set u a ractical cost function for our roblem, assume either the case where x is sarse in current domain (i.e. D = Φ), or the dictionary Ψ is square (n = m). 428

3 Note, however, the extension to the case of redundant dictionaries (n < m) is easy and will be exlained later in the sequel. If we then exress G in terms of D and relace it in (7) we obtain the following unconstrained minimization roblem, ˆD = argmin D T D I 2 F (8) D To minimize (8), we adot a gradient-descent method. To do this, we first define the corresonding error as, E = D T D I 2 F= Tr{( D T D I)( D T D I) T } (9) where Tr{ } denotes the trace oeration. Then, we need to comute the gradient of E with resect to elements of D, denoted by d i j, E E d i j = 4 D( D T D I) () Once the gradient of E is obtained, the solution for (8) can be described as an iterative rocess to udate D D η E, where η > is the stesize which can be fixed or variable. Consequently, the full descrition of udate equation in each iteration is exressed as, D (i+) = D (i) ˆη D (i) ( D T (i) D (i) I) () where i is the iteration index and ˆη is the new stesize after merging the scaler 4 in () with η. The roosed algorithm starts with an initial random D () and then iteratively udating D. In addition, a normalization ste for columns of D, denoted by d j with j =,2...n, is required at each iteration: d j (i+) d j (i) / d j (i) 2 (2) After K iterations, when the convergence condition(s) is(are) met, ˆD = D (K) is given as the solution for (8). Finally, if x is sarse in its current domain, ˆΦ = ˆD would actually be the required measurement matrix and the algorithm is terminated. However, if x is sarse over Ψ nn, then an inverse or seudoinverse is required to obtain the measurement matrix ˆΦ = ˆDΨ. Note that the gradient descent methods, mainly do not guarantee a global minimum, but can normally rovide a local minimum. The seudocode of the roosed algorithm is given in Algorithm. In order to ext the above algorithm for the case of sarse signals over a dictionary Ψ nm, with n < m, we only need to consider Ψ in converting (7) to (8). Since G = D T D = Ψ T Φ T ΦΨ, we modify the minimization roblem and obtain, ˆΦ = argmin Ψ T Φ T ΦΨ I 2 F (3) Φ The gradient of the error is then comuted, E φ i j = 4ΦΨ(Ψ T Φ T ΦΨ I)Ψ T (4) and the udate equation which directly udates Φ is exressed as: Φ (i+) = Φ (i) ηφ (i) Ψ(Ψ T Φ T (i) Φ (i)ψ I)Ψ T. (5) Algorithm : Gradient-descent otimization. Inut: Sarse reresentation basis Ψ nn (if necessary), Stesize η, Maximum number of iterations K. Outut: Measurement matrix Φ n. begin Initialize D to a random matrix. for k= to K do for j= to n do d j d j / d j 2 D D ηd(d T D I) if Ψ nn has been given as inut then ˆΦ DΨ else ˆΦ D 4. SIMULATION RESULTS In this section we illustrate some results of our exeriments to show the ability of the roosed method in otimizing the measurement matrix and consequently its effect in the reconstruction rocess. The results are encouraging and show that the idea of otimizing the measurement matrix can increase the erformance in CS framework. In the first exeriment we built u a random dictionary Ψ (2 4) and a random Φ ( 2) both with i.i.d Gaussian elements. We then ran the roosed algorithm with η =.2 and 6 number of iterations to otimize Φ. We also alied Elad s algorithm [2] to the same Φ and Ψ. The arameters we used for were: the shrinkage arameter γ =.5, a fixed threshold t =.25 and 6 number of iterations. Figure shows the distribution of absolute off-diagonal elements of G for this exeriment. As seen from the figure, after alying the roosed method, the distribution becomes denser with more coefficients close to zero. This change verifies a smaller coherence between Φ and Ψ, which is also confirmed by having µ av =.75 and µ mx =.3887 in this exeriment, comared to µ av =.48 and µ mx =.4995, for unotimized Φ. imroves the coherence, almost similarly; µ av =.87 and µ mx = However, large µ av in, also reorted in [3], is due to an undesired eak around.5 in Figure, which may weaken the RIP conditions [3]. This drawback has been well mitigated in the roosed method as can be noticed from Figure. The same results are also seen for random i.i.d Gaussian Φ ( 2) and Discrete cosine transform (DCT) Ψ (2 2), shown in Figure (b). In the next exeriment we generated a set of sarse signals with the length of n = 2, at random locations and random amlitudes, for nonzero samles. We chose DCT bases for Ψ, with the size of 2 2, i.e. n = m = 2. In this exeriment we used η =. and 5 number of iterations. For we used γ =.95, relative threshold t = 2% and number of iterations: First, we fixed the sarsity level to s = 8 and varied from 2 to 8 in taking the measurements y = Φ n Ψ nn α. We then alied three reconstruction methods: IHT, OMP, and BP on all signals. This exeriment was reeated for each [2 8], and then the relative error 429

4 Distribution Distribution Amlitude Φ: Random, and Ψ: Random Amlitude (b) Φ: Random, and Ψ: DCT (b) Figure : Distribution of off-diagonal elements of G. rate between the reconstructed signals and the original signals were comuted. The result of this exeriment is shown in Figure 2. It is seen that otimization of Φ has a considerable influence in reducing the reconstruction error. In addition, it is observed that the roosed method works better comared to the work in [2]. It is also seen that the error rate is almost similar for both roosed and Elad s, when using BP for recustruction (Figure 2 (c)). Second, in order to evaluate the erformance of the roosed method against changes in cardinality, we fixed = 3 and varied the sarsity level from to 2, and then reconstructed x by alying IHT and OMP for all sarse signals. The comuted relative error rate for this exeriment is shown in Figure 3. Again we see imrovement comared to unotimized case. In the last exeriment, we studied the effects of the roosed otimization on samling and reconstruction of real images. Due to huge number of ixels, it is mainly imossible to rocess the whole image, directly. Hence, we alied the multi-scale strategy roosed in [9] and [2], considering wavelet transform as the reresenting dictionary to sarsify the inut image. We used symmlet8 wavelet with coarsest scale at 4 and the finest scale at 5, and followed the same rocedure as in [9]. As an illustrative examle, the Mondrian image of size was comressed using unotimized and otimized measurement matrices. Following the same rocedure in [9] to kee the aroximation coefficients, the whole image was then comressed to 87 samles. Then, the image was reconstructed using HALS CS method [9, 2]. As can be seen from Figure 4, the reconstruction error comuted as ε = ˆx x 2 / x 2, where ˆx is the reconstructed image, is less when we comress the image using the roosed otimized measurement matrix. We also recorded the running time of this exeriment where a deskto comuter with a Core 2 Duo CPU of 3. GHz, and 2 G Bytes of RAM was used. It is seen from Figure 4 that adding the otimization ste increases the running time, as (c) Figure 2: vs. the number of measurements, using IHT, (b) OMP, and (c) BP for reconstruction s s Figure 3: vs. the sarsity level s, using IHT and (b) OMP for reconstruction. exected. However, less comutation time of the roosed method comared to Elad s is noticeable. (b) 5. DISCUSSION AND CONCLUSION In this aer the roblem of comressive sensing is investigated. A new gradient-based aroach is roosed in which the aim is to otimize the measurement matrix in order to decrease the mutual coherence. An extension to the roosed algorithm is also resented which is suitable for sarse signals with resect to overcomlete dictionaries. The results of our simulations on both real and synthetic signals are romising and confirm that otimization of the measurement matrix increases the erformance, which is introduced in terms of reconstruction error and the number of measurements taken. In ractice, however, the roosed method is still not alicable to very large-scale roblems. Lower comlexity of the roosed method, comared to the revious methods, indi- 43

5 (c) ε =.57, t = 9.37 sec (b) ε =.68, t = 2.3 sec (d) ε =.63, t = sec. Figure 4: Reconstruction of Mondrian image using HALS CS method. Original image. Reconstruction with (b) no otimization, (c) roosed otimization, and (d) Elad s otimization of the measurement matrix. cates the ossibility of aroriateness of such methods for very large-scale roblems. This fact has not been reorted in the literature yet and needs to be challenged. REFERENCES [] D. Donoho. Comressed sensing. Information Theory, IEEE Transactions on, 52(4):289 36, Aril 26. [2] E. J. Candès, J. Romberg, and T. Tao. Robust uncertainty rinciles: exact signal reconstruction from highly incomlete frequency information. IEEE Transactions on Information Theory, 52(2):489 59, February 26. [3] M. Lustig, D. Donoho, J.M. Santos, and J.M. Pauly. Comressed sensing MRI. Signal Processing Magazine, IEEE, 25(2):72 82, March 28. [4] S.F. Cotter and B.D. Rao. Sarse channel estimation via matching ursuit with alication to equalization. Communications, IEEE Transactions on, 5(3): , Mar 22. [5] J. Bobin, J.-L. Starck, and R. Ottensamer. Comressed sensing in astronomy. Selected Toics in Signal Processing, IEEE Journal of, 2(5):78 726, Oct. 28. [6] E. J. Candès, J. K. Romberg, and T. Tao. Stable signal recovery from incomlete and inaccurate measurements. Communications on Pure and Alied Mathematics, 59(8):27 223, 26. [7] E. J. Candès and M. B. Wakin. An introduction to comressive samling. IEEE Signal Processing Magazine, 25(2):2 3, March 28. [8] J. A. Tro and A. C. Gilbert. Signal recovery from random measurements via orthogonal matching ursuit. Information Theory, IEEE Transactions on, 53(2): , Dec. 27. [9] D. Donoho, Y. Tsaig, I. Drori, and J-l. Starck. Sarse solution of underdetermined linear equations by stagewise orthogonal matching ursuit. Technical reort, 26. [] T. Blumensath and M. Davies. Iterative thresholding for sarse aroximations. Journal of Fourier Analysis and Alications, 24. [] D. Donoho and P. B. Stark. Uncertainty rinciles and signal recovery. SIAM Journal on Alied Mathematics, 49(3):96 93, 989. [2] M. Elad. Otimized rojections for comressed sensing. Signal Processing, IEEE Transactions on, 55(2): , Dec. 27. [3] J.M. Duarte-Carvajalino and G. Sairo. Learning to sense sarse signals: Simultaneous sensing matrix and sarsifying dictionary otimization. Image Processing, IEEE Transactions on, 8(7):395 48, July 29. [4] P. Vinuelas-Peris and A. Artes-Rodriguez. Sensing matrix otimization in distributed comressed sensing. In Statistical Signal Processing, 29. SSP 9. IEEE/SP 5th Worksho on, ages , Aug. 29. [5] V. Abolghasemi, S. Sanei, S. Ferdowsi, F. Ghaderi, and A. Belcher. Segmented comressive sensing. In Statistical Signal Processing, 29. SSP 9. IEEE/SP 5th Worksho on, ages , Aug. 29. [6] M. Aharon, M. Elad, and A. Bruckstein. K-SVD: An algorithm for designing overcomlete dictionaries for sarse reresentation. Signal Processing, IEEE Transactions on, 54(): , 26. [7] J.A. Tro. Greed is good: algorithmic results for sarse aroximation. Information Theory, IEEE Transactions on, 5(): , Oct. 24. [8] D. L. Donoho and M. Elad. Otimally sarse reresentation in general (nonorthogonal) dictionaries via l- minimization. PNAS, (5): , March 23. [9] A. Cichocki, R. Zdunek, A. H. Phan, and S. Amari. Nonnegative Matrix and Tensor Factorizations: Alications to Exloratory Multi-way Data Analysis and Blind Source Searation. John Wiley, 29. [2] Y. Tsaig and D. L. Donoho. Extensions of comressed sensing. Signal Processing, 86(3):549 57, 26. Sarse Aroximations in Signal and Image Processing. [2] S. Sanei, A.H. Phan, Jen-Lung Lo, V. Abolghasemi, and A. Cichocki. A comressive sensing aroach for rogressive transmission of images. In Digital Signal Processing, 29 6th International Conference on, ages 5, Jul