On the theoretical analysis of cross validation in compressive sensing

Transcription

1 MITSUBISHI ELECTRIC RESEARCH LABORATORIES On the theoretical analysis of cross validation in copressive sensing Zhang, J.; Chen, L.; Boufounos, P.T.; Gu, Y. TR May 2014 Abstract Copressive sensing (CS) is a data acquisition technique that easures sparse or copressible signals at a sapling rate lower than their Nyquist rate. Results show that sparse signals can be reconstructed using greedy algoriths, often requiring prior knowledge such as the signal sparsity or the noise level. As a substitute to prior knowledge, cross validation (CV), a statistical ethod that exaines whether a odel overfits its data, has been proposed to deterine the stopping condition of greedy algoriths. This paper analyses cross validation in a general copressive sensing fraework. Furtherore, we provide both theoretical analysis and nuerical siulations for a cross-validation odification of orthogonal atching pursuit, referred to as OMP-CV, which has good perforance in sparse recovery. ICASSP 2014 This work ay not be copied or reproduced in whole or in part for any coercial purpose. Perission to copy in whole or in part without payent of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by perission of Mitsubishi Electric Research Laboratories, Inc.; an acknowledgent of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payent of fee to Mitsubishi Electric Research Laboratories, Inc. All rights reserved. Copyright c Mitsubishi Electric Research Laboratories, Inc., Broadway, Cabridge, Massachusetts 02139

2

3 ON THE THEORETICAL ANALYSIS OF CROSS VALIDATION IN COMPRESSIVE SENSING Jinye Zhang Laing Chen Petros T. Boufounos Yuantao Gu State Key Laboratory on Microwave and Digital Counications Tsinghua National Laboratory for Inforation Science and Technology Departent of Electronic Engineering, Tsinghua University, Beijing , CHINA Mitsubishi Electric Research Laboratories, Cabridge, MA 02139, USA ABSTRACT Copressive sensing (CS) is a data acquisition technique that easures sparse or copressible signals at a sapling rate lower than their Nyquist rate. Results show that sparse signals can be reconstructed using greedy algoriths, often requiring prior knowledge such as the signal sparsity or the noise level. As a substitute to prior knowledge, cross validation (CV), a statistical ethod that exaines whether a odel overfits its data, has been proposed to deterine the stopping condition of greedy algoriths. This paper analyses cross validation in a general copressive sensing fraework. Furtherore, we provide both theoretical analysis and nuerical siulations for a cross-validation odification of orthogonal atching pursuit, referred to as OMP-CV, which has good perforance in sparse recovery. Index Ters Copressed sensing, signal reconstruction, cross validation, orthogonal atching pursuit 1. INTRODUCTION Copressive sensing (CS) is a data acquisition technique that easures sparse or copressible signals at sapling rate close to their intrinsic inforation rate rather than the Nyquist rate [2,3]. Broadly speaking, it consists of two ain building blocks: encoding an N- diensional k-sparse signal x by coputing its M N linear projections and decoding the signal using various sparse recovery ethods, such as linear prograing [4 6] or greedy algoriths [7 11]. While greedy algoriths are often preferred for their low coputational coplexity, ost of the require prior inforation such as signal sparsity or noise level for accurate reconstruction, without which the reconstructed signal ay fail to copletely reconstruct the signal or overfit the noise [17]. Such inforation not available in practice, cross validation (CV) [12 16], a statistical ethod that exaines whether a odel overfits the data, has been proposed as an alternative solution [17], where, in general, easureents are separated into reconstruction easureents and CV easureents; the forer are used to reconstruct the signal via a greedy algorith, while the latter to copute the stopping criterion. It has also been shown that CV can be used to estiate the recovery error. Specifically, using the Johnson-Lindenstrauss lea, we can derive theo- More details including proofs for the leas and theores in this paper can be found in [1]. This work was partially supported by the National Progra on Key Basic Research Project (973 Progra 2013CB329201) and the National Natural Science Foundation of China (NSFC ). Petros T. Boufounos is exclusively supported by Mitsubishi Electric Research Laboratories. The corresponding author of this paper is Yuantao Gu (gyt@tsinghua.edu.cn). retical bounds on the estiation error, quantifying how the nuber of CV easureents cv influences the estiation quality [18]. In this paper we provide an analysis of CV perforance in general CS probles. Furtherore, we analyse a variation of the orthogonal atching pursuit (OMP) algorith, called OMP-CV, which exploits CV to deterine the recovered signal sparsity. To analyse CV in general CS probles, we need to address the following two probles. First, given a recovered signal, we are interested in deterining how close it is to the original input signal. Second, given two candidate recovered signals, we are interested in deterining which of the two has lower recovery error. Both questions could be answered using CV. Given a signal, the recovery error ε x can be estiated by its CV residual ϵ cv; we attept to deterine the accuracy and probability the CV residual could provide bounds on the recovery error. Furtherore, given two signals, the coparison of two recovery errors can be estiated by their respective CV residuals; we are interested in the probability the coparison of the error and the CV residual are consistent with each other. In other words, these probles could be forulated as: 1. (Recovery error estiation) Given a recovered signal, with what accuracy and what probability could its CV residual provide bounds on its recovery error? 2. (Recovery error coparison) Given two recovered signals, with what probability could the coparison of their CV residuals correctly reflect their recovery errors? These probles are referred to as general CV probles in the reainder of the article. To analyse both probles, we first calculate the probability distribution of CV residuals. By transforing this distribution into inequalities that hold with certain probability, we directly answer the questions above. Our results for both probles are given in Section 3. We also analyse the OMP-CV algorith, described in Table 1. In this analysis we refer to the algorith output, which is the recovered signal with the sallest CV residual, as the OMP-CV output. The recovered signal with the sallest recovery error is referred to as the oracle optiu. Our ain result shows that the recovery error of the OMP-CV output is very close to that of the oracle optiu with high probability given that the oracle optiu recovers all indices in the support set of the signal. Note that OMP-CV includes two alost separate procedures: one is reconstructing the signal using OMP, the other is estiating the recovery error by CV. To achieve the ain result, we first analyse the internal structure of two recovered signals acquired in different iterations of OMP. We then study how their CV residuals correspond to their recovery errors using the techniques we developed for the general CV proble. Finally we generalize the recovery error coparison between two recovered signals, as generated

4 by the OMP iterations, to the coparison of all recovered signals. Thus we can estiate how close is the OMP-CV output to the oracle optiu. The reinder of the paper is organized as follows. Section 2 forulates the proble and describes the OMP-CV algorith. Section 3 gives the analysis of CV, while Section 4 gives the analysis of OMP-CV. Nuerical siulations are given in Section 5. Finally, Section 6 discusses our findings and concludes. 2. BACKGROUND 2.1. Notation and Proble Forulation We consider an unknown k-sparse signal x R N observed using M linear easureents corrupted by additive noise. Let T be its support set, using T = k to denote the size of T. The vector x T contains the coefficients of x indexed by T. To ipleent the CVbased odification, we separate the M by N sapling atrix to a reconstruction atrix A R N and a CV atrix A cv R cv N. Measureents are also separated accordingly, to reconstruction easureents y R and CV easureents y cv R cv. Our analysis only considers Gaussian sapling atrices and additive Gaussian noise, i.e., y = Ax + n, n = σ na n, y cv = A cvx + n cv, n cv = σ na cv,n, where eleents of A, A cv, a n, and a cv,n are i.i.d norally distributed with ean zero and variance 1/ so that the sensing atrix will have unit colun nor. For notational siplicity, we also define: Definition 1. (Generalized sapling atrix and input signal): Let A g [A, a n] and x g [x,σ n], then y = Ax + n = A gx g, (1) where A g is called the generalized sapling atrix, and x g is called the generalized input signal. In describing the OMP algorith, we use ˆx p to denote the recovered signal in the p-th iteration, and T p to denote its support set. The difference between the recovered signal ˆx p and the input signal x is denoted using x p and the corresponding recovery error using ε p x. The generalized versions of x p and ε p x are x p g and ε p g, respectively, and ϵ p cv denotes the CV residual of ˆx p. In other words, x p x ˆx p, ε p x x p 2 2, x p g [( x p ),σ n], ε p g x p g 2 2, ϵ p cv y cv A cvˆx p 2 2. To ake the analysis ore clear, we ephasize that in this paper, the input signal is considered as deterinistic while the sapling atrix and noise are rando. Without loss of clarity and for notational siplicity, the rando variables and their realizations are denoted by sae notation OMP-CV: Algorith Description The OMP-CV algorith, described in Table 1, is a noise- and sparsity-robust greedy recovery algorith that cobines OMP and CV [17]. The path it follows can be interpreted intuitively: as the signal estiate iproves, the recovery error and CV residual decrease; when the recovered signal starts to overfit the noise, the Table 1. OMP-CV Algorith Input: Reconstruction atrix A, CV atrix A cv, reconstruction easureents y, CV easureents y cv, iteration ties d; Output: The recovered signal ˆx. Initialization: Set p =1, ϵ 0 cv = y cv 2 2; Repeat: Copute ˆx p using an OMP iteration; Copute ϵ p cv = A cvˆx p y cv 2 2; Increent p by 1; Until: p d Copute o cv = argin ϵ p cv; p Return: ˆx = ˆx ocv. recovery error increases. The CV residual detects the change by starting to increase siultaneously. That is, CV residuals provide a good estiate for recovery errors and, further, the OMP-CV output is close to the oracle optiu. There are several advantages of OMP-CV. First, it does not require prior inforation such as noise level or sparsity. Instead, only the axiu nuber of iterations, d, is required as input 1. This can be set two or three ties larger than the sparsity k without hurting recovery perforance. Second, the algorith provides an estiate of the recovery error in its residual. Additionally, by properly setting cv, the recovery perforance of OMP-CV copetes with that of OMP when cobined with accurate inforation on the noise level. Earlier work has provided experiental support on the recovery perforance of OMP-CV [17, 18]. However, to the best of our knowledge, theoretical analysis on this algorith does not exist. Our work presents such analysis in Section 4. In addition, we also provide nuerical siulations in Section 5 to support our theoretical results. 3. CROSS VALIDATION IN COMPRESSIVE SENSING This section describes our results for general CV probles, as described in Section 1. We start with calculating the probability distribution of ϵ cv, i.e. Lea 1. Let ˆx be a recovered signal and ε x be its recovery error. Provided that cv is sufficiently large 2, then ϵ cv = y cv A cvˆx 2 2 N(µ, σ 2 ), (2) where µ = cv (εx + σ2 n), and σ 2 = 2cv (ε 2 x + σn) 2 2. The condition provided that cv is sufficiently large is used in one approxiation step of the proof which uses the Central Liit Theore (CLT). The actual probability distribution of ϵ cv converges absolutely to (2) as cv increases and the approxiation error becoes negligible very fast. In practical cases the nuber of CV easureents cv is often tens or hundreds and the approxiation is very good. The precision of the approxiation is also supported by the siulation result in Section 5.1. The sae condition is also 1 Note that d cannot be greater than, since regular OMP will produce zero residual after that and conclude. 2 An cv exceeding tens, which is coon in CS, eets the requireent.

5 required in Lea 2, Theore 1, and Theore 2 for the sae reason. One iediate consequence of Lea 1 is that ϵ cv can be used to provide an estiate of ε x in the for of an inequality that holds with certain probability. In particular: Theore 1. (Recovery error estiation): Provided that cv is sufficiently large, with probability erf( λ 2 ) the following holds h(λ, +)ϵ cv σ 2 n ε x h(λ, )ϵ cv σ 2 n, (3) where h(λ, ±) is a function related to λ defined as h(λ, ±) 1, (4) cv 2 1 ± λ cv and erf(u) is the error function of noral distribution, erf(u) 1 π u u e t2 dt. (5) Theore 1 provides an answer to Prob. 1 bounding the recovery error ε x by the interval [ h(λ, +)ϵcv σn,h(λ, 2 )ϵ cv σn 2 ]. (6) The difference of the upper bound and the lower bound, 2λ 2 ϵ cv, (7) cv cv 2λ2 cv is roughly proportional to 1/ 2/3 cv and becoes tighter as cv increases. For Prob. 2, we copute the distribution of ϵ cv = ϵ p cv ϵ q cv. Lea 2. Let ˆx p and ˆx q be two recovered signals, ε p g and ε q g be the their generalized recovery error, and ρ g xp g, x q g x p g 2 x q g 2. (8) Provided that cv is sufficiently large, then where µ = cv ϵ cv = ϵ p cv ϵ q cv N(µ, σ 2 ), (9) (εp g ε q g), and σ 2 = 2cv [(ε p g) 2 +(ε q g) 2 2ρ 2 gε p gε q g]. 2 Lea 2 brings us closer to deterining the probability that ϵ p cv >ϵ q cv given that ε p x >ε q x. In particular, we obtain: Theore 2. (Recovery error coparison) Let ˆx p and ˆx q be two recovered signals. If ε p x ε q x, it holds with probability Φ(λ) that ϵ p cv ϵ q cv, where λ is given by 1 λ = 2 2 cv [ 1+2(1 ρ 2 ε p gε q g g) (ε p g ε q g) 2 ], (10) and Φ(u) is the cuulative distribution function (CDF) of standard noral distribution, Φ(u) 1 2π u e t2 2 dt. (11) Theore 2 answers Prob. 2 by giving the probability that the coparison of CV residuals correctly reflect the coparison of the recovery errors. This probability increases with cv, ρ 2 g, and ε p g/ε q g. In other words, as the nuber of CV easureents increases, as recovered signals are ore correlated, and as the recovery error difference increases, this probability also increases. 4. SPARSITY- AND NOISE- ROBUST OMP This section provides our analysis on OMP-CV. Before stating our ain theore, we first define: Definition 2. (Ratio of unrecovered signal and noise): The ratio α p R, defined as α p x T \T p 2 σ n, (12) easures to what extent the signal ˆx has not been recovered by ˆx p. Next, we provide our ain theore for OMP-CV: Theore 3. In OMP-CV, assue that the oracle optiu is ˆx o and T T o. For any recovered signal ˆx p other than ˆx o : if T \T p, then ϵ o cv <ϵ p cv with probability Φ(λ), where λ cv 2 1 g(αp ); (13) if T \T p = and if ˆx p is the OMP-CV output, then with probability greater than {1 (d k)[1 Φ(λ 0)]} we have ε p g C 1ε o g, (14) where g(α p ) is roughly proportional to 1/(α p ) 2, λ 0 is a constant chosen to decide the probability with which (14) holds, and C 1 is only related to λ 0 and cv. Theore 3 supports the recovery perforance of OMP-CV. It divides recovered signals into two categories. For ˆx p with T \T p, the probability [1 Φ(λ)] decays sharply as α p increases. Since this is the probability that ϵ p cv <ϵ o cv, it is nearly ipossible for such ˆx p to be the OMP-CV output. For exaple, [1 Φ(λ)] would be less than 0.5% with α p = 1, and further drop to % with α p =2. As a result, the OMP-CV output recovers all indices of the support set with overwheling probability. Further, if the OMP-CV output recovers all indices of the support set, its recovery error can be bounded by ε o g with high probability showing that the recovery error of OMP-CV output is very close to that of the oracle optiu. Reark 1. Paraeter details of Theore 3 are: g(α p β 1(α p ) 2 + β 2 )= β 1(α p ) 2 + β 2 +ax((α p ) 2 β 3α p β 4, 0) 2 β [ 1/ (α p ) 2 ] + β 1, C 1 =2C C0 2 + C0, C 0 β 5λ 2 0/( cv 2λ 2 0), where betas are decided by RIP constant [4] of the sapling atrix. E.g., if δ d < 0.1, the values of betas are: β 1 =2.08, β 2 = β 3 = 0.03, β 4 =0.02, and β 5 = Further, if, e.g., (d k) =100, setting cv =48and λ 0 =4, produces a nuerical for of (14): with probability 99.7% we have ε p g 1.47ε o g (15) Apart fro the perforance analysis on OMP-CV, we also note that the recovery error of OMP-CV can be estiated using its CV residual by directly applying Theore 1.

6 1.5 Siulation result Theoretical result (a) average recovery error / db OMP CV OMP Recovery error with = Siulation result Theoretical result (b) Fig. 1. Validation for Lea 1 and Lea 2. Fig.1(a) validates Lea 1 by plotting the siulation result of probability distribution of ϵ cv, while Fig.1(b) validates Lea 2 by plotting that of ϵ cv (red curve). The siulation results agree well with the theoretical results, which are plotted in black as reference. The Kullback-Leibler divergences of the theoretical results fro the siulation results are and for Fig. 1(a) and Fig. 1(b) respectively. 5. NUMERICAL SIMULATION This section gives our siulation results. 3 Section 5.1 siulates the probability distribution of rando variables described in Lea 1 and Lea 2 to support both leas. Section 5.2 provides siulations on the perforance of OMP-CV and OMP as a function of cv. The siulations deonstrate the recovery perforance of OMP-CV. Throughout this section, Gaussian signals are used where non-zero entries are generated following the standard Gaussian distribution Validation for Lea 1 and Lea 2 As previously noted, CLT is used to approxiate the probability distribution of ϵ cv in Lea 1 and ϵ cv in Lea 2. The siulations in this section attept to validate this approxiation. In both siulations, paraeters are set as N = 512, = 96, cv = 48, and k =50. With recovered signals (ˆx for Lea 1; ˆx p and ˆx q for Lea 2) fixed, the rando CV atrix along with its noise is realized 1E5 ties and the probability distributions of rando variables (ϵ cv for Lea 1; ϵ cv for Lea 2) are calculated. The experient results, shown in Fig.1, indicate that the siulation results agree well with the theoretical prediction. This validates our approxiation and supports both leas Recovery Perforance of OMP-CV with Variation of CV Measureents Nuber Given a fixed total nuber of easureents M, there is a tradeoff between and cv [17]. On one hand, increasing will reduce 3 The code for siulation is available at Fig. 2. The trade Off between and cv. Paraeters are fixed as N =1000, k =50, σn 2 =0.1, and M =400. cv varies fro 100 to 10 with step 10 while = M cv. OMP-CV outperfors OMP except when cv is too sall. In addition, using sae nuber of easureents, OMP-CV has recovery perforance siilar to OMP (recovery error with = 400) with paraeters appropriately set even though the prior knowledge is required for OMP. the reconstruction error. On the other hand, increasing cv will iprove the CV estiate, and thus ake the OMP-CV output closer to the oracle optiu. This siulation epirically investigates the recovery perforance of OMP-CV as cv varies. In this experient, we set N =1000, k =50, σn 2 =0.1, and M =400. cv varies fro 100 to 10 with step 10 and we have = M cv. For OMP-CV, easureents are used for reconstruction, while cv easureents are used for CV. For OMP, easureents are used for reconstruction and the terination is based on residual with the accurate noise level given. In addition, the recovery perforance, where all M easureents are used for reconstruction using OMP, is given for coparison. We average 1000 repetitions for experients of each paraeter setting. The experient result, plotted in Fig.2, shows the best perforance of OMP-CV lies in the region where cv is neither too sall nor too large. OMP-CV outperfors OMP except when cv is very sall, indicating that CV-based terination is better than residual-based terination, even if the latter uses an exact noise level. Additionally, note that with the sae nuber of easureents at hand (M easureents for both OMP-CV and OMP), OMP-CV can achieve recovery perforance siilar to OMP with paraeters appropriately set, even though prior knowledge is required for OMP. In this sense, OMP-CV outperfors OMP. 6. CONCLUSION This paper presents a theoretical study of CV in copressive sensing, providing analysis of general CV probles as well as analysis of the OMP-CV algorith. As a highly practical algorith, OMP- CV could reconstruct the signal without prior knowledge such as sparsity or noise level; its perforance is supported in this paper both theoretically and epirically. Additionally, our results on general CV probles could also be used to apply CV to other CS-based reconstruction algoriths. CV sacrifices a sall aount of easureents to estiate the reconstruction error. In a nutshell, this technique akes it possible for greedy algoriths to reconstruct the signal without prior knowledge like the sparsity or noise level. In future work, we would like to extend our analysis on CV by studying the use of CV in other greedy sparse recovery algoriths.

7 7. REFERENCES [1] Jinye Zhang, Laing Chen, Petros T. Boufounos, and Yuantao Gu, Cross validation in copressive sensing and its application on op-cv algorith, subitted to IEEE for possible publication, available at [2] David L Donoho, Copressed sensing, Inforation Theory, IEEE Transactions on, vol. 52, no. 4, pp , [3] Eanuel J Candès, Copressive sapling, in Proceedings oh the International Congress of Matheaticians: Madrid, August 22-30, 2006: invited lectures, 2006, pp [4] Eanuel J Candes and Terence Tao, Decoding by linear prograing, Inforation Theory, IEEE Transactions on, vol. 51, no. 12, pp , [5] Eanuel Candes, Mark Rudelson, Terence Tao, and Roan Vershynin, Error correction via linear prograing, in Foundations of Coputer Science, FOCS th Annual IEEE Syposiu on. IEEE, 2005, pp [6] Eanuel J Candès, Justin Roberg, and Terence Tao, Robust uncertainty principles: Exact signal reconstruction fro highly incoplete frequency inforation, Inforation Theory, IEEE Transactions on, vol. 52, no. 2, pp , [7] Joel A Tropp and Anna C Gilbert, Signal recovery fro rando easureents via orthogonal atching pursuit, Inforation Theory, IEEE Transactions on, vol. 53, no. 12, pp , [8] Deanna Needell and Roan Vershynin, Unifor uncertainty principle and signal recovery via regularized orthogonal atching pursuit, Foundations of coputational atheatics, vol. 9, no. 3, pp , [9] David L Donoho, Yaakov Tsaig, Iddo Drori, and J-L Starck, Sparse solution of underdeterined systes of linear equations by stagewise orthogonal atching pursuit, Inforation Theory, IEEE Transactions on, vol. 58, no. 2, pp , [10] Wei Dai and Olgica Milenkovic, Subspace pursuit for copressive sensing signal reconstruction, Inforation Theory, IEEE Transactions on, vol. 55, no. 5, pp , [11] Deanna Needell and Joel A Tropp, Cosap: Iterative signal recovery fro incoplete and inaccurate saples, Applied and Coputational Haronic Analysis, vol. 26, no. 3, pp , [12] Pierre A Devijver and Josef Kittler, Pattern recognition: A statistical approach, Prentice/Hall International Englewood Cliffs, NJ, [13] Seyour Geisser, Predictive interference: an introduction, vol. 55, CRC Press, [14] Ron Kohavi et al., A study of cross-validation and bootstrap for accuracy estiation and odel selection, in IJCAI, 1995, vol. 14, pp [15] Richard R Picard and R Dennis Cook, Cross-validation of regression odels, Journal of the Aerican Statistical Association, vol. 79, no. 387, pp , [16] Jun Shao, Linear odel selection by cross-validation, Journal of the Aerican statistical Association, vol. 88, no. 422, pp , [17] Petros Boufounos, Marco F Duarte, and Richard G Baraniuk, Sparse signal reconstruction fro noisy copressive easureents using cross validation, in Statistical Signal Processing, SSP 07. IEEE/SP 14th Workshop on. IEEE, 2007, pp [18] Rachel Ward, Copressed sensing with cross validation, Inforation Theory, IEEE Transactions on, vol. 55, no. 12, pp , 2009.