arxiv: v1 [cs.it] 20 Jul 2016

Transcription

1 ONSAGER-CORRECTED DEEP LEARNING FOR SPARSE LINEAR INVERSE PROBLES ark Borgerding and Philip Schniter Dept. of ECE, The Ohio State Universit, Columbus, OH arxiv: v1 [cs.it] 2 Jul 216 ABSTRACT Deep learning has gained great popularit due to its widespread success on man inference problems. We consider the application of deep learning to the sparse linear inverse problem encountered in compressive sensing, where one seeks to recover a sparse signal from a small number of nois linear measurements. In this paper, we propose a novel neural-network architecture that decouples prediction errors across laers in the same wa that the approximate message passing (AP algorithm decouples them across iterations: through Onsager correction. Numerical experiments suggest that our learned AP network significantl improves upon Gregor and Le- Cun s learned ISTA network in both accurac and complexit. 1 Index Terms Deep learning, compressive sensing, sparse coding, approximate message passing. 1. INTRODUCTION We consider the problem of recovering a signals C N from a nois linear measurement = Φs+n C, (1 where Φ C N represents a linear measurement operator and n C a noise vector. In man cases of interest, N. We will assume that the signal vector s has a sparse representation in a known orthonormal basis Ψ C N N, i.e., that s = Ψx for some sparse vector x C N. Thus we define A ΦΨ C N, write (1 as = Ax+n, (2 and seek to recover an sparse x from. In the sequel, we will refer to this problem as the sparse linear inverse problem. Note that the resulting estimate x of x can be easil converted into an estimate ŝ ofsvia ŝ = Ψ x. The sparse linear inverse problem has received enormous attention over the last few ears, in part because it is central to compressive sensing [1] and sparse coding [2]. an methods have been developed to solve this problem. ost of the existing methods involve a reconstruction algorithm that inputs a pair(,a and produces an sparse estimate x. A mriad of such algorithms have been proposed, including both sequential (e.g., greed and iterative varieties. Some relevant algorithms will be reviewed in Section Section 2.1. Recentl, a different approach to solving this problem has emerged from the field of deep learning, whereb a neural network with man laers is trained using a set of D examples 2 1 This work was supported b the NSF I/UCRC grant IIP Since orthonormal Ψ implies x = Ψ H s, training examples of the form {( (d,s (d } can be converted to {( (d,x (d } D d=1 via x(d = Ψ H s (d. {( (d,x (d } D d=1. Once trained, the network can be used to predict the sparse x that corresponds to a given input. Note that knowledge of the operator A is not necessar here. Previous work (e.g., [3 6] has shown that the deep-learning approach to solving sparse linear inverse problems has the potential to offer significant improvements, in both accurac and complexit, over the traditional algorithmic approach. Relation to prior work: In this paper, we show how recent advances in iterative reconstruction algorithms suggest modifications to traditional neural-network architectures that ield improved accurac and complexit when solving sparse linear inverse problems. In particular, we show how Onsager correction, which lies at the heart of the approximate message passing (AP algorithm [7], can be emploed to construct deep networks with increased accurac and computational efficienc (i.e., fewer laers needed to produce an accurate estimate. To our knowledge, the application of Onsager correction to deep neural networks is novel. 2. ITERATIVE ALGORITHS AND DEEP LEARNING 2.1. Iterative Algorithms One of the best known algorithmic approaches to solving the sparse linear inverse problem is through solving the convex optimization problem [8, 9] 1 x = argmin x 2 Ax 2 2 +λ x 1, (3 where λ > is a tunable parameter that controls the tradeoff between sparsit and measurement fidelit in x. The convexit of (3 leads to provabl convergent algorithms and bounds on the performance of the estimate x (see, e.g., [1] ISTA One of the simplest approaches to solving (3 is the iterative softthresholding algorithm (ISTA [11], which consists of iterating the steps (for t =,1,2,... and x = v t = A x t x t+1 = η ( x t +βa H v t;λ, (4a (4b where β (,1/ A 2 2] is a stepsize, v t is the iteration-t residual measurement error, and η( ;λ : C N C N is the soft thresholding denoiser that operates componentwise as: [η(r;λ] j = sgn(r jmax{ r j λ,}. (5

2 FISTA Although ISTA is guaranteed to converge under β (,1/ A 2 2 [12], it converges somewhat slowl and so man modifications have been proposed to speed it up. Among the most famous is fast ISTA (FISTA [13], v t = A x t (6a x t+1 = η ( x t +βa H v t + t 2 t+1 ( xt xt 1;λ, (6b which converges in roughl an order-of-magnitude fewer iterations than ISTA (see Fig. 1. average NSE [db] ISTA FISTA AP AP Recentl, the approximate message passing (AP algorithm [7, 14] was applied to (3, giving v t = A x t +b tv t 1 x t+1 = η ( x t +A H v t;λ t, for x =, v 1 =, t =,1,2,..., and (7a (7b b t = 1 xt (8 λ t = α v t 2. (9 Here, α is a tuning parameter that has a one-to-one correspondence with λ in (3 [14]. Comparing AP to ISTA, we see two major differences: i AP s residual v t in (7a includes the Onsager correction term b tv t 1, and ii AP s denoising threshold λ t in (7b takes the prescribed, t-dependent value (9. We now describe the rationale behind these differences. When A is a tpical realization of a large i.i.d. (subgaussian random matrix with entries of variance 1, the Onsager correction decouples the AP iterations in the sense that the input to the denoiser, r t x t +A H v t, can be modeled as 3 r t = x+n(,σ 2 ti N with σ 2 t = 1 vt 2 2. (1 In other words, the Onsager correction ensures that the denoiser input is an additive white Gaussian noise (AWGN corrupted version of the true signal x with known AWGN variance σt 2. (See Fig. 5. The resulting problem, known as denoising, is well understood. For example, for an independent known priorp(x = N j=1 pj(xj, the mean-squared error (SE-optimal denoiser 4 is simpl the posterior mean estimator (i.e., x t+1,j = E{x j r t,j;σ t}, which can be computed in closed form for man p j(. In the more realistic case that p j( are unknown, we ma be more interested in the minimax denoiser, i.e., minimizer of the maximum SE over an assumed famil of priors. Remarkabl, for sparse priors, i.e.,p j(x j = (1 γδ(x j+γ p j(x j withγ (,1 and arbitrar unknown p j(, soft-thresholding (5 with a threshold proportional to the AWGN standard deviation (i.e., λ t = ασ t as in (9 is nearl minimax optimal [14]. Thus, we can interpret the AP algorithm (7 as a nearl minimax approach to the sparse linear inverse problem. 3 The AP model (1 is provabl accurate in the large-sstem limit (i.e.,, N with /N converging to a fixed positive constant [15]. 4 AP with SE-optimal denoising was first described in [16] iterations Fig. 1. Average NSE versus iteration number for AP, FISTA, ISTA (from left to right Comparison of ISTA, FISTA, and AP We now compare the average per-iteration behavior of ISTA, FISTA, and AP for an A drawn i.i.d. N(, 1. In our experiment, all quantities were real-valued, the problem dimensions were N = 5 and = 25, the elements of x were drawn i.i.d. N(,1 with probabilit γ =.1 and were otherwise set to zero, and the noise n was drawn i.i.d. N(, v with v set to ield a signal-to-noise ratio (SNR of 4 db. Recall that ISTA, FISTA, and AP all estimate x b iterativel minimizing (3 for a chosen value of λ (selected via α in the case of AP. We chose the minimax optimal value of α for AP (which is since γ =.1 [14] and used the corresponding λ for ISTA and FISTA. Figure 1 shows the average normalized SE (NSE versus iterationt, where NSE x t x 2 2/ x 2 2 and 1 realizations of (x,n were averaged. We see that AP requires roughl an order-of-magnitude fewer iterations than FISTA, which requires roughl an order-of-magnitude fewer iterations than ISTA Deep Learning In deep learning [17], training data {( (d,x (d } D d=1 composed of (feature,label pairs are used to train the parameters of a deep neural network with the goal of accuratel predicting the label x of a test feature. The deep network acceptsand subjects it to man laers of processing, where each laer consists of a linear transformation followed b a non-linearit. Tpicall, the label space is discrete (e.g., is an image and x is its class in {cat, dog,..., tree}. In our sparse linear inverse problem, however, the labels x are continuous and high-dimensional (e.g., C N or R N. Remarkabl, Gregor and LeCun demonstrated in [3] that a well-constructed deep network can accuratel predict even labels such as ours. The neural network architecture proposed in [3] is closel related to the ISTA algorithm discussed in Section To understand the relation, we rewrite the ISTA iteration (4 as x t+1 = η ( S x t +B;λ { B βa H with (11 S I N BA and unfold iterations t = 1...T, resulting in the T -laer feedforward neural network shown in Fig. 2. Whereas ISTA uses the

3 B η S + η S + η S + η x x 1 x 2 x 4 3 Fig. 2. The feed-forward neural network constructed b unfolding T =4 iterations of ISTA. values of S and B prescribed in (11 and a common value of λ at all laers, Gregor and LeCun [3] proposed to use laer-dependent thresholds λ [λ 1,λ 2,...,λ T] and learn both the thresholds λ and the matrices B,S from the training data {( (d,x (d } D d=1 b minimizing the quadratic loss L T(Θ = 1 D D x T( (d ;Θ x (d 2. (12 2 d=1 Here, Θ = [B,S,λ] denotes the set of learnable parameters and x T( (d ;Θ the output of the T -laer network with input (d and parameters Θ. The resulting approach was coined learned ISTA (LISTA. Relative to existing algorithms for the sparse linear inverse problem with optimall tuned regularization parameters (e.g., λ or α, LISTA generates estimates of comparable SE with significantl fewer matrix-vector multiplications. As an example, for the problem described in Section 2.1.4, LISTA took onl 16 laers to reach an NSE of 35 db, whereas AP took 25 iterations. (ore details will be given in Section 4. Other authors have also applied ideas from deep learning to the sparse linear inverse problem. For example, [4] extended the approach from [3] to handle structured sparsit and dictionar learning (when the training data are { (d } D d=1 and A is unknown. ore recentl, [6] extended [3] from the l 2+l 1 objective of (3 to the l 2+l objective, and [5] proposed to learn the SE-optimal scalar denoising function η b learning the parameters of a B-spline. The idea of unfolding an iterative algorithm and learning its parameters via training data has also been used to recover non-sparse signals. For example, it has been applied to speech enhancement [18], image deblurring [19], image super resolution [2], compressive imaging [21, 22], and video compressive sensing [23]. 3. LEARNED AP Recall that LISTA involves learning the matrix S I N BA, where B C N and A C N. As noted in [3], when < N/2, it is advantageous to leverage the I N BA structure of S, leading to network laers of the form shown in Fig. 3, with firstlaer inputs x = and v =. Although not considered in [3], the network in Fig. 3 allows both A and B to var with the laer t, allowing for modest improvement (as will be demonstrated in Section The LAP Network We propose to construct a neural network from unfolded AP (7 with tunable parameters {A t,b t,α t} T 1 t= learned from training data. The hope is that a learned AP (LAP will require fewer laers than LISTA, just as AP tpicall requires man fewer iterations than ISTA to converge. Figure 4 shows one laer of the LAP network. Comparing Fig. 4 to Fig. 3, we see two main differences: x t + r η( ;λ x t+1 t t v t + v t+1 B t Fig. 3. Thetth laer of the LISTA network, with tunable parameters {A t,b t,λ t} T 1 t=. x t + r t x η( ; t+1 αt 2 v t λ t + + v t+1 B t b t+1 Fig. 4. The tth laer of the LAP network, with tunable parameters {A t,b t,α t} T 1 t= 1. LAP includes a feed-forward path from v t to v t 1 that is not present in LISTA. This path implements an Onsager correction whose goal is to decouple the laers of the network, just as it decoupled the iterations of the AP algorithm (recall Section LAP s denoiser threshold λ t = α t v t 2/ varies with the realizationv t, whereas LISTA s is constant. Note that LAP is built on a generalization of the AP algorithm (7 wherein the matrices (A,A H manifest as (A t,b t at iteration t. An important question is whether this generalization preserves the independent-gaussian nature (1 of the denoiser input error the ke feature of AP. It can be shown that the desired behavior does occur when i A t = β ta with scalar β t and ii B t = A H C t with appropriatel scaled C t. Thus, for LAP, we impose 5 A t = β ta and learn onl β t, and we initialize B t appropriatel before learning. From Fig. 4, we see that the β t scaling withina t can be moved to the denoiser η( ; under a suitable re-definition of x t, and we take this approach in LAP. Thus, the tth laer of LAP can be summarized as x t+1 = β ( x α tη t +B tv t; t v t 2 (13a A t A t v t+1 = A x t+1 + βt xt+1 vt, (13b with first-laer inputs x = and v =. Figure 5(c shows the QQplot of LAP s denoiser input error ( x t + B tv t x. The shape of the plot confirms that the error is Gaussian Learning The LAP Parameters Our proposed learning procedure is described below, where it is assumed that A is known. But if A was unknown, it could be estimated using a least-squares fit to the training data. Empiricall, we find that there is essentiall no difference in final test SE between LAP networks where i A is known, ii A is estimated via leastsquares, or iiiais learned using back-propagation to minimize the loss L T from ( Tied{B t} In the tied case, B t = B t, and so the adjustable parameters are Θ = [ ] B,{α t} T 1 t=,{βt}t 1 t=. We proposed to learn Θ b 5 Here A refers to the true measurement matrix from (2. If A is unknown, it can be learned from the training data as described in Section 3.2.

4 (a ISTA (b AP (c LAP Fig. 5. QQplots of the denoiser input error evaluated at the first iteration t for which NSE( x t < 15 db. Note ISTA s error is heav tailed while AP s and LAP s errors are Gaussian due to Onsager correction. starting with a 1-laer network and growing it one laer at a time. For the first laer (i.e., t =, we initialize B as the regularized pseudo-inverse average NSE [db] LISTA tied LISTA untied LAP tied LAP untied laers Fig. 6. Test NSE versus laer for i.i.d. Gaussian A. B = γ 1 A H( AA H +I 1, (14 with γ chosen so that tr(ab = N, and we use backpropagation to learn the values of α,β that minimize the loss L from (12. Then, for each new laer t = 1,...,T 1, we 1. initializeb,α t,β t at the values from laer t 1, 2. optimize α t,β t alone using backpropagation, and 3. re-optimize all parameters B,{α i} t i=,{β i} t i= using backpropagation to minimize the loss L t from ( Untied {B t} In the untied case, B t is allowed to var across laers t. We propose the same learning procedure as above, but with one exception: when initializing the parameters for each new laer, we initializeb t at the regularized pseudo-inverse (14 rather than the learned value B t Structured B t otivated b the <N case, we recommend constraining B t = A H C t and learning onl C t C, which reduces the number of free parameters in B t from N to 2. After these B t are learned, there is no need to represent them in factored form, although doing so ma be advantageous if A has a fast implementation (e.g., FFT. Empiricall, we find that there is essentiall no difference in final test SE between LAP networks where ib t is unconstrained and ii B t = A H C t. However, the learning procedure is more efficient in the latter case. 4. NUERICAL RESULTS We now evaluate the performance of LISTA and LAP on the sparse linear inverse problem described in Section The data were generated as described in Section 2.1.4, with training mini-batches of sized=1 and a single testing mini-batch of size1 (drawn independentl of the training data. The training and testing methods were implemented in Pthon using TensorFlow [24] with the ADA optimizer [25]. For LAP, we performed the learning as described in Section 3.2. For LISTA, we used the same approach to learn untied [ B,S,{λ t} T 1 t= ] and tied {Bt,S t,λ t} T 1 t=, with no constraints ons t orb t (because we found that adding constraints degraded performance. average NSE [db] LISTA tied LISTA untied LAP tied LAP untied laers Fig. 7. Test NSE versus laer for A with condition number 15. Figure 6 shows average test NSE x t x 2 2/ x 2 versus laer t for the same i.i.d. Gaussian A and test data used to create Fig. 1, allowing a direct comparison. The figure shows LAP significantl outperforming LISTA and AP in convergence time and final NSE. For example, to reach 34 db NSE, tied-lap took onl7laers, tied-lista took15, and AP took 23. Figure 7 shows the results of a similar experiment, but where the singular values ofawere replaced b a geometric sequence that ielded A 2 F = N and a condition-number of15. For thisa, AP diverged but LAP did not, due in part to the preconditioning effect of (14. The figures also show that the untied versions of LAP and LISTA ielded small improvements over the tied versions, but at the cost of a T -fold increase in parameter storage plus significantl increased training time. 5. CONCLUSION We considered the application of deep learning to the sparse linear inverse problem. otivated b the AP algorithm, we proposed the use of Onsager correction in deep neural networks, for the purpose of decoupling and Gaussianizing errors across laers. Empirical results demonstrated improved accurac and efficienc over Gregor and LeCun s LISTA [3].

5 6. REFERENCES [1] Y. C. Eldar and G. Kutniok, Compressed Sensing: Theor and Applications. New York: Cambridge Univ. Press, 212. [2] B. A. Olshausen and D. J. Field, Sparse coding with an overcomplete basis set: A strateg emploed b v1, Vision Research, vol. 37, pp , [3] K. Gregor and Y. LeCun, Learning fast approximations of sparse coding, in Proc. Int. Conf. ach. Learning, pp , 21. [4] P. Sprechmann, P. Bronstein, and G. Sapiro, Learning efficient structured-sparse models, in Proc. Int. Conf. ach. Learning, pp , 212. [5] U. Kamilov and H. ansour, Learning optimal nonlinearities for iterative thresholding algorithms, IEEE Signal Process. Lett., vol. 23, pp , a 216. [6] Z. Wang, Q. Ling, and T. S. Huang, Learning deep l encoders, in Proc. AAAI Conf. Artificial Intell., pp , 216. [7] D. L. Donoho, A. aleki, and A. ontanari, essage passing algorithms for compressed sensing, Proc. Nat. Acad. Sci., vol. 16, pp , Nov. 29. [8] R. Tibshirani, Regression shrinkage and selection via the lasso, J. Ro. Statist. Soc. B, vol. 58, no. 1, pp , [9] S. S. Chen, D. L. Donoho, and. A. Saunders, Atomic decomposition b basis pursuit, SIA J. Scientific Comput., vol. 2, no. 1, pp , [1] E. Candès, J. Romberg, and T. Tao, Stable signal recover from incomplete and inaccurate measurements, Communications on Pure and Applied athematics, vol. 59, no. 8, pp , 26. [11] A. Chambolle, R. A. DeVore, N. Lee, and B. J. Lucier, Nonlinear wavelet image processing: Variational problems, compression, and noise removal through wavelet shrinkage, IEEE Trans. Image Process., vol. 7, pp , ar [12] I. Daubechies,. Defrise, and C. D. ol, An iterative thresholding algorithm for linear inverse problems with a sparsit constraint,, Commun. Pure & Appl. ath., vol. 57, pp , Nov. 24. [13] A. Beck and. Teboulle, A fast iterative shrinkagethresholding algorithm for linear inverse problems, SIA J. Imag. Sci., vol. 2, no. 1, pp , 29. [14] A. ontanari, Graphical models concepts in compressed sensing, in Compressed Sensing: Theor and Applications (Y. C. Eldar and G. Kutniok, eds., Cambridge Univ. Press, 212. [15]. Baati and A. ontanari, The dnamics of message passing on dense graphs, with applications to compressed sensing, IEEE Trans. Inform. Theor, vol. 57, pp , Feb [16] D. L. Donoho, A. aleki, and A. ontanari, essage passing algorithms for compressed sensing: I. otivation and construction, in Proc. Inform. Theor Workshop, (Cairo, Egpt, pp. 1 5, Jan. 21. [17] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. IT Press, 216. [18] J. R. Hershe, J. Le Roux, and F. Weninger, Deep unfolding: odel-based inspiration of novel deep architectures, Tech. Rep. TR , itsubishi Electric Research Labs, 214. (See also arxiv: [19] U. Schmidt and S. Roth, Shrinkage fields for effective image restoration, in Proc. IEEE Conf. Comp. Vision Pattern Recog., pp , 214. [2] C. Dong, C. C. Lo, K. He, and X. Tang, Image superresolution using deep convolutional networks, IEEE Trans. Pattern Anal. ach. Intell., vol. 38, pp , Feb [21] A. ousavi, A. Patel, and R. Baraniuk, A deep learning approach to structured signal recover, in Proc. Allerton Conf. Commun. Control Comput., pp , 215. [22] K. Kulkarni, S. Lohi, P. Turaga, R. Kerviche, and A. Ashok, ReconNet: Non-iterative reconstruction of images from compressivel sensed random measurements, in Proc. IEEE Conf. Comp. Vision Pattern Recog., 216. (see also arxiv: [23]. Iliadis, L. Spinoulas, and A. K. Katsaggelos, Deep full-connected networks for video compressive sensing, in arxiv:163:493, 216. [24]. Abadi, A. Agarwal, P. Barham, et al., TensorFlow: Largescale machine learning on heterogeneous sstems, 215. Software available from tensorflow.org. [25] D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, in Proc. Internat. Conf. on Learning Repres., 215. (see also arxiv: