Incremental Reformulated Automatic Relevance Determination

Transcription

1 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 60, NO. 9, SEPTEMBER Incrementa Reformuated Automatic Reevance Determination Dmitriy Shutin, Sanjeev R. Kukarni, and H. Vincent Poor Abstract In this work, the reationship between the incrementa version of sparse Bayesian earning (SBL) with automatic reevance determination (ARD) a fast margina ikeihood maximization (FMLM) agorithm and a recenty proposed reformuated ARD scheme is estabished. The FMLM agorithm is an incrementa approach to SBL with ARD, where the corresponding objective function the margina ikeihood is optimized with respect to the parameters of a singe component provided that the other parameters are fixed; the corresponding maximizer is computed in cosed form, which enabes a very efficient SBL reaization. Wipf and Nagarajan have recenty proposed a reformuated ARD (R-ARD) approach, which optimizes the margina ikeihood using auxiiary upper bounding functions. The resuting agorithm is then shown to correspond to a series of reweighted -constrained convex optimization probems. This correspondence estabishes and anayzes the reationship between the FMLM and R-ARD schemes. Specificay, it is demonstrated that the FMLM agorithm reaizes an incrementa approach to the optimization of the R-ARD objective function. This reationship aows deriving the R-ARD pruning conditions simiar to those used in the FMLM scheme to anayticay detect components that are to be removed from the mode, thus reguating the estimated signa sparsity and acceerating the agorithm convergence. Index Terms Automatic reevance determination, fast margina ikeihood maximization, sparse Bayesian earning. I. INTRODUCTION During the ast decade research on sparse signa representations has received considerabe attention [1] [5]. With a few minor variations, the genera goa of sparse reconstruction is to optimay estimate the parameters of the foowing canonica mode: t =8w + (1) where t 2 N is a vector of targets, 8 =[ 1 ;...; L ] is a dictionary matrix with L coumns corresponding to component vectors 2 N, =1;...;L, and w =[w1;...;w L ] T is a vector of unknown weights with ony K nonzero entries, i.e., w is assumed to be K-sparse. The additive perturbation is typicay assumed to be a white Gaussian random vector with zero mean and covariance matrix I, where Manuscript received October 04, 21; revised March 12, 22; accepted May, 22. Date of pubication May 21, 22; date of current version August 07, 22. The associate editor coordinating the review of this manuscript and approving it for pubication was Prof. Raviv Raich. This research was supported in part by an Erwin Schrödinger Postdoctora Feowship, Austrian Science Fund (FWF) Project J2909-N23, in part by the U.S. Army Research Office under Grant W911NF , the U.S. Office of Nava Research under Grant N , and in part by the Center for Science of Information (CSoI), an NSF Science and Technoogy Center, under grant agreement CCF D. Shutin is with the Department of Eectrica Engineering, Princeton University Princeton, NJ USA, and aso with the German Aerospace Center, Institute of Communications and Navigation, Wessing 82234, Germany (e-mai: dshutin@gmai.com; dmitriy.shutin@dr.de). S. R. Kukarni is with the Department of Eectrica Engineering, Princeton University, Princeton, NJ USA (e-mai: kukarni@princeton.edu). H. V. Poor is with Princeton University, Schoo of Engineering and Appied Science, Princeton, NJ USA (e-mai: poor@princeton.edu). Coor versions of one or more of the figures in this correspondence are avaiabe onine at Digita Object Identifier /TSP is a noise precision parameter. Imposing constraints on the mode parameters w is a key to sparse signa modeing (see, e.g., [4]). Sparse Bayesian earning (SBL) [5] [9] is a famiy of empirica Bayes techniques that find a sparse estimate of w by modeing the weights using a hierarchica prior p(wj)p() = L =1 p(w j )p( ), 1 where p(w j ) is a Gaussian probabiity density function (pdf) with zero mean and precision parameter aso caed the sparsity parameter that reguates the width of this pdf. Different approaches to SBL vary mainy in the way the hyperprior p( ) is specified (see, e.g., [6] and [9]). Automatic reevance determination (ARD) represents a cass of SBL agorithms in which the hyperprior p( ) is assumed to be fat or noninformative. The weights w are estimated using the posterior pdf p(wjt; ;), which can be computed anayticay; the sparsity parameters and the noise precision are typicay found by maximizing the margina ikeihood function 2 p(tj;)= p(tjw; )p(wj)dw. In the origina reevance vector machine (RVM) agorithm the atter optimization is reaized iterativey using the expectation-maximization (EM) agorithm [6]. Unfortunatey, the EM agorithm is known to converge rather sowy and a number of improvements have been proposed in [11] and [12] to address the sow convergence of the RVM agorithm. In [11], an efficient incrementa scheme has been proposed to maximize the margina ikeihood. The authors demonstrate that the maximizer of the margina ikeihood function with respect to the sparsity parameter of a singe component can be computed anayticay provided the sparsity parameters for the components and the parameter are fixed; moreover, this maximizer is shown to take either finite or infinite vaues. This observation has ead to two important practica consequences. First, it aows constructing an efficient agorithm, termed fast margina ikeihood maximization (FMLM), that incrementay maximizes the margina ikeihood and prunes components 3 by cycing through them in a round-robin fashion. It is important to note that the same mechanism can be used to incrementay buid up the mode compexity by incorporating new components in the mode. Second, the anaytica anaysis of the conditions that detect the divergence of sparsity parameters pruning conditions reveas the dependency of the estimated signa sparsity on the amount of additive noise [13]; this aows for an empirica adjustment of these conditions, which has been shown to significanty acceerates the convergence rate of the sparse inference. Another approach to maximize the margina ikeihood has been proposed in [12]. In contrast to the incrementa FMLM approach, in [12] the margina ikeihood is optimized jointy over the sparsity parameters of a components via minimization of auxiiary upper bounding convex functions [14]. This approach, which the authors in [12] termed a reformuated ARD (R-ARD), exhibits a number of usefu features. Specificay, the objective function of the R-ARD scheme is convex; moreover, the R-ARD soution can be computed jointy for a eements of w via a series of weighted `1-constrained east-squares optimization probems. The atter property is important from a theoretica standpoint as it effectivey reates the SBL with ARD to more traditiona non-bayesian sparse earning methods, e.g., basis pursuit denoising and minimum `1-norm methods [3]. The downside of the scheme is a reativey high computationa cost as it requires soving a series of `1-constrained optimization probems. The atter fact has motivated us 1 It is aso possibe to extend the SBL prior formuation to priors invoving three ayers of hierarchy (see, e.g., [9] or [10]). 2 This is equivaent to assuming a fat hyperpriors p( ) and p( ); 8. 3 An infinite vaue of the hyperparameter forces the posterior vaue of the component weight to zero X/$ IEEE

2 4978 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 60, NO. 9, SEPTEMBER 22 to seek for a more efficient reaization of the R-ARD scheme. Inspired by the efficiency and anaytica tractabiity of incrementa ARD soutions, we investigate if R-ARD scheme can be impemented in an incrementa setting. In this work, we report the resuts of these investigations. Specificay, we demonstrate that the R-ARD and FMLM agorithms are reated; in fact, the FMLM agorithm reaizes an incrementa optimization of the R-ARD objective function. In other words, FMLM agorithm represents an aternative optimization strategy for the R-ARD objective cost function. It is not unreasonabe to assume the existence of the reationship between the two schemes since both R-ARD and FMLM share a simiar ARD-motivated effective cost function, yet no study has been performed to formay estabish this connection. This reationship aows, on the one hand, for a better understanding of the performance of cassica sparse earning schemes in the presence of noise due to the anaytica tractabiity of the FMLM pruning conditions and their dependency on the component s signa-to-noise ratio (SNR) [13], and, on the other hand, for empirica adjustment of R-ARD scheme based on a predefined SNR eve that acceerates the convergence rate of the agorithm. The rest of the correspondence is organized as foows. In Section II, we give an outine of the SBL signa mode and the standard RVM agorithm, foowed a brief summary of the FMLM scheme. Its reationship to the R-ARD agorithm is outined in Section III. Finay, in Section IV, we present a sma simuation exampe that iustrates the distinction between the incrementa FMLM and R-ARD schemes. Throughout this correspondence, we sha make use of the foowing notation. Vectors are represented as bodface owercase etters, e.g., x, and matrices as bodface uppercase etters, e.g., X. For vectors and matrices, (1) T denotes the transpose. Finay, for a random vector x, N(xja; B) denotes a mutivariate Gaussian pdf with mean a and covariance matrix B. II. SPARSE BAYESIAN LEARNING A standard soution to SBL with ARD is a two step procedure that aternates between i) estimating the weight vector w and ii) estimating the corresponding sparsity parameters and noise precision.given an estimate of the noise precision parameter ^ and sparsity parameters ^, an estimate of the weight vector w is obtained as a mode of the posterior pdf p(wjt; ^; ^ )=p(tjw; ^ )p(wj^), which can be shown to be a Gaussian pdf p(wjt; ^; ^ ) = N(wjw; S) with the mean w and covariance matrix S given by S = ^ 8 T 8 + A and w =^ S8 T t (2) where A =diag() a diagona matrix with the eements of on the main diagona. The estimates of and are computed by maximizing the margina ikeihood function [6] p(tj;)= p(tjw; )p(wj)dw =N(tj0; I +8A 8 T ): (3) The maximizers ^ and ^ of (3) can be obtained using the EM agorithm where the weights w are used as compete data (see [6] for more detais). The corresponding estimation expressions are then given as and =(jw j 2 + S ) (4) N ^ = kt 0 8wk 2 +tr(s8 T 8) (5) where w is the th eement of the vector w, and S is the th eement on the main diagona of the matrix S. The update expressions (2) and (4) (5) are then repeatedy evauated unti convergence [6]. In cases when t aows for a sparse representation in terms of 8, the sparsity parameters of some of the components wi diverge, forcing the posterior vaue of the corresponding weights to converge towards zero and, thus, encouraging a sparse soution. Unfortunatey, due to the EM-based maximization of the margina ikeihood the rate at which sparsity parameters diverge is ow. Many iterations are needed for the hyperparameters to reach a threshod at which they can be treated as numericay infinite. 4 This has motivated to use of aternative, more efficient schemes to maximize (3) with respect to. A. Fast Margina Likeihood Maximization One possibe strategy to optimize (3) more efficienty consists of computing its optimum with respect to a singe sparsity parameter assuming that the other sparsity parameters ^ =[1;...; ; +1;...; L] T and the noise precision parameter ^ are fixed. Specificay, the ogarithm of p(tj ; ^ ; ) in (3) can be decomposed as og p(tj ; ^ ; ^ )=L( ; ^ ; )=L(^ ; ^ ) where og( ) 0 og( 0 s )+ q2 + s (6) s = T C ; q = T C t; (7) C =^ I + k k T k ; (8) k6= and L(^ ; ^ ) is a part of the margina og-ikeihood og p(tj ; ^ ;) that is independent of. Then, the maximum of (6) with respect to is obtained at [11] = s2 (q 2 0 s ) ; q 2 >s 1; q 2 s : Using (9), the margina ikeihood p(tj ; ^ ; ) can be maximized incrementay with respect to the sparsity parameter of one component at a time. Observe that the resut (9) not ony aows for determining if a particuar eement in w is zero, but aso dramaticay acceerates the rate of SBL convergence since the optimum of the margina ikeihood can be computed anayticay. Another important consequence of (9) is that the ratio w 2 = q 2 =s 2 can be shown to be equa to the squared posterior estimate of the th weight w computed when =0; furthermore, S corresponds to the posterior variance of this weight (see [13] for more detais). It foows then that the ratio w 2 =S = q 2 =s is an estimate of the component s SNR. The condition q 2 >s in (9) is then equivaent to the condition w 2 =S > 1, which has a very intuitive interpretation: components with the estimated SNR beow 1 (or equivaenty beow 0 db) are removed from the mode since according to (9) the corresponding sparsity parameter that maximizes the margina ikeihood p(tj ; ^ ; ) is infinite. Obviousy, this interpretation can be extended by requiring that w 2 =S exceeds some other predefined SNR eve 1, which has been shown to improve sparse signa estimation when the actua signa-to-noise ratio is known [13]. 4 For instance, this threshod can be set to 10 or 10. (9)

3 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 60, NO. 9, SEPTEMBER III. REFORMULATED ARD AND INCREMENTAL ARD SCHEMES Let us now anayze the reationship between the R-ARD approach to SBL proposed in [12] and the FMLM scheme. To be consistent with the notation adopted in [12], we define =[ 1 ;...; L ] T as a prior variance vector with eements =, =1;...;L. The margina ikeihood function can then be represented as (see [6], [7], and [12]) og p(tj; ^ )=0 og jcj0t T C t +const (10) where C = ^ I + L =1 T and the const term coectivey represents terms that are independent of. In [12], the authors propose to optimize (10) using auxiiary upper bounding functions. Specificay, they show that the objective function L() =0 og p(tj; ) can be upper-bounded as L(;z) =z T 0 g 3 (z)+t T C t L() (11) where z =[z 1 ;...;z L ] T and g 3 (z) is the concave conjugate of og jcj defined by the duaity reationship g 3 (z) =min z T 0 og jcj [14]. Then, for fixed at some estimate, the tightest upper bound on L() is obtained by minimizing L(;z)j = = L(z; ) over z. The corresponding minimizer can be found in cosed form: ^z = arg min L(z; ^) = diagf8 T ^C 8g (12) z where diagf8 T C 8g is a vector of diagona entries of 8 T C 8 and C = ^ I + L =1 T. Now, by fixing z at z, the bound L(;z)j z=z = L(; z) is minimized with respect to by soving = arg min L(; z) = arg min z T + t T C t: (13) The R-ARD agorithm then aternates between (12) and (13) unti convergence. Note that the objective function (13) is convex; in [12], it has been shown that it can be efficienty optimized using a series of weighted `1-constrained east squares optimizations. Obviousy, a variety of aternative optimization strategies can be derived to optimize L() more convenienty. In what foows, we demonstrate that the incrementa FMLM approach discussed in Section II-A can aso be used to optimize the upper bound L(;z) in (11). Let us consider the optimization of the bound L(;z) in (11) in an incrementa setting. First, consider the optimization of L(; z) with respect to a singe parameter assuming that k, k 6=, are known and fixed at their estimated vaues. By defining =[ 1 ;...; ; +1 ;...; L ] T and noting that the matrix C can be decomposed as C =^ I + k6= k k T k + T, we rewrite (13) in the foowing form: L( ; ; z) = k6= =^z 0 ^z k k + t T C t +^z 0 q 2 + s q 2 + s + const (14) where the parameters q and s are defined in (7), C is defined in (8), and const is a part of L( ; ; z) that is independent of.wenow compute the vaue of that minimizes (14) by computing dl( ; ; z) q 2 =^z 0 d (1 + s ) 2 and equating the resut to zero. Soving for gives two stationary points of L( ; ; z) at ;1 = 0jq j0 p^z and ;2 = jq j0 p^z : (15) s p^z s p^z Notice that the first soution ;1 is aways negative and, thus, infeasibe 5 ; the second soution is positive provided jq j > p^z. Let us study these stationary points in more detai. It is known that a stationary point 3 is a oca minimum of L( ; ; z) if, and ony if, d 2 L( ; ; z) d 2 = = 2q 2 s (1 + s ) 3 = > 0: (16) By evauating (16) at 3 = ;1, it is easy to see that the sign of the second derivative at this point is negative, i.e., ;1 is not a oca minimum of (14). In contrast, ;2 in (15) is a oca minimum. However, it is ony a feasibe optimum provided jq j > p^z. Thus, p = arg min L( ; ; z) = jq j0 s p ^z ^z jq j > p^z ; no feasibe so. jq j p^z : (17) Notice a simiarity between the structure of the soution (17) and that in (9). As we wi see ater, the condition jq j > p^z is in fact reated to the pruning conditions of the FMLM agorithm. Once the optimum of (14) with respect to is found, we return to (12) and re-estimate z using an updated vector, in which the th eement has been computed using (17). What wi happen if the updates (12) and (17) are repeated ad infinitum for some fixed component? To answer this question we note that the vaue of in (17) depends expicity ony on the th eement of z. Appying the Woodbury matrix identity [15] to the inverse of C = ^ I + L =1 T and combining the resut with (12) we can rewrite the th eement of z as ^z =^ T 0 ^ 2 T 8S8T : (18) The resut (18) aows us to estabish a reationship between the vaue of ^z and the parameter s in (7). Specificay, it can be shown (see [11, eq. (23) and (24)]) that these parameters are reated through the foowing transformation: ^z = s ( + s ) : (19) Now, if (19) and (17) are repeatedy evauated ad infinitum, then 6 ^z! s2 q 2 and! s 02 (q 2 0 s ): (20) It can be seen that the vaue of in (20) coincides with the vaue of in (9). Notice that this resut is vaid provided jq j > p^z, i.e., when in (17) is a feasibe optimum. However, for ^z = s 2 =q 2, the condition jq j > p^z is equivaent to the condition q 2 > s, which in turn guarantees the positivity of in (20) and coincides with the pruning condition in (9). Thus, the FMLM agorithm reaizes an incrementa optimization of the R-ARD objective function. Moreover, due to the convexity of the (13) such incrementa optimization is guarantied to converge to a oca optimum of the R-ARD objective function. Furthermore, the pruning condition in (9) aso determines if the minimum of (13) is achieved at a feasibe soution. Let us point out that the connection between the R-ARD and the FMLM scheme can be expoited to deveop SNR-based adjustments of the R-ARD simiar to those used for FMLM agorithm. Specificay, by combining (19) and = s 02 (q 2 0 s ) from (20) with the adjusted 5 Reca that modes the prior variance of the weight w and must be nonnegative. 6 We eave this resut without proof. It can be easiy verified by substituting a feasibe soution for from (17) into (19) and soving for ^z, which gives ^z = s =q. Inserting the atter resut in (17) gives (20).

4 4980 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 60, NO. 9, SEPTEMBER 22 Fig. 1. (a) Estimated NMSE and (b) number of estimated components versus SNR. (c) Nonzero eements of a sparse estimate of a weight vector for 30 db SNR. The potted vaues in (c) correspond to a singe reaization of the earning agorithms. pruning condition q 2 =s >, where 1 is the pruning SNR eve, it can be shown that the FMLM pruning condition q 2 =s > is equivaent to > 0 1 z : (21) Condition (21) can be used as a pruning condition for the R-ARD scheme. In other words, the sparsity eve of the R-ARD agorithm can be adjusted based on (21) and some preseected sensitivity 1 expressed in the units of SNR. In the next experimenta section, we demonstrate the usefuness of such an adjustment. IV. A SIMPLE ILLUSTRATIVE EXAMPLE In this section, we contrast the performance of the incrementa ARD scheme, i.e., of the FMLM agorithm, to that of the R-ARD agorithm. Note that whie the atter finds a soution via a series of weighted `1-constrained 7 east-squares optimizations as suggested in [12], the FMLM agorithm optimizes the same R-ARD objective function incrementay, with the optimum at each step computed anayticay. We wi aso compare the performance of FMLM and R-ARD to the standard RVM agorithm with noninformative hyperpriors that uses EM agorithm to estimate mode parameters [6], as we as to the FMLM and R-ARD agorithms that use SNR-adjusted pruning rues q 2 =s > in (9) and (21), respectivey; we wi refer to the atter two schemes as FMLM-adj and R-ARD-adj schemes, respectivey. For both FMLM-adj and R-ARD-adj schemes, the adjustment threshod is set to = 3:16, = 1;...;L, which corresponds to 5 db SNR. This adjustment has been used in a simuations. Our goa here is to construct a simpe scenario where the distinctions between the R-ARD approach to SBL and the incrementa FMLM, as we as the impact of SNR-based adjustment can be easiy demonstrated. We acknowedge, however, that the performed experiments do not by any means represent a comprehensive comparison of the methods, which cannot be incuded due to the correspondence ength constraints. As performance measures, we compute the normaized mean-square error (NMSE) and the estimated number of components versus SNR; additionay, we demonstrate the rate at which components are removed from the mode and the convergence rate of the estimated weight vector w as a function of the agorithm iteration index. The estimated quantities are averaged over 100 independent agorithm runs. Note that in order to estimate the number of components with the RVM and R-ARD agorithms, it is necessary to specify a pruning threshod for the sparsity parameters, =1;...; L, which is essentiay a numerica way of detecting their divergence. Specificay, when for some component 7 The ` -constrained optimization is impemented using a simpe Matab sover for ` -reguarized east squares probems, which uses a og-barrier method [14] to optimize the corresponding objective function. The duaity gap for the og-barrier sover is set to 10. The software is avaiabe onine at boyd/1\_s/. an estimate of exceeds the pruning threshod, the corresponding component is removed from the mode. Let us stress that, in the case of the incrementa approach, such a pruning threshod is not needed: the pruning conditions in (9) essentiay detect the divergence of sparsity parameters; the same aso hods for the R-ARD-adj agorithm. In our impementation of the RVM and R-ARD agorithms, we set this threshod to We note that, in the case of the R-ARD agorithm, this is aso needed since in the presence of noise the soution obtained by `1-constrained sover might not be exacty sparse; in fact, some eements in w might become very sma, but nonetheess numericay sti arger than 0. To further simpify the anaysis of the simuation resuts, we wi assume the noise precision parameter ^ to be known and fixed in a simuations. A simpe compressive samping toy probem is considered. We assume that the components 2 N, =1;...;L, are generated by drawing N sampes from a Gaussian distribution with zero mean and variance 1= p N. A sparse vector w is constructed by setting K eements of w to 61 at random ocations. The target vector t is then generated according to (1). In this experiment, we set N = 100, K =10, and L = 200. In Fig. 1, we pot the estimated NMSE and the estimated number of components versus SNR. Notice that the best performance is achieved for R-ARD-adj agorithm, foowed cosey by the FMLM-adj scheme; the normaized mean square-error (NMSE) performance of the other agorithms is indistinguishabe. The reason for such a good performance of the R-ARD-adj is the adjusted pruning condition that eads to the improved estimated signa sparsity, as can be seen in Fig. 1(b): the R-ARD-adj agorithm estimates the correct number of components amost over the whoe tested SNR range; in contrast, other schemes underestimate the true signa sparsity. In the case of the FMLM-adj scheme, this adjustment is ess effective. This can be expained by the gain of the R-ARD scheme due to the joint estimation of the weight vector w. Let us point out that the underestimation of the signa sparsity is due to the fact that when noise is present some of the estimated weights take very sma, yet nonzero vaues, as demonstrated in Fig. 1(c). Ceary, with an appropriate threshoding it is possibe to remove these weak noisy components to recover the true sparsity. Yet in practica situations the discrepancy between zero and nonzero estimated weights might not be so distinct as in Fig. 1(c) and finding an optima threshod might be quite chaenging. The proposed adjustments of the pruning conditions, originay used in [13] for FMLM and extended here in (21) to the R-ARD scheme, readiy provide a way to optimay seect this threshod based on the anaytica anaysis of the inference expressions; surprisingy, these adjustments give quite accurate resuts. Unfortunatey, the space constraints prohibit us from a more detaied study of the adjusted pruning conditions. Now, in Fig. 2, we demonstrate the performance of the compared agorithms versus the number of update iterations; specificay, we compute the `2-norm of the difference between the estimated vectors w at

5 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 60, NO. 9, SEPTEMBER Fig. 2. Convergence of the weight vector w and estimated signa sparsity versus the number of agorithm iterations for (a),(b) 0 db SNR and (c),(d) 30 db SNR. two consecutive iterations as we as the number of estimated components versus the agorithm iteration index for 0 db and 30 db SNR. Here, each iteration incudes a compete update of a L components present in the mode. Note that, for the R-ARD agorithm, a singe update iteration incudes a singe `1-constrained optimization, which is aso an iterative scheme. The reported number of iterations for the R-ARD agorithm is obtained assuming that an `1-constrained optimization is soved instantaneousy. Again we observe that the adjusted schemes outperform the other tested methods: due to the used adjustment, the irreevant components are removed aready during the eary iterations and roughy ten updates are needed for the remaining weights to converge. As expected, the RVM agorithm performs the worst among the compared schemes. The performance of R-ARD and FMLM schemes is comparabe, yet the former is computationay more demanding since it requires soving a series of `1-constrained optimizations; obviousy, in this respect the incrementa approach is computationay much more attractive. Note that the convergence rate of R-ARD improves as the SNR grows. In contrast, the rate at which the incrementa FMLM agorithm removes components seems to be amost independent of the SNR. V. CONCLUSION In this work, a reationship between the incrementa fast margina ikeihood maximization (FMLM) agorithm and the reformuated ARD (R-ARD) agorithm has been estabished. We have demonstrated that the FMLM agorithm reaizes an incrementa optimization of the R-ARD objective function. The pruning condition of the FMLM agorithm that determines whether the maximum of the margina ikeihood with respect to a singe sparsity parameter is achieved at a finite optimum coincides with the condition that guarantees the existence of a feasibe minimum of the R-ARD objective function with respect to this sparsity parameter. Based on this reationship the empirica adjustments of the pruning conditions, previousy proposed for the FMLM agorithm, can be derived for the R-ARD scheme as we. This provides an additiona degree of freedom for the R-ARD scheme to contro the estimated signa sparsity based on the signa-to-noise ratio. Experimenta resuts based on a simpe compressive samping toy probem indicate that the use of adjustments significanty boosts the convergence rate of the scheme and reduces the underestimation of the true signa sparsity. [4] M. Wakin, An introduction to compressive samping, IEEE Signa Process. Mag., vo. 25, no. 2, pp , Mar [5] D. G. Tzikas, A. C. Likas, and N. P. Gaatsanos, The variationa approximation for Bayesian inference, IEEE Signa Process. Mag., vo. 25, no. 6, pp , Nov [6] M. Tipping, Sparse Bayesian earning and the reevance vector machine, J. Mach. Learn. Res., vo. 1, pp , Jun. 20. [7] D. Wipf and B. Rao, Sparse Bayesian earning for basis seection, IEEE Trans. Signa Process., vo. 52, no. 8, pp , Aug [8] M. Figueiredo, Adaptive sparseness for supervised earning, IEEE Trans. Pattern Ana. Mach. Inte., vo. 25, no. 9, pp , [9] N. L. Pedersen, C. N. Manchon, D. Shutin, and B. H. Feury, Appication of Bayesian hierarchica prior modeing to sparse channe estimation, presented at the IEEE Int. Conf. Commun. (ICC), 22. [10] A. Lee, F. Caron, A. Doucet, and C. Homes, A hierarchica Bayesian framework for constructing sparsity-inducing priors [Onine]. Avaiabe: preprint arxiv: [11] M. E. Tipping and A. C. Fau, Fast margina ikeihood maximisation for sparse Bayesian modes, presented at the 9th Int. Workshop Artif. Inte. Stat., Key West, FL, Jan [12] D. Wipf and S. Nagarajan, A new view of automatic reevance determination, presented at the 21 Annu. Conf. Neura Inf. Process. Syst., Vancouver, BC, Canada, Dec [13] D. Shutin, T. Buchgraber, S. R. Kukarni, and H. V. Poor, Fast variationa sparse Bayesian earning with automatic reevance determination for superimposed signas, IEEE Trans. Signa Process., vo. 59, no. 12, pp , Dec. 21. [14] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, MA: Cambridge Univ. Press, [15] G. H. Goub and C. F. Van Loan, Matrix Computations, 3rd ed. Batimore, MD: The Johns Hopkins Univ. Press, REFERENCES [1] R. Baraniuk, Compressive sensing, IEEE Signa Process. Mag., vo. 24, no. 4, pp , Ju [2] D. Donoho, For most arge underdetermined systems of inear equations, the minima e-1 norm soution is aso the sparsest soution, Commun. Pure App. Math., vo. 59, no. 6, pp , Jun [3] S. S. Chen, D. L. Donoho, and M. A. Saunders, Atomic decomposition by basis pursuit, SIAM J. Sci. Comput., vo. 20, no. 1, pp , 1998.