Compressive Sensing Over Networks

Transcription

1 Forty-Eighth Annual Allerton Conference Allerton House, UIUC, Illinois, USA Septeber 29 - October, 200 Copressive Sensing Over Networks Soheil Feizi MIT Eail: sfeizi@it.edu Muriel Médard MIT Eail: edard@it.edu Michelle Effros Caltech Eail: effros@caltech.edu Abstract In this paper, we deonstrate soe applications of copressive sensing over networks. We ake a connection between copressive sensing and traditional inforation theoretic techniques in source coding and channel coding. Our results provide an explicit trade-off between the rate and the decoding coplexity. The key difference of copressive sensing and traditional inforation theoretic approaches is at their decoding side. Although optial decoders to recover the original signal, copressed by source coding have high coplexity, the copressive sensing decoder is a linear or convex optiization. First, we investigate applications of copressive sensing on distributed copression of correlated sources. Here, by using copressive sensing, we propose a copression schee for a faily of correlated sources with a odularized decoder, providing a trade-off between the copression rate and the decoding coplexity. We call this schee Sparse Distributed Copression. We use this copression schee for a general ulticast network with correlated sources. Here, we first decode soe of the sources by a network decoding technique and then, we use a copressive sensing decoder to obtain the whole sources. Then, we investigate applications of copressive sensing on channel coding. We propose a coding schee that cobines copressive sensing and rando channel coding for a high-snr point-to-point Gaussian channel. We call this schee Sparse Channel Coding. We propose a odularized decoder providing a trade-off between the capacity loss and the decoding coplexity. At the receiver side, first, we use a copressive sensing decoder on a noisy signal to obtain a noisy estiate of the original signal and then, we apply a traditional channel coding decoder to find the original signal. I. INTRODUCTION Data copression has been a research topic of any scholars in past years. However, recently, the field of copressive sensing, originated in [], [2] and [3], looked at the copression proble fro another point of view. In this paper, we try to ake a bridge between copressive sensing which ay be viewed as a signal processing technique and both source coding and channel coding. By using this connection, we propose soe non-trivial practical coding schees that provide a trade-off between the rate and the decoding coplexity. Copressive sensing has provided a low coplexity approxiation to the signal reconstruction. Inforation theoretic has been ostly concerned with accuracy of the signal reconstruction under rate constraints. In this paper, we seek to provide new connections which use copressive sensing for traditional inforation theory probles such as Slepian- Wolf copression and channel coding in order to provide This aterial is based upon work under subcontract C, ITMANET project and award No supported by AFSOR. echaniss to explicitly trade-off between the decoding coplexity and the rate. Soe previous work investigated the connection between copressive sensing and inforation theory. For exaple, reference [4] studied the iniu nuber of noisy easureents required to recover a sparse signal by using Shannon inforation theory bounds. Reference [5] investigated the contained inforation in noisy easureents by viewing the easureent syste as an inforation theoretic channel and using the rate distortion function. Also, reference [6] studied the trade-offs between the nuber of easureents, the signal sparsity level, and the easureent noise level for exact support recovery of sparse signals by using an analogy between support recovery and counication over the Gaussian ultiple access channel. In this paper, we want to investigate applications of copressive sensing on source coding and channel coding. First, in Section III, we consider distributed copression of correlated sources. In this section, by using copressive sensing, we propose a copression schee for a Slepian- Wolf proble with a faily of correlated sources. The proposed decoder is a odularized decoder, providing a trade-off between the copression rate and the decoding coplexity. We call this schee Sparse Distributed Copression. Then, We use this copression schee for a general ulticast network with correlated sources. Here, we first decode soe of the sources by a network decoding technique and then, we use the copressive sensing decoder to obtain the whole sources. Then, in Section IV, we investigate applications of copressive sensing on channel coding. We propose a coding schee that cobines copressive sensing and rando channel coding for a high-snr point-to-point Gaussian channel. We call this schee Sparse Channel Coding. We propose a odularized decoder, providing a trade-off between the capacity loss and the decoding coplexity (i.e., the higher the capacity loss, the lower the decoding coplexity.). The idea is to add intentionally soe correlation to transitted signals in order to decrease the decoding coplexity. At the receiver side, first, we use a copressive sensing decoder on a noisy signal to obtain a noisy estiate of the original signal and then, we apply a traditional channel coding decoder to find the original signal. II. TECHNICAL BACKGROUND AND PRIOR WORK In this section, we review soe prior results, in both copressive sensing and distributed copression, which will /0/$ IEEE 29

2 be used in the rest of the paper. A. Copressive Sensing Background Let X R n be a signal vector. We say this signal is k-sparse if k of its coordinates are non-zero and the rest are zero. Define α = k n as the sparsity ratio of the signal. Suppose Φ is a n n easureent atrix (hence, Y = ΦX is the easureent vector.). If Φ is non-singular, by having Y and Φ, one can recover X. However, [2] and [3] showed that, one can recover the original sparse signal by having far fewer easureents than n. Suppose for a given n and k, satisfies the following: M Mn ENCODER ENCODER M tx Mn tx R Rn DECODER M, M2,,Mn ρk log(n/k) () where ρ is a constant. Let S be the power set of {, 2,..., }, and S represent a eber of S with cardinality (e.g., S = {, 2,..., }). A n atrix Φ S is fored of rows of Φ whose indices belong to S. Φ S represents an incoplete easureent atrix. Reference [7] showed that, if satisfies () and Φ S satisfies the Restricted Isoetry Property (which will be explained later), having these easureents is sufficient to recover the original sparse vector. In fact, it has been shown in [7] that, X is the solution of the following linear prograing: in X L (2) subject to Y = Φ S X, where. L represents the L nor of a vector. Let us refer to the coplexity of this optiization by CXcs noiseless (, n). We say Φ S satisfies the restricted isoetry property (RIP) iff there exists 0 < δ k < such that, for any vector V R n, we have: ( δ k ) V 2 L 2 Φ S V 2 L 2 ( + δ k ) V 2 L 2. (3) Reference [7] showed if, δ k + δ 2k + δ 3k <, (4) then, the optiization (2) can recover the original k-sparse signal. This condition leads to (). Intuitively, if Φ S satisfies the RIP condition, it preserves the Euclidean length of k sparse signals. Hence, these sparse signals cannot be in the null-space of Φ S. Moreover, any k subset of coluns of Φ S are nearly orthogonal if Φ S satisfies RIP. Most of atrices which satisfy RIP are rando atrices. Reference [8] showed a connection between copressive sensing, n- widths, and the Johnson-Lindenstrauss Lea. Through this connection, [9] provided an elegant way to find δ k by using the concentration theore for rando atrices and proposed soe data-base friendly atrices (binary atrices) satisfying RIP. Now, suppose our easureents are noisy (i.e., Y = Φ S X + Z, where Z L2 = ϵ.). Then, reference [] showed that if Φ S satisfies RIP (3) and X is sufficiently sparse, the Fig.. A depth one ulticast tree with n sources. following optiization recovers the original signal up to the noise level: in X L (5) subject to Y Φ S X L2 ϵ. Hence, if X is a solution of (5), then X X L2 βϵ, where β is a constant. To have this, the original vector X should be sufficiently sparse. Specifically, [] showed that, if δ 3k + 3δ 4k < 2, (6) then, the results hold. We refer to the coplexity of this optiization by CXcs noisy (, n). B. Distributed Copression Background In this section, we briefly review soe prior results in distributed copression. In Section III, we will use these results along with copressive sensing results, aking a bridge these two subjects. Consider the network depicted in Figure. This is a depth one tree network with n sources. Suppose each source i, has essages referred by M i, wishes to send to the receiver. To do this, each source i sends its essage with a rate R i (i.e., say i has essages of source i. This source aps i to {, 2,..., 2 Ri }.). Let R = n i= R i. According to Slepian- Wolf copression ([0]), one needs to have, R H(M, M 2,..., M n ). (7) By using linear network coding, reference [] extended Slepian-Wolf copression for a general ulticast proble. Note that, the proposed decoders in [0] and [] (a iniu entropy or a axiu probability decoder) have high coplexity ([2]). For a depth one tree network, references [3], [4], [5], [6], [7] and [8] provide soe low coplexity decoding techniques for Slepian-Wolf copression. However, none of these ethods can explicitly provide a trade-off between the rate and the decoding coplexity. For a general ulticast network, it has been shown in [9] that, there is no separation between Slepian-Wolf copression and network 30

3 coding. For this network, a low coplexity decoder for an optial copression schee has not been proposed yet. III. COMPRESSIVE SENSING AND DISTRIBUTED SOURCE CODING In this part, we propose a distributed copression schee for correlated sources providing a trade-off between the decoding coplexity and the copression rate. References [0] and [] propose optial distributed copression schees for correlated sources, for depth-one trees and general networks, respectively. However, their proposed copression schees need to use a iniu entropy or a axiu probability decoder, which has high coplexity ([2]). On the other hand, copressive sensing provides a low coplexity decoder, by using a convex optiization, to recover a sparse vector fro incoplete and containated observations (e.g., [], [2], [3] and [7]). Our ai is to use copressive sensing over networks to design copression schees for correlated sources. Note that, our proposed techniques can be applied on different network topologies, with different assuptions and constraints. Here, to illustrate the ain ideas, first we consider a noiseless tree network with depth one and with correlated sources. Then, in Section III-B, we extend our techniques to a general ulticast network. A. A one-stage tree with n-correlated sources Consider the network shown in Figure, which is a one-stage noiseless ulticast tree network with n-sources. First, consider a case where sources are independent. Hence, the proble reduces to a classical source coding proble. Say each source is transitting with a rate R i. Denote R = n i= R i. It is well-known that the following su-rate for this network is the iniu required su-rate: R n H(M i ), (8) i= where M i is the essage rando variable of source i. Let us refer to the coplexity of its decoder at the receiver as CX indep (n), since sources are independent. Now, we consider a case where sources are correlated. We forulate the correlation of sources as follows: Definition. Suppose µ i,t represents a realization of the i th source essage rando variable at tie t (i.e., a realization of M i at tie t). Hence, the vector µ t = {µ,t,..., µ n,t } is the sources essage vector at tie t. We drop the subscript t when no confusion arises. Suppose sources are correlated so that there exists a transfor atrix under which the sources essage vector is k-sparse (i.e., there exists a n n transfor atrix Φ and a k-sparse vector µ t such that µ t = Φµ t.). Let S be the power set of {, 2,..., n}, and S denote ebers of S whose cardinality are. Suppose Φ S, defined as in Section II-A, satisfies RIP (3), where satisfies () for a given n and k. We have, µ t,s = Φ S µ t, where µ t,s is coponents of µ t whose indices belong to S. We refer to these sources as k-sparsely correlated sources. If we know zero locations of µ t and if the transfor atrix is non-singular, by having each subset containing k sources, the whole vector µ t and µ t can be coputed. Hence, by the data processing inequality, we have, H(M i,..., M ik ) + nh b (α) = H(M,..., M n ), (9) for and i < i 2 <... < i k n where M ij is the essage of source i j. Also, by [2], having sources, where satisfies () for a given k and n, allows us to recover the whole vector µ t and therefore, µ t. Hence, H(M i,..., M i ) = H(M,..., M n ). (0) Exaple 2. For an exaple of k-sparsely correlated sources, suppose each coordinates of µ t is zero with probability α, otherwise it is uniforly distributed over {, 2,..., 2 R }. Suppose the entries of Φ are independent realizations of Bernoulli rando variables: Φ i,j = { with prob. 2 with prob. 2. () Thus, as shown in [9], Φ S satisfies RIP (3). Also, we have, H(M,..., M n ) = nh b (α) + k(r + ) nh b (α) + kr. (2) The entropy of each source rando variable for large n can be coputed approxiately as follows: H(M i ) R + log(k). (3) 2 Note that, the individual entropies of k-sparsely correlated sources are roughly the sae. Now, we want to use copressive sensing to have a distributed source coding providing a trade-off between the copression rate and the decoding coplexity. We shall show that, if sources are k-sparsely correlated, if we have of the at the receiver at each tie t, by using a linear prograing, all sources can be recovered at that tie. Hence, instead of sending n correlated sources, one needs to transit of the where << n. We assue that the entropies of sources are the sae. Hence, each source transits with probability γ = n. Therefore, by the law of large nubers, for large enough n, we have transitting sources. For the case of non-equal source entropies, one can a priori choose sources with the lowest entropy as the transitting sources. In Table III-A, we copare four different schees: The first schee which Theore 3 is about, is called Sparse Distributed Copression with independent coding (SDCIC). The idea is to send sources fro n of the, assuing they are independent (i.e., their correlation is not used in coding). In this case, at the receiver, first we decode these transitted essages 3

4 TABLE I COMPARISON OF DISTRIBUTED COMPRESSION METHODS OF CORRELATED SOURCES Copression Methods Miniu Min-Cut Rate Decoding Coplexity SDCIC j= ) CX indep() + CX noiseless H(Mij cs (, n) Slepian-Wolf (SW) H(M,..., M n) CX sw(n) Cobination of SDC and SW H(M,..., M n) CX sw() + CXcs noiseless (, n) Naive Correlation Ignorance n i= H(Mi) CXindep(n) M 2 M M, M 2,,M n M, M 2,,M n and then, we use a copressive sensing decoder to recover the whole sources. The decoding coplexity is CX indep () + CXcs noiseless (, n). The required in-cut rate between each receiver and sources is ). In the second schee referred in Table III-A as the Slepian-Wolf coding, one perfors an optial source coding on correlated sources. The required in-cut rate between sources and the receiver is H(M,..., M n ) which is less than ). However, at the receiver, we need to have a Slepian-Wolf decoder with coplexity CX sw (n). One can have a cobination of sparse distributed copression and Slepian-Wolf copression. To do this, instead of n sources, we transit of the by using a Slepian-Wolf coding. The required in-cut rate would be H(M i,..., M i ) which by (0), H(M,..., M n ) = H(M i,..., M i ) ). At the receiver, first we use a Slepian-Wolf decoder with coplexity CX sw () to decode the transitted sources inforation. Then, we use a copressive sensing decoder to recover the whole sources. The fourth ethod is a naive way of transitting n correlated sources so that we have an easy decoder at the receiver. We call this schee a Naive Correlation Ignorance ethod. In this ethod, we siply ignore the correlation in the coding schee. In this naive schee, the required in-cut rate is n i= H(M i) with a decoding coplexity equal to CX indep (n). Note that, in sparse distributed copression with independent coding, the required in-cut rate is uch less than this schee ( ) << n i= H(M i)). The following theore is about sparse distributed copression with independent coding: Theore 3. For a depth one tree network with n-sources, k-sparsely correlated, the required in-cut rate between the receiver and sources in sparse distributed copression with independent coding (SDCIC) is, R H(M ij ), (4) j= where is the sallest nuber satisfies () for a given n and k and i j is the index of j th active source. Also, the decoding coplexity of SDCIC ethod is CX indep () + CX noiseless cs (, n). Proof: For a given n (nuber of sources) and k (the M n Fig. 2. M, M 2,,M n A general ulticast network with n sources. sparsity factor), we choose to satisfy (). Now, suppose each source is transitting its block with probability γ where γ = n. Hence, by the law of large nubers, the probability that we have transitting sources blocks goes to one for sufficiently large n. Without loss of generality, say sources,..., are transitting (i.e., active sources) and other sources are idle. If R i H(M i ) for active sources and zero for idle sources, active sources can transit their essages to the receiver and the required in-cut rate between the receiver and sources would be R ). Let S indicate active sources indices. First, at the receiver, we decode these active sources. The coplexity of this decoding part is CX indep () because we did not use their correlation in the coding schee. Hence, we have µ t,s, active sources essages at tie t at the receiver (in this case, µ t,s = (µ,t,..., µ,t )). Then, by having µ t,s and using the sparsity of µ t, the following optiization can recover the whole sources: in µ t L (5) subject to µ t,s = Φ S µ t. The overall decoding coplexity is CX indep () + (, n). CX noiseless cs B. A General Multicast Network with Correlated Sources In this section, we extend results of Section III-A to a general noiseless ulticast network. Note that, for a depth one tree network, references [3], [4], [5], [6], [7] and [8] provide soe low coplexity decoding techniques for Slepian-Wolf copression. However, none of these ethods can explicitly provide a trade-off between the rate and the decoding coplexity. For a general ulticast network, it has been shown in [9] that, there is no separation between Slepian-Wolf copression and network coding. For this network, a low coplexity decoder for an optial copression schee has not been proposed yet. There are few practical approaches and they rely on sall topologies [20]. In this section, by using copressive sensing, we propose a coding schee which provides a trade-off between the copression rate and the decoding coplexity. 32

5 Consider a ulticast network shown in Figure 2 which has n sources and T receivers. If sources are independent, reference [2] showed that the iniu required in-cut rate between each receiver and sources is as (8). This rate can be achieved by using linear network coding over the network. Reference [] showed that rando linear network coding can perfor arbitrarily closely to this rate bound. Say (n) is the coplexity of its decoder. Now, consider a case when sources are correlated. Reference [] extended Slepian-Wolf copression result ([0]) to a ulticast network by using network coding. The proposed decoder is a iniu entropy or a axiu probability decoder whose coplexity is high ([2]). We refer to its CX nc indep coplexity by CX nc sw(n). In this section, we consider sources to be k-sparsely correlated. We want to use copressive sensing along with network coding to propose a distributed copression schee providing a trade-off between the copression rate and the decoding coplexity. In the following, we copare four distributed copression ethods for a general ulticast network: In the first schee, we use sparse distributed copression with independent coding. With high probability, we have active sources. Then, we perfor network coding on these active sources over the network, assuing they are independent. At the receiver, first we decode these active sources and then, by a copressive sensing decoder, we recover the whole sources. The decoding coplexity is CXindep nc ()+CXnoiseless cs (, n). The required in-cut rate between each receiver and sources is ). In the second schee, we use an extended version of Slepian-Wolf copression for a ulticast network ([22]). Reference [22] considers a vector linear network code that operates on blocks of bits. The required in-cut rate between sources and the receiver is H(M,..., M n ). At the receiver, one needs to have a iniu entropy or a axiu probability decoder with coplexity CXsw(n). nc A cobination of sparse distributed copression and Slepian-Wolf copression can provide a trade-off between the copression rate and the decoding coplexity. For exaple, instead of n sources, one can transit of the by using a Slepian-Wolf coding for ulticast networks. The required in-cut rate would be H(M i,..., M i ). At the receiver, first we use a Slepian- Wolf decoder with coplexity CXsw() nc to decode active sources. Then, we use a copressive sensing decoder to recover the whole sources. In a naive correlation ignorance ethod, we siply ignore the correlation in the coding schee. In this naive schee, the required in-cut rate is n i= H(M i) with a decoding coplexity equal to CXindep nc (n). The following theore is about sparse distributed copression with independent coding for a ulticast network: Theore 4. For a general ulticast network with n-sources, SOURCE/NETWORK DECODER CS DECODER (a) (b) CS DECODER CHANNEL DECODER Fig. 3. A odularized decoder for a) copressive sensing with source coding, (b) copressive sensing with channel coding. k-sparsely correlated, the required in-cut rate between each receiver and sources in sparse distributed copression with independent coding is, R H(M ij ), (6) j= where is the sallest nuber satisfies () for a given n and k. Also, the decoding coplexity is CXindep nc () + (, n). CX noiseless cs Proof: The proof is siilar to the one of Theore 3. The only difference is that, here, one needs to perfor linear network coding on active sources over the network without using the correlation aong the. At the receiver, first, these active sources blocks are decoded (the decoding coplexity of this part is CXindep nc ()). Then, a copressive sensing decoder (5) is used to recover the whole sources. Reark: In this section, we explained how copressive sensing can be used with distributed source coding. At the receiver side, first, we decode the active sources by a network decoding technique and then, we use a copressive sensing decoder to recover the whole sources. This high level odularized decoding schee of sparse distributed copression is depicted in Figure 3-a. On the other hand, when we use copressive sensing for channel coding (which will be explained in detail in Section IV), we switch these two decoding odules (as shown in Figure 3-b). In other words, first, we use a copressive sensing decoder on a noisy signal to obtain a noisy estiate of the original signal and then, we use a channel decoder to find the original essage fro this noisy estiate. IV. COMPRESSIVE SENSING AND CHANNEL CODING In this section, we ake a bridge between copressive sensing, which is ore a signal processing technique, and channel coding. We consider a high-snr Gaussian pointto-point channel depicted in Figure 4. We propose a coding schee that cobines copressive sensing and rando channel coding with a odularized decoder to provide a tradeoff between the capacity loss and the decoding coplexity (i.e., the higher the capacity loss, the lower the decoding 33

6 M i ENCODER Fig. 4. W i A Point-to-Point Gaussian Channel Z Y=W i +Z coplexity.). We call this schee Sparse Channel Coding. We add intentionally soe correlation to transitted signals to decrease the decoding coplexity. This is an exaple of how copressive sensing can be used with channel coding and could be extended to other types of channels. Consider a point to point channel with additive Gaussian noise with noise power N and transission power constraint P, shown in Figure 4. Suppose we are in a high SNR regie P (i.e., N ). The capacity of this channel is, C 2 log ( P ). (7) N An encoding and a decoding schee for this channel can be found in [2]. As explained in [2], at the receiver, one needs to use a -diensional axiu likelihood decoder where, the code length, is arbitrarily large. We refer to the coplexity of such a decoder as CX l (). Our ai is to design a coding schee to have a trade-off between the rate and the decoding coplexity. Suppose is arbitrarily large. Choose n and k to satisfy (6). Definition 5. A (2 R, ) sparse channel code for a pointto-point Gaussian channel with power constraint P satisfies the following: Encoding process: essage i fro the set {, 2,..., 2 R } is assigned to a -length vector W i, satisfying the power constraint; that is, for each codeword, we have, Wij 2 P, (8) j= where W ij is the j th coordinate of codeword W i. Decoding process: the receiver receives Y, a noisy version of the transitted codeword W i (i.e., Y = W i + Z, where Z is a Gaussian noise vector with i.i.d. coordinates with power N). A decoding function g aps Y to the set {, 2,..., 2 R }. The decoding coplexity is CX l (k) + CXcs noisy (, n). A rate R is achievable if Pe 0 as, where, Pe = 2 R 2 R P r(g(y) i M = i), (9) i= and M represents the transitted essage. Theore 6. For a high-snr Gaussian point-to-point channel described in Definition 5, the following rate is achievable, while the decoding coplexity is CX l (k)+cxcs noisy (, n): R = k C + n H b(α) + k 2 log( ), (20) β( + δ k ) where C is the channel capacity (7), α, β and δ k are paraeters defined in Section II-A, and H b (.) is the binary entropy function. Before expressing the proof, let us ake soe rearks: if we have () (i.e., /n = ρα log(/α)), the achievable rate can be approxiated as follows: R log(/α) C + α log(/α) H b(α) (2) where α = k/n is the sparsity ratio. The first ter of (2) shows that the capacity loss factor is a log function of the sparsity ratio. The ter α log(/α) H b(α) is the rate gain that we obtain by using copressive sensing in our schee. Note that, since the SNR is sufficiently high, the overall rate would be less than the capacity. The coplexity ter CX l (k) is an exponential function, while CXcs noisy (, n) is a polynoial. Proof: We use rando coding and copressive sensing arguents to propose a concatenated channel coding schee satisfying (20). First, we explain the encoding part: Each essage i is apped randoly to a k-sparse vector X i with length n so that its non-zero coordinates are i.i.d. drawn fro N (0, k(+δ k ) P ). n is chosen to satisfy (6), for a given k and. To obtain W i, we ultiply X i by a n atrix (Φ S ) satisfying the RIP condition (3) (i.e., W i = Φ S X i ). We send W i through the channel. In fact, it is a concatenated channel coding schee [23]. The outer layer of this coding is the sparsity pattern of X i. The inner layer of this coding is based on rando coding around each sparisty pattern. To satisfy transission power constraint P, we generate each non-zero coordinate of X i by N (0, k(+δ k ) P ). Note that, the encoding atrix Φ S is known to both encoding and decoding sides. At the receiver, we have a noisy version of the transitted signal. Suppose essage i has been transitted. Hence, Y = W i + Z = Φ S X i + Z. The decoding part is as follows: Since X i is k-sparse, first, we use a copressive sensing decoder to find X i such that X i X i 2 L 2 βn, as follows, where β is the paraeter of (5): in X L (22) subject to Y Φ S X i 2 L 2 N. We assue that the coplexity of this convex optiization is saller than the one of a axiu likelihood decoder. Since we are in a high-snr regie, by having X i, we can find the sparsity pattern of X i. It gives us the outer layer code of essage i. The higher the outer layer code rate, the higher the required SNR. We shall develop this 34

7 arguent with ore detail later (26). Having the sparsity pattern of X i, we use a axiu likelihood decoder in a k-diensional space (i.e., nonzero coordinates of X i ) to find non-zero coordinates of X i (the inner layer code). The coplexity of this decoder is denoted by CX l (k). Before presenting the error probability analysis, let us discuss different ters of (20). Since in our coding schee, for each essage i, we send a k-sparse signal by sending a -length vector, we have the fraction k before the capacity ter C. The capacity ter C in (20) coes fro the inner layer coding schee. The ter n H b(α) coes fro the outer layer coding schee. Note that, since α is a sall nuber, H b (α) is close to one. Also, the ratio n depends on the outer layer code rate. Here, we assued the SNR is sufficiently high so that we can use all possible outer k 2 log( β(+δ k ) layer codes. The ter ) is because of the power constraint and the RIP condition. Note that, if one perfors a tie-sharing based channel coding, a rate k C is achievable with a decoding coplexity CX l (k). By using the copressive sensing, we obtain additional rate ters because of the outer layer coding with approxiately the sae coplexity. We proceed by the error probability analysis. Define Ξ i as the sparsity pattern of X i (i.e., Ξ ij = 0 if X ij = 0, otherwise, Ξ ij =.). Also, define Ξ i as the decoded sparsity pattern of essage i. Without loss of generality, assue that essage was transitted. Thus, Y = Φ S X + Z. Define the following events: E 0 = { Wj 2 > P } j= E i = { n X i X 2 L 2 β n N} E p = { Ξ Ξ }. (23) Hence, an error occurs when E 0 occurs (i.e., the power constraint is violated), or E p occurs (i.e., the outer layer code is wrongly decoded), or E c occurs (i.e., the underlying sparse signal of the transitted essage X and its decoded noisy version X are in a distance greater than the noise level), or one of E i occurs while i. Say M is the transitted essage (in this case M = ) and ˆM is the decoded essage. Let E denote the event M ˆM. Hence, P r(e M = ) = P r(e) R 2 = P r(e 0 Ep E c E j ) (24) j=2 P r(e 0 ) + P r(e p ) + P r(e c ) + P r(e j ), 2 R j=2 by union bounds of probabilities. We bound these probabilities ter by ter as follows: By the law of large nubers, we have P r( X 2 L 2 > +δ k P +ϵ) 0 as n. By using the RIP condition (3) and with probability one, we have, Φ S X 2 L 2 ( + δ k ) X 2 L 2 P. (25) Hence, P r(e 0 ) 0 as n. After using the copressive sensing decoder entioned in (22), we have X such that n X X 2 L 2 β n N. Suppose this error is uniforly distributed over different coordinates. We use a threshold coparison to deterine the sparsity pattern Ξ. The probability of error in deterining Ξ deterines how high the SNR should be. Intuitively, if we use all possible outer layer codes (all possible sparsity patterns), we should not ake any error in deterining the sparsity pattern of each coordinate j (i.e., Ξ j ). Hence, we need sufficiently high SNR for a given n. On the other hand, if we decrease the outer layer code rate, the required SNR would be lower than the case before. Here, to illustrate how the required SNR can be coputed, we assue that we use all possible sparsity patterns. Say E pj is the event of aking an error in deterining whether the j th coordinate is zero or not. We say Ξ j = 0 if X ij < τ. Otherwise, Ξ j =. Hence, for a given threshold τ, we have, τ P r(e pj Ξ j = 0) = ϕ( ) β n N τ P r(e pj Ξ j = ) = 2ϕ( ) P + βn N where ϕ(.) is the cuulative distribution function of a noral rando variable. Hence, P r(e p ) = P r(ep c ) (26) = ( τ ) n( α) ϕ( ) β n N ( τ 2ϕ( ) ) nα P + β n N Note that, one can choose τ and P N large enough to have P r(e p ) arbitrarily sall. By [2], P r(e) c is zero. At the last step, we need to bound P r(e j ) for j. We have, P r(e j ) = P r(ξ j = Ξ )P r(e j Ξ j = Ξ ) + P r(ξ j Ξ )P r(e j Ξ j Ξ ) = α k ( α) n k P r(e j Ξ j = Ξ ).(27) Note that, P r(e j Ξ j Ξ ) goes to zero in a high SNR regie. By bounding the noise power of non-zero coordinates by βn (the whole noise power) and using rando coding arguent in a k diensional space, we 35

8 have, P r(e j Ξ j = Ξ ) ( + P β(+δ k ) N ), (28) where indicates the right hand side is less or equal than the left one in an exponential rate. Therefore, by (27) and (28), and doing soe anipulations, we have, P (E j ) 2 n(h b(α)+ α 2 log( P β(+δ k ) N )), (29) for j = 2,..., 2 R. Thus, in a sufficiently high SNR regie, by (24) and (29), we have, P e = P r(e) = P r(e M = ) (30) 2 R 2 n(h b(α)+ α 2 log( P β(+δ k ) N )) 2 (R n H b(α) k C k 2 log( β(+δ k ) )). For any given ϵ > 0, we can choose n large enough to have R k C + n H b(α) + k 2 log( β(+δ k ) ) ϵ achievable. V. CONCLUSIONS In this paper, we deonstrated soe applications of copressive sensing over networks. We ade a connection between copressive sensing and traditional inforation theoretic techniques in source coding and channel coding in order to provide echaniss to explicitly trade-off between the decoding coplexity and the rate. Although optial decoders to recover the original signal copressed by source coding have high coplexity, the copressive sensing decoder is a linear or convex optiization. First, we investigated applications of copressive sensing on distributed copression of correlated sources. For a depth one tree network, references ([3]-[8]) provide soe low coplexity decoding techniques for Slepian-Wolf copression. However, none of these ethods can explicitly provide a trade-off between the rate and the decoding coplexity. For a general ulticast network, reference [] extended Slepian-Wolf copression for a ulticast proble. It is showed in [9] that, there is no separation between Slepian-Wolf copression and network coding. For this network, a low coplexity decoder for an optial copression schee has not been proposed yet. Here, by using copressive sensing, we proposed a copression schee for a faily of correlated sources with a odularized decoder providing a trade-off between the copression rate and the decoding coplexity. We called this schee Sparse Distributed Copression. We used this copression schee for a general ulticast network with correlated sources. Here, we first decoded soe of the sources by a network decoding technique and then, we used a copressive sensing decoder to obtain the whole sources. Next, we investigated applications of copressive sensing on channel coding. We proposed a coding schee that cobines copressive sensing and rando channel coding for a high-snr point-to-point Gaussian channel. We called this schee Sparse Channel Coding. Our coding schee provides a odularized decoder to have a trade-off between the capacity loss and the decoding coplexity. The idea is to add intentionally soe correlation to transitted signals in order to decrease the decoding coplexity. At the decoder side, first we used a copressive sensing decoder to get an estiate of the original sparse signal, and then we used a channel coding decoder in the subspace of non-zero coordinates. REFERENCES [] E. Candès, J. Roberg, and T. Tao, Stable signal recovery fro incoplete and inaccurate easureents, Counications on Pure and Applied Matheatics, vol. 59, no. 8, pp , [2] E. Candès and J. Roberg, Sparsity and incoherence in copressive sapling, Inverse Probles, vol. 23, p. 969, [3] D. Donoho, Copressed sensing, IEEE Transactions on Inforation Theory, vol. 52, no. 4, pp , [4] M. Akçakaya and V. Tarokh, Shannon theoretic liits on noisy copressive sapling, arxiv, vol. 7. [5] S. Sarvotha, D. Baron, and R. Baraniuk, Measureents vs. bits: Copressed sensing eets inforation theory, in Proceedings of 44th Allerton Conf. Co., Ctrl., Coputing, [6] Y. Jin, Y.-H. Ki, and B. D. Rao, Perforance tradeoffs for exact support recovery of sparse signals, in 200 IEEE International Syposiu on Inforation Theory, ISIT 200, TX, US, 200, pp [7] E. Candes and T. Tao, Decoding by linear prograing, IEEE Transactions on Inforation Theory, vol. 5, no. 2, pp , [8] R. Baraniuk, M. Davenport, R. DeVore, and M. Wakin, A siple proof of the restricted isoetry property for rando atrices, Constructive Approxiation, vol. 28, no. 3, pp , [9] D. Achlioptas, Database-friendly rando projections: Johnson- Lindenstrauss with binary coins, Journal of Coputer and Syste Sciences, vol. 66, no. 4, pp , [0] D. Slepian and J. K. Wolf, Noiseless coding of correlated inforation sources, IEEE Trans. Inf. Theory, vol. 9, no. 4, pp , Jul [] T. Ho, M. Médard, R. Koetter, D. R. Karger, M. Effros, J. Shi, and B. Leong, A rando linear network coding approach to ulticast, IEEE Trans. Inf. Theory, vol. 52, no. 0, pp , Oct [2] I. Csiszar and J. Körner, in Inforation Theory: Coding Theores for Discrete Meoryless Systes. Acadeic Press, New York, 98. [3] S. Pradhan and K. Rachandran, Distributed source coding using syndroes (DISCUS): Design and construction, in dcc. Published by the IEEE Coputer Society, 999, p. 58. [4] T. Colean, A. Lee, M. Médard, and M. Effros, Low-coplexity approaches to Slepian Wolf near-lossless distributed data copression, IEEE Transactions on Inforation Theory, vol. 52, no. 8, [5] R. Gallager, Low-density parity-check codes, Inforation Theory, IRE Transactions on, vol. 8, no., pp. 2 28, 962. [6] T. Tian, J. Garcia-Frias, and W. Zhong, Copression of correlated sources using LDPC codes, in Data Copression Conference, Proceedings. DCC 2003, [7] C. Berrou, A. Glavieux, P. Thitiajshia et al., Near Shannon liit error-correcting coding and decoding, in Proceedings of ICC, vol. 93, 993, pp [8] J. Garcia-Frias, Copression of correlated binary sources using turbo codes, IEEE Counications Letters, vol. 5, no. 0, pp , 200. [9] A. Raaoorthy, K. Jain, P. Chou, and M. Effros, Separating distributed source coding fro network coding, Inforation Theory, IEEE Transactions on, vol. 52, no. 6, pp , [20] G. Maierbacher, J. Barros, and M. Médard, Practical source-network decoding, in Wireless Counication Systes, ISWCS th International Syposiu on. IEEE, 2009, pp [2] R. Ahlswede, N. Cai, S.-Y. R. Li, and R. W. Yeung, Network inforation flow, IEEE Trans. Inf. Theory, vol. 46, pp , [22] R. Koetter and M. Medard, An algebraic approach to network coding, IEEE/ACM Trans. on networking, vol., no. 5, pp , Oct [23] G. Forney, Concatenated codes. Citeseer,