Limits on Support Recovery with Probabilistic Models: An Information-Theoretic Framework

Transcription

1 Limits on Support Recovery with Probabiistic Modes: An Information-Theoretic Framewor Jonathan Scarett and Voan Cevher arxiv:5.744v3 cs.it 3 Aug 6 Abstract The support recovery probem consists of determining a sparse subset of a set of variabes that is reevant in generating a set of observations, and arises in a diverse range of settings such as compressive sensing, and subset seection in regression, and group testing. In this paper, we tae a unified approach to support recovery probems, considering genera probabiistic modes reating a sparse data vector to an observation vector. We study the information-theoretic imits of both exact and partia support recovery, taing a nove approach motivated by threshoding techniques in channe coding. We provide genera achievabiity and converse bounds characterizing the trade-off between the error probabiity and number of measurements, and we speciaize these to the inear, -bit, and group testing modes. In severa cases, our bounds not ony provide matching scaing aws in the necessary and sufficient number of measurements, but aso sharp threshods with matching constant factors. Our approach has severa advantages over previous approaches: For the achievabiity part, we obtain sharp threshods under broader scaings of the sparsity eve and other parameters e.g., signa-to-noise ratio compared to severa previous wors, and for the converse part, we not ony provide conditions under which the error probabiity fais to vanish, but aso conditions under which it tends to one. Index Terms Support recovery, sparsity pattern recovery, information-theoretic imits, compressive sensing, non-inear modes, -bit compressive sensing, group testing, phase transitions, strong converse I. INTRODUCTION The support recovery probem consists of determining a sparse subset of a set of variabes that is reevant in producing a set of observations, and arises frequenty in discipines such as group testing,, compressive sensing CS 3, and subset seection in regression 4. The observation modes can vary significanty among these discipines, and it is of considerabe interest to consider these in a unified fashion. This can be done via probabiistic modes reating the sparse vector β R p to a singe observation Y R in the foowing manner: Y S = s, X = x, β = b P Y XS β S x s, b s, The authors are with the Laboratory for Information and Inference Systems LIONS, Écoe Poytechnique Fédérae de Lausanne EPFL emai: jonathan.scarett,voan.cevher}@epf.ch. This wor was supported in part by the European Commission under Grant ERC Future Proof, SNF and SNF CRSII-47633, and EPFL Feows Horizon grant This wor was presented in part at the IEEE Internationa Symposium on Information Theory 5, and at the ACM-SIAM Symposium on Discrete Agorithms 6. where S,..., p} represents the set of reevant variabes, X R p is a measurement vector, X S respectivey, β S is the subvector of X respectivey, β S containing the entries indexed by S, and P Y XS β S is a given probabiity distribution. Given a coection of measurements Y R n and the corresponding measurement matrix X R n p with each row containing a singe measurement vector, the goa is to find the conditions under which the support S can be recovered either perfecty or partiay. In this paper, we study the information-theoretic imits for this probem, characterizing the number of measurements n required in terms of the sparsity eve and ambient dimension p regardess of the computationa compexity. Such studies are usefu for assessing the performance of practica techniques and determining to what extent improvements are possibe. Before proceeding, we state some important exampes of modes that are captured by. Linear Mode: The inear mode 5, 6 is ubiquitous in signa processing, statistics, and machine earning, and in itsef covers an extensive range of appications. Each observation taes the form Y = X, β + Z, where, denotes the inner product, and Z is additive noise. An important quantity in this setting is the signa-tonoise ratio SNR E X,β EZ, and in the context of support recovery, the smaest non-zero absoute vaue β min in β has aso been shown to pay a ey roe 5, 7, 8. Quantized Linear Modes: Quantized variants of the inear mode are of significant interest in appications with hardware imitations. An exampe that we wi consider in this paper is the -bit mode 9, given by Y = sign X, β + Z, 3 where the sign function equas if its argument is nonnegative, and if it is negative. Group Testing: Studies of group testing probems began severa decades ago,, and have recenty regained significant attention,, with appications incuding medica testing, database systems, computationa bioogy, and faut detection. The goa is to determine a sma number of defective items within a arger subset of items. The items invoved in a singe test are indicated by X, } p, and each observation taes the form } Y = X i = } Z, 4 i S

2 with S representing the defective items, Y indicating whether the test contains at east one defective item, and Z representing possibe noise here denotes moduo- addition. In this setting, one can thin of β as deterministicay having entries equaing one on S, and zero on S c. The above exampes highight that captures both discrete and continuous modes. Beyond these exampes, severa other non-inear modes are captured by, incuding the ogistic, Poisson, and gamma modes. A. Previous Wor and Contributions Numerous previous wors on the information-theoretic imits of support recovery have focused on the inear mode 5, 7, 8, 3. The main aim of these wors, and of that the present paper, is to deveop necessary and sufficient conditions for which an error probabiity vanishes as p. However, there are severa distinctions that can be made, incuding: Random measurement matrices 5, 7, 8, 3 vs. arbitrary measurement matrices 6, 8, 9; Exact support recovery 5, 7, 8, 3 vs. partia support recovery 5, 6, ; Minimax characterizations for β in a given cass 5, 7, 8, 3 vs. average performance bounds for random β 4, 6,. Perhaps the most widey-studied combination of these is that of minimax characterizations for exact support recovery with random measurement matrices. In this setting, within the cass of vectors β whose non-zero entries have an absoute vaue exceeding some threshod β min, necessary and sufficient conditions on n are avaiabe with matching scaing aws 7, 8. See aso 3, 4 for informationtheoretic studies of the inear mode with a mean square error criterion. Compared to the inear mode, research on the information-theoretic imits of support recovery for noninear modes is reativey scarce. The system mode that we have adopted foows those of a ine of wors seeing mutua information characterizations of sparsity probems,, 4, 5, though we mae use of significanty different anaysis techniques. Simiary to these wors, we focus on random measurement matrices and random non-zero entries of β. Other wors considering non-inear modes have used vasty different approaches such as reguarized M-estimators 6, 7 and approximate message passing 8. High-eve Contributions: We consider an approach using threshoding techniques ain to those used in information-spectrum methods 9, thus providing a new aternative to previous approaches based on maximumieihood decoding and Fano s inequaity. Our ey contributions and the advantages of our framewor are as foows:. Considering both exact and partia support recovery, we provide non-asymptotic performance bounds appying to genera probabiistic modes, aong with a procedure for appying them to specific modes cf. Section III-B.. We expicity provide the constant factors in our bounds, aowing for more precise characterizations of the performance compared to wors focusing on scaing aws e.g., see 5, 8,. In severa cases, the resuting necessary and sufficient conditions on the number of measurements coincide up to a mutipicative + o term, thus providing exact asymptotic threshods sometimes referred to as phase transitions 4, 3 on the number of measurements. 3. As evidenced in our exampes outined beow, our framewor often eads to such exact or near-exact threshods for significanty more genera scaings of, SNR, etc. compared to previous wors. 4. The majority of previous wors have deveoped converse resuts using Fano s inequaity, eading to necessary conditions for Perror. In contrast, our converse resuts provide necessary conditions for Perror. The distinction between these two conditions is important from a practica perspective: One may not expect a condition such as Perror to be significant, whereas the condition Perror is inarguaby so. Contributions for Specific Modes: An overview of our bounds for specific modes is given in Tabe I, where we state the derived bounds with the asymptoticay negigibe terms omitted. A of the modes and their parameters are defined precisey in Section IV; in particuar, the functions f,..., f 9 and the remainder terms, 9 are given expicity, and are easy to evauate. We proceed by discussing these contributions in more detai, and comparing them to various existing resuts in the iterature:. Linear mode In the case of exact recovery, we recover the exact threshods on the required number of measurements given by Jin et a. 7, as we as handing a broader range of scaings of β min := min β i : β i } see Section IV-A for detais and strengthening the converse by considering the more stringent condition Perror. Our resuts for partia recovery provide near-matching necessary and sufficient conditions under scaings with = op, thus compementing the extensive study of the scaing = Θp by Reeves and Gastpar 5, 6.. -bit mode We provide two surprising observations regarding the -bit mode: Coroary 3 provides a ow- SNR setting where the quantization ony increases the asymptotic number of measurements by a factor of π, whereas Coroary 4 provides a high-snr setting where the scaing aw is stricty worse than the inear mode. Simiar behavior wi be observed for partia recovery Coroaries and 5 by numericay comparing the bounds for various SNR vaues. 3. Group testing Asymptotic threshods for group testing with = Θ were given previousy by Mayutov and Atia and Saigrama. However, for the case that, the sufficient conditions of that introduced additiona ogarithmic factors. In

3 3 Mode Resut Parameters Distributions Sufficient n for Perror Necessary n for Perror + Discrete β og p Cor. = op S max og p + Gaussian X =,... f max =,... f Linear Cor. Cor. 3 -bit Cor. 4 Group testing Genera discrete observations Cor. 5, = op Partia recovery of proportion α = Θ Low SNR = Θp High SNR, = op Partia recovery of proportion α Cor. 6 = Θp θ Cor. 7 Cor. 8 = Θp θ Noisy crossover probabiity ρ, = op Partia recovery of proportion α Gaussian β S Gaussian X Discrete β S Gaussian X Fixed β S Gaussian X Gaussian β S Gaussian X Fixed β S Bernoui X Fixed β S Bernoui X Fixed β S Bernoui X for various scaings α og p max α α, f α og p max =,... f 3 α α og p max α α, f α og p max =,... f 3 within a factor π of inear mode within a factor π of inear mode - Ωp og p compared to Θp for inear mode α og p max α α, f 5 α α α og p max α α, f 5 α og p p og f 6 θ f 6θ = og for θ 3 og og p f 7 θ f 7θ = og H ρ for sma θ og p og H ρ Cor. 9 Arbitrary Arbitrary - max =,... og p og H ρ α og p og H ρ og p + f Tabe I: Overview of main resuts for exact or partia support recovery under various observation modes. In the necessary and sufficient number of measurements, asymptoticay negigibe terms have been omitted. A quantities are defined precisey in Section IV. contrast, we obtain matching Θ og p scaing aws for any subinear scaing of the form = Op θ θ,. Moreover, for sufficienty sma θ we obtain exact threshods. In particuar, for the noiseess p setting we show that n og measurements are both necessary and sufficient for θ 3. This is in fact the same threshod as that for adaptive group testing 3, thus proving that non-adaptive Bernoui measurement matrices are asymptoticay optima even when adaptivity is aowed; this was previousy nown ony in the imit as θ 3. For the noisy case, we prove an anaogous caim for sufficienty sma θ. A shortened and simpified version of this paper focusing excusivey on group testing can be found in Genera discrete observations Our converse for the case of genera discrete observations Coroary 9 recovers that of Tan and Atia 5 for the case that β S is fixed, strengthens it due to a smaer remainder term 9, and provides a generaization to the case that β S is random. B. Structure of the Paper In Section II, we introduce our system mode. In Section III, we present our main non-asymptotic achievabiity and converse resuts for genera observation modes, and the procedure for appying them to specific probems. Severa appications of our resuts to specific modes are presented in Section IV. The proofs of the genera bounds are given in Section V, and concusions are drawn in Section VI. C. Notation We use upper-case etters for random variabes, and owercase variabes for their reaizations. A non-bod character may be a scaar or a vector, whereas a bod character refers to a coection of n scaars e.g., Y R n or vectors e.g., X R n p. We write β S to denote the subvector of β at the coumns indexed by S, and X S to denote the submatrix of X containing the coumns indexed by S. The compement with respect to,..., p} is denoted by c. The symbo means distributed as. For a given joint distribution P XY, the corresponding margina distributions are denoted by P X and P Y, and simiary for conditiona marginas e.g., P Y X. We write P for probabiities, E for expectations, and Var for variances. We use usua notations for the entropy e.g., HX and mutua information e.g., IX; Y, and their conditiona counterparts e.g., HX Z, IX; Y Z. Note that H may aso denote the differentia entropy for continuous random variabes; the distinction wi be cear from the context. We define the binary entropy function H ρ := ρ og ρ ρ og ρ, and the Q-function Qx := PW x W N,. We mae use of the standard asymptotic notations O, o, Θ, Ω and ω. We define the function + =

4 4 max, }, and write the foor function as. The function og has base e. A. Mode and Assumptions II. PROBLEM SETUP Reca that p denotes the ambient dimension, denotes the sparsity eve, and n denotes the number of measurements. We et S be the set of subsets of,..., p} having cardinaity. The ey random variabes in our setup are the support set S S, the data vector β R p, the measurement matrix X R n p, and the observation vector Y R n. The support set S is assumed to be equiprobabe on the p subsets within S. Given S, the entries of βs c are deterministicay set to zero, and the remaining entries are generated according to some distribution β S P βs. We assume that these non-zero entries foow the same distribution for a of the p possibe reaizations of S, and that this distribution is permutation-invariant. The measurement matrix X is assumed to have i.i.d. vaues on some distribution P X. We write P n p X, to denote the corresponding i.i.d. distributions for matrices, and we write PX as a shorthand for PX. Given S, X, and β, each entry of the observation vector Y is generated in a conditionay independent manner, with the i-th entry Y i distributed according to Y i S = s, X i = x i, β = b P Y XS β S x s i, b s, 5 for some conditiona distribution P Y XS β S. We again assume symmetry with respect to S, namey, that P Y XS β S does not depend on the specific reaization, and that the distribution is invariant when the coumns of X S and the entries of β S undergo a common permutation. Given X and Y, a decoder forms an estimate Ŝ of S. Simiary to previous wors studying information-theoretic imits on support recovery, we assume that the decoder nows the system mode. We consider two reated performance measures. In the case of exact support recovery, the error probabiity is given by P e := PŜ S, 6 and is taen with respect to the reaizations of S, β, X, and Y; the decoder is assumed to be deterministic. We aso consider a ess stringent performance criterion requiring that ony d max entries of S are successfuy recovered, for some d max,..., }. Foowing 5, 6, the error probabiity is given by P e d max := P S\Ŝ > d max Ŝ\S > d max. 7 Note that if both S and Ŝ have cardinaity with probabiity one, then the two events in the union are identica, and hence either of the two can be removed. For carity, we formay state our main assumptions as foows: Extensions to more genera aphabets beyond R are straightforward. A The support set S is uniform on the p subsets of,..., p} of size, and the measurement matrix X is i.i.d. on some distribution P X. A The non-zero entries β S are distributed according to P βs, and this distribution is permutation-invariant and the same for a reaizations of S. A3 The observation vector Y is conditionay i.i.d. according to P Y XS β S, and this distribution is the same for a reaizations of S, and invariant to common permutations of the coumns of X S and entries of β S. A4 The decoder is given X, Y, and aso nows the system mode incuding, P Y XS β S, and P βs. Our main goa is to derive necessary and sufficient conditions on n and as functions of p such that P e or P e d max vanishes as p. Moreover, when considering converse resuts, we wi not ony be interested in conditions under which P e, but aso conditions under which the stronger statement P e hods. In particuar, we introduce the terminoogy that the strong converse hods if there exists a sequence of vaues n, indexed by p, such that for a η >, we have P e when n n + η, and P e when n n η. This is reated to the notion of a phase transition 4, 3. More generay, we wi refer to conditions under which P e as strong impossibiity resuts, not necessariy requiring matching achievabiity bounds. That is, the strong converse concusivey gives a sharp threshod between faiure and success, whereas a strong impossibiity resut may not. It wi prove convenient to wor with random variabes that are impicity conditioned on a fixed vaue of S, say s =,..., }. We write P βs and P Y Xsβ s in pace of P βs and P Y XS β S to emphasize that S = s. Moreover, we define the corresponding joint distribution P βsx sy b s, x s, y := P βs b s P Xx s P Y Xsβ s y x s, b s, 8 and its mutipe-observation counterpart P βsx syb s, x s, y := P βs b s P n X x sp n Y X sβ s y x s, b s. 9 where P n Y X sβ s, b s is the n-fod product of P Y Xsβ s, b s. Except where stated otherwise, the random variabes β s, X s, Y and β s, X s, Y appearing throughout this paper are distributed as β s, X s, Y P βsx sy β s, X s, Y P βsx sy, with the remaining entries of the measurement matrix being distributed as X s c P n p X, and with β s c = deterministicay. That is, we condition on a fixed S = s except where stated otherwise. For notationa convenience, the main parts of our anaysis are presented with P βs, P X and P Y Xsβ s representing probabiity mass functions PMFs, and with the corresponding averages written using summations. However, except where stated otherwise, our anaysis is directy appicabe to case

5 5 that these distributions instead represent probabiity density functions PDFs, with the summations repaced by integras where necessary. The same appies to mixed discretecontinuous distributions. Message S Channe State βs PβS Codeword Output XS Y PY n XSβS Encoder Channe Decoder Estimate Ŝ B. Information-Theoretic Definitions Before introducing the required definitions for support recovery, it is instructive to discuss threshoding techniques in channe coding studies. These commenced in eary wors such as 34, 35, and have recenty been used extensivey in information-spectrum methods 9, 36. Channe Coding: We first reca the mutua information, which is ubiquitous in information theory: IX; Y := x,y P XY x, y og P Y Xy x. P Y y In deriving asymptotic and non-asymptotic performance bounds, it is common to wor directy with the ogarithm, ıx; y := og P Y Xy x, 3 P Y y which is commony nown as the information density. The threshoding techniques wor by manipuating probabiities of events of the form n i= ıx i; Y i γ and n i= ıx i; Y i > γ. For the former, one can perform a change of measure from the conditiona distribution Y given X to the unconditiona distribution of Y, with a mutipicative constant e γ. For the atter, one can simiary perform a change of measure from Y to Y X. Hence, in both cases, there is a simpe reation between the conditiona and unconditiona probabiities of the output sequences. Using these methods, one can get upper and ower bounds on the error probabiity such that the dominant term is n P ıx i ; Y i IX; Y + ζ n 4 n i= for some ζ n = o. Assuming that X i, Y i } n i= has some form of i.i.d. structure, one can anayze this expression using toos from probabiity theory. The aw of arge numbers yieds the channe capacity C = max PX IX; Y, and refined characterizations can be obtained using variations of the centra imit theorem 37. Among the channe coding iterature, our anaysis is most simiar to that of mixed channes 9, Sec. 3.3, where the reation between the input and output sequences is not i.i.d., but instead conditionay i.i.d. given another random variabe. In our setting, β s wi pay the roe of this random variabe. See Figure for a depiction of this connection. Support Recovery: As in, 4, we wi consider partitions of the support set s S into two sets s dif and s eq. As wi be seen in the proofs, s eq wi typicay correspond to an overap between s and some other set s i.e., s s, whereas s dif wi correspond to the indices in one set but not the other e.g., s\s. There are ways of performing such a partition with s dif. Figure : Connection between support recovery and coding over a mixed channe. For fixed s S and a corresponding pair s dif, s eq, we introduce the notation P Y Xsdif X seq y x sdif, x seq := P Y Xs y x s 5 P Y Xsdif X seq β s y x sdif, x seq, b s := P Y Xsβ s y x s, b s, 6 where P Y Xs is the margina distribution of 9. Whie the eft-hand sides of 5 6 represent the same quantities for any such s dif, s eq, it wi sti prove convenient to wor with these in pace of the right-hand sides. In particuar, this aows us to introduce the margina distributions P Y Xseq y x seq := x sdif P n X x s dif P Y Xsdif X seq y x sdif, x seq 7 P Y Xseq β s y x seq, b s := x sdif P Xx sdif P Y Xsdif X seq β s y x sdif, x seq, b s, 8 where := s dif. Using the preceding definitions, we introduce two information densities. The first contains probabiities averaged over β s, ıx sdif ; y x seq := og P Y X sdif X seq y x sdif, x seq, 9 P Y Xseq y x seq whereas the second conditions on β s = b s : n ı n x sdif ; y x seq, b s := ıx s i dif ; y i x i s eq, b s, i= where the singe-etter information density is ıx sdif ; y x seq, b s := og P Y X sdif X seq β s y x sdif, x seq, b s. P Y Xseq β s y x seq, b s As mentioned above, we wi generay wor with discrete random variabes for carity of exposition, in which case the ratio is between two PMFs. In the case of continuous observations the ratio is instead between two PDFs, and more generay this can be repaced by the Radon-Niodym derivative as in the channe coding setting 37. Averaging with respect to the random variabes in conditioned on β s = b s yieds a conditiona mutua information, which we denote by I sdif,s eq b s := IX sdif ; Y X seq, β s = b s.

6 6 This quantity wi pay a ey roe in our bounds, which wi typicay have the form p s P e P n max dif, 3 s dif,s eq I sdif,s eq β s as wi be made more precise in the subsequent sections. III. GENERAL ACHIEVABILITY AND CONVERSE BOUNDS In this section, we provide genera resuts hoding for arbitrary modes satisfying the assumptions given in Section II. Each of the resuts for exact recovery has a direct counterpart for partia recovery. For carity, we focus on the former throughout Sections III-A and III-B, and then proceed with the atter in Section III-C. A. Initia Non-Asymptotic Bounds Here we provide our main non-asymptotic upper and ower bounds on the error probabiity. These bounds bear a strong resembance to anaogous bounds from the channe coding iterature 9; in each case, the dominant term invoves tai probabiities of the information density given in. The mean of the information density is the mutua information in, which thus arises naturay in the subsequent necessary and sufficient conditions on n upon showing that the deviation from the mean is sma with high probabiity. The procedure for doing this given a specific mode wi be given in Section III-B. We start with our achievabiity resut. Here and throughout this section, we mae use of the random variabes defined in. Theorem. For any constants δ > and γ, there exists a decoder such that P e P ı n X sdif ; Y X seq, β s og s dif,s eq : s dif p +og s dif where P γ := P og P Y X s,β s Y X s, β s P Y Xs Y X s δ Proof: See Section V-A. } +γ +P γ+δ, s dif 4 > γ. 5 Remar. The probabiity in the definition of P γ is not an i.i.d. sum, and the techniques for ensuring that P γ vary between different settings. The foowing approaches wi suffice for a of the appications in this paper:. In the case that P βs is discrete, P Y Xs y x s = b s P βs b s P Y Xs,β s y x s, b s, and it foows that γ = og min bs P βs b s = P γ =. 6 Moreover, this can be strengthened by noting from the proof of Theorem that γ may depend on β s, and choosing γb s = og P βs b accordingy. s. Defining I := Iβ s ; Y X s 7 V := Var og P Y X s,β s Y X s, β s, 8 P Y Xs Y X s we have for any δ > that V γ = I + = P γ δ. 9 δ This foows directy from Chebyshev s inequaity. 3. Defining I,+ := E og P Y X sβ s Y X s, β s +, 3 P Y Xs Y X s we have for any δ > that γ = I,+ δ = P γ δ. 3 This foows directy from Marov s inequaity. The proof of Theorem is based on a decoder the searches for a unique support set s such that ıx sdif ; y x seq > γ sdif 3 for some γ } = and a partitions s dif, s eq of s with s dif. Since the numerator in 9 is the ieihood of y given x sdif, x seq, this decoder can be thought of a weaened version of the maximum-ieihood ML decoder. Lie the ML decoder, computationa considerations mae its impementation intractabe. The foowing theorem provides a genera non-asymptotic converse bound. Theorem. Fix δ >, and et s dif b s, s eq b s be an arbitrary partition of s =,..., } with s dif depending on b s R. For any decoder, we have P e P ı n X sdif β s; Y X seqβ s, β s p + sdif β s og + og δ δ. 33 s dif β s Proof: See Section V-B. The proof of Theorem is based on Verdú-Han type bounding techniques 36. B. Techniques for Appying Theorems and The bounds presented in the preceding theorems do not directy revea the number of measurements required to achieving a vanishing error probabiity. In this subsection, we present the steps that can be used to obtain such conditions. We provide exampes in Section IV. The idea is to use a concentration inequaity to bound the first term in 4 or 33, which is possibe due to the fact

7 7 that each summation ı n is conditionay i.i.d. given β s. We proceed by providing the detais of these steps separatey for the achievabiity and converse. We start with the former.. Observe that, conditioned on β s = b s, the mean of ı n X sdif ; Y X seq, β s is ni sdif,s eq b s, where I sdif,s eq b s is defined in.. Fix δ,, and suppose that for a fixed vaue b s of β s, we have for a s dif, s eq that og p + og s dif δ + γ s dif n δ I sdif,s eq b s, 34 and P ı n X sdif ; Y X seq, b s n δ I sdif,s eq b s βs = b s ψ sdif n, δ 35 for some functions ψ } = e.g., these may arise from Chebyshev s inequaity or Bernstein s inequaity 38, Ch.. Combining these conditions with the union bound, we obtain P ı n X sdif ; Y X seq, β s s dif,s eq : s dif p og +og s dif δ s dif = } +γ βs = b s ψ n, δ Observe that the condition in 34 can be written as og p s dif + og δ n s dif + γ. 37 I sdif,s eq b s δ We summarize the preceding findings in the foowing. Theorem 3. For any constants δ >, δ, and γ, and functions ψ } = ψ : Z R R, define the set Bδ, δ, γ := b s : 35 and 37 hod for a Then we have P e P β s / Bδ, δ, γ + s dif, s eq with s dif }. 38 = ψ n, δ +P γ+δ. 39 Remar. The preceding arguments remain unchanged when δ aso depends on = s dif. We eave this possibe dependence impicit throughout this section, since a fixed vaue wi suffice for a but one of the modes considered in Section IV. In the case that 35 hods for a b s or more generay, within a set whose probabiity under P βs tends to one and the fina three terms in 39 vanish, the overa upper bound approaches the probabiity, with respect to P βs, that 37 fais to hod. In many cases, the second ogarithm in the numerator therein is dominated by the first. It shoud be noted that the condition that the second term in 39 vanishes can aso impose conditions on n. For most of the exampes presented in Section IV, the condition in 37 wi be the dominant one; however, this need not aways be the case, and it depends on the concentration inequaity used in 35. The appication of Theorem is done using simiar steps, so we provide ess detai. Fix δ >, and suppose that, for a fixed vaue b s of β s, the pair s dif, s eq = s dif b s, s eq b s is such that p + sdif og og δ n+δ I sdif,s s dif eq b s, 4 and P ı n X sdif ; Y X seq, b s n + δ I sdif,s eq b s βs = b s ψ s dif n, δ 4 for some function ψ s dif. Combining these conditions, we see that the first probabiity in 33, with an added conditioning on β s = b s, is ower bounded by ψ s dif n, δ. In the case that ψ is defined for mutipe vaues corresponding to different vaues of b s, we can further ower bound this by max ψ n, δ. Next, we observe that 4 hods if and ony if n og p + s dif s dif I sdif,s eq b s + δ og δ. 4 Recaing that the partition s dif, s eq is an arbitrary function of β s, we can ensure that this coincides with og p + s dif s n max dif og δ 43 s dif,s eq : s dif I sdif,s eq b s + δ by choosing each pair s dif, s eq as a function of b s to achieve this maximum. Finay, we note that the maximum over in the abovederived term max ψ n, δ may be restricted to any set L,..., } provided that s dif is constrained simiary in 43; one simpy chooses the partition s dif b s, s eq b s so that = s dif aways ies in this set. Putting everything together, we have the foowing. Theorem 4. For any set L,..., }, constants δ > and δ >, and functions ψ } L ψ : Z R R, define the set B δ, δ := b s : 4 and 4 hod for a s dif, s eq with s dif L }. 44 Then we have P e P β s B δ, δ max L ψ n, δ δ. 45 If the pair s dif, s eq had been fixed in Theorem, as opposed to being a function of β s, then we woud have ony obtained a weaer resut with the statement for a s dif, s eq

8 8 Procedure : Steps for Obtaining Necessary and Sufficient Conditions on n from Theorems 3 and 4. Identify a Typica Set Construct a sequence of typica sets T β R of non-zero entries, indexed by p, such that P β s T β, thus restricting the vectors b s for which ıx sdif ; Y X seq, b s needs to be characterized.. Bound the Information Density Tai Probabiities Using a concentration inequaity for i.i.d. summations e.g., Chebyshev, Bernstein, bound the tai probabiities in 35 and 4 for each s dif, s eq and b s T β, with a fixed constant δ. Upon maing these dependent on s dif, s eq, b s ony through := s dif, the bounds are denoted by ψ n, δ and ψ n, δ. 3. Contro the Remainder Terms By suitabe rearrangements, find conditions on n under which the terms ψ n, δ and max L ψ n, δ in 39 and 45 vanish, thus ensuring that their contribution is negigibe. Simiary, choose δ to vanish with p so that its contribution is negigibe, and for the achievabiity part, choose γ such that the remainder term P γ vanishes cf. Remar. 4. Combine and Simpify Combine the previous steps as foows: a Construct the set of non-zero entries Bδ, δ, γ R respectivey, B δ, δ in 38 respectivey, 44; b Deduce from 39 respectivey, 45 and Step 3 that P e Pβ s / Bδ, δ, γ + o respectivey, P e Pβ s B δ, δ + o; c From the properties of the typica set T β in Steps, deduce that P e respectivey, P e when n satisfies 37 respectivey, 4 for a b s T β ; d Augment this condition on n with Step 3. in 44 repaced by a fixed pair. Assuming that the remainder terms in 45 are insignificant, this weaer resut is of the form P e max sdif,s eq P n fs dif, s eq, β s rather than P e P n max sdif,s eq fs dif, s eq, β s. This can ead to significanty different bounds on the sampe compexity, and the distinction is crucia in our appications in Section IV. As described in the proof in Section V, the ey to obtaining this difference is in appying a refined version of an argument based on a genie. The genera steps in appying Theorems 3 and 4 to specific probems are outined in Procedure. In our experience, the choice of T β in the first step of Procedure usuay comes naturay given the specific mode. On the other hand, it is often ess straightforward to find a sufficienty powerfu concentration inequaity in Step. A simpe choice is Chebyshev s inequaity, which expresses ψ and ψ in terms of I s dif,s eq b s see and the corresponding variances of the information densities. This choice is usuay effective for the converse, wheres the achievabiity part typicay requires sharper concentration inequaities such as Bernstein s inequaity, due to the combinatoria terms in 39. C. Extensions to Partia Recovery We now turn to the partia support recovery criterion in 7. The changes in the anaysis required to generaize Theorems and to this setting are given in Section V-C; rather than repeating each of these, we focus our attention on the resuting anaogues of Theorems 3 and 4. Theorem 5. For any constants δ >, δ, and γ >, and functions ψ } =d max+ ψ : Z R R, define the set Bδ, δ, γ := b s : 35 and 37 hod for a Then we have s dif, s eq with s dif d max +,..., } }. 46 P e d max P β s / Bδ, δ, γ + =d max+ where P is defined in 5. ψ n, δ + P γ + δ, 47 For the converse part, 4 is repaced by n og p + s dif dmax s dif og p sdif d= d d og δ, I sdif,s eq b s 48 and we have the foowing anaog of Theorem 4. Theorem 6. For any set L d max +,..., }, constants δ > and δ,, and functions ψ } L ψ : Z R R, define the set B δ, δ := b s : 4 and 48 hod for a s dif, s eq with s dif L }. 49 Then we have P e d max P β s B δ, δ max L ψ n, δ δ. 5 The appications of Theorems 5 and 6 foow identica steps to Procedure. However, it wi be seen that the restriction s dif > d max can in fact consideraby simpify these steps, since it removes the need to obtain concentration inequaities for smaer vaues of s dif. D. Comparison to Fano s Inequaity Most previous wors on the information-theoretic imits of sparsity recovery have made use of Fano s inequaity 39, Sec... For this reason, we provide here a discussion on the reative merits of this approach and our approach. To this end, we consider the foowing bound, which can be

9 9 obtained by combining the anaysis of, 4 with our refined genie argument: P e P βs b s max, ni } s dif b s,s eqb sb s + b s og p + s dif b s s dif b s 5 in the notation of Theorem. By anayzing this bound simiary to Section III-B, we obtain for any δ > that P e δ P β s B Fanoδ ogp +, 5 where B Fanoδ := b s : n og p + s dif s dif δ I sdif,s eq b s } for a s dif, s eq with s dif. 53 A simiar resut for partia recovery can aso be derived by incorporating the arguments from 6 and the present paper. As discussed in the introduction, the ey advantage of Theorem 4 is that it provides a more precise characterization of how far the error probabiity is from zero, and in particuar, the conditions under which P e strong impossibiity resuts. On the other hand, the bound on P e in 5 is aways bounded away from one for fixed δ, and becomes increasingy wea for sma δ. The advantage of Fano s inequaity is that it ony requires the mutua information to be computed, whereas our approach aso requires the appication of a concentration inequaity. This, in turn, typicay requires the variance of the information density to be characterized, which is not aways straightforward. However, as discussed foowing Procedure, the main difficuty associated with these concentration inequaities is typicay in finding one which is sufficienty powerfu for the achievabiity part. Thus, the added difficuty in the converse may not add to the overa difficuty in deriving matching achievabiity and converse bounds. IV. APPLICATIONS TO SPECIFIC MODELS In this section, we present appications of Theorems 3 6 to the inear 5, -bit 9, and group testing modes, and to more genera modes with discrete observations 5. Throughout the section, we mae use of genera concentration inequaities given in Appendix A. We aso mae use of the foowing variance quantity: V sdif,s eq b s := Var ıx sdif ; Y X seq, b s β s = b s. 54 A. Linear Mode with Discrete β s Here we consider the inear mode, where each observation taes the form Y = X, β + Z, 55 where Z N, σ for some σ >. Without oss of generaity, we consider the fixed support set s =,..., }. Foowing the setup of 7, we et β s be a uniformy random permutation of a fixed vector b,..., b, and we choose P X N,. Since both the measurement matrices and the noise are Gaussian, the mutua information in is given by 39, Ch. I sdif,s eq b s = og + σ. 56 i s dif b i Throughout this subsection, we denote b min := min i b i and b max := max i b i. We assume that σ = Θ, and that b min = Θb max and < b min = O; note that b min = o is aowed. The steps of Procedure are as foows. Step : We triviay choose the typica set T β to contain a vectors on the support of P βs. Step : We mae use of the foowing concentration inequaity based on Bernstein s inequaity. Proposition. Under the preceding setup for the inear mode, we have for a s dif, s eq and b s that ı P n X ; Y X, b sdif seq s ni b sdif,seq s nδ βs = b s δ n exp 4αs, 57 dif + δα sdif where with σ s dif := i s dif b i. α sdif := σ s dif σ + σ sdif σ + σ s dif 58 Proof: See Appendix B. Setting δ = δ I sdif,s eq b s, it foows that in 35 and 4 we can set ψ n, δ = ψ n, δ = max s dif,s eq,b s : s dif = δ I sdif,s exp eq b s n 4α sdif + δ I sdif,s eq b s α sdif. 59 Step 3: In accordance with the first item of Remar, we set γ as in 6 so that P γ =. We focus on the conditions on n under which the term = ψ n, δ in 39 vanishes; the term containing ψ in 45 can be handed in a simiar yet simper fashion. By the assumptions σ = Θ and b max = Θb min, we readiy obtain I sdif,s eq b s = Θog+b min and α s dif = Θmin, b min } using 56 and 58, where = s dif. Using these growth rates and upper bounding the summation in 39 by times the corresponding maximum, we see that = ψ n, δ provided that the foowing hods for some sufficienty sma constant ζ depending on δ : n og + b min min, b min } + og + b min min, b min }ζ og og 6 for a. We now treat two cases separatey: If b min = o, the first term in 6 behaves as Θnb min ; by rearranging, we concude that it suffices

10 that n and n = Ω og b with a sufficienty arge min impied constant. If b min = Ω, the first term in 6 behaves as Ωn, and it thus suffices that n and n = Ω with a sufficienty arge impied constant. Thus, the overa condition that we require is n and og n = Ω and n = Ω, 6 b min with sufficienty arge impied constants. For the converse, the anaogous condition to 6 contains ony the first term on the eft-hand side the difference being due to the fact that the combinatoria term in 39 is not present in 45, and a simiar argument reveas that it suffices that n = ω b. min Step 4: Combining the preceding steps and appying asymptotic simpifications, we obtain the foowing. Coroary. Under the preceding setup for the inear mode with σ = Θ, b min = Θb max, b min = O, = op, and m β distinct eements in b,..., b, we have P e as p provided that og p s n max dif + η, 6 s dif og + σ i s dif b i under any one of the foowing additiona conditions: i = Θ; ii = oog p and m β = Θ; iii = Oog p θ for some θ >, and m β = ; iv = Θp θ for some θ,, b min = Θ og, and mβ =. Conversey, without any additiona conditions, we have P e as p whenever n max s dif for some η >. og p + s dif s dif og + σ i s dif b i η 63 Proof: The converse part foows from 4 with δ sufficienty sowy. To chec the condition n = ω b min stated foowing 6, we may assume without oss of generaity that 63 hods with equaity, since the decoder can aways choose to ignore additiona measurements. When equaity hods, we observe that for the worst-case s dif with =, the denominator therein behaves as Ob min since b min = O and the numerator behaves as Θog p, and hence, the condition n = ω b is satisfied. min For the achievabiity part, we first use 37 to obtain og p s n max dif + og s dif + og mβ + η, s dif og + σ i s dif b i 64 where the fina term in the numerator arises from 6 since P βs b s is the same for a permutations of b,..., b, and is ower bounded by m β. Observe that the first term in the numerator behaves as Θ s dif og p for each of the cases in the coroary statement, and the second term behaves as Θ og + s dif og s dif. In cases i iii, we have og = oog p, and it immediatey foows that the numerator in 64 is dominated by the first term, and hence, the others can be factored into η in 6. Moreover, in case i, both conditions in 6 are dominated by the objective in 64 with := s dif =, which behaves as Θ og p b. In cases ii iii, the first min condition in 6 is again dominated by the term in 64 with =. The second condition is dominated by the term with =, which behaves as Θ og p og+b = Ω og p min og. In case iv, the first term in the numerator of 64 may not be dominant for sma := s dif, since og = Θog p. However, by observing that the objective scaes as Θ og p og+b and using the assumed scaing of b min min, it is readiy verified that the maximum can ony be achieved with = Θ. For any such maximizer, we have og p = Θ og p, and hence, the second term in the numerator of 64 can be factored into η, as it behaves as O. The two conditions in 6 are identica under the given scaing of b min, and are dominated by the objective in 6 with =, which behaves as Θ og og og. In the case that b min = Θ, the threshods given in Coroary coincide with those given in the main resuts of 7. Our framewor has the advantage of handing the case that b min = o, as we as providing the strong converse P e instead of the wea converse P e. However, it shoud be noted that the achievabiity parts of 7 have the notabe advantage of using a decoder that does not depend on the distribution of β s. On first gance, the bounds in 6 63 may appear to be difficut to evauate, since the maximizations are over non-empty subsets s dif. However, it is in fact ony of them that need to be computed, since for any given = s dif the maximizing s dif is the one with the smaest corresponding vaue of i s dif b i. Comparison to the LASSO: Conditions for the support recovery of the computationay tractabe LASSO agorithm were given by Wainwright 6. Severa comparisons to the information-theoretic imits were given in 5, 6 in terms of scaing aws; here we compement these comparisons by briefy discussing the corresponding constant factors. For simpicity, we focus on the case that the non-zero entries are a equa to a common vaue b = c β for some constant c β representing the per-sampe SNR and is poy-ogarithmic in p, corresponding to case iii of Coroary. The resuts of 6 state that LASSO requires at east og p+o measurements regardess of c β, and that this bound is aso achievabe in the imit as c β. On the other hand, Coroary reveas that for the optima decoder, the coefficient to og p can be arbitrariy sma provided that c β is arge enough. More precisey, appying some simpe manipuations to 6, we find that the coefficient to α og p is sup α,, where α represents the ratio s dif og+c βα. It is easy to verify that the maximum is achieved at α =, yieding the constant og+c β. We concude that the LASSO provaby yieds a suboptima constant when c β >, and fais to achieve the optima ogarithmic decay. However,

11 it shoud be noted that our decoder requires nowedge of and c β, whereas the LASSO does not except possiby via their roe in determining the reguarization parameter. B. Linear Mode with Gaussian β S and Partia Recovery In this subsection, we consider the setup of Section IV-A with two changes: We et the distribution of β s be continuous rather than discrete, and we consider partia recovery instead of exact recovery. More specificay, we et β s be i.i.d. on N, σβ for some variance σ β, and we consider the recovery condition in 7 with d max = α 65 for some α, not varying with p. We again choose P X N,. We assume σβ = c β for some c β > not depending on p, corresponding to a fixed per-sampe SNR. We begin with the foowing auxiiary resut. Proposition. Under the preceding setup for the inear mode, the quantities I and V defined in 7 8 satisfy I og + nσ β σ 66 V n. 67 Proof: See Appendix B. We now proceed with the steps of Procedure with the suitabe changes from exact recovery to partia recovery, cf. Section III-C. Step : Our choice of the typica set T β is based on the foowing proposition characterizing the behavior of the α entries of β s having the smaest magnitude for fixed α. We define the random variabe β s to be the permutation of β s whose entries are isted in increasing order of magnitude. Proposition 3. For any α,, we have im σ β with probabiity one, where gα := α β s i = gα 68 i= α Fχ u + du, 69 and F χ is the cumuative distribution function of a χ random variabe with one degree of freedom. Proof: Letting ˆF be the empirica distribution of the vaues } β σβ i, we have from the Giveno-Cantei i= theorem 4, Thm. 9. that sup u ˆF u F χ u amost surey. This immediatey impies that the sum of the α smaest vaues in } β σβ i, normaized by the number of vaues, converges amost surey to the integra of F χ u from to α. It is easiy verified graphicay that this integra can equivaenty be written as 69. Based on this resut and its proof, we set T β to be the set of vectors b s such that sup u ˆF u F χ u ɛ, where ɛ is chosen to decay sufficienty sowy so that Pβ s T β. Thus, within the typica set, the empirica distribution of the non-zero entries cosey foows a χ random variabe. An important consequence of this choice of typica set regards the behavior of the mutua information in 56. For a fixed set size s dif, the partition s dif, s eq minimizing this mutua information is the one with the smaest vaue of i s dif b i. Within the typica set, we immediatey obtain from Proposition 3 that the corresponding mutua information behaves as foows when s dif = α : I sdif,s eq b s og + c β σ gα, 7 where we reca that c β = σβ is a constant. Step : We again mae use of Proposition and its subsequent expression for ψ andψ in 59. V Step 3: We choose γ = I + δ as in 9 for some δ >, thus ensuring that P γ δ. For the terms in Theorems 5 6 containing ψ and ψ, we first note that since we are considering partia recovery, we may focus on vaues of = s dif greater than α. By our choice of T β, we may aso focus on reaizations b s of β s satisfying 68. For such reaizations, we have for a s dif with s dif = = Θ that i s dif b i = Ω, which impies that αs dif = Θ in 58 and I sdif,s eq b s = Ω in 56. The anaogous condition to 6 thus simpifies to ni for some I = Ω, giving the foowing condition under which the second term in 39 vanishes: n = Ω, 7 with a sufficienty arge impied constant. For the converse part, it suffices to have the weaer condition n = ω. Step 4: Combining the above steps, we get the foowing. Coroary. Under the preceding setup for the inear mode with, = op, σβ = c β for some c β >, and d max = α for some α,, we have P e d max as p provided that n max α α, α og p og + c β σ gα + η 7 for some η >, where g is defined in 69. Conversey, P e d max as p whenever n for some η >. max α α, α α og p η 73 og + c β σ gα Proof: The condition in 7 is obtained using 37 and 7. By the assumption = op, the numerator in 7 coincides with og p α up to remainder terms in Stiring s approximation that can be factored into η. The factor og δ s dif in 37 has been factored into η; this is vaid when δ sufficienty sowy due to the fact that og s dif = O, whereas again using the assumption = op the numerator in 7 behaves as ω. We caim that the factor γ = I + V δ resuting

12 from 9 can aso be factored into η for some vanishing sequence of parameters δ indexed by p. To see this, we consider without oss of generaity the worst-case setting in which 7 hods with equaity. We readiy obtain n = Θ og p, which in turn impies from Proposition that I = O og + σβ og p = O og og p and V = O og p. Thus, I V + δ is dominated by the numerator of 7 if δ is chosen to decay as for exampe Θ og. The fact that n = Θ og p aso impies 7. The converse bound in 7 is obtained simiary using 37, except for the term α in the numerator. To see how this arises, we consider an arbitrary vaue of α α, and set = α ; the case α = α foows by continuity. The term og p + is handed in the same way as the term og p above, so we focus on the term og d max p d= d d. This is upper bounded by max d=,...,dmax og + d max p d d. Simiary to the achievabiity part, we can factor og + d max og d into η, so we are eft with og p d max. Approximating this using Stiring s approximation as before, and recaing that d max = α, we obtain the desired term α og p. Whie the achievabiity and converse bounds in Coroary do not have the same constants, the two are simiar, and aways have the same scaing aws. In the imit as c β, we have og + c β σ gα = og c β + o; in this case, the maxima in 7 73 are both achieved with α, and hence, the two bounds coincide to within a mutipicative factor of α. Coroary is reated to the setting studied by Reeves and Gastpar 5, 6, but considers = op instead of = Θp. Despite this difference, it is instructive to compare the bounds upon etting the impied constant in the Θp scaing tend to zero. A carefu comparison reveas that the converse bounds coincide in this imit, whereas our achievabiity bound is sighty better, in that the anaogous bound in 5 mutipies c β σ gα by.7; see 5, Eq. and 6, Eq. 5. In Section IV-E, we present some numerica resuts for this setting. C. -bit Mode with Discrete β S We now turn to the quantized counterpart of 55: Y = sign X S, β S + Z. 74 As in Section IV-A, we fix s =,..., } and et β s be a uniformy random permutation of a fixed vector b,..., b, and we set P X N,. We again write the minimum and maximum absoute vaues of b i } i= as b min and b max. The foowing proposition gives the required characterizations on the mutua information terms and the corresponding variance terms. Reca the binary entropy function H and the Q-function Q defined in Section I-C. Proposition 4. Under the preceding setup for the -bit mode, we have the foowing: i The mutua information I sdif,s eq b s is given by I sdif,s eq b s = E H Q W H Q W i s eq b i σ + i s dif b i σ i s b i, 75 where W N,. ii If = Θ, σ = Θ, b min = Θb max, and b min = o, then I sdif,s eq b s = + o. 76 πσ i s dif b i iii If = Θp, σ = Θ, and the entries of b s a equa a common vaue b such that b = Θ og p p, then the mutua information quantities I sdif,s eq b s with s dif = a equa a common vaue I satisfying I = b σ π b og p = Θ p E σ W og QW QW + o 77, 78 where W N,. iv The variance V sdif,s eq b s defined in 54 satisfies V sdif,s eq b s c σ + min, σ i s dif b i + for some universa constant c. i s dif b i σ b i i s dif } σ i s eq b i 79 Proof: See Appendix C. Beow we present two coroaries corresponding to different scaings of and the SNR, namey, those given in parts ii and iii of Proposition 4. We proceed by simutaneousy presenting the steps of Procedure for both settings. Step : As in Section IV-A, we choose the trivia typica set T β containing a vectors on the support of P βs. Step : We mae use of Chebyshev s inequaity in Proposition 9 in Appendix A. Choosing δ = δ I sdif,s eq b s in 43, it foows that we may set ψ n, δ = ψ n, δ V sdif,s = max eq b s s dif,s eq,b s : s dif = nδ I s dif,s eq b s. 8 Step 3: We again choose γ as in 6 so that P γ =. Consider the setting described in part ii of Proposition 4. Under the scaings therein, 76 and 79 both behave as. Hence, and using 8 and the fact that = Θ, Θb min the second term in 39 vanishes provided that n = ω b min. 8