Analyzing the Structure of Multidimensional Compressed Sensing Problems through Coherence

Transcription

1 Analyzing the Structure of Multiimensional Compresse Sensing Problems through Coherence A. Jones University of Cambrige B. Acock Purue Univesity A. Hansen University of Cambrige Abstract In previous work it was establishe that asymptotic incoherence can be use to facilitate subsampling in a variety of continuous compresse sensing problems an the coherence structure of certain oneimensional Fourier sampling problems was etermine. This paper extens the analysis of asymptotic incoherence to cover multiimensional reconstruction problems. It is shown that Fourier sampling an separable wavelet sparsity in any imension can yiel the same optimal asymptotic incoherence as in one imensional case. Moreover in two imensions the coherence structure is compatible with many stanar two imensional sampling schemes that are currently in use. However, in higher imensional problems with poor wavelet smoothness we emonstrate that there are consierable restrictions on how one can subsample from the Fourier basis with optimal incoherence. This can be remeie by using a sufficiently smooth generating wavelet. It is also shown that using tensor bases will always provie suboptimal ecay marre by problems associate with imensionality. The impact of asymptotic coherence on the ability to subsample is emonstrate with some simple two imensional numerical experiments. 1 Introuction Exploiting aitional structure has always been central to the success of compresse sensing, ever since it was introuce by Canès, Romberg & Tao [8] an Donoho [12]. Sparsity an incoherence has allowe us to solve convex optimisation problems with uniform ranom subsampling. Recently [2] the more general notions of sparsity in levels & asymptotic incoherence were introuce to solve a greater variety of inverse problems using subsampling in levels. There is a wie variety of problems that lack incoherence an instea posses asymptotic incoherence; Magnetic Resonance Imaging (MRI) [14,25], X-ray Compute Tomography [9,27] an Electron Microscopy [22,23] to name a few. This phenomena often originates from the inverse problems being base upon integral transforms, for example, reconstructing a function f from pointwise evaluations of its Fourier transform. In compresse sensing, such a transform is combine with an appropriate sparsifying transformation associate to a basis or frame, giving rise to an infinite measurement matrix U. The coherence 1 of U C N N or U C N N is efine by µ(u) = max i,j N U ij 2, µ(u ) = max i,j=1,...,n U ij 2. Asymptotic incoherence is the phenomena of when µ(p N U), µ(up N ) 0, N, (1.1) where PN enotes the projection onto the inices N + 1, N + 2,... As a general rule, the faster asymptotic incoherence ecays the more we are able to subsample (see (1.5)). The stuy of more precise notions of coherence has also been consiere for the one an two imensional iscrete Fourier sampling, separable Haar sparsity problems in [20]. This paper focuses on stuying the structure of (1.1) in continuous multiimensional inverse problems an the impact this has on the ability to effectively subsample. In previous work [18], the structure of incoherence was analyze as a general problem an theoretical limits on how fast it can ecay over all such inverse problems were establishe. Furthermore, the notion of optimal ecay was introuce, which escribes the fastest asymptotic incoherence ecay possible for a 1 Incoherence refers to small coherence. 1

2 given inverse problem. The notion of an optimal orering was also introuce, which acte as a set of instructions on how to actually attain this optimal incoherence ecay rate by orering the sampling basis. Optimal ecay rates an optimal orerings were etermine for the one-imensional Fourier-wavelet an Fourier-polynomial cases an the former was foun to attain the theoretically optimal 2 incoherence ecay rate of N 1. Figure 1: Fourier - Separable Haar Cases: Incoherence Structures in Different Dimensions Wavelet Basis Column Coherence Fourier Frequency (a) Coherence Matrix for the 1D case Fourier Frequency (b) 1D Column Coherences (c) 2D analogue of (b) () Isosurface of 3D Case In (b), the coherences are calculate by taking the maxima over the columns in (a), emonstrating ecay that scales with frequency. In 2D this ecay roughly matches that of the norm of the frequency as seen in (c). However in 3D there are hyperbolic spikes aroun the coorinate axes that lea to poor incoherence ecay (see ()) when using sampling patterns with rotational invariance or linear scaling. In the black an white plots, white inicates larger absolute value. The optimal orerings in these one imensional cases matche the levele schemes that was alreay use for subsampling. For example, when sampling from the 1D Fourier basis, the sampling levels are usually orere accoring to increasing frequency. In multiple imensions there is no such consensus, instea many ifferent sampling patterns are use, especially when it comes to 2D sampling patterns where raial lines [10], spirals [17] or other k-space trajectories are use. There are also a variety of other sampling techniques use in even higher imensional (3-10D) problems, such as in the fiel of NMR spectroscopy [5]. If one esires to exploit asymptotic incoherence to its fullest it must be unerstoo whether the optimal orering is consistent with the sampling pattern that one intens to use. This paper etermines optimal orerings for the case of Fourier sampling an (separable) wavelet sparsity in any imension. It is shown that the optimal ecay is always that of the one-imensional case, an moreover in two imensions the optimal orerings are compatible with the structure of the 2D sampling patterns mentione above. However, in higher imensions problems with poor wavelet smoothness, such as the three imensional separable Haar case, the class of optimal orerings 3 are no longer rotationally invariant (as in Figure 1), hinering the ability to subsample with traitional sampling schemes. It is also shown that using a pair of tensor bases in general leas to optimal incoherence ecay that is always anisotropic an suboptimal. We shoul mention here that for many inverse problems in higher imensions, using separable wavelets as a reconstruction basis fairs poorly against other bases such as shearlets [21] an curvelets [6] for approximating images with curve-like features. However, it is not our goal to focus on a particular reconstruction basis in this paper, instea we wish to emonstrate how the incoherence structure can vary for ifferent bases an the impact this has on its application in compresse sensing problems, for goo or for worse. 1.1 Setup & Key Concepts : Incoherence, Sparsity & Orerings Throughout this paper we shall work in an infinite imensional separable Hilbert space H, typically H = L 2 ([ 1, 1] ), with two close infinite imensional subspaces V 1, V 2 spanne by orthonormal bases B 1, B 2 respectively, V 1 = Span{f B 1 }, V 2 = Span{f B 2 }. We call (B 1, B 2 ) a basis pair. If we are to form the change of basis U = (U i,j ) i,j N we must list the two bases, which leas to following efinitions: Definition 1.1 (Orering). Let S be a set. Say that a function ρ : N S is an orering of S if it is bijective. 2 Optimal in the sense of over all inverse problems that has U an isometry. Furthermore, it is the fastest ecay as a power of N. 3 Technically we mean strongly optimal here (see Definition 1.8). 2

3 Definition 1.2 (Change of Basis Matrix). For a basis pair (B 1, B 2 ), with corresponing orerings ρ : N B 1 an τ : N B 2, form a matrix U by the equation U m,n := τ(n), ρ(m). (1.2) Whenever a matrix U is forme in this way we write U := [(B 1, ρ), (B 2, τ)]. Stanar compresse sensing theory says that if x C N is s-sparse, i.e. x has at most s nonzero components, then, with probability exceeing 1 ɛ, x is the unique minimiser to the problem min η l 1 subject to P Ω Uη = P Ω Ux, η C N where P Ω is the projection onto span{e j : j Ω}, {e j } is the canonical basis, Ω is chosen uniformly at ranom with Ω = m an m C µ(u) N s log(ɛ 1 ) log(n), (1.3) for some universal constant C > 0 (see [7] an [1]). In [2] a new theory of compresse sensing was introuce base on the following three key concepts: Definition 1.3 (Sparsity in Levels). Let x be an element of either C N or l 2 (N). For r N let M = (M 1,..., M r ) N r with 1 M 1 <... < M r an s = (s 1,..., s r ) N r, with s k M k M k 1, k = 1,..., r, where M 0 = 0. We say that x is (s, M)-sparse if, for each k = 1,..., r, k := supp(x) {M k 1 + 1,..., M k }, satisfies k s k. We enote the set of (s, M)-sparse vectors by Σ s,m. Definition 1.4 (Multi-level sampling scheme). Let r N, N = (N 1,..., N r ) N r with 1 N 1 <... < N r, m = (m 1,..., m r ) N r, with m k N k N k 1, k = 1,..., r, an suppose that Ω k {N k 1 + 1,..., N k }, Ω k = m k, k = 1,..., r, are chosen uniformly at ranom, where N 0 = 0. We refer to the set as an (N, m)-multilevel sampling scheme. Ω = Ω N,m := Ω 1... Ω r Definition 1.5 (Local coherence). Let U be an isometry of either C N or l 2 (N). If N = (N 1,..., N r ) N r an M = (M 1,..., M r ) N r with 1 N 1 <... N r an 1 M 1 <... < M r the (k, l) th local coherence of U with respect to N an M is given by µ N,M (k, l) = where N 0 = M 0 = 0 an P a b µ(p N k 1 N k UP M l 1 M l ) µ(p N k 1 N k U), k, l = 1,..., r, (1.4) enotes the projection matrix corresponing to inices {a + 1,..., b}. [2] provie the following estimate 4 regaring the local number of measurements m k in the k th level in orer to obtain a goo reconstruction with probability 1 ɛ : ( r ) m k C log(ɛ 1 ) µ N,M (k, l) s l log (N), k = 1,..., r. (1.5) N k N k 1 l=1 In particular, the sampling strategy (i.e. the parameters N an m) is now etermine through the local sparsities an incoherences. Since the local coherence (1.4) is rather ifficult to analyze in its current form, we boun it above by the following: µ N,M (k, l) = µ(p N k 1 N k UP M l 1 M l ) µ(p N k 1 N k U) (1.6) min(µ(p N k 1 N k U), µ(up M l 1 M l )) µ(p N k 1 N k U) (1.7) min(µ(pn k U), µ(upm l )) µ(pn k U) (1.8) 4 C > 0 is a universal constant 3

4 It is arguably (1.8) rather than (1.7) that is the roughest boun here, however we shall see that this becomes effectively an equality in what follows. The crucial improvement of (1.8) over (1.6) is that it is completely in terms of the asymptotic incoherences µ(pn U), µ(up N ), which epen only on the orerings of B 1, B 2 respectively, rather than both of them. Furthermore, we can treat the two problems of maximizing the ecay of µ(pn U), µ(up N ) separately an then combine the two resulting orerings together at the en. Next we escribe how one etermines the fastest ecay of µ(pn U). In [18] this was one via the notion of optimality up to constants: Definition 1.6 (Optimal Orerings). Let ρ 1, ρ 2 : N B 1 be any two orerings of a basis B 1 an τ any orering of a basis B 2. Let U 1 := [(B 1, ρ 1 ), (B 2, τ)], U 2 := [(B 1, ρ 2 ), (B 2, τ)] as in (1.2). Also let Q N := PN 1. If there is a constant C > 0 such that µ(q N U 1 ) C µ(q N U 2 ), N N, then we write ρ 1 ρ 2 an say that ρ 1 has a faster ecay rate than ρ 2 for the basis pair (B 1, B 2 ). ρ 1 is sai to be an optimal orering of (B 1, B 2 ) if ρ 1 ρ 2 for all other orerings ρ 2 of B 1. It was shown in [18] that optimal orerings always exist. Optimal orerings are use to give us the optimal ecay rate: Definition 1.7 (Optimal Decay Rate). Let f, g : N R >0 be ecreasing functions. We write f g to mean there is a constant C > 0 such that f(n) C g(n), N N. If both f g an g f hols, we write f g. Now suppose that ρ : N B 1 is an optimal orering for the basis pair (B 1, B 2 ) an we let U = [(B 1, ρ), (B 2, τ)] be a corresponing incoherence matrix (with some orering τ of B 2 ). Then any ecreasing function f : N R >0 which satisfies f g, where g is efine by g(n) = µ(q N U), N N, is sai to represent the optimal ecay rate of the basis pair (B 1, B 2 ). Notice that the optimal ecay rate is unique up to the equivalence relation efine on the set of ecreasing functions f : N R >0. We also have a stronger notion of optimality, which gives us finer etails on the exact ecay: Definition 1.8 (Strong Optimality). Let U = [(B 1, ρ), (B 2, τ)] an π N enote the projection onto the single inex N. If f represents the optimal ecay rate of the basis pair (B 1, B 2 ) then ρ is sai to be strongly optimal if the function g(n) := µ(π N U) satisfies f g. Estimates in terms of the row incoherence µ(π N U) have use before in [20], where it was calle the local coherence. If ρ is a strongly optimal orering, U = [(B 1, ρ), (B 2, τ)] an f represents the optimal ecay of (B 1, B 2 ) then µ(q N U) C 1 f(n) C 2 µ(π N U) C 2 µ(p N 1+M N 1 U), N, M N, for some constants C 1 (ρ), C 2 (ρ) > 0, which can then be use to justify (1.8). We shall introuce the Fourier basis here as it is use in all of the examples iscusse in this paper: Definition 1.9 (Fourier Basis). If we efine χ k (x) = ɛ exp(2πiɛkx) 1 [( 2ɛ) 1,(2ɛ) 1 ](x), k Z, then the (χ k ) k Z form a basis 5 B f (ɛ) of L 2 ([ (2ɛ) 1, (2ɛ) 1 ]). We can form a -imensional basis of L 2 ([ (2ɛ) 1, (2ɛ) 1 ] ) by taking tensor proucts (see Section 4) χ k := χ kj, k Z, an setting B f (ɛ) = {χ k : k Z }. It shall be convenient to ientify B f (ɛ) with Z using the function 5 The little f here stans for Fourier. j=1 λ : B f Z, λ (χ k ) := (λ(χ k1 ),..., λ(χ k )) = (k 1,..., k ) = k. (1.9) 4

5 1.2 Main Results The primary focus of the paper is to stuy the Fourier-Separable wavelet case where we use any Daubechies wavelet 6. The efinition of a separable wavelet basis B sep is provie in Section 5. The main results on these cases are summarize below: Theorem The optimal ecay rate of both (Bf, B sep) an (Bsep, Bf ) is represente by f(n) = N 1. The optimal ecay rate of (Bsep, Bf ) is obtaine by using an orering τ that is consistent with the wavelet levels an this orering is strongly optimal. In 2D the optimal ecay rate of (Bf, B sep) is obtaine by using an orering ρ that satisfies, for some constants C 1, C 2 > 0 an some norm on R, C 1 N 1/ max( λ (ρ(n)), 1) C 2 N 1/, N N. (1.10) In fact ρ is strongly optimal if an only if (1.10) hols. An orering satisfying (1.10) is calle a linear orering. The class of linear orerings are rotation invariant an compatible with sampling schemes base on linearly scaling a fixe shape from the origin (see Section 5.3). In higher imensions linear orerings can fail to be optimal, however with a sufficiently smooth wavelet an orering is strongly optimal if an only if (1.10) hols. This theorem is a simplifie version of Theorem Optimal orerings in the case of high imensions an poor wavelet smoothness can be foun by interpolating between the case of (1.10) an the hyperbolic cross, which generates semi-hyperbolic orerings (see Definition 5.14). It is also shown that if a linear orering is optimal then the wavelet use must have some egree of smoothness proportional to the imension; in 3D it is C 0, 5D it is C 1, 7D it is C 2, etc. (see Section 5.6). We also consier the general case of a pair of tensor bases where we escribe a metho of fining optimal orerings an ecay rates for all multiimensional cases, given that the incoherence structure of the one-imensional case is known. In particular, we have the following result for the Fourier-Tensor wavelet case: Theorem Let B w be a tensor wavelet basis. The optimal ecay rate of both (B f, B w) an (B w, B f ) is represente by f(n) = log 1 (N) N 1. Optimal orerings for both cases are constructe using the hyperbolic cross on the original one-imensional optimal orerings. This theorem is a restatement of Theorem This emonstrates the typical curse of imensionality which is lacking in the separable case. The ifferences between the two incoherence structures of the Fourier- Tensor wavelet an Fourier-Separable wavelet cases are teste in 2D in Section Outline for the Remainer Some key tools that we use to fin optimal orerings are given in Section 2. Those familiar with [18] can skip the majority of this section except for the concept of characterization. We then cover the general tensor case an introuce hyperbolic orerings in Section 4. In Section 5 we iscuss the separable cases, first covering how to optimally orer the wavelet basis before quickly moving on to the central problem of fining optimal orerings of the Fourier basis. Linear orerings are introuce first, then we justify the nee for semihyperbolic orerings. Finally we move onto some simple compresse sensing experiments, one emonstrating the benefits of multilevel subsampling an one showing the impact of iffering incoherence structures between the 2D tensor an separable cases. 2 Tools for Fining Optimal Orerings & Theoretical Limits on Optimal Decay The first tool is perhaps the most important, as it is a very easy way to ientify a strongly optimal orering: 6 We use the convention that Daubechies wavelets inclues the Haar wavelet. 5

6 Lemma ): Let (B 1, B 2 ) be a basis pair an τ any orering of B 2. Furthermore, let B 1 have an orering ρ 1 : N B 1, an efine U 1 := [(B 1, ρ 1 ), (B 2, τ)]. Suppose that that there is a ecreasing function f 1 : N R >0 such that f 1 (N) µ(π N U 1 ), N N. Then if ρ 2 : N B 1 is an orering, U 2 = [(B 1, ρ 2 ), (B 2, τ)] an f 2 : N R >0 is a function with µ(q N U 2 ) f 2 (N), N N, then f 1 (N) f 2 (N) for every N N. 2): Let ρ be an orering of B 1 with U := [(B 1, ρ), (B 2, τ)] an f : N R 0 be a ecreasing function with f(n) 0 as N. If, for some constants C 1, C 2 > 0, we have C 1 f(n) µ(π N U) C 2 f(n), N N, (2.1) then ρ is a strongly optimal orering an f is a representative of the optimal ecay rate. Proof. See Lemma 2.11 in [18]. Definition 2.2 (Best orering). Let (B 1, B 2 ) be a basis pair. Then any orering ρ : N B 1 is sai to be a best orering if for any other orering τ of B 2 an U = [(B 1, ρ), (B 2, τ)] we have that the function g(n) := µ(π N U) is ecreasing. Notice that any best orering is also a strongly optimal orering. We shall nee the notion of a best briefly to prove Lemma 2.6. Lemma 2.3. Suppose that we have a basis pair (B 1, B 2 ) with two orerings ρ : N B 1, τ : N B 2 of B 1, B 2 respectively. If U = [(B 1, ρ), (B 2, τ)] satisfies then a best orering exists. Proof. See Lemma 2.10 in [18]. µ(π N U) 0 as N, Throughout this paper we woul like to efine an orering accoring to a particular property of the basis but this property may not be enough to specify a unique orering. To eal with this issue we introuce the notion of consistency: Definition 2.4 (Consistent orering). Let F : S R where S is a set. We say that an orering ρ : N S is consistent with F if F (f) < F (g) ρ 1 (f) < ρ 1 (g), f, g S. The notion of consistency becomes important if we want to convert bouns on the coherence into optimal orerings: Definition ) Suppose F : S R satisfies #{x S : F (x) K} < for all K > 0, σ : N S is consistent with 1/F an F (σ(n)) 0 as N. Then any ecreasing function f : N R >0 such that f F σ is sai to represent the fastest ecay of F. 2.) Suppose (B 1, B 2 ) is a basis pair an ι : S B 1 a bijection. If there exists a function F : S R an a constant C 1 > 0 such that sup ι(s), g 2 C 1 F (s), s S, (2.2) g B 2 then F is sai to ominate the optimal ecay of (B 1, B 2 ). If the inequality is reverse we say F is ominate by the optimal ecay of (B 1, B 2 ). Furthermore, if there is a constant C 2 > 0 such that C 2 F (s) sup g B 2 ι(s), g 2 C 1 F (s), s S, (2.3) then F is sai to characterize the optimal ecay of (B 1, B 2 ). 6

7 Lemma ): Suppose f is a representative of the optimal ecay rate for the basis pair (B 1, B 2 ), ι : S B 1 is a bijection, F : S R ominates the optimal ecay of (B 1, B 2 ), σ : N S is consistent with 1/F an U = [(B 1, ι σ), (B 2, τ)]. Then if g represents the fastest ecay of F then f, µ(π U) g. 2): If F is instea is ominate by the optimal ecay of (B 1, B 2 ) then f, µ(π U) g. 3): If F now characterizes the optimal ecay of (B 1, B 2 ) then f, µ(π U) g an therefore ρ is a strongly optimal orering for the basis pair (B 1, B 2 ) if an only if F (ι 1 ρ( )) g. Proof. 1.) We may assume, without loss of generality, that g(n) 0 as N else there is nothing to prove as f(n), µ(π N U) are boune functions of N. Therefore, a best orering exists by Lemma 2.3. (2.2) becomes (for C 1 > 0 a constant), µ(π N U) C 1 F (σ(n)) C 1 g(n), N N. Since g is ecreasing we have µ(q N U) C 1 g(n) an therefore we can apply part 1) of Lemma 2.1 to f 1 = f an f 2 = g (using a best orering as ρ 1 an ρ 2 = ι σ) to euce that f g. 2.) (2.2) reverse becomes µ(π N U) C F (σ(n)) C 1 g(n), N N. Therefore we can apply part 1) of Lemma 2.1 to f 1 = g an f 2 = f (using ρ 1 = ι σ, ρ 2 an optimal orering) to euce that f g. 3.) Notice that if F characterizes the optimal ecay of (B 1, B 2 ) then (2.3) becomes C 2 g(n) C 2 F (σ(n)) µ(π N U) C 1 F (σ(n)) C 1 g(n), N N, an we can then apply part 2) of Lemma 2.1 to show f, µ(π U) g. If we let U := [(B 1, ρ), (B 2, τ)] then (2.3) becomes C 2 F (ι 1 ρ(n)) µ(π N U ) C 1 F (ι 1 ρ(n)), N N, an the result follows from Definition 1.8. Before moving on, we recall from [18] some results on the fastest optimal ecay rate for a basis pair: Theorem 2.7. Let U B(l 2 (N)) be an isometry. Then N µ(q NU) iverges. Proof. See Theorem 2.14 in [18]. Corollary 2.8. Let U B(l 2 (N)) be any isometry. Then there oes not exist an ɛ > 0 such that µ(q N U) = O(N 1 ɛ ), N. It turns out that Theorem 2.7 cannot be improve without imposing aitional conitions on U: Lemma 2.9. Let f, g : N R be any two strictly positive ecreasing functions an suppose that N f(n) iverges. Then there exists U B(l 2 (N)) an isometry with Proof. See Lemma 2.16 in [18]. µ(q N U) f(n), µ(uq N ) g(n), N N. (2.4) If we restrict our ecay function to be a power law, i.e. f(n) := CN α for some constants α, C > 0 then the largest possible value of α > 0 such that (2.4) hols for an isometry U is α = 1. This gives us a notion of the fastest optimal ecay rate as a power of N over all pairs of bases where the span of B 2 lies in the span of B 1. 3 One-imensional Bases an Incoherence Results Before we begin our review of the one-imensional cases by quickly going the one-imensional bases an orerings that we shall be working with to construct multi-imensional bases an orerings in Section 4. 7

8 3.1 Fourier Basis We recall the one-imensional Fourier basis B f (ɛ) = (χ k ) k Z from Definition 1.9. Definition 3.1 (Stanar orering). We efine F f : B f N {0} by F f (χ k ) = k an say that an orering ρ : N B f is a stanar orering if it is consistent with F f (recall Definition 2.4). 3.2 Stanar Wavelet Basis Take a Daubechies wavelet ψ an corresponing scaling function φ in L 2 (R) with We write Supp(φ) = Supp(ψ) = [ p + 1, p]. φ j,k (x) = 2 j/2 φ(2 j x k), V j := Span{φ j,k : k Z}, ψ j,k (x) = 2 j/2 ψ(2 j x k), W j := Span{ψ j,k : k Z}. With the above notation, (V j ) j Z is the multiresolution analysis for φ, an therefore V j V j+1, V j+1 = V j W j, L 2 (R) = j Z V j, where W j here is the orthogonal complement of V j in V j+1. For a fixe J N we efine the set 7 B w := Supp(φ J,k ) ( 1, 1), φ J,k, ψ j,k : Supp(ψ j,k ) ( 1, 1), j N, j J, k Z, (3.1) Let ρ be an orering of B w. Notice that since L 2 (R) = V J j=j W j for all f L 2 (R) with supp(f) [ 1, 1] we have f = c n ρ(n) for some (c n ) n N l 2 (N). n=1 Definition 3.2 (Levele orering (stanar wavelets)). Define F w : B w R by { j, if f W j F w (f) =, 1, if f V J an say that any orering τ : N B w is a levele orering if it is consistent with F w. Notice that F w (ψ j,k ) = j. We use the name levele here since requiring an orering to be levele means that you can orer however you like within the iniviual wavelet levels themselves, as long as you correctly orer the sequence of wavelet levels accoring to scale. Suppose that U = [(B f (ɛ), ρ), (B w, τ)] for orerings ρ, τ. If we require U to be an isometry we must impose the constraint (2ɛ) J+1 (p 1) otherwise the elements in B w o not lie in the span of B f (ɛ). For convenience we rewrite this as ɛ I J,p where 3.3 Bounary Wavelet Basis I J,p := (0, (2 + 2 J+2 (p 1)) 1 ]. We now look at an alternative way of ecomposing a function f L 2 ([ 1, 1]) in terms of a wavelet basis, which involves using bounary wavelets [26, Section 7.5.3]. The basis functions all have support containe within [ 1, 1], while still spanning L 2 [ 1, 1]. Furthermore, the bounary wavelet basis retains the ability to reconstruct polynomials of orer up to p 1 from the corresponing stanar wavelet basis. We shall not go into great etail here but we will outline the construction; we take, along with a Daubechies wavelet ψ an 7 w here stans for wavelet. 8

9 corresponing scaling function φ with Supp(ψ) = Supp(φ) = [ p + 1, p], bounary scaling functions an wavelets (using the same notation as in [26] 8 ) φ left n, φ right n, ψn left, ψn right, n = 0,, p 1. Like in the stanar wavelet case we shift an scale these functions, φ left j,n(x) = 2 j/2 φ left n (2 j (x + 1)), φ right j,n (x) = 2j/2 φ right n (2 j (x 1)). We are then able to construct neste spaces, (Vj int such that L 2 ([ 1, 1]) =, Vj+1 int = V j int V int j+1 by efining V int j W int j j=j V j int = Span = Span ) j J, (W int) j J for a fixe base level J log 2 (p), W int j j with W int j { φ left j,n, φ right j,n : n = 0,, p 1 }, φ j,k k Z s.t. Supp(φ j,k ) ( 1, 1) { ψ left j,n, ψ right j,n : n = 0,, p 1 }. ψ j,k k Z s.t. Supp(ψ j,k ) ( 1, 1) the orthogonal complement of V int j We then take the spanning elements of VJ int int an the spanning elements of Wj for every j J to form the basis B bw (bw for bounary wavelets ). Definition 3.3 (Levele orering (bounary wavelets)). Define F w : B bw R by the formula { j, if f Wj int F bw (f) = 1, if f VJ int. Then we say that an orering τ : N B bw of this basis is a levele orering if it is consistent with F bw. 3.4 Legenre Polynomial Basis If (p n ) n N enotes the stanar Legenre polynomials on [ 1, 1] (so p n (1) = 1 an p 1 (x) = 1 for x [ 1, 1]) then the L 2 -normalise Legenre polynomials are efine by p n = n 1/2 p n an we write B p := ( p n ) n=1 (the p here stans for polynomial ). B p is alreay orere; call this the natural orering. 3.5 Incoherence Results for One-imensional Bases Next we recall the one-imensional incoherence results prove in [18], which shall be use to prove the corresponing multi-imensional tensor results in Section 4: Theorem 3.4. Let ρ be a stanar orering of B f (ɛ) with ɛ I J,p, τ a levele orering of B w an U = [(B f (ɛ), ρ), (B w, τ)]. Then we have, for some constants C 1, C 2 > 0 the ecay in C 1 N µ(π N U), µ(uπ N ) C 2, N N, (3.2) N The same conclusions also hol if the basis B w is replace by B bw an the conition ɛ I J,p by ɛ (0, 1/2]. Theorem 3.5. Let ρ be a stanar orering of B f (ɛ) with ɛ (0, 0.45], τ a natural orering of B p an U = [(B f (ɛ), ρ), (B p, τ)]. Then we have, for some constants C 1, C 2 > 0 the ecay C 1 N µ(π 2/3 N U), µ(uπ N ), C 2, N N. (3.3) N 2/3 8 We use [ 1, 1] instea of [0, 1] as our reconstruction interval here, but everything else is the same. 9

10 4 Multiimensional Tensor Cases 4.1 General Estimates Definition 4.1 (Tensor basis). Suppose that B is an orthonormal basis of some space T L 2 (R) (i.e. T is a subspace L 2 (R)) an we alreay have an orering ρ : N B. Define ρ : N j=1 T L2 (R ) by the formula (m N ) ( ) ρ (m)(x) := ρ(m j ) (x) = ρ(m j )(x j ). j=1 This gives a basis of j=1 T L2 (R ) because of the formula ρ (m), ρ (n) L2 (R ) = j=1 ρ(m j ), ρ(n j ) L 2 (R). (4.1) We call B := (ρ (m)) m N a tensor basis. The function ρ is sai to be the -imensional inexing inuce by ρ. Notice that ρ is not an orering unless = 1. Now suppose that we have two one-imensional bases B 1, B 2 with corresponing optimal orerings ρ 1, ρ 2. Let ρ 1, ρ 2 be the -imensional inexings inuce by ρ 1, ρ 2 of the bases B 1, B 2. What are optimal orerings of the basis pair (B 1, B 2) an what is the resulting optimal ecay rate? Some insight is given by the following Lemma: Lemma 4.2. Let (B 1, B 2 ) be a pair of bases with corresponing tensor bases B 1, B 2. Let ρ 1 be a strongly optimal orering of B 1 an ρ 1 be the -imensional inexing inuce by ρ 1. Finally, for some orering τ of B 2, let U = [(B 1, ρ 1 ), (B 2, τ)]. Then if f represents the optimal ecay rate corresponing to the basis pair (B 1, B 2 ) we have, for some constants C 1, C 2 > 0, C1 f(n i ) sup ρ 1(n), g 2 = g B2 j=1 µ(π ni U) C2 f(n i ), n N. (4.2) Consequently, if we let ι := ρ 1 then F (n) := f(n i) characterizes the optimal ecay of (B 1, B 2 ). Proof. Let τ enote the -imensional inexing inuce by τ. Then by breaking the own the tensor prouct into terms an using the bijectivity of τ we have sup ρ 1(n), g 2 = sup ρ 1(n), τ (m) 2 = sup g B2 m N m N = sup ρ 1 (n i ), τ(m) 2 = m N ρ 1 (n i ), τ(m i ) 2 µ(π ni U). Therefore (4.2) follows from applying the efinition of a strongly optimal orering to each term in the prouct. Lemma 4.2 says that if we have a strongly optimal orering for the basis pair (B 1, B 2 ) then we can use Lemma 2.6 to fin all strongly optimal orerings for the corresponing tensor basis pair (B 1, B 2). In particular, we have Corollary 4.3. We use the framework of the previous Lemma. Let σ : N N be consistent with 1/F. Then an orering ρ is strongly optimal for the basis pair (B 1, B 2) if an only if there are constants C 1, C 2 > 0 such that C 1 F (σ(n)) F ((ρ 1) 1 ρ(n)) C 2 F (σ(n)), N N. Suppose that we have a strongly optimal orering ρ 1 of B 1 such that the optimal ecay rate is a power of N, namely that f(n) = n α for some α > 0, which is the case for the one imensional examples we covere in Section 3. The above Lemma tells us that to fin the optimal ecay rate we shoul take an orering σ : N N that is consistent with 1/F (n) := 1/f(n i) = nα i which is equivalent to being consistent with 1/F 1/α (n) = n i. This motivates the following: 10

11 Definition 4.4 (Corresponing to the hyperbolic cross). Define F H : N R by F H (n) = n i. Then we say a bijective function σ : N N correspons to the hyperbolic cross if it is consistent with F H. The name hyperbolic cross originates from its use in approximation theory [3, 11]. We now claim that if σ correspons to the hyperbolic cross an 2, then ( 1)!N σ(n) i log 1 (N + 1) Next we procee to prove this claim. as N. (4.3) Definition 4.5. For N let f (x) = x log 1 x be efine on [1, ). We efine g as the inverse function of f on [1, ), an so g : [0, ) [1, ). Furthermore, we efine x h (x) := log 1, x [1, ). (4.4) (x + 1) Lemma 4.6. The following hols: 1.) g (x)/h (x) 1 as x. 2.) Let f(x) = x log 1 x + xp(log(x)) + β with p a polynomial of egree at most 2, β R an let g be its inverse function efine for large x R +. Then we also have g(x)/h (x) 1 as x. Proof. 1.) For notational convenience we shall prove the equivalent result g (x) log 1 (x) 1 as x. x By taking logarithms we change the problem from stuying the asymptotics of a fraction to the asymptotics of the ifference log(g (x)) log(h (x)) = log(g (x)) log x + ( 1) log log x 0 as x. (4.5) With this in min we notice that the function log(g ) (efine on [0, )) is the inverse function of e (x) := f (exp(x)) = x 1 exp x (efine on [0, )). (x ( 1) log x) 1 Notice that for x large we have e (x ( 1) log x) = exp(x) exp(x) which x 1 implies that x ( 1) log x log(g (exp(x))). Now if we let ɛ > 0 then we euce that e (x ( 1) log x + ɛ) = (x ( 1) log x + ɛ) 1 x 1 exp(x + ɛ) exp(x) for x large. This implies that x ( 1) log x + ɛ log(g (exp(x))) for x large. We therefore conclue that for all x sufficiently large we have x ( 1) log x log(g (exp(x))) x ( 1) log x + ɛ, from which (4.5) follows since ɛ > 0 is arbitrary. 2.) Notice that by part 1. it suffices to show that g(x)/g (x) 1 as x. Again, we shall show this by taking logarithms, reucing the proof to showing log( g(x)) log(g (x)) 0 as x. Notice that log( g(x)) is the inverse function, efine for large x, of Then since ẽ(x) := f(exp(x)) = x 1 exp(x) + p(x) exp(x) + β, ẽ (x) = x 1 exp(x) + (( 1) x 2 + p (x) + p(x)) exp(x), we can use the hypothesis that p is of a lower orer than x 1 to show that for every ɛ > 0, there is an L(ɛ) > 0 such that for all x L(ɛ) we have ɛ ẽ (x ɛ) ẽ(x) e (x) = p(x) exp(x) + β. We therefore euce from the mean value theorem that for x exp(l(ɛ)) we have ẽ(log(g (x)) ɛ) e (log(g (x))) =x ẽ(log(g (x)) + ɛ) log(g (x)) ɛ log( g(x)) log(g (x)) + ɛ, where we applie log( g) to the inequality in the last step (this preserves the inequality since log( g) is an increasing function of x for x large). 11

12 Lemma ). For every N we have R N := N 1 i (log(n) log(i)) = log+1 N + O(log N) N. (4.6) 2). Let S (N) for, N N be efine by { S (N) := # m N : } m i N. (4.7) Then for every N, there exists polynomials p, p both of egree 1 with ientical leaing coefficient 1/( 1)! such that Np (log(n)) S (N) Np (log(n)). (4.8) 3). If we let σ : N N correspon to the hyperbolic cross then (4.3) hols. Proof. 1). Let I N := N 1 1 x (log(n) log(x)) x. Since the integran is a ecreasing function of x (with N fixe) we fin that by the Maclaurin integral test that 0 R N I N log (N). This means that showing (4.6) is equivalent to showing that N 1 1 x (log(n) log(x)) x = log+1 N + O(log N). Now, by expaning out the factors of the integran an integrating (recall that the integral of x 1 log k x is 1 k+1 logk+1 x) the integral becomes log +1 (N) i=0 ( ) 1 ( 1) i. i + 1 i Since i+1( 1 ) ( i = 1 +1 ) +1 i+1 we see that the sum simplifies to 1 +1 an we are one. 2). We use inuction on the imension. The case = 1 is immeiate since p 1 (x) = p 1 (x) = 1 satisfies inequality (4.8). Therefore suppose that inequality (4.8) hols for imension = k. We shall exten the result to = k + 1 using the equality: S k+1 (N) = N ( N ) S k. (4.9) i This equality follows from rewriting the set efining S k+1 as the following isjoint union: { k+1 } m N k+1 : m i N = { N k m N k+1 : m k+1 = j, m i j=1 } N. i Upper Boun: We may assume without loss of generality that p k has all coefficients positive. Therefore, by replacing N i with N i an using the upper boun in (4.8), we can upper boun equation (4.9) by N N ( ( N )) i p k log i N N 1 i p k(log(n) log(i)). We can then get the require upper boun by applying part 1) of the lemma to each term in the polynomial; for example the highest orer term becomes N N i 1 (k 1)! (log(n) log(i))k 1 N k! logk N + CN log k 1 N, N N, for some constant C > 0 sufficiently large. The other terms in p k are hanle similarly. 12

13 Lower Boun: Notice that without loss of generality we can assume all the coefficients of p k apart from the leaing coefficient are negative. Using the lower boun in (4.8), we can lower boun equation (4.9) by N N ( ( N )) p i k log. i This means we can tackle the < k 1 orer terms in the same way as in the upper boun since we can replace N i with N i (recall we have assume these terms are negative). Now we are left with bouning the highest orer term: N N 1 ( N ) N N i (k 1)! (log ) k 1 [ ( N ) = i i (k 1)! log ( ( N ) ( N )) ] k. log log i i i (4.10) Therefore expaning ( ) out the binomial term, setting the sign of all terms except the first to be negative, an ) 1 for every i, N we get the lower boun noticing log N i log ( N i N N 1 ( N ) i (k 1)! logk i N k 1 j=0 N i ( ) k j 1 ( N (k 1)! logj i From here we can replace N i by N i for the right term, N i by N i 1 on the left term an use part 1) of the lemma again to prove the lower boun. 3.) From the secon part of the lemma we know that for some egree 1 polynomials p, p with leaing coefficient 1/( 1)! we have Np (log(n)) S (N) Np (log(n)). Now notice that if m N then because of consistency we must have S (F H (σ(m)) 1) m since σ must first list all the terms n in N with F H (n) F H (σ(m)) 1 before listing σ(m). Likewise we must have m S (F H (σ(m))) since the S (F H (σ(m))) terms with F H (n) F H (σ(m)), n N must be liste by σ first, incluing m, before any others. Consequently we euce (F H (σ(m)) 1)p (log(f H (σ(m)) 1)) m F H (σ(m))p (log(f H (σ(m)))). (4.11) We now treat both sies separately. Looking at the LHS we get the estimate F H (σ(m)) 1 g (m), where g (m) is the inverse function (efine for large m) of f (x) := 1 ( 1)! x log 1 (x) + (egree 2 poly)(log(x)), an so we may apply part 2. of Lemma 4.6 to euce F H (σ(m)) h (( 1)!m) (1 + ɛ(m)), where ɛ(m) 0 as m. The right han sie is hanle similarly to get the same asymptotic lower boun on F H (σ(m)), namely F H (σ(m)) h (( 1)!m) (1 + ɛ(m)), where ɛ(m) 0 as m. Since h (( 1)!x) ( 1)!h (x) 1 as x the proof is complete. (4.3) allows us to etermine the optimal ecay rate for when the optimal one imensional ecay rate is a power of N. Theorem 4.8. Returning to the framework of Corollary 4.3, if f(n) = n α for n N, F (n) = f(n) for n N an σ : N N correspons to the hyperbolic cross then ( ) α F (σ(n)) = σ(n) i ( ( 1)! h (N) ) α, N. (4.12) Consequently h α is representative of the optimal ecay rate for the basis pair (B1, B2). Furthermore, an orering ρ is strongly optimal for the basis pair (B1, B2) if an only if there are constants C 1, C 2 > 0 such that ( ) C 1 h (N) (ρ 1) 1 ρ(n) C 2 h (N), N N. (4.13) i ). 13

14 Proof. (4.12) follows immeiately from (4.3). This implies that F σ h α. The statement on the optimal ecay rate then follows from the characterization result from Lemma 4.2 applie to Lemma 2.6. The statement on strongly optimal orerings follows from Corollary 4.3. Definition 4.9. Using the framework of Lemma 4.2, any orering ρ : N B 1 such that (4.13) hols is calle a hyperbolic orering. with respect to ρ 1. Notice that by (4.12) that if σ : N N correspons to the hyperbolic cross then ρ 1 σ is hyperbolic with respect to ρ 1. We now apply Theorem 4.8 to the one-imensional cases we have alreay covere: Fourier-Wavelet Case Theorem We use the setup of Lemma 4.2. Suppose B 1 = B f (ɛ), B 2 = B w for some ɛ I J,p, ρ 1 is a stanar orering of B 1 an τ 1 is a levele orering of B 2. Let U = [(B 1, ρ), (B 2, τ)] where ρ, τ is hyperbolic with respect to ρ 1, τ 1 respectively. Then we have, for some constants 9 C 1, C 2 > 0, C 1 log 1 (N + 1) N µ(π N U ), µ(u π N ) C 2 log 1 (N + 1), N N. (4.14) N The above also hols if the basis B w is replace by B bw an the conition ɛ I J,p by ɛ (0, 1/2]. Proof. Inequality (4.14) follows from applying Theorem 3.2 to Theorem Fourier-Polynomial Case Theorem We use the setup of Lemma 4.2. Suppose B 1 = B f (ɛ), B 2 = B p for some ɛ (0, 0.45], ρ 1 is a stanar orering of the Fourier basis an τ 1 is the natural orering of the polynomial basis. Let U = [(B 1, ρ), (B 2, τ)] where ρ, τ is hyperbolic with respect to ρ 1, τ 1 respectively. Then we have, for some constants C 1, C 2 > 0, that C 1 (log 1 (N + 1)) 2/3 N 2/3 µ(π N U ), µ(u π N ) C 2(log 1 (N + 1)) 2/3 N 2/3, N N. (4.15) Proof. Inequality (4.15) follows from applying Theorem 3.3 to Proposition Examples of Hyperbolic Orerings The generalisation introuce by Definition 4.9, apart from allowing us to characterise all orerings that are strongly optimal, may seem to fulfil little other purpose. However, as we shall see in this section, this efinition amits orerings which in specific cases are very natural an appear a little less abstract than an orering erive from the hyperbolic cross. Example 4.1 (Hyperbolic Cross in Z ) Our first example is unremarkable but nonetheless important. In imensions, take B1 := Bf as a -imensional tensor Fourier basis. Recall we can ientify this basis with Z using the function λ. Suppose that we efine a function H : Z R by H (m) = max( m i, 1), (4.16) an say that a bijective function σ : N Z correspons to the hyperbolic cross in Z if it is consistent with H. Figure 2 shows the first few contour lines of H in two imensions. With this efinition we can then prove the analogous result of Lemma 4.7: Lemma Let σ : N Z correspon to the hyperbolic cross an let h be as in (4.3). Then we have max( σ(m) i, 1) ( 1)! 2 h (m) as m. (4.17) Moreover, if ρ 1 is a stanar orering of B f an σ : N Z correspons to the hyperbolic cross. Then λ 1 σ is a hyperbolic orering with respect to ρ 1. 9 Constants that are epenent on ɛ. 14

15 Figure 2: Hyperbolic Fourier Orering in Two Dimensions: A Contour Plot of H Proof. Let R (n) enote the number of lattice points in the hyperbolic cross of size n in Z, namely R (n) := #{m Z : max( m i, 1) n}. Call the set in the above efinition H (n). If we remove the hyperplanes {m i = 0} for every i from H (n), we are left with 2 quarants in Z which are congruent to set in equation (4.7). From the secon part of Lemma 4.7 we therefore have R (n) 2 np (log(n)). Next notice that the intersection of H (n) with each hyperplane {m i = 0} can be ientifie with H 1 (n) an so we also have the upper boun R (n) 2 np (log(n)) + R 1 (n) R (n) nr (log(n)), for some egree 1 polynomial r with leaing coefficient 2 ( 1)! we see that for some polynomials r, r of egree 1 with leaing coefficient Therefore for m N since we have nr (log(n)) R (n) nr (log(n)). R (H (σ(m)) 1) m R (H (σ(m))),. Combining the upper an lower bouns 2 ( 1)! we have (H (σ(m)) 1)r (log(h (σ(m)) 1)) m H (σ(m))r (log(h (σ(m)))). Consequently we can apply Lemma 4.6 to both sies to erive (4.17) like in the proof of Lemma 4.7. For the last part of the Lemma notice that since ρ 1 is a stanar orering then max( λ 1 ρ 1 (N), 1) N. This means that the bouns on µ(π N U) in Theorem 3.4 can be rephrase as (for some constants C 1, C 2 > 0) C 1 (max( n, 1)) 1 sup g B w λ 1 1 (n), g 2 C 2 (max( n, 1)) 1, n Z, an by Lemma 4.2 this extens to the D tensor case: C 1 (max( n i, 1)) 1 sup g B w λ 1 (n), g 2 C 2 (max( n i, 1)) 1, n Z. (4.18) This escribes a characterization of the optimal ecay of (Bf (ɛ), B w). Lemma 2.6 tells us that λ 1 σ is strongly optimal for (Bf (ɛ), B w), which by Theorem 4.8 is hyperbolic with respect to ρ 1. 15

16 Example 4.2 (Tensor Wavelet Orering) Now we look at an example of a less obvious hyperbolic orering. We first introuce some notation to escribe a tensor wavelet basis: For j N, k Z let φ 0 j,k := φ k, φ 1 j,k := ψ j,k. Now for s {0, 1}, j N, k Z efine Ψ s j,k := φ si j i,k i. Then it follows that for J N fixe, we have Supp(φ si Bw j i,k i ) ( 1, 1) i, := Ψ s j,k : s i = 0 j i = J, s i = 1 j i J j N, s {0, 1}, k Z, (4.19) The same approach can be applie to the bounary wavelet basis B bw to generate a bounary tensor wavelet basis Bbw, although we must inclue the extra bounary terms, which can be one by letting s {0, 1, 2, 3} where φ 2 J,n woul be a bounary scaling function term an φ3 j,n a bounary wavelet term. Lemma Let ρ 1 be any levele orering of a one-imensional Haar 10 wavelet basis B w. Setting j = j i efine F hyp : B w R by the formula F hyp (f) = j if f = Ψ s j,k. Then any orering ρ : N B w that is consistent with F hyp is a hyperbolic orering with respect to ρ 1. Remark 4.1 Such an orering ρ is use to implement a tensor wavelet basis in Section 6. Proof. By recalling inequality (3.10) in [18] or by using Lemma 5.3 in the case = 1 we know that there are constants C 1, C 2 > 0 such that for ρ 1 (N) = φ s j,k, s {0, 1}, j N, k Z, Therefore, writing ρ 1(m) = Ψ s(m) j(m),k(m), C 1 2 j(m) C 1 2 j N C 2 2 j, N N. m i C2 2 j(m), m N. Consequently if we rewrite this with an actual orering ρ(n) = Ψ s(n) j(n),k(n) for N N we euce C 1 2 j(n) ( ) (ρ 1) 1 ρ(n) C2 2 j(n), (4.20) i an so we have reuce the problem to etermining how j(n) scales with N. Notice that from our orering of the wavelet basis that j(n) is a monotonically increasing function in N an moreover, for every value of j(n) there are r (j(n))2 j(n) terms 11 in Bw with this value of j(n) in the wavelet basis, where { } r (N) := # (j, s) N {0, 1} : j = N, j i J, (s i 1)(j i J) = 0 i = 1,...,, which is a polynomial of egree 1. With this in min notice we can efine, consistent 12 for n N, n J, T (x) = x r (i)2 i := p (x)2 x + α, i=j 10 For the sake of simplicity we only work with the Haar wavelet case, although we coul cover the bounary wavelet case with the same argument. 11 This is where we use that the support of the Haar wavelet is [0, 1] an so there are 2 j shifts of φ j,0, ψ j,0 in B w. 12 To see how this is possible, take the formula for the geometric series expansion an ifferentiate repeately. 16

17 for some egree 1 polynomial p an constant α. By the consistency property of ρ we euce the inequality T (j(n) 1) N T (j(n)) j(n) 1 T 1 (N) j(n) 2 j(n) 1 2 T 1 (N) 2 j(n). Notice that 2 T 1 (x) is the inverse function of T (log 2 x) which is of the form x p (log 2 x) + α, Therefore, applying parts 2. & 3. of Lemma 4.6 gives, for some constants D 1, D 2 > 0 an N large, (1 + ɛ 1 (N)) D 1 2 j(n) g (N) (1 + ɛ 2 (N)) D 2 2 j(n), (4.21) where ɛ 1 (N), ɛ 2 (N) 0 as N. Combining this with (4.20) shows that we have a hyperbolic orering. 4.2 Plotting Tensor Coherences Let us consier a simple illustration of this theory applie to a 2D tensor Fourier-Wavelet case (B 2 f, B2 w). We can ientify the 2D Fourier Basis B 2 f with Z2 using the function λ 2, so the row incoherences can also be ientifie with Z 2 an therefore they can be image irectly in 2D, as in Figure 3. Figure 3: 2D Tensor Fourier - Tensor Haar Incoherences (a) Original Coherences 0 (b) Hyperbolic Scaling 0 (c) Scale Coherences (Prouct of (a) & (b)) 0 We show the subset { 250, 249,..., 249, 250} 2 Z 2. Notice that the scale coherences have no vanishing values (no pure black) an no values that blow up (no pure white, baring the center value which = 1) inicating that we have characterise the coherence in terms of the hyperbolic scaling use. Formally this is shown by equation (4.18). The coherences shown in the Figure are square roote to reuce contrast (i.e. we image µ(π N U) instea of µ(π N U)). 5 Multiimensional Fourier - Separable Wavelet Case We repeat the notation of the one-imensional case, with scaling function φ (in one imension) & Daubechies wavelet ψ: φ j,k (x) = 2 j/2 φ(2 j x k), ψ j,k (x) = 2 j/2 ψ(2 j x k). We can construct a -imensional scaling function Φ by taking the tensor prouct of φ with itself, namely Φ(x) := ( j=1 ) φ (x) = φ(x j ), x R, j=1 which has corresponing multiresolution analysis (Ṽj) j Z with iagonal scaling matrix A R with A i,j = 2δ i,j. Let φ 0 := φ, φ 1 := ψ an for s {0, 1}, j J, k Z where J N is fixe efine the functions Ψ s j, := 17 φ si j,k i. (5.1)

18 If we write (for s {0, 1} \ {0}, j J) W s j := Span{Ψ s j,k : k Z }. Then it follows that Ṽ j+1 = Ṽj Wj s, s {0,1} \{0} L 2 (R ) = ṼJ s {0,1} \{0} j J W s j. This correspons to taking 2 1 wavelets for our basis in imensions (see [26]). As before we take the spanning functions from the above whose support has non-zero intersection with [ 1, 1] as our basis B 2 (calle a separable wavelet basis ): Bsep := Supp(φ si j,k i ) ( 1, 1) Ψ s j,k : s = 0 j = J, j N, s {0, 1}, k Z i,, (5.2) Remark 5.1 We can also construct a separable bounary wavelet basis in the same manner like in the oneimensional case however, for the sake of simplicity, we stick to the above relatively simple construction throughout (although all the coherence results we cover here also hol for the separable bounary wavelet case as well). 5.1 Orering the Separable Wavelet Basis We note a few key equalities from the one-imensional case that will come in hany: Fφ j,k (ω) = e 2πi2 j kω 2 j/2 Fφ(2 j ω), Fψ j,k (ω) = e 2πi2 j kω 2 j/2 Fψ(2 j ω), (5.3) where F here enotes the Fourier Transform, i.e. for f L 2 (R ) we efine Ff(ω) = f(x)e 2πiω x x, ω R. R Recall χ k from Definition 1.9. We observe that by (5.1) Ψ s j,k, χ n = ɛ /2 FΨ s j,k(ɛn) = ɛ /2 sup n N Ψ s j,k, χ n 2 = ɛ 2 j Fφ si j,k i (ɛn i ), n Z, sup Fφ si (ɛ2 j n) 2. n N By careful treatment of the prouct term we can etermine the optimal ecay of (Bsep, Bf (ɛ)), using the following result: Proposition 5.1. There are constants C 1, C 2 > 0 such that for all ɛ I J,p, Ψ s j,k B sep we have C 1 ɛ 2 j sup n N Ψ s j,k, χ n 2 C 2 ɛ 2 j. Consequently, fixing ɛ, the function F power : B sep R efine by F power (Ψ s j,k ) = 2 j characterizes the optimal ecay of (B sep, B f (ɛ)). Proof. Let A = max(sup ω R Fφ(ω) 2, sup ω R Fψ(ω) 2 ). Then (5.4) gives us the upper boun sup Ψ s j,k, χ n 2 ɛ 2 j A. n N This leaves the lower boun. This can be achieve if we can show that there exists constants D 1, D 2 > 0 such that for all ɛ I J,p 13 (5.4) F 1 (ɛ) := sup Fφ(ɛ2 J n) D 1, n N F 2 (ɛ) := sup Fψ(ɛ2 J n) D 2. (5.5) n N 13 notice that replacing J with j J below woul have been reunant. 18

19 By the Riemann-Lebesgue Lemma the functions F 1, F 2 are continuous on I J,p an F 1 (ɛ) sup Fφ(ω) > 0 as ɛ 0. ω R Likewise for F 2. Therefore F 1, F 2 can be extene to continuous functions over the close interval I J,p {0}. Since F 1 (ɛ) > 0, F 2 (ɛ) > 0 for every ɛ I J,p 14 the infimums over I J,p {0} are attaine an are strictly positive, proving (5.5) an the lower boun. Let F level : Bsep R be efine by F level (Ψ s j,k ) = j. Lemma 2.6 tells that an orering that is consistent with 1/F power, i.e. consistent with F level will be strongly optimal. Definition 5.2. We say that an orering ρ : N B sep is levele if it is consistent with F level. Lemma 5.3. Let ρ : N B sep be levele. Then there are constants D 1, D 2 > 0 such D 1 N 2 F level(ρ(n)) D 2 N. (5.6) Proof. Let a N enote the length of the support of φ, ψ. Notice that for each j N an s {0, 1}, there are (2 j+1 + a 1) shifts of Ψ s j,0 whose support lies in [ 1, 1]. For convenience we use the notation j(n) := F level (ρ(n)) an shall also be using the simple bouns 2 j(n)+1 2 j(n)+1 +a 1 2 j(n)+a. Now for every N N with j(n) > J, we must have ha all the terms of the form f B sep, F level (f) = j(n) 1 come before N in the levele orering an there are at least (2 1) 2 j(n) of these terms, implying that (2 1) 2 j(n) N. This completes the upper boun for j(n) > J. Likewise for every N N with j(n) J there can be no more than j(n) 2 2 (i+a) 2 2 (j(n)+a+1) = 2 (a+2) 2 j(n), i=j terms such that F level (f) j(n). This shows that N 2 (a+2) 2 j(n), completing the upper boun for j(n) > J. Extening (5.6) to all N N (i.e. j(n) J) is trivial since we have only omitte finitely many terms so a change of constants will suffice. Corollary 5.4. Any orering ρ of B sep that is levele is strongly optimal for the basis pair (B sep, B f (ɛ)). Furthermore, the optimal ecay rate of (B sep, B f (ɛ)) is represente by the function f(n) = N 1. Proof. Lemma 2.6 applie to Proposition 5.1 tells us that ρ is strongly optimal an moreover the optimal ecay rate is represente by F power (ρ(n)) which by Lemma 5.3 is of orer N Orering the Fourier Basis - Hyperbolic & Linear Orerings We now want to fin the optimal ecay rate of (Bf (ɛ), B sep) which means looking at orerings of the Fourier basis. It might be tempting to try an exten the stanar orering efinition from the one imensional Fourier basis. Recall as well that, using the function λ efine in (1.9), orering Bf (ɛ) is equivalent to orering Z. If we let s {0, 1}, j N, k Z, then in orer to boun the coherence µ(π N U) we nee to be bouning terms of the form Ψ s j,k, λ 1 (n) 2 = ɛ 2 j Fφ si (2 j ɛn i ) 2. (5.7) In the one-imensional case in [18] the following ecay property of the Fourier transform of the scaling function φ was use: 14 if it was zero then since the span of B f (ɛ) covers L2 [ 1, 1] we woul euce that φ, ψ has no support in [ 1, 1] which is a contraiction. 19

20 Lemma 5.5. If φ is any Daubechies scaling function with corresponing mother wavelet ψ then there exists a constant K > 0 such that for all ω R \ {0}, Fφ(ω), Fψ(ω) K ω. (5.8) Furthermore, suppose that for some α > 0 we have, for some constant K > 0, the ecay Fφ(ω) K ω α for all ω R \ {0}. Then, for a larger constant K > 0, Fψ(ω) K ω α for all ω R \ {0}. Proof. The first result is a irect result of Lemma 3.5 in [18]. The last statement follows immeiately from the equality (taken from equation (3.14) in [18]): Fψ(2ω) = m 0 (ω + 1/2) Fφ(ω), (5.9) where m 0 is the low pass filter corresponing to φ which satisfies m 0 (ω) 1 for all ω R. Therefore let us first consier the case where we use (5.8) to boun every term in the prouct, giving us (n Z, n i 0, i = 1,..., ) Ψ s j,k, λ 1 (n) 2 ɛ 2 j K 2 ɛ2 j n i = K2 n i. (5.10) Making ajustments to prevent iviing by zero by using sup ω R max( Fφ(ω), Fψ(ω) ) 1 15, this can then be rephrase as sup g, λ 1 (n) 2 max(k2, 1) g Bsep max( n i, 1), n Z. (5.11) This tells us that the function F hyp : Z R, F hyp (n) = ( max( n i, 1)) 1 ominates the optimal ecay of (Bf, B sep) (see Definition 2.5 for the efinition of omination). Therefore if we want to maximise the utility of this boun then we shoul use an orering σ of Z so that max( σ(n) i, 1) is increasing, namely an orering corresponing to the hyperbolic cross in Z (see Example 4.1). However, using such an orering will not give us the N 1 ecay rate that we got from the one imensional case: Proposition 5.6. Let σ : N Z correspon to the hyperbolic cross in Z an efine an orering ρ of Bf (ɛ) by ρ := λ 1 σ, where ɛ I J,p. Next let U = [(Bf (ɛ), ρ), (B sep, τ)] for any orering τ an fix ɛ. Then there are constants C 1, C 2 > 0 C 1 log 1 (N + 1) N µ(q N U) C 2 log 1 (N + 1), N N. N As this result is primarily for motivation, its proof is left to the appenix. Since this approach gives us suboptimal results, we return to our boun of (5.7). Instea of using (5.8) on every term in the prouct, why not just use it once on the term that give us the best ecay instea? To boun the remaining terms we can simply use sup ω R max( Fφ(ω), Fψ(ω) ) 1. This approach gives us the following boun Ψ s j,k, λ 1 (n) 2 ɛ 2 j min,... K 2 ɛ2 j n i = ɛ 1 2 ( 1)j K 2 max,..., n i, n Z. (5.12) As we shall see in Lemma 5.9, choosing ρ so that we maximise the growth of the max,..., n i leas to max,..., n i E N 1/ for some constant E > 0 an so (5.12) is boune above by constant N 1/, which is very poor ecay. However, if we instea replace (5.8) by the stronger conition Fφ(ω) 15 For φ this follows from Proposition 1.11 in [16]. We exten this to ψ using equation (5.9) K, ω R \ {0}. (5.13) ω /2 20

21 then we can obtain the following upper boun 16 Ψ s j,k, ρ(n) 2 ɛ 2 j K 2 min,... ɛ2 j n i = K 2 max,..., n i. (5.14) Let us write n := max,..., n i. The above can be rephrase as sup g, λ 1 (n) 2 max(k2, 1) g B max( n sep, 1), n Z. (5.15) Therefore we euce that F lin : Z R, F lin (n) = (max( n, 1)) 1 ominates the optimal ecay of (B f (ɛ), B sep). In fact in can be shown that F lin also characterizes the optimal ecay (i.e. a lower boun of the same form is possible) by using the following preliminary Lemma: Lemma 5.7. For any compactly supporte wavelet ψ there exists an R N such that for all q R, (q N) we have L q := inf ω [2 (q+1),2 q ] Fψ(ω) > 0. (5.16) Proof. See Lemma 3.6 in [18]. Proposition 5.8. We fix the choice of wavelet basis B sep an recall the function λ : B f (ɛ) Z from (1.9). 1.) Then there are constants C 1 (φ) > 0, D(J) > 0 such that for all ɛ I J,p an n Z with n Dɛ 1 we have sup g, λ 1 (n) 2 C 1 g B n. (5.17) sep Therefore (by fixing ɛ) the function F lin is ominate by the optimal ecay of (B f (ɛ), B sep). 2.) Suppose that φ satisfies (5.13). Then there is a constant C 2 (φ) > 0 such that for all ɛ I J,p an n Z, sup g B sep g, λ 1 (n) 2 C 2 max( n, 1). (5.18) Therefore (by fixing ɛ) the function F lin characterizes the optimal ecay of (B f (ɛ), B sep). Proof. 2.) Follows from (5.15). 1.) If we set j = log 2 ɛ n + q for some q N fixe we observe that ɛ2 j n i [0, 2 q ] for every i = 1,..., an, since we are using the max norm, ɛ2 j n i [2 q 1, 2 q ] for at least one i, say i. Set s i = 0 for i i an s i = 1. Then, assuming j J, by (5.7) we have the lower boun. Ψ s j,0, λ 1 (n) 2 2 (q+1) n 2 (q+1) n Fφ si (ɛ2 j n i ) 2 inf Fψ(ω) 2 ω (2 q 1,2 q ] inf ω [0,2 q ] Fφ(ω) 2( 1). (5.19) Recall that by Lemma 5.7 there exists a q N such that L q > 0 an inf ω [0,2 q ] Fφ(ω) > 0 17 an therefore (5.18) follows as long as j J. To ensure that j = log 2 (ɛ n ) + q satisfies j J we must therefore impose the constraint that n is sufficiently large. j J is satisfie if J log 2 (ɛ n ) n 2 J ɛ 1. Remark 5.2 If = 2 then (5.13) always hols by Lemma 5.5. This means we have characterize every 2D Separable wavelet case (for Daubechies Wavelets). 16 noting that (5.13) also hols for ψ by (5.9). 17 We are using the fact that Fφ(0) = 1 an continuity of Fφ here which follows from φ L 1 (R). 21

22 Remark 5.3 A similar upper boun in two imensions base on the norm of n Z 2 has alreay been consiere in a iscrete framework for separable Haar wavelets [20]. Let F norm (n) := max( n, 1). By 2.6 we know that if (5.13) hols then the optimal ecay of (Bf, B sep) is etermine by the fastest growth of F norm. This motivates the following: Lemma 5.9. Let σ : N Z be consistent with F norm. Then there are constants E 1, E 2 > 0 such that E 1 N 1/ max( σ(n), 1) E 2 N 1/, N N. (5.20) Proof. If σ(n) = L 2, then σ must have enumerate beforehan all points m in Z with m L 1 an there are (2L 1) of such points. This means that N (2L 1) σ(n) N 1/ + 1, N N. 2 which proves the upper boun when σ(n) = L 2. The lower boun is tackle similarly by noting σ must first list all m Z with m L, incluing σ(n) which shows N (2L + 1) σ(n) N 1/ 1, N N. 2 This proves (5.9) for σ(n) = L 2. Extening this to all N N is trivial since we have only omitte finitely many terms, so changing the constants will suffice since all terms are strictly positive. Definition 5.10 (Linear Orering). Any orering ρ : N B f (ɛ) such that σ = λ ρ satisfies (5.20) is calle a linear orering. Corollary Assuming (5.13) hols for the scaling function corresponing to B sep, an orering ρ of B f (ɛ) is strongly optimal for the basis pair (B f (ɛ), B sep) if an only if it is linear. Furthermore, the optimal ecay rate of (B f (ɛ), B sep) is represente by the function f(n) = N 1. Proof. If we apply part 2.) of Proposition 5.8 to Lemma 2.6 we kow that if σ : N Z is consistent with 1/F lin = F norm, i.e. consistent with F norm, then F lin (σ( )) = 1/F norm(σ( )) represents the optimal ecay rate. Lemma 5.9 tells us that this optimal ecay is 1/(N 1/ ) = 1/N. Furthermore, Lemma 2.6 says that an orering ρ is strongly optimal for (B f (ɛ), B sep) if an only if F lin (λ ρ( )) F lin (σ( )) which hols if an only if F norm (λ ρ( )) F norm (σ( )), namely ρ is linear. Corollary 5.11 gives us the same optimal ecay as in one imension, which is in contrast to the multiimensional tensor case, where the best we can o is have 1 extra log factors. Example 5.1 If we o not have conition (5.13) then our argument can break own very baly: For Haar wavelets we have an explicit formula for the Fourier transform of the one-imensional mother wavelet, Fφ(ω) = exp(2πiω) 1. 2πiω Therefore we have that (5.13) is not satisfie for 3 an furthermore we have (for ɛ < 1 an J N fixe) Fφ(ɛ2 J k) 1 2πɛk, (5.21) for infinitely many k N. Now consier the case of D separable Haar wavelets with a linear orering ρ of the Fourier Basis. Then, for m N such that λ ρ(m) = (λ ρ(m) 1, 0,, 0) we know that by (5.21) there are infinitely many m such that Φ, ρ(m) 2 = ɛ Fφ(ɛ2 J λ ρ(m) 1 ) 2 Fφ(0) 2 ( 1) ɛ 1 (2πɛ λ ρ(m) 1 ) 2 ɛ 2 E 4π 2 m, (5.22) 2/ for some constant E using Lemma 5.9. Therefore an upper boun of the form Constant N 1 is not possible for a linear scaling scheme if 3. This can be rectifie by applying a semi-hyperbolic scaling scheme, as in the next subsection. 22

23 5.3 Examples of Linear Orerings - Linear Scaling Schemes A wie variety of sampling schemes that are commonly use happen to be linear. In particular we emonstrate that sampling accoring to how a shape scales linearly from the origin always correspons to a linear orering (see Figure 4): Figure 4: A Simple Linear Scaling Scheme (a) Scaling shape (b) Sampling accoring to a linear scaling scheme with scaling shape (a) Definition Let D R be boune with 0 in its interior an efine S D : Z : R S D (x) := inf { κ > 0 : x κd }. An orering σ : N Z is sai to correspon to a linear scaling scheme with scaling shape D if it is consistent with S D. Furthermore, an orering ρ : N Bf (ɛ) is sai to correspon to a linear scaling scheme with scaling shape D if it is consistent with S D λ. Remark 5.4 If we put a norm on Z an take an orering consistent with this norm then this orering correspons to a linear scaling scheme with scaling shape {x R : x = 1}. Lemma Let ρ : N Bf (ɛ) correspons to a linear scaling scheme with scaling shape D. Then ρ is linear. Proof. Let σ = λ ρ. Because the scaling shape D is boune an contains 0 in its interior we have that there exists constants C 1, C 2 > 0 such that C 1 S D C 2 S where S is efine to be the unit hypercube, i.e. S := {x R : x = 1}. Therefore if σ(n) = L, then since D C 2 S we have that S D (σ(n)) LC2 1. Applying this to C 1S D we euce that σ must have enumerate beforehan all points m in Z with m < LC 1 C2 1 an there are at least (2(LC 1 C2 1 1) + 1) of such points. This means that N (2( σ(n) C 1 C 1 2 1) + 1) σ(n) N 1/ + 1 2C 1 C 1 2 which proves the upper boun. The lower boun is tackle similarly to prove (5.20) D Separable Incoherences N 1/ C 1 C 1, N N. By Remark 5.2 we have shown that linear orerings are strongly optimal for all 2D Fourier - separable wavelet cases, so this is a goo point to have a quick look at a few of these in Figure Semi-Hyperbolic Orerings By Example 5.1 we know that if (5.13) oes not hol then our approach of using a linear orering can fail. We therefore return once more to (5.7). Let us now try to use an approach that is halfway between our 2 23

24 Figure 5: 2D Fourier - Separable Wavelet Coherences (a) Haar Coherences (b) Scale Haar Coherences (c) Linear Scaling use for Both Bases () Daubechies16 Coherences 0 (e) Scale Daubechies16 Coherences 0 We show the subset { 250, 249,..., 249, 250} 2 Z 2. Notice again that the scale coherences are boune above zero an below 1 inicating that we have characterise the incoherence in terms of the linear scaling use, as shown in Proposition 5.8. The incoherences shown in the Figure are square roote to reuce contrast. two previous linear/hyperbolic approaches. Let r {1,..., 1} be fixe. We shall first impose a ecay conition that is stronger than (5.8) but weaker than (5.13): Fφ(ω) K, ω R \ {0}. (5.23) ω /2r Instea of just taking out the ominant term of the prouct in (5.7), let us take out the r smallest terms: Ψ s j,r, λ 1 (n) 2 ɛ 2 j min ( = K 2r i 1,...,i r {1,...,} i 1<...<i r max i 1,...,i r {1,...,} i 1<...<i r r r=1 K 2 /r ɛ2 j n ir r /r, n ir ) n Z, n i 0, i = 1,...,. r=1 (5.24) Again we can exten this boun to all n Z : ( sup g, λ 1 (n) 2 max(k 2r, 1) g Bsep max i 1,...,i r {1,...,} i 1<...<i r We euce that the function F hyp,r : Z R, F hyp,r (n) = ominates the optimal ecay of of (B f, B sep). Definition Let us efine, for r, N, r the function H,r (n) := max i 1,...,i r {1,...,} i 1<...<i r r /r, max( n ir, 1)) n Z. (5.25) r=1 ( ) /r r max i1,...,i r {1,...,} r=1 max( n i r, 1) i 1<...<i r r max( n ij, 1), n Z. j=1 24

25 Then we say an orering σ : N Z is semi-hyperbolic of orer r in imensions if it is consistent with H,r. Figure 6 presents some isosurface plots of H 3,r for the various values of r Notice that a semi-hyperbolic orering of orer in imensions correspons to the hyperbolic cross in Z (see Example 4.1). Furthermore, if σ : N Z is a semi-hyperbolic orering of orer 1 in imensions then, by Remark 5.4, σ correspons to a linear scaling scheme because H,1 (n) = n for the componentwise max norm on R. Like in the linear an hyperbolic cases iscusse in the previous sections, we want to etermine how H,r (σ(n)) scales with n N. Figure 6: Isosurfaces of H 3,r, r = 1, 2, 3 escribing the three types of orering available in 3D (a) Case r = 1 (Linear) ; Isosurface value=10. (b) Case r = 2 (Semi-Hyperbolic) ; Isosurface value=20. (c) Case r = 3 (Hyperbolic) ; Isosurface value=20. Lemma ). Let r, N, r 1 be fixe. Let us efine Then there is a constant C > 0 such that S,r (n) := #{m Z : H,r (m) n}, n N. n /r S,r (n) C n /r, n N. 2). If σ : N Z is semi-hyperbolic of orer r with r 1 then there are constants C 1, C 2 > 0 such that C 1 n r/ H,r (σ(n)) C 2 n r/, n N. Proof. 1). For notational simplicity we prove the same bouns but with S,r replace by the smaller set S,r (n) := #{m N : H,r (m) n}, n N. The same bouns for S,r then follows immeiately, albeit with a larger constant C > 0. The lower boun is straightforwar since the set efining S,r (n) contains the set {m N : m i n 1/r, i = 1,.., }. We prove the upper boun by inuction on r. The case r = 1 is clear because S,1 (n) is simply the number of points insie a -imensional hypercube with sie length n. Suppose the result hols for r = r 1. We use the following set inclusion: {m Z : H,r (m) n} {m N : m i n 1/r, i = 1,.., } {m N : n 1/r m i n, H 1,r 1( m i ) n/m i }, (5.26) where m i here refers to m with the ith entry remove. The carinality of the first set on the right is just n /r an so we are one if we can show that for some constant C > 0, #{m N : n 1/r m 1 n, H 1,r 1((m 2,..., m )) n/m 1 } Cn /r, n N 25

26 We achieve this by applying our inuctive hypothesis: #{m N : n 1/r m 1 n, H 1,r 1((m 2,..., m )) n/m 1 } n n S 1,r 1( n/i ) C ( ) ( 1)/(r n/i 1) i= n 1/r C n ( 1)/(r 1) n n 1/r 2 i= n 1/r x ( 1)/(r 1) x C n ( 1)/(r 1) (n 1/r 2) (1 ( 1)/(r 1)), (noting r 1) (5.27) where C > 0 is some constant. We can replace (n 1/r 2) by n 1/r in the above by changing the constant C an assuming n > 2 r. Finally, we notice that the exponents a to the esire expression: 1 r ( r 1 1 ) r 1 = 1 r 1 r r (r 1) = r. This gives the require upper boun for n > 2 r. Since the terms involve are all positive, we can just increase the constant C to inclue the cases n 2 r. This shows that the result hols for r = r an the inuction argument is complete. 2.) By consistency we know that S,r (H,r (σ(n)) 1) n S,r (H,r (σ(n))), n N an therefore we can irectly apply part 1 to euce from which the result follows. (H,r (σ(n)) 1) /r n C (H,r (σ(n))) /r, n N, Arme with this result, we can now completely tackle the separable wavelet case. Proposition Suppose that the scaling function φ corresponing to the separable wavelet basis Bsep, satisfies (5.23) for some constant K 0 an r {1,..., 1}. Next let σ : N Z be semi-hyperbolic of orer r in imensions an ρ := λ 1 σ. Finally, we let U = [(B f (ɛ), ρ), (B sep, τ)], where τ is an orering of Bsep. Let us also fix ɛ I J,p. Then there are constants C 1, C 2 such that C 1 N µ(q NU) C 2 N, N N Furthermore it follows that the orering ρ is optimal for the basis pair (B f (ɛ), B sep). Proof. Applying part 1.) from Proposition 5.8 (with ɛ fixe) to part 2.) of Lemma 2.6 immeiately gives us the lower boun for the semihyperberbolic orering since this boun also hols for the optimal ecay rate. Furthermore this lower boun hols for any other orering an therefore if we have the upper boun then the orering ρ is automatically optimal. We now focus on the upper boun. By (5.25) we know that the optimal ecay of (Bf (ɛ), B sep) is ominate by F hyp,r. Therefore by part 1.) of Lemma 2.6 if σ : N Z is consistent with 1/F hyp,r, i.e. σ is semihyperbolic of orer r then we can boun the row incoherence µ(π N U) by F hyp,r (σ(n)) = H /r,r (σ(n)) (N r/ ) /r = N 1 by Lemma Since N 1 is ecreasing this boun extens to µ(q N U). Finally we can summarise our results on the (B f (ɛ), B sep) case as follows: Theorem Let ρ be a Linear orering of the -imensional Fourier basis B f (ɛ) with ɛ I J,p, τ a levele orering of the -imensional separable wavelet basis B sep an U = [(B f (ɛ), ρ), (B sep, τ)]. Furthermore, suppose that the ecay conition (5.8) hols for the wavelet basis. Then, keeping ɛ > 0 fixe, we have, for some constants C 1, C 2 > 0 the ecay C 1 N µ(π NU), µ(uπ N ), C 2 N N N. (5.28) 26

27 Let us now instea replace ρ by a semi-hyperbolic orering of orer r in imensions with r {1,..., 1} an assume the weaker ecay conition (5.23). Then, keeping ɛ > 0 fixe, we have, for some constants C 1, C 2 > 0 the ecay C 1 N µ(q NU), µ(uπ N ), C 2 N N, (5.29) N an furthermore ρ is optimal for the basis pair (B f (ɛ), B sep). Since, for any separable Daubechies wavelet basis, (5.23) always hols for r = 1 any semi-hyperbolic orering of orer 1 in imensions will prouce (5.29). Proof. (5.28) follows from Corollaries 5.4 an (5.29) follows from Corollary 5.4 an Proposition To show that (5.29) always hols for a 1 egree semi-hyperbolic orering in imensions, we note that the weakest ecay on the scaling function φ is Fφ(ω) K ω 1 (see Lemma 5.5) an therefore (5.23) is automatically satisfie for r = Optimal Orerings & Wavelet Smoothness Theorem 5.17 emonstrates how certain egrees of smoothness, in terms of ecay of the Fourier transform, allows us to show certain orerings are optimal an this smoothness requirement becomes increasingly more emaning as the imension increases. But if a certain orering is optimal for the basis pair (B f, B sep), oes this mean that the wavelet must also have some egree of smoothness as well? The answer to this question turns out to be yes, an it is the goal of this section to prove this result. We shall rely heavily on the following simple result from [19, Thm. 9.4]: Theorem Let 18 f : R/Z C be continuous an for k Z efine ˆf(k) = 1 f(x) exp(2πikx) x. If 0 k= k ˆf(k) < then f C 1. Consequently 19 if k= k n ˆf(k) < then f C n. Now the main result itself: Theorem Let σ : N Z be semihyperbolic of orer r < in imensions an let ρ := λ 1 σ : N B f (ɛ) where ɛ I J,p. Then if ρ is optimal for the basis pair (B f (ɛ), B sep) then φ C l for any l N {0} with l + 1 < /2r. Proof. By Theorem 5.17 we know that the optimal ecay rate for the basis pair is N 1, therefore if ρ is optimal we must have, for some constant C 1 > 0, the boun sup g, λ 1 σ(n) 2 C 1 N 1, N N. g Bsep Next since σ is semihyperbolic we also know that, by Lemma 5.15, there is a constant C 2 > 0 such that Consequently we euce, H,r (σ(n)) C 2 N r/, N N. sup g, λ 1 σ(n) 2 C 1 N 1 C 1 C /r 2 H /r,r (σ(n)), N N. g Bsep sup g B sep g, λ 1 (n) 2 C 1 C /r 2 ( maxi1,...,i r {1,...,} r j=1 max(n i j, 1) ), /r n Z i 1<...<i r. (5.30) Letting g = Ψ s J,0 where s = {0,..., 0} an n = (k, 0,..., 0) for k Z we see that (5.30) becomes ɛ 2 J Fφ(2 J ɛk) 2 C 1C /r 2, k Z. (5.31) max( k, 1) /r Since the scaling function φ has compact support in [ p + 1, p] an ɛ I J,p, φ J,0 can be viewe as a function on R/Z an (5.31) escribes a boun on the Fourier coefficients of φ. Formally, if we write ϕ(x) := 18 R/Z enotes the unit circle which we write as [0, 1) with the quotient topology inuce by M : R [0, 1), M(x) = x(mo 1). 19 Using f (k) = (2πik) 1 ˆf(k). 27

28 φ(2 J ɛ 1 (x 1/2)), then since ɛ I J,p we have that ϕ is supporte in [0, 1] an (5.31) becomes, for some constant D(ɛ, J, p) > 0: Fϕ(k) 2 = ϕ(k) 2 D, k Z. max( k, 1) /r If φ C 0 then the result follows from Theorem If φ / C 0, i.e. φ correspons to a Haar wavelet basis, then (5.31) cannot hol with /2r > 1 as this woul contraict (5.21). Corollary Let the scaling function φ corresponing to the Daubechies wavelet basis Bsep be fixe. Then for every orer r N, there exists a imension N, > r such that for all, we have that a semihyperbolic orering σ of orer r in imensions is such that ρ = λ 1 σ is not optimal for the basis pair (Bf (ɛ), B sep). Proof. If this result was not true then we woul euce by Theorem 5.19 that the wavelet φ satisfies φ C, which is a contraiction because no compactly supporte wavelet can be infinitely smooth [16, Thm. 3.8]. 5.7 Hierarchy of Semihyperbolic Orerings One other notable point from Theorem 5.17 is that we can have multiple values of r such that if σ is semihyperbolic of orer r in imensions then ρ = λ 1 σ is optimal for the basis pair (B f, B sep), so which one shoul we choose? We know that in the case of sufficient smoothness linear orerings are strongly optimal an therefore this suggests that the lower the orer r the stronger the optimality result. We now seek to prove this conjecture. Lemma Let r, r, N, r r. Then for all n Z we have that H,r r (n) Hr,r (n). Proof. Let n Z be fixe. For each j = 1,.. let i j enote the jth largest terms of the form max( n ij, 1). Observe that r r H,r (n) = max( n ij, 1), H,r (n) = max( n ij, 1), j=1 Hr,r (n) H,r r (n) = j=1 r j=1 max( n i j, 1) r r r j=r+1 max( n i j, 1). r Finally we observe that the numerator an enominator have the same number (r(r r)) of terms in the prouct an that each term in the numerator is greater than each term in the enominator, proving the inequality. Corollary Let r, r, N, r r < an σ, σ be semihyperbolic of orers r, r in imensions respectively. If ρ = λ 1 σ is optimal for the basis pair (Bf (ɛ), B sep) then so is ρ = λ 1 σ. Proof. Recalling (5.30) we know that there is a constant C > 0 such that sup g, λ 1 σ(n) 2 C H /r,r (σ(n)), N N, g Bsep sup g, λ 1 σ (N) 2 C H /r,r (σ (N)), N N, g Bsep where we have use Lemma 5.21 on the secon line. We then apply Lemma 5.15 to euce the result. Remark 5.5 Corollary 5.22 tells us that if there are several orers r that give us optimality then the smallest r possible, say r, is the strongest result. 28

29 Figure 7: 3D Fourier - Separable Wavelet Incoherence Isosurface Plots (a) Daubechies4 - Isosurface Value = (b) Daubechies8 - Isosurface Value = (c) Haar - Isosurface Value = () 3D slice of 5D Daubechies4 - Isosurface Value = We raw the isosurface plots over the subset { 50, 49,..., 49, 50} 3 Z 3. These pictures shoul be compare with the orering plots in Figure 6. Notice that for the smoother wavelets in (a) & (b), the growth matches that of a linear orering however the 3D Haar case lacks this smoothness, resulting in semi-hyperbolic scaling in (c). If we keep the wavelet basis fixe an let the imension increase, the scaling becomes increasingly hyperbolic, as seen in () an prove in Corollary D Separable Incoherences We have foun optimal orerings for every multiimensional Fourier- separable wavelet case however, we have not shown that (apart from in the linear case with sufficient Fourier ecay) that the orering is strongly optimal an we have not characterize the ecay. Therefore it is of interest to see how the incoherence scales in further etail by irectly imaging them in 3D. We o this by rawing levels sets in Z 3, as seen in Figure 7. 6 Asymptotic Incoherence an Compresse Sensing in Levels We now return to the original compresse sensing problem which was escribe in the introuction of this paper an aim to stuy how asymptotic incoherence can influence the ability to subsample effectively. We shall be working exclusively in 2D for this section. Consier the problem of reconstructing a function f L 2 ([ 1, 1] 2 ) from its samples { f, g : g 29

30 Bf 2(2 1 )}. The function f is reconstructe as follows: Let U := [(Bf 2(2 1 ), ρ), (B 2, τ)] for some orerings ρ, τ an a reconstruction basis B 2. The number 2 1 is present here to ensure the span of B f contains L 2 ([ 1, 1] 2 ). Next let Ω N enote the set of subsamples from Bf 2(2 1 ) (inexe by ρ), P Ω the projection operator onto Ω an ˆf := ( f, ρ(m) ) m N. We then attempt to approximate f by n=1 x nτ(n) where x l 1 (N) solves the optimisation problem min x 1 subject to P Ω Ux = P Ω ˆf. (6.1) x l 1 (N) Figure 8: Simple Resolution Phantom Experiment (a) Rasterize Phantom Resolution = (b) Reconstruction from pattern A L 1 error = (c) Reconstruction from pattern B L 1 error = () Rasterize Phantom - Closeup (e) Closeup of (b) (f) Closeup of (c) (g) Sampling Pattern A Number of Samples: (h) Sampling Pattern B Number of Samples: Samples are from the subset { 200, 199,..., 199, 200} 2 Z 2. Notice that the checkerboar feature are capture by the levele sampling pattern but not by pattern (a), even though it uses fewer samples. Reconstructions are at a resolution of Since the optimisation problem is infinite imensional we cannot solve it numerically so instea we procee as in [1] an truncate the problem, approximating f by R n=1 x nτ(n) (for R N large) where x = ( x n ) R n=1 now solves the optimisation problem min x C R x 1 subject to P Ω UP R x = P Ω ˆf. (6.2) 30

31 We shall be using the SPGL1 package [4] to solve (6.2) numerically. 6.1 Demonstrating the Benefits of Multilevel Subsampling We shall first emonstrate irectly how subsampling in levels is beneficial in situations with asymptotic incoherence (µ(q N U) 0) but poor global incoherence (µ(u) is relatively large). The image f that we will attempt to reconstruct is mae up of regions efine by Bezier curves with one egree of smoothness, as in [15]. This image is intene is moel a resolution phantom 20 which is often use to calibrate MRI evices [13]. A rasterization of this phantom is provie in image (a) of Figure 8. We reconstruct with 2D separable Haar wavelets, orere accoring to its resolution levels, from a base level of 0 up to a highest resolution level of 8. The Fourier basis is orere by the linear consistency function H 2,1, which gives us a square leveling structure when viewe in Z 2. We choose these orerings because we know that they are both strongly optimal for the corresponing bases, an therefore shoul allow reasonable egrees of subsampling when given an (asymptotically) sparse problem. By looking at Figure 8, we observe that subsampling in levels (pattern (b)) allows to pick up features that woul be otherwise impossible from a irect linear reconstruction from the first number of samples (pattern (a)) an moreover the L 1 error is smaller. 6.2 Tensor vs Separable - Fining a Fair Comparison We woul like to stuy how ifferent asymptotic incoherence behaviours can impact how well one can subsample. In 2D it woul be unwise to compare 2 ifferent separable wavelet bases, since we know that they have the same optimal orerings an ecay rates in 2D (see Corollary 5.11). Therefore we are left with comparing a separable wavelet basis to a tensor basis. The incoherence ecay rates for the 2D Haar cases are shown in the table below for Linear an Hyperbolic orerings of the Fourier basis B 2 f : 2D Haar Basis Incoherence Decay Rates Orering Tensor Separable Linear N 1/2 N 1 Hyperbolic log(n + 1) N 1 log(n + 1) N 1 Table 1: The ecay rates for the hyperbolic case comes from Theorem 4.10 an Proposition 5.6. For the linear case, the separable result comes from Theorem 5.17 an the tensor result can be euce from Lemma 5.9 applie to (4.18), although we o not provie the etails here. Observe that for linear orerings, there is a large iscrepancy between the ecay rates, however they are the same for hyperbolic orerings. Therefore, comparing separable an tensor reconstructions appears to be a goo metho for testing the behaviour of iffering spees of asymptotic incoherence. However, there is one serious problem, namely the choice of image f that we woul like to reconstruct. Recall from (1.5) that the ability to subsample epens on both the coherence structure of the pair of bases an the sparsity structure of the function f we are trying to reconstruct. Ieally, to isolate the effect of asymptotic incoherence we woul like to choose an f that has the same sparsity structure in both a tensor an separable wavelet basis. If f was chosen to be the resolution phantom like before then the tensor wavelet approximation woul be a poor comparison to that of the separable wavelet reconstruction (ue to a poor resolution structure). Therefore we nee to choose a function that we expect to reconstruct well in tensor wavelets, for example a tensor prouct of one imensional functions. Such an example is provie by NMR spectroscopy [24, Eqn. (5.24)]. A 2D spectrum is sometimes moelle as a prouct of 1D Lorentzian functions: r f(x) = L 2,p(i),s(i) (x), x, p(i), s(i) R 2, L 2,p,s (x) = L p1,s 1 (x 1 ) L p2,s 2 (x 2 ), x, p, s R 2 s L p,s = s 2, x, p, s R. + (x p) 2 20 resolution here refers to resolving a signal from a MRI evice. (6.3) 31

32 Figure 9: Spectrum Moel an Full Sampling Reconstructions (a) Rasterize Spectrum (b) Separable Reconstruction L 1 error = (c) Tensor Reconstruction L 1 error = Reconstructions uses all samples from the subset { 200, 199,..., 199, 200} 2 Z 2. Images are at a resolution of Haar wavelets are use for tensor an separable cases. Observe that both reconstructions match the original very closely an have similar L 1 approximation errors. We consier a specific spectrum f of the above form. By looking at Figure 9 we observe that, without any subsampling from the subset { 200, 199,..., 199, 200} 2 Z 2, the tensor an separable Haar wavelet reconstructions have almost ientical L 1 errors, suggesting that this problem oes not bias either reconstruction basis. We orer the tensor an separable reconstruction bases using their corresponing level base orerings, which are efine in Lemma 4.13 an Definition 5.2 respectively. For separable wavelets we start at a base level of J = 0 an stop at level 8 (so we truncate at the first wavelet coefficients) an for tensor wavelets we start at level J = 0 an stop at level Figure 10: Sampling Patterns (a) Linear Sampling Pattern (b) Hyperbolic Sampling Pattern (Boxe in) Samples are from the subset { 200, 199,..., 199, 200} 2 Z 2. White inicates sample is taken. We are now going to test how well these two bases perform uner subsampling with ifferent orerings of Bf 2. Two subampling patterns, one base on a linear orering an another on a hyperbolic orering, are presente in Figure 10. Ieally the hyperbolic subsampling pattern woul not be restricte by the { 200, 199,..., 199, 200} 2 but this is numerically unfeasible. Let us first consier what happens when using pattern (a) (see Figure 11). Notice that the separable reconstruction performs far better than the tensor reconstruction an therefore is more tolerant to subsampling with a linear orering than the tensor case. This is unsurprising as the tensor problem suffers from noticeably large 1/ N incoherence when using a linear orering when compare to the 1/N separable ecay rate. Of course we shoul have fully consiere the sparsity of these two problems which also factor into the ability to subsample, however f was specifically chosen because it was sparse in the tensor basis an moreover we have seen that it provies a comparable reconstruction to the separable case when taking a full set of { 200, 199,..., 199, 200} 2 samples. Next we observe what happens when using the pattern (b) (Figure 12). There is now a stark contrast to the linear case, in that both separable an tensor cases provie very similar reconstructions an furthermore the 21 When the problem was truncate at higher wavelet resolutions the improvement in reconstruction quality was negligible. 32

33 Figure 11: Reconstructions from Linear Sampling Pattern (a) Separable Reconstruction L 1 error = (b) Separable Closeup (c) Tensor Reconstruction L 1 error = () Tensor Closeup L 1 errors are very close. This suggests that both problems have similar susceptibility to subsampling when using hyperbolic sampling, which is reflecte by their ientical rates of incoherence ecay with hyperbolic orerings. 7 Appenix Proof of Proposition 5.6: (5.11) applie to part 1.) of Lemma 2.6 shows that the ecay of µ(π N U) is boune above by 22 F hyp (σ(n)) = 1/H (σ(n)) 1/h (N), which gives us the upper boun for µ(q N U) since 1/h (N) is ecreasing. For the lower boun, we focus on terms of the form λ ρ(m) = (t,..., t) for some t N an we set, for a fixe q N s = (1,..., 1), j := ɛ log 2 t + q, where we assume for now that j J is satisfie. This gives us Ψ s j,0, ρ(m) 2 = ɛ 2 j Fψ(ɛ2 j t) (1+q) t Fψ(ɛ2 ( ɛ log 2 t +q) t) (1+q) t L2 q (using (5.7)). (7.1) Let m now be arbitrary with max( λ ρ(m) i, 1) = M 1 an let t = M 1/ + 1. Because ρ correspons to the hyperbolic cross there exists an m > m such that max( λ ρ(m ) i, 1) = t where λ ρ(m ) = (t,..., t). Notice that t E()M for some constant epenent on the imension. 22 for the efinitions of H, h see (4.4) an (4.16). 33

34 Figure 12: Reconstructions from Hyperbolic Sampling Pattern (a) Separable Reconstruction L 1 error = (b) Separable Closeup (c) Tensor Reconstruction L 1 error = () Tensor Closeup Furthermore, (7.1) hols for m = m if we have that j J, which is satisfie if m is sufficiently large. Therefore we euce by (7.1) that µ(q m U) Ψ s j,0, ρ(m ) (1+q) t L2 q 1 E2 (1+q)+1 M L2 q 1 = E2 (1+q) max( λ ρ(m) i, 1) L2 q This proves the lower boun. References C E2 (1+q) h (m) L2 q (using (4.17), C > 0 some constant). [1] B. Acock an A. C. Hansen. Generalize sampling an infinite-imensional compresse sensing. Technical report NA2011/02, DAMTP, University of Cambrige, [2] B. Acock, A. C. Hansen, C. Poon, an B. Roman. Breaking the coherence barrier: A new theory for compresse sensing. Preprint, [3] K. I. Babenko. On the approximation of functions perioic functions of several variables by trigonometric polynomials. Dokl. Aka. Nauk SSSR, 132: , [4] E. Berg an M. Frielaner. Probing the pareto frontier for basis pursuit solutions. SIAM Journal of Scientific Computing, 31(2): , [5] M. Billeter an V. Orekhov. Novel Sampling Approaches in Higher Dimensional NMR. Springer,