ABSTRACT. Title: Wavelet-space solution of the Poisson equation: An algorithm for use in particle-incell simulations

Transcription

1 ABSTRACT Name: Benjamin Sprague Department: Physics Title: Wavelet-space solution of the Poisson equation: An algorithm for use in particle-incell simulations Major: Physics Degree: Master of Science Approved by: Date: Thesis Director NORTHERN ILLINOIS UNIVERSITY

2 ABSTRACT A particle-in-cell (PIC) approach is useful for simulations involving large numbers of particles interacting via electric charge or gravitation. The interaction potential for such forces is described by the Poisson equation 2 U = Cρ, where C is some constant which establishes the sign and magnitude of the interaction. By depositing the particle charge distribution into cells and solving for the potential on the grid of cells, the solution of the Poisson equation is made more tractable for large numbers of particles. This work presents a wavelet-based algorithm for the solution of the Poisson equation in PIC simulations which exhibits better scaling properties than the traditional Green s function and FFT approach. This algorithm also provides unique possibilities for adaptive resolution PIC simulations and parallelization of the Poisson solver. The algorithm described in this work is derived in a self-contained manner from theory to implementation, with the intent of encouraging the adoption of this wavelet-space technique by researchers with no prior experience in wavelet techniques. The algorithm was implemented in a test program to verify correctness, and over 1000 test runs of the application were executed to examine the numerical accuracy and scaling properties of the algorithm. These tests demonstrate that the wavelet-space approach is a competitive algorithm for the solution of the Poisson equation in a particle-in-cell simulation.

3 NORTHERN ILLINOIS UNIVERSITY WAVELET-SPACE SOLUTION OF THE POISSON EQUATION: AN ALGORITHM FOR USE IN PARTICLE-IN-CELL SIMULATIONS A THESIS SUBMITTED TO THE GRADUATE SCHOOL IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE MASTER OF SCIENCE DEPARTMENT OF PHYSICS BY BENJAMIN SPRAGUE c 2008 Benjamin Sprague DEKALB, ILLINOIS AUGUST 2008

4 Certification: In accordance with departmental and Graduate School policies, this thesis is accepted in partial fulfillment of degree requirements. Thesis Director Date

5 ACKNOWLEDGEMENTS Work supported by the Office of Naval Research, Department of Defense, under contract N with Northern Illinois University and by the Department of Education under contract P116Z with Northern Illinois University.

6 TABLE OF CONTENTS Page LIST OF TABLES LIST OF FIGURES LIST OF APPENDICES vi vii ix Chapter 1 INTRODUCTION AND PROBLEM STATEMENT Problem Scale Improvement of Wavelet-Based Particle-in-Cell Algorithm Goals and Chapter Outline WEIGHTED RESIDUAL METHOD AND SOLVING PDE IN A BASIS Fourier Basis, Green s Functions, etc INTERPOLATING WAVELETS Transform and Inverse What Is the Wavelet Basis? SPECIAL TOPICS Refinement Relations Dual and Primal Wavelet Transform Orthonormal Wavelets Multidimensional Wavelets Continuous Real Space to Wavelet Space Human-Readable Representation of Vectors

7 v Chapter Page 5 THE ALGORITHM Prior Work Representation of Operators, Calculating Operators in Wavelet Space Non-Standard Operator D Operators Implementation of 3D Operators in Non-Standard Form Preconditioning and Temporal Coherence Multiresolution Grids Monopole Approximation for Boundary Conditions Parallelization Opportunities IMPLEMENTATION AND ALGORITHM TESTING Features and Code Structure Testing: Parameters and Measurements Testing: Demonstration of Solutions Testing: Effects of Radial Extent and Monopole BCs CONCLUSIONS Future Applications BIBLIOGRAPHY APPENDICES

8 LIST OF TABLES Table Page 1 Linear prediction filter Linear update filter Interpolating wavelet prediction filters Interpolating wavelet base filters Linear prediction filter on boundary Linear prediction filter on boundary, no S

9 LIST OF FIGURES Figure Page 1 Particle-in-cell algorithm Basic forward wavelet transform Sampled linear function Single-stage wavelet transform Basic inverse wavelet transform Recursive forward wavelet transform Fully transformed linear function with perfect prediction filter Transformed linear function zero BCs Transformed linear function periodic BCs Transformed linear function interpolation BCs Lifted wavelet transforms Wavelet functions of linear interpolating wavelets Wavelet functions of linear interpolating wavelets doubled resolution Wavelet functions higher order families Shape and frequency spectrum of linear interpolating wavelets Shape and frequency spectrum of 8th-order interpolating wavelets Wavelet transform example Standard form wavelet space operator Non-standard form wavelet space operator Pseudo code for non-standard form operator top scale Pseudo code for non-standard form operator child scales Results of preconditioning and temporal coherence

10 viii Figure Page 23 Multiresolution grid used for boundary conditions Code testing input functions Dipole decay with boundary layers Error vs. radial extent Number of points vs. radial extent Maple 10 code for generating operator base filters

11 LIST OF APPENDICES Appendix Page A WAVELET COEFFICIENTS AND OPERATOR COEFFICIENTS B DERIVATION OF WAVELET TRANSFORM IDENTITIES C INHERENT PRECONDITIONER IN A BIORTHOGONAL BASIS D A BRIEF PRIMER ON DIRAC NOTATION E INTERPOLATING WAVELETS ON AN INTERVAL

12 CHAPTER 1 INTRODUCTION AND PROBLEM STATEMENT Improving the detail of the N-body space charge calculation is a crucial step in increasing the fidelity of particle accelerator and galaxy dynamics simulations. For physical problems in which the Coulomb force is dominant, accurately approximating the Poisson equation 2 U = Cρ is essential in correctly evaluating the forces acting on each particle in the simulation. Several approaches have been taken for the solution of the Poisson equation in such N-body simulations. They may be classified as either integral or differential equation approaches. The differential forms directly solve the Poisson equation in differential form, while the integral forms formulate the Poisson equation as an integral equation and evaluate the summed effect of the charge distribution at each point in the computational domain. The brute-force N-body approach is an integral solution, useful for small N. Each particle simply sums the forces applied to it by each of the other particles. This approach is simple to implement and can scale well in parallel, but it scales poorly with the number of particles N. This naive approach to N-body simulation scales as O(N 2 ) for each time step, making it prohibitive for use with standard computer hardware for N much larger than An improved integral approach is obtained by evaluating nearby forces with a direct N-body approach, while approximating long-range forces by a multipole expansion. This tree approach also has good parallel scaling potential, and it scales significantly better with particle count. Typical tree codes scale as O(N log N) and are frequently used in galactic dynamics simulations.

13 2 For very large particle numbers, the Green s function plus FFT approach is often used [15]. This approach shares characteristics with both integral and differential solutions of the Poisson equation. The particle density is deposited on a grid of cells, and the potential is evaluated on this grid using the differential Poisson equation. This technique is often called a particle-in-cell approach. Because a Green s function is used to solve the equation, however, the solution proceeds using a convolution in a manner analogous to an integral solution. This approach always scales as O(N) with respect to the number of particles, because the individual particles are used only for carrying the dynamics information between time steps. It scales as O(N g log N g ) with respect to the number of grid points, with the advantage that the number of grid points is typically chosen to be an order of magnitude less than the particle count. The typical algorithm of a particle-in-cell code is shown in Figure 1. The particles are initially constructed with some distribution in space and momentum. They are advanced forward in time and then binned onto a computational grid. This binning process is very similar to constructing a 3D histogram of the charge density. Once this charge density function is obtained, the Poisson equation is solved on this grid, the forces are interpolated back to the particles, and the particles advance forward in time again. This loop is repeated for each time step of the simulation as the particles travel through the accelerator. This thesis is concerned only with the step of solving the Poisson equation to find the forces on the particles caused by their own charge distribution. By replacing the Poisson equation solver, an alternative particle-in-cell approach was developed by Terzić et al. in 2007, and it is a fully differential solution to the N-body problem [28]. It uses orthogonal wavelet bases defined over the computational grid to evaluate the differential equation, and an iterative conjugate gradient solver to find the solution. Because this algorithm uses an iterative solver, and information from the previous time step, it is adaptive in time and can very quickly find a solution if the charge density changes slowly with respect to the time step.

14 3 Particle-In-Cell Advance particles in time Bin particles to grid Interpolate force to particles Solve Poisson equation on grid Figure 1: Particle-in-cell algorithm. 3

15 4 1.1 Problem Scale In particle accelerator codes, the number of particles used is typically on the order of 10 5 or With particle counts in the millions, the brute force scaling of O(N 2 ) is simply impractical for most hardware. Particle-in-cell codes can manage these high particles counts because the particle deposition schemes are simply O(N). In addition, the particle management routines of particle-in-cell codes can operate very efficiently in parallel since the particles only interact through the grid. The grid solution for an FFT-based particlein-cell scales as O(N g log N g ) with approximately ten particles per cell. This means that grid sizes are typically on the order of 32 3 or In order to increase the number of particles in future simulations, this grid size needs to increase as well to maintain the optimum number of particles per cell. This requires an improvement in the scaling of the algorithms with respect to the number of grid points, as well as with respect to the number of parallel processors employed in the simulation. 1.2 Improvement of Wavelet-Based Particle-in-Cell Algorithm The initial wavelet-based Poisson solver for particle-in-cell applications constructed by Terzić et al. showed great promise for improving the speed of future accelerator codes and was competitive with the standard FFT plus Green s function algorithm for current typical problem sizes [28]. It used a preconditioner in wavelet space to ensure rapid convergence of the iterative solution and it took advantage of previous time step information to create an adaptive-in-time approach to the solution of Poisson s equation in a particlein-cell environment. This current work is a continuation of that effort, and is focused on overcoming the difficulties of this earlier algorithm in the effort to enhance the ability of a

16 5 wavelet-based algorithm to provide adaptive solutions which can produce additional detail for the particle-in-cell simulator. Specifically, the original wavelet-based solver utilized the standard form of wavelet operators, which are simple to implement but hide much of the inherent sparsity of the operator in the wavelet basis. Also, the scales are tightly coupled in this form, and it is not obvious how to separate them in order to construct a parallel algorithm. Second, this initial wavelet-based particle-in-cell algorithm had difficulty specifying boundary conditions in wavelet space. A set of Green s functions had to be evaluated at each time step, which was computationally expensive and difficult to arrange for efficient parallel execution. This caused the boundary conditions evaluation to consume a significant portion of the solver s execution time, even during those stages where not much had changed losing some of the benefits of the algorithm s otherwise adaptive-in-time nature. The goal of this current work is to construct an algorithm which retains the strengths of the work of Terzić et al.: adaptivity in time and fast solution of Poisson s equation, while improving these two key areas to produce an efficient algorithm which can be easily adapted for parallel execution. 1.3 Goals and Chapter Outline With the major foundational works in modern wavelet theory dating back to the early 1990 s [7], it is becoming increasingly less tenable to consider it a new mathematical technique merely on the premise of age. Also, many of the major ideas behind solving differential equations in wavelet bases have been well known for at least ten years [4, 5, 6, 11, 13, 14, 17], yet the slow adoption of these techniques for solving physical problems is cause to evaluate the reasons for such a hesitant response. In an effort to encourage

17 6 the adoption of these powerful techniques, this thesis is constructed with the following concepts in mind: Wavelet theory has many interrelated properties and techniques for manipulation of wavelets which must be mastered in order to be comfortable working with wavelet bases. Wavelets are a generalization of many different techniques, and it is easy to get lost in the (often irrelevant) details of the basis. With this in mind, this work will attempt to constrain the discussion to a minimum working set of required knowledge, keeping the focus on things required for the current application, and will identify important techniques as well as equations to enable the reader to grasp the concepts more thoroughly. Wavelet methods are computational in nature and are only useful when implemented in software. This thesis will attempt to make the results as repeatable as possible by including notes on the special considerations which arise when implementing a real wavelet algorithm filter boundary conditions and their effect, preserving a symmetric Laplacian operator, special cases when generalizing from 1D to 3D, etc. Wavelet techniques are often misconstrued as being new or magic. This work will attempt to help physicists understand the connections from wavelet theory to common mathematical concepts, and will derive equations and concepts from basic principles wherever possible rather than simply using equations from other sources without explaining their meaning.

18 Organization This work begins with a quick review of the basic mathematics involved in computationally solving partial differential equations in a basis set, and moves quickly into describing the basis of choice for this algorithm interpolating wavelets. In Chapter 4, several additional mathematical techniques are introduced which are essential to the understanding of the wavelet particle-in-cell Poisson equation algorithm. This chapter is followed by a chapter describing the algorithm in detail. Chapter 6 describes a test implementation of the algorithm which was used to verify the mathematics of the earlier chapters as well as to examine the feasibility of the new boundary condition schemes introduced in this thesis. The final chapter reviews the contributions of this thesis and examines possible directions for future development.

19 CHAPTER 2 WEIGHTED RESIDUAL METHOD AND SOLVING PDE IN A BASIS Approximating a differential equation with any basis function expansion must first begin with the initial continuous equation: 2 (u + u bc ) = ρ. (2.1) where u is the potential inside the computational domain and u bc is the potential on the outside. 1 Eqn. (2.1) can be reinterpreted as a requirement that the residual R would vanish everywhere: R = 2 (u + u bc ) ρ = 0. (2.2) If a u, u bc and ρ can be found which causes R to vanish in all of continuous space, then the problem is solved. When working with analytic functions for ρ, this can often be done by using the integral form of the equation. However, solving the problem for general ρ requires approximating with a basis function expansion. The particular form of the basis functions is irrelevant at this point, but the goal is to make the residual orthogonal to the space spanned by the basis functions. Specifically, we want φ i R = φ i, R = φ i ( x)r( x)d 3 x = 0, i. (2.3) Note the use of Dirac s Bra-Ket notation to represent the inner product, and its equivalence to the integral norm. I will continue to use the Dirac notation throughout this thesis. So, we want to reduce the residual to be orthogonal to every basis vector (function) in the 1 This separation in the potential will be explained in further detail in Section 5.8.

20 9 expansion space given by the φ i. In the limit of a complete basis, it can be seen that this restriction will be equivalent to the continuous requirement (Eqn. 2.2). Because the ultimate goal is to establish a relationship between u and ρ, we must next examine the residual in detail. Again, due to the inability of finite minds or computers to represent a continuum of values, we need to expand ρ, u and u bc in some set of functions in order to work with them. For our purposes, it is convenient if they all are expanded in the same set of basis functions: ρ = j ρ j Φ j, u = j u j Φ j, u bc = j u bc j Φ j. (2.4) Inserting these expressions into the requirement on R gives: φ i R = φ i 2 j (u j + u bc j )Φ j φ i j ρ j Φ j, = j (u j + u bc j ) φ i 2 Φ j j ρ j φ i Φ j = 0, i. (2.5) We now have an algebraic linear equation to solve, which is a well-researched problem in computational science. This will be more apparent if we write: ρ = ρ 0 ρ 1 ρ n, u = u 0 u 1 u n, u bc = u bc 0 u bc 1 u bc n, O := φ i Φ j, L := φ i 2 Φ j. (2.6) Our discretized differential equation becomes L( u + u bc ) = O ρ, (2.7)

21 10 and finding u becomes a problem of simply inverting the matrix L: u = L 1 (O ρ L u bc ). (2.8) This matrix inversion is handled readily by several common linear algebra algorithms. Note that choice of φ i and Φ i are arbitrary, and the general form of the resulting linear equation does not depend on their properties. However, in order for L to be invertible, it must be square, which requires that the number of basis functions be the same for both expansions. One common choice for basis functions is to pick φ i = δ( x α i), restricting the residual to be zero at discrete points. This is called a collocation method and is useful for finding a solution which is valid on a grid. It also often results in a less dense representation of the Laplacian operator matrix and is the approach taken by Goedecker [11]. Another common choice is to choose φ i = Φ i, and the resulting weighted residual method is known as the Galerkin method. The largest advantage of this approach is that the resulting Laplacian operator is a symmetric matrix, which is a requirement for many linear system solvers. Arias et al. use this approach and are able to use the quickly converging conjugate gradient method [1, 17], while Goedecker s approach limits him to variations of the slower steepest descent algorithm, although with a much more sparse operator [12]. 2.1 Fourier Basis, Green s Functions, etc. Another basis of academic interest for the weighted residual method is the Fourier basis. Using φ ω = Φ ω = e iωx results in a diagonal overlap matrix and diagonal Laplacian matrix and can therefore be trivially inverted. The downside of this approach lies in the

22 11 periodicity that is assumed in the expansion of functions into the Fourier basis. Unless the basis expansion is carried out to infinite frequencies, the expansion will result in a function which repeats periodically. The inherent periodicity of the Fourier basis usually makes it undesirable for use with the differential form of the Poisson equation due to the difficulty of defining spatially localized boundary conditions. Instead, the Poisson equation in integral form is used with a Green s function, and a pilgrimage through frequency space is used to turn the convolution integral into a simple algebraic expression [15]. This approach has the advantage of allowing arbitrary boundary conditions to be applied using the Green s function, yet only needing a computational grid of the size of the charge distribution of interest. The disadvantages of this approach is that in order to avoid periodic wrapping of the Green s function, the frequency-space calculation must be performed on a doubled grid, and it still requires a fully dense grid inside of the charge region.

23 CHAPTER 3 INTERPOLATING WAVELETS One of the main purposes of a wavelet basis is compact storage of general data such as music and photos and data files [27]. It is in this realm of data compression that the wavelet scheme is easiest to conceptualize. The goal of a basis-function compression scheme is to represent the input data with as few basis function coefficients as possible. This is generally best accomplished when the basis function set shares many similarities with the data (f( x)) to be compressed. A more mathematically formal way to state this is to say that φ i f is large for only a very few i, and zero for all others. One example of this is an infinitely periodic signal. In a delta-function (time sampling) basis set, this signal would be significantly dense, even if it were compressed by storing only a single period. However, in a Fourier series expansion, this signal may require only very few coefficients to accurately represent it. In the opposite extreme, a highly localized delta function signal is sparse in a time sampling space, but infinitely dense in Fourier space. This space / frequency localization dichotomy is of mathematical interest, but physical signals are nearly always localized in space, and often are significantly localized in frequency as well. There are many common methods of compactly representing signals in simple physical systems, often using energy or angular momentum eigenkets such as gauss-hermite polynomials and spherical harmonics. For the general case, however, none of these analytical basis sets have the full localization in space that is typical of physical quantities such as charge and mass distributions. This need for a basis set which compactly describes physical quantities in general is a major driving force in the success of wavelet methods.

24 Transform and Inverse Wavelets are most easily understood through the transform between discrete physical space and wavelet space. Interpolating wavelets are the main focus of this research, and their transform will be described in detail. This description will follow the lifting scheme wavelet description of Sweldens [26, 27], though there are many ways to represent a wavelet transform. This transform occurs after the signal has already been converted into a discrete sample of the physical system at some resolution level h, which can have a different value for each dimension considered. The goal of the transformation is to reduce the problem to the fewest number of large non-zero coefficients as possible by providing a way to recover the value of the original signal via some interpolation scheme. In 1D, the method taken to accomplish this is to first separate the even-indexed coefficients from the oddindexed coefficients (see Figure 2). Then the even-indexed coefficients (which will be called Smooth or S coefficients) are interpolated in some manner to make a prediction of what the odd-indexed coefficients (called Detail or D coefficients) should contain. This prediction is subtracted from the D coefficients, and the first stage of the wavelet transform is complete. s k s k+1 Sp P d k Figure 2: Basic forward wavelet transform.

25 14 As an example, consider a linear function as the input function sampled as in Figure 3, and a simple wavelet transform utilizing a linear interpolation / prediction scheme. 1 The prediction scheme would be simply to take any two adjacent S coefficients, take their average, and predict that the odd D coefficient between them should equal their average. This corresponds to the prediction filter shown in Table 1. The coordinates are such that a 0 offset refers to the S coefficient to the left of the current D coefficient, and a 1 offset refers to the next S coefficient, which is to the right of the current D coefficient. The prediction is then tested by subtracting the prediction from the actual value of the D coefficient. Of course, in the special case of a linear input function, this prediction scheme will be perfect, and all of the D coefficients will be zero as in Figure 4. This transform is completely invertible all that is required is to perform the prediction again from the S coefficients and add the prediction back on to the D coefficients, then merge the S and D coefficients back into one single data set. The block diagram for this operation is shown in Figure 5. With a perfect prediction filter, the goal to compress this input data set has now achieved a 2:1 compression ratio with no losses. Cutting the storage requirement in half is a good start, but with such a simple input function, it is not yet very impressive. Further compression is achieved by recursively splitting the remaining zero-level S 0 coefficients into -1 level S 1 and D 1 coefficients and predicting these D 1 coefficients as well, as Figure 6 shows. There are now two separate scales of information, and only the D coefficients are kept for the lower scale 2, while the S and D coefficients are kept at the highest scale. 1 A simpler wavelet transform is possible, using a 0th order constant-interpolating scheme, which results in an un-lifted Haar transform. 2 The numbering convention for levels is an unfortunate relic of wavelet literature. Up in level is Down in scale. This work will attempt to preserve the distinction between the two. Numeric values are always levels, while the terms higher, lower, up, down, large, small will always refer to scales. The author would have preferred to simply re-number levels to match scales and derive the self-consistent set of equations, but then the reader would be unable to connect with the vast majority of other wavelet literature. It would be analogous to re-labeling the electron charge as positive in an electronics text so that currents would make more intuitive sense.

26 15 10 Linear Input Function Wavelet Transform Output s 0 0 s 0 0 s 0 2 s 0 3 s 0 4 s 0 5 s 0 6 s 0 7 Figure 3: Sampled linear function. Table 1: Linear prediction filter Offset 0 1 Value

27 16 10 Linear Input Function Wavelet Transform Output s 0 0 d 0 0 s 0 1 d 0 1 s 0 2 d 0 2 s 0 3 d 0 3 Figure 4: Single-stage wavelet transform. Transform is of a linear function using a perfect prediction filter. s k P M s k+1 d k Figure 5: Basic inverse wavelet transform.

28 17 This recursive pyramid algorithm can continue until there is only one S n and D n point remaining, as long as the original data set contained a power of 2 number of points (see Figure 7). The naming convention for scales is such that increasing scale corresponds to decreasing resolution and decreasing number of S and D coefficients. Each scale has half as many coefficients, but they are positioned twice as far apart as the coefficients at the next lower scale. The frequency information contained in a scale is then half of that which is contained in the next lower scale. In fact, the wavelet transform stage can be considered as a high frequency / low frequency filter where the S coefficients are given the low frequency information and the D coefficients are given the high frequency data. The separation into scales is also a separation in frequency. s k-1 s k Sp P s k+1 Sp P d k-1 d k Figure 6: Recursive forward wavelet transform. The wavelet compression scheme will result in only one single non-zero coefficient (S n ) if the prediction scheme is perfect, as in Figure 7. This seems to violate the basic idea that it takes two constants to define a line. However, the perceptive reader may have noticed that up until this point the boundary conditions have been ignored. Because our filter simply took the average of two points on either side of the D coefficient, we have been

29 18 10 Linear Input Function Wavelet Transform Output s -2 0 d 0 0 d -1 0 d 0 1 d -2 0 d 0 2 d -1 1 d 0 3 Figure 7: Fully transformed linear function with perfect prediction filter.

30 19 assuming that the prediction scheme could correctly guess what the S coefficient values were for points after of the last D coefficient. For higher order interpolation schemes, this problem is more obvious, as a wider filter is used, and more S coefficients are taken into account that are often outside of the region of sampled data. There are many ways to handle boundary conditions with wavelet expansions, and the different types are appropriate for different uses. The simplest boundary condition scheme is so simply assume that everything outside of the region of interest is zero. This scheme is the easiest to implement, but often results in large D coefficients along boundaries due to the implied discontinuity between the measured non-zero data and the assumption of zero outside of the boundary (see Figure 8). Periodic boundary conditions often give the same boundary difficulties for non-periodic data sets, as Figure 9a shows. For periodic or symmetric data sets however, it is often advantageous in terms of compression to choose periodic or mirrored boundary conditions (Figure 9b). In the second generation wavelet scheme [26, 27], one can also define a prediction scheme which takes the finite interval into account. This can give a better general-case solution to the problem of data compression with wavelets (see Figure 10), but it makes the prediction filter dependent on the spatial coordinate rather than being uniform across the whole region of interest. See Appendix E for an example of defining the linear interpolating wavelets on an interval Lifting It is sometimes desirable to have the multiple scales of coefficients conserve some quantities, such as zeroth, first, second, or nth moments. In order to accomplish this, we need to update the sequence of S coefficients in some way in order to preserve these moments. This can be accomplished by adding an update stage to the transform after the prediction stage has occurred. As Figure 11 shows, this change is easily invertable

31 20 10 Linear Input Function Wavelet Transform Output s -2 0 d 0 0 d -1 0 d 0 1 d -2 0 d 0 2 d -1 1 d 0 3 (a) Linear input function. 3 Linear Input Function Wavelet Transform Output s -2 0 d 0 0 d -1 0 d 0 1 d -2 0 d 0 2 d -1 1 d 0 3 (b) Periodic linear function. Figure 8: Transformed linear function zero BCs.

32 21 10 Linear Input Function Wavelet Transform Output s -2 0 d 0 0 d -1 0 d 0 1 d -2 0 d 0 2 d -1 1 d 0 3 (a) Linear input function. 3 Linear Input Function Wavelet Transform Output s -2 0 d 0 0 d -1 0 d 0 1 d -2 0 d 0 2 d -1 1 d 0 3 (b) Periodic linear function. Figure 9: Transformed linear function periodic BCs.

33 22 10 Linear Input Function Wavelet Transform Output s -2 0 d 0 0 d -1 0 d 0 1 d -2 0 d 0 2 d -1 1 d 0 3 (a) Linear input function. 3 Linear Input Function Wavelet Transform Output s -2 0 d 0 0 d -1 0 d 0 1 d -2 0 d 0 2 d -1 1 d 0 3 (b) Periodic linear function. Figure 10: Transformed linear function interpolation BCs.

34 23 using the same scheme as before: mirror the diagram from left to right, and change the signs of the operations. The update stage is also known as a lifting stage, and there are techniques for defining lifting update stages to preserve multiple moments in between the scales of the transformation [26, 27]. For the simple case of linear interpolating wavelets, to conserve the zeroth moment (average), one defines the update filter as in Table 2. In this case, each D coefficient simply writes 1/4 of its value to the two S coefficients surrounding it. This again is subject to boundary condition issues, especially if the prediction filter changes near the boundaries. See Sweldens [26, 27] for more details on deriving update filters in the lifting scheme. s k s k s k+1 Sp P U U P M s k+1 d k d k (a) Forward transform. (b) Inverse transform. Figure 11: Lifted wavelet transforms. 3.2 What Is the Wavelet Basis? Recalling that the goal is to create a wavelet basis set, one may wonder why only the coefficients have been mentioned thus far. If this wavelet transform does convert from the

35 24 Table 2: Linear update filter Offset 0 1 Value Note: This filter is designed to preserve the average.

36 25 sampled set into another basis, then what do the basis functions look like, and what are their properties? Because the wavelet basis is in fact defined through the transform and inverse transform, an obvious way to examine the basis functions is to simply set only one coefficient equal to unity in the wavelet basis, and perform an inverse transformation in order to see what the function will become in coordinate space. In Figure 12, we can see several such functions drawn. The S coefficients will produce functions called scaling functions, which will be represented by φ k i (where k denotes the level3, and i is the index within that level), and the D coefficients produce functions called wavelet functions, denoted with ψ k i. While these functions appear to be only coarse discrete functions, by adding additional lower scales of zero coefficients, they can be interpolated to any resolution and are defined on any fractional value that can be represented by a computer (Figure 13). These scaling (φ k i ) and wavelet (ψk i ) functions behave as pseudo-continuous functions and are defined on any rational number that can be obtained as a binary fraction (0.d 1 d 2 d 3 d 4 = i d i2 i ). Therefore, we can treat them as functions of the coordinate space (φ k i (x) and ψk i (x)). Though they are not defined on the set of irrational numbers, they are defined on the rational numbers which are arbitrarily nearby. Since only rational numbers are definable in a finite computer representation, we may treat these functions φ k i (x) and ψk i (x) as continuous functions of x for any computational purposes. Because the shape of the individual wavelet functions does not depend on its scale or its index, we can write all of the functions φ k i (x) as translations and scalings of some base scaling function φ(x). Similarly, the wavelet functions are all translations and dilations of some function ψ(x) (often referred to as the Mother Wavlet ) [11]. Formally, this is 3 Note that k follows the numbering for levels rather than scales. See Footnote 2 on page 14 for a discussion of the difference.

37 26 1 Wavelet Coefficients 1 ϕ s 0 0 d 2 0 d 1 0 d 2 1 d 0 0 d 2 2 d 1 1 d Wavelet Coefficients 1 ψ s 0 0 d 2 0 d 1 0 d 2 1 d 0 0 d 2 2 d 1 1 d Wavelet Coefficients 1 ψ s 0 0 d 2 0 d 1 0 d 2 1 d 0 0 d 2 2 d 1 1 d Wavelet Coefficients 1 ψ s 0 0 d 2 0 d 1 0 d 2 1 d 0 0 d 2 2 d 1 1 d Figure 12: Wavelet functions of linear interpolating wavelets. Left column is input wavelet coefficients, and right column is the result of the inverse wavelet transform of those coefficients (with zero for the boundary conditions), showing the wavelet or scaling function associated with that coefficient.

38 27 1 Wavelet Coefficients 1 ϕ s 0 0 d 2 0 d 1 0 d 2 1 d 0 0 d 2 2 d 1 1 d Wavelet Coefficients 1 ψ s 0 0 d 2 0 d 1 0 d 2 1 d 0 0 d 2 2 d 1 1 d Wavelet Coefficients 1 ψ s 0 0 d 2 0 d 1 0 d 2 1 d 0 0 d 2 2 d 1 1 d Wavelet Coefficients 1 ψ s 0 0 d 2 0 d 1 0 d 2 1 d 0 0 d 2 2 d 1 1 d Figure 13: Wavelet functions of linear interpolating wavelets doubled resolution. This figure is identical to Figure 12, with the addition of another layer of d 3 i coefficients which are set to zero. The result is a double-resolution view of the same wavelet and scaling functions from Figure 12. This resolution doubling can be repeated to view the functions to any desired resolution.

39 28 expressed as φ k i (x) = φ(2 k x i), (3.1) ψ k i (x) = ψ(2 k x i). (3.2) Another unique attribute of these functions is that they are strictly limited in support. Beyond some finite range ±x, they are identically equal to zero. This is a feature which is not achievable with analytic functions constructed as a power series representation. Even the most rapidly decaying analytic function has some small non-zero tail region with infinite extent. The compact support of the wavelet functions is an advantage in two ways. First, it more accurately matches distributions of physical systems avoiding the introduction of spurious tail regions into the expansion, and allowing an inverse transform with perfect reconstruction. Second, it simplifies many operations in wavelet space, as wavelet functions which are far apart in real space have zero overlap and can have no direct interactions with each other through spatially local operators. Higher order wavelets, constructed with higher order interpolation schemes, have larger support and more complicated φ(x) and ψ(x) functions, but all wavelet schemes possess this property of compact support (Figure 14). So the wavelet basis set has achieved one of the goals demanded of it: localization in space. In addition, the fact that the φ k i and ψi k have some finite non-zero support rather than being a δ function implies, through the time-frequency uncertainty principle, that they may be localized in some frequency band as well. Note that because the wavelet function is not periodic, it will require an infinite frequency range to fully represent it, but if the wavelet function is a fairly smooth function (has a large number of continuous derivatives), the decay of Fourier coefficients toward infinity will be rapid. For n continuous derivatives of ψi k(x), the Fourier coefficients will decay as 1/ωn+1 at large ω [11]. This

40 29 1 ψ ψ (a) Linear interpolating wavelets. (b) 4th-order interpolating wavelets. 1 ψ ψ (c) 6th-order interpolating wavelets. (d) 8th-order interpolating wavelets. Figure 14: Wavelet functions higher order families. Notice that while the higher order functions do have wider support, their support is still finite.

41 30 can be seen in Figures 15 and 16 as the higher order and smoother functions decay much more rapidly in the higher frequency range. In the low-frequency regime, the Fourier components can be made to decay as well if certain constraints are met by the wavelet functions [11]. The Fourier transform of the wavelet function is given by Ψ(ω) = ψ(x)e iωx dx, (3.3) and at ω = 0, this becomes simply Ψ(ω = 0) = ψ(x)dx. So, the Fourier spectrum will decay down to zero at the origin if the wavelet function has a vanishing zero moment. Furthermore, expanding Ψ(ω) as a power series around the origin gives Ψ(ω) = l=0 ω l l! d l Ψ(ω) dω l (3.4) ω=0, where the derivatives d l Ψ(ω) are given by dω l ω=0 d l Ψ(ω) dω l = dl ω=0 dω l ψ(x)e iωx dx = ψ(x)( ix) l e iωx dx = ( i) l x l ψ(x)dx. ω=0, ω=0, (3.5) Therefore the series expansion says that if ψ(x) has m vanishing moments, then Ψ(ω) will vanish as ω m for ω 0. As Figures 15 and 16 show, the lifted wavelet functions with more vanishing moments are more localized in frequency near zero. Acquiring vanishing moments in the wavelet functions is accomplished through modifying the update stage of the wavelet transform. Notice that in the inverse transform of Figure 11b, the update stage can affect both even and odd coefficients, and can cause a change in the shape of the resulting ψ(x) function, because it depends on D coefficients

42 ψ k=4,i=5 (x) ψ k=5,i=20 (x) ψ k=6,i=50 (x) ψ k=4,i=5 (x) ψ k=5,i=20 (x) ψ k=6,i=50 (x) (a) (b) ψ k=4,i=5 (x) ψ k=5,i=20 (x) ψ k=6,i=50 (x) ψ k=4,i=5 (x) ψ k=5,i=20 (x) ψ k=6,i=50 (x) (c) (d) Figure 15: Shape and frequency spectrum of linear interpolating wavelets. (a) and (b) are linear interpolating wavelets, (c) and (d) are lifted linear interpolating wavelets with the first two moments vanishing.

43 ψ k=4,i=5 (x) ψ k=5,i=20 (x) ψ k=6,i=50 (x) ψ k=4,i=5 (x) ψ k=5,i=20 (x) ψ k=6,i=50 (x) (a) (b) ψ k=4,i=5 (x) ψ k=5,i=20 (x) ψ k=6,i=50 (x) ψ k=4,i=5 (x) ψ k=5,i=20 (x) ψ k=6,i=50 (x) (c) (d) Figure 16: Shape and frequency spectrum of 8th-order interpolating wavelets. (a) and (b) are 8th-order interpolating wavelets, (c) and (d) are lifted 8th-order interpolating wavelets with the first eight moments vanishing.

44 33 as input. Because the wavelet expansion is a complete basis which is separated into two spaces at each step, namely scaling function space (φ k i ) and wavelet function space (ψk i ), if one of these two spaces conserves a quantity, then the functions in the other space must be orthogonal to this quantity. For example, if the scaling function space is to conserve the average, then the wavelet functions must have a zero average because they are all orthogonal to the zeroth moment. So it is equivalent to say either that the wavelet function has been lifted so that it has a zero moment, or that the scaling function space has been lifted so that it conserves the this moment. This is another purpose of the lifting stage of the transformation to further localize the wavelet functions in frequency space by adding more vanishing moments to ψ(x).

45 CHAPTER 4 SPECIAL TOPICS This chapter will address several diverse topics and historical notation from wavelet theory. Much of this information, especially the refinement relations with the alternate h and g filter notation and the concept of a dual wavelet space, will be critical to the understanding of later chapters of this work. 4.1 Refinement Relations Running a single S coefficient through one stage of the inverse transform (Fig. 11b), one can see that any function φ k i is simply a linear combination of functions φk+1 j at the lower scale. 1 In the same way, the wavelet functions ψ k i can also be written as a combination of scaling functions φ k+1 j. This result can be expressed directly as a set of refinement relations between base functions (defined in Eqns. 3.1 and 3.2): φ(x) = j h j φ(2x j), (4.1) ψ(x) = j g j φ(2x j), (4.2) where the range of j depends on the size of the h and g filters. It is often useful to rephrase these equations in terms of relations between functions at adjacent levels. By 1 Again, level k + 1 is a lower scale than k. See Footnote 2 on page 14.

46 35 using Eqns. 3.1 and 3.2, the following can be obtained: φ k i (x) = j ψ k i (x) = j h j φ k+1 2i+j (x), (4.3) g j φ k+1 2i+j (x). (4.4) Wavelet families can be defined entirely by these refinement relations, and this notation is often used in literature [1, 7, 11]. These h and g filters can be derived, as mentioned, from the action of the prediction and update filters in the inverse wavelet transform. 2 The resulting expressions are (using P for the predict filter and U for the update filter) 3 : h 2i = δ i,0, (4.5) h 2i+1 = P i, g 2i = U i, g 2i+1 = δ i,0 j U ( j i) P j, where again, the index j runs over all of the non-zero values of U ( j i) P j, and U i and P i are defined to be zero outside of the size of their established filter range. This new notation allows us to compute integrals involving wavelet functions, which will be useful when defining the O and L matrices later. As an example, we can compute 2 While h and g can be derived from the prediction and update filters, it also possible to construct valid h and g filters which have no finite P and U filter representation. This is the case with orthogonal wavelets, for example [26]. 3 These relations were derived by following a delta-function sequence δ i,0 through either the S or D branch of the inverse transform, and observing what information came out as S coefficients on the lower scale. This is how the h and g coefficients of Eqns. 4.1 and 4.2 are defined.

47 36 the normalization condition for the scaling functions via 1 = φ(x)dx, = h j φ(2x j)dx, j = ( 1 h j 2 j = 1 h j, 2 j ) φ(x)dx, 2 = j h j. (4.6) In this fashion, we can often obtain relationships between the filter coefficients in the discrete set which are equivalent to the integral relationships in the continuous functions, and we can perform integrals over wavelet functions even though they are defined only by their refinement relationships Dual Space The wavelet bases which have been introduced up to this point are not orthogonal. In other words, the overlap matrix O := φ i Φ j from Chapter 2 contains possibly significant off-diagonal components. Because there are many cases where orthonormality is useful for simplifying expressions, we introduce another wavelet basis which is orthogonal (dual) to the first (primal) wavelet basis.

48 37 This new dual basis is composed of dual scaling ( φ k i ) and dual wavelet ( ψ k i ) functions derived from a base scaling function and mother wavelet φ k i (x) = 2 k φ(2 k x i), (4.7) ψ k i (x) = 2 k ψ(2 k x i), (4.8) which obey a set of refinement relations similar to that of the primal wavelets: φ(x) = 2 j h j φ(2x j), (4.9) ψ(x) = 2 j g j φ(2x j), (4.10) φ k i (x) = j ψ k i (x) = j h φk+1 j 2i+j (x), (4.11) g φk+1 j 2i+j (x). (4.12) Notice that there is now a four defining filters for the wavelet family: h, g, h, and g. The constraints between the two bases will cause only two of these to be independent. Also, the factors of 2 and 2 k in these relations will become apparent shortly. These two bases are required to satisfy the following biorthogonality constraints on a single level k: φ k i φ k j = δ i,j, φ k i ψj k = 0, ψ i k φ k j = 0, ψ i k ψj k = δ i,j,

49 38 and if these are satisfied, the refinement relations between bases guarantee an even stricter set of constraints between multiple levels 4 : φ k i φ k j = δ i,j, (4.13) φ k i ψj l = 0, k l, (4.14) ψ i k φ l j = 0, k l, (4.15) ψ i k ψj l = δ i,j δ k,l. (4.16) A few questions remain: First, why are the definitions and refinement relations for the dual functions different from those for the primal functions? Second, what are the relationships between h, g, h, and g, and how do we find the dual functions given only the primal functions? The answer to the first question lies in the orthonormality constraint on the scaling functions: δ i,j = φ k i (x) φ k j (x), = 2 k φ(2 k x i) φ(2 k x j), = 2 k φ(2 k x i)φ(2 k x j)dx, = 2 k φ(x i)φ(x j)(2 k )dx, (4.17) = φ 0 i (X) φ 0 i (X). Therefore the factor of 2 between scales of dual functions is just a normalization factor to ensure that φ k i (x) φk i (x) = 1 for all levels k. Note that because this factor is only a normalization factor, it could have been shared symmetrically between φ and φ as a 2 4 This is because any function φ l, φ l, φ l, or ψ l can be written as a sum of scaling functions φ l+1 or φ l+1 recursively at a lower scale until l = k and the above expressions for φ k i ψ k j = ψ k i φ k j = 0 can be invoked.