Computing Steerable Principal Components of a Large Set of Images and their Rotations

Transcription

1 Computing Steerable Principal Components of a Large Set of Images and their Rotations Colin Ponce and Amit Singer 1 Abstract We present here an efficient algorithm to compute the principal component analysis (PCA) of a large image set consisting of images and, for each image, the set of its uniform rotations in the plane. We do this by pointing out the bloc circulant structure of the covariance matrix and utilizing that structure to compute its eigenvectors. We also demonstrate the advantages of this algorithm over similar ones with numerical experiments. Although it is useful in many settings, we illustrate the specific application of the algorithm to the problem of cryo-electron microscopy. EDICS Category: TEC-PRC Image & Video Processing Techniques I. INTRODUCTION In image processing and computer vision applications, often one is not interested in the raw pixels of images used as input, but wishes to transform the input images into a representation that is meaningful to the application at hand. This usually comes with the added advantage of requiring less space to represent each image, effectively resulting in compression. Because less data is required to store each image, algorithms can often operate on images in this new representation more quicly. One common transformation is to project each image onto a linear subspace of the image space. This is typically done using Principal Component Analysis (PCA), also nown as the Karhunen-Loéve expansion. The PCA of a set of images produces the optimal linear approximation of these images in the sense that it minimizes the sum of squared reconstruction errors. As an added benefit, PCA often serves to reduce noise in a set of images. Colin Ponce is with the Department of Computer Science, Coell University, Ithaca, NY 1485 USA, cponce@cs.coell.edu Amit Singer is with the Department of Mathematics and PACM, Princeton University, Fine Hall, Washington Road, Princeton NJ USA, amits@math.princeton.edu

2 2 PCA is typically computed as the singular value decomposition (SVD) of the data matrix X of image vectors, or as the eigenvector decomposition of the covariance matrix C XX T. One then rans eigenvectors according to their eigenvalues and projects each image onto the linear subspace spanned by the top n eigenvectors. One can then represent each image with only n values, instead of the original number of pixel values. There are a number of applications in which each of an image s planar rotations are interesting as well as the original images. It is usually unreasonable due to both time and space constraints to replicate each image many times at different rotations and to compute the resulting covariance matrix for eigenvector decomposition. To cope with this, Teague, [19] and then Khotanzad and Lu [1], developed a means of creating rotation-invariant image approximations based on Zeie polynomials. While Zeie polynomials are not adaptive to the data, PCA produces an optimal data adaptive basis (in the least squares sense). This idea was first utilized in the 199 s, when in optics, Hilai and Rubinstein [8] developed a method of computing an invariant Karhunen-Loéve expansion, which they compared to the Zeie polynomials expansion. Independently, a similar method to produce an approximate set of steerable eels was developed by Perona [14] in the context of computer vision and machine leaing. Later Uenohara and Kanade [2] produced, and Par [12] corrected, an algorithm to efficiently compute the PCA of an image and its set of uniform rotations. Jogan, Zagar, and Leonardis [9] then developed an algorithm to compute the PCA of a set of images and their uniform rotations. The natural way to represent images when rotating them is to sample them on a polar grid consisting of N θ radial lines at evenly spaced angles and N r samples along each radial line. The primary advantage of [9] is that the running time of the algorithm increases nearly linearly with both the size of the image (N r below) and the number of rotations (N θ below). The disadvantage, however, is that the running time experiences cubic growth with respect to the number of images P in the set. The result is an algorithm with a running time of O(P 2 N r N θ log(n r N θ ) + N θ P 3 ), which is impractical for large sets of images. We present here an alteate algorithm that grows cubically with the size of the image, but only linearly with respect to the number of rotations and the number of images. Due to this running time, our algorithm is appropriate for computing the PCA of a large set of images with a running time of O(P N θ N r (N r + log(n θ )) + N θ Nr 3 ). Although this algorithm is generally useful in many applications, we discuss its application to cryoelectron microscopy [3]. In cryo-em, many copies of a molecule are embedded in a sheet of ice so thin that images of the molecule of interest are disjoint when viewing the sheet flat. The sheet is then imaged with an electron microscope, destroying the molecules in the process. The result is a set of projection

3 3 images of the molecule taen at unnown random orientations with extremely low signal-to-noise ratio. The cryo-em problem, then, is to reconstruct the three-dimensional structure of the molecule using thousands of these images as input. Because the images are so noisy, preprocessing must be done to remove noise from the images. The first step is usually to compute the PCA of the image set [1], [21], a step which both reduces noise and compresses the image representation. This is where our algorithm comes in, because in cryo-em every projection image is equally liely to appear in all possible in-plane rotations. After PCA, class averaging is usually performed to estimate which images are taen from similar viewing angles and to average them together in order to reduce the noise [3], [17], [22]. A popular method for obtaining class averages requires the computation of rotationally invariant distances between all P (P 1)/2 pairs of images [13]. The principal components we compute in this paper provide a way to accelerate this large-scale computation. This particular application will be discussed in a separate publication [18]. Other applications for computing the PCA of a large set of images with their uniform rotations include those discussed in [9]. A set of full-rotation panoramic images were taen by a robot at various points around a room. Those panoramic images and their uniform rotations were then processed using PCA. A new image taen at an unnown location in the room was then template matched against the set of images already taen to determine the location in the room from which the new image was taen. Because these images are liely to be somewhat noisy, it is important to perform PCA on them to be able to effectively template match against the training set. The rest of this paper is organized as follows. In Section II, we derive the principles behind the algorithms. In Section III, we describe an efficient implementation of our algorithm and discuss its space and computational complexity. In Section IV, we present a method for applying this algorithm to images sampled in rectangular arrays. In Section V, we present numerical experiments comparing our algorithm to other existing algorithms. Finally, in Section VI, we present an example of its use in cryo-em. II. DERIVATION Consider a set of P images represented in polar coordinates. Each image is conveniently stored as an N r N θ matrix, where the number of columns N θ is the number of radial lines, and the number of rows N r is the number of samples along each radial line. Note that N θ and N r must be the same for all images. Then each column represents a radial line beginning in the center of the image, and each row represents a circle at a given distance from the center. The first circle is at distance from the center, and so the center value is repeated N θ times as the first row of the matrix.

4 4 We can create a rotation of an image simply by rotating the order of the columns. Moving the leftmost column to the rightmost and cyclically shifting all other columns of the matrix, for example, represents a rotation of N θ. In this way we can generate all N θ rotations of each image. So let matrix I i,l represent the l th rotation of image i, i P 1, l N θ 1. Note that we may assume the average pixel value in each image is, for otherwise we subtract the mean pixel value from all pixels. Let J i,l be a vector created by row-major stacing of the entries of image I i,l. That is, J i,l N θm+n Ii,l m,n, m N r 1, n N θ 1. Then construct the N r N θ P N θ data matrix X by stacing all J i,l side-by-side to create X J, J P 1,Nθ 1 and set the covariance matrix C XX T. Note that C is of size N r N θ N r N θ. The principal components that we wish to compute are the eigenvectors of C. This can be computed directly through the eigenvector decomposition of C, or through the SVD of X. We will now detail an alteative and more efficient approach for computing these eigenvectors exploiting the bloc circulant structure of C. Consider C i, the covariance matrix given by just image i and its rotations, J i,... J i,nθ 1 : (1) Note that C i θ 1 J i,n J i,nt. (2) C P 1 i C i. (3) Now, let C i,α,β be an N θ N θ bloc of C i corresponding to the outer product of row α of I i,,..., I i,nθ 1 in J i,,..., J i,nθ 1 with row β of I i,,..., I i,nθ 1. We will now show that C i is of the form C i,, C i,,nr 1 C i....., (4) C i,nr 1, C i,nr 1,Nr 1 where each C i,α,β is an N θ N θ circulant matrix. Because image i and its rotations comprise a full rotation of row α and a full rotation of row β, θ 1 Cn i,α,β 1,n 2 θ 1 Iα,n i,n 1 I i,n β,n 2 I i, α,n 1+n Ii, β,n 2+n, (5)

5 5 where each row of I i, is considered a circular vector, so that all indices are taen modulo N θ. But, due to the circular property of the vector, for any integer c, N θ 1 C i,α,β n 1+c,n 2+c I i, α,n 1+c+n Ii, β,n 2+c+n Ci,α,β n 1,n 2. (6) This means that C i,α,β is a circulant matrix. But this analysis applies to any rows α and β. Therefore, C i consists of N r N r tiles of N θ N θ circulant matrices, as in equation 4. Now, the sum of circulant matrices is also a circulant matrix. Define C α,β as C α,β P 1 i C i,α,β. (7) Thus, C α,β is an N θ N θ circulant matrix. Then C is of the form C, C,Nr 1 C (8) C Nr 1, C Nr 1,Nr 1 A nown fact is that the eigenvectors V, V 1,..., V Nθ 1 of any N θ N θ circulant matrix A are the columns of the discrete Fourier transform matrix [5], given by V n e ın/nθ,, n N θ 1, (9) where ı 1. The first coordinate of any V is 1, so the eigenvalues λ, λ 1,..., λ Nθ 1 of a circulant matrix A, for which AV λ V, are given by λ (AV ) θ 1 n ı N A,n e θ Â(), N θ 1, (1) where A is the first row in matrix A, and  is its discrete Fourier transform. These eigenvalues are, in general, complex. It follows that for the matrix C there are N θ eigenvalues associated with each of its N 2 r circulant tiles. We denote these eigenvalues λ α,β θ 1 λ α,β satisfying C α,β V λ α,β V. Note that θ 1 C α,β n N,n e ı θ ( P 1 i C i,α,β,n ) P n ı N e θ λ i,α,β. (11) i

6 6 Now construct a vector W,q of length N r N θ by concatenating multiples c,q α times. That is, W,q c,q V. c,q N r 1 V These W,q vectors will be our eigenvectors for C. The exact values of c,q the index set for q, will be determined below. of the vector V N r, N θ 1. (12), c,q 1,..., c,q N r 1, as well as Consider the result of the matrix-vector multiplication CW,q. Along each tile-row of C (α α ), each circulant submatrix C α,β is multiplied by an eigenvector c,q β V. Thus, ( N r 1 β λ,β c,q β ) V CW,q. (. (13) N r 1 β λnr 1,β c,q β ) V So, requiring W,q to be an eigenvector of C satisfying CW,q µ,q W,q is equivalent to r 1 β λ α,β c,q β µ,q c,q α, for α, 1,..., N r 1. (14) But this in tu is equivalent to the eigenvector decomposition problem for the matrix λ, λ,nr 1 Λ (15) λ Nr 1, λ Nr 1,Nr 1 That is, the vector W,q is an eigenvector of C with eigenvalue µ,q if the vector c,q (c,q,..., c,q N r 1 )T is an eigenvector of the matrix Λ, satisfying Λ c,q µ,q c,q. Next, we show that the matrix Λ is Hermitian. To that end, note that the covariance matrix C is by definition symmetric, so column γ of the N θ N θ submatrix C α,β is equal to row γ of submatrix C β,α. Now, set γ. Then, by viewing the rows of the tile C α,β as circular vectors, the circulant property of the tiles implies that The eigenvalues of C β,α, then, are C β,α,n Cα,β n, Cα,β, n, for n,..., N θ 1. (16) θ 1 λ β,α (C β,α V ) N θ 1 C β,α,n n N e ı θ C α,β n N, n e ı θ, (17)

7 7 which, by the change of variables n n, becomes where λ α,β is real for all n 1, n 2. θ 1 λ β,α n C α,β,n n N eı θ λ α,β, (18) refers to the complex conjugate of λ α,β. Note that this property is true only because Cn α,β 1,n 2 Thus, Λ is a Hermitian matrix. As a result, all of Λ s eigenvalues are real. Furthermore, these eigenvalues are the µ,q so q, 1,..., N r 1 and the c,q are complex-valued. So, for some fixed, Λ has N r linearly independent eigenvectors that in tu give N r linearly independent eigenvectors W,q of C, through (12). Furthermore, the the vectors V, V 1,..., V Nθ 1 are linearly independent. From this it follows that the constructed eigenvectors W,q are linearly independent. Now, consider an eigenvector W,q, N θ 1, q N r 1 as an eigenimage. That is, consider the eigenimage as an N r N θ matrix R,q c,q V T. c,q N r 1 V T. (19) This is of the same form as the original images, in which each row is a circle and each column is a radial line, with the first row being the center of the image. The coefficients c,q r may write c,q r f,q (r)e ıϕ,q (r) for r N r 1. Therefore, R,q is of the form R,q (r, θ) f,q (r)e ıϕ,q (r) e ıθ for θ, /N θ,..., (N θ 1)/N θ, r, 1,..., N r 1. The real and imaginary parts of R,q are eigenimages, given by R,q c (r, θ) R(R,q (r, θ)) are complex, so we f,q (r)e ı( θ+ϕ,q (r)), (2) f,q (r) cos ( θ + ϕ,q (r)), (21) R,q s (r, θ) I(R,q (r, θ)) for r, 1,..., N r 1, θ, /N θ,..., (N θ 1)/N θ. f,q (r) sin ( θ + ϕ,q (r)), (22)

8 8 Algorithm Computational Complexity Ours O(P N θ N r(n r + log(n θ )) + N θ Nr 3 ) JZL O(P 2 N rn θ log(n rn θ ) + N θ P 3 ) SVD min{n rp 2 Nθ 3, P Nr 2 Nθ 3 } Eig. Decomp. P N rnθ 2 + (N rn θ ) 3 TABLE I COMPARISON OF COMPUTATIONAL COMPLEXITIES OF DIFFERENT PCA ALGORITHMS. Thus, each eigenimage R,q produces two eigenimages R,q c Nθ 1 λ α,β and Rs,q. It may appear as though this gives us 2N r N θ eigenimages, which is too many. However, consider equation (11) for V N 1 V : ( N C α,β ( )n θ 1 P 1 ) N,n e ı θ C i,α,β,n e ı n N θ Therefore, Λ Λ, Thus, i P 1 λ i,α,β i λ α,β. (23) Λ c,q Λ c,q µ,q c,q. (24) But because Λ is Hermitian, µ,q is real, so µ,q µ,q. Therefore, the eigenvalues of Λ are the same as those for Λ, and the eigenvectors of Λ are the complex conjugates of the eigenvectors of Λ. Note also that if or Nθ 2, which are also the values of such that Λ has real eigenvectors. Therefore, each eigenvector from or Nθ 2 contributes one real eigenimage, while eigenvectors from every other value of each contribute two. Thus, linear combinations of the eigenvectors of the first Nθ Λ matrices results in N r N θ eigenimages. Note also that when N θ is odd, eigenvectors for each contribute one real eigenimage, while eigenvectors for every other each contribute two. Thus, counting eigenimages for, 1,..., Nθ 1 2 produces N r N θ eigenimages, as required. III. IMPLEMENTATION AND COMPUTATIONAL COMPLEXITY The first step in this algorithm is to compute the covariance matrix C i of each image I i and its rotations. However, it is not necessary to compute the actual covariance matrix. Instead, we need only compute the eigenvalues λ i,α,β of each circulant tile C i,α,β of the covariance matrix. From equations (1) and (11) it follows that the eigenvalues are equal to the discrete Fourier transform of the top row of a tile C i,α,β.

9 9 Note that, by equation (5), N θ 1 C i,α,β,m I i, α,ni i, β,m+n, (25) where m,..., N θ 1. Now, the convolution theorem for cross-correlations states that, for two sequences f and g, if f g indicates the cross-correlation of f and g, then Therefore, the eigenvalues λ i,α,β λ i,α,β f g ˆf ĝ. (26) can be computed efficiently as ( Nθ 1 ) ( Nθ 1 Iα,ne i, n ı N θ Îi, ) I i, n N β,ne ı θ α () Îi, (). (27) β This can be computed efficiently with the Fast Fourier Transform. In addition, because the covariance matrix C i is symmetric, λ i,α,β and sum so that Λ is as defined in (15). λ i,β,α. We can in this way compute N θ different N r N r matrices Λ i, Λ P 1 i Λ i, (28) Note that in some implementations, such as in MATLAB, complex eigenvectors of unit norm are always retued. As a result, real eigenvectors, which occurs when or Nθ 2, will have twice the norm of the other eigenvectors. One must therefore normalize these eigenvectors by dividing them by 2. Computation of the Fourier transforms Îi, α can be performed in time O(N θ log(n θ )). Because we do this for every α, computation of the necessary Fourier transforms taes total time O(N r N θ log(n θ )). In addition, it uses space O(N θ N r ). Computation of a matrix Λ has computational complexity O(N 2 r ), but because we compute N θ of them, the overall time of O(N θ N 2 r ). In addition, the space complexity of this step is O(N θ N 2 r ). Each of these steps must be repeated for each image i, and then the P matrices Λ i,, (i,..., P 1) must be summed together for each image. Thus the computational complexity up to this point is O(P N θ N r (N r + log(n θ ))) and the space complexity is O(N θ N 2 r ). Note that the computation of Λ for each image i is completely independent of each other image. Thus, these computations can be performed in parallel to further improve performance. Such a parallel implementation requires space O(P N θ N 2 r ).

10 1 At this point we need to perform eigenvector decomposition on each Λ to determine the eigenvectors of C. The computational complexity of this step is O(N θ Nr 3 ) and the space complexity is O(P N θ N r ). Thus, the computational complexity of the entire algorithm is, O(P N θ N r (N r + log(n θ )) + N θ Nr 3 ), (29) and the total space complexity is max(n θ Nr 2, P N θ N r ). (3) The algorithm is summarized in Algorithm 1, and a comparison of the computational complexity of this algorithm with others is shown in Table I. IV. APPLICATION TO RECTANGULAR IMAGES Although this algorithm is based on a polar representation of images, it can be implemented efficiently on images represented in the standard Cartesian manner efficiently and without loss of precision using the two dimensional polar Fourier transform and the Radon transform. The two dimensional Radon transform of a function I(x, y) along a line τ θ through the origin is given by the projection of I(x, y) along the direction τ θ [11]. The Radon transform is defined on the space of lines τ θ in R 2, so we may write R(t, τ θ ) τ θ I(x, y) ds I(x, y)δ(x cos(θ) + y sin(θ) t) dxdy (31) The Fourier projection-slice theorem states that the one-dimensional Fourier transform of the projection of an image along τ θ along τ θ is equal to the slice of the two dimensional Fourier transform of the image taen [11]. To see this, let r R 2 be any point on the plane, and let r u + s, u τ θ, s τ θ (32) Also, tae w τθ. Note that r w u w, since s w. Then ˆR θ (w) I(u + s) ds e ıu w du τθ τ θ I(u + s)e ıu w dsdu τθ τ θ I(r)e ır w dr R 2 Î(w) (33)

11 11 Algorithm 1 PCA of a Set of Images and Their Rotations Require: P image matrices I,, I 1,,..., I P 1, of size N r N θ in polar form. 1: Set λ α,β, α, β,..., N r 1,,..., N θ 1 2: for i,..., P 1 do 3: Compute the Fourier transforms Îi, α () for α,..., N r 1,,..., N θ 1. 4: for α, β,..., N r 1 do 5: Compute the eigenvalues λ i,α,β of C i,α,β using (27). 6: end for 7: Update Λ Λ + Λ i, for,..., N θ 1 as in (28). 8: end for 9: for,..., Nθ 2 do 1: Compute the eigenvector decomposition of matrix Λ to generate eigenvectors c,q and eigenvalues µ,q for q,..., N r 1. 11: for q,..., N r 1 do 12: Construct eigenvector W,q as in (12), using eigenvector c,q of Λ. 13: end for 14: end for 15: for,..., Nθ 2, q,..., N r 1 do 16: Rearrange W,q as an image matrix as in (19) and compute its real and imaginary parts Rc,q Rs,q. 17: If or Nθ 2, normalize eigenimages if necessary by dividing by 2. 18: end for 19: retu R,q c and R,q s as in (21) and (22). and The Fourier projection-slice theorem provides an accurate way for computing the Radon transform by sampling the image s two dimensional Fourier transform on a polar grid and then taing inverse one-dimensional Fourier transforms along lines through the origin. Because the two dimensional Fourier transform is discrete, performing the normal two dimensional FFT and interpolating it to a polar grid leads to approximation errors. Instead, we use a nonequally-spaced FFT described in [2], [6]. This step has a computational complexity of O((mL) 2 log(l) + N r N θ log(n r N θ ) log( 1 ɛ )), where L is the number of pixels on the side of a square true image, N r is the number of pixels in a radial line in polar form,

12 12 N θ is the number of radial lines, m is an oversampling factor used in the nonequally-spaced FFT, and ɛ is the required precision. 1 The Fourier transform is a unitary transformation by Parseval s theorem. Therefore, the principal components of the normal two-dimensional FFT ed images are simply the two-dimensional FFT of the principal component images. The two-dimensional polar Fourier transform, however, is not a unitary transformation, because the frequencies are nonequally spaced: they are sampled on concentric circles of N r equally spaced radii, such that every circle has a constant number N θ of samples instead of being proportional to its circumference, or equivalently, to its radius. In order to mae it into a unitary transformation, we have to multiply the two-dimensional Fourier transform Î( w cos θ, w sin θ) of image I by w. In other words, all images are filtered with the radially symmetric wedge filter w, or equivalently, are convolved with the inverse two-dimensional Fourier transform of this filter. After multiplying the images two-dimensional polar Fourier transforms by w, we tae the onedimensional inverse Fourier transform of every line that passes through the origin. From the Fourier projection-slice theorem, these one-dimensional inverse Fourier transforms are equal to the line projections of the convolved images. That is, the collection of one-dimensional inverse Fourier transforms of a given w -filtered image, is the two-dimensional Radon transform of the filtered image. These two-dimensional Radon transformed filtered images are real valued and are given in polar form that can be organized into matrices as described at the beginning of Section II. We can then utilize the algorithm described above to compute the principal components of such images. Moreover, since the transformation from the original images to the Radon transformed filtered images is unitary, we are guaranteed that the computed principal components are simply the Radon transform of the filtered principal components images. After computing all of the polar Radon principal components we desire, we must convert them bac to rectangular coordinates. We cannot simply invert the Radon transform because the pixels of each eigenimage are not uniformly sampled, maing the problem ill-formed. However, for the inversion we can utilize the special form of the Radon eigenimages R,q c and R,q s from (21) and (22), given by R,q c (r, θ) f,q (r) cos ( θ + ϕ,q (r)) (34) R,q s (r, θ) f,q (r) sin ( θ + ϕ,q (r)) (35) 1 We would lie to than Sholnisy for sharing with us his code for the two dimensional polar Fourier transform. In his code, m 2 and ɛ is single precision accuracy.

13 13 To compute the inverse Radon transform of such functions we utilize the projection-slice theorem. In principle, this is done by first computing the Fourier transform of each radial line, dividing by w, and then taing the 2-D inverse Fourier transform of that polar Fourier image to produce the true eigenimage. Let us first compute the 2-D polar Fourier transform by taing one-dimensional discrete Fourier transform along each radial line of the function in (34). Function (34) is only defined for r, but we must compute the DFT along an entire line through the origin. We must therefore expand this definition to allow for any real r. Let g,θ c (x) Rc,q ) if x Rc,q + π) if x < for some fixed θ. Note that we should really denote this function gc,q,θ, but since q is fixed throughout this section, we drop it for convenience of notation. Then, f,q (x) cos ( θ + ϕ,q (x)) if x gc,θ (x) f,q ( x) cos ( (θ + π) + ϕ,q (x)) if x < Define g,θ s similarly for Rs,q. Note that g,θ c and g,θ s functions if is odd. The discrete Fourier transform can then be defined as where r w. Note that F,θ c (r) F,θ c ( r) r 1 r 1 (36) (37) are even functions if is even, and odd g,θ c (n)e ı, (38) g,θ c (n)e ı F,θ c (r) (39) because gc,θ (n) is real. Furthermore, with the change of variables n n, we find that, if is even, and if is odd, F,θ c ( r) F,θ c ( r) Therefore, if is even, then F,θ c and odd. r 1 n (N r 1) r 1 n (N r 1) g,θ c ( n )e i F,θ c (r) (4) g,θ c ( n )e i F,θ c (r) (41) (r) is real and even, and if is odd, then Fc,θ (r) is pure imaginary Then the 2-D Fourier transform of the true image can be defined as (A proof of this formula can be found in Appendix A.) F c (r, θ) F,θ c,, (r) cos ( θ)fc (r) sin ( θ)fs (r) (42)

14 14 and Fs (r, θ) Fs,θ (r) sin ( θ)fc,, (r) + cos ( θ)fs (r) (43) As shown above, these functions are either real or pure imaginary, depending on. Next we will tae the 2-D inverse Fourier transform of 1 r F c (r, θ) (where r w and 1 r is the filter) to obtain true eigenimages Ic and I s. Define H c as the order Hanel transform of r 1 Fc, (r): H c(s) and define Hs as the order Hanel transform of r 1 Fs, (r): where J is the Bessel function. H s (s) r 1 r F, c (r)j (rs) dr (44) r 1 r F, s (r)j (rs) dr, (45) Then the true real eigenimages can be described (in polar form) with the two equations (A proof of this formula can be found in Appendix B.) I1 (s, α) ( β() cos I2 (s, α) ( β() sin ((α + π 2 ) ) H c(s) + sin ((α + π 2 ) ) H c(s) cos ((α + π ) ) 2 ) Hs (s) ((α + π ) ) 2 ) Hs (s) (46) (47) where β() 1 2 ( ( 1 + ( 1) + ı 1 ( 1) )), (48) so that β() 1 when is even, and β() ı when is odd. In practice, Hc and H s are computed using standard numerical procedures [7], [4], [15] and from the polar form (46)-(47) it is a straightforward manner to obtain the images values on the Cartesian grid. V. NUMERICAL EXPERIMENTS We performed numerical experiments to test the speed and accuracy of our algorithm against similar algorithms. We performed this test on a UNIX environment with four processors, each of which was a Dual Core AMD Opteron(tm) Processor 275 running at 2.2 GHz. These experiments were performed in MATLAB with randomly generated pixel data in each image. We did this because the acutal output of the algorithms are not important in measuring runtime, and runtime is not affected by the content of the images. The images are assumed to be on a polar grid as described at the beginning on Section II.

15 15 P This Paper JZL SVD Eig. Decomp N/A N/A N/A N/A N/A N/A N/A N/A TABLE II COMPARISON OF RUNNING TIMES UNDER INCREASING NUMBER OF IMAGES P, IN SECONDS. N r 1, N θ 72. We compared our algorithm against three other algorithms: 1) The algorithm described in [9], which we will refer to as JZL, 2) SVD of the data matrix, and 3) computation of the covariance matrix C and then eigenvector decomposition of C. Note that our algorithm as well as JZL compute the PCA of each image and its planar rotations, while SVD and eigenvector decomposition do not tae rotations into account and are therefore not as accurate. In Table II one can see a comparison of the running times of various algorithms. In this experiment, images of size N r 1 and N θ 72 were used. For image counts P of at least 64 and less than 12288, our algorithm is fastest, at which point eigenvector decomposition becomes faster. However, the goal of this paper is to wor with large numbers of images, so our algorithm is not designed to compete for very small P. In addition, eigenvector decomposition does not tae planar rotations into account. Thus, eigenvector decomposition only computes the principal components of P images, whereas our algorithm

16 16 N r This Paper JZL SVD Eig. Decomp N/A N/A N/A TABLE III COMPARISON OF RUNNING TIMES UNDER INCREASING NUMBER OF PIXELS PER RADIAL LINE N r, IN SECONDS. P 5 AND N θ 72. actually computes the principal components of P N θ images. In addition, the calculation of the covariance matrix was made faster because MATLAB maes use of parallel processors to compute matrix products. In Table III one can see a comparison of the running times of various algorithms as P 5 and N θ 72 are held constant, and N r is increased. Even though the running time of our algorithm shows cubic growth here, it is still far faster than either JZL or eigenvector decomposition. Singular value decomposition performs more quicly than our algorithm here; however, it is not considering image rotations. Note that the large jump in running time for eigenvector decomposition with N r 384 is due to the fact that for large N r, the N r N θ N r N θ covariance matrix became too large to store in memory, causing the virtual memory to come into use, causing large delays. In Table IV one can see a comparison of the running times of various algorithms as P 5 and N r 1 are held constant, and N θ is increased. Again, our algorithm is significantly faster than any of the others. We now consider the numerical accuracy of our algorithm. We compare the resulting eigenvectors of SVD against those of our algorithm and eigenvector decomposition of the covariance matrix. However,

17 17 N θ This Paper JZL SVD Eig. Decomp N/A N/A N/A N/A TABLE IV COMPARISON OF RUNNING TIMES UNDER INCREASING NUMBER OF ROTATIONS N θ, IN SECONDS. P 5 AND N r 1. these eigenvectors often occur in pairs, and any two orthonormal eigenvectors that span the correct eigensubspace are valid eigenvectors. Therefore, we cannot simply compute the distance between eigenvectors of different methods, because the paired eigenvectors produced may be different. To measure the accuracy of the algorithms, then, we compute the Grassmannian distance between two eigensubspaces: If a subspace V is spanned by orthonormal vectors v 1,..., v, then the projection matrix onto that subspace is defined by P v i1 v ivi T. The Grassmannian distance between two subspaces V and W is defined as the largest eigenvalue magnitude of the matrix (P V P W ). In our first experiment, we artificially generated 5 projection images of the E. Coli 5s ribosomal subunit, as described in the next section, and convert them into w-filtered Radon images as described in Section IV with N r 1 and N θ 36. We then perform PCA on these Radon images and each of their 36 rotations using the four different algorithms. We then pair off principal components with identical eigenvalues, and compute the Grassmannian distance between each set of principal components and those computed by SVD. In our second experiment, we do the same thing but using images consisting entirely of Gaussian noise with mean and variance 1. The results of both experiments can be seen in Table V.

18 18 (A) (B) Princ. Comps This Paper JZL Eig. Decomp. Princ. Comps This Paper JZL Eig. Decomp & & & & & & & & TABLE V COMPARISON OF GRASSMANNIAN DISTANCES BETWEEN SVD EIGENSUBSPACES AND OTHER METHODS. (A) IMAGES OF ARTIFICIALLY GENERATED PROJECTIONS OF THE E. Coli 5S RIBOSOMAL SUBUNIT. (B) IMAGES OF WHITE NOISE. Finally, in our third experiment, we compare the true eigenimages that are produced by our algorithm using the procedure described in Section IV with those produced by performing SVD on a full set of images and rotations. To do this, we produce artificially generated 75 projection images of the E. Coli 5s ribosomal subunit and produced 36 uniform rotations of each one 2. We then performed SVD on this set of 27 images, and compared the results to running our algorithm on the original set of 75 images. The results can be seen in Figure 1. Eigenimages 2 and 3 are rotated differently in our algorithm than in SVD. This is because the two eigenimages have the same eigenvalue, and different rotations correspond to different eigenvectors in the two-dimensional subspace spanned by the eigenvectors. Eigenimage 4 is also rotated differently, which is again due to paired eigenvalues; the fifth eigenimage is not shown. VI. CRYO-ELECTRON MICROSCOPY Cryo-electron microscopy is a technique used to image and determine the three dimensional structure of molecules. Many copies of some molecule of interest are frozen in a sheet of ice that is thin enough so that, when viewing the sheet from its normal direction, the molecules do not typically overlap. This sheet is then imaged using a transmission electron microscope. The result is a set of noisy projection images of the molecule of interest, taen at random orientations of the molecule. This process usually destroys the molecules of interest, preventing one from tuing the sheet of ice to image a second time at a nown orientation. The goal is to then to tae a large set of such images, often 1, or more, and from that set deduce the three dimensional structure of the molecule. 2 We would lie to than Yoel Sholnisy for providing us with his code for rotating images. It produces a rotated image by upsampling using FFT, then rotating using bilinear interpolation, and finally downsampling again to the original pixels.

19 (A) (B) (C) Fig. 1. The number above each column represents which eigenimages is shown in that column. (A) The eigenimages produced by running SVD on the original set of 75 images. (B) The eigenimages produced by running SVD on the set of 27 rotated images. (C) The eigenimages produced by our algorithm. Several sample projection images are shown in Figure 2(A). These images were artificially generated by taing projections from random orientations of the E. Coli 5s ribosomal subunit, which has a nown structure. Gaussian noise was then added to create a signal-to-noise ratio of Var(signal)/Var(noise) 1/5. P 25 such images of size were generated, and then transformed into Radon images of size N θ 72 and N r 1. Note that in a real application of cryo-em, images often have much lower signal-to-noise ratios, and many more images are used. We then applied to this set of images the algorithm described in Sections II, III, and IV. The first 1 w -filtered Radon eigenimages are shown in Figure 2(B). Note that eigenimages 2 and 3 are an example of paired eigenimages. Paired eigenimages have the same eigenvalue, and occur as eigenvectors in paired Hermitian matrices Λ and Λ, as in equations (21) and (22). Note that eigenimages 1 and 6 have a frequencies of, which results in only a real eigenimage and so is not paired with another. Figure 2(C) shows the conversion of the Radon eigenimages in Figure 2(B) to true eigenimages. This was done with the procedure described in Section IV. Eigenimages 2 and 3 are paired with frequency 1, and eigenimages 4 and 5 are paired with frequency 2. They are described by equations (46) and (47). Eigenimage 1 is described by equation (46) only. These images can be compared in their general

20 2 shape to the eigenimages shown in [16, p. 117] where the SVD was used to compute the eigenimages for all possible rotations of KLH projection images. We remar that the true eigenimages are shown just for illustrative purposes: as will be shown in [18], the principal components of the two-dimensional w -multiplied polar Fourier transformed images are sufficient to accelerate the computation of the rotationally invariant distances [13], but the true eigenimages are not required. ACKNOWLEDGMENTS We would lie to than Yoel Sholnisy for providing us his code for computing the two dimensional Fourier transform on a polar grid, as well as his code for rotating images. We would also lie to than Zhizhen Zhao for helpful discussions and for proofreading the paper. The project described was supported by Award Number R1GM92 from the National Institute of General Medical Sciences. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of General Medical Sciences or the National Institutes of Health.

21 21 (A) Noisy Projections (B) ω -filtered Radon Eigenimages (C) True Eigenimages 22748, 3685, , , , , , , , , 139, 1279, , , , 3 193, 5 193, 5 94, 2 94, 2 882, 3 Fig. 2. (A) Artificially generated projections of the E. Coli 5s ribosomal subunit, with Gaussian noise added to create a signal-to-noise ratio of 1/5. (B) The first ten w -filtered Radon eigenimages. (C) The first twenty true eigenimages. Below each true eigenimage is its eigenvalue µ and frequency, respectively.

22 22 APPENDIX A PROOF OF 2-D FOURIER TRANSFORM FORMULA In this section, we prove formulas (42) and (43). To that end, consider the operation in equation (38): F,θ c (r) N r 1 + R,q N r 1 + r 1 R,q g,θ c (n)e ı c ( n, θ + π)e ı c (n, θ )e ı f( n) cos ( (θ + π) + ϕ( n))e ı f(n) cos ( θ + ϕ(n))e ı f( n) [cos ( (θ + π)) cos (ϕ( n)) sin ( (θ + π)) sin (ϕ( n))] e ı N r 1 + f(n) [cos ( θ ) cos (ϕ(n)) sin ( θ ) sin (ϕ(n))] e ı cos ( (θ + π)) sin ( (θ + π)) r 1 + cos ( θ ) f(n) cos (ϕ(n))e ı f( n) cos (ϕ( n))e ı f( n) sin (ϕ( n))e ı r 1 sin ( θ ) f(n) sin (ϕ(n))e ı (49)

23 23 Note that and, similarly, cos ( (θ + π)) cos ( θ )( 1) cos ( θ ) cos ( θ ) sin ( (θ + π)) sin ( θ ) f( n) cos (ϕ( n))e ı f( n) cos (ϕ( n))e ı f( n)( 1) cos (ϕ( n))e ı f( n) cos ( π + ϕ( n))e ı (5) f( n) sin (ϕ( n))e ı f( n) sin ( π + ϕ( n))e ı (51)

24 24 Therefore, it follows that equation (49) becomes Fc,θ (r) cos ( θ ) sin ( θ ) f( n) cos ( π + ϕ( n))e ı r 1 + cos ( θ ) f(n) cos (ϕ(n))e ı r 1 sin ( θ ) f(n) sin (ϕ(n))e ı cos ( θ ) sin ( θ ) cos ( θ ) sin ( θ ) N r 1 + f( n) sin ( π + ϕ( n))e ı R,q c R,q c N r 1 + r 1 r 1 R,q s g, ( n, π)e ı (n, )e ı R,q s ) ( n, π)e ı (n, )e ı c (n)e ı g, s (n)e ı ) cos ( θ )F, c (r) sin (θ )F, s (r) (52) Therefore, define the 2-D Fourier transform of the true image as F c (r, θ) F,θ c As similar manipulation shows that,, (r) cos ( θ)fc (r) sin ( θ)fs (r) (53) Fs (r, θ) Fs,θ (r) sin ( θ)fc,, (r) + cos ( θ)fs (r) (54) As shown above, these functions are either real or pure imaginary, depending on.

25 25 APPENDIX B PROOF OF THE RADON INVERSION FORMULA In this appendix we show that the true eigenimages I 1 and I 2 are given by equations (46) and (47). To do this, we tae the 2-D inverse Fourier transform of F, c (r, θ). First, let F, c (w 1, w 2 ) 1 F w 2 1 +w c ( w w2 w2 2, arctan w 2 ). Then I c(x 1, x 2 ) F, c (w 1, w 2 )e ı(w1,w2) (x1,x2) dw 1 dw 2 (55) With the change of variables x 1 s cos α, x 2 s sin α and w 1 r cos θ, w 2 r sin θ we obtain I c(s, α) F, r c (r, θ)eı(r cos θ,r sin θ) (s cos α,s sin α) r drdθ 1 ( r cos ( θ)fc, (r) ) sin ( θ)fs, (r) e ı(r cos θ,r sin θ) (s cos α,s sin α) r drdθ 1 ( cos ( θ)f, r c (r) ) sin ( θ)fs, (r) e ırs cos (θ α) r drdθ ( e ıθ + e ıθ ) 1 2 r F c, (r)eırs cos (θ α) r drdθ 1 2ı ( e ıθ e ıθ ) 1 2ı r 1 ( F, r c (r) + r 1 F, r s (r) + r F s, (r)e ırs cos (θ α) r drdθ e ı(θ+rs cos(θ α)) dθ ) e ı(θ rs cos(θ α)) dθ dr ( e ı(θ+rs cos(θ α)) dθ ) e ı(θ rs cos(θ α)) dθ dr (56)

26 26 Now, with the change of variables θ α + π 2 θ, e ı(θ+rs cos(θ α)) dθ e ı(α+ π 2 ) α+ π 2 and with the change of variables θ θ α + π 2, α+ π 2 e ı(θ rs sin(θ )) dθ e ı(α+ π 2 ) J (rs) (57) e ı(θ rs cos(θ α)) dθ e ı(α π 2 ) α+ π 2 + α+ π 2 where J is the Bessel function. Thus, I c becomes e ı(θ rs sin(θ )) dθ e ı(α π 2 ) J (rs) (58) Ic(s, α) (e ı(α+ π2 ) + e ı(α π2 )) 2 ( e ı(α+ π2 ) + e ı(α π2 )) 2ı ( e ı(α+ π ) 2 + e ı(α π )) 2 H 2 c(s) + 2ı r 1 F, r c (r)j (rs) dr r 1 F, r s (r)j (rs) dr ( e ı(α+ π 2 ) e ı(α π 2 )) H s (s) (59) where Hc (s) is the order Hanel transform of r 1 Fc, (r), and H s (s) is the order Hanel transform 1 of r Fs, (r). Now note that e ı(α π 2 ) e ı(α+ π 2 )+ıπ e ıπ e ı(α+ π 2 ) (6) As a result, Ic(s, α) ( e ı(α+ π ) 2 + e ıπ e ı(α+ π )) 2 H 2 c(s) + ( e ı(α+ π ) 2 e ıπ e ı(α+ π )) 2 H 2ı s (s) (61)

27 27 If is even, then e ıπ 1, so I c(s, α) 2 + 2ı cos ( e ı(α+ π ) 2 + e ı(α+ π )) 2 Hc(s) ( e ı(α+ π ) 2 e ı(α+ π )) 2 Hs (s) ((α + π ) 2 ) Hc(s) + sin This is a real-valued function because Fc, and Fs, are real when is even. ((α + π 2 ) ) H s (s) (62) If is odd, then e ıπ 1, so Ic(s, α) ( e ı(α+ π ) 2 e ı(α+ π )) 2 H 2 c(s) + ( e ı(α+ π ) 2 + e ı(α+ π )) 2 H 2ı s (s) ı sin ((α + π ) 2 ) Hc(s) ı cos ((α + π ) 2 ) Hs (s) (63) While this appears imaginary, it is also real because Fc, and Fs,, and therefore Hc and H s, are imaginary when is odd. A similar manipulation shows that, when is even, Is (s, α) sin ((α + π ) 2 ) Hc(s) + cos ((α + π ) 2 ) Hs (s) (64) and when is odd, I s (s, α) ı cos ((α + π ) 2 ) Hc(s) + ı sin ((α + π ) 2 ) Hs (s) (65) Again, both (64) and (65) are real. However, note that (62) is a complex scalar multiple of (65). Scalar multiples of real values are unimportant, so we may combine these two into the single formula ( I1 (s, α) β() cos ((α + π ) 2 ) Hc(s) + sin ((α + π ) ) 2 ) Hs (s) (66) where β() 1 2 ( ( 1 + ( 1) + ı 1 ( 1) )), (67) so that β() 1 when is even, and β() ı when is odd. Similarly, (63) is a complex scalar multiple of (64), so we may combine these two into the single formula ( I2 (s, α) β() sin ((α + π ) 2 ) Hc(s) cos ((α + π ) ) 2 ) Hs (s) (68)

28 28 REFERENCES [1] Borland, L. van Heel, M. (199) Classification of Image Data in Conjugate Representation Spaces. Joual of the Optical Society of America A, 7 (4), pp [2] Dutt, A., and Rohlin, V. (1993) Fast Fourier Transorms for Nonequispaced Data. SIAM Joual on Scientific Computing, 14 (6), pp [3] Fran, J. (26) Three-Dimensional Electron Microscopy of Macromolecular Assemblies: Visualization of Biological Molecules in Their Native State, 2nd edition, Oxford University Press, New Yor. [4] Gardner, D. G., Gardner, J. C., Lausch, G., Meine, W. W. (1959) Method for the Analysis of Multi-Component Exponential Decays. J. Chem. Phys., 31, pp. 987 [5] Golub, G. H., and Van Loan, C. F. (1996) Matrix Computations, Johns Hopins University Press. [6] Greengard, L., and Lee, J.-Y. (24) Accelerating the Nonuniform Fast Fourier Transform. SIAM Review, 46(3), pp [7] Guizar-Sicairos, M., and Gutierrez-Vega, J.C. (24) Computation of Quasi-Discrete Hanel Transforms of Integer Order for Propagating Optical Wave Fields. J. Opt. Soc. Am. A, 21, pp [8] Hilai, R., and Rubinstein, J. (1994) Recognition of rotated images by invariant Karhunen-Loeve expansion. J. Opt. Soc. Am. A, 11, pp [9] Jogan, M., Zagar, E., and Leonardis, A. (23) Karhunen-Loéve expansion of a Set of Rotated Templates. IEEE Transactions on Image Processing, 12 (7), pp [1] Khotanzad, A., and Lu, J. H. (1989) Object recognition using a neural networ and invariant Zeie features. Proceedings of the IEEE Conference on Vision and Patte Recognition, pp [11] Natterer, F., (21) The Mathematics of Computerized Tomography, the Society for Industrial and Applied Mathematics. [12] Par, R. (22) Correspondence. IEEE Transactions on Image Processing, 11 (3), pp [13] Pencze, P.A., Zhu, J., and Fran, J. (1996) A common-lines based method for determining orientations for N > 3 particle projections simultaneously. Ultramicroscopy 63, pp [14] Perona, P. (1995) Deformable Keels for Early Vision. IEEE Transactions on Patte Analysis and Machine Intelligence, 17 (5), pp [15] Siegmann, A. E. (1977) Quasi Fast Hanel Transform. Opt. Lett., 1, pp [16] Sigworth, F. J. (24) Classical Detection Theory and the Cryo-EM Particle Selection Problem. Joual of Structural Biology, 145, pp [17] Singer, A., Zhao, Z., Sholnisy, Y., and Hadani, R. (submitted for publication) Viewing Angle Classification of Cryo- Electron Microscopy Images Using Eigenvectors. [18] Singer, A. and Zhao, Z. (in preparation). [19] Teague, M. R. (198) Image Analysis via the General Theory of Moments. J. Opt. Soc. Am., 7, pp [2] Uenohara, M. and Kanade, T. (1998) Optimal Approximation of Uniformly Rotated Images: Relationship Between Karhunen-Loeve Expansion and Discrete Cosine Transform. IEEE Transactions on Image Processing, 7 (1), pp [21] van Heel, M., Fran, J. (1981) Use of Multivariate Statistics in Analysing the Images of Biological Macromolecules. Ultramicroscopy, 6, pp

29 29 [22] van Heel, M., Gowen, B., Matadeen, R., Orlova, E. V., Finn, R., Pape, T., Cohen, D., Star, H., Schmidt, R., Schatz, M., and Patwardhan, A. (2) Single-particle electron cryo-microscopy: towards atomic resolution. Quarterly Reviews of Biophysics 33 (4), pp