Preconditioning regularized least squares problems arising from high-resolution image reconstruction from low-resolution frames

Transcription

1 Linear Algebra and its Applications 39 (2004) wwwelseviercom/locate/laa Preconditioning regularized least squares problems arising from high-resolution image reconstruction from low-resolution frames Fu-Rong Lin a,b,, Wai-Ki Ching b,2, Michael K Ng b,,3 a Department of Mathematics, Shantou University, Shantou, Guangdong 55063, PR China b Department of Mathematics, The University of Hong Kong, Pokfulam Road, Hong Kong, PR China Received April 2003; accepted 22 January 2004 Submitted by S van Huffel Abstract In this paper, we study the problem of reconstructing a high-resolution image from multiple undersampled, shifted, degraded frames with subpixel displacement errors from multisensors Preconditioned conjugate gradient methods with cosine transform based preconditioners and incomplete factorization based preconditioners are applied to solve this image reconstruction problem Numerical examples are given to demonstrate the efficiency of these preconditioners We find that cosine transform based preconditioners are effective when the number of shifted low-resolution frames are large, but are less effective when the number is small However, incomplete factorization based preconditioners work quite well independent of the number of shifted low-resolution frames 2004 Elsevier Inc All rights reserved Keywords: High-resolution; Image reconstruction; Regularization; Cosine transform preconditioner; Incomplete Cholesky factorization preconditioner Corresponding author Tel: ; fax: addresses: frlin@stueducn (F-R Lin), mng@mathshkuhk (MK Ng) Supported in part by the Guangdong Provincial Natural Science Foundation of China No Research supported in part by RGC Grant No HKU 726/02P and HKU CRCG Grant No , and Research supported in part by Hong Kong Research Grants Council Grant Nos HKU 730/02P and 7046/03P, and HKU CRCG Grant Nos , and /$ - see front matter 2004 Elsevier Inc All rights reserved doi:006/jlaa

2 50 F-R Lin et al / Linear Algebra and its Applications 39 (2004) Introduction An image acquisition system composed of an array of sensors, where each sensor has a subarray of sensing elements of suitable size, has recently been popular for increasing the spatial resolution with high signal-to-noise ratio beyond the performance bound of technologies that constrain the manufacture of imaging devices The attainment of superresolution from a sequence of degraded undersampled images could be viewed as reconstruction of the high-resolution image from a finite set of its projections on a sampling lattice This can then be formulated as a constrained optimization problem whose solution is obtained by minimizing a cost function [3,8,9,9,2,22] The image acquisition scheme is important in the modeling of the degradation process The need for model accuracy is undeniable in the attainment of superresolution along-with the design of the algorithm whose robust implementation will produce the desired quality in the presence of model parameter uncertainty To keep the presentation focused and of reasonable size, data acquisition with multisensors instead of, say, a video camera is considered Multiple undersampled images of a scene are often obtained by using multiple identical image sensors which are shifted relative to each other by subpixel displacements [,6,0] The resulting highresolution image reconstruction problem using a set of low-resolution images captured by the image sensors is interesting because it is closely related to the design of high-definition television (HDTV) and very high-definition (VHD) image sensors CCD image sensor arrays, where each sensor consists of a rectangular subarray of sensing elements, produce discrete images whose sampling rate and resolution are determined by the physical size of the sensing elements If multiple CCD image sensor arrays are shifted relative to each other by subpixel values, the reconstruction of high-resolution images is sometimes modeled as in [] Let g i, i =,,m be the low-resolution frames and z be the high-resolution image We have H i z = g i + η i, i =,,m, () where η i is the noise of g i and H i, i =,,m are structured matrices which will be specified in Section 2 The high-resolution image reconstruction problem is equivalent to find z such that it can be modeled as a minimization problem with regularization: min z m H i z g i α Lz 2 2, (2) i= where L is the discretization of the first order differential operator Here Lz 2 2 is a functional which measures the regularity (the difference between pixel values) of z and the regularization parameter α is used to control the degree of regularity of the solution This regularization functional has been used in [,6] for the reconstruction of high-resolution images

3 F-R Lin et al / Linear Algebra and its Applications 39 (2004) The minimization problem (2) is equivalent to the linear system ( m ) m H t i H i + αl t L z = H t i g i i= i= Ng et al [6] used cosine transform based preconditioners to precondition the above linear system When the number of shifted low-resolution images is equal to four (ie, m = 4) in the 2-by-2 sensor setting and these four shifted low-resolution images are shifted relative to each other by the half-pixel value, they showed that the conjugate gradient method, when applied to solving the cosine preconditioned system, converges superlinearly We note that under the noiseless condition, the four shifted low-resolution images are sufficient to reconstruct the high-resolution image perfectly In [7], Ng and Sze further modified cosine transform based preconditioners to handle some special cases where the number of shifted low-resolution images is equal to two Numerical results showed that the performance of these cosine transform based preconditioners are quite good for some special cases However, the cosine transform based preconditioners do not work well in general On the other hand, in the literature, there is no theoretical and experimental results for cosine transform preconditioners when the number of shifted low-resolution images is large We note that the quality of the reconstructed image is better when there are more shifted low-resolution images (see the numerical results in Section 4) There are two aims of this paper The first aim is to extend cosine transform based preconditioners for the high-resolution image reconstruction when the number of shifted low-resolution images is large The other one is to propose and develop incomplete Cholesky factorization based preconditioners for the high-resolution image reconstruction problem Incomplete Cholesky factorization based preconditioners was commonly employed to precondition the linear system arising from partial differential equations In this paper, we consider this type of preconditioner for the blurring type problem Our experimental results show that the performance of incomplete Cholesky factorization based preconditioners is quite efficient independent of the number of shifted low-resolution images However, the cosine transform based preconditioners are effective when the number of shifted low-resolution images is large When the number of shifted low-resolution images is small, cosine transform based preconditioners do not work well The outline of the paper is as follows In Section 2, we briefly give a mathematical formulation of the problem In Section 3, we study cosine transform based preconditioners and incomplete Cholesky factorization based preconditioners Finally, numerical results and concluding remarks are given in Section 4 2 The high-resolution image reconstruction model In this section, we give a brief introduction of the mathematical model for the high-resolution image reconstruction, see Bose and Boo [] for details

4 52 F-R Lin et al / Linear Algebra and its Applications 39 (2004) Suppose that we have m sensors, each sensor has N N 2 sensing elements (pixels) and the size of each sensing element is T T 2 Wethenhavem images of resolution N N 2 (low-resolution images) Our aim is to reconstruct an image of resolution M M 2 (high-resolution image), where M = L N and M 2 = L 2 N 2 In order to have some information to resolve the high-resolution image, there are subpixel displacements between the sensors More precisely, there exist integers u i [0,L ], v i [0,L 2 ], and real numbers ɛi x,ɛy i ( 2, 2 ), such that the horizontal and vertical displacements of the ith sensor are given by: di x = T (u i + ɛi x L ) and dy i = T 2 (v i + ɛ y i L ) 2 Here (u i,v i ) is the sensor position of the ith sensor, and ɛi x and ɛ y i denote respectively the normalized horizontal and vertical displacement errors We note that the parameters ɛi x and ɛ y i can be obtained by manufacturers during camera calibration The estimation method of these displacement errors was discussed in [5] An efficient algorithm for the estimation of these displacement errors is given in [5] Let f be the original scene The observed low-resolution image g i is modeled by: g i [n,n 2 ] = T T 2 ( ) T2 n d y i T 2 (n 2 2 ) +d y i ( ) T n + 2 +di x ) T (n 2 +di x f(x,y)dx dy + η i [n,n 2 ] (3) for n =,,N and n 2 =,,N 2 InFig,weshowthegenerationofthe low-resolution image pixel value from the high-resolution image pixel values Here η i is the noise corresponding to the ith sensor Similarly, the high-resolution image z is modeled by: low-resolution image pixel high-resolution image pixel /4 /2 /4 /2 /2 /4 /2 /4 Fig The generation of the low-resolution image pixel for a 2-by-2 sensor array with the exact half-pixel value displacement

5 F-R Lin et al / Linear Algebra and its Applications 39 (2004) z[n,n 2 ]= L L 2 T T 2 ( ) n T 2 /L 2 ( ) n 2 2 T 2 /L 2 ) T /L ( n 2 ( n + 2 )T /L f(x,y)dx dy (4) for n =,,M and n 2 =,,M 2 Let g i, η i,andz be the corresponding vectors obtained by using a column by column ordering for g i, η i,andz, respectively, we have g i = H i z + η i, where H i is the blurring matrix corresponding to the ith sensor [] The stencil for a L -by-l 2 sensor array is given by L L ɛ x [ 2 + ɛ y 2 ɛ y] 2 ɛ x For instance, the stencil for a 2-by-2 sensor array is given by 4 ( )( ) ( ) ( )( ) 2 + ɛ 2 x + ɛ 2 y + ɛ 2 x + ɛ 2 x ɛ y ( ) ( ) 2 + ɛ y 2 ɛ y ( )( ) ( ) ( )( ), 2 ɛ 2 x + ɛ 2 y ɛ 2 x ɛ 2 x ɛ y see also Fig Now we can state the reconstruction problem as follows: find z minimizing m g i H i z 2 2 (5) i= Since the minimization problem (5) is ill-conditioned or even singular in general and there exists noise in the low-resolution images, the classical Tikhonov regularization is used More precisely, we solve the problem min z R M M 2 m g i H i z α Lz 2 2, (6) i= where the regularization parameter α is a small positive number controlling the degree of regularity of the solution, and L is discretization of the first order differential operator, ie 2 L t L = I M 2 M M I M2

6 54 F-R Lin et al / Linear Algebra and its Applications 39 (2004) Let the positions of all sensors be denoted by S =[(u,v ), (u 2,v 2 ),,(u m,v m )] Let #(u, v) denote the number of sensors with position (u, v) in S for integers u [0,L ] and v [0,L 2 ] Define M(S) = #(u, v) and m(s) = min #(u, v) (7) 0 u L 0 v L 2 max 0 u L 0 v L 2 We note that for the image reconstruction problem in [,6], M(S) = m(s) = and for the image reconstruction problem in [7], M(S) = andm(s) = 0 Now the proposed model (6) can handle the cases where M(S) > 2 Image boundary Because of the blurring process (cf (3)), the boundary values of g i are also affected by the values of f outside the scene Thus in solving z from (6), we need some assumptions on the values of f outside the scene Bose and Boo [] imposed the zero boundary condition outside the scene, ie, assuming a dark background outside the scene in the image reconstruction The ringing effects will occur at the boundary of the reconstructed images if f is indeed not zero close to the boundary The problem is more severe if the image is reconstructed from a large sensor array since the number of pixel values of the image affected by the sensor array increases, see [6] Let d u,l be the l vector with zero entries except its (u + )th entry being equal to (for instance, d,4 = (0,, 0, 0) t ) Under the zero boundary condition, the blurring matrix corresponding to the ith sensor can be written as H i = H y i H x i, (8) where H x i = (I M /L d t u i,l ) H x (ɛi x ) and H y i = (I M2 /L 2 d t v i,l 2 ) H y (ɛ y i ) Here H x (ɛi x) is an M M banded Toeplitz matrix with bandwidth L + : h x+ i 0 h x+ i L h x i 0 h x i with h x± i = 2 ± ɛx i

7 F-R Lin et al / Linear Algebra and its Applications 39 (2004) The M 2 M 2 banded blurring matrix H y (ɛ y i ) is defined similarly We recall that a matrix T =[t ij ] n i,j= is called a Toeplitz matrix if t ij = t i j for i, j =,,nin many applications, Toeplitz matrices are generated by a function p(θ) = t k e ikθ, k= which is called a generating function For the above Toeplitz matrix, the generating function is given by p(θ) = L e ikθ + h x+ uv L eilθ + h x uv e il θ (9) k= (L ) Ng et al [6] have considered using the Neumann boundary condition on the image It assumes that the scene immediately outside is a reflection of the original scene at the boundary Numerical results have shown that the Neumann boundary condition gives better reconstructed high-resolution images than that by the zero or periodic boundary conditions Under the Neumann boundary condition, the blurring matrices are given by H i = [(I M2 /L 2 d t v i,l2 )H y (ɛ y i ) ] [(I M /L d t u i,l )H x (ɛ x i ) ], which is similar to (8) Here H x (ɛ x i ) and Hy (ɛ y i ) are M M and M 2 M 2 Toeplitzplus-Hankel matrices respectively: h x+ i 0 h x i 0 q q h x+ i L + q h x+ i h x L i h x, i q q q 0 h x+ 0 h x i i (0) and H y (ɛ y i ) is similar We recall that a matrix H =[h ij ] n i,j= is called a Hankel matrix if h ij = h i+j for i, j =,,n For the sake of simplicity, we define H x i = (I M /L d t u i,l )H x (ɛ x i ) and Hy i = (I M2 /L 2 d t v i,l 2 )H y (ɛ y i ) Let E =[(ɛ x,ɛy ),,(ɛx m,ɛy m)], and m A(S, E,α) = [(H y i )t H y i ] [(Hx i )t H x i ]+αlt L, () i=

8 56 F-R Lin et al / Linear Algebra and its Applications 39 (2004) g = m (H y i Hx i )t g i i= We see that when the Neumann boundary condition is applied, the optimization problem (6) is equivalent to A(S, E,α)z = g (2) In the next section, we consider solving (2) by preconditioned conjugate gradient (PCG) methods We remark that Ng and Sze [7] have considered the image reconstruction problem where M(S) = andm(s) = 0 For comparison, we briefly introduce the mathematical model proposed in [7] Let D u,l be an l l diagonal matrix with all zero diagonals except that the (u + )th diagonal is equal to and D u,v = (I M2 /L 2 D v,l2 ) (I M /L D u,l ) Then the blurringmatrixcorresponding to theith sensor under the Neumann boundary condition is given by H(ɛi x,ɛy i ) = D u i,v i (H y (ɛ y i ) Hx (ɛi x )) We note that the idea is to intersperse the low-resolution image g i to form an M M 2 image g i : assign g i [n,n 2 ] to g i [L (n ) + u +,L 2 (n 2 ) + v + ] and zero to other positions of g i Thus, g i = H(ɛi x,ɛy i )z + η i The blurring matrix for the whole set of sensors is made up of blurring matrices from each sensor: m H(S, E) = H(ɛi x,ɛy i ) (3) i= With the Tikhonov regularization, the problem becomes: (H(S, E) t H(S, E) + αl t L)z = H(S, E) t g, (4) where g = m i= g i and g = m i= g i is called the observed image It is not difficult to check that the systems (2) and (4) are equivalent if M(S) = Thus, our model is an extension of the model in [7] For the cases where M(S) =, the observed image is given by { g[l (n ) + u i +,L 2 (n 2 ) + v i + ] =g i [n,n 2 ] for i =,,m, g[l (n ) + u +,L 2 (n 2 ) + v + ] =0 for #(u, v) = 0 (5) If M(S) 2, that is, there exist integers u, v such that #(u, v) 2, then g[l (n ) + u +,L 2 (n 2 ) + v + ] is set to the average of values of [n,n 2 ]th pixel of the low-resolution images with (u, v) sensor position

9 F-R Lin et al / Linear Algebra and its Applications 39 (2004) Fig 2(a) shows the method of forming a 4 4image g with a 2 2 sensor array each having a 2 2 sensing elements, ie, L = 2, L 2 = 2, M = M 2 = 4, N = N 2 = 2 and T = T 2 = 2 This is the case for high-resolution image reconstruction and S =[(0, 0), (0, ), (, 0), (, )] Fig 2(b) shows a 4 2image g with 2 sensors each having a 2 2 sensing elements The sensor positions are (0, 0) and (, 0) respectively, ie, S =[(0, 0), (, 0)] This case corresponds to the sensor taking the same scene of the original image but is slightly displaced in the horizontal direction with respect to the reference sensor In Fig 2(c), we consider the case where S =[(0, 0), (0, )] This case corresponds to the sensor which is slightly displaced in the vertical direction with respect to the sensor with position (0, 0) In Fig 2(d), we consider the case of two sensors where the sensor with position (, ) is slightly displaced in the diagonal direction with respect to the sensor with position (0, 0) In this case, we have S =[(0, 0), (, )] 3 The construction of preconditioners In this section, we discuss the construction of preconditioners for the linear system (2) We consider cosine transform based preconditioners and incomplete Cholesky factorization based preconditioners 3 Cosine transform based preconditioners Let C n be the n n discrete cosine transform matrix, ie the (i, j)th entry of C n is given by ( ) 2 δi (i )(2j )π cos, i, j n, n 2n where δ ij is the Kronecker delta Note that the matrix-vector product C n x can be computed in O(n log n) operations by using the fast cosine transform, see Sorensen and Burrus [20, p 557] For an n n matrix B, the cosine transform preconditioner c(b) of B is defined to be the matrix C t n C n that minimizes C t n C n B F in Frobenius norm [2] Clearly, the cost of computing c(b) y for any vector y is of O(nlog n) operations For banded matrices, like the matrices defined in () and (3), the cost of constructing c( ) is of O(n) [2], where n = M M 2 In this paper, we propose usingc(a(s, E,α)) as preconditioner for (2) We note that when M(S) = the preconditioner can be written as c(h(s, E) t H(S, E)) + αl t L Obviously, this preconditioner is different from the one proposed in [7] c(h(s, E)) t c(h(s, E)) + αl t L The following theorems imply that our new preconditioner can be more efficient than the preconditioner proposed in [7]

10 58 F-R Lin et al / Linear Algebra and its Applications 39 (2004) Fig 2 Observed images: (a) high-resolution image reconstruction; (b) high-resolution image reconstruction (horizontal displacement of the sensor); (c) high-resolution image reconstruction (vertical displacement of the sensor); (d) high-resolution image reconstruction (diagonal displacement of the sensor)

11 F-R Lin et al / Linear Algebra and its Applications 39 (2004) Theorem Let H ɛ be a Toeplitz-plus-Hankel matrix h + 0 h 0 q q h + L h + q h + L h, q q q 0 h + 0 h where h ± = /2 ± ɛ Then we have H t ɛ H ɛ = A + A 2, where A is a Toeplitz-plus-Hankel matrix w w 2 w n w 2 w 3 w n 0 A = w 2 w w 3 q q q w n w2 + q q q w n q q q w 3 w n w 2 w 0 w n w 3 w 2 and A 2 is a rank 2 matrix (6) A 2 = L 4ɛ L 2 O n 2L+2 4ɛ L Here w i,i=,,nare the entries of w = (2L 5/2 + 2ɛ 2, 2L 3, 2L 4,,, /4 ɛ 2, 0,,0) t /L 2, and m and O m are m m matrices with all entries and 0 respectively Proof We first note that H t ɛ H ɛ can be written as a sum of a banded Toeplitz matrix and a sparse matrix H t ɛ H ɛ = (T + E) L2 Let l = L, note that h + + h =, and the Toeplitz matrix T is generated by (cf (9)) l k= (l ) = e ikθ + h + e ilθ + h e ilθ l k= (l ) e ikθ 2 + (e ilθ + e ilθ ) l k= (l ) l k= (l ) e ikθ + h e ilθ + h + e ilθ e ikθ

12 60 F-R Lin et al / Linear Algebra and its Applications 39 (2004) = = +(h + e ilθ + h e ilθ )(h e ilθ + h + e ilθ ) 2(l ) k= 2(l ) 2l + k= 2l (2l k )e ikθ + k= e ikθ e ikθ + ((h + ) 2 + (h ) 2 ) + h + h (e 2ilθ + e 2ilθ ) (2l 2 ) 2l + 2ɛ2 + 2 k= (2l k) cos(kθ) + 2 ( ) 4 ɛ2 cos(2lθ) In other words, T is a symmetric Toeplitz matrix with the first column given by w The matrix E is a sparse matrix with only the upper-left block E( : 2L 2, : 2L 2) and the lower-right block E(n 2L + 3 : n, n 2L + 3 : n) are non-zero It can be verified that and E( : 2L 2, : 2L 2) w 2 w 3 w 2L w 3 q q 0 = q q + w 2L 0 0 E(n 2L + 3 : n, n 2L + 3 : n) q q w 2L = q q + 0 w 2L w 2 ( ) 4ɛL O L O L O L ( ) OL O L O L 4ɛ L Thus, the result of the theorem follows Using the results and algorithms of [7], one can check that the matrix A can be diagonalized by the cosine transform matrix, ie c(a ) = A and the cosine transform preconditioner for A 2 is the zero matrix, ie c(a 2 ) = O n Hence, H t ɛ H ɛ c(h t ɛ H ɛ) = A 2 is a rank two matrix, ie the spectrum of H t ɛ H ɛ c(h t ɛ H ɛ) is clustered around 0 Furthermore, when ɛ is small, the matrix A 2 is also a small norm matrix On the other hand, it is easy to check that c(h ɛ ) = H 0 and it follows that the spectrum of H t ɛ H ɛ c(h t ɛ )c(h ɛ) = H t ɛ H ɛ H t 0 H 0

13 F-R Lin et al / Linear Algebra and its Applications 39 (2004) is not clustered around 0 if ɛ not small enough Therefore, as preconditioners for H t ɛ H ɛ, c(h t ɛ H ɛ) is better than c(h ɛ ) t c(h ɛ ) Based on the above discussions, we can easily prove the following theorem for the two-dimensional case (the block Toeplitz-plus-Hankel with Toeplitz-plus-Hankel block case), which states that when M(S) = m(s) = and all subpixel displacement errors are the same, then c(a(s, E,α)) A(S, E,α)is a low rank matrix with respect to the matrix size of A(S, E,α) Theorem 2 Let H ɛ and H δ be two Toeplitz-plus-Hankel matrices of order M and M 2 respectively, cf (6) We have (H t ɛ H ɛ) (H t δ H δ) c(h t ɛ H ɛ) c(h t δ H δ) = A 2M +2M 2 +4, where A 2M +2M 2 +4 is an (M M 2 ) (M M 2 ) matrix of rank at most 2M + 2M Furthermore, if both ɛ and δ are small enough, then A 2M +2M 2 +4 is also a small norm matrix According to Theorem 2, we expect that the proposed cosine transform preconditioner works well for block Toeplitz-plus-Hankel with Toeplitz-plus-Hankel block systems arising from the high-resolution image reconstruction problem when M(S) = m(s) = 32 Incomplete factorization based preconditioners Besides the cosine transform based preconditioner, we study incomplete factorization based preconditioners for the high-resolution image reconstruction problem Given a symmetric matrix A and a symmetric sparsity pattern S, an incomplete Cholesky factor of A is a lower triangular matrix Q such that A = QQ T + V, q ij = 0 if(i, j) S, v ij = 0 if (i, j) S Meijerink and van der Vorst [4] considered two choices of S, the standard setting of S to the sparsity pattern of A, and a setting that allowed more fill Many variations are possible In [4], it is proved that if A is an M-matrix, then the incomplete Cholesky factorization exists for any predetermined sparsity pattern S Manteuffel [3] extended this result to H -matrix which positive diagonal elements We note that an n n matrix A is called an M-matrix if A is invertible and all entries of the inverse of A are non-negative A matrix A is called an H -matrix if the associated matrix M(A): { [A]ij, i = j, [M(A)] ij = [A] ij, i /= j is an M-matrix We are interested in the incomplete factorization with S being the sparsity pattern of A This fails if a negative diagonal element is encountered One can increase any non-positive pivot to a positive threshold during the factorization process However,

14 62 F-R Lin et al / Linear Algebra and its Applications 39 (2004) this may result in a very poor preconditioner, see for instance, the example in Section 3 of [] It is important to modify the diagonal elements before we encounter a negative pivot There are several modifications to the incomplete Cholesky factorization of the form A + E, wheree is a diagonal matrix, see for instance [4,8] In this paper, we will apply the shifted incomplete Cholesky factorization of Manteuffel [2,3] for the scaled matrix  = D /2 AD /2, D = diag([a],,[a] nn ) The idea is to apply the incomplete Cholesky factorization to the matrix  + βi, where I is the identity matrix and β 0 It is obvious that for sufficiently large β,  + βi is an H -matrix and therefore the incomplete factorization exists However, a large β will result in a poor preconditioner The minimal value of β is an interesting issue for further research In summary, we have the following algorithm: Shifted incomplete Cholesky factorization of Manteuffel Choose β S > 0 Compute  = D /2 AD /2,whereD = diag([a],,[a] nn ) Set β 0 = 0 For k = 0,,, Compute the incomplete Cholesky factorization of  k =  + β k IIfsuccessful set β = β k and exit Set β k+ = max(2β k,β S ) End The main features of the shifted incomplete Cholesky factorization are that the memory requirement is predictable (limited memory) and the computational cost is not more than ( + log 2 (max(2β,β S )/β S )), where is the cost of one incomplete Cholesky factorization with the sparsity pattern of A Note that the coefficient matrix A(S, E,α)(cf ()) is a block band matrix with band blocks, where the bands are not more than (4L 3) and (4L 2 3) respectively, it follows that = O(L 2 L2 2 M M 2 ) Numerical results in Section 4 show that the shifted incomplete Cholesky factorization performs well (see Table 3) and that the value of ( + log 2 (max(2β,β S )/β S )) is small for the high-resolution image reconstruction problem (see Table 4) 33 Comparison of the preconditioners In this subsection, we compare the condition numbers and the spectra of the preconditioned matrices of cosine transform preconditioner with the incomplete Cholesky factorization preconditioner We have randomly tested a number of different ɛ (ɛ x i and ɛ y i are chosen randomly between 0 and0), the results are similar In Table, we show the condition numbers for the following situations of 2 2sensor:

15 F-R Lin et al / Linear Algebra and its Applications 39 (2004) (i) S =[(0, 0), (0, )], (ii) S 2 =[(0, 0), (, )], (iii) S 3 =[(0, 0), (0, ), (, )], (iv) S 4 =[(0, 0), (0, ), (, 0), (, )], (v) S 5 =[(0, 0), (0, ), (, 0), (, ), (0, 0)], (vi) S 6 =[(0, 0), (0, ), (, 0), (, ), (0, 0), (0, )], (vii) S 7 =[(0, 0), (0, ), (, 0), (, ), (0, 0), (, )], (viii) S 8 =[(0, 0), (0, ), (, 0), (, ), (0, 0), (0, ), (, )], (ix) S 9 =[(0, 0), (0, ), (, 0), (, ), (0, 0), (0, ), (, 0), (, )] For each situation, we show one numerical result In our tests, we set M = M 2 = 32 In Table, κ, κ 2,andκ 3 denote the condition numbers of the coefficient matrix A(S, E,α), c(a(s, E,α)) A(S, E,α),andQ t Q A(S, E,α)respectively, where Q is the factor of the shifted incomplete Cholesky factorization of A(S, E,α)In the shifted incomplete Cholesky factorization, we set β S = 00 We observe that the preconditioned matrices with incomplete Cholesky factorization preconditioners are quite well-conditioned for all situations As to the cosine transform preconditioners, the preconditioned matrices are very well-conditioned for the cases S i where S i S 4 (m(s i ) ),ies i for i = 4,,9 while they are quite ill-conditioned for the cases S i where S i S 4 (m(s i ) = 0), ies i for i =, 2, 3 We show in Fig 3 the spectra of the preconditioned matrices It is easy to see that the spectra of preconditioned matrices are not clustered around one Therefore, the improvement of the convergence of the PCG method lies in the improvement of the condition numbers of the preconditioned matrices 4 Numerical results and concluding remarks In this section, we compare the performance of cosine transform based preconditioners and incomplete Cholesky factorization based preconditioners Nine situations Table Condition numbers for different S: M = M 2 = 32 S κ κ 2 κ 3 κ κ 2 κ 3 α = α = S S S S S S S S S

16 64 F-R Lin et al / Linear Algebra and its Applications 39 (2004) Fig 3 Spectra of the preconditioned matrices with cosine transform preconditioners (left) and incomplete Cholesky factorization preconditioners (right) of 2-by-2 sensor array are considered: S i, i =,,9, (cf Section 33) In the tests, the parameters ɛi x and ɛ y i are chosen randomly between 0 and 0 Gaussian white noises with signal-to-noise ratios of 50 and 30 db are added to the low-resolution images The optimal regularization parameter α is chosen such that it minimizes the relative error of the reconstructed image z r (α) to the original image z, ie, it minimizes z z r (α) 2 (7) z 2 In the conjugate gradient methods, we use the zero vector as the initial guess and the stopping criteria is r (j) 2 r (0) < 0 6, 2 where r (j) is the normal equations residual after j iterations The data in Tables 2 4 are averages of 20 randomly generated problems The original image is shown in Fig 4(a) One of the low-resolution images is shown in Fig 4(b) (50 db case) The observed noisy images and reconstructed images for S i, i =,,9 are also shown in Fig 4 (50 db case) We can see that all reconstructed images are much better than the observed images Table 2 shows the optimal regularization parameters and the corresponding relative errors for the nine situations We can clearly see that the relative error becomes smaller when the number of low-resolution images increases Furthermore, the optimal regularization parameter is proportional to the number of sensor positions covered by low-resolution images Table 3 shows the performance of the cosine transform based preconditioners and the incomplete Cholesky factorization based preconditioners proposed in Section 3

17 F-R Lin et al / Linear Algebra and its Applications 39 (2004) Table 2 Regularization parameters and relative errors S α Relative error α Relative error SNR=50 db SNR=30 db S S S S S S S S S Table 3 Number of iterations required for convergence S N C IC N C IC SNR=50 db SNR=30 db S S S S S S S S S Table 4 Values of ( + log 2 (max(2β,β S )/β S )) for α = S S S 2 S 3 S 4 S 5 S 6 S 7 S 8 S 9 + log 2 (max(2β,β S )/β S ) In Table 3, the symbols N, C, andic denote the PCG methods without preconditioner, with cosine transform based preconditioners and with incomplete Cholesky factorization based preconditioners respectively We see from Table 3 that the incomplete Cholesky factorization based preconditioner is quite efficient for all sensor arrays The cosine transform preconditioner is more efficient than the incomplete Cholesky factorization preconditioner when m(s i ) = while it is not efficient when m(s i ) = 0(S, S 2,andS 3 ) This observation is consistent with the numerical results in Table We remark that in Table the value of the regularization parameter is fixed at or 5 0 3, while in Table 3, the optimal regularization parameter

18 66 F-R Lin et al / Linear Algebra and its Applications 39 (2004) Fig 4 Images: (a) original; (b) low-resolution; (c) observation for S ; (d) restoration for S ; (e) observation for S 2 ; (f) restoration for S 2 ; (g) observation for S 3 ; (h) restoration for S 3 (i) observation for S 4 ; (j) restoration for S 4 ; (k) observation for S 5 ; (l) restoration for S 5 (m) observation for S 6 ; (n) restoration for S 6 ; (o) observation for S 7 ; (p) restoration for S 7 (q) observation for S 8 ; (r) restoration for S 8 ; (s) observation for S 9 ; (t) restoration for S 9 based on (7) is used for each S i We note that for S 4 S 9, the regularization parameter α is larger than that for S S 3 Finally in Table 4 we show the values of ( + log 2 (max(2β,β S )/β S )) for different sensor arrays with regularization parameter α = We see from the

19 F-R Lin et al / Linear Algebra and its Applications 39 (2004) table that ( + log 2 (max(2β,β S )/β S )) are small for all problems we tested (for α = , all the values are equal to ) It follows that the total cost of the shifted incomplete Cholesky factorization which is given by ( + log 2 (max(2β,β S )/β S ))O (L 2 L2 2 M M 2 ) is well bounded by O(L 2 L2 2 M M 2 ) In this paper we study the problem of reconstructing a high-resolution image from multiple undersampled, shifted, degraded frames with subpixel displacement errors We apply the PCG method with cosine transform based preconditioners and incomplete factorization based preconditioners to solve this reconstruction problem Numerical results show that cosine transform based preconditioners are effective when m(s) = (the number of shifted low-resolution frames are large), but are less effective when m(s) = 0 (the number of shifted low-resolution frames is small) However, incomplete factorization based preconditioners work quite well independent of the number of shifted low-resolution frames References [] NK Bose, KJ Boo, High-resolution image reconstruction with multisensors, Internat J Imag Syst Technol 9 (998) [2] RH Chan, MK Ng, Conjugate gradient method for Toeplitz system, SIAM Rev 38 (996) [3] R Chan, T Chan, M Ng, W Tang, C Wong, Preconditioned iterative methods for high-resolution image reconstruction with multisensors, in: F Luk (Ed), Proceedings to the SPIE Symposium on Advanced Signal Processing: Algorithms, Architectures, and Implementations, San Diego, CA, July 998, vol 346, pp [4] A Forsgren, PE Gill, W Murray, Computing modified Newton directions using a partial Cholesky factorization, SIAM J Sci Comput 6 (995) [5] H Fu, J Barlow, A regularized total least squares algorithm for high resolution image reconstruction, Linear Algebra Appl, this issue [6] G Jacquemod, C Odet, R Goutte, Image resolution enhancement using subpixel camera displacement, Signal Process 26 (992) [7] T Kailath, V Olshevsky, Displacement structure approach to discrete trigonometric transform based preconditioners of G Strang and T Chan type, Calcolo 33 (996) [8] E Kaltenbacher, RC Hardie, High resolution infrared image reconstruction using multiple, low resolution, aliased frames, in: Proceedings of IEEE 996 National Aerospace and Electronic Conference on NAECON, vol 2, 996, pp [9] SP Kim, NK Bose, HM Valenzuela, Recursive reconstruction of high resolution image from noisy undersampled multiframes, IEEE Trans Acoust Speech Signal Process 38 (6) (990) [0] T Komatsu, K Aizawa, T Igarashi, T Saito, Signal processing based method for acquiring very high resolution images with multiple cameras and its theoretical analysis, IEE Proc 40 (3, Part I) (993) 9 25 [] CJ Lin, JJ Moré, Incomplete Cholesky factorization with limited memory, SIAM J Sci Comput 2 (999) [2] TA Manteuffel, Shifted incomplete Cholesky factorization, in: Sparse Matrix Proceedings 978, SIAM, Philadelphia, 979, pp 4 6 [3] TA Manteuffel, An incomplete factorization technique for positive definite linear systems, Math Comput 34 (980)

20 68 F-R Lin et al / Linear Algebra and its Applications 39 (2004) [4] JA Meijerink, HA van der Vorst, An iterative solution method for linear systems of which the coefficient matrix is a symmetric M-matrix, Math Comput 3 (977) [5] M Ng, NK Bose, J Koo, Constrained total least squares computations for high resolution image reconstruction with multisensors, Internat J Imag Syst Technol 2 (2002) [6] M Ng, R Chan, T Chan, A Yip, Cosine transform preconditioners for high resolution image reconstruction, Linear Algebra Appl 36 (2000) [7] M Ng, KN Sze, Preconditioned iterative methods for super-resolution image reconstruction with multisensors, in: Symposium on Advanced Signal Processing: Algorithms, Architectures and Implementations, San Diego CA, July 2000, in: F Luk (Ed), Proceedings to the SPIE, vol 46, 2000, pp [8] RB Schnabel, E Eskow, A new modified Cholesky factorization, SIAM J Sci Statist Comput (990) [9] RR Schultz, RL Stevenson, Extraction of high-resolution frames from video sequences, IEEE Trans Image Process 5 (6) (996) [20] H Sorensen, C Burrus, Fast DFT and convolution algorithms, in: S Mitra, J Kaiser (Eds), Handbook of Signal Processing, Wiley, New York, 993, pp [2] AM Tekalp, MK Ozkan, MI Sezan, High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration, in: Proceedings of IEEE International Conference on Acoustic, Speech, and Signal Processing, III, San Francisco, CA, March 992, pp [22] RY Tsai, TS Huang, Multiframe image restoration and registration, Adv Comput Vis Image Process (984)