Image fusion based on bilateral sharpness criterion in DT-CWT domain

Transcription

1 Int. J. Computational Vision and Robotics, Vol. 4, Nos. 1/2, Image fusion based on bilateral sharpness criterion in DT-CWT domain Priya Ranjan Muduli* Department of Electrical Engineering, Indian Institute of Technology, Kharagpur , India *Corresponding author Umesh Chandra Pati Department of Electronics and Communication Engineering, National Institute of Technology, Rourkela , India Abstract: Since last few decades, multi sensor image fusion has been an emerging field of research in remote sensing, medical imaging and variety of computer vision applications. The primary objective of image fusion lies in the formation of a perceptually enhanced image from several multi sensor images using an appropriate fusion rule. The discrete wavelet transform (DWT)-based image fusion techniques have been popular due to less redundancy, low computations and perfect reconstruction with short support filters. But, it suffers severely from lack of directionality, shift variance, oscillations and aliasing problems. These issues have been overcome by means of Q-shift dual-tree complex wavelet transform (DT-CWT)-based image fusion. In this paper, an improved DT-CWT-based image fusion technique has been proposed to compose a resultant image with better perceptual as well as quantitative image quality indices. A bilateral sharpness based weighting scheme has been implemented for the high frequency coefficients taking both gradient and its phase coherence in account. A normalised maximum gradient weighting scheme is implemented for low frequency wavelet components. The fusion results demonstrate that the proposed fusion technique is more effective and competitive in terms of entropy, total standard deviation, average gradient measure and edge intensity measure. Keywords: bilateral-sharpness-criterion; discrete wavelet transform; DWT; dual-tree complex wavelet transform; DT-CWT; image-fusion; phase-congruency; multi resolution analysis; MRA. Reference to this paper should be made as follows: Muduli, P.R. and Pati, U.C. (2014) Image fusion based on bilateral sharpness criterion in DT-CWT domain, Int. J. Computational Vision and Robotics, Vol. 4, Nos. 1/2, pp Biographical notes: Priya Ranjan Muduli received his BTech degree in Electronics and Telecommunication Engineering from ITER, Bhubaneswar, India in He pursued his MTech degree in Electronics and Instrumentation from National Institute of Technology, Rourkela, India in Currently, he is Copyright 2014 Inderscience Enterprises Ltd.

2 162 P.R. Muduli and U.C. Pati working as a Research Scholar in the Department of Electrical Engineering, Indian Institute of Technology, Kharagpur, India. He has publications in refereed journals and conference proceedings. His research interests are in image and video processing, computer vision, body area networking, instrumentation and signal processing. Umesh Chandra Pati obtained his BE in Electrical Engineering from National Institute of Technology, Rourkela, Odisha, India. He received both his MTech and PhD degrees in Electrical Engineering from Indian Institute of Technology, Kharagpur, India. Currently, he is serving as an Associate Professor in the Department of Electronics and Communication Engineering, National Institute of Technology, Rourkela. His current areas of interest are image processing, computer vision, instrumentation and signal processing. He has authored/edited two books and published more than 60 research papers in international as well as national journals and conference proceedings. He is a member of IEEE. 1 Introduction Image fusion is basically a technique of merging several images from multi-modal sources with respective complementary information to form a new image, which carries all the common as well as complementary features of individual images. Image fusion has an extensive area of application such as, multi-spectral remote sensing, target detection, military surveillance systems medical imaging and so on. Various algorithms have been proposed for effective fusion of multi-source images such as simple averaging, maximum and minimum fusion rules. Afterwards, the fusion performance is improved by the introduction of principal component analysis (PCA) and Morphological processing algorithms (Krishnamoorthy and Soman, 2010). Again, the fusion algorithms such as Brovey technique, PCA and Intensity-Hue-Saturation fail due to the characteristics spectral losses and colour deformation. Data fusion by means of pyramidal decomposition results in growth of redundancy and orientation deficiency (Akerman, 1992). Further, with the development of discrete wavelet transform (DWT)-based image fusion techniques, the perceptual quality has been enhanced upholding the spectral information contents. The real DWT has the property of good compression of signal energy. Perfect reconstruction is possible using short support filters. The unique feature of DWT is the absence of redundancy and very low computation (Chao et al., 2004). Therefore, DWT has been used extensively for multi resolution analysis (MRA)-based image fusion. The DWT primarily suffers from the various problems (Selesnick et al., 2005) such as oscillations, aliasing, shift variance and lack of directionality. The ringing artefacts introduced by DWT are also completely eliminated by the implementation of dual tree complex wavelet (DT-CWT)-based image fusion methods (Yu, 2008). In this paper, an improved version of dual tree wavelet transform-based image fusion algorithm is proposed. The fusion process is implemented using efficient fusion rules for high frequency coefficients as well as low frequency coefficients depending on their characteristics. Since the background texture information is primarily highlighted by the low frequency components, a maximum normalised gradient of the neighbourhood pixels-based weighing rule is implemented (Tian et al., 2011). Here, the structural content of an image I(r, c) is measured effectively using a gradient covariance matrix of a region

3 Image fusion based on bilateral sharpness criterion in DT-CWT domain 163 specified by means of a local window of suitable size. The difference between eigenvalues of the gradient covariance matrix yields the image gradient strength which is further normalised. Again, the bilateral sharpness measurement for high frequency wavelet components comprises of both gradient strength criterion and corresponding phase coherence measurement. The phase criterion provides robustness towards noise and gradient-based illumination variation. The robustness of the proposed method is verified from low light television (LLTV) and forward-looking-infrared (FLIR) image fusion, multi-spectral satellite image fusion and CT-MR image fusion. The paper is organised as follows: Section 2 gives detail description of the DT-CWT Transform. The proposed image fusion method as well as the bilateral sharpness-weighting criterion is vividly described in the Section 3. Simulation results along with quality assessment of fused image are obtained by means of some non-referential quality measures in Section 4. Finally, the paper is concluded in Section 5. 2 Dual-tree complex wavelet transforms The dual-tree complex wavelet transform (DT-CWT) was first introduced by Nick Kingsbury in the year The DT-CWT has been one of the most popular transform domain techniques due to some of the unique features such as, good shift invariance, good directional selectivity in two-dimensions as well as three-dimensions, perfect reconstruction using short support filters, limited redundancy and low computation. The DT-CWT basically utilises two real DWTs. The first one yields the real part of the transform while the second one yields the imaginary part. In order to accomplish the perfect reconstruction, it is required to process signals with the help of wavelets. The DT-CWT (Selesnick et al., 2005) accomplishes this by using two filter banks and thus two bases. With the help of two filter banks {g 0 (n), h 0 (n)} and {g 1 (n), h 1 (n)} four DWTs, F hh, F gg, F gh and F hg are generated. The F gh component is extracted from the filters g i (n) along the rows and filters h i (n) along columns. Since at each decomposition level of DWT, three sub-bands are produced, there can be 12 sub-bands to generate six directionally selective complex sub-bands, which are approximately analytic. These complex sub-bands are basically oriented at ±15, ±45, and ±75. Mathematically the two dimensional DT-CWT decomposition of an image I(x, y) can be expressed by means of the complex shifted and dilated mother wavelet ψ(x) and scaling function φ(x) as: (1) θ θ j0, l j0, l j, l j, l 2 l z θ Θ j j 2 0 l z I( x, y) = S φ ( xy, ) + C ψ ( xy, ) The scaling function is expressed as: r i jl, j0, l j0 φ ( x) = φ ( x) + 1 φ ( x) (2) The mother wavelet function is: r i jl, jl, jl, ψ ( x) = ψ ( x) + 1 ψ ( x) (3) where z is the natural number set.

4 164 P.R. Muduli and U.C. Pati S represents the scaling coefficient with shifting of j and dilation of l j 0, l C jl, shows the complex wavelet coefficients r, i are the indexing for real and imaginary parts respectively. The directionality of the six complex sub bands generated is guided along θ Θ = { ± 15, ± 45, ± 75 }. Thus, the two dimensional DT-CWT Transform gives rise to one real low-pass image along with six complex high-pass sub-images at each decomposition level. To obtain the final DT-CWT outcome, the difference between the filters in the two trees is estimated. It is basically implemented by a delayed version of one sample between the level 1 filters of the real as well as imaginary decomposition trees. The outcomes of subsequent levels are obtained by the help of alternate odd-length and even-length linear-phase filters. Unluckily the odd and even filtering techniques often suffer from the problems like: unsymmetrical sub-sampling structure, frequency responses variation between decomposition trees (Kingsbury, 2000). To beat these issues, a Q-shift dual tree (Kingsbury, 2003) is implemented for this research work. This novel algorithm proposes that, all the filters beyond level 1 has to be of even length without any rigid linear phase condition with a group delay of nearly a quarter samples (q). Likewise, other filters beyond level 1 are imitated from the orthonormal prototype set. A symmetric sub-sampling arrangement takes care of the shift invariance as well as the directional selectivity property of DT-CWT. The dual tree filters for Q-shift wavelet decomposition is shown in Figure 1. Figure 1 Dual tree filter bank for the Q-shift wavelet transform 3 Proposed image fusion using DT-CWT At every decomposition level of DT-CWT, six directional high frequency wavelet coefficients are generated along with two low frequency coefficients. Generally, the wavelet coefficients of each band are blended using some suitable fusion rules. A new fused image is reconstructed using inverse DT-CWT. The complete fusion process is shown in Figure 2.

5 Image fusion based on bilateral sharpness criterion in DT-CWT domain 165 Figure 2 Flowchart for the proposed fusion method Image 1 Q-shift DT-CWT Bilateral gradient-based image fusion Inverse DT-CWT Fused image Image 2 Q-shift DT-CWT 3.1 Fusion rule for low frequency coefficients In an image, the background texture information is primarily highlighted by the low frequency components. Appropriate fusion selection rule for low frequency components is one of the important criterions during the fusion process. The rules can be classified as maximum selection rule or weighted rule. In this research work, we have implemented the weighted fusion scheme for low frequency components. The normalised weighting factor selection for this research work is inspired by the maximum gradient-based sharpness of the neighbourhood pixels (Tian et al., 2011) Gradient-based sharpness criterion Sharpness has always been considered as one of the prime norm for image quality measurement. The sharpness and information content of an image primarily depends on the strength measures. Image fusion algorithms based on simple normalised aggregation of input images fails since, the concerned high frequency regions are also weighted equally along with the unimportant regions. To overcome such issues, an improved version of the novel weighting criterion (Tian et al., 2011) has been implemented in this paper. Here, to measure the structural contents of an image I(r, c) effectively, a gradient covariance matrix of a region is specified by means of a local window of size M N (Agrawal et al., 2006). The gradient covariance matrix is given by: 2 Ir (,) rc Ir(,) rcic(,) rc w w C = 2 Ir(,) r c Ic(,) r c Ic (,) r c w w where I r (r, c) and I c (r, c) represent the row-gradient and column-gradient of the image respectively. The gradient covariance matrix can be represented as: (4) T T λ1 0 v 1 C = VDV = ( v1 v2) 0 λ T 2 v 2 (5)

6 166 P.R. Muduli and U.C. Pati where the 2 2 matrix V consists of the eigen vectors v 1 and v 2 along its columns. The λ matrix D =. Where λ 1 and λ 2 represents the eigenvalues of the gradient 0 λ2 covariance matrix (Wee and Paramesran, 2007) The image gradient strength is represented as: Grc (,) = λ λ (6) 1 2 The maximum gradient strengths of input images can be calculated as: ( ( G r c )) Gmax = max max (, ) The maximum gradient strengths are computed for individual image A and image B using the above formula. Now, the normalised weighting factors W A for image A is laid out by: G W max A A = G max A + G max B (7) Similarly, the normalised weighting factor for image B can be represented as: G W max B B = G max A + G max B (8) The weights W A and W B corresponding to the image A and image B respectively, are applied to the low frequency wavelet coefficients for the fusion process. 3.2 Fusion rule for high frequency coefficients Recently, most of the fusion rules meant for high frequency coefficients are based on neighbourhood characteristics such as neighbourhood energy, variance, etc. The high frequency components basically describe the detail information of an image. In this paper, the fusion of high frequency components is implemented by means of an effective weighting scheme proposed by Tian et al. (2011). For high frequency wavelet component fusion process, the bilateral gradient-based sharpness weighting method is implemented. This bilateral sharpness measurement comprises of both gradient strength criterion from equation (6) and corresponding phase coherence measurement. The local phase coherence criterion plays a vital role with respect to human visual perception towards the fusion response (Kovesi, 1999). The phase criterion is also robust towards noise and gradient-based illumination variation. The phase coherence for image gradient can be presented as: ( ) Prc (,) = cos θ(,) rc θ (,) rc (9) where θ(r, c) evaluated from principal vector v 1, is the phase information at coordinates (r, c). θ (,) rc is the average phase of neighbouring pixels.

7 Image fusion based on bilateral sharpness criterion in DT-CWT domain 167 The maximum phase coherence value corresponds to the edge pixels. The bilateral sharpness criterion is developed using the gradient sharpness criterion in equation (6) and its corresponding phase coherence in equation (9). This can be expressed as: S = G α (,) r c P β (,) r c (10) The factors α and β can be tuned to some suitable values to maximise the contribution of suitable sharpness criterions. In this experimental work, α and β values are set to 1 and 0.5 respectively confined in an window size of w = 5 (Tian et al., 2011) 4 Simulation results and discussion The Bilateral sharpness criterion-based fusion in DT-CWT domain has been performed using various multi-sensor images from a standard image database of Dr. Oliver Rockinger. The input images are pre-registered. The robustness of the proposed fusion technique is verified successfully with some multi-sensor images such as: LLTV sensor image, FLIR sensor image, multispectral remote sensing images and medical images such as CT, MR images. The original input images and their corresponding fusion results using the proposed technique are depicted in Figures 3, 4 and 5. Figure 3 (a) LLTV sensor image (b) FLIR sensor image (c) fused image using proposed method (a) (b) (c) Figure 4 (a) Multispectral sensor image A (b) Multispectral sensor image B (c) Fused image using proposed method (a) (b) (c)

8 168 P.R. Muduli and U.C. Pati Figure 5 (a) CT image (b) MRI image (c) fused image using proposed method (a) (b) (c) Table 1 Quantitative assessment for fusion of navigation images (LLTV and FLIR) Methods Quality indices Entropy Average gradient Edge intensity Standard deviation DWT DT-CWT Proposed method Table 2 Quantitative assessment for fusion of multispectral remote sensing images Methods Quality indices Entropy Average gradient Edge intensity Standard deviation DWT DT-CWT Proposed method Table 3 Quantitative assessment for fusion of medical images (CT and MR) Quality indices Average Edge Standard Entropy Methods gradient intensity deviation DWT DT-CWT Proposed method The quantitative evaluation of this research work using various multi-sensor images has been shown in Tables 1, 2 and 3. The performance comparison of the proposed method is accomplished with DWT and DT-CWT transform in terms of some non-referential image quality measures such as entropy, average gradient, edge intensity and standard deviation. The superiority as well as robustness of the proposed image fusion technique is evidently justified from the fused image quality assessment tables. Some of the major non-referential image quality measures are discussed below. 4.1 Entropy Entropy (E) is considered as one of the vital image quality index to evaluate the information content in an image.

9 Image fusion based on bilateral sharpness criterion in DT-CWT domain 169 It is formulated as: N i= 0 ( ) log ( ) E = p x p x (11) i i where x i is the grey level value at i th pixel with corresponding probability p. The entropy value is larger for images containing more the information. 4.2 Average gradient The detail contrast and texture variation in an image is usually indexed by means of average gradient values and is given as: i= f f ( M 1)( N 1) + 1 x y g = ( M 1)( N 1) (12) Edge intensity The measurement of sharp discontinuities in an image can be considered as one of the image quality assessment parameters. It can be easily accomplished using the Sobel edge detection algorithm. It uses horizontal differentiation kernel g x and a vertical differentiation kernel g y, which are presented as: g x = 2 0 2, (13) For an image I, the edge intensity values are given as: 2 2 ( x y ) S = G + G (14) where G I* g, G = I* g x = x y y` 4.4 Standard deviation The standard deviation is considered as one of the best metrics for contrast value measurement for an image. High contrast level of an image can be making out from high standard deviation value. It can be formulated as: 1 N j = ji j N i = 1 ( ) 2 (15) σ x μ where the mean pixel value μ j 1 N = x N i = 1 ji

10 170 P.R. Muduli and U.C. Pati 5 Conclusions An enhanced fusion scheme proposed in this paper implements a bilateral gradient-based sharpness-weighting criterion in DT-CWT. The proposed fusion technique compensates all the shortcomings of DWT by the implementation of Q-shift DT-CWT. It also removes the ringing artefacts introduced in the fused image by assigning suitable weighting schemes to high pass wavelet coefficients and low pass coefficients independently. The normalised maximum gradient-based sharpness criterion for low frequency coefficients enhances the background texture information as well as improves the quality of the blurred regions in the fusion result. The most vital information contents concealed in the high frequency coefficients are also boosted up by the implementation of bilateral sharpness criterion. From the image quality assessment tables, it is clear that the proposed fusion technique outperforms other methods in terms of entropy, average gradient, edge intensity and standard deviation. References Agrawal, A., Raskar, R. and Chellappa, R. (2006) Edge suppression by gradient field transformation using cross-projection tensors, Proc. IEEE Int. Conf. on Computer Vision and Pattern Recognition, Vol. 8, pp Akerman, A. (1992) Pyramid techniques for multisensor fusion, Proc. SPIE, Vol. 1828, pp Chao, R., Zhang, K. and Li, Y-J. (2004) An image fusion algorithm using wavelet transform, ACTA ELECTRONICA SINICA, Vol. 32, No. 5, pp Kingsbury, N. (2000) A dual-tree complex wavelet transform with improved orthogonally and symmetry properties, ICIP, Vol. 2, pp Kingsbury, N. (2003) Design of q-shift complex wavelets for image processing using frequency domain energy minimization, Int. Conference on Image Processing, Vol. 1, pp Kovesi, P. (1999) Videre: A Journal of Computer Vision Research, Vol. 1, No. 3, MIT Press Journals, Cambridge, Massachusetts. Krishnamoorthy, S. and Soman, K.P. (2010) Implementation and comparative study of image fusion algorithms, Int. J. of Computer Applications ( ), Vol. 9, No. 2, pp Selesnick, I.W., Baraniuk, R.G. and Kingsbury, N. (2005) The dual-tree complex wavelet transform, IEEE Signal Processing Magazine, Vol. 22, No. 6, pp Tian, J. et al. (2011) Multi-focus image fusion using a bilateral gradient-based sharpness criterion, Journal of Optics Communications, Elsevier, Vol. 284, No. 1, pp Wee, C.Y. and Paramesran, R. (2007) Measure of image sharpness using eigen values, Information Sciences, Elsevier, Vol. 177, No. 12, pp Yu, R. (2008) Theory of dual-tree complex wavelets, IEEE Transactions on Signal Processing, Vol. S6, No. 9, pp