Coherent use of opposing lenses for axial resolution increase. II. Power and limitation of nonlinear image restoration

Transcription

1 M. Nagorni and S. W. Hell Vol. 18, No. 1/January 2001/J. Opt. Soc. Am. A 49 Coherent use of opposing lenses for axial resolution increase. II. Power and limitation of nonlinear image restoration Matthias Nagorni and Stefan W. Hell High Resolution Optical Microscopy Group, Max-Planck-Institute for Biophysical Chemistry, D Göttingen, Germany Received January 28, 2000; revised manuscript received August 7, 2000; accepted August 10, 2000 We analyze the ability of nonlinear image restoration to remove interference artifacts in microscopes that enlarge the axial optical bandwidth through coherent counterpropagating waves. We calculate the images of an elaborate test object as produced by confocal, standing-wave, incoherent illumination interference image interference, and 4Pi confocal microscopes, and we subsequently investigate the extent to which the initial object can be restored by the information allowed by their optical transfer function. We find that nonlinear restoration is successful only if the transfer function is sufficiently contiguous and has amplitudes well above the noise level, as is mostly the case in a two-photon excitation 4Pi confocal microscope Optical Society of America OCIS codes: , , , , , INTRODUCTION The superposition of coherent counter propagating waves in fluorescence microscopy, as it is utilized in a standingwave microscope (SWM), 1,2 a 4Pi confocal microscope, 3,4 and an incoherent illumination interference image interference microscope (I 5 M), 5,6 increases the available optical frequencies and the axial bandwidth of the optical transfer function (OTF) of a microscope. The concomitant interference produces a point-spread function (PSF) with a sharper main maximum. Hence these microscopes have the potential of increasing the axial resolution in far-field fluorescence microscopy. Unfortunately, for all aperture angles available, the PSF of these microscopes is characterized by multiple maxima or sidelobes, which not only blur the object but also replicate it in the image. This results in ghost images or interference artifacts in the raw data. Therefore image deconvolution or restoration is essential to render unambiguous imaging and genuine axial resolution improvement. According to Fourier theory, the periodic occurrence of maxima in the PSF manifests itself as periodic depressions in the OTF. 7 In the SWM, the depressions are even frequency gaps. In both the case of gaps and of depressions where the signal is masked by noise, the newly gained higher-frequency parts are disconnected from the low frequencies also found in single-lens systems. In these cases, the removal of the interference artifacts by linear deconvolution is impossible, and the higher frequencies cannot be exploited without introducing artifacts. 8 The artifacts can be conceived as caused by the fringes introduced by the missing frequencies. We have shown that the OTF s of a SWM, a 4Pi confocal microscope, and an I 5 M differ fundamentally in contiguity and in the heights of the periodic depression of the transfer functions. 7 Whereas the OTF of the two-photon excitation 4Pi confocal microscope is contiguous, 9 with rather elevated periodic depressions (19% 32% of the main maximum), the OTF of the I 5 M features rather broad and shallow depressions, which render linear deconvolution challenging. 7 The OTF of the SWM is disconnected by gaps, in which case linear deconvolution is virtually unfeasible. Fortunately, there is a further option to restore the data, which is nonlinear deconvolution, or image restoration The principal idea behind this approach is to retrieve the missing frequencies of the OTF from a priori knowledge of the object. Such a priori knowledge is the fact that in fluorescence microscopy the object and the image function are strictly nonnegative. Utilizing maximum-likelihood estimation image restoration, an initial study has been carried out for the SWM, concluding that the artifact removal is hardly feasible for the conditions provided by its OTF. 8 It is now interesting to investigate to what extent the nonnegativity of the object and the full knowledge of the PSF can be used to remove the interference artifacts in a 4Pi confocal microscope and an I 5 M in comparison with their use in the SWM. Of the known nonlinear image algorithms, we have elected the Richardson Lucy (RL) algorithm, which has been shown to be a good choice for images dominated by Poisson noise TEST OBJECT As the imaging properties of the SWM, the 4Pi confocal microscope, and the I 5 M are fully defined by the complex OTF, we can investigate their performance by calculating the resulting three-dimensional (3D) images of a welldefined test object. We have designed an elaborate 3D test object consisting of four lines, two in the XZ plane and two in the XY plane, encompassing a sphere that con /2001/ $ Optical Society of America

2 50 J. Opt. Soc. Am. A/ Vol. 18, No. 1/ January 2001 M. Nagorni and S. W. Hell Fig. 1. Test object: (a) axial section (XZ) encompassing the optical axis Z, designed to contain all axial distances of interest, and (b) YZ and XY sections of the 3D test object. tains point objects (Fig. 1). Since the potential artifacts are expected to occur along the optical axis, our 3D test object is designed to contain all axial distances of interest. Therefore the most important parts of the test object are the two lines emanating from a common point in the top of the XZ section, shown in Fig. 1(a). A similar line object was added in the XY plane [Fig. 1(b)]. These line objects can be considered the idealization of the filamentous actin or microtubules in a mammalian cell. The sphere in the test object is filled with randomly distributed points. This part of the object can be considered an idealized nuclear membrane containing hybridization spots. The dimensions of the test object are 256 pixels and 128 pixels in the axial and lateral directions, respectively. The pixels are cubic with a cube length of 20 nm. We simulated raw image data of the different microscopes by a 3D convolution of the test object with the PSF calculated in Ref. 7. To include the effect of noise, we set the global maximum of the data to 100 counts; subsequently, the pixel values were randomized by a Poisson noise filter, rendering data corresponding to the anticipated Poisson statistics. 3. MAXIMUM-LIKELIHOOD RESTORATION The calculated raw image data were restored by the iterative RL algorithm. 11 This algorithm has been shown to implement maximum-likelihood estimation for images dominated by Poisson noise. 12,16 If g is the measured image and h is the PSF, then the RL iteration for the estimate fˆ is given by fˆ m,n k 1 g k fˆ m,n i i,j j k r h r,s fˆ i r,j s s h i m, j n. (1) We applied Tikhonov regularization to the RL algorithm, as proposed by Conchello and McNally. 17 In the RL Conchello algorithm, the regularized estimate fˆreg is derived from the unregularized estimate fˆ according to k fˆ k 1 1/2, (2) fˆ reg where is the regularization parameter. This transformation is applied after each iteration of the RL algorithm. After each iteration, we also calculated the I divergence 18 between the true object f and the estimate fˆ according to I f, fˆ i f i, j j ln f i, j fˆi, j f i, j fˆi, j. (3) Minimizing the I divergence is equivalent to maximizing the likelihood of the object estimate. 15 Therefore we stopped the iterations after the I divergence reached a minimum. 4. RESULTS The axial sections (XZ images) of the simulated raw data of the confocal microscope, the SWM, the I 5 M, and the three types of 4Pi confocal microscopes give an impression of the amount of blur and replication introduced by the PSF in each case (Fig. 2, left column). Note that flat standing-wave excitation and the concomitant lack of lateral discrimination of the excitation light lead to an extended halo in the raw data of the SWM and the I 5 M. Whereas in the 4Pi microscopes the threefold replication of the object can be clearly observed, this is not possible in the SWM and the I 5 M. The reason is the separability of the 4Pi confocal PSF into a lateral and an axial function, which allows one to separate this PSF into a blurring peak and a lobe function responsible for replication. 7,19 The ghost images of the object are strong in the singlephoton 4Pi confocal microscope of type C, but they are weaker in the two-photon excitation 4Pi confocal microscope of type A and almost vanish in the type C data. Since the replication part is almost negligible in type C, the effect of the blur can be clearly observed for this mode: The lateral blur yields two weak spots inside the sphere,

3 M. Nagorni and S. W. Hell Vol. 18, No. 1/January 2001/J. Opt. Soc. Am. A 51 Fig. 2. Raw image data of the object in Fig. 1 calculated for the confocal microscope, the SWM, the I 5 M, the 4Pi type C microscope, the two-photon 4Pi type A microscope, and the two-photon 4Pi type C microscope (left column), shown in the same section as that in Fig. 1(a). The data were normalized to a maximum of 100 counts before a Poisson noise filter was applied. Results of the RL restoration are given in the right column. The number of iterations is indicated in the lower right corner. Observe the structure of the raw image data and the varying degree of success of the restoration.

4 52 J. Opt. Soc. Am. A/ Vol. 18, No. 1/ January 2001 M. Nagorni and S. W. Hell which originate from point objects located 80 nm above and below the axial section shown. InaRL Conchello restoration, two parameters can be varied: the number of iterations and the regularization parameter. If the iterations are stopped after the minimumofthei divergence is reached, remains the only free parameter. We restored the data with different and finally elected a value that led to a smooth solution that was in good agreement with the original test object. For the single-photon and two-photon microscopes, two slightly different s were chosen, that is, and , respectively, which reflects the fact that the FWHM of the maxima of the single-photon 4Pi confocal type C PSF is smaller than that of the twophoton 4Pi confocal type A PSF. First, we note that the restored confocal images are given as a resolution reference. The OTF of the confocal microscope is convex, so that the artifact produced in the restored image is just a blur along the optical axis (Fig. 2, right column). The strong effect of the PSF on the number of iterations can be seen when comparing the restoration of the three different types of 4Pi confocal data. It turns out that the stronger the replication effect of the sidelobes, the more the number of iterations needed to reach the minimum of the I divergence. In the case of the SWM and the I 5 M, we stopped the iterations after 2000 steps because we could not find a minimum of the I divergence. For the I 5 M, we increased the number of iterations to 8000 without finding a minimum. The structure of the OTF of the SWM and the I 5 M shown in Ref. 7 already indicates that their raw data cannot be restored without artifacts. In the restored image of the SWM data, the lines have almost disappeared, and the sphere shows residual replications. In the restored I 5 M data, the lines have a strong shadow of artifacts. Inside the sphere, a benzene-ring-like structure appears together with an unexpected strong spot in the center of the sphere. Considering that, because of optimized low-intensity excitation by a lamp, the signal-to-noise ratio (SNR) in some cases might be higher in the I 5 M than in the other systems, we examined the extent to which the increase of the SNR of the raw data improves the restoration result in the I 5 M. For this purpose, we generously increased the assumed signal of the I 5 M raw data 100-fold, so that the RL Conchello algorithm was applied to a maximum signal level of 10 4 counts in the I 5 M [Fig. 3(a)]. The higher signal level indeed allowed us to decrease drastically. We set 10 8 and stopped after 2000 iterations. Many of the artifacts are now eliminated, but strong artifacts such as the strong central spot in the sphere, as well as residual replications of the lines, remain. The success of the restoration of the I 5 M data is enhanced if the object happens to lack those frequencies that are only weakly transferred by its OTF. For example, in Fig. 3(b) we investigated an object XZ section that was offset 80 nm in the Y direction with respect to the section shown in Fig. 3(a); it is displayed in the upper right panel of Fig. 3(b). Because of the 80-nm offset, this section does not contain the two bright lines anymore. The raw I 5 M image data in Fig. 3(a) still indicate the presence of the lines because the 80-nm offset is smaller than the lateral resolution of the I 5 M. The restoration of the I 5 M images now works impressively well because the section does not contain weakly transferred spatial frequencies. Although the value of also influences the sidelobe removal, we found that at a given SNR, it was not possible to substantially reduce the residual lobe artifacts by decreasing. For too small a, the solution will not be smooth and will contain individual pixels with highintensity values. If the color lookup table is intentionally normalized to these values, it appears as though the lobe artifacts were reduced (Fig. 4). However, this is not confirmed by the corresponding profiles of the restored data. Fig. 3. Calculated I 5 M raw data (left) and their RL restored counterparts (right). To account for a potentially higher signal in this microscope, we assumed an average maximum of 10,000 counts, which is 100 times higher than that in Fig. 2. Whereas in (a) the restored result exhibits artifacts, the restoration shown in (b) is successful, because the imaged object section (upper right inset) does not contain unfavorable spatial frequencies.

5 M. Nagorni and S. W. Hell Vol. 18, No. 1/January 2001/J. Opt. Soc. Am. A 53 Fig. 4. Comparison of 4Pi confocal type C images obtained by restoring with (left), which is slightly too high, and (right), which is too low. The profiles reveal that the residual lobe artifacts are not significantly affected. The position of the profiles in the image is indicated by the arrows. The two highest peaks correspond to the line object and the sphere; the two weaker peaks are artifacts. Fig. 5. Calculated raw data (left) of the 4Pi confocal type C, two-photon 4Pi confocal type A, and two-photon 4Pi confocal type C microscopes assuming a pinhole size of 87% of the backprojected fluorescence Airy disk. The comparison with the object data in Fig. 1 discloses a remarkable performance of the combination of two-photon 4Pi confocal microscopy with image restoration. The contiguity of the 4Pi confocal OTF renders the restoration well applicable. Whereas the restored image of the single-photon 4Pi confocal microscope of type C contains weak residual lobe artifacts, that of the two-photon 4Pi confocal microscope of type A currently in use is nearly free of them. An exception is a weak artifact produced between the lines above the sphere. Its position coincides with the crossing point of the sidelobes of the lines in the raw data. The restoration of the two-photon 4Pi confocal microscope of type C does not contain artifacts; it is easily observed that the lateral blur is reduced by the restoration as well. The simulated data of Fig. 1 were obtained by convolving the test object with the PSF calculated for the case of an infinitesimally small pinhole, but of course in practice a finite pinhole diameter is used. The OTF calculated for a finite pinhole diameter of 87% of the backprojected Airy disk is shown in Ref. 7. This diameter is typical for our experiments with the two-photon excitation 4Pi confocal microscope of type A. The analysis of the restored data of the 4Pi confocal microscope of type C (Fig. 5, upper row) reveals that the images exhibit increased artifacts as a result of the finite pinhole size. In the case of the twophoton excitation 4Pi confocal microscope of type A (Fig. 5, middle row), the restored image shows only weak artifacts, which are axially spaced at the distance of the side-

6 54 J. Opt. Soc. Am. A/ Vol. 18, No. 1/ January 2001 M. Nagorni and S. W. Hell lobes. The restoration of the two-photon excitation 4Pi confocal microscope of type C (Fig. 5, bottom row) remains virtually artifact free. 5. DISCUSSION Of the microscopes investigated in this paper by data simulation, only the standard confocal and 4Pi confocal microscopes lead to reliable information about the structure of the object to be imaged. Whereas nonlinear image restoration is useful in improving the removal of interference artifacts, in the images of the I 5 M and even more so in the SWM, nonnegligible artifacts cannot be avoided. We also found that the removal of artifacts fails even at a signal level 100 times higher than that in the 4Pi confocal microscope. Hence the missing frequencies in the OTF of these microscopes precluded the restoration of a faithful object function, even for high signals. It appears that in many cases a large part of the object frequencies that are not transferred by the OTF cannot be satisfactorily recovered by a nonlinear image restoration using the nonnegativity constraint as a priori object information. Still, object constellations can be found that can be fully restored in the I 5 M and in some cases also in the SWM without severe artifacts. This is the case for structures that happen to lack the frequencies that are not covered by the OTF, as has been exemplified in Fig. 3. Clearly, in this case the OTF deficiencies are not effective. However, the study with our test object demonstrates that a successful restoration of some objects does not imply its general viability. 6. CONCLUSION The coherent use of opposing objective lenses for axial resolution increase is successful only if the effective OTF is contiguous and its amplitudes are well above the noise level. This condition is not fulfilled in the SWM, partially fulfilled in the I 5 M, acceptably well fulfilled in the single-photon 4Pi confocal microscope of type C if the pinhole is small enough, and well fulfilled in the two-photon excitation 4Pi confocal microscope. In the latter case, and especially in the case of coherent two-photon excitation and detection, any object suitable for imaging by opposing lenses can be gained with fundamentally increased spatial resolution. ACKNOWLEDGMENTS We gratefully acknowledge funding by the Deutsche Forschungsgemeinschaft, Bonn (He-1977). Address correspondence to Stefan W. Hell at the location on the title page or by , shell@gwdg.de. REFERENCES 1. F. Lanni, Applications of Fluorescence in the Biomedical Sciences, 1st ed. (Liss, New York, 1986). 2. B. Bailey, D. L. Farkas, D. L. Taylor, and F. Lanni, Enhancement of axial resolution in fluorescence microscopy by standing-wave excitation, Nature (London) 366, (1993). 3. S. W. Hell, Double-scanning microscope, European patent (December 18, 1990). 4. S. Hell and E. H. K. Stelzer, Properties of a 4Pi-confocal fluorescence microscope, J. Opt. Soc. Am. A 9, (1992). 5. M. G. L. Gustafsson, D. A. Agard, and J. W. Sedat, Sevenfold improvement of axial resolution in 3D widefield microscopy using two objective lenses, in Three-Dimensional Microscopy: Image Acquisition and Processing II, T.Wilson, and C. J. Cogswell, eds., Proc. SPIE 2412, (1995). 6. M. G. L. Gustafsson, D. A. Agard, and J. W. Sedat, I 5 M: 3D widefield light microscopy with better than 100 nm axial resolution, J. Microsc. (Oxford) 195, (1999). 7. M. Nagorni and S. W. Hell, Coherent use of opposing lenses for axial resolution increase in fluorescence microscopy. I. Comparative study of concepts, J. Opt. Soc. Am. A 18, (2001). 8. V. Krishnamurthi, B. Bailey, and F. Lanni, Image processing in 3-D standing wave fluorescence microscopy, in Three-Dimensional Microscopy: Image Acquisition and Processing III, C. J. Cogswell, G. S. Kino, and T. Wilson, eds., Proc. SPIE 2655, (1996). 9. M. Gu and C. J. R. Sheppard, Three-dimensional transfer functions in 4Pi confocal microscopes, J. Opt. Soc. Am. A 11, (1994). 10. B. R. Frieden, Restoring with maximum likelihood and maximum entropy, J. Opt. Soc. Am. 62, (1972). 11. W. H. Richardson, Bayesian-based iterative method of image restoration, J. Opt. Soc. Am. 62, (1972). 12. L. B. Lucy, An iterative technique for the rectification of observed distributions, Astron. J. 79, (1974). 13. T. J. Holmes, Maximum-likelihood image restoration adapted for non-coherent optical imaging, J. Opt. Soc. Am. A 5, (1988). 14. T. J. Holmes, Expectation maximization restoration of band-limited, truncated point-process intensities with application in microscopy, J. Opt. Soc. Am. A 6, (1989). 15. D. L. Snyder, T. J. Schutz, and J. A. O Sullivan, Deblurring subject to nonnegative constraints, IEEE Trans. Signal Process. 40, (1992). 16. G. M. P. Van Kempen, L. J. Van Vliet, P. J. Verveer, and H. T. M. van der Voort, A quantitative comparison of image restoration methods for confocal microscopy, J. Microsc. (Oxford) 185, (1997). 17. J.-A. Conchello and J. G. McNally, Fast regularization technique for expectation maximization algorithm for optical sectioning microscopy, in Three-Dimensional Microscopy: Image Acquisition and Processing III, C. J. Cogswell, G. S. Kino, and T. Wilson, eds., Proc. SPIE 2655, (1996). 18. I. Csiszar, Why least squares and maximum entropy? An axiomatic approach to inference for linear inverse problems, Ann. Stat. 19, (1991). 19. M. Schrader, K. Bahlmann, G. Giese, and S. W. Hell, 4Piconfocal imaging in fixed biological specimens, Biophys. J. 75, (1998).