Phaseless Imaging Algorithms and Applications

Transcription

1 Phaseless Imaging Algorithms and Applications James R. Fienup Robert E. Hopkins Professor of Optics University of Rochester Institute of Optics Presented to: Institute for Mathematics and Applications University of Minnesota August 2017 JRF -1

2 Outline Introductory phase retrieval example Astronomical imaging Mathematical background The problem Iterative transform algorithms Nonlinear optimization algorithms Recent application examples Lensless coherent optical imaging Non-crystallographic (x-ray) diffraction JRF -2

3 Passive Imaging of Astronomical/Space Objects Problem: atmospheric turbulence causes phase errors, limits resolution 1 arc-sec 5*10 6 rad. λ/ r o for λ = 0.5 μm and r o = 10 cm Best resolution as though through only 10 cm aperture (vs. Keck 10 m) Solutions: Hubble Space Telescope (2.4 m diameter), $2 B; JWST $8 B Adaptive optics + laser guide star, $10 s M Optical interferometry, $10 s M Stellar speckle interferometry, << $1 M aberrated optical system } object JWST Naval Prototype Optical Interferometer Lick Observatory blurred image JRF -3

4 Labeyrie s Stellar Speckle Interferometry 1. Record blurred images: g k (x,y ) = f (x, y) s k (x,y), k = 1,..., K where s k (x, y) is k th point-spread function due to atmospheric turbulence 2. Fourier transform: G k (u,v ) = F (u,v )S k (u,v ), where S k (u, v) is k th optical transfer function k = 1,..., K 3. Magnitude square and average: 1 K G K k (u,v) 2 = F(u,v ) 2 1 K k =1 K k =1 1 K 4. Measure or determine transfer function S K k (u,v ) 2 k =1 atmospheric model or measure reference star 5. Divide by 1 K K S k (u,v ) 2 S k (u,v ) 2 to get F (u,v ) 2 = squared Fourier magnitude k =1 A. Labeyrie, "Attainment of Diffraction Limited Resolution in Large Telescopes by Fourier Analysing Speckle Patterns in Star Images," Astron. and Astrophys. 6, (l970). JRF -4

5 Image Reconstruction from Simulated Speckle Interferometry Data Labeyri s stellar speckle interferometry gives this J.R. Fienup, "Phase Retrieval Algorithms: A Comparison," Appl. Opt. 21, (1982). JRF -5

6 Reconstruction from Fourier Magnitude Fourier magnitude of astronomical object can come from Stellar speckle interferometry Michelson stellar (amplitude) interferometry Intensity interferometry (Hanbury Brown Twiss) For coherently illuminated objects: lensless imaging e.g. x-ray diffraction, laser remote sensing From MRI, etc., Any imaging modality where when Fourier phase is lost Have Fourier magnitude; Want to reconstruct an image from it JRF -6

7 Fourier transform: F (u,v) = Inverse transform: f (x,y) = F(u,v ) = Phase retrieval problem: Phase Retrieval Basics f (x,y)e i 2π (ux +vy ) dxdy Given F (u,v ) and some constraints on f (x,y ), Reconstruct f (x,y ), or equivalently retrieve ψ (u,v ) [ ] = F (u,v)e iψ (u,v ) = F f(x,y ) F (u,v)e i 2π (ux +vy ) dudv = F 1 F (u,v ) [ ] F [ f (x,y)] = F [ e ic f (x x o,y y o )] = F [ e ic f * ( x x o, y y o )] (Inherent ambiguities: phase constant, images shifts, twin image all result in same data) Autocorrelation: [ ] r f (x,y ) = f ( x, y )f * ( x x, y y )d x d y = F 1 F (u,v ) 2 Patterson function in crystallography is an aliased version of the autocorrelation Need Nyquist sampling of the Fourier intensity to avoid aliasing JRF -7

8 Constraints in Phase Retrieval Nonnegativity constraint: f(x, y) 0 True for ordinary incoherent imaging, x-ray diffraction, MRI, etc. Not true for wavefront sensing or coherent imaging The support of an object is the set of points over which it is nonzero Meaningful for imaging objects of finite extent on dark backgrounds Wavefront sensing through a known aperture Essential for complex-valued objects Atomiticity when have angstrom-level resolution For crystals -- not applicable for coarser-resolution, single-particle Object intensity constraint (wish to reconstruct object phase) E.g., measure wavefront intensity in two planes (Gerchberg-Saxton) If available, supersedes support constraint Statistical information Maximum entropy, total variations, sparsity, sharpness, compactness, weak object phase, smooth object phase, etc. JRF -8

9 Phase is More Important than Amplitude G FT[G] phase{ft[g]} FT -1 [ FT[G] exp[i phase{ft[m]}]] M FT[M] phase{ft[m]} FT -1 [ FT[M] exp[i phase{ft[g]}]] JRF -9

10 Is Phase Retrieval Possible? 1970 s: Some said, Can t be done. The solution won t be unique. JRF -10

11 First Phase Retrieval Result (a) Original object, (b) Fourier modulus data, (c) Initial estimate (d) (f) Reconstructed images number of iterations: (d) 20, (e) 230, (f) 600 Reference: J.R. Fienup, Optics Letters, Vol 3., pp (1978). JRF -11

12 Iterative Transform Algorithm START: Initial Estimate g F F { } G = G e i φ = Gerchberg-Saxton algorithm if have measured intensities in both domains Constraints: Support, (Nonnegativity) Form New Input Using Image Constraints Satisfy Fourier Domain Constraints Measured Data: Magnitude, F Measured intensity g' 1 i F { } G' = F e F 1 φ Hybrid Input-Output version g g k +1 ( x,y ) = k ( x,y ), g k g k ( x,y ) β g k ( x,y ), g k ( x,y ) satisfies constraints x,y ( ) violates constraints JRF -12

13 Error Reduction = Projection onto Sets Constraint Set #1 (non-convex) Start Constraint Set #2 (convex) b a S If is a, convex: b S, If a, b S, then c = t a + (1 t) b S, then 0 t c 1= t a + ( 1 t )b S, 0 t 1 JRF -13

14 Error reduction algorithm Error-Reduction and HIO ER: g k +1 ( x ) = g k x ( ), x S & g k ( x ) 0 0, otherwise Satisfy constraints in object domain Equivalent to projection onto (nonconvex) sets algorithm Proof of convergence (weak sense) In practice: slow, prone to stagnation, gets trapped in local minima Hybrid-input-output algorithm HIO: g k +1 ( x ) = g k x g k x ( ), x S & g k ( x ) 0 ( ) β g k ( x ), otherwise Uses negative feedback idea from control theory β is feedback constant No convergence proof (can increase errors temporarily) In practice: much faster than ER, can climb out of local minima H. Takajo, T. Takahashi et al, J. Opt. Soc. Am. A (1997 & 1998) JRF -14

15 Progression of Iterations Initial support constraint Initial estimate 1 iteration 5 iterations 20 iterations 25 iterations 35 iterations 45 iterations 55 iterations JRF -15

16 Error Metric versus Iteration Number JRF -16

17 Autocorrelation Support = Sum of Support Sets Complex Fourier transform F ( u) = f ( x )e i 2πux dx Object Support S = { x : f ( x ) > 0} Set addition (Minkowski sum) and multiplication a X + by a x + b y : x X and y Y r f Autocorrelation support A = ( ) y S = F [ f ( x )] { } In r f x, f (y + x) is f (x) translated by y ( x ) is a weighted sum of translated versions of f (x), weighted by f (y) ( S y ) = S S = { x y : x,y S} u, x may be multidimensional Autocorrelation function r f ( x ) = f ( y )f ( y x )dy = f ( y )f ( y + x )dy = F 1 F u ( ) 2 assuming f is real, nonnegative A can have points missing if f is complex-valued or bipolar Solve the inverse problem: Given A, find the smallest set L inside of which all S fit, where A = S S JRF -17

18 Bounds on Object Support Autocorrelation: r f (x,y ) = f ( x, y )f * ( x x, y y )d x d y = F 1 F (u,v ) 2 a 1 0 Object Support a 2 Autocorrelation Support a 1 0 a 2 Triple Intersection of Autocorrelation Supports Triple-Intersection Rule: [Crimmins, Fienup, & Thelen, JOSA A 7, 3 (1990)] JRF -18

19 Triple Intersection for Triangle Object Object (a) Support Autocorrelation Support (b) Triple Intersection (c) Support Constraint Alternative (d) Object Support Family of solutions for object support from autocorrelation support Use upper bound for support constraint in phase retrieval JRF -19

20 Shrinkwrap 1 Example Initial Support Guess Optimize With Initial Support Guess Blur, Threshold, and Update Support [1] Marchesini, S., He, H., Chapman, H. N., Hau-Riege, S. P., Noy, A., Howells, M. R., Weierstall, U., and Spence, J. C. H., Xray image reconstruction from a diffraction pattern alone, Phys. Rev. B 68, (2003). JRF -20

21 Find set of optimization parameters, g (image pixel values), to minimize an objective function consisting of data consistency and prior knowledge g x,y Data consistency (example) Amplitude only: Phase Retrieval by Nonlinear Optimization ( ) = argmin Example priors Non-negativity Support g Φ data g ( ) ω i = weights on prior knowledge ( ) + ω i Φ prior,i g ( ) = W ( u,v ) FT g i ( ) 2 Φ data g 2 F u,v W = weighting (optional) Φ prior2 = Φ prior1 = ( x,y ) S { } ( x,y ) g( x,y )<0 g( x,y ) 2 g( x,y ) 2 S = points in support 2 F 2 = measured Fourier intensity Can also implement support by only varying values of g within S JRF -21

22 Minimize Error Metric, e.g.: Nonlinear Optimization Algorithms Employing Gradients E = Contour Plot of Error Metric u W u ( ) G( u) 2 F ( u) 2 2, G u [ ] ( ) = F g ( x ) Repeat three steps: Parameter 2 c a 1. Compute gradient: E, E, p 1 p 2 2. Compute direction of search b 3. Perform line search Parameter 1 Gradient methods: (Steepest Descent) Conjugate Gradient BFGS/Quasi-Newton JRF -22

23 E = u W u Optimizing over where G W ( ) G( u) F ( u) 2, G u ( u) = W u g(x ) = g R For point-by-point pixel (complex) value, g(x), For point-by-point phase map, θ(x), For Zernike polynomial coefficients, P [ ] can be a single FFT or multiple-plane Fresnel transforms with phase factors and obscurations Analytic Gradients ( x ) + i g I ( x ) = m o ( x )e iθ( x ), θ ( x ) = a j Z j ( x ) ( ) F ( u) G(u) F ( u) G ( u) [ ] ( ) = P g ( x ) E θ(x ) E g(x ) = 2 Im g x Analytic gradients very fast compared with calculation by finite differences J.R. Fienup, Phase-Retrieval Algorithms for a Complicated Optical System, Appl. Opt. 32, (1993). J.R. Fienup, J.C. Marron, T.J. Schulz and J.H. Seldin, Hubble Space Telescope Characterized by Using Phase Retrieval Algorithms, Appl. Opt (1993). A.S. Jurling and J.R. Fienup, Applications of Algorithmic Differentiation to Phase Retrieval Algorithms, J. Opt. Soc. Am. A (June 2014). J j =1 { } = 2 Im g W * (x ) { ( ) g W * ( x )} E = 2 Im g ( x ) g W * ( x ) Z a j x j x, and g W (x ) = P G W u ( ) ( ) JRF -23

24 Object: electron density function Periodic, continuous X-ray Crystallography Measurements Fourier intensity, undersampled Bragg peaks are half-nyquist sampled Missing phase Prior information Nonnegativity (usually) Atomiticity = object consists of a finite number of atoms within a unit cell Just need to recover the 3-D locations and strengths of a number of electron peaks To reconstruct an electron density function, need to retrieve the Fourier phase solve the phase retrieval problem An early form of computational imaging and of compressive sensing JRF -24

25 X-ray Crystallography Examples of Notable Achievements 29 Nobel Prizes associated with crystallography Bragg solved the first crystal structure, salt, in 1914 ² Nobel Prize in Physics, 1915 "for their services in the analysis of crystal structure by means of X-rays Sir William Henry Bragg & William Lawrence Bragg Nobel Prize in Medicine, 1962 ² "for their discoveries concerning the molecular structure of nucleic acids and its significance for information transfer in living material [DNA] Francis Harry Compton Crick, James Dewey Watson, and Maurice Hugh Frederick Wilkins Nobel Prize in Chemistry, 1985 ² for their outstanding achievements in the development of direct methods for the determination of crystal structures Herbert A. Hauptman and Jerome Karle JRF -25

26 Image Reconstruction from X-Ray Non-Crystallographic Diffraction Intensity micron size or smaller Target Detector array (CCD) Coherent X-ray beam Computer Image Nano-object (electron micrograph) Far-field diffraction pattern (Fourier intensity) J.N. Cederquist, J.R. Fienup, J.C. Marron, and R.G. Paxman, Phase Retrieval from Experimental FarField Speckle Data, Optics Lett. 13, (August 1988). J. Miao, P. Charalambous, J. Kirz, and D. Sayre, Extending the methodology of x-ray crystallography to allow imaging of micrometre-sized non-crystalline specimens, Nature (London) 400, (1999). H. Chapman et al., High-resolution ab initio 3-D x-ray diffraction microscopy, JOSAA 23, (2006). JRF -26

27 First Coherent Diffractive Imaging Experiment JRF -27

28 Ptychography (Transverse Translation Diversity) Moving the sample with respect to a known illumination pattern can provide suitably diverse measurements Makes phase retrieval more robust to stagnation, noise and ambiguities Fourier intensity measurements (one for each of several mask positions) Plane-wave Moveable object H. M. L. Faulkner and J. M. Rodenburg, Movable aperture lensless transmission microscopy: a novel phase retrieval algorithm, Phys. Rev. Lett. 93, (2004). M. Guizar-Sicairos and J. R. Fienup, Phase retrieval with transverse translation diversity: a nonlinear optimization approach, Opt. Express 16, (2008). JRF -28

29 Ptychography Reconstruction Example 98 Mega-pixel phase image of an Eiger detector array with 45 nm resolution M. Guizar-Sicairos et al., Highthroughput ptychography using Eiger - Scanning X-ray nanoimaging of extended regions, Optics Express 22, 14859, June 10, JRF -29

30 Fourier Ptychography G. Zheng, Breakthroughs in Photonics 2013: Fourier Ptychographic Imaging, IEEE Photonics J. 6, (April 2014) JRF -30

31 Unconventional Pupil-Plane Imaging Active Conformal Thin Imaging System Coherent illumination, sensed by a thin, conformal array of detectors Light weight, small depth, day/night With no imaging optics, a wider aperture fits on an airplane Giving finer resolution Phase retrieval used to form image Illumination-pattern constraint Coherent Illuminator Conformal Array of Detectors Conventional, depth ~ D T Focal plane detector array D T Real 2- D aperture or 1-D array with aperture synthesis Imaging lens ACTIS, depth ~ D P << D T Aircraft fusilage Discrete detectors Aircraft fusilage Fresnel Lenses D p D T D T D p JRF -31

32 Solution to Support Constraint Issues: Illumination Pattern with Separated Parts Uniqueness more likely for separated support (areas with zero in between) m T.R. Crimmins and J.R. Fienup, "Uniqueness of Phase Retrieval for Functions with Sufficiently Disconnected Support," J. Opt. Soc. Am. 73, (1983) Phase retrieval works particularly well for support constraints with separated parts m J.R. Fienup, "Reconstruction of a Complex-Valued Object from the Modulus of Its Fourier Transform Using a Support Constraint," J. Opt. Soc. Am. A 4, (1987) Parts separated in x and y give fringes in detected speckle pattern m m If one separated part was a delta-function, then would be a hologram Somehow makes phase more readily available JRF -32

33 Image Reconstruction Example Complex-valued SAR Image Illumination optics 1/4 diameter of receiver array + Hanning weighting Extended scene Laser illumination pattern Illuminated scene Far-field speckle intensity Pattern at sensor array Includes aperture weighting Reconstructed image JRF -33

34 PROCLAIM 3-D Imaging Phase Retrieval with Opacity Constraint LAser IMaging tunable laser direct-detection array M.F. Reiley, R.G. Paxman J.R. Fienup, K.W. Gleichman, and J.M. Marron, 3-D Reconstruction of Opaque Objects from Fourier Intensity Data, Proc. SPIE , pp (1997) ν 1, ν 2,... ν n collected data set initial estimate from locator set phase retrieval algorithm 3-D FFT reconstructed object reflecting, opaque object is on 2-D manifold embedded in 3-D volume sparse in z! JRF -34

35 Imaging Correlography Get incoherent-image information from coherent speckle pattern Estimate 3-D incoherent-object Fourier squared magnitude Like Hanbury-Brown Twiss intensity interferometry F I (u,v,w ) 2 D k (u,v,w ) I o [ ] D k (u,v,w ) I o [ ] k (autocovariance of coherent speckle pattern) Easier phase retrieval: have nonnegativity constraint on incoherent image Coarser resolution since intensity interferometry SNR lower JRF -35

36 Object for Laboratory Experiments ST Object. The three concentric discs forming a pyramid can be seen as dark circles at their edges. The small piece on one of the two lower legs was removed before this photograph was taken. JRF -36

37 3-D Laser Fourier Intensity Laboratory Data Data cube: 1024x1024 CCD pixels x 64 wavelengths Shown at right: 128x128x64 sub-cube (128x128 CCD pixels at each of 64 wavelengths) JRF -37

38 Coherent Image Reconstructed by ITA from One 128x128x64 Sub-Cube JRF -38

39 Imaging through Scattering Media Gather light scattered onto detector array, with or without lens Autocorrelate to get autocorrelation of object equivalent to Fourier intensity Phase retrieval to reconstruct incoherent image Related to Labeyrie s technique JRF -39

40 Summary of Phase Retrieval Phase retrieval can give diffraction-limited images Despite atmospheric turbulence or aberrated optics Despite lack of Fourier or Fresnel phase measurements Can work for many imaging modalities Passive, incoherent or laser imaging correlography Nonnegativity constraint Support constraint may be loose and derived from given data Active, coherent imaging Complex-valued image, so NOT nonnegative Support constraint: must be tight and of special type or helped by: 3-D, or reflective opaque object, or correlography Can be used for wavefront sensing, using simple hardware For atmospheric turbulence, telescope aberrations (HST, JWST), eye aberrations, optical metrology Can be used for beam characterization (laser, x-ray) Allows imaging system to be corrected for improved imagery JRF -40

41 need 10 s of nm alignment over 6.5 m James Webb Space Telescope 18 segments in folded-up configuration JRF -41