POSITIVITY CONSTRAINT IMPLEMENTATIONS FOR MULTIFRAME BLIND DECONVOLUTION RECONSTRUCTIONS

Transcription

1 POSITIVITY CONSTRAINT IMPLEMENTATIONS FOR MULTIFRAME BLIND DECONVOLUTION RECONSTRUCTIONS A DISSERTATION SUBMITTED TO THE GRADUATE DIVISION OF THE UNIVERSITY OF HAWAI I AT MĀNOA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY IN ELECTRICAL ENGINEERING DECEMBER 014 B Paul A. Billings Dissertation Coittee: Todd Reed, Chairperson Kungi Bae Olga Boric-Lubece Yingfei Dong Aaron Ohta Gar Varner Kewords: Paraeterization, Inverse Proble, Deconvolution

2 ABSTRACT Reconstruction of iager degraded b atospheric turbulence is inherentl an ill-posed proble. Multiple solutions can be found which satisf the easured data to the etent allowed b the noise statistics. The setric nature of a spatiall invariant iaging sste gives rise to an abiguit between object and distortion. Furtherore, portions of the object spatial frequenc inforation are often attenuated beond usefulness. Research in this field is priaril concerned with resolving this abiguit and aing a qualit estiate of the object in a tiel fashion. Man of the ore advanced iage reconstruction algoriths are iterative algoriths, seeing to iniize error/cost or aiize lielihood/conditional probabilit. The estiates, of course, are rando variables, and regularization and constraints are often eploed to guide solutions or reduce effects of overfitting. While positivit is an often eploed constraint, one can conceive of various was to achieve this beond the tpical approach of squaring the search paraeters. We evaluate several functional paraeterizations as well as the use of an optiizer eploing search boundaries. We also consider the order of application in conjunction with a soothness constraint. Perforance is quantified b etrics, including RMS error spatial doain, RMS phase error curves spatial frequenc doain, and tiing to characterize coputational deands. We find the RMS phase error curves to be the ost valuable as the assess reconstruction accurac as a function of level of detail. Of the paraeterizations evaluated, ipleentation of object positivit through squaring the paraeters prior to application of a soothness constraint achieved the best blend of accurac and runtie eecution speed. Developent of the theoretical bacground in a coon fraewor contributed to reuse of constituent quantities, both in the developent of the lielihood and an gradient epressions as well as ipleentation of those epressions into coputer code. Such reuse prootes accurac as well as ease of new developent. ii

3 TABLE OF CONTENTS ABSTRACT... ii List of Tables... v List of Figures... vi List of Sbols/Abbreviations... i 1 INTRODUCTION Bacground General Discussion Multi-Frae Blind Deconvolution Constrained Estiation Nonlinear Search Other Approaches to Iprove Perforance Research Description Significance of the Stud Theoretical Developent Lielihood Gradients Gradients w.r.t. Object Paraeters Gradients w.r.t. Point Spread Function Paraeters Noise Models Object Paraeterizations Pielated Object Soothness Constraint Positivit through the Square Positivit through the Eponential Positivit through the Absolute Value Positivit through Bounds Constraints... 3 iii

4 .3.7 Gradient Suar Object Paraeters Point Spread Function Paraeterization Phase Errors at the Aperture, Zernie Epansion Lielihood Maiization Siulation of Data Atospheric turbulence Sapling Noiseless Iage Foration Noise Accurac Metrics RMS Error Spatial Doain RMS Phase Error Spatial Frequenc Doain Results Siulation of iage data Ipleentation of constraints Iage Reconstruction Satellite Reconstructions, 40 db SNR Satellite Reconstructions, 0 db SNR Spoe Reconstructions, 40 db SNR Spoe Reconstructions, 0 db SNR Observations and Metric Results Suar/Conclusions References... 8 iv

5 LIST OF TABLES Table 1: Coon approaches used to iprove reconstruction perforance Table : Object paraeterizations to investigate Table 3: Lielihood gradient equation suar Table 4: Sapling reference raifications Table 5: Paraeterization run-tie etrics Table 6: Paraeterization accurac etrics, Satellite object... 7 Table 7: Paraeterization accurac etrics, Spoe object v

6 LIST OF FIGURES Figure 1: Point source short eposure iages through turbulence... Figure : a tpical raw frae and b 16-frae MFBD reconstruction... 4 Figure 3: True objects used in the iage generation Figure 4: Areas of significant spatial frequenc content Figure 5: Eaple object estiates that are a overfit and b regularized... 6 Figure 6: SNR-based sieve/filter design in the spatial frequenc doain... 7 Figure 7: Eaple error of a single reconstruction Figure 8: Indicator as Mu of spectral locations of interest Figure 9: Eaple phase differences Figure 10: Eaple RMSPE curve Figure 11: Tpical siulated iager, Satellite object with SNR=40 db... 5 Figure 1: Tpical siulated iager, Satellite object with SNR=0 db... 5 Figure 13: Tpical siulated iager, Spoe object with SNR=40 db Figure 14: Tpical siulated iager, Spoe object with SNR=0 db Figure 15: Square 1 reconstructions, Satellite object, 40 db SNR Figure 16: Square reconstructions, Satellite object, 40 db SNR Figure 17: Ep 1 reconstructions, Satellite object, 40 db SNR Figure 18: Ep reconstructions, Satellite object, 40 db SNR Figure 19: Abs 1 reconstructions, Satellite object, 40 db SNR vi

7 Figure 0: Abs reconstructions, Satellite object, 40 db SNR Figure 1: Bounds reconstructions, Satellite object, 40 db SNR Figure : Square 1 reconstructions, Satellite object, 0 db SNR Figure 3: Square reconstructions, Satellite object, 0 db SNR Figure 4: Ep 1 reconstructions, Satellite object, 0 db SNR Figure 5: Ep reconstructions, Satellite object, 0 db SNR Figure 6: Abs 1 reconstructions, Satellite object, 0 db SNR... 6 Figure 7: Abs reconstructions, Satellite object, 0 db SNR... 6 Figure 8: Bounds reconstructions, Satellite object, 40 db SNR Figure 9: Square 1 reconstructions, Spoe object, 40 db SNR Figure 30: Square reconstructions, Spoe object, 40 db SNR Figure 31: Ep 1 reconstructions, Spoe object, 40 db SNR Figure 3: Ep reconstructions, Spoe object, 40 db SNR Figure 33: Abs 1 reconstructions, Spoe object, 40 db SNR Figure 34: Abs reconstructions, Spoe object, 40 db SNR Figure 35: Bounds reconstructions, Spoe object, 40 db SNR Figure 36: Square 1 reconstructions, Spoe object, 0 db SNR Figure 37: Square reconstructions, Spoe object, 0 db SNR Figure 38: Ep 1 reconstructions, Spoe object, 0 db SNR Figure 39: Ep reconstructions, Spoe object, 0 db SNR Figure 40: Abs 1 reconstructions, Spoe object, 0 db SNR Figure 41: Abs reconstructions, Spoe object, 0 db SNR Figure 4: Bounds reconstructions, Spoe object, 40 db SNR vii

8 Figure 43: RMSPE for the Satellite object, 40 db SNR data Figure 44: RMSPE for the Satellite object, 0 db SNR data Figure 45: RMSPE for the Spoe object, 40 db SNR data Figure 46: RMSPE for the Spoe object, 0 db SNR data Figure 47: RMSPE for the Spoe object, 0 db SNR data, F 8 region viii

9 LIST OF SYMBOLS/ABBREVIATIONS Ter/Sbol Definition/Description * Denotes convolution in various equations A AMOS B ASIC b c.c. d D D d θ Φ f Denotes correlation in various equations Single paraeter in the MLE discussion Air Force Maui Optical and Supercoputing observator Bias level counts of a detector Application-specific integrated circuit Blur ernel or soothness constraint Cople conjugate Vector of all easured data Aperture diaeter of the iaging sste Distance fro the origin function The th easured iage Angular saple spacing subscripted to denote relevanc Linear saple spacing subscripted to denote relevanc The th basis function Focal length of the iaging sste I Fourier transfor FPGA FWHM g Γ H Field prograable gate arra Full width at half a The th noiseless odel iage Covariance atri The th pupil function cople aplitude i

10 Ter/Sbol Definition/Description h The th coherent point spread function I Identit atri I Indicator function L Lielihood function L-BFGS-B An optiization algorith with bounds constraint Penalt function weight λ Wavelength of illuination Mean rate in a Poisson PDF µ Mean subscripted to denote specific contet M An indicator function with values 0 or 1 MAP Maiu a posteriori MFBD Multi-frae blind deconvolution ML MLE σ A, A Maiu lielihood estiation N µ Noral Gaussian distribution with specified ean and variance N A, N o, N p Nuber of piels across the iage, object, and pupil arras n The noise distortion of the th iage o Object intensit values O.T. Other ters to be neglected P Pea value of the noiseless iage P Penalt function ipleenting a soft constraint p Paraeters describing the object PDF Probabilit densit function PSF Point spread function R Pupil radius Range between object and iaging sste pupil Choles factorization of a covariance atri r 0 Atospheric seeing paraeter r The residual of the th frae

11 Ter/Sbol RMS RMSE RMSPE RV σ w σ d s SNR τ 0 τ 0 θ u W p, W o WGN w.r.t. ω 0 ψ z Definition/Description Root ean square error RMS phase error spatial frequenc doain Rando variable General noise variance Read noise of the detector The th point spread function Signal to noise ratio Atospheric decorrelation tie Atospheric decorrelation tie Vector of all paraeters Two diensional indeing variable, tpicall spatial frequenc Spatial width or etent of the pupil and object White Gaussian noise With respect to Fundaental frequenc for the discrete Fourier transfor Two diensional indeing variable Two diensional indeing variable Phase of the aberrated wavefront across the aperture Two diensional indeing variable i

12 1 INTRODUCTION 1.1 Bacground General Discussion With an ground based incoherent iaging sste, phase aberrations are often the liiting factor in achieving resolution.[1] These aberrations arise fro a nuber of causes, with atospheric turbulence being one of the largest contributors. As light propagates fro the object to detector, inor variations in the inde of refraction of air cause phase delas between various ras, potentiall giving rise to a degraded iage. Of priar interest is near-field turbulence, when the iaging sste captures these ras ver soon after the are distorted. This is ver tpical with ground-to-space iaging applications. David Fried studied this issue in depth [] and noted that there is spatial correlation to the teperature variations. He developed the concept of a characteristic size, tered the atospheric seeing paraeter and denoted b r 0, within which the inde of refraction could be regarded as the sae. While varing with tie of da, local terrain, and other factors, tpical values of r 0 are 1 to 0 c for visible wavelength illuination. The aperture of an optical sste gathers the light originating fro the various points in the scene. As long as the aperture diaeter is approiatel less than r 0, all light ras captured in a given instant will have undergone about the sae refraction b the atosphere. Convenientl, little distortion will occur beond possibl a unifor shift of the iage. Unfortunatel, the drive to eaine dier objects and longer ranges requires the capture of ore photons, which is achieved with increasingl larger apertures. The drawbac to larger apertures is that not all the ras that are captured have eperienced the sae refraction, having gone through different areas of air and turbulence. The ras fro each pocet of air will eperience a different phase dela which causes a local tilt of the wavefront, giving rise to a distorted iage. The larger the aperture, the ore pocets of air 1

13 are involved, and the ore distorted the iage becoes. Thus, the ratio of aperture to seeing size D/ r 0 is a easure of how distorted an iage will be. Eaple iages of a single point source object are shown in Figure 1. Brighter piels are shown with a greater aount of blac throughout this docuent. B definition, these are the Point Spread Functions PSFs for those particular atospheric realizations. a b Figure 1: Point source short eposure iages through turbulence for a D/ r 0 = 16 and b D/ r 0 = 1 Furtherore, Fried noted a teporal correlation of the inde of refraction, with the decorrelation tie denoted b τ 0. Iaging sensors have finite eposure ties. As long as the eposure tie is approiatel less than τ 0, the atosphere is frozen and onl spatial variations need to be considered. Long eposures fro large apertures are etreel blurred due to the low-pass nature of this integration. Several decades ago, Laberie noted that the spatial frequenc content of short eposure iager etended nearl out to the diffraction liit of the optics well beond that of long eposure iages.[3] He pioneered the field of specle iaging, deonstrating that inforation was not lost, erel scrabled. The tric to unscrabling la in the deterination of the phase aberrations.

14 Measureent of these phase aberrations is possible e.g., using a Shac-Hartann wavefront sensor, and the etension of these ideas lead to the area of adaptive optics.[4,5] This is not a cure-all, however. Measureent errors, delas, iperfect correction, as well as greatl increased sste copleit and cost often prohibit going down this road. On the other hand, several ethods eist to deterine and correct these phase aberrations directl fro the iager itself using various transforations that are invariant to the effects of atospheric turbulence [6], e.g., Kno-Thopson [7] or triple-correlation/bispectru [8]. One drawbac to these ethods is that the norall require nowledge of the average pupil aplitude function coonl obtained fro separate easureents of a point source with the assuption that the atospheric characteristics are siilar during the two ties of observation. Another technique to deal with the phase distortions is phase-diversit, which was first entioned b Gonsalves [9,10] and generalized b Paan et al. [11] to accoodate an arbitrar nuber of diversit easureents. Two or ore iages are recorded siultaneousl, differing onl b a nown aberration coonl defocus. Estiation theor is used to deterine the ost liel coon object giving rise to the different iages. This was later etended to phase-diverse specle iaging b using ultiple diversit iages recorded at different ties.[1] Each of these additional sets of iages is ade after the atosphere has decorrelated. Diversit is the e to these estiation approaches. As with ost tpes of estiation, eploing ultiple uncorrelated easureents reduces the errors. In our case of incoherent iaging, there is a single coon object that gives rise to each of these different iages, which is a ver powerful constraint. One of the drawbacs to phase-diverse specle iaging is that the captured photons ust be split aong the various siultaneous eposure iage planes. This reduces the signal to noise ratio of an given iage, which can be a ajor proble for faint objects. The degenerate case is a single-channel phase-diverse specle approach in other words, it uses diversit in tie onl. Others have derived this technique fro a different direction, and we use their ter in the reainder: ulti-frae blind deconvolution MFBD.[13-18] The phrase ulti-frae is due to the use of ultiple short eposure fraes collected at different ties through independent atospheric realizations. To support the assuption of a coon object, the scene ust not 3

15 change appreciabl throughout the iage enseble. Deconvolution is the process of estiating the object b observing the convolution of object and nown point spread function PSF. The PSF is said to be deconvolved fro the easured iage data, ielding the object. Blind is used to denote the fact that we do not now the point spread functions i.e., the atospheric distortions, and the are jointl estiated along with the object Multi-Frae Blind Deconvolution We assue space-invariant incoherent iaging, regarding our iages as the twodiensional convolution of a single object and the various PSFs. Furtherore, we use the discrete representation to avoid the intractabilit of estiating analog wavefors. The th noiseless iage in the enseble is fored as: = o ' s g = o * s ' 1 where o is the object, s is the th point spread function, and is a two-diensional coordinate variable. The operator * denotes convolution. The goal, of course, is to estiate o and s fro nois data d, and the following for for the data is assued: ' d = g n + a b Figure : a tpical raw frae and b 16-frae MFBD reconstruction, D/r 0 =16, SNR=40 db 4

16 There are two schools of thought with regard to an estiation approach. In one case, aiu lielihood ML estiation, the paraeters of interest are regarded as deterinistic quantities; the onl source of randoness is the noise. The second approach, aiu a posteriori MAP estiation, regards each of the paraeter values as a realization of an underling rando process. While the distinction is a bit subtle, MAP estiation has the potential for ore accurate estiates if the underling rando processes can be statisticall described. The following sections contain a brief review of these two tpes of estiation [19], priaril to establish notation and draw coparisons that are coonl used in the iage reconstruction field Maiu Lielihood Estiation A siple eaple will be used to illustrate the aiu lielihood fraewor to estiate the unnown paraeters of interest. The fundaental basis of MLE is that the paraeters are non-rando. The onl source of randoness is the noise which corrupts the various data, denoted b w. For this proble, we assue a odel with a single paraeter A to describe the various easureents dn: d n = A + w n 3 The noise is assued to be white Gaussian noise WGN or Gaussian distributed with zero ean, variance σ w, and uncorrelated fro saple to saple. As a result, the data will also be Gaussian distributed but with a ean of A. The probabilit densit function PDF of d is paraeterized b the variable A and will be denoted with a seicolon. p d n; A = 1 πσ w 5 1 d n A ep σ w Often independence of the noise saples is assued which allows the PDF of a set of rando variables to be fored as the product of their arginal distributions. This also holds true in this particular case of uncorrelated Gaussian RVs. Denoting the set of N easureents b d, its PDF is jointl Gaussian distributed in N diensions: 1 1 p d ; A = p d n; A = ep d n A N n σ 5 w n πσ w 4

17 For fied values of dn, this can be regarded as a function of the paraeter A, and this function is called the lielihood function. The goal of aiu lielihood estiation is to identif the location of the pea in the lielihood function; i.e., identif the paraeter value ost liel to have given rise to the easured data: Aˆ ML = arg a p d; A 6 A This location is unchanged b onotonic transforation, so in practice the logarith of the PDF is used as the lielihood function. This is ore correctl called the log-lielihood function, but is often shortened as it, too, satisfies the intuitive nature of lielihood. Further, ters which do not contain the paraeters of interest are ignored as the do not affect the location of the pea the erel shift the entire function up or down, allowing et another forulation of the lielihood function. These ters will be denoted O.T. for clarit. L 1 Aˆ ln p d; Aˆ = d n Aˆ O.T. d = + 7 σ Note the diensionalit of this function is equal to the nuber of paraeters N θ, one in this case. Also note the change in notation where the pipe is used to denote given the realization of a particular rando variable. Finding the location of the pea of the lielihood function is relativel straight forward, involving equating the derivative to zero and solving for the paraeter of interest. When ultiple paraeters are involved, the solution is the point in the N θ -diensional space where all partial derivatives equate to zero. For this siple eaple, we have: d L daˆ w Aˆ d ˆ 1 d 1 A d = d n Aˆ + O.T. = d n Aˆ 0 σ d L daˆ n = 0 = ˆ n w da σ w n Solving for Â, we find the aiu lielihood estiate to be: 8 A ˆ 1 ML = d n N 9 n 6

18 1.1.. Maiu A Posteriori Estiation Maiu lielihood estiation is often utilized due to its siplicit. It needs no other inforation other than a odel and data. As previousl entioned, however, it stands to reason that ore accurate estiates a be possible if ore inforation is available. In the aiu a posteriori fraewor, the paraeters of interest are regarded as realizations of rando variables, the possible values of which can be described in a statistical anner. For eaple, that in addition to the N saples of data, we also now that the rando variable A is norall distributed as N σ µ. A, A As before, we wish to obtain our estiate as the solution to an equation involving our estiate. As can be guessed fro the nae of the fraewor, we see to identif the location of the aiu of the a posteriori distribution: A d Aˆ = MAP arg a p 10 We turn to Baes Theore which relates the conditional probabilities of these rando quantities: A A d p p A d = p d A 11 p The first ter pd A was derived in the aiu lielihood section though with the notation pd; A, and the net ter is the ratio of the unconditional densities. This is ver intuitivel satisfing, as it effectivel aplifies or attenuates various parts of the aiu lielihood forulation in accordance with the a priori inforation. As before, we can define several related lielihood functions based on pa d, here using the logarith and ignoring ters not involving A since the do not influence the location of the function pea: L A ˆ = ln p Aˆ d = ln p d Aˆ + ln p Aˆ + O.T. d 1 Substitution of the densities gives a lielihood function with two useful ters: 1 1 Aˆ = ˆ d n A Â - µ O.T. L d A + 13 σ σ w n A 7

19 We note a data-driven ter, which was present in the aiu lielihood forulation as well as a second ter which reduces the lielihood as the estiate deviates fro the a priori epected value. Proceeding as before, we equate the derivative of the lielihood function to zero in order to find the estiate. Ka addresses this derivation in Eaple 10.1 in [19] where he finds the aiu a posteriori estiate to be: Aˆ σ ˆ A AML + σ MLµ A MAP = 14 σ A + σ ML where σ = σ N is the variance of the aiu lielihood estiate. Fro this, we see the ML w MAP estiate is a weighted average of the data-driven aiu lielihood estiate and the a priori estiate the ean µ A. Eaination of Eq. 14 shows a natural fusion or 8 Â ML blending of inforation as the nuber of saples grows. With few data points, the estiate is heavil influenced b the a priori inforation; with an points, the data-driven estiate doinates Constrained Estiation We contend the priar difference in ML and MAP estiation is the injection of a priori inforation. The MAP estiation procedure can leverage ore than the nois easureents, and thus should be ore accurate than ML. In practice, however, it is difficult to describe the statistical nature of the underling rando processes i.e., specif the prior densities. Consider the case of the object brightness. We usuall now little ore than that the piel values ust be positive. While it is possible to deterine an upper bound requiring nowledge of the source intensit, attenuation of an intervening edia, positions of source and object, the angularldependent reflectance of the unnown object, and so forth, this is usuall intractable, coputationall prohibitive, or sipl not a tight enough bound to be useful ML with Soft Constraints Neglecting the philosophical differences of rando vs. deterinistic paraeters, we have seen the ML and MAP estiator equations are rearabl siilar. Indeed, under the Gaussian noise assuption, the MAP estiator sipl has an additional ter, which serves to lower the lielihood score of the ML estiator. This has been generalized to the use of penalt ters

20 to reject undesirable solutions. We broaden our notation to include a nuber of paraeters organized into a vector θ. L soft θ ˆ d L θˆ d λp θˆ d = 15 ML For instance a solution containing negative object piel values can be ade less desirable b augenting the ML equation with a ter that increases with ore or larger negative values: ˆ d = P o Pθ = o 16 { o < 0 } Soothness in the intensit is usuall a desirable propert for an an-ade objects ecept around edges. One ter which penalizes rapid spatial intensit variations is based on the Laplacian operator: ˆ d = P o = o Pθ 17 The Laplacian operator is often approiated b the difference of a point and the average of its neighbors. Multiple constraints can sipl be taced on as additional ters, each weighted with its own strength coefficient. On the surface, this is all ver pleasing. In each of these, ore inforation is present than the raw easureents, hopefull ielding ore accurate estiates. The choice of penalt strength λ, b design, will greatl influence the solution, but this is unfortunatel a double-edged sword. An optial value is usuall proble specific and often ust be deterined i.e., tuned b trial and error or soe variant of supervised learning techniques.[0,1] Multiple constraints can be especiall probleatic if there are undesired cancellation or aplification effects. Penalt ters are tpes of soft constraints because soe violation is possible or even desired ML with Hard Constraints In contrast, a hard constraint will alwas be satisfied. There is no gra area, and therefore, no tuning constant or constants with which the researcher ust fiddle i.e., the various λs. To ipleent hard constraints, the proble is redefined in such a anner that the constraint ust be satisfied. Consider for eaple, the positivit technicall, the non- 9

21 negativit of the object intensit values. Following Biraud s approach [], we can redefine the object values to be the square of the paraeters of interest: o = p 18 With this change, the object intensit is alwas non-negative. A hard constraint has been applied, hopefull ielding ore accurate solutions Nonlinear Search A side effect of an hard constraints is that the lielihood equation is obviousl nonlinear in ters of the paraeters of interest. Indeed, b the var nature of the incoherent iaging proble, the object and PSF paraeters cobine in a nonlinear anner through the convolution operator. Even further, the use of a realistic noise assuption e.g., Poisson also ields a nonlinear equation. Ultiatel, closed for epressions for the paraeter estiates are not tpicall available, andating the use of nuerical techniques i.e., iterative to identif the paraeters which aiize the lielihood function. Consider the nature of this lielihood function. If the recorded iages are 1818 piels, a tpical ipleentation requires soe siteen thousand variables paraeters to describe the object intensit alone. While there are soe techniques to reduce this e.g., a size assuption, a..a. a support constraint, the lielihood function is nevertheless ver ultidiensional. Recall that the goal of the estiation process is to identif the location of the pea of a ulti-diensional hper-surface. The aiization of a ultidiensional function is no trivial atter. The dependenc of the function on the paraeters plas a crucial role in how difficult that optiization is. It is natural to wonder if these hard constraints a have ade it ore difficult to find the solution pea. Are there other paraeterizations of positivit that give rise to a hper-surface that are ore easil searched? What about using the original forulation directl searching for o but using an optiizer with bounds restrictions i.e., onl searching in the area o 0? Other Approaches to Iprove Perforance The ajorit of the effort in this field has been, in one for or another, an attept to increase perforance. Perforance is a bit vague, but perhaps rightl so, as eecution tie and qualit are easil traded. Consider an advanceent that reduces the tie per iteration. After 10

22 this iproveent is ipleented, the researcher can stop sooner for the sae qualit or the can run for the sae length of tie and achieve better qualit. Conversel, an advanceent that aes larger progress toward the solution need not run as long to be acceptable. Individual circustances will usuall dictate which aspect is the higher priorit. The approaches to iproveent can be grouped into two categories as described in Table 1: hardware/software developent and those related to the iage reconstruction algoriths theselves. The ost obvious solution to iproveent is sipl to run on faster hardware. The perforance differences when eecuted on coputers that are just a few ears ore recent are aazing. In this categor, we also lup iproveents due ore efficient software developent ethods such as reusing coputed quantities or use of copiled languages. Additionall, one can tae advantage of parallel resources. MFBD reconstructions are etreel aenable to the ebarrassingl parallel approach because individual reconstructions tpicall do not depend on each other. If latenc can be tolerated, this often achieves aiu throughput. Soeties, however, it is desired to ae a single reconstruction as fast as possible iniizing latenc, and finer grained parallelis is warranted. MFBD is also quite aenable to this approach, as the individual fraes are tpicall independent, with a bit of consolidation for the coon object calculations, as will be seen later in the theoretical developent. Lastl, dedicated ipleentation of parts of the processing logic in hardware can pa dividends. For eaple, we will see that the nuber of Fourier transfors plas a e role in runtie per iteration. Specialized arra processors, field prograable gate arras FPGAs, or application-specific integrated circuits ASICs are all approaches that can accelerate part of the processing. The second categor of iproveent is to ae iproveents in the algoriths theselves. Man of these attepts involve the incorporation of a priori inforation. This research is one eaple of this categor. Additional constraints serve to reduce the nuber of potential solutions which a ae it easier for the search algorith to find an acceptable solution. 11

23 Object positivit, soothness, and support are all etreel coon constraints. Beond that, approaches are quite varied. Phase-diversit is coon for iaging of the sun where reduced SNR due to the partitioning of photons onto ultiple detectors is less of a concern.[3,4] Others have used Kologorov statistics as prior inforation for the point spread functions.[5] In the case of nown object classes, incorporation of a priori inforation on the power spectru of the object can iprove reconstructions as well.[6] Other research has ade use of the tendenc for the high spatial frequenc content of natural scenes to be sparse i.e., an frequencies have a agnitude near zero.[7,8] The ethod b which the soothness constraint is achieved also varies. While an use convolution with a relativel sooth ernel, others have sought to iniize the total variation roughl ain to path or arc length as a function varies.[9,30] An additional constraint or assuption often iposed on the PSFs is that of phase-onl distortions in the aperture. We discuss this in section.4.1. Originall adopted in order to reduce the nuber of degrees of freedo, reconstructions eploing this assuption were found to often be fairl accurate even in the presence of agnitude distortions scintillation at the aperture.[31] Particulars of the iaging sste can also suggest constraints. For eaple, truncation of the object b the edge of the field of view can be eplicitl odeled with a guardband [3,33] and object inforation outside of the field of view estiated fro inforation that was blurred into the frae b the atospheric distortion. Such a constraint is crucial to avoid ringing artifacts fro the periodic assuption iposed b the use of the discrete Fourier transfor while processing. Detector piel saturation and grascale quantization [34] of the raw iager can be eplicitl odeled and reconstructions iproved as a result. The piel sapling or angular field of view of the iages can be odeled and a de-aliased object estiated at a higher spatial sapling rate.[35,36] Another possibilit is to start the search fro a point closer to the final solution possibl leveraging other solution techniques e.g., bispectru [8,37] or an MFBD estiate based on previous fraes in the foration of the initial estiate. Finall, iproveents a be found through the use of different optiization algoriths. Man of the ver earl MFBD papers describe searching the paraeter space through iterativel appling constraints in the spatial and 1

24 frequenc doains, a specialized case of generalized projections.[15,38] Later approaches e.g., Lane [16] cast the proble in ters of iniizing a cost or error function using an iterative search algorith such as steepest-descent, conjugate gradient, or epectation aiization search algoriths. Presentl, conjugate gradient and liited eor quasi-newton ethods of optiization of a lielihood or cost function see to doinate tpical use, but these are certainl not the onl approaches.[39] In addition to the search technique, the quantities being sought have changed over tie. The earl forulations circa id-1970s and 1980s alternated between the object and PSF paraeters. However, in this research, lie that of an others after that tiefrae, we search for the object and PSF paraeters siultaneousl. Interestingl, soe of the ost recent research into search techniques has resurrected the notion of focusing on one set of paraeters at a tie.[40,41] Table 1: Coon approaches used to iprove reconstruction perforance Hardware/Software Developent Algorith Enhanceents Newer coputers Constraints / a priori inforation Ipleentation choices Positivit, soothness, support, setr, Copiled languages phase-onl distortions, etc. Floating point coprocessors Multiple processors parallelis FPGAs / ASICs Multiple easureents fraes or wavelength Iproved initial estiates Search or optiization technique For contet, this research sees to increase perforance b cobining aspects of these two categories. Fro the Software Developent side, there are an approaches to ipleenting or achieving a desired end result. Fro the Algorith side, we adopt one of the ost coon constraints as that desired end result. These are bolded in Table 1. Specificall, we investigate the particular anner in which positivit is ipleented in the reconstruction software. The hpothesis is that perforance will var across the different ipleentation approaches. 13

25 1. Research Description In the Bacground section, we noted it is desirable to augent aiu lielihood estiation with constraints. In this stud, we wish to investigate the ipleentation of a positivit constraint in an estiation procedure. While we consider four paraeterizations for evaluation, the near universal use of a soothness constraint b other researchers andates its inclusion in this wor as well. As will be described in section.3, perforance is affected b the order of application of the positivit and soothness constraints, and both orderings are evaluated for each tpe of positivit ipleentation. The following approaches are considered with equation nubers taen fro section.3: Table : Object paraeterizations to investigate Description Paraeterization Eq Square 1 p * b 56 Square [ ] p * b 57 Ep 1 ep p * b 6 Ep ep[ * b ] p 63 Abs 1 p * b 71 Abs p * b 7 Bounds p * b 8 In the Bounds case, a bounds-constrained optiizer is used to search over the region where p is non-negative, and soothness ust be applied after positivit. In all cases, the soothness constraint is achieved b convolution with a blur ernel b. In the author s discussions with other researchers and literature search, the Square paraeterization is universall eploed. Criteria used for evaluation are each ethod s qualit of reconstruction root ean square or RMS spatial doain error and the RMS phase error in the spatial frequenc doain and coputational deand tie required for a constant nuber of search iterations. Etensive attepts at code optiization were perfored for ore accurate coparison of the techniques as opposed to their software ipleentations. 14

26 Perforance of each object paraeterization was evaluated across four data sets. These data sets are the cobination of two different objects or scenes and two different levels of signal to noise ratio corresponding to two levels of object brightness. The selected objects differed significantl in shape and spatial frequenc content shown in Figure 3 and Figure 4. a Figure 3: True objects used in the iage generation: a Satellite and b Spoe objects b a b Figure 4: Areas of significant spatial frequenc content based on the true spectru of a Satellite and b Spoe objects 15

27 1.3 Significance of the Stud In the broadest sense, we wish to optiize reconstruction perforance through the use of constraints. In our discussions with various researchers, an of the who eploed a positivit constraint used the technique of squaring the soothed paraeters i.e., the Square paraeterization, but soething better a be available. The notion of better a be tied to reconstruction qualit or speed of reconstruction. Advances in sensor technolog have given rise to increasingl larger arras. While the coputation capabilit of coputers has also advanced, the coputational burden of nonlinear iage reconstruction is decidedl non-trivial. The state of the art has turned to assive parallelization to decrease the tie spent on a given reconstruction, and near real-tie results are now available. Even a odest savings in coputational copleit will pa dividends due to the ultiple iterations that are tpicall required. It a be possible that with a less cople lielihood surface, fewer iterations would be required to achieve the sae lielihood or result. We first developed a generalized, theoretical fraewor that is conducive to eploration of various ipleentations of hard constraints. This fraewor isolated the choice of noise odel fro the choice of object or PSF paraeterization, ielding a consistent procedure for evaluating these constraints. While we applied this fraewor specificall to object positivit, it is b no eans liited to that. Other constraints for either the object or PSF can be eained equall as well. We then evaluated several ipleentations of object positivit and their interaction with a coon soothness constraint and found a clear benefit to using soething other than the standard ethod. The benefit in this case was faster eecution with no discernable degradation to the accurac of reconstruction, as will be shown in Section 3 Results. 16

28 THEORETICAL DEVELOPMENT.1 Lielihood Gradients We continue the developent of Multi-frae Blind Deconvolution within the aiu lielihood fraewor b considering the nature of the noise corrupting our noiseless iages. We assue a zero-ean noise distribution. The average value, then, at a particular piel will equal that of noiseless iage g. We also assue the noise is independent fro piel to piel and fro iage to iage. As a reinder, g is an iplicit function of the object and th point spread function atospheric distortion. Following the developent of section , we see to deterine a lielihood function for our set of easureents. In the case of MFBD, those easureents coprise the set of piel values in the iages {d }, so the lielihood function is: L = o, { sˆ } { d } = ln p { d }; { g } ln p d ; g ˆ 19 The ML estiates of the object and PSFs are found b identifing ˆ o and { ˆ } s that aiize the lielihood function. This is found b finding the point at which the partial derivatives of the lielihood function vanish. We use the chain rule etensivel throughout this developent and also drop the dependenc and estiate/hat notation on the lielihood function. We start b developing an epression for the lielihood gradient with respect to w.r.t. the object paraeters. 17

29 Gradients w.r.t. Object Paraeters Since we plan to discuss several object paraeterizations see section 1.1.5, we will first investigate the lielihood with respect to a particular object piel value and then consider the effect of the object paraeterizations on that piel value. We recall the chain rule for of the derivative with respect to o, where is a diensional indeing variable: = o g g L o L 0 The last derivative is that of the spatiall-invariant incoherent iaging equation: [ ] * s z s z o o s o o o g z = = = 1 The first ter depends on the particular noise odel assued. For now, denote this ter b: g L r It will soon be seen that this ter acts as a residual ter that describes how well the estiate atches the data. With this definition, we have the lielihood gradient with respect to the th piel in the object estiate as: = s r o L 3 The suation over is recognized as the definition of a correlation giving the lielihood gradient as an accuulation of the correlations between the residuals and PSFs: = s r o L 4 We reind the reader that unlie the convolution operator *, the correlation operator is not coutative.

30 19 At this point, we recall that the th piel of the object estiate o is itself a function of our search paraeters p discussed in section Eploing the chain rule one last tie, we find the lielihood gradient with respect to our object paraeters of interest as: p o s r p o o L p L = = 5 In the reaining developent, we derive epressions for r under three different noise odels section.. We also copute the gradients of the various object paraeterizations under investigation section.3. As a practical atter, these specific instances erel plug into the general epression given in Eq Gradients w.r.t. Point Spread Function Paraeters In addition to the object paraeters, blind deconvolution sees to estiate the paraeters describing the various point spread functions. Due to the coutative propert of the convolution operator, the lielihood gradient with respect to the PSF paraeters is siilar to that for the object paraeters. The fundaental difference is that a given PSF paraeter is specific to a particular frae, and the lielihood gradient w.r.t. that paraeter will onl contain contributions fro its associated frae. We are first concerned with the derivative of the noiseless iaging equation with respect to the th piel value in the th PSF. This is the analog of Eq. 1. = = z z z o z s s z o z s s s g 6 While we iplicitl interchanged the order of suation and derivative operators in the object developent, here we show this epression to highlight the subscripts on the PSFs. This gradient is zero for, and we can represent this with an indicator function I- also nown as a unit saple sequence[5]. = = 0 0, 0 1, w w w I 7 Using this notation, we have: I o I z I z o s g z = = 8

31 This indicator function will filter the suation over that was ept in the previous developent, and a correlation is recognized as before. The gradient with respect to the PSF piel value is thus: L = r o I = r o 9 s The final step is to consider that the PSF piel values could be represented b soe function of the PSF paraeters. As an eaple, positivit as well as noralization could be ipleented. For now, we consider the gradient with respect to a particular paraeter q j, which is the j th paraeter of the th PSF: L = q j L s s q = j s r o q j 30. Noise Models We net consider the effect of several noise odels. The particular choice of noise odel will dictate the for of the lielihood function as well as the residual ter r in the lielihood gradients Poisson Noise The quantized nature of the incoing illuination gives rise to rando arrival ties of the photons. Equivalentl, the nuber of photons landing on our detector during a fied eposure tie is rando. The average nuber of photons to arrive on a particular piel is described b the noiseless iage g. An particular realization, however, is governed b a Poisson distribution. Recall the probabilit densit function for a Poisson rando variable n with an average nuber of events λ: p Poisson n λ λ e n; λ = 31 n! Arguabl a probabilit ass function ; however, we will generalize to other non-discrete PDFs later and view densit function as the ore general ter. 0

32 Fro 19 and ignoring ters that are irrelevant to aiization, a Poisson noise odel gives a lielihood function of d ; g = L = ln p d ln g g 3 Poisson The gradient residual ter is then: Poisson L d r = = 1 33 g g Now that we have a concrete epression for r, we can see the residual nature that contributes to the overall gradient coputations. When the object and PSF estiates cause the estiated noiseless or rendered iage to be coparable to the easured iage, the gradient becoes sall. Fro that standpoint, those estiates are viewed as close to optial. In practice, it is undesirable for this ter to be eactl zero due to a tendenc to fit the noise as described in section.3.. Fortunatel, overfitting is less of a proble as ore fraes are used. One final coent is that these epressions are ill-defined as g approaches zero. Consequentl, care ust be taen during ipleentation to avoid nueric issues in such regions for reconstructions of space-based scenes...1. Gaussian Noise Most iaging detectors tpicall have noise associated with the read-out and associated electronics as well as noise due to theral variations. This noise is ver well described b a Gaussian probabilit densit function with a constant variance. Use of this densit is appropriate when read noise is relativel large doinating the photon noise or the scene has relativel low contrast and the photon noise consequentl has approiatel constant variance. 1 d µ 1 σ p Gaussian d; µ, σ = e 34 σ π With the realization that the ean value of a piel equals that of the noiseless iage, 19 can be used to find a tentative constant-variance Gaussian lielihood function of [ d g ] 1 L Gaussian, 1 = ln p d ; g = lnσ 35 σ 1

33 We eplicitl note here that the effect of a constant variance σ is a bias and scaling but does not alter the location of the pea of the lielihood function. Thus, an equivalent lielihood function that can be aiized is: L Gaussian = [ d g ] The previous noise odels address two different sources of noise, and the effectiveness of each odel will depend greatl on the relative contributions of those noise sources to the iager. It is reasonable to epect that the Poisson noise odel will be ore accurate when read noise is negligible; but unfortunatel, read noise is not alwas negligible. Also in an cases, piel values are biased functions of the nuber of photons captured during the eposure tie of the iage; i.e., a piel eposed to an effective intensit level of zero has an output value greater than zero counts. In the quest for a ore faithful odel of the entire iaging sste, an iaging sstes can afford to characterize their detectors and estiate the bias signal B and read noise variance σ d. Accounting for these effects separatel fro the object intensit will ield a ore accurate photoetric characterization of the object itself. We present one final noise odel that allows the incorporation of detector noise characteristics in a straightforward anner as well as preserving the signal-dependent and thus usuall spatiall dependent noise that is present due to photon arrivals. Due to the Central Liit Theore [4], we note that the PDF of a Poisson rando variable with an epected count λ of ore than about 100 is ver well approiated b a Gaussian curve with variance and ean equal to that epected count. We also can leverage the fact that the variance of the su of two Gaussian rando variables is equal to the su of the variances, allowing eas cobination of the 36 Fro this, we see that this noise odel is intuitivel satisfing fro a least-squarederror interpretation. Maiizing this lielihood function is equivalent to iniizing the squared error. The gradient residual function easil found to be: Gaussian r L = = [ d g ] 37 g Finall, we see these two ters L and r reain well defined as g tends toward zero Spatiall Dependent Gaussian Noise

34 3 effects of detector read noise variance σ d and photon noise variance g. We assue equal eposure ties for all fraes in the enseble and therefore epect equal read-noise bias and variance for each piel. Looing bac at 35, we can no longer eliinate the variance fro the lielihood function since it is not constant to all ters of the suation. We instead consider a spatiall dependent variance: varphoton noise varreadnoise g d + = + = σ σ 38 Thus, the lielihood function with a spatiall-dependent Gaussian noise odel with constant bias signal B is given b: [ ] ln ; ln B g d g d p L SVGaussian σ σ + = = 39 To copute the lielihood gradient, we first substitute an interediate variable w : B g d w = 40 so that the lielihood function can be written as: ln w L SVGaussian σ σ + = 41 and the derivative with respect to an arbitrar noiseless iage piel g is then: 1 4 g g w w g w g L σ σ σ σ σ + = 4 We net note the derivatives of w and σ with respect to the arbitrar piel are nonzero onl for = and =, which can be reflected with indicator functions as in.1.: I I w g = 43 I I g = + σ 44

35 4 These indicator functions will reduce the suations in the lielihood gradient to ters involving the indices and onl, and we can distribute the negative in front of the suations to find: 1 4 w w g L σ σ σ + = 45 For a slightl sipler epression, we can factor w out of the first ter: 1 4 w w g L σ σ σ + = 46 and eplo a coon denoinator of σ : 1 w w g L σ σ + = 47 We now realize that and are arbitrar indices and replace the with and to atch the previous definition of the gradient residual epression. Additionall, we substitute the definition of w to find the residual function without that interediate variable as: 1 B g d B g d g L r SVGaussian σ σ + = = 48 Finall, we note that the ajorit of real-world detectors will have a non-zero read noise variance. In contrast to the Poisson forulation which had g alone in the denoinator of the residual function, inclusion of an estiate of the read noise variance g d + = σ σ will greatl increase the nuerical stabilit of L and r for near-zero piel values in the noiseless iage g. A siilar liiting effect is seen on low-aplitude spectral ters when perforing Wiener filtering.[43] As one can iagine, near-zero values are liel to be coon in g when reconstructing iages of orbiting objects..3 Object Paraeterizations At this point, we have eained the lielihood function for several noise odels and have epressions for the lielihood gradients which involve a residual ter and the partial derivative of the object intensities with respect to the object paraeters. In this section we

36 discuss several paraeterizations of the object in order to achieve positivit. In addition, we also discuss a ver iportant constraint, that of soothness. The forward or rendering process will convert fro the set of object paraeter values {p} to the set of object intensit values {o} used in the calculation of the noiseless iage estiates..3.1 Pielated Object The first paraeterization ethod is not uch of a stretch of the iagination. The object piel values are identicall equal to the paraeter values: o = p 49 With an unconstrained optiization ethod, it therefore is possible for object piels to tae on negative values. This is undesirable considering that these piels represent intensit values. In practice, an of the estiated piel values will be positive b virtue of the lielihood function tring to atch the ostl positive data. However, ringing near edges where the bacground intensit is ver low or even zero e.g., ground-based iager of objects in orbit will liel be present with estiated intensities below zero. Secondl, such a paraeterization has a great an degrees of freedo, and reconstructions have a tendenc to have rapid variations between adjacent piels. Such rapid variations are not tpicall present in the true object scene but are artifacts of estiating an answer fro nois data.[44,45] Historicall, the ost coon regularizing technique to reduce these artifacts when using an iterative estiator was to abort the iterations before this occurred. Of course, choosing that stopping point was quite difficult and certainl data dependent. Another technique utilizes a priori nowledge of the object pdf or penalt ters in the lielihood function as discussed above. Including a ter that is sensitive to rapid variations e.g., gradient or localized variance will guide the search awa fro the ottled solution..3. Soothness Constraint A different technique, first developed in the area of toograph, uses the concept of sieves [46] or a sall soothing ernel [11,47]. As this is etreel coon, we also eplo this technique to reduce the tendenc for the aiization procedure to overfitting the noise. 5

37 a b Figure 5: Eaple object estiates that are a overfit and b regularized Fro the a priori inforation viewpoint, a sieve enforces soe aount of correlation between piels within a given neighborhood. Abrupt transitions are reduced under the assuption that either ost real objects do not have such sharp changes or that in ost instances ou re willing to sacrifice soe of that sharpness to avoid the overfitting proble. The basic technique is to perfor a weighted average or soothing of the object paraeters, which is often ipleented as a convolution of the paraeters and a blur ernel: o = p * b 50 Tpicall, a -diensional, radiall setric, Gaussian blur ernel is used as the ernel, and the full width at half a FWHM is one easure of the aount of blurring that will be induced. The width selected is based on the average signal-to-noise ratio SNR of the piel values, with larger widths being used for iager with lower SNR values. 6

38 .3..1 Low-pass Filter Viewpoint A second wa to loo at a soothness constraint is as a low-pass filter operation. B taing the Fourier transfor of 50, we see the object spectru eperiences a filtering of its spatial frequencies. A Gaussian spatial blur ernel, for eaple, will greatl attenuate the higher spatial frequencies and tend to pass the lower frequencies. O u = P u B u 51 In fact, this is an ecellent ethod b which to select the width of the blur ernel. In Figure 6, we have depicted the spectral aplitude of a hpothetical, one-diensional object signal blue along with that of an eaple noise process realization green. F cut Y cut Figure 6: SNR-based sieve/filter design in the spatial frequenc doain The point at which the strength of the signal equals that of the noise SNR = 1 aes for a natural cutoff frequenc beond which restoration of the signal is tpicall not attepted. A Gaussian curve can be constructed red such that it has soe noinal value at that cutoff frequenc. In this eaple, 0.1 was piced for the filter value, providing for at least 10 db of attenuation in the noise-doinated areas of the spatial frequenc doain. This selected cutoff frequenc has a direct relation to the FWHM of the spatial ernel, and the filtering operation can 7

39 be perfored in either the spatial doain b convolution or the frequenc doain b ultiplication and then inverse transforing..3.. Basis Function Viewpoint We have identified another wa to view this soothness constraint. In this alternate viewpoint, the object is regarded as a superposition of sooth functions. This can be seen b starting fro the definition of the soothed object: = p b o = p * b 5 The arguent to the blur function erel serves to reflect and shift it across the D arra p. We can treat these reflected and shifted versions of b as a collection of functions indeed b a shift inde subscript: Φ b 53 Rewriting in this anner, it becoes obvious that the object is a weighted superposition of the functions in the set { Φ }, and the weights are given b the paraeters we see to estiate: = p Φ o 54 Fro here, it is a short leap to conceive of a different set of functions with other desirable properties e.g., sharp edges, flat sooth regions, rotational setr, etc. that would encopass ore sophisticated regularization. Depending on the functions in the set, fewer paraeters a be needed to adequatel describe the object as well. One distinct advantage of the set of shifted copies, however, is that the speed of coputation is potentiall ver fast due to ipleentation as a convolution either spatiall or via the Fourier doain..3.3 Positivit through the Square We now coe to the heart of this research and eplore a positivit constraint through the use of the square of the estiated paraeters. This is a hard constraint, as it is ipossible to obtain a negative value for real p: o sq = p 55, 0 8

40 Due to the ubiquitous desire to regularize solutions with a soothness constraint, it does little good to consider this particular paraeterization. Instead, we consider two possibilities that differ in the order that positivit and the soothness constraint are applied: o sq = p * b 56, 1 o sq = [ p * b ] 57, Be aware, however, that the Square 1 forulation can technicall perit negatives if b has negative values. Clearl when b is a Gaussian ernel this will not occur, but other choices of b could easil introduce negative values. The above epressions are used to for g and copute the lielihood. As described in 5, the gradient residual function requires the partial derivative of these epressions with respect to the estiated object paraeters. For, we have o sq, 1 o sq,1 = p z b z = p b 58 p p and substitution into 5 gives the lielihood gradient as L psq 1 For, we have o sq,, z = p r s b 59 osq, = p z b z = [ p * b ] b p p 60 z and substitution into 5 gives the lielihood gradient as L psq, = r s [ p * b ] b 61 9

41 .3.4 Positivit through the Eponential The second ethod of ipleenting positivit is through the use of an eponential function, achieving a hard constraint for real p. As before, we eaine this constraint in conjunction with a soothness constraint due to widespread use and caution that Ep 1 a include negative values in the case where b contains negative values. o = ep p * b 6 ep, 1 o = ep[ p * b ] 63 ep, The above epressions are used to for g and copute the lielihood. As described in 5, the gradient residual function requires the partial derivative of these epressions with respect to the estiated object paraeters. For o, we have ep, 1 o ep,1 = ep p z b z = ep p b 64 p p z and substitution into 5 gives the lielihood gradient as L = ep p p ep, 1 r s b 65 For o, we use an interediate variable wz: ep, so that the object can be written as: = p b z w z = p z * b z 66 o = epw 67 ep, and the derivative of the object with respect to this variable is: o ep, = epw I z 68 w z With the chain rule, we find oep, oep, w z = = epw I z b z = epw b 69 p w z p z and substitution into 5 gives the lielihood gradient as z L = p ep, r s o b 70 30

42 .3.5 Positivit through the Absolute Value The third ethod of ipleenting positivit is through the use of the absolute value function, achieving a hard constraint. As before, we eaine this constraint in conjunction with a soothness constraint due to widespread use and caution that Abs 1 a include negative values in the case where b contains negative values. o = p * b 71 ep, 1 o = p * b 7 ep, The above epressions are used to for g and copute the lielihood. As described in 5, the gradient residual function requires the partial derivative of these epressions with respect to the estiated object paraeters. First, we recall the odified derivative of the absolute value function: d du d f u = sgn f u f u du where the signu or sign function is defined as: , > 0 sgn = 0, = , < 0 In truth, the derivative of fu is undefined where fu = 0. However, we realize that if fu = 0 + a nuber slightl greater than 0, the optiization process will see to reduce fu, and if fu = 0 - a nuber slightl less than 0 the gradient search will see to increase fu. Therefore, in such a case, the locall optial value ust be fu = 0, so we sipl define the derivative for the purposes of the gradient search to be 0. The signu function is a convenient ethod to achieve this in practice. For, we have o abs, 1 o abs,1 = p z b z = sgn p b 75 p p and substitution into 5 gives the lielihood gradient as L pabs 1, z = sgn p r s b 76

43 For, we use an interediate variable wz: o abs, so that the object can be written as: = p b z w z = p z * b z 77 o abs = w 78, and the derivative of the object with respect to this variable is: o abs, = sgn w I z 79 w z With the chain rule, we find oabs, oabs, w z = = sgn w I z b z = sgn w b 80 p w z p z and substitution into 5 gives the lielihood gradient as L p abs, z = r s sgn[ p * b ] b Positivit through Bounds Constraints The final ethod of ipleenting positivit is through the use of an optiizer with bounds constraints. We have alread touched upon this paraeterization in conjunction with the soothness constraint see 50, which is repeated here: o bc = p * b 8 In this case, we cannot change the application order of soothness and positivit, since the positivit is handled b the optiizer directl and is applied before soothness. Care ust be taen that b is positive. The above epression is used to for g and copute the lielihood. As described in 5, the gradient residual function requires the partial derivative of this epression with respect to the estiated object paraeters. This derivative is then: obc = p z b z = b 83 p p z 3

44 33 and substitution into 5 gives the lielihood gradient as = p L bc b s r Gradient Suar Object Paraeters Table 3 lists the various paraeterizations along with their lielihood gradients: Table 3: Lielihood gradient equation suar Description Paraeterization eq Lielihood Gradient wrt. Object eq Square 1 * b p 56 b s r p 59 Square [ ] * b p 57 [ ] * b b p s r 61 Ep 1 * ep b p 6 ep b s r p 65 Ep [ ] * ep b p 63 b o s r 70 Abs 1 * b p 71 sgn b s r p 76 Abs * b p 7 [ ] * sgn b b p s r 81 Bounds * b p 8 b s r 84.4 Point Spread Function Paraeterization We now leave discussion of the object and turn to the point spread functions. In blind deconvolution, the PSF paraeters are estiated along with the object paraeters. Positivit and unit-energ are two constraints that are alost alwas eploed. In addition, soe correlation between piels is also desirable in light of the phsics that are present. The aperture as will

45 anifest itself as a convolution in the spatial doain. Just as with the object, we have an epression for the lielihood gradient with respect to the PSF paraeters which involve a residual ter fro the noise odel and the partial derivative of the PSF intensities with respect to the PSF paraeters. In this section we discuss one paraeterization of the PSF that is often used and which achieves the above entioned constraints of positivit, unit-energ, and piel correlation..4.1 Phase Errors at the Aperture, Zernie Epansion This paraeterization ethod directl leverages the fact that there are phase errors induced b the atosphere in the near-field of the iaging sste. Additionall, the optical paraeters of the aperture such as cutoff frequenc diffraction liit and central obscuration are naturall odeled. To further liit the degrees of freedo, the phase distortions are odeled paraeterized as a superposition of classical optical aberrations. For eaple, tip, tilt, defocus, astigatis, and so forth, with the set of PSF paraeters {q n} corresponding to the weighting of the n th aberration in the th frae. Zernie polnoials are used to describe the various phase aberrations. The point spread function for an incoherent iaging sste is given b the agnitudesquared of the inverse Fourier transfor I 1 [ ] of the pupil function. Representing the pupil function for the th frae b H u, its inverse transfor b h, and a superscript * to denote the cople conjugate, we have the following relations for coputing the PSFs fro the pupil functions: s = * = h h h 85 = I 1 1 h [ H u ] = H uep iω 0u 86 N u 34

46 35 As previousl entioned, we represent the phase of the cople pupil function as a superposition of Zernie polnoial functions, the set of which we denote as } { u n Φ with weights {q n}. The agnitude of the pupil function u H is assued nown, and tpicall this is just a as with values of 1 to represent the pupil and 0 to represent the central obscuration and edges of the aperture. The pupil function is then: Φ = n n u n q i u H u H ep 87 Along with the object, the above epressions are used to for g and copute the lielihood. As described in 30, the gradient residual function requires the partial derivative of s epressions with respect to the estiated PSF paraeters. Those PSF paraeters are buried soewhat deep, but the chain rule will see us through. Using c.c. to represent the cople conjugate of the previous ters,.. * * c c j q h h h h j q j q s + = = 88 We now focus on the derivative of h via the chain rule: = r u r r j q u H u H h j q h 89 The first ter is: ep 1 ep r I u i N v i v H N u H u H h v r r = = ω ω 90 The second ter is: ep r I u i u H u n q i u H j q j q u H j r n n r r r Φ = Φ = 91 Cobining, the derivative of h is: ep ep I u i u u H N i r I u i u H r I u i N j q h j u r u j r Φ = Φ = ω ω 9

47 Substitution into 88 gives us the PSF derivative we see: * s ih = H u uep i 0u I c. c. Φ j ω + 93 q j N u One last bit of siplification is in order before further substitution. Consider a general cople nuber can be represented as a+ib. We note that our derivative epression is of the for i a + ib + c. c = ia b + ia b = b, bringing our final PSF derivative to: * s h = I H u Φ uep i 0u I j ω 94 q j N u This is then substituted into 30 to give the lielihood gradient as: L = q j * h o I N u Φ j uep iω 0u I 95 r H u We note that Φ u is a real factor and can be factored out, while the correlation is also j real and can be inserted into the I{} operator. Recognizing the suation over is an inverse Fourier transfor, our siplified lielihood gradient is finall: 1 * { H u I { h [ r o ] } L = Φ j u I 96 q j.5 Lielihood Maiization u With the epression for the lielihood function, the desire is to find the object and PSF paraeters ielding the aiu value of that function. In order to identif the aiu lielihood, we turn to nuerical optiization techniques. There is no best algorith for searching this lielihood space, unfortunatel. The fact we are able to analticall define the lielihood gradients, however, greatl iprove the pace at which we can proceed and allows the use of the ore powerful techniques. Fro the standpoint of the solution, it should not atter what algorith is used. After all, the location of the pea doesn t depend on how we traverse the hper surface assuing a global pea, of course. However, since we care about finding the pea or soething reasonabl close to it in a tiel fashion, we choose to use a relativel standard approach for 36

48 this tpe of proble. B and large, the two ost coonl used approaches are conjugate gradient and liited eor quasi-newton ethods. In our case, we require an optiizer that ipleents bounds constraints and leverage the wor b acadeia with the use of the L-BFGS-B pacage fro Northwestern Universit.[48] Fro their description: L-BFGS-B is a liited eor algorith for solving large nonlinear optiization probles subject to siple bounds on the variables. It is intended for probles in which inforation on the Hessian atri is difficult to obtain, or for large dense probles. L-BFGS-B can also be used for unconstrained probles, and in this case perfors siilarl to its predecessor, algorith L-BFGS Harwell routine VA15. Additionall, while quasi-newton ethods require storage of additional values, the also tend to have faster convergence rates than the conjugate gradient ethods.[49] We set the nuber of BFGS corrections that are saved to 10 in the iddle of the recoend range but did not pursue evaluation at other settings given the success of reconstructions at that setting. One additional consideration with nuerical optiization is that of scaling. A proble is poorl scaled if the changes in the function value f are ver uch larger for changes in in a certain direction copared to another direction. For eaple, in a two diensional function, a unit change along each diension causes function values of 10 and This arises in our case because we are searching for object paraeters as well as the PSF paraeters. Tpical object paraeter values could range upwards to several thousand, while the coefficients for a Zernie polnoial epansion are tpicall between zero and one. Discussions with other researchers indicate that the consensus is to noralize the data before searching and then scale the resultant object estiate. The coon scale factor is the aiu piel value contained within the enseble of data. Consequentl, the object paraeter values are significantl better atched to those of the PSF paraeters. Convenientl, quasi-newton ethods are also quite insensitive to an residual scaling issues.[49] One additional technique that a be useful for ore scale-sensitive search algoriths is to alternate the tpe of paraeters being searched. For eaple, soe nuber of iterations where the object paraeters are held constant and PSF paraeters are adjusted followed b soe 37

49 nuber of iterations where the PSF paraeters are held constant and object paraeters are adjusted, then repeating the ccle. In practice, the L-BFGS-B code we used requires the coputation of the lielihood function and gradients once per iteration. In other words, the current object and PSF paraeters are rendered into object and PSF estiates and convolved to generate the various noiseless iage estiates. The specific choice of noise distribution Poisson, Gaussian, or Spatiall Varing Gaussian dictates the lielihood and gradient equations, the evaluations of which are provided to the optiizer. In this wor, we selected the Gaussian noise odel for all reconstructions. The iterative approach andates an initial estiate for the object and PSF paraeters. In this wor, we set the initial object estiate to the average of the iages in the enseble after their centroids had been aligned. We then used the inverse of the paraeterization epressions in order to initialize the actual paraeters. Since the inverse of the soothness constraint is illdefined, this part of the inverse logic was oitted. The initial PSF paraeters Zernie polnoial coefficients were set at zero with the eception of the tip and tilt ters i.e., the vertical and horizontal shift for each frae. These paraeters were initialized based on the centroid of each frae..6 Siulation of Data.6.1 Atospheric turbulence Much of the accurac of the siulated iager is a result of several decades of wor b an researchers. We priaril leverage the wor b Noll [50] who described a odal epansion of the phase perturbations due to light propagation through an inhoogeneous atosphere. Zernie polnoials for an orthonoral basis for functions defined on the unit circle, which is especiall apropos given that ost optical sstes have circular apertures. Using Zernie polnoials, we for a series epansion to describe the phase of the aberrated wavefront across an aperture of radius R as: N Rρ, θ = ρ, θ ψ a i Z i 97 i= 1 38

50 Because of the underling Gaussian distributed nature of ψrρ,θ at ever point in space, the rando variables a i are also Gaussian distributed with soe covariance atri Γ a. The eleents of the covariance atri of these coefficients have been calculated for inde of refraction fluctuations characterized b the Kologorov power spectral densit and are given b: [50] 5 / 3 D ni + n j i / 1/ 8/ [ ni + 1 n j + 1 ] π δ a a 0 i j = r Γ14/3 Γ[ ni + n j 5/3 / ] Γ[ n n + 17 /3 / ] Γ[ n n + 17 /3 / ] Γ[ n + n i j j i i i j j + 3/3 / ], for even i-j 98 a = 0, for odd i-j 99 i a j In order to generate a vector of rando coefficients which have this covariance, we perfor a Choles factorization of the covariance atri Γ a into the product of two square atrices: RR T Γ a = 100 We for a vector b of rando draws of a zero ean, unit variance, Gaussian rando variable, which ields the covariance of b as the identit atri: { bb T } = I Now use b to generate a new vector a as: E 101 a = Rb 10 We confir the covariance of this vector a atches the desired covariance: E T T T T T { aa } = E{ Rb Rb } = E{ Rbb R } = E{ RR } = Γa 103 Although the covariance epression is rather coplicated, it is a fairl straightforward process to obtain coefficients that will generate statisticall accurate phase variations in the covariance sense. The resultant phase screen fors the basis of a point spread function as outlined in the first part of section.4.1. This process is repeated for ever frae of iager to be siulated, ielding a set of independent atospheric realizations corresponding to Kologorov turbulence. 39

51 .6. Sapling Care ust be taen to accuratel siulate the iaging process, which is coplicated b the fact the siulation is conducted in a discrete environent. There are three planes of interest with their associated piilated arras: 1 object, pupil, and 3 iage planes. There are two choices for the basis of the sapling: 1 detector piel spacing and object piel spacing. Associated with these are the angular saple spacings at the detector and object. For a linear saple spacing piel pitch d on the detector and effective sste focal length f, the angular saple spacing at the detector is: d θ = d 104 f The real-world object has an etent W, and it s a siple atter to define the piel size i.e., the saple spacing in the object plane for an arra size of N o piels as: W o = 105 N As seen fro the iaging sste pupil at a range of R, the object piels have an angular etent of It is ver liel that d o = o W θ o = 106 R RN o o θ θ, which andates a choice of which saple spacing to use in the iage foration process. If the detector piel spacing is used as the basis, then the object ust be resapled prior to use in order to adjust the angular sapling at the object. If the detector piel spacing is used as the basis, then the iage ust be resapled as a final step to provide the sapling dictated b the detector. The resapling in each case will entail either 1 averaging across several saller piels and dealing with partial piel effects or interpolation, depending on choice of reference as shown in Table 4. Additionall, the choice of basis for the angular saple spacing detector or iage has raifications for the sapling of the pupil function. If the detector angular spacing is used, the pupil diaeter in piels within the arra is convenientl constant. If the object angular spacing is used, the pupil diaeter in piels will be a function of range to the object. The constant pupil 40

52 size is attractive for perforance optiization as well as siplicit, but we recoend choosing the saller saple spacing as the reference. Table 4: Sapling reference raifications Sapling Reference Detector/Iage Object Ref Spacing < Other spacing Interpolate object 1 st Average iage last Ref Spacing > Other Spacing Average object 1 st Interpolate iage last Pupil Diaeter piels Constant Changes with range Lastl, we note that the optical sste with aperture diaeter D effectivel ields a lowpass filtering operation with a hard cutoff at a spatial frequenc of D/λ [rad -1 ], ipling a iniu full-width at half-a FWHM resolution of λ/d [rad]. Nquist sapling considerations would have the siulation angular saple spacing not to eceed half that or: λ θ Nquist = 107 D At this point, we proceed with the assuption that the siulation will use the angular saple spacing of the object, and will later revisit the case where the detector spacing is chosen as the reference. Neglecting brightness effects and assuing we are in the far-field i.e., the Fraunhofer region, Fourier optics tells us that the field aplitude at the pupil is a scaled in lateral etent version of the Fourier transfor of the aplitude at the object plane.[51] Siilarl, the field aplitude at the iage plane is a scaled version of the Fourier transfor of the aplitude at the pupil. The sapling at each of the planes ust tae this scaling into account. The following developent uses one-diensional notation for siplicit. We also regard a representative, single-lens iaging sste in which pupil and aperture are interchangeable. It is iportant throughout this that the reader separate the notion of iage/pupil fro spatial/fourier doain. While this is tpical in the coputer vision fields, it will not serve us here. In point of fact, the aperture in a real iaging sste e.g., irror or lens has a spatial etent which ust be considered. Further, it also has inforation that after a Fourier transfor 41

53 ust be viewed as spatial inforation, which is suggestive of being in the Fourier doain to start. In the following, i is a coordinate in the iage plane, p is a coordinate in the pupil plane, o is a coordinate in the object plane, λ is the wavelength and f is the effective focal length distance fro pupil to detector. Denoting the Fourier transfor of the pupil aplitude b Pu and neglecting brightness scaling factors, we start with the intensit at the iage plane fro Fourier optics theor as: I = P u i u = i λf = i P λf 108 B inspection, we have an epression relating the Fourier doain saple spacing in the pupil to the spatial doain saple spacing in the iage as: i u = 109 λf The aperture is assued to be circular with phsical etent D and is represented in the pupil arra b a circle with a diaeter of N p piels. Thus, the spatial doain saple spacing in the pupil plane is: D p = 110 N We assue the pupil arra is of size N A, and therefore the spatial etent of the entire pupil arra is: W p A p = N 111 We also now that the Fourier doain saple spacing in the pupil is equal to the inverse of this spatial etent: W p p 1 u = 11 Equating 109 and 11, we have the following relation: i 1 = λf N A p 113 4

54 Rearranging and cobining with the spatial sapling of the pupil 110, we see the angular saple spacing of the iage is: λ λ N p 114 D N i θ i = = = f N A p For the purpose of the siulation, we equate the angular saple spacings of the object and iage and also assue equal sized arras for convenience N o = N A. Recall that if the actual saple spacings are different, one is adjusted either prior or after iage foration. λ N D N p A = W RN We can rearrange this epression to find the size of the aperture in piels that is required to accuratel siulate this scenario: N p A A 115 D W = 116 λ R In other words, the nuber of saples across the aperture is the product of the optical cutoff frequenc and the angular etent of the scene. For eaple, to accuratel siulate the iage of a 6 scene at a range of 500 as seen b the 1.6 telescope on Haleaala in the visible wavelength λ=0.5 µ, one would need a circle of diaeter 1.6*6/0.5e-6/500e3 ~ 38 piels to represent the pupil. Looing bac at the original iage epression in 108, we can view the transfor of the iage the optical transfer function when the source is a point source as proportional to the autocorrelation of the pupil aplitude. Recall that a correlation can be coputed via the Fourier doain as: I f g F u G * u When the functions f and g are equal i.e., an autocorrelation, the Fourier side of the 117 relation reduces to F u, which atches the for of the iaging equation. To avoid wraparound effects fro that autocorrelation corrupting the optical transfer function, the arra size N A should be at least twice the pupil etent N p. N A N p

55 An alternate view of this is that the Nquist criterion is being preserved as substitution into 114 gives λ N p λ N p λ θi = θi 119 D N D N D A Given the above sapling relations are aintained, an iage can be calculated with 1 with possible averaging if the detector angular saple spacing differs fro the reference. If, on the other hand, the angular saple spacing of the detector and therefore the final, desired siulated iage is used as the reference, then the object ust be resapled first. This effectivel changes the value of the object etent W in the above developent with the result that the adjusted object spacing will atch that of the detector. This is, in fact, eactl what happens in a real-world iaging sste with a fied field of view, in which the lateral etent of the scene within the field of view naturall changes as a function of range. Using the detector saple spacing as the reference and thus We assue narrow-band, spatiall-invariant, incoherent iaging. As a result, the noiseless iage is the convolution of the object and sste point spread function as denoted in 1. For the arra sizes tpicall involved, this convolution is actuall calculated via ultiplication in the Fourier doain. Due to the discrete nature, this technicall achieves a socalled circular convolution. [5] However, we set the support of the pupil function at half that of the arra size to avoid wraparound effects i.e., ielding a noral convolution. 44 p θ = θ, 114 is anipulated to find the epression for the nuber of piels across the pupil. Siilar to the previous epression, the nuber of piels across the pupil is seen to be the product of the optical cutoff frequenc and the angular etent of the iage. N p D = θd N A 10 λ In the wor here, we sidestep the ajorit of the sapling issues copletel and sipl set N p = N A /, which ields critical sapled iager and assue the detector angular saple spacing atches that of the object. Resapling of the object or iage is not necessar. The specific values of scene etent, range, etc. is not unique, of course, and so specific values are iaterial to our purposes..6.3 Noiseless Iage Foration i d

56 The notion of narrow-band iaging is not a well-defined notion. How narrow ust the band be? Solar iaging, for instance, will coonl use etreel narrow filters, even less than a single angstro wide. Soe applications, are content with a hundred nanoeters. Still others desire to use ost of the visible spectru with color iager. As we have seen fro the sapling developent, the wavelength of illuination plas a role in the size of the intensit pattern being sapled. Another overwheling aspect is that the object intensities are rarel the sae in different bands. Fro the reconstruction side, MFBD assues a single, coon object, which certainl does not atch realit for wide band iager. One of the siple approaches for color iager, is to separatel reconstruct the red, green, and blue channels. Unfortunatel, it is ver difficult to register these separate estiates and colorfringing will result. Results are often less than satisfactor because there are an ore unnowns than for narrowband data a factor of three for RGB channels. A ore sophisticated approach will leverage the fact that the atospheric aberrations and thus the point spread functions scale with wavelength and are not, in fact, independent between the channels. This is nown in the industr as wavelength diversit. While these ultiple loos at the sae atospheric distortion iprove things considerabl for the atospheric paraeter estiates, ultiple wavelength-dependent objects usuall have to be estiated. As far as siulation is concerned, siulating wideer band iager is a atter of cobining the results of several possibl an narrow-band siulations. As long as the wavelength is considered in the sapling, it coes down to the tradeoff between accurac and run-tie i.e., nuber of bands to siulate and then cobine. That said, however, one is often able to obtain significant iproveent even though the narrowband assuption is violated. The individual circustance will dictate if the errors inherent in such an approach are tolerable. In the wor here, we use a single wavelength in all siulations and reconstructions..6.4 Noise Once a noiseless iage g has been siulated, noise is added to each piel. The rando arrival tie of photons and thus the brightness during a fied eposure interval is assued to have a Poisson distribution with a ean rate equal to that of the piel values in the noiseless iage. Recall the variance of a Poisson rando variable is equal to this ean rate. Additionall, noise with a Gaussian distribution with a constant variance is applied to siulate 45

57 the effects of detector read noise. In this wor, we assue a read noise standard deviation of 10 counts and desire a pea signal to noise ratio. Thus, the signal is the pea value P of the noiseless iage. The total noise variance is the su of the constituent noise variances: P for the photon noise and σ d for the detector read noise. It is coon to specif the desired signal to noise ratio in units of db SNRdb, so we have: B the definition of the signal to noise ratio: This can be solved to give an epression for the brightness P: SNRdb /10 SNR = P SNR = 1 P + σ 46 d SNR + SNR SNR + 4σ d 1 P = 13 Once the pea value P is coputed, we scale the entire noiseless iage g appropriatel. The noise in each piel is assued independent and rando draws are ade against a Poisson rando variable with ean g and a zero-ean Gaussian rando variable with variance σ d for a cobined noise ter of: and the final nois iage is given b:.7 Accurac Metrics.7.1 RMS Error Spatial Doain n = Poisson g + Gaussian0, σ 14 d = g n 15 + The first etric we consider is the root ean square RMS error in the spatial doain. Since these data were siulated, we have the luur of nowing the true answer a perfect reconstruction process would have generated. One issue is that the rando shift that is present in each of the actual reconstructions coplicates the accurate coputation of this etric. While the reconstructions can be correlated against the true iage for alignent, this approach erel leverages an integer-piel estiate of the shift. Since the siulated atospheric distortions were d

58 not liited to integer tip & tilt aberrations i.e., shifts, there is no reason to epect that the sapled representation of the reconstructed object necessaril be aligned with the original saples of the true object. Here we switch notation slightl, and refer to the reconstructed object as o ˆ with a hat and the true object as o. Figure 7: Eaple error of a single reconstruction shows a slight residual vertical shift Figure 7 shows the error [ ˆ o ] o of an eaple reconstruction after alignent b spatial correlation. A slight vertical shift is evident due to the white negative values at the top edge and blac positive values at the botto edge of the solar panels. In the aggregate, however, we epect siilar errors to be present across all reconstruction ethods, inflating all of the b approiatel the sae aount. As a gross figure of erit, we can copute the total RMS error as: RMSE = 1 N N, oˆ o 16 where N is the nuber of reconstructions, and N is the nuber of piels in an iage. The result is a scalar representing the reconstruction accurac of each paraeterization. Tae note, however, that these results should not be copared across the two SNR cases since the object brightnesses are vastl different. 47

59 .7. RMS Phase Error Spatial Frequenc Doain In addition to the RMS error in the spatial doain, we also consider the RMS phase error RMSPE coputed in the spatial frequenc doain. This etric ephasizes the shape of the reconstruction rather than the particular intensit levels. Additionall, we focus on the area of the spectru that is eaningful to the true object b using an indicator as Mu which is unit where the agnitude of the true spectru is within orders of agnitude of the pea agnitude and zero elsewhere. Figure 8 shows the spectral areas of interest for the two objects used in this stud. The RMSPE etric is coputed solel in these areas where the true object contains a reasonable level of signal. a b Figure 8: Indicator as Mu of spectral locations of interest based on the true spectru of a Satellite and b Spoe objects At first blush, unfortunatel, the phase is also corrupted b the rando spatial doain shifts, as can be seen fro the Fourier shift propert: [51] f I jπ u F 0 ep 0 u 17 Thus, a shift in the spatial doain anifests itself as a two-diensional phase rap in the frequenc doain. Note the several blac/white transitions in Figure 9a where the phase difference changes between ±π. However, the signal to noise ratio of the frequenc piels around DC i.e., the origin is etreel high, and the phase rap can be estiated fro the first non- 48

60 zero frequenc piels in the vertical and horizontal directions. Of critical iportance, however, is that this phase rap characterizes an subpiel shift as well, and copensation provides insight into the phase error when the estiated and true objects are well aligned. Figure 9: Eaple phase differences when the estiated and true objects are a isaligned and b aligned Another aspect of this etric is the characterization of reconstructed resolution or level of detail, since resolution increases as accurate spatial frequenc content increases. To aintain insight into the various spatial frequencies, we copute the etric within narrow annular regions of approiatel constant spatial frequenc agnitude. Using u as a D indeing variable spatial doain coordinate, and the function Du to denote the distance of that location fro the origin, the r th annular region is defined b r < Du < r+1. We copute the etric as: argoˆ u argo u M u 1 r< D u < r+ 1 RMSPE r = 18 N M u r< D u < r+ 1 1/ 49

61 In other words, we copute the RMS phase error within each annular region subject to the ased area of interest and then average across the reconstructions. The result is a curve for each algorith relating the fidelit of reconstruction as a function of spatial frequenc. Figure 10: Eaple RMSPE curve relating fidelit of reconstruction to spatial frequenc Optical sstes have been characterized in various was over the ears, and one of the fundaental easures of sste qualit is at what spatial frequenc is the phase error greater than sa, 1/8 of a wave. A wave is sipl an error of π radians. Regarding our reconstruction process as part of the iaging sste, the RMSPE curves provide this inforation; the higher the frequenc, the better. In our case, we noralize the frequenc between 0 and 1, where 1 represents the highest frequenc possible in our sste r a the Nquist frequenc, and denote this etric b F 8 for 1/8 wave error: 1 in RMSPE r > π r F 8 8 = 19 r a 50

62 3.1 Siulation of iage data 3 RESULTS The iage siulation process [53,54] leverages the fact that the priar, shape-distorting aberrations in an iaging sste are due to phase errors in the near-field. For each iage, the siulation draws a rando realization of coefficients. These coefficients are distributed with a Kologorov spectru which has been shown to odel the statistics of the atospheric distortions etreel well. These coefficients are then used to weight a set of basis functions described b Zernie polnoials [55]. The accuulation of weighted polnoials describes the phase distortions present at the aperture of the iaging sste. The turbulence strength described b the relation of Fried s seeing paraeter r 0 and the aperture diaeter D plas a critical role here as well. For all iager, I used a D/r 0 ratio of 16 to reflect a coon case eperienced b the observator on Halealaa 1.6 telescope and 10 c seeing. This is on the relativel poor side as will be seen in the iager. During the process of siulating the raw iager, I was reinded of soething Gonsalves said during a conference presentation: Siulations are often dooed to success. While this stud is priaril concerned with the perforance iproveents brought about b various paraeterizations of positivit, we used well established theor in the siulation of data to iniize the chances of a false-success. In the final analsis, though, I did not want to recoend a paraeterization if it would onl wor in a sunn-da scenario with a high signal to noise ratio i.e., a bright object. As a result, we considered signal to noise ratios of 0 db and 40 db. Gaussian read noise with a standard deviation of 10 counts was used for all iager. In addition, signal-dependent, Poisson noise was applied to odel the rando arrival ties of the photons during the effective integration tie. Object brightness was set to ield a cobined pea SNR of the atosphericall distorted signal of the aforeentioned 40 db or 0 db. 51

63 An incoherent iaging sste is assued in the generation of the various point spread functions PSFs and resultant iager [56]. Data were siulated for the two different objects shown in Figure 3. Each data set consisted of 1,600 independent realizations of the atospheric turbulence. Figure 11 and Figure 1 show tpical eaples of the siulated raw iager at various signal to noise ratios: a Figure 11: Tpical siulated iager, Satellite object with SNR=40 db b a b Figure 1: Tpical siulated iager, Satellite object with SNR=0 db 5

64 a Figure 13: Tpical siulated iager, Spoe object with SNR=40 db b a b Figure 14: Tpical siulated iager, Spoe object with SNR=0 db 3. Ipleentation of constraints Each of the equations in Table 3 was ipleented in the reconstruction code. A great deal of effort was spent in the theoretical developent to identif convolutions and correlations in the siplified epressions. For the arra sizes involved, it is significantl faster to copute 53

65 these with the use of the fast Fourier transfor. During ipleentation, further attention was spent in iniizing the nuber of calls to the transfor routines, resulting in the etensive use of interediate cache-lie arras. The transfors necessar for the PSFs s or the residuals r are iaterial to the purpose of this research and are ecluded fro the count. Table 5 shows the nuber of object-related transfors and eecution tie required in the final ipleentation. The relative tiing as a percentage change copared to that required b Square 1 and Square are denoted b the two % Ch coluns. Table 5: Paraeterization run-tie etrics Desc. Render FFTs Gradient FFTs Per Frae Render Tie µs Gradient Tie Per Frae µs Cobined Tie Per 16 Fraes s % Ch Fro Square1 % Ch Fro Square Square ± ± % 4 Square ± ± % 0% 6 Ep ± ± % -30% 3 Ep ± 9 11 ± % 3% 7 Abs ± ± % -46% Abs ± ± % -% 5 Bounds ± ± % -49% 1 Ran The tiing data were estiated with repeated evaluations in conjunction with the coputer s real-tie cloc. This is necessaril an asnchronous approach, of course, leading to slight differences in consecutive tiing trials. The uncertaint quoted above is the 95% confidence interval about the estiate of the ean. Furtherore, the nuber of trials was adjusted such that that confidence interval was no ore than 10 µs. 3.3 Iage Reconstruction With seven object paraeterizations, two true objects, and two SNR settings, there are twent-eight cases to consider. In all cases, a non-overlapping enseble of 16 fraes were used. In other words, fraes 1-16 were used in the first reconstruction, fraes 17-3 were used in the second, etc. With a data set of 1,600 fraes, 100 reconstructions were generated for each case, and 150 iterations were used in the optiization. A soothness constraint of 0.5 piels and.0 54

66 piels was used for the 40 db and 0 db SNR cases respectivel. The initial object estiates were based on the centroid-aligned average of the raw fraes in each enseble. The PSFs were paraeterized b phase-onl descriptions of the distortions at the aperture. These were epanded in a series epansion of 91 Zernie polnoial ters for each frae. The tip and tilt paraeters were initialized based on the difference of the centroids of the initial object estiate and each raw frae. The reainder of the paraeters were initialized to 0. The resultant reconstructions were not required to be centered in the arra. Indeed, the tip/tilt horizontal and vertical shifts required is part of the PSF estiation, and the eact position of the object centroid is relativel arbitrar. The relative isalignent between reconstructions, however, does pose a proble when coparing reconstructions. To itigate this issue, all reconstructions in a set were aligned in the spatial doain b cross correlation using the true object as a reference. The figures in the following sections show the average and the first individual reconstruction for each case. Visual assessent of qualit in the spatial doain is subjective at best, and we are liited to broad coparisons with regard to perceived resolution and accurac. Following these figures, however, we present RMS phase error plots, fro which a quantitative resolution assessent is ore easil ade. 55

67 3.3.1 Satellite Reconstructions, 40 db SNR First we show the Satellite reconstructions at 40 db SNR. a b Figure 15: Square 1 reconstructions, Satellite object, 40 db SNR a set average, b individual reconstruction a b Figure 16: Square reconstructions, Satellite object, 40 db SNR a set average, b individual reconstruction 56

68 a b Figure 17: Ep 1 reconstructions, Satellite object, 40 db SNR a set average, b individual reconstruction a b Figure 18: Ep reconstructions, Satellite object, 40 db SNR a set average, b individual reconstruction 57

69 a b Figure 19: Abs 1 reconstructions, Satellite object, 40 db SNR a set average, b individual reconstruction a b Figure 0: Abs reconstructions, Satellite object, 40 db SNR a set average, b individual reconstruction 58

70 a b Figure 1: Bounds reconstructions, Satellite object, 40 db SNR a set average, b individual reconstruction 59

71 3.3. Satellite Reconstructions, 0 db SNR Net we show the Satellite reconstructions at the reduced SNR of 0 db. We note additional blurring copared to the 40 db reconstructions. a b Figure : Square 1 reconstructions, Satellite object, 0 db SNR a set average, b individual reconstruction a b Figure 3: Square reconstructions, Satellite object, 0 db SNR a set average, b individual reconstruction 60