ransation Microscopy (RAM) for super-resoution imaging. Zhen Qiu* 1,,3, Rhodri S Wison* 1,3, Yuewei Liu 1,3, Aison Dun 1,3, Rebecca S Saeeb 1,3, Dongsheng Liu 4, Coin Ricman 1,3, Rory R Duncan 1,3,5, Weiping Lu 1,3,5 *Authors contributed equay to the wor. 1 Institute of Bioogica Chemistry, Biophysics and Bioengineering, Heriot-Watt University, Edinburgh, EH14 4AS, Present address: Centre for Neuroregeneration Chanceor's Buiding 49 Litte France Crescent Edinburgh EH16 4SB, 3 Edinburgh Super-Resoution Imaging Consortium, www.esric.org. 4 Department of Chemistry, singhua University, Beijing, China 5 o whom correspondence shoud be addressed; emai: W.Lu@hw.ac.u, eephone: +44(0)131 451 3065 or R.R.Duncan@hw.ac.u, teephone: +44 (0)131 451 3414
Suppementary Notes 1. Method of RAM he fu fow chart for the method of RAM can be found in Suppementary Figure 5. Here we discuss in more detai each of the steps in the RAM restoration process. 1.1 Energy function Foowing the genera form of Eq. () in Methods, we define the energy function as Nc Oc ( h) 1 I M E ( I ) J - P CI wh ( ) f L, (1) 1 ct1 c t h1 which comprises the first term measuring the difference between the ow-resoution observations and predicted high-resoution counterpart and the second term characterizing image structures to reguarize the minimization process. In the second term, comprising intensities of a pies, i.e., ( he function h ) L represents a patch of W+1 pies centred around and is a vector L [ ( ), ( 1), ( ), ( )] I W I w I I W. () f L is the h th principa component (PC) score, which is the projection of the patch L onto a ow-dimensiona space using principe component anaysis (PCA). PCA has been widey used in computer vision automatic feature etraction from compe data sets 1. f L characterizes nonoca features of images through anaysis of patches and can ( Here, h ) be considered as a generaization of the first- and second-order nonoca differences (NLDs) in (1) our previous study. For eampe, equivaent to the 1 st () -order NLD, whie NLD. f L for patches containing an edge feature is f L for a ridge/bob feature represents the nd -order ( As discussed in Methods, noninear features characterized by h ) f L, particuary those underying fine structures are prone to noise contamination. It is therefore advantageous to group and characterize patches of simiar structures in order to enhance the robustness of feature etraction. We appy a traditiona K-means custering agorithm 3 to the image patches, 1,... L, to partition a pies N N conditions 3, into c N separate groups tt 1,, Nc c, under the
( Given the custering resut in Eq.(3), PC score h ) each patch group by using PCA as ct Nu Nc t1ct N. (3) cp cq Nu, p q f L is estimated for pie beonging to X [ L,, L, ] U V, c 1 V v, v, v,..., v 0 0 1 h W1 h β f ( ),, f ( ), L L VL t, (4) where V is a matri whose coumns are made by eigenvectors vh of the matri X X, β is a coumn vector whose h th eement is h f ( L ). he summation in the second term of Eq. (1) is therefore appied first to patches in a group of simiar features and then to a different groups. he number of PC scores used for feature modeing, O c t, in Eq. (1) is critica in superresoution restoration, because a sma number may miss ey structures, whie a arge number can resut in over-fitting, namey turning noise into features. o overcome the probem, ess ( informative h ) f L, which are often sma in vaues, shoud be discarded. We appy a simpe hard shrinage method 4, f h h f ( L ) f ( L ) t ( L ). (5) 0 ese h c where the threshod c t is determined by the noise eve of the patches in group c t. he highest ( order of the PC scores h ) f L for a pies t average number of non-zero eements among a vectors c t c in the group is then estimated as the β, that is, O ct β ct C t 0, (6)
where the 0 -norm β of the vector β is the number of its non-zero eements and 0 represents the number of pies in the group c t. he weighing coefficient wh ( ) in Eq. (1) is C t define as, h O ct r w ( ) f ( L ) f ( L ), which enhances the effects of arge scores. h r1 1. Energy minimization We minimize the energy function of Eq. (1), I arg min E( I ), to estimate the desired de( I ) high-resoution image. his can be done by soving the gradient equation, 0. We di rewrite Eq. (1) in a matri-vector form before differentiation. We first define a matri, NN D h R, which corresponds to the h th ( PC score h ) I f L of a pies (patches), D h v W1, h 0 0 0 v 0 W, h 0 0 0 N v N W, h N (7) where v h, is the h th eigenvector defined by Eq. (4). We further define a coumn vector, N1 δ R th, whose eement is the ony nonzero eement with unit vaue so that the eement of any vector v v v v N (1), ( ), ( ) can be written as th v( ) δ v. (8) Combining Eq.(4), (7) and (8), we can rewrite the scaar variabe of the PC score as f ( h) L δ DI. (9) h he function E(I ) can then be epressed by using Eq. (9) in the foowing matri-vector form,
M N 1 1 E ( I ) δ P C I J N O c c t I ( ) wh δ DI h ct1 c t h1 which aows us to directy computer the gradient,, (10) de( I ) d I A S C P A P C I I M 1 M PCS, I PCS, 1 C P A J, (11) where the N N diagona matrices, A PCS, and S, PCS, are given as A δ D I D δ δ D N Oc O c t ct PCS, wh ( ) wh ( ) h h h ct1 h1 c t h1 S δ DI δ I δ D N Oc O c t ct ddh PCS, wh ( ) wh ( ) h h ct1 h1 c t h1 di, (1) and the N N diagona matri, A, is given as A diag Φ 1, Φ δ P C I J δ P C I J N de( I ) he minimization, i.e., 0, eads to the foowing equation, di. (13) M M I A PCS, I S PCS, C P A P C I CP A J (14) 1 1 which is a noninear equation of I because A PCS,, S PCS, and A aso invove the variabe I, so wi have mutipe soutions that correspond to oca and goba minima of the energy function E(I ). We here appy a modified iterative reweighted east squares (MIRLS) method 5, 6 that
iterativey estimates between A PCS,, S, PCS, A and I. o do so, we first fi I, so from Eq.(1), dd di h 0 and S PCS, goes to 0. We can then rewrite Eq. (14) as M 1 I, B Q A I PCS, (15) where the matrices B and Q are given respectivey as B C P A J Q C P A P C I. (16) We finay rewrite the noninear equation Eq. (15) as B Q A I 1 1 I1 PCS, B Q I A PCS, I B Q A I M M IM PCS, M / M / M / M. (17) We note that Eq.(17) has more constraint than Eq. (15) since the unnown image I now must satisfy a set of M equations simutaneousy, i.e., satisfy each of the M observations, instead the sum of the M observations. he soution of Eq. (17) wi be more iey to get coser to the ground truth than that of Eq. (15). 1.3 Agorithm Pipeines he main steps of MIRLS for soving Eq. (17) are: (a) Initiaization: Let I = J and I = σ n, where LR observation J, the burring matri P, the correspondence matri C and the noise Standard deviation σ n are nown. (b) Computer the weight matrices B, A PCS,, Q by Eq. (1) and (16) based on the current estimate I. (c) For each observation :
1 (c1) Sove the equation I ˆ argmin, B Q to obtain an intermediate soution I ˆ,. (c) Use the above I ˆ, as the initia vaue to refine the soution by soving 1 equation I ˆ, arg min I A,, / PCS I M B. (d) he soution I is obtained by a weighted average of I, 1,... M M M M I w1 ( ) I, (1), w ( ) I, (),... wn ( ) I, ( N) (18) 1 1 1 where the weight vector is given as i 1 M w '( I ( i) I ( i)),... '( I ( i) I ( i)) / C, i 1,..., N (19) and is the derivative of the robust function. C is a normaization factor. his step enforces that the mutipe soutions I, by step (c) shoud be simiar to each other. (e) Go to step (c) if Eq. (14) cannot be satisfied using the current estimation I ; otherwise update the parameter I according to the residua noise in I 7. (f) he iteration stops when I converges to a certain eve, measured by MSDN between two adjacent images, and is considered to be the restored image; otherwise go to step (b) to compute again the weight matrices with an updated I. he intermediate soution ˆ argmin 1, I B Q in Step (c1) can be soved by many approaches, such as conjugate gradient (CG), Wiener Fiter, or shrinage method. We sove it by using an iterative Wiener Fiter method with a sight modification 8 as 1 ˆ 1 pre pre I, I, P P eps ( P J P P I, ) (0)
where eps is a sma constant to mae sure the stabiity of the matri inverse, pre I, is the soution for the previous iteration. Given the intermediate soution, I ˆ,, we can then sove in step (c) the equation ˆ 1 I arg min A I / M B (1), I PCS,, which is supposed to be aso soved by Wiener fiter but its stabiity cannot be guaranteed. We therefore revise the equation by adding a reguarization term in Eq. (1) as where 1, I PCS,, PCS,, 1 1-1 I arg min A Iˆ / M B Root( A ) I () is the 1 norm and operator Root PCS, A indicates a new matri whose eements are square root of the corresponding eements of the matri A PCS,. We can then sove Eq. () using a we-nown method of east-absoute-shrinage-and-seection-operator (asso) 9 as, where 1 1 I ( ) D Root A sgn Root A D I, 1 PCS, PCS, 1, 1 Root A D I / M PCS, 1, I sgn is a sign function and is a shrinage function given as 6,, (3) a a 0 a (4) 0 ese References 1. Joiffe, I. Principa component anaysis. (Wiey Onine Library, 005).. Maira, J., Bach, F., Ponce, J., Sapiro, G. & Zisserman, A. in Proc. CVPR 009 7-79 (IEEE, 009). 3. Jain, A.K., Murty, M.N. & Fynn, P.J. Data custering: a review. Acm Computing Surveys (CSUR) 31, 64-33 (1999).
4. Ead, M. Why simpe shrinage is sti reevant for redundant representations? IEEE ransactions on Information heory 5, 5559-5569 (006). 5. Daubechies, I., Defrise, M. & De Mo, C. An iterative threshoding agorithm for inear inverse probems with a sparsity constraint. Communications on pure and appied mathematics 57, 1413-1457 (004). 6. Daubechies, I., DeVore, R., Fornasier, M. & Güntür, C.S.n. Iterativey reweighted east squares minimization for sparse recovery. Commun. Pure App. Math. 63, 1-38 (010). 7. Osher, S., Burger, M., Godfarb, D., Xu, J. & Yin, W. An iterative reguarization method for tota variation-based image restoration. Mutiscae Modeing & Simuation 4, 460-489 (005). 8. Hiery, A.D. & Chin, R.. Iterative Wiener fiters for image restoration. Signa Processing, IEEE ransactions on 39, 189-1899 (1991). 9. ibshirani, R. Regression shrinage and seection via the asso. Journa of the Roya Statistica Society. Series B (Methodoogica), 67-88 (1996).
Suppementary Figure S1
Suppementary Figure S
Suppementary Figure S3
Suppementary Figure S4
Suppementary Figure S5
Suppementary Figure Captions Suppementary Figure S1. est resuts on a -D 8-bit resoution chart (104 104 pies). (a-b) A LR ISO 133 resoution chart and the restored resut by our method. (c) he pea signa to noise ratio (PSNR) versus the frame number of LR images used for restoration, noise Std σn = 0 and PSF Stds σpsf = 5, 10, 15 pies. (d) A comparison of PSNRs by RAM, ALG, RSR and ZM versus the Noise Std when the PSF Stds σpsf are 5, 10, 15, respectivey. 64 LR images are used. (e) A zoom region of the ground truth. (f) A singe LR image from the same zoom region, with Gaussian-shape PSF Std σpsf = 10 and AWGN of Std σn = 0. (g)-(j) Restoration resuts by RAM, ALG, RSR and ZM, respectivey. Suppementary Figure S.est resuts on -D synthetic ce data. (a)-(b) A synthetic HR ce image (31 384 pies) and its LR observation corrupted with noise contamiation of Std σn= 0 and PSF burring of Std σpsf = 31 (pies). 1-D intensity profies of the five structures in the images are aso potted as green curves. (c) Restored image by RAM and the intensity profie of the five structures. (d) FWHM ratio between the LR and restored images for the five structures, respectivey. (e) FWHM ratio between the LR and restored structures versus the Std of noise in the observations. he number of LR frames and PSF Std are fied to be 64 and σpsf = 31 (pies). (f) FWHM ratio between the LR and restored structures versus the number of
LR images for different noise eves in the observations of Std σn = 10, 0 and 30, respecitivey. he Std of PSF is set to be σpsf = 31 (pies). Suppementary Figure S3. Resoution of gated stimuated emission depetion microscopy. a) op row: Confoca and deconvoved gsed images of 0nm beads, scae bar 1µm. Bottom row: Zoom area of boed regions from the same fied of view, scae bar 100nm. b) Line intensity profies aong the indicated ines in (a). c) bo-andwhiser pots indicating the improved resoution by gsed (mean FWHM of 44 nm) over confoca (mean FWHM of 04 nm). his gsed data was obtained with 100% depetion power and -6.5ns gating. d) Intensity ine profies of a bead imaged with progressivey restricted time gating (100% depetion power used) or e) progressivey increasing depetion power (gating set to -6.5ns for gsed imaging). Suppementary Figure S4. Dependence of the RAM restoration on its operating conditions. Upper row: A zoom area of the actin fiaments from the CLSM image Fig 4a (eft) and two RAM restored images of the same area with high (midde) and ow (right) denoising threshod, whist other parameters remain the same. he atter shows more compe structures, incuding fiament bundes, whist in the presence of more visibe noise. here is a good correation between these images as mared by boes. Lower row: A zoom area of the DAPI stained nucei from the CLSM image Fig 4a (eft), two RAM restored images of the same area with high (midde) and ow (right) denoising threshod. he atter reviews more detaied structure of the nucei.
Suppementary Figure S5. Fow chart of RAM. Upper bo depicts the imaging mode (eqn. 1) and the ower bo describes the iterative energy minimization process (eqn. ).