A STATISTICAL APPROACH TO IMAGE WARPING. Chris Glasbey. Biomathematics and Statistics Scotland

Transcription

1 A STATISTICAL APPROACH TO IMAGE WARPING Chris Glasbey Biomathematics and Statistics Scotland

2 IMAGE ANALYSIS: the extraction of information from pictures Which line is longer? Who is this? 2

3 The human eye/brain is a superb image analyser So, why use a computer? better for quantitative tasks (repeatable/non-subjective) cheaper, faster and less tedious capable of different techniques 3

4 An example of non-automated image analysis: Cross-section of turbinate bone = Using sno-pake and black ink. 4

5 But automating image analysis is hard, because all a computer sees is: Could you recognise Mona Lisa from this? 5

6 Question 1: Can we superimpose microscope images? brightfield DIC phase contrast 6

7 Question 2: Can we distinguish between species of fish? haddock whiting 7

8 Question 3: Are the E. coli samples already in database? Pulsed-field gel electrophoresis (PFGE) database 8

9 Who is this? 9

10 George W. Bush Arnold Schwarzenegger Image morphing c 10

11 These are all examples of IMAGE WARPING j k i (i, j) f l (k, l) Y µ Y x µ f(x) OR Y (i,j) µ f(i,j) where f(i, j) =(k, l) is a matching criterion f is warping function, either parametric or with distortion penalty 11

12 Polynomial warps Translation k = i + a 00 l = j + b 00 Translation + Dilation k = ci + a 00 l = cj + b 00 Procrustes k = ci cos θ + cj sin θ + a 00 l = ci sin θ + cj cos θ + b 00 Translation + Rotation k = i cos θ + j sin θ + a 00 l = i sin θ + j cos θ + b 00 Bilinear k = a 10 i + a 01 j + a 11 ij + a 00 l = b 10 i + b 01 j + b 11 ij + b 00 Polynomial k = a mn i m j n l = b mn i m j n Affine k = a 10 i + a 01 j + a 00 l = b 10 i + b 01 j + b 00 Perspective k =(a 10 i + a 01 j + a 00 )/(c 10 i + c 01 j +1) l =(b 10 i + b 01 j + b 00 )/(c 10 i + c 01 j +1) 12

13 Parametric warping between species of fish: D Arcy Thompson, On Growth and Form (1917) 13

14 Distortion penalties have been motivated by: elastic membranes optical or fluid flow diffusion Markov random fields thin plate splines ( kriging) 14

15 An example of thin plate splines: Venus Venus warped average Arad et al. (CVGIP: Graphical Models and Image Processing, 1994) 15

16 PLAN 0. Introduction 1. Matching criteria 2. Distortion penalties 3. Algorithms 4. Summary 16

17 1. Matching criteria Question 1: Can we superimpose microscope images? The Fourier representation of Y is Y x = ω A ω cos θω +2πω T x x X. Y A (amplitude) θ (phase) 17

18 µ A (µ) θ (µ) Consider a second image, µ, and difference the phases θ θ (µ) 18

19 We propose a Fourier-von Mises model to estimate translation α θ ω θ ω (µ) M(2πω T α, κ ω ) where precision κ ω (A ω ) γ 1 A (µ) γ 2 exp γ3 ω + γ 4 ω 2 ω 19

20 We propose a Fourier-von Mises model to estimate translation α θ ω θ ω (µ) M(2πω T α, κ ω ) where precision κ ω (A ω ) γ 1 A (µ) γ 2 exp γ3 ω + γ 4 ω 2 ω The log-likelihood L ω κ ω cos θω θ (µ) ω +2πω T α generalises the cross-covariance C(α) = x µ xy x+α = ω A ωa ω (µ) cos θω θ ω (µ) and the phase correlation C (α) = cos θω θ (µ) ω ω +2πω T α +2πω T α for special cases γ =(1, 1, 0, 0) and γ =(0, 0, 0, 0) 20

21 Alignment of multiple subimages Similarity criterion s.d. α 1 α 2 Covariance Phase-correlation Fourier-von Mises log-likelihood

22 Pseudo-coloured composite image 22

23 Landsat Thematic Mapper (TM) images of east coast of Scotland band 1 band 2 band 3 band 4 band 5 band 7 23

24 Phase differences between Fourier transforms of pairs of bands 24

25 mean angle precision omega_2 Phase difference between bands 1 and 2 The linear trend in phase differences is affected by aliasing at high frequencies 25

26 We have developed a method for aligning images, which takes account of aliasing We use a parametric model for the power- and cross-spectra of the multivariate, stochastic process in continuous-space A side benefit is improved interpolation: de-aliasing using information from other bands 26

27 Tay bridge fields original cubic interpolation new result 27

28 PLAN 0. Introduction 1. Matching criteria 2. Distortion penalties 3. Algorithms 4. Summary 28

29 Question 2: Can we distinguish between species of fish? one species another species 29

30 We estimate the warp, f =(f 1,f 2 ), to minimise a penalised sum of squares P = Yx µ 2 x f(x) + λd(f,c) where D is a distortion penalty such that D(f,C) =0 f C > 0 f / C. For the commonly used thin-plate-spline penalty D = 2 i=1 2 2 j=1 k=1 C is the set of affine transformations 2 f i x j x k 2 dx 30

31 For shape comparisons, we wish to restrict C to Procrustes similarity transformations (S). We do this by modifying an existing (base) distortion penalty (D B ): D(f,S) = min g S D B(f g) If then D B (f) = 2 i=1 2 j=1 f i x j 2 dx D(f,S) = D B (f) 2n 1 n 2 ( α α 2 12), where the images are of size n 1 n 2 and α 11 = 1 f 1 + f 2 2n 1 n 2 x 1 x 2 α 12 = 1 f 1 f 2 2n 1 n 2 x 2 x 1 dx, dx. 31

32 Using D(f,S), all values of P for within species comparisons are less than those for between species comparisons: between within Whereas, if the thin-plate-spline penalty is used, within species values of P are no smaller than between species comparisons: P between within P (λ chosen to maximise studentised difference between sets of comparisons) 32

33 haddock 1 haddock 2 warping haddock 1 impersonating haddock 2 33

34 haddock 1 whiting 1 warping haddock 1 impersonating whiting 1 34

35 We formed average fish by warping into alignment 8 haddock and 8 whiting average haddock average whiting 35

36 Distances between average fish and 16 used in training plus 4 more (circled) 36

37 PLAN 0. Introduction 1. Matching criteria 2. Distortion penalties 3. Algorithms 4. Summary 37

38 Question 3: Are the E. coli samples already in database? Track markers Y µ We seek the warping function (f) and lane labels (l) that minimise C(f,l) = i j (Yij µ i+fij,l j ) 2 + λ 1 (f ij f i 1,j ) 2 + λ 2 (f ij f i,j 1 ) 2 38

39 Three Generalisations of DP A) Iterated DP (Leung et al, 2004) 0) Initialise f =0 1) For each column j in turn, use DP to minimise C(f,l) w.r.t. f.j and l j, given current values of all other f s and l s, i.e. minimise i Yij µ 2 i+fij,l j + λ1 (f ij f i 1,j ) 2 { + λ 2 (fij f i,j 1 ) 2 +(f ij f i,j+1 ) 2} where (f i f i 1 )=0.2, 0.4,...,1.8 2) Repeat (1) until convergence OR DP-IDP: replace step (0) by 0) For each column j in turn, use DP to minimise w.r.t. f.j and l j 2 Y ij µ i+fij,l + λ1 (f j ij f i 1,j ) 2 i 39

40 B) Stochastic IDP 0) Initialise f =0and T =10 5 1) Simulate e ijfl U[0,T] for all i, j, f, l Then, for each column j in turn, use DP to minimise i Yij µ 2 +eij,fij i+fij,l j,l j + λ 1 (f ij f i 1,j ) 2 { + λ 2 (fij f i,j 1 ) 2 +(f ij f i,j+1 ) 2} w.r.t. f.j and l j 2) Reduce T αt, and repeat (1) until f unchanged, with α such that {# iterations} 2 6,

41 C) Forward-Backwards (FB) sampler + simulated annealing 0) Initialise f =0and T =10 5 1) For each column j in turn, use FB (Eddy, 1995; Scott, 2002) to sample f.j and l j with probability e C(f,l)/T, given current values of all other f s and l s, i.e. sample from exp 1 T i (Y ij µ i+fij,lj ) 2 + λ1 (f ij f i 1,j ) 2 + λ 2 { (fij f i,j 1 ) 2 +(f ij f i,j+1 ) 2} 2) Reduce T αt, and repeat (1) until f unchanged 41

42 Gel 1 Gel 2 C(f,l) IDP DP IDP stoch IDP FB SA C(f,l) IDP DP IDP stoch IDP FB SA CPU (sec) CPU (sec) 42

43 Y f 1 ij,j, where f 1 is an inverse function in the first index, defined such that f f 1 ij,j i. gel 1 gel 2 gel 3 43

44 Re-ordered class 1 class 2 class 3 class 4 new class Arrowed columns visually identified as belonging to E. coli strains not in database 44

45 4. Summary We have considered some aspects of the broad subject of image warping: Fourier matching criteria null-set distortion penalties iterated dynamic programming algorithms With applications in: microscopy remote sensing digital photography bioinformatics For further details, see papers on 45