What Causes Image Intensity Changes?

Transcription

1 Ede Detection l Why Detect Edes? u Information reduction l Replace imae by a cartoon in which objects and surface markins are outlined create line drawin description l These are the most informative parts of the imae u Bioloical plausibility l Initial staes of mammalian vision systems involve detection of edes and local features u Applications l Object reconition, stereo, texture analysis, motion analysis, imae enhancement, imae compression 1 What Causes Imae Intensity Chanes? l Many types of physical events cause intensity chanes u Surface reflectance discontinuity - chane in the fraction of liht incident on the surface that is reflected to viewer u Illumination discontinuity - shadow u Surface orientation (normal) discontinuity u Depth discontinuity - at occludin contour, where surface orientation is perpendicular to line of siht 2 1

2 What type of real-world ede is this contrast border? 3 Need lobal information too 4 2

3 What Does an Ede Look Like? l Step l Ramp l Roof l Line (bar) 5 6 3

4 fine scale hih threshold 7 coarse scale, hih threshold 8 4

5 coarse scale low threshold 9 Ede Detectors in the HVS Simple (border) Complex l Sum activity from an array of oriented simple cells 10 5

6 Orientation Columns in V1 l A hypercolum = complete set of orientations 11 Craik-O Brien-Cornsweet Illusion 12 6

7 Koffka Rin 13 Luminance Differences Affect our Perceptions l Artists use the technique of equiluminance to blur outlines and suest motion. We cannot perceive the edes of objects where object and backround have the same luminance. If parts of a paintin are equiluminant, their positions become ambiuous. They may seem to shift position or to float l Equiluminant colors have special properties, e.., they can make a paintin appear unstable 14 7

8 Equiluminant Colors l An object that can be seen by both subdivisions of the visual system will be perceived accurately. But if the two subdivisions are not balanced in their response to an object, it may look peculiar. l For example, an object defined by equiluminant colors can be seen by the What system but is invisible (or poorly seen) by the Where system. It may seem flat, it may seem to shift position, or it may seem to float ambiuously because there is too little luminance contrast to provide adequate information about its three-dimensional shape, its location in space, or its motion (or lack of it). l Conversely, somethin defined by very low contrast contours is seen by the Where system but not the What system and may seem to have depth and spatial oranization but no clear shape. 15 Detail from Richard Anuszkiewicz s Plus Reversed, 1960 The red and blue seem to move around because they are equiluminant 16 8

9 Ede Detection Goals l Good detection: Low false alarm rate and low false dismissal rate maximize sinal-to-noise (S/N) ratio l Good localization: Mark point closest to center of true ede minimize distance between marked point and center l Uniqueness: Only one response to a sinle ede l Good property measurement: Orientation, contrast, etc. 17 Ede Operator Properties l Shift invariant (translation invariant, position invariant) u If (x,y) = Op[f(x,y)] then (x-a, y-b) = Op[f(x-a, y-b)] l Isotropic (rotation invariant) vs. non-isotropic l Derivative order (if differentiation-based method) l Linear vs. non-linear u Op[a f 1 (x,y) + b f 2 (x,y)] = a Op[f 1 (x,y)] + b Op[f 2 (x,y)] = a 1 (x,y) + b 2 (x,y) l Scale (operator neihborhood size) u (x,y) = Op[f(x+a, y+b), -k < a, b < k ] l Convolution (linear and shift-invariant) 18 9

10 Ede and Local Feature Detection Methods l Gradient-based ede detection l Ede detection by function fittin l Second derivative ede detectors l Ede linkin and the construction of the chain raph 19 1D Ede Detection l An ideal ede is a step function I(x) x I (x) x 20 10

11 1D Ede Detection I (x) x l The first derivative of I(x) has a peak at the ede l The second derivative of I(x) has a zero crossin at the ede 21 1D Ede Detection l More realistically, imae edes are blurred and the reions that meet at those edes have noise or variations in intensity u Blur - hih first derivatives near edes u Noise - hih first derivatives within reions that meet at edes I(x) I (x) x x 22 11

12 Ede Detection in 2D l Let I(x,y) be the imae intensity function. It has derivatives in all directions u I(x, y)/ x = lim I(x+ x, y) - I(x, y) / x I(u+1, v) - I(u,v) u Gradient of I(x, y) is a vector I(x, y) = [ I/ x, I/ y] T specifyin the direction of reatest rate of chane in intensity (i.e., perpendicular to the ede s direction) u From radient can determine the direction in which the first derivative is hihest, and the manitude of the first derivative in that direction u Manitude = [( I/ x) 2 + ( I/ y) 2 ] 1/2 u Direction = tan -1 ( I/ y)/( I/ x) 23 Computin First Derivative l To compute first derivative in direction θ, calculate from linear combination of derivatives from any two non-collinear directions x = x cos θ - y sin θ y y θ x x y = x sin θ + y cos θ I/ x = I/ x x/ x + I/ y y/ y = I/ x cos θ + I/ y sin θ I/ y = - I/ x sin θ + I/ y cos θ ( I/ x ) 2 + ( I/ y ) 2 = ( I/ x) 2 + ( I/ y) 2 So, sum of squares of first derivative is isotropic, non-linear Similarly, all derivatives of odd ordcr raised to an even power are isotropic 24 12

13 Gradient l Gradient = [ I/ x, I/ y] T l What direction is first derivative a maximum? u Set / θ ( I/ x ) = 0, and solve for θ / θ ( I/ x cos θ + I/ y sin θ) = 0 θ = tan -1 ( I/ y / I/ x) l Gradient direction is perpendicular to ede direction l Gradient manitude is isotropic I = I x 2 I + y 2 25 Ede Detection in 2D l With a diital imae, the partial derivatives are replaced by finite differences: u x I = I(u+1, v) - I(u, v) u y I = I(u, v) - I(u, v+1) l An alternative (Sobel) u sobel_x I = I(u+1, v+1) + 2I(u+1, v) + I(u+1, v-1) - I(u-1, v+1) - 2I(u-1, v) - I(u-1, v-1) u sobel_y I = I(u-1, v-1) + 2I(u, v-1) + I(u+1, v-1) - I(u-1, v+1) - 2I(u, v+1) - I(u+1, v+1) l Roberts s Cross u + I = I(u, v) - I(u+1, v-1) u - I = I(u, v-1) - I(u+1, v) v u 26 13

14 Gradient Operator Example l I = l x = -1 1 l x I = *

15 15 30 Directional Ede Operators l Kirsch 8-direction masks l Gradient manitude = l Gradient direction = 45 armax k n = k = k = k = k k n n max =0,...7

16 Directional Ede Operators l Robinson masks r 0 1 = r 2 7 = l Nevatia and Babu used 6 5x5 masks, detectin orientations at multiples of Ede Detection in 2D l How do we combine the directional derivatives to compute the radient manitude? u Use the root mean square (RMS) as in the continuous case u Sum of the absolute values of the directional derivatives u Maximum of the absolute values of the directional derivatives l Advantaes of the latter u Avoids computin square root (althouh this can be done usin table lookups) u Keeps result in the same rane as the oriinal imae u Gives a manitude that is invariant with respect to orientation of the ede in the imae ( sobel_x overestimates diaonal edes and + overestimates horizontal and vertical ede manitudes) 32 16

17 Finite Differences and Noise l Finite difference filters respond stronly to noise u obvious reason: imae noise results in pixels that look very different from their neihbors l Generally, the larer the noise the stroner the response l What is to be done? u intuitively, most pixels in imaes look quite a lot like their neihbors u this is true even at an ede; alon the ede they re similar, across the ede they re not u suests that smoothin the imae should help, by forcin pixels different from their neihbors (=noise pixels?) to look more like their neihbors 33 Finite Differences Respondin to Noise Increasin noise -> (this is zero mean additive aussian noise) 34 17

18 Combinin Smoothin and Differentiation - Fixed Scale l Local operators like the Roberts operator ive hih responses to any intensity variation u local surface texture u noise from the sensin process l If the picture is first smoothed by an averain process, then these local variations are removed and what remains are the prominent edes l Example: I 2x2 (u,v) = 1/4[I(u,v) + I(u+1,v) + I(u,v+1) + I(u+1,v+1)] 35 Smoothin - Basic Problems l What function should be used to smooth or averae the imae before differentiation? u Mean smoothin (aka box filters or uniform smoothin or averain) l easy to compute (linear) l for lare smoothin neihborhoods assins too much weiht to points far from an ede u Median smoothin l doesn t blur corners l unaffected by outliers u Gaussian smoothin l [1/(2πσ 2 )] e -(u2 + v 2 )/2σ

19 Example: Smoothin by Averain 38 19

20 Smoothin with a Gaussian 39 The Mystery of the Mona Lisa Smile l Mararet Livinstone has conjectured that the elusive smile is because of the differences in resolution in the fovea and the periphery of the retina Low-pass filtered Middle-pass Hih-pass 40 20

21 Chanin small details can affect the mood we perceive. A few pixels of chane switches her from happy to sad. Addin random noise also chanes our perception of her expression

22 Smoothin and Convolution l The convolution of two functions, f(x) and (x), is defined as h(x) = (x ) f(x-x ) dx = (x) * f(x) l When f and are discrete and when is nonzero only over a finite rane [-n, n], this interal is replaced by the summation h(u) = (i) f(u-i) 43 Example of 1D Convolution f h h(4) 12 2 = i= ( i) f (4 i) Note: Values in h have been divided by = 13 = ( 2) f (6) + ( 1) f (5) + (0) f (4) + (1) f (3) + (2) f (2) 44 22

23 Smoothin and Convolution l These interals and summations extend simply to functions of two variables: h(u,v) = (u,v) * f(u,v) = (i,j) f(u-i, v-j) l Convolution computes the weihted sum of the ray levels in each n x n neihborhood of the imae, f, usin the kernel matrix of weihts, l Convolution is a linear operator because u *(af 1 + bf 2 ) = a(*f 1 ) + b(*f 2 ) 45 Convolution 46 23

24 D Convolution (4,4) (1,1) (4,5) (1,0) (4,6) 1) (1, (5,4) (0,1) (5,5) (0,0) (5,6) 1) (0, (6,4) 1,1) ( (6,5) 1,0) ( (6,6) 1) 1, ( ),5 (5 ), ( (5,5) f f f f f f f f f j i f j i h i j = = = = 48 Smoothin and Convolution

25 Properties of Convolution l Commutative l Associative l Cascadable l Linear, shift-invariant l Any linear, shift-invariant operation can be implemented as a convolution l Convolution in spatial domain = multiplication in frequency domain. I.e., = f * h iff G = F H, where * denotes convolution and F = (f) l An alternative for computin : = -1 (F H) 49 Combinin Smoothin and Ede Detection l Let S = l x = -1 1 l Smooth, then differentiate: x * (S * I), where I is oriinal imae and * is convolution = ( x * S) * I l x * S = = Prewitt_x

26 Back to Smoothin Functions l To smooth an imae usin a Gaussian filter, e -(u2 + v 2 )/2σ 2, we must u Choose an appropriate value for σ, which controls how quickly the Gaussian falls to near 0 l Small σ produces filter which drops to near 0 quickly - can be implemented usin small diital array of weihts l Lare σ produces a filter which drops to near 0 slowly - will be implemented usin a larer size diital array of weihts u Determine the size weiht array needed to adequately represent that Gaussian l To represent 98.76% of the area under the Gaussian usin σ=σ 0 use a mask of size 5σ 0 x 5σ 0 l In practice, size of mask is often closer to size 3σ x 3σ l Weiht array needs to be of odd size (2k+1 x 2k+1) to allow for symmetry 51 Gaussian Smoothin l To smooth an imae usin a Gaussian filter we must u Sample the Gaussian by interatin it over the square pixels of the array of weihts and multiplyin by a scale factor to obtain inteer weihts u Because we have truncated the Gaussian, the weihts will not sum to 1.0 x scale factor l In flat areas of the imae we expect our smoothin filter to leave the imae unchaned l But if the filter weihts do not sum to 1.0 x scale factor, it will either amplify (> 1.0) or de-amplify (< 1.0) the imae 52 26

27 Gaussian Smoothin l Gaussian smoothin steps u Normalize the weiht array by dividin each entry by the sum of the all of the entries (interal equal to 1) u Convert to inteers l Advantaes of Gaussian filterin u Rotationally symmetric (for lare filters) u Filter weihts decrease monotonically from central peak, ivin most weiht to central pixels u Simple and intuitive relationship between size of σ and size of objects whose edes will be detected in imae. u The Gaussian is separable 53 Buildin Discrete Gaussian Kernels l Given σ and w, compute real-valued weihts for w x w kernel array l Divide all entries by minimum weiht and round result to nearest inteer, obtainin kernel G l Compute normalization factor n = weihts l I' = 1/n (G * I) 54 27

28 Separability l A function (x, y) is separable if (x, y) = 1 (x) 2 (y) l The Gaussian function is separable: e -[(x2 + y 2 )/2σ 2 ] = e -[x2 /2σ 2 ] * e -[y2 /2σ 2 ] l First convolve the imae with a 1D horizontal filter l Then convolve the result of the first convolution with a 1D vertical filter l For a k x k Gaussian filter, 2D convolution requires k 2 multiplications and k 2-1 additions per pixel l But usin the separable filters, we reduce this to 2k multiplications and 2k-1 additions per pixel 55 Separability I = = = x = = 11 2 = = = = =

29 Advantaes of Gaussians l Cascadin Gaussians: Convolution of a Gaussian with itself is another Gaussian u So, we can first smooth an imae with a small Gaussian u Then, we convolve that smoothed imae with another small Gaussian and the result is equivalent to smoother the oriinal imae with a larer Gaussian u If we smooth an imae with a Gaussian havin standard deviation σ twice, then we et the same result as smoothin the imae with a Gaussian havin standard deviation 2σ 57 Combinin Smoothin and Differentiation - Fixed Scale l Non-maxima suppression - Retain a point as an ede point if u its radient manitude is hiher than a threshold u its radient manitude is a local maximum in the radient direction l Lateral inhibition process 58 29

30 Summary of Basic Ede Detection Steps l Smooth the imae to reduce the effects of local intensity variations u Choice of smoothin operator practically important l Differentiate the smoothed imae usin a diital radient operator that assins a manitude and direction of the radient at each pixel u Mathematically, we can apply the diital radient operator to the diital smoothin filter, and then just convolve the differentiated smoothin filter to the imae u Requires usin a slihtly larer smoothin array to avoid border effects l Threshold the radient manitude to eliminate low contrast edes 59 Summary of Basic Ede Detection Steps l Apply a non-maximum suppression step to thin the edes to sinle pixel wide edes u Smoothin will produce an imae in which the contrast at an ede is spread out in the neihborhood of the ede u Thresholdin operation will produce thick edes 60 30

31 Canny Ede Operator l Desin operator that is optimal with respect to ood detection, ood localization, and unique response criteria l Good detection criterion: Maximize SNR, which is the ratio of the step size of a step ede over the standard deviation of the 0-mean, additive Gaussian noise l Good localization criterion: Maximize inverse distance of detected ede from true ede l Optimal 1D step-ede detector = detector that maximizes the product of the two terms above subject to third criterion l First derivative of a Gaussian is within 20% of optimal 61 Directional Derivatives l Isotropic operators smooth across edes Poor localization l Instead, averae pixels alon ede direction only Use a set of directional derivatives at a lare number of orientations etc

32 Canny Ede Operator l Gaussian smooth imae: G * I l Estimate radient direction (i.e., ede normal), n, at each pixel: ( G * I) n = ( G * I) l Compute first derivative in the radient direction, n. That is, G/ n * I l Suppress non-maxima u If value is not a local max in direction n, then chane value to 0 u Or, find zero-crossins in direction n. I.e., zero-crossins of ([ G/ n] * I)/ n = 2 (G * I)/ n 2 63 Canny Ede Operator l To et rid of spurious edes created by noise (false positives) and to fill in missin edes (false neatives) use hysteresis thresholdin: u Define two thresholds, t hih and t low u Mark all pixels with ede manitude > t hih as ede pixels u For each marked pixel, search in ede direction, both ways, markin all pixels with ede manitude > t low as ede pixels. Stop search when a pixel is reached with ede manitude < t low 64 32

33 Laplacian Ede Detectors l Directional second derivative in direction of radient has a zero crossin at radient maximum l Can approximate directional second derivative with Laplacian: 2 I = 2 I/ x I/ y 2 l Laplacian is lowest order linear isotropic operator l Diital approximation (2nd forward difference) is u 2 I(u,v) = [I(u+1,v) + I(u-1,v) + I(u,v+1) + I(u,v-1)] - 4I(u,v) = [I(u+1,v) - I(u,v)] - [I(u,v) - I(u-1,v)] + [I(u,v+1) - I(u,v)] - [I(u,v) - I(u,v-1)] Laplacian Examples l I = l 2 = I = l I = I =

34 Problems with Laplacian of Small Size l Responds most stronly at isolated points (spots) noise sensitive l Responds stronly at lines and endpoints of lines l Responds relatively less at edes l Ramp edes detected at two ends, not once at center 67 34

35 Usin Laplacian for Ede Enhancement l Given a blurred imae I = 1/5 (I * ) l 2 I I - I l - k 2 I - k(-1/5 (I - I )) = k/5 I + I (1 - k/5) I when k=5 l ==> Can deblur an imae by subtractin a multiple of its Laplacian l Simulates Mach Band effect in human visual system l Analoous to unsharp maskin technique in photoraphy 69 Mach Bands l In 1865, Mach discovered that we perceive an ede (chane of intensity) at lines that separate surface reions with identical intensity on both sides but different intensity derivatives l It is caused by the lateral inhibition of the receptors in the eyes 70 35

36 Mach Bands Actual brihtness Perceived by you 71 Laplacian Ede Detectors l Laplacians are also combined with smoothin for ede detectors u Take the Laplacian of a Gaussian-smoothed imae - called the Laplacian-of-Gaussian (LoG), Mexican Hat operator, Difference-of-Gaussians (DoG), Marr- Hildreth, 2 G operator u Locate the zero-crossin of the operator l these are pixels whose LoG is positive and which have neihbor s whose LoG is neative or zero u Usually, measure the radient or directional first derivatives at these points to eliminate low contrast edes 72 36

37 Laplacian of Gaussian (LoG) 2 G σ (x, y) = -[1/2πσ 4 ] (2 - (x 2 + y 2 )/σ 2 ) e -(x2 + y 2 )/2σ

38 LoG Properties l Linear, shift invariant convolution l Circularly symmetric isotropic l Size of LoG operator approximately 6σ x 6σ l LoG is separable l LoG G σ1 - G σ2, where σ 1 = 1.6σ 2 l Analoous to spatial oranization of receptive fields of retinal anlion cells, with a central excitatory reion and an inhibitory surround

39 Receptive Field + - u Reion on retina that influences activity of a anlion cell u Firin rate of neuron u Excitatory center, inhibitory surround 77 u Ganlion cells detect differences in luminance at a border, line, or spot u Retina is NOT a spot-meter 78 39

40 Lateral Inhibition Explanation dark band liht band B < A B inhibited by surround D > E D less inhibited by surround 79 Hermann Grid l Illustrates lateral inhibition 80 40

41 Visual illusions 81 Lateral Inhibition Explanation Inhibited less by white stripes Inhibited more by white stripes 82 41

42 The Scale Space Problem l Usually, any sinle choice of σ does not produce a ood ede map u a lare σ will produce edes from only the larest objects, and they will not accurately delineate the object because the smoothin reduces shape detail u a small σ will produce many edes and very jaed boundaries of many objects l Scale-space approaches u detect edes at a rane of scales [σ 1, σ 2 ] u combine the resultin ede maps l trace edes detected usin lare σ down throuh scale space to obtain more accurate spatial localization 83 42

43 43

44 44

45 Pyramids l Very useful for representin imaes l Pyramid is built by usin multiple copies of imae l Each level in the pyramid is 1/4 the area of previous level, i.e., each dimension is 1/2 resolution of previous level 89 Pyramids 45

46 91 Gaussian Pyramid 92 46

47 Gaussian Pyramid l Multiresolution, low-pass filter l Hierarchical convolution u G0 = input imae u G k (u, v) = w(m, n) G k-1 (u-m, v-n) u G k (u, v) = G k (2u, 2v), 0 < u, v < 2 N-k ; smooth ; sub-sample l w is a small (e.., 5 x 5) separable eneratin kernel, e.., 1/16 [ ] l Cascadin w equivalent to applyin lare kernel u Effective size of kernel at level k = 2M(2 k - 1) + 1, where w has width 2M+1 u Example: Let M=1. If k=1 then equivalent size = 5; k=2 then equivalent size = 13; k=3 then equivalent size = Gaussian Pyramids 2 2 = m= 2n = 2 l ( u, v) w( m, n) l 1(2u + m,2v + n) l = REDUCE [ l 1 ] 94 47

48 Reduce (1D) 2 l = wˆ( m) l + m=2 ( u) (2) = wˆ( 2) l wˆ (0) wˆ (0) l1 l1 (4) + wˆ(1) (2) = wˆ ( 2) l l1 l1 (4) + wˆ (1) 1(2u m) (4 2) + wˆ ( 1) l1 (2) + wˆ ( 1) l1 l1 (4 + 1) + wˆ (2) l1 (5) + wˆ (2) (3) + l1 (6) (4 1) + l1 (4 + 2) 95 Reduce 96 48

49 49 97 Gaussian Pyramids [ 1 ],, = n l n l EXPAND ) 2, 2 ( ), ( ), ( ,, q v p u q p w v u p q n l n l = = = 98 Expand (1D) ) 2 ( ) ˆ ( ) ( 2 2 1,, p u p w u p n l n l = = ) ( ˆ (2) ) ( ˆ (1) ) 2 4 ( ˆ (0) ) ( 1) ˆ( ) ( 2) ˆ ( (4) 1, 1, 1, 1, 1,, = n l n l n l n l n l n l w w w w w (3) ˆ(2) (2) ˆ (0) (1) 2) ˆ( 4) ( 1, 1, 1,, + + = n l n l n l n l w w w

50 Expand 99 Convolution Mask [ w( 2), w( 1), w(0), w(1), w(2)]

51 Convolution Mask l Separable w ( m, n) = wˆ ( m) wˆ( n) l Symmetric wˆ ( u) = wˆ ( u) [ c, b, a, b, c] 101 Convolution Mask l The sum of mask should be 1: a + 2 b + 2c = 1 l All nodes at a iven level must contribute the same total weiht to the nodes at the next hiher level: a + 2 c = 2b

52 Convolution Mask w$ ( 0) = a 1 w$ ( 1) = w$ ( 1) = 4 1 a w$ ( 2) = w$ ( 2) = Alorithm l Apply 1D mask to alternate pixels alon each row of imae l Apply 1D mask to each pixel alon alternate columns of resultant imae from previous step

53 Laplacian Pyramids l Similar to results of ede detection l Most pixels are 0 l Can be used for imae compression: L L 2 = 2 L 1 = 1 EXPAND[ 2 ] EXPAND[ 3 ] 3 = 3 EXPAND[ 4 ]

54 Laplacian Pyramid l Computes a set of bandpass filtered versions of imae l L k = G k - w * G k = G k - Expand(G k+1 ) l L N = G N (apex of Laplacian pyr = apex of Gaussian pyr) l Separates features by their scale (size) l Enhances features l Compact representation l L k = (G 0 - G 1 ) + (G 1 - G 2 ) (G N-1 - G N ) + G N = G

55 Gaussian and Laplacian Pyramids 109 Codin usin the Laplacian Pyramid Compute Gaussian pyramid 1 2, 3,, 4 Compute Laplacian pyramid L L L L = 1 = = = EXPAND [ EXPAND [ EXPAND [ Code Laplacian pyramid ] ] ]

56 Decodin usin Laplacian pyramid l Decode Laplacian pyramid l Compute Gaussian pyramid from Laplacian pyramid: 4 = L 4 [ + L 3 = EXPAND 4] 2 = EXPAND[ 3 ] + L = EXPAND[ 2 ] + L 1 l 1 is reconstructed imae 111 Imae Compression Laplacian Pyramid After Quantization Level

57 Imae Compression Oriinals Encoded 113 Imae Compositin by Pyramid Blendin l Given: Two 2 n x 2 n imaes l Goal: Create an imae that contains left half of imae A and riht half of imae B l Alorithm u Compute Laplacian pyramids, LA and LB, from imaes A and B u Compute Laplacian pyramid LS by copyin left half of LA and riht half of LB. Pyramid nodes down the center line = averae of correspondin LA and LB nodes blend alon center line u Expand and sum levels of LS to obtain output imae S

58 Combinin Apple & Orane 115 Combinin Apple & Orane (usin Pyramids)

59 Imae Compositin from Arbitrary Reions l Given: Two 2 n x 2 n imaes and one 2 n x 2 n binary mask l Goal: Output imae containin imae A where mask=1, and imae B where mask=0 l Alorithm: u Construct Laplacian pyramids LA and LB from imaes A and B u Construct Gaussian pyramid GR from mask R u Construct Laplacian pyramid LS: LS k (u, v) = GR k (u, v) LA k (u, v) + (1 - GR k (u, v)) LB k (u, v) u Expand and sum levels of LS to obtain output imae S

60 60

61 61

62 Oriented ( Steerable ) Pyramids l Laplacian pyramid is orientation-independent l Apply an oriented filter to determine orientations at each layer u by clever filter desin, we can simplify synthesis u this represents imae information at a particular scale and orientation

63 125 From Shiftable MultiScale Transforms, by Simoncelli et al., IEEE Transactions on Information Theory,

64 Analysis 127 Synthesis

65 Ede Detection by Function Fittin: Facet Model l General approach u Fit a function to each pixel s neihborhood of the imae u Use the radient of the function as the diital radient of the imae neihborhood l Example: Fit a plane to a 2 x 2 neihborhood u z = ax + by + c; z is ray level u Gradient is then (a 2 + b 2 ) 1/2 u Neihborhood points are I(x,y), I(x+1,y), I(x,y+1) and I(x+1,y+1) l Need to minimize E(a,b,c) = ΣΣ [a(x+i) + b(y+j) + c - I(x+i,y+j)] 2 l Solve this and similar problems by: u Differentiatin with respect to a,b,c, set results to 0, and u Solve for a,b,c in resultin system of equations 129 Ede Detection by Function Fittin l E/ a = ΣΣ2[a(x+i) + b(y+j) + c - I(x+i,y+j)](x+i) l E/ b = ΣΣ2[a(x+i) + b(y+j) + c - I(x+i,y+j)](y+j) l E/ c = ΣΣ2[a(x+i) + b(y+j) + c - I(x+i,y+j)] l It is easy to verify that a = [I(x+1,y) + I(x+1,y+1) - I(x,y) - I(x,y+1)]/2 b = [I(x,y+1) + I(x+1,y+1) - I(x,y) - I(x+1,y)]/2 l a and b are the partial derivatives with respect to x and y a = -1 1 b =

66 Ede Detection by Function Fittin l Could also fit a hiher order surface than a plane u With a second order surface we could find the (linear) combination of pixel values that corresponds to the hiher order derivatives, which can also be used for ede detection l Would ordinarily use a neihborhood larer than 2 x 2 u Better fit u For hih deree functions need more points for the fit to be reliable 131 Ede Linkin and Followin l Group ede pixels into chains and chains into lare pieces of object boundary. u can use the shapes of lon ede chains in reconition l slopes l curvature l corners l Basic steps u thin connected components of edes to one pixel thick u find simply connected paths u link them at corners into a raph model of imae contours u compute local and lobal properties of contours and corners

67 Findin Simply Connected Chains l Goal: create a raph-structured representation (chain raph) of the imae contours u vertex for each junction in the imae u ede connectin vertices correspondin to junctions that are connected by an chain; ede labeled with chain u b α β a s c φ d f δ e ε η r h j β b α c a δ e ε d φ f η 133 Creatin the Chain Graph l Alorithm: iven binary imae, E, of thinned edes u create a binary imae, J, of junctions and end points l points in E that are 1 and have more than two neihbors that are 1 or exactly one neihbor that is a 1 u create the imae E-J = C(chains) l this imae contains the chains of E, but they are broken at junctions u perform a connected component analysis of C. For each component store in a table T: l its end points (0 or 2) l the list of coordinates joinin its end points u For each point in J: l create a node in the chain raph, G, with a unique label

68 Creatin the Chain Graph l For each chain in C u if that chain is a closed loop (has no end points) l choose one point from the chain randomly and create a new node in G correspondin to that point l mark that point as a loop junction to distinuish it from other junctions l create an ede in G connectin this new node to itself, and mark that ede with the name of the chain loop u if that chain is not a closed loop, then it has two end points l create an ede in G linkin the two points from J adjacent to its end points 135 Creatin the Chain Graph l Data structure for creatin the chain raph l Biest problem is determinin for each open chain in C the points in J that are adjacent to its end points u sophisticated solution miht use a hierarchical data structure, like a k-d tree, to represent the points in J u more direct solution is to create imae J in which all 1 s are marked with their unique labels u For each chain in C l Examine the 3 x 3 neihborhood of each end point of C in J l Find the name of the junction or end point adjacent to that end point from this 3 x 3 neihborhood

69 Findin Internal Corners of Chains l Chains are only broken at junctions u But important features of the chain miht occur at internal points u Example: closed loop correspondin to a square - would like to find the natural corners of the square and add them as junctions to the chain raph (splittin the chains at those natural corners) l Curve sementation u similar to imae sementation, but in a 1D form l Local methods, like ede detectors l Global methods, like reion analyzers 137 Local Methods of Curve Sementation l Natural locations to sement contours are points where the slope of the curve is chanin quickly u These correspond, perceptually, to corners of the curve l To measure the chane in slope we are measurin the curvature of the curve u Straiht line has 0 curvature u Circular arc has constant curvature correspondin to 1/r l Can estimate curvature by fittin a simple function (circular arc, quadratic function, cubic function) to each neihborhood of a chain, and usin the parameters of the fit to estimate the curvature at the center of the neihborhood

70 Formulas for Curvature l Consider movin a point, P, alon a curve u Let T be the unit tanent vector as P moves l T has constant lenth (1) l The direction of T, φ, chanes from point to point unless the curve is a straiht line l Measure this direction as the anle between T and the x-axis T = dr / ds, s distance alon curve P φ R 139 Formulas for Curvature l The curvature, κ, is the instantaneous rate of chane of φ with respect to s, distance alon the curve u κ = dφ / ds u ds = [dx 2 + dy 2 ] 1/2 u φ = tan -1 dy/dx T = dr / ds, s distance alon curve P φ R

71 Formulas for Curvature l Now dφ / dx = d 2 y dx ( dy dx )2 and so ds / dx = κ = dφ / ds = 1+ ( dy dx )2 d 2 y dφ / dx ds / dx = dx 2 [1+ ( dy dx )2 ] 3/ Example - Circle l For the circle u s = aθ u φ = θ + π/2 u so κ = dφ/ds = dθ/adθ = 1/a a θ s φ

72 Local Methods of Curve Sementation l There are also a wide variety of heuristic methods to estimate curvature like local properties u For each point, p, alon the curve u Find the points k pixels before and after p on the curve (p +k, p -k ) and then measure l the anle between pp +k and pp -k θ l the ratio s/t l Similar problems to ede detection u what is the appropriate size for k? t k s k u how do we combine the curvature estimates at different scales? u boundary problems near the ends of open curves - not enouh pixels to look out k in both directions