Lifting Detail from Darkness

Transcription

1 Lifting Detail from Darkness J.P.Lewis Disney The Secret Lab Lewis / Detail from Darkness p.1/38

2 Brightness-Detail Decomposition detail image image intensity Separate detail by Wiener filter Represent brightness by membrane PDE Unsharp masking 2.0 Topics: Wiener filter, Laplace PDE, multigrid Lewis / Detail from Darkness p.2/38

3 Brightness-Detail Decomposition detail image image intensity Modify detail, keep brightness. Example: texture replacement (subject to limitations of 2D technique) Modify brightness, keep detail. Example: 102 Dalmatians spot removal Lewis / Detail from Darkness p.3/38

4 Motivation: 102-Dalmatians Lewis / Detail from Darkness p.4/38

5 Motivation: 102-Dalmatians..."It turned out to be a huge job, much bigger than any of us had imagined."..."any spot that procedural could get rid of would help"... Jody Duncan, Out, Out, Damned Spot, Cinefex 84 (January 2001), p. 56. Lewis / Detail from Darkness p.5/38

6 No easy trick Lewis / Detail from Darkness p.6/38

7 No easy trick Hypothetical spot luminance profile, blurred luminance (heavy), and unsharp mask (bottom). Lewis / Detail from Darkness p.7/38

8 No easy trick Spot transition width is quite different on opposite sides of the same spot. Lewis / Detail from Darkness p.8/38

9 Wiener optimal linear estimator; for Gaussian data is optimal period. same principles can be applied as filter interpolator predictor apply in frequency domain or spatially, recursive setting Kalman Lewis / Detail from Darkness p.9/38

10 Wiener 1D y = x + n ˆx = ay E E[ax + by] = ae[x] + be[y] E[xn] = 0 x: the unknown, y: observation ˆx is estimate. Find best a expectation operator E is linear noise not correlated with signal Lewis / Detail from Darkness p.10/38

11 Wiener 1D min a E[(ˆx x) 2 ] Find a that minimizes expected err 2 min a E[a 2 y 2 2ayx + x 2 ] expand square, ˆx min a E[a 2 (x + n) 2 2a(x + n)x + x 2 ] expand y Expand products; E[xn] = 0: min a E[a 2 (x 2 + 2xn + n 2 ) 2a(x 2 + xn) + x 2 ] = 0 = a 2aE[x2 + n 2 ] 2E[x 2 ] a = E[x2 ] E[x 2 +n 2 ] minimize Lewis / Detail from Darkness p.11/38

12 Wiener principle a = E[x2 ] E[x 2 + n 2 ] Signal variance divided by signal variance + noise variance. Lewis / Detail from Darkness p.12/38

13 How to use Wiener Interpret noise as detail, signal as brightness. Lewis / Detail from Darkness p.13/38

14 Membrane 2 u = 0 ( 2 = 2 x y 2) scattered interp for when there are lots of data (c.f. radial basis, other: invert N 2 matrix for N data points) minimizes integrated gradient-squared (roughness) minimizes small-deflection approx. to surface area Lewis / Detail from Darkness p.14/38

15 Membrane Left - impulses in L pattern. Black means no data, intepolate. Right - membrane interpolation. Lewis / Detail from Darkness p.15/38

16 Membrane: minimize roughness roughness R = u 2 du (u k+1 u k ) 2 for a particular k: dr = d [(u k u k 1 ) 2 + (u k+1 u k ) 2 ] du k du k = 2(u k u k 1 ) 2(u k+1 u k ) = 0 u k+1 2u k + u k 1 = 0 or 2 u = 0 Lewis / Detail from Darkness p.16/38

17 Laplacian mask 1D: D: Lewis / Detail from Darkness p.17/38

18 Membrane as matrix eqn 2 u = 0 rewrite 2 as matrix Mu = 0 rows looks like:... 1,......, 1, 4, 1,......, 1, ,......, 1, 4, 1,......, 1, ,......, 1, 4, 1,......, 1,... M is huge - number of pixels in the region, squared! But M is sparse, five diagonal bands. Lewis / Detail from Darkness p.18/38

19 Membrane: relaxation Mu = 0 is too large to solve by conventional matrix inverse, solve by relaxation. u k+1 2u k + u k 1 = 0 u k 0.5(u k+1 + u k 1 ) Lewis / Detail from Darkness p.19/38

20 Membrane: boundary conditions For interpolation some u r,c are known/specified rather than free. In setting up the linear system, subtract these from both sides of the eq, so the known quantities move to the rhs. 1 h 2 (u +0 + u 0 + u 0+ + u 0 4u 00 ) = 0 Say u +0 is known/fixed, then 1 h 2 (u 0 + u 0+ + u 0 4u 00 ) = 1 h 2u +0 Lewis / Detail from Darkness p.20/38

21 Membrane artifact Lewis / Detail from Darkness p.21/38

22 Membrane vs. Thin Plate Left - membrane interpolation, right - thin plate. Lewis / Detail from Darkness p.22/38

23 Multigrid Ax = b approximate solution ˆx = x + e r: residual, e: error r = Aˆx b r = Ax + Ae b but Ax = b so r = Ae Residual has lower frequencies, so e can be solved at low res and subtracted from ˆx to give x. Lewis / Detail from Darkness p.23/38

24 Multigrid results before: several minutes, > 1/2 gig of memory after: several seconds, memory not noticed Lewis / Detail from Darkness p.24/38

25 Algorithm steps Lewis / Detail from Darkness p.25/38

26 Recovered fur Lewis / Detail from Darkness p.26/38

27 Recovered fur: detail Lewis / Detail from Darkness p.27/38

28 Versus hand cloning (manual) Lewis / Detail from Darkness p.28/38

29 Versus hand cloning (auto) Lewis / Detail from Darkness p.29/38

30 Versus hand cloning (manual), edge Image has been sharpened Lewis / Detail from Darkness p.30/38

31 Versus hand cloning (auto), edge Image has been sharpened Lewis / Detail from Darkness p.31/38

32 Other applications Lewis / Detail from Darkness p.32/38

33 Conclusions+Demo image alterations produced quickly with no artistic skill (cloning requires some paint skill) produces consistent effects across frames (cloning may chatter unless done skillfully) Lewis / Detail from Darkness p.33/38

34 Demo Lewis / Detail from Darkness p.34/38

35 Title Lewis / Detail from Darkness p.35/38

36 Title Lewis / Detail from Darkness p.36/38

37 Title Lewis / Detail from Darkness p.37/38

38 Title Lewis / Detail from Darkness p.38/38

39 Title Lewis / Detail from Darkness p.39/38