Color2Gray: Salience-Preserving Color Removal

Transcription

1 Color2Gray: Salience-Preserving Color Removal Amy Gooch Sven Olsen Jack Tumblin Bruce Gooch Slides by Gooch et al.

2 Color Removal Color Grayscale

3 Color Removal Color Proposed Algorithm

4 Isoluminant Colors Color Grayscale

5 Traditional Methods Contrast enhancement & Gamma Correction Doesn t help with isoluminant values Photoshop Grayscale PSGray + Auto Contrast Proposed Algorithm

6 Overview Conversion to grayscale is dimensionality reduction From tristimulus values to single channel => Loss of information Luminance Axis Goal: Maintain salient features in color image

7 Human color perception: Relative differences Color Illusion by Lotto and Purves

8 Human color perception: Relative differences Color Illusion by Lotto and Purves

9 Algorithm Overview Convert to Perceptually Uniform Space CIE L*a*b*

10 Algorithm Overview Convert to Perceptually Uniform Space CIE L*a*b* Initialize grayscale image, g, with L channel For each pair of neighboring pixels i, j Compute Luminance distance Compute Chrominance distance δ ij Adjust g to incorporate both luminance and chrominance differences

11 Parameters μ : Radius of neighboring pixels α : Max chrominance offset θ : Map chromatic difference to increases or decreases in luminance values

12 μ : Neighborhood Size μ = 2 μ = 16 μ= entire image

13 μ : Neighborhood Size μ = 16 μ= entire image

14 α: Chromatic variation maps to luminance variation Luminance (L*) values are in the range [0,100] Chrominance values are in the range [-500,500] (a*) and [-200,200] (b*) Need to remap chrominance differences to range [-α, α]

15 α: Chromatic variation maps to luminance variation α α crunch(x) = α * tanh(x/α) α = 5 α = 10 α = 25

16 Effect of α α = 5 α = 15 α = 25 α = 35 α = 45 α = 55 α = 65 α = 75 α = 85 α = 95

17 θ: Sign of chromatic difference Luminance Distance: ΔL ij = L i -L j Chrominance Distance: ΔC ij Problem: ΔC ij is unsigned

18 +b* C 2 -a* +a* C 1 Color Space b*

19 Map chromatic difference to increases or decreases in luminance values Original...

20 Color Difference Space +ΔC 1, Δb* v θ v θ = (cos θ, sin θ) -Δa* θ +Δa* - + -Δb* sign( ΔC i,j ) = sign(δc i,j. v θ )

21 Rasche et al. Photoshop Grayscale

22 Rasche et al. Photoshop Grayscale

23 θ = 45 Photoshop Grayscale θ = 225

24 θ = 135 θ = 45 θ = 0 Grayscale

25 Photoshop Grayscale

26 How to Combine Chrominance and Luminance ΔL ij if ΔL ij > crunch( ΔC ij ) δ(α,θ) ij = crunch( ΔC ij ) if ΔC ij. ν θ 0 crunch(- ΔC ij ) otherwise α... α crunch(x) = α * tanh(x/α)

27 Color2Grey Algorithm Optimization: min ΣΣ ( (g i -g j ) - δ i,j ) 2 i i+μ j=i-μ If δ ij = ΔL then ideal image is g Otherwise, selectively modulated by ΔC ij

28 Implementation Performance Image of size S x S O(μ 2 S 2 ) or O(S 4 ) for full neighborhood case 12.7s 100x100 image 65.6s 150x150 image 204.0s 200x200 image GPU implementation Athlon CPU O(S 2 ) ideal, really O(S 3 ) 2.8s 100x s 150x150 NVIDIA GeForce GT s 200x200

29 Results Original Photoshop Grey Color2Grey Color2Grey + Color

30 Original PhotoshopGrey Color2Grey

31 Original PhotoshopGrey Color2Grey+Color

32 Original PhotoshopGrey Color2Grey

33 Original PhotoshopGrey Color2Grey+Color

34 Original PhotoshopGrey Color2Grey

35 Original PhotoshopGrey Color2Grey+Color

36 Original PhotoshopGrey Color2Grey

37 Original PhotoshopGrey Color2Grey+Color

38 Original PhotoshopGrey Color2Grey

39 Original PhotoshopGrey Color2Grey+Color

40 Validate "Salience Preserving" Original PhotoshopGrey Color2Grey Apply Contrast Attention model by Ma and Zhang 2003

41 Validate "Salience Preserving" Original PhotoshopGrey Color2Grey

42 Color Harmonization Daniel Cohen-Or, Olga Sorkine, Ran Gal, Tommer Leyvand Tel Aviv University Ying-Qing Xu Microsoft Research Asia Slides by Cohen-Or et al.

43 What is color harmony? Harmonic colors are pleasing to the eye. They engage the human observer and give a sense of order and balance in the visual experience.

44 Coloring/recoloring recoloring

45 Harmonization technique i type V type L type I type T type Y type X type N type Hue histogram

46 Harmonization technique

47 Applications Facilitating the choice of harmonic colors automatic harmonization may be used instead of careful choice of color palettes

48 Applications Matching the colors coming from different sources (cut-and-paste)

49 Applications Matching the colors coming from different sources (cut-and-paste)

50 HSV color space Perceptually meaningful dimensions: Hue, Saturation, Value (Intensity)

51 Formal definition of color harmony? Mathematical formulation has been developing together with color theory Newton, Goethe, Young, Maxwell Itten [1960]: harmony means relationships on the hue wheel: 2-color harmony: complementary colors 3-color harmony: equilateral triangle N-color harmony: equilateral N-gon

52 Formal definition of color harmony? Matsuda [1995]: extensive empirical studies, derived 8 hue templates Tokumaru et al. [2002] developed a fuzzy system to evaluate the harmony of color schemes i type V type L type I type T type Y type X type N type

53 Harmonic scheme The templates can be arbitrarily rotated Harmonic scheme is template type T m + specific orientation α i type V type L type I type T type Y type X type N type

54 Harmony score To evaluate the harmony of an input image X we analyze its hue histogram: Every pixel p contributes its saturation S(p) to the bin of the hue H(p)

55 Harmony function The harmony of image X w. r. t. harmonic scheme (T m, α) : F( X,( T, )) m α = H( p) ETm( α )( p) S( p) p X arc length distance hue of p closest sector border saturation of p H(p) E Tm(α) (p)

56 Best template Automatically compute α that minimizes F(X, (T m, α)) for each template T m The best-fitting harmonic scheme: ( T,α ) = argmin F X,( T,α) m 0 0 ( m,α) ( ) m

57 Harmonization Given (T m, α) we shift the hues so that the hue histogram is contained in (T m, α)

58 Color shifting The hue of pixel p is shifted to its associated sector E Tm(α) (p) The amount of squeezing is controlled by a Gaussian fall-off function

59 Color coherency If we define E Tm(α) (p) simply as the closest template sector to H(p), we get coloring discontinuity: Graph-cut solution

60 Graph-cut optimization To make the coloring more coherent we assign E Tm(α) (p) by optimizing the labeling V E( V) = λ E ( V) + E ( V) 1 2 Favors short distance to the template sector Favors coherent labeling of neighboring pixels

61 Graph-cut optimization To make the coloring more coherent we assign E Tm(α) (p) by optimizing the labeling V E( V) = λ E ( V) + E ( V) E ( ) ( ) ( ( )) ( ) 1 V = H p H V p S p p 1 2 ( ) 1 2 = δ max { pq, } N E ( V) V( p), V( q) H( p) H( q) S ( p, q)

62 Overcoming segmentation problems The graph-cut may fail when an object in the image has several connected components

63 Overcoming segmentation problems User-assisted fix: scribbling on the erroneously labeled area Re-compute the labeling

64 Example result

65 Results choosing colors

66 Results cut and paste The background is harmonized according to the best-fitting harmonic template of the pasted foreground

67 Results cut and paste The background is harmonized according to the best-fitting harmonic template of the pasted foreground original harmonized harmonized

68 Results Text over a poster

69 Results Text over a poster

70 Results

71 Discussion Nature is already harmonic original best-fitting template poorly-fitting template

72 Discussion Cannot improve a masterpiece! Wassily Kandinsky, Composition VII, 1913

73 Discussion Grayish colors will remain such

74 Summary The proposed method enhances the harmony of colors in a given image Operates by fitting the image hues into a given harmonic distribution Especially useful for artificial colors, cutand-paste settings and collages that combine imagery from different sources

75 Example-based recoloring J.-F. Lalonde and A. Efros. Using Color Compatibility for Assessing Image Realism. ICCV 2007.