WEEK 5: BRIGHTNESS* PERCEPTION

Transcription

1 CN530 S CN530 S WEEK 5: BRIGHTNESS* PERCEPTION 1) Brightness perception: Quantifying percepts 2) Isomorphistic and nonisomorphistic theories 3) Craik-O Brien-Cornsweet (COCE) effect 4) Retinex algorithm 5) Brightness assimilation 6) Grossberg and Todorovic (T) implementation of BCS/FCS *We will also be talking about LIGHTNESS PERCEPTION. First, spend some time at interactive lightness demos at: Even a few moments at should convince you that there is a tremendous amount of complexity in the domain of brightness and lightness perception. To date, no model has given a comprehensive account of all known perceptual effects. We will focus initially on models that do a reasonable job on a subset of known effects, conceding limits on scope from the outset. Later, we will briefly describe some promising paths for extension of models. THE PLAN CN530 S ) T (1987) meta-analysis of models of brightness perception, especially with regard to motivation for filling-in 2) Cornsweet's analysis of the Craik-O'Brien-Cornsweet Effect 3) Land's Retinex theory early form: more computationally intuitive later form: more biologically plausible Note: We will be considering the achromatic analog of Land s (chromatic) Retinex theory. CN530 S MODELS OF BRIGHTNESS PERCEPTION (After T,1987) Styles of theories: Nonisomorphistic Cognitive Mechanistic (Filling-in and Integration theories) Goal: To describe magnitude, M, of brightness sensation as a function of luminance, L: M = f(l) What about f( )? Is it... linear,... a power function,...logarithmic?... (at least!) monotone? 4) Background for G & T (1988) simulations

2 The previous panel sidesteps a major issue: CN530 S What is the spatial layout of the region whose brightness magnitude we are to judge with respect to its surrounding regions? As we will later see, everything affects this judgment: Whether there is one surrounding region or many... Whether the region to be judged has the highest or lowest luminance, or some intermediate luminance... The figure-ground relationships in the scene... Slant in depth, corners in depth, curvature in depth... CN530 S Note: There is much confusion about the nature and even the number of dimensions of achromatic color experience: Brightness -- dim to bright vs.. Lightness -- black to white Can one have a dim white?... a bright black? Katz (1935) also includes pronouncedness and insistence as dimensions of achromatic color, specifically to get at distinctions such as white surface in bright light vs. white surface in dim light. For now, we will consider only the simplest spatial arrangements. CN530 S CN530 S WOULDN T IT BE NICE? BUT, HANG ON... Desired simplicity: Perceived magnitude is always some function of luminance. E.g. power law: M = kl n is often found when an observer is asked to report the subjective magnitude (e.g. brightness) associated with a test region, relative to some constant background ( magnitude estimation ). Compare last panel to Weber/Fechner paradigm for just noticeable differences (JNDs), where luminance of a target region and the background covary, yielding a logarithmic law, e.g. M = k log (L). Recall: I I = C In any case, we still desire, for some f, M = f(l). Is inverse of f well-defined? For that matter, is f well-defined?

3 CN530 S UNFULFILLED DESIRE: BACKGROUND EFFECTS a a b b In other words, (1) you are trying to find a function that relates the magnitude of a visual stimulus in some region (luminance, given by the height of the six central plateaus in the above plots) to the magnitude of some internal sensation. (2) The answer you get depends on the way you set up the backgrounds, as well as on the magnitude of the stimulus, in the sense that the shape of the resulting response function differs. c c CN530 S TROUBLE AHEAD, TROUBLE BEHIND 1) Many values of L can map to one value of M. (Consider brightness constancy! ) 2) One value of L can map to many values of M, as in Craik--O Brien--Cornsweet Effect. Bottom line: Power and log laws only work for (simple situations, such as s) target on a homogeneous background; to explain more requires tasting the fruit of the tree of knowledge. M L M M L M a L a b c d L b c d now what?? SAME L, DIFFERENT M CN530 S CRAIK-O'BRIEN-CORNSWEET EFFECT CN530 S All the diamonds in the same row as the one marked 1 have the same luminance as all the ones in the row marked 2. (!) long-range interactions in what we see in a region. After Todorovic, é 1987 Figure source: Logvinenko A D, 1999 "Lightness induction revisited" Perception

4 Consider COCE: CN530 S BRIGHTNESS AND SPATIAL FREQUENCY Nonisomorphistic account (e.g. Cornsweet): Illusion is based on attenuation of low spatial frequency information -- cf. MTF. e.g. COCE different M's Square wave is composed of sine waves at many spatial frequencies, and we are not very sensitive to the low spatial frequency components. same L's So: a luminance step and a luminance cusp are (nearly) functionally equivalent. (Cf. Cornsweet, p 346, Arend & Goldstein, p 67.) Bonus question: What would be a good direct test of this idea? CN530 S NONISOMORPHISTIC ACCOUNT OF COCE Luminance: Input amplitude X spatial frequency: CSF weighting Weighted input X spatial frequency Reconstructed profile (C calls this brightness!) CN530 S CRITIQUE OF NONISOMORPHISTIC ACCOUNT OF COCE A HIGHER STANDARD CN530 S Nonisomorphistic account: Through early visual processing: square wave: luminance step goes to neural cusp COCE: luminance cusp also goes to neural cusp. Q.E.D. Todorovic Don t just ask why they look alike, but also what they look like! Nonisomorphistic: Stimulus: Isomorphistic: Note: The transformation of C s account on the previous panel -- attenuation of low spatial frequencies -- can also be phrased in terms of the natural result of center-surround processing. Neural (early): Bridge locus (neural): Percept: Filling in (neural): Proponents of isomorphistic account argue that we also need an explanation for why any neural cusp looks like a brightness step. Note: Equal parts of early representation yield unequal parts of percept... HOW?

5 GREAT MINDS THINK ALIKE s c T (1987) shows that the configuration of vs luminance discontinuities ( edges ) can have c profound effects on brightness, even in places distant from the cusp. luminance CN530 S CLAIM: Both the step edges ( s ) and the cusp ( c ) are implicated in experiencing the COCE in the above configuration. NOTE: Cornsweet s account only deals with the cusp. Resulting conclusions: 1) A filling-in process is needed (to turn cusp into a step.) 2) Boundaries are needed to contain filling-in. (Sound familiar?) s PSYCHOPHYSICS IN THE FIRST CENTURY B.C. Note that the spinning disk apparatus -- the standard means of generating smoothly varying luminance profiles before the computer-controlled CRT -- guarantees that a closed region bounded entirely by a luminance cusp is generated! (from Cornsweet, 1970, p 273) Compare to the T (1987) configuration of COCE. intensity CN530 S position THE THIRD DIMENSION CN530 S RETINEX THEORY, I Recall the proposed strategy for discounting the illuminant: CN530 S same luminance profile in horizontal cross-section of the two figures See Dan Kersten s web demos Knill, D. C., & Kersten, D. (1991). Apparent surface curvature affects lightness perception. Nature, 351, I R I R First recover relative reflectances (ratios) near rapid (i.e., discontinuous) changes in image intensity, then reconstruct perceived color values in region interiors. unequal image intensities at region boundaries

6 CN530 S LAND! LAND! Besides the experiments with Mondrians, Land (1977) describes: 1) Take a black and white slide of a natural scene. 2) Take another slide of the same scene on (the same type of) monochromatic film, but insert a filter that admits only red (long wave) light in front of the camera lens. 3) Project both slides on a screen, superimposed, and in registration, but use normal (white) light to project the first slide and long wave light only to project the second slide. Paradoxical result: Many (veridical) colors (green, blue, yellow, orange*, etc.) besides red can be perceived. Such demonstrations formed foundations of Retinex premises. *The Stroop effect strikes again! RETINEX PROPOSITIONS In summary the three propositions of Retinex Theory [not to be confused with the early Retinex algorithm] are: CN530 S I. The composition of the light from an area in an image does not specify the color of that area. II. The color of a unit area is determined by a trio of numbers each computed on a single waveband to give for that waveband the relationship between the unit area and the rest of the unit areas in the scene. III. The trio of numbers, the three R Λ s, as computed by the Retinex algorithm, are the designators for the point in Retinex three-space which is the color of the unit area. (from Land, 1977) RETINEX ALGORITHM Summary of Retinex algorithm (older or early ) version) from Land, 1986a: CN530 S RETINEX EXAMPLE 1 CN530 S While Retinex was conceived in the chromatic domain, the achromatic Retinex algorithm is an obvious generalization. Note multiple paths Here Land describes a running product of those ratios sufficiently different from 1 -- product instead of sum, because the log transform was skipped.

7 CN530 S DISCOUNTING THE ILLUMINANT A LA RETINEX Example with shallow gradient of illumination: CN530 S Note: In the original caption to the two preceding figures, Land points out that the images used in the computation have far more than 18 X 18 pixels. The numerical values that are printed represent a sparse sampling of the actual image. Thus, a transition from one pixel to the next goes from 55.8 to The resetting of this ratio to 1.0, instead of before carrying out the running product calculation is the soul of the Retinex algorithm. NOTE: Retinex -- and many other models -- assume that changes in image intensity owing to illumination effects are more gradual than changes owing to physical reflectances. CN530 S SOURCES OF VARIATION IN IMAGE INTENSITY Simple computer graphics algorithms for rendering typically compute dot products of vectors normal to a surface patches with others pointing to the viewpoint or light source. Results are weighted by parameters that express the characteristics of reflectance (diffuse, specular, wavelength...) of a surface being modeled. Still difficult to model: effects of rough surface interflections and cast shadows; the latter require qualitatively different algorithms (e.g. ray tracing, radiosity,...) L N V CN530 S HOW PLAUSIBLE? Although Land (1977) describes the scanning algorithm as physiologically plausible (!), he later described a more biologically plausible, version of the Retinex algorithm, in which the designators are computed by simple filtering with kernels having very narrow ON centers, and very very broad OFF surrounds. (Land, 1986b) kernel: image Question: Look around you on a sunny day. Are the outlines of cast shadows qualitatively fuzzier than the outlines of objects?

8 HOMOPHOBIA CN530 S While the computational core of the (early, especially) Retinex algorithm is somewhat obscure, it can be viewed as a special case of homomorphic filtering*. (Oppenheim, 1967; see Ch. 12 of A.V. Oppenheim and R. W. Schafer, Discrete-time signal processing. Englewood Cliffs, NJ: Prentice Hall, 1989.) The Retinex recipe can be viewed as: 1) Filter the image at many scales (e.g. Fourier analysis) 2) Throw out the outputs of all the large filters ( slow-varying spatial components) 3) Deconvolve the remaining components (e.g. Fourier synthesis) Note similarity to Cornsweet s argument re: COCE! * With thanks to Mike Cohen. CN530 S INTERLUDE: DEFINITION OF ASSIMILATION Consider bars of three luminances on a background of a fourth (intermediate) luminance. Brightness assimilation: Brightness of medium luminance ( gray ) bars varies in the same direction as the brightness of the nearby bars; they become more similar ( assimilate ) to the perceived color of their neighbors than would be the case if presented alone. Effect is opposite in direction to that of brightness contrast. Is this effect just blurring of high-spatial frequency regions? low: medium: high: BRIGHTNESS: To contrast or to assimilate; that is the question! MORE ASSIMILATION CN530 S Once again, we encounter a word whose usage among researchers is inconsistent. CN530 S Assimilation is often used purely descriptively to refer to any color perception effect -- including achromatic color (lightness) or brightness -- in which a perceived color (A) is more similar to some nearby color (B) than would be expected if the part of the stimulus corresponding to (A) occurred alone. Others use the word assimilation to refer only to instances where the more similar color effect can be arguably traced to mixing of signals by early receptive fields, because the stimulus parts (A) and (B) are too close together -- often they are adjacent -- to be fully resolved.

9 CN530 S WHETHER TIS NOBLER... What determines whether the brightness of a visual entity (e.g., line, disk, blob, whatever) will contrast with the brightness of surrounding entities or assimilate toward their brightness?* The following claims have been advanced, at some point, by some people, about assimilation. NOTE: They may be incorrect! Claim 1: Perceptual objects contrast with other objects; parts of an object or objects in a group assimilate to each other. Claim 2: Contrast occurs between a figure and its ground; assimilation occurs among objects within a background. Claim 3: Contrast occurs within attended parts of a scene; unattended parts of a scene assimilate to each other. * Memo to Piers: This will remain a thermos question until an adequate theory of brightness is formulated. CN530 S FOOD FOR THOUGHT How does the visual system treat a simple bull s eye display? from Gilchrist 1994 Note: Most simple models (e.g., shunting network, Cornsweetstyle filtering theories, even G & T, 1988) implicitly treat a bull s eye as depicted on the left, while the visual system chooses the interpretation on the right. Bottom line: There may be no escaping the issues of figure/ground and depth, even for the simplest 2-D displays. CN530 S ALTERNATIVES TO RETINEX CN530 S Names from the classical lightness literature to consult with respect to the propositions of the previous panel include, but are not limited to: Beck, Coren, Hochberg, Kanizsa, Kardos If pursuing these questions, try to keep in mind that one does not have to leap all the way to anything goes cognitive explanations just because your simplest filter model cannot handles the relevant distinctions. An intermediate level of autonomous perceptual organization remains to be explored. The intuitions of Land s (1977) Retinex theory -- specifically, those concerning edges and relatively homogeneous regions - - could be developed in several ways. Consider: Diffusive filling-in as a mechanism for an isomorphistic account of brightness phenomena. Recall from Lecture 1: Filling-in is a fix for featural noise suppression (!) so we do not experience a world of colored edges. Issues: inherent seriality? one-shot alternative? Fischl/Schwartz Role of complementary representations -- boundaries and (color/brightness) features

10 CN530 S BCS and FCS CN530 S Note: The questions of whether we really need filling-in, of whether filling-in is performed by a diffusion-like process, of how to computationally speed up the computation of a diffusion-like process, and what are the real biological time delays involved in any filling-in process have received a great deal of attention in the published literature in recent years. The CN530 syllabus contains a great number of pointers to more comprehensive and more recent treatments of these and related questions than we will be able to examine in depth in the course. If you decide to pursue these questions, please make use of these pointers. Next, consider BCS/FCS interaction: Many aspects of BCS will be put off until the next weeks. For now: What are BCS signals for? How does the branching from MP and rejoining work? OBJECT RECOGNITION SYSTEM (ORS) e.g. ART variants Learns and remembers perceptual codes BOUNDARY CONTOUR SYSTEM (BCS) grouping completion sharpening FEATURE CONTOUR SYSTEM (FCS) filling-in of brightness, color MONOCULAR PREPROCESSING (MP) shunting center-surround network gain control (dynamic range) INPUT IMAGERY eye, or sensor data ICONOGRAPHY 101 CN530 S Recall the diagram on this panel and try to locate its themes in the following diagram (next panel): Hint: Not all of the themes are in the next panel! NO CHEATING! G & T (1988) macrocircuit: E CN530 S cooperation 2 F C 4 + D filling-in (diffusion?) Boundaries: Completion Surfaces: Filling-in oriented unoriented inward outward insensitive to sensitive to direction-of-contrast direction-of-contrast B A

11 KEY CN530 S FOR THOSE WHO INSIST CN530 S FCS Filling-in here F Concentrate on this step today. C E + D hypercolumn/blob interactions and/or V4 monocular vs. binocular theory Be careful! LGN? F 6 C 5 4?? + + E + D + + complex? B MP: 1 monocular preprocessing 2 3 A Simplified BCS No competition No long-range completion Next: Describe G & T algorithm in excruciating detail. B 2 3 A 1 retina? simple A few points about G & T, 88: CN530 S Q. It is neither the first nor the latest and greatest instantiation of a filling-in model. Why then study it? A. As a model it has a good granularity for exploring the transition from the highly abstract treatment of Grossberg s shunting networks (circa 70s and early 80s) to the more detailed local circuit models of today. That is, it s degree of complexity as a model is much higher without being out of reach for a first-semester course. Similarly, degree of fidelity to underlying physiology is also much greater (than that of the stand-alone shunting network) without being so detailed that you d need to spend half a year researching the latest physiology data just to understand it. Also, as research articles go, it is relatively tutorial in its exposition. All this said, G & T, 88 is not the best we can do for modeling lightness and brightness. Paths to further exploration are contained in the syllabus, in next week s lecture, and are also available via further discussion. LEVEL 1 Let I ij e {1,2,...9} (input quantized to nine levels) For image: To filter with Gaussian : extend the outermost pixel values. (This is not the only way to handle image boundary artifacts.) CN530 S

12 LEVEL 2 CN530 S C SUPPRESSION OR SURPRISE? CN530 S Build DOG kernels for distance-dependent shunting network with: B 2 A Shape of weighted gaussians used in 1-D simulations: C pqij = C exp{- α 2 log2[(p - i) 2 + (q - j) 2 ]} (A4) E pqij = E exp{- β 2 log2[(p - i) 2 + (q - j) 2 ]} (A5) Q: Why the log 2? Note: log 2 stands for a CONSTANT -- the log of 2; you are not meant to take the log of what s in the square brackets [ ]. excitatory inhibitory α < β C > E Do these parameters support featural noise suppression? How can you tell? Why did G & T use these parameters? ON AGAIN, OFF AGAIN CN530 S LEVEL 3 CN530 S Simple scheme for oriented detectors G & T define ON and OFF cells by: (k ) H pqij C pqij = E pqij (A8) E pqij = C pqij (A9) k =? m k nk Q1: How biologically accurate is this? k =? G pqij Q2: What happens to OFF-cell output in simulations? A self-similar (sic) basis for simple cells. Cf. Watson & Ahumada (1989) and others. NOTE: This scheme is also a DOG, but from shifted gaussians a.k.a. DOOG -- difference of offset gaussians

13 MODEL SIMPLE CELLS CN530 S LEVELS 4 AND 5 CN530 S ( k ) ( k ) F = G H ( A13) pqij pqij pqij G is at the center; H gets displaced, relative to orientation, k. Level 4: Complex cells -- via simple rectification zijk = Yijk + Yij [ k + ( K/ 2)] ( A19) Note that subscript of second Y should indicate mod K. G pqij = exp{- γ 2 [(p - i) 2 + (q - j) 2 ]} (A14) Compare γ to α, β of Level 2 gaussians... Level 5: Sum boundary signals across orientations at each position. Q: Does this step require a separate neural layer? For G & T s purposes, BCS structural scale << FCS (Level 2) structural scale. BOUNDARY-GATED DIFFUSION CN530 S BCS/FCS DIFFUSION CN530 S d dt S ij = MS ij + [ S pq S ij] P pqij + X ij ( A 22) P pqij Locate: (1) source, (2) sink, (3) variable diffusion coefficient, (4) density gradient (conductance) BCS output FCS diffusion centersurround (FCS) pq, N S ij F ij ij δ = 1 + ε Z + Z ( pq ij) Bij ( A23) Just average between boundaries. -- NOT!!! This system is nonconservative. (To be discussed further.) Aside: A closed form solution for the equilibrium of the G & T diffusion exists; it involves inversion of a large banded matrix. QUESTION 1: What s a syncytium? QUESTION 2 (Schwartz): Is diffusion plausible? timescale? Cf: anisotropic diffusion; geometry-driven diffusion, conductance-based diffusion (Malik and Perona, 1987,... )

14 SIMPLE EXAMPLE CN530 S BRIGHTNESS CONSTANCY CN530 S G & T, Fig 10a Functional scales: BCS peaks are spatially narrower than FCS peaks (and troughs). NOTE: This is a contingent statement about G & T simulations, NOT a deep truth about BCS/FCS (despite Retinex motivation!) G & T, Fig 10b: Note ratio-sensitive (!) edges in FCS for discounting the illuminant. Compare this simulation of brightness constancy to the UMAP non-distance-dependent network argument! CN530 S WATCH WHAT WE DO, NOT WHAT WE SAY A single scale simulation in G & T (1988) contains three important scales of interaction. (Cf. Orwell: War is peace. ) BCS: (smallest) responds with fine localization to input discontinuities; ignores homogeneous regions and shallow gradients. FCS, Level 2: For these simulations, responds somewhat similarly to BCS (aside from orientation and rectification!), but with significantly wider spatial kernels. Responds to somewhat lower spatial frequencies ; is more likely to exhibit interactions (e.g. peak shift, smoothing) in the presence of two nearby discontinuities. BCS/FCS interaction by diffusion, Level 6: Largest scale, at least potentially. More importantly, functional scale of diffusion is almost arbitrary, depending on layout over the entire scene of discontinuities registered by BCS and FCS. Recall: Structural scale of diffusion equation is given by nearest neighbor interactions! Bottom line: The power of the procedure comes from the flexibility afforded by the chameleon-like functional scale of the contextsensitive BCS/FCS interaction.