Σ S(x )δ x. x. Σ S(x )δ x ) x Σ F(S(x )δ x ), by superposition x Σ S(x )F(δ x ), by homogeneity x Σ S(x )I x. x. Σ S(x ) I x (y ) = x

Transcription

1 4. Vision: LST approach - Flicker, Spatial Frequency Channels I. Goals Stimulus representation System identification and prediction II. Linear, shift-invariant systems Linearity Homogeneity: F(aS) = af(s) Superposition: F(S + T) = F(S) + F(T) Shift-invariance Given stimulus S and system F, Define T( ) = S( + 0), S = F(S), T = F(T) Fisshift-invariant if T ( ) = S ( + 0) for all S, and 0 Delta functions Define δ to be an impulse at location S = Σ S( )δ. A weighted sum of basis functions δ. Note that I m using summation notation, although for continuous images an integration would be the correct notation. Let I = F(δ 0). This is called the impulse response of F. By shift-invariance, F(δ ) is a shifted version of I. Call it I. Thus, for arbitrary stimulus S we have F(S) = F( = = = Σ S( )δ ) Σ F(S( )δ ), by superposition Σ S( )F(δ ), by homogeneity Σ S( )I. Thus, the response is predictable for all stimuli given the impulse response. Each point in the stimulus generates its own replica of the impulse response weighted by the value of the stimulus at that point. These replicas sum to produce the complete response. At a given location y the value is: F(S)[y ] = Σ S( ) I (y ) = Σ S( ) I(y ) = S*I. This operation is called the convolution of S and I. Another way of thinking about this system is as a series of identical receptive fields R applied to the input: F(S)[y ] = Σ R( ) S(y + ) = Σ R( y )S( ). That is, R( ) = I( ), so this is an equivalent description. The receptive field is the

2 impulse response reflected about all the aes. Issue: Time and Causality The Fourier transform Sine wave gratings Frequency f Amplitude A Phase φ Orientation θ L(,y) = L 0 (1 + A cos(2πf(cosθ +ysinθ) φ)). Intuition Just like δ functions: (1) Any stimulus is a weighted sum of sine waves (2) Weights are determined by cross-correlating with the stimulus (3) Different frequencies capture different levels of detail (4) Orientation is just that (5) Phase is used to code location The details + F(ω) = f()e i2πω d f() = Notes: + F(ω)e i2πω dω i = 1 e i2πω = cos2πω + isin2πω, So: (1) if is in degrees, ω is in cycles/degree, (2) comple numbers are used to store the cosine and sine phases in the same comple number. Any other phase may be derived from these. (3) The rest is analogous to the δ function case. For n-dimensional case, change to and ω to ω. Eamples Sine wave grating Square wave grating Line Rotated line Edge Properties of the Fourier transform If f( ) transforms to F(ω ), (written: f F), Similarity theorem: f(a ) 1 F( ω ) a a Linearity: f + g F + G Shift theorem: f( a) e i2πaω F Convolution: f*g FG, fg F*G Power: f 2 d = F 2 dω Autocorrelation: f * (u)f(u + )du F 2, the power spectrum Derivative: f i2πωf.

3 The payoff, for linear, shift-invariant T with impulse response i: (1) T(f) = f * i (2) Thus, T(f) F I (3) This implies that T applied to a grating of frequency ω is another grating of frequency ω, only the amplitude may be scaled and the phase shifted by fied amounts (4) I is called the MTF of modulation transfer function of the linear filter T. More eamples Details III. Other representations Pulse sinc() = sin <=<3<3<. Sampling array, Sampling theorem, Nyquist frequency, aliasing One dimension: auditory signals, 1-d gratings, flicker Two dimensions: /y, /t Three dimensions: full motion analysis /y/t Low pass, band pass, high pass, orientationally tuned filters Optical MTF Point spread function Line spread function Flicker sensitivity as a linear filter followed by peak detection CSF or Contrast sensitivity function Window of visibility Convolution in statistics: sums of random variables Fourier series vs. Fourier transform Continuous vs. discrete transform/series Matlab issues: origin, fftshift, how to get power spectrum Fourier transform as orthogonal transform/change of basis/change of aes Local Fourier transforms Gaussian and Laplacian pyramids Gabor functions Sine wave times a Gaussian Transform is a shifted Gaussian Localization properties in space and spatial frequency DOGs (Difference of Gaussians) and DOOGs (Difference of Offset Gaussians) Wavelets and quadrature mirror filters (QMFs) The corte transform IV. Some applications Retinal sampling and aliasing (Yellott, Williams, et al.) Filtering and visual illusions (Ginsberg et al.) Discriminability models Watson: window of visibility Carlson & Klopfenstein A look ahead: spatial frequency channels versus the visual system as a (local) Fourier analyzer

4 V. Some provisos Nonisotropic - the oblique effect Nonisoplanatic - eccentricity effects/scaling Not constant with light level - adaptation is nonlinear Contrast nonlinearities

5 Convolution Eample Using the Impulse Response Let I(t) be the impulse response of the linear system F, i.e., I(t) is the output F(t) of the system when the input S(t) isaunit impulse. More complicated inputs result in an output that is the convolution of the input S(t) with the impulse response I(t). This can be thought of as the sum of copies of the impulse response starting at different times, and scaled by the value of the input at that time, as pictured below. Thus, for this view, we think of the input as a series of nonoverlapping impulses. Input S(t): Impulse response I(t): 1 I(t t 1 ): 2 I(t t 2 ): 3 I(t t 3 ): 8 I(t t 8 ): 9 I(t t 9 ): 14 I(t t 14 ): Σ 14 i I(t t i ): i=1 Note the change of vertical scale in the last graph. The response to the ith input impulse is denoted i I(t t i ) = I(t t i ) S(t i ) i,where t i is the time of the impulse, I is the shape of the impulse, and S(t i ) i is the size of the impulse. The output is the sum of these impulse responses, Σ 14 i I(t t i ) = Σ 14 I(t t i ) S(t i ) i F(t) = I(t t )S(t )dt = S(t t ) I(t )dt. i=1 i=1

6 Convolution versus Correlation Using the Receptive Field In another view, we flip the impulse response to create the system s receptive field R(t) = I( t). The output of the system at a given time is the output of the receptive field (RF), shifted to that time, correlated with the system input. that is, F(t) = I(t t )S(t )dt = R(t t)s(t )dt Note that the illustrated receptive field is causal, that is, the current output only depends on inputs from the past. Input S(t): RF 1 (t): RF 1 (t)s(t): RF 2 (t): RF 2 (t)s(t): RF 8 (t): RF 8 (t)s(t): RF 9 (t): RF 9 (t)s(t): F (t):

7

8

9

10

11 310 J. Opt. Soc. Am. A/Vol. 2, No. 2/February 1985 J. P. H. van Santen and G. Sperling c(, t) C(fw) bw0 A. He - ~~~~~~~? 0 He ~~~~~ dl; 0 I_ 3 e d E -~~~~~~~~~ 2 U, I I v -..~~~~~~~~ 2/ I Fig. 5. Eamples of two-dimensional functions considered in this paper and their Fourier transforms. For the functions c(, t), the horizontal dimension is, the vertical is t, and the shading indicates the value of c(, t). For the Fourier transforms C(f, w), the horizontal dimension is spatial frequency f; the vertical dimension is temporal frequency w; the shading indicates the magnitude of the comple number C(f, w). The one-dimensional motion interpretation (, t) rather than the two-dimensional space (, y) interpretation is used in the following descriptions. Functions (a) to (e) are periodic; one period of an infinite two-dimensional mosaic is shown. (a) Right-drifting sinusoid. (b) Right-drifting sinusoid with double the spatial frequency and double the temporal frequency of (a) and hence the same velocity. (c) Spatially uniform, stationary flicker. (d) Left-moving sinusoid. (e) A sampled (stroboscopic) motion display derived from (d). The motion path is ambiguous; three possible paths (1, 2, 3) and their transforms are shown. The contrast functions (f-j) are not periodic in but are spatially confined to the interval shown; functions (h-j) are also temporally confined. (f) Continuously present, stationary single line. (g) Continuously present, stationary random bar pattern. (h) A briefly flashed sinusoidal grating pattern. (i) A two-flash presentation of a random bar pattern moved to the right. U) A two-flash presentation of a sinusoidal grating pattern moved to the right.

12 Eercises (1) The diagrams below show an input to a (possibly linear) filter and the output of that filter. The problem is to characterize the filters. Fourteen different filters are described in the following figure. Choose the one that fits best for each filter, A 1 to A 19. Input Filter Output Input Filter Output A 1 A 11 A 2 A 12a A 12b A 3 A 13 sharp corners rounded corners A 4 A 14 A 5 A 15 A 6 A 16 A 7 A 17 A 8 A 18a A 18b A 9 A 19 A 10

13 Impulse Response Possibly linear, but none of the above 14. Nonlinear

14 (2) The hole-in-the-head peeker is a visually oriented animal. It often is used in theoretical studies of nerve recording because several of its brain nerves course out through holes in the head to where they can be recorded conveniently. In a recent study of h. peeker s response to a visual flicker stimulus (one-percent sine wave modulation), the maimum firing rate (ma) and the minimum firing rate (min) were recorded for each frequency. Below we show graphs of A(f) = ma(f) min(f) for nerves having peeker hold numbers of 71 and A(f) A(f) f (Hz) f (Hz) The Hungarian investigators Leseiw and Lebuh proposed that these nerves actually are in series, with 71 preceding 81. Assume they are correct, and assume that the visual system is approimately linear for these small modulations. (a) Draw the MTF of the brain system between 71 and 81. (b) Draw a possible impulse response. (c) What additional information do you need to specify the impulse response uniquely? (d) Does your solution provide evidence that 71 precedes 81, as opposed to vice versa? Eplain? (e) Draw the probable response that would be recorded at 71 and at 81 to the following stimuli: (1) Impulse: (2) Square wave: (3) Low frequency sine wave: (4) High frequency sine wave:

15 (3) An eperimenter measures the DeLange characteristic shown below (where M is the threshold contrast modulation; a value of M =1 means 100% contrast): log 10 (M) log 10 (f) (Hz) (a) Draw a block diagram of a complete system that would behave similarly. Label and define the properties of all the blocks. (b) What is the CFF (for 100% sine wave modulation) in Hz? (c) Draw a picture, approimately to scale, of a 1 Hz threshold stimulus and the CFF stimulus (amplitude vs time). (d) Draw a possible impulse response of the linear portion of your system. (e) The eperimenter superimposes increments of 1 second duration on a steady 10 Td background. What is the threshold amplitude of these increments (Trolands)? Eplain or show calculations. (f) Etend your diagram to model reaction times to step increments. (g) Suppose you are interested in the onset asynchrony required for the observer to discriminate a single impulse from a pair of impulses. Suppose your model is that the observer classifies the output as deriving from two input pulses if the output looks like two pulses. How do these judgments depend on the intensity and timing of a brief pair of impulses? (h) Comparer the variation of reaction times and of simultaneity judgments to impulses and to steps as a function of stimulus amplitude.

16 (4) A, B and C illustrate three systems. Each system is composed of three identical linear stages in series (the stages may differ from A to B to C, however). (a) The output of each stage is shown encircled to the right of each stage, and the impulse response of each stage is to be shown in the square representing that stage. Fill in the missing outputs and impulse responses. (b) (Creative...) Consider the three systems A, B and C as well as the following three stimuli: S 1 = 1 msec flash of light S 2 = 1 sec tone of frequency 100 Hz S 3 = a brief flash of a numeral Try to invent a psychological theory (i.e., a psychological of physiological process that corresponds to each of the boes - different boes may correspond to the same or to different processes) and an eperimental situation such that if one of the above stimuli were presented as the input impulse, the eperimental outcome would be represented by the output shown at the far right. Use each system and each output at least once. (Hint: consider reaction time.) A Same as A Same as A B Same as B Same as B Normal σ 2 = σ 2 = σ 2 = 75 C Same as C Same as C

17 (5) A theory of perceptual aftereffects is proposed to apply to the strength of such effects as the waterfall illusion or the McCollough effect. The aioms of the theory are vague (and use at least two meanings of the word effect ), like many psychological theories. They are: (i) A stimulus picture produces an increment in perceptual effect which is proportional to its contrast. (ii) For a given contrast and for very small durations (less than 1 sec) the effect produced is proportional to eposure duration. (iii) Effects of repeated eposures are additive. (iv) Effect decays eponentially with time constant τ = 10 sec. (a) Draw a graph of the magnitude of the effect as a function of time following a very brief eposure. (b) Same, for a prolonged eposure. (c) A clever eperimenter presents stimuli whose contrast fades out eponentially with time constant τ. Draw the approimate graph of the magnitude of the effect. (Etra credit, difficult: At what time is it maimum?) (d) A clever eperimenter modulates the contrast of the pictures viewed sinusoidally in time. Draw a graph of what the theory predicts for his/her results: (1) effect of viewing one stimulus, (2) effects of different modulation frequencies compared. Label (describe) coordinates as precisely as you can. (e) The cleverest eperimenter (self-appraised) collects data which show that the optimum interval between two equal eposures for a maimum effect is zero, i.e., any delay reduces the effect. Does this result disprove the theory? (f) If yes to (e), what should the result have been? If no to (e), invent a simple eperimental result to (c), (d), or (e) to powerfully disprove the theory. (6) Using any convenient package that can compute Fourier transforms, multiply images, etc. (e.g. HIPS, MATLAB or MATHEMATICA), try your hand at any of the following (your choice, but please do at least one): (a) Take two natural images A and B. Produce an image that is based on the phase of the Fourier coefficients of image A and the magnitude of the Fourier coefficients of image B. What happened, and why? (b) Produce and eercise operators that do the following: (1) enhance vertical edges, (2) enhance horizontal edges, (3) blur isotropically, (4) enhance high frequencies (approimately) isotropically. (c) Take a natural image. Subsample the image by setting all piel values whose and y coordinates are not multiples of, say, 4 to zero. Compute the Fourier transform of the original image and of the subsampled image. Eplain the results.