A Neural Model of the Scintillating Grid Illusion: Disinhibition and Self-Inhibition in Early Vision

Transcription

1 LETTER Communicated by Sidney Lehky A Neural Model of the Scintillating Grid Illusion: Disinhibition and Self-Inhibition in Early Vision Yingwei Yu yingwei@tamu.edu Yoonsuck Choe choe@tamu.edu Department of Computer Science, Texas A&M University, College Station, Texas , U.S.A A stationary display of white discs positioned on intersecting gray bars on a dark background gives rise to a striking scintillating effect the scintillating grid illusion. The spatial and temporal properties of the illusion are well known, but a neuronal-level explanation of the mechanism has not been fully investigated. Motivated by the neurophysiology of the Limulus retina, we propose disinhibition and self-inhibition as possible neural mechanisms that may give rise to the illusion. In this letter, a spatiotemporal model of the early visual pathway is derived that explicitly accounts for these two mechanisms. The model successfully predicted the change of strength in the illusion under various stimulus conditions, indicating that low-level mechanisms may well explain the scintillating effect in the illusion. 1 Introduction The scintillating grid illusion consists of bright discs superimposed on intersections of orthogonal gray bars on a dark background (see Figure 1A; Schrauf, Lingelbach, & Wist, 1997). In this illusion, illusory dark spots are perceived as scintillating within the white discs. Several important properties of the illusion have been discovered and reported inrecent years. (1) The discs that are closer to a fixation show less scintillation (Schrauf et al., 1997), which might be due to the fact that receptive fields in the periphery are larger than those in the fovea. As shown in Figure 2, if the periphery of the scintillating grid is correspondingly scaled up, the scintillation effect is diminished. Note that the diminishing effect is not due to the polar arrangement alone, as can be seen in Figure 1B. (2) The illusion is greatly reduced or even abolished both with steady fixation and by reducing the contrast between the constituent grid elements (Schrauf et al., 1997). (3) As speed of motion is increased (either efferent eye movement or afferent grid movement), the strength of scintillation decreased (Schrauf, Wist, & Ehrenstein, 2). (4) The presentation duration of the grid also plays a role Neural Computation 18, (26) C 26 Massachusetts Institute of Technology

2 522 Y. Yu and Y. Choe A B Scintillating grid Polar arrangement Figure 1: The scintillating grid illusion and its polar variation. (A) The original scintillating grid illusion is shown (redrawn from Schrauf et al., 1997). (B) A polar variation of the illusion. The scintillating effectis still strong in the polar arrangement (cf. Kitaoka, 23). in determining the strength of illusion. The strength first increases when the presentation time is less than about 22 ms, but it slowly decreases once the presentation duration is extended beyond that (Schrauf et al., 2). What kind of neural process may be responsible for such a dynamic illusion? The scintillating grid can be seen as a variation of the Hermann grid illusion where the brightness level of the intersecting bars is reduced (a representative example of simultaneous contrast; Gerrits, & Vendrik, 197). The illusory dark spots in the Hermann grid can be explained by feedforward lateral inhibition mechanism, commonly modeled with difference of gaussian (DOG) filters (Spillmann, 1994). Thus, DOG filters may seem like a plausible mechanism contributing to the scintillating grid illusion. However, they are not exactly fit to explain the complexities of the scintillating grid illusion because of the following reasons. (1) The DOG model cannot account for the change in the strength of scintillation over different brightness and contrast conditions, as shown in the experiments by Schrauf et al., 1997). (2) Furthermore, DOG cannot explain the basic scintillation effect, which has a temporal dimension to it. Thus, the feedforward lateral mechanism represented by DOG fails to fully explain the scintillating effect. Anatomical and physiological observations show that the centersurround property in early visual processing may not be strictly feedforward: the process involves a recurrent inhibition, which leads to disinhibition. Moreover, the process also includes self-inhibition the inhibition of the cell itself. For example, Hartline et al. used Limulus (horseshoe crab) optical cells to demonstrate disinhibition and self-inhibition in the retina

3 A Neural Model of Scintillating Grid Illusion 523 Figure 2: A variation without the scintillating effect. The grids toward the periphery are significantly scaled up, which results in the abolishment of the scintillating effect when stared at in the middle (see Raninen & Rovamo, 1987, for a similar input scaling approach to alter perceptual effects). This is because the scintillating grid illusion highly depends on the size of the receptive fields. In the fovea, the receptive field size is small, and in the periphery, the receptive field size is relatively larger. (Note that Kitaoka, 23, presented a similar plot, but there, the periphery was not significantly scaled up such that the scintillating effect was preserved.) (Hartline & Ratliff, 1957). Disinhibition and self-inhibition have been discovered in mammals and other vertebrates as well. As for disinhibition, it has been found in the retina of cats (Li et al., 1992; Kolb & Nelson, 1993), tiger salamanders (Roska, Nemeth, & Werblin, 1998), and mice (Frech, Perez- Leon, Wassle, & Backus, 21). For example, Kolb and Nelson (1993) have shown that the A2 amacrine cells in the cat retina contribute to lateral inhibition among ganglion cells, and they can play a role in disinhibition. With regard to self-inhibition, Hartveit (1999) found that depolarization of a rod bipolar cell in rat retina evokes a feedback response to the same cell, thus indicating that a mechanism similar to those in the Limulus may exist in mammalian vision. Also, Nirenberg and Meister (1997) have shown that transient and motion-sensitive responses in ganglion cells may be produced by self-inhibitory feedback of the amacrine cells in mouse retina (for similar results, see Berry, Brivanlou, Jordan, & Meister, 1999; Victor, 1987; Crevier & Meister, 1998). Computational models also suggested that self-inhibition

4 524 Y. Yu and Y. Choe may exist in cells sensitive to light-dark contrast (Neumann, Pessoa, & Hanse, 1999). Disinhibition can effectively reduce the amount of inhibition where there is a large area of bright input, and self-inhibitioncan give rise to oscillations in the response over time. Thus, the combination of those two mechanisms, disinhibition and self-inhibition, may provide an explanation for the intriguing scintillating grid illusion. In this letter, we present a model based on disinhibition and selfinhibition in the early visual pathway to explain the scintillating grid and its various spatiotemporal properties reported by Schrauf et al. (1997, 2). Our model is, to our knowledge, the first attempt at computationally modeling the scintillating grid illusion at a neuronal level. In the next section, we begin with a brief review of disinhibition and self-inhibition and introduce our model in detail. Next we present the main results, followed by discussion and conclusion. 2 Disinhibition and Self-Inhibition in Early Vision In this section, we review the basic mechanism of disinhibition and selfinhibition in the Limulus optical cells. The study by Hartline and Ratliff (1957) was the first to show that lateral inhibition exists at the very first stage in visual processing between optical cells in the Limulus. Furthermore, they showed that the lateral interaction has a nontrivial ripple effect. That is, the final response of a specific cell can be considered as the overall effect of the response from itself and from all other cells directly or indirectly connected to that cell. As a result, the net response of a cell can be enhanced or reduced due to such an inhibitory interaction depending on the surrounding context. For example, inhibition of an inhibitory neuron X will release the target of X from inhibition, thus allowing the latter to fire (or increase its firing rate). This process is known as disinhibition, and it has been shown that such a mechanism may be more accurate than lateral inhibition alone in explaining subtle brightnesscontrast effects (see, e.g., Yu, Yamauchi, & Choe 24). Self-inhibition is also found in Limulus optical cells (Ratliff, Hartline, & Miller, 1963). When a depolarizing step input is applied to the cell, a transient peak in firing rate can be observed, which is followed by a rapid decay to a steady rate. The self-inhibition effect is due to synaptic interactions, which produce a negative feedback into the cell itself. The self-inhibition mechanism is illustrated by the process in which cells go from an initial transient peak to a stable equilibrium plateau, which is a form of neural adaptation: each impulse discharge acts recurrently to delay the discharge of the next impulse within the same neuron (Hartline, Wager, & Ratliff, 1956; Hartline & Ratliff, 1957). Together with the feedback to neighboring cells, the feedback to oneself may be essential for explaining the evolution of the temporal dynamics observed in the scintillating grid illusion.

5 A Neural Model of Scintillating Grid Illusion A Spatiotemporal Model of Disinhibition and Self-Inhibition Hartline and his colleagues developed an early computational model of the response in the Limulus retina. The Hartline-Ratliff equation describing disinhibition in Limulus can be summarized as follows (Hartline & Ratliff, 1957, 1958; Stevens, 1964), r m = ɛ m K s r m n m v mn (r n t mn ), (3.1) where r m is the final response of the mth ommatidium, K s its self-inhibition rate, ɛ m itsexcitation level, v mn the inhibitory weight from another ommatidium n, andt mn its threshold. When equation 3.1 is used to calculate the evolution of responses in a network of cells, the effect of disinhibition arises. Brodie, Knight, and Ratliff (1978) extended this equation to derive a spatio-temporal filter, where the input was assumed to be a sinusoidal grating. The reason for limiting the input in such a way was to make tractable the explicit calculation of the responses. As a result, the derived model was applicable only to sinusoidal grating inputs. In addition to model selfinhibition, which gives the temporal property, they replaced the constant self-inhibition rate in equation 3.1 with a time-dependent term (cf. K s (t) in section 3.1). Their model is perfect in accounting for the responses in Limulus retina, but it is limited to a single spatial frequency channel input. Because of this, their extended model cannot be applied to complex images containing a broader band of spatial frequency, which is typically the case for visual illusions such as the scintillating grid illusion. In the following section, we extend the Hartline-Ratliff equation using a different strategy to derive a filter that can address these issues, while remaining tractable. 3.1 A Simplified Model in One Dimension. Rearranging equation 3.1 by omitting the threshold, we have (1 + Ks) r m w mn r n = ɛ m, (3.2) where w mn is the strength of interaction (either excitatory or inhibitory) from cell m to n.note that w mn extends the definition of v mn in equation 3.1 to allow excitatory as well as inhibitory contributions. To generalize the model to n inputs, the responses of n cells can be expressed in matrix form as (Yu et al., 24), Kr = e, (3.3)

6 526 Y. Yu and Y. Choe where r is the output vector of size n, e the input vector of size n,andk the n n weight matrix: 1 + K s (t) w(1).. w(n 1) w(1) 1 + K s (t).. w(n 2) K = , (3.4) w(n 1) K s (t) where K s (t) is the self-inhibition rate at time t, andw( ) is the connection weight, which is a function of the distance between the cells. (Note that unlike in our previous models (Yu et al., 24; Yu & Choe, 24a) the introductionof the time-varying term K s (t) allows for the model to have a temporal behavior.) For the convenience of calculation, we assume K s (t) here approximately equals the self-inhibition rate of a single cell. The exact derivation of K s (t) is as follows (Brodie et al., 1978): K s (t) = y(t) r(t), (3.5) where a y(t) is the amount of self-inhibition at time t, andr(t) is the response at time t for this cell. We know that the Laplace transform y(s) ofy(t) has the following property: y(s) = r(s)t s (s), T s (s) = k 1 + sτ, (3.6) (3.7) where k is the maximum value K s (t) can reach and τ the time constant. By assuming that the input e(t) is a step input for this cell, the Laplace transform of e(t) can be written as e(s) = I s, (3.8) where I is a constant representing the strength of the light stimulus. From the definition of y(t), we know that dr dt = e(t) y(t). (3.9)

7 A Neural Model of Scintillating Grid Illusion 527 To solve the response r(t), we can apply Laplace transform and plug in e(s) and y(s): r(s) = ( I s r(s) k 1 + sτ Solving this equation, we get r(s) = I s ) 1 s. (3.1) sτ + 1 τs 2 + s + k. (3.11) By substituting r(s) andt(s) in equation 3.6 with equations 3.7 and 3.11, we get y(s) = I s (sτ + 1) k (τs 2 + s + k) (1 + sτ). (3.12) Then, by inverse Laplace transform, we can get y(t) andr(t). Finally, the exact expression for K s (t) can be obtained by evaluating equation 3.5: K s (t) = 4kτ 1 + (1 4k)h(t)cos(ωt) 2kh(t)ω sin(ωt) 4kτ 1 + (1 4k)h(t)cos(ωt) + (4kτ 2)h(t)ω sin(ωt), (3.13) where ( h(t) = τ exp 1 ), (3.14) 2τt and 4kτ 1 ω =. 2τ (3.15) An intuitive way of understanding the above expression is to see it as a division of two convolutions, K s (t) = e(t) g(t) e(t) f (t), (3.16) where g(t) and f (t) are impulse response functions derived from the above and the convolution operator (see the appendix for details). Figure 3 shows several curves plotting the self-inhibition rate under different parameter conditions. As discovered in Limulus by Hartline and Ratliff (1957, 1958), self-inhibition is strong (k = 3), while lateral contribution is weak

8 528 Y. Yu and Y. Choe K s (t) k=3, τ=.3 k=3, τ=.1 k=2, τ=1. k=2, τ= time: t Figure 3: Self-inhibition rate. The evolution of the self-inhibition rate K s (t) (y-axis) over time (x-axis) is shown for various parameter configurations (see equations 3.5 to 3.7). The parameter k defines the peak value of the curve, and τ defines how quickly the curve converges to a steady state. For all computational simulations in this article, the values k = 3andτ =.3 wereused. (.1 or less). Hartline and Ratliff (1957, 1958) experimentally determined these values, whereas they left τ as a free parameter. Now we have a model for the response of cells arranged in one dimension, but for realistic visual inputs, we need a 2D model. In the following section, we provide details about extending the 1D model above to 2D. 3.2 Extending the Model to Two Dimensions. The 1D model can be easily extended to two dimensions. We simply serialize the input and output matrices to vectors to fit in the 1D model we have. The weight matrix K can then be defined to represent the weight K ij from cell j to cell i based on their distance in 2D, following the DOG distribution (Marr & Hildreth, 198): w( i, j ) when i j K ij =, (3.17) 1 + K s (t) when i = j w(x) = k c exp ( x2 σ 2 c ) k s exp ( x2 σ 2 s ), (3.18) where i, j is the Euclidean distance between cell i and j; k c and k s are the scaling constants that determine the relative scale of the excitatory and inhibitory distributions, set to 1/( 2πσ c ) and 1/( 2πσ s ); and σ c and σ s their

9 A Neural Model of Scintillating Grid Illusion 529 widths. The σ values were indirectly specified, as a fraction of the receptive field size ρ: σ c = ρ/24 and σ s = ρ/6. Finally, the response vector r can be derived from equation 3.3 as follows (Yu et al., 24): r = K 1 e, (3.19) and we can apply reverse serialization to get the vector r back into a 2D matrix form. Figure 4 shows a single row of the weight matrix K, corresponding to a weight matrix (when reverse serialized) of a cell in the center of the 2D retina, at various time points. The plot shows that the cell in the center can be influenced by the inputs from locations far away, outside its classical receptive field area. The initial state shown in Figure 4A looks bumpy (and somewhat random), but on closer inspection we can observe concentric rings of ridges as in Figure 4B. (The apparent bumpiness along the ridges is due to the aliasing effect caused by the square boundary of the receptive field.) Another noticeable feature is that the spatial extent of excitation and inhibition shrinks over time (from Figure 4A to 4F). This may seem inconsistent with the notion of persisting inhibitory surround, but in fact the spatiotemporal property of on-off receptive fields shows such a diminishing lateral influence over time (in retinal ganglion cells, Jacobs & Werblin, 1998; and also in the lateral geniculate nucleus, Cai, DeAngelis, & Freeman, 1997). 4 Methods To match the behavior of the model to psychophysical data, we need to measure the degree of the illusory effect in the scintillating grid. Measuring the strength of the overall scintillation effect is difficult because it involves many potential factors, such as the change in the perceived brightness of the discs over time and the perceived number of scintillating dark spots. (In fact, in Schrauf et al., 1997, 2, subjects were simply asked to report the strength of illusion on a scale of 1 to 5 without any reference to these various factors.) For our computational experiments, one simple yet meaningful measure of the strength of scintillation can be derived based on the changes in the perceived contrast. More specifically, we are interested in the change over time in the relative contrast of the disc versus the gray bars: S(t) = C(t) C(), (4.1) where S(t) is the perceived strength t time units from the last eye movement or the time of initial presentation of the scintillating grid stimulus (time t in

10 53 Y. Yu and Y. Choe A B t=.7 t=.31 C D t=.62 t=.124 E F t=.2452 t=.895 Figure 4: Disinhibition filter at various time points. The filter (i.e., the connection weights) of the central optical cell shown at different time steps (k = 3,τ =.3,ρ = 2). The self-inhibition rate evolved over time as follows: (A) K s (t) =.3, (B) K s (t) =.15, (C) K s (t) =.29, (D) K s (t) =.54, (E) K s (t) =.99, and (F) K s (t) = Initially, a large ripple extends over a long distance from the center (beyond the classical receptive field), but as time goes on, the longrange influence diminishes. In other words, the effective receptive field size is reduced over time due to the change in self-inhibition rate. (Note that the visual field size is shown above is for better visualization, compared to 3 3 in the actual experiments.)

11 A Neural Model of Scintillating Grid Illusion 531 our model is on an arbitrary scale) and C(t) is the contrast between the disc and the gray bars in the center row of the response matrix: C(t) = R disc(t) R min (t) R bar (t) R min (t), (4.2) where R disc (t) is the response at the center of the disc region, R bar (t) the response at the center of either of the gray bar regions, and R min (t) the minimum response in the output at time t. In other words, the function of perceived strength of illusion S(t) is defined as the relative disc-to-bar contrast at time t as compared to its initial value at time. Using this measure, in the experiments below, we tested our model under various experimental conditions, mirroring those in Schrauf et al. (1997, 2). In all calculations, the effect of illusion was measured on an image consisting of a single isolated grid element of size 3 3 pixels. The disc at the center had a diameter of 8, and the bars had a width of 6. The model parameters k = 3andτ =.3 were fixed throughout all experiments, and so was the pattern where the background luminance was set to 1, the gray bar to 5, and the white disc to 1, unless stated otherwise. Dependent on the experimental condition under consideration, the model parameters (receptive field size ρ) and/or the stimulus conditions (such as the duration of exposure to the stimulus and/or brightness of different components of the grid) were varied. The units of the receptive field size, the width of the bar, and the diameter of the disc were all equivalent in pixels on the receptor surface, where each pixel corresponds to one photo receptor. The details of the variations are provided in section 5. 5 Experiments and Results 5.1 Experiment 1: Perceived Brightness as a Function of Receptive Field Size. In the scintillating grid illusion, the scintillating effect is most strongly present in the periphery of the visual field. As we stated earlier, this may be due to the fact that the receptive field size is larger in the periphery than in the fovea, thus matching the scale of the grid. If there is a mismatch in the scale of the grid and the receptive field size, the illusory dark spot would not appear. For example, in Figure 2, the input is scaled up in the periphery, thus creating a mismatch between the peripheral receptive field sizeand the scale of the grid. As a result, the scintillating effect is abolished. Conversely, if the receptive field size is reduced in size with no change to the input, the perceived scintillation would diminish (as it happens in the center of gaze in the original scintillating grid; see Figure 1). To verify this point, we tested our model with different receptive field sizes while the input grid size was fixed. As shown in Figure 5A, smaller receptive fields result in almost no darkening effect in the white disc.

12 532 Y. Yu and Y. Choe A B C D E ρ =3 ρ=6 ρ=9 ρ=12 ρ=15 F G C(t=.1) 7 k=3, τ= Receptive field size: ρ Normalized response ρ=3 ρ=6 ρ=9 ρ=12 ρ= Position Figure 5: Response under various receptive field sizes. The response of our model to a single grid element in the scintillating grid is shown, under various receptive field sizes at t =.1. (A E) The responses of the model are shown when the receptive field size was increased from ρ = 3 to 6, 9, 12, and 15. Initially, the disc in the center is bright (simulating the fovea), but as ρ increases, it becomes darker (simulating the periphery). (F) The relative brightness level of the central disc compared to the gray bar C(t) is shown (see equation 4.2). The contrast decreases as ρ increases, indicating that the disc in the center becomes relatively darker than the gray bar region. The contrast drops abruptly until around ρ = 6 and then gradually decreases. (G) The normalized responses of the horizontal cross section of A to F are shown. For normalization, the darkest part and the gray bar region of the horizontal cross-section were scaled between. and 1.. When ρ is small (=3), the disc in the center is very bright (the plateau in the middle in the black ribbon), but it becomes dark relative to the gray bars as ρ increases (white ribbon). As the receptive field size grows, the dark spot becomes more prominent (see Figures 5B to 5E). The cross-sections of Figures 5A to 5E are shown in figure 5G. Figure 5F shows the bar-disc contrast C(t) over different receptive field sizes (ρ), where a sudden drop in contrast can be observed around ρ = 6. Note that at this point, C(t) is still above 1., suggesting that the disc in the center is brighter than the gray bars. However, C(t) is not an absolute measure of perceived brightness since it does not count in the vividly bright halo around the disc (already visible in Figure 5B). Thus, our interpretation that low C(t) (close to 1. or below) means a perceived dark spot may be reasonable.

13 A Neural Model of Scintillating Grid Illusion 533 In sum, these results could be an explanation as to why there is no scintillating effect in Figure 2. In the original configuration, the peripheral receptive fields were large enough to give rise to the dark spot; however, in the new configuration, they are not large enough, and thus no dark spot can be perceived. 5.2 Experiment 2: Perceived Brightness as a Function of Time. In this experiment, the response of the model at different time steps was measured. A B C D E t=.1 t=.1 t=.8 t=1.6 t=1. F C(t) G k=3, τ=.3 5 Normalized Response t=1 t=1.6 t=.8 t=.1 t= time: t Position Figure 6: Response at various time points. The response of the model to an isolated scintillating grid element is shown over time. The parameters used for this simulation were: receptive field size = 6 (represents the periphery), k = 3, and τ =.3. The plots demonstrate a single blinking effect of the white disc. (A) In the beginning when the self-inhibition rate is small, the illusory dark spot can be seen in the central disc (K s (t) =.495). (B E) As time goes on, the illusory dark spot disappears as the self-inhibition rate increases: K s (t) =.4521, K s (t) = 2.417, K s (t) = 3.214, and K s (t) = 3, respectively. (F) The relative brightness level of the central disc compared to the gray bar C(t) is shown (see equation 4.2). The results demonstrate an increase in the relative perceived brightness of the center disc as time progresses. (G) The normalized responses of the horizontal cross-section of A to E are shown. Normalization was done as described in Figure 5G. In the beginning (t =.1), the disc region in the middle is almost level with the flanking gray bar region (white ribbon near the bottom). However, as time goes on, the plateau in the middle rises, signifying that the disc in the center is becoming perceived as brighter.

14 534 Y. Yu and Y. Choe In Figures 6A to 6E, five snapshots are shown. In the beginning, the dark spot can clearly be observed in the center of the disc, but as time goes on, it gradually becomes brighter. Figure 6F plots the relative brightness of the disc compared to the bars as a function of time, which shows a rapid increase to a steady state. Such a transition from dark to bright corresponds to a single scintillation. (Note that the opposite effect, bright to dark, is achieved by refreshing of the neurons via saccades.) Figure 6G shows the actual response level in a horizontal cross-section of the response matrix shown in Figures 6A to 6E. Initially, the response to the disc area shown as the sunken plateau in the middle is relatively low compared to that to the gray bars represented by the flanking areas (bottom trace, white ribbon). However, as time passes by, the difference in response between the two areas dramatically increases (top trace, black ribbon). Again, the results show a nice transition from a perception of a dark spot to that of a bright disc. 5.3 Experiment 3: Strength of Scintillation as a Function of Luminance. The strength in perceived illusion can be affected by changes in the luminance of the constituent parts of the scintillating grid, such as the gray bar, disc, and the dark background (see figures 7A and 7C; Schrauf et al., 1997). Figures 7B and 7D show a variation in response in our model under such stimulus conditions. Our results show a close similarity to the experimental results by Schrauf et al. (1997). As the luminance of the gray bar increases,the strength of illusion increases, but after reaching about 4% of the disc brightness, the strength gradually declines (see Figure 7B), consistent with experimental results (see Figures 7A). Such a decrease is due to disinhibition, which cannot be explained by DOG (Yu et al., 24). When the luminance of the disc was increased, the model (see Figure 7D, right) demonstrated a similar increase in the scintillating effect as in the human experiment (see Figure 7C, right). When the disc has a luminance lower than the bar, a Hermann grid illusion occurs (Schrauf et al., 1997). Both the human data (see Figure 7C, left) and the model results (see Figure 7D, left) showed an increase in the Hermann grid effect when the disc became darker. Note that disinhibition plays an important role here, especially for the bar luminance experiments (see Figures 7A and 7B). In standard DOG, which lacks the recurrent inhibitory interaction, the illusory effect will monotonically increase with the increase in the luminance of the gray bars. However, with disinhibition, the increasing illusory effect will reach a critical point followed by a decline. (See section 6 for details.) 5.4 Experiment 4: Strength of Scintillation as a Function of Motion Speed and Presentation Duration. As we have seen above, the scintillating effect has both a spatial and atemporal component. Combining these two may give rise to a more complex effect. Schrauf et al. (2) demonstrated that such an effect in fact exists. They conducted experiments under

15 A Neural Model of Scintillating Grid Illusion 535 A Mean rated strength Human Bar luminance (cd/m 2 ) Schrauf et al. (1997) B Strength of scintillation: S Model Bar brightness in input Model C Mean rated strength Hermann (Human) Scintillating (Human) Disc luminance (cd/m 2 ) Schrauf et al. (1997) D Strength of scintillation: S Hermann (Model) Scintillating (Model) Disc brightness in input Model Figure 7: Strength of scintillation under various luminance conditions. (A) Mean rated strength of scintillation in human experimentsis shown as a function of disc luminance (Schrauf et al., 1997). (B) Scintillation effects in the model shown as a function of bar luminance. (C) Mean rated strength of scintillation in human experimentsis plotted as a function of bar luminance (Schrauf et al., 1997). The plot shows results from two separate experiments: the Hermann grid on the left and the scintillating grid on the right. (D) The Hermann grid and scintillation effects in the model are plotted as functions of disc luminance. Under both conditions, the model results closely resemble those in human experiments. For B and D, the strength of the scintillation effectin the model was calculated as S = C( ) C(), where C( ) is the steady-state value of C(t) (see equation 4.1). The illusion strength in the Hermann grid portion in D was calculated as S = 1/C( ) 1. The reciprocal was used because in the Hermann grid, the intersection is darker than the bars, whereas in the scintillating grid, it is the other way around (the disc is brighter than the bars). three conditions: (1) smooth pursuit movements executed across a stationary grid (efferent condition), (2) grid motion at an equivalent speed while the eyes are held stationary (afferent condition), and (3) brief exposure of a stationary grid while the eyes remained stationary. For conditions 1 and 2,

16 536 Y. Yu and Y. Choe A B Mean rated strength 4.5 Human Speed of motion (deg/s) Schrauf et al. (2) Strength of scintillation: S.55 Model Speed of motion (pixel/s) Model C Mean rated strength 4.5 Human Presentation duration (s) Schrauf et al. (2) D Strength of scintillation: S Model Model Presentation duration: t 4 Figure 8: Strength of scintillation under varying speed and presentation duration. (A) Mean rated strength of the illusion as a function of the speed of stimulus movement (Schrauf et al., 2). (B) Scintillation effect as a function of the speed of motion (v) in the model is shown. The receptive field size was 6, and the strength of scintillation was calculated as S(v 1 ) = C(v 1 ) C(). (C) Mean rated strength of the illusion as a function of the duration of exposure (Schrauf et al., 2). (D) Scintillation effect as a function of presentation duration (t) in the model. The receptive field size was 6, and the strength of scintillation was computed as S(t) = C(t) C(). In both cases (A B and C D) the curves show a very similar trend. both afferent and efferent motion produced very similar results: the strength of scintillation gradually decreased as the speed of motion increased (see Figure 8A). For condition 3, the strength of illusion abruptly increases, coming to a peak at around 2 ms, and then slowly decreases (see Figure 8C). We tested our model under these conditions to verify if temporal dynamics induced by self-inhibition can accurately account for the experimental results. First, we tested the model when either the input or the eye was moving (assuming that conditions 1 and 2 above are equivalent). In our experiments,

17 A Neural Model of Scintillating Grid Illusion 537 instead of directly moving the stimulus, we estimated the effect of motion in the following manner. Let v be the corresponding speed of motion, either afferent or efferent. From this, we can calculate the amount of time elapsed before the stimulus (or the eye) moves on to a new location. For a unit distance, the elapsed time t is simply an inverse function of motion speed v; thus, the effect of illusion can be calculated as S(v 1 ). Figure 8B shows the results from our model, which closely reflects the experimental results in Figure 8A. Note that we used a single flash because we modeled the input as a single step function (see equation 3.8). So the experiment may have been slightly different from the real one where grids come in and go out of view, corresponding to multiple step inputs. However, most of the perceived effect was accountable under this simplifying assumption, as shown in the results, indicating that the perceived dynamics within a single grid element can play an important role in determining the overall effect under moving conditions. Next, we tested the effect of stimulus flash duration on our model behavior. Figure 8D shows our model s prediction of the brightness as a function of the presentation duration. In this case, given a duration of d, the strength of illusion can be calculated as S(d). The perceived strength initially increases abruptly up to around t = ; then it slowly decreases until it reaches a steady level. Again, the computational results closely reflect those in human experiments (see Figure 8C). The initial increase might be due to the fact that the presentation time is within the time period required for one scintillation, and the slow decline may be due to no new scintillation being produced after the first cycle as the eyes were fixated, so that the overall perception of the scintillating strength declines. Again, this experiment is an approximation of the real condition, but we can make a reasonable assumption that the effect at the end of the input flash duration is what is perceived. For example, consider turning off the stimulus at a different point of time in Figure 6F (which in fact shows a close similarity to Figure 8D). Finally, note that because of the relationship between speed of motion and elapsed time mentioned above, the data presented in Figures 8B and 8D are identical (i.e., from the same simulation), except for the appropriate transformation in the x-axis. This is why we see the small wiggle in the beginning (on the left) in Figure 8B, which corresponds to the peak near t = and the slight decrease to a steady state in Figure 8D (on the right). In summary, our model based on disinhibition and self-inhibition was able to accurately replicate experimental data under various temporal conditions. 6 Discussion The main contribution of this work was to provide, to our knowledge, the first neurophysiologically grounded computational model to replicate the scintillating grid illusion. We have demonstrated that disinhibition and

18 538 Y. Yu and Y. Choe self-inhibition are sufficient mechanisms to explain a broad range of spatiotemporal phenomena observed in psychophysical experiments with the scintillating grid. The DOG filter failed to account for the change in the strength of scintillation, because it does not incorporate the disinhibition mechanism or the dynamics of self-inhibition. Disinhibition can effectively reduce the amount of inhibition in the case where there is a large area of bright input (Yu et al., 24). Therefore, a DOG filter without a disinhibition mechanism cannot explain why the dark illusory spots in the scintillating grid are perceived to be much darker than those in the Herman grid. The reason is that the DOG filter predicts that the white bars in the Hermann grid should give stronger inhibition to its intersection than the gray bars in the scintillating grid to its disc. Thus, according to DOG, the intersection in the Herman grid should appear darker than that in the scintillating grid, which is contrary to fact. However, with a disinhibition mechanism, since disinhibition is stronger in the Hermann grid than in the scintillating grid (because the bars are brighter in the Hermann grid, there is more disinhibition), the inhibition in the center of the Hermann grid is weaker than that in the scintillating grid. Thus, the center appears brighter (because of weaker inhibition) in the Hermann grid than in the scintillating grid due to disinhibition. Regarding the issue of dynamics, the lack of a self-inhibition mechanism in the DOG filter causes it to fail to explain the temporal properties of the scintillation. There are certain issues with our model that may require further discussion. In our simulations, we used a step input with an abrupt stimulus onset. In a usual viewing condition, the scintillating grid as a whole is presented, and when the gaze moves around, the scintillating effect is generated. All the while, the input is continuously present, without any discontinuous stimulus onset. Thus, the difference in the mode of stimulus presentation could be a potential issue. However, as Schrauf et al. (2) observed, what causes the scintillation effect is not the saccadic eye movement per se, but the transient stimulation that the movement brings about. Thus, such a transient stimulation can be modeled as a step input, and the results of our model may well be an accurate reflection of the real phenomena. Another concern is about the way we measured the strength of the scintillation effect in the model. In our model, we were mostly concerned about the change in the perceived brightness of the disc over time (see equation 4.1), whereas in psychophysical experiments, other measures of the effect have been incorporated, such as the perceived number of scintillating dark spots (Schrauf et al., 2). However, one observation is that the refresh rate of the stimulus depends on the number of saccades in a given amount of time. Considering that a single saccade triggers an abrupt stimulus onset, we can model multiple saccades as a series of step inputs in our simulations. Since our model perceives one scintillation per stimulus onset, the frequency of flickering reported in the model can be modulated exactly by changing the number of stimulus onsets in our simulations. A related issue is the use

19 A Neural Model of Scintillating Grid Illusion 539 of a single grid element (instead of a whole array) in our experiments. It may seem that the scintillation effect would require at least a small array (say 2 2) of grid elements. However, as McAnany and Levine (24) have shown, even a single grid element can elicit the scintillating effect quite robustly; thus, the stimulus condition in our simulations may be sufficient to model the target phenomenon. The model also gives a couple of interesting predictions (both brought to our attention by Rufin VanRullen). The first prediction is that a scintillating effect will occur only in an annular region in the visual field surrounding the fixation point, where the size of the receptive field matches that of the grid element size. However, this does not seem to be the case under usual viewing conditions. Our explanation for this apparent shortcoming of the model is that the size of the usual scintillating grid image is not large enough to go beyond the outer boundary of the annular region. Our explanation can be tested in two ways: test the strength of illusion with (1) a very large scintillating grid image where the grid element size remains the same, or with (2) the usual sized image with a reduced gridelement size (similar in manner to the input scaling approach by Raninen & Rovamo, 1987). We expect that the annular region will become visible in both cases, where no scintillating effect is observed beyond the outer boundary of the annular region. The second prediction is that the scintillation would be synchronous, due to the same time course followed by the neurons responding to each scintillating grid element. Again, this is quite different from our perceived experience, which is more asynchronous. In our observation, the asynchronicity is largely due to the random nature of eye movement. If that is true, the scintillating effect will become synchronous if eye movement is suppressed. That is, if we fixate on one location of the scintillating grid while the stimulus is turned on and off periodically (or alternatively, we can blink our eyes to simulate this), all illusory dark spots would seem to appear all at the same time, in a synchronous manner. (In fact, this seems to be the case in our preliminary experiments.) Then why is our experience asynchronous? The reason that we perceive the scintillation to be asynchronous may be that when we move our gaze from point X to point Y in a long saccade, first the region surrounding X, and then, later, the region surrounding Y scintillates. This will give the impression that the scintillating effect is asynchronous. In sum, the predictions of the model are expected to be consistent with experiments under similar conditions as in the computational simulations. Further psychophysical experiments may have to be conducted to test the model predictions more rigorously. Besides the technical issues as discussed, more fundamental questions need to be addressed. Our model was largely motivated by the pioneering work by Hartline et al. in the late 195s. However, the animal model they used was the Limulus, an invertebrate with compound eyes; thus, the applicability of our extended model in human visual phenomena may

20 54 Y. Yu and Y. Choe be questionable. However, disinhibition and self-inhibition, the two main mechanisms in the Limulus, have been discovered in mammals and other vertebrates. Mathematically, the recurrent inhibitory influence in the disinhibition mechanism and the self-inhibitory feedback are the same in both the limulus and in mammals. Therefore, our model based on the Limulus may generalize to human vision. Finally, an important question is whether our bottom-up model accounts for the full range of phenomena in the scintillating grid illusion. Why should the scintillating effect originate only from such a low level in the visual pathway? In fact, recent experiments have shown that part of the scintillating effect can arise based on top-down, covert attention (VanRullen & Dong, 23). The implication of VanRullen and Dong s study is that although the scintillation effect can originate in the retina, it can be modulated by later stages in the visual hierarchy. This is somewhat expected because researchers have found that the receptive field properties (which may include the size) can be effectively modulated by attention (for a review, see Gilbert, Ito, Kapadia, & Westheimer, 2). It is unclear how exactly such a mechanism can affect brightness-contrast phenomena that depend on the receptive field size at such a very low level; thus, it may require further investigation. Schrauf and Spillmann (2) also pointed out a possible involvement of a later stage by studying the illusion in stereo depth. But as they admitted, the major component of the illusion may be retinal in origin. Regardless of these issues, modeling spatiotemporal properties at the retinal level may be worthwhile by serving as a firm initial stepping-stone on which a more complete theory can be constructed. 7 Conclusion In this letter, we presented a neural model of the scintillating grid illusion, based on disinhibition and self-inhibition in early vision. The two mechanisms inspired by neurophysiology were found to be sufficient in explaining the multifaceted spatiotemporal properties of the modeled phenomena. We expect that our model can be extended to the latest results that indicate a higher-level involvement in the illusion, such as that of attention. Appendix: Derivation of Self-Inhibition Rate K s (t) The exact formula for K s (t) can be derived as follows: K s (t) = y(t) r(t), (A.1)

21 A Neural Model of Scintillating Grid Illusion 541 where y(t) is the amount of self-inhibition at time t and r(t) theresponse at time t for this cell. We know that the Laplace transform r(s) of r(t) has the following property: r(s)s = e(s) r(s)t s (s), T s (s) = k 1 + τ, (A.2) (A.3) where T s (s) is a transfer function, k the maximum value K s (t) can reach, τ the time constant, and e(s) the Laplace transform of the step input of this cell: e(s) = 1 s. (A.4) By rearranging equation A.2, we can solve for r(s) to obtain 1 r(s) = e(s). s + T s (A.5) Therefore,r(t) canbetreated asthestepinput function e(t) convolved with an impulse response function, r(t) = e(t) f (t), (A.6) where is the convolution operator, and f (t) = L 1 [ 1 s + T s ], (A.7) where L 1 is the inverse Laplace transform operator. Solving equation A.7, we get f (t) as a superposition of two exponential functions: f (t) = 1 C ( C1 exp(c 2 t) + C 2 exp(c 1 t) ), (A.8) where C = 1 4τk, C 1 = (C + 1)/2, and C 2 = (C 1)/2. The function y(t) can also be obtained in a similar manner as shown above: y(s) = r(s)t s (s). (A.9)

22 542 Y. Yu and Y. Choe Impulse response t f(t) g(t) Figure 9: Impulse response functions f (t) andg(t). By substituting r(s) with the right-hand side in equation A.5, we have T s y(s) = e(s). s + T s (A.1) Therefore, y(t) can also be treated as the step input function e(t) convolved with an impulse response function g(t) in time domain: y(t) = e(t) g(t), (A.11) where g(t) is a sine-modulated exponentially decaying function: [ ] g(t) = L 1 Ts = 6 5 exp( 5t) sin( 5t). (A.12) s + T s Hence, the final form of K s (t) can then be calculated as a division of two convolutions as follows: K s (t) = e(t) g(t) e(t) f (t). (A.13) Figure 9 shows the impulse response functions f (t) and g(t). The exact formula in equation 3.13 was derived based on the above derivation. Acknowledgments We thank Takashi Yamauchi, Rufin VanRullen, and an anonymous reviewer for helpful discussions and Jyh-Charn Liu for his support. This research

23 A Neural Model of Scintillating Grid Illusion 543 was funded in part by the Texas Higher Education Coordinating Board ATP grant and by the National Institute of Mental Health Human Brain Project, grant 1R1-MH A preliminary version of the material presented here appeared in Yu and Choe (24b) as an abstract. References Berry II, M. J., Brivanlou, I. H., Jordan, T. A., & Meister, M. (1999). Anticipation of moving stimuli by the retina. Nature, 398, Brodie, S., Knight, B. W., & Ratliff, F. (1978). The spatiotemporal transfer function of the limulus lateral eye. Journal of General Physiology, 72, Cai, D., DeAngelis, G. C., & Freeman, R. D. (1997). Spatiotemporal receptive field organization in the lateral geniculate nucleus of cats and kittens. Journal of Neurophysiology, 78, Crevier, D. W., & Meister, M. (1998). Synchronous period-doubling in flicker vision of salamander and man. Journal of Neurophysiology, 79, Frech, M. J., Perez-Leon, J., Wassle, H., & Backus, K. H. (21). Characterization of the spontaneous synaptic activity of amacrine cells in the mouse retina. Journal of Neurophysiology, 86, Gerrits, H. J., & Vendrik, A. J. (197). Simultaneous contrast, filling-in process and information processing in man s visual system. Experimental Brain Research, 26, Gilbert, C., Ito, M., Kapadia, M., & Westheimer, G. (2). Interactions between attention,context and learning in primary visual cortex.vision Research, 4, Hartline, H. K., & Ratliff, F. (1957). Inhibitory interaction of receptor units in the eye of Limulus. JournalofGeneralPhysiology,4, Hartline, H. K., & Ratliff, F. (1958). Spatial summation of inhibitory influences in the eye of Limulus, and the mutual interaction of receptor units. Journal of General Physiology, 41, Hartline, H. K., Wager, H., & Ratliff, F. (1956). Inhibition in the eye of Limulus. Journal of General Physiology, 39, Hartveit, E. (1999). Reciprocal synaptic interactions between rod bipolar cells and amacrine cells in the rat retina. Journal of Neurophysiology, 81, Jacobs, A. L., & Werblin, F. S. (1998). Spatiotemporal patterns at the retinal output. Journal of Neurophysiology, 8, Kitaoka, A. (23). Trick eyes 2.Tokyo:Kanzen. Kolb, H., & Nelson, R. (1993). Off-alpha and off-beta ganglion cells in the cat retina. Journal of Comparative Neurology, 329, Li, C. Y., Zhou, Y. X., Pei, X., Qiu, F. T., Tang, C. Q., & Xu, X. Z. (1992). Extensive disinhibitory region beyond the classical receptive field of cat retinal ganglion cells. Vision Research, 32, Marr, D., & Hildreth, E. (198). Theory of edge detection. Proceedings of the Royal Society of London B, 27, McAnany, J. J., & Levine, M. W. (24). The blanking phenomenon: A novel form of visual disappearance. Vision Research, 44,

24 544 Y. Yu and Y. Choe Neumann, H., Pessoa, L., & Hanse, T. (1999). Interaction of on and off pathways for visual contrast measurement. Biological Cybernetics, 81, Nirenberg, S., & Meister, M. (1997). The light response of retinal ganglion cells is truncated by a displaced amacrine circuit. Neuron, 18, Raninen, A., & Rovamo, J. (1987). Retinal ganglion-cell density and receptive-field size as determinants of photopic flicker sensitivity across the human visual field. Journal of the Optical Society of America A, 4, Ratliff, F., Hartline, H. K., & Miller, W. H. (1963). Spatial and temporal aspects of retinal inhibitory interaction. Journal of the Optical Society of America, 53, Roska, B., Nemeth, E., & Werblin, F. (1998). Response to change is facilitated by a three-neuron disinhibitory pathway in the tiger salamander retina. Journal of Neuroscience, 18, Schrauf, M., Lingelbach, B., & Wist, E. R. (1997). The scintillating grid illusion. Vision Research, 37, Schrauf, M., & Spillmann, L. (2). The scintillating grid illusion in stereo depth. Vision Research, 4, Schrauf, M., Wist, E. R., & Ehrenstein, W. H. (2). The scintillating grid illusion during smooth pursuit, stimulus motion, and brief exposure in humans. Neuroscience Letters, 284, Spillmann, L. (1994). The Hermann grid illusion: A tool for studying human perceptive field organization. Perception, 23, Stevens, C. F. (1964). A quantitative theory of neural interactions: Theoretical and experimental investigations. Unpublished doctoral dissertation, Rockefeller Institute. VanRullen, R., & Dong, T. (23). Attention and scintillation. Vision Research, 43, Victor, J. (1987). The dynamics of the cat retina x cell centre. Journal of Physiology, 386, Yu, Y., & Choe, Y. (24a). Angular disinhibition effect in a modified Poggendorff illusion. In K. D. Forbus, D. Gentner, T. Regier (Eds.), Proceedings of the 26th Annual Conference of the Cognitive Science Society (pp ). Mahwah, NJ: Erlbaum. Yu, Y., & Choe, Y. (24b). Explaining the scintillating grid illusion using disinhibition and self-inhibition in the early visual pathway. In Society for Neuroscience Abstracts. Program No Washington, DC: Society for Neuroscience. Yu, Y., Yamauchi, T., & Choe, Y. (24). Explaining low-level brightness-contrast illusions using disinhibition. In A. J. Ijspeert, M. Murata, N. Wakamiya (Eds.), Biologically inspired approaches to advanced Information technology (BioADIT 24) (pp ). Berlin: Springer. Received February 18, 25; accepted September 8, 25.