Computation in the Higher Visual Cortices: Map-Seeking Circuit Theory and Application to Machine Vision

Transcription

1 Coputation in the Higher Visual Cortices: Map-Seeking Circuit Theory and Application to Machine Vision David Arathorn Center for Coputational Biology Montana State University and General Intelligence Corporation PO Box 738, Bozean, MT, 5977 Abstract Map-Seeking Circuit theory is a biologically based coputational theory of vision applicable to difficult achine vision probles such as recognition of 3D objects in arbitrary poses aid distractors and clutter, as well as to non-recognition probles such as terrain interpretation. It provides a general coputational echanis for tractable discovery of correspondences in assive transforation spaces by exploiting an ordering property of superpositions. The latter allows a set of transforations of an input iage to be fored into a sequence of superpositions which are then culled to a coposition of single appings by a copetitive process which atches each superposition against a superposition of inverse transforations of eory patterns. The architecture that perfors this is based on a nuber of neuroanatoical features of the visual cortices, including reciprocal dataflows and inverse appings.. Introduction The echanis described here evolved fro an effort to reverse engineer the visual cortices to create a viable achine vision echanis. Many of the techniques used in conventional coputational vision are precluded either by the liitations of plausible neuronal circuitry or by psychophysical characteristics of biological vision. For exaple, it can be reasonably argued that the repertoire of coputations that can be ipleented by realistic neurons is liited to cobination (suing of signals fro dendritic branches, linear over a liited range) and analog gating (pseudo-ultiplication locally in thin branches of dendrites)[], thresholding and clipping, copetition (via lateral inhibition) and apping (via synaptic interconnectivity patterns). On the other hand, neuroanatoy offers a rich hint of a solution in the vast neuronal resources allocated to creating reciprocal topdown and botto-up pathways. More specifically, this reciprocal pathway architecture appears to be organized with reciprocal, co-centered fan outs in the opposing directions [2]. Visual psychophysics provides such a vast array of hints as to how the aalian visual syste is organized that it is difficult to organize the. However, individual psychophysical behaviors preclude soe of the favored coputational approaches of achine vision. For exaple, the variety of kinetic depth effects which allow us to distinguish 3D surfaces defined by oving sets of indistinguishable dots akes any solution by deterining pairwise correspondences virtually intractable. Siilarly, our ability to recognize distant objects which the fovea can represent with less than a dozen cycles of resolution is a psychophysical behavior which precludes the invariant feature approaches often used in achine vision. The hope of bioietic vision is that by discovering and applying the principles which drive biological vision we ay at least approach if not exceed the capabilities of aalian vision. Thus neuroanatoy, neurophysiology and visual psychophysics ust be allowed to provide both the constraints and treasure ap for the search. The general echanis described here was evolved by this strategy. It has coe to be called a ap-seeking circuit because its atheatical expression has an isoorphic ipleentation in quite realistic neuronal circuitry [3]. The ost striking features of the architecture of both, seen in Figure, are the reciprocal forward and backward pathways a signature of the biological cortices with their forward and inverse apping sets. In atheatical for, as seen in equations 6-9 below, it provides a parsionious, coputationally explicit theoretical echanis for the accuulating body of neurobiological evidence that top-down expectations drive vision a conclusion also iposed by atheatical necessity in what is otherwise an ill-posed proble. The practical effectiveness of the sae equations will also be deonstrated. Proceedings of the 33rd Applied Iagery Pattern Recognition Workshop (AIPR 4) /4 $ 2. IEEE

2 forward aps: t input iage orprevious layer atch cop backward superposition: b forward superposition: f atch inverse aps: t ' () eory patterns: w k eory superposition atch Figure. Data flow in ap-seeking circuit 2. The proble The abstract proble solved by the ap-seeking circuit is the discovery of a coposition of transforations between an input pattern and a stored pattern (or between two input patterns, as in the case of stereovision). In general the transforations express the generating process of the proble. Define correspondence c between vectors r and w through a coposition of transforations 2 l l l l t, t,, t where t t, t,, t j j2 j jl 2 nl i c() j = tj () r, w eq. i i= where the coposition operator is defined l = tj t l j t j r t ji ( r) = i=, l = r et C be an diensional atrix of values of c(j) whose diensions are n n. The proble, then is to find x= arg ax c( j) eq. 2 The indices x specify the sequence of transforations that best correspondence between vectors r and w. The proble is that C is too large a space to be searched by conventional eans. Instead, ap-seeking circuits search a superposition space Q defined nl Q : l= l l l l Q( G + ) = gi ti, gi ti l= l=, i r i w eq. 3 where = [ x ] =, = of t in layer, l x [, ] ti G g x n n is nuber l g, is adjoint of ti. Q(G) is the hypersurface defining the value of the inner product of forward and backward superpositions for all values of g. In Q space, the solution lies along a single axis in each layer. Superposition culling uses the coponents of grad Q to copute a path in steps g tothe axis in each layer l which corresponds to the best fitting transforation t xl,wherex l is the l th eleent of x in eq. 2. QG + l l l l = tj gi ti, gi ti g l=, r l=, w eq. 4 j i i Q( G) Q( G) g = f,, eq. 5 g g n The function f preserves the axial coponent and reduces the others: in neuronal ters, lateral inhibition. This reforulation of the proble into the superposition space Q perits a search with resources proportional to the su of sizes of the diensions of C instead of their product. The price for oving the proble into superposition space is that collusions of coponents of the superpositions can result in better atches for incorrect appings than for the appings of the correct solution. The ordering property of superpositions [4] gives a probabilistic description of the occurrence of collusion for pattern vectors which satisfy the distribution properties of decorrelating encodings, for which there is reasonable Proceedings of the 33rd Applied Iagery Pattern Recognition Workshop (AIPR 4) /4 $ 2. IEEE

3 neurobiological evidence [5]. The foral stateent of the superposition ordering property has two cases: n a) If a superposition s= vi is fored fro a set of i= sparse vectors vi V,thenforavectorv k which is not part of the set fro which superposition is fored, vk V, the following relationship expresses the ordering property of superpositions: > Pincorrect where = P( vi s> vk s), Pincorrect = P( vi s vk s) and as n b) If three superpositions n r= u, i s= v and j s = v i= j= k = are fored fro three sets of sparse vectors ui R, v j S and vk S where R S= and R S = vq then the following relationship expresses the superposition ordering property: > Pincorrect where = P( r s > r s), Pincorrect = P( r s r s) and as n, The proof [6] of these akes use of the assuption of pdf or pf equality fi( q) = fk ( q) where fi( q) = P( vi ( s vi) = q), fk ( q) = P( vk ( s vi) = q) For a given sparsity of iage encoding the probability of the occurrence of collusion decreases with the decrease in nubers of contributing coponents in the superposition(s). Thus the strategy for exploiting the superposition space reliably is to allow convergence to quickly prune the nuber of contributors to the superpositions before coitting to a solution. For proble spaces which ake collusion less likely the convergence can be allowed to proceed ore quickly. At the liit, in proble spaces which preclude collusion, the first iteration gives the solution. The role of the ordering property in convergence to a correct solution can be understood in how it dictates the shape of the surface Q, specifically the relationship of the coponents of grad Q in eq. 5. That is, the largest coponent of grad Q in each layer will correspond with the correct apping with increasing probability where ore of the eleents of G are near zero. The decoposition of the aggregate transforation into subtransforations proves to be ore than a k atheatical convenience. The specific characteristics of the subtransforations turn out to carry cognitively critical inforation. In vision apping classes specify pose, distance, and location in the visual field or yield the paraeters for surface orientation for shape-fro-viewdisplaceent coputation. 3. The ap-seeking solution A ap-seeking circuit is coposed of several transforation or apping layers between the input at one end and a eory layer at the other. Other than the pattern of individual connections which ipleent the set of appings in each layer, the layers theselves are ore or less identical. The copositional structure is evident in the siplicity of the equations (eqs. 6-9 below) which define a circuit of any diension. In a ulti-layer circuit of layers plus eory with n l appings in layer l the apping coefficients g are updated by the recurrence : (, ) + gi = cop gi ti f b for=, i= nl eq. 6 where atch operator u v = q, q is a scalar easure of goodness-of-atch between u and v, and f and b are defined below. When is a dot product the second arguent of cop is Q/g in eq. 4. The function cop is a realization of lateral inhibition function f in eq. 5. k2 qi cop( gi, qi) = ax, gi k h eq. 7 ax q if ax q> t where h = t if ax q t The forward path signal for layer is coputed nl l l f = g t f for = eq. 8 j= j j The backward path signal for layer is coputed nl l l + g j t j ( b ) for = j= b = eq. 9 z ( wk f ) wk for = + k In above, f l l is the input signal, t j, t j are the j th forward and backward appings for the l th layer, w k is the k th eory pattern, z is a non-linearity applied to the response of each eory. g l is the set of apping l coefficients g j for the l th layer each of which is associated with apping t l j and is odified over tie by the expression that is the second arguent of the copetition function. cop is any function (one is illustrated) which leaves the axiu g in g l unchanged and oves the other values of g toward zero in proportion to their difference fro the axiu eleent. Constants k and k 2 control the speed of convergence. Proceedings of the 33rd Applied Iagery Pattern Recognition Workshop (AIPR 4) /4 $ 2. IEEE

4 The apping gain coefficient g, is a relative easure of the probability that the singleton pattern corresponds to a contributor to the superposition, or that the two superpositions both contain a corresponding pattern. The copetition process guarantees that each layer will converge to a single apping with a non-zero g coefficient if the input pattern can be atched to one of the eory patterns, or that it will converge to all zero coefficients for all appings in all layers if no acceptable atch (i.e. sub-threshold) can be established. This process of superposition culling converges in tie (or steps) largely independent of the nuber of initially active appings. It is the ordering property of superpositions, discussed above, that causes the best set of appings ultiately to be selected with high probability at the end of convergence. This probability increases as the convergence proceeds since it is inversely related to the nuber of contributors to the superposition. Since there is a non-zero probability of selection of a eory pattern or transforation which provides a poor atch when in fact a good atch is available, the failure ode of the ap-seeking circuit is to converge to a false negative. In practice this is rarely seen. Recognizing 2D projections of 3D objects The space of transforations with which a 3D object ay project onto the retina, even when liited by discretization and liitation of range, in practice exceeds. By usual ethods this space is intractably large to search exhaustively. However, the use of superpositions and decoposition, as proposed in ap-seeking circuit theory, allows this space to be searched in biologically (and technologically) realistic tie. An exaple of a recognition proble in this doain is seen in Figure 2: given a 3D odel of a particular odel of tank, distinguish the correct target undeterred by occlusion, noise and distractors. The ap-seeking circuit used in the deonstration in Figs. 2, 3 has four layers of transforational appings: translation (22 pixels by steps of pixel), rotation in plane (-5 to +5 by ),scaling(.4to.4bysteps of.25) and 3D projection (aziuth -9 to +9 by 5, elevation to 6 by 5). A zero-th, initial, layer is used to process the gray scale iage into an oriented-edge representation which is used in the 2D doain layers. (a) T8-3D eory odel (b) CG input iage (c)iter (d)iter3 (e)iter2 (f)finalodelpose Figure 2. Occluded target (T-8) with siilar distractor, coputer generated scene. (a) T8-3D eory odel encoded as norals and edges; (b) CG input iage: occluded target and distractors; (c-e) isolation of target in layer, iterations, 3, 2; (f) deterination of pose in final iteration, layer 4 backward. Proceedings of the 33rd Applied Iagery Pattern Recognition Workshop (AIPR 4) /4 $ 2. IEEE

5 This zero-th layer also provides the capability to operate in very low resolution and noisy conditions, as will be discussed below. Figure 2c-e illustrate the evolution of the forward signal fro layer during noral convergence of the circuit to the correct sequence of transforations in layers -4 respectively, which deterine the location, scale, rotation-in-viewing-plane and (nearly) correct 3D pose of the target. Figure 2f is the final projection of the odel. Recognition in real operating conditions In real environents, biological as well as achine vision is called upon to identify objects at distances or in conditions which liit the resolution to or fewer cycles on the long axis of the object. At such low resolutions there are no alignable features other than the shape of the object itself, and even its own boundaries are sufficiently blurred as to prevent generating reliable edges in a feedforward anner. However, the 3D odel in eory can be used to hypothesize top-down, in biological parlance, possible locations of edges in blurred iage. The inverseappings of the 3D odel to the possible projections, scalings, rotations and translations creates a set of edge hypotheses on the backward path out of layer into layer. In layer these hypotheses are used to gate the input iage. As convergence proceeds, the edge hypotheses are reduced to a single edge hypothesis that best fits the grayscale input iage. Figure 3 shows one of a series of tests of the sae circuit used in the deonstration in Figure 2 applied to deliberately blurred iages fro the Fort Carson Iagery Data Set. The 3D eory odel used in these tests is a siplified representation of an M-6 tank without a barrel. The circuit has no difficulty distinguishing the location and orientation of the tank, despite distractors and background clutter. Perforance On coputer generated iagery, such as seen in Figure 2, the ap-seeking circuit always located the correct target unless deliberately induced to fail by severely reducing target contrast relative to the contrast of a siilar nontarget vehicle or by oving it to an extree peripheral location relative to distractor. In all cases the circuit was able to deterine orientation to about 5. Using the Fort Carson visual spectru input iagery, both blurred (a) 3D odel of M-6 tank (b)source iage (c) input iage - blurred (d) iter (e) iter 3 (f) iter 2 (g) final odel pose Figure 3. Target (M6) with distractor vehicles, Fort Carson scene. (a) M6 3D eory odel; (b) source iage; (c) Gaussian blurred input iage; (d-f) isolation of target in layer, iterations, 3, 2; (g) pose deterination in final iteration, layer 4 backward. Note odel pose is presented left-right irrored to reflect irroring deterined in layer 3. M-6 odel courtesy Colorado State University. Proceedings of the 33rd Applied Iagery Pattern Recognition Workshop (AIPR 4) /4 $ 2. IEEE

6 and full resolution, the ap-seeking circuit always located the target and in about 6% of cases correctly deterined orientation to about 5. Most of the incorrect orientations were syetries of the correct orientation, i.e. projections with very siilar silhouette to the correct projection. On Fort Carson IR iagery, soe incorrect identifications of vehicles occurred with low contrast iagery. When contrast was noralized and IR hotspots reoved, perforance was siilar to perforance on visual spectru iagery. [5] B.A. Olshausen, D.J. Field, Eergence of Siple-Cell Receptive Field Properties by earning a Sparse Code for Natural Iages, Nature, 38, 996 pp67-69 [6] D. Arathorn, A Solution to the Generalized Correspondence Proble Using an Ordering Property of Superpositions, subitted 24. 3D odel of T-8 and WW-2 tank courtesy 3DCafe.co 4. Conclusion Object recognition fro 3D eory odels proves to be a natural application for ap-seeking circuits, particularly in operating conditions that have proved extreely difficult for other ethods, as exeplified by the low-resolution or otherwise degraded iagery illustrated in Figure 3. This is a copelling indication that drawing inspiration fro plausible biological visual circuitry is a path of research for achine vision that will result in robustness and capability ore nearly typical of biological vision. Biology has been chary with its circuit engineering. It is generally believed that cortical architecture is quite siilar throughout the brain, suggesting that a coon coputational repertoire ay be at work in a variety of perceptual, otor and cognitive functions. The proble solved by ap-seeking circuits, the discovery of appings between patterns, certainly applies to other visual probles, such as 2D iage recognition and shape-fro-view displaceent (otion or stereo). Nevertheless there reains uch work to do to accoodate the full range of biological visual capabilities with ap-seeking circuits, and only tie will tell whether they ay be extended to such capabilities as recognition under deforation, generalization and recognition by coponents. References [] A. Polsky, B. Mel, J. Schiller, Coputational Subunits in Thin Dendrites of Pyraidal Cells, Nature Neuroscience 7(6), 24 pp [2] A. Angelucci, B. evitt, E. Walton, J.M. Hupé, J. Bullier, J. und, Circuits for ocal and Global Signal Integration in Priary Visual Cortex, Journal of Neuroscience, 22(9), 22 pp [3] D. Arathorn, Map-Seeking Circuits in Visual Cognition, Palo Alto, Stanford University Press, 22 [4] D. Arathorn, Map-Seeking: Recognition Under Transforation Using A Superposition Ordering Property. Electronics etters 37(3), 2 pp64-65 Proceedings of the 33rd Applied Iagery Pattern Recognition Workshop (AIPR 4) /4 $ 2. IEEE