Abstract. 1 Introduction

Transcription

1 A Cortically-Plausible Inverse Proble Solving Method Applied to Recognizing Static and Kineatic 3D Objects David W. Arathorn Center for Coputational Biology, Montana State University Bozean, MT 5977 General Intelligence Corporation Abstract Recent neurophysiological evidence suggests the ability to interpret biological otion is facilitated by a neuronal irror syste which aps visual inputs to the pre-otor cortex. If the coon architecture and circuitry of the cortices is taken to iply a coon coputation across ultiple perceptual and cognitive odalities, this visual-otor interaction ight be expected to have a unified coputational basis. Two essential tasks underlying such visual-otor cooperation are shown here to be siply expressed and directly solved as transforation-discovery inverse probles: (a) discriinating and deterining the pose of a pried 3D object in a real-world scene, and (b) interpreting the 3D configuration of an articulated kineatic object in an iage. The recently developed ap-seeking ethod provides a atheatically tractable, cortically-plausible solution to these and a variety of other inverse probles which can be posed as the discovery of a coposition of transforations between two patterns. The ethod relies on an ordering property of superpositions and on decoposition of the transforation spaces inherent in the generating processes of the proble. Introduction A variety of brain tasks can be tersely posed as transforation-discovery probles. Vision is replete with such probles, as is lib control. The proble of recognizing the 2D projection of a known 3D object is an inverse proble of finding both the visual and pose transforations relating the iage and the 3D odel of the object. When the object in the iage ay be one of any known objects another step is added to the inverse proble, because there are ultiple candidates each of which ust be apped to the input iage with possibly different

2 transforations. When the known object is not rigid, the deterination of articulations and/or orphings is added to the inverse proble. This includes the general proble of recognition of biological articulation and otion, a task recently attributed to a neuronal irror-syste linking visual and otor cortical areas []. Though the aggregate transforation space iplicit in such probles is vast, a recently developed ethod for exploring vast transforation spaces has allowed soe significant progress with a siple unified approach. The ap-seeking ethod [2,4] is a general purpose atheatical procedure for finding the decoposition of the aggregate transforation between two patterns, even when that aggregate transforation space is vast and there is no prior inforation is available to restrict the search space. The proble of concurrently searching a large collection of eories can be treated as a subset of the transforation proble and consequently the sae ethod can be applied to find the best transforation between an input iage and a collection of eories (nubering at least thousands in practice to date) during a single convergence. In the last several years the ap-seeking ethod has been applied to a variety of practical probles, ost of the related to vision, a few related to kineatics, and soe which do not correspond to usual categories of brain functions. The generality of the ethod is due to the fact that only the appings are specialized to the task. The atheatics of the search, whether expressed in an algorith or in a neuronal or electronic circuit, do not change. Fro an evolutionary biological point of view this is a satisfying characteristic for a odel of cortical function because only the connectivity which ipleents the appings ust be varied to specialize a cortex to a task. All the rest organization and dynaics would reain the sae across cortical areas. Figure. Data flow in ap-seeking circuit

3 Cortical neuroanatoy offers ephatic hints about the characteristics of its solution in the vast neuronal resources allocated to creating reciprocal top-down and bottoup pathways. More specifically, recent evidence suggests this reciprocal pathway architecture appears to be organized with reciprocal, co-centered fan outs in the opposing directions [3], quite possibly ipleenting inverse appings. The data flow of ap-seeking coputations, seen in Figure, is architecturally copatibility with these features of cortical organization. Though not within the scope of this discussion, it has been deonstrated [4] that the atheatical expression of the ap-seeking ethod, seen in equations 6-9 below, has an isoorphic ipleentation in neuronal circuitry with reasonably realistic dendritic architecture and dynaics (e.g. copatible with [5] ) and oscillatory dynaics. 2 The basis for tractable transforation-discovery The related probles of recognition/interpretation of 2D iages of static and articulated kineatic 3D objects illustrate how cleanly significant vision probles ay be posed and solved as transforation-discovery inverse probles. The visual and pose (in the sense of orientation) transforations, t visual and t pose, between a given 3D odel and the extent of an input iage containing a 2D projection P(o ) of an object o appable to can be expressed pose ( ) j k ( ) P o t t t T t T eq. visual visual visual pose pose = j, k If we now consider that the odel ay be constructed by the one-to-any apping of a base vector or feature e, and that arbitrarily other odels i ay be siilarly constructed by different appings, then the transforation t foration corresponding to the correct eory converts the eory database search proble into another transforation-discovery proble with one ore coposed transforation visual ( ) ( e) P o = t t t t T pose foration foration j k foration ( e) foration t = M eq. 2 Finally, if we allow a orphable object to be constructed by a generative odel, whose various configurations or articulations ay be generated by a coposition of transforations t generative of soe root or seed feature e, the proble of explicitly recognizing the particular configuration of orph becoes a transforationdiscovery proble of the for visual ( ( )) ( e) P C o t t t t T pose generative generative generative l = j k l l eq. 3 These unifying forulations are only useful, however, if there is a tractable ethod of solving for the various transforations. That is what the ap-seeking ethod provides. Abstractly the proble is the discovery of a coposition of transforations between two patterns. In general the transforations express the generating process of the proble. Define correspondence c between vectors r and 2 l l l l w through a coposition of transforations t j, t,, where j t 2 j t, 2,, j t t t l nl This illustrates that foring a superposition of eories is equivalent to foring superpositions of transforations. The first is a ore practical realization, as seen in Figure. Though not deonstrated in this paper, the ulti-eory architecture has proved robust with or ore eory patterns fro real-world datasets (paper in preparation).

4 i ( j) = t j ( r), w eq. 4 c i i= where the coposition operator is defined l t ji i=, ( r) l = t t t = l = r j j j ( r) et C be an diensional atrix of values of c(j) whose diensions are n n. The proble, then is to find x = arg ax c( j ) eq. 5 The indices x specify the sequence of transforations that best correspondence between vectors r and w. The proble is that C is too large a space to search for x by conventional eans. Instead, a continuous ebedding of C perits a search with resources proportional to the su of sizes of the diensions of C instead of their product. C is ebedded in a superposition dot product space Q defined nl Q : l= + Q g t g t eq. 6 ( G l l l l ) = i i ( r ), i i ( w ) l = l =, i i where G = [ gx ] =, x = n n is nuber of t in layer, x [, ] l l t is adjoint of t. i i g, In Q space, the solution to eq. 5 lies along a single axis in the set of axes represented each row of G. That is, g =<,, ux,, > ux > which corresponds to the best fitting transforation t x, where x is the th index in x in eq. 5. This state is reached fro an initial state G = [ ] by a process tered superposition culling in which the coponents of grad Q are used to copute a path in steps g, ( G) Q + l l l l = t j gi ti ( ), gi ti ( ) g l=, r l=, w eq. 7 j i i ( G) Q( G) Q g = f,, eq. 8 g g n The function f preserves the axial coponent and reduces the others: in neuronal ters, lateral inhibition. The resulting path along the surface Q can be thought of as a high traverse in contrast to the gradient ascent or descent usual in optiization ethods. The price for oving the proble into superposition dot product space is that collusions of coponents of the superpositions can result in better atches for incorrect appings than for the appings of the correct solution. If this occurs it is alost always a teporary state early in the convergence. This is a consequence of the ordering property of superpositions (OPS) [2,4], which, as applied here, describes the characteristics of the surface Q. For exaple, let three n = i= superpositions r u, i s = j j= v and s = k = v be fored fro three sets of sparse k

5 vectors ui R, v j S and vk S where R S = and R S = v q. Then the following relationship expresses the OPS: ( r s r s), incorrect ( r s r s) define P = P > P = P then correct P > P or P >.5 correct incorrect correct and as n, P. correct Applied to eq. 8, this eans that for superpositions coposed of vectors which satisfy the distribution properties of sparse, decorrelating encodings 2 (a biologically plausible assuption [6]), the probability of the axiu coponents of grad Q oving the solution in the correct direction is always greater than.5 and increases toward. as the G becoes sparser. In other words, the probability of the occurrence of collusion decreases with the decrease in nubers of contributing coponents in the superposition(s), and/or the decrease in their gating coefficients G. 3 The ap-seeking ethod and application A ap-seeking circuit (MSC) is coposed of several transforation or apping layers between the input at one end and a eory layer at the other, as seen in Figure. The copositional structure is evident in the siplicity of the equations (eqs. 9-2 below) which define a circuit of any diension. In a ulti-layer circuit of layers plus eory with n l appings in layer l the forward path signal for layer is coputed n j= ( ) f = g t f for = eq. 9 j j The backward path signal for layer is coputed b nl + g j t j ( b ) for = j= = nw z ( w k f ) w k or gk w k or w k k = The apping coefficients g are updated by the recurrence + ( ( f ) b ) ( f w ) g : = κ g, t for =, i = n i i i l g : = κ g, for k = n (optional) + + i i k w for = + eq. eq. where atch operator u v = q, q is a scalar easure of goodness-of-atch between u and v, and ay be non-linear. When is a dot product, the second arguent of is the sae as Q/g in eq. 7. The copetition function is a realization of lateral inhibition function f in eq. 8. It ay optionally be applied to the eory layer, as seen in eq.. k2 qi κ( gi, qi ) = ax, gi k ax q eq. 2 2 A restricted case of the superposition ordering property using non-sparse representation is exploited by HRR distributed eory. See [7] for an analysis which is also applicable here.

6 Thresholds are norally applied to q and g, below which they are set to zero to speed convergence. In above, f is the input signal, t j, t j are the j th forward and backward appings for the th layer, w k is the k th eory pattern, z( ) is a nonlinearity applied to the response of each eory. g is the set of apping coefficients g j for the th layer, each of which is associated with apping t j and is odified over tie by the copetition function ( ). Recognizing 2D projections of 3D objects under real operating conditions (a) 3D eory odel (b)source iage (c) input iage - blurred (d) iter (e) iter 3 (f) iter 2 (g) final odel pose Figure 2. Recognizing target aong distractor vehicles. (a) M6 3D eory odel; (b) source iage, Fort Carson Data Set; (c) Gaussian blurred input iage; (d-f) isolation of target in layer, iterations, 3, 2; (g) pose deterination in final iteration, layer 4 backward - presented left-right irrored to reflect irroring deterined in layer 3. M-6 odel courtesy Colorado State University. Real world probles of the for expressed in eq. often present objects at distances or in conditions which so liit the resolution that there are no alignable features other than the shape of the object itself, which is sufficiently blurred as to prevent generating reliable edges in a feed-forward anner (e.g. Fig. 2c). In the ap-seeking approach, however, the top-down (in biological parlance) inverseappings of the 3D odel are used to create a set of edge hypotheses on the backward path out of layer into layer. In layer these hypotheses are used to gate the input iage. As convergence proceeds, the edge hypotheses are reduced to a single edge hypothesis that best fits the grayscale input iage. Figure 2 shows this process applied to one of a set of deliberately blurred iages fro the Fort Carson Iagery Data Set. The MSC used four layers of visual transforations: 4,4 translational, 3 rotational, 4 scaling, 48 3D projection. The MSC had no difficulty distinguishing the location and orientation of the tank, despite distractors and background clutter: in all tests in the dataset target was correctly located. In effect, once pried with a top-down expectation, attentional behavior is an eergent property of application of the ap-seeking ethod to vision [8].

7 Adapting generative odels by transforation The direct-atching hypothesis of the interpretation of biological otion] holds that we understand actions when we ap the visual representation of the observed action onto our otor representation of the sae action. [] This apping, attributed to a neuronal irror-syste for which there is gathering neurobiological evidence (as reviewed in []), requires a echanis for projecting between the visual space and the constrained skeletal joint paraeter (kineatic) space to disabiguate the 2D projection of body structure. 3 Though this proble has been solved to various degrees by other coputational ethods, a review of which is beyond the scope of this discussion, to the author s knowledge none of these have biological plausibility. The present purpose is to show how siply the proble can be expressed by the generative odel interpretation proble introduced in eq. 3 and solve by ap-seeking circuits. An idealized exaple is the proble of interpreting the shape of a featureless snake articulated into any configuration, as appears in Fig. 3. (a) (b) spine (e) (c) (d) Figure 3. Projection between visual and kineatic spaces with two ap-seeking circuits. (a) input view, (b) top view, (c) projection of 3D occluding contours, (d,e) projections of relationship of occluding contours to generating spine. The solution to this proble involves two coupled ap-seeking circuits. The kineatic circuit has layers which odel the ultiple degrees of freedo (here two angles, variable length and optionally variable radius fro spine to surface) of each of the connected spine segents. The other circuit deterines the visual transforations, as seen in the earlier exaple. The surface of the articulated cylinder is apped fro an axial spine. The points where that surface is tangent to the viewpoint vectors define the occluding contours which, projected in 2D, becoe the object silhouette. The proble is to find the articulations, segent lengths (and optionally segent diaeter) which account for the occluding contour atching the silhouette in the input iage. In the MSC solution, the initial state all possible articulations of the snake spine are superposed, and all the occluding contours fro a range of viewing angles are projected into 2D. The latter superposition serves as the backward input to the visual space ap-seeking circuit. Since the snake surface is deterined by all of the layers of the kineatic circuit, these are projected in 3 The application of the ap-seeking ethod to solve lib inverse kineatics under constraints of joint angle range and external obstacles was first deonstrated in [4].

8 parallel to for the backward (biologically top-down) 2D input to the visual transforation-discovery circuit. A atching operation between the contributors to the 2D occluding contour superposition and the forward transforations of the input iage odulates the gain of each apping in the kineatic circuit via a i in eqs. 3, 4 (odified fro eq. ). In eqs. 3, 4 K indicates kineatic circuit, V indicates visual circuit. K K VK K K K K K + gi : = cop gi, ai t i f b for =, i = nl eq. 3 eq. 4 VK V K K 3D 2D surface + ai = f t t ti b The process converges concurrently in both circuits to a solution, as seen in Figure 3. The atch of the occluding contours and the input iage, Figure 3a, is seen in Figure 3b,c, with its three diensional structure is clarified in Figure 3d. Figure 3e shows a view of the 3D structure as deterined directly fro the apping paraeters defining the snake spine after convergence. A variety of snake configurations have been successfully tested. 4 Conclusion The investigations reported here expand the envelope of vision-related probles aenable to a pure transforation-discovery approach ipleented by the apseeking ethod. The recognition of static 3D odels, as seen in Figure 2, and other probles [9] solved by MSC have been well tested with real-world input. Nuerous variants of Figure 3 have deonstrated the applicability of MSC to recognizing generative odels of high diensionality, and the principle has recently been applied successfully to real-world doains. Consequently, the research to date does suggest that a single cortical coputational echanis could span a significant range of the brain s visual and kineatic coputing. References [] G. Rizzolati,. Fogassi, V. Gallese, Neurophysiological echaniss underlying the understanding and iitation of action, Nature Reviews Neuroscience, 2, 2, [2] D. Arathorn, Map-Seeking: Recognition Under Transforation Using A Superposition Ordering Property. Electronics etters 37(3), 2 pp64-65 [3] A. Angelucci, B. evitt, E. Walton, J.M. Hupé, J. Bullier, J. und, Circuits for ocal and Global Signal Integration in Priary Visual Cortex, Journal of Neuroscience, 22(9), 22 pp [4] D. Arathorn, Map-Seeking Circuits in Visual Cognition, Palo Alto, Stanford Univ Press, 22 [5] A. Polsky, B. Mel, J. Schiller, Coputational Subunits in Thin Dendrites of Pyraidal Cells, Nature Neuroscience 7(6), 24 pp [6] B.A. Olshausen, D.J. Field, Eergence of Siple-Cell Receptive Field Properties by earning a Sparse Code for Natural Iages, Nature, 38, 996 pp67-69 [7] T. Plate, Holographic Reduced Representation, CSI publications, Stanford, California, 23 [8] D. Arathorn,, Meory-driven visual attention: an eergent behavior of ap-seeking circuits, in Neurobiology of Attention, Eds Itti, Rees G, Tsotsos J, Acadeic/Elsevier, 25 [9] C. Vogel, D. Arathorn, A. Parker, and A. Roorda, "Retinal otion tracking in adaptive optics scanning laser ophthaloscopy", Proceedings of OSA Conference on Signal Recovery and Synthesis, Charlotte NC, June 25.