University of Genova - DITEN Smart Patrolling 1
Smart Patrolling Detection of the intruder Tracking of the intruder A cognitive node will active an operator, describing on his mobile terminal the characteristic of the intruder: Position Weapons detention The operator will be supported in his actions ANALYSIS SENSING
Mapping of Observed Interaction model onto ICE Cognitive node DECISION ACTION ANALYSIS SENSING DECISION ACTION ANALYSIS SENSING
Applications of the empathic interaction model for motion analysis in imagery streams The probabilistic model for empathic interaction analysis can be used to describe human activities at different semantic resolution levels Example: two level empathic interaction model: High level trajectory analysis high semantic level, low resolution Low level sub-parts analysis low semantic level, high resolution
Emphatic behavior modeling and recognition at different semantic levels Multi-level feature hierarchical structure for emotional human behavior analysis High level global environment position of entities in the scene E1 action Perceived E2 action environment E2 action Perceived E1 action global emotion obs. space actions/trajectories E1 E1 emotion Perceived E2 emotion Shared emotion observation space E2 emotion Perceived E1 emotion E2 E local environment shape local emotion obs. space shape deformations E11 E11 action Perceived E12 emotion environment Perceived actions E11 emotion E12 emotion E Shared emotion observation space E12 action Perceived E11 emotion Low level
Emphatic interaction modeling and recognition by trajectory analysis Modeling human activity in video is one of the most challenging task in many video analytics applications as video-surveillance, ambient intelligence, etc. Models should be enough discriminative to distinguish between considered actions but also general to include behavior variability and to handle noise introduced by low level analysis modules Trajectories are very common descriptors of movement pattern and their representation as 2D sequences of points into a map have been widely used for action recognition purposes A bio-inspired model derived from neuro-physiological studies on the generation of consciousness in the human brain is used to describe the emotional causal relationships taking place between the two entities.
Emphatic interaction modeling and recognition by trajectory analysis Modeling and recognition of empathic human interactions by trajectory analtysis is performed considering two semantic levels of information: The environment level is related to the position of the entities in the scene: X P (t),x C (t) E1 action Perceived E2 action environment Perceived E1 action E2 action E1 emotion E2 emotion The emotion observation level is related to actions and movements of the entities in the scene: E1 Shared emotion observation space E2 Sx P (t),sx C (t) Perceived E2 emotion Perceived E1 emotion ε P ε C
Emphatic interaction modeling and recognition by trajectory analysis These neuro-physiological model of empathic interactions are the inspiration to define an appropriate representation of interactions by defining the causal relationships occurring between them. Dynamic Bayesian Networks [14] are used to encode in an appropriate probabilistic way the temporal evolution of the interactions. States modifications of the Autobiographical Memory in this context are the state of the DBN and they are represented as events described as zone changes of two entities in a map: Proto Event Core Event [14] K. P. Murphy, Dynamic Bayesian Networks : representation, inference and learning, Ph.D. dissertation, Berkeley, CA, USA, 2002. 8
Map Generation ITM Algorithm To create the Emotion Observation Space, dimensionality reduction techniques are needed: in this example spatial track information is reduced to a concise and abstract domain i.e. a topological map Trajectories Map The space can be discretized using available observations with Topology Representing Networks (TRN) [15] algorithms Tracks are used to partition the space using vector quantization, clustering map areas and creating links between the centers of neighboring regions [15] T. M. Martinetz, Competitive hebbian learning rule forms perfectly topology preserving maps, ICANN 93, 1993, pp.427 434. 9
Map Generation ITM Algorithm The Instantaneous Topological Map (ITM) [16] is a TRN appropriate to discretize spatial trajectories and in general temporal correlated data. Given a set of trajectories observations P t ={p 1,,p t } the ITM Algorithm incrementally and online produces a set of nodes N and the corresponding connecting edges E by minimizing the quantization error: where v i ϵ R d (d<n) is the reference vector that represents the data x ϵ R n ; d(x, v i ) is the distance measure and used for minimization and l i are the Voronoi regions that partition the space [16] J. Jockusch and H. Ritter, An instantaneous topological map for correlated stimuli, in IJCNN, vol. 1, 1999, pp. 529 534. 10
Map Generation ITM Algorithm The coarseness τ of the topological map of the environment defines the semantic significance of detected events, i.e. transitions from a map zone to another. τ = 2000 τ = 400 τ = 100 In general not all the human behaviors are best recognized using the same topological decomposition of the environment. 11
Emphatic interaction modeling and recognition by trajectory analysis The causal relationships between the behavior of two entities can be encoded in two conditional probability densities (CPDs) that describe coupling of movements of two interacting entities, i.e. an Autobiographical Memory : To train the DBN a sequences of coupled proto and core events related to the two moving people are collected first using a simulator and then updated using real video sequences. ε ={ε 1P ; ε 1C ; ε 2P ; ε 2C ; ; ε NP ; ε NC } 12
Emphatic interaction modeling and recognition by trajectory analysis The two CPDs that represent the causal relationships can be derived into an AM t model the coupled DBN by considering as input data only sequences of triplets For example a smoothing voting procedure (like Parzen method) can be used to estimate the frequency of occurrence of the events ε t P(j,k) and ε t C(n,o) considering the triplets 13
Emphatic interaction modeling and recognition by trajectory analysis The time of occurrence of events associated with triplets can also be learned to store probabilistic information about the relative time of occurrence of causal events within a triplet. The probability distribution of the time of the second and third event in a triplet from the first event is represented by a Gaussian Mixture Model (GMM) The GMM is estimated from observed data using the algorithm proposed by Figueiredo and Jain [17] that automatically selects the number of components [17] M. A. T. Figueiredo and A. K. Jain, (2002) Unsupervised learning of finite mixture models, IEEE PAMI, vol. 24, no. 3, pp. 381 396, 2002. 14
Results Different types of interactions have been taken into account: guardian-intruder trajectories (GI); meeting trajectories (MM); meet-and-leave trajectory (ML); non interacting trajectories (NI); running (RR); running-walk (RW). A context generator has been designed in order to generate a sufficient number of samples to create the map and train the DBNs. 15
Results Real world trajectory of two meeting people Meeting Moving Together 16
Results Classification of Guard Intruder trajectory combining subsequent triplets Trajectory Classification 0,6 0,533061 0,5 0,4 0,327947 0,3 Probability 0,2 0,1 0,0500933 0 0,0428111 0,0460875 0 NI GI MM ML RR RW 17
Results Classification of Meet & Leave trajectory combining subsequent triplets Trajectory Classification 0,6 0,525739 0,5 0,4 0,3 0,286959 Probability 0,2 0,1 0 0,0993323 0,0774359 0,00841864 0,00211551 NI GI MM ML RR RW 18
Mapping onto a ICE Cognitive Node Event detection: the topological map is divided in a set of zones Z1 Zp in order to obtain a schematic description of the environment. An event Є(b,c,t,v) is defined as a zone transition from zone Zb to zone Zc of a moving object having speed v at time t. In particular are defined: proto events related to cooperative cognitive entities (operators) Єp(bp,cp,tp,vp) core events related to non cooperative cognitive entities (potential suspicious people) Єc(bc,cc,tc,vc) Interaction assessment: it is possible to define an interaction (IB) between two or more entities in case events caused by one or more of such entities are influenced by events related to other entities. An interaction is thus defined by the temporal sequence of events representing invovled entities. I = {Єc(bc,cc,t1,vc), Єp(bp,cp,t2,vp), } t1 t2
Cognitive Security/Safety Anomaly detection: the interaction l is classified according to acquired experience by the cognitive system stored and coded in the Autobiographical Memory of each node. Strategy selection: within a set of available strategies S = {s1, s2,,sn} the optimal strategy so is selected to face a potential dangerous situation. Action selection: the strategy so = {a0 an} is defined as the set of potential actions selected by the system in any possible incoming situation. The proper action ao is selected and suggested to the operators acting in the environment or directly sent to the actuators.