From Language towards Formal Spatial Calculi Parisa Kordjamshidi Martijn Van Otterlo Marie-Francine Moens Katholieke Universiteit Leuven Computer Science Department CoSLI August 2010 1
Introduction Goal: What is placed Where Key points Multimodal environment Unrestricted language Formal spatial representation Relational machine learning Spatial extraction and reasoning Current experiments & Future plans 2
Motivation(Multimodal Environment) 1. so from here exactly opposite is my desk. 2. and next to that left of that is my computer, perhaps a meter away. 3. (breathing) ähm. 4. next to that at the wall is my kitchen, first there is my fridge all the way to the right. Room Description [Bateman, et.al, 2006] 3
Two level of semantics Language side and Spatial Role Labeling: Automatic labeling of natural language with a set of spatial roles. Space side and Spatial Formalizing Automatic mapping of the output of SpRL to formal relations in spatial calculi. 4
Spatial role labeling task Trajectory Landmark Landmark Give me that book on the table in the living room. ON(book,table) IN(table, living room) Spatial Indicator Trajectory Spatial Indicator IN(book,living room)? 5
Labeling the parse tree 6
Holistic spatial semantics The entity whose location or motion is of relevance. A binary component; whether there is perceived motion or not. Motion Trajector Landmark Path The reference entity in relation to which the location or motion of the trajector is determined. In terms of its beginning, middle and end. The direction along the axes provided by the different frames of reference. Direction Frame of Reference Region A region of space which is defined in relation to a landmark. Three types of frame of reference: intrinsic, relative or absolute. 7
Relational representation TRAJECTOR(id, token) LANDMARK(id, token, path) SPATIAL-INDICATOR(id, token,general-type, specific-type, spatial-value) MOTION-INDICATOR(id, token) SR(id, trajector, landmark, spatial-indicator, frame-of-reference, motion-indicator) 8
(Example: The vase is in the living room on the table under the window.) <TRAJECTOR id= 1 > The vase </TRAJECTOR> <LANDMARK id= 1 path= ZERO > the living room </LANDMARK> <LANDMARK id= 2 path= ZERO > the table </LANDMARK> <LANDMARK id= 3 path= ZERO >the window </LANDMARK > <SPATIAL-INDICATOR id= 1 general-type= REGION specific-type= RCC8 spatial-value= NTPP > in </SPATIAL-INDICATOR> <SPATIAL-INDICATOR id= 2 general-type= REGION specific-type= RCC8 spatial-value= EC > on </SPATIAL-INDICATOR> <SPATIAL-INDICATOR id= 3 general-type= DIRECTION specific-type= RELATIVE spatial-value= BELOW > under </SPATIAL-INDICATOR> <SR id= 1 trajector= 1 landmark= 1 spatial-indicator= 1 frame-of-reference= INTRINSIC motion-indicator= NIL /> <SR id= 2 trajector= 1 landmark= 2 spatial-indicator= 2 frame-of-reference= INTRINSIC motion-indicator= NIL /> <SR id=3 trajector=1 landmark=3 spatial-indicator=3 frame-of-reference=intrinsic motionindicator=nil/> [Ref. LREC2010] 9
Spatial reasoning Extraction for Reasoning or Reasoning for Extraction? The vase is on the table to your left. on(vase,table) left(table, you) left(vase,you)? The vase is on the ground to your left. on(vase, ground) left(vase,you) on(you,ground)? 10
Mapping to formal spatial relations (Spatial formalizing) Spatial Relations Region Direction Distance Qualitative RCC8 Absolute Relative Quantitative close... PO DC TPP-1 TPP EC NTPP NTPP-1 EQ West South East North SW SW NW LEFT RIGHT FRONT BELOW BEHIND ABOVE 30 km... SE 11
GUM ontology 12
Mapping to spatial calculi and next to that left of that is my computer, perhaps a meter away. Computer Desk LeftProjectionExt [distance: 1m] next-to(computer,that) DC(computer,that) left-of(computer,that) LEFT(computer, that) a-meter-away(computer,that) DC(computer,that) + In this way: Distance(computer, that, a meter) first: For reasoning we can gain reasoning based on RCC+ Directions+ Distances. It means a combination of spatial calculus. second: These extracted relations are uncertain, we need learning from examples + probabilistic logical inference. third: these capacities are gathered in statistical relational learning
Representation towards spatial reasoning 1. opposite(desk, here) FRONT(desk, here) 2. next-to(computer,that) DC(computer,that) left-of(computer,that) LEFT(computer, that) a-meter-away(computer,that) DC(computer,that) + Distance(computer, that, a meter) 4. next-to(kitchen,computer) DC(kitchen,computer) at(kitchen, wall) EC(kitchen,wall) first(fridge,nil) ClOSE( fridge, you) right(fridge,you) RIGHT(fridge,you) 5. In(water-can, corner) TPP(water-can, corner) In(flowers, corner) TPP(flowers, corner) 1. er, so from here exactly opposite is my desk. 2. and next to that left of that is my computer, perhaps a meter away. 3. (breathing) ähm, (1) [er]. 4. next to that at the wall is my kitchen, first there is my fridge all the way to the right. 5. in the corner where there s also my flowers, is my watering can. 6. and there is my stove, and the table with my sugar bowl is also at the side of the fridge. 7. left of that. 8. yes, and er - so to say behind me but left, er, is the entrance door 9. then further to the right of me directly is my dining table. 10. and er, how is it,from my position let s say between er twelve o clock and three o clock there is directly the TV set 11. and er, to the right of that is my sofa,which I cannot sit on you know, only the guests 12. and also my,er, beautiful coffee table. 14
Representation towards spatial reasoning Utterance Locatum Relatum GUM Category 1 Desk Self NonprojectionAxial: opposite 2 Computer Desk LeftProjectionExt [distance: 1m] 4 Kitchen Computer HorizontalProjectionExt: next 4 Kitchen Wall ExternalConnection: at 4 Fridge Kitchen RightProjectionInt: rightmost 4 Fridge Corner Containment: in 5 Houseplant Corner Utterance Containment: NL Spatial in relation Formal Spatial Relation 6 Stove There 1 opposite(desk, ExternalConnection: here) at FRONT(desk, here) 6 {Stove, kitchen table} Fridge 2 next-to(computer,that) HorizontalProjectionExt: side of DC(computer,that) 6 7 {Stove, kitchen table} Fridge left-of(computer,that) LeftProjectionExt LEFT(computer, that) 9 Entrance Self BackProjectionExt a-meter-away(computer,that) DC(computer,that) + 10 Dining table Self RightProjectionExt Distance(computer, that, a meter) 4 next-to(kitchen,computer) DC(kitchen,computer) at(kitchen, wall) EC(kitchen,wall) first(fridge, NIL) CLOSE( fridge, you) right(fridge,you) RIGHT(fridge,you) 5 In(water-can, corner) TPP(water-can, corner) In(flowers, corner) TPP(flowers, corner) 15
Grounded spatial communication A table-top scene : the apples are on the table A street-scale scene : the car is in front of the church 1. Selection of appropriate reference object(s) 2. Adoption of appropriate reference frame 3. Use of correct spatial prepositions 4.... 16
Spatial Puzzle Text Spatial Role Labeling Mapping NL to Spatial Ontology Mapping to Formal Spatial Relations Discourse Level Extracts Spatial Reasoning Grounding Spatial Relations Image/ Video 17
Current experiments (Preposition disambiguation) The book is on the table. TPP Data on: spatial sense Sentence Linguistically motivated features extracted from parser, dependency tree, semantic role labeler. Preposition Disambigation using Machine Learning(Mallet) The meeting is on Monday. Preposition with spatial sense on: temporal sense 18
Current experiments (Extraction of trajector and landmark) GUM Data Sentence Linguistically motivated features extracted from parser, dependency tree, semantic role labeler. Argument Identification and Classification The book is on the table. Classify the words with respect to the detected spatial indicator on. Spatial Indicator Word-based Constituent-based Sentence (parse tree) labeled with spatial roles Word-based the: None book: Trajector is : None the : None table: Landmark Sequence-tagging 19
Current experiments Preposition disambiguation gaining 87.4% accuracy, outperforming the previous works by 1.3% increase. SemEval07 data (Litkowski and Hargraves, 2007).. gaining 88% accuracy over spatial prepositions. Extraction of trajector and landmark 20
Conclusion & Future directions Conclusion Introduced two semantic levels for mapping language to spatial calculi Covering spatial semantics including dynamic and static spatial information Promising experimental results 21
Conclusion & Future directions Ongoing work and Future directions Getting annotated corpus for the whole scheme (currently working on CLEF, IAPRTC-12 Image Benchmark) Statistical relational learning for the whole automatic mapping (injecting background knowledge to the system in a logical formalization) Spatial reasoning, combining multimodal information 22
Thank you! Questions? 23