ONR Mine Warfare Autonomy Virtual Program Review September 7, 2017

Transcription

1 ONR Mine Warfare Autonomy Virtual Program Review September 7, 2017 Information-driven Guidance and Control for Adaptive Target Detection and Classification Silvia Ferrari Pingping Zhu and Bo Fu Mechanical and Aerospace Engineering Cornell University

2 Outline Introduction MCM Motivation Directional Information Gain UUV-sonar Imaging Frames of Reference UUV-sonar Feature Extraction CNN-SVM ATR UUV-sonar Bayesian Modeling UUV-sonar Information Value Learning Conclusions and Q&A 2

3 Vehicle Path Planning and Control Traditional paradigm: Proprioceptive and exteroceptive sensor (output) used as feedback to vehicle in support of vehicle navigation objectives. Guidance Vehicle Output Control 3

4 Sensor Navigation and Control New paradigm: Vehicle is used to gather information (output) to support sensing objectives, such as target acquisition, or DCLT. Guidance Sensor Output Control Research challenges: Represent sensor objectives in closed-form Computational geometry; information theory Environmental and target feedback (output) Significant uncertainties; Bayesian updates Information-driven guidance and control Couplings between sensor measurements and vehicle dynamics 4

5 MCM Motivation and Application 5

6 Classification-driven Path Planning

7 Influence of UUV Position and Orientation Forward-looking Sonar: Side-scan Sonar: The sensor FOV, denoted by S 3, is defined as a compact subset of the workspace, W, in which the robot can obtain sonar measurements. Motivation: Sonar is directional and the information gain depends on the sonar FOV geometry, position, and orientation relative to the target.

8 Surveillance Region and Oceanic Currents LISC Real CODAR-Measured Current Field (100 naut mi-nj Coast) [COOL, Rutgers University] 8

9 Directional Information Gain 9

10 Visibility Problem For a polygonal obstacle B in a workspace W, consider a sensor on A observing a target T. Determine obstacle-generated cone K. Then, for a sensor FOV, S, we seek to find a shadow region, D, such that the visibility region is R Rs Solution: Construct a polygon with sensor at point P: S = {P,V 1,,V N }, where V 1,,V N are the obstacle vertices. Compute the convex hull of S: S S CS conv( S) kixi ( i : ki 0 ki i 1 i 1 Visibility region for one target: R = CS\B R S = (S\K) ((S K)\(D B)) 1)

11 Visibility Region in Closed Form Obstacle cone in closed-form: Construct unit vectors k i, k j Find boundary unit vectors k 1 and k 2 : k 1 PV PV p p k 2 PV PV q q Compute obstacle cone K: K cone( kˆ, kˆ ) { ˆ ˆ 2 x x r c1k 1 c2k 2, c1, c2 D (S K)\(CS) 1 Visibility region for multiple targets (i) and obstacles (j): 0} R (i) = CS (i) \B (i); D (i) = (S K (i) )\CS (i) R S = i R S(i) = i {(S\K (i) ) ((S K (i) )\(D (i) B (i) )}

12 Distance in meters 5 10 Distance in meters 5 10 Example: Art Gallery Problem Sensor at Single Location Sensor at Multiple Location T Distance in meters Distance in meters :B :T :D :K :R s

13 UUV-Sonar Directional Information Gain 13

14 UUV Kinematics Frames of Reference F W y uuv (North) θ uuv O A Assume: v = constant UUV States: x(t)= [x uuv (t), y uuv (t), uuv (t)] T UUV Dynamics: x v cos( y uuv uuv v sin( uuv uuv ) ) F A (East) O W x uuv

15 Distance in meters UUV Frames Relative to Sonar Image UUV Heading Direction Target: T UUV at x(t 2 ), t 2 > t 1 r O A Vehicle Frame F A = F A (x(t)) UUV at x(t 1 ) O A Sonar FOV FOV = FOV(x(t)) Distance in meters

16 Vehicle Frame and Image Frame Consider the image segmentation as the target geometry T. Then, the actual target geometry, orientation, and shape are viewed as hidden random features. (North) UUV Heading θ uuv T O A O T O A O T F A : Vehicle Frame F T : Image Frame T = T (x): Target Image Region Distance in meters

17 Image Frame and Target Frame F W (North) F T : Image Frame θ uuv θ T F T : Target Frame y T y T O T O T T θ Aspect Angle: θ = θ uuv - θ T (East) O W x T x T

18 Image and Target Distance Vectors F W (North) F A : Vehicle Frame O A r F T : Image Frame F T : Target Frame r i O T r O T True Distance: r = r i + r = [(x T - x), (y T - y)] T (East) O W

19 UUV-Sonar Feature Extraction 19

20 Automatic Target Recognition an Detection Training Phase: Train SVM classifier with Sonar Image I s and true classification Y T Testing Phase: Matched filter for image segmentation and CNN+SVM for target classification

21 CNN for Sonar Image Features Extraction Convolution + ReLU Pooling Fully-Connected (FC) Layer Input Image Convolution + ReLU Pooling I s Z R 4096 Automatic segmentation is performed via matched filter Matched filter provides better performance than Markov random fields (MRFs) Deep learning features extracted from segmentation by Pre-trained AlexNet AlexNet CNN provides better performance than other features extraction techniques such as HOG and LBP.

22 CNN Performance: Classification Accuracy Legend: TP: True Positive FP: False Positive TN: True Negative FN: False Negative G i : Image Group i Classification Accuracy: CA n TP n n TP FN n n TN FP n TN Probability of detection (Matched Filter performance): 88.31% * Zhu, P., Issacs, J., Fu, B., Ferrari, S., Deep Learning Feature Extraction for Target Recognition and Classification in Underwater Sonar Images 56th IEEE Conference on Decision and Control, Melbourne, Australia (Accepted).

23 CNN Performance: True Positive Rate Legend: TP: True Positive FP: False Positive TN: True Negative FN: False Negative G i : Image Group i True Positive Rate (TPR): TPR n TP ntp n FP Probability of detection (Matched Filter performance): 88.31% * Zhu, P., Issacs, J., Fu, B., Ferrari, S., Deep Learning Feature Extraction for Target Recognition and Classification in Underwater Sonar Images 56th IEEE Conference on Decision and Control, Melbourne, Australia (Accepted).

24 CNN Performance: False Alarms Legend: TP: True Positive FP: False Positive TN: True Negative FN: False Negative G i : Image Group i False Positive Rate (FPR): FPR n FP nfp n TN Probability of detection (Matched Filter performance): 88.31% * Zhu, P., Issacs, J., Fu, B., Ferrari, S., Deep Learning Feature Extraction for Target Recognition and Classification in Underwater Sonar Images 56th IEEE Conference on Decision and Control, Melbourne, Australia (Accepted).

25 UUV-Sonar Information Value 25

26 Distance (m) Distance (m) UUV-Sonar Information Value F W Minefield: 520 Targets, 20 Mine objects Distance (m) Sonar Image Sonar Image Distance (m) O W *Courtesy of Jason Isaacs, NSWC Panama City, FL

27 Key Variable Description Name Description Name Description X uuv Vehicle Position X T Target Position uuv Vehicle Orientation T Target Orientation r = [ d, ] d Relative Distance Relative Orientation Y T True Classification Estimated Classification Ŷ Features F ={S,H,W,L} Assume Constant S Object Shape L Object Length W Object Width H Object Height I s Segmented Image z CNN output Features M Sensor Mode E Noise level

28 Key Variables and Causal Relationships Target Features Vehicle Path Planning Image Generation Information Value Function Learning CNN Features Extraction and SVM Classification

29 Information Value Function Learning Assume information gain can be modeled as Expected Entropy Reduction (EER) N 1 ( i) EER( yˆ pri, Fˆ, rˆ pri, rˆ post ) H pri ( yˆ pri, Fˆ, rˆ pri ) H post ( yˆ pri, yˆ ˆ ˆ post, Fˆ, rpri, rpost ) N i 1 Example: Prior Image location ˆ {[0,75),[5 / 4,7 / 4,)} r pri Target: rˆpost Posterior Image location ˆ {[75,150],[ / 4, / 4,)} r post ŷ pri rˆpri ( ) yˆ i post : given in training data : yˆ 1 : yˆ 0 Discretization: r = [d, ]; d = {[0,75), [75,150]}; = {[-π/4, π /4), [π /4, 3 π /4), [3 π /4, 5 π /4), [5 π /4, 7 π /4]}

30 Information Value Function Learning Continuous probabilities (PDF s) are difficult to learn, random variables are first discretized. Conditional probability table (CPT) is estimated from dataset: P( Yˆ Y,S, L, W, H, r) T Entropy H is calculated based on estimated conditional probability Bayesian Network Model, Features F = {S,H,W,L} Information Value Function (EER) is learned from the dataset images and Features.

31 Bayesian Network Model n Training data set: D {( i, li, wi, hi, di, i, yt, i, yˆ i )} i Random variables: S, L, W, H, D,, YT,andYˆ Realization variables: s, l, w, h, d,, y,and yˆ i i i i i i T s 1 Bayesian Network Model, Features F = {S,H,W,L} Probability parameters learned from dataset: Prior: P( Fˆ, rˆ) Likelihood: Y T P( yˆ, Fˆ, rˆ) Y T S = {Cylindrical, Rectangular}; V = (LWH) 1/3 = {[0, 0.14), [0.14, 0.3), [0.3, 1.1), [1.1, 1.7]} r = [d, ]; d = {[0,75), [75,150]}; = {[- /4, /4), [ /4, 3 /4), [3 /4, 5 /4), [5 /4, 7 /4]}

32 BN Parameters Learning Bayesian Network Model, Features F = {S,H,W,L} i : Variable Node index j: Parent configuration index k: Value index of variable r i : Number of values of variable N ijk : Count number in data set Bayesian parameter estimation: Parameters are random variables Multinomial likelihood of training data set: P(D ) Dirichlet Prior of the parameter: Hyper parameter: Posterior of the parameter: MAP estimate: N n ijk ikj ( k ijk ) P( α) k ( ijk ) Dirichlet( ; ikj ri ( ikj k 1 ( Nijk P( D, α) ikj ijk ˆ MAP ijk j N ' ijk ( N ij ijk 1),..., ijk ) ij1 ijri [ ij 1,..., ijr i ijk 1) ' ' ) ij k α ijk ij k ]

33 Posterior Probability Learning Posterior probabilities calculated via Bayes rule: Given image I pri P( Y T 1 yˆ pri, Fˆ, rˆ pri ) P( Y T 1 Fˆ, rˆ ˆ pri ) P( y pri Y P( yˆ Fˆ, rˆ ) pri pri T 1, Fˆ, rˆ pri ) Given image I pri and I post P( Y T P( Y T 1 yˆ pri 1 Fˆ, rˆ, yˆ pri post, rˆ, Fˆ, rˆ post ) P( yˆ P( yˆ, rˆ, yˆ Y ) post Instantiations can also be calculated: P ˆ pri pri pri post 1, Fˆ, rˆ ) ( ˆ pri P y Fˆ, rˆ, rˆ ) post F ˆ, rˆ ) and P( yˆ pri, yˆ post Fˆ, rˆ, ˆ pri rpost ) ( y pri pri T pri post Y T 1, Fˆ, rˆ post )

34 Results 34

35 Probability of Correct Classification Probability of Correct Classification Influence of UUV position on Classification Dataset A: Low Frequency Probability of correct classification (total): Cylindrical object 91.11% Rectangular object 83.02% Relative Distance d (m) Relative Orientation (rad)

36 Probability of Correct Classification Probability of Correct Classification Influence of UUV position on Classification Dataset A: High Frequency Probability of correct classification (total): Cylindrical object 87.04% Rectangular object 86.27% Relative Distance d (m) Relative Orientation (rad)

37 Probability of Correct Classification Probability of Correct Classification Influence of UUV position on Classification Dataset B: Low Frequency Probability of correct classification (total): Cylindrical object 70.97% Rectangular object 43.48% Relative Distance d (m) Relative Orientation (rad)

38 Probability of Correct Classification Probability of Correct Classification Influence of UUV position on Classification Dataset B: High Frequency Probability of correct classification (total): Rectangular object 86.23% Cylindrical object 77.03% Relative Distance d (m) Relative Orientation (rad)

39 Information Value Function Learning Assume information gain can be modeled as Expected Entropy Reduction (EER) N 1 ( i) EER( yˆ pri, Fˆ, rˆ pri, rˆ post ) H pri ( yˆ pri, Fˆ, rˆ pri ) H post ( yˆ pri, yˆ ˆ ˆ post, Fˆ, rpri, rpost ) N i 1 Example: Prior Image location ˆ {[0,75),[5 / 4,7 / 4,)} r pri Target: rˆpost Posterior Image location ˆ {[75,150],[ / 4, / 4,)} r post ŷ pri rˆpri ( ) yˆ i post : given in training data : yˆ 1 : yˆ 0 Discretization: r = [d, ]; d = {[0,75), [75,150]}; = {[-π/4, π /4), [π /4, 3 π /4), [3 π /4, 5 π /4), [5 π /4, 7 π /4]}

40 Distance in meters Distance in meters Results: Learned Information 150 Target #2: y T = 1, s = Rectangular, l = 1.25m, w = 0.61m, h = 0.47 m Correct classification ( ˆ 1) y pri EER Incorrect classification ( ˆ y pri 0) EER O T O T 0.1 O A O A Distance in meters Distance in meters F T : Target Frame F A : Vehicle Frame

41 Distance in meters Distance in meters Results: Learned Information Gain 150 Target #230: y T = 0, s = Circular, l = 0.05m, w = 0.20 m, h = 0.27 m Correct classification ( ˆ 0) Incorrect classification ( ˆ 1) y pri EER y pri EER O A O T O T 0.1 O A Distance in meters Distance in meters F T : Target Frame F A : Vehicle Frame

42 Probability of Correct Classification CNN-SVM Learning with limited data Percentage of Data used for Training

43 MCM Unmanned Autonomy Evaluation Simulator (MUAES)

44 Conclusions MCM information-driven path planning and control: Directional information gain Automatic CNN-SVM target recognition and detection New reference frames and problem formulation Bayesian model of UUV-sonar sensing problem Information gain function learning MUAES visualization and testing Ongoing and Future Work: Information-driven MCM acquisition and classification MCM path planning and control Validation in MUAES-Sonar Imaging 44

45 Acknowledgments Collaborators: Jason Isaacs, Ph.D. Naval Surface Warfare Center Panama City, FL Drew Lucas, Ph.D. Naval Surface Warfare Center Panama City, FL Matt Bays, Ph.D. Naval Surface Warfare Center Panama City, FL Pingping Zhu, Ph.D. LISC, MEMS, Duke Bo Fu, Ph.D. LISC, MAE, Cornell This work was funded by ONR Code 321OE, MCM Autonomy Program 45