Multivariate Quantiles for Rapid Spatio-temporal Threat Detection

Transcription

1 Multivariate Quantiles for Rapid Spatio-temporal Threat Detection Snigdhansu Chatterjee 1 Benjamin E. Bagozzi 2 Ujjal Kumar Mukherjee 3 1 School of Statistics, University of Minnesota 2 Dept. of Political Science & International Relations, University of Delaware 3 Dept. of Business Administration, University of Illinois This research is partially supported by the National Science Foundation with grants DMS , , and

2 Univariate quantiles Suppose X R is a random variable. For any α (0, 1), the α th quantile Q α is the number below which X is observed with probability α, i.e. Q α = inf{q : P [X q] α. Theorem If X is (absolutely) continuous with cumulative distribution function F ( ), then F (X ) Uniform(0, 1), and there is a one-to-one relationship between α and Q α. We should not use this co-ordinatewise for multivariate data, since this ignores all dependency patterns and is inferior for statistical inference.

3 Univariate quantiles: an alternative view The median is the (unique) minimizer of Ψ(q) = E X q.

4 Univariate quantiles: an alternative view The median is the (unique) minimizer of Ψ(q) = E X q. (An extension) The α th quantile Q α is the (unique) minimizer of Ψ(q) = E{ X q + (2α 1)(X q)}.

5 Univariate quantiles: an alternative view The median is the (unique) minimizer of Ψ(q) = E X q. (An extension) The α th quantile Q α is the (unique) minimizer of Ψ(q) = E{ X q + (2α 1)(X q)}. (Alternative notation) Define u = 2α 1 ( 1, 1). The u th quantile Q u is the (unique) minimizer of Ψ(q) = E{ X q + u(x q)}

6 Univariate quantiles: an alternative view The median is the (unique) minimizer of Ψ(q) = E X q. (An extension) The α th quantile Q α is the (unique) minimizer of Ψ(q) = E{ X q + (2α 1)(X q)}. (Alternative notation) Define u = 2α 1 ( 1, 1). The u th quantile Q u is the (unique) minimizer of Ψ(q) = E{ X q + u(x q)} = E{ X q + u T (X q)}

7 Univariate quantiles: an alternative view The median is the (unique) minimizer of Ψ(q) = E X q. (An extension) The α th quantile Q α is the (unique) minimizer of Ψ(q) = E{ X q + (2α 1)(X q)}. (Alternative notation) Define u = 2α 1 ( 1, 1). The u th quantile Q u is the (unique) minimizer of Ψ(q) = E{ X q + u(x q)} = E{ X q + u T (X q)} = E{ X q + < u, X q >}.

8 Univariate quantiles: an alternative view The median is the (unique) minimizer of Ψ(q) = E X q. (An extension) The α th quantile Q α is the (unique) minimizer of Ψ(q) = E{ X q + (2α 1)(X q)}. (Alternative notation) Define u = 2α 1 ( 1, 1). The u th quantile Q u is the (unique) minimizer of Ψ(q) = E{ X q + u(x q)} = E{ X q + u T (X q)} = E{ X q + < u, X q >}. Define quantiles in any inner-product space as minimizers of Ψ u (q) = E{ X q + < u, X q >}. (Haldane (1948), Chaudhuri (1996).)

9 Univariate to multivariate quantiles Univariate quantiles: For every u {x : x < 1} R, Q(u) minimizes Ψ u (q) = E [ X q + < u, X q >]. Write x = x u u/ u + x u. Generally, for some λ 0, the generalized spatial quantile (GSQ) are: 1. indexed by vectors in the unit ball u B p = {x : x < 1}, and 2. the u-th quantile Q(u) is the minimizer of [ Ψ uλ (q) = E X u q u { 1 + λ(x u q u ) 2 X u q u 2} 1/2 ] + u (X u q u ).

10 Normal probability density function Standard Normal Distribution

11 Normal quantile Standard Normal Distribution Q(0.9)=1.68 Support of Distn

12 Univariate quantile mapping Probabilities Support of Distn Q(0.9)=

13 Bivariate quantiles Domain u Support of Distn Q(u)

14 Earlier work: Bahadur representation of GSQ Theorem The following asymptotic Bahadur-type representation holds with probability 1 for any u: n 1/2 ( ˆQ(u) Q(u)) = n 1/2 H 1 S n + O(n (1+s)/4 (log n) 1/2 (log log n) (1+s)/4 ) as n. (Apologies for not including the details.)

15 Projection quantiles Generalized spatial quantiles minimize: [ Ψ uλ (q) = E X u q { u 1 + λ(x u q u) 2 X u q u 2} ] 1/2 + u (Xu q u). Set λ = 0 to get projection quantiles. Computationally extremely simple, no limitations from sample size and dimension (high p, low n allowed). Projection quantiles based confidence sets have exact coverage. Works on infinite-dimensional spaces.

16 Earlier work: Projection quantiles Theorem Projection quantiles have a one-to-one relationship with the unit ball, like univariate quantiles.

17 Example: simulated data plots Figure: Simulated data with a few GSQ (covered areas are deliberately different)

18 What is data-depth? Suppose F is a cumulative distribution function corresponding to the random variable X R p. A data-depth is a function of R p and measures on R p such that there exists θ R p such that D(θ, F) D(θ + t(x θ), F) for any x R p and and any t (0, 1). Multivariate quantiles naturally yield data-depths.

19 Example: a depth plot Z X Y 2 1 Y X Z X Y Y X Figure: Perspective and contour plots of projection depth on top, simplicial at bottom.

20 GSQ-depths are great for classification Figure: A simulated 2-class classification problem with GSQ-depth classifier

21 GSQ-depth based classification: some results Method CPU Time Accuracy GSQ Random Forest SVM LDA Logit Table: Arcene classification without feature selection (neural nets did not converge)

22 Our proposal: Rapid spatio-temporal threat detection We propose to use multivariate quantiles and data depths for rapid, spatio-temporal, threat detection, and for characterizing what is normal. We are not reliant on an assumed geometry of the data. Since projection quantiles can be computed very fast on high-dimensional data, rapid detection is possible.

23 Applications: Spatio-temporal Political Violence Political event data, machine coded from newswire reports Measures who did what to whom, and where/when E.g.: civilian government protest; rebel military bombings Applications: protest & anti-gov. violence in Mexico & Africa. Daily/weekly/monthly data geolocated to municipalities or cities Monthly Instances of Anti Gov Violence in Mexico Anti Government Violence Time

24 Applications: Predictors of Political Violence Large numbers spati-temporal predictors encompassing: Event data-derived pentaclass measures of (material and verbal) conflict and cooperation among various actor pairings (e.g., criminal Mexican civilian; Mexican civilian Mexican gov.) Social, political, and demographic measures: homicides, population, elections, economic deprivation, commodities prices & trade, etc. Text-based social data exhaust, collected from online sources: Electronic access to information (ATI) requests made by non-u.s. citizens to their national governments (e.g., Mexican gov. agencies) Geo-tagged twitter data pertaining to civil unrest (e.g., food and water security concerns within urban areas in Africa)