Perfect Power Law Graphs: Generation, Sampling, Construction, and Fitting

Transcription

1 Perfect Power Law Graphs: Generation, Sampling, Construction, and Fitting Jeremy Kepner SIAM Annual Meeting, Minneapolis, July 9, 2012 This work is sponsored by the Department of the Air Force under Air Force Contract #FA C Opinions, interpretations, recommendations and conclusions are those of the authors and are not necessarily endorsed by the United States Government.

2 Outline Introduction Sampling Sub-sampling Reuter s Data Summary D4M- 2

3 Goals Develop a background model for graphs based on perfect power law Examine effects of sampling such a power law Develop techniques for comparing real data with a power law model D4M- 3

4 Detection Theory DETECTION OF SIGNAL IN NOISE THRESHOLD DETECTION OF SUBGRAPHS IN GRAPHS Example subgraph of interest: Fully connected (complete) SIGNAL NOISE SIGNAL H 0 H 1 NOISE N-D SPACE ASSUMPTIONS Background (noise) statistics Foreground (signal) statistics Foreground/background separation Model reality Example background model: Powerlaw graph Can we construct a background model based on power law degree distribution? D4M- 4

5 Perfect Power Law Matrix Definition N out Adjacency/Incidence Matrix N in A M = ΣA edges n(d out ) number of vertices Vertex Out Degree Distribution -α out n(d in ) number of vertices Vertex In Degree Distribution -α in in degree, d in out degree, d out Graph represented as a rectangular sparse matrix Can be undirected, multi-edged, self-loops, disconnected, hyper edges, Out/in degree distributions are independent first order statistics Only constraint: Σ n(d out ) d out = Σ n(d in ) d in = M D4M- 5

6 Power Law Distribution Construction Perfect power law matlab code n 1 function [di ni] = PPL(alpha,dmax,Nd)! logdi = (0:Nd) * log(dmax) / Nd;! di = unique(round(exp(logdi)));! logni = alpha * (log(dmax) - log(di));! ni = round(exp(logni));! n(d i ) = n 1 /d i α Parameters alpha = slope dmax = largest degree vertex Nd = number of bins (before unique) d max integer logarithmic Simple algorithm naturally generates perfect power law Smooth transition from integer to logarithmic bins Poor man s slope estimator: α = log(n 1 )/log(d max ) D4M- 6

7 Power Law Edge Construction Power law vertex list matlab code! function v = PowerLawEdges(di,ni);! A1 = sparse(1:numel(di),ni,di);! A2 = fliplr(cumsum(fliplr(a1),2));! [tmp tmp d] = find(a2);! A3 = sparse(1:numel(d),d,1);! A4 = fliplr(cumsum(fliplr(a3),2));! [v tmp tmp] = find(a4);! Degree distribution independent of Vertex labels Edge pairing Edge order random edge pairs random vertex labels Algorithm generates list of vertices corresponding to any distribution All other aspects of graph can be set based on desired properties D4M- 7

8 Fitting α, N, M Power law model works for any α > 0, d max > 1, N d > Allowed N and M for α = 1.3 Desire distribution that fits α, N, M Can invert formulas N = Σ i n(d i ) M = Σ i n(d i ) d i M M Highly non-linear; requires a combination of Exhaustive search, simulated annealing, and Broyden s algorithm Given α, N, M can solve for N d and d max Not all combinations of α, N, M are consistent with power law 10 2 N N D4M- 8

9 Outline Introduction Sampling Sub-sampling Reuter s Data Summary D4M- 9

10 Graph Construction Effects Generate a perfect power law NxN randomize adjacency matrix A α = 1.3, d max = 1000, N d = 50 N = 18K, M = 84K Make undirected, unweighted, with no self-loops A = triu(a + A );! A = double(logical(a));! A = A - diag(diag(a)); count 10 4 measured input model degree Graph theory best for undirected, unweighted graphs with no self-loops Often clean up real data to apply graph theory results Process mimics bent broom distribution seen in real data sets D4M- 10

11 Power Law Recovery Procedure Compute α, N, M from measured Fit perfect power law to these parameters Rebin measured data using perfect power law degree bins count 10 4 measured α 10 3 model fit rebin degree Perfect power law fit to cleaned up graph can recover much of the shape of the original distribution D4M- 11

12 Correlation Construction Effects Generate a perfect power law NxN randomize incidence matrix E α = 1.3, d max = 1000, N d = 50 N = 18K, M = 84K Make unweighted and use to form correlation matrix A with no self-loops count 10 4 measured input model E = double(logical(e));! A = triu(e * E);! A = A - diag(diag(a)); degree Correlation graph construction from incidence matrix results in a bent broom distribution that strongly resembles a power law D4M- 12

13 Power Law Lost Procedure Compute α, N, M from measured Fit perfect power law to these parameters Rebin measured data using perfect power law degree bins count 10 4 measured α model fit 10 3 rebin degree Perfect power law fit to correlation shows non-power law shape Reveals witches nose distribution D4M- 13

14 Power Law Preserved In degree is power law α = 1.3, d max = 1000, N d = 50 N = 18K, M = 84K Out degree is constant N = 16K, M = 84K Edges/row = 5 (exactly) count 10 4 measured α model fit rebin Make unweighted and use to form correlation matrix A with no self-loops degree Uniform distribution on correlated dimension preserves power law shape D4M- 14

15 Edge Ordering: Densification Compute M/N cumulatively and piecewise for 2 orderings Linear Random By definition M/N goes from 1 to infinity for finite N M/N Linear Elimination of multi-edges reduces M and causes M/N to grow more slowly 2 random fraction of total edges Densification is the observation that M/N increases with N Densification is a natural byproduct of randomly drawing edges from a power law distribution Linear ordering has constant M/N D4M- 15

16 Edge Ordering: Power Law Exponent (α) Compute α cumulatively and piecewise for 2 orderings Linear Random α Edge ordering and sampling have large effect on the power law exponent random random cumulative linear cumulative linear fraction of total edges Power law exponent is fundamental to distribution Strongly dependent on edge ordering and sample size D4M- 16

17 Outline Introduction Sampling Sub-sampling Reuter s Data Summary D4M- 17

18 Sub-Sampling Challenge Anomaly detection requires good estimates of background Traversing entire data sets to compute background counts is increasingly prohibitive Can be done at ingest, but often is not Can background be accurately estimated from a sub-sample of the entire data set? D4M- 18

19 Sampling a Power Law Whole distribution Generate power law Select fraction of edges 1/40 sample D4M- 19

20 Linear Degree Estimate Whole distribution Divide measured degree by fraction Accurate for high degree Overestimates low degree Can we do better? Linear estimate D4M- 20

21 Non-Linear Degree Estimate Whole distribution Assume power law input Create non-linear estimate Matches median degree Non-Linear estimate D4M- 21

22 Outline Introduction Sampling Sub-sampling Reuter s Data Summary D4M- 22

23 Reuter s Incidence Matrix Entities extracted from Reuter s Corpus E(i,j) = # times entity appeared in document LOCATION ORGANIZTION PERSON TIME _2304.txt _12989.txt _23589.txt _33974.txt _44444.txt _54917.txt _65253.txt _75717.txt DOCUMENT Ndoc = Nent = M = _86206.txt _96462.txt _ txt _ txt Four entity classes with different statistics _ txt E _ txt LOCATION ORGANZATION PERSON TIME _ txt _ txt _ txt _ txt _ txt _ txt LOCATION/addis LOCATION/el LOCATION/malaysia ababsalvado LOCATION/seychelles PERSON/abdullah PERSON/ashley PERSON/christian bad PERSON/drew giles PERSON/henry ma PERSON/jimmy peck PERSON/khaled wilson PERSON/michael hayes salam PERSON/paul PERSON/ryan pric wells PERSON/terrell beck TIME/ owen TIME/ TIME/ TIME/ TIME/ Fit power law model to each entity class D4M- 23

24 E(:,PERSON) Degree Distribution 10 6 E(:,PERSON) document distribution 10 5 measured α model fit rebin 10 5 E(:,PERSON) entity distribution 10 4 measured α model fit rebin n(d doc ) n(d ent ) d doc d ent M N M/N α M fit N fit M fit /N fit Document Entity D4M- 24

25 Procedure Make unweighted and use to form correlation matrix A with no selfloops E(:,PERSON) t x E(:,PERSON) E = double(logical(e));! A = triu(e * E);! A = A - diag(diag(a)); Perfect power law fit to correlation shows non-power law shape Reveals witches nose distribution D4M- 25

26 Document Densification Constant M/N consistent with sequential ordering of documents D4M- 26

27 Entity Densification Increasing M/N consistent with random ordering of entities D4M- 27

28 Document Power Law Exponent (α) location organization person time α document fraction of total edges Increasing α consistent with sequential ordering of documents D4M- 28

29 Entity Power Law Exponent (α) location organization person time 1.5 α entity fraction of total edges Decreasing α consistent with random ordering of entities D4M- 29

30 Summary Developed a background model for graphs based on perfect power law Can be done via simple heuristic Reproduces much of observed phenomena Examine effects of sampling such a power law Lossy, non-linear transformation of graph construction mirrors many observed phenomena Traditional sampling approaches significantly overestimate the probability of low degree vertices Assuming a power law distribution it is possible to construct a simple non-linear estimate that is more accurate Develop techniques for comparing real data with a power law model Can fit perfect power-law to observed data Provided binning for statistical tests D4M- 30

31 Acknowledgements Nicholas Arcolano Michelle Beard Nadya Bliss Bob Bond Matthew Schmidt Ben Miller Bill Arcand Bill Bergeron David Bestor Chansup Byun, Matt Hubbell Pete Michaleas Julie Mullen Andy Prout Albert Reuther Tony Rosa Charles Yee D4M- 31

32 D4M- 32 Appendix

33 Sub-Sampling Formula f = fraction of total edges sampled n 1 = # of vertices of degree 1 d max = maximum degree Allowed slope: ln(n 1 )/ln(d max /f) < α < ln(n 1 )/ln(d max ) Cumulative distribution P(α,d) = (f 1-α d max α / n 1 ) Σ i<d i 1-α e -fi Find α* such that P(α*, ) = 1 Find d 50% such that P(α*,d 50% ) = ½ Compute K = 1/(1 + ln(d 50% )/ln(f)) Non-linear estimate of true degree of vertex v from sample d(v) d(v) = d(v) / f1-1/(k d(v)) D4M- 33