High-dimensional probability and statistics for the data sciences

Size: px

Start display at page:

Download "High-dimensional probability and statistics for the data sciences"

Mervyn Flowers
5 years ago
Views:

1 High-dimensional probability and statistics for the data sciences Larry Goldstein and Mahdi Soltanolkotabi Motivation August 21, 2017 Motivation August 21, 2017

2 Please ask questions!

3 Traditionally mathematics driven by physics...

4 Traditionally mathematics driven by physics... Kepler s law of planetary motion

5 Traditionally mathematics driven by physics... Kepler s law of planetary motion Newtonian Mechanics

6 Traditionally mathematics driven by physics... Kepler s law of planetary motion Newtonian Mechanics General relativity

7 Todays mathematics is driven by something else...

8 Todays mathematics is driven by something else... Can you guess?

9 Todays mathematics is driven by something else... Can you guess?

10 What is different about the data we have today?

11 What is different about the data we have today? Size? Data deluge?

12 Ye Olde Data Deluge

13 What is different about the data we have today? variety and complexity

14 What is different about the data we have today? variety and complexity Web Data

15 What is different about the data we have today? variety and complexity Web Data Text Data Social network data Video Data Image data

16 What is different about the data we have today? variety and complexity

17 What is different about the data we have today? variety and complexity Scientific Data

18 What is different about the data we have today? variety and complexity Remote Sensing Data Brain Data Scientific Data Genomic Data Sensor Network Data

19 What is different about the data we have today? variety and complexity

20 What is different about the data we have today? variety and complexity variety of platforms

21 Challenge

22 Challenge Conclusion Scientific Data Has Become So Complex, We Have to Invent New Math to Deal With It

23 Need new Math... Mathematics Driven by Data

24 Probability at the heart of data science

25 Example I: Community detection and stochastic block models

26 Standard Machine 1 Construct affinity matrix W between samples weighted graph ( ) W i,j = exp x i x j 2 l 2 2σ 2 2 Construct clusters by applying spectral clustering to W.

27 Spectral clustering Affinity matrix W (N N) Degree matrix D=diag(d i ) d i = j W ij Normalized graph Laplacian (symmetric form) L = I D 1/2 W D 1/2 (N N)

28 Spectral clustering Affinity matrix W (N N) Degree matrix D=diag(d i ) d i = j W ij Normalized graph Laplacian (symmetric form) L = I D 1/2 W D 1/2 (N N) v 1 v 2 v 3 r 1 v 11 v 12 v 13 r 2 v 21 v 22 v r N v N1 v N2 v N3 Dim. reduction: N k Spectral clustering (1) Build V R N k with first k lowest eigenvectors of L as columns (2) Interpret ith row of V as new data point r i in R k representing observation i (3) Apply k-means clustering to the points {r i } Fantastic tutorial: U. von Luxburg

29 Spectral Clustering: Ideal case Graph EigVs rows in R 3

30 Spectral Clustering: non-ideal case Graph EigVs rows in R 3

31 Adjacency matrix of random graphs Adjacency matrix of Graph A ij = { 1 with prob. Pij 0 otherwise

32 Adjacency matrix of random graphs Adjacency matrix of Graph A ij = { 1 with prob. Pij 0 otherwise Probability matrix for stochastic block model k clusters of size n/k 0 q p 1.

33 Where s Waldo?

34 Where are the clusters? n = 200, k = 4, p = 0.7, q = 0.3.

35 Where are the clusters? n = 200, k = 4, p = 0.6, q = 0.4.

36 Where are the clusters? Permuted version n = 200, k = 4, p = 0.6, q = 0.4.

37 Will it work? Eigenvalues of the normalized Laplacian with n = 200, k = 4, p = 0.6, q = Eigenvalues index of eigenvalue

38 Eigen vectors Top two eigenvectors of the normalized Laplacian n = 200, k = 4, p = 0.6, q = second entry first entry 10 2

39 Will it work? Eigenvalues of the normalized Laplacian with n = 200, k = 4, p = 0.7, q = Eigenvalues index of eigenvalue

40 Eigen vectors Top two eigenvectors of the normalized Laplacian n = 200, k = 4, p = 0.7, q = nd entry first entry 10 2

41 Example II: Learning models from data and uniform concentration results

42 Learn model from training data Main abstraction of machine learning: parameter estimation Data (n samples) Features: x 1, x 2,..., x n R p. Response/class: y 1, y 2,..., y n. Input features Model with parameter θ Output Face Bicycle Guitar

43 Mathematical abstraction: Empirical Risk Minimization

44 Mathematical abstraction: Empirical Risk Minimization If we had infinite input/output data according to a distribution D θ = arg min θ Θ L(θ) := E (x,y) D [l(f θ (x), y)]

45 Mathematical abstraction: Empirical Risk Minimization If we had infinite input/output data according to a distribution D θ = arg min θ Θ L(θ) := E (x,y) D [l(f θ (x), y)] Given a data set (x i, y i ) R d R find the best function that fits this data ˆθ = arg min L(θ) := 1 n l(f θ (x i ), y i ) θ Θ n i=1

Mathematical abstraction: Empirical Risk Minimization If we had infinite input/output data according to a distribution D θ = arg min θ Θ L(θ) := E (x,y) D [l(f θ (x), y)] Given a data

46 Mathematical abstraction: Empirical Risk Minimization If we had infinite input/output data according to a distribution D θ = arg min θ Θ L(θ) := E (x,y) D [l(f θ (x), y)] Given a data set (x i, y i ) R d R find the best function that fits this data ˆθ = arg min L(θ) := 1 n l(f θ (x i ), y i ) θ Θ n How good is fˆ? That is a new sample x how well can f ˆ(x) estimate y? i=1

47 Uniform concentration For a fixed θ we know 1 n n l(f θ (x i ), y i ) E (x,y) D [l(f θ (x), y)]. i=1

48 Uniform concentration For a fixed θ we know Questions? 1 n n l(f θ (x i ), y i ) E (x,y) D [l(f θ (x), y)]. i=1 Can we guarantee this for all θ Θ? 1 n sup l(f θ (x i ), y i ) E (x,y) D [l(f θ (x), y)] n δ θ Θ i=1

49 Uniform concentration For a fixed θ we know Questions? 1 n n l(f θ (x i ), y i ) E (x,y) D [l(f θ (x), y)]. i=1 Can we guarantee this for all θ Θ? 1 n sup l(f θ (x i ), y i ) E (x,y) D [l(f θ (x), y)] n δ θ Θ i=1 much more difficult that proving this for one point asymptotically (law of large numbers)

50 Uniform concentration For a fixed θ we know Questions? 1 n n l(f θ (x i ), y i ) E (x,y) D [l(f θ (x), y)]. i=1 Can we guarantee this for all θ Θ? 1 n sup l(f θ (x i ), y i ) E (x,y) D [l(f θ (x), y)] n δ θ Θ i=1 much more difficult that proving this for one point asymptotically (law of large numbers) How many data samples do we need as a function of Θ, f and δ.

51 Course Logistics

52 Goals Learn modern techniques in probability Concentration in high-dimensions geared towards applications for data sciences, statistics and machine learning

53 Why is this course needed? Distinctions with other courses? Advanced probability courses (while very technical e.g. cover measure theory) do not cover some of the most useful techniques in modern probability Discuss analogous results to low-dimensions in high-dimensions (law of large numbers, concentration, etc.) Over the past 5-10 years there has been tremendous progress simplifying many proofs Focus on the most useful techniques

54 Background and disclaimer Prerequisites: EE 599 enrollees (EE 441 and EE 503) MATH 605 enrollees (MATH 505a or MATH 507a)

55 Background and disclaimer Prerequisites: EE 599 enrollees (EE 441 and EE 503) MATH 605 enrollees (MATH 505a or MATH 507a) More mathy than EE 441 and EE 503 (but not abstract e.g. don t care about measure theory, Hilbert spaces, and Banach spaces) We cover a lot of material and many applications

56 Background and disclaimer Prerequisites: EE 599 enrollees (EE 441 and EE 503) MATH 605 enrollees (MATH 505a or MATH 507a) More mathy than EE 441 and EE 503 (but not abstract e.g. don t care about measure theory, Hilbert spaces, and Banach spaces) We cover a lot of material and many applications Do I need to know these applications? Do I need to know measure theory or Morse theory? Do I need to be a math graduate student? Do I need to be an electrical engineering student?

57 Background and disclaimer Prerequisites: EE 599 enrollees (EE 441 and EE 503) MATH 605 enrollees (MATH 505a or MATH 507a) More mathy than EE 441 and EE 503 (but not abstract e.g. don t care about measure theory, Hilbert spaces, and Banach spaces) We cover a lot of material and many applications Do I need to know these applications? Do I need to know measure theory or Morse theory? Do I need to be a math graduate student? Do I need to be an electrical engineering student? Answer: Absolutely not.

58 Logistics Class: Mon, Wed 10:30-11:50 PM, VKC 256. Instructor office hours: Larry: Monday 12-1:30, Wednesday 3:30-5, KAP 406D Mahdi: Monday and Wednesday 5:30-7 PM EEB 422 Course website: blackboard Grading % 10 participation % 90 Homework

59 Logistics Class: Mon, Wed 10:30-11:50 PM, VKC 256. Instructor office hours: Larry: Monday 12-1:30, Wednesday 3:30-5, KAP 406D Mahdi: Monday and Wednesday 5:30-7 PM EEB 422 Course website: blackboard Grading % 10 participation % 90 Homework Lowest homework will be dropped

60 Logistics Class: Mon, Wed 10:30-11:50 PM, VKC 256. Instructor office hours: Larry: Monday 12-1:30, Wednesday 3:30-5, KAP 406D Mahdi: Monday and Wednesday 5:30-7 PM EEB 422 Course website: blackboard Grading % 10 participation % 90 Homework Lowest homework will be dropped We don t care where you find the solution just write the proof in your own language (no plagarism) Course Policy: Use of sources (people, books, internet, etc.) without citation results in failing grade.

61 Textbook Required textbook High-Dimensional Probability: An Introduction with Applications in Data Science. Roman Vershynin. Additional textbooks Concentration Inequalities: A Non-asymptotic Theory of Independence. Stephane Boucheron, Gabor Lugosi, Pascal Massart The Concentration of measure phenomenon. Michel Ledoux

62 Why you should not take this class Probability is not your thing. Eclectic topics.

63 Why you should not take this class Probability is not your thing. Eclectic topics. This class is rated P...

1.1 Administrative Stuff

601.433 / 601.633 Introduction to Algorithms Lecturer: Michael Dinitz Topic: Introduction, Karatsuba/Strassen Date: 9/4/18 1.1 Administrative Stuff Welcome to Algorithms! In this class you will learn the