Sample sta*s*cs and linear regression. NEU 466M Instructor: Professor Ila R. Fiete Spring 2016

Transcription

1 Sample sta*s*cs and linear regression NEU 466M Instructor: Professor Ila R. Fiete Spring 2016

2 Mean {x 1,,x N } N samples of variable x hxi 1 N NX i=1 x i sample mean mean(x) other notation: x

3 Binned version of mean {x 1,,x N } N samples of variable x {c 1, c B },B bins {n 1, n B } counts per bin hxi 1 N BX i=1 n i c i sample mean

4 Variance {x 1,,x N } h(x hxi) 2 i 1 N 1 NX (x i hxi) 2 sample variance i=1 a measure of the scajer /spread of the data around its mean value homework: show that h(x hxi) 2 i = hx 2 i hxi 2

5 Standard devia*on {x 1,,x N } p h(x hxi) 2 standard deviation

6 Covariance {x 1,,x N }{y 1,,y N } N samples each of variables x, y C(x, y) 1 N 1 NX (x i hxi)(y i hyi) i=1 sample covariance (C(x, x) is simply sample variance of x)

7 Covariance: what does it measure? C(x, y) 1 N 1 NX (x i hxi)(y i hyi) i=1 If x, y both deviate from their means together (both up then both down) then terms in sum are posi*ve, C(x,y) > 0. If x,y deviate from their means independent of each other, then terms in the sum are randomly posi*ve and nega*ve, C(x,y) ~=0. If x,y deviate from their means in opposite direc*ons, then terms in sum are nega*ve, C(x,y) < 0. Literally, covariance is a measure of co- varia*on.

8 4 3 Covariance example I x, y independent x = randn(1000, 1) y = randn(1000, 1) 2 1 C(x, y) =0.009; C(x, x) =1.069 y x x>0,y around 0 without bias

9 4 3 Covariance example II x, y independent x = 0.2 randn(1000, 1) y = 0.2 randn(1000, 1) 2 1 y x C(x, y) =0.001; C(x, x) =0.0407

10 Covariance example III 2.5 x, y not independent x = randn(1000, 1) y = 0.5 x randn(1000, 1) y x>0,y > x C(x, x) =0.907; C(x, y) =0.464; C(y, y) =0.469

11 Alterna*ve nota*on Mean: hxi, x, µ x, E(x) Variance: hx 2 i hxi 2, x 2 x 2, 2 x, var(x), C(x, x) Covariance: hxyi hxihyi, xy xȳ, Standard devia*on p hx2 i hxi 2, 2 xy, cov(x), C(x, y) q x 2 x 2, x, std(x)

12 Pearson s correla*on coefficient (x, y) = (x hxi)(y hyi) p h(x hxi)2 ih(x hxi) 2 i (x, y) = C(x, y) x y shorter- form nota*on

13 Pearson s correla*on coefficient and covariance only measure linear dependency from: hjps://en.wikipedia.org/wiki/correla*on_and_dependence

14 Robust sta*s*cs? Mean, variance are easy to compute, widely used/useful. But not robust: sensi*ve to outliners. More robust alterna*ve to mean: median.

15 Applica*on LINEAR REGRESSION IN TERMS OF SAMPLE STATISTICS

16 Regression: curve- fi`ng Scalar explanatory variable (X) and response variable (Y); N samples {(x 1,y 1 ), (x 2,y 2 ),, (x N,y N )} ỹ(x) =w 0 + w 1 x + + w M x M = MX j=0 w j x j free parameters: (w 0,w 1,,w M )

17 Linear least- squares regression E = 1 2 = 1 2 = 1 2 NX [ỹ(x n ; w) y n ] 2 n=1 NX [ n=1 MX j=0 w j x j n y n ] 2 NX [w 0 + w 1 x n y n ] 2 n=1 To solve for best w 0, w 1 : M=1 for linear regression de dw 0 =0, de dw 1 =0

18 Linear least- squares regression E = 1 2 NX [w 0 + w 1 x n y n ] 2 n=1 de N dw 0 = X [w 0 + w 1 x n y n ] n=1 = Nw 0 + Nw 1 hxi Nhyi =0 w 0 + w 1 hxi hyi =0 (1)

19 Linear least- squares regression E = 1 2 NX [w 0 + w 1 x n y n ] 2 n=1 de N dw 1 = X [w 0 + w 1 x n y n ]x n n=1 = Nw 0 hxi + Nw 1 hx 2 i Nhxyi =0 w 0 hxi + w 1 hx 2 i hxyi =0 (2)

20 Linear least- squares regression w 1 C(x, y) = C(x, x) w 0 = hyi w 1 hxi slope y intercept In homework: check matlab s polyfit with this op*mal expression for linear- least squares fi`ng.

21 Linear least- squares regression w 1 C(x, y) = C(x, x) w 0 = hyi w 1 hxi slope y intercept Contrast with w 1 : Pearson s correla*on (x, y) = C(x, y) x y Different normaliza*ons: Different correla*on coefficient for same slope but different amounts of x,y- scajer. Same correla*on for different slopes and different x,y scajer. Correla*on: more strongly penalizes y- scajer, more weakly penalizes x- scajer.

22 Slope versus Pearson s correla*on coefficient same slope different ρ from: hjps://en.wikipedia.org/wiki/correla*on_and_dependence different slope, same ρ

23 Applica*on BACK TO SAMPLE STATISTICS: MULTIVARIATE

24 Mul*ple variables: covariance matrix {x 1,,x N } N samples of the αth variable x α K different variables x α, labeled by α, β = {1,,K}: C 1 N 1 K K dim since K variables NX (x i hx i)(x i hx i) i=1 = cov(x,x ) sample covariance matrix

25 Covariance matrix (α,β) element is covariance between x α, x β. Diagonal of covariance matrix is variance of each variable: var(x α ) or C(x α, x α ). K 2 entries total, but only half of off- diagonal terms are independent because of symmetry (C(x β, x α )= C(x α, x β )). Thus only (K 2 - K)/2 + K = K(K+1)/2 independent terms. Q s: How do do linear regression in mul*variate case? Will it involve covariance matrix?

26 4 3 Covariance example I x, y independent x = randn(1000, 1) y = randn(1000, 1) 2 1 y C = apple x

27 Covariance example III 2.5 x, y not independent x = randn(1000, 1) y = 0.5 x randn(1000, 1) y C = apple x

28 Summary Defined sample mean and variance of a variable Defined covariance between a pair of variables Solved op*mal (least- squares) linear regression between two variables in terms of mean, covariance Covariance matrix: covariance between all K(K+1)/2 unique pairs of K variables