Python 데이터분석 보충자료. 윤형기

Transcription

1 Python 데이터분석 보충자료 윤형기

2 단순 / 다중회귀분석 Logistic Regression 회귀분석 REGRESSION

3 Regression 개요 single numeric D.V. (value to be predicted) 과 one or more numeric I.V. (predictors) 간의관계식. "regression" = process of fitting lines to data (Galton) also used for hypothesis testing, determining whether data indicate that a presupposition is more likely to be true or false. 다양한모델에적용 SLR MLR GLM Link functions Logistic regression, Poisson regression, 3

4 단순회귀분석 단순회귀분석 종속변수 = the variable to be predicted (y). 독립변수 = explanatory variable = The predictor (x). 대상 : only a straight-line relationship between 2 variables 회귀직선식의결정 deterministic regression model is y = β 0 + β 1 x probabilistic regression model is y = β 0 + β 1 x + ε 4

5 OLS (Ordinary Least Squares) 5

6 잔차분석 6

7 추정값의표준오차 error 분석을위해잔차 (= 개별 point 에대한 estimation errors) 계산대신 standard error of the estimate 이용. SSE is in part a function of the number of pairs of data being used to compute the sum, which lessens the value of SSE as a measurement of error. 더좋은지표 = standard error of the estimate (s e ) is a standard deviation of the error of the regression model. ( 정규분포 empirical rule: 68% 가 μ+ 1σ 범위, 95% 가 μ+ 2σ 범위. regression 의 assumption 도 for a given x, error terms ~ ND() ) 이제 error terms ~ ND(), s e 는 error 의 s.d., AVG error =0 이므로» 68% of the error values (residuals) should be within 0 ±1s e» 95% of the error values (residuals) should be within 0 ±2s e. s e provides a single measure of magnitude of errors in model. 또한 outlier 식별에이용. ( 예 : outside ±2s e or ±3s e ) 7

8 결정계수 R 2 = I.V. (x) 가 variability of D.V. (y) 를얼마나설명하는가» r 2 =0 r 2 = 1 D.V. (y) has a variation, measured by SS of y (SS yy ):» SS yy =SSR +SSE» If each term is divided by SSyy, the resulting equation is r 2 is proportion of y variability explained by regression model: Relationship Between r and r 2 r 2 = (r) 2» coeff t of correlation & determination 회귀모델기울기의가설검정 & 모델전반의 Testing 기울기 r = (r) 2 8

9 계수추정 OLS chooses β 0 and β 1 to minimize the RSS, using some calculus

10 추정된계수의 Accuracy (Q) μ 추정치가얼마나정확한가? (A) SE(μ) (=standard error of μ) 를계산 즉, β 0 와 β 1 에대한표준오차의계산 residual standard error RSE = RSS (n 2).

11 선형모델의 Accuracy Residual Standard Error R 2 Statistic r = Cor(x, y) R 2 = r 2

12 다중회귀분석 SLR 과 MLR 단순회귀모델 : y =β 0 + β 1 x +ε 다중회귀모델 : y =β 0 + β 1 x 1 + β 2 x β k x k +ε 독립변수를가진 MR Model (First Order) y = β 0 + β 1 x 1 + β 2 x 2 +ε Constant & coefficients 는표본으로부터추출 : y =b 0 +b 1 x 1 +b 2 x 2 response surface / response plane 회귀모델과계수에대한유의성검정 <Regression 모델의 adequacy 분석 > 모델전반의검정 단순회귀 ; t test of slope of the regression line to see if 0. ( 즉, whether I.V. contribute significantly in predicting D.V. ) 다중회귀 ; an analogous test makes use of F statistic. 12

13 13

14 회귀계수에대한 Significance Tests individual significance tests for each regression coefficient with t test. H 0 : β 1 =0 H 0 : β 2 =0 H 0 : β k =0 H a : β 1 0 H a : β 2 0 H a : β k 0 d.f. for each of individual tests of regression coefficients are n - k - 1. 추정치의잔차와표준오차및 R 2 Residuals = error of the regression model 활용 : outlier 탐지, regression 분석시 assumptions 검정 SSE 와 Standard Error of the Estimate = 추정값의표준오차 = 추정표준오차 ( 표준추정오차 )= 차이의표준오차 = 최적선에대한산포도에서점들의분산도 = y 를중심으로실제 y 점수분포가 ( 회귀선에의한 ) 어느정도인가표시 SSE =Σ(y - y) 2 회귀분석의가정 (error terms ~ ND(0) + 경험칙 ( 대략잔차의 68% 가 ±1se 범위, 95% 가 ±2se 범위 ) 회귀모델의데이터 fitting 정도를측정하는데 standard error of estimate 가유용. 14

15 계수추정 (MLR)

16 주요이슈 (1) Response-Predictors 간의관계성여부? 가설검정 H 0 : β 1 = β 2 = = β p =0 Ha: at least one β j is non-zero. F-statistic 계산 : 단, TSS = (y i y) 2 and RSS = (y i y i ) 2. IF H 0 is true (=response-predictors 간 no relationship) THEN F 값은 1 에근접 IF H a is true, THEN E{(TSS - RSS)/p} >σ 2, so we expect F > 1.

17 (2) 변수별중요도결정 Variable Selection Mallow s Cp, Akaike information criterion (AIC), Bayesian information criterion (BIC), adjusted R 2 그런데 2 p 모델 Forward selection Backward selection Mixed selection

18 (3) Model Fit In SLR, R 2 = 설명변수와상관계수간의상관계수의제곱 In MLR, it equals Cor(Y, Y.) 2 fitted linear model 의특징 : maximizes this correlation among all possible linear models. p-value 를통해 R 2 의개선정도를계수화 RSE 의정의 : Thus, models with more variables can have higher RSE if the decrease in RSS is small relative to the increase in p.

19 (4) Predictions β 0, β 1,..., β p 의 true value 를안다해도 random error 로인해완벽한예측은불가능. ( 즉, irreducible error) confidence interval prediction interval

20 기타의주요이슈 Interaction terms Non-linear effects Multicollinearity Model Selection 20

21 mathematical transformation 을통한 Non-linear models <first-order model> one independent variable: y = β 0 + β 1 x 1 +ε two independent variables: y = β 0 + β 1 x 1 + β 2 x 2 +ε <polynomial regression model> ; contain squared, cubed, or higher powers of the predictor variable(s) and contain response surfaces that are curvilinear. Yet, they are still special cases of the general linear model given in formula: y = β 0 + β 1 x 1 + β 2 x β k x k +ε <second-order model with one independent variable> y = β 0 + β 1 x 1 + β 2 x ε <Quadratic model> 次數가 2 차 (=polynomial equation of degree 2) = a special case of the general linear model curvilinear regression by recoding the data before the multiple regression analysis is attempted. Quadratic form (2 차형식 ) quadratic curve (2 차곡선 ) X T AX = x1 x2 a b c d x1 x2 = ax bx 1 x 2 + xc 1 x 2 + dx 2 2 Non-linear 특화모델 ( 後述 ) 21

22 Model Transformation 개념 exponential model log antilog inverse model Tukey 의 Ladder of Transformations 22

23 Non-linear Relationships 의예 Polynomial regression

24 Regression 분석에서의 interaction Interaction 항목을별도의독립변수로검토 interaction predictor variable can be designed by multiplying the data values of one variable by the values of another variable, y = β 0 + β 1 x 1 + β 2 x 2 + β 3 x 1 x 2 +ε (x 1 x 2 term = interaction term). Even though this model has 1 as the highest power of any one variable, it is considered to be a second-order equation because of the x 1 x 2 term. 24

25 모델구축 : Search 절차 회귀모델개발 : (i) maximize explained proportion of the deviation of y values. (ii) Be as parsimonious as possible. Search 절차 All Possible Regressions ( 모든가능한조합의회귀분석 ) If a data set contains k independent variables, all possible regressions will determine 2 k -1 different models. Stepwise Regression ( 단계적회귀분석 ) single predictor variable 에서시작해서 adds and deletes predictors one step at a time, examining the fit of the model at each step until no more significant predictors remain outside the model. STEP 1/2/3: Forward Selection ( 전진선택법 ) = stepwise regression 과동일. 단, once a variable is entered into the process, it is never dropped out. Backward Elimination ( 후진제거법 ) 25

26 Multicollinearity ( 다중공선성 ) = 2 이상독립변수가 highly correlated. (2 개 : collinearity; 여러개 : multicollinearity) 1. It is difficult to interpret the estimates of the regression coeff ts. 2. Inordinately small t values for regression coefficients may result. 3. S.D. of regression coefficients are overestimated. 4. The algebraic sign of estimated regression coefficients may be the opposite of what would be expected for a particular predictor value. multicollinearity 문제는 regression 계수를평가하는 t 값에도영향. Multicollinearity can result in an overestimation of s.d. of the regression coefficients t values tend to be underrepresentative when multicollinearity is present. ( 접근법 ) correlation matrix 를조사하여가능한 intercorrelations 를탐색. Stepwise regression to prevent the problem of multicollinearity. 26

27 Interaction 개념 예 : When the effect on Y of increasing X1 depends on another X2. Advertising 예 : TV and radio advertising both increase sales. Sales = b 0 + b 1 TV + b 2 Radio+ b 3 TV Radio Parameter Estimates Term Estimate Std Error t Ratio Prob> t Intercept <.0001 * TV <.0001 * Radio * TV*Radio e <.0001 * 27

28 Dummy coding 예 : men and women (category listings) Code as indicator variables (dummy variables); Male=0, Female=1. Suppose we want to include income and gender. β 2 = average extra balance each month that females have for given income level. Males are the baseline. 28

29 Line for women Regression equation female: salary = position males: salary = position Different intercepts Same slopes Line for men Position Regression coefficients Coefficient Std Err t-value p-value Constant Income Gender_Female

30 모델평가와변수선택 개념 독립변수의개수가많을경우이를축소하여단순화 즉, OLS fitting 에대한 alternative fitting 을통한 MSE 최소화 필요성 Prediction Accuracy Model Interpretability Subset Selection Stepwise Selection Choosing the Optimal Model Shrinkage Methods Ridge Regression The Lasso

31 1. Prediction Accuracy X 와 Y 의관계가선형이고 n >>p 일때는비교적 low bias, low variance ( 단, n= # of observations, p= # of predictors) 그러나 n» p when, OLS fit can have high variance and may result in overfitting and poor estimates on unseen observations, when n < p, the variability of the least squares fit increases dramatically, and the variance of these estimates in infinite 2. Model Interpretability 독립변수 X 의개수가많을경우이들의 Y 에대한효과가감소 Leaving these variables in the model makes it harder to see the big picture, i.e., the effect of the important variables The model would be easier to interpret by removing (i.e. setting the coefficients to zero) the unimportant variables 31

32 Solution Subset Selection 전체 p 개의설명변수 X 의일부분 (subset) 을식별해낸후이를이용해서모델 fitting 예 : best subset selection, stepwise selection Shrinkage Shrink the estimates coefficients towards zero reduces variance Some of the coefficients may shrink to exactly zero, and hence shrinkage methods can also perform variable selection 예 : Ridge regression, Lasso 차원축소 (Dimension Reduction) Involves projecting all p predictors into an M-dimensional space where M < p, and then fitting linear regression model 예 : Principle Components Regression 32

33 Best Subset Selection One simple approach = take the subset with the smallest RSS or the largest R 2. 단, 모델의변수가많아질수록 R 2 증가 (== smallest RSS) 예 33

34 Measures of Comparison Add penalty to RSS for the number of variables (complexity) 종류 Adjusted R 2 AIC (Akaike information criterion) BIC (Bayesian information criterion) C p (equivalent to AIC for linear regression) 34

35 Stepwise Selection 배경 Best Subset Selection is computationally intensive especially when we have a large number of predictors (large p) More attractive methods: Forward Stepwise Selection: Begins with the model containing no predictor, and then adds one predictor at a time that improves the model the most until no further improvement is possible Backward Stepwise Selection: Begins with the model containing all predictors, and then deleting one predictor at a time that improves the model the most until no further improvement is possible 35

36 Shrinkage Methods Ridge Regression Ordinary Least Squares (OLS) estimates β by minimizing Ridge Regression uses a slightly different equation Tuning parameter λ is a positive value. has the effect of shrinking large values of β towards zero. It turns out that such a constraint should improve the fit, because shrinking the coefficients can significantly reduce their variance Notice that when λ = 0, we get the OLS! 36

37 As λ increases, standardized coefficients shrinks towards 0. 효과 OLS estimates generally have low bias but can be highly variable. In particular when n and p are of similar size or when n < p, then the OLS estimates will be extremely variable The penalty term makes the ridge regression estimates biased but can also substantially reduce variance 37

38 효과 일반적으로, RR estimates will be more biased than OLS but have lower variance Ridge regression will work best in situations where the OLS estimates have high variance If p is large, using best subset selection approach requires searching through enormous numbers of possible models For Ridge Regression, for any given λ, we only need to fit one model and the computations turn out to be very simple Ridge Regression can even be used even when p > n! 38

39 Lasso 개념 ( 배경 ) Ridge Regression isn t perfect the penalty term will never force any of the coefficients to be exactly zero. Thus, the final model will include all variables, which makes it harder to interpret LASSO 역시유사하지만 penalty term 이다름 Penalty term Ridge Regression minimizes The LASSO estimates β by minimizing the 39

40 Tuning parameter λ 의선택 Select a grid of potential values, use cross validation to estimate the error rate on test data (for each value of λ) and select the value that gives the least error rate 40