CE3502. Environmental Monitoring, Measurements & Data Analysis. Points from previous lecture

Transcription

1 CE35. Environmental Monitoring, Measurements & Data Analysis Regression and Correlation Analysis 11 February 9 Points from previous lecture Noise in environmental data can obscure trends; Smoothing is one mechanism for removing noise; Smoothing can help reveal trends and cyclic features in data; Two common means of smoothing are moving averages and exponentiallyweighted moving averages; 1

2 Grades to date Presentations % 1.8% 7 Lab 1: % Lab : % Lab 3: % Lab 4: 8 + 8% Lab 5: % 9 Lab 1: % Lab : % Class Schedule 5 /1-13 Lab 4. Wastewater SRP calibration MEMO Plotting, Correlation, Regression, Phosphorus Confidence intervals, percentiles, outliers, Snow Chap. 7 pp , , Chap. 5 pp , , , pp , /17- Lab 5. Snow water content MEMO Presentations on lab 4 Detection limits, Lake O hypotheses Chap. 6 pp , 413-4, pp / /3-6 Lab. 6. Project orientation FULL REPORT No LAB Presentations on lab 5 Presentations on lab 6 Exam review EXAM

3 Motivation: Regression & Correlation 1. Frequently, engineers and scientists need to know the quantitative relationship between two variables (e.g., rainfall intensity and runoff); {empirical models}. Frequently, scientists and engineers need to determine if two factors are statistically associated with each other (e.g., illness and exposure to a pollutant, t CO emissions i and climate warming); 3

4 Rainfall Intensity-Duration-Frequency Curves for the State of Michigan David Watkins & Dennis Johnson Great Lakes Comparison: Phosphorus Figure taken from Great Lakes Atlas, Canadian Gov't & U.S. EPA, 3 rd Ed.,

5 Limnology & Oceanography 1974, 19(5): Historical changes in Lake Superior? Total Phos sphorus Conc. (mg m -3 ) Year 5

6 Correlation: P and Chlorophyll in Lake Superior Annual me ean Chlorophyll conc. (mg m -3 ) Annual mean TP conc. (mg m -3 ) Nitrate (NO 3- ) in Lake Superior 4 NO3-N (ppb) Year 6

7 NO3-N (ppb) y = 3.9x R =.88 Predictive Models Year 4 3 NO3 - (ppb b) 1 Model Concentration ppb Measured Concentration ppb Year Theory: Least squares regression 1. Consider two variables (x, y) that may be linearly related through an expression such as: y = A + Bx where A and B are constants whose values are unknown; The method used to determine values of A and B and hence. The method used to determine values of A and B and hence to define the straight line that best fits the data (x 1,y 1 ), (x,y ) (x n,y n ) is called linear regression, and the technique used most frequently is Least-squares fitting. 7

8 Theory: Least Squares Data: x i,y i Equation: y i = A + Bx i Y variable Residual (error): ε = y i y i = y i (A+Bx i ) Sums of Squares of Errors: SSE = Σ(ε ) Least-squares: Minimizes SSE residual Xvariable A = Theory: Least Squares Calculation of A and B: d xiid yii d xiid xiyii n xi xi d i d i d i d id i B n x y x y i i i i = n x x d ii d ii Note: x i,y i are pairs of corresponding measurements made at the same time and location (e.g., x = time,,y = [NO 3- ]) 8

9 Regression in Excel: 1 Trendline ) Chlorophyll (mg/m Chlorophyll (mg/m 3 ) y =.6x -.93 R = Total P (mg/m 3 ) Total P (mg/m 3 ) Chart Add trendline options: Show equation on chart Show r on chart Regression in Excel:. Data Analysis Year Chlorophyll T.P. mg/m3 mg/m Tools Data Analysis Regression 9

10 Theory: Correlation 1. How closely are two variables associated with one another?. How good is the fit of the regression equation to the data? The correlation coefficient, r, answers both questions. r sxy = ss x y Covariance of x and y Covariance is the product of the deviations from the means: n s 1 = ( x x)( y y) n xy i i 1 Deviation of y from its mean Theory: Correlation Combining expressions for s xy, s y, and s x : r = L NM dxi xidyi yi dxi xi dyi yi O QP 1 / Note: r will be between een 1 and 1; If r is close to + 1 then the x and y data lie close to a straight line (defined by A and B) and are highly correlated; If r is close to zero then the points do not lie along a line, and x and y are not correlated. 1

11 What is good enough? Table 1. The probability, P (%), that for n samples of two uncorrelated variables (x,y) a value of r greater than r would occur. N r : Example 1: Is correlation good enough? Chlorophyll (mg/m 3 ) y =.6x -.93 R =.383 N = 19 r =.383 r = Total P (mg/m 3 ) P <.5% that this correlation resulted from chance Therefore, regression is significant at > 99.5% confidence level 11

12 Is the slope significantly different from zero?.9 < Slope <.43 Slope = (+ 95% CI) YES! Application: Regression & Correlation Rain duration and intensity were measured at a weather station in central Illinois. Perform a least-squares fit on the data and assess the correlation. Rain event Duration (min) Intensity (in/hr)

13 Is it linear? 7 Intensity (in/hr) y= -.83x 83x R = Duration (min.) r =.98, n = 7 Statistics table P <.6% that this is chance Regression is significant at > 99.4% confidence level Pearson Product-Moment Correlation Coefficient Table of Critical Values df=n- N = number of pairs of data Level of significance for two-tailed test

14 dicted ty Error in pre intensit.6.4 slope -.89 It Intercept. t Residuals Residuals 4 6 Storm duration (min) Residuals must be randomly distributed. The systematic pattern shown here violates an assumption of regression analysis and implies that the regression is not valid, particularly outside the range in which data were collected. 14

15 Theory: Least Squares Uncertainty in A and B: A and B are ESTIMATES for constants that make the best linear fit for the data; Just as each value of x and y has uncertainties due to systematic (bias) and random (precision) errors, so also do A and B have uncertainties; The uncertainty in A and B will lead to an uncertainty in any value of y predicted with the regression equation. Assume: measured values of x have less error than y n Then: 1 sy' = cyi A+ Bxi n 1 h Theory: Least Squares sy'dd xi i d ii d ii Error in A (A + s A ): s A = n x x Error in B (B + s B ): s B = ns y' d ii d ii n x x 15

16 Error in calculated X What if we want to use the regression line to calculate unknown x values? Can we use the uncertainty in y, Aand B to estimate the uncertainty in x? 1. Absorba ance (au) y =.54x +. R² = Conc. (mg/l) Goal: Calculate uncertainty in x Information available: A + ε a, B + ε b, s y or s pred, y Approach: x x x x a b y ε = ε + ε + ε da db dy y A x' x' x' Given: x ' = what is? What is? What is? B A B y 16