ENVE3502. Environmental Monitoring, Measurements & Data Analysis. Points from previous lecture

Transcription

1 ENVE35. Environmental Monitoring, Measurements & Data Analysis Regression and Correlation Analysis Points from previous lecture Noise in environmental data can obscure trends; Smoothing is one mechanism for removing noise; Smoothing can help reveal trends and cyclic features in data; Two common means of smoothing are moving averages and exponentiallyweighted moving averages; 1

2 Grades to date Presentations % % Lab Reports: Lab 1: % Lab : % Lab 3: % Lab 4: 8 + 8% Lab 5: % Reports: Reports: % % % 8% Class Schedule Week Monday Wednesday Friday 4 1/3 /1 Lecture /3 Lab 4 Presentations Lab Regression, Correlation Wastewater SRP memo 5 /6 Presentations Lab 3 6 /13 Presentations Lab 4 /8 Lec. CI, percentiles /15 Lec. Detection Limit 7 / Lecture / Exam review 8 /7 Presentations Lab 5 /9 EXAM /1 Winter Carnival /17 Lab 5 Snow water content memo /4 Lab 6 Lake DO Full report 3/ Presentations Lab 6

3 Motivation: Regression & Correlation 1. Frequently, scientists and engineers need to determine if two factors are statistically associated with each other (e.g., illness and exposure to a pollutant, CO emissions and climate warming); Correlation analysis. Frequently, engineers and scientists need to know the quantitative titti relationship lti between bt two variables ibl (e.g., rainfall intensity and runoff); Regression analysis {empirical models} 3

4 Rainfall Intensity-Duration-Frequency Curves for the State of Michigan David Watkins & Dennis Johnson Great Lakes Comparison: Phosphorus Figure taken from Great Lakes Atlas, Canadian Gov't & U.S. EPA, 3 rd Ed.,

5 Limnology & Oceanography 1974, 19(5): Historical changes in Lake Superior? Total Phos sphorus Conc. (mg m -3 ) Year 5

6 Correlation: P and Chlorophyll in Lake Superior Annual me ean Chlorophyll conc. (mg m -3 ) Annual mean TP conc. (mg m -3 ) r =.38, P <.1 r =.78, P <.5 Nitrate (NO 3- ) in Lake Superior 4 NO3-N (ppb) Year 6

7 NO3-N (ppb) y = 3.9x R =.88 Predictive Models Year 4 3 NO3 - (ppb b) 1 Model Concentration ppb Measured Concentration ppb Year Theory: Least squares regression 1. Consider two variables (x, y) that may be linearly related through an expression such as: y = A + Bx where A and B are constants whose values are unknown; The method used to determine values of A and B and hence. The method used to determine values of A and B and hence to define the straight line that best fits the data (x 1,y 1 ), (x,y ) (x n,y n ) is called linear regression, and the technique used most frequently is Least-squares fitting. 7

8 Theory: Least Squares Data: x i,y i Equation: y i = A + Bx i Y variable Residual (error): ε = y i y i = y i (A+Bx i ) Sums of Squares of Errors: SSE = Σ(ε ) Least-squares: Minimizes SSE residual Xvariable A = Theory: Least Squares Calculation of A and B: d xiid yii d xiid xiyii n xi xi d i d i d i d id i B n x y x y i i i i = n x x d ii d ii Note: x i,y i are pairs of corresponding measurements made at the same time and location (e.g., x = time,,y = [NO 3- ]) 8

9 Regression in Excel: 1 Trendline ) Chlorophyll (mg/m Chlorophyll (mg/m 3 ) y =.6x -.93 R = Total P (mg/m 3 ) Total P (mg/m 3 ) Chart Add trendline options: Show equation on chart Show r on chart Regression in Excel:. Data Analysis Year Chlorophyll T.P. mg/m3 mg/m Tools Data Analysis Regression 9

10 Theory: Correlation 1. How closely are two variables associated with one another?. How good is the fit of the regression equation to the data? The correlation coefficient, r, answers both questions. r sxy = ss x y Covariance of x and y Covariance is the product of the deviations from the means: n s 1 = ( x x)( y y) n xy i i 1 Deviation of y from its mean Theory: Correlation Combining expressions for s xy, s y, and s x : r = L NM dxi xidyi yi dxi xi dyi yi O QP 1 / Note: r will be between 1 and 1; If r is close to + 1 then the x and y data lie close to a straight line (defined by A and B) and are highly correlated; If r is close to zero then the points do not lie along a line, and x and y are not correlated. r is called the regression coefficient and represents the percentage of the variance explained by the independent variable; 1

11 What is good enough? Table 1. The probability, P (%), that for n samples of two uncorrelated variables (x,y) a value of r greater than r would occur. N r : Example 1: Is correlation good enough? Chlorophyll (mg/m 3 ) y =.6x -.93 R =.383 N = 19 r =.383 r = Total P (mg/m 3 ) P <.5% that this correlation resulted from chance Therefore, regression is significant at > 99.5% confidence level 11

12 Is the slope significantly different from zero?.9 < Slope <.43 Slope = (+ 95% CI) YES! Application: Regression & Correlation Rain duration and intensity were measured at a weather station in central Illinois. Perform a least-squares fit on the data and assess the correlation. Rain event Duration (min) Intensity (in/hr)

13 Is it linear? 7 Intensity (in/hr) y= -.83x 83x R = Duration (min.) r =.98, n = 7 D.F. = n = 5 Statistics table P <.6% that this is chance Regression is significant at > 99.4% confidence level Pearson Product-Moment Correlation Coefficient Table of Critical Values df=n- N = number of pairs of data Level of significance for two-tailed test

14 dicted ty Error in pre intensit.6.4 slope -.89 It Intercept. t Residuals Residuals 4 6 Storm duration (min) Residuals must be randomly distributed. The systematic pattern shown here violates an assumption of regression analysis and implies that the regression is not valid, particularly outside the range in which data were collected. 14

15 Theory: Least Squares Uncertainty in A and B: A and B are ESTIMATES for constants that make the best linear fit for the data; Just as each value of x and y has uncertainties due to systematic (bias) and random (precision) errors, so also do A and B have uncertainties; The uncertainty in A and B will lead to an uncertainty in any value of y predicted with the regression equation. Assume: measured values of x have less error than y n Then: 1 sy' = cyi A+ Bxi n 1 h Theory: Least Squares sy'dd xi i d ii d ii Error in A (A + s A ): s A = n x x Error in B (B + s B ): s B = ns y' d ii d ii n x x 15

16 Excel Output error tabulation Error in calculated X What if we want to use the regression line to calculate unknown x values? Can we use the uncertainty in y, Aand B to estimate the uncertainty in x? 1. Absorba ance (au) y =.54x +. R² = Conc. (mg/l) 16

17 Goal: Calculate uncertainty in x Information available: A + ε a, B + ε b, s y or s pred, y Approach: x x x x a b y ε = ε + ε + ε da db dy y A x' x' x' Given: x ' = what is? What is? What is? B A B y Using the regression equation for the standard curve and the absorbance values of the unknowns, determine the concentrations of the unknowns. Use the error propagation technique outlined in the lab handout to determine the uncertainty in your calculated concentrations for the unknowns. Calculate the mean and standard deviation for the concentrations for each sample based on the four measurements by different groups. How does the uncertainty (standard error) that you estimated above compare with the standard deviation that you just calculated? 17