Non-Liner & Logistic Regression If the sttistics re boring, then you've got the wrong numbers. Edwrd R. Tufte (Sttistics Professor, Yle University)
Regression Anlyses When do we use these? PART 1: find reltionship between response vrible (Y) nd predictor vrible (X) (e.g. Y~X) PART 2: use reltionship to predict Y from X Simple liner regression: y = b + m*x y = β 0 + β 1 * x 1 Multiple liner regression: y = β 0 + β 1 *x 1 + β 2 *x 2 + β n *x n Non liner regression: when line just doesn t fit our dt Logistic regression: when our dt is binry (dt is represented s 0 or 1)
Non-liner Regression Curviliner reltionship between response nd predictor vribles The right type of non-liner model re usully conceptully determined bsed on biologicl considertions For strting point we cn plot the reltionship between the 2 vribles nd visully check which model might be good option There re obviously MANY curves you cn generte to try nd fit your dt
response (y) response (y) response (y) response (y) Exponentil Curve Non-liner regression option #1 Exponentil: y = + bc x Rpid incresing/decresing chnge in Y or X for chnge in the other Ex: bcteri growth/decy, humn popultion growth, infection rtes (humns, trees, etc.) 0 < c < 1 c > 1 +b +b 0 < c < 1 c > 1 -b -b
response (y) response (y) response (y) response (y) Logrithmic Curve Non-liner regression option #2 Logrithmic: y = + bx c Rpid incresing/decresing chnge in Y or X for chnge in the other Ex: survivl thresholds, resource optimiztion -c +c -c +b +b +c -b -b
response (y) response (y) Hyperbolic Curve Non-liner regression option #3 Hyperbolic: y = + b x + c Rpid incresing/decresing chnge in Y or X for chnge in the other Ex: survivl of function of popultion Similr to exponentil nd logrithmic curve but now we hve 2 symptotes +b -b c c
response (y) response (y) Prbolic Curve Non-liner regression option #4 Prbolic: y = + b x c 2 Rpid incresing/decresing chnge in Y or X for chnge in the other followed by the reverse trend Ex: survivl of function of n environmentl vrible Upwrd Prbolic Downwrd Prbolic +b -b c c
response (y) Gussin Curve Non-liner regression option #5 Gussin: y = b x c 2 Resembles norml distribution Ex: survivl of function of n environmentl vrible Where 0 < b < 1 b c
response (y) Sigmoidl Curve Non-liner regression option #6 Signoidl: y = 1 + b x c + d Stbility in Y followed by rpid increse then stbility gin Ex: restricted growth, lerning response, threshold hs to occur for response effect Where b > 1 nd c > 1 c b d
response (y) response (y) Michelis Menten Curve Non-liner regression option #7 Michelis Menten: y = x b + x Rpid incresing/decresing chnge in Y or X for chnge in the other Ex: biologicl process s function of resource vilbility Similr to exponentil nd logrithmic curve but now we hve 2 prmeters this model comes from kinetics/physiology 1 2 -b b
Non-Liner Regression Curve Fitting Procedure: 1. Plot your vribles to visulize the reltionship. Wht curve does the pttern resemble? b. Wht might lterntive options be? 2. Decide on the curves you wnt to compre nd run non-liner regression curve fitting. You will hve to estimte your prmeters from your curve to hve strting vlues for your curve fitting function 3. Once you hve prmeters for your curves compre models with AIC 4. Plot the model with the lowest AIC on your point dt to visulize fit Non-liner regression curve fitting in R: instll.pckges("minpck.lm") nlslm(responsey~model, strt=list(strting vlues for model prmeters))
Non-Liner Regression Output from R Non-liner model tht we fit Simplified logrithmic with slope=0 Estimtes of model prmeters Residul sum-of-squres for your non-liner model Number of itertions needed to estimte the prmeters
Non-Liner Regression Curve Fitting Procedure: 1. Plot your vribles to visulize the reltionship. Wht curve does the pttern resemble? b. Wht might lterntive options be? 2. Decide on the curves you wnt to compre nd run non-liner regression curve fitting. You will hve to estimte your prmeters from your curve to hve strting vlues for your curve fitting function 3. Once you hve prmeters for your curves compre models with AIC 4. Plot the model with the lowest AIC on your point dt to visulize fit Non-liner regression curve fitting in R: instll.pckges("minpck.lm") nlslm(responsey~model, strt=list(strting vlues for model prmeters))
Akike s Informtion Criterion (AIC) How do we decide which model is best? In the 1970s he used informtion theory to build numericl equivlent of Occm's rzor Hirotugu Akike, 1927-2009 Occm s rzor: All else being equl, the simplest explntion is the best one For model selection, this mens the simplest model is preferred to more complex one Of course, this needs to be weighed ginst the bility of the model to ctully predict nything AIC considers both the fit of the model nd the model complexity Complexity is mesured s number prmeters or the use of higher order polynomils Allows us to blnce over- nd under-fitting in our modelled reltionships We wnt model tht is s simple s possible, but no simpler A resonble mount of explntory power is trded off ginst model complexity AIC mesures the blnce of this for us
Akike s Informtion Criterion (AIC) AIC in R AIC is useful becuse it cn be clculted for ny kind of model llowing comprisons cross different modelling pproches nd model fitting techniques Model with the lowest AIC vlue is the model tht fits your dt best (e.g. minimizes your model residuls) Output from R is single AIC vlue Akike s Informtion Criterion in R to determine best model: AIC(nlsLM(responseY~MODEL1, strt=list(strting vlues))) AIC(nlsLM(responseY~MODEL2, strt=list(strting vlues))) AIC(nlsLM(responseY~MODEL3, strt=list(strting vlues)))
Non-Liner Regression Curve fitting Use the prmeter estimtes outputted from nlslm() to generte curve for plotting
Non-Liner Regression Assumptions NLR mke no ssumptions for normlity, equl vrinces, or outliers However the ssumptions of independence (sptil & temporl) nd design considertions (rndomiztion, sufficient replictes, no pseudorepliction) still pply We don t hve to worry bout sttisticl power here becuse we re fitting reltionships All we cre bout is if or how well we cn model the reltionship between our response nd predictor vribles
Non-Liner Regression R 2 for goodness of fit Clculting n R 2 is NOT APPROPIATE for non-liner regression Why? For liner models, the sums of the squred errors lwys dd up in specific mnner: SS Regression + SS Error = SS Totl Therefore R 2 = SS Regression SSTotl which mthemticlly must produce vlue between 0 nd 100% But in nonliner regression SS Regression + SS Error SS Totl Therefore the rtio used to construct R 2 is bis in nonliner regression Best to use AIC vlue nd the mesurement of the residul sum-of-squres to pick best model then plot the curve to visulize the fit
Logistic Regression (.k. logit regression) Reltionship between binry response vrible nd predictor vribles Logistic Model: y = eβ 0+β 1 x 1 +β 2 x 2 + +β n x n 1 e β 0 +β 1 x 1 +β 2 x 2 + +β nx n Logit Model Binry response vrible cn be considered clss (1 or 0) Yes or No Present or Absent The liner prt of the logistic regression eqution is used to find the probbility of being in ctegory bsed on the combintion of predictors Predictor vribles re usully (but not necessrily) continuous But it is hrder to mke inferences from regression outputs tht use discrete or ctegoricl vribles
Binomil distribution vs Norml distribution Key difference: Vlues re continuous (Norml) vs discrete (Binomil) As smple size increses the binomil distribution ppers to resemble the norml distribution Binomil distribution is fmily of distributions becuse the shpe references both the number of observtions nd the probbility of getting success - vlue of 1 Wht is probbility of x success in n independent nd identiclly distributed Bernoulli trils? Bernoulli tril (or binomil tril) - rndom experiment with exctly two possible outcomes, "success" nd "filure", in which the probbility of success is the sme every time the experiment is conducted
Logistic Regression vs Liner Regression Liner Regression - references the Gussin (norml) distribution - uses ordinry lest squres to find best fitting line the estimtes prmeters tht predict the chnge in the dependent vrible for chnge in the independent vrible Logistic regression - references the Binomil distribution - estimtes the probbility (p) of n event occurring (y=1) rther then not occurring (y=0) from knowledge of relevnt independent vribles (our dt) - regression coefficients re estimted using mximum likelihood estimtion (itertive process)
Mximum likelihood estimtion How coefficients re estimted for logistic regression Complex itertive process to find coefficient vlues tht mximizes the likelihood function Likelihood function - probbility for the occurrence of observed set of vlues X nd Y given function with defined prmeters Process: 1. Begins with tenttive solution for ech coefficient 2. Revise it slightly to see if the likelihood function cn be improved 3. Repets this revision until improvement is minute, t which point the process is sid to hve converged
Logistic Regression vs Liner Regression Liner Regression - references the Gussin (norml) distribution - uses ordinry lest squres to find best fitting line the estimtes prmeters tht predict the chnge in the dependent vrible for chnge in the independent vrible Logistic regression - references the Binomil distribution - estimtes the probbility (p) of n event occurring (y=1) rther then not occurring (y=0) from knowledge of relevnt independent vribles (our dt) - regression coefficients re estimted using mximum likelihood estimtion (itertive process) Simple Logistic Regression in R: lm(response~predictor, fmily="binomil") summry(lm(response~predictor, fmily="binomil")) Multiple Logistic Regression in R: lm(response~predictor1+predictor2+ +predictorn, fmily="binomil") summry(lm(response~predictor1+predictor2+ +predictorn, fmily="binomil"))
Logistic Regression (.k. logit regression) Output from R Estimte of model prmeters (intercept nd slope) Stndrd error of estimtes AIC vlue for the model Tests the null hypothesis tht the coefficient is equl to zero (no effect) A predictor tht hs low p-vlue is likely to be meningful ddition to your model becuse chnges in the predictor's vlue re relted to chnges in the response vrible A lrge p-vlue suggests tht chnges in the predictor re not ssocited with chnges in the response
Logistic Regression (.k. logit regression) Pseudo R 2 for goodness of fit In liner regression, the reltionship between the dependent nd the independent vribles is liner However this ssumption is not mde in logistic regression so we cnnot use the clcultion R 2 = SS Regression SSTotl - REMEMBER we re not using sum-of-squres to estimte our prmeters we re using mximum likelihood estimtion We cn however clculte pseudo R 2 - Lots of options on how to do this, but the best for logistic regression ppers to be McFdden's clcultion R 2 = 1 lnl M FULL lnl M intercept L = Estimted likelihood Estimting McFdden s pseudo R 2 in R: mod=lm(response~predictor,fmily="binomil") mcf.r2=1-mod$devince/mod$null.devince NOTE: Pseudo R 2 will be MUCH lower thn R 2 vlues!
Logistic Regression (.k. logit regression) Assumptions Logistic regression mke no ssumptions for normlity, equl vrinces, or outliers However the ssumptions of independence (sptil & temporl) nd design considertions (rndomiztion, sufficient replictes, no pseudorepliction) still pply Logistic regression ssumes the response vrible is binry (0 & 1) We don t hve to worry bout sttisticl power here becuse we re fitting reltionships All we cre bout is if or how well we cn model the reltionship between our response nd predictor vribles
Importnt to Remember A non-liner or logistic reltionship DOES NOT imply custion! AIC or pseudo R 2 implies reltionship rther thn one or multiple fctors cusing nother fctor vlue Be creful of your interprettions!