Explaining the Stein Paradox
|
|
- Maximillian Pearson
- 5 years ago
- Views:
Transcription
1 Explanng the Sten Paradox Kwong Hu Yung 1999/06/10 Abstract Ths report offers several ratonale for the Sten paradox. Sectons 1 and defnes the multvarate normal mean estmaton problem and ntroduces Sten s paradox. Sectons 3 8 advances the Galtonan perspectve and explans Sten s paradox usng regresson analyss. Sectons 9 and 10 approaches Sten s paradox through the conventonal emprcal Bayes approach. The closng sectons 11 and 1 compares admssblty to equvarance and to mnmaxty as crtera for smultaneous estmaton. 1 Estmatng Mean of Multvarate Normal Sten s paradox arses n estmatng the mean of a multvarate normal random varable. Let X 1, X,..., X k be ndependent normal random varables, such that X N(θ, 1). All k random varables have a common known varance, but ther unknown means dffer and vary separately. In other words, (X 1, X,..., X k ) N(θ, I), where θ = (θ 1, θ,..., θ k ) and I s the k k dentty matrx. Consder estmatng the mean θ under squarederror loss L(θ, d) = k (θ d ) = θ d. A natural and ntutve estmate of θ would be X tself. Charles Sten showed that ths naïve estmator ˆθ = X s not even admssble. For k 3, the obvous estmate X s domnated by ˆθ JS = (1 k S )X, where S = X. (1) The James-Sten estmator ˆθ JS shrnks the naïve estmate towards zero by a factor 1 k S, where S = X depends on the other random varables. Although the rsk s optmal for k, more generally, for k 3 and 0 < c < (k ), any estmator of the form ˆθ JS = (1 c S )X () has unformly smaller rsk for all θ. Smlarly, for k 4 and 0 < c < (k 3), the famly of Efron- Morrs estmator where ˆθ EM = X + (1 c S )(X X), (3) S = (X X) (4) domnates the naïve estmator X. In ths case, the optmal c = k 3. The Efron-Morrs estmators shrnk towards the mean X rather than 0. Because each X s ndependently dstrbuted, estmatng each θ as separate experments would seem ntutve. Paradoxcally, combnng observatons actually provdes a much better estmate of the means. So Sten s paradox states not only that the natural estmator X s not admssble, but also that X s domnated by ˆθ JS, an estmator whch combnes ndependent experments. Sten s Paradox Volates Intuton On the surface, Sten s paradox seems unacceptable. Intutvely each ndependent experment should not affect the other experments. For example, consder
2 X as test scores of each student n a class. Suppose that X N(θ, 1), where θ s the theoretcal IQ of student. Gven that each student takes hs test ndependently, the natural estmate of each student s IQ would be hs ndvdual test score. Snce each student took the test ndvdually and separately, justce dctates that each student be judged on hs personal performance. In partcular, the natural estmate of θ would ntutvely be X. Yet accordng to Sten s paradox, the performance of the entre class should be factored nto the estmate of each ndvdual s IQ. Rather than estmate Tom s IQ by hs personal test score, Sten s estmator shrnks Tom s IQ by a factor based on the standard devaton of all test scores. Sten s paradox asserts that makng use of test scores of the whole class provdes a better estmate of each ndvdual s IQ. 3 Galtonan Perspectve on Sten s Paradox Consder pars {(X, θ ) = 1,,..., k}, where the X s are known but the θ s are unknown. Snce X N(θ, 1), wrte X = θ + Z, where Z N(0, 1). Because X are mutually ndependent, the Z are also mutually ndependent. Francs Galton over a century ago offered a perspectve that can shed lght on Sten s paradox. On a graph wth X as the horzontal axs and θ as the vertcal axs, the pont (X, θ ) would le near the dagonal lne θ = X. Because of the error term Z, the pont (X, θ ) s horzontally shfted from the dagonal lne θ = X by the random dstance Z. Snce E X = θ and V ar X = 1/k, the mean X should be centered about θ. In other words, ( X, θ) should le near the center dagonal lne θ = X on the X/θ coordnate system. 4 Regresson Analyss Pcture Gven an X, the correspondng θ can be estmated usng standard regresson ˆθ = E[θ X]. The naïve estmate ˆθ = X, apparently agrees wth ˆX = E[X θ] = θ, the regresson lne of X on θ. However, the regresson lne of X on θ s not approprate n estmatng θ on the bass of X. Instead the opposte regresson lne should be used. Usng the regresson of θ on X would provde a more meanngful estmate of θ. Snce the dstrbuton of θ gven X s unknown, the regresson lne ˆθ = E[θ X] must be approxmated. 5 Lnear Regresson Motvates the Efron-Morrs Estmator The optmal estmator ˆθ = E[θ X] s unavalable because the dstrbuton of θ gven X s unknown. Instead, restrct attenton to only lnear estmators of the form ˆθ = a + bx, = 1,,..., k. (5) Mnmzng the loss functon L(θ, ˆθ) = k (θ ˆθ ) s equvalent to fndng the least squares ft to {(X, θ ) = 1,,..., k} n the X/θ coordnate system. The usual least squares ft would be ˆθ = θ + ˆβ(X X), where ˆβ = (X X)(θ ˆθ) (Xj ˆX) (6) f the ponts {(X, θ ) = 1,,..., k} were actually known. Snce θ are unknown, the ponts {(X, θ ) = 1,,..., k} are not avalable and so the least squares coeffcents cannot be computed exactly. Instead, consder estmatng the coeffcents usng only the avalable data X 1, X,..., X k. Snce X N(θ, I) s a full-rank exponental famly, X s complete suffcent for θ. So X s not only an unbased estmator of θ but also the UMVU estmator of θ. The numerator of ˆβ can be estmated by (X X) (k 1) snce E[ (X X) (k 1)] = E[ (X X)(θ θ)] = (θ θ). An estmate of ˆβ would be (X X) (k 1) (Xj X) = 1 k 1 (Xj X). (7)
3 The estmate of the least square lnear ft becomes ˆθ EM = X + (1 k 1 (Xj X) )(X X), (8) whch s just an Efron-Morrs estmator. Although the constant c = k 1 s not the optmal choce among the Efron-Morrs famly, the estmator suggested by the estmated least square lnear ft nonetheless domnates the naïve estmator X. 6 Lnear Regresson Motvates the James-Sten Estmator The James-Sten estmator can also be found through a smlar analyss. In the X/θ coordnate system, consder usng the regresson of θ on X to get an estmator ˆθ = E[θ X]. Once agan the dstrbuton of θ gven X s unavalable. Ths tme, restrct attenton to only estmators proportonal to X, of the form ˆθ = bx. (9) Mnmzng the loss functon L(θ, ˆθ) = (θ ˆθ ) s equvalent to fndng the least squares ft through the orgn. The least squares estmator s ˆβ = θ X X. (10) Snce E[ X k] = E[ θ X] = θ s an estmate of the numerator of ˆβ, the least square ft through the orgnal can be estmated by ˆθ JS = (1 k X )X, (11) whch has the form of the James-Sten estmator wth c = k nstead of the optmal c = k. 7 Shrnkage Estmators Domnate The regresson analyss pcture provdes a smple route toward the shrnkage estmators of James-Sten and Efron-Morrs. The Galtonan perspectve also explans why the shrnkage estmators are preferred over the naïve estmator. The naïve estmator X corresponds to an estmate of the regresson lne ˆX = E[X θ] = θ. To predct θ on the bass of X, however, the proper regresson lne to use s ˆθ = E[θ X]. Shrnkage estmators are just estmates of the proper regresson lne. When k = 1,, the two regresson lnes meet at each pont (X, θ ). In ths case, the James-Sten shrnkage estmator does not domnate the usual estmator. The superorty of the James-Sten estmator becomes apparently only when the two regresson lnes dffer, for k 3. 8 Mnmzng the Rsk Drectly From the Galtonan vewpont, Sten s estmator s just an estmate of the least squares ft to the regresson of θ on X n the X/θ coordnate system. The least squares ft mnmzes the loss functon, whch n turn mnmzes the rsk. A smlar strategy follows from mnmzng the rsk drectly. Among the class of estmators ˆθ = bx, the rsk s functon R(θ, bx) = E θ [ acheves ts mnmum at (θ bx ) ] (1) = kb + (1 b) θ, (13) b = to gve the mnmum rsk θ k + θ (14) R(θ, b θ X) = k < k. (15) k + θ In actualty, θ s the unknown estmand. Snce E[ X ] = (θ + 1) = θ + k, (16) 3
4 estmatng the numerator and denomnator of b separately gves ˆb = X k X = 1 k X. (17) As before, the estmatng the optmal b leads to the shrnkng factor. 9 Emprcal Bayes Approach to the James-Sten Estmator The emprcal Bayes pcture offers an alternatve explanaton for Sten s paradox. From the Bayesan standpont, consder X θ N(θ, 1) (18) θ N(0, τ ), (19) where each s an ndependent experment. Then the posteror densty of θ gven X s θ X N((1 B)X, 1 B), where B = 1 τ + 1. (0) Under squared-error loss, the Bayes estmator s ˆθ = E[θ X ] = (1 B)X. In the context of the orgnal problem, however, the Bayes estmator s unsatsfactory because τ s unknown. Snce X θ N(θ, 1), wrte X = θ + Z, where Z s ndependent of θ and Z N(0, 1). Then X N(0, τ + 1). The natural estmator of B = 1/(τ + 1) s therefore ˆB = c S, where S = X. (1) Thus, the emprcal Bayes argument gves ˆθ = (1 ˆB)X = (1 c )X, () S whch has the form of the James-Sten shrnkage estmator. 10 Emprcal Bayes Approach to the Efron-Morrs Estmator The emprcal Bayes pcture also provdes a route towards the Efron-Morrs estmator. From the Bayesan standpont, consder ndependent experments whch n turn gves X θ N(θ, 1) (3) θ N(µ, τ ), (4) θ X N(µ + (1 B)(X µ), 1 B) (5) X N(µ, 1 B ), where B = 1/(τ + 1).(6) Usng ˆµ = X and ˆB = c/ (X j X) gves the Efron-Morrs estmator ˆθ = X c + (1 (Xj X) )(X X). (7) 11 Equvarance and Mnmaxty n Smultaneous Estmaton Consder X 1 N(θ 1, 1), wth unknown estmand θ 1 R. The estmator ˆθ 1 = X 1 s nvarant under the translaton group G 1 = {g a g a X 1 = X 1 +a}. The class of all equvarant estmators has form X 1 + b. Under the nvarant loss functon L 1 (θ 1, d 1 ) = (θ 1 d 1 ), ˆθ 1 = X 1 s the mnmum rsk equvarance estmator because the normal dstrbuton s symmetrc about ts mean. Now generalze to X N(θ, I), where unknown estmand θ R 1 R... R k has components varyng separately. Consder the translaton group G = G 1 G... G k. The class of all equvarant estmators have form X + b, where constant b R k. Under the nvarant loss functon L(θ, d) = L (θ, d ) = θ d, the mnmum rsk equvarant estmator s X, agan because the normal dstrbuton s symmetrc about ts mean. 4
5 Now reconsder the one-dmensonal estmaton problem under the mnmax crteron. Under the onedmensonal squared-error loss functon L 1 (θ 1, d 1 ) = (θ 1 d 1 ), the Bayes estmator correspondng to the pror θ 1 N(µ, b ) s ˆθ 1 = X 1 + µ/b 1 + 1/b, (8) wth posteror rsk r 1 = 1/(1 + 1/b ). As b, r 1 1. Snce R(θ 1, X 1 ) = E θ1 (θ 1 X 1 ) = 1 for all θ 1, the usual least favorable sequence argument shows that X 1 s mnmax for estmatng θ 1. For the multvarate estmaton problem, take multvarate pror θ N(µ, b I). Then under the multvarate squared-error loss functon L(θ, d) = L (θ, d ) = θ d, the Bayes estmator s ˆθ = (ˆθ 1, ˆθ,..., ˆθ k ), where ˆθ = X + µ/b 1 + 1/b. (9) The posteror rsk becomes r = r = k/(1 + 1/b ). As b, r k. Snce R(θ, X) = E θ (θ X ) = k for all θ, the vector X s mnmax for θ. The above arguments for equvarance and for mnmaxty apply to many other smultaneous estmaton stuatons. Under qute general assumptons, mnmum rsk equvarance and mnmaxty both extend by components. 1 Concluson: Paradox of Admssblty n Smultaneous Estmaton Unlke mnmum rsk equvarance and mnmaxty, the admssblty of component estmators does not extend to the admssblty of the smultaneous estmator. Sten s paradox ponts out clearly that the admssblty of X 1 does not mply the admssblty of the vector X. The Galtonan pcture and the Bayesan pcture offer nsght on why admssblty does not extend by components. In the Galtonan perspectve, the ndependent samples X are coupled through regresson of {(X, θ )} n the X/θ coordnate system. As s typcal n regresson analyss, the predcton of any sngle pont s nfluenced by ts neghborng ponts. In the Bayes perspectve, the ndependent experments X θ are coupled through the common pror for all the θ. In estmatng the common varance of θ, all the ndependent samples X are pooled together to provde a more accurate emprcal Bayes estmate. Mnmum rsk equvarance and mnmaxty are both strong optmalty crtera. Mnmum rsk equvarance wth respect to a transtve group s a concrete property whch defnes a total orderng on the space of equvarant estmators. Mnmaxty s also a concrete property whch defnes a total orderng on the space of estmators. So mposng mnmum rsk equvarance or mnmaxty on component estmators s usually suffcent to guarantee that the same concrete property wll hold over the product space. Admssblty s not a concrete optmalty crteron. Snce the rsk functons of two estmators may cross, comparson of rsk functons n ther entrety does not defne a total orderng on the space of estmators. Because admssblty s a weak crteron representng the absence of optmalty, the product of admssble estmators does not guarantee admssblty. On the other hand, nadmssblty s a concrete optmalty crteron. So n general, the product of nadmssble estmators wll reman nadmssble. 13 References 1. Efron, B. and Morrs, C. N. (1977). Sten s paradox n statstcs. Scentfc Amercan. 36 (5) Lehmann, E. L. (1983). Theory of Pont Estmaton. Wley, New York. 3. Stgler, S. M. (1990). The 1998 Neyman Memoral Lecture: A Galtonan Perspectve on Shrnkage Estmators. Statstcal Scence. 5 (1)
/ n ) are compared. The logic is: if the two
STAT C141, Sprng 2005 Lecture 13 Two sample tests One sample tests: examples of goodness of ft tests, where we are testng whether our data supports predctons. Two sample tests: called as tests of ndependence
More informationChapter 7 Generalized and Weighted Least Squares Estimation. In this method, the deviation between the observed and expected values of
Chapter 7 Generalzed and Weghted Least Squares Estmaton The usual lnear regresson model assumes that all the random error components are dentcally and ndependently dstrbuted wth constant varance. When
More informationSTAT 3008 Applied Regression Analysis
STAT 3008 Appled Regresson Analyss Tutoral : Smple Lnear Regresson LAI Chun He Department of Statstcs, The Chnese Unversty of Hong Kong 1 Model Assumpton To quantfy the relatonshp between two factors,
More informationEstimation: Part 2. Chapter GREG estimation
Chapter 9 Estmaton: Part 2 9. GREG estmaton In Chapter 8, we have seen that the regresson estmator s an effcent estmator when there s a lnear relatonshp between y and x. In ths chapter, we generalzed the
More information2E Pattern Recognition Solutions to Introduction to Pattern Recognition, Chapter 2: Bayesian pattern classification
E395 - Pattern Recognton Solutons to Introducton to Pattern Recognton, Chapter : Bayesan pattern classfcaton Preface Ths document s a soluton manual for selected exercses from Introducton to Pattern Recognton
More informationComposite Hypotheses testing
Composte ypotheses testng In many hypothess testng problems there are many possble dstrbutons that can occur under each of the hypotheses. The output of the source s a set of parameters (ponts n a parameter
More informationChapter 11: Simple Linear Regression and Correlation
Chapter 11: Smple Lnear Regresson and Correlaton 11-1 Emprcal Models 11-2 Smple Lnear Regresson 11-3 Propertes of the Least Squares Estmators 11-4 Hypothess Test n Smple Lnear Regresson 11-4.1 Use of t-tests
More informationDepartment of Quantitative Methods & Information Systems. Time Series and Their Components QMIS 320. Chapter 6
Department of Quanttatve Methods & Informaton Systems Tme Seres and Ther Components QMIS 30 Chapter 6 Fall 00 Dr. Mohammad Zanal These sldes were modfed from ther orgnal source for educatonal purpose only.
More informationLecture 6: Introduction to Linear Regression
Lecture 6: Introducton to Lnear Regresson An Manchakul amancha@jhsph.edu 24 Aprl 27 Lnear regresson: man dea Lnear regresson can be used to study an outcome as a lnear functon of a predctor Example: 6
More informationComparison of the Population Variance Estimators. of 2-Parameter Exponential Distribution Based on. Multiple Criteria Decision Making Method
Appled Mathematcal Scences, Vol. 7, 0, no. 47, 07-0 HIARI Ltd, www.m-hkar.com Comparson of the Populaton Varance Estmators of -Parameter Exponental Dstrbuton Based on Multple Crtera Decson Makng Method
More informationLecture 3 Stat102, Spring 2007
Lecture 3 Stat0, Sprng 007 Chapter 3. 3.: Introducton to regresson analyss Lnear regresson as a descrptve technque The least-squares equatons Chapter 3.3 Samplng dstrbuton of b 0, b. Contnued n net lecture
More information4 Analysis of Variance (ANOVA) 5 ANOVA. 5.1 Introduction. 5.2 Fixed Effects ANOVA
4 Analyss of Varance (ANOVA) 5 ANOVA 51 Introducton ANOVA ANOVA s a way to estmate and test the means of multple populatons We wll start wth one-way ANOVA If the populatons ncluded n the study are selected
More informationCIS526: Machine Learning Lecture 3 (Sept 16, 2003) Linear Regression. Preparation help: Xiaoying Huang. x 1 θ 1 output... θ M x M
CIS56: achne Learnng Lecture 3 (Sept 6, 003) Preparaton help: Xaoyng Huang Lnear Regresson Lnear regresson can be represented by a functonal form: f(; θ) = θ 0 0 +θ + + θ = θ = 0 ote: 0 s a dummy attrbute
More informationWhy Bayesian? 3. Bayes and Normal Models. State of nature: class. Decision rule. Rev. Thomas Bayes ( ) Bayes Theorem (yes, the famous one)
Why Bayesan? 3. Bayes and Normal Models Alex M. Martnez alex@ece.osu.edu Handouts Handoutsfor forece ECE874 874Sp Sp007 If all our research (n PR was to dsappear and you could only save one theory, whch
More information1. Inference on Regression Parameters a. Finding Mean, s.d and covariance amongst estimates. 2. Confidence Intervals and Working Hotelling Bands
Content. Inference on Regresson Parameters a. Fndng Mean, s.d and covarance amongst estmates.. Confdence Intervals and Workng Hotellng Bands 3. Cochran s Theorem 4. General Lnear Testng 5. Measures of
More informationLecture Notes on Linear Regression
Lecture Notes on Lnear Regresson Feng L fl@sdueducn Shandong Unversty, Chna Lnear Regresson Problem In regresson problem, we am at predct a contnuous target value gven an nput feature vector We assume
More informationFeb 14: Spatial analysis of data fields
Feb 4: Spatal analyss of data felds Mappng rregularly sampled data onto a regular grd Many analyss technques for geophyscal data requre the data be located at regular ntervals n space and/or tme. hs s
More informationSee Book Chapter 11 2 nd Edition (Chapter 10 1 st Edition)
Count Data Models See Book Chapter 11 2 nd Edton (Chapter 10 1 st Edton) Count data consst of non-negatve nteger values Examples: number of drver route changes per week, the number of trp departure changes
More informationPsychology 282 Lecture #24 Outline Regression Diagnostics: Outliers
Psychology 282 Lecture #24 Outlne Regresson Dagnostcs: Outlers In an earler lecture we studed the statstcal assumptons underlyng the regresson model, ncludng the followng ponts: Formal statement of assumptons.
More informationChapter 6. Supplemental Text Material
Chapter 6. Supplemental Text Materal S6-. actor Effect Estmates are Least Squares Estmates We have gven heurstc or ntutve explanatons of how the estmates of the factor effects are obtaned n the textboo.
More information8/25/17. Data Modeling. Data Modeling. Data Modeling. Patrice Koehl Department of Biological Sciences National University of Singapore
8/5/17 Data Modelng Patrce Koehl Department of Bologcal Scences atonal Unversty of Sngapore http://www.cs.ucdavs.edu/~koehl/teachng/bl59 koehl@cs.ucdavs.edu Data Modelng Ø Data Modelng: least squares Ø
More informationMATH 829: Introduction to Data Mining and Analysis The EM algorithm (part 2)
1/16 MATH 829: Introducton to Data Mnng and Analyss The EM algorthm (part 2) Domnque Gullot Departments of Mathematcal Scences Unversty of Delaware Aprl 20, 2016 Recall 2/16 We are gven ndependent observatons
More informationLinear Regression Analysis: Terminology and Notation
ECON 35* -- Secton : Basc Concepts of Regresson Analyss (Page ) Lnear Regresson Analyss: Termnology and Notaton Consder the generc verson of the smple (two-varable) lnear regresson model. It s represented
More informationSolutions Homework 4 March 5, 2018
1 Solutons Homework 4 March 5, 018 Soluton to Exercse 5.1.8: Let a IR be a translaton and c > 0 be a re-scalng. ˆb1 (cx + a) cx n + a (cx 1 + a) c x n x 1 cˆb 1 (x), whch shows ˆb 1 s locaton nvarant and
More informationStatistics for Economics & Business
Statstcs for Economcs & Busness Smple Lnear Regresson Learnng Objectves In ths chapter, you learn: How to use regresson analyss to predct the value of a dependent varable based on an ndependent varable
More informationFall 2012 Analysis of Experimental Measurements B. Eisenstein/rev. S. Errede
Fall 0 Analyss of Expermental easurements B. Esensten/rev. S. Errede We now reformulate the lnear Least Squares ethod n more general terms, sutable for (eventually extendng to the non-lnear case, and also
More informationThe Ordinary Least Squares (OLS) Estimator
The Ordnary Least Squares (OLS) Estmator 1 Regresson Analyss Regresson Analyss: a statstcal technque for nvestgatng and modelng the relatonshp between varables. Applcatons: Engneerng, the physcal and chemcal
More informationLINEAR REGRESSION ANALYSIS. MODULE IX Lecture Multicollinearity
LINEAR REGRESSION ANALYSIS MODULE IX Lecture - 31 Multcollnearty Dr. Shalabh Department of Mathematcs and Statstcs Indan Insttute of Technology Kanpur 6. Rdge regresson The OLSE s the best lnear unbased
More informationLecture 2: Prelude to the big shrink
Lecture 2: Prelude to the bg shrnk Last tme A slght detour wth vsualzaton tools (hey, t was the frst day... why not start out wth somethng pretty to look at?) Then, we consdered a smple 120a-style regresson
More informationStatistics for Managers Using Microsoft Excel/SPSS Chapter 13 The Simple Linear Regression Model and Correlation
Statstcs for Managers Usng Mcrosoft Excel/SPSS Chapter 13 The Smple Lnear Regresson Model and Correlaton 1999 Prentce-Hall, Inc. Chap. 13-1 Chapter Topcs Types of Regresson Models Determnng the Smple Lnear
More informationComparison of Regression Lines
STATGRAPHICS Rev. 9/13/2013 Comparson of Regresson Lnes Summary... 1 Data Input... 3 Analyss Summary... 4 Plot of Ftted Model... 6 Condtonal Sums of Squares... 6 Analyss Optons... 7 Forecasts... 8 Confdence
More informationLinear Approximation with Regularization and Moving Least Squares
Lnear Approxmaton wth Regularzaton and Movng Least Squares Igor Grešovn May 007 Revson 4.6 (Revson : March 004). 5 4 3 0.5 3 3.5 4 Contents: Lnear Fttng...4. Weghted Least Squares n Functon Approxmaton...
More informationEcon107 Applied Econometrics Topic 3: Classical Model (Studenmund, Chapter 4)
I. Classcal Assumptons Econ7 Appled Econometrcs Topc 3: Classcal Model (Studenmund, Chapter 4) We have defned OLS and studed some algebrac propertes of OLS. In ths topc we wll study statstcal propertes
More informationChapter 14 Simple Linear Regression
Chapter 4 Smple Lnear Regresson Chapter 4 - Smple Lnear Regresson Manageral decsons often are based on the relatonshp between two or more varables. Regresson analss can be used to develop an equaton showng
More informationSection 8.3 Polar Form of Complex Numbers
80 Chapter 8 Secton 8 Polar Form of Complex Numbers From prevous classes, you may have encountered magnary numbers the square roots of negatve numbers and, more generally, complex numbers whch are the
More informationStatistics for Business and Economics
Statstcs for Busness and Economcs Chapter 11 Smple Regresson Copyrght 010 Pearson Educaton, Inc. Publshng as Prentce Hall Ch. 11-1 11.1 Overvew of Lnear Models n An equaton can be ft to show the best lnear
More informationMaximum Likelihood Estimation (MLE)
Maxmum Lkelhood Estmaton (MLE) Ken Kreutz-Delgado (Nuno Vasconcelos) ECE 175A Wnter 01 UCSD Statstcal Learnng Goal: Gven a relatonshp between a feature vector x and a vector y, and d data samples (x,y
More informationLINEAR REGRESSION ANALYSIS. MODULE IX Lecture Multicollinearity
LINEAR REGRESSION ANALYSIS MODULE IX Lecture - 30 Multcollnearty Dr. Shalabh Department of Mathematcs and Statstcs Indan Insttute of Technology Kanpur 2 Remedes for multcollnearty Varous technques have
More informationClassification as a Regression Problem
Target varable y C C, C,, ; Classfcaton as a Regresson Problem { }, 3 L C K To treat classfcaton as a regresson problem we should transform the target y nto numercal values; The choce of numercal class
More informationComputing MLE Bias Empirically
Computng MLE Bas Emprcally Kar Wa Lm Australan atonal Unversty January 3, 27 Abstract Ths note studes the bas arses from the MLE estmate of the rate parameter and the mean parameter of an exponental dstrbuton.
More informationStat 543 Exam 2 Spring 2016
Stat 543 Exam 2 Sprng 206 I have nether gven nor receved unauthorzed assstance on ths exam. Name Sgned Date Name Prnted Ths Exam conssts of questons. Do at least 0 of the parts of the man exam. I wll score
More informationStat260: Bayesian Modeling and Inference Lecture Date: February 22, Reference Priors
Stat60: Bayesan Modelng and Inference Lecture Date: February, 00 Reference Prors Lecturer: Mchael I. Jordan Scrbe: Steven Troxler and Wayne Lee In ths lecture, we assume that θ R; n hgher-dmensons, reference
More informationLossy Compression. Compromise accuracy of reconstruction for increased compression.
Lossy Compresson Compromse accuracy of reconstructon for ncreased compresson. The reconstructon s usually vsbly ndstngushable from the orgnal mage. Typcally, one can get up to 0:1 compresson wth almost
More informationGoodness of fit and Wilks theorem
DRAFT 0.0 Glen Cowan 3 June, 2013 Goodness of ft and Wlks theorem Suppose we model data y wth a lkelhood L(µ) that depends on a set of N parameters µ = (µ 1,...,µ N ). Defne the statstc t µ ln L(µ) L(ˆµ),
More informationChapter 9: Statistical Inference and the Relationship between Two Variables
Chapter 9: Statstcal Inference and the Relatonshp between Two Varables Key Words The Regresson Model The Sample Regresson Equaton The Pearson Correlaton Coeffcent Learnng Outcomes After studyng ths chapter,
More informationMLE and Bayesian Estimation. Jie Tang Department of Computer Science & Technology Tsinghua University 2012
MLE and Bayesan Estmaton Je Tang Department of Computer Scence & Technology Tsnghua Unversty 01 1 Lnear Regresson? As the frst step, we need to decde how we re gong to represent the functon f. One example:
More informationThe Second Anti-Mathima on Game Theory
The Second Ant-Mathma on Game Theory Ath. Kehagas December 1 2006 1 Introducton In ths note we wll examne the noton of game equlbrum for three types of games 1. 2-player 2-acton zero-sum games 2. 2-player
More informationx = , so that calculated
Stat 4, secton Sngle Factor ANOVA notes by Tm Plachowsk n chapter 8 we conducted hypothess tests n whch we compared a sngle sample s mean or proporton to some hypotheszed value Chapter 9 expanded ths to
More informationKernel Methods and SVMs Extension
Kernel Methods and SVMs Extenson The purpose of ths document s to revew materal covered n Machne Learnng 1 Supervsed Learnng regardng support vector machnes (SVMs). Ths document also provdes a general
More informationU.C. Berkeley CS294: Beyond Worst-Case Analysis Luca Trevisan September 5, 2017
U.C. Berkeley CS94: Beyond Worst-Case Analyss Handout 4s Luca Trevsan September 5, 07 Summary of Lecture 4 In whch we ntroduce semdefnte programmng and apply t to Max Cut. Semdefnte Programmng Recall that
More informationThe Order Relation and Trace Inequalities for. Hermitian Operators
Internatonal Mathematcal Forum, Vol 3, 08, no, 507-57 HIKARI Ltd, wwwm-hkarcom https://doorg/0988/mf088055 The Order Relaton and Trace Inequaltes for Hermtan Operators Y Huang School of Informaton Scence
More informationECONOMICS 351*-A Mid-Term Exam -- Fall Term 2000 Page 1 of 13 pages. QUEEN'S UNIVERSITY AT KINGSTON Department of Economics
ECOOMICS 35*-A Md-Term Exam -- Fall Term 000 Page of 3 pages QUEE'S UIVERSITY AT KIGSTO Department of Economcs ECOOMICS 35* - Secton A Introductory Econometrcs Fall Term 000 MID-TERM EAM ASWERS MG Abbott
More informationStat 543 Exam 2 Spring 2016
Stat 543 Exam 2 Sprng 2016 I have nether gven nor receved unauthorzed assstance on ths exam. Name Sgned Date Name Prnted Ths Exam conssts of 11 questons. Do at least 10 of the 11 parts of the man exam.
More informationChapter 15 Student Lecture Notes 15-1
Chapter 15 Student Lecture Notes 15-1 Basc Busness Statstcs (9 th Edton) Chapter 15 Multple Regresson Model Buldng 004 Prentce-Hall, Inc. Chap 15-1 Chapter Topcs The Quadratc Regresson Model Usng Transformatons
More information8 : Learning in Fully Observed Markov Networks. 1 Why We Need to Learn Undirected Graphical Models. 2 Structural Learning for Completely Observed MRF
10-708: Probablstc Graphcal Models 10-708, Sprng 2014 8 : Learnng n Fully Observed Markov Networks Lecturer: Erc P. Xng Scrbes: Meng Song, L Zhou 1 Why We Need to Learn Undrected Graphcal Models In the
More informationx i1 =1 for all i (the constant ).
Chapter 5 The Multple Regresson Model Consder an economc model where the dependent varable s a functon of K explanatory varables. The economc model has the form: y = f ( x,x,..., ) xk Approxmate ths by
More informationANOMALIES OF THE MAGNITUDE OF THE BIAS OF THE MAXIMUM LIKELIHOOD ESTIMATOR OF THE REGRESSION SLOPE
P a g e ANOMALIES OF THE MAGNITUDE OF THE BIAS OF THE MAXIMUM LIKELIHOOD ESTIMATOR OF THE REGRESSION SLOPE Darmud O Drscoll ¹, Donald E. Ramrez ² ¹ Head of Department of Mathematcs and Computer Studes
More informationGaussian process classification: a message-passing viewpoint
Gaussan process classfcaton: a message-passng vewpont Flpe Rodrgues fmpr@de.uc.pt November 014 Abstract The goal of ths short paper s to provde a message-passng vewpont of the Expectaton Propagaton EP
More informationUsing T.O.M to Estimate Parameter of distributions that have not Single Exponential Family
IOSR Journal of Mathematcs IOSR-JM) ISSN: 2278-5728. Volume 3, Issue 3 Sep-Oct. 202), PP 44-48 www.osrjournals.org Usng T.O.M to Estmate Parameter of dstrbutons that have not Sngle Exponental Famly Jubran
More informationMIMA Group. Chapter 2 Bayesian Decision Theory. School of Computer Science and Technology, Shandong University. Xin-Shun SDU
Group M D L M Chapter Bayesan Decson heory Xn-Shun Xu @ SDU School of Computer Scence and echnology, Shandong Unversty Bayesan Decson heory Bayesan decson theory s a statstcal approach to data mnng/pattern
More informationProperties of Least Squares
Week 3 3.1 Smple Lnear Regresson Model 3. Propertes of Least Squares Estmators Y Y β 1 + β X + u weekly famly expendtures X weekly famly ncome For a gven level of x, the expected level of food expendtures
More informationSIMPLE LINEAR REGRESSION
Smple Lnear Regresson and Correlaton Introducton Prevousl, our attenton has been focused on one varable whch we desgnated b x. Frequentl, t s desrable to learn somethng about the relatonshp between two
More informationComputation of Higher Order Moments from Two Multinomial Overdispersion Likelihood Models
Computaton of Hgher Order Moments from Two Multnomal Overdsperson Lkelhood Models BY J. T. NEWCOMER, N. K. NEERCHAL Department of Mathematcs and Statstcs, Unversty of Maryland, Baltmore County, Baltmore,
More informationGeneralized Linear Methods
Generalzed Lnear Methods 1 Introducton In the Ensemble Methods the general dea s that usng a combnaton of several weak learner one could make a better learner. More formally, assume that we have a set
More informationa. (All your answers should be in the letter!
Econ 301 Blkent Unversty Taskn Econometrcs Department of Economcs Md Term Exam I November 8, 015 Name For each hypothess testng n the exam complete the followng steps: Indcate the test statstc, ts crtcal
More informationChapter 5 Multilevel Models
Chapter 5 Multlevel Models 5.1 Cross-sectonal multlevel models 5.1.1 Two-level models 5.1.2 Multple level models 5.1.3 Multple level modelng n other felds 5.2 Longtudnal multlevel models 5.2.1 Two-level
More informationxp(x µ) = 0 p(x = 0 µ) + 1 p(x = 1 µ) = µ
CSE 455/555 Sprng 2013 Homework 7: Parametrc Technques Jason J. Corso Computer Scence and Engneerng SUY at Buffalo jcorso@buffalo.edu Solutons by Yngbo Zhou Ths assgnment does not need to be submtted and
More informationFeature Selection: Part 1
CSE 546: Machne Learnng Lecture 5 Feature Selecton: Part 1 Instructor: Sham Kakade 1 Regresson n the hgh dmensonal settng How do we learn when the number of features d s greater than the sample sze n?
More informationLectures - Week 4 Matrix norms, Conditioning, Vector Spaces, Linear Independence, Spanning sets and Basis, Null space and Range of a Matrix
Lectures - Week 4 Matrx norms, Condtonng, Vector Spaces, Lnear Independence, Spannng sets and Bass, Null space and Range of a Matrx Matrx Norms Now we turn to assocatng a number to each matrx. We could
More informationBasic Business Statistics, 10/e
Chapter 13 13-1 Basc Busness Statstcs 11 th Edton Chapter 13 Smple Lnear Regresson Basc Busness Statstcs, 11e 009 Prentce-Hall, Inc. Chap 13-1 Learnng Objectves In ths chapter, you learn: How to use regresson
More informationBiostatistics 360 F&t Tests and Intervals in Regression 1
Bostatstcs 360 F&t Tests and Intervals n Regresson ORIGIN Model: Y = X + Corrected Sums of Squares: X X bar where: s the y ntercept of the regresson lne (translaton) s the slope of the regresson lne (scalng
More informationBayesian predictive Configural Frequency Analysis
Psychologcal Test and Assessment Modelng, Volume 54, 2012 (3), 285-292 Bayesan predctve Confgural Frequency Analyss Eduardo Gutérrez-Peña 1 Abstract Confgural Frequency Analyss s a method for cell-wse
More informationRockefeller College University at Albany
Rockefeller College Unverst at Alban PAD 705 Handout: Maxmum Lkelhood Estmaton Orgnal b Davd A. Wse John F. Kenned School of Government, Harvard Unverst Modfcatons b R. Karl Rethemeer Up to ths pont n
More informationLaboratory 1c: Method of Least Squares
Lab 1c, Least Squares Laboratory 1c: Method of Least Squares Introducton Consder the graph of expermental data n Fgure 1. In ths experment x s the ndependent varable and y the dependent varable. Clearly
More informationGlobal Sensitivity. Tuesday 20 th February, 2018
Global Senstvty Tuesday 2 th February, 28 ) Local Senstvty Most senstvty analyses [] are based on local estmates of senstvty, typcally by expandng the response n a Taylor seres about some specfc values
More informationLinear regression. Regression Models. Chapter 11 Student Lecture Notes Regression Analysis is the
Chapter 11 Student Lecture Notes 11-1 Lnear regresson Wenl lu Dept. Health statstcs School of publc health Tanjn medcal unversty 1 Regresson Models 1. Answer What Is the Relatonshp Between the Varables?.
More informationThe Gaussian classifier. Nuno Vasconcelos ECE Department, UCSD
he Gaussan classfer Nuno Vasconcelos ECE Department, UCSD Bayesan decson theory recall that we have state of the world X observatons g decson functon L[g,y] loss of predctng y wth g Bayes decson rule s
More informationStatistics MINITAB - Lab 2
Statstcs 20080 MINITAB - Lab 2 1. Smple Lnear Regresson In smple lnear regresson we attempt to model a lnear relatonshp between two varables wth a straght lne and make statstcal nferences concernng that
More informationwhere I = (n x n) diagonal identity matrix with diagonal elements = 1 and off-diagonal elements = 0; and σ 2 e = variance of (Y X).
11.4.1 Estmaton of Multple Regresson Coeffcents In multple lnear regresson, we essentally solve n equatons for the p unnown parameters. hus n must e equal to or greater than p and n practce n should e
More information2016 Wiley. Study Session 2: Ethical and Professional Standards Application
6 Wley Study Sesson : Ethcal and Professonal Standards Applcaton LESSON : CORRECTION ANALYSIS Readng 9: Correlaton and Regresson LOS 9a: Calculate and nterpret a sample covarance and a sample correlaton
More informationLimited Dependent Variables
Lmted Dependent Varables. What f the left-hand sde varable s not a contnuous thng spread from mnus nfnty to plus nfnty? That s, gven a model = f (, β, ε, where a. s bounded below at zero, such as wages
More informationEconomics 130. Lecture 4 Simple Linear Regression Continued
Economcs 130 Lecture 4 Contnued Readngs for Week 4 Text, Chapter and 3. We contnue wth addressng our second ssue + add n how we evaluate these relatonshps: Where do we get data to do ths analyss? How do
More informationChapter 8 Indicator Variables
Chapter 8 Indcator Varables In general, e explanatory varables n any regresson analyss are assumed to be quanttatve n nature. For example, e varables lke temperature, dstance, age etc. are quanttatve n
More informationLaboratory 3: Method of Least Squares
Laboratory 3: Method of Least Squares Introducton Consder the graph of expermental data n Fgure 1. In ths experment x s the ndependent varable and y the dependent varable. Clearly they are correlated wth
More informationHere is the rationale: If X and y have a strong positive relationship to one another, then ( x x) will tend to be positive when ( y y)
Secton 1.5 Correlaton In the prevous sectons, we looked at regresson and the value r was a measurement of how much of the varaton n y can be attrbuted to the lnear relatonshp between y and x. In ths secton,
More informationISSN: ISO 9001:2008 Certified International Journal of Engineering and Innovative Technology (IJEIT) Volume 3, Issue 1, July 2013
ISSN: 2277-375 Constructon of Trend Free Run Orders for Orthogonal rrays Usng Codes bstract: Sometmes when the expermental runs are carred out n a tme order sequence, the response can depend on the run
More information3.1 Expectation of Functions of Several Random Variables. )' be a k-dimensional discrete or continuous random vector, with joint PMF p (, E X E X1 E X
Statstcs 1: Probablty Theory II 37 3 EPECTATION OF SEVERAL RANDOM VARIABLES As n Probablty Theory I, the nterest n most stuatons les not on the actual dstrbuton of a random vector, but rather on a number
More informationENG 8801/ Special Topics in Computer Engineering: Pattern Recognition. Memorial University of Newfoundland Pattern Recognition
EG 880/988 - Specal opcs n Computer Engneerng: Pattern Recognton Memoral Unversty of ewfoundland Pattern Recognton Lecture 7 May 3, 006 http://wwwengrmunca/~charlesr Offce Hours: uesdays hursdays 8:30-9:30
More informationLecture 20: Hypothesis testing
Lecture : Hpothess testng Much of statstcs nvolves hpothess testng compare a new nterestng hpothess, H (the Alternatve hpothess to the borng, old, well-known case, H (the Null Hpothess or, decde whether
More informationUncertainty as the Overlap of Alternate Conditional Distributions
Uncertanty as the Overlap of Alternate Condtonal Dstrbutons Olena Babak and Clayton V. Deutsch Centre for Computatonal Geostatstcs Department of Cvl & Envronmental Engneerng Unversty of Alberta An mportant
More informationPolynomial Regression Models
LINEAR REGRESSION ANALYSIS MODULE XII Lecture - 6 Polynomal Regresson Models Dr. Shalabh Department of Mathematcs and Statstcs Indan Insttute of Technology Kanpur Test of sgnfcance To test the sgnfcance
More informationIntroduction to Regression
Introducton to Regresson Dr Tom Ilvento Department of Food and Resource Economcs Overvew The last part of the course wll focus on Regresson Analyss Ths s one of the more powerful statstcal technques Provdes
More informationCS 2750 Machine Learning. Lecture 5. Density estimation. CS 2750 Machine Learning. Announcements
CS 750 Machne Learnng Lecture 5 Densty estmaton Mlos Hauskrecht mlos@cs.ptt.edu 539 Sennott Square CS 750 Machne Learnng Announcements Homework Due on Wednesday before the class Reports: hand n before
More informationand V is a p p positive definite matrix. A normal-inverse-gamma distribution.
OSR Journal of athematcs (OSR-J) e-ssn: 78-578, p-ssn: 39-765X. Volume 3, ssue 3 Ver. V (ay - June 07), PP 68-7 www.osrjournals.org Comparng The Performance of Bayesan And Frequentst Analyss ethods of
More informationLecture 4 Hypothesis Testing
Lecture 4 Hypothess Testng We may wsh to test pror hypotheses about the coeffcents we estmate. We can use the estmates to test whether the data rejects our hypothess. An example mght be that we wsh to
More information10-701/ Machine Learning, Fall 2005 Homework 3
10-701/15-781 Machne Learnng, Fall 2005 Homework 3 Out: 10/20/05 Due: begnnng of the class 11/01/05 Instructons Contact questons-10701@autonlaborg for queston Problem 1 Regresson and Cross-valdaton [40
More informationLecture 9: Linear regression: centering, hypothesis testing, multiple covariates, and confounding
Recall: man dea of lnear regresson Lecture 9: Lnear regresson: centerng, hypothess testng, multple covarates, and confoundng Sandy Eckel seckel@jhsph.edu 6 May 8 Lnear regresson can be used to study an
More informationLecture 9: Linear regression: centering, hypothesis testing, multiple covariates, and confounding
Lecture 9: Lnear regresson: centerng, hypothess testng, multple covarates, and confoundng Sandy Eckel seckel@jhsph.edu 6 May 008 Recall: man dea of lnear regresson Lnear regresson can be used to study
More informationEstimating the Fundamental Matrix by Transforming Image Points in Projective Space 1
Estmatng the Fundamental Matrx by Transformng Image Ponts n Projectve Space 1 Zhengyou Zhang and Charles Loop Mcrosoft Research, One Mcrosoft Way, Redmond, WA 98052, USA E-mal: fzhang,cloopg@mcrosoft.com
More informationEcon107 Applied Econometrics Topic 9: Heteroskedasticity (Studenmund, Chapter 10)
I. Defnton and Problems Econ7 Appled Econometrcs Topc 9: Heteroskedastcty (Studenmund, Chapter ) We now relax another classcal assumpton. Ths s a problem that arses often wth cross sectons of ndvduals,
More information