Maximum likelihood fitting through least squares algorithms

Transcription

1 Maximum likelihood fitting through least squares algorithms Rasmus Bro, Nicholaos Sidiropoulos & Age Smilde Chemometrics Group, The Royal Veterinary and Agricultural University, Denmark, Department of Electronic and Computer Engineering, Chania - Crete, Greece, nikos@telecom.tuc.gr Process Analysis and Chemometrics, University of Amsterdam, asmilde@its.chem.uva.nl

2 Maximum likelihood fitting through least squares algorithms Background Algorithm Majorization MILES Application Conclusion

3 MILES: Background Objective in least squares fitting Minimize summed squared residual error MLR : σ ( bz, y) = y Zb = ( y Zb) T ( y Zb) LS PCA : T T σ ( T, P Z) = vecz vec ( TP ) = ( vecz vec ( TP )) T ( vecx LS T vec ( TP )) General : σ ( m x) = x m = ( x m) T ( x m) LS

4 Vectorizing a model Example PCA e T e = e T Ie Least squares

5 Weighted LS Weighted least squares Different size of each residual Ex. auto-scaling in PCA Ex. Weighted linear regression Ex. Generalized least squares w w w 3 w 4 e T W T We w J

6 Error covariance Error covariance Sometimes errors are correlated E.g. if one residual is high, another is too Example from Wentzell et al. 999 Five replicate spectra Residuals are consistently of same size for one spectrum Absorbance error x 0-3 Errors of five replicates nm Importance A residual is fine if neighbor is similar A residual is bad if neighbor is dissimilar (indicates systematic variation that should be modeled) Wentzell PD, Lohnes MT, Maximum likelihood principal component analysis with correlated measurement errors: theoretical and practical considerations, Chemom Intell Lab Syst, 999, 45,

7 Error covariance e T W T We Maximum likelihood Allows for optimal estimates even when errors are covarying Minimize e T W T We w w w 3 w 4 w 4 w 4J w J

8 Algorithm Deriving an algorithm Problem: σ (, ) = ( ) = ( ) T T ( ) ML m x W W x m x m W W x m Some models easy to fit but in general, no closed-form solution Iterative majorization provides a possible general solution Loss function. 0.6 σ ML (m x,w) σ maj (m x,w,m c c) o Kiers HAL, Majorization as a tool for optimizing a class of matrix functions, Psychometrika, 990, 55, Heiser WJ, Convergent computation by iterative majorization: Theory and applications in multidimensional data analysis, Recent advances in descriptive multivariate analysis, (Ed. Krzanowski,WJ), 995, Parameter 5 0 m 5 m C+ m C

9 MILES maximum likelihood Algorithm MILES (Maximum likelihood via Iterative Least squares EStimation) Enables weighted least squares and maximum likelihood fitting of any model which has a least squares algorithm. Initialize model, m 0, with LS, set c := 0;. T q = m + / β W W( x m ) c 3. m c+ = argmin m q m ϒ 4. c := c+; go to step until convergence F c Calculate q Fit LS model to q instead of to data

10 q = m + W W x m c T / β ( c) Exampe MILES-PCA Initialize model, m 0, using centered LS-PCA model (m 0 =vectp T ) of the data, and set c := 0;. T q= m + / β W W( x m ) c c. Q = reshape(q); 3. T, P are found from LS-PCA of Q, 4. m c+ = vectp T ; c := c+; 5. Continue till convergence

11 ε Example: fluorescence samples containing L-phenylalanine, L-3,4- dihydroxyphenylalanine (DOPA),,4-dihydroxybenzene & L- tryptophan Baunsgaard D, Factors affecting 3-way modelling (PARAFAC) of fluorescence landscapes, The Royal Veterinary & Agricultural University, 999

12 Example: fluorescence samples containing L-phenylalanine, L-3,4dihydroxyphenylalanine (DOPA),,4-dihydroxybenzene & Ltryptophan ε Baunsgaard D, Factors affecting 3-way modelling (PARAFAC) of fluorescence landscapes, The Royal Veterinary & Agricultural University, 999

13 ε Example: fluorescence samples containing L-phenylalanine, L-3,4- dihydroxyphenylalanine (DOPA),,4-dihydroxybenzene & L- tryptophan Three types of unwanted variation Measurement error (~iid Gaussian) Rayleigh and Raman scatter Non-chemical area Baunsgaard D, Factors affecting 3-way modelling (PARAFAC) of fluorescence landscapes, The Royal Veterinary & Agricultural University, 999

14 Defining weights Three types of unwanted variation Measurement error (~iid Gaussian) Rayleigh and Raman scatter Non-chemical area Weights are not statistically based, but based on knowledge of artefacts

15 Example: fluorescence RAW DATA MILES interpretation of data MILES PARAFAC Least squares PARAFAC Artifact

16 Example: fluorescence Loading Loading Emission spectra from 00 resamplings Emission /nm Least squares missing Maximum likelihood R. Bro, N. D. Sidiropoulos, and A. K. Smilde. Maximum likelihood fitting using simple least squares algorithms. Journal of Chemometrics, 00

17 MILES in general MILES - general algorithm applicable to all problems where least squares algorithm exist Very simple to implement Enables simple test of algorithms Not fast (can be optimized) Examples on MILES (in matlab) and applications at