Nonlinear Estimation. Professor David H. Staelin

Transcription

1 Nonlinear Estimation Professor Davi H. Staelin Massachusetts Institute of Technology Lec22.5-1

2 [ DD 1 2] ˆ = 1 Best Fit, "Linear Regression" Case I: Nonlinear Physics Data Otimum Estimator P() ˆ D 1 D 2 Sloe P{} Parameter Probability Distribution 0 Minimum Square Error (MSE) is usually the "otimum" sought. It is imortant that the "training" ata use for regression is similar to, but ineenent of, the "test" ata use for erformance evaluation. Lec J1

3 Case II: Non-Gaussian Statistics Best linear estimator Maximum a osteriori robability "MAP" estimator A 0 ˆ A P{ A } Best Nonlinear Estimator (avois ˆ < 0). ˆ A Minimizes MSE given A Note: Linear estimator can set <0 ˆ (which is non-hysical): nonlinear estimator aroaches correct asymtotes as Note: Physics here is linear. Can use olynomials etc. for ˆ(), or recursive linear estimators uating D. ± Lec J2

4 Linear Estimates for Nonlinear Problems Say = a + a + a an = b + b + b ( ) Otimum Linear Estimate 2 1 ( ) 2 ( ) ( 1 2) - sace ab 2 1 ( ) Then = b b /b 1 2 Linear estimates are subotimum for 1 or 2 alone. = a + a+ a b b /b = c + c + c (efines lane in,, Therefore ˆ = = c0 + 1 c2 2 / c 1 where c1 = a1 0 b ˆ 1, 2 can be linear in 1, 2 an (noiseless case) erfect, even though, is nonlinear ( ) 1 2 Truth (without noise) Otimum Linear Estimate In general, the more varie the ata, the smaller the erformance ga between linear estimators an nonlinear ones, which are a suerset. 2 Lec J3

5 Linear Estimates for Nonlinear Problems, ( 1 2) ( 1 2) ( 2 1) ( ) Plane ˆ 1, 2 is the linear estimator 1 ( ) 2 Plane ˆ, has two 1 2 angular egrees of freeom to use for minimizing ( ). Lec J4

6 Generalization to Nth-Orer Nonlinearities 2 n Let 1 = c1+ a11 + a a1n 2 n 2 = c2 + a21+ a a2n 2 n n = cn + an1+ an2 + + ann where 1, 2,, n are observe noise-free ata relate to by an nth-orer olynomial. Claim: In non-singular cases there exists an exact linear estimator ˆ = D + constant. Note: For erfect linear estimation the number of ifferent observations must equal or excee n, the maximum orer of the olynomials characterizing the hysics. Lec J5

7 Nonlinear Estimators for Nonlinear Problems ˆ = augmente via olynomials 2) Rank reuction first via KLT 3) Neural nets Aroaches:1) D aug where is Case (1): = ( ˆ ) 4) Iterations with D 1,,,,,,, for examle aug Case (2): Use of KLT to Reuce Rank Otion A: KLT Nonlinear Augmentation aug Daug ˆ Lec Dro Useful if is of High Orer L1

8 Karhunen-Loeve Transform (KLT) Lec = K transforms a noisy reunant ata vector into a reuce-rank vector with most variance in, least [ ] 1 in n, where E i j =δλ ij i, an λi λj> i λ t t C = E = K λ2 K λn e.g. h 10KM T(h) 2 3 KLT The KLT is essentially the same as Princial Comonent Analysis (PCA) an Emirical Orthogonal Functions (EOF). 1 Basis Functions Are Rows of K L2

9 Nonlinear Estimators Using Rank Reuction Otion B: Nonlinear augmentation KLT D ˆ Dro Useful if is very nonlinear Otion C: KLT Nonlinear augmentation aug KLT aug D aug ˆ ro ro Lec L3

10 Neural Networks (NN) NN ˆ Neural nets comute comlex olynomials with great efficiency, simlicity. 1 1 N W 01 W 11 W N1 W NM Σ Σ 1 M 1 M NN coefficients comute via relaxation otimization back roagation. NN 1 2 WNM NN NN 3 ˆ "Hien Layers" NN may be recursive or urely Fee-Forwar (FFNN) Lec L4

11 Neural Networks (NN) Weights W ij are foun by guessing, then erturbing to reuce cost function on ˆ over "training set" of known,. Training can be teious. Commercial software tools are commonly use. Degrees of freeom in training set shoul excee the number of weights by a moest factor of ~ Risks of "overtraining" are reuce by monitoring NN erformance on ineenent test ata. The more nonlinear roblems nee more layers an more internal noes, as etermine emirically. Neural nets can also erform recognition tasks. Simle neural nets are easier to train correctly, so merge with linear techniques, e.g. NN Linear Estimator ˆ Lec L5

12 Accommoating Nonlinearity via Iteration A. Library Retrievals Do Di i Dk Dk Do Successive D i Chosen from Library as f i Lec M1

13 Accommoating Nonlinearity via Iteration B. Physics Recomutation o "First Guess" + + D i + o n i= 1 + i n ˆ i + 1 W Nonlinear Physics Essential only that D always yiels right irection (so errors shrink). in i Lec M2

14 Accommoating Nonlinearity via Iteration C. Satial Iteration Sace i Works well if ata is smooth. i Delay + + i i + D + i 1 i 1 Delay i Alternate Metho (Use if > Threshol i Lec M3

15 Multi-Dimensional Estimation e.g., Atmosheric Temerature Profiles ( ) ( ) = ( ) 1) Tˆ h,x,y f x,y ( ) ( ) = ( ) ( ± ± ) 2) Tˆ h,x,y f x,y, x,y,... Estimate imrove by aitional correlations an egrees of freeom. Lec M4

16 Genetic Algorithms Lec ) Characterize algorithm by segmente character string, e.g., a binary number. 2) Significance of string is efine, e.g., A. As weights in a linear estimator or neural net. B. It characterizes the architecture of a neural net. 3) Many algorithms (strings) are teste, an the metrics for each are evaluate for some ensemble of test cases. 4) Ranomly combine elements of best algorithms to form new ones; some ranom changes may be mae too. 5) Iterate stes (3) an(4) until asymmtotic otimum is reache. Can be use for attern recognition, signal etection, arameter estimation, an other uroses. 6) Proer choice of "gene" size accelerates convergence (genes swae as units). M5