10. Multi-objective least squares

Transcription

1 L Vandenberghe ECE133A (Winter 2018) 10 Multi-objective least squares multi-objective least squares regularized data fitting control estimation and inversion 10-1

2 Multi-objective least squares we have several objectives J 1 = A 1 x b 1 2,, J k = A k x b k 2 A i is an m i n matrix, b i is an m i -vector we seek one x that makes all k objectives small usually there is a trade-off: no single x minimizes all objectives simultaneously Weighted least squares formulation: find x that minimizes λ 1 A 1 x b λ k A k x b k 2 coefficients λ 1,, λ k are positive weights weights λ i express relative importance of different objectives without loss of generality, we can choose λ 1 = 1 Multi-objective least squares 10-2

3 Solution of weighted least squares weighted least squares is equivalent to a standard least squares problem minimize λ1 A 1 λ2 A 2 λk A k x λ1 b 1 λ2 b 2 λk b k 2 solution is unique if the stacked matrix has linearly independent columns each matrix A i may have linearly dependent columns (or be a wide matrix) it the stacked matrix has linearly independent columns, the solution is ˆx = ( λ 1 A T 1 A λ k A T k A k ) 1 ( λ1 A T 1 b λ k A T k b k ) Multi-objective least squares 10-3

4 Example with two objectives A 1 and A 2 are 10 5 minimize A 1 x b λ A 2 x b ˆx 1 (λ) ˆx 2 (λ) ˆx 3 (λ) ˆx 4 (λ) ˆx 5 (λ) λ plot shows weighted least squares solution ˆx(λ) as function of weight λ Multi-objective least squares 10-4

5 Example with two objectives minimize A 1 x b λ A 2 x b J2(λ) λ = J 1 (λ) J 2 (λ) 6 4 λ = 1 λ = λ J 1 (λ) left figure shows J 1 (λ) = A 1ˆx(λ) b 1 2 and J 2 (λ) = A 2ˆx(λ) b 2 2 right figure shows optimal trade-off curve of J 2 (λ) versus J 1 (λ) Multi-objective least squares 10-5

6 Outline multi-objective least squares regularized data fitting control estimation and inversion

7 Motivation consider linear-in-parameters model ˆf(x) = θ 1 f 1 (x) + + θ p f p (x) we assume f 1 (x) is the constant function 1 we fit the model ˆf(x) to examples (x (1), y (1) ),, (x (N), y (N) ) large coefficient θ i makes model more sensitive to changes in f i (x) keeping θ 2,, θ p small helps avoid over-fitting this leads to two objectives: J 1 (θ) = N ( ˆf(x (k) ) y (k) ) 2, J 2 (θ) = k=1 p j=2 θ 2 j primary objective J 1 (θ) is sum of squares of prediction errors Multi-objective least squares 10-6

8 Weighted least squares formulation minimize J 1 (θ) + λj 2 (θ) = λ is positive regularization parameter N ( ˆf(x (k) ) y (k) ) 2 + λ k=1 equivalent to least squares problem: minimize [ A 1 λa2 ] θ [ y d 0 ] 2 p j=2 θ 2 j with y d = (y (1),, y (N) ), A 1 = 1 f 2 (x (1) ) f p (x (1) ) 1 f 2 (x (2) ) f p (x (2) ) 1 f 2 (x (N) ) f p (x (N) ), A 2 = stacked matrix has linearly independent columns (for positive λ) value of λ can be chosen by out-of-sample validation or cross-validation Multi-objective least squares 10-7

9 Example Train Test x solid line is signal used to generate synthetic (simulated) data 10 blue points are used as training set; 20 red points are used as test set we fit a model with five parameters θ 1,, θ 5 : ˆf(x) = θ θ k+1 cos(ω k x + φ k ) (with given ω k, φ k ) k=1 Multi-objective least squares 10-8

10 Result of regularized least squares fit Train Test RMS error versus λ 2 1 Coefficients versus λ θ 1 θ 2 θ 3 θ 4 θ λ λ minimum test RMS error is for λ around 008 increasing λ shrinks the coefficients θ 2,, θ 5 dashed lines show coefficients used to generate the data for λ near 08, estimated coefficients are close to these true values Multi-objective least squares 10-9

11 Outline multi-objective least squares regularized data fitting control estimation and inversion

12 Control y = Ax + b x is n-vector of actions or inputs y is m-vector of results or outputs relation between inputs and outputs is known affine function goal is to choose inputs to optimize different objectives on x and y Multi-objective least squares 10-10

13 Optimal input design Linear dynamical system y(t) = h 0 u(t) + h 1 u(t 1) + h 2 u(t 2) + + h t u(0) output y(t) and input u(t) are scalar we assume input u(t) is zero for t < 0 coefficients h 0, h 1, are the impulse response coefficients output is convolution of input with impulse response Optimal input design optimization variable is the input sequence x = (u(0), u(1),, u(n)) goal is to track a desired output using a small and slowly varying input Multi-objective least squares 10-11

14 Input design objectives minimize J t (x) + λ v J v (x) + λ m J m (x) track desired output y des over an interval [0, N]: J t (x) = N (y(t) y des (t)) 2 t=0 use a small and slowly varying input signal: J m (x) = N u(t) 2, J v (x) = t=0 (u(t + 1) u(t)) 2 N 1 t=0 Multi-objective least squares 10-12

15 Tracking error N J t (x) = (y(t) y des (t)) 2 t=0 = A t x b t 2 with A t = h h 1 h h 2 h 1 h h N 1 h N 2 h N 3 h 0 0 h N h N 1 h N 2 h 1 h 0, b t = y des (0) y des (1) y des (2) y des (N 1) y des (N) Multi-objective least squares 10-13

16 Input variation and magnitude Input variation J v (x) = (u(t + 1) u(t)) 2 = Dx 2 N 1 t=0 with D the N (N + 1) matrix D = Input magnitude J m (x) = N u(t) 2 = x 2 t=0 Multi-objective least squares 10-14

17 Example λ v = 0, small λ m u(t) t y(t) t larger λ v u(t) 0 2 y(t) t t Multi-objective least squares 10-15

18 Outline multi-objective least squares regularized data fitting control estimation and inversion

19 Estimation Linear measurement model y = Ax + v n-vector x contains parameters that we want to estimate m-vector v is unknown measurement error or noise m-vector y contains measurements m n matrix A relates measurements and parameters Least squares estimate: use as estimate of x the solution ˆx of minimize Ax y 2 Multi-objective least squares 10-16

20 Regularized estimation add other terms to Ax y 2 to include information about parameters Example: Tikhonov regularization minimize Ax y 2 + λ x 2 goal is to make Ax y small with small x equivalent to solving (A T A + λi)x = A T y solution is unique (if λ > 0) even when A has linearly dependent columns Multi-objective least squares 10-17

21 Signal denoising observed signal is n-vector 15 y = x + v x is unknown signal v is noise Least squares denoising yk k minimize n 1 x y 2 + λ (x i+1 x i ) 2 i=1 goal is to find slowly varying signal ˆx, close to observed signal y Multi-objective least squares 10-18

22 Matrix formulation minimize [ I λd ] x [ y 0 ] 2 D is (n 1) n finite difference matrix D = equivalent to linear equation (I + λd T D)x = y Multi-objective least squares 10-19

23 Trade-off the two objectives ˆx(λ) y and Dˆx(λ) for varying λ ˆx(λ) y Dˆx(λ) Dˆx(λ) 1 05 λ = λ λ = 10 2 λ = ˆx(λ) y Multi-objective least squares 10-20

24 Three solutions 15 λ = λ = 10 2 ˆx(λ)k 1 ˆx(λ)k k k 15 λ = 10 5 ˆx(λ) y for λ 0 ˆx(λ) avg(y)1 for λ ˆx(λ)k 1 λ 10 2 is good compromise Multi-objective least squares k

25 Image deblurring y = Ax + v x is unknown image, y is observed image A is (known) blurring matrix, v is (unknown) noise images are M N, stored as MN-vectors blurred, noisy image y deblurred image ˆx Multi-objective least squares 10-22

26 Least squares deblurring minimize Ax y 2 + λ( D v x 2 + D h x 2 ) 1st term is data fidelity term: ensures Aˆx y 2nd term penalizes differences between values at neighboring pixels D h x 2 + D v x 2 M N 1 = (X i,j+1 X ij ) 2 + i=1 j=1 M 1 i=1 N (X i+1,j X ij ) 2 j=1 if X is the M N image stored in the MN-vector x Multi-objective least squares 10-23

27 Differencing operations in matrix notation suppose x is the M N image X, stored column-wise as MN-vector x = (X 1:M,1, X 1:M,2,, X 1:M,N ) horizontal differencing: (N 1) N block matrix with M M blocks D h = I I I I I I vertical differencing: N N block matrix with (M 1) M blocks D v = D D D, D = Multi-objective least squares 10-24

28 Deblurred images Multi-objective least squares λ = 10 6 λ = 10 4 λ = 10 2 λ=

29 Tomography y = Ax + v x represents values of some quantity in a region of interest of n voxels (pixels) y represents measurements of the integral along lines through the region y i = n A ij x j + v i i=1 A ij is the length of the intersection of the line in measurement i with voxel j x 1 x 2 x 6 line for measurement i Multi-objective least squares 10-26

30 Tomographic reconstruction minimize Ax y 2 + λ( D v x 2 + D h x 2 ) D v and D h are defined as in image deblurring example on page Example left: 4000 lines (100 points, 40 lines per point) right: object placed in the square region at the center of the picture on the left region of interest is divided in pixels Multi-objective least squares 10-27

31 Regularized least squares reconstruction λ = 10 2 λ = 10 1 λ = 1 λ = 5 λ = 10 λ = 100 Multi-objective least squares 10-28