6. Approximation and fitting

Transcription

1 6. Approximation and fitting Convex Optimization Boyd & Vandenberghe norm approximation least-norm problems regularized approximation robust approximation 6

2 Norm approximation minimize Ax b (A R m n with m n, is a norm on R m ) interpretations of solution x = argmin x Ax b : geometric: Ax is point in R(A) closest to b estimation: linear measurement model y = Ax + v y are measurements, x is unknown, v is measurement error given y = b, best guess of x is x optimal design: x are design variables (input), Ax is result (output) x is design that best approximates desired result b Approximation and fitting 6

3 examples least-squares approximation ( ): solution satisfies normal equations A T Ax = A T b (x = (A T A) A T b if rank A = n) Chebyshev approximation ( ): can be solved as an LP minimize t subject to t Ax b t sum of absolute residuals approximation ( ): can be solved as an LP minimize T y subject to y Ax b y Approximation and fitting 6 3

4 Penalty function approximation minimize φ(r ) + + φ(r m ) subject to r = Ax b (A R m n, φ : R R is a convex penalty function) examples quadratic: φ(u) = u deadzone-linear with width a:.5 log barrier quadratic φ(u) = max{, u a} φ(u).5 deadzone-linear log-barrier with limit a: φ(u) = { a log( (u/a) ) u < a otherwise u Approximation and fitting 6 4

5 example (m =, n = 3): histogram of residuals for penalties φ(u) = u, φ(u) = u, φ(u) = max{, u a}, φ(u) = log( u ) 4 p = p = Log barrier Deadzone r shape of penalty function has large effect on distribution of residuals Approximation and fitting 6 5

6 Huber penalty function (with parameter M) replacements φ hub (u) = { u u M M( u M) u > M linear growth for large u makes approximation less sensitive to outliers φhub(u).5.5 f(t) u 5 5 t left: Huber penalty for M = right: affine function f(t) = α + βt fitted to 4 points t i, y i (circles) using quadratic (dashed) and Huber (solid) penalty Approximation and fitting 6 6

7 Least-norm problems minimize x subject to Ax = b (A R m n with m n, is a norm on R n ) interpretations of solution x = argmin Ax=b x : geometric: x is point in affine set {x Ax = b} with minimum distance to estimation: b = Ax are (perfect) measurements of x; x is smallest ( most plausible ) estimate consistent with measurements design: x are design variables (inputs); b are required results (outputs) x is smallest ( most efficient ) design that satisfies requirements Approximation and fitting 6 7

8 examples least-squares solution of linear equations ( ): can be solved via optimality conditions x + A T ν =, Ax = b minimum sum of absolute values ( ): can be solved as an LP minimize T y subject to y x y, Ax = b tends to produce sparse solution x extension: least-penalty problem φ : R R is convex penalty function minimize φ(x ) + + φ(x n ) subject to Ax = b Approximation and fitting 6 8

9 Regularized approximation minimize (w.r.t. R +) ( Ax b, x ) A R m n, norms on R m and R n can be different interpretation: find good approximation Ax b with small x estimation: linear measurement model y = Ax + v, with prior knowledge that x is small optimal design: small x is cheaper or more efficient, or the linear model y = Ax is only valid for small x robust approximation: good approximation Ax b with small x is less sensitive to errors in A than good approximation with large x Approximation and fitting 6 9

10 Scalarized problem minimize Ax b + γ x solution for γ > traces out optimal trade-off curve other common method: minimize Ax b + δ x with δ > Tikhonov regularization minimize Ax b + δ x can be solved as a least-squares problem [ minimize δi A ]x solution x = (A T A + δi) A T b [ b ] Approximation and fitting 6

11 Optimal input design linear dynamical system with impulse response h: y(t) = t h(τ)u(t τ), τ= t =,,...,N input design problem: multicriterion problem with 3 objectives. tracking error with desired output y des : J track = N t= (y(t) y des(t)). input magnitude: J mag = N t= u(t) 3. input variation: J der = N t= (u(t + ) u(t)) track desired output using a small and slowly varying input signal regularized least-squares formulation minimize J track + δj der + ηj mag for fixed δ, η, a least-squares problem in u(),..., u(n) Approximation and fitting 6

12 example: 3 solutions on optimal trade-off curve (top) δ =, small η; (middle) δ =, larger η; (bottom) large δ u(t) t 4 u(t) u(t) t t y(t).5.5 y(t) 5 5 t.5.5 y(t) 5 5 t t 5 Approximation and fitting 6

13 Signal reconstruction minimize (w.r.t. R +) ( ˆx x cor, φ(ˆx)) x R n is unknown signal x cor = x + v is (known) corrupted version of x, with additive noise v variable ˆx (reconstructed signal) is estimate of x φ : R n R is regularization function or smoothing objective examples: quadratic smoothing, total variation smoothing: φ quad (ˆx) = n (ˆx i+ ˆx i ), φ tv (ˆx) = i= n i= ˆx i+ ˆx i Approximation and fitting 6 3

14 quadratic smoothing example.5 ˆx.5 x ˆx xcor ˆx i 3 original signal x and noisy signal x cor 4 i 3 4 three solutions on trade-off curve ˆx x cor versus φ quad (ˆx) Approximation and fitting 6 4

15 total variation reconstruction example ˆxi x ˆxi 5 5 xcor ˆxi 5 i 5 original signal x and noisy signal x cor 5 i 5 three solutions on trade-off curve ˆx x cor versus φ quad (ˆx) quadratic smoothing smooths out noise and sharp transitions in signal Approximation and fitting 6 5

16 ˆx x ˆx 5 5 xcor ˆx 5 i 5 original signal x and noisy signal x cor 5 i 5 three solutions on trade-off curve ˆx x cor versus φ tv (ˆx) total variation smoothing preserves sharp transitions in signal Approximation and fitting 6 6

17 Robust approximation minimize Ax b with uncertain A two approaches: stochastic: assume A is random, minimize E Ax b worst-case: set A of possible values of A, minimize sup A A Ax b tractable only in special cases (certain norms, distributions, sets A) example: A(u) = A + ua x nom minimizes A x b 8 x nom x stoch minimizes E A(u)x b with u uniform on [,] r(u) 6 4 x stoch x wc x wc minimizes sup u A(u)x b figure shows r(u) = A(u)x b u Approximation and fitting 6 7

18 stochastic robust LS with A = Ā + U, U random, EU =, EUT U = P minimize E (Ā + U)x b explicit expression for objective: E Ax b = E Āx b + Ux = Āx b + Ex T U T Ux = Āx b + x T Px hence, robust LS problem is equivalent to LS problem minimize Āx b + P / x for P = δi, get Tikhonov regularized problem minimize Āx b + δ x Approximation and fitting 6 8

19 worst-case robust LS with A = {Ā + u A + + u p A p u } minimize sup A A Ax b = sup u P(x)u + q(x) where P(x) = [ A x A x A p x ], q(x) = Āx b from page 5 4, strong duality holds between the following problems maximize Pu + q subject to u minimize subject to t + λ I P q P T λi q T t hence, robust LS problem is equivalent to SDP minimize subject to t + λ I P(x) q(x) P(x) T λi q(x) T t Approximation and fitting 6 9

20 example: histogram of residuals r(u) = (A + u A + u A )x b with u uniformly distributed on unit disk, for three values of x.5. x rls frequency.5..5 x tik x ls r(u) x ls minimizes A x b x tik minimizes A x b + x (Tikhonov solution) x wc minimizes sup u A x b + x Approximation and fitting 6