p(z) = 1 a e z/a 1(z 0) yi a i x (1/a) exp y i a i x a i=1 n i=1 (y i a i x) inf 1 (y Ax) inf Ax y (1 ν) y if A (1 ν) = 0 otherwise

Transcription

1 Dustn Lennon Math 582 Convex Optmzaton Problems from Boy, Chapter 7 Problem 7.1 Solve the MLE problem when the nose s exponentally strbute wth ensty p(z = 1 a e z/a 1(z 0 The MLE s gven by the followng: p(y a x = [ (1/a exp = (1/a n exp Ths means the ML problem can be expresse as ( ( y a x n a y a x a ] 1(y a x > 0, y Ax nf x n (y a x y Ax Equvalently, nf x 1 (y Ax 0 Ax y Ths leas to the Lagrangan: L(x, ν = 1 (y Ax + ν (Ax y = 1 y ν y + (ν A 1 Ax An the Lagrange ual, g(ν = (1 ν y f A (1 ν = 0 otherwse Problem 7.2 Gven the lnear measurement moel, y = Ax + v wth unform nose 1/(2α f z α p(z = 0 otherwse Show that the ont ML estmates of x an α are foun by solvng the l -norm approxmaton problem mn Ax y The MLE s compute as the maxmum of ( 1 1( α a 2α x y α 1

2 Ths s expresse as, max x,α ( 1 n 2α Ax y < α Equvalently, we can solve the mnmzaton problem after a monotonc transformaton mn x,α α Ax y < α An snce α s a slack varable, the result follows. Problem 7.3 Estmate the parameters, a, b n a probt moel where v s a zero mean Gaussan varable: y = 1 a u + b + v 0 0 a u + b + v > 0 If y = 1, then a u + b v. Ths event has probablty gven by 1 Φ(a u + b = Φ( a u b where Φ(z = z 1 2π ( exp s2 2 s. Smlarly, for y = 0, a u + b v has probablty gven by Φ(a u + b. We maxmze the lkelhoo equaton: Φ(a u + b Φ( a u b y =0 or, equvalently, maxmze the log-lkelhoo equaton: y log Φ(a u + b + log Φ( a u b y y =0 Φ s an ntegral of a log-concave functon, hence log-concave. Thus the log lkelhoo s convex an has the form of a penalty approxmaton problem. Problem 7.4a Jont estmaton of covarance an mean for a multvarate normal strbuton. Let R be the covarance an a the mean. Defne Y an µ to be the respectve estmates of R an a: Frst we state the log-lkelhoo functon µ = 1 N N y k, Y = 1 N N (y k µ(y k µ l(r, a = (Nn/2 log(2π (N/2 log et R (1/2 N (y k a R 1 (y k a an note that the last term can be rewrtten: N N (y k a R 1 (y k a = (y k µ + µ a R 1 (y k µ + µ a = (y k µ R 1 (y k µ + 2 (y k µ R 1 (µ a + (µ a R 1 (µ a }} =0 = tr(r 1 (y k µ(y k µ + N(µ a R 1 (µ a = N tr(r 1 Y + N(µ a R 1 (µ a 2

3 Hence, l(r, a = (Nn/2 log(2π (N/2 log et R N 2 tr(r 1 Y N 2 (µ a R 1 (µ a We maxmze the log-lkelhoo by takng the matrx ervatve wth respect to R an graent wth respect to a an settng them to zero: a (µ a R 1 (µ a = 2R 1 (a µ log et R = R 1 R R tr(r 1 Y = R 1 Y R 1 where the last two enttes are erve by varatonal methos (see Appenx A n Boy. Settng the ervatves equal to zero yels the followng ML estmates: a = µ R = Y Problem 7.5a Markov Chan Estmaton. Defne the transton probablty matrx as P = prob (y(t + 1 = y(t = where n P = 1. We wrte the lkelhoo functon: P (Y 1 N P (y( = k y( 1 = k 1 =2 If we enote n as the number of transtons from to, we can wrte the above as P (Y 1, P n Wth the constrant, ths yels a Lagrangan of the log lkelhoo: L(P, ν 1,..., ν n = c + n log P + ( ν 1 P Takng ervatves, we obtan L P = n P ν = 0 An summng over wth the constrant yels ν = n n. Thus the MLE s P = n n whch can be nterprete as the number of observe transtons from to ve by the total number of vsts to state. Problem 7.5b We a the constrant of a known equlbrum strbuton, q where 1 q = 1 an P q = q. Ths amounts to ang a constrant to the Lagrangan: L(P, ν, µ 1,..., µ n = c + n log P + ( ν 1 P + µ q P q 3

4 Ths s convex snce both constrants are lnear n P. Problem 7.6 Conser a normalze ranom varable, X, an a shfte an scale ranom varable Y = X+b a. We assume that X has ensty functon, p(x. Then the ensty functon of Y can be compute as Hence the lkelhoo s gven by p Y (y = ap X (ay b ap X (ay b an the log lkelhoo s convex snce p X s log-convex. We compute ML estmates for a, b for the Laplace strbuton, p x (x = exp( 2 x, for whch we maxmze the log lkelhoo: n log a 2 n ay b It s equvalent to solve max a,b max max a b n log a 2 n ay b The nner maxmzaton yels b = a mean(y 1,..., y n. We can solve for a by takng ervatves. Frst efne S = y mean(y 1,..., y n. Problem 7.7 n log a 2aS n a 2S = 0 a = n 2S b = n mean(y 1,..., y n 2S X Posson (µ 1,..., n}. These are the types of events. p probablty that the th evce etects an event of type. There are m evces. Y number of events of type etecte by evce. Not rectly observe. Y = n =0 y s the number of events etecte by evce. These are the observatons. Goal: estmate µ va maxmum lkelhoo. P (Y = k = P (Y = k X = n P (X = n [( ] [ n e = p k k (1 p n k µ µ n ] n! [ e µ e = e µp (n k! (µ (1 p n k µ p (µ p k ] k! }} =1 Posson (µ p 4

5 The sum of nepenent Posson ranom varables wth means λ 1,, λ n s a Posson ranom varable wth mean λ λ n. Hence, ( n Y Posson µ p Now form the maxmum lkelhoo estmate, =0 L (µ y,..., y m = The prmal problem s then: m exp ( n µ p ( n =0 µ p y y! l(µ = log L (µ y,..., y m } ( m n m n } = µ p + y log µ p log (y! = µ q + m y log (µ p log (y!} mn l(µ such that µ 0 Ths problem s convex, as t s an affne functon of µ mnus the log of an affne functon of µ. We compute the Lagrangan whch s fferentable n µ. L(µ, λ = µ q m y log (µ p log (y!} µ λ Problem 7.8 Estmaton usng sgn measurements. Measurements are gven by (y, a, b. We estmate x n the moel: y = sgn(a x + b + v where y = ±1 an v s a log-concave IID nose term. Ths problem s very smlar to queston 7.3. Let F (x enote the strbuton functon. Then the MLE s gven by (1 F ( a x b F ( a x b y y = 1 an the log-lkelhoo s log((1 F ( a x b + log(f ( a x b y y = 1 Snce the ensty s assume to be log concave, t follows that maxmzng the log-lkelhoo s a convex optmzaton problem. 5