10-716: Advanced Machine Learning Spring Lecture 13: March 5

Transcription

1 10-716: Advaced Machie Learig Sprig 019 Lecture 13: March 5 Lecturer: Pradeep Ravikumar Scribe: Charvi Ratogi, Hele Zhou, Nicholay opi Note: Lae template courtey of UC Berkeley EECS dept. Diclaimer: hee ote have ot bee ubjected to the uual crutiy reerved for formal publicatio. hey may be ditributed outide thi cla oly with the permiio of the Itructor. I the previou clae aw that i the oiy ettig we ca o loger expect to achieve perfect recovery. Itead we focu o boudig the l error betwee a Lao olutio ˆθ ad the ukow regreio vector θ log p ˆθ θ = c, where deote the parity, p deote the ambiet dimeio ad i the umber of ample obtaied. Note that thi i a huge reductio from the liear rate i p to a logarithmic rate of covergece. We eed to udertad if covergece of l orm i a ufficiet metric for recovery of pare liear model. For example, coider the cae where θ = 0, ˆθ = 1 I. So, ˆθ θ 1, but Sˆθ = {1,, p} = Sθ. hi example illutrate that jut a good covergece rate i ot ufficiet, the upport et hould match too kow a variable electio coitecy. We will how i the ret of the lecture that uder ome aumptio the Lao olutio eure that the upport et are the ame i the limitig cae, ie, Sˆθ = Sθ Lao Solutio A1 Aumptio 1: Retricted Eigevalue Coditio tate that, if S = upportθ the λ S S mi c mi > hi aumptio i alo kow a the idetifiability coditio. We ote that the retricted eigevalue coditio aloe oly aure l covergece. Aother way to tate thi i c mi > 0 : S c = A Aumptio : he irrepreetability aumptio tate that S S 1 S j 1 1 α < 1 j S c hi aumptio ca be udertood a the olutio ŵ for the ordiary leat quare OLS problem ŵ = arg if w j. hi coditio implie that the vector j i ot too related to the upport vector i ad hece caot be repreeted by them. Keepig thee aumptio i mid, we tate oe of the mai theorem for the Lao olutio. 13-1

2 13- Lecture 13: March 5 heorem 13.1 Coider a S-pare liear regreio model for which the deig matrix atifie coditio A1 ad A. he for ay choice of regularizatio parameter uch that the Lagragia Lao ha the followig propertie: λ α S c Π S w a uique ˆθ that olve the lao problem. Sˆθ Sθ. 3. ˆθ θ S S 1 S w + λ S 1 S = r 4. No fale excluio of all j Sθ for which θ j > r I equatio 13.4, λ give a upper boud o the maximum oie level i the p elemet i S c. o udertad thi theorem, we firt look at the what the projectio matrix Π S i equatio 13.4 mea. Π S = S S S 1 S, where S i a matrix ad hece Π S i a matrix. he projectio matrix that o pre-multiplyig project a vector u to the S pace. hi ca be ee a follow - ˆβ = arg if u Sβ β = S S 1 S u S ˆβ = S S S 1 S u 13.5 Note that Π S = I Π S Π S c. Poit 4 follow from 3 i that if ˆθ θ r, the for a fale excluio to occur θj would have to be maller tha r to go udetected, which directly lead to the cocluio i tatemet Side ote: Norm Defiitio Before provig the ret of theorem, let have a quick recap of the defiitio of differet matrix ad vector orm Vector l p orm : u p = p 1 j=1 u j p p Matrix operator orm : A p = up u 0 Au p u p Spectral Norm : A = max igular valuea Matrix Ifiity orm : A = max j p k=1 A jk Variable Selectio Coitecy for the Lao with Gauia Noie heorem 13.1 i a reult that applie to ay et of liear regreio equatio. Now, uppoe that the oie i.i.d. i Gauia, i.e., w i N0, σ. here are a few part to thi example:

3 Lecture 13: March Part 1: We will firt derive a upper boud for the lower boud of λ i heorem Part : he, lookig at Property 3 i heorem 13.1, we will derive a upper boud for Put part 1 ad together ad make ome obervatio. 1. Part 1: I order to boud S cπ S w which ha dimeio p 1, we coider oe elemet i the vector at a time, ad the boud the maximum acro all elemet. Let Z j refer to the jth etry of the vector: Z j = j Π S w = a w for ome j S c, ad where a = j Π S. Sice a i determiitic ad w i i Gauia, we kow that Z j i Gauia with mea 0 ad variace σ a. hat i, Z j N0, σ a where a = j Π S. If you aume that the colum are ormalized, i.e., max p j=1 j c, the ice projectio i ever expaive the L orm ever icreae, a = j Π S he, Z j i ubgauia with parameter cσ. Uig the Gauia tail boud ad uio boud, we get j c. P max z j > t j S c = max j S c z j cσ t p exp c σ logp + δ w.p. 1 exp δ hu, S cπ S w logp cσ + δ. Part : We alo have to boud 1 a 1 vector. o do thi, we itroduce z j = e j 1 = a w 1 for ome j {1,,..., } where a = e j. Note that e j i the jth tadard bai vector 0 everywhere ad a 1 i the jth dimeio. A before, we compute a :

4 13-4 Lecture 13: March 5 a = a a = e j 1 1 e j 1 e j = 1 e j 1 1 = c mi / where c mi = 1. hu, uig a imilar boud o the maximum abolute value of z j, we ca boud the l orm a follow: 1 { } 1 log + σ cmi Puttig both part together: We have bouded with high probability that S cπ S w max j S c z j cσ logp + δ ad that 1 { } 1 log + σ cmi Pluggig ito heorem 13.1, we ee that we get withi log p of the bet poible rate Side Note: Sub-gradiet Let f be a covex fuctio. Whe f i differetiable, the lie fθ = fθ 0 + fθ 0 θ θ 0 lie below fθ. However, thi require differetiability. z i a ub-gradiet of f at θ 0 iff fθ fθ 0 + z, θ θ 0 θ Θ. Whe f i ot differetiable, we could have everal z value that atify thi. he ub-differetial i the et of all uch z value. A ub-gradiet i oe uch z value. If fθ i the L-1 orm, the whe θ i o-zero, it i differetiable ad it gradiet i the ig of θ. Whe θ = 0, ay taget lie with lope betwee 1 to 1 lie below fθ ee Figure

5 Lecture 13: March Figure 13.1: Subgradiet for the L1 orm fuctio. Whe olvig for a tatioary poit mi θ fθ, oe ormally et the derivative to 0 fθ = 0 ad the olve. Whe f i ot differetiable, ca itead require that 0 i i the ub-gradiet of θ Proof of Lagragia Lao Propertie heorem 13.1 o olve 1 y θ 1 + λ θ 1, let u ue the ub-gradiet: θ y + λ z = 0 for ome z ˆθ 1. We eed to how that for ay ˆθ that atifie the above, ˆθ S c = 0 i.e., o irrelevat coordiate are picked. Will cotruct a ˆθ, ẑ pair uch that the coditio are already atified: 1. ˆθ c = 0.. he firt part of the tatioary coditio i atified. 3. ẑ ˆθ 1 with high probability. We ue a cotructive procedure, called the primal-dual wite techique. hi create a ˆθ, ẑ pair which i primal-dual optimal ad atifie the required coditio. he cotructio procedure i a follow: 1. ˆθ c = 0.. Set ˆθ by olvig: ˆθ = if θ y θ + λ θ Chooe ẑ δ ˆθ 1 uch that 1 ˆθ y + λ ẑ = c ˆθ y + λ ẑ = 0. Referece [waiwright] M. Waiwright, Chapter 7, High Dimeioal Statitic, Prereleae, 019