INRODUCION hs dssetaton s the eadng of efeences [1], [] and [3]. Faas lemma s one of the theoems of the altenatve. hese theoems chaacteze the optmalt condtons of seveal mnmzaton poblems. It s nown that these theoems of the altenatve ae all equvalent n the sense that these theoems can easl be deved fom each othe. Usuall these sstems consst of lnea nequaltes and/o equaltes called a pmal sstem and a dual sstem. A theoem of the altenatve assets that ethe the pmal sstem has a soluton o the dual sstem has a soluton, but neve both. he optmalt condtons ae also gven b the dualt theoem. A standad technque fo povng a dualt theoem lnng two constaned optmzaton poblems s to appl an appopate theoem of altenatve. he chapte 1 contans esults fom [1]. We show that, usng smple logcal aguments, a dualt theoem s equvalent to a theoem of the altenatve. hese aguments do not need assumpton of lnea space stuctue. he abstact theoem of altenatve s as follows: Let X and Y be abta non empt sets, and let f and g be abta etended eal valued functons defned on X and Y, espectvel. Fo each, consde the statements : I thee ests X II thee ests Y such that such that g f,. he followng statement s an abstact theoem of the altenatve nvolvng the pas (f, X) and (g, Y) : 1
(a) fo all,, eactl one of I, II holds. he statement (a) s ust ntoduced as a logcal statement. Net, the followng logcal statement s an abstact dualt theoem (d) nf f ma g X Y hs statement means nf sup Y g s attaned n Y. X f sup g and sup Y g g fo some.e., Y In chapte, we elaboate an elementa poof of Faas lemma gven n []. he poof s based on elementa aguments. FARKAS LEMMMA : Let A be a eal m n mat and let c be a eal non-zeo n-vecto. hen ethe the pmal sstem (1) A 0 and c 0 has a soluton () n R o the dual sstem A c and 0 has a soluton R m, but neve both. As Faas lemma s one of theoems of the altenatve, t s not possble that both sstems ae solvable. he queston of whch of the two sstems s solvable s answeed b consdeng the bounded least squaes poblem mnmze A c subect to 0 We ae thanful to Pof. Acha Da, Hdologcal Sevces, Isael fo encouagng dscussons and hs helpful comments.
Let m R and let A c denote the coespondng esdual vecto. hen we pove that f 0 then solves second sstem othewse solves fst sstem. he estence of a pont that solves above BLS s gven b a smple teatve algothm. he last chapte 3 gves vaous theoems of the altenatve namel due to Gale, Godan, Steme s, Motzn and Da. Fst, we pove what s called as the altenatve fom of Faas lemma (AFFL) usng uce s theoem. hen we show that t s equvalent to Faas lemma. he statement of AFFL s Let M m n R and c n R be abta. hen ethe o (A) hee ests v 0 such that M v c (B) hee ests w 0 such that Mw 0 and c w 0. and the statement of uce s theoem s : Let A be an abta sew-smmetc mat. hee ests u 0 such that Au 0 and u Au 0. Note that as A s sew-smmetc, u Au 0. Hence f u 0 then Au 0 and Au 0 mples u 0. hs s stct complementates between u and Au. he above mentoned theoems of the altenatve ae meel specal cases of ths theoem and ma be poved smpl b substtutng the appopate matces fo M. 3
CHAPER 1 DUALIY HEOREMS AND HEOREMS OF ALERNAIVE Let X and Y be abta non-empt sets and let f and g be abta etended eal valued functons defned on X and Y espectvel. Fo each ϵ (-, + ], consde the statements I hee ests X II hee ests Y such that f such that g. he followng logcal statement s an abstact theoem of the altenatve nvolvng the pas f, X and g, Y : fo all,, eactl one of Consde two abstact optmzaton poblems. he pmal poblem as I, II holds. (1) nf X f () and the dual poblem as sup Y g (3) Fo these poblems, the abstact dualt theoem s the followng logcal statement. nf X f = ma Y g (4) hs statement means nf sup Y g s attaned n Y. X f sup g and sup Y g g fo some.e., Y 4
Nethe (1) no (4) has an eal content untl the pas f, X and, g Y ae assgned specfc ntepetatons, o stuctue, and also hpotheses ae gven unde whch (1) o (4) s tue. he theo of dual optmzaton poblems nvolves epesentng f and g n tems of some othe functon L: X Y, such that and sup, f L fo all X (5) Y nf L, g fo all Y (6) X hen the pmal s nf f nf sup L, (7) X X Y and the dual s L sup g sup nf, (8) Y he abstact dualt theoem s Y X nf sup L, X Y ma nf L, (9) Y X hs s called the abstact mnma theoem. Note : he smbol denotes negaton. Lemma 1.1 : he followng two statements ae equvalent. (a 1 ) fo all,, II I II II I II I I 5
(d 1 ) [Wea dualt theoem] nf f sup g X Y Poof : [(a 1 ) (d 1 )] Snce Let Y and put g. If then obvousl nf X f. Y s abta and LHS s ndependent of, nf sup Now f,, b (a 1 ) f thee s Y X, f. hs gves nf X f. [(d 1 ) (a 1 )] g X such that g Hence, nf f sup g. Let, be such that Hence, nf f sup g g and 0. X Y II holds. hee s 0 0 X f g. Y then fo all Y Y such that II follows. Lemma 1. : he followng two statements ae equvalent. (a ) [the non-tval half of the theoem of the altenatve] fo all ϵ (-, + ], I II I II (d ) thee ests Y Poof : [(a ) (d )] such that g nf f g X f fo each Y. Now If nf X f then cleal nf X suppose nf X f >. hen I holds fo nf X f. mples that II holds. hus thee ests Y such that g ests Y such that g nf f. hs s (d ). X heefoe (a ),.e., thee 6
[(d ) (a )] Let (d ) holds.e., thee ests Y such that g nf X f. Suppose, I X, f. hs be such that holds.e., fo all gves nf X f. hen b (d ), thee s some Y such that g f II holds. nf. hs means X Followng poposton gves the basc logcal pncple between the dualt theoem and the theoem of altenatves. Poposton 1.3: (1) holds f and onl f (4) holds. Poof: he condton (1) s equvalent to (a 1 ) and (a ) and (4) s equvalent to (d 1 ) and (d ). Hence b pevous two lemmas, we get (1) f and onl f (4). Rema : he f half of the poposton gves a geneal ecpe fo theoems of the altenatve. 7
CHAPER AN ELEMENARY PROOF OF FARKAS LEMMA Faas Lemma.1 : Let A be a eal m n mat and let c be a eal nonzeo n- vecto. hen ethe the pmal sstem A 0 and c 0 (1) has a soluton fo n R o the dual sstem A c and 0 () has a soluton fo R m, but neve both. Rema. : Note that, c 0 0. Whle c 0. Now f both (1) and () holds then A c means 0 as 0 c A A contadcts c 0. Hence ethe of the sstem has a soluton but neve both. Now the queston that whch of the two sstems s solvable s answeed b consdeng bounded least squaes (BLS) poblem mnmze such that A 0 c (3) whee s the Eucldean nom. Let m R and let A c (4) s called the esdual vecto fo the BLS. 8
Lemma.3 : Let hen m R and let solves (3) f and onl f 0, be the esdual vecto fo the BLS (3). and A 0 and satsf the condtons A 0 (5) Poof : Let solves (3). Consde one paametc quadatc functon f A e c (6) a, 1, m Whee, a denotes the th ow of A, s a eal vaable and e denotes the th column of m m unt mat. As such that solves (3), f 0 f 0. Hence, 0 solves the poblem mnmze f fo all eal subect to 0 Note that, f A e c A e c A e c A a a A c A a c f a a A a c a a A c a a 9
0 a f As the devatve vanshes at the mnmum pont, mples, a 0. 0.e.,, Net, f we can wte 0. e., 0, and 0 0 0 f f f f (Whee 0 as 0 ) to get f f 0 of 0 holds. a 0 then fo suffcentl small contadctng the mnmalt 6,heoem4.1.. hus, 0 mples a 0. heefoe (5) Convesel, assume that (5) holds and let z be an abta pont n such that z 0. Put Now, u z. hen cleal A 0, A 0 and z 0, u A z A A z A 0. 0 mples u 0. Fom (5) as R m A z c A u c A c A u A c A u A c A u A c A u u A A c 10
shows that solves (3). Note : Now, combnng (4) and (5), we have c A A whch gves followng theoem. heoem.4 : Let coespondng esdual vecto. If solves (1) and c. solve (3) and let 0 then A c denote the solves (). Othewse, Rema.5 : he theoem tells us that whch of the two sstems, n Faas Lemma, s solvable b consdeng the BLS. Now, what emans s the estence of a pont that solves (3). Estence of : he estence of a pont smple teatve algothm whose followng two steps. that solves (3) s acheved b ntoducng a Step1 - Solvng an unconstaned least squaes poblem. Let begnnng of th teaton, 1,, conssts of 1,, m 0 denote the cuent estmate of soluton at the th teaton. Defne A c 0 and w 0 v 11
he numbe of ndces n w s denoted b t. Now let A be the t n mat whose ows ae a, w. Fo smplct assume that A a1,, a t, w 1,, t and 1,, v t m. Case () : t 0 (.e., 0 ) then sp to step (to chec optmalt and movng awa fom dead pont). Case () : 0 (.e., A c movng awa fom dead pont). Let the t-vecto w w w squaes poblem ) then sp to step (to chec optmalt and 1,, t solves the unconstaned least mnmze A w. Clam : 0 solves ths poblem f and onl f A 0. Let f w A w, ( w : unestcted) A w A w If 0 solves the poblem then A w 0 A w A w A w A 0 (snce w s unestcted). Convesel, f A 0, then A w A w A w A w 0 1
A w.e., 0 solves the poblem. In ths case sp to step. m Othewse, defne a non-zeo seach decton u R b u w fo 1,, t and u 0 fo t 1,, m and the net pont s gven b u 1 whee 0 s the lagest numbe n the nteval 0, 1 u feasble. hs mples followng : that eeps the pont fo 1,, t to be w 0, we must have ma w 0 w (7) and mn w 0 w (8) Futhe note that when w 0, an 0 wll wo and when w 0 an 0 wll wo. Net, the mnmum n (8) s postve o zeo. Cleal, ths mnmum shall also wo fo (7). Hence we can wte mn 1 ( w ) w 0 Step : estng optmalt and movng awa fom a dead pont. Hee t 0 o 0 o A 0. When t 0, 0, so that A does not est. heefoe, n all we can assume that A 0. hs mples followng clam. Clam : If A 0 then solves the poblem 13
mnmze mze A C Subect subectto to 0 fo v and and 0 fo w (9) [[7], theoem 3L, page 156] In ths case, s called a dead pont (t does not change). o test whethe s optmal. Compute nde such that a mn a v. If a 0 then a 0 fo all v. We have a 0 fo all w (snce A 0 ). Hence A 0 hus, and algothm temnates.. Futhe b defntons of v and satsf (5). heefoe, b lemma.3, Othewse (f a 0 ), the net pont s defned as a Note that, 0. a a a 1 a a e w, A 0. solves (3) and Clam: 1 mnmzes Note that f A e c f a a f a a a a 14
and f a 0. heoem.6 : he above algothm temnates n fnte numbe of steps. Poof: It s clea fom followng popetes. (a) he obectve functon s stctl deceasng at each teaton. snce a 0, 1 and 1 mnmzes f. Hence, A c A 1 c. (b) If 1 then 1, u 0, 1,, t. u So t does not change. Hence 1 s a dead pont. Othewse, fo u l l, cleal 1 0 l.e., t deceases. Hence, t s not possble to pefom moe than m teatons wthout eachng a dead pont. (c) Each tme we each a dead pont, the cuent pont solves (9). (d) hee s a fnte numbe of such poblems (9). Because of (a), t s not possble to get the same poblem twce. Coolla.7 : Let hen the vecto and be as n theoem.4 and assume that solves the steepest descent poblem 0. 15
mnmze c (10) subect to A 0 and 1 (11) Poof: Let satsf the constants (11). hen, as 0, A 0. he Cauch Schwatz nequalt gves, hs gves c A A Snce, c, the vecto solves the steepest descent poblem. 16
CHAPER - 3 ON HEOREMS OF HE ALERNAIVE In ths chapte we gve vaous theoems of the altenatve namel due to Gale, Godan, Steme s, Motzn and Da. Fst, Altenatve fom of Faas Lemma (AFFL) s poved usng uce s theoem. he above mentoned theoems of the altenatve ae meel specal cases of ths theoem. uce s heoem 3.1: If B s a eal sew-smmetc mat thee ests a non-negatve vecto u such that Bu s non-negatve and u postve. poof : [[4], theoem 3.4]. heoem 3. ( AFFL: Altenatve fom of Faas Lemma ) : Bu s stctl Let m n R M and c n R be abta. hen ethe o (A) thee ests v 0 such that M v c (.e., M v c 0 ) (B) thee ests w 0 such that Mw 0 and c w 0 but not both (A) and (B) hold. Poof : Consde the followng sew-smmetc mat B 0 0 0 0 M c M c 0 (1) 17
Now, b uce s heoem 3.1, we get u u Bu > 0. Now, v t w 0, such that Bu 0 and Bu = 0 0 0 0 M c M c 0 v t w 0 Mw 0, c w 0, M v + ct 0. () hen u Bu > 0 gves v t w + Mw c w M v ct > 0 v + Mw > 0, t - c w > 0, w - Case () : t = 0, () gves Mw 0 and (3) gves Case () : t = 1, () gves M v c. hs s (A). Futhe, f (A) and (B) hold togethe then b (A) thee ests w 0 such that Mw 0 and But Mw 0 and v 0 gves est togethe. M v + ct > 0. (3) c w < 0. Hence, (B) holds. c w < 0.heefoe, v M v Mw c and b (B) c w < 0. v Mw 0. Hence, both (A) and (B) cannot 18
heoem 3.3 : Faas Lemma and AFFL ae equvalent. Poof : [ ] Accodng to Faas Lemma, onl one of the followng sstem holds: Fo A (C) A 0, R mn and b R n, c 0 (D) A c and 0 In (C), eplace b to get A 0, Cleal, (D) mples (A) of the AFFL. [ ] In (B) of AFFL, eplace w b c 0. hs s the (B) of AFFL. to get (C). Now, appl AFFL to M A A, c b b to get (D). In ths case v. Notaton : We denote b e the vecto whose eve element s 1. Gales heoem 3.4 : Ethe 1 A thee ests n R such that A b o 1 B thee ests 0 such that A 0 and b 1. Poof : Let M A A, v 1, c = b and w. Substtutng these values n (A) of AFFL, we get, 19
A A 1 b,.e., A1 A b..e., A b fo 1. 1 hs s A. Now, same substtutons n (B) of AFFL gves A A 0 and b 0 Now, eplace b A 0 and b 0. b to get b 1. Godan s heoem 3.5 : Ethe A thee ests n R such that A 0 o B thee ests 0 such that A 0 and e 0. Poof : Put b e n Gale s theoem 3.4. Note : Godan s theoem ensues that at least one component of s stctl postve. 0
Faas Lemma 3.6 : Let A m n R and n br be abta. hen ethe 3 A thee ests 0 such that A b o 3 B thee ests n R such that 0 A and b 0. Poof: heoem 3.3. Steme s heoem 3.7 : Ethe 4 A thee ests z 0 such that A z 0 o 4 B thee ests n R such that 0 A and A 0. Poof : Put 0 b A e n Faas Lemma 3.6. hen equaton ( A 3 ) becomes A e. Put z e 0 3. hs gves A. Now, equaton B gves fo b 0, e A 0. Now, note that 4 e A 1 1 _ 1 1 a a _ m a 0 a 0 1
As a 0, thee s atleast one such that a 0. heefoe, A 0. 4 hs gves B. Motzn s heoem 3.8: Let A A1 A A 3, whee A m n R and 3 1 m m. hen ethe one of the followng holds. 5 A thee ests R n such that A1 0, A 0 and A3 0. 5 B thee ests 1 0, 0 and 3 R m3 such that and e 1 0. 3 1 A 0 Poof : Let A A A A M A1 A A3 A3 1 3 3 c e 0 0 0, B AFFL, we have ethe 1 v such that M v c o thee s w 1 z1 z 0, gves such that Mw 0 and c w 0. hus, M v c A A A A A A A A 1 1 3 3 3 3 1 e 0 0
A1 e 0 and A 0 and A3 0, whee 1. 5 hs gves A. Now Mw 0 mples A A A A A A A A 1 3 3 1 3 3 1 z1 z 0 A1 1 A A3 z1 A3 z 0 and A 1 1 A A3 z1 A3 z 0 A 1 1 A A 3 3 0, whee 3 z1 z Net, c w 0 gves e 1 0. 5 1 e 0 0 0 z1 z 0, hs s B. Notaton : B we mean the vecto whose elements ae. Da s heoem 3.9 : Let d m R be an stctl postve vecto. hen ethe 6 A thee ests such that m R satsfng d d A b o 6 B thee ests n R such that b d A 0. 3
Poof : Let M A A I I, A A I I 1 v, c b b d d and w 1 z1 z. he AFFL condton M v c gves A A I I A A I I 1 b b d d A b, d, d whee 1. hs gves Now, Mw 0 gves A. 6 A A I I A A I I 1 z1 z 0 A z1 z whee, 1. Futhe, A z1 z z1 z, snce z1 0, z 0 Now, z1 z z1 z., theefoe 4
b b d d 1 z1 z 0, b d z z Snce d 0 and A z1 z 1 0 we have d A d z z Addng ths nequalt to above nequalt we get 6 Net, A gves d A b 0.. 1 0 b A A A A A But snce d d, d so that whch contadcts 6 b d A B. Hence both A and 6 B do not hold togethe. 6 5
REFERENCES [1] Dualt heoems and heoems of he Altenatve, L. McLnden, Poceedngs of the Amecan Mathematcal Socet, V. 53, 17-175, 1975. [] An Elementa Poof of Faas Lemma, Acha Da, SIAM REV. V. 39, No. 3, 503-507, 1997. [3] On theoems of the altenatve. C. G. Boden, Optmzaton methods and Softwae, V.16, N0.1, 101-111. [4] A smple algebac poof of Faas s lemma and elated theoems, C. G. Boden, Optmzaton Methods and Softwae, V.8, No.3, 185-199, 1998. [5] Optmzaton b Vecto Space Methods, D. G. Luenbege, Wle, New Yo, 1969. [6] Non lnea Pogammng : heo and Algothms, M. S. Bazaaa, Hanf D. Sheal, C. M. Shett, John Wle & Sons. [7] Lnea Algeba and ts applcatons, Glbet Stang, homson leanng Inc, 1998. 6
7