Lecture 14 (03/27/18). Channels. Decoding. Preview of the Capacity Theorem.
|
|
- Flora Booth
- 6 years ago
- Views:
Transcription
1 Lecture 14 (03/27/18). Channels. Decodng. Prevew of the Capacty Theorem. A. Barg The concept of a communcaton channel n nformaton theory s an abstracton for transmttng dgtal (and analog) nformaton from the sender to the recpent over a nosy medum. Examples of physcal channels are wreless lnks, cable communcatons (optcal, coaxal, etc.), wrtng onto dgtal meda (flash, magnetc), and many more. Let X, Y be fnte sets. A mappng W : X Y s called stochastc f the mage of x X s a random varable takng values n Y. Denote P (x y) by W (y x), the probablty of y condtonal on the gven nput x. Defnton: A dscrete memoryless channel (DMC) s a stochastc mappng W : X Y. We use the letter W to refer both to the channel tself and to the probablty dstrbuton W (y x). The sets X and Y are called the nput alphabet and the output alphabet of W, respectvely. The channel s represented by a stochastc matrx whose rows are labelled by the elements of X (nput letters) and columns by the elements of Y (output letters). By defnton y Y W (y x) = 1 for any x X. Examples. 1. Z-channel (called so because ts dagram resembles the letter Z). ( ) 1 ε ε W = Bnary symmetrc channel (BSC(p) ) W : {0, 1} {0, 1} ( ) 1 p p W = p 1 p 3. Bnary erasure channel (BEC((p)) W : {0, 1} {0, 1,?} ( ) 1 p p 0 W = 0 p 1 p There are more examples n the textbook. Defnton: Let M be a fnte set of cardnalty M and let f : M X n be a mappng. A code C of length n over the alphabet X s the mage of f n X n. We say that a message m M s encoded nto a codeword x m C f f(m) = x m. The set of codewords {x 1,..., x M } s called a channel code 1. The number R = 1 n log M s called the rate of the code C. Below we denote general n-vectors by x n, y n and keep the above notaton for the codewords. The codewords are transmtted over the channel. Ths means the followng. The mappng W s extended from X to X n usng the memoryless property of W : n W n (y n x n ) = W (y n x n ), where x n = (x n 1,..., x n n), y n = (y1 n,..., yn). n =1 The result of transmttng the codeword x m over the channel W s a vector y n Y n wth probablty W n (y n x m ). Messages are encoded and transmtted as codewords to provde the recpent wth the functonalty of correctng errors that may occur n the channel. Error correcton s performed by a decoder,.e., a mappng g : Y n M. The decoder s a determnstc mappng constructed so as to mnmze the probablty of ncorrect recovery of transmtted messages. 1 Sometmes the term code s used to refer to f and then the set of codewords C s called the codebook. 1
2 2 Optmal decoders. We brefly dscuss optmal decodng rules. Let Pr(m) be a probablty dstrbuton on M. Let y n be the receved vector,.e., the output of the channel. The posteror probablty that the transmtted message was m equals (1) P (m y n ) = Pr(m)W n (y n x m ) P (y n, ) where P (y n ) = M m=1 Pr(m)W n (y n x m ). Assume that g(y n ) = m, then the error probablty s p e = 1 P (m y n ). To mnmze p e decode y to m such that (2) P (m y n ) P (m y n ) for all m m (tes are broken arbtrarly). Ths rule s called the maxmum aposteror probablty (MAP) decoder. If Pr(m) = 1/M s unform, then the MAP decoder s equvalent to the maxmum lkelhood (ML) decoder g ML gven by g(y n ) = m f W n (y n x m ) W n (y n x m ) for all m m. To see ths, use the Bayes formula (1) n (2). If Pr(m) = 1/M s not unform, then the ML decoder s generally suboptmal. ML and MAP decoders are computatonally very hard because of the large search nvolved n fndng g(y). Prevew of the Shannon capacty theorem. The followng dscusson s nformal. It uses the smple case of the BSC to explan the nature of channel capacty n geometrc terms. Consder transmsson over W =BSC(p), p < 1 /2. Let d H (x n, y n ) = { : x y } be the Hammng dstance between the (bnary n-dmensonal) vectors x n and y n. Let x n be the transmtted vector and y n the receved vector. The typcal value of the dstance d H (x n, y n ) np. In other words, Pr{ d H (x n, y n ) np nα} s small, where α > 0 s a small number. Therefore defne the decoder value g(y n ) as follows: f there s a unque codevector x m C such that d H (x m, y n ) np nα, then g(y n ) = x m, otherwse put g(y) = x 1 (or any other arbtrary codevector). Below we call vectors y n whose dstance from x m s about np typcal for x m. The number of typcal vectors y n {0, 1} n for a gven x n s (3) {y n {0, 1} n : d H (x n, y n ) np nλ} =. Lemma 1. Let 1 λ 1/2, then (4) Proof : 1 = (λ + (1 λ)) n =0 1 n + 1 2nh(λ) λ (1 λ) n =0 =0 2 nh(λ). : np nλ (1 λ) n( λ ) λn = 2 nh(λ) 1 λ =0
3 3 We would lke the sets T α (x m ) {y n Y n : d H (x m, y n ) np nα} for dfferent x m to be dsjont. Note that n(p+α) T α ( ) =. Suppose frst that =n(p α) (5) M T α ( ) (4) 1 n + 1 M2nh(p α) > 2 n, then a pont y n n the output space s typcal on average for an exponentally large number of codevectors, namely for A = 2 n(r+h 2(p α) o(1)) /2 n = 2 n(r+h 2(p α) 1 o(1)) = 2 nε codevectors, where we denoted ε = R + h 2 (p) α 1. Ths means that a sgnfcant proporton of ponts y n s typcal for exponentally many codewords. In ths case decodng wth low error probablty s mpossble (for nstance, the maxmum error probablty s close to 1). We observe that (5) mples the followng nequalty: R > 1 h(p) + α where α s small f so s α. Thus f ths nequalty s true, the error probablty s large. At the same tme, f the cumulatve volume of the typcal sets around the codewords satsfes M T α ( ) = 2 nr T α ( ) (4) 2 n(r+h 2(p+α)),.e., s less than 2 n (the total sze of the output space), then there can exst codes n whch decodng s correct wth large probablty. We wll show by random choce that such codes ndeed exst. We notce that there s a dvdng pont R = 1 h 2 (p) between the low and hgh error probablty of decodng. Ths value of the rate s called the capacty of the channel W. We can also flp the queston by askng Threshold nose level: We are gven a code C of rate R. Assumng that C s chosen optmally for transmsson over a BSC(p), what s the largest p for whch the code can guarantee relable transmsson? The above argument shows that the maxmum p s p th = h 1 2 to vsualze the dependence on the rate). (1 R) (Plot the functon h 1 2 (z)
4 4 Lecture 15 (03/29/18). Channel capacty The followng s a formalzaton of the dscusson n the prevous secton. Consder BSC(p),.e., a stochastc mappng W : X Y, X = Y = {0, 1} such that Let W (y x) = (1 p)1 {x=y} + p1 {x y}. T α (x n ) = {y n : d H (x n, y n ) np nα}, α > 0. Let C = {x n 1,..., x n M } be a code (below we omt the superscrpt n from the notaton for the codewords). Let D(x m ) {y n {0, 1} n : m W n (y n x m ) W n (y n x m )} be the decson regon for the codeword x m. Denote by λ m = W n (y n x m ) y n D(x m) c the error probablty of decodng condtonal on transmttng the mth codeword and let be the maxmum error probablty of the code C. λ max (C) = max λ m m Theorem 2. (Shannon s capacty theorem for the BSC, lower bound) Gven ε > 0, γ > 0, p < 1/2 and R 1 h(p) γ, there exsts n 0 = n 0 (ε, γ) such that for any n n 0 there exsts a code C {0, 1} n of cardnalty 2 Rn, whose maxmum error probablty of decodng on a BSC(p) satsfes λ max ε. Proof : Let M = 2 R n, where R = R + 1 n. Choose C = {x 1,..., x M } {0, 1} Mn by randomly assgnng codewords to the messages wth unform probablty Pr(f(m) = x m ) = 2 n ndependently of each other (below we use the notaton x m for codewords, omttng the superscrpt x n m). Suppose that y n s the receved vector. Let us use the followng decoder g : Y n M : f there s a unque codeword x m C such that y n T α (x m ), assgn g(y n ) = m. In all other stuatons put g(y n ) = 1. (Ths mappng s called the typcal pars decoder). Let Z m = 1 {(yn T α (x m ))}, m = 1,..., M be the ndcator random varable of the event {y n s typcal for x m }. Suppose that the transmtted vector s x 1. The probablty of error λ 1 satsfes We have 2 M λ 1 Pr{Z 1 = 0} + Pr{ Z m 1}. m=2 Pr{Z 1 = 0} = Pr{y n T α (x 1 )} = Pr{ d H (x 1, y n ) np > nα} Chebyshev nequalty np(1 p) (nα) 2 = p(1 p)n δ 2 The Chebyshev nequalty states that for any random varable X wth fnte expectaton and varance Var(X) we have Pr{ X EX a} Var X a 2.
5 5 by takng α = n (1 δ)/2, δ > 0. By takng n suffcently large, we can guarantee that Pr{Z 1 = 0} β, where β > 0 s arbtrarly small. Next for m 2 Pr{Z m = 1} = T α( ) 2 n (4) 2 n(1 h(p+α)). Use the unon bound: M Pr{ Z m 1} M Pr{Z m = 1} 2 n(1 R h(p+α)) 2 n(α γ+ 1 n ), m=2 where we wrte h(p+α) = h(p)+α, and α s small f α s small (note that α > 0 snce p < 1/2). By takng a suffcently large n we can ensure that α < γ 1 n, and so Pr{ M m=2 Z m 1} β. Now let use compute the average probablty of error over all codeword assgnments f : P e = E F λ(c) = 1 M P r(c) λ m (C) M (6) = 1 M C m=1 M P r(c)λ m (C) = C C m=1 Pr(C)λ 1 (C) 2β where Pr(C) = M m=1 Pr{f(m) = x m}. Here F s the random mappng. Snce we go over all the mappngs, the sum C P r(c)λ m(c) does not depend on m. By (6) there exsts a code C for whch the error probablty averaged over M codewords satsfes λ(c) 2β. By the Markov nequalty, {m M : λ m (C ) 2λ(C } M/2. Thus, there exst at least M/2 messages 3 whose codewords n C are decoded wth error probablty λ m (C ) 4β. Denote ths set of codewords by C and take β = ε/4. Thus there s a code C of cardnalty M 2 = 2n(R 1 n ),.e., of rate R wth λ max ( C ) ε. Observe that the probablty λ max falls as n ε. By usng the optmal (.e., MAP) decoder together rather than the typcal pars decoder t s possble to show that there exst much better codes for the BSC. Theorem 3. For any rate R, 0 R < 1 h 2 (p) there exsts a sequence of codes C, = 1, 2... of growng length n such that log C R n and (7) λ max (C ) 2 n{d(δ(r) p))(1 o(1)} where D(x y) = x log x y 1 x + (1 x) log 1 y, δ(r) h 1 2 (1 R), and o(1) 0 as n. We wll omt the proof. Note that the declne rate of the maxmum error probablty s a much faster (exponental) functon of the code length n than n the above argument. We took a loss by usng a suboptmal decoder (and a smpler proof). Fnte-length scalng. The effcency of our transmsson desgn can be measured by the number of messages that can be transmtted relably. Suppose that the code rate s R = 1 h 2 (p) γ, where γ > 0 s small and p s the transmsson probablty of the BSC. Suppose moreover that we requre 3 Ths dea s called expurgaton.
6 6 that λ max = ε. We already understand that we wll have to choose a suffcently long code. What s the smallest n that can guarantee ths? As R 1 h(p), we have δ(r) p. We have D(δ(1 h(p)) p) = D(p p) = 0, D δ(δ p) (1 δ)p = log = 0, D δ (δ p) = 1 1 δ(1 p) δ=p ln 2 δ(1 δ). If R = 1 h 2 (p) γ then δ = p + γ where γ s some small number. Expandng D nto a power seres n the neghborhood of δ = p we obtan D(δ p) = 1 2 D δ (δ p)(δ p) 2 + o((δ p) 2 ) = O((δ p) 2 ) p From (7) we obtan (8) n log(1/ε) (δ p) 2 (constants omtted). Let us rephrase ths by fndng how γ (gap to capacty) depends on the code length n. To answer ths, rewrte (8) as follows: δ p (log(1/ε)) 1 /2 1 n Now substtute δ = h 1 (1 R) to fnd that R 1 h(p O(n 1 /2 )), or ( 1 R 1 h(p) O n ). To conclude: Proposton 4. The gap-to-capacty for optmal codes scales as n 1/2. The outcome of ths calculaton s called fnte-length scalng of the code sequence on the BSC. The same order of scalng s true for optmal codes used on any bnary-nput dscrete memoryless channel (DMC). We return to the textbook, and state the Shannon capacty theorem for a DMC (pp , 200). Then we consder examples (pp ), and then (next class) prove the capacty theorem.
Lecture 3: Shannon s Theorem
CSE 533: Error-Correctng Codes (Autumn 006 Lecture 3: Shannon s Theorem October 9, 006 Lecturer: Venkatesan Guruswam Scrbe: Wdad Machmouch 1 Communcaton Model The communcaton model we are usng conssts
More informationError Probability for M Signals
Chapter 3 rror Probablty for M Sgnals In ths chapter we dscuss the error probablty n decdng whch of M sgnals was transmtted over an arbtrary channel. We assume the sgnals are represented by a set of orthonormal
More informationECE 534: Elements of Information Theory. Solutions to Midterm Exam (Spring 2006)
ECE 534: Elements of Informaton Theory Solutons to Mdterm Eam (Sprng 6) Problem [ pts.] A dscrete memoryless source has an alphabet of three letters,, =,, 3, wth probabltes.4,.4, and., respectvely. (a)
More informationChapter 7 Channel Capacity and Coding
Chapter 7 Channel Capacty and Codng Contents 7. Channel models and channel capacty 7.. Channel models Bnary symmetrc channel Dscrete memoryless channels Dscrete-nput, contnuous-output channel Waveform
More informationChapter 7 Channel Capacity and Coding
Wreless Informaton Transmsson System Lab. Chapter 7 Channel Capacty and Codng Insttute of Communcatons Engneerng atonal Sun Yat-sen Unversty Contents 7. Channel models and channel capacty 7.. Channel models
More informationModule 3 LOSSY IMAGE COMPRESSION SYSTEMS. Version 2 ECE IIT, Kharagpur
Module 3 LOSSY IMAGE COMPRESSION SYSTEMS Verson ECE IIT, Kharagpur Lesson 6 Theory of Quantzaton Verson ECE IIT, Kharagpur Instructonal Objectves At the end of ths lesson, the students should be able to:
More information2E Pattern Recognition Solutions to Introduction to Pattern Recognition, Chapter 2: Bayesian pattern classification
E395 - Pattern Recognton Solutons to Introducton to Pattern Recognton, Chapter : Bayesan pattern classfcaton Preface Ths document s a soluton manual for selected exercses from Introducton to Pattern Recognton
More informationÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE
ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE School of Computer and Communcaton Scences Handout 0 Prncples of Dgtal Communcatons Solutons to Problem Set 4 Mar. 6, 08 Soluton. If H = 0, we have Y = Z Z = Y
More informationRandomness and Computation
Randomness and Computaton or, Randomzed Algorthms Mary Cryan School of Informatcs Unversty of Ednburgh RC 208/9) Lecture 0 slde Balls n Bns m balls, n bns, and balls thrown unformly at random nto bns usually
More informationEGR 544 Communication Theory
EGR 544 Communcaton Theory. Informaton Sources Z. Alyazcoglu Electrcal and Computer Engneerng Department Cal Poly Pomona Introducton Informaton Source x n Informaton sources Analog sources Dscrete sources
More informationLecture Notes on Linear Regression
Lecture Notes on Lnear Regresson Feng L fl@sdueducn Shandong Unversty, Chna Lnear Regresson Problem In regresson problem, we am at predct a contnuous target value gven an nput feature vector We assume
More informationprinceton univ. F 13 cos 521: Advanced Algorithm Design Lecture 3: Large deviations bounds and applications Lecturer: Sanjeev Arora
prnceton unv. F 13 cos 521: Advanced Algorthm Desgn Lecture 3: Large devatons bounds and applcatons Lecturer: Sanjeev Arora Scrbe: Today s topc s devaton bounds: what s the probablty that a random varable
More informationStanford University CS254: Computational Complexity Notes 7 Luca Trevisan January 29, Notes for Lecture 7
Stanford Unversty CS54: Computatonal Complexty Notes 7 Luca Trevsan January 9, 014 Notes for Lecture 7 1 Approxmate Countng wt an N oracle We complete te proof of te followng result: Teorem 1 For every
More informationCommunication with AWGN Interference
Communcaton wth AWG Interference m {m } {p(m } Modulator s {s } r=s+n Recever ˆm AWG n m s a dscrete random varable(rv whch takes m wth probablty p(m. Modulator maps each m nto a waveform sgnal s m=m
More informationExpected Value and Variance
MATH 38 Expected Value and Varance Dr. Neal, WKU We now shall dscuss how to fnd the average and standard devaton of a random varable X. Expected Value Defnton. The expected value (or average value, or
More informationCSCE 790S Background Results
CSCE 790S Background Results Stephen A. Fenner September 8, 011 Abstract These results are background to the course CSCE 790S/CSCE 790B, Quantum Computaton and Informaton (Sprng 007 and Fall 011). Each
More informationLecture 3. Ax x i a i. i i
18.409 The Behavor of Algorthms n Practce 2/14/2 Lecturer: Dan Spelman Lecture 3 Scrbe: Arvnd Sankar 1 Largest sngular value In order to bound the condton number, we need an upper bound on the largest
More informationVQ widely used in coding speech, image, and video
at Scalar quantzers are specal cases of vector quantzers (VQ): they are constraned to look at one sample at a tme (memoryless) VQ does not have such constrant better RD perfomance expected Source codng
More informationLecture 3: Probability Distributions
Lecture 3: Probablty Dstrbutons Random Varables Let us begn by defnng a sample space as a set of outcomes from an experment. We denote ths by S. A random varable s a functon whch maps outcomes nto the
More informationMATH 5707 HOMEWORK 4 SOLUTIONS 2. 2 i 2p i E(X i ) + E(Xi 2 ) ä i=1. i=1
MATH 5707 HOMEWORK 4 SOLUTIONS CİHAN BAHRAN 1. Let v 1,..., v n R m, all lengths v are not larger than 1. Let p 1,..., p n [0, 1] be arbtrary and set w = p 1 v 1 + + p n v n. Then there exst ε 1,..., ε
More informationFinding Dense Subgraphs in G(n, 1/2)
Fndng Dense Subgraphs n Gn, 1/ Atsh Das Sarma 1, Amt Deshpande, and Rav Kannan 1 Georga Insttute of Technology,atsh@cc.gatech.edu Mcrosoft Research-Bangalore,amtdesh,annan@mcrosoft.com Abstract. Fndng
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 12 10/21/2013. Martingale Concentration Inequalities and Applications
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.65/15.070J Fall 013 Lecture 1 10/1/013 Martngale Concentraton Inequaltes and Applcatons Content. 1. Exponental concentraton for martngales wth bounded ncrements.
More informationState Amplification and State Masking for the Binary Energy Harvesting Channel
State Amplfcaton and State Maskng for the Bnary Energy Harvestng Channel Kaya Tutuncuoglu, Omur Ozel 2, Ayln Yener, and Sennur Ulukus 2 Department of Electrcal Engneerng, The Pennsylvana State Unversty,
More informationLecture 4: November 17, Part 1 Single Buffer Management
Lecturer: Ad Rosén Algorthms for the anagement of Networs Fall 2003-2004 Lecture 4: November 7, 2003 Scrbe: Guy Grebla Part Sngle Buffer anagement In the prevous lecture we taled about the Combned Input
More information3.1 Expectation of Functions of Several Random Variables. )' be a k-dimensional discrete or continuous random vector, with joint PMF p (, E X E X1 E X
Statstcs 1: Probablty Theory II 37 3 EPECTATION OF SEVERAL RANDOM VARIABLES As n Probablty Theory I, the nterest n most stuatons les not on the actual dstrbuton of a random vector, but rather on a number
More informationMore metrics on cartesian products
More metrcs on cartesan products If (X, d ) are metrc spaces for 1 n, then n Secton II4 of the lecture notes we defned three metrcs on X whose underlyng topologes are the product topology The purpose of
More informationMaximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models
ECO 452 -- OE 4: Probt and Logt Models ECO 452 -- OE 4 Maxmum Lkelhood Estmaton of Bnary Dependent Varables Models: Probt and Logt hs note demonstrates how to formulate bnary dependent varables models
More informationLossy Compression. Compromise accuracy of reconstruction for increased compression.
Lossy Compresson Compromse accuracy of reconstructon for ncreased compresson. The reconstructon s usually vsbly ndstngushable from the orgnal mage. Typcally, one can get up to 0:1 compresson wth almost
More informationLectures - Week 4 Matrix norms, Conditioning, Vector Spaces, Linear Independence, Spanning sets and Basis, Null space and Range of a Matrix
Lectures - Week 4 Matrx norms, Condtonng, Vector Spaces, Lnear Independence, Spannng sets and Bass, Null space and Range of a Matrx Matrx Norms Now we turn to assocatng a number to each matrx. We could
More informationLimited Dependent Variables
Lmted Dependent Varables. What f the left-hand sde varable s not a contnuous thng spread from mnus nfnty to plus nfnty? That s, gven a model = f (, β, ε, where a. s bounded below at zero, such as wages
More informationNP-Completeness : Proofs
NP-Completeness : Proofs Proof Methods A method to show a decson problem Π NP-complete s as follows. (1) Show Π NP. (2) Choose an NP-complete problem Π. (3) Show Π Π. A method to show an optmzaton problem
More informationLecture 10 Support Vector Machines II
Lecture 10 Support Vector Machnes II 22 February 2016 Taylor B. Arnold Yale Statstcs STAT 365/665 1/28 Notes: Problem 3 s posted and due ths upcomng Frday There was an early bug n the fake-test data; fxed
More informationModule 9. Lecture 6. Duality in Assignment Problems
Module 9 1 Lecture 6 Dualty n Assgnment Problems In ths lecture we attempt to answer few other mportant questons posed n earler lecture for (AP) and see how some of them can be explaned through the concept
More informationDifferentiating Gaussian Processes
Dfferentatng Gaussan Processes Andrew McHutchon Aprl 17, 013 1 Frst Order Dervatve of the Posteror Mean The posteror mean of a GP s gven by, f = x, X KX, X 1 y x, X α 1 Only the x, X term depends on the
More informationChannel Encoder. Channel. Figure 7.1: Communication system
Chapter 7 Processes The model of a communcaton system that we have been developng s shown n Fgure 7.. Ths model s also useful for some computaton systems. The source s assumed to emt a stream of symbols.
More informationTornado and Luby Transform Codes. Ashish Khisti Presentation October 22, 2003
Tornado and Luby Transform Codes Ashsh Khst 6.454 Presentaton October 22, 2003 Background: Erasure Channel Elas[956] studed the Erasure Channel β x x β β x 2 m x 2 k? Capacty of Noseless Erasure Channel
More informationDC-Free Turbo Coding Scheme Using MAP/SOVA Algorithms
Proceedngs of the 5th WSEAS Internatonal Conference on Telecommuncatons and Informatcs, Istanbul, Turkey, May 27-29, 26 (pp192-197 DC-Free Turbo Codng Scheme Usng MAP/SOVA Algorthms Prof. Dr. M. Amr Mokhtar
More informationEntropy Coding. A complete entropy codec, which is an encoder/decoder. pair, consists of the process of encoding or
Sgnal Compresson Sgnal Compresson Entropy Codng Entropy codng s also known as zero-error codng, data compresson or lossless compresson. Entropy codng s wdely used n vrtually all popular nternatonal multmeda
More informationAPPENDIX A Some Linear Algebra
APPENDIX A Some Lnear Algebra The collecton of m, n matrces A.1 Matrces a 1,1,..., a 1,n A = a m,1,..., a m,n wth real elements a,j s denoted by R m,n. If n = 1 then A s called a column vector. Smlarly,
More informationComplete subgraphs in multipartite graphs
Complete subgraphs n multpartte graphs FLORIAN PFENDER Unverstät Rostock, Insttut für Mathematk D-18057 Rostock, Germany Floran.Pfender@un-rostock.de Abstract Turán s Theorem states that every graph G
More informationprinceton univ. F 17 cos 521: Advanced Algorithm Design Lecture 7: LP Duality Lecturer: Matt Weinberg
prnceton unv. F 17 cos 521: Advanced Algorthm Desgn Lecture 7: LP Dualty Lecturer: Matt Wenberg Scrbe: LP Dualty s an extremely useful tool for analyzng structural propertes of lnear programs. Whle there
More informationEstimation: Part 2. Chapter GREG estimation
Chapter 9 Estmaton: Part 2 9. GREG estmaton In Chapter 8, we have seen that the regresson estmator s an effcent estmator when there s a lnear relatonshp between y and x. In ths chapter, we generalzed the
More informationMatrix Approximation via Sampling, Subspace Embedding. 1 Solving Linear Systems Using SVD
Matrx Approxmaton va Samplng, Subspace Embeddng Lecturer: Anup Rao Scrbe: Rashth Sharma, Peng Zhang 0/01/016 1 Solvng Lnear Systems Usng SVD Two applcatons of SVD have been covered so far. Today we loo
More informationDepartment of Computer Science Artificial Intelligence Research Laboratory. Iowa State University MACHINE LEARNING
MACHINE LEANING Vasant Honavar Bonformatcs and Computatonal Bology rogram Center for Computatonal Intellgence, Learnng, & Dscovery Iowa State Unversty honavar@cs.astate.edu www.cs.astate.edu/~honavar/
More informationThe Gaussian classifier. Nuno Vasconcelos ECE Department, UCSD
he Gaussan classfer Nuno Vasconcelos ECE Department, UCSD Bayesan decson theory recall that we have state of the world X observatons g decson functon L[g,y] loss of predctng y wth g Bayes decson rule s
More informationCalculation of time complexity (3%)
Problem 1. (30%) Calculaton of tme complexty (3%) Gven n ctes, usng exhaust search to see every result takes O(n!). Calculaton of tme needed to solve the problem (2%) 40 ctes:40! dfferent tours 40 add
More informationComposite Hypotheses testing
Composte ypotheses testng In many hypothess testng problems there are many possble dstrbutons that can occur under each of the hypotheses. The output of the source s a set of parameters (ponts n a parameter
More informationInner Product. Euclidean Space. Orthonormal Basis. Orthogonal
Inner Product Defnton 1 () A Eucldean space s a fnte-dmensonal vector space over the reals R, wth an nner product,. Defnton 2 (Inner Product) An nner product, on a real vector space X s a symmetrc, blnear,
More informationANSWERS. Problem 1. and the moment generating function (mgf) by. defined for any real t. Use this to show that E( U) var( U)
Econ 413 Exam 13 H ANSWERS Settet er nndelt 9 deloppgaver, A,B,C, som alle anbefales å telle lkt for å gøre det ltt lettere å stå. Svar er gtt . Unfortunately, there s a prntng error n the hnt of
More informationChapter 5. Solution of System of Linear Equations. Module No. 6. Solution of Inconsistent and Ill Conditioned Systems
Numercal Analyss by Dr. Anta Pal Assstant Professor Department of Mathematcs Natonal Insttute of Technology Durgapur Durgapur-713209 emal: anta.bue@gmal.com 1 . Chapter 5 Soluton of System of Lnear Equatons
More information6. Stochastic processes (2)
6. Stochastc processes () Lect6.ppt S-38.45 - Introducton to Teletraffc Theory Sprng 5 6. Stochastc processes () Contents Markov processes Brth-death processes 6. Stochastc processes () Markov process
More informationxp(x µ) = 0 p(x = 0 µ) + 1 p(x = 1 µ) = µ
CSE 455/555 Sprng 2013 Homework 7: Parametrc Technques Jason J. Corso Computer Scence and Engneerng SUY at Buffalo jcorso@buffalo.edu Solutons by Yngbo Zhou Ths assgnment does not need to be submtted and
More information6. Stochastic processes (2)
Contents Markov processes Brth-death processes Lect6.ppt S-38.45 - Introducton to Teletraffc Theory Sprng 5 Markov process Consder a contnuous-tme and dscrete-state stochastc process X(t) wth state space
More informationConsider the following passband digital communication system model. c t. modulator. t r a n s m i t t e r. signal decoder.
PASSBAND DIGITAL MODULATION TECHNIQUES Consder the followng passband dgtal communcaton system model. cos( ω + φ ) c t message source m sgnal encoder s modulator s () t communcaton xt () channel t r a n
More informationProblem Set 9 Solutions
Desgn and Analyss of Algorthms May 4, 2015 Massachusetts Insttute of Technology 6.046J/18.410J Profs. Erk Demane, Srn Devadas, and Nancy Lynch Problem Set 9 Solutons Problem Set 9 Solutons Ths problem
More informationLecture 4: Universal Hash Functions/Streaming Cont d
CSE 5: Desgn and Analyss of Algorthms I Sprng 06 Lecture 4: Unversal Hash Functons/Streamng Cont d Lecturer: Shayan Oves Gharan Aprl 6th Scrbe: Jacob Schreber Dsclamer: These notes have not been subjected
More informationEcon107 Applied Econometrics Topic 3: Classical Model (Studenmund, Chapter 4)
I. Classcal Assumptons Econ7 Appled Econometrcs Topc 3: Classcal Model (Studenmund, Chapter 4) We have defned OLS and studed some algebrac propertes of OLS. In ths topc we wll study statstcal propertes
More informationFirst Year Examination Department of Statistics, University of Florida
Frst Year Examnaton Department of Statstcs, Unversty of Florda May 7, 010, 8:00 am - 1:00 noon Instructons: 1. You have four hours to answer questons n ths examnaton.. You must show your work to receve
More informationAffine transformations and convexity
Affne transformatons and convexty The purpose of ths document s to prove some basc propertes of affne transformatons nvolvng convex sets. Here are a few onlne references for background nformaton: http://math.ucr.edu/
More informationU.C. Berkeley CS294: Beyond Worst-Case Analysis Luca Trevisan September 5, 2017
U.C. Berkeley CS94: Beyond Worst-Case Analyss Handout 4s Luca Trevsan September 5, 07 Summary of Lecture 4 In whch we ntroduce semdefnte programmng and apply t to Max Cut. Semdefnte Programmng Recall that
More informationPulse Coded Modulation
Pulse Coded Modulaton PCM (Pulse Coded Modulaton) s a voce codng technque defned by the ITU-T G.711 standard and t s used n dgtal telephony to encode the voce sgnal. The frst step n the analog to dgtal
More informationCS 798: Homework Assignment 2 (Probability)
0 Sample space Assgned: September 30, 2009 In the IEEE 802 protocol, the congeston wndow (CW) parameter s used as follows: ntally, a termnal wats for a random tme perod (called backoff) chosen n the range
More informationConvergence of random processes
DS-GA 12 Lecture notes 6 Fall 216 Convergence of random processes 1 Introducton In these notes we study convergence of dscrete random processes. Ths allows to characterze phenomena such as the law of large
More informationLecture 5 Decoding Binary BCH Codes
Lecture 5 Decodng Bnary BCH Codes In ths class, we wll ntroduce dfferent methods for decodng BCH codes 51 Decodng the [15, 7, 5] 2 -BCH Code Consder the [15, 7, 5] 2 -code C we ntroduced n the last lecture
More informationPh 219a/CS 219a. Exercises Due: Wednesday 23 October 2013
1 Ph 219a/CS 219a Exercses Due: Wednesday 23 October 2013 1.1 How far apart are two quantum states? Consder two quantum states descrbed by densty operators ρ and ρ n an N-dmensonal Hlbert space, and consder
More informationFor now, let us focus on a specific model of neurons. These are simplified from reality but can achieve remarkable results.
Neural Networks : Dervaton compled by Alvn Wan from Professor Jtendra Malk s lecture Ths type of computaton s called deep learnng and s the most popular method for many problems, such as computer vson
More informationErrors for Linear Systems
Errors for Lnear Systems When we solve a lnear system Ax b we often do not know A and b exactly, but have only approxmatons  and ˆb avalable. Then the best thng we can do s to solve ˆx ˆb exactly whch
More informationMarkov Chain Monte Carlo (MCMC), Gibbs Sampling, Metropolis Algorithms, and Simulated Annealing Bioinformatics Course Supplement
Markov Chan Monte Carlo MCMC, Gbbs Samplng, Metropols Algorthms, and Smulated Annealng 2001 Bonformatcs Course Supplement SNU Bontellgence Lab http://bsnuackr/ Outlne! Markov Chan Monte Carlo MCMC! Metropols-Hastngs
More informationMaximizing the number of nonnegative subsets
Maxmzng the number of nonnegatve subsets Noga Alon Hao Huang December 1, 213 Abstract Gven a set of n real numbers, f the sum of elements of every subset of sze larger than k s negatve, what s the maxmum
More informationStanford University CS359G: Graph Partitioning and Expanders Handout 4 Luca Trevisan January 13, 2011
Stanford Unversty CS359G: Graph Parttonng and Expanders Handout 4 Luca Trevsan January 3, 0 Lecture 4 In whch we prove the dffcult drecton of Cheeger s nequalty. As n the past lectures, consder an undrected
More informationGoogle PageRank with Stochastic Matrix
Google PageRank wth Stochastc Matrx Md. Sharq, Puranjt Sanyal, Samk Mtra (M.Sc. Applcatons of Mathematcs) Dscrete Tme Markov Chan Let S be a countable set (usually S s a subset of Z or Z d or R or R d
More information10-701/ Machine Learning, Fall 2005 Homework 3
10-701/15-781 Machne Learnng, Fall 2005 Homework 3 Out: 10/20/05 Due: begnnng of the class 11/01/05 Instructons Contact questons-10701@autonlaborg for queston Problem 1 Regresson and Cross-valdaton [40
More informationU.C. Berkeley CS294: Beyond Worst-Case Analysis Handout 6 Luca Trevisan September 12, 2017
U.C. Berkeley CS94: Beyond Worst-Case Analyss Handout 6 Luca Trevsan September, 07 Scrbed by Theo McKenze Lecture 6 In whch we study the spectrum of random graphs. Overvew When attemptng to fnd n polynomal
More informationThe Multiple Classical Linear Regression Model (CLRM): Specification and Assumptions. 1. Introduction
ECONOMICS 5* -- NOTE (Summary) ECON 5* -- NOTE The Multple Classcal Lnear Regresson Model (CLRM): Specfcaton and Assumptons. Introducton CLRM stands for the Classcal Lnear Regresson Model. The CLRM s also
More informationMaximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models
ECO 452 -- OE 4: Probt and Logt Models ECO 452 -- OE 4 Mamum Lkelhood Estmaton of Bnary Dependent Varables Models: Probt and Logt hs note demonstrates how to formulate bnary dependent varables models for
More informationDynamic Programming. Preview. Dynamic Programming. Dynamic Programming. Dynamic Programming (Example: Fibonacci Sequence)
/24/27 Prevew Fbonacc Sequence Longest Common Subsequence Dynamc programmng s a method for solvng complex problems by breakng them down nto smpler sub-problems. It s applcable to problems exhbtng the propertes
More informationECE559VV Project Report
ECE559VV Project Report (Supplementary Notes Loc Xuan Bu I. MAX SUM-RATE SCHEDULING: THE UPLINK CASE We have seen (n the presentaton that, for downlnk (broadcast channels, the strategy maxmzng the sum-rate
More informationSupplementary material: Margin based PU Learning. Matrix Concentration Inequalities
Supplementary materal: Margn based PU Learnng We gve the complete proofs of Theorem and n Secton We frst ntroduce the well-known concentraton nequalty, so the covarance estmator can be bounded Then we
More informationNAME and Section No.
Chemstry 391 Fall 2007 Exam I KEY (Monday September 17) 1. (25 Ponts) ***Do 5 out of 6***(If 6 are done only the frst 5 wll be graded)*** a). Defne the terms: open system, closed system and solated system
More informationMATH 241B FUNCTIONAL ANALYSIS - NOTES EXAMPLES OF C ALGEBRAS
MATH 241B FUNCTIONAL ANALYSIS - NOTES EXAMPLES OF C ALGEBRAS These are nformal notes whch cover some of the materal whch s not n the course book. The man purpose s to gve a number of nontrval examples
More informationMin Cut, Fast Cut, Polynomial Identities
Randomzed Algorthms, Summer 016 Mn Cut, Fast Cut, Polynomal Identtes Instructor: Thomas Kesselhem and Kurt Mehlhorn 1 Mn Cuts n Graphs Lecture (5 pages) Throughout ths secton, G = (V, E) s a mult-graph.
More informationSELECTED PROOFS. DeMorgan s formulas: The first one is clear from Venn diagram, or the following truth table:
SELECTED PROOFS DeMorgan s formulas: The frst one s clear from Venn dagram, or the followng truth table: A B A B A B Ā B Ā B T T T F F F F T F T F F T F F T T F T F F F F F T T T T The second one can be
More informationHomework Assignment 3 Due in class, Thursday October 15
Homework Assgnment 3 Due n class, Thursday October 15 SDS 383C Statstcal Modelng I 1 Rdge regresson and Lasso 1. Get the Prostrate cancer data from http://statweb.stanford.edu/~tbs/elemstatlearn/ datasets/prostate.data.
More informationLecture 9: Converse of Shannon s Capacity Theorem
Error Correctng Codes: Combnatorcs, Algorthms and Alcatons (Fall 2007) Lecture 9: Converse of Shannon s Caacty Theorem Setember 17, 2007 Lecturer: Atr Rudra Scrbe: Thanh-Nhan Nguyen & Atr Rudra In the
More informationFinding Primitive Roots Pseudo-Deterministically
Electronc Colloquum on Computatonal Complexty, Report No 207 (205) Fndng Prmtve Roots Pseudo-Determnstcally Ofer Grossman December 22, 205 Abstract Pseudo-determnstc algorthms are randomzed search algorthms
More informationSpectral Graph Theory and its Applications September 16, Lecture 5
Spectral Graph Theory and ts Applcatons September 16, 2004 Lecturer: Danel A. Spelman Lecture 5 5.1 Introducton In ths lecture, we wll prove the followng theorem: Theorem 5.1.1. Let G be a planar graph
More informationThe strict priority scheduler
6. SP and GPS schedulers Pag. The strct prorty scheduler The strct prorty scheduler s very smple and effcent In the followng, we wll carry out the analyss of ths scheduler to determne the formulas for
More informationDigital Modems. Lecture 2
Dgtal Modems Lecture Revew We have shown that both Bayes and eyman/pearson crtera are based on the Lkelhood Rato Test (LRT) Λ ( r ) < > η Λ r s called observaton transformaton or suffcent statstc The crtera
More informationTHE ARIMOTO-BLAHUT ALGORITHM FOR COMPUTATION OF CHANNEL CAPACITY. William A. Pearlman. References: S. Arimoto - IEEE Trans. Inform. Thy., Jan.
THE ARIMOTO-BLAHUT ALGORITHM FOR COMPUTATION OF CHANNEL CAPACITY Wllam A. Pearlman 2002 References: S. Armoto - IEEE Trans. Inform. Thy., Jan. 1972 R. Blahut - IEEE Trans. Inform. Thy., July 1972 Recall
More informationGeneralized Linear Methods
Generalzed Lnear Methods 1 Introducton In the Ensemble Methods the general dea s that usng a combnaton of several weak learner one could make a better learner. More formally, assume that we have a set
More informationOn complexity and randomness of Markov-chain prediction
On complexty and randomness of Markov-chan predcton Joel Ratsaby Department of Electrcal and Electroncs Engneerng Arel Unversty Arel, ISRAEL Emal: ratsaby@arelacl Abstract Let {X t : t Z} be a sequence
More informationModule 2. Random Processes. Version 2 ECE IIT, Kharagpur
Module Random Processes Lesson 6 Functons of Random Varables After readng ths lesson, ou wll learn about cdf of functon of a random varable. Formula for determnng the pdf of a random varable. Let, X be
More information= z 20 z n. (k 20) + 4 z k = 4
Problem Set #7 solutons 7.2.. (a Fnd the coeffcent of z k n (z + z 5 + z 6 + z 7 + 5, k 20. We use the known seres expanson ( n+l ( z l l z n below: (z + z 5 + z 6 + z 7 + 5 (z 5 ( + z + z 2 + z + 5 5
More informationWhich Separator? Spring 1
Whch Separator? 6.034 - Sprng 1 Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng 3 Margn of a pont " # y (w $ + b) proportonal
More informationApproximate Smallest Enclosing Balls
Chapter 5 Approxmate Smallest Enclosng Balls 5. Boundng Volumes A boundng volume for a set S R d s a superset of S wth a smple shape, for example a box, a ball, or an ellpsod. Fgure 5.: Boundng boxes Q(P
More informationWinter 2008 CS567 Stochastic Linear/Integer Programming Guest Lecturer: Xu, Huan
Wnter 2008 CS567 Stochastc Lnear/Integer Programmng Guest Lecturer: Xu, Huan Class 2: More Modelng Examples 1 Capacty Expanson Capacty expanson models optmal choces of the tmng and levels of nvestments
More informationExcess Error, Approximation Error, and Estimation Error
E0 370 Statstcal Learnng Theory Lecture 10 Sep 15, 011 Excess Error, Approxaton Error, and Estaton Error Lecturer: Shvan Agarwal Scrbe: Shvan Agarwal 1 Introducton So far, we have consdered the fnte saple
More informationIntroduction to information theory and data compression
Introducton to nformaton theory and data compresson Adel Magra, Emma Gouné, Irène Woo March 8, 207 Ths s the augmented transcrpt of a lecture gven by Luc Devroye on March 9th 207 for a Data Structures
More information2.3 Nilpotent endomorphisms
s a block dagonal matrx, wth A Mat dm U (C) In fact, we can assume that B = B 1 B k, wth B an ordered bass of U, and that A = [f U ] B, where f U : U U s the restrcton of f to U 40 23 Nlpotent endomorphsms
More information