Lecture 9: Converse of Shannon s Capacity Theorem
|
|
- Vivian Barton
- 5 years ago
- Views:
Transcription
1 Error Correctng Codes: Combnatorcs, Algorthms and Alcatons (Fall 2007) Lecture 9: Converse of Shannon s Caacty Theorem Setember 17, 2007 Lecturer: Atr Rudra Scrbe: Thanh-Nhan Nguyen & Atr Rudra In the last lecture, we stated Shannon s caacty theorem for the BSC, whch we restate here: Theorem 0.1. Let 0 < 1/2 be a real number. For every 0 < ε 1/2, the followng statements are true for large enough nteger n: () There exsts a real δ > 0, an encodng functon E : {0, 1} k {0, 1} n, and a decodng functon D : {0, 1} n {0, 1} k, where k (1 H( + ε))n such that the followng holds for every m {0, 1} k : P r [D(E(m) + e) m] 2 δn. nose e of BSC () If k (1 H() + ε)n then for every encodng and decodng functons E : {0, 1} k {0, 1} n and D : {0, 1} n {0, 1} k the followng s true for some m {0, 1} k : P r [D(E(m) + e) m] 1/2. nose e of BSC In today s lecture, we wll rove art () of Theorem Prelmnares Before we begn wth the roof we wll need a few results, whch we dscuss frst. 1.1 Chernoff Bound Chernoff bound states a bound on the tal of a certan dstrbuton that wll be useful for us. Here we state the verson of the Chernoff bound that we wll need. Prooston 1.1. For 1,, n, let X be a bnary random varable that takes a value of 1 wth robablty and a value of 0 wth robablty 1. Then the followng bounds are true: () P r [ n 1 X (1 + ε)n] e ε2 n/3 () P r [ n 1 X (1 ε)n] e ε2 n/3 Note that the exectaton of the sum n 1 X s n. The bound above states that the robablty mass s tghtly concentrated around the mean. 1
2 1.2 Volume of Hammng Balls We wll also need good uer and lower bounds on the volume of a Hammng ball. Recall that V ol q (0, n) B q (0, ρn) n ( n ) (q 1). We wll rove the followng result: Prooston 1.2. Let q 2 be an nteger and q () V ol q (0, n) q Hq()n () V ol q (0, n) q Hq()n o(n) where recall that H q (x) x log q (q 1) x log q x (1 x) log q (1 x). be a real. Then for large enough n: Proof. We start wth the roof of (). Consder the followng sequence of relatons: 1 ( + (1 )) n n (1 ) n (1) n (1 ) n (2) n ( ) ( ) n (q 1) (1 ) n q 1 n ( ( ) n )(q 1) (1 ) n (q 1)(1 ) n ( ( ) n n )(q 1) (1 ) n (3) (q 1)(1 ) n ( ) n (q 1) (1 ) (1 )n. (4) q 1 In the above, (1) follows from the bnomal exanson. (2) follows by drong some terms from the summaton and (3) follows from that facts that 1 (as q 2 and 1/2) and (q 1)(1 ) n 1 (for large enough ( n). Rest of the stes follow from rearrangng the terms. n As q Hq()n q 1) (1 ) (1 )n, (4) mles that 1 V ol q (0, n)q Hq()n, whch roves (). We now turn to the roof of art (). For ths art, we wll need Strlng s aroxmaton for n! ( n ) n 2πn e λ 1 (n) < n! < ( n ) n 2πn e λ 2 (n), e e 2
3 where 1 λ 1 (n) 12n + 1 and λ 2(n) 1 12n. By the Strlng s aroxmaton, we have the followng nequalty: n! n (n)!((1 )n)! (n/e) n > (n/e) n ((1 )n/e) 1 e λ 1(n) λ 2 (n) λ 2 ((1 )n) (1 )n 2π(1 )n 1 l(n), (5) n (1 ) (1 )n where l(n) eλ 1 (n) λ 2 (n) λ 2 ((1 )n) 2π(1 )n. Now consder the followng sequence of relatons that comlete the roof: V ol q (0, n) (q 1) n (6) n (q 1) n > l(n) (7) n (1 ) (1 )n q Hq()n o(n). (8) In the above (6) follows by only lookng at one term. (7) follows from (5) whle (8) follows from the defnton of H q ( ) and the fact that for large enough n, l(n) s q o(n). 2 Converse of Shannon s Caacty Theorem for BSC We wll now rove art () of Theorem 0.1: the roof of the other art wll be done n the next lecture. Frst, we note that there s nothng to rove f 0, so for the rest of the roof we wll assume that > 0. For the sake of contradcton, assume that the followng holds for every m {0, 1} k : P r [D(E(m) + e) m] 1/2. nose e of BSC Fx an arbtrary message m {0, 1} k. Defne D m to be the set of receved words that are decoded to m by D, that s, D m {y D(y) m}. Note that by our assumton, the followng s true (where from now on we omt the exlct deendence of the robablty on the BSC nose for clarty): P r [E(m) + e D m ] 1/2. (9) 3
4 Further, by the Chernoff bound, P r[e(m) + e S m ] 2 Ω(γ2 n), (10) where S m s the shell of radus [(1 γ)n, (1 + γ)n] around E(m), that s, S m B 2 (E(m), (1 + γ)n) \ B 2 (E(m), (1 γ)n). (We wll set γ > 0 n terms of ε and at the end of the roof.) (9) and (10) along wth the unon bound mly the followng: P r [E(m) + e D m S m ] Ω(γ2 n) 1 4, (11) where the last nequalty holds for large enough n. Next we uer bound the robablty above to obtan a lower bound on D m S m. It s easy to see that where P r [E(m) + e D m S m ] D m S m max, max max P r[e(m) + e y] max d (1 ) n d. y S m d [(1 γ)n,(1+γ)n] It s easy to check that d (1 ) n d s decreasng n d for 1/2. Thus, we have ( ) γn ( ) γn 1 1 max (1 γ)n (1 ) n (1 γ)n n (1 ) (1 )n 2 nh(). Thus, we have shown that ( ) γn 1 P r [E(m) + e D m S m ] D m S m 2 nh(), whch by (11) mles that D m S 1 ( ) γn nh(). (12) Next, we consder the followng sequence of relatons: 2 n D m (13) m {0,1} k D m S m {0,1} k 1 ( ) γn 1 (14) 4 m {0,1} k 2 H()n 2 k 2 H()n γ log(1/ 1)n 2 > 2 k+h()n εn. (15) 4
5 In the above (13) follows from the fact that for m 1 m 2, D m1 and D m2 are dsjont. (14) ε follows from (12). (15) follows for large enough n and f we ck γ 2 log( 1 1). (Note that as 0 < < 1/2, γ Θ(ε).) (15) mles that k < (1 H() + ε)n, whch s a contradcton. The roof of art () of Theorem 0.1 s comlete. Remark 2.1. It can be verfed that the roof above can also work f the decodng error robablty s bounded by 2 βn (nstead of the 1/2 n art () of Theorem 0.1) for small enough β β(ε) > 0. 5
Lecture 3: Shannon s Theorem
CSE 533: Error-Correctng Codes (Autumn 006 Lecture 3: Shannon s Theorem October 9, 006 Lecturer: Venkatesan Guruswam Scrbe: Wdad Machmouch 1 Communcaton Model The communcaton model we are usng conssts
More informationLecture 14 (03/27/18). Channels. Decoding. Preview of the Capacity Theorem.
Lecture 14 (03/27/18). Channels. Decodng. Prevew of the Capacty Theorem. A. Barg The concept of a communcaton channel n nformaton theory s an abstracton for transmttng dgtal (and analog) nformaton from
More informationEigenvalues of Random Graphs
Spectral Graph Theory Lecture 2 Egenvalues of Random Graphs Danel A. Spelman November 4, 202 2. Introducton In ths lecture, we consder a random graph on n vertces n whch each edge s chosen to be n the
More informationfind (x): given element x, return the canonical element of the set containing x;
COS 43 Sprng, 009 Dsjont Set Unon Problem: Mantan a collecton of dsjont sets. Two operatons: fnd the set contanng a gven element; unte two sets nto one (destructvely). Approach: Canoncal element method:
More informationRandomness and Computation
Randomness and Computaton or, Randomzed Algorthms Mary Cryan School of Informatcs Unversty of Ednburgh RC 208/9) Lecture 0 slde Balls n Bns m balls, n bns, and balls thrown unformly at random nto bns usually
More informationManaging Capacity Through Reward Programs. on-line companion page. Byung-Do Kim Seoul National University College of Business Administration
Managng Caacty Through eward Programs on-lne comanon age Byung-Do Km Seoul Natonal Unversty College of Busness Admnstraton Mengze Sh Unversty of Toronto otman School of Management Toronto ON M5S E6 Canada
More informationprinceton univ. F 17 cos 521: Advanced Algorithm Design Lecture 7: LP Duality Lecturer: Matt Weinberg
prnceton unv. F 17 cos 521: Advanced Algorthm Desgn Lecture 7: LP Dualty Lecturer: Matt Wenberg Scrbe: LP Dualty s an extremely useful tool for analyzng structural propertes of lnear programs. Whle there
More informationLecture 10: May 6, 2013
TTIC/CMSC 31150 Mathematcal Toolkt Sprng 013 Madhur Tulsan Lecture 10: May 6, 013 Scrbe: Wenje Luo In today s lecture, we manly talked about random walk on graphs and ntroduce the concept of graph expander,
More informationLearning Theory: Lecture Notes
Learnng Theory: Lecture Notes Lecturer: Kamalka Chaudhur Scrbe: Qush Wang October 27, 2012 1 The Agnostc PAC Model Recall that one of the constrants of the PAC model s that the data dstrbuton has to be
More informationPARTIAL QUOTIENTS AND DISTRIBUTION OF SEQUENCES. Department of Mathematics University of California Riverside, CA
PARTIAL QUOTIETS AD DISTRIBUTIO OF SEQUECES 1 Me-Chu Chang Deartment of Mathematcs Unversty of Calforna Rversde, CA 92521 mcc@math.ucr.edu Abstract. In ths aer we establsh average bounds on the artal quotents
More informationSupplementary Material for Spectral Clustering based on the graph p-laplacian
Sulementary Materal for Sectral Clusterng based on the grah -Lalacan Thomas Bühler and Matthas Hen Saarland Unversty, Saarbrücken, Germany {tb,hen}@csun-sbde May 009 Corrected verson, June 00 Abstract
More informationLecture 5 Decoding Binary BCH Codes
Lecture 5 Decodng Bnary BCH Codes In ths class, we wll ntroduce dfferent methods for decodng BCH codes 51 Decodng the [15, 7, 5] 2 -BCH Code Consder the [15, 7, 5] 2 -code C we ntroduced n the last lecture
More informationModule 2. Random Processes. Version 2 ECE IIT, Kharagpur
Module Random Processes Lesson 6 Functons of Random Varables After readng ths lesson, ou wll learn about cdf of functon of a random varable. Formula for determnng the pdf of a random varable. Let, X be
More informationThe Number of Ways to Write n as a Sum of ` Regular Figurate Numbers
Syracuse Unversty SURFACE Syracuse Unversty Honors Program Capstone Projects Syracuse Unversty Honors Program Capstone Projects Sprng 5-1-01 The Number of Ways to Wrte n as a Sum of ` Regular Fgurate Numbers
More informationA NOTE ON THE DISCRETE FOURIER RESTRICTION PROBLEM
A NOTE ON THE DISCRETE FOURIER RESTRICTION PROBLEM XUDONG LAI AND YONG DING arxv:171001481v1 [mathap] 4 Oct 017 Abstract In ths aer we establsh a general dscrete Fourer restrcton theorem As an alcaton
More informationTAIL BOUNDS FOR SUMS OF GEOMETRIC AND EXPONENTIAL VARIABLES
TAIL BOUNDS FOR SUMS OF GEOMETRIC AND EXPONENTIAL VARIABLES SVANTE JANSON Abstract. We gve explct bounds for the tal probabltes for sums of ndependent geometrc or exponental varables, possbly wth dfferent
More informationarxiv: v1 [math.co] 1 Mar 2014
Unon-ntersectng set systems Gyula O.H. Katona and Dánel T. Nagy March 4, 014 arxv:1403.0088v1 [math.co] 1 Mar 014 Abstract Three ntersecton theorems are proved. Frst, we determne the sze of the largest
More informationAn application of generalized Tsalli s-havrda-charvat entropy in coding theory through a generalization of Kraft inequality
Internatonal Journal of Statstcs and Aled Mathematcs 206; (4): 0-05 ISS: 2456-452 Maths 206; (4): 0-05 206 Stats & Maths wwwmathsjournalcom Receved: 0-09-206 Acceted: 02-0-206 Maharsh Markendeshwar Unversty,
More informationLecture 3 January 31, 2017
CS 224: Advanced Algorthms Sprng 207 Prof. Jelan Nelson Lecture 3 January 3, 207 Scrbe: Saketh Rama Overvew In the last lecture we covered Y-fast tres and Fuson Trees. In ths lecture we start our dscusson
More informationGenericity of Critical Types
Genercty of Crtcal Types Y-Chun Chen Alfredo D Tllo Eduardo Fangold Syang Xong September 2008 Abstract Ely and Pesk 2008 offers an nsghtful characterzaton of crtcal types: a type s crtcal f and only f
More informationModule 3 LOSSY IMAGE COMPRESSION SYSTEMS. Version 2 ECE IIT, Kharagpur
Module 3 LOSSY IMAGE COMPRESSION SYSTEMS Verson ECE IIT, Kharagpur Lesson 6 Theory of Quantzaton Verson ECE IIT, Kharagpur Instructonal Objectves At the end of ths lesson, the students should be able to:
More informationLecture 3. Ax x i a i. i i
18.409 The Behavor of Algorthms n Practce 2/14/2 Lecturer: Dan Spelman Lecture 3 Scrbe: Arvnd Sankar 1 Largest sngular value In order to bound the condton number, we need an upper bound on the largest
More informationOn the Connectedness of the Solution Set for the Weak Vector Variational Inequality 1
Journal of Mathematcal Analyss and Alcatons 260, 15 2001 do:10.1006jmaa.2000.7389, avalable onlne at htt:.dealbrary.com on On the Connectedness of the Soluton Set for the Weak Vector Varatonal Inequalty
More informationEdge Isoperimetric Inequalities
November 7, 2005 Ross M. Rchardson Edge Isopermetrc Inequaltes 1 Four Questons Recall that n the last lecture we looked at the problem of sopermetrc nequaltes n the hypercube, Q n. Our noton of boundary
More informationExcess Error, Approximation Error, and Estimation Error
E0 370 Statstcal Learnng Theory Lecture 10 Sep 15, 011 Excess Error, Approxaton Error, and Estaton Error Lecturer: Shvan Agarwal Scrbe: Shvan Agarwal 1 Introducton So far, we have consdered the fnte saple
More informationComputational and Statistical Learning theory Assignment 4
Coputatonal and Statstcal Learnng theory Assgnent 4 Due: March 2nd Eal solutons to : karthk at ttc dot edu Notatons/Defntons Recall the defnton of saple based Radeacher coplexty : [ ] R S F) := E ɛ {±}
More information18.1 Introduction and Recap
CS787: Advanced Algorthms Scrbe: Pryananda Shenoy and Shjn Kong Lecturer: Shuch Chawla Topc: Streamng Algorthmscontnued) Date: 0/26/2007 We contnue talng about streamng algorthms n ths lecture, ncludng
More informationCOS 511: Theoretical Machine Learning
COS 5: Theoretcal Machne Learnng Lecturer: Rob Schapre Lecture #0 Scrbe: José Sões Ferrera March 06, 203 In the last lecture the concept of Radeacher coplexty was ntroduced, wth the goal of showng that
More informationBernoulli Numbers and Polynomials
Bernoull Numbers and Polynomals T. Muthukumar tmk@tk.ac.n 17 Jun 2014 The sum of frst n natural numbers 1, 2, 3,..., n s n n(n + 1 S 1 (n := m = = n2 2 2 + n 2. Ths formula can be derved by notng that
More informationDerivatives of Value at Risk and Expected Shortfall
Dervatves of Value at Rsk and Exected Shortfall November 2003 Hans Rau-Bredow hans.rau-bredow@mal.un-wuerzburg.de Dr. Hans Rau-Bredow Unversty of Cologne; Unversty of Würzburg Leo Wesmantel Str. 4 D-97074
More informationError Probability for M Signals
Chapter 3 rror Probablty for M Sgnals In ths chapter we dscuss the error probablty n decdng whch of M sgnals was transmtted over an arbtrary channel. We assume the sgnals are represented by a set of orthonormal
More information1 Definition of Rademacher Complexity
COS 511: Theoretcal Machne Learnng Lecturer: Rob Schapre Lecture #9 Scrbe: Josh Chen March 5, 2013 We ve spent the past few classes provng bounds on the generalzaton error of PAClearnng algorths for the
More informationRefined Coding Bounds for Network Error Correction
Refned Codng Bounds for Network Error Correcton Shenghao Yang Department of Informaton Engneerng The Chnese Unversty of Hong Kong Shatn, N.T., Hong Kong shyang5@e.cuhk.edu.hk Raymond W. Yeung Department
More informationJoint Decoding of Content-Replication Codes for Flash Memories
Ffty-thrd Annual Allerton Conference Allerton House, UIUC, Illnos, USA Setember 29 - October 2, 2015 Jont Decodng of Content-Relcaton Codes for Flash Memores Qng L, Huan Chang, Anxao (Andrew) Jang, and
More informationAdvanced Topics in Optimization. Piecewise Linear Approximation of a Nonlinear Function
Advanced Tocs n Otmzaton Pecewse Lnear Aroxmaton of a Nonlnear Functon Otmzaton Methods: M8L Introducton and Objectves Introducton There exsts no general algorthm for nonlnear rogrammng due to ts rregular
More information1 The Mistake Bound Model
5-850: Advanced Algorthms CMU, Sprng 07 Lecture #: Onlne Learnng and Multplcatve Weghts February 7, 07 Lecturer: Anupam Gupta Scrbe: Bryan Lee,Albert Gu, Eugene Cho he Mstake Bound Model Suppose there
More informationNon-Ideality Through Fugacity and Activity
Non-Idealty Through Fugacty and Actvty S. Patel Deartment of Chemstry and Bochemstry, Unversty of Delaware, Newark, Delaware 19716, USA Corresondng author. E-mal: saatel@udel.edu 1 I. FUGACITY In ths dscusson,
More informationWhat Independencies does a Bayes Net Model? Bayesian Networks: Independencies and Inference. Quick proof that independence is symmetric
Bayesan Networks: Indeendences and Inference Scott Daves and ndrew Moore Note to other teachers and users of these sldes. ndrew and Scott would be delghted f you found ths source materal useful n gvng
More informationAlgorithms for factoring
CSA E0 235: Crytograhy Arl 9,2015 Instructor: Arta Patra Algorthms for factorng Submtted by: Jay Oza, Nranjan Sngh Introducton Factorsaton of large ntegers has been a wdely studed toc manly because of
More informationprinceton univ. F 13 cos 521: Advanced Algorithm Design Lecture 3: Large deviations bounds and applications Lecturer: Sanjeev Arora
prnceton unv. F 13 cos 521: Advanced Algorthm Desgn Lecture 3: Large devatons bounds and applcatons Lecturer: Sanjeev Arora Scrbe: Today s topc s devaton bounds: what s the probablty that a random varable
More informationFeature Selection: Part 1
CSE 546: Machne Learnng Lecture 5 Feature Selecton: Part 1 Instructor: Sham Kakade 1 Regresson n the hgh dmensonal settng How do we learn when the number of features d s greater than the sample sze n?
More informationCOS 511: Theoretical Machine Learning. Lecturer: Rob Schapire Lecture #16 Scribe: Yannan Wang April 3, 2014
COS 511: Theoretcal Machne Learnng Lecturer: Rob Schapre Lecture #16 Scrbe: Yannan Wang Aprl 3, 014 1 Introducton The goal of our onlne learnng scenaro from last class s C comparng wth best expert and
More informationSpectral Graph Theory and its Applications September 16, Lecture 5
Spectral Graph Theory and ts Applcatons September 16, 2004 Lecturer: Danel A. Spelman Lecture 5 5.1 Introducton In ths lecture, we wll prove the followng theorem: Theorem 5.1.1. Let G be a planar graph
More informationLecture Space-Bounded Derandomization
Notes on Complexty Theory Last updated: October, 2008 Jonathan Katz Lecture Space-Bounded Derandomzaton 1 Space-Bounded Derandomzaton We now dscuss derandomzaton of space-bounded algorthms. Here non-trval
More informationThe Expectation-Maximization Algorithm
The Expectaton-Maxmaton Algorthm Charles Elan elan@cs.ucsd.edu November 16, 2007 Ths chapter explans the EM algorthm at multple levels of generalty. Secton 1 gves the standard hgh-level verson of the algorthm.
More information10-801: Advanced Optimization and Randomized Methods Lecture 2: Convex functions (Jan 15, 2014)
0-80: Advanced Optmzaton and Randomzed Methods Lecture : Convex functons (Jan 5, 04) Lecturer: Suvrt Sra Addr: Carnege Mellon Unversty, Sprng 04 Scrbes: Avnava Dubey, Ahmed Hefny Dsclamer: These notes
More informationMaximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models
ECO 452 -- OE 4: Probt and Logt Models ECO 452 -- OE 4 Maxmum Lkelhood Estmaton of Bnary Dependent Varables Models: Probt and Logt hs note demonstrates how to formulate bnary dependent varables models
More informationMATH 829: Introduction to Data Mining and Analysis The EM algorithm (part 2)
1/16 MATH 829: Introducton to Data Mnng and Analyss The EM algorthm (part 2) Domnque Gullot Departments of Mathematcal Scences Unversty of Delaware Aprl 20, 2016 Recall 2/16 We are gven ndependent observatons
More informationFirst day August 1, Problems and Solutions
FOURTH INTERNATIONAL COMPETITION FOR UNIVERSITY STUDENTS IN MATHEMATICS July 30 August 4, 997, Plovdv, BULGARIA Frst day August, 997 Problems and Solutons Problem. Let {ε n } n= be a sequence of postve
More informationSome congruences related to harmonic numbers and the terms of the second order sequences
Mathematca Moravca Vol. 0: 06, 3 37 Some congruences related to harmonc numbers the terms of the second order sequences Neşe Ömür Sbel Koaral Abstract. In ths aer, wth hels of some combnatoral denttes,
More informationSTEINHAUS PROPERTY IN BANACH LATTICES
DEPARTMENT OF MATHEMATICS TECHNICAL REPORT STEINHAUS PROPERTY IN BANACH LATTICES DAMIAN KUBIAK AND DAVID TIDWELL SPRING 2015 No. 2015-1 TENNESSEE TECHNOLOGICAL UNIVERSITY Cookevlle, TN 38505 STEINHAUS
More informationMaximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models
ECO 452 -- OE 4: Probt and Logt Models ECO 452 -- OE 4 Mamum Lkelhood Estmaton of Bnary Dependent Varables Models: Probt and Logt hs note demonstrates how to formulate bnary dependent varables models for
More informationWeek 2. This week, we covered operations on sets and cardinality.
Week 2 Ths week, we covered operatons on sets and cardnalty. Defnton 0.1 (Correspondence). A correspondence between two sets A and B s a set S contaned n A B = {(a, b) a A, b B}. A correspondence from
More informationOnline Classification: Perceptron and Winnow
E0 370 Statstcal Learnng Theory Lecture 18 Nov 8, 011 Onlne Classfcaton: Perceptron and Wnnow Lecturer: Shvan Agarwal Scrbe: Shvan Agarwal 1 Introducton In ths lecture we wll start to study the onlne learnng
More informationSUPPLEMENT TO ROP: MATRIX RECOVERY VIA RANK-ONE PROJECTIONS 1
SUPPLEMENT TO ROP: MATRIX RECOVERY VIA RANK-ONE PROJECTIONS BY T. TONY CAI and ANRU ZHANG Deartment of Statstcs The Wharton School Unversty of Pennsylvana We rove n ths sulement the techncal lemmas used
More informationSection 8.3 Polar Form of Complex Numbers
80 Chapter 8 Secton 8 Polar Form of Complex Numbers From prevous classes, you may have encountered magnary numbers the square roots of negatve numbers and, more generally, complex numbers whch are the
More informationMLE and Bayesian Estimation. Jie Tang Department of Computer Science & Technology Tsinghua University 2012
MLE and Bayesan Estmaton Je Tang Department of Computer Scence & Technology Tsnghua Unversty 01 1 Lnear Regresson? As the frst step, we need to decde how we re gong to represent the functon f. One example:
More informationA random variable is a function which associates a real number to each element of the sample space
Introducton to Random Varables Defnton of random varable Defnton of of random varable Dscrete and contnuous random varable Probablty blt functon Dstrbuton functon Densty functon Sometmes, t s not enough
More information1 Convex Optimization
Convex Optmzaton We wll consder convex optmzaton problems. Namely, mnmzaton problems where the objectve s convex (we assume no constrants for now). Such problems often arse n machne learnng. For example,
More informationThe Order Relation and Trace Inequalities for. Hermitian Operators
Internatonal Mathematcal Forum, Vol 3, 08, no, 507-57 HIKARI Ltd, wwwm-hkarcom https://doorg/0988/mf088055 The Order Relaton and Trace Inequaltes for Hermtan Operators Y Huang School of Informaton Scence
More informationEntropy Coding. A complete entropy codec, which is an encoder/decoder. pair, consists of the process of encoding or
Sgnal Compresson Sgnal Compresson Entropy Codng Entropy codng s also known as zero-error codng, data compresson or lossless compresson. Entropy codng s wdely used n vrtually all popular nternatonal multmeda
More informationMaximizing the number of nonnegative subsets
Maxmzng the number of nonnegatve subsets Noga Alon Hao Huang December 1, 213 Abstract Gven a set of n real numbers, f the sum of elements of every subset of sze larger than k s negatve, what s the maxmum
More informationChapter 1. Probability
Chapter. Probablty Mcroscopc propertes of matter: quantum mechancs, atomc and molecular propertes Macroscopc propertes of matter: thermodynamcs, E, H, C V, C p, S, A, G How do we relate these two propertes?
More informationFor now, let us focus on a specific model of neurons. These are simplified from reality but can achieve remarkable results.
Neural Networks : Dervaton compled by Alvn Wan from Professor Jtendra Malk s lecture Ths type of computaton s called deep learnng and s the most popular method for many problems, such as computer vson
More informationIntroduction to Random Variables
Introducton to Random Varables Defnton of random varable Defnton of random varable Dscrete and contnuous random varable Probablty functon Dstrbuton functon Densty functon Sometmes, t s not enough to descrbe
More informationAppendix B. Criterion of Riemann-Stieltjes Integrability
Appendx B. Crteron of Remann-Steltes Integrablty Ths note s complementary to [R, Ch. 6] and [T, Sec. 3.5]. The man result of ths note s Theorem B.3, whch provdes the necessary and suffcent condtons for
More informationLinear, affine, and convex sets and hulls In the sequel, unless otherwise specified, X will denote a real vector space.
Lnear, affne, and convex sets and hulls In the sequel, unless otherwse specfed, X wll denote a real vector space. Lnes and segments. Gven two ponts x, y X, we defne xy = {x + t(y x) : t R} = {(1 t)x +
More informationU.C. Berkeley CS294: Spectral Methods and Expanders Handout 8 Luca Trevisan February 17, 2016
U.C. Berkeley CS94: Spectral Methods and Expanders Handout 8 Luca Trevsan February 7, 06 Lecture 8: Spectral Algorthms Wrap-up In whch we talk about even more generalzatons of Cheeger s nequaltes, and
More informationSelf-complementing permutations of k-uniform hypergraphs
Dscrete Mathematcs Theoretcal Computer Scence DMTCS vol. 11:1, 2009, 117 124 Self-complementng permutatons of k-unform hypergraphs Artur Szymańsk A. Paweł Wojda Faculty of Appled Mathematcs, AGH Unversty
More information1 Bref Introducton Ths memo reorts artal results regardng the task of testng whether a gven bounded-degree grah s an exander. The model s of testng gr
On Testng Exanson n Bounded-Degree Grahs Oded Goldrech Det. of Comuter Scence Wezmann Insttute of Scence Rehovot, Israel oded@wsdom.wezmann.ac.l Dana Ron Det. of EE { Systems Tel Avv Unversty Ramat Avv,
More informationLecture 20: Lift and Project, SDP Duality. Today we will study the Lift and Project method. Then we will prove the SDP duality theorem.
prnceton u. sp 02 cos 598B: algorthms and complexty Lecture 20: Lft and Project, SDP Dualty Lecturer: Sanjeev Arora Scrbe:Yury Makarychev Today we wll study the Lft and Project method. Then we wll prove
More information6) Derivatives, gradients and Hessian matrices
30C00300 Mathematcal Methods for Economsts (6 cr) 6) Dervatves, gradents and Hessan matrces Smon & Blume chapters: 14, 15 Sldes by: Tmo Kuosmanen 1 Outlne Defnton of dervatve functon Dervatve notatons
More informationEXPONENTIAL AND MOMENT INEQUALITIES FOR U-STATISTICS
EXPONENTIAL AND MOMENT INEQUALITIES FOR U-STATISTICS EVARIST GINÉ, RAFA L LATA LA AND JOEL ZINN ABSTRACT A Bernsten-tye exonental nequalty for (generalzed) canoncal U-statstcs of order s obtaned and the
More informationLecture 4: November 17, Part 1 Single Buffer Management
Lecturer: Ad Rosén Algorthms for the anagement of Networs Fall 2003-2004 Lecture 4: November 7, 2003 Scrbe: Guy Grebla Part Sngle Buffer anagement In the prevous lecture we taled about the Combned Input
More informationFirst Year Examination Department of Statistics, University of Florida
Frst Year Examnaton Department of Statstcs, Unversty of Florda May 7, 010, 8:00 am - 1:00 noon Instructons: 1. You have four hours to answer questons n ths examnaton.. You must show your work to receve
More information11 Tail Inequalities Markov s Inequality. Lecture 11: Tail Inequalities [Fa 13]
Algorthms Lecture 11: Tal Inequaltes [Fa 13] If you hold a cat by the tal you learn thngs you cannot learn any other way. Mark Twan 11 Tal Inequaltes The smple recursve structure of skp lsts made t relatvely
More informationComplex Numbers, Signals, and Circuits
Complex Numbers, Sgnals, and Crcuts 3 August, 009 Complex Numbers: a Revew Suppose we have a complex number z = x jy. To convert to polar form, we need to know the magntude of z and the phase of z. z =
More informationA Mathematical Theory of Communication. Claude Shannon s paper presented by Kate Jenkins 2/19/00
A Mathematcal Theory of Communcaton Claude hannon s aer resented by Kate Jenkns 2/19/00 Publshed n two arts, July 1948 and October 1948 n the Bell ystem Techncal Journal Foundng aer of Informaton Theory
More informationFinding Dense Subgraphs in G(n, 1/2)
Fndng Dense Subgraphs n Gn, 1/ Atsh Das Sarma 1, Amt Deshpande, and Rav Kannan 1 Georga Insttute of Technology,atsh@cc.gatech.edu Mcrosoft Research-Bangalore,amtdesh,annan@mcrosoft.com Abstract. Fndng
More informationSELECTED SOLUTIONS, SECTION (Weak duality) Prove that the primal and dual values p and d defined by equations (4.3.2) and (4.3.3) satisfy p d.
SELECTED SOLUTIONS, SECTION 4.3 1. Weak dualty Prove that the prmal and dual values p and d defned by equatons 4.3. and 4.3.3 satsfy p d. We consder an optmzaton problem of the form The Lagrangan for ths
More informationFoundations of Arithmetic
Foundatons of Arthmetc Notaton We shall denote the sum and product of numbers n the usual notaton as a 2 + a 2 + a 3 + + a = a, a 1 a 2 a 3 a = a The notaton a b means a dvdes b,.e. ac = b where c s an
More informationVapnik-Chervonenkis theory
Vapnk-Chervonenks theory Rs Kondor June 13, 2008 For the purposes of ths lecture, we restrct ourselves to the bnary supervsed batch learnng settng. We assume that we have an nput space X, and an unknown
More informationMatching Dyadic Distributions to Channels
Matchng Dyadc Dstrbutons to Channels G. Böcherer and R. Mathar Insttute for Theoretcal Informaton Technology RWTH Aachen Unversty, 5256 Aachen, Germany Emal: {boecherer,mathar}@t.rwth-aachen.de Abstract
More informationLecture 4: Proof of Shannon s theorem and an explicit code
CSE 533: Error-Correcting Codes (Autumn 006 Lecture 4: Proof of Shannon s theorem and an explicit code October 11, 006 Lecturer: Venkatesan Guruswami Scribe: Atri Rudra 1 Overview Last lecture we stated
More informationn α j x j = 0 j=1 has a nontrivial solution. Here A is the n k matrix whose jth column is the vector for all t j=0
MODULE 2 Topcs: Lnear ndependence, bass and dmenson We have seen that f n a set of vectors one vector s a lnear combnaton of the remanng vectors n the set then the span of the set s unchanged f that vector
More information8.4 COMPLEX VECTOR SPACES AND INNER PRODUCTS
SECTION 8.4 COMPLEX VECTOR SPACES AND INNER PRODUCTS 493 8.4 COMPLEX VECTOR SPACES AND INNER PRODUCTS All the vector spaces you have studed thus far n the text are real vector spaces because the scalars
More informationDimensionality Reduction Notes 1
Dmensonalty Reducton Notes 1 Jelan Nelson mnlek@seas.harvard.edu August 10, 2015 1 Prelmnares Here we collect some notaton and basc lemmas used throughout ths note. Throughout, for a random varable X,
More informationINTEGRAL p-adic HODGE THEORY, TALK 14 (COMPARISON WITH THE DE RHAMWITT COMPLEX)
INTEGRAL -ADIC HODGE THEORY, TALK 4 (COMPARISON WITH THE DE RHAMWITT COMPLEX) JOAQUIN RODRIGUES JACINTO (NOTES BY JAMES NEWTON). Recollectons and statement of theorem Let K be a erfectod eld of characterstc
More informationDeterminants Containing Powers of Generalized Fibonacci Numbers
1 2 3 47 6 23 11 Journal of Integer Sequences, Vol 19 (2016), Artcle 1671 Determnants Contanng Powers of Generalzed Fbonacc Numbers Aram Tangboonduangjt and Thotsaporn Thanatpanonda Mahdol Unversty Internatonal
More information6.842 Randomness and Computation February 18, Lecture 4
6.842 Randomness and Computaton February 18, 2014 Lecture 4 Lecturer: Rontt Rubnfeld Scrbe: Amartya Shankha Bswas Topcs 2-Pont Samplng Interactve Proofs Publc cons vs Prvate cons 1 Two Pont Samplng 1.1
More informationLecture Notes on Linear Regression
Lecture Notes on Lnear Regresson Feng L fl@sdueducn Shandong Unversty, Chna Lnear Regresson Problem In regresson problem, we am at predct a contnuous target value gven an nput feature vector We assume
More informationLecture 19. Endogenous Regressors and Instrumental Variables
Lecture 19. Endogenous Regressors and Instrumental Varables In the prevous lecture we consder a regresson model (I omt the subscrpts (1) Y β + D + u = 1 β The problem s that the dummy varable D s endogenous,.e.
More informationatri/courses/coding-theory/book/
Foreword This chapter is based on lecture notes from coding theory courses taught by Venkatesan Guruswami at University at Washington and CMU; by Atri Rudra at University at Buffalo, SUNY and by Madhu
More informationMatrix Approximation via Sampling, Subspace Embedding. 1 Solving Linear Systems Using SVD
Matrx Approxmaton va Samplng, Subspace Embeddng Lecturer: Anup Rao Scrbe: Rashth Sharma, Peng Zhang 0/01/016 1 Solvng Lnear Systems Usng SVD Two applcatons of SVD have been covered so far. Today we loo
More informationMore metrics on cartesian products
More metrcs on cartesan products If (X, d ) are metrc spaces for 1 n, then n Secton II4 of the lecture notes we defned three metrcs on X whose underlyng topologes are the product topology The purpose of
More informationProblem Set 9 Solutions
Desgn and Analyss of Algorthms May 4, 2015 Massachusetts Insttute of Technology 6.046J/18.410J Profs. Erk Demane, Srn Devadas, and Nancy Lynch Problem Set 9 Solutons Problem Set 9 Solutons Ths problem
More informationLecture 14: Bandits with Budget Constraints
IEOR 8100-001: Learnng and Optmzaton for Sequental Decson Makng 03/07/16 Lecture 14: andts wth udget Constrants Instructor: Shpra Agrawal Scrbed by: Zhpeng Lu 1 Problem defnton In the regular Mult-armed
More informationAPPENDIX A Some Linear Algebra
APPENDIX A Some Lnear Algebra The collecton of m, n matrces A.1 Matrces a 1,1,..., a 1,n A = a m,1,..., a m,n wth real elements a,j s denoted by R m,n. If n = 1 then A s called a column vector. Smlarly,
More informationNumerical Algorithms for Visual Computing 2008/09 Example Solutions for Assignment 4. Problem 1 (Shift invariance of the Laplace operator)
Numercal Algorthms for Vsual Computng 008/09 Example Solutons for Assgnment 4 Problem (Shft nvarance of the Laplace operator The Laplace equaton s shft nvarant,.e., nvarant under translatons x x + a, y
More informationDECOMPOSITION OF SPACES OF DISTRIBUTIONS INDUCED BY HERMITE EXPANSIONS
DECOMPOSITION OF SPACES OF DISTRIBUTIONS INDUCED BY HERMITE EXPANSIONS PENCHO PETRUSHEV AND YUAN XU Abstract. Decomoston systems wth radly decayng elements needlets) based on Hermte functons are ntroduced
More information