Lecture 14: Graph Entropy
|
|
- Charles Adams
- 5 years ago
- Views:
Transcription
1 15-859: Iformatio Theory ad Applicatios i TCS Sprig 2013 Lecture 14: Graph Etropy March 19, 2013 Lecturer: Mahdi Cheraghchi Scribe: Euiwoog Lee 1 Recap Bergma s boud o the permaet Shearer s Lemma Number of triagles i a grpah with l edges. 2 Motivatio ad Defiitio of Graph Etropy So far i this course, we have leared two aspects to codig theory - source codig ad chael codig. Graph etropy ca be thought as a combiatorial extesio of source codig. Suppose that we are give a source which emits oe symbol x V. The source codig theorem says that if symbols are i.i.d. ad the umber of symbols is large, it is possible to achieve Rate H(X) ad this is the best to hope for. This result is based o the requiremet that wheever we have two sequeces of symbols (x 1,..., x t ) ad (y 1,.., y t ), which are differet i at least oe symbol, the ecoder should assig differet codewords for them; otherwise at least oe of them caot be recovered. What does happe if we relax this strict requiremet ad allow some cofusio (i.e. it is okay to use the same codeword for certai pairs of strigs)? As the requiremet is relaxed, we might hope for a better rate. The graph etropy studies this questio by represetig such requiremets by graphs symbol Case We still have a source that emits a symbol i V, ad a graph G = (V, E) such that {a, b} E if a ad b must be distiguished. This graph represets the requiremet that for ay ecoder Ec : V {0, 1} R, {a, b} E : Ec(a) Ec(b) How small R ca be i this settig? This settig is exactly equal to the well-studied graph (vertex) colorig problem, where the goal is to color each vertex so that o edge has both edpoits with the same color (each color correspods to a codeword). Let χ(g) be the miimum umber of colors eeded for G. The best R = log χ(g). If G = K, which meas every symbol must be distiguished, χ(g) = ad R OP T = log. 1
2 2.2 Multi-symbol Case We ow assume that the source emits t i.i.d. symbols, each accordig to distributio p o V. Defiitio 2.1. (x 1,..., x t ) is distiguishable from (y 1,..., y t ) if i [t] such that (x i, y i ) E. Let G t = (V t, E t ) where V t = {(v 1,..., v t ) : v i V } {(v 1,..., v t ), (w 1,..., w t )} E if ad oly if i such that {v i, w i } E. We ca see (v 1,..., v t ) ad (w 1,..., w t ) are distiguishable whe {(v 1,..., v t ), (w 1,..., w t )} E t. Let p t (v 1,..., v t ) = Π i [t] p(v i ) be the probability of (v 1,..., v t ). As i the origial source codig theorem, we might decide to igore small fractio of vertices accordig to this distributio ad color the rest of the graph with a small umber of colors. Asymptotically, we take t ad allow a error parameter ɛ. If ɛ = 0 (i.e. error-free code), the best achievable rate is log χ(g t ) lim t t If ɛ > 0, we defie etropy of G as the best achievable rate allowig ɛ error, amely H(G, p) = lim mi t U V t p t (U) 1 ɛ where G t (U) is the subgraph of G t iduced by U. proved that 1. Limit exists 2. Limit is idepedet of ɛ (0, 1). log χ(g t (U)) t Körer, who itroduced this defiitio, 3. H(G, p) = mi (X,Y ) I(X; Y ) where X V is a radom vertex whose margial distributio is p, ad Y V is a radom idepedet set of vertices such that X Y always. Y is a idepedet set if for all v, v Y, {v, v } / E. Note that 3 implies 1 ad 2. Oe rough ituitio is that ay colorig of G partitios V ito idepedet sets, ad as we use a fewer umber of colors, the size of each idepedet set will be larger. This colorig aturally defies the joit distributio (X, Y ) - pick X V accordig to p, ad let Y be the set of vertices with the same color with X. I(X; Y ) = H(X) H(X Y ) also gets smaller as the size of Y icreases, so this roughly explais how colorig is related to a I(X; Y ). 3 Examples of Graph Etropy From ow o, p is the uiform distributio o V. I this case defie H(G) to be H(G, uiform). To prove a upper boud o H(G), it is eough to fid a joit distributio (X, Y ) such that I(X; Y ) is small. 2
3 3.1 Empty Graph I a graph with o edge, Y ca be V always regardless of X. H(G) I(X; Y ) H(Y ) = 0 Sice H(G) 0 by defiitio, H(G) = Complete Graph I a complete graph K, give X, Y has to be {X} sice it is the oly set that cotais X ad is idepedet. This uique distributio gives H(G) = I(X; Y ) = H(X) H(X Y ) = H(X) = log. 3.3 Bipartite ad r-partite Graph Suppose we have a complete bipartite graph K m, with partitios A ad B such that A = m, B =. Give X, we take Y = A if x A, ad Y = B if x B. Usig this joit distributio, H(G) I(X; Y ) = H(X) H(X Y ) = log(m + ) m m + log m log = h( m + m + ) where h is the biary etropy fuctio. O the other had, for ay joit distributio (X, Y ), we see that Y A if X A, ad Y B if X B. Therefore, H(X Y ) P r[x A] log A + P r[x B] log B = This shows that H(G) h( m+ ), ad therefore H(G) = h( m m + log m + m + log m+ ) Geerally, if we have r-partite graph where V = [] [r] ad E = {(i, j), (k, l) : j l}, followig the same argumet, we ca coclude that H(G) = log r. The bipartite graph with m = is a special case with H(G) = h( 1 2 ) = log 2 = 1. 4 Properties of Graph Etropy 4.1 Subadditivity Lemma 4.1. Let G 1 = (V, E 1 ), G 2 = (V, E 2 ) ad G = (V, E 1 E 2 ). The H(G) H(G 1 )+H(G 2 ). Proof. Take joit distributio (X, Y 1, Y 2 ) such that H(G 1 ) = I(X; Y 1 ) H(G 2 ) = I(X; Y 2 ) Coditioed o X, Y 1 ad Y 2 are idepedet. 3
4 Y 1 Y 2 is idepedet o G, ad it cotais X. Therefore, (X, Y 1 Y 2 ) is a valid distributio for G. H(G) I(X; Y 1 Y 2 ) I(X; Y 1, Y 2 ) (follows from data processig iequality) = H(Y 1, Y 2 ) H(Y 1, Y 2 X) = H(Y 1, Y 2 ) H(Y 1 X) H(Y 2 X) (Y 1 Y 2 coditioed o X) H(Y 1 ) + H(Y 2 ) H(Y 1 X) H(Y 2 X) = H(G 1 ) + H(G 2 ) 4.2 Mootoicity Lemma 4.2. Let G = (V, E), F = (V, E ), E E. The H(G) H(F ). Proof. Sice G has fewer edges (less strict requiremets) tha F, (X, Y ) achievig H(F ) is feasible for H(G). 4.3 Disjoit Uio Lemma 4.3. Let G 1,..., G t are coected compoets of G ad ρ i := V (G i) V (G). The H(G) = i [k] ρ i H(G i ) Proof. First we show that H(G) ρ i H(G i ). Take a joit distributio (X, Y ) such that H(G) = I(X; Y ), ad let Y i = Y V (G i ). Defie l(x) : V (G) [k] such that l(x) = i iff x V (G i ). H(G) = I(X; Y 1,..., Y k ) = I(X, l(x); Y 1,..., Y k ) (X determies (X, l(x))) = I(l(X); Y 1,..., Y k ) + I(X; Y 1,..., Y k l(x)) (Chai rule) P r[l(x) = i]i(x; Y 1,..., Y k l(x) = i) (Expad oly the secod term) i [k] = i [k] ρ i (I(X; Y i l(x) = i) + I(X; Y 1,..., Y i 1, Y i+1,..., Y k l(x) = i, Y i )) (Chai rule) i [k] ρ i I(X; Y i l(x) = i) (Igore the secod term) i [k] ρ i H(G i ) (Defiitio of H(G i )) which completes the proof that H(G) ρ i H(G i ). For the other directio, let p i be a joit distributio (X, Y i ) that achieves H(G i ) = I(X; Y i ). We defie a joit distributio (X, Y ) such that 4
5 1. Pick Y 1,..., Y k idepedetly accordig to p 1,..., p k. 2. Pick i [k] with probability ρ i. 3. Sample X accordig to p i (X Y i ). We wat to show that I(X; Y ) = ρ i H(G i ). We are goig to use the same proof; we oly eed to check that the three ieqaulities above ideed hold as equalities. 1. We chose i = l(x) idepedetly from Y 1,..., Y k ; so I(l(X); Y 1,..., Y k ) = 0 ad the first iequality holds with equality. 2. Our choice of X oly depeds o i ad Y i, so I(X; Y 1,..., Y i 1, Y i+1,..., Y k l(x) = i, Y i ) = 0 ad the secod ieqaulity holds with equalty. 3. By the choice of p i, I(X; Y i ) = H(G i ) for each i. Therefore, H(G) I(X; Y ) = ρ i H(G i ). With the lower boud above, we ca coclude that H(G) = ρ i H(G i ). 5
Lecture 6: Source coding, Typicality, and Noisy channels and capacity
15-859: Iformatio Theory ad Applicatios i TCS CMU: Sprig 2013 Lecture 6: Source codig, Typicality, ad Noisy chaels ad capacity Jauary 31, 2013 Lecturer: Mahdi Cheraghchi Scribe: Togbo Huag 1 Recap Uiversal
More informationLecture 16: Monotone Formula Lower Bounds via Graph Entropy. 2 Monotone Formula Lower Bounds via Graph Entropy
15-859: Iformatio Theory ad Applicatios i TCS CMU: Sprig 2013 Lecture 16: Mootoe Formula Lower Bouds via Graph Etropy March 26, 2013 Lecturer: Mahdi Cheraghchi Scribe: Shashak Sigh 1 Recap Graph Etropy:
More informationLecture 2. The Lovász Local Lemma
Staford Uiversity Sprig 208 Math 233A: No-costructive methods i combiatorics Istructor: Ja Vodrák Lecture date: Jauary 0, 208 Origial scribe: Apoorva Khare Lecture 2. The Lovász Local Lemma 2. Itroductio
More informationLecture 15: Strong, Conditional, & Joint Typicality
EE376A/STATS376A Iformatio Theory Lecture 15-02/27/2018 Lecture 15: Strog, Coditioal, & Joit Typicality Lecturer: Tsachy Weissma Scribe: Nimit Sohoi, William McCloskey, Halwest Mohammad I this lecture,
More informationNotes for Lecture 11
U.C. Berkeley CS78: Computatioal Complexity Hadout N Professor Luca Trevisa 3/4/008 Notes for Lecture Eigevalues, Expasio, ad Radom Walks As usual by ow, let G = (V, E) be a udirected d-regular graph with
More information1 Review and Overview
DRAFT a fial versio will be posted shortly CS229T/STATS231: Statistical Learig Theory Lecturer: Tegyu Ma Lecture #3 Scribe: Migda Qiao October 1, 2013 1 Review ad Overview I the first half of this course,
More informationSequences and Series of Functions
Chapter 6 Sequeces ad Series of Fuctios 6.1. Covergece of a Sequece of Fuctios Poitwise Covergece. Defiitio 6.1. Let, for each N, fuctio f : A R be defied. If, for each x A, the sequece (f (x)) coverges
More informationEntropy and Ergodic Theory Lecture 5: Joint typicality and conditional AEP
Etropy ad Ergodic Theory Lecture 5: Joit typicality ad coditioal AEP 1 Notatio: from RVs back to distributios Let (Ω, F, P) be a probability space, ad let X ad Y be A- ad B-valued discrete RVs, respectively.
More informationEntropy Rates and Asymptotic Equipartition
Chapter 29 Etropy Rates ad Asymptotic Equipartitio Sectio 29. itroduces the etropy rate the asymptotic etropy per time-step of a stochastic process ad shows that it is well-defied; ad similarly for iformatio,
More informationEmpirical Process Theory and Oracle Inequalities
Stat 928: Statistical Learig Theory Lecture: 10 Empirical Process Theory ad Oracle Iequalities Istructor: Sham Kakade 1 Risk vs Risk See Lecture 0 for a discussio o termiology. 2 The Uio Boud / Boferoi
More informationProblem Set 2 Solutions
CS271 Radomess & Computatio, Sprig 2018 Problem Set 2 Solutios Poit totals are i the margi; the maximum total umber of poits was 52. 1. Probabilistic method for domiatig sets 6pts Pick a radom subset S
More informationHOMEWORK 2 SOLUTIONS
HOMEWORK SOLUTIONS CSE 55 RANDOMIZED AND APPROXIMATION ALGORITHMS 1. Questio 1. a) The larger the value of k is, the smaller the expected umber of days util we get all the coupos we eed. I fact if = k
More informationEntropies & Information Theory
Etropies & Iformatio Theory LECTURE I Nilajaa Datta Uiversity of Cambridge,U.K. For more details: see lecture otes (Lecture 1- Lecture 5) o http://www.qi.damtp.cam.ac.uk/ode/223 Quatum Iformatio Theory
More informationLecture 11: Channel Coding Theorem: Converse Part
EE376A/STATS376A Iformatio Theory Lecture - 02/3/208 Lecture : Chael Codig Theorem: Coverse Part Lecturer: Tsachy Weissma Scribe: Erdem Bıyık I this lecture, we will cotiue our discussio o chael codig
More informationThe Borel hierarchy classifies subsets of the reals by their topological complexity. Another approach is to classify them by size.
Lecture 7: Measure ad Category The Borel hierarchy classifies subsets of the reals by their topological complexity. Aother approach is to classify them by size. Filters ad Ideals The most commo measure
More informationCS284A: Representations and Algorithms in Molecular Biology
CS284A: Represetatios ad Algorithms i Molecular Biology Scribe Notes o Lectures 3 & 4: Motif Discovery via Eumeratio & Motif Represetatio Usig Positio Weight Matrix Joshua Gervi Based o presetatios by
More informationLecture 9: Hierarchy Theorems
IAS/PCMI Summer Sessio 2000 Clay Mathematics Udergraduate Program Basic Course o Computatioal Complexity Lecture 9: Hierarchy Theorems David Mix Barrigto ad Alexis Maciel July 27, 2000 Most of this lecture
More informationShannon s noiseless coding theorem
18.310 lecture otes May 4, 2015 Shao s oiseless codig theorem Lecturer: Michel Goemas I these otes we discuss Shao s oiseless codig theorem, which is oe of the foudig results of the field of iformatio
More informationACO Comprehensive Exam 9 October 2007 Student code A. 1. Graph Theory
1. Graph Theory Prove that there exist o simple plaar triagulatio T ad two distict adjacet vertices x, y V (T ) such that x ad y are the oly vertices of T of odd degree. Do ot use the Four-Color Theorem.
More informationAdvanced Stochastic Processes.
Advaced Stochastic Processes. David Gamarik LECTURE 2 Radom variables ad measurable fuctios. Strog Law of Large Numbers (SLLN). Scary stuff cotiued... Outlie of Lecture Radom variables ad measurable fuctios.
More informationLecture 9: Expanders Part 2, Extractors
Lecture 9: Expaders Part, Extractors Topics i Complexity Theory ad Pseudoradomess Sprig 013 Rutgers Uiversity Swastik Kopparty Scribes: Jaso Perry, Joh Kim I this lecture, we will discuss further the pseudoradomess
More information62. Power series Definition 16. (Power series) Given a sequence {c n }, the series. c n x n = c 0 + c 1 x + c 2 x 2 + c 3 x 3 +
62. Power series Defiitio 16. (Power series) Give a sequece {c }, the series c x = c 0 + c 1 x + c 2 x 2 + c 3 x 3 + is called a power series i the variable x. The umbers c are called the coefficiets of
More informationMA131 - Analysis 1. Workbook 3 Sequences II
MA3 - Aalysis Workbook 3 Sequeces II Autum 2004 Cotets 2.8 Coverget Sequeces........................ 2.9 Algebra of Limits......................... 2 2.0 Further Useful Results........................
More informationLecture 7: Channel coding theorem for discrete-time continuous memoryless channel
Lecture 7: Chael codig theorem for discrete-time cotiuous memoryless chael Lectured by Dr. Saif K. Mohammed Scribed by Mirsad Čirkić Iformatio Theory for Wireless Commuicatio ITWC Sprig 202 Let us first
More informationMath 216A Notes, Week 5
Math 6A Notes, Week 5 Scribe: Ayastassia Sebolt Disclaimer: These otes are ot early as polished (ad quite possibly ot early as correct) as a published paper. Please use them at your ow risk.. Thresholds
More informationApplication to Random Graphs
A Applicatio to Radom Graphs Brachig processes have a umber of iterestig ad importat applicatios. We shall cosider oe of the most famous of them, the Erdős-Réyi radom graph theory. 1 Defiitio A.1. Let
More informationUniversal source coding for complementary delivery
SITA2006 i Hakodate 2005.2. p. Uiversal source codig for complemetary delivery Akisato Kimura, 2, Tomohiko Uyematsu 2, Shigeaki Kuzuoka 2 Media Iformatio Laboratory, NTT Commuicatio Sciece Laboratories,
More informationSeunghee Ye Ma 8: Week 5 Oct 28
Week 5 Summary I Sectio, we go over the Mea Value Theorem ad its applicatios. I Sectio 2, we will recap what we have covered so far this term. Topics Page Mea Value Theorem. Applicatios of the Mea Value
More informationEECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1
EECS564 Estimatio, Filterig, ad Detectio Hwk 2 Sols. Witer 25 4. Let Z be a sigle observatio havig desity fuctio where. p (z) = (2z + ), z (a) Assumig that is a oradom parameter, fid ad plot the maximum
More informationAsymptotic Coupling and Its Applications in Information Theory
Asymptotic Couplig ad Its Applicatios i Iformatio Theory Vicet Y. F. Ta Joit Work with Lei Yu Departmet of Electrical ad Computer Egieerig, Departmet of Mathematics, Natioal Uiversity of Sigapore IMS-APRM
More informationLet us give one more example of MLE. Example 3. The uniform distribution U[0, θ] on the interval [0, θ] has p.d.f.
Lecture 5 Let us give oe more example of MLE. Example 3. The uiform distributio U[0, ] o the iterval [0, ] has p.d.f. { 1 f(x =, 0 x, 0, otherwise The likelihood fuctio ϕ( = f(x i = 1 I(X 1,..., X [0,
More informationChapter 3. Strong convergence. 3.1 Definition of almost sure convergence
Chapter 3 Strog covergece As poited out i the Chapter 2, there are multiple ways to defie the otio of covergece of a sequece of radom variables. That chapter defied covergece i probability, covergece i
More informationIf a subset E of R contains no open interval, is it of zero measure? For instance, is the set of irrationals in [0, 1] is of measure zero?
2 Lebesgue Measure I Chapter 1 we defied the cocept of a set of measure zero, ad we have observed that every coutable set is of measure zero. Here are some atural questios: If a subset E of R cotais a
More informationIt is always the case that unions, intersections, complements, and set differences are preserved by the inverse image of a function.
MATH 532 Measurable Fuctios Dr. Neal, WKU Throughout, let ( X, F, µ) be a measure space ad let (!, F, P ) deote the special case of a probability space. We shall ow begi to study real-valued fuctios defied
More informationDisjoint Systems. Abstract
Disjoit Systems Noga Alo ad Bey Sudaov Departmet of Mathematics Raymod ad Beverly Sacler Faculty of Exact Scieces Tel Aviv Uiversity, Tel Aviv, Israel Abstract A disjoit system of type (,,, ) is a collectio
More informationLecture Notes for Analysis Class
Lecture Notes for Aalysis Class Topological Spaces A topology for a set X is a collectio T of subsets of X such that: (a) X ad the empty set are i T (b) Uios of elemets of T are i T (c) Fiite itersectios
More information# fixed points of g. Tree to string. Repeatedly select the leaf with the smallest label, write down the label of its neighbour and remove the leaf.
Combiatorics Graph Theory Coutig labelled ad ulabelled graphs There are 2 ( 2) labelled graphs of order. The ulabelled graphs of order correspod to orbits of the actio of S o the set of labelled graphs.
More informationLecture 12: November 13, 2018
Mathematical Toolkit Autum 2018 Lecturer: Madhur Tulsiai Lecture 12: November 13, 2018 1 Radomized polyomial idetity testig We will use our kowledge of coditioal probability to prove the followig lemma,
More informationPRACTICE PROBLEMS FOR THE FINAL
PRACTICE PROBLEMS FOR THE FINAL Math 36Q Fall 25 Professor Hoh Below is a list of practice questios for the Fial Exam. I would suggest also goig over the practice problems ad exams for Exam ad Exam 2 to
More informationÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE
ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE School of Computer ad Commuicatio Scieces Hadout Iformatio Theory ad Sigal Processig Compressio ad Quatizatio November 0, 207 Data compressio Notatio Give a set
More informationLecture 7: October 18, 2017
Iformatio ad Codig Theory Autum 207 Lecturer: Madhur Tulsiai Lecture 7: October 8, 207 Biary hypothesis testig I this lecture, we apply the tools developed i the past few lectures to uderstad the problem
More informationIncreasing timing capacity using packet coloring
003 Coferece o Iformatio Scieces ad Systems, The Johs Hopkis Uiversity, March 4, 003 Icreasig timig capacity usig packet colorig Xi Liu ad R Srikat[] Coordiated Sciece Laboratory Uiversity of Illiois e-mail:
More informationDefinition An infinite sequence of numbers is an ordered set of real numbers.
Ifiite sequeces (Sect. 0. Today s Lecture: Review: Ifiite sequeces. The Cotiuous Fuctio Theorem for sequeces. Usig L Hôpital s rule o sequeces. Table of useful its. Bouded ad mootoic sequeces. Previous
More informationChapter 6 Infinite Series
Chapter 6 Ifiite Series I the previous chapter we cosidered itegrals which were improper i the sese that the iterval of itegratio was ubouded. I this chapter we are goig to discuss a topic which is somewhat
More informationLecture 2: April 3, 2013
TTIC/CMSC 350 Mathematical Toolkit Sprig 203 Madhur Tulsiai Lecture 2: April 3, 203 Scribe: Shubhedu Trivedi Coi tosses cotiued We retur to the coi tossig example from the last lecture agai: Example. Give,
More information6.895 Essential Coding Theory October 20, Lecture 11. This lecture is focused in comparisons of the following properties/parameters of a code:
6.895 Essetial Codig Theory October 0, 004 Lecture 11 Lecturer: Madhu Suda Scribe: Aastasios Sidiropoulos 1 Overview This lecture is focused i comparisos of the followig properties/parameters of a code:
More informationDistribution of Random Samples & Limit theorems
STAT/MATH 395 A - PROBABILITY II UW Witer Quarter 2017 Néhémy Lim Distributio of Radom Samples & Limit theorems 1 Distributio of i.i.d. Samples Motivatig example. Assume that the goal of a study is to
More informationMath 155 (Lecture 3)
Math 55 (Lecture 3) September 8, I this lecture, we ll cosider the aswer to oe of the most basic coutig problems i combiatorics Questio How may ways are there to choose a -elemet subset of the set {,,,
More informationIntro to Learning Theory
Lecture 1, October 18, 2016 Itro to Learig Theory Ruth Urer 1 Machie Learig ad Learig Theory Comig soo 2 Formal Framework 21 Basic otios I our formal model for machie learig, the istaces to be classified
More informationLecture 2: Concentration Bounds
CSE 52: Desig ad Aalysis of Algorithms I Sprig 206 Lecture 2: Cocetratio Bouds Lecturer: Shaya Oveis Ghara March 30th Scribe: Syuzaa Sargsya Disclaimer: These otes have ot bee subjected to the usual scrutiy
More informationProblem Set 4 Due Oct, 12
EE226: Radom Processes i Systems Lecturer: Jea C. Walrad Problem Set 4 Due Oct, 12 Fall 06 GSI: Assae Gueye This problem set essetially reviews detectio theory ad hypothesis testig ad some basic otios
More information1. Universal v.s. non-universal: know the source distribution or not.
28. Radom umber geerators Let s play the followig game: Give a stream of Ber( p) bits, with ukow p, we wat to tur them ito pure radom bits, i.e., idepedet fair coi flips Ber( / 2 ). Our goal is to fid
More informationREGRESSION WITH QUADRATIC LOSS
REGRESSION WITH QUADRATIC LOSS MAXIM RAGINSKY Regressio with quadratic loss is aother basic problem studied i statistical learig theory. We have a radom couple Z = X, Y ), where, as before, X is a R d
More informationLecture 12: September 27
36-705: Itermediate Statistics Fall 207 Lecturer: Siva Balakrisha Lecture 2: September 27 Today we will discuss sufficiecy i more detail ad the begi to discuss some geeral strategies for costructig estimators.
More informationINFORMATION THEORY AND STATISTICS. Jüri Lember
INFORMATION THEORY AND STATISTICS Lecture otes ad exercises Sprig 203 Jüri Lember Literature:. T.M. Cover, J.A. Thomas "Elemets of iformatio theory", Wiley, 99 ja 2006; 2. Yeug, Raymod W. "A first course
More informationWeek 5-6: The Binomial Coefficients
Wee 5-6: The Biomial Coefficiets March 6, 2018 1 Pascal Formula Theorem 11 (Pascal s Formula For itegers ad such that 1, ( ( ( 1 1 + 1 The umbers ( 2 ( 1 2 ( 2 are triagle umbers, that is, The petago umbers
More informationLecture 4: Unique-SAT, Parity-SAT, and Approximate Counting
Advaced Complexity Theory Sprig 206 Lecture 4: Uique-SAT, Parity-SAT, ad Approximate Coutig Prof. Daa Moshkovitz Scribe: Aoymous Studet Scribe Date: Fall 202 Overview I this lecture we begi talkig about
More informationLecture 1: Basic problems of coding theory
Lecture 1: Basic problems of codig theory Error-Correctig Codes (Sprig 016) Rutgers Uiversity Swastik Kopparty Scribes: Abhishek Bhrushudi & Aditya Potukuchi Admiistrivia was discussed at the begiig of
More informationLecture 3 The Lebesgue Integral
Lecture 3: The Lebesgue Itegral 1 of 14 Course: Theory of Probability I Term: Fall 2013 Istructor: Gorda Zitkovic Lecture 3 The Lebesgue Itegral The costructio of the itegral Uless expressly specified
More informationInformation Theory and Coding
Sol. Iformatio Theory ad Codig. The capacity of a bad-limited additive white Gaussia (AWGN) chael is give by C = Wlog 2 ( + σ 2 W ) bits per secod(bps), where W is the chael badwidth, is the average power
More informationHere, e(a, B) is defined as the number of edges between A and B in the n dimensional boolean hypercube.
Lecture 2 Topics i Complexity Theory ad Pseudoradomess (Sprig 2013) Rutgers Uiversity Swastik Kopparty Scribes: Amey Bhagale, Mrial Kumar 1 Overview I this lecture, we will complete the proof of formula
More informationLecture 19: Convergence
Lecture 19: Covergece Asymptotic approach I statistical aalysis or iferece, a key to the success of fidig a good procedure is beig able to fid some momets ad/or distributios of various statistics. I may
More informationf n (x) f m (x) < ɛ/3 for all x A. By continuity of f n and f m we can find δ > 0 such that d(x, x 0 ) < δ implies that
Lecture 15 We have see that a sequece of cotiuous fuctios which is uiformly coverget produces a limit fuctio which is also cotiuous. We shall stregthe this result ow. Theorem 1 Let f : X R or (C) be a
More informationThe Growth of Functions. Theoretical Supplement
The Growth of Fuctios Theoretical Supplemet The Triagle Iequality The triagle iequality is a algebraic tool that is ofte useful i maipulatig absolute values of fuctios. The triagle iequality says that
More information4 The Sperner property.
4 The Sperer property. I this sectio we cosider a surprisig applicatio of certai adjacecy matrices to some problems i extremal set theory. A importat role will also be played by fiite groups. I geeral,
More informationRandomized Algorithms I, Spring 2018, Department of Computer Science, University of Helsinki Homework 1: Solutions (Discussed January 25, 2018)
Radomized Algorithms I, Sprig 08, Departmet of Computer Sciece, Uiversity of Helsiki Homework : Solutios Discussed Jauary 5, 08). Exercise.: Cosider the followig balls-ad-bi game. We start with oe black
More informationNotes 19 : Martingale CLT
Notes 9 : Martigale CLT Math 733-734: Theory of Probability Lecturer: Sebastie Roch Refereces: [Bil95, Chapter 35], [Roc, Chapter 3]. Sice we have ot ecoutered weak covergece i some time, we first recall
More informationLarge holes in quasi-random graphs
Large holes i quasi-radom graphs Joaa Polcy Departmet of Discrete Mathematics Adam Mickiewicz Uiversity Pozań, Polad joaska@amuedupl Submitted: Nov 23, 2006; Accepted: Apr 10, 2008; Published: Apr 18,
More informationFall 2013 MTH431/531 Real analysis Section Notes
Fall 013 MTH431/531 Real aalysis Sectio 8.1-8. Notes Yi Su 013.11.1 1. Defiitio of uiform covergece. We look at a sequece of fuctios f (x) ad study the coverget property. Notice we have two parameters
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More informationOn Random Line Segments in the Unit Square
O Radom Lie Segmets i the Uit Square Thomas A. Courtade Departmet of Electrical Egieerig Uiversity of Califoria Los Ageles, Califoria 90095 Email: tacourta@ee.ucla.edu I. INTRODUCTION Let Q = [0, 1] [0,
More informationMath 220B Final Exam Solutions March 18, 2002
Math 0B Fial Exam Solutios March 18, 00 1. (1 poits) (a) (6 poits) Fid the Gree s fuctio for the tilted half-plae {(x 1, x ) R : x 1 + x > 0}. For x (x 1, x ), y (y 1, y ), express your Gree s fuctio G(x,
More informationECE 6980 An Algorithmic and Information-Theoretic Toolbox for Massive Data
ECE 6980 A Algorithmic ad Iformatio-Theoretic Toolbo for Massive Data Istructor: Jayadev Acharya Lecture # Scribe: Huayu Zhag 8th August, 017 1 Recap X =, ε is a accuracy parameter, ad δ is a error parameter.
More informationSection 5.1 The Basics of Counting
1 Sectio 5.1 The Basics of Coutig Combiatorics, the study of arragemets of objects, is a importat part of discrete mathematics. I this chapter, we will lear basic techiques of coutig which has a lot of
More informationSingular Continuous Measures by Michael Pejic 5/14/10
Sigular Cotiuous Measures by Michael Peic 5/4/0 Prelimiaries Give a set X, a σ-algebra o X is a collectio of subsets of X that cotais X ad ad is closed uder complemetatio ad coutable uios hece, coutable
More informationLecture 11: Pseudorandom functions
COM S 6830 Cryptography Oct 1, 2009 Istructor: Rafael Pass 1 Recap Lecture 11: Pseudoradom fuctios Scribe: Stefao Ermo Defiitio 1 (Ge, Ec, Dec) is a sigle message secure ecryptio scheme if for all uppt
More informationPairs of disjoint q-element subsets far from each other
Pairs of disjoit q-elemet subsets far from each other Hikoe Eomoto Departmet of Mathematics, Keio Uiversity 3-14-1 Hiyoshi, Kohoku-Ku, Yokohama, 223 Japa, eomoto@math.keio.ac.jp Gyula O.H. Katoa Alfréd
More informationConvergence of random variables. (telegram style notes) P.J.C. Spreij
Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space
More informationSequences A sequence of numbers is a function whose domain is the positive integers. We can see that the sequence
Sequeces A sequece of umbers is a fuctio whose domai is the positive itegers. We ca see that the sequece 1, 1, 2, 2, 3, 3,... is a fuctio from the positive itegers whe we write the first sequece elemet
More informationThe multiplicative structure of finite field and a construction of LRC
IERG6120 Codig for Distributed Storage Systems Lecture 8-06/10/2016 The multiplicative structure of fiite field ad a costructio of LRC Lecturer: Keeth Shum Scribe: Zhouyi Hu Notatios: We use the otatio
More informationChapter 6 Principles of Data Reduction
Chapter 6 for BST 695: Special Topics i Statistical Theory. Kui Zhag, 0 Chapter 6 Priciples of Data Reductio Sectio 6. Itroductio Goal: To summarize or reduce the data X, X,, X to get iformatio about a
More information(A sequence also can be thought of as the list of function values attained for a function f :ℵ X, where f (n) = x n for n 1.) x 1 x N +k x N +4 x 3
MATH 337 Sequeces Dr. Neal, WKU Let X be a metric space with distace fuctio d. We shall defie the geeral cocept of sequece ad limit i a metric space, the apply the results i particular to some special
More information2.1. The Algebraic and Order Properties of R Definition. A binary operation on a set F is a function B : F F! F.
CHAPTER 2 The Real Numbers 2.. The Algebraic ad Order Properties of R Defiitio. A biary operatio o a set F is a fuctio B : F F! F. For the biary operatios of + ad, we replace B(a, b) by a + b ad a b, respectively.
More informationA sequence of numbers is a function whose domain is the positive integers. We can see that the sequence
Sequeces A sequece of umbers is a fuctio whose domai is the positive itegers. We ca see that the sequece,, 2, 2, 3, 3,... is a fuctio from the positive itegers whe we write the first sequece elemet as
More informationInformation Theory Tutorial Communication over Channels with memory. Chi Zhang Department of Electrical Engineering University of Notre Dame
Iformatio Theory Tutorial Commuicatio over Chaels with memory Chi Zhag Departmet of Electrical Egieerig Uiversity of Notre Dame Abstract A geeral capacity formula C = sup I(; Y ), which is correct for
More informationCarleton College, Winter 2017 Math 121, Practice Final Prof. Jones. Note: the exam will have a section of true-false questions, like the one below.
Carleto College, Witer 207 Math 2, Practice Fial Prof. Joes Note: the exam will have a sectio of true-false questios, like the oe below.. True or False. Briefly explai your aswer. A icorrectly justified
More informationBeurling Integers: Part 2
Beurlig Itegers: Part 2 Isomorphisms Devi Platt July 11, 2015 1 Prime Factorizatio Sequeces I the last article we itroduced the Beurlig geeralized itegers, which ca be represeted as a sequece of real umbers
More informationInfinite Sequences and Series
Chapter 6 Ifiite Sequeces ad Series 6.1 Ifiite Sequeces 6.1.1 Elemetary Cocepts Simply speakig, a sequece is a ordered list of umbers writte: {a 1, a 2, a 3,...a, a +1,...} where the elemets a i represet
More informationLecture 10 October Minimaxity and least favorable prior sequences
STATS 300A: Theory of Statistics Fall 205 Lecture 0 October 22 Lecturer: Lester Mackey Scribe: Brya He, Rahul Makhijai Warig: These otes may cotai factual ad/or typographic errors. 0. Miimaxity ad least
More informationMATH 112: HOMEWORK 6 SOLUTIONS. Problem 1: Rudin, Chapter 3, Problem s k < s k < 2 + s k+1
MATH 2: HOMEWORK 6 SOLUTIONS CA PRO JIRADILOK Problem. If s = 2, ad Problem : Rudi, Chapter 3, Problem 3. s + = 2 + s ( =, 2, 3,... ), prove that {s } coverges, ad that s < 2 for =, 2, 3,.... Proof. The
More informationSquare-Congruence Modulo n
Square-Cogruece Modulo Abstract This paper is a ivestigatio of a equivalece relatio o the itegers that was itroduced as a exercise i our Discrete Math class. Part I - Itro Defiitio Two itegers are Square-Cogruet
More informationEDGE-COLORINGS AVOIDING RAINBOW STARS
EDGE-COLORINGS AVOIDING RAINBOW STARS CARLOS HOPPEN, HANNO LEFMANN, KNUT ODERMANN, AND JULIANA SANCHES Abstract. We cosider a extremal problem motivated by a paper of Balogh [J. Balogh, A remark o the
More informationn=1 a n is the sequence (s n ) n 1 n=1 a n converges to s. We write a n = s, n=1 n=1 a n
Series. Defiitios ad first properties A series is a ifiite sum a + a + a +..., deoted i short by a. The sequece of partial sums of the series a is the sequece s ) defied by s = a k = a +... + a,. k= Defiitio
More informationLecture 2 February 8, 2016
MIT 6.854/8.45: Advaced Algorithms Sprig 206 Prof. Akur Moitra Lecture 2 February 8, 206 Scribe: Calvi Huag, Lih V. Nguye I this lecture, we aalyze the problem of schedulig equal size tasks arrivig olie
More informationLecture XVI - Lifting of paths and homotopies
Lecture XVI - Liftig of paths ad homotopies I the last lecture we discussed the liftig problem ad proved that the lift if it exists is uiquely determied by its value at oe poit. I this lecture we shall
More informationInformation Theory and Statistics Lecture 4: Lempel-Ziv code
Iformatio Theory ad Statistics Lecture 4: Lempel-Ziv code Łukasz Dębowski ldebowsk@ipipa.waw.pl Ph. D. Programme 203/204 Etropy rate is the limitig compressio rate Theorem For a statioary process (X i)
More informationSTAT Homework 1 - Solutions
STAT-36700 Homework 1 - Solutios Fall 018 September 11, 018 This cotais solutios for Homework 1. Please ote that we have icluded several additioal commets ad approaches to the problems to give you better
More informationBalanced coloring of bipartite graphs
Balaced colorig of bipartite graphs Uriel Feige Shimo Koga Departmet of Computer Sciece ad Applied Mathematics Weizma Istitute, Rehovot 76100, Israel uriel.feige@weizma.ac.il Jue 16, 009 Abstract Give
More informationLecture 23: Minimal sufficiency
Lecture 23: Miimal sufficiecy Maximal reductio without loss of iformatio There are may sufficiet statistics for a give problem. I fact, X (the whole data set) is sufficiet. If T is a sufficiet statistic
More information