Growth Rate of Spatially Coupled LDPC codes
|
|
- Betty Townsend
- 5 years ago
- Views:
Transcription
1 Growth Rate of Spatially Coupled LDPC codes Workshop on Spatially Coupled Codes and Related Topics at Tokyo Institute of Technology 2011/2/19
2 Contents 1. Factor graph, Bethe approximation and belief propagation 2. Relation between annealed free energy and belief propagation 3. Growth rate of spatially coupled LDPC codes and threshold saturation phenomenon Here, growth rate is G(ω) = lim N 1 N loge[z(ω)] Z(ω): the number of codewords of relative weight ω [0,1]. 2 / 34
3 Factor graph, Bethe approximation and belief propagation 3 / 34
4 Factor graph Factor graph: bipartite graph which defines probability measure p(x) = 1 f a (x a ) Z a Z := f a (x a ), (partition function) f a (x a ) : X r a R 0 x X n a 4 / 34
5 Gibbs free energy p(x) = 1 Z f a (x a ) Approximation by simple distribution q of p which is defined by factor graph a D(q p) = x q(x)log q(x) p(x) ( ) = logz x q(x) log f a (x a ) + x q(x) log q(x) =: logz +U(q) H(q) =: logz +F Gibbs (q) a U(p): internal energy H(p): entropy F Gibbs (p): Gibbs free energy 5 / 34
6 Mean field approximation and Bethe approximation Mean field approximation q(x) = i b i (x i ) Degree of freedom is reduced from q n to nq Bethe approximation d i : degree of variable node i q(x) = ab a(x a ) i b i(x i ) d i 1 When factor graph is tree, Bethe approximation can be exact 6 / 34
7 Bethe free energy U(q) = x a ( ) q(x) log f a (x a ) a b a (x a )logf a (x a ) =: U Bethe ({b a }) x a b(x) ab a(x a ) i b i(x) d i 1 H(b) = x x = a b(x) log b(x) b(x) log ab a(x a ) i b i(x) d i 1 x a b a (x a )logb a (x a )+ =: H Bethe ({b i },{b a }) (d i 1) i i b i (x i )logb i (x i ) 7 / 34
8 Minimization of Bethe free energy F Bethe ({b i },{b a }) := U Bethe ({b a }) H Bethe ({b i },{b a }) minimize : F Bethe ({b i },{b a }) subject to : b i (x i ) 0, i b a (x a ) 0, a b i (x i ) = 1 i b a (x a ) = 1 a x a \x i b a (x a ) = b i (x i ), a, i a 8 / 34
9 Stationary point of Lagrangian of Bethe free energy [Yedidia, Freeman, and Weiss 2005] L := F Bethe ({b i },{b a })+ [ ] γ a b a (x a ) 1 + [ ] γ i b i (x) 1 a x a i x + λ ai (x i ) b i (x i ) b a (x a ) a i a x i x a \x i Stationary points of Lagrangian is fixed points of BP b a (x a ) f a (x a ) m i a (x i ) b i (x i ) i a i a m a i (x i ) where m i a (x i ) m c i (x i ) c i\a m a i (x i ) x a \x i f a (x a ) m j a (x j ) j a\i 9 / 34
10 Relation between annealed free energy and belief propagation 10 / 34
11 Random regular factor graph ensemble Factor graph: bipartite graph which defines probability measure µ(x) = 1 f a (x a ) Z a Z := f a (x a ), x X n a (partition function) Random (l, r)-regular factor graph ensemble: l: degree of variable nodes, r: degree of factor nodes Random ensemble of factor graphs 11 / 34
12 Annealed free energy Factor graph: bipartite graph which defines probability measure µ(x) = 1 f a (x a ) Z a Z := f a (x a ), x X n a (partition function) Random (l, r)-regular factor graph ensemble: l: degree of variable nodes, r: degree of factor nodes Random ensemble of factor graphs (Quenched) free energy: lim N 1 N E[logZ] Annealed free energy: lim N 1 N loge[z] 12 / 34
13 Contribution to partition function of particular types {v x } x X : the number of variable nodes of value x X is v x {u x } r: x X the number of factor nodes of value x Xr is u x Z = f(x a ) x X N = E[N({v},{u})] = {v},{u} a N({v},{u}) x X r f(x) ux. ( )( N l N ) r x X (v xl)!. {v x } x X {u x } x X r (Nl)! lim N 1 N loge[z({ν},{µ})] = l r H({µ}) (l 1)H({ν})+ l r x X r µ(x)logf(x). 13 / 34
14 Annealed free energy of fixed type and Bethe free energy F Bethe ({b i },{b a }) = b a (x a )logf a (x a ) a x a + b a (x a )logb a (x a ) (d i 1) a x a i i b i (x i )logb i (x i ) lim N 1 N loge[z({ν},{µ})] = l µ(x)logf(x)+ l r r H({µ}) (l 1)H({ν}). x X r 14 / 34
15 Maximization of the exponents of contributions maximize : l r H({µ}) (l 1)H({ν})+ l r subject to : ν(x) 0, x X µ(x) 0, x X r ν(x) = 1 x X µ(x) = 1 x X r 1 r µ(x) = ν(z), z X r k=1 x\x k x k =z x X r µ(x)logf(x) 15 / 34
16 The stationary condition The stationary condition is where ν(x) m f v (x) l µ(x) f(x) r m v f (x i ) i=1 m v f (x) m f v (x) l 1 r m f v (x) f(x) m v f (x j ). j k k=1 x\x k x k =x If f(x) is invariant under any permutation of x X r m f v (x) x\x 1 x 1 =x 16 / 34 f(x) j 1m v f (x j ).
17 Annealed free energy Theorem 1. lim N 1 N loge[z] = max (m f v,m v f ) S where S denotes the set of saddle points, and where { } l r logz f +logz v l logz fv. Z v := x Z f := x Z fv := x m f v (x) l r f(x) m v f (x i ) i=1 m f v (x)m v f (x). 17 / 34
18 Number of solutions If r k=1 x\x k x k =x f(x) is constant among all x X, the uniform message m f v (x) m v f (x) is a saddle point. The contribution of the uniform message is lim N 1 N loge[z(ν,µ)] = logq + l r log ( Nf q r ) (design rate) where q := X N f := x f(x). For CSP i.e., f(x) {0,1}, the expected number of solutions is about q N ( Nf q r ) l r N. This intuitively means all constraints are independent. 18 / 34
19 Contribution to partition function of fixed variable type Z({ν}) := {µ} Z({ν},{µ}) lim N 1 N loge[z({ν})] { = sup {µ} l r H({µ}) (l 1)H({ν})+ l r x X r µ(x)logf(x) } where {µ} satisfies 1 r µ(x) 0, x X r µ(x) = 1 x X r r µ(x) = ν(z), z X k=1 x\x k x k =z Convex optimization problem with linear constraints. 19 / 34
20 The stationary condition The stationary condition is µ(x) f(x) r i=1 m v f (x i ) where ν(x) h(x)m f v (x) l m v f (x) h(x)m f v (x) l 1 r m f v (x) f(x) m v f (x j ). j k k=1 x\x k x k =x If f(x) is invariant under any permutation of x X r m f v (x) x\x 1 x 1 =x 20 / 34 f(x) j 1m v f (x j ).
21 Growth rate of contribution to partition function of fixed variable type Theorem 2. lim N 1 N loge[z({ν})] { = max (m f v,m v f ) S l r logz f +logz v l logz fv x ν(x) log h(x) } where S denotes the set of saddle points, and where Z v := x Z f := x Z fv := x h(x)m f v (x) l r f(x) m v f (x i ) i=1 m f v (x)m v f (x). 21 / 34
22 Growth rate of regular LDPC codes G(ω) = l r log 1+zr 2 +log [ e h ( 1+y 2 where ω := 1 2ω and ω = tanh(h+l tanh 1 (y)) y = z r 1 ) l +e h ( 1 y z = tanh(h+(l 1)tanh 1 (y)). 2 ) l ] l log 1+yz 2 ω h This result can be easily understood from correspondings ω = ν(0) ν(1) y = m f v (0) m f v (1) z = m v f (0) m v f (1) h(x) = e ( 1)x h 22 / 34
23 Growth rate of regular LDPC codes (5,10) (6,10) (7,10) (8,10) (9,10) (10,10) (11,10) G w 23 / 34
24 Growth rate of binary CSP k=1 k=2 k= G w 24 / 34
25 Growth rate of (3,2)-regular-3-coloring / 34
26 Replica theory This story continues into the replica theory (see the paper in arxiv). But, we don t deal with it here. E[logZ] = loge[zn ] n n=0 lim N 1 E[logZ] = lim N N 1 N lim n 0? 1 = lim n 0 n lim N loge[z n ] n 1 N loge[zn ] The replica method is methematically not rigorous e.g., exchange of limits, analytic continuation of n. 26 / 34
27 Growth rate of spatially coupled LDPC codes and threshold saturation phenomenon 27 / 34
28 Protograph ensemble The similar results also holds for protograph ensemble [Vontobel 2010] In this morning, Kenta has explained Definition of protograph ensemble Definition of spatially coupled LDPC codes Threshold saturation phenomenon of EXIT curve 28 / 34
29 Growth rate of spatially coupled LDPC codes [ G(ω) = 1 l 2L+1 r + L i= L log L+l 1 j= L [ log e h l 1 ω = 1 2L+1 k=0 ( log 1+ l 1 k=0 z j,k 2 ( 1+yi,k L i= L z j,k = tanh h+ y i,k = z i+k,k r l 1 2 tanh l 1 L r l ) )+e h l 1 l 1 i= Lk=0 ( k =0,k k l 1 k =0,k k h+ k=0 ( 1 yi,k 2 ) ] ( ) ] 1+yi,k z i+k,k log ω h. 2 l 1 k=0 tanh 1 (y i,k ) tanh 1( y j k,k z i+k,k r l ) ) 29 / 34
30 ω versus h L=2,4,8,16,32,64,128,256 (5,10) h ω 30 / 34
31 ω versus h: Derivative of growth rate 1 L=2,4,8,16,32,64,128,256 (5,10) 0.5 h ω 31 / 34
32 Growth rate L=2,4,8,16,2048 (5,10) G(ω) ω 32 / 34
33 Conclusion Contribution to annealed free energy of particular type has similar form of Bethe free energy. The stationary condition of maximization problem for annealed free energy is similar to equation of belief propagation. There exists threshold saturation phenomenon in the calculation of growth rate of spatially coupled LDPC codes. We now can calculate annealed free energy of any coupled factor graphs. Effect of boundary condition is not obvious. BP iterations does not necessarily converge (even for uncoupled cases). 33 / 34
34 Acknowledgment The basic idea that the growth rate approaches to the concave hull is given by Nicolas Macris. I acknowledges Hamed Hassani and Toshiyuki Tanaka for encouragement and discussion. arxiv: , paper about connection between Bethe and annealed free energies (submitted to ISIT 2011). Joint paper with Hamed and Nicolas about growth rate of coupled LDPC codes was submitted to ISIT / 34
Information, Physics, and Computation
Information, Physics, and Computation Marc Mezard Laboratoire de Physique Thdorique et Moales Statistiques, CNRS, and Universit y Paris Sud Andrea Montanari Department of Electrical Engineering and Department
More informationVariational algorithms for marginal MAP
Variational algorithms for marginal MAP Alexander Ihler UC Irvine CIOG Workshop November 2011 Variational algorithms for marginal MAP Alexander Ihler UC Irvine CIOG Workshop November 2011 Work with Qiang
More informationEstimating the Capacity of the 2-D Hard Square Constraint Using Generalized Belief Propagation
Estimating the Capacity of the 2-D Hard Square Constraint Using Generalized Belief Propagation Navin Kashyap (joint work with Eric Chan, Mahdi Jafari Siavoshani, Sidharth Jaggi and Pascal Vontobel, Chinese
More information14 : Theory of Variational Inference: Inner and Outer Approximation
10-708: Probabilistic Graphical Models 10-708, Spring 2014 14 : Theory of Variational Inference: Inner and Outer Approximation Lecturer: Eric P. Xing Scribes: Yu-Hsin Kuo, Amos Ng 1 Introduction Last lecture
More informationLower Bounds on the Graphical Complexity of Finite-Length LDPC Codes
Lower Bounds on the Graphical Complexity of Finite-Length LDPC Codes Igal Sason Department of Electrical Engineering Technion - Israel Institute of Technology Haifa 32000, Israel 2009 IEEE International
More informationUNDERSTANDING BELIEF PROPOGATION AND ITS GENERALIZATIONS
UNDERSTANDING BELIEF PROPOGATION AND ITS GENERALIZATIONS JONATHAN YEDIDIA, WILLIAM FREEMAN, YAIR WEISS 2001 MERL TECH REPORT Kristin Branson and Ian Fasel June 11, 2003 1. Inference Inference problems
More information13 : Variational Inference: Loopy Belief Propagation and Mean Field
10-708: Probabilistic Graphical Models 10-708, Spring 2012 13 : Variational Inference: Loopy Belief Propagation and Mean Field Lecturer: Eric P. Xing Scribes: Peter Schulam and William Wang 1 Introduction
More information14 : Theory of Variational Inference: Inner and Outer Approximation
10-708: Probabilistic Graphical Models 10-708, Spring 2017 14 : Theory of Variational Inference: Inner and Outer Approximation Lecturer: Eric P. Xing Scribes: Maria Ryskina, Yen-Chia Hsu 1 Introduction
More informationProbabilistic and Bayesian Machine Learning
Probabilistic and Bayesian Machine Learning Day 4: Expectation and Belief Propagation Yee Whye Teh ywteh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit University College London http://www.gatsby.ucl.ac.uk/
More informationProbabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Variational Inference II: Mean Field Method and Variational Principle Junming Yin Lecture 15, March 7, 2012 X 1 X 1 X 1 X 1 X 2 X 3 X 2 X 2 X 3
More informationSpatially Coupled LDPC Codes
Spatially Coupled LDPC Codes Kenta Kasai Tokyo Institute of Technology 30 Aug, 2013 We already have very good codes. Efficiently-decodable asymptotically capacity-approaching codes Irregular LDPC Codes
More information12 : Variational Inference I
10-708: Probabilistic Graphical Models, Spring 2015 12 : Variational Inference I Lecturer: Eric P. Xing Scribes: Fattaneh Jabbari, Eric Lei, Evan Shapiro 1 Introduction Probabilistic inference is one of
More informationStructured Variational Inference
Structured Variational Inference Sargur srihari@cedar.buffalo.edu 1 Topics 1. Structured Variational Approximations 1. The Mean Field Approximation 1. The Mean Field Energy 2. Maximizing the energy functional:
More information5. Density evolution. Density evolution 5-1
5. Density evolution Density evolution 5-1 Probabilistic analysis of message passing algorithms variable nodes factor nodes x1 a x i x2 a(x i ; x j ; x k ) x3 b x4 consider factor graph model G = (V ;
More informationOn the number of circuits in random graphs. Guilhem Semerjian. [ joint work with Enzo Marinari and Rémi Monasson ]
On the number of circuits in random graphs Guilhem Semerjian [ joint work with Enzo Marinari and Rémi Monasson ] [ Europhys. Lett. 73, 8 (2006) ] and [ cond-mat/0603657 ] Orsay 13-04-2006 Outline of the
More informationPhase transitions in discrete structures
Phase transitions in discrete structures Amin Coja-Oghlan Goethe University Frankfurt Overview 1 The physics approach. [following Mézard, Montanari 09] Basics. Replica symmetry ( Belief Propagation ).
More informationEE229B - Final Project. Capacity-Approaching Low-Density Parity-Check Codes
EE229B - Final Project Capacity-Approaching Low-Density Parity-Check Codes Pierre Garrigues EECS department, UC Berkeley garrigue@eecs.berkeley.edu May 13, 2005 Abstract The class of low-density parity-check
More information17 Variational Inference
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms for Inference Fall 2014 17 Variational Inference Prompted by loopy graphs for which exact
More informationRecitation 9: Loopy BP
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms For Inference Fall 204 Recitation 9: Loopy BP General Comments. In terms of implementation,
More information(a) Write down the value of q and of r. (2) Write down the equation of the axis of symmetry. (1) (c) Find the value of p. (3) (Total 6 marks)
1. Let f(x) = p(x q)(x r). Part of the graph of f is shown below. The graph passes through the points ( 2, 0), (0, 4) and (4, 0). (a) Write down the value of q and of r. (b) Write down the equation of
More informationCS Lecture 13. More Maximum Likelihood
CS 6347 Lecture 13 More Maxiu Likelihood Recap Last tie: Introduction to axiu likelihood estiation MLE for Bayesian networks Optial CPTs correspond to epirical counts Today: MLE for CRFs 2 Maxiu Likelihood
More informationProbabilistic Graphical Models. Theory of Variational Inference: Inner and Outer Approximation. Lecture 15, March 4, 2013
School of Computer Science Probabilistic Graphical Models Theory of Variational Inference: Inner and Outer Approximation Junming Yin Lecture 15, March 4, 2013 Reading: W & J Book Chapters 1 Roadmap Two
More informationConnecting Belief Propagation with Maximum Likelihood Detection
Connecting Belief Propagation with Maximum Likelihood Detection John MacLaren Walsh 1, Phillip Allan Regalia 2 1 School of Electrical Computer Engineering, Cornell University, Ithaca, NY. jmw56@cornell.edu
More informationA CLASSROOM NOTE: ENTROPY, INFORMATION, AND MARKOV PROPERTY. Zoran R. Pop-Stojanović. 1. Introduction
THE TEACHING OF MATHEMATICS 2006, Vol IX,, pp 2 A CLASSROOM NOTE: ENTROPY, INFORMATION, AND MARKOV PROPERTY Zoran R Pop-Stojanović Abstract How to introduce the concept of the Markov Property in an elementary
More informationBayesian Machine Learning - Lecture 7
Bayesian Machine Learning - Lecture 7 Guido Sanguinetti Institute for Adaptive and Neural Computation School of Informatics University of Edinburgh gsanguin@inf.ed.ac.uk March 4, 2015 Today s lecture 1
More informationLecture 18 Generalized Belief Propagation and Free Energy Approximations
Lecture 18, Generalized Belief Propagation and Free Energy Approximations 1 Lecture 18 Generalized Belief Propagation and Free Energy Approximations In this lecture we talked about graphical models and
More informationMessage passing and approximate message passing
Message passing and approximate message passing Arian Maleki Columbia University 1 / 47 What is the problem? Given pdf µ(x 1, x 2,..., x n ) we are interested in arg maxx1,x 2,...,x n µ(x 1, x 2,..., x
More informationMachine Learning for Data Science (CS4786) Lecture 24
Machine Learning for Data Science (CS4786) Lecture 24 Graphical Models: Approximate Inference Course Webpage : http://www.cs.cornell.edu/courses/cs4786/2016sp/ BELIEF PROPAGATION OR MESSAGE PASSING Each
More informationProbabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Variational Inference IV: Variational Principle II Junming Yin Lecture 17, March 21, 2012 X 1 X 1 X 1 X 1 X 2 X 3 X 2 X 2 X 3 X 3 Reading: X 4
More informationCS Lecture 19. Exponential Families & Expectation Propagation
CS 6347 Lecture 19 Exponential Families & Expectation Propagation Discrete State Spaces We have been focusing on the case of MRFs over discrete state spaces Probability distributions over discrete spaces
More informationProbabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Variational Inference III: Variational Principle I Junming Yin Lecture 16, March 19, 2012 X 1 X 1 X 1 X 1 X 2 X 3 X 2 X 2 X 3 X 3 Reading: X 4
More informationConcavity of reweighted Kikuchi approximation
Concavity of reweighted ikuchi approximation Po-Ling Loh Department of Statistics The Wharton School University of Pennsylvania loh@wharton.upenn.edu Andre Wibisono Computer Science Division University
More informationThreshold Saturation on Channels with Memory via Spatial Coupling
Threshold Saturation on Channels with Memory via Spatial Coupling Shrinivas Kudekar and Kenta Kasai New Mexico Consortium and Center for Non-linear Studies, Los Alamos National Laboratory, NM, USA Email:
More informationECE 4400:693 - Information Theory
ECE 4400:693 - Information Theory Dr. Nghi Tran Lecture 8: Differential Entropy Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Lecture 1 / 43 Outline 1 Review: Entropy of discrete RVs 2 Differential
More informationLDPC Codes. Intracom Telecom, Peania
LDPC Codes Alexios Balatsoukas-Stimming and Athanasios P. Liavas Technical University of Crete Dept. of Electronic and Computer Engineering Telecommunications Laboratory December 16, 2011 Intracom Telecom,
More information13 : Variational Inference: Loopy Belief Propagation
10-708: Probabilistic Graphical Models 10-708, Spring 2014 13 : Variational Inference: Loopy Belief Propagation Lecturer: Eric P. Xing Scribes: Rajarshi Das, Zhengzhong Liu, Dishan Gupta 1 Introduction
More informationInformation Geometric view of Belief Propagation
Information Geometric view of Belief Propagation Yunshu Liu 2013-10-17 References: [1]. Shiro Ikeda, Toshiyuki Tanaka and Shun-ichi Amari, Stochastic reasoning, Free energy and Information Geometry, Neural
More informationMin-Max Message Passing and Local Consistency in Constraint Networks
Min-Max Message Passing and Local Consistency in Constraint Networks Hong Xu, T. K. Satish Kumar, and Sven Koenig University of Southern California, Los Angeles, CA 90089, USA hongx@usc.edu tkskwork@gmail.com
More informationLecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016
Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016 1 Entropy Since this course is about entropy maximization,
More informationInformation Theory. Lecture 5 Entropy rate and Markov sources STEFAN HÖST
Information Theory Lecture 5 Entropy rate and Markov sources STEFAN HÖST Universal Source Coding Huffman coding is optimal, what is the problem? In the previous coding schemes (Huffman and Shannon-Fano)it
More informationThe Generalized Distributive Law and Free Energy Minimization
The Generalized Distributive Law and Free Energy Minimization Srinivas M. Aji Robert J. McEliece Rainfinity, Inc. Department of Electrical Engineering 87 N. Raymond Ave. Suite 200 California Institute
More informationDecomposition Methods for Large Scale LP Decoding
Decomposition Methods for Large Scale LP Decoding Siddharth Barman Joint work with Xishuo Liu, Stark Draper, and Ben Recht Outline Background and Problem Setup LP Decoding Formulation Optimization Framework
More informationComplex Systems Methods 2. Conditional mutual information, entropy rate and algorithmic complexity
Complex Systems Methods 2. Conditional mutual information, entropy rate and algorithmic complexity Eckehard Olbrich MPI MiS Leipzig Potsdam WS 2007/08 Olbrich (Leipzig) 26.10.2007 1 / 18 Overview 1 Summary
More informationLow-Density Parity-Check Codes
Department of Computer Sciences Applied Algorithms Lab. July 24, 2011 Outline 1 Introduction 2 Algorithms for LDPC 3 Properties 4 Iterative Learning in Crowds 5 Algorithm 6 Results 7 Conclusion PART I
More informationTHERE is a deep connection between the theory of linear
664 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 2, FEBRUARY 2007 Griffith Kelly Sherman Correlation Inequalities: A Useful Tool in the Theory of Error Correcting Codes Nicolas Macris, Member,
More informationExpectation Propagation Algorithm
Expectation Propagation Algorithm 1 Shuang Wang School of Electrical and Computer Engineering University of Oklahoma, Tulsa, OK, 74135 Email: {shuangwang}@ou.edu This note contains three parts. First,
More information1. Definition of a Polynomial
1. Definition of a Polynomial What is a polynomial? A polynomial P(x) is an algebraic expression of the form Degree P(x) = a n x n + a n 1 x n 1 + a n 2 x n 2 + + a 3 x 3 + a 2 x 2 + a 1 x + a 0 Leading
More informationJunction Tree, BP and Variational Methods
Junction Tree, BP and Variational Methods Adrian Weller MLSALT4 Lecture Feb 21, 2018 With thanks to David Sontag (MIT) and Tony Jebara (Columbia) for use of many slides and illustrations For more information,
More informationExpectation propagation for symbol detection in large-scale MIMO communications
Expectation propagation for symbol detection in large-scale MIMO communications Pablo M. Olmos olmos@tsc.uc3m.es Joint work with Javier Céspedes (UC3M) Matilde Sánchez-Fernández (UC3M) and Fernando Pérez-Cruz
More informationOne-Bit LDPC Message Passing Decoding Based on Maximization of Mutual Information
One-Bit LDPC Message Passing Decoding Based on Maximization of Mutual Information ZOU Sheng and Brian M. Kurkoski kurkoski@ice.uec.ac.jp University of Electro-Communications Tokyo, Japan University of
More informationSpatially-Coupled MacKay-Neal Codes Universally Achieve the Symmetric Information Rate of Arbitrary Generalized Erasure Channels with Memory
Spatially-Coupled MacKay-Neal Codes Universally Achieve the Symmetric Information Rate of Arbitrary Generalized Erasure Channels with Memory arxiv:1516736v1 [csit] 27 Jan 215 Abstract This paper investigates
More informationInformation Projection Algorithms and Belief Propagation
χ 1 π(χ 1 ) Information Projection Algorithms and Belief Propagation Phil Regalia Department of Electrical Engineering and Computer Science Catholic University of America Washington, DC 20064 with J. M.
More informationPerformance Analysis and Code Optimization of Low Density Parity-Check Codes on Rayleigh Fading Channels
Performance Analysis and Code Optimization of Low Density Parity-Check Codes on Rayleigh Fading Channels Jilei Hou, Paul H. Siegel and Laurence B. Milstein Department of Electrical and Computer Engineering
More informationFactor Graphs and Message Passing Algorithms Part 1: Introduction
Factor Graphs and Message Passing Algorithms Part 1: Introduction Hans-Andrea Loeliger December 2007 1 The Two Basic Problems 1. Marginalization: Compute f k (x k ) f(x 1,..., x n ) x 1,..., x n except
More informationECEN 655: Advanced Channel Coding
ECEN 655: Advanced Channel Coding Course Introduction Henry D. Pfister Department of Electrical and Computer Engineering Texas A&M University ECEN 655: Advanced Channel Coding 1 / 19 Outline 1 History
More information5. Sum-product algorithm
Sum-product algorithm 5-1 5. Sum-product algorithm Elimination algorithm Sum-product algorithm on a line Sum-product algorithm on a tree Sum-product algorithm 5-2 Inference tasks on graphical models consider
More informationAn instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if. 2 l i. i=1
Kraft s inequality An instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if N 2 l i 1 Proof: Suppose that we have a tree code. Let l max = max{l 1,...,
More informationJoint Iterative Decoding of LDPC Codes and Channels with Memory
Joint Iterative Decoding of LDPC Codes and Channels with Memory Henry D. Pfister and Paul H. Siegel University of California, San Diego 3 rd Int l Symposium on Turbo Codes September 1, 2003 Outline Channels
More information13: Variational inference II
10-708: Probabilistic Graphical Models, Spring 2015 13: Variational inference II Lecturer: Eric P. Xing Scribes: Ronghuo Zheng, Zhiting Hu, Yuntian Deng 1 Introduction We started to talk about variational
More informationStatistical Physics on Sparse Random Graphs: Mathematical Perspective
Statistical Physics on Sparse Random Graphs: Mathematical Perspective Amir Dembo Stanford University Northwestern, July 19, 2016 x 5 x 6 Factor model [DM10, Eqn. (1.4)] x 1 x 2 x 3 x 4 x 9 x8 x 7 x 10
More informationLecture 5 Channel Coding over Continuous Channels
Lecture 5 Channel Coding over Continuous Channels I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw November 14, 2014 1 / 34 I-Hsiang Wang NIT Lecture 5 From
More informationOn the Typicality of the Linear Code Among the LDPC Coset Code Ensemble
5 Conference on Information Sciences and Systems The Johns Hopkins University March 16 18 5 On the Typicality of the Linear Code Among the LDPC Coset Code Ensemble C.-C. Wang S.R. Kulkarni and H.V. Poor
More informationGame Theory and its Applications to Networks - Part I: Strict Competition
Game Theory and its Applications to Networks - Part I: Strict Competition Corinne Touati Master ENS Lyon, Fall 200 What is Game Theory and what is it for? Definition (Roger Myerson, Game Theory, Analysis
More informationPowers of Tensors and Fast Matrix Multiplication
Powers of Tensors and Fast Matrix Multiplication François Le Gall Department of Computer Science Graduate School of Information Science and Technology The University of Tokyo Simons Institute, 12 November
More informationChapter 2: Entropy and Mutual Information. University of Illinois at Chicago ECE 534, Natasha Devroye
Chapter 2: Entropy and Mutual Information Chapter 2 outline Definitions Entropy Joint entropy, conditional entropy Relative entropy, mutual information Chain rules Jensen s inequality Log-sum inequality
More informationLecture 6: Gaussian Channels. Copyright G. Caire (Sample Lectures) 157
Lecture 6: Gaussian Channels Copyright G. Caire (Sample Lectures) 157 Differential entropy (1) Definition 18. The (joint) differential entropy of a continuous random vector X n p X n(x) over R is: Z h(x
More informationBeyond Log-Supermodularity: Lower Bounds and the Bethe Partition Function
Beyond Log-Supermodularity: Lower Bounds and the Bethe Partition Function Nicholas Ruozzi Communication Theory Laboratory École Polytechnique Fédérale de Lausanne Lausanne, Switzerland nicholas.ruozzi@epfl.ch
More informationHash Property and Fixed-rate Universal Coding Theorems
1 Hash Property and Fixed-rate Universal Coding Theorems Jun Muramatsu Member, IEEE, Shigeki Miyake Member, IEEE, Abstract arxiv:0804.1183v1 [cs.it 8 Apr 2008 The aim of this paper is to prove the achievability
More informationMessage Passing and Junction Tree Algorithms. Kayhan Batmanghelich
Message Passing and Junction Tree Algorithms Kayhan Batmanghelich 1 Review 2 Review 3 Great Ideas in ML: Message Passing Each soldier receives reports from all branches of tree 3 here 7 here 1 of me 11
More informationCodes on Graphs, Normal Realizations, and Partition Functions
Codes on Graphs, Normal Realizations, and Partition Functions G. David Forney, Jr. 1 Workshop on Counting, Inference and Optimization Princeton, NJ November 3, 2011 1 Joint work with: Heide Gluesing-Luerssen,
More informationLinear Response for Approximate Inference
Linear Response for Approximate Inference Max Welling Department of Computer Science University of Toronto Toronto M5S 3G4 Canada welling@cs.utoronto.ca Yee Whye Teh Computer Science Division University
More informationConcavity of reweighted Kikuchi approximation
Concavity of reweighted Kikuchi approximation Po-Ling Loh Department of Statistics The Wharton School University of Pennsylvania loh@wharton.upenn.edu Andre Wibisono Computer Science Division University
More informationCOMPSCI 650 Applied Information Theory Apr 5, Lecture 18. Instructor: Arya Mazumdar Scribe: Hamed Zamani, Hadi Zolfaghari, Fatemeh Rezaei
COMPSCI 650 Applied Information Theory Apr 5, 2016 Lecture 18 Instructor: Arya Mazumdar Scribe: Hamed Zamani, Hadi Zolfaghari, Fatemeh Rezaei 1 Correcting Errors in Linear Codes Suppose someone is to send
More informationProbabilistic Graphical Models Lecture Notes Fall 2009
Probabilistic Graphical Models Lecture Notes Fall 2009 October 28, 2009 Byoung-Tak Zhang School of omputer Science and Engineering & ognitive Science, Brain Science, and Bioinformatics Seoul National University
More informationMGMT 69000: Topics in High-dimensional Data Analysis Falll 2016
MGMT 69000: Topics in High-dimensional Data Analysis Falll 2016 Lecture 14: Information Theoretic Methods Lecturer: Jiaming Xu Scribe: Hilda Ibriga, Adarsh Barik, December 02, 2016 Outline f-divergence
More informationGraphical Models and Kernel Methods
Graphical Models and Kernel Methods Jerry Zhu Department of Computer Sciences University of Wisconsin Madison, USA MLSS June 17, 2014 1 / 123 Outline Graphical Models Probabilistic Inference Directed vs.
More informationSparse Superposition Codes for the Gaussian Channel
Sparse Superposition Codes for the Gaussian Channel Florent Krzakala (LPS, Ecole Normale Supérieure, France) J. Barbier (ENS) arxiv:1403.8024 presented at ISIT 14 Long version in preparation Communication
More informationLOW-density parity-check (LDPC) codes were invented
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 54, NO 1, JANUARY 2008 51 Extremal Problems of Information Combining Yibo Jiang, Alexei Ashikhmin, Member, IEEE, Ralf Koetter, Senior Member, IEEE, and Andrew
More information18.310A Final exam practice questions
18.310A Final exam practice questions This is a collection of practice questions, gathered randomly from previous exams and quizzes. They may not be representative of what will be on the final. In particular,
More informationPseudocodewords from Bethe Permanents
Pseudocodewords from Bethe Permanents Roxana Smarandache Departments of Mathematics and Electrical Engineering University of Notre Dame Notre Dame, IN 46556 USA rsmarand@ndedu Abstract It was recently
More informationBelief Propagation, Information Projections, and Dykstra s Algorithm
Belief Propagation, Information Projections, and Dykstra s Algorithm John MacLaren Walsh, PhD Department of Electrical and Computer Engineering Drexel University Philadelphia, PA jwalsh@ece.drexel.edu
More informationON THE MINIMUM DISTANCE OF NON-BINARY LDPC CODES. Advisor: Iryna Andriyanova Professor: R.. udiger Urbanke
ON THE MINIMUM DISTANCE OF NON-BINARY LDPC CODES RETHNAKARAN PULIKKOONATTU ABSTRACT. Minimum distance is an important parameter of a linear error correcting code. For improved performance of binary Low
More informationIntroduction to Graphical Models. Srikumar Ramalingam School of Computing University of Utah
Introduction to Graphical Models Srikumar Ramalingam School of Computing University of Utah Reference Christopher M. Bishop, Pattern Recognition and Machine Learning, Jonathan S. Yedidia, William T. Freeman,
More informationVariational Inference (11/04/13)
STA561: Probabilistic machine learning Variational Inference (11/04/13) Lecturer: Barbara Engelhardt Scribes: Matt Dickenson, Alireza Samany, Tracy Schifeling 1 Introduction In this lecture we will further
More informationDoes Better Inference mean Better Learning?
Does Better Inference mean Better Learning? Andrew E. Gelfand, Rina Dechter & Alexander Ihler Department of Computer Science University of California, Irvine {agelfand,dechter,ihler}@ics.uci.edu Abstract
More informationEE376A: Homework #3 Due by 11:59pm Saturday, February 10th, 2018
Please submit the solutions on Gradescope. EE376A: Homework #3 Due by 11:59pm Saturday, February 10th, 2018 1. Optimal codeword lengths. Although the codeword lengths of an optimal variable length code
More informationLecture 5 - Information theory
Lecture 5 - Information theory Jan Bouda FI MU May 18, 2012 Jan Bouda (FI MU) Lecture 5 - Information theory May 18, 2012 1 / 42 Part I Uncertainty and entropy Jan Bouda (FI MU) Lecture 5 - Information
More informationExpectation Consistent Free Energies for Approximate Inference
Expectation Consistent Free Energies for Approximate Inference Manfred Opper ISIS School of Electronics and Computer Science University of Southampton SO17 1BJ, United Kingdom mo@ecs.soton.ac.uk Ole Winther
More informationOQ4867. Let ABC be a triangle and AA 1 BB 1 CC 1 = {M} where A 1 BC, B 1 CA, C 1 AB. Determine all points M for which ana 1...
764 Octogon Mathematical Magazine, Vol. 24, No.2, October 206 Open questions OQ4867. Let ABC be a triangle and AA BB CC = {M} where A BC, B CA, C AB. Determine all points M for which 4 s 2 3r 2 2Rr AA
More informationBelief Propagation on Partially Ordered Sets Robert J. McEliece California Institute of Technology
Belief Propagation on Partially Ordered Sets Robert J. McEliece California Institute of Technology +1 +1 +1 +1 {1,2,3} {1,3,4} {2,3,5} {3,4,5} {1,3} {2,3} {3,4} {3,5} -1-1 -1-1 {3} +1 International Symposium
More informationOn Achievable Rates and Complexity of LDPC Codes over Parallel Channels: Bounds and Applications
On Achievable Rates and Complexity of LDPC Codes over Parallel Channels: Bounds and Applications Igal Sason, Member and Gil Wiechman, Graduate Student Member Abstract A variety of communication scenarios
More informationInference as Optimization
Inference as Optimization Sargur Srihari srihari@cedar.buffalo.edu 1 Topics in Inference as Optimization Overview Exact Inference revisited The Energy Functional Optimizing the Energy Functional 2 Exact
More informationLECTURE 3. Last time:
LECTURE 3 Last time: Mutual Information. Convexity and concavity Jensen s inequality Information Inequality Data processing theorem Fano s Inequality Lecture outline Stochastic processes, Entropy rate
More informationThe Ising model and Markov chain Monte Carlo
The Ising model and Markov chain Monte Carlo Ramesh Sridharan These notes give a short description of the Ising model for images and an introduction to Metropolis-Hastings and Gibbs Markov Chain Monte
More informationLearning Target: I can sketch the graphs of rational functions without a calculator. a. Determine the equation(s) of the asymptotes.
Learning Target: I can sketch the graphs of rational functions without a calculator Consider the graph of y= f(x), where f(x) = 3x 3 (x+2) 2 a. Determine the equation(s) of the asymptotes. b. Find the
More informationStatus of Knowledge on Non-Binary LDPC Decoders
Status of Knowledge on Non-Binary LDPC Decoders Part I: From Binary to Non-Binary Belief Propagation Decoding D. Declercq 1 1 ETIS - UMR8051 ENSEA/Cergy-University/CNRS France IEEE SSC SCV Tutorial, Santa
More informationIntroduction to Graphical Models. Srikumar Ramalingam School of Computing University of Utah
Introduction to Graphical Models Srikumar Ramalingam School of Computing University of Utah Reference Christopher M. Bishop, Pattern Recognition and Machine Learning, Jonathan S. Yedidia, William T. Freeman,
More informationSlepian-Wolf Code Design via Source-Channel Correspondence
Slepian-Wolf Code Design via Source-Channel Correspondence Jun Chen University of Illinois at Urbana-Champaign Urbana, IL 61801, USA Email: junchen@ifpuiucedu Dake He IBM T J Watson Research Center Yorktown
More informationBounds on Achievable Rates of LDPC Codes Used Over the Binary Erasure Channel
Bounds on Achievable Rates of LDPC Codes Used Over the Binary Erasure Channel Ohad Barak, David Burshtein and Meir Feder School of Electrical Engineering Tel-Aviv University Tel-Aviv 69978, Israel Abstract
More informationLecture 10 Polynomial interpolation
Lecture 10 Polynomial interpolation Weinan E 1,2 and Tiejun Li 2 1 Department of Mathematics, Princeton University, weinan@princeton.edu 2 School of Mathematical Sciences, Peking University, tieli@pku.edu.cn
More information