Information Recovery from Pairwise Measurements
|
|
- Lionel Hart
- 5 years ago
- Views:
Transcription
1 Information Recovery from Pairwise Measurements A Shannon-Theoretic Approach Yuxin Chen, Changho Suh, Andrea Goldsmith Stanford University KAIST Page 1
2 Recovering data from correlation measurements A large collection of data instances In many applications, it is difficult/infeasible to measure each variable directly feasible to measure pairwise correlation Page 2
3 Motivating application: multi-image alignment Structure from motion: estimate 3D structures from 2D image sequences Key step: joint alignment input: (noisy) estimates of relative camera poses goal: jointly recover all camera poses Page 3
4 Motivating application: graph clustering Real-world networks exhibit community structures input: pairwise similarities between members goal: uncover hidden clusters Page 4
5 This talk: recovery from pairwise difference measurements Goal: recover a collection of variables {x i } Can only measure several pairwise difference x i x j (broadly defined) Page 5
6 This talk: recovery from pairwise difference measurements Goal: recover a collection of variables {x i } Can only measure several pairwise difference x i x j (broadly defined) Examples: joint alignment x i : (angle θ i, position z i ) relative rotation/translation (θ i θ j, z i z j ) Page 5
7 This talk: recovery from pairwise difference measurements Goal: recover a collection of variables {x i } Can only measure several pairwise difference x i x j (broadly defined) Examples: joint alignment x i : (angle θ i, position z i ) relative rotation/translation (θ i θ j, z i z j ) graph partition x i : membership (which partition it belongs to) { 1, if i, j same partition cluster agreement: x i x j = 0, else. Page 5
8 This talk: recovery from pairwise difference measurements Goal: recover a collection of variables {x i } Can only measure several pairwise difference x i x j (broadly defined) Examples: joint alignment x i : (angle θ i, position z i ) relative rotation/translation (θ i θ j, z i z j ) graph partition x i : membership (which partition it belongs to) { 1, if i, j same partition cluster agreement: x i x j = 0, else. pairwise maps, parity reads,... Page 5
9 A fundamental-limit perspective? A flurry of activity in recovery algorithm design convex program combinatorial spectral method Page 6
10 A fundamental-limit perspective? A flurry of activity in recovery algorithm design convex program combinatorial spectral method What are the fundamental recovery limits? min. sample complexity? how noisy the measurements can be? Page 6
11 A fundamental-limit perspective? A flurry of activity in recovery algorithm design convex program combinatorial spectral method What are the fundamental recovery limits? min. sample complexity? how noisy the measurements can be? So far mostly studied in a model-specific manner Seek a more unified framework Page 6
12 Problem setup: a Shannon-theoretic framework x 3 x 1 x 4 x 5 x 2 x 6 y 26 x i {0,, M 1} x 7 Information network n vertices discrete inputs w/ alphabet size: could scale with n M Page 7
13 Problem setup: a Shannon-theoretic framework y 34 x 3 x 1 y 13 y 12 measurement graph G x 4 y 35 y 15 x 5 y 24 y 27 x 6 x 2 y 26 measurements of x 1 x 2, x 1 x 3, x 1 x 5, x 7 y 67 Pairwise difference measurements truth: x i x j measurements: y ij (arbitrary alphabet) can be corrupted by noise, distortion,... Graphical representation observe y ij (i, j) G Page 7
14 Problem setup: a Shannon-theoretic framework x1 - x2 x1 x3 x1 - x5 x2 x4 x5 x6 x7 x2 - x7 x6 - x7 p (yij xi-xj ) channel y12 channel y15 channel y27 channel y67 Channel-decoding perspective each measurement is modeled by an i.i.d. channel transition prob. P (yij xi xj ) Page 7
15 Problem setup: a Shannon-theoretic framework x1 - x2 x1 x3 p (yij xi-xj ) channel y12 channel y15 channel y27 channel y67 x1 - x5 x2 x4 x5 x6 x7 x2 - x7 x6 - x7 Goal: recover {xi} exactly (up to global offset) Unified framework for decoding model capture similarities among various applications Page 7
16 What factors dictate hardness of recovery? x 1 x 2 = 1 channel y 12 P 1 x 1 x 2 x 1 x 2 = 2 channel y 12 P 2 P l := P( y ij x i x j = l) Channel distance/resolution Captured by KL( P l P k ) or Hellinger( P l P k ) or... Page 8
17 What factors dictate hardness of recovery? x 1 x 2 = 1 channel y 12 P 1 x 1 x 2 x 1 x 2 = 3 channel y 12 P 3 P l := P( y ij x i x j = l) Minimum channel distance/resolution min l k KL( P l P k ) := KL min or min l k Hellinger( P l P k ) := Hellinger min or... Uncoded input Page 8
18 What factors dictate hardness of recovery? Graph connectivity measurement graph G o Impossible to recover isolated vertices Page 9
19 What factors dictate hardness of recovery? Graph connectivity o Over-sparse connectivity is fragile measurement graph G Page 9
20 What factors dictate hardness of recovery? Graph connectivity measurement graph G o Sufficient connectivity removes fragility! Page 9
21 Agenda Page 10
22 Main result: Erdos-Renyi random graph Erdos-Renyi graph G(n, p obs ). Each edge (i, j) is present independently w.p. p obs (p obs = 1) (p obs = 0.3) Page 11
23 Main result: Erdos-Renyi random graph Erdos-Renyi graph G(n, p obs ). Each edge (i, j) is present independently w.p. p obs (p obs = 1) (p obs = 0.3) ML decoding works if Hellinger min > 2 log n + 4 log M p obs n Page 11
24 Main result: Erdos-Renyi random graph Erdos-Renyi graph G(n, p obs ). Each edge (i, j) is present independently w.p. p obs (p obs = 1) (p obs = 0.3) ML decoding works if Hellinger min > 2 log n + 4 log M p obs n Converse: no method works if KL min < log n p obs n Page 11
25 Main result: Erdos-Renyi random graph Erdos-Renyi graph G(n, p obs ). Each edge (i, j) is present independently w.p. p obs (p obs = 1) (p obs = 0.3) ML decoding works if Hellinger min > Converse: no method works if 2 log n + 4 log M p obs n KL min < log n p obs n non-asymptotic! Page 11
26 Main result: Erdos-Renyi random graph (p obs = 1) (p obs = 0.3) Page 12
27 Main result: Erdos-Renyi random graph (p obs = 1) (p obs = 0.3) In the hard regime where dp l dp k 1: KL min 2 Hellinger min Page 12
28 Main result: Erdos-Renyi random graph (p obs = 1) (p obs = 0.3) In the hard regime where dp l dp k 1: KL min 2 Hellinger min Recovery conditions ML works if Hellinger min 2 log n + 4 log M > p obs n Impossible if Hellinger min < log n 2p obs n Page 12
29 Main result: Erdos-Renyi random graph (p obs = 1) (p obs = 0.3) In the hard regime where dp l dp k 1: KL min 2 Hellinger min Fundamental recovery condition (assuming M poly(n)) Hellinger min log n p obs n Page 12
30 Main result: Erdos-Renyi random graph (p obs = 1) (p obs = 0.3) In the hard regime where dp l dp k 1: KL min 2 Hellinger min Fundamental recovery condition (assuming M poly(n)) Hellinger min log n p obs n avg-degree Hellinger min log n Page 12
31 Intuition Fundamental recovery condition (Erdos-Renyi graphs). avg-degree Hellinger min log n [x i x j ] 1 i,j n Page 13
32 Intuition Fundamental recovery condition (Erdos-Renyi graphs). avg-degree Hellinger min log n [x i x j ] 1 i,j n hypotheses: H 0 : x = [0, 0,, 0] H 1 : x = [1, 0,, 0] H 0 and H 1 differ only at the highlighted region ( avg-degree pieces of info) Page 13
33 Intuition Fundamental recovery condition (Erdos-Renyi graphs). avg-degree Hellinger min log n [x i x j ] 1 i,j n hypotheses: H 0 : x = [0, 0,, 0] H 2 : x = [0, 1,, 0] H 0 and H 2 differ only at the highlighted region ( avg-degree pieces of info) Page 13
34 Intuition Fundamental recovery condition (Erdos-Renyi graphs). avg-degree Hellinger min log n (1) [x i x j ] 1 i,j n hypotheses: H 0 : x = [0, 0,, 0] H n : x = [0, 0,, 1] n minimally-separated hypotheses needs at least log n bits the consequence of uncoded inputs Page 13
35 Minimal sample complexity Fundamental recovery condition (Erdos-Renyi graphs). avg-degree Hellinger min log n Sample complexity: n avg-degree Page 14
36 Minimal sample complexity Fundamental recovery condition (Erdos-Renyi graphs). avg-degree Hellinger min log n Sample complexity: n avg-degree Min sample complexity n log n Hellinger min Page 14
37 How general this limit is? Fundamental recovery condition (Erdos-Renyi graphs). avg-degree Hellinger min log n Can we go beyond Erdos-Renyi graphs? Page 15
38 Main results: homogeneous graphs random geometric graph (generalized) ring Page 16
39 Main results: homogeneous graphs random geometric graph (generalized) ring Fundamental recovery condition (various homogeneous graphs). avg-degree Hellinger min log n Homogeneous graphs: min-degree max-degree mincut balanced cut-set distributions Page 16
40 Main results: homogeneous graphs random geometric graph (generalized) ring Fundamental recovery condition (various homogeneous graphs). avg-degree Hellinger min log n Depend almost only on graph sparsity Page 16
41 Main results: general graphs mincut = 5 Page 17
42 Main results: general graphs mincut = 5 Information across the minimum cut set: mincut Hellinger min Page 17
43 Main results: general graphs Recovery conditions mincut = 5 ML works if mincut Hellinger min τ cut + log n + log M Impossible if mincut Hellinger min τ cut + mincut max-degree log n Page 17
44 Cut-homogeneity exponent τ cut captures 1 growth rate of the cut-set distribution the ratio mincut avg-degree 1 τ cut := maxk 1 k N (k mincut), where N (K) := {cut : cut-size K} Page 18
45 Cut-homogeneity exponent τ cut captures 1 growth rate of the cut-set distribution the ratio mincut avg-degree In general: τ cut 1 For homogeneous graphs: τ cut log n 1 τ cut := maxk 1 k N (k mincut), where N (K) := {cut : cut-size K} Page 18
46 Summary of main results mincut Hellinger min (1 log n) gap log n avg-deg Hellinger min log n gap 1 avg-deg Hellinger min log n gap 4(1 + 2 log M log n ) Page 19
47 Concrete application: stochastic block model Stochastic block model: 2 clusters edge densities: within-cluster: p = α log n n across-cluster: q = β log n n (q < p) adjacency matrix Page 20
48 Concrete application: stochastic block model Stochastic block model: 2 clusters edge densities: within-cluster: p = α log n n across-cluster: q = β log n n (q < p) adjacency matrix Our theory: feasible if impossible if α β > 2 α β < 1/2 Fundamental limit (Abbe et al. and Mossel et al.): α β > 2 Page 20
49 Concluding remarks A unified framework to determine recovery limits Interplay between IT and graph theory Tighten the pre-constants? Arxiv: Page 21
Information Recovery from Pairwise Measurements: A Shannon-Theoretic Approach
Information Recovery from Pairwise Measurements: A Shannon-Theoretic Approach Yuxin Chen Statistics Stanford University Email: yxchen@stanfordedu Changho Suh EE KAIST Email: chsuh@kaistackr Andrea J Goldsmith
More informationThe Projected Power Method: An Efficient Algorithm for Joint Alignment from Pairwise Differences
The Projected Power Method: An Efficient Algorithm for Joint Alignment from Pairwise Differences Yuxin Chen Emmanuel Candès Department of Statistics, Stanford University, Sep. 2016 Nonconvex optimization
More informationStatistical and Computational Phase Transitions in Planted Models
Statistical and Computational Phase Transitions in Planted Models Jiaming Xu Joint work with Yudong Chen (UC Berkeley) Acknowledgement: Prof. Bruce Hajek November 4, 203 Cluster/Community structure in
More informationRobust Principal Component Analysis
ELE 538B: Mathematics of High-Dimensional Data Robust Principal Component Analysis Yuxin Chen Princeton University, Fall 2018 Disentangling sparse and low-rank matrices Suppose we are given a matrix M
More informationHigh-dimensional graphical model selection: Practical and information-theoretic limits
1 High-dimensional graphical model selection: Practical and information-theoretic limits Martin Wainwright Departments of Statistics, and EECS UC Berkeley, California, USA Based on joint work with: John
More informationSpectral Method and Regularized MLE Are Both Optimal for Top-K Ranking
Spectral Method and Regularized MLE Are Both Optimal for Top-K Ranking Yuxin Chen Electrical Engineering, Princeton University Joint work with Jianqing Fan, Cong Ma and Kaizheng Wang Ranking A fundamental
More informationHigh-dimensional graphical model selection: Practical and information-theoretic limits
1 High-dimensional graphical model selection: Practical and information-theoretic limits Martin Wainwright Departments of Statistics, and EECS UC Berkeley, California, USA Based on joint work with: John
More informationLecture 4 Noisy Channel Coding
Lecture 4 Noisy Channel Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 9, 2015 1 / 56 I-Hsiang Wang IT Lecture 4 The Channel Coding Problem
More informationELE 538B: Mathematics of High-Dimensional Data. Spectral methods. Yuxin Chen Princeton University, Fall 2018
ELE 538B: Mathematics of High-Dimensional Data Spectral methods Yuxin Chen Princeton University, Fall 2018 Outline A motivating application: graph clustering Distance and angles between two subspaces Eigen-space
More informationCOMMUNITY DETECTION IN SPARSE NETWORKS VIA GROTHENDIECK S INEQUALITY
COMMUNITY DETECTION IN SPARSE NETWORKS VIA GROTHENDIECK S INEQUALITY OLIVIER GUÉDON AND ROMAN VERSHYNIN Abstract. We present a simple and flexible method to prove consistency of semidefinite optimization
More information9 Forward-backward algorithm, sum-product on factor graphs
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms For Inference Fall 2014 9 Forward-backward algorithm, sum-product on factor graphs The previous
More informationJointly Clustering Rows and Columns of Binary Matrices: Algorithms and Trade-offs
Jointly Clustering Rows and Columns of Binary Matrices: Algorithms and Trade-offs Jiaming Xu Joint work with Rui Wu, Kai Zhu, Bruce Hajek, R. Srikant, and Lei Ying University of Illinois, Urbana-Champaign
More informationMGMT 69000: Topics in High-dimensional Data Analysis Falll 2016
MGMT 69000: Topics in High-dimensional Data Analysis Falll 2016 Lecture 14: Information Theoretic Methods Lecturer: Jiaming Xu Scribe: Hilda Ibriga, Adarsh Barik, December 02, 2016 Outline f-divergence
More informationEstimating Unknown Sparsity in Compressed Sensing
Estimating Unknown Sparsity in Compressed Sensing Miles Lopes UC Berkeley Department of Statistics CSGF Program Review July 16, 2014 early version published at ICML 2013 Miles Lopes ( UC Berkeley ) estimating
More informationGraph Partitioning Using Random Walks
Graph Partitioning Using Random Walks A Convex Optimization Perspective Lorenzo Orecchia Computer Science Why Spectral Algorithms for Graph Problems in practice? Simple to implement Can exploit very efficient
More informationADMM and Fast Gradient Methods for Distributed Optimization
ADMM and Fast Gradient Methods for Distributed Optimization João Xavier Instituto Sistemas e Robótica (ISR), Instituto Superior Técnico (IST) European Control Conference, ECC 13 July 16, 013 Joint work
More informationIntroduction to graphical models: Lecture III
Introduction to graphical models: Lecture III Martin Wainwright UC Berkeley Departments of Statistics, and EECS Martin Wainwright (UC Berkeley) Some introductory lectures January 2013 1 / 25 Introduction
More informationInformation-Theoretic Limits of Group Testing: Phase Transitions, Noisy Tests, and Partial Recovery
Information-Theoretic Limits of Group Testing: Phase Transitions, Noisy Tests, and Partial Recovery Jonathan Scarlett jonathan.scarlett@epfl.ch Laboratory for Information and Inference Systems (LIONS)
More information(Classical) Information Theory III: Noisy channel coding
(Classical) Information Theory III: Noisy channel coding Sibasish Ghosh The Institute of Mathematical Sciences CIT Campus, Taramani, Chennai 600 113, India. p. 1 Abstract What is the best possible way
More informationLinear inverse problems on Erdős-Rényi graphs: Information-theoretic limits and efficient recovery
Linear inverse problems on Erdős-Rényi graphs: Information-theoretic limits and efficient recovery Emmanuel Abbe, Afonso S. Bandeira, Annina Bracher, Amit Singer Princeton University Abstract This paper
More informationQuantum Sphere-Packing Bounds and Moderate Deviation Analysis for Classical-Quantum Channels
Quantum Sphere-Packing Bounds and Moderate Deviation Analysis for Classical-Quantum Channels (, ) Joint work with Min-Hsiu Hsieh and Marco Tomamichel Hao-Chung Cheng University of Technology Sydney National
More informationSubmodularity in Machine Learning
Saifuddin Syed MLRG Summer 2016 1 / 39 What are submodular functions Outline 1 What are submodular functions Motivation Submodularity and Concavity Examples 2 Properties of submodular functions Submodularity
More informationSHARED INFORMATION. Prakash Narayan with. Imre Csiszár, Sirin Nitinawarat, Himanshu Tyagi, Shun Watanabe
SHARED INFORMATION Prakash Narayan with Imre Csiszár, Sirin Nitinawarat, Himanshu Tyagi, Shun Watanabe 2/40 Acknowledgement Praneeth Boda Himanshu Tyagi Shun Watanabe 3/40 Outline Two-terminal model: Mutual
More informationMixed Membership Stochastic Blockmodels
Mixed Membership Stochastic Blockmodels (2008) Edoardo M. Airoldi, David M. Blei, Stephen E. Fienberg and Eric P. Xing Herrissa Lamothe Princeton University Herrissa Lamothe (Princeton University) Mixed
More informationLecture 8: Shannon s Noise Models
Error Correcting Codes: Combinatorics, Algorithms and Applications (Fall 2007) Lecture 8: Shannon s Noise Models September 14, 2007 Lecturer: Atri Rudra Scribe: Sandipan Kundu& Atri Rudra Till now we have
More informationCompressed Sensing and Linear Codes over Real Numbers
Compressed Sensing and Linear Codes over Real Numbers Henry D. Pfister (joint with Fan Zhang) Texas A&M University College Station Information Theory and Applications Workshop UC San Diego January 31st,
More informationData Rate Theorem for Stabilization over Time-Varying Feedback Channels
Data Rate Theorem for Stabilization over Time-Varying Feedback Channels Workshop on Frontiers in Distributed Communication, Sensing and Control Massimo Franceschetti, UCSD (joint work with P. Minero, S.
More informationLecture 7. Union bound for reducing M-ary to binary hypothesis testing
Lecture 7 Agenda for the lecture M-ary hypothesis testing and the MAP rule Union bound for reducing M-ary to binary hypothesis testing Introduction of the channel coding problem 7.1 M-ary hypothesis testing
More informationMatrix estimation by Universal Singular Value Thresholding
Matrix estimation by Universal Singular Value Thresholding Courant Institute, NYU Let us begin with an example: Suppose that we have an undirected random graph G on n vertices. Model: There is a real symmetric
More informationPredictive Hypothesis Identification
Marcus Hutter - 1 - Predictive Hypothesis Identification Predictive Hypothesis Identification Marcus Hutter Canberra, ACT, 0200, Australia http://www.hutter1.net/ ANU RSISE NICTA Marcus Hutter - 2 - Predictive
More informationReconstruction in the Sparse Labeled Stochastic Block Model
Reconstruction in the Sparse Labeled Stochastic Block Model Marc Lelarge 1 Laurent Massoulié 2 Jiaming Xu 3 1 INRIA-ENS 2 INRIA-Microsoft Research Joint Centre 3 University of Illinois, Urbana-Champaign
More information1 Regression with High Dimensional Data
6.883 Learning with Combinatorial Structure ote for Lecture 11 Instructor: Prof. Stefanie Jegelka Scribe: Xuhong Zhang 1 Regression with High Dimensional Data Consider the following regression problem:
More informationA Random Dot Product Model for Weighted Networks arxiv: v1 [stat.ap] 8 Nov 2016
A Random Dot Product Model for Weighted Networks arxiv:1611.02530v1 [stat.ap] 8 Nov 2016 Daryl R. DeFord 1 Daniel N. Rockmore 1,2,3 1 Department of Mathematics, Dartmouth College, Hanover, NH, USA 03755
More informationPlanted Cliques, Iterative Thresholding and Message Passing Algorithms
Planted Cliques, Iterative Thresholding and Message Passing Algorithms Yash Deshpande and Andrea Montanari Stanford University November 5, 2013 Deshpande, Montanari Planted Cliques November 5, 2013 1 /
More informationSolving Corrupted Quadratic Equations, Provably
Solving Corrupted Quadratic Equations, Provably Yuejie Chi London Workshop on Sparse Signal Processing September 206 Acknowledgement Joint work with Yuanxin Li (OSU), Huishuai Zhuang (Syracuse) and Yingbin
More informationLower Bounds on the Graphical Complexity of Finite-Length LDPC Codes
Lower Bounds on the Graphical Complexity of Finite-Length LDPC Codes Igal Sason Department of Electrical Engineering Technion - Israel Institute of Technology Haifa 32000, Israel 2009 IEEE International
More informationReconstruction in the Generalized Stochastic Block Model
Reconstruction in the Generalized Stochastic Block Model Marc Lelarge 1 Laurent Massoulié 2 Jiaming Xu 3 1 INRIA-ENS 2 INRIA-Microsoft Research Joint Centre 3 University of Illinois, Urbana-Champaign GDR
More informationSHARED INFORMATION. Prakash Narayan with. Imre Csiszár, Sirin Nitinawarat, Himanshu Tyagi, Shun Watanabe
SHARED INFORMATION Prakash Narayan with Imre Csiszár, Sirin Nitinawarat, Himanshu Tyagi, Shun Watanabe 2/41 Outline Two-terminal model: Mutual information Operational meaning in: Channel coding: channel
More informationLearning Markov Network Structure using Brownian Distance Covariance
arxiv:.v [stat.ml] Jun 0 Learning Markov Network Structure using Brownian Distance Covariance Ehsan Khoshgnauz May, 0 Abstract In this paper, we present a simple non-parametric method for learning the
More informationCertifying the Global Optimality of Graph Cuts via Semidefinite Programming: A Theoretic Guarantee for Spectral Clustering
Certifying the Global Optimality of Graph Cuts via Semidefinite Programming: A Theoretic Guarantee for Spectral Clustering Shuyang Ling Courant Institute of Mathematical Sciences, NYU Aug 13, 2018 Joint
More informationHigh dimensional Ising model selection
High dimensional Ising model selection Pradeep Ravikumar UT Austin (based on work with John Lafferty, Martin Wainwright) Sparse Ising model US Senate 109th Congress Banerjee et al, 2008 Estimate a sparse
More information11. Learning graphical models
Learning graphical models 11-1 11. Learning graphical models Maximum likelihood Parameter learning Structural learning Learning partially observed graphical models Learning graphical models 11-2 statistical
More informationfor some error exponent E( R) as a function R,
. Capacity-achieving codes via Forney concatenation Shannon s Noisy Channel Theorem assures us the existence of capacity-achieving codes. However, exhaustive search for the code has double-exponential
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 11 Project
More informationat Some sort of quantization is necessary to represent continuous signals in digital form
Quantization at Some sort of quantization is necessary to represent continuous signals in digital form x(n 1,n ) x(t 1,tt ) D Sampler Quantizer x q (n 1,nn ) Digitizer (A/D) Quantization is also used for
More informationUpper Bounds on the Capacity of Binary Intermittent Communication
Upper Bounds on the Capacity of Binary Intermittent Communication Mostafa Khoshnevisan and J. Nicholas Laneman Department of Electrical Engineering University of Notre Dame Notre Dame, Indiana 46556 Email:{mhoshne,
More informationNetworks and Their Spectra
Networks and Their Spectra Victor Amelkin University of California, Santa Barbara Department of Computer Science victor@cs.ucsb.edu December 4, 2017 1 / 18 Introduction Networks (= graphs) are everywhere.
More informationSparse Subspace Clustering
Sparse Subspace Clustering Based on Sparse Subspace Clustering: Algorithm, Theory, and Applications by Elhamifar and Vidal (2013) Alex Gutierrez CSCI 8314 March 2, 2017 Outline 1 Motivation and Background
More informationOptimisation Combinatoire et Convexe.
Optimisation Combinatoire et Convexe. Low complexity models, l 1 penalties. A. d Aspremont. M1 ENS. 1/36 Today Sparsity, low complexity models. l 1 -recovery results: three approaches. Extensions: matrix
More informationECE 8201: Low-dimensional Signal Models for High-dimensional Data Analysis
ECE 8201: Low-dimensional Signal Models for High-dimensional Data Analysis Lecture 7: Matrix completion Yuejie Chi The Ohio State University Page 1 Reference Guaranteed Minimum-Rank Solutions of Linear
More information5. Density evolution. Density evolution 5-1
5. Density evolution Density evolution 5-1 Probabilistic analysis of message passing algorithms variable nodes factor nodes x1 a x i x2 a(x i ; x j ; x k ) x3 b x4 consider factor graph model G = (V ;
More informationExploiting Sparsity for Wireless Communications
Exploiting Sparsity for Wireless Communications Georgios B. Giannakis Dept. of ECE, Univ. of Minnesota http://spincom.ece.umn.edu Acknowledgements: D. Angelosante, J.-A. Bazerque, H. Zhu; and NSF grants
More information1.1 Basis of Statistical Decision Theory
ECE598: Information-theoretic methods in high-dimensional statistics Spring 2016 Lecture 1: Introduction Lecturer: Yihong Wu Scribe: AmirEmad Ghassami, Jan 21, 2016 [Ed. Jan 31] Outline: Introduction of
More informationQUANTIZATION FOR DISTRIBUTED ESTIMATION IN LARGE SCALE SENSOR NETWORKS
QUANTIZATION FOR DISTRIBUTED ESTIMATION IN LARGE SCALE SENSOR NETWORKS Parvathinathan Venkitasubramaniam, Gökhan Mergen, Lang Tong and Ananthram Swami ABSTRACT We study the problem of quantization for
More informationCommunity Detection on Euclidean Random Graphs
Community Detection on Euclidean Random Graphs Abishek Sankararaman, François Baccelli July 3, 207 Abstract Motivated by applications in online social networks, we introduce and study the problem of Community
More informationCommunity Detection. fundamental limits & efficient algorithms. Laurent Massoulié, Inria
Community Detection fundamental limits & efficient algorithms Laurent Massoulié, Inria Community Detection From graph of node-to-node interactions, identify groups of similar nodes Example: Graph of US
More informationLayered Synthesis of Latent Gaussian Trees
Layered Synthesis of Latent Gaussian Trees Ali Moharrer, Shuangqing Wei, George T. Amariucai, and Jing Deng arxiv:1608.04484v2 [cs.it] 7 May 2017 Abstract A new synthesis scheme is proposed to generate
More informationIntroduction to Graphical Models. Srikumar Ramalingam School of Computing University of Utah
Introduction to Graphical Models Srikumar Ramalingam School of Computing University of Utah Reference Christopher M. Bishop, Pattern Recognition and Machine Learning, Jonathan S. Yedidia, William T. Freeman,
More informationMAP Examples. Sargur Srihari
MAP Examples Sargur srihari@cedar.buffalo.edu 1 Potts Model CRF for OCR Topics Image segmentation based on energy minimization 2 Examples of MAP Many interesting examples of MAP inference are instances
More informationOptimization in Information Theory
Optimization in Information Theory Dawei Shen November 11, 2005 Abstract This tutorial introduces the application of optimization techniques in information theory. We revisit channel capacity problem from
More informationsparse and low-rank tensor recovery Cubic-Sketching
Sparse and Low-Ran Tensor Recovery via Cubic-Setching Guang Cheng Department of Statistics Purdue University www.science.purdue.edu/bigdata CCAM@Purdue Math Oct. 27, 2017 Joint wor with Botao Hao and Anru
More information(Classical) Information Theory II: Source coding
(Classical) Information Theory II: Source coding Sibasish Ghosh The Institute of Mathematical Sciences CIT Campus, Taramani, Chennai 600 113, India. p. 1 Abstract The information content of a random variable
More informationRobust Camera Location Estimation by Convex Programming
Robust Camera Location Estimation by Convex Programming Onur Özyeşil and Amit Singer INTECH Investment Management LLC 1 PACM and Department of Mathematics, Princeton University SIAM IS 2016 05/24/2016,
More informationSuperposition Encoding and Partial Decoding Is Optimal for a Class of Z-interference Channels
Superposition Encoding and Partial Decoding Is Optimal for a Class of Z-interference Channels Nan Liu and Andrea Goldsmith Department of Electrical Engineering Stanford University, Stanford CA 94305 Email:
More informationMessage passing and approximate message passing
Message passing and approximate message passing Arian Maleki Columbia University 1 / 47 What is the problem? Given pdf µ(x 1, x 2,..., x n ) we are interested in arg maxx1,x 2,...,x n µ(x 1, x 2,..., x
More informationSpectral Clustering. Guokun Lai 2016/10
Spectral Clustering Guokun Lai 2016/10 1 / 37 Organization Graph Cut Fundamental Limitations of Spectral Clustering Ng 2002 paper (if we have time) 2 / 37 Notation We define a undirected weighted graph
More informationSparse analysis Lecture VII: Combining geometry and combinatorics, sparse matrices for sparse signal recovery
Sparse analysis Lecture VII: Combining geometry and combinatorics, sparse matrices for sparse signal recovery Anna C. Gilbert Department of Mathematics University of Michigan Sparse signal recovery measurements:
More informationComputational Lower Bounds for Statistical Estimation Problems
Computational Lower Bounds for Statistical Estimation Problems Ilias Diakonikolas (USC) (joint with Daniel Kane (UCSD) and Alistair Stewart (USC)) Workshop on Local Algorithms, MIT, June 2018 THIS TALK
More informationCompressed Sensing and Sparse Recovery
ELE 538B: Sparsity, Structure and Inference Compressed Sensing and Sparse Recovery Yuxin Chen Princeton University, Spring 217 Outline Restricted isometry property (RIP) A RIPless theory Compressed sensing
More informationBasic models and questions in statistical network analysis
Basic models and questions in statistical network analysis Miklós Z. Rácz with Sébastien Bubeck September 12, 2016 Abstract Extracting information from large graphs has become an important statistical
More informationStatistical Measures of Uncertainty in Inverse Problems
Statistical Measures of Uncertainty in Inverse Problems Workshop on Uncertainty in Inverse Problems Institute for Mathematics and Its Applications Minneapolis, MN 19-26 April 2002 P.B. Stark Department
More informationCombining geometry and combinatorics
Combining geometry and combinatorics A unified approach to sparse signal recovery Anna C. Gilbert University of Michigan joint work with R. Berinde (MIT), P. Indyk (MIT), H. Karloff (AT&T), M. Strauss
More informationEntropies & Information Theory
Entropies & Information Theory LECTURE I Nilanjana Datta University of Cambridge,U.K. See lecture notes on: http://www.qi.damtp.cam.ac.uk/node/223 Quantum Information Theory Born out of Classical Information
More information1 Background on Information Theory
Review of the book Information Theory: Coding Theorems for Discrete Memoryless Systems by Imre Csiszár and János Körner Second Edition Cambridge University Press, 2011 ISBN:978-0-521-19681-9 Review by
More informationThe non-backtracking operator
The non-backtracking operator Florent Krzakala LPS, Ecole Normale Supérieure in collaboration with Paris: L. Zdeborova, A. Saade Rome: A. Decelle Würzburg: J. Reichardt Santa Fe: C. Moore, P. Zhang Berkeley:
More informationInderjit Dhillon The University of Texas at Austin
Inderjit Dhillon The University of Texas at Austin ( Universidad Carlos III de Madrid; 15 th June, 2012) (Based on joint work with J. Brickell, S. Sra, J. Tropp) Introduction 2 / 29 Notion of distance
More informationRank Minimization over Finite Fields
Rank Minimization over Finite Fields Vincent Y. F. Tan, Laura Balzano and Stark C. Draper Dept. of ECE, University of Wisconsin-Madison, Email: {vtan@,sunbeam@ece.,sdraper@ece.}wisc.edu LIDS, Massachusetts
More informationIntroduction to Graphical Models. Srikumar Ramalingam School of Computing University of Utah
Introduction to Graphical Models Srikumar Ramalingam School of Computing University of Utah Reference Christopher M. Bishop, Pattern Recognition and Machine Learning, Jonathan S. Yedidia, William T. Freeman,
More informationGossip algorithms for solving Laplacian systems
Gossip algorithms for solving Laplacian systems Anastasios Zouzias University of Toronto joint work with Nikolaos Freris (EPFL) Based on : 1.Fast Distributed Smoothing for Clock Synchronization (CDC 1).Randomized
More information19.1 Problem setup: Sparse linear regression
ECE598: Information-theoretic methods in high-dimensional statistics Spring 2016 Lecture 19: Minimax rates for sparse linear regression Lecturer: Yihong Wu Scribe: Subhadeep Paul, April 13/14, 2016 In
More informationIndependence and chromatic number (and random k-sat): Sparse Case. Dimitris Achlioptas Microsoft
Independence and chromatic number (and random k-sat): Sparse Case Dimitris Achlioptas Microsoft Random graphs W.h.p.: with probability that tends to 1 as n. Hamiltonian cycle Let τ 2 be the moment all
More informationCompressibility of Infinite Sequences and its Interplay with Compressed Sensing Recovery
Compressibility of Infinite Sequences and its Interplay with Compressed Sensing Recovery Jorge F. Silva and Eduardo Pavez Department of Electrical Engineering Information and Decision Systems Group Universidad
More informationIEOR 265 Lecture 3 Sparse Linear Regression
IOR 65 Lecture 3 Sparse Linear Regression 1 M Bound Recall from last lecture that the reason we are interested in complexity measures of sets is because of the following result, which is known as the M
More informationQuilting Stochastic Kronecker Graphs to Generate Multiplicative Attribute Graphs
Quilting Stochastic Kronecker Graphs to Generate Multiplicative Attribute Graphs Hyokun Yun (work with S.V.N. Vishwanathan) Department of Statistics Purdue Machine Learning Seminar November 9, 2011 Overview
More informationLecture 12. Block Diagram
Lecture 12 Goals Be able to encode using a linear block code Be able to decode a linear block code received over a binary symmetric channel or an additive white Gaussian channel XII-1 Block Diagram Data
More informationNew ways of dimension reduction? Cutting data sets into small pieces
New ways of dimension reduction? Cutting data sets into small pieces Roman Vershynin University of Michigan, Department of Mathematics Statistical Machine Learning Ann Arbor, June 5, 2012 Joint work with
More informationExtreme eigenvalues of Erdős-Rényi random graphs
Extreme eigenvalues of Erdős-Rényi random graphs Florent Benaych-Georges j.w.w. Charles Bordenave and Antti Knowles MAP5, Université Paris Descartes May 18, 2018 IPAM UCLA Inhomogeneous Erdős-Rényi random
More informationBOUNDS ON THE NUMBER OF INFERENCE FUNCTIONS OF A GRAPHICAL MODEL
BOUNDS ON THE NUMBER OF INFERENCE FUNCTIONS OF A GRAPHICAL MODEL SERGI ELIZALDE AND KEVIN WOODS Abstract. Directed and undirected graphical models, also called Bayesian networks and Markov random fields,
More informationModel-Based Compressive Sensing for Signal Ensembles. Marco F. Duarte Volkan Cevher Richard G. Baraniuk
Model-Based Compressive Sensing for Signal Ensembles Marco F. Duarte Volkan Cevher Richard G. Baraniuk Concise Signal Structure Sparse signal: only K out of N coordinates nonzero model: union of K-dimensional
More informationThe sequential decoding metric for detection in sensor networks
The sequential decoding metric for detection in sensor networks B. Narayanaswamy, Yaron Rachlin, Rohit Negi and Pradeep Khosla Department of ECE Carnegie Mellon University Pittsburgh, PA, 523 Email: {bnarayan,rachlin,negi,pkk}@ece.cmu.edu
More informationLarge Deviations for Channels with Memory
Large Deviations for Channels with Memory Santhosh Kumar Jean-Francois Chamberland Henry Pfister Electrical and Computer Engineering Texas A&M University September 30, 2011 1/ 1 Digital Landscape Delay
More informationLecture 11: Polar codes construction
15-859: Information Theory and Applications in TCS CMU: Spring 2013 Lecturer: Venkatesan Guruswami Lecture 11: Polar codes construction February 26, 2013 Scribe: Dan Stahlke 1 Polar codes: recap of last
More informationThe Lovász ϑ-function in Quantum Mechanics
The Lovász ϑ-function in Quantum Mechanics Zero-error Quantum Information and Noncontextuality Simone Severini UCL Oxford Jan 2013 Simone Severini (UCL) Lovász ϑ-function in Quantum Mechanics Oxford Jan
More informationList Decoding of Reed Solomon Codes
List Decoding of Reed Solomon Codes p. 1/30 List Decoding of Reed Solomon Codes Madhu Sudan MIT CSAIL Background: Reliable Transmission of Information List Decoding of Reed Solomon Codes p. 2/30 List Decoding
More informationEstimating Latent Variable Graphical Models with Moments and Likelihoods
Estimating Latent Variable Graphical Models with Moments and Likelihoods Arun Tejasvi Chaganty Percy Liang Stanford University June 18, 2014 Chaganty, Liang (Stanford University) Moments and Likelihoods
More informationCheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 Outlines Overview Introduction Linear Algebra Probability Linear Regression
More informationSpectral Perturbation of Small-World Networks with Application to Brain Disease Detection
Spectral Perturbation of Small-World Networks with Application to Brain Disease Detection Chenhui Hu May 4, 22 Introduction Many real life systems can be described by complex networks, which usually consist
More informationPerformance of small signal sets
42 Chapter 5 Performance of small signal sets In this chapter, we show how to estimate the performance of small-to-moderate-sized signal constellations on the discrete-time AWGN channel. With equiprobable
More informationCommunity detection in stochastic block models via spectral methods
Community detection in stochastic block models via spectral methods Laurent Massoulié (MSR-Inria Joint Centre, Inria) based on joint works with: Dan Tomozei (EPFL), Marc Lelarge (Inria), Jiaming Xu (UIUC),
More informationShannon s noisy-channel theorem
Shannon s noisy-channel theorem Information theory Amon Elders Korteweg de Vries Institute for Mathematics University of Amsterdam. Tuesday, 26th of Januari Amon Elders (Korteweg de Vries Institute for
More information