Springer Berlin Heidelberg New York Barcelona Budapest Hong Kong London Milan Paris Santa Clara Singapore Tokyo

Transcription

1 Springer Series in Information Sciences 32 Editor: Thomas S. Huang Springer Berlin Heidelberg New York Barcelona Budapest Hong Kong London Milan Paris Santa Clara Singapore Tokyo

2 Springer Series in Information Sciences Editors: Thomas S. Huang Teuvo Kohonen Manfred R. Schroeder Managing Editor: H. K. V. Lotsch 30 Self-Organizing Maps By T. Kohonen 2nd Edition 31 Music and Schema Theory Cognitive Foundations of Systematic Musicology By M. Leman 32 The Maximum Entropy Method By N. Wu 33 Steps Towards 3D Active Vision By T. Vieville 34 Calibration and Orientation of Cameras in Computer Vision Editors: A. Orlin and T. S. Huang 35 Speech Processing: Fundamentals and Applications By B. S. Atal and M. R. Schroeder Volumes 1-29 are listed at the end of the book.

3 Nailong Wu The Maximum Entropy Method With 53 Figures Springer

4 Dr. Nailong Wu Department of Physics and Astronomy York University Petrie Science Building 4700 Keele Street Toronto, Ontario, M3J JP3, Canada Series Editors: Professor Thomas S. Huang Department of Electrical Engineering and Coordinated Science Laboratory, University of Illinois, Urbana, IL 61801, USA Professor Teuvo Kohonen Helsinki University of Technology, Neural Networks Research Centre, Rakentajanaukio 2C, FIN-02 I50 Espoo, Finland Professor Dr. Manfred R. Schroeder Drittes Physikalisches lnstitut, UniversiUit Gottingen, Biirgerstrasse 42-44, D Gottingen, Germany Managing Editor: Dr.-Ing. Helmut K. V. Lotsch Springer-Verlag, Tiergartenstrasse 17, D Heidelberg, Germany ISSN: OnO-678X IS BN-13: e-isb -13: DOl: / Springer-Verlag Berlin Heidelberg New York Library of Congress Cataloging.in-Publication Data. Wu, Nailong, The maximum entropy method I Nailong Wu. p. em. - (Springer series in information sciences; 32) Includes bibliographical references and index. ISBN (hardcover: alk. paper) I. Maximum entropy method. 2. Spectral theory (Mathematics) 3. Signal processing. 4. Mathematical physics. l. TItle. II. Series. Q370.W '.54-dc This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thercof is permilled only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer Verlag. Violations are liable for prosecution under the German Copyright Law. Springer-Verlag Berlin Heidelberg 1997 Softcover reprint of the hardcover 1st edition 1997 The use of general descriptive names, registered names, trademarks, etc. in this publication docs not imply, even inlhe absence ofa specific statement, that such names arc exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Camera ready by author Cover design: design & prod"crion GmbH, Heidelberg Production editor: P. Treiber, Heidelberg SPIN: / I 0 - Printed on acid-free paper

5 Preface Forty years ago, in 1957, the Principle of Maximum Entropy was first introduced by Jaynes into the field of statistical mechanics. Since that seminal publication, this principle has been adopted in many areas of science and technology beyond its initial application. It is now found in spectral analysis, image restoration and a number of branches of mathematics and physics, and has become better known as the Maximum Entropy Method (MEM). Today MEM is a powerful means to deal with ill-posed problems, and much research work is devoted to it. My own research in the area of MEM started in 1980, when I was a graduate student in the Department of Electrical Engineering at the University of Sydney, Australia. This research work was the basis of my Ph.D. thesis, The Maximum Entropy Method and Its Application in Radio Astronomy, completed in As well as continuing my research in MEM after graduation, I taught a course of the same name at the Graduate School, Chinese Academy of Sciences, Beijing from 1987 to Delivering the course was the impetus for developing a structured approach to the understanding of MEM and writing hundreds of pages of lecture notes. Between 1991 and 1994, I was a member of the Image Restoration Project at the Space Telescope Science Institute in the United States of America, where I applied MEM as well as other methods to restore images from the Hubble Space Telescope. The research results, including image restoration software, were presented in a number of published papers and internal technical reports. It was these three activities: my Ph.D. thesis, the lecture notes, and the recently published papers and reports, that formed the basis for this monograph. Before actually rolling up my sleeves to start writing the manuscript, I thought for a while about the areas to be covered, the style of writing, the prospective readers and so on. Based on my learning, teaching and researching experience, I decided to select spectral analysis, image restoration, some branches in mathematics, and statistical mechanics for the purposes of

6 VI Preface elaborating upon the principle and demonstrating applications of MEM. A step-by-step style of writing is used for enhanced readability, so that this monograph can be used by both beginners and more experienced researchers in MEM. Relevant discussions and comments are included after, rather than interleaved with, the presentation of the main results. I have tried my best to follow the above guidelines throughout the book. This monograph consists of five chapters, two appendices, a list of references and an index. Chapter 1 is an overview of MEM. This was written especially with the needs of beginners in mind. Chapters 2, 3 are devoted to MEM and its applications, respectively, in spectral analysis and image restoration, including many algorithms in practical use. Chapter 4 is concerned, both experimentally and theoretically, with the analysis and comparison of three schools of thought on MEM in their basic ideas, properties and applications. Chapter 5 presents MEM and its applications to mathematics (including the solution of moment problems, integral equations and partial differential equations) and statistical mechanics. Appendices A, B provide the minimum relevant knowledge of cepstral analysis and image restoration, respectively, for understanding Chap. 3. This monograph will be found to be very useful as a reference for researchers, as a textbook for graduate students in a one-semester course, or as lecture notes for MEM disciples in an intensive course of some forty hours. Writing a monograph on MEM has been a somewhat challenging task for me. Much effort has been made to maximize the readability and to bring the contents up to date. My greatest satisfaction from publication of this monograph will come if it contributes to the further development of MEM theory and applications. Acknowledgments. My first thanks are due to Professor Trevor Cole, Department of Electrical Engineering at the University of Sydney, Australia, for his supervision of the research on MEM in my graduate studies, and for his continuous encouragement. I would like to acknowledge the assistance received from my colleagues at the following institutions during the preparation of the manuscript: the Space Telescope Science Institute, the United States of America; Dominion Radio Astrophysical Observatory, Canada; and York University, Canada. In particular, I would like to express my appreciation to Professor John Caldwell, Department of Physics and Astronomy at York University, for his generous support. I am deeply grateful to Dr. Clifford Maldonado, for the time he took to read over the manuscript, and for the corrections and suggestions he made to improve the presentation of this monograph.

7 Preface VII Finally, I would like to thank the Series Editor, Professor Thomas Huang, for his recommendation of the publication of this monograph, and the Managing Editor, Dr. Helmut Lotsch and his editorial staff at Springer-Verlag, for their kind assistance and friendly cooperation. Toronto, Canada February, 1997 Nailong Wu

8 Table of Contents 1. Introduction What is the Maximum Entropy Method Definition of Entropy Rationale of the Maximum Entropy Method Present and Future Research Maximum Entropy Method MEMI and Its Application in Spectral Analysis Definition and Expressions of Entropy HI Approach Approach Discussion Formulation and Solution Formulation Solution Discussion Equivalents and Signal Model ACF Extension Subject to the Nonnegativity Constraint Principle of MCE AR Process (Signal Model) Bayesian Method Wiener Filter and Approximation Theoretic Approach Algorithms and Numerical Example (Given ACF) Levinson's Recursion for I-D Noiseless Data Lim-Malik Algorithm for 2-D Noiseless Data Wernecke-D'Addario Algorithm for 2-D Noisy Data Numerical Example Algorithms and Numerical Example (Given Time Series) Burg Algorithm Marple Algorithm Other Fast Algorithms Numerical Example

9 X Table of Contents 2.6 Order Selection FPE Criterion AIC Criterion Other Criteria Summary Maximum Entropy Method MEM2 and Its Application in Image Restoration Definition and Expressions of Entropy H MLM Direct Definition Method Discussion Formulation and Implicit Solution Formulation Implicit Solution Iterative Algorithm _ Discussion Explicit Solution Explicit Solution Discussion Examples Equivalents and Signal Model ACF Extension Subject to the Nonnegativity Constraint Principle of MCE Exponential Process (Signal Model) Bayesian Method MLM R - A Procedure Statements of the MEM2 Problem R - A Procedure Example Algorithms and Numerical Examples (I) Frieden Algorithm Gull-Daniell Algorithm Revised GD Algorithm Simplified Newton-Raphson Algorithm Numerical Example Algorithms and Numerical Examples (II) Skilling-Bryan Algorithm Differential Equation Approach Algorithms and Numerical Examples (III) MEMjMemSys5 Package MEM Task in IRAF Restoration with Variable Resolution 184

10 Table of Contents Xl Numerical Examples Other Algorithms Analysis and Comparison of the Maximum Entropy Method Generalized MEM Formulation of GMEM 192 4:1.2 "Entropy" Expressions in GMEM Properties of GMEM Expressions of Entropy Solution's Properties Existence Uniqueness Consistency Statistical Properties Resolution Enhancement and Data Extension (Experimental Results) Examples Resolvability in 1-D Spectral Estimation Resolvability in 2-D Spectral Estimation Superresolution and Spectral Line Splitting Resolution Enhancement and Data Extension (Theoretical Analysis) Data Extension in MEM1 and MEM Resolution Enhancement of MEM1 and MEM MEM1 and MEM2 Spectra at Low SNR Line Splitting of MEM Peak Location and Relative Power Estimation (Experimental Results) Peak Location (Given ACF) Peak Location (Given Time Series) Relative Power Estimation (Given ACF) Summary and Comments Peak Location and Relative Power Estimation (Theoretical Analysis) Interference Between Peaks Causes Peak Shifting Explanation of the Peak Shifting in MEM1 Spectra Relative Power Estimation for MEM Summary for Sects Comments on the Three Schools of Thought on MEM Applications of the Maximum Entropy Method in Mathematics and Physics ", Solution of Moment Problems General Theory 252