Generalised density function estimation using moments and the characteristic function

Size: px

Start display at page:

Download "Generalised density function estimation using moments and the characteristic function"

Horace Neal
5 years ago
Views:

Generalised density function estimation using moments and the characteristic function Gerhard Esterhuizen Thesis presented in partial fulfilment of the

1 Generalised density function estimation using moments and the characteristic function Gerhard Esterhuizen Thesis presented in partial fulfilment of the requirements for the degree of Master of Science in Electronic Engineering at the University of Stellenbosch Supervisor: Prof. J.A. du Preez April 2003

2 Declaration I, the undersigned, hereby declare that the work contained in this thesis is my own original work and that I have not previously in its entirety or in part submitted it at any university for a degree. Signature: March 2003 i

3 Abstract Probability density functions (PDFs) and cumulative distribution functions (CDFs) playa central role in statistical pattern recognition and verification systems. They allow observations that do not occur according to deterministic rules to be quantified and modelled. An example of such observations would be the voice patterns of a person that is used as input to a biometric security device. In order to model such non-deterministic observations, a density function estimator is employed to estimate a PDF or CDF from sample data. Although numerous density function estimation techniques exist, all the techniques can be classified into one of two groups, parametric and non-parametric, each with its own characteristic advantages and disadvantages. In this research, we introduce a novel approach to density function estimation that attempts to combine some of the advantages of both the parametric and non-parametric estimators. This is done by considering density estimation using an abstract approach in which the density function is modelled entirely in terms of its moments or characteristic function. New density function estimation techniques are first developed in theory, after which a number of practical density function estimators are presented. Experiments are performed in which the performance of the new estimators are compared to two established estimators, namely the Parzen estimator and the Gaussian mixture model (GMM). The comparison is performed in terms of the accuracy, computational requirements and ease of use of the estimators and it is found that the new estimators does combine some of the advantages of the established estimators without the corresponding disadvantages. II

4 Opsomming Waarskynlikheids digtheidsfunksies (WDFs) en Kumulatiewe distribusiefunksies (KDFs) speel 'n sentrale rol in statistiese patroonherkenning en verifikasie stelsels. Hulle maak dit moontlik om nie-deterministiese observasies te kwantifiseer en te modelleer. Die stempatrone van 'n spreker wat as intree tot 'n biometriese sekuriteits stelsel gegee word, is 'n voorbeeld van so 'n observasie. Ten einde sulke observasies te modelleer, word 'n digtheidsfunksie afskatter gebruik om die WDF of KDF vanaf data monsters af te skat. Alhoewel daar talryke digtheidsfunksie afskatters bestaan, kan almal in een van twee katagoriee geplaas word, parametries en nie-parametries, elk met hul eie kenmerkende voordele en nadele. Hierdie werk Ie 'n nuwe benadering tot digtheidsfunksie afskatting voor wat die voordele van beide die parametriese sowel as die nie-parametriese tegnieke probeer kombineer. Dit word gedoen deur digtheidsfunksie afskatting vanuit 'n abstrakte oogpunt te benader waar die digtheidsfunksie uitsluitlik in terme van sy momente en karakteristieke funksie gemodelleer word. Nuwe metodes word eers in teorie ondersoek en ontwikkel waarna praktiese tegnieke voorgele word. Hierdie afskatters het die vermoe om 'n wye verskeidenheid digtheidsfunksies af te skat en is nie net ontwerp om slegs sekere families van digtheidsfunksies optimaal voor te stel nie. Eksperimente is uitgevoer wat die werkverrigting van die nuwe tegnieke met twee gevestigde tegnieke, naamlik die Parzen afskatter en die Gaussiese mengsel model (GMM), te vergelyk. Die werkverrigting word gemeet in terme van akkuraatheid, vereiste numeriese verwerkingsvermoe en die gemak van gebruik. Daar word bevind dat die nuwe afskatters wei voordele van die gevestigde afskatters kombineer sonder die gepaardgaande nadele. iii

5 Acknowledgements I would like to thank the following people: Prof. J.A. Du Preez, my supervisor, for his patient guidance. Zelda Weitz for her unfailing love and encouragement. My parents and family for their education and support. My grandmother and Mr and Mrs Weitz for providing a home away from home. Dr Dave Weber and the DSP Lab for providing a creative learning environment. Chari Botha for his technical advice. Pieter Nel and Koos Hugo for False Bay sailing. iv

6 Contents 1 Introduction Motivation and topicality Density function estimation Our research Background Random variables Pattern classification Hypothesis tests Existing techniques Non-parametric estimators Parametric estimators Other techniques Requirements for a new estimator Objectives Contributions Overview of the document 19 2 Novel probability density function estimators Introduction Motivation Definitions and Background Moments Characteristic function Fourier series Estimators based on moments Motivation. 27 v

7 2.4.2 A PDF in terms of moments The anti-derivative Numerical integration techniques Fourier series approximation Estimating moments from sample data Estimators based on the characteristic function Motivation A PDF in terms of a characteristic function Fourier series Limiting case where N x Conclusions Comparison between characteristic function and moments techniques Comparison with the Parzen estimator and Gaussian mixture model (GMM) 59 3 Novel cumulative distribution function estimators 3.1 Introduction 3.2 Motivation. 3.3 Definitions and background 3.4 Estimators based on moments A CDF in terms of moments Numerical integration techniques Fourier series approximation Estimators based on the characteristic function A CDF in terms of a characteristic function Fourier series 3.6 Conclusions... 4 Experimental results 4.1 Introduction Experimental setup Input data Estimation error measure PDF and CDF estimate using moments PDF and CDF estimate using characteristic function VI

8 4.2.5 Parzen estimator Gaussian Mixture Model Mean estimation error PDF estimators CDF estimators Computational requirements PDF Estimators CDF Estimators Training requirements Application to speaker verification Conclusions Conclusions and recommendations Conclusions Recommendations A A review of the Fourier transform 120 B Summary of algorithms 123 vii

9 List of Tables 4.1 PDF estimators: mllllmum mean estimation errors, minimum combined mean estimation errors and corresponding combined standard deviations (in brackets) from 100 samples (loox Kullback-Leibler divergence is indicated) PDF estimators: minimum mean estimation errors, minimum combined mean estimation errors and corresponding combined standard deviations (in brackets) from 1000 samples (loox Kullback-Leibler divergence is indicated) PDF estimators: the effect of selecting a sub-optimal working point CDF estimators: minimum mean estimation errors, minimum combined mean estimation errors and corresponding combined standard deviation (in brackets) from 100 samples (lox integral absolute difference is indicated) CDF estimators: minimum mean estimation errors, minimum combined mean estimation errors and corresponding combined standard deviation (in brackets) from 1000 samples (lox integral absolute difference is indicated) Gradient (xl00) characterising the relationship between the computation time and the parameter value Comparison between GMM and CF technique in a speaker verification application Feature matrix for all the estimators. 113 A.l Selected Fourier transform properties.. B.l PDF estimate from moments using Fourier series. B.2 PDF estimate from samples using Fourier series.. B.3 CDF estimate from moments using Fourier series. B.4 CDF estimate from samples using Fourier series Vlll

10 List of Figures 2.1 Successive Taylor series approximations to characteristic function of Gaussian PDF using 5, 14, and 32 terms The smoothing effect of a Hamming window on the characteristic function Leakage introduced by rectangular windowing of the characteristic function Discretisation of the characteristic function Reconstructing a PDF directly from data samples, using a triangular windowing function Relationship between f(x), f'(x)and f"(x) Mean estimation error: PDF estimate using GMM from 100 samples Mean estimation error: PDF estimate using Parzen estimator from 100 samples Mean estimation error: PDF estimate using characteristic function from 100 samples Mean estimation error: PDF estimate using moments from 100 samples Typical PDF estimates Examples of over-fitted PDF estimates (obtained from 100 samples) Mean estimation error: CDF estimate using GMM from 100 samples Mean estimation error: CDF estimate using Parzen estimator from 100 samples Mean estimation error: CDF estimate using characteristic function from 100 samples Mean estimation error: CDF estimate using moments from 100 samples Typical CDF estimates Comparison of normalised execution times of PDF estimators: Pentium III 700 MHz 104 ix

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

Oplos van kwadratiese vergelykings: die vind van die vergelyking *

OpenStax-CNX module: m39143 1 Oplos van kwadratiese vergelykings: die vind van die vergelyking * Free High School Science Texts Project Based on Solving quadratic equations: nding the equation by Free