Rate-Distortion Theory of Finite Point Processes

Transcription

1 Rate-Distortion Theory of Finite Point Processes Günther Koiander, Dominic Schuhmacher, and Franz Hawatsch, Feow, IEEE Abstract We study the compression of data in the case where the usefu information is contained in a set rather than a vector, i.e., the ordering of the data points is irreevant and the number of data points is unnown. Our anaysis is based on rate-distortion theory and the theory of finite point processes. We introduce fundamenta information-theoretic concepts and quantities for point processes and present genera ower and upper bounds on the rate-distortion function. To enabe a comparison with the vector setting, we concretize our bounds for point processes of fixed cardinaity. In particuar, we anayze a fixed number of unordered Gaussian data points and show that we can significanty reduce the required rates compared to the best possibe compression strategy for Gaussian vectors. As an exampe of point processes with variabe cardinaity, we study the best possibe compression of Poisson point processes. For the specific case of a Poisson point process with uniform intensity on the unit square, our ower and upper bounds are separated by ony a sma gap and thus provide a good characterization of the rate-distortion function. Index Terms Source coding, data compression, point process, rate-distortion theory, Shannon ower bound. I. INTRODUCTION The continuing growth of the amount of data to be stored and anayzed in many appications cas for efficient methods for representing and compressing arge data records []. In the iterature on data compression, one aspect was hardy considered: the fact that we are often interested in sets and not in ordered ists, i.e., vectors, of data points. Our goa in this paper is to study the optima compression of finite sets, aso caed point patterns, in an information-theoretic framewor. More specificay, we consider a sequence of independent and identicay distributed i.i.d. point patterns, and we want to cacuate the minima rate i.e., number of representation bits per point pattern for a given upper bound on the expected distortion. For this anaysis, we need distributions on point patterns. Fortunatey, these and other reevant mathematica toos are provided by the theory of finite point processes PPs [], [3]. The theory and appications of PPs have a ong history, and in most fieds using this concept such as, e.g., forestry [4], epidemioogy [5], and astronomy [6] significant amounts of data in the form of point patterns are coected, stored, and G. Koiander is with the Acoustics Research Institute, Austrian Academy of Sciences, 040 Vienna, Austria e-mai: goiander@fs.oeaw.ac.at. D. Schuhmacher is with the Institute of Mathematica Stochastics, Georg-August-Universität Göttingen, Göttingen, Germany e-mai: dominic.schuhmacher@mathemati.uni-goettingen.de. F. Hawatsch is with the Institute of Teecommunications, TU Wien, 040 Vienna, Austria e-mai: franz.hawatsch@nt.tuwien.ac.at. This wor was supported by the FWF under grant P7370-N30 and by the WWTF under grant MA Copyright c 08 IEEE. Persona use of this materia is permitted. However, permission to use this materia for any other purposes must be obtained from the IEEE by sending a request to pubs-permissions@ieee.org. processed. Thus, we beieve that ossy source coding may be of great interest in these fieds. Furthermore, the recenty studied probem of super-resoution [7], [8] or more generay atomic norm minimization [9] resuts in a point pattern in a continuous aphabet and is often described by some statistica properties. In this setting, one frequenty deas with noisy signas, and thus an additiona distortion resuting from ossy compression may be acceptabe. As a more expicit exampe, consider a database of minutiae patterns in fingerprints [0], []. Minutiae are endpoints and bifurcations of ridge ines on the finger. Typica data consists of x- and y-positions of points in some reative coordinate system and may we incude further information such as the ange of the orientation fied at the minutiae or the minutia types. For simpicity, we consider here ony the positions; any additiona information can be incorporated by a suitabe adaptation of the distortion measure. A fingerprint of good quaity typicay contains about minutiae [0]. For many minutia-based agorithms for fingerprint matching, the order in which the minutiae are stored is irreevant [] and a fingerprint can thus be represented as a point pattern. Furthermore, different pressures appied during the acquisition of a fingerprint ead to varying oca deformations and thus varying minutiae for the same finger. Hence, in most appications, a sma additiona distortion due to compression wi be acceptabe. Because the exact ocations as we as the number of minutiae acquired for the same finger may vary, the squared OSPA metric as defined further beow in 7 appears we suited for measuring the distortion between minutiae patterns. A. Prior Wor Information-theoretic wor on PPs is scarce. An extension of entropy to PPs is avaiabe [3, Sec. 4.8], but apparenty the mutua information between PPs was never anayzed in detai athough it is defined by the genera quantizationbased definition of mutua information [, eq. 8.54] or its equivaent form in 5 beow. A simiar quantity was recenty considered for a specia case in [3, Th. VI.]. However, this quantity deviates from the genera definition of mutua information, because the joint distribution in [3, eq. 5] impies a fixed association between the points in the two PPs invoved. Source coding resuts for PPs are avaiabe amost excusivey for infinite Poisson PPs on R [4] [7]. However, that setting considers ony a singe PP rather than an i.i.d. sequence of PPs. More specificay, the sequence considered for ratedistortion RD anaysis in [4] [7] is the growing vector of the smaest n points of the PP. This approach was aso adopted in [8], where the motivation was simiar to that of the present paper but the main objective was to study the asymptotic behavior of the RD function as the cardinaity of the data set

2 grows infinite. It was shown in [8] that the expected distortion divided by the number of points in the data set converges to zero even for zero-rate coding. Athough we are interested in the nonasymptotic scenario, the motivation given in [8] and the fact that the per-eement distortion increases significanty ess fast than in the vector case are of reevance to our wor. In channe coding, PPs were used in optica communications [9], [0] and for genera timing channes []. However, the PPs considered are again on R and in most cases Poisson PPs. A different source coding setting for point patterns was presented in [], [3]. There, the goa was not to reconstruct the points, but to find a covering consisting of intervas of a points. There is a tradeoff between the description ength of the covering set and its Lebesgue measure, both of which are desired to be as sma as possibe. For discrete aphabets, an agorithm compressing mutisets was presented in [4]. However, from an information-theoretic viewpoint, the coection of a muti-sets in a discrete aphabet is just another discrete set and thus sufficienty addressed by the standard theory for discrete sources. To the best of our nowedge, the RD function for i.i.d. sequences of PPs has not been studied previousy. In fu generaity, such a study requires the definition of a distortion function between sets of possiby different cardinaity. A pertinent and convenient definition of a distortion function between point patterns was proposed in [5] in the context of target tracing see 7 beow. B. Contribution and Paper Organization In this paper, we are interested in ossy compression of i.i.d. sequences of PPs of possiby varying cardinaity. We obtain bounds on the RD function in a genera setting and anayze the benefits that a set-theoretic viewpoint provides over a vector setting. Our resuts and methods are based on the measuretheoretic fundamentas of RD theory [6]. As the information-theoretic anaysis of PPs is not we estabished, we present expressions of the mutua information between dependent PPs, which can be used in upper bounds on the RD function. Our main contribution is the estabishment of upper and ower bounds on the RD function of finite PPs on R d. The upper bounds are based either on the RD theorem for memoryess sources [6] or on codeboos constructed by a variant of the Linde-Buzo-Gray agorithm [7]. The ower bounds are based on the characterization of mutua information as a supremum [8], which is cosey reated to the Shannon ower bound [9, eq ]. To iustrate our resuts, we compare the setting of a PP of fixed cardinaity with that of a vector of the same dimension and find that the RD function in the PP setting is significanty ower. Furthermore, we concretize our bounds for Poisson PPs and, in particuar, consider a Poisson PP on the unit square in R, for which our bounds convey an accurate characterization of the RD function. The paper is organized as foows. In Section II, we present some fundamentas of PP theory. In particuar, we introduce pairs of dependent PPs, which are reevant to an informationtheoretic anaysis but not common in the statistica iterature. In Section III, the mutua information between PPs is studied in detai, and some toos from measure-theoretic RD theory that can be used in the anaysis of PPs are introduced. In Sections IV and V, we present ower and upper bounds on the RD function of PPs in a genera setting. These bounds are appied to PPs of fixed cardinaity in Section VI and to Poisson PPs in Section VII. In Section VIII, we summarize our resuts and suggest future research directions. C. Notation Bodface owercase etters denote vectors. A vector x x T x T T R d with x i R d wi often be denoted as x,..., x or, more compacty, as x :. Sets are denoted by capita etters, e.g., A. The set A + x is defined as {y + x : y A}. The compement of a set A is denoted as A c, and the cardinaity as A. The indicator function A is given by A x if x A and A x 0 if x / A. The Cartesian product A A A of sets A i, i,..., is denoted as i A i. Sets of sets are denoted by caigraphic etters e.g., A. Mutisets, i.e., sets with not necessariy distinct eements, are distinguished from sets in that we use A, B, C to denote sets and,, Z to denote mutisets. For a set A and a mutiset, we denote by A the mutiset {x : x A}, which conforms to the cassica intersection if is a set but contains x A more than once if contains x more than once. Simiary, the cardinaity of a mutiset gives the tota number of the not necessariy distinct eements in. The set of nonnegative integers {0} N is denoted as N 0, the set of positive rea numbers as R +, and the set of nonnegative rea numbers as R 0. Sans serif etters denote random quantities, e.g., x is a random vector and is a random mutiset or PP. We write E x [ ] for the expectation operator with respect to the random variabe x and simpy E[ ] for the expectation operator with respect to a random variabes in the argument. Pr[x A] denotes the probabiity that x A. L d denotes the d-dimensiona Lebesgue measure and B d the Bore σ-agebra on R d. For measures µ, ν on the same measurabe space, µ ν means that µ is absoutey continuous with respect to ν, i.e., that νa 0 impies µa 0 for any measurabe set A. A random vector x on R d is understood to be measurabe with respect to B d. The differentia entropy of a continuous random vector x with probabiity density function g is denoted as hx or hg, and the entropy of a discrete random variabe x is denoted as Hx. The ogarithm to the base e is denoted og. For a function f : A B and a set C B, f C denotes the inverse image {x A : fx C}. Finay, we indicate by, e.g., 4 that the equaity hods due to 4. II. POINT PROCESSES AS RANDOM SETS OF VECTORS In this section, we present basic definitions and resuts from PP theory. In the cassica iterature on this subject, PPs are defined as random counting measures [3, Def. 9..VI]. Athough this approach is very genera and mathematicay eegant, we wi use a more appied viewpoint and interpret PPs as random mutisets, i.e., coections of a random number of random vectors that are not necessariy distinct. These mutisets are assumed to be finite in the sense that they have a finite cardinaity with probabiity one.

3 3 Definition : A point pattern on R d is defined as a finite mutiset R d, i.e., <. The coection of a point patterns on R d is denoted as N. Our goa is to compress point patterns under certain constraints imiting an expected distortion. To this end, we have to define random eements on N and, in turn, a σ-agebra. Definition : We denote by S the σ-agebra on N generated by the coections of mutisets N B { N : B } for a B B d and a N 0. A. Finite Point Processes The random variabes on N, S are caed finite spatia PPs on R d, hereafter simpy referred to as PPs. Foowing [, Sec. 5.3], a PP can be constructed by three steps: Let be a discrete random variabe on N 0 with probabiity mass function p. For each N, et x be a random vector on R d with probabiity measure P and the foowing symmetry property: x x,..., x with x i R d has the same distribution as x,..., x for any permutation on {,..., }. 3 The random variabe is defined by first choosing a reaization of the random cardinaity according to p. Then, for 0, a reaization x : x,..., x of x is chosen according to P, and this reaization is converted to a point pattern via the mapping φ : R d N ; x : {x,..., x }. For 0, we set. More compacty, this procedure corresponds to constructing as { if 0 φ x if. Remar 3: In principe, it is not necessary to start with symmetric random vectors x. Indeed, the mapping φ erases any order information the vector x might have, and thus we woud obtain a PP even for nonsymmetric x. However, for our information-theoretic anaysis, it wi turn out to be usefu to have access to the symmetric random vectors x and the symmetric probabiity measures P. Note that this does not impy a restriction on the PPs we consider, as any random vector can be symmetrized before using it in the PP construction. The probabiity measure on N, S induced by is denoted as P and satisfies P A Pr[ A] p 0 A + p P φ A for any measurabe set A N i.e., A S. This construction indeed resuts in a measurabe [3, Ch. 9]. According to, an integra with respect to P or, equivaenty, an expectation with respect to can be cacuated as g dp E[g] N p 0g + p 0g + p p E [ g φ x ] gφ x : dp R d x : for any integrabe function g : N R. In particuar, by 3 with g A g, we obtain g dp p 0 A g + p A for any A S. B. Pairs of Point Processes 3 gφ x : dp φ A x : 4 For information-theoretic considerations, it is convenient to have a simpe definition of the joint distribution of two PPs. Thus, simiar to the construction of, we define a pair of generay dependent PPs, as random eements on the product space N N as foows. Let, be a discrete random variabe on N 0 N 0 N 0 with probabiity mass function p,. For each, N 0 \{0, 0}, et x, y,, be a random vector on R d + with probabiity measure P,, and the foowing symmetry property: x, y,, x,..., x, y,..., y with x i, y j R d has the same distribution as x,..., x, y,..., y for any permutations on {,..., } and on {,..., }. Note that for the cases 0 and 0, we have x, y 0,, y,..., y and x, y,0 x,..., x, respectivey., 3 The random variabe, is defined by first choosing a reaization, of the random cardinaities, according to p,. Then, for,, with 0 or 0, a reaization x :, y : is chosen accord- x,..., x, y,..., y of x, y,, ing to P,,, and this reaization is converted to a pair of point patterns via the mapping φ, : R d + N ; x :, y : {x,..., x }, {y,..., y } 5 if, N, or φ 0, : R d N ; y :, {y,..., y } 6 if 0 and N, or φ,0 : R d N ; x : {x,..., x }, 7 This expression can be shown by the standard measure-theoretic approach of defining an integra in turn for indicator functions, simpe functions, nonnegative measurabe functions, and finay a integrabe functions [, Sec. A.4].

4 4 if N and 0. For, 0, 0, we set,,. More compacty, the overa procedure corresponds to constructing, as, if, 0, 0,, φ, x, y, if,, 0, 0. As we wi often use inverse images of the mapping φ, in our proofs, we state some properties of φ, A for A N in Appendix A. The probabiity measure on N, S S induced by, wi be denoted as P, and satisfies P, A p, 0, 0 A, + p,, P,, φ, A 8, N 0, 0,0 for any measurabe A N i.e., A S S. An integra with respect to P, or, equivaenty, an expectation with respect to, can be cacuated as g, dp,, E[g, ] N p, 0, 0g, + p,, E [ g,] φ, x, y,, N 0, 0,0 p, 0, 0g, +, N 0, 0,0 p,, g φ, x :, y : dp,, x :, y : 9 R d + for any integrabe function g : N R. As in the singe-pp case, g, A, g, resuts in an integra expression simiar to 4 for any measurabe set A N. The symmetry of the random vectors x, y,, impies that the corresponding probabiity measures P,, are symmetric in the foowing sense. Let and be permutations on {,..., } and {,..., }, respectivey, and define ψ, : R d + R d + ; x :, y : x,..., x, y,..., y. 0 Then, for any measurabe A R d + P,,, A P, ψ, A. We wi aso be interested in margina probabiities. For a pair of PPs,, the margina PP is defined by the probabiity measure P A P, AN for a measurabe sets A N. The corresponding probabiity measures P for N satisfy p P B p,, 0P,0, B + N p,, P,, B R d for Bore sets B R d, where for N 0 p N 0 p,,. 3 The definition of the margina PP is anaogous. We caution that the probabiity measures P and P are in genera not the marginas of P,,. Indeed, P depends on P,, for a N 0 with p,, 0 and, simiary, P depends on P,, for a N 0 with p,, 0. In particuar, the probabiity measures of the marginas x,, and y,, of x, y,, generay are not equa to P and P, respectivey. We wi often consider the case of i.i.d. PPs. Two PPs and are independent if P, P P, i.e., Pr[, A A ] Pr[ A ] Pr[ A ] for a A, A S. Furthermore, and are identicay distributed if their measures P and P are equa. A definitions and resuts in this subsection can be readiy generaized to more than two PPs. In particuar, we wi consider sequences of i.i.d. PPs in Section III-D. C. Point Processes of Fixed Cardinaity There are two major differences between spatia PPs and random vectors: first, the number of eements in a point pattern is a random quantity whereas the dimension of a random vector is deterministic; second, there is no inherent order of the eements of a point pattern. PPs of fixed cardinaity differ from random vectors ony by the second property. More specificay, we say that a PP is of fixed cardinaity if p Pr[ ] for some given N we do not consider the trivia case 0. The set of a possibe reaizations of is denoted as N, i.e., N { N : }. The probabiity measure P for a PP of fixed cardinaity simpifies to cf. P A P φ A, i.e., it is simpy the induced measure of P under the mapping φ. Simiary, a pair of PPs, is caed of fixed cardinaity,, if p,, for some given, N, i.e., Pr[ ] and Pr[ ]. The corresponding probabiity measure P, satisfies cf. 8 P, A P,, φ, A. Because p,, 0 for,,, 3 impies p p,, and, simiary, p p,,. Thus, simpifies to P, B P, B R d for Bore sets B R d. Anaogousy, we obtain P, B P, R d B for Bore sets B R d. Hence, the probabiity measures of the marginas x,, and y,, of x, y,, are given by P and, respectivey. P D. Point Processes of Equa Cardinaity A setting of particuar interest to our study are pairs of PPs that have equa but not necessariy fixed cardinaity. More specificay, we say that a pair of PPs, has equa cardi-

5 5 naity if p,, 0 for, i.e., Pr[ ]. The corresponding probabiity measure P, satisfies cf. 8 P, A p, 0, 0 A, + p,, P,, φ, A. A significant simpification can be observed for the margina probabiities. Because p,, 0 for, 3 impies p p,,, and simpifies to P B P,, B R d for N. Thus, the probabiity measure of the margina x,, of x, y,, is given by P and we wi write more compacty x x,,. Anaogousy, we define y y,,, and thus can rewrite x, y,, as x, y,, x, y. 4 III. MUTUAL INFORMATION AND RATE-DISTORTION FUNCTION FOR POINT PROCESSES Mutua information is a genera concept that can be appied to arbitrary probabiity spaces athough it is most commony used for continuous or discrete random vectors. To obtain an intuition about the mutua information between PPs, we wi anayze severa specia settings that wi aso be reevant ater. The basic definition of mutua information is for discrete random variabes [, eq..8] and readiy extended to arbitrary random variabes by quantization [, eq. 8.54]. By the Gefand-agom-Perez theorem [30, Lem. 5..3], mutua information can be expressed in terms of a Radon-Niodym derivative: for two random variabes and on the same probabiity space, dp, I; og dp P, dp,, 5 if P, P P and I; ese. A. Genera Expression of Mutua Information Using 5, we can express the mutua information between PPs as a sum of Kubac-Leiber divergences KLDs. We reca that the KLD between two probabiity measures µ and ν on the same measurabe space Ω is given as [3, Sec..3] { D KL µ ν Ω og dµ dν x dµx if µ ν 6 ese. As a preiminary resut, we present a characterization of the dp Radon-Niodym derivative, dp P for a pair of PPs,. A proof is given in Appendix B. Lemma 4: Let, be a pair of PPs. The foowing two properties are equivaent: i P, P P ; ii For a, N such that p,, 0, we have P,, P p,, 0 0, we have P,0, P ; for a N such that P ; and for a N such that p, 0, 0, we have P 0,, P. We wi use 5 mainy for PPs and thus use the notation of PPs. However, it is aso vaid for random vectors. Furthermore, if the equivaent properties i and ii hod, then dp, dp P θ, where θ, : N R 0 satisfies 3 θ,, p, 0, 0 p 0p 0 θ, φ x :, p,, 0 p p 0 θ,, φ y : p, 0, p 0p θ, φ x :, φ y : p,, p p d P dp,, P 7a dp,0, x dp : 7b dp 0,, y dp : 7c x :, y :. 7d Here, the right-hand sides of 7 are understood to be zero if p,, 0. Using Lemma 4, we can decompose the mutua information between PPs into KLDs between measures associated with random vectors. The foowing theorem is proved in Appendix C. Theorem 5: The mutua information I; for a pair of PPs, is given by I; I ; +,0 p,, 0D KL P, P + 0, p, 0, D KL P, P N +, p,, D KL P, P P. N 8, Note that in genera D KL P, P P cannot be represented as a mutua information because the probabiity measures P and P are not the marginas of P,,. However, for a pair of PPs of fixed cardinaity or of equa cardinaity, a representation as mutua information is possibe. B. Mutua Information for Point Processes of Fixed Cardinaity For a pair of PPs, of fixed cardinaity, i.e., p,, for some, N see Section II-C, we can reate the mutua information to the mutua information between random vectors. Indeed, since P,, the probabiity measures of x, y,, and y,,, respectivey, we obtain D KL, P, and P are and its marginas, x, P,, P, P 3 Note that the functions φ are not one-to-one and thus, e.g., for x : x : with φ x : φ x :, 7b might seem to give contradictory vaues for θ,,. However, due to our symmetry assumptions on P,,, P, and P see Sections II-A and II-B, a Radon-Niodym derivatives on the right-hand side of 7 can be chosen symmetric and thus the vaues of θ, given in 7 are consistent.

6 6 I x,, ; y,,. Inserting this into 8 whie recaing that p,, 0 for,, and noting that I ; 0, Theorem 5 simpifies significanty. Coroary 6: Let, be a pair of PPs of fixed cardinaity,, for some, N. Then I; I x,, ; y,,. We can aso start with an arbitrary random vector x, y on R d + without assuming any symmetry properties. In that case, the mutua information between x and y cannot be competey described by the associated pair of PPs φ x, φ y but we aso have to consider random permutations, i.e., discrete random variabes t x and t y that specify the order of the vectors in x and y, respectivey. More specificay, for a point pattern {x,..., x } where the indices are chosen according to a predefined tota order e.g., exicographica of the eements, a permutation specifies the vector 4 x,..., x R d. Using this convention, the random vector x can be equivaenty represented by the associated PP φ x and a random permutation t x specifying the order of the eements reative to the predefined tota order, i.e., x t x φ x. Appying further the tie-brea rue that t x i < t x j if x i x j and i < j, there is a one-to-one reation between the random vector x and the pair φ x, t x. Simiary, we can represent y by the pair φ y, t y. This eads to the foowing expression of the mutua information between PPs of fixed cardinaity. Lemma 7: Let, be a pair of PPs of fixed cardinaity,, for some, N. Furthermore, et x, y be a random vector on R d + such that, has the same distribution as φ x, φ y. Then I; I x ; y I t x ; φ y φ x I x ; t y φ y 9 I x ; y I t x ; φ y φ x I φ x ; t y φ y I t x ; t y φ x, φ y 0 where t x and t y are the random permutations associated with the vectors in x and y, respectivey. Proof: Due to the one-to-one reation between x and φ x, t x, and between y and φ y, t y, we have I x ; y I φ x, t x ; φ y, t y. Using the chain rue for mutua information [30, Cor ] three times, we thus obtain I x ; y I φ x ; φ y + I t x ; φ y φ x + I x ; t y φ y I φ x ; φ y + I t x ; φ y φ x + I φ x ; t y φ y + I t x ; t y φ x, φ y. 4 Here and in what foows, we use the same symbo for both the permutation on {,..., } and the associated mapping : N R d, and we refer to both as permutation. Because the distributions of, and φ x, φ y are equa, we have I; I φ x ; φ y. Hence, impies 9 and impies 0. C. Mutua Information for Point Processes of Equa Cardinaity If, is a pair of PPs of equa cardinaity see Section II-D, the mutua information I; sti simpifies significanty compared to the genera case. Coroary 8: Let, be a pair of PPs of equa cardinaity, i.e., p,, 0 for. Then I; H + p I x ; y. 3 Proof: We have and thus see [, eq..4] I ; H. Furthermore, we have p,, 0 for. Thus, by Theorem 5, we obtain I; H + p,, D KL P,, P P. According to 3, p p,,. Because P, P, and P,, are the probabiity measures of x, y,, and y, respectivey, 4 impies and its marginas, x, D KL P, P P I x ; y, which concudes the proof. As in the case of fixed cardinaity, we can start with arbitrary vectors x, y without assuming symmetry. Combining Coroary 8 with the expression of mutua information provided by Lemma 7, this approach yieds the foowing resut. Theorem 9: Let, be a pair of PPs of equa cardinaity, i.e., p,, 0 for. For every N, et x, y be random vectors such that, φ x, φ y and φ x, φ y have the same distribution. Then I; H + p I x ; y I t x ; I x ; t y 4 H + p I x ; y I t x ; I ; t y I t x ; t y, 5 where t x and t y are the random permutations associated with the vectors in x and y, respectivey. Proof: Because the distributions of, and φ x, φ y are equa, we have I ; I φ x ; φ y. Using this equaity and appying Lemma 7 to the pair of PPs,, we obtain I φ x ; φ y I x ; y I t x ; I x ; t y

7 7 I x ; y I t x ; I ; t y I t x ; t y,. On the other hand, appying Coroary 6 to the pair of PPs of fixed cardinaity φ x, φ y, we have I φ x ; φ y I x ; y. Combining these equaities and inserting into 3 concudes the proof. D. Rate-Distortion Function for Point Processes We summarize the main concepts of RD theory [, Sec. 0] in the PP setting. For two point patterns, N, et ρ: N N R 0 be a measurabe distortion function, i.e., ρ, quantifies the distortion incurred by changing to. A source generates i.i.d. copies [j], j N of a PP on R d. Loosey speaing, the RD function R [j]j N, ρd gives the smaest possibe encoding rate for maximum expected distortion D. In mathematica terms, R [j]j N, ρd is the infimum of a R > 0 such that for a ε > 0 there exists an n N and a source code, i.e., a measurabe mapping g n : N n N n, satisfying og g n N n nr and E [ n n j ρ [j], [j] ] D + ε, where [],..., [n] g n [],..., [n] N n. Foowing common practice, we wi write R [j]j N, ρd briefy as RD. Furthermore, we specify the source by ony one PP and tacity assume that [j] j N consists of i.i.d. PPs with the same distribution as. Remar 0: In the vector case, ρx, y is usuay defined based on x y, e.g., the squared-error distortion ρx, y x y. However, in the case of point patterns and, this convenient construction is not possibe because there is no meaningfu definition of as a difference between point patterns. This resuts in a significanty more invoved anaysis and construction of source codes. For simpicity, we wi assume ρ, 0 for a N. Moreover, we wi use some genera theorems for the characterization of RD functions, which can aso be appied to the setting of PPs. These theorems require that there exists a reference point pattern A N such that E[ρ, A ] <. This condition is satisfied, e.g., if the distortion between N and the empty set is a inear function of the cardinaity cf. 7, i.e., ρ, c, and the PP has finite expected cardinaity E[ ] <. The RD theorem for genera i.i.d. sources [6, Th and Th. 7..5] states that for a given source PP and distortion function ρ, the RD function can be cacuated as RD inf I ; 6,: E[ρ,] D where the infimum is taen over a pairs of PPs, such that has the same distribution as and E[ρ, ] D. The expression 6 is usefu for the derivation of upper bounds on the RD function see Section V. Another characterization of the RD function that is more usefu for the derivation of ower bounds see Section IV is [8, Th..3] RD max max og α s dp sd 7 s 0 α s >0 N where the inner maximization is over a positive functions α s : N R + satisfying α s e sρ, dp 8 RD max s 0 N for a N. Let us assume that the measures P are absoutey continuous with respect to L d with Radon-Niodym dp derivatives f dl d, i.e., the x are continuous random vectors. Then, 7 and 8 can equivaenty be written as foows. Using 3 with g og α s, 7 becomes max p 0 og α s + p α s >0 og α s φ x : f R d x : dx : sd 9 where dx : is short for dl d x : and the inner maximization is over a positive functions α s : N R + satisfying using 3 with g α s e sρ, in 8 p 0α s e sρ, + p α s φ x : e sρφ x :, f x : dx : R d for a N. IV. LOWER BOUNDS 30 Lower bounds on the RD function are notoriousy hard to obtain. The ony we-estabished ower bound is the Shannon ower bound, which is based on the characterization of the RD function given in 7, 8. More specificay, by omitting in 7 the maximization over α s and using any specific positive function α s satisfying 8 yieds the ower bound RD max og α s dp sd. s 0 N The standard approach [9, Sec. 4] is to set α s f γs, where f dp dq is the Radon-Niodym derivative of P with respect to some bacground measure Q on the given measurabe space N, S that satisfies P Q, and γs, s 0 is a suitaby chosen function. In this standard approach, γs is chosen independenty of the cardinaity of, which is too restrictive for the construction of usefu ower bounds for PPs. Hence, we tae a sighty different approach and define α s f γ s with appropriate functions γ s that depend on. More specificay, we propose the foowing bound. Theorem : Let be a PP on R d and assume that for a N, the measures P are absoutey continuous with respect to L d dp with Radon-Niodym derivatives f dl d, i.e., x are continuous random vectors with probabiity density functions f. For any measurabe sets A R d

8 8 satisfying P A, i.e., f x : 0 for L d - amost a x : A c, the RD function is ower-bounded according to RD p h f + max p og γ s sd 3 s 0 0 where γ are any functions satisfying { e sρ, if 0 γ s A e sρφ x :, dx : if N for a N and s 0. RD max s 0 3 Proof: The characterization of the RD function in 9 impies that for any α s satisfying 30, p 0 og α s + p a max s 0 og α s φ x : f R d p 0 og α s + p og α s φ x : f A x : dx : sd x : dx : sd 33 where a hods because we assumed that f x : 0 for L d -amost a x : A c. Using functions γ satisfying 3, we construct α s as if f f α s φ x : x :γ s x : 0 34 if f x : 0 for x : R d and Due to 3, the functions γ satisfy α s γ 0 s. 35 p 0 γ 0 s e sρ, + p A γ s e sρφ x :, dx : p 0 + p for a N, which is recognized as the condition 30 evauated for the functions α s given by 34 and 35. Inserting 34 and 35 into 33 gives RD max s 0 p 0 og f A γ 0 s + x : og p f x :γ s dx : sd max p 0 og γ 0 s s 0 + p h f which is equivaent to 3. og γ s sd V. UPPER BOUNDS We wi use two different approaches to cacuate upper bounds on the RD function. The first is based on the RD theorem, i.e., expression 6, whereas the second uses concrete codes and the operationa interpretation of the RD function. A. Upper Bounds Based on the Rate-Distortion Theorem Let be a PP defined by the cardinaity distribution p and the random vectors x see Section II-A. To cacuate upper bounds, we can construct an arbitrary pair of PPs, see Section II-B such that has the same distribution as and E[ρ, ] D. According to 6, we then have RD I ;. However, it is often easier to construct vectors x, y that do not satisfy the symmetry properties we assumed in the construction of pairs of PPs in Section II-B. The foowing coroary to Theorem 9 shows that in the case where x is a symmetrized version of x, we can construct upper bounds on RD based on x, y. Coroary : Let be a PP on R d defined by the cardinaity distribution p and the random vectors x. Furthermore, for each N, et x, y be a random vector on R d such that φ x has the same distribution as φ x. Finay, assume that p E [ ρ, ] D 36 with φ y. Then RD H + p I x ; y I t x ; I x ; t y 37 H + p I ; t y I t x I x ; y I t x ; t y, ; where t x and t y are the random permutations associated with the vectors in x and y, respectivey. Proof: We construct a pair of PPs, of equa cardinaity. First, we define the cardinaity distribution as p,, 0 for and p,, p for N 0. Next, we define the random vectors x, y, φ x, φ y and φ x such that, φ y

9 9 have the same distribution. By 4 and 5, we then obtain for the pair of PPs, I ; H + p I x ; y I t x ; I x ; t y 38 H + p I x ; y I t x ; I ; t y I t x ; t y,. 39 Furthermore, because has the same distribution as φ x, the construction of, impies that φ x has the same distribution as φ x too, and, in turn, the PPs and have the same distribution. Since 36 impies E[ρ, ] p E [ ρ, ] D, we obtain by 6 that RD I ;, which in combination with 38 and 39 concudes the proof. B. Codeboo-Based Upper Bounds It is we nown [, Sec. 0.] that the RD function for a given PP can be easiy upper-bounded based on its operationa interpretation if we are abe to construct good source codes. Let be a PP and assume that there exists a source code g : N N such that gn M. If E[ρ, g] D, then the RD function at D satisfies R D og M. 40 The construction of good source codes, even in the vector case, is a difficut optimization tas. In the case of PPs, this tas is further compicated by the absence of a meaningfu vector space structure of sets; even the definition of a mean of point patterns is not straightforward. Our construction is motivated by the Loyd agorithm [3], which, for a given number M of codewords i.e., eements in gn, aternatey finds M centers j N and constructs an associated partition {A j } j,...,m of N. The resuting centers,..., M can be used as codewords and the associated source code g is defined as g : N N ; arg min ρ, j. j {,..., M } That is, a point pattern N is encoded into the center point pattern j that is cosest to in the sense of minimizing ρ, j. In our setting, the Loyd agorithm can be formaized as foows: Input: PP ; distortion function ρ: N N R 0 ; number M N of codewords. Initiaization: Draw M different initia codewords j N according to the distribution of. Step : Find a partition of N into M disjoint subsets A j such that the distortion incurred by changing A j to j is ess than or equa to the distortion incurred by changing to any other j, j j, i.e., ρ, j ρ, j for a j j and a A j. Step : For each j {,..., M}, find a new codeword associated with A j that has the smaest expected distortion from a point patterns in A j, i.e., a center point pattern j repacing the previous j satisfying j arg min N E Aj [ρ, ]. Repeat Step and Step unti some convergence criterion is satisfied. Output: codeboo {,..., M }. Unfortunatey, cosed-form soutions for Steps and do not exist in genera. A woraround is an approach nown in vector quantization as Linde-Buzo-Gray LBG agorithm [7]. Here, a codeboo is constructed based on a given set A of source reaizations. We can generate the set A by drawing i.i.d. sampes of. Adapted to our setting, the agorithm can be stated as foows: Input: a set A N containing A < point patterns; distortion function ρ: N N R 0 ; number M of codewords. Initiaization: Randomy choose M different initia codewords j A. Step : Find a partition of A into M disjoint subsets A j such that the distortion incurred by changing A j to j is ess than or equa to the distortion incurred by changing to any other j, j j, i.e., ρ, j ρ, j for a j j and a A j. Step : For each j {,..., M}, find a new codeword associated with A j that has the smaest average distortion from a point patterns in A j, i.e., a center point pattern j N repacing the previous j satisfying j arg min N A j A j ρ,. 4 Repeat Step and Step unti some convergence criterion is satisfied. Output: codeboo {,..., M }. Step can be performed by cacuating A M times a distortion ρ, j. However, Step is typicay computationay unfeasibe: in many cases, finding a center point pattern of a finite coection A j of point patterns according to 4 is equivaent to soving a muti-dimensiona assignment probem, which is nown to be NP-hard. Hence, we wi have to resort to approximate or heuristic soutions. Note that we do not have to sove the optimization probem exacty to obtain upper bounds on the RD function. We merey have to construct a source code that can be anayzed, no matter what heuristics or approximations were used in its construction. A convergence anaysis of the proposed agorithm appears to be difficut, as even the convergence behavior of the Loyd agorithm in R d is not competey understood [33], [34]. VI. POINT PROCESSES OF FIED CARDINALIT In this section, we present ower and upper bounds on the RD function for PPs of fixed cardinaity as discussed in Section II-C. We thus restrict our anaysis to source codes and distortion functions on the subset N { N : } N. The assumption of fixed cardinaity eads to more concrete bounds and enabes a comparison with the vector viewpoint.

10 0 A. Lower Bound For a PP of fixed cardinaity, the RD ower bound in Theorem simpifies as foows. Coroary 3: Let be a PP on R d of fixed cardinaity, i.e., p for some N. Assume that the measure P is absoutey continuous with respect to L d dp with Radon-Niodym derivative f. Then the RD dl d function is ower-bounded according to RD h f + max s 0 og γ s sd where γ is any function satisfying γ s e sρφ x :, dx : R d for a N. We can obtain a simper bound by considering a specific distortion function. For point patterns {x,..., x } and {y,..., y } of equa cardinaity, we define the distortion function as ρ, min x i y i 4 i where the minimum is taen over a permutations on {,..., }. This is a natura counterpart of the cassica squared-error distortion function of vectors. We note that a generaization to sets, of different cardinaities and the incusion of a normaization factor / eads to the squared optima subpattern assignment OSPA metric defined in [5]. The idea of the foowing ower bound is that a source code for PPs can be extended to a source code for vectors, by additionay specifying an ordering. Theorem 4: Let be a PP on R d of fixed cardinaity and et x be a random vector on R d such that φ x has the same distribution as. Then the RD function for and distortion function ρ is ower-bounded in terms of the RD function R vec for x and squared-error distortion according to RD R vec D og. 43 Proof: Let D > 0 be fixed. For any R > RD and ε > 0, the operationa definition of the RD function see Section III-D impies that there exists an n N and a source code g n : N n N n such that og g n N n nr and [ n ] E ρ [j], [j] D + ε 44 n j where [],..., [n] g n [],..., [n] N n and the [j] are i.i.d. copies of. We define a vector source code g vec,n : R d n R d n for sequences of ength n in R d by the foowing procedure. For a sequence x: [],..., x : [n] with x : [j] R d, we first map each vector x : [j] to the corresponding point pattern [j] φ x: [j] { x [j],..., x [j] }. Then we use the source code g n to obtain an encoded sequence of point patterns [],..., [n] gn [],..., [n]. Finay, we map each point pattern [j] { y [j],..., y [j] } to a vector y [j] [j],..., y [j] [j] by a permutation [j] such that the squared error i xi [j] y [j]i [j] is minimized, i.e., [j] arg min [j] i xi [j] y [j]i [j]. Based on this construction, the eements y [] [],..., y [] [],..., y [n] [n],..., y [n] [n] in the range of g vec,n are sequences in g n N n with the eements of each component [j] ordered according to some permutation [j]. As there are possibe orderings for each component, we have gvec,n R d n n g n N n n e nr e nr+og. Furthermore, for y: [],..., y : [n] g vec,n x: [],..., x : [n], we have n x : [j] y : [j] j 4 n j min [j] x i [j] y [j]i [j] i n ρ [j], [j] j and thus for y [],..., y [n] g vec,n x [],..., x [n], where the x [j] are i.i.d. copies of x, [ n E x [j] y [j] ] [ n ] E ρ [j], [j] n n j j 44 D + ε. Hence, for an arbitrary R R + og > RD + og and ε > 0, we constructed a source code g vec,n such that og g vec,n R d n n R and the expected average distortion between x [],..., x [n] and g vec,n x [],..., x [n] is ess than or equa to D + ε. According to the operationa definition of the RD function in Section III-D, we hence obtain R vec D RD + og. The offset og corresponds to the maxima information that a vector contains in addition to the information present in the set, i.e., the maxima information provided by the ordering of the eements. Indeed, the information content of the ordering is maxima if a of the possibe orderings are equay iey, in which case it is given by og. For D 0, the bound 43 shows that the asymptotic behavior of the RD function RD for sma distortions is simiar to the vector case, i.e., R vec D. In particuar, we expect that an anaysis of the RD dimension [35] of PPs can be based on 43 and the asymptotic tightness of the Shannon ower bound in the vector case [36]. On the other hand, 43 does not aow us to anayze the RD function for, as the resuting bounds quicy fa beow zero. Let us combine the bound 43 with the cassica Shannon ower bound for a random vector x with probabiity density function f x and squared-error distortion, which is given by [9, eq ] R vec D h f x d πd + og. 45 d In particuar, for a PP on R d of fixed cardinaity whose measure P is absoutey continuous with respect

11 dp dl d to L d with Radon-Niodym derivative f, setting x x, and combining 45 with Theorem 4 gives RD h f d πd + og og. 46 d The same resut can aso be obtained by concretizing Coroary 3 for the distortion function ρ. B. Upper Bound Based on the Rate-Distortion Theorem We can aso concretize the upper bounds from Section V for PPs of fixed cardinaity. Coroary becomes particuary simpe. Coroary 5: Let be a PP on R d of fixed cardinaity. Denote by x the associated symmetric random vector on R d. Furthermore, et x, y be any random vector on R d such that φ x has the same distribution as φ x. Finay, assume that E [ ρ φ x, φ y ] D. 47 Then the RD function for distortion function ρ, at distortion D, is upper-bounded according to RD I x ; y I t x ; φ y φ x I x ; t y φ y 48 where t x and t y are the random permutations associated with the vectors in x and y, respectivey. We can simpify 47 by using the foowing reation of ρ to the squared-error distortion of vectors: ρ φ x :, φ y : min x i y i i x i y i i x : y :. 49 Thus, E [ x y ] D impies 47. This shows that the upper bound I x ; y on the RD function of a random vector x based on an arbitrary random vector y satisfying E [ x y ] D, is aso an upper bound on the RD function of the corresponding fixed-cardinaity PP φ x. C. Codeboo-Based Upper Bounds We can aso obtain upper bounds by constructing expicit source codes using the variation of the LBG agorithm proposed in Section V-B. We wi specify the two iteration steps of that agorithm for point patterns of fixed cardinaity. In Step, for a given set A N of point patterns and M center point patterns j, we have to associate each point pattern A with the center point pattern j with minima distortion ρ, j. This requires an evauation of ρ, j for each A and each j {,..., M}. If severa distortions ρ, j are minima for a given, we choose the one with the smaest index j. A point patterns A that are associated with the center point pattern j are coected in the set A j. In Step, for each subset A j A, we have to find an updated center point pattern j of minima average distortion from a point patterns in A j, i.e., j arg min ρ, N A j. 50 A j We can reformuate 50 as the tas of finding an optima permutation corresponding to an ordering of each point pattern A j according to the foowing emma. A proof is given in Appendix D. Lemma 6: Let A j N be a finite coection of point patterns in R d of fixed cardinaity N, i.e., for a A j, we have { x,..., x } with x i R d. Then a center point pattern 5 j arg min N A j A j ρ, is given as j {x,..., x } with x i x A j i 5 A j where the coection of permutations { } A j { } Aj arg min { } Aj i A j A j is given by x i x i. 5 By Lemma 6, the minimization probem in 50 is equivaent to a muti-dimensiona assignment probem MDAP. Indeed, a coection of permutations { } Aj corresponds to a choice of ciques 6 C i { x i : A j}, i,...,. 53 Thus, for each point pattern A j, assigns each of the vectors in to one of different ciques, such that no two vectors in are assigned to the same cique. Each resuting cique hence contains C i A j vectors one from each A j and each vector x beongs to exacty one cique for a A j. Note that the union of a A j is the same as the union of a ciques C i, i.e., A j i C i. The reation between the ciques C i and the point patterns A j is iustrated in Figure. We define the cost of a cique C i as the sum x x x i x i. x C i x C i A j A j 54 Finding the coection of ciques {Ci } i,..., with minima sum cost, i.e., {C i } i,..., arg min {C i} i,..., i x C i x C i x x 55 is then equivaent to finding the optima coection of permutations in 5. Moreover, according to its definition in 54, the cost of a cique can be decomposed into a sum of 5 The center point pattern is not necessariy unique. 6 An MDAP can aso be formuated as a graph-theoretic probem where a cique corresponds to a compete subgraph [37].

12 x y 4 z 4 x C z x 3 C y 3 C 3 y z z 3 Fig.. Ciques C i for A j {,, Z}, 4, and permutations { } A j {,, Z } with,, 3 3, 4 4; 3, 4, 3, 4 ; and Z 4, Z, Z 3, Z 4 3. Note that, e.g., 4 expresses the fact that y C 4. squared distances, each between two of its members. Thus, the minimization in 55 is an MDAP with decomposabe costs. Athough finding an exact soution to such an MDAP is unfeasibe for arge and arge cique sizes C i A j, there exist various heuristic agorithms producing approximate soutions [37] [39]. Because we are mainy interested in the case of a arge cique size, we wi merey use a variation of the basic singe-hub heuristic and the muti-hub heuristic presented in [37]. Since these heuristic agorithms are used in Step of the proposed LBG-type agorithm, we wi abe the corresponding steps as..3. The cassica singe-hub heuristic is based on assigning the vectors x i in each point pattern A j to the different ciques using a permutation by minimizing the sum of squared distances between the vectors x i and the vectors x i of one hub point pattern. More specificay, the agorithm corresponding to Step of the LBG agorithm is given as foows. Input: a coection A j of point patterns; each point pattern A j contains points in R d. Initiaization: Choose a point pattern A j caed the hub and define i i for i,...,. Step.: For each { x,..., x } Aj \ { }, find the best assignment between the vectors in and, i.e., a permutation x. 56 arg min i x 4 y C 4 i x i Step.: Define the cique C i according to 53, i.e., C i { x i : A j}, i,...,. 57 Step.3: Define the approximate center point pattern ˆ j as the union of the centers arithmetic means of a ciques, i.e., ˆ j } { xci with x Ci x. 58 C i i x C i Output: approximate center point pattern ˆ j. The above heuristic requires ony A j optima assignments between point patterns. However, the accuracy of the resuting approximate center point pattern ˆ j strongy depends on the choice of the hub and can be very poor for certain choices of. The more robust muti-hub heuristic [37] performs the singe-hub agorithm with a the A j as aternative hubs, which can be shown to increase the compexity to Aj Aj optima assignments, i.e., by a factor of A j /. We here propose a different heuristic that has amost the same compexity as the singe-hub heuristic but is more robust. The idea of our approach is to repace the best assignment to the singe hub by an optima assignment to the approximate center point pattern of the subsets defined in the preceding steps. More specificay, we start with a hub A j and, as in the singe-hub heuristic, search for the best assignment see 56 with between the vectors in a randomy chosen A j \ { } and. Then, we cacuate the center point pattern ˆ of and as in 57 and 58 but with A j repaced by {, }. In the next step, we choose a random 3 A j \ {, } and find the optima assignment 3 between the vectors in 3 and ˆ. An approximate center point pattern ˆ 3 of,, and 3 is then cacuated as in 57 and 58 but with A j repaced by {,, 3 }. Equivaenty, we can cacuate ˆ 3 as a weighted center point pattern of 3 and ˆ. We proceed in this way with a the point patterns in A j, aways cacuating the optima assignment r r 4, 5,... between the vectors in r and the approximate center point pattern ˆ r of the previous r point patterns. A forma statement of the agorithm is as foows. Input: a coection A j of point patterns; each point pattern A j contains points in R d. Initiaization: Randomy order the point patterns A j, i.e., choose a sequence,..., Aj where the r are a the eements of A j. Set the initia subset center point pattern ˆ to. For r,..., A j : Step.: Find the best assignment between the vectors in r and ˆ r, i.e., a permutation r arg min i x r i ˆ r x i. Step.: Generate an updated approximate subset center point pattern ˆ r { x ˆ r,..., x ˆ r } according to x ˆ r i r r s for i,...,. s i r ˆr x i r x s + x r r i Output: approximate center point pattern ˆ j ˆ Aj. 59 As in the case of the singe-hub heuristic, we ony have to perform A j optima assignments. The compexity of the additiona center update 59 is negigibe. On the other hand, the muti-hub agorithm can be easiy paraeized whereas our agorithm wors ony sequentiay. D. Exampe: Gaussian Distribution As an exampe, we consider the case of a PP on R d of fixed cardinaity whose points are independenty distributed according to a standard Gaussian distribution on R d,