On the relation between the relative earth mover distance and the variation distance (an exposition)

Transcription

1 On the relation between the relative earth mover distance and the variation distance (an eposition) Oded Goldreich Dana Ron Februar 9, 2016 Summar. In this note we present a proof that the variation distance up to relabeling is upperbounded b the relative earth mover distance (to be defined below). The relative earth mover distance was introduced b Valiant and Valiant [VV11], and was etensivel used in their work. The foregoing claim was made in [VV11], but was not used there. The claim appears a special case of [VV15, Fact 1] (i.e., the case of τ 0). The proof we present is merel an elaboration of (this special case of) the proof presented b Valiant and Valiant in [VV15, Apd A]. 1 Definitions We start b introducing some definitions and notations. Definition 1 (Histograms and relative histograms for distributions) For a distribution p : [n] [0,1], the corresponding histogram, denoted h p :[0,1] N, such that h p () def {i [n] : p(i) } for each [0,1]. The corresponding relative histogram, denoted h R p :[0,1] R, satisfies h R p () h p () for ever [0,1]. That is, h p () equals the number of elements in p that are assigned probabilit mass, whereas h R p () equals the total probabilit mass assigned to these elements. Hence, h p(0) ma be positive, whereas h R p (0) is alwas zero. For a non-negative function h, let S(h) def { : h() > 0} denote the support of h. Observe that for an distribution p : [n] [0,1] we have that S(h h p) p() n and S(h R p ) hr p () 1. Also note that S(h R p ) S(h p) \ {0}. The following definition interprets the distance between non-negative functions h and h as the cost of transforming h into h b moving m(,) units from in h to in h (for ever S(h) and S(h )), where the cost of moving a single unit from to is either or log(/) (depending on the distance). Partiall supported b the Israel Science Foundation (grant No. 671/13). Department of Computer Science, Weizmann Institute of Science, Rehovot, Israel. oded.goldreich@weizmann.ac.il Department of EE Sstems, Tel-Aviv Universit, Ramat-Aviv, Israel. danar@eng.tau.ac.il 1

2 Definition 2 (Earth-Mover Distance and Relative Earth-Mover Distance) For a pair of non-negative functions h and h over [0,1] such that S(h) h() S(h ) h (), the earth-mover distance between them, denoted EMD(h,h ), is the minimum of m(,), S(h) S(h ) taken over all non-negative functions m:s(h) S(h ) R that satisf: For ever S(h), it holds that S(h) m(,) h(), and For ever S(h ), it holds that S(h ) m(,) h (). The relative earth-mover distance between h and h, denoted REMD(h,h ), is the minimum of S(h) S(h ) subject to the same constraints on m as for EMD. m(,) log(/), The term earth-mover comes from viewing the functions as piles of earth, where for each S(h) there is a pile of size h() in location and similarl for each S(h ) there is a pile of size h () in location. The goal is to transform the piles defined b h so as to obtain the piles defined b h, with minimum transportation cost. Specificall, m(, ) captures the possibl fractional number of units transferred from pile in h to pile in h. For EMD the transportation cost of a unit from to is while for REMD it is log(/). In what follows, for a pair of distributions p and q over [n] we shall appl EMD to the corresponding pair of histograms h p and h q, and appl REMD to the corresponding relative histograms h R p and hr q. Variation distance up to relabeling, as defined net, is a natural notion in the contet of testing properties of smmetric distributions (i.e., properties that are invariant under relabeling of the elements of the distribution). Definition 3 (Variation Distance up to Relabeling) For two distributions p and q over n, the variation distance up to relabeling between p and q, denoted VDR(p,q), is the minimum over all permutations σ over [n] of 1 n p(i) q(σ(i)). 2 2 Proofs i1 Our goal is to present a proof of the following result. Theorem 4 (special case of Fact 1 in [VV15]) For ever two distributions p and q over [n], it holds that VDR(p,q) REMD(h R p,h R q ). The proof will consist of two steps (captured b lemmas): 2

3 1. VDR(p,q) 1 2 EMD(h p,h q ). 2. EMD(h p,h q ) 2 REMD(h R p,h R q ). Actuall, we start with the second step. Lemma 5 For ever two distributions p and q over [n], EMD(h p,h q ) 2 REMD(h R p,hr q ). The following proof shows how to construct, for ever transportation function m used for the relative histograms (h R p and h R q ) a corresponding transportation function m for the corresponding histograms (h p and h q ) such that the EMD cost of m is at most twice the REMD cost of m. Proof: It will be convenient to consider two distributions, p and q that are slight variations of p and q, respectivel. The are both defined over [2n], where p(i) p(i) and q(i) q(i) for ever i [n], and p(i) q(i) 0 for ever i [2n] \ [n]. Since h R ep hr p and hr eq hr q, we have that REMD(h R ep,hr eq ) REMD(hR p,hr q ). As for h ep and h eq, the agree with h p and h q, respectivel, everwhere ecept on 0, where h ep (0) h p (0) + n and h eq (0) h q (0) + n, so EMD(h ep,h eq ) EMD(h p,h q ) as well. Therefore, it suffices to show that EMD(h ep,h eq ) 2 REMD(h R ep,hr eq ). Let m be a function over S(h R ep ) S(hR eq ) that satisfies the constraints stated in Definition 2 for the pair of histograms h R ep and hr eq. We net show that there eists a non-negative function m over S(h ep ) S(h eq ) that satisfies the constraints stated in Definition 2 for the pair of histograms h ep and h eq, and also satisfies S(h ep ) S(h eq ) m(,) 2 S(h R ep ) S(h R eq ) log(/). (1) Note that the range of m is [0,1], since it is defined over relative histograms, while m is not upper bounded b 1. However, the constraints on the two functions are related since for ever S(h R ep ) S(h ep)\{0} it is required that S(h R eq ) m (,)/ h ep () S(h eq ) m(,) and for ever S(h R eq ) S(h eq) \ {0} it is required that S(h R ep ) m (,)/ h eq () S(h eq ) m(,). (Indeed, m is also subjected to constraints on 0 and 0, whereas m is not.) We now define the function m. For each S(h R ep ), initialize m(,0) to 0 and similarl for each S(h eq ), initialize m(0,) to 0. For ever pair (,) S(h R ep ) S(hR eq ), if m (,) 0, then m(,) 0, and otherwise we do the following. If >, let m(,) be set to / and increase m(0,) b m (0,) def / / > 0. Observe that m(,) ( ) (1 /) m (0,). Therefore, the contribution to the left-hand-side of Equation (1) is m(,) ( ) + m (0,) ( 0) 2 (1 /) < 2 log(/), where the last inequalit is due to the fact that f(z) log z + (1/z) 1 > lnz + (1/z) 1 is positive for all z > 1. 3

4 If <, let m(,) be set to / and increase m(,0) b m (,0) def / / > 0. Similarl to the previous case, m(,) ( ) m (,0), and the contribution to the left-hand-side of Equation (1) is m(,) ( ) + m (,0) ( 0) 2 (1 /) < 2 log(/). If, let m(,) / ( /). In this case both m(,) 0 and log(/) 0. Finall, we set m(0,0) h ep (0) S(h R ) m(0,). To see that m(0,0) 0, note that since eq h ep (0) n while m(0,) m (0,) / n. S(h R eq ) S(h R eq ) S(h R ep ) (,1] S(h R eq ) S(h R ep ) (,1] B combining the contribution of all pairs, as defined above, Equation (1) holds. It remains to verif that m satisfies the constraints in Definition 2. For each S(h ep ) \ {0}, m(,) m(,0) + m(,) + m(, ) S(h eq ) S(h eq ) (0,] S(h R eq ) (,1] m (,0) + S(h R eq ) (,1] S(h R eq ) ( 1 1 ) Similarl, for each S(h eq ) \ {0}, m(,) m(0,) + S(h ep ) S(h ep ) (0,] h ep (). S(h R eq ) (,1] m (0,) + S(h R eq ) (,1] S(h R ep ) ( 1 1 ) S(h eq ) (,1] S(h R eq ) (0,] m(,) + + m(,) + h eq (). S(h R eq ) (0,] S(h ep ) (,1] S(h R eq ) (0,] m(,) + + S(h R eq ) (0,] S(h R eq ) (,1] m(,) m(, ) + S(h R eq ) (,1] S(h R eq ) (,1] m(,) + S(h R eq ) (,1] We defined m(0,0) such that S(h eq ) m(0,) m(0,0) + S(h R eq ) m(0,) h ep(0), and S(h ep ) m(,0) S(h ep ) eq and the proof is completed. m(,) S(h ep ) S(h eq )\{0} m(,) 2n S(h eq ) h eq () h eq (0), 4

5 Lemma 6 For ever two distributions p and q over [n], VDR(p,q) 1 2 EMD(h p,h q ). Intuitivel, there is a one-to-one correspondence between integer-valued transportation functions m as in Definition 2 and the relabeling permutations σ used in Definition 3. The core of the following proof is showing that integer-value transportation functions m obtain the minimum for EMD. Proof: Consider a constrained version of the earth-mover distance in which we also require that m(,) is an integer for ever S(h p ) and S(h q ), and denote this distance measure b IEMD. Using the definition of VDR and IEMD, one can verif that VDR(q,p) 1 2 IEMD(h p,h q ), since there is a correspondence between the permutation σ used in Definition 3 and the integer movement in EMD. (The factor of 1/2 is due to the fact that the variation distance between distributions equals half the L 1 -norm between them.) It therefore remains to prove that EMD(h p,h q ) IEMD(h p,h q ); that is, the function m that obtains the minimum of the EMD objective function has integer values. To this end, we define a specific integer-valued function m (based on a simple iterative assignment procedure), and show that it is optimal. Initiall, m(,) 0 for ever S(h p ) and S(h q ). We also initialize s() h p () for ever S(h p ), and d() h q () for ever S(h q ). (Intuitivel, s() is the suppl of, and d() is the demand of.) Note that S(h s() n p) S(h q) d(). In each iteration, we consider the smallest S(h p ) for which s() > 0 and the smallest S(h q ) for which d() > 0, set m(, ) min{s(), d()} and reduce both s() and d() b m(, ). Hence, all intermediate values of m (as well as s and d) are integers. (We note that an equivalent definition of m can be obtained b considering the mapping σ from [n] to [n] that maps the i th smallest p-value to the i th smallest q-value.) 1 B its construction, the function m satisfies the constraints of Definition 2. To verif that the resulting function m is an optimal setting for EMD, consider an other non-negative function l over S(h p ) S(h q ) that satisfies the constraints of Definition 2. Actuall, among all such functions l consider onl those that agree with m on the longest prefi of pairs (,) according to the leicographical order on pairs, and let (, ) be the first pair on which l and m differ; that is, l(, ) m(, ) whereas l(,) m(,) for ever (,) < (, ). Furthermore, among all such functions l, select one for which l(, ) m(, ) is minimal. We shall show that l m. Assume towards the contradiction that l m, and let (, ) be as above. We first prove that l(, ) < m(, ). Towards this end, we consider the suppl of and the demand of just before m(, ) is determined; that is, s( ) h p ( ) < m(,) and d( ) h q ( ) < m(, ). Recalling that m(, ) min(s( ),d( )), we note that if l(, ) > m(, ) s( ), then l(,) < m(,) + l(, ) > h p ( ), which means that l violates a constraint of Definition 2. A similar contradiction is obtained b assuming that l(, ) > m(, ) d( ), when in this case we get l(, ) > h q ( ). Having shown that l(, ) < m(, ). we now derive a function l that violates the minimalit of l. Specificall, l(, ) < m(, ) (combined with l(,) m(,) for ever (,) < (, )) implies that there eists > such that l(, ) > m(, ) and > such that l(, ) > m(, ). Letting c min(m(,) l(, ),l(, ),l(, )) > 0, define 1 That is, letting π p and π q be permutations over [n] such that p(π p(i)) p(π p(i + 1)) and q(π q(i)) q(π q(i + 1)) for ever i [n 1], define σ(π p(i)) π q(i) for ever i [n]. 5

6 l as equal to l on all pairs ecept for the following four pairs that satisf l (, ) l(, ) + c, l (, ) l(, ) c, l (, ) l(, ) c, and l (, ) l(, ) + c. Then, l preserves the constraints of Definition 2, but l (,) l(,) m(,) for ever (,) < (, ) and l (, ) m(, ) l(, ) m(, ) c, in contradiction to the choice of l, since c > 0. 3 Comments As noted in [VV11], there eist distributions p and q for which VDR(h p,h q ) REMD(h R p,h R q ). The source of this phenomenon is the unbounded cost of transportation under the REMD (i.e., transforming a unit of mass from to costs log(/) ). For eample, for an ǫ [0,0.5], consider the pair (p,q) such that p is uniform over [n] (i.e., p(i) 1/n for ever i [n]) and q is etremel concentrated on a single point in the sense that q(n) 1 ǫ and q(i) ǫ/(n 1) for ever i [n 1]. Then, the variation distance between p and q is n 1 n ǫ, but the REMD is at least n 1 n log(1/ǫ). This phenomenon is reflected in the proof of Lemma 5 at the point we used the inequalit 1 (1/z) < log z for z > 1. This inequalit becomes more crude when z grows. References [VV11] G. Valiant and P. Valiant. Estimating the unseen: an n/ log(n)-sample estimator for entrop and support size, shown optimal via new CLTs. In Proceedings of the Fourt-Third Annual ACM Smposium on the Theor of Computing (STOC), pages , See ECCC TR for the algorithm, and TR for the lower bound. [VV15] G. Valiant and P. Valiant. Instance optimal learning. CoRR, abs/ ,