Proof of Proposition 1

Transcription

1 A Proofs of Propositions,2,. Before e look at the MMD calculations in various cases, e prove the folloing useful characterization of MMD for translation invariant kernels like the Gaussian an Laplace kernels. Lemma. For translation invariant kernels, there exists a pf s such that MMD 2 p, q s Φ p Φ q 2, here Φ p, Φ q enote the characteristic functions of p, q respectively. Proof. From efinition of MMD 2, e have MMD 2 p, q kx, x pxpx x,x x,x x,x kx, x pxqx xx. From Bochner s theorem see Ruin 962 for translation invariant kernels, e kno kx, x sei x e i x here s is the fourier transform of the kernel. Substituting the above equality in the efinition of MMD 2, e have the require result. Proof of Proposition Proof. Since Gaussian kernel is a translation invariant kernel, e can use Lemma to erive the MMD 2 in this case. It is ell-knon that the Fourier transform s of Gaussian kernel is Gaussian istribution. Substituting the characteristic function of normal istribution in Lemma, e have MMD 2 p, q 2 /2π 2 2 /2 iµ Σ/2 iµ Σ/2 2 2 /2π Σ 2 2 /2 iµ iµ /2π Σ + 2 I/2 2 iµ µ 2 iµ 2 µ 2 2 /2π Σ + 2 I/2 iµ µ 2 2 The thir step follos from efinition of complex conjugate. In hat follos, e o the folloing change of variable u Σ + 2 I/2 /2. Consier the folloing term: Σ + 2 I/2 iµ µ 2 u u u + iµ µ 2 Σ + 2 I/2 /2 u Σ + 2 I/2 /2 u Σ + 2 I/2 /2 µ µ 2 Σ + 2 I/2 µ µ 2 /4 u iσ + 2 I/2 /2 µ µ 2 /2 2 u u π Σ + 2 I/2 /2 µ µ 2 Σ + 2 I/2 µ µ 2 /4 The secon step follos from ell-knon theory of change of variables see Theorem 26D of Fremlin 2. By substituting the above equality in Equation 2, e get the require result. Proof of Proposition 2 Before e elve into the etails of the result, e prove the folloing useful propositions. Proposition 4. Let, R + an λ R. Suppose, then e have, an hen, e have, x x x x e λ / / + / + e λ / / / e λ / / / + e λ / / + / e λ / / + / + λ e λ / + e λ / / + /

2 Proof. We sho this hen λ as an example proof: x x λ x λ x x + + λ x x x λ λ x x x e λ/ e λ/+λ/ + e λ/ e λ/+λ/ e λ/ + / + / / / / + / Also, hen, e obtain the same ression for the first an last terms. Hoever, the mile term has the folloing constant integran, thereby, leaing to the require ression. λ x x x λ e λ /. λ Proposition 5. Let, R + an µ R. Then e have, x x here ψ /. 2 e µ / ψ + µ / µ ψ ψ 2 + ψ 2 + O Proof. We first integrate ith respect to x using the Proposition 4 to get 4 2 x µ x xx ψe µ / + e µ / µ 2 2 O e x / / + / + e x / / / e x / / / + e x / / + / µ 2 x µ x We then integrate these terms once again using both parts of Proposition 4 to get the first equality. We simplify the secon equation in the folloing manner: ψ + µ / 2 e µ / + ψe µ / + e µ / µ 2 + µ 2 ψ + µ / / µ / + µ2 /2 + µ / + µ2 /2 2 µ µ +O 2 2 O 2 ψ 2 µ2 2 + µ ψ µ2 2 + µ2 2 2 µ µ +O 2 2 O 2 ψ 2 µ2 + µ2 /2 ψ µ µ O 2 2 O 2 ψ 2 + µ ψ µ 2 µ µ + O + ψ2 2 2 O 2 µ ψ ψ 2 + ψ 2 + O µ µ 2 2 O 2

3 Proof Proposition 2. Recall that e use Laplace kernel, i.e., kx, x x x /. By using the efinition of MMD 2, e have MMD 2 pxpx + qxqx 2pxqx kx, x xx. x,x Consier the term x,x pxqx kx, x xx. The other terms can be calculate in a similar manner. Let ψ / an β + ψ/2/ + ψ 2. We have, x,x pxqx kx, x xx i β i x i,x i µ 2 i β 4β + ψ 2 + O µ 2 4β + ψ + O x x 4 2 x µ x x i xi µ i µ i O β 2 2 µ i β 2 2 O β 2 µ i β 2 The first step follos from the fact that both Laplace kernel an Laplace istribution ecompose over the coorinates. The secon step follos from Proposition 5. Substituting the above ression in Equation, e get, MMD 2 β µ 2 β 2 + ψ O µ β µ O 2. Proof of Proposition Suppose P i N, 2 N, a 2 an Q i N, 2 N, b 2. If a, b are of the same orer as then the meian heuristic ill still pick for banith of the Gaussian kernel. First e note that for istributions ith the same mean, by Taylor s theorem, KLP, Q 2 trσ Σ loget Σ / et Σ 2 a2 /b 2 loga 2 /b 2 a2 /b The MMD 2 can be erive approximate using + x n + nx for small x as / 2 /2 + 4a2 / b2 / a2 + b 2 / / 2 / / 2 /2 + 2a 2 / b 2 / a 2 + b 2 / a2 / 2 + 2b2 / / 2 /2 a 2 / 2 b 2 / 2 2 b 4 / / 2 /2 a2 /b 2 2 If is chosen by the meian heuristic optimal in this case, e see that this is smaller than KL by 4 2 e/b 4. If it is chosen as constant, it can be onentially smaller than KL.

4 B Verifying accuracy of approximate MMDs calculate in Propositions,2,. In the proofs an corollaries of erivations of MMD in Propositions,2,, e use many Taylor approximations in orer to get a more interpretable formula. Here e sho that our approximate formulae, hile being interpretable, are also very accurate. We provie empirical results emonstrating the quality of the approximations use in Section 4. In particular, e compare the estimate value of the MMD using large sample size so that the sample MMD is a very goo estimate of population MMD an the approximations provie in Section 4. As observe in Figure 8, the approximations are quite close to the estimate value, thereby valiating the quality of our approximations MMD logmmd an Overlap..5 an Overlap.5 MMD 2 7 x logmmd log log True mm mm log Figure 8: Top left: MMD vs, for Gaussian istributions an Gaussian kernel ith optimal banith, as estimate from ata an approximate by formula. Top right: same but for LogMMD. Mile left: MMD vs, for Laplace kernel ith optimal banith, estimate from ata an approximate by formula. Mile right: same but for LogMMD. The Log Plots also sho the right scaling that ecays as / ith the right choice of banith. Bottom: LogMMD vs, for Gaussian kernel ith optimal banith, for Gaussians ith same mean an ifferent variances. The straight line is our final approximation in the theorem. The other to are the true MMD by formula, an the MMD from ata.

5 C Biase MMD for Gaussian Distribution In the previous sections, e provie results for unbiase MMD estimator an empirically prove that the poer of the test base on the estimator ecreases ith increasing imension. We report results for the biase MMD estimator in this section an sho that it exhibits similar behavior ,.5 an. Overlap log / log /.5 Meian log.5 an Meian Overlap Poer , Meian,.75 an Overlap log / Meian Figure 9: Plots for Biase MMD ith Gaussian kernel, hen the ata is ran from to Gaussians ith 2 an constant mean separation µ µ 2 2. With respect to the selection of banith, the poer of Biase MMD has similar behavior as Unbiase MMD. As seen in Figure 9, the poer of the biase MMD ecreases in exactly the same fashion as unbiase MMD. We also observe similar behavior ith other examples.