Relative Entropy and Score Function: New Information Estimation Relationships through Arbitrary Additive Perturbation
|
|
- Susanna Ferguson
- 5 years ago
- Views:
Transcription
1 Relative Entropy an Score Function: New Information Estimation Relationships through Arbitrary Aitive Perturbation Dongning Guo Department of Electrical Engineering & Computer Science Northwestern University Evanston, IL 60208, USA Abstract This paper establishes new information estimation relationships pertaining to moels with aitive noise of arbitrary istribution. In particular, we stuy the change in the relative entropy between two probability measures when both of them are perturbe by a small amount of the same aitive noise. It is shown that the rate of the change with respect to the energy of the perturbation can be expresse in terms of the mean suare ifference of the score functions of the two istributions, an, rather surprisingly, is unrelate to the istribution of the perturbation otherwise. The result hols true for the classical relative entropy or Kullback Leibler istance, as well as two of its generalizations: Rényi s relative entropy an the f-ivergence. The result generalizes a recent relationship between the relative entropy an mean suare errors pertaining to Gaussian noise moels, which in turn supersees many previous information estimation relationships. A generalization of the e Bruijn ientity to non-gaussian moels can also be regare as conseuence of this new result. I. INTRODUCTION To ate, a number of connections between basic information measures an estimation measures have been iscovere. By information measures we mean notions which escribe the amount of information, such as entropy an mutual information, as well as several closely relate uantities, such as ifferential entropy an relative entropy also known as information ivergence or Kullback Leibler istance. By estimation measures we mean key notions in estimation theory, which inclue in particular the mean suare error MSE an Fisher information, among others. An early such connection is attribute to e Bruijn ] which relates the ifferential entropy of an arbitrary ranom variable corrupte by Gaussian noise an its Fisher information: δ h X + δ N 2 J X + δ N for every δ 0, where X enotes an arbitrary ranom variable an N N 0, enotes a stanar Gaussian ranom variable inepenent of X throughout this paper. Here JY enotes the Fisher information of its istribution with respect to w.r.t. the location family. The e Bruijn ientity is euivalent to a recent connection between the input output mutual information an the minimum mean-suare This work is supporte by the NSF uner grant CCF an DARPA uner grant W9NF error MMSE of a Gaussian moel 2]: γ I X; γ X + N 2 mmse P X, γ 2 where X P X an mmse P X, γ enotes the MMSE of estimating X given γ X + N. The parameter γ 0 is unerstoo as the signal-to-noise ratio SNR of the Gaussian moel. By-proucts of formula 2 inclue the representation of the entropy, ifferential entropy an the non-gaussianness measure in relative entropy in terms of the MMSE 2] 4]. Several generalizations an extensions of the previous results are foun in 5] 7]. Moreover, the erivative of the mutual information an entropy w.r.t. channel gains have also been stuie for non-aitive-noise channels 7], 8]. Among the aforementione information measures, relative entropy is the most general an versatile in the sense that it is efine for istributions which are iscrete, continuous, or neither, an all the other information measures can be easily expresse in terms of relative entropy. The following relationship between the relative entropy an Fisher information is known 9]: Let p θ } be a family of probability ensity functions pfs parameterize by θ R. Then Dp θ+δ p θ δ 2 /2 Jp θ + o δ 2 3 where Jp θ is the Fisher information of p θ w.r.t. θ. In a recent work 0], Verú establishe an interesting relationship between the relative entropy an mismatche estimation. Let mse Q P, γ represent the MSE for estimating the input X of istribution P to a Gaussian channel of SNR eual to γ base on the channel output, with the estimator assuming the prior istribution of X to be Q. Then 2 γ D P N 0, γ Q N 0, γ mse Q P, γ mmsep, γ where the convolution P N 0, γ represents the istribution of X + N/ γ with X P. Obviously mse Q P, γ mmsep, γ if Q is ientical to P. The formula is particularly satisfying because the left-han sie l.h.s. is an informationtheoretic measure of the mismatch between two istributions, whereas the right-han sie r.h.s. measures the mismatch using an estimation-theoretic metric, i.e., the increase in the 4
2 estimation error ue to the mismatch. In another recent work, Narayanan an Srinivasa ] consier an aitive non-gaussian noise channel moel an provie the following generalization of the e Bruijn ientity: δ h X + δ V δ0 JX where the pf of V is symmetric about 0, twice ifferentiable, an of unit variance but otherwise arbitrary. The significance of 5 is that the erivative oes not epen on the etaile statistics of the noise. Thus, if we view the ifferential entropy as a manifol of the istribution of the perturbation δ V, the geometry of the manifol appears to be locally a bowl which is uniform in every irection of the perturbation. In this work, we consier the change in the relative entropy between two istributions when both of them are perturbe by an infinitesimal amount of the same arbitrary aitive noise. We show that the rate of this change can be expresse as the mean-suare ifference of the score functions of the istributions. Note that the score function is an important notion in estimation theory, whose mean suare is the Fisher information. Like formula 5, the new general relationship turns out to be inepenent of the noise istribution. The general relationship is foun to hol for both the classical relative entropy or Kullback Leibler istance an the more general Rényi s relative entropy, as well as the general f-ivergence ue to Csiszár an inepenently Ali an Silvey 2]. In the special case of Gaussian perturbations, it is shown that, 2, 4, 5 can all be obtaine as conseuence of the new result. II. MAIN RESULTS Theorem : Let Ψ enote an arbitrary istribution with zero mean an variance δ. Let P an Q be two istributions whose respective pfs p an are twice ifferentiable. If P Q an δ z DP Ψ Q Ψ 2 2 E P pz log pz z δ0 + ] 0 6 pz log pz 2 z 7 z log pz log Z 2} 8 where the expectation in 8 is taken with respect to Z P. The classical relative entropy Kullback-Leibler istance is efine for two probability measures P Q as P DP Q log P. 9 Q When the corresponing ensities exist, it is also customary to enote the relative entropy by Dp. The notation / δ in Theorem can be unerstoo as taking erivative w.r.t. the variance of the istribution Ψ with its shape fixe, i.e., Ψ is the istribution of δ V with the ranom variable V fixe. We note that the r.h.s. of 8 oes not epen on the istribution of V, i.e., the change in the relative entropy ue to small perturbation is proportional to the variance of the perturbation but inepenent of its shape. Thus the notation / δ is not ambiguous. For every function f, let f enote its erivative f for notational convenience. For every ifferentiable pf p, the function log px p x/px is known as its score function, hence the r.h.s. of 8 is the mean suare ifference of two score functions. As the previous result 4, this is satisfying because both sies of 8 represent some error ue to the mismatch between the prior istribution supplie to the estimator an the actual istribution p. Obviously, if p an are ientical, both sies of the formula are eual to zero; otherwise, the erivative is negative i.e., perturbation reuces relative entropy. Consier now the Rényi relative entropy, which is efine for two probability measures P Q an every α > 0 as D α P Q α P α log P 0 Q where D P Q is efine as the classical relative entropy DP Q because α D α P Q DP Q. Theorem 2: Let the istributions P, Q an Ψ be efine the same way as in Theorem. Let δ enote the variance of Ψ. If P Q an δ D αp Ψ Q Ψ α 2 p α z α z ] 0 z δ0 + log pz 2 p α z α 2 z z pα u α u u z. Note that as α, the r.h.s. of 2 becomes the r.h.s. of 7. We also point out that, similar to that in 7, the outer integral in 2 can be viewe as the mean suare ifference of two scores log pz log Z with the pf of Z being proportional to p α z α z. Theorems an 2 are uite general because conitions 6 an are satisfie by most istributions of interest. For example, if both p an belong to the exponential family, the erivatives in 6 an also vanish exponentially fast. Not all istributions satisfy those conitions. This is because that although the functions pz an pz logpz/z integrate to an DP Q respectively, they nee not vanish. For example, pz may consist of narrower an narrower Gaussian pulses of the same height as z, so that not only pz oes not vanish, but p z is unboune. Another family of generalize relative entropy, calle the f- ivergence, was introuce by Csiszár an inepenently by
3 Ali an Silvey see e.g., 2]. It is efine for P Q as P I f P Q f Q. 3 Q Theorem 3: Let the istributions P, Q an Ψ be efine the same way as in Theorem. Let δ enote the variance of Ψ. Suppose the secon erivative of f exists an is enote by f. If P Q an δ I f z zf P Ψ Q Ψ δ0+ 2 ] pz 0 4 z py yf py 2 y. y y 5 The integral in 5 can still be expresse in terms of the ifference of the score functions because py/y py/y log py log y]. 6 Note that the special case of ft t log t correspons to the Kullback Leibler istance, whereas the case of ft t t / correspons to the Tsallis relative entropy 3]. Inee, Theorem is a special case of Theorem 3. III. PROOF The key property that unerlies Theorems 3 is the following observation of the local geometry of an aitive-noiseperturbe istribution mae in ]: Lemma : Let the pf p of a ranom variable Z be twice ifferentiable. Let p δ enote the pf of Z + δ V where V is of zero mean an unit variance, an is inepenent of Z. Then for every y R, as δ 0 +, δ p δy δ py. 7 y2 Formula 7 allows the erivative w.r.t. the energy of the perturbation δ to be transforme to the secon erivative of the original pf. In Appenix we provie a brief proof for Lemma which is slightly ifferent than that in ]. Note that Lemma oes not reuire the istribution of the perturbation to be symmetric as is reuire in ]. In the following we first prove Theorem 3, which implies Theorem as a special case, an prove Theorem 2. A. Proof for Theorem 3 Let V be a ranom variable with fixe istribution P V. For convenience, we use a shorthan p δ to enote the pf of the ranom variable Y Z + δ V with Z, V P P V. Similarly, let δ enote the pf of Y Z + δ V with Z, V Q P V. Clearly, δ I f P Ψ Q Ψ δ I f p δ δ. 8 For any single-variable function g, let g an g enote its first an secon erivative respectively. Consier now δ I f p δ δ pδ y δ yf y 9 δ δ y δ y pδ y f + δ y δ δ y δ f pδ y y 20 δ y δ y pδ y f p ] δy δ δ y δ y f pδ y δ y + p δy f pδ y y. 2 δ δ y Invoking Lemma on 2 yiels δ I f p δ δ py p yf δ0 + 2 y py + y f py ] py y y f y. y 22 To procee, we reorganize the integran in 22 to the esire form. The key techniue is integration by parts, which we carry out implicitly with the help of a moest amount of foresight. For convenience, we use p an as shorthan for py an y respectively. We use the fact g h gh gh to rewrite the integran in 22 as p f p + f p + f p p f p p f p p f p ] f ] p ]] p f p p ] p f. 23 Combining the first two terms an simplifying the last term on the r.h.s. of 23 yiel ] ] ] p p f p f + p p f. 24 The first term in 24 integrates to zero by assumption 4. The last two terms in 24 can be combine to obtain ] p p f p 2 p f. 25 Collecting the results from 22 to 25, we have I f P δ Q δ δ 2 δ0 + y py y 2 py f y y which is euivalent to 5. Hence the proof of Theorem We note that the preceing calculation is tantamount to two uses of integration by parts. The treatment here, however, reuires the minimum regularity conitions on the ensities p an.
4 B. Proof for Theorem 2 Consier now δ D αp δ δ α α δ log p α δ y α δ y y 27 p α δ yα δ y y δ / p α δ y α δ y y. 28 By Lemma, the integral in the numerator in 28 can be written as α α pδ y p δ y pδ y δ y α + α y δ y δ δ y δ α α py p y + α α py y y 2 y 2 y 29 at δ 0 +. Note that implies that α α py py α p y + α y 30 y y vanishes as y. Using integration by parts, we further euate the integral on the r.h.s. of 29 to α α p 2 p + α α p y. 3 y 2 y The integran in 3 can be written as α 2 α p α α p p p In the following we omit the coefficient α α/2 to write the remaining terms in 32 as α 2 α p p p p p 2 p 2 p α 2 α 2p p α α + p α α 2 33 log p log 2 p α α. 34 Collecting the preceing results from 28 to 34, we have establishe 2 in Theorem 2. IV. RECOVERING EXISTING INFORMATION ESTIMATION RELATIONSHIPS USING THEOREM A. Mutual information an MMSE We first use Theorem to recover formula 2 establishe in 2]. For convenience, consier the following alternative Gaussian moel: Z X + σw 35 where X an W N 0, are inepenent so that σ 2 represents the noise variance. It suffices to show the following result, which is euivalent to 2, σ 2 IX; X + σw 2σ 4 mmse P X, σ For any x R, let P Z Xx enote the istribution of Z as the output of the moel 35 conitione on X x. The mutual information can be expresse as IX; X + σw IX; Z DP Z X P Z P X 37 which is the average of DP Z Xx P Z over x accoring to the istribution P X, which oes not epen on σ 2. Let N N 0, be inepenent of Z. Consier the erivative of DP Z Xx P Z w.r.t. σ 2, or euivalently, by introucing a small perturbation, δ D P Z+ δn Xx PZ+ δn δ0 + 2 E log pz Xx Z log p Z Z 2 } 38 ue to Theorem, where the expectation is over P Z Xx, which is a Gaussian istribution centere at x with variance σ 2. The first score is easy to evaluate: log p Z Xx Z x Z/σ 2. The secon score is etermine by the following simple variation of a result ue to Esposito 4] see also Lemma 2 in 2]: Lemma 2: log p Z z E X Z z} z/σ 2. Clearly, the r.h.s. of 38 becomes E PZ Xx x E X Z} 2 } /2σ 4, 39 the average of which over x is eual to the r.h.s. of 36. Thus 36 is establishe, an so is 2. B. Differential Entropy an MMSE Consier again the moel 35. It is not ifficult to see DP Z+ δ N N 0, σ 2 + δ 2 log 2πe σ 2 + δ + EX2 /2 σ 2 + δ h Z + δ N 40. By Theorem an Lemma 2, we have δ D P Z+ δ N N 0, σ 2 + δ δ0 + E X Z} z 2 E σ 2 + z } 2 σ 2 4 E E X Z} 2} / 2σ Plugging into 40, we have σ 2 hz 2 log2πe + σ2 2 + σ 2 σ 2 EX2 E X Z} 2 2σ Note that EX 2 E X Z} 2 mmsep X, /σ 2. Moreover, hx hz σ0, an hz 2 log2πe + σ2 vanishes as σ 2. Therefore, by integrating w.r.t. σ 2 from
5 0 to, we obtain hx 2 log 2πe s 2 mmse P X, s ss + s 44 which is euivalent to the integral expression in 2], in which we use the SNR as the integral variable. C. Relative Entropy an MMSE The connection between relative entropy an MMSE 4 can also be regare as a special case of Theorem. Consier again the moel 35 an apply Theorem. We have 2 δ D P X+ δ N Q X+ δ N E P log p Z X + δ N log Z X + } 2 δ N. By Lemma 2, the r.h.s. of 45 can be rewritten as E P EP X Z} E Q X Z} 2} 45 E P X EQ X Z} X E P X Z}] 2} 46 E P X EQ X Z} 2} + E P X EP X Z} 2} 2E P X E Q X Z}X E P X Z}}. 47 Using the orthogonality of X E P X Z} an every function of Z uner probability measure P, we can replace E Q X Z} in the last term by E P X Z} which are both functions of Z, an continue the euality as E P X EQ X Z} 2} + E P X EP X Z} 2} 2E P X E P X Z}X E P X Z}} E P X EQ X Z} 2} E P X EP X Z} 2} 48 mse Q X P X, γ mmse P X, γ 49 where γ /δ. Hence yiels the esire formula 4. D. Differential Entropy an Fisher Information The generalize e Bruijn ientity 5 can be recovere basically by inspection of 8. Consier a istribution Q Z which is uniform on m, m] with m being a large number an vanishes smoothly outsie the interval e.g., a raise-cosine function with roll-off. Then Q Z+σN remains essentially uniform, so that log Z z 0 over almost all the probability mass of P Z. As m, 8 reuces to 5. V. CONCLUDING REMARKS The relationships connecting the score function an various forms of relative entropy shown in this paper are the most general for aitive-noise moels to this ate. It is by now clear that such erivative relationships between basic informationan estimation-theoretic measures rely on neither the normality of the aitive perturbation, nor the logarithm functional in classical information measures. The results, however, o not irectly translate into integral relationships unless the noise is Gaussian, which has the infinite ivisibility property. VI. ACKNOWLEDGEMENTS The author woul like to thank Sergio Verú for sharing an earlier conjecture of his on the relationship between the Kullback Leibler istance an MSE, which inspire this work. APPENDIX PROOF OF LEMMA Proof: Recall that p δ enotes the istribution of Y X + δ V. Denote the characteristic function of p δ as ϕu, δ E e iuy }. 50 Due to inepenence of X an V, ϕs, δ E e iux} } E e iu δ V 5 k } iu δ V ϕs, 0 E 52 k! k0 k ] ϕs, 0 + δiu2 iu δ + EV k 53 2 k! k3 where we have use the assumptions that V has zero mean an unit variance. Note also that the series sum in 53 vanishes as oδ. Taking the inverse Fourier transform on both sies of 53 yiels p δ y p 0 y + 2 δ 2 y 2 p 0y + oδ. 54 Hence Lemma is prove as p 0 y py. REFERENCES ] A. J. Stam, Some ineualities satisfie by the uantities of information of Fisher an Shannon, Information an Control, vol. 2, pp. 0 2, ] D. Guo, S. Shamai, an S. Verú, Mutual information an minimum mean-suare error in Gaussian channels, IEEE Trans. Inform. Theory, vol. 5, pp , Apr ] D. Guo, S. Shamai, an S. Verú, Proof of entropy power ineualities via MMSE, in Proc. IEEE Int. Symp. Inform. Theory, pp. 0 05, Seattle, WA, USA, July ] S. Verú an D. Guo, A simple proof of the entropy power ineuality, IEEE Trans. Inform. Theory, pp , May ] M. Zakai, On mutual information, likelihoo ratios, an estimation error for the aitive Gaussian channel, IEEE Trans. Inform. Theory, vol. 5, pp , Sept ] D. Guo, S. Shamai, an S. Verú, Aitive non-gaussian noise channels: Mutual information an conitional mean estimation, in Proc. IEEE Int. Symp. Inform. Theory, Aelaie, Australia, Sept ] D. Palomar an S. Verú, Representation of mutual information via input estimates, IEEE Trans. Inform. Theory, vol. 53, pp , Feb ] C. Measson, A. Montanari, an R. Urbanke, Maxwell Construction: The Hien Brige Between Iterativew an Maximum a Posteriori Decoing, IEEE Trans. Inform. Theory, vol. 54, pp , ] S. Kullback, Information Theory an Statistics. New York: Dover, ] S. Verú, Mismatche estimation an relative entropy, in Proc. IEEE Int. Symp. Inform. Theory, Seoul, Korea, ] K. R. Narayanan an A. R. Srinivasa, On the thermoynamic temperature of a general istribution ] I. Csiszr, Axiomatic characterizations of information measures, Entropy, vol. 0, pp , ] C. Tsallis, Possible generalization of Boltzmann-Gibbs statistics, Journal of Statistical Physics, vol. 52, pp , ] R. Esposito, On a relation between etection an estimation inecision theory, Information an Control, vol. 2, pp. 6 20, 968.
Linear First-Order Equations
5 Linear First-Orer Equations Linear first-orer ifferential equations make up another important class of ifferential equations that commonly arise in applications an are relatively easy to solve (in theory)
More informationSturm-Liouville Theory
LECTURE 5 Sturm-Liouville Theory In the three preceing lectures I emonstrate the utility of Fourier series in solving PDE/BVPs. As we ll now see, Fourier series are just the tip of the iceberg of the theory
More informationTable of Common Derivatives By David Abraham
Prouct an Quotient Rules: Table of Common Derivatives By Davi Abraham [ f ( g( ] = [ f ( ] g( + f ( [ g( ] f ( = g( [ f ( ] g( g( f ( [ g( ] Trigonometric Functions: sin( = cos( cos( = sin( tan( = sec
More informationLeast-Squares Regression on Sparse Spaces
Least-Squares Regression on Sparse Spaces Yuri Grinberg, Mahi Milani Far, Joelle Pineau School of Computer Science McGill University Montreal, Canaa {ygrinb,mmilan1,jpineau}@cs.mcgill.ca 1 Introuction
More informationTime-of-Arrival Estimation in Non-Line-Of-Sight Environments
2 Conference on Information Sciences an Systems, The Johns Hopkins University, March 2, 2 Time-of-Arrival Estimation in Non-Line-Of-Sight Environments Sinan Gezici, Hisashi Kobayashi an H. Vincent Poor
More informationSchrödinger s equation.
Physics 342 Lecture 5 Schröinger s Equation Lecture 5 Physics 342 Quantum Mechanics I Wenesay, February 3r, 2010 Toay we iscuss Schröinger s equation an show that it supports the basic interpretation of
More informationMath 342 Partial Differential Equations «Viktor Grigoryan
Math 342 Partial Differential Equations «Viktor Grigoryan 6 Wave equation: solution In this lecture we will solve the wave equation on the entire real line x R. This correspons to a string of infinite
More informationEuler equations for multiple integrals
Euler equations for multiple integrals January 22, 2013 Contents 1 Reminer of multivariable calculus 2 1.1 Vector ifferentiation......................... 2 1.2 Matrix ifferentiation........................
More informationImplicit Differentiation
Implicit Differentiation Thus far, the functions we have been concerne with have been efine explicitly. A function is efine explicitly if the output is given irectly in terms of the input. For instance,
More information. Using a multinomial model gives us the following equation for P d. , with respect to same length term sequences.
S 63 Lecture 8 2/2/26 Lecturer Lillian Lee Scribes Peter Babinski, Davi Lin Basic Language Moeling Approach I. Special ase of LM-base Approach a. Recap of Formulas an Terms b. Fixing θ? c. About that Multinomial
More informationCapacity Analysis of MIMO Systems with Unknown Channel State Information
Capacity Analysis of MIMO Systems with Unknown Channel State Information Jun Zheng an Bhaskar D. Rao Dept. of Electrical an Computer Engineering University of California at San Diego e-mail: juzheng@ucs.eu,
More informationThe total derivative. Chapter Lagrangian and Eulerian approaches
Chapter 5 The total erivative 51 Lagrangian an Eulerian approaches The representation of a flui through scalar or vector fiels means that each physical quantity uner consieration is escribe as a function
More informationRobust Forward Algorithms via PAC-Bayes and Laplace Distributions. ω Q. Pr (y(ω x) < 0) = Pr A k
A Proof of Lemma 2 B Proof of Lemma 3 Proof: Since the support of LL istributions is R, two such istributions are equivalent absolutely continuous with respect to each other an the ivergence is well-efine
More informationThe continuity equation
Chapter 6 The continuity equation 61 The equation of continuity It is evient that in a certain region of space the matter entering it must be equal to the matter leaving it Let us consier an infinitesimal
More informationTopic 7: Convergence of Random Variables
Topic 7: Convergence of Ranom Variables Course 003, 2016 Page 0 The Inference Problem So far, our starting point has been a given probability space (S, F, P). We now look at how to generate information
More informationGeneralization of the persistent random walk to dimensions greater than 1
PHYSICAL REVIEW E VOLUME 58, NUMBER 6 DECEMBER 1998 Generalization of the persistent ranom walk to imensions greater than 1 Marián Boguñá, Josep M. Porrà, an Jaume Masoliver Departament e Física Fonamental,
More informationCalculus of Variations
16.323 Lecture 5 Calculus of Variations Calculus of Variations Most books cover this material well, but Kirk Chapter 4 oes a particularly nice job. x(t) x* x*+ αδx (1) x*- αδx (1) αδx (1) αδx (1) t f t
More informationA. Incorrect! The letter t does not appear in the expression of the given integral
AP Physics C - Problem Drill 1: The Funamental Theorem of Calculus Question No. 1 of 1 Instruction: (1) Rea the problem statement an answer choices carefully () Work the problems on paper as neee (3) Question
More informationELEC3114 Control Systems 1
ELEC34 Control Systems Linear Systems - Moelling - Some Issues Session 2, 2007 Introuction Linear systems may be represente in a number of ifferent ways. Figure shows the relationship between various representations.
More informationThe Entropy of Random Finite Sets
The Entropy of Ranom Finite Sets Mohamma Rezaeian an Ba-Ngu Vo Department of Electrical an Electronic Engineering, University of Melbourne, Victoria, 300, Australia rezaeian, b.vo@ee.unimelb.eu.au Abstract
More informationAn Analytical Expression of the Probability of Error for Relaying with Decode-and-forward
An Analytical Expression of the Probability of Error for Relaying with Decoe-an-forwar Alexanre Graell i Amat an Ingmar Lan Department of Electronics, Institut TELECOM-TELECOM Bretagne, Brest, France Email:
More informationP(x) = 1 + x n. (20.11) n n φ n(x) = exp(x) = lim φ (x) (20.8) Our first task for the chain rule is to find the derivative of the exponential
20. Derivatives of compositions: the chain rule At the en of the last lecture we iscovere a nee for the erivative of a composition. In this lecture we show how to calculate it. Accoringly, let P have omain
More informationMath 1B, lecture 8: Integration by parts
Math B, lecture 8: Integration by parts Nathan Pflueger 23 September 2 Introuction Integration by parts, similarly to integration by substitution, reverses a well-known technique of ifferentiation an explores
More informationQubit channels that achieve capacity with two states
Qubit channels that achieve capacity with two states Dominic W. Berry Department of Physics, The University of Queenslan, Brisbane, Queenslan 4072, Australia Receive 22 December 2004; publishe 22 March
More informationLecture XII. where Φ is called the potential function. Let us introduce spherical coordinates defined through the relations
Lecture XII Abstract We introuce the Laplace equation in spherical coorinates an apply the metho of separation of variables to solve it. This will generate three linear orinary secon orer ifferential equations:
More informationLecture 2 Lagrangian formulation of classical mechanics Mechanics
Lecture Lagrangian formulation of classical mechanics 70.00 Mechanics Principle of stationary action MATH-GA To specify a motion uniquely in classical mechanics, it suffices to give, at some time t 0,
More informationTutorial on Maximum Likelyhood Estimation: Parametric Density Estimation
Tutorial on Maximum Likelyhoo Estimation: Parametric Density Estimation Suhir B Kylasa 03/13/2014 1 Motivation Suppose one wishes to etermine just how biase an unfair coin is. Call the probability of tossing
More informationSeparation of Variables
Physics 342 Lecture 1 Separation of Variables Lecture 1 Physics 342 Quantum Mechanics I Monay, January 25th, 2010 There are three basic mathematical tools we nee, an then we can begin working on the physical
More informationSurvey Sampling. 1 Design-based Inference. Kosuke Imai Department of Politics, Princeton University. February 19, 2013
Survey Sampling Kosuke Imai Department of Politics, Princeton University February 19, 2013 Survey sampling is one of the most commonly use ata collection methos for social scientists. We begin by escribing
More informationPDE Notes, Lecture #11
PDE Notes, Lecture # from Professor Jalal Shatah s Lectures Febuary 9th, 2009 Sobolev Spaces Recall that for u L loc we can efine the weak erivative Du by Du, φ := udφ φ C0 If v L loc such that Du, φ =
More information1 Heisenberg Representation
1 Heisenberg Representation What we have been ealing with so far is calle the Schröinger representation. In this representation, operators are constants an all the time epenence is carrie by the states.
More informationLectures - Week 10 Introduction to Ordinary Differential Equations (ODES) First Order Linear ODEs
Lectures - Week 10 Introuction to Orinary Differential Equations (ODES) First Orer Linear ODEs When stuying ODEs we are consiering functions of one inepenent variable, e.g., f(x), where x is the inepenent
More informationSome Examples. Uniform motion. Poisson processes on the real line
Some Examples Our immeiate goal is to see some examples of Lévy processes, an/or infinitely-ivisible laws on. Uniform motion Choose an fix a nonranom an efine X := for all (1) Then, {X } is a [nonranom]
More informationQuantum mechanical approaches to the virial
Quantum mechanical approaches to the virial S.LeBohec Department of Physics an Astronomy, University of Utah, Salt Lae City, UT 84112, USA Date: June 30 th 2015 In this note, we approach the virial from
More information1 dx. where is a large constant, i.e., 1, (7.6) and Px is of the order of unity. Indeed, if px is given by (7.5), the inequality (7.
Lectures Nine an Ten The WKB Approximation The WKB metho is a powerful tool to obtain solutions for many physical problems It is generally applicable to problems of wave propagation in which the frequency
More informationPhysics 5153 Classical Mechanics. The Virial Theorem and The Poisson Bracket-1
Physics 5153 Classical Mechanics The Virial Theorem an The Poisson Bracket 1 Introuction In this lecture we will consier two applications of the Hamiltonian. The first, the Virial Theorem, applies to systems
More informationLecture 1b. Differential operators and orthogonal coordinates. Partial derivatives. Divergence and divergence theorem. Gradient. A y. + A y y dy. 1b.
b. Partial erivatives Lecture b Differential operators an orthogonal coorinates Recall from our calculus courses that the erivative of a function can be efine as f ()=lim 0 or using the central ifference
More information6 General properties of an autonomous system of two first order ODE
6 General properties of an autonomous system of two first orer ODE Here we embark on stuying the autonomous system of two first orer ifferential equations of the form ẋ 1 = f 1 (, x 2 ), ẋ 2 = f 2 (, x
More informationIPA Derivatives for Make-to-Stock Production-Inventory Systems With Backorders Under the (R,r) Policy
IPA Derivatives for Make-to-Stock Prouction-Inventory Systems With Backorers Uner the (Rr) Policy Yihong Fan a Benamin Melame b Yao Zhao c Yorai Wari Abstract This paper aresses Infinitesimal Perturbation
More informationBayesian Estimation of the Entropy of the Multivariate Gaussian
Bayesian Estimation of the Entropy of the Multivariate Gaussian Santosh Srivastava Fre Hutchinson Cancer Research Center Seattle, WA 989, USA Email: ssrivast@fhcrc.org Maya R. Gupta Department of Electrical
More informationJointly continuous distributions and the multivariate Normal
Jointly continuous istributions an the multivariate Normal Márton alázs an álint Tóth October 3, 04 This little write-up is part of important founations of probability that were left out of the unit Probability
More informationd dx But have you ever seen a derivation of these results? We ll prove the first result below. cos h 1
Lecture 5 Some ifferentiation rules Trigonometric functions (Relevant section from Stewart, Seventh Eition: Section 3.3) You all know that sin = cos cos = sin. () But have you ever seen a erivation of
More informationG4003 Advanced Mechanics 1. We already saw that if q is a cyclic variable, the associated conjugate momentum is conserved, L = const.
G4003 Avance Mechanics 1 The Noether theorem We alreay saw that if q is a cyclic variable, the associate conjugate momentum is conserve, q = 0 p q = const. (1) This is the simplest incarnation of Noether
More informationJUST THE MATHS UNIT NUMBER DIFFERENTIATION 2 (Rates of change) A.J.Hobson
JUST THE MATHS UNIT NUMBER 10.2 DIFFERENTIATION 2 (Rates of change) by A.J.Hobson 10.2.1 Introuction 10.2.2 Average rates of change 10.2.3 Instantaneous rates of change 10.2.4 Derivatives 10.2.5 Exercises
More informationθ x = f ( x,t) could be written as
9. Higher orer PDEs as systems of first-orer PDEs. Hyperbolic systems. For PDEs, as for ODEs, we may reuce the orer by efining new epenent variables. For example, in the case of the wave equation, (1)
More information19 Eigenvalues, Eigenvectors, Ordinary Differential Equations, and Control
19 Eigenvalues, Eigenvectors, Orinary Differential Equations, an Control This section introuces eigenvalues an eigenvectors of a matrix, an iscusses the role of the eigenvalues in etermining the behavior
More informationA PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
A PAC-Bayesian Approach to Spectrally-Normalize Margin Bouns for Neural Networks Behnam Neyshabur, Srinah Bhojanapalli, Davi McAllester, Nathan Srebro Toyota Technological Institute at Chicago {bneyshabur,
More informationLower Bounds for the Smoothed Number of Pareto optimal Solutions
Lower Bouns for the Smoothe Number of Pareto optimal Solutions Tobias Brunsch an Heiko Röglin Department of Computer Science, University of Bonn, Germany brunsch@cs.uni-bonn.e, heiko@roeglin.org Abstract.
More informationarxiv: v1 [physics.flu-dyn] 8 May 2014
Energetics of a flui uner the Boussinesq approximation arxiv:1405.1921v1 [physics.flu-yn] 8 May 2014 Kiyoshi Maruyama Department of Earth an Ocean Sciences, National Defense Acaemy, Yokosuka, Kanagawa
More informationA note on asymptotic formulae for one-dimensional network flow problems Carlos F. Daganzo and Karen R. Smilowitz
A note on asymptotic formulae for one-imensional network flow problems Carlos F. Daganzo an Karen R. Smilowitz (to appear in Annals of Operations Research) Abstract This note evelops asymptotic formulae
More informationThe Sokhotski-Plemelj Formula
hysics 25 Winter 208 The Sokhotski-lemelj Formula. The Sokhotski-lemelj formula The Sokhotski-lemelj formula is a relation between the following generalize functions (also calle istributions), ±iǫ = iπ(),
More informationLeaving Randomness to Nature: d-dimensional Product Codes through the lens of Generalized-LDPC codes
Leaving Ranomness to Nature: -Dimensional Prouct Coes through the lens of Generalize-LDPC coes Tavor Baharav, Kannan Ramchanran Dept. of Electrical Engineering an Computer Sciences, U.C. Berkeley {tavorb,
More informationOptimal Control of Spatially Distributed Systems
Optimal Control of Spatially Distribute Systems Naer Motee an Ali Jababaie Abstract In this paper, we stuy the structural properties of optimal control of spatially istribute systems. Such systems consist
More informationNOTES ON EULER-BOOLE SUMMATION (1) f (l 1) (n) f (l 1) (m) + ( 1)k 1 k! B k (y) f (k) (y) dy,
NOTES ON EULER-BOOLE SUMMATION JONATHAN M BORWEIN, NEIL J CALKIN, AND DANTE MANNA Abstract We stuy a connection between Euler-MacLaurin Summation an Boole Summation suggeste in an AMM note from 196, which
More informationFirst Order Linear Differential Equations
LECTURE 6 First Orer Linear Differential Equations A linear first orer orinary ifferential equation is a ifferential equation of the form ( a(xy + b(xy = c(x. Here y represents the unknown function, y
More informationError Floors in LDPC Codes: Fast Simulation, Bounds and Hardware Emulation
Error Floors in LDPC Coes: Fast Simulation, Bouns an Harware Emulation Pamela Lee, Lara Dolecek, Zhengya Zhang, Venkat Anantharam, Borivoje Nikolic, an Martin J. Wainwright EECS Department University of
More informationWUCHEN LI AND STANLEY OSHER
CONSTRAINED DYNAMICAL OPTIMAL TRANSPORT AND ITS LAGRANGIAN FORMULATION WUCHEN LI AND STANLEY OSHER Abstract. We propose ynamical optimal transport (OT) problems constraine in a parameterize probability
More informationPerturbation Analysis and Optimization of Stochastic Flow Networks
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. XX, NO. Y, MMM 2004 1 Perturbation Analysis an Optimization of Stochastic Flow Networks Gang Sun, Christos G. Cassanras, Yorai Wari, Christos G. Panayiotou,
More informationIterated Point-Line Configurations Grow Doubly-Exponentially
Iterate Point-Line Configurations Grow Doubly-Exponentially Joshua Cooper an Mark Walters July 9, 008 Abstract Begin with a set of four points in the real plane in general position. A to this collection
More informationThe Principle of Least Action
Chapter 7. The Principle of Least Action 7.1 Force Methos vs. Energy Methos We have so far stuie two istinct ways of analyzing physics problems: force methos, basically consisting of the application of
More informationImage Denoising Using Spatial Adaptive Thresholding
International Journal of Engineering Technology, Management an Applie Sciences Image Denoising Using Spatial Aaptive Thresholing Raneesh Mishra M. Tech Stuent, Department of Electronics & Communication,
More informationON THE OPTIMALITY SYSTEM FOR A 1 D EULER FLOW PROBLEM
ON THE OPTIMALITY SYSTEM FOR A D EULER FLOW PROBLEM Eugene M. Cliff Matthias Heinkenschloss y Ajit R. Shenoy z Interisciplinary Center for Applie Mathematics Virginia Tech Blacksburg, Virginia 46 Abstract
More informationMath Notes on differentials, the Chain Rule, gradients, directional derivative, and normal vectors
Math 18.02 Notes on ifferentials, the Chain Rule, graients, irectional erivative, an normal vectors Tangent plane an linear approximation We efine the partial erivatives of f( xy, ) as follows: f f( x+
More informationEntanglement is not very useful for estimating multiple phases
PHYSICAL REVIEW A 70, 032310 (2004) Entanglement is not very useful for estimating multiple phases Manuel A. Ballester* Department of Mathematics, University of Utrecht, Box 80010, 3508 TA Utrecht, The
More informationState-Space Model for a Multi-Machine System
State-Space Moel for a Multi-Machine System These notes parallel section.4 in the text. We are ealing with classically moele machines (IEEE Type.), constant impeance loas, an a network reuce to its internal
More informationClosed and Open Loop Optimal Control of Buffer and Energy of a Wireless Device
Close an Open Loop Optimal Control of Buffer an Energy of a Wireless Device V. S. Borkar School of Technology an Computer Science TIFR, umbai, Inia. borkar@tifr.res.in A. A. Kherani B. J. Prabhu INRIA
More informationQuantum Mechanics in Three Dimensions
Physics 342 Lecture 20 Quantum Mechanics in Three Dimensions Lecture 20 Physics 342 Quantum Mechanics I Monay, March 24th, 2008 We begin our spherical solutions with the simplest possible case zero potential.
More informationSpace-time Linear Dispersion Using Coordinate Interleaving
Space-time Linear Dispersion Using Coorinate Interleaving Jinsong Wu an Steven D Blostein Department of Electrical an Computer Engineering Queen s University, Kingston, Ontario, Canaa, K7L3N6 Email: wujs@ieeeorg
More informationGaussian processes with monotonicity information
Gaussian processes with monotonicity information Anonymous Author Anonymous Author Unknown Institution Unknown Institution Abstract A metho for using monotonicity information in multivariate Gaussian process
More informationSwitching Time Optimization in Discretized Hybrid Dynamical Systems
Switching Time Optimization in Discretize Hybri Dynamical Systems Kathrin Flaßkamp, To Murphey, an Sina Ober-Blöbaum Abstract Switching time optimization (STO) arises in systems that have a finite set
More informationPartial Differential Equations
Chapter Partial Differential Equations. Introuction Have solve orinary ifferential equations, i.e. ones where there is one inepenent an one epenent variable. Only orinary ifferentiation is therefore involve.
More informationConservation Laws. Chapter Conservation of Energy
20 Chapter 3 Conservation Laws In orer to check the physical consistency of the above set of equations governing Maxwell-Lorentz electroynamics [(2.10) an (2.12) or (1.65) an (1.68)], we examine the action
More information'HVLJQ &RQVLGHUDWLRQ LQ 0DWHULDO 6HOHFWLRQ 'HVLJQ 6HQVLWLYLW\,1752'8&7,21
Large amping in a structural material may be either esirable or unesirable, epening on the engineering application at han. For example, amping is a esirable property to the esigner concerne with limiting
More informationA Review of Multiple Try MCMC algorithms for Signal Processing
A Review of Multiple Try MCMC algorithms for Signal Processing Luca Martino Image Processing Lab., Universitat e València (Spain) Universia Carlos III e Mari, Leganes (Spain) Abstract Many applications
More informationCalculus in the AP Physics C Course The Derivative
Limits an Derivatives Calculus in the AP Physics C Course The Derivative In physics, the ieas of the rate change of a quantity (along with the slope of a tangent line) an the area uner a curve are essential.
More informationarxiv:math/ v1 [math.pr] 19 Apr 2001
Conitional Expectation as Quantile Derivative arxiv:math/00490v math.pr 9 Apr 200 Dirk Tasche November 3, 2000 Abstract For a linear combination u j X j of ranom variables, we are intereste in the partial
More informationLATTICE-BASED D-OPTIMUM DESIGN FOR FOURIER REGRESSION
The Annals of Statistics 1997, Vol. 25, No. 6, 2313 2327 LATTICE-BASED D-OPTIMUM DESIGN FOR FOURIER REGRESSION By Eva Riccomagno, 1 Rainer Schwabe 2 an Henry P. Wynn 1 University of Warwick, Technische
More informationThe derivative of a function f(x) is another function, defined in terms of a limiting expression: f(x + δx) f(x)
Y. D. Chong (2016) MH2801: Complex Methos for the Sciences 1. Derivatives The erivative of a function f(x) is another function, efine in terms of a limiting expression: f (x) f (x) lim x δx 0 f(x + δx)
More informationControl Volume Derivations for Thermodynamics
Control olume Derivations for Thermoynamics J. M. Powers University of Notre Dame AME 327 Fall 2003 This ocument will give a summary of the necessary mathematical operations necessary to cast the conservation
More informationIntroduction to the Vlasov-Poisson system
Introuction to the Vlasov-Poisson system Simone Calogero 1 The Vlasov equation Consier a particle with mass m > 0. Let x(t) R 3 enote the position of the particle at time t R an v(t) = ẋ(t) = x(t)/t its
More informationAssignment 1. g i (x 1,..., x n ) dx i = 0. i=1
Assignment 1 Golstein 1.4 The equations of motion for the rolling isk are special cases of general linear ifferential equations of constraint of the form g i (x 1,..., x n x i = 0. i=1 A constraint conition
More informationA Note on Exact Solutions to Linear Differential Equations by the Matrix Exponential
Avances in Applie Mathematics an Mechanics Av. Appl. Math. Mech. Vol. 1 No. 4 pp. 573-580 DOI: 10.4208/aamm.09-m0946 August 2009 A Note on Exact Solutions to Linear Differential Equations by the Matrix
More informationCalculus of Variations
Calculus of Variations Lagrangian formalism is the main tool of theoretical classical mechanics. Calculus of Variations is a part of Mathematics which Lagrangian formalism is base on. In this section,
More informationALGEBRAIC AND ANALYTIC PROPERTIES OF ARITHMETIC FUNCTIONS
ALGEBRAIC AND ANALYTIC PROPERTIES OF ARITHMETIC FUNCTIONS MARK SCHACHNER Abstract. When consiere as an algebraic space, the set of arithmetic functions equippe with the operations of pointwise aition an
More informationSummary: Differentiation
Techniques of Differentiation. Inverse Trigonometric functions The basic formulas (available in MF5 are: Summary: Differentiation ( sin ( cos The basic formula can be generalize as follows: Note: ( sin
More informationIntroduction to Markov Processes
Introuction to Markov Processes Connexions moule m44014 Zzis law Gustav) Meglicki, Jr Office of the VP for Information Technology Iniana University RCS: Section-2.tex,v 1.24 2012/12/21 18:03:08 gustav
More information3.2 Differentiability
Section 3 Differentiability 09 3 Differentiability What you will learn about How f (a) Might Fail to Eist Differentiability Implies Local Linearity Numerical Derivatives on a Calculator Differentiability
More informationA. Exclusive KL View of the MLE
A. Exclusive KL View of the MLE Lets assume a change-of-variable moel p Z z on the ranom variable Z R m, such as the one use in Dinh et al. 2017: z 0 p 0 z 0 an z = ψz 0, where ψ is an invertible function
More informationAPPROXIMATE SOLUTION FOR TRANSIENT HEAT TRANSFER IN STATIC TURBULENT HE II. B. Baudouy. CEA/Saclay, DSM/DAPNIA/STCM Gif-sur-Yvette Cedex, France
APPROXIMAE SOLUION FOR RANSIEN HEA RANSFER IN SAIC URBULEN HE II B. Bauouy CEA/Saclay, DSM/DAPNIA/SCM 91191 Gif-sur-Yvette Ceex, France ABSRAC Analytical solution in one imension of the heat iffusion equation
More informationA new proof of the sharpness of the phase transition for Bernoulli percolation on Z d
A new proof of the sharpness of the phase transition for Bernoulli percolation on Z Hugo Duminil-Copin an Vincent Tassion October 8, 205 Abstract We provie a new proof of the sharpness of the phase transition
More informationOnline Appendix for Trade Policy under Monopolistic Competition with Firm Selection
Online Appenix for Trae Policy uner Monopolistic Competition with Firm Selection Kyle Bagwell Stanfor University an NBER Seung Hoon Lee Georgia Institute of Technology September 6, 2018 In this Online
More informationComputing Exact Confidence Coefficients of Simultaneous Confidence Intervals for Multinomial Proportions and their Functions
Working Paper 2013:5 Department of Statistics Computing Exact Confience Coefficients of Simultaneous Confience Intervals for Multinomial Proportions an their Functions Shaobo Jin Working Paper 2013:5
More informationSharp Thresholds. Zachary Hamaker. March 15, 2010
Sharp Threshols Zachary Hamaker March 15, 2010 Abstract The Kolmogorov Zero-One law states that for tail events on infinite-imensional probability spaces, the probability must be either zero or one. Behavior
More information4.2 First Differentiation Rules; Leibniz Notation
.. FIRST DIFFERENTIATION RULES; LEIBNIZ NOTATION 307. First Differentiation Rules; Leibniz Notation In this section we erive rules which let us quickly compute the erivative function f (x) for any polynomial
More informationarxiv:hep-th/ v1 3 Feb 1993
NBI-HE-9-89 PAR LPTHE 9-49 FTUAM 9-44 November 99 Matrix moel calculations beyon the spherical limit arxiv:hep-th/93004v 3 Feb 993 J. Ambjørn The Niels Bohr Institute Blegamsvej 7, DK-00 Copenhagen Ø,
More informationTEMPORAL AND TIME-FREQUENCY CORRELATION-BASED BLIND SOURCE SEPARATION METHODS. Yannick DEVILLE
TEMPORAL AND TIME-FREQUENCY CORRELATION-BASED BLIND SOURCE SEPARATION METHODS Yannick DEVILLE Université Paul Sabatier Laboratoire Acoustique, Métrologie, Instrumentation Bât. 3RB2, 8 Route e Narbonne,
More informationMulti-edge Optimization of Low-Density Parity-Check Codes for Joint Source-Channel Coding
Multi-ege Optimization of Low-Density Parity-Check Coes for Joint Source-Channel Coing H. V. Beltrão Neto an W. Henkel Jacobs University Bremen Campus Ring 1 D-28759 Bremen, Germany Email: {h.beltrao,
More informationII. First variation of functionals
II. First variation of functionals The erivative of a function being zero is a necessary conition for the etremum of that function in orinary calculus. Let us now tackle the question of the equivalent
More informationarxiv: v3 [cs.lg] 3 Dec 2017
Context-Aware Generative Aversarial Privacy Chong Huang, Peter Kairouz, Xiao Chen, Lalitha Sankar, an Ram Rajagopal arxiv:1710.09549v3 [cs.lg] 3 Dec 2017 Abstract Preserving the utility of publishe atasets
More informationDot trajectories in the superposition of random screens: analysis and synthesis
1472 J. Opt. Soc. Am. A/ Vol. 21, No. 8/ August 2004 Isaac Amiror Dot trajectories in the superposition of ranom screens: analysis an synthesis Isaac Amiror Laboratoire e Systèmes Périphériques, Ecole
More information