Section 4.2. Types of Differentiation

Transcription

1 42 Types of Differentiation 1 Section 42 Types of Differentiation Note In this section we define differentiation of various structures with respect to a scalar, a vector, and a matrix Definition Let vector y be a function of scalar variable x so that y = y(x) = [y 1,y 2,,y n ] = [y 1 (x),y 2 (x),,y n (x)] The derivative of vector y with respect to scalar x is the vector y = [ y1, y 2,, y ] n = [y 1,y 2,,y n] Let matrix Y be a function of scalar variable x so that Y = Y (x) = [y ij (x)] The derivative of matrix Y with respect to scalar x is the matrix [ ] Y = yij = [y ij ] Note To define differentiation with respect to a vector x of an object Φ which is a function of a vector, we use the usual definition: Φ(x + ty) Φ(x) lim where y is any conformable vector with x (this will give the derivative of Φ in the direction of y) Notice that we can use the usual ε/δ definition of limit, provided we have a metric on the objects Φ We start with scalar valued functions

2 42 Types of Differentiation 2 Definition Let f : R n R be a scalar valued function of a vector The derivative of scalar valued function f with respect to vector x = [x 1,x 2,,x n ] is [ f f =, f,, f ] 1 2 n This derivative is the gradient of f, denoted f (sometimes read dell f ) Note Let A be a given (constant) n n matrix and x R n The quadratic form x T Ax is a scalar and the derivative of the quadratic form in the direction y is (x + ty) T A(x + ty) x T Ax lim (x T + ty T )A(x + ty) x T Ax = lim x T Ax + ty T Ax + tx T Ay + t 2 y T Ay x T Ax = lim = lim(y T Ax + x T AY + ty T Ay) = y T Ax + x T Ay t 0 = y T Ax + (x T Ay) T since x T Ay is a 1 1 matrix and so is symmetric = y T Ax + y T A T x = y T (A + A T )x Notice that x 2 2 = xt x = x T Ix so this example shows that the directional derivative of the Euclidean norm exists in all directions y Note We now consider derivatives of vector valued functions of vectors, f : R n R m

3 42 Types of Differentiation 3 Definition Let f : S R m where S R n With f = f(x 1,x 2,,x n ) = (f 1 (x 1,x 2,,x n ),f 2 (x 1,x 2,,x n ),,f m (x 1,x 2,,x n )), define the matrix gradient f/ as the n m matrix with (i,j) entry f i / j : f 1 f f = f 1 2 f 2 2 f 1 n f 2 n This is also sometimes denoted f (Gentle uses the notation f T /) The m n matrix f) T = ( f) T is the Jacobian of f (denoted by Gentle as f/ T ) f m 1 f m 2 f m n Note You have seen the Jacobian when dealing with substitution in a multiple integral (it sort of plays the role of du in one variable u-substitution) See my online Calculus 3 notes for Substitutions in Multiple Integrals : edu/gardnerr/2110/notes-12e/c15s8pdf Definition Let Y be a p q matrix of functions f ij : R n R, Y = [f ij ] where f ij = f ij (x 1,x 2,,x m ) Define the derivative of Y as Y = [ f ij] That is, Y/ is a three dimensional object (or a matrix of vectors) of dimension p q n where the (i,j,k) entry is f ij / k For fixed k we denote the p q matrix with (i,j) entry as the (i,j,k ) entry of Y/ as Y/ k

4 42 Types of Differentiation 4 Note The linearity and chain rule properties of partial derivatives allow us to establish rules of differentiation in these new settings For example, if Y = [f ij ] where f ij : R n R is a square nonsingular matrix then we show in Exercise 43 that Y 1 = Y 1 ( ) Y Y 1 In Exercise 42A we show for Y = [f ij ] that [tr(y )] = tr ( ) Y Definition Let f : R n R be a scalar valued function of a vector The matrix H with (i,j) entry as is the Hessian of f, denoted H: i j H = n 1 Gentle denotes the Hessian as 2 f and x 2 n 2 2 f T 1 n 2 n 2 n Note In the case that f = f(x,y) then det(h) = det 2 y y y 2 = 2 f 2 y 2 ( 2 ) 2 f y (assuming the mixed partials are equals, which is really an assumption of continuity) and det(h) is related to the curvature of the surface z = f(x, y); see The Gauss Curvature in Detail from my Differential Geometry notes: etsuedu/gardnerr/5310/5310pdf/dg1-6pdf (see the example on pages 3 and

5 42 Types of Differentiation 5 4) Because of this, the Hessian (which is often defined as det(h) instead of H itself) can be used to find extrema and saddle points of z = f(x,y) in a style similar to the use of the Second Derivative Test for functions y = f(x) See Extreme Values and Saddle Points from Calculus 3: notes-12e/c14s7pdf Note Gentle lists f/ for several types of functions f(x) where x is an n- vector Let a be a constant scalar, b a constant conformable vector, and A a constant conformable matrix Then we have (with corrections to Gentle as given in the Errata): f(x) ax b T x x T b x T x xx T b T Ax x T Ab f/ ai b b 2x x I + I x A T b Ab x T Ax (A + A T )x exp(( 1/2)x T Ax) exp(( 1/2)x T Ax)Ax if A = A T x 2 2 2x V (x) 2x/(n 1) The proofs of some of these are to be given in Exercise 42B

6 42 Types of Differentiation 6 Definition Let X be a matrix of independent variables and f a function of X so that f = f(x) = f([x ij ]) The derivative of function f with respect to matrix X is the matrix f [ ] f = ij Theorem 421 Differentiation of scalar valued function f satisfies the following (1) f T = ( f ) T (2) For X square and f(x) = tr(x), f = I (3) For AX a square matrix where A is constant, [tr(ax)] (4) [tr(xt X)] = 2X (5) With a and b constant vectors, [at Xb] = abt (6) [det(x)] = (adj(x)) T = A T Note A quick search of the internet reveals that there are a number of definitions for the derivative of a matrix with respect to a matrix Inspired by Gentle s general approach, we take the following definition Definition Let X be a matrix of independent variables and let Y be a matrix of functions of X, Y = [y ij ] = [y ij (X)] The derivative of matrix Y with respect to matrix X is the matrix Y [ ] Y = ij

7 42 Types of Differentiation 7 Note Gentle claims that that the following hold Given his cryptic statement of equation (415) on page 155, I am not totally confident that all of the following hold under our definition of a matrix with respect to a matrix Note Gentle lists f/ for several types of functions f(x) where X is a matrix Let a and b be a conformable vectors, and A a constant conformable matrix Then we have: f(x) f/ Justification a T Xb ab T Theorem 421(5) tr(ax) A T Theorem 421(3) tr(x T X) 2X Theorem 421(4) BX XC BXC I B C T I C T B If X is square and invertible as required, then: f(x) f/ Justification tr(x) I Theorem 421(2) tr(x k ) kx k 1 Exercise 42C(1) tr(bx 1 C) (X 1 CBX 1 ) T det(x) detx(x 1 ) T Exercise 42C(2) log(det(x)) (X 1 ) T Exercise 42C(3) detx k kdet(x) k (X 1 ) T Exercise 42C(4) BX 1 C (X 1 C) T BX 1 Revised: 4/1/2018