B553 Lecture 3: Multivariate Calculus and Linear Algebra Review

Transcription

1 B553 Lecture 3: Multivariate Calculus and Linear Algebra Review Kris Hauser December 30, 2011 We now move from the univariate setting to the multivariate setting, where we will spend the rest of the class. In this lecture we review some fundamental concepts from multivariate calculus and linear algebra that will serve as a foundation for understanding multivariate optimization techniques. 1 Metric spaces We would like to extend concepts that are familiar to us from working wiht real numbers R to other spaces of interest. For example, through much of this class we will operate in n-dimensional Cartesian spaces R n. Elements of R n (points) are simply tuples of n real numbers. In this section we extend our understanding the notions of sequences, limits, derivatives, and closed and open intervals to a more general setting of metric spaces. A metric space S is an arbitrary abstract set equipped with a metric d(x, y) that measures a sense of distance between points. The metric is required to be 1) nonnegative (d(x, y) 0), 2) reflexive (d(x, y) = 0 iff x = y) and 3) symmetric (d(x, y) = d(y, x)). It is also required to satisfy the triangle inequality: d(x, y) d(x, z) + d(z, y) for all x, y, z. Examples: R is a metric space under the absolute distance metric d(x, y) = x y. Any Cartesian space R n is a metric space under the Euclidean distance metric (and indeed, many other metrics). 1

2 Any set is a metric space under the discrete metric: d(x, y) = 0 if x = y, and d(x, y) = 1 otherwise. We can easily generalize the notions of convergence and limits of sequences into the metric space setting. This is done simply by replacing the absolute distances x y with the metric d(x, y). Limits of a function also generalize, but only in the standard case (taken relative to some point c). The one-sided and asymptotic limits do not generalize in the same way, nor do the notions of minimum/maximum/infimum/supremum. Neighborhood. For a metric space (S, d), the r-neighborhood of a point x S is the set N r (x) = y S d(x, y) < r. For a Cartesian space R n and the Euclidean metric, this is a ball of dimension n and radius r centered on x, not including its surface. Open set. An open set A is a subset of a metric space (S, d) such that every point x A has an r-neighborhood completely contained within A with some r > 0. More precisely, for all x, there exists an r > 0 such that N r (x) A. (Note that r is allowed to depend on x.) Closed set. A closed set A is a subset of a metric space (S, d) such that its complement S \ A is an open set. (Note: typically the universe of possible elements S is known and is therefore left implicit.) Examples: The half-open interval (a, b] is neither open nor closed. Any finite set of points {x 1,..., x n } closed in R. The empty set is both open and closed. R is both open and closed (taking R as the universe). The union of any number of open sets is open. The intersection of any finite number of open sets is open. The union of any finite number of closed sets is closed. The intersection of any number of closed sets is closed. 2

3 Closure. The closure of a set A, denoted cl(a) is the set of all points x S such that for any r > 0, N r (x) A. From this definition, we can see that A cl(a), cl(a) is a closed set, and that A = cl(a) iff A is a closed set. Interior. The interior of a set A, denoted int(a) is the set of all points x S such that there exists an r > 0 such that N r (x) A. From this definition, we can see that int(a) A, int(a) is an open set, and that A = int(a) iff A is an open set. Boundary. The boundary of a set A, denoted A, is defined as cl(a)\int(a). Or, equivalently, it is the set of points for which all r-neighborhoods with r > 0 contain at least one point of A and one point of its complement. 2 Vector Spaces Cartesian spaces are not only metric spaces, but also vector spaces. Vector spaces are distinguished from generic metric spaces by the ability to transform vectors by addition and scaling, as well as a notion of directionality. Vectors are typically thought of by most engineers and computer scientists as tuples of real numbers. In truth, these numbers are just an interpretation of a more abstract essential concept they are primarily defined by how they transform under linear operations. In more advanced math topics (e.g., functional analysis) the abstract concepts are crucial, but in this class here we will primarily stick with the layman s definition. 2.1 Vectors In our layman s definition, an n-dimensional vector x is a tuple of real numbers x = (x 1,..., x n ) R n. We will use boldface notation only temporarily to help distinguish between vectors and real numbers. Note that in the future we will typically drop the boldface. The space of vectors R n is known as the vector space. Vectors can be added and subtracted component-wise, can be multipled by elements of R, and can be divided by elements of R \ {0}. There is a zero vector 0 with all elements equal to zero. Each element has a component-wise negation, x. We will on occasion refer to the standard basis vectors e 1,..., e n, where e i has all elements equal to 0 except for the i th element, which is equal to 1. 3

4 Dot product. The dot product is a function that takes two vectors x = (x 1,..., x n ) and y = (y 1,..., y n ) and returns a real number, and is given by the expression n x y = x i y i. (1) For example, x e i = x i. Orthogonal vectors. Two vectors whose dot product is identically zero are called orthogonal. For example, e i e j = 0 for all i j, and so e i and e i are orthogonal. Norms. A norm describes some notion of vector magnitude. The standard Euclidean norm is defined as x = x x. Some identities are 0 = 0, x > 0 if x 0, and cx = c x for all real values c. Unit vectors. A unit vector is a vector with unit norm: x = 1. Euclidean distance. The euclidean distance between two vectors is given by the norm of the difference between the vectors: d(x, y) = x y. It satisfies all the criteria of a metric, and hence vector spaces are metric spaces. L k norms. Besides the euclidean norm, we will occasionally use a more general set of L k norms (and their induced distance metrics). The L k norm is given by the expression: ( n ) 1/k x k = x i k. (2) In particular the L 1, L 2, and L norms are of greatest interest. The L 2 norm is simply the Euclidean norm. The L 1 norm is given by the sum of absolute values of the components of x. It gives rise to the Manhattan distance metric d(x, y) = x y 1. The L norm is given by the maximum absolute value of the components of x: x = max x i. (3),...,n Line equations. A line through points a, b is specified by the equation x(u) = (1 u)a + ub, for u R. The line segment between a and b is obtained by restricting u to [0, 1]. Plane equations. All points x on a (hyper)plane through the origin 0 satisfy the equation x a = 0, where a is nonzero vector orthogonal to the 4

5 plane surface. In fact any vector ca, where c is a nonzero scalar, defines the same plane. All points x on a plane that passes through another point b satisfy the equation x a = b a. A unique representation for all possible planes is x u = b where u is a unit vector orthogonal to the plane and b is a nonnegative offset that determines the distance away from the origin. Linear Combinations. x is a linear combination of a set of vectors {a 1,..., a m } if m x = u i a i (4) for some set of numbers u 1,..., u m. Span. The span of {a 1,..., a m }, denoted Span(a 1,..., a m ) is the set of all vectors x that are linear combinations of {a 1,..., a m }. Linear dependence/independence. A set of vectors {a 1,..., a m } is linearly dependent if at least one vector is a linear combination of the remaining vectors. If the set is not linearly dependent, then it is called linearly independent. Every set of m > n vectors is linearly dependent. Basis. A set of n unit vectors that are each mutually orthogonal and that span R n is known as an orthogonal basis for R n. 3 Scalar Functions on Vector Spaces We will often work with functions that map many variables (a vector) to a single number (a scalar). A scalar field f : R n R, is a real-valued function of n real-valued variables. We can either write the arguments explicitly as f(x 1,..., x n ) or as a single vector argument as f(x). The same notions of continuity, minima, and maxima that apply to univariate functions also apply to scalar fields. Continuity. A scalar field f is continuous on a domain S R n if for every point x S and ɛ > 0, there exists a neighborhood with radius δ around x that satisfies f(x) f(y) < ɛ for all y N δ (x). Extreme value theorem. A continuous scalar field f attains a minimum and maximum value on a closed set S R n. Local minima. x is said to be a local minimum of a scalar field f on an 5

6 open set S R n if there exists a neighborhood N r (x) around x such that f(x) < f(y) for all y N r (x) \ {x}. A similar definition holds for local maxina. 3.1 Partial derivatives Partial derivatives allow taking derivatives of a scalar field with respect to certain individual arguments. The partial derivative of f with respect to its argument x i at the values (x 1,..., x n ) = (a 1,..., a n ) is given by f f(a 1,..., a i 1, a i + h, a i+1,..., a n ) f(a 1,..., a n ) (a 1,..., a n ) = lim x i h 0 h (5) In other words, this is the deriative in the direction of the a i while fixing all of the other a j s, i j, constant. We can write this definition more compactly in vector notation: f f(a + he i ) f(a) (a) = lim x i h 0 h A third possible interpretation is that we are taking the derivative of the function g i (x) = f(a 1,..., a i 1, x, a i+1,..., a n ) (7) at x = a i. In other words, g i(x) x=ai = f x i (a). (Note that some other texts may use the notation f i to denote the partial derivative with respect to the i th argument to f. I find this notation to be extremely confusing so we will use the f x i notation throughout this class.) Partial differentiation operator. The notation x i refers to the partial differentiation operator on the i th argument to f. For shorthand, I may occasionally write. i Finding partial derivatives. The rules for computing regular derivatives can also be applied when computing partial derivatives. The only difference is that all arguments, except for the one being differentiated, are treated as constants. As an exmaple, consider The partial derivatives are: (6) f(x 1, x 2 ) = x k 1e cx 2. (8) x 1 f(x 1, x 2 ) = kx k 1 1 e cx 2 (9) 6

7 because x 2 is treated as a constant, and x 2 f(x 1, x 2 ) = cx k 1e cx 2 (10) because x 1 is treated as a constant. Be careful when a variable appears in multiple arguments. For example, given a function f(x, g(x)), the partial derivative of f with respect to x 1 does not use the partial derivative with respect to x 2, or the derivative of g in its calculation. Instead, you should treat the second argument as a constant y and proceed as though it has no dependence on x. You should incorporate those derivatives when computing the total derivative with respect to x, which is denoted d or. The total derivative treats dx the expression as a single function, h(x) = f(x, g(x)), and involves a chain rule with both partial derivatives: d dx f(x, g(x)) = h (x) = x 1 f(x, g(x)) + x 2 f(x, g(x))g(x). (11) Directional derivatives. The directional derivative is another type of derivative that measures the rate of change of the scalar field f as you move from a in a direction d. You can think about moving along a ray x(u) = a + ud standing at a and beginning to move with velocity d. The directional derivative measures the slope of the field in your desired direction. Precisely, the directional derivative of f at a in direction d is the value: f(a + hd) f(a) d f(a) = lim. (12) h 0 h Note that the standard partial derivative i f(a) is equivalent to ei f(a). Gradient. A fundamental result in multivariate calculus is that all directional derivatives can be expressed as a dot product of d with a vector known as the gradient of f. The gradient of f, denoted f(a), is the vector of all partial derivatives of f: f(a) = We will prove the key result that ( x 1 f(a),..., ) f(a). (13) x n d f(a) = ( f(a)) d (14) 7

8 We will need a lemma that directional derviatives are linear, specifically that: cd f(a) = c d f(a) (15) and for any scalar c, vectors d, d 1, and d 2. First, scaling: d+ei f(a) = d f(a) + ei f(a) (16) f(a + chd) f(a) cd f(a) = lim h 0 h f(a + chd) f(a) = c lim h 0 ch f(a + chd) f(a) ch f(a + hd) f(a) h 0 = c lim (ch) 0 = c lim = c d f(a) h (17) Now, let s look at summing. d1 +d 2 f(a) = lim h 0 f(a + hd 1 + hd 2 ) f(a) h (18) Now let g h (x) = f(a + hd 1 + xd 2 ) and evaluate the Taylor expansion of g h (x) around x = 0: g h (x) = f(a + hd 1 ) + g h(x) x=0 x + O(x 2 ) (19) So d1 +d 2 f(a) = lim h [f(a + hd 1) + g h(0)h + O(h 2 ) f(a)] ( ) 1 = lim h 0 h [f(a + hd 1) f(a) + O(h 2 )] + g h(0) h 0 1 = d1 f(a) + lim h 0 g h(x) x=0. (20) The desired result is obtained by noting that lim h 0 g h (x) x=0 is identically d2 f(a). 8

9 Returning to the original question, we use linearity of the directional derivative to obtain n ( f(a)) d = d i ei f(a) as desired. QED. = n di e i f(a) = n d ie i f(a) = d f(a) (21) First-order Taylor expansion of a scalar field. If a scalar field f is differentiable, we can use the gradient to define the first-order Taylor expansion of f about a point a: f(x) = f(a) + ( f(a)) (x a) + O( x a 2 ). (22) Note that the equation f(x) = f(a) + f(a) (x a) defines a plane in the (x, f) space. This says that the first two terms of the Taylor expansion give the plane that is tangent to the surface defined by f at a. Moreover, when x is close to a, the fit is quite good. It therefore serves a role that is very much like the tangent line found by the Taylor expansion of a univariate function. [We will examine higher-order Taylor expansions in future lectures.] Differentiation rules. Here, let f and g be scalar fields, let h be a univariate function, and let c be a scalar. Linearity. (f(x) + cg(x)) = f(x) + c g(x). Chain rule. h(f(x)) = h (f(x)) f(x). Multiplication rule. (fg)(x)) = f(x) g(x) + g(x) f(x). There is another version of the chain rule that is occasionally useful. Let y(u) be a vector space curve parameterized by u R. Then d du (f(y(u))) = f(y(u)) y (u). (23) In other words, you get the directional derivative of f in the direction y (u). 9

10 4 Caveats Most of the techniques we will look at in class operate only on vector spaces. But sometimes we would like to work on spaces that are not vector spaces. For example, the spaces of angles and rigid-body rotations are not vector spaces, but they are important in computer vision, robotics, and biomechanics. The spaces of directions (unit vectors) or lines are also important in many fields, but they too are not vector spaces. To perform optimization in such spaces S using a vector space framework, we must be clever in embedding S into a vector space V, with constraints that eliminate regions of V that do not map to sensical points in S. For example, unit vectors in R n can be represented by their components, but constrained to the unit hypersphere x = 1. 5 Exercises 1. Prove from first principles that the unit ball in the plane R 2, B = {(x, y) x 2 + y 2 1} is closed. Prove that the unit ball without its boundary, B = {(x, y) x 2 + y 2 < 1}, is open. 2. The Manhattan distance in R n is defined as follows: n d(x, y) = x i y i, (24) where x = (x 1,..., x n ) and y = (y 1,..., y n ). In other words, it adds up each of the absolute distances along each of the coordinate axes. Prove that the Manhattan distance is a metric. 3. What is the closure of the set {1/n n = 1, 2,...}? What is its interior? What is its boundary? 4. What is the closure of the set {(x, y) x > 3.5, y < 2}? What is its interior? What is its boundary? 5. The y-level set of a scalar field f is the set of all points x such that f(x) = y. Show that if the gradient f(x) is nonzero, then near x the f(x)-level set is a curve, and furthermore this curve is orthogonal to f(x) when it passes through at x. 10