The Complex Gradient Operator and the CR-Calculus

Transcription

1 arxiv: v1 [math.oc] 26 Jun Introduction The Complex Gradient Operator and the CR-Calculus June 25, 2009 Ken Kreutz Delgado Electrical and Computer Engineering Jacobs School of Engineering University of California, San Diego VERSION UCSD-ECE275CG-S2009v1.0 Copyright c , All Rights Reserved Often signals and system parameters are most conveniently represented as complex-valued vectors. This occurs, for example, in array processing [1], as well as in communication systems [7] when processing narrowband signals using the equivalent complex baseband representation [2]. Furthermore, in many important applications one attempts to optimize a scalar real-valued measure of performance over the complex parameters defining the signal or system of interest. This is the case, for example, in LMS adaptive filtering where complex filter coefficients are adapted on line. To effect this adaption one attempts to optimize the performance measure by adjustments of the coefficients along its stochastic gradient direction [16, 23]. However, an often confusing aspect of complex LMS adaptive filtering, and other similar gradient-based optimization procedures, is that the partial derivative or gradient used in the adaptation of complex parameters is not based on the standard complex derivative taught in the standard mathematics and engineering complex variables courses [3]-[6], which exists if and only if a function of a complex variable z is analytic in z. 1 This is because a nonconstant real-valued function of a complex variable is not (complex) analytic and therefore is not differentiable in the standard textbook complex-variables sense. 1 I.e., complex-analytic. 1

2 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v1.0 2 Nonetheless, the same real-valued function viewed as a function of the real-valued real and imaginary components of the complex variable can have a (real) gradient when partial derivatives are taken with respect to those two (real) components. In this way we can shift from viewing the real-valued function as a non-differentiable mapping between C and R to treating it as a differentiable mapping between R 2 and R. Indeed, the modern graduate-level textbook in complex variables theory by Remmert [12] continually and easily shifts back and forth between the real function R 2 R or R 2 perspective and the complex function C C perspective of a complex or real scalar-valued function, f(z) = f(r) = f(x, y), of a complex variable z = x + j y, z C r = ( ) x R 2. y In particular, when optimizing a real-valued function of a complex variable z = x + j y one can work with the equivalent real gradient of the function viewed as a mapping from R 2 to R in lieu of a nonexistent complex derivative [14]. However, because the real gradient perspective arises within a complex variables framework, a direct reformulation of the problem to the real domain is awkward. Instead, it greatly simplifies derivations if one can represent the real gradient as a redefined, new complex gradient operator. As we shall see below, the complex gradient is an extension of the standard complex derivative to non-complex analytic functions. Confusing the issue is the fact that there is no one unique way to consistently define a complex gradient which applies to (necessarily non-complex-analytic) real-valued functions of a complex variable, and authors do not uniformly adhere to the same definition. Thus it is often difficult to resolve questions about the nature or derivation of the complex gradient by comparing authors. Given the additional fact that typographical errors seem to be rampant these days, it is therefore reasonable to be skeptical of the algorithms provided in many textbooks especially if one is a novice in these matters. An additional source of confusion arises from the fact that the derivative of a function with respect to a vector can be alternatively represented as a row vector or as a column vector when a space is Cartesian, 2 and both representations can be found in the literature. In this note we carefully distinguish between the complex cogradient operator (covariant derivative operator [22]), which is a row vector operator, and the associated complex gradient operator which is a vector operator which gives the direction of steepest ascent of a real scalar-valued function. Because of the constant back-and-forth shift between a real function ( R-calculus ) perspective and a complex function ( C-calculus ) perspective which a careful analysis of nonanalytic complex functions requires [12], we refer to the mathematics framework underlying the derivatives given in this note as a CR-calculus. In the following, we start by reviewing some of the properties of standard univariate analytic functions, describe the CR-calculus for univariate nonanalytic functions, and then develop a multivariate second order CR-calculus appropriate for optimizing scalar real-valued cost functions of a complex parameter vector. We end the note with some examples. 2 I.e., is Euclidean with identity metric tensor.

3 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v The Derivative of a Holomorphic Function Let z = x + jy, for x, y real, denote a complex number and let f(z) = u(x, y) + j v(x, y) be a general complex-valued function of the complex number z. 3 In standard complex variables courses it is emphasized that for the complex derivative, f (z) = lim z 0 f(z + z) f(z), z to exist in a meaningful way it must be independent of the direction with which z approaches zero in the complex plane. This is a very strong condition to be placed on the function f(z). As noted in an introductory comment from the textbook by Flanigan [6]: You will learn to appreciate the difference between a complex analytic function (roughly a complex-valued function f(z) having a complex derivative f (z)) and the real functions y = f(x) which you differentiated in calculus. Don t be deceived by the similarity of the notations f(z), f(x). The complex analytic function f(z) turns out to be much more special, enjoying many beautiful properties not shared by the run-of-the-mill function from ordinary real calculus. The reason [ ] is that f(x) is merely f(x) whereas the complex analytic function f(z) can be written as f(z) = u(x,y) + iv(x,y), where z = x + iy and u(x,y), v(x,y) are each real-valued harmonic functions related to each other in a very strong way: the Cauchy-Riemann equations u x = v y In summary, the deceptively simple hypothesis that v x = u y. (1) f (z) exists forces a great deal of structure on f(z); moreover, this structure mirrors the structure of the harmonic u(x,y) and v(x,y), functions of two real variables. 4 In particular the following conditions are equivalent statements about a complex function f(z) on an open set containing z in the complex plane [6]: 3 Later, in Section 3, we will interchangeably alternate between this notation and the more informative notation f(z, z). Other useful representations are f(u, v) and f(x, y). In this section we look for the (strong) conditions for which f : z f(z) C is differentiable as a mapping C C (in which case we say that f is C-differentiable), but in subsequent sections we will admit the weaker condition that f : (x, y) (u, v) be differentiable as a mapping R 2 R 2 (in which case we say that f is R-differentiable); see Remmert [12] for a discussion of these different types of differentiability. 4 Quoted from page 2 of reference [6]. Note that in the quote i = 1 whereas in this note we take j = 1 following standard electrical engineering practice.

4 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v1.0 4 The derivative f (z) exists and is continuous. The function f(z) is holomorphic (i.e, complex-analytic in z). 5 The function f(z) satisfies the Cauchy-Riemann conditions (1). All derivatives of the function f(z) exist and f(z) has a convergent power series. Furthermore, it is a simple consequence of the Cauchy-Riemann conditions that f(z) = u(x, y) + j v(x, y) is holomorphic only if the functions u(x, y) and v(x, y) both satisfy Laplace s equation 2 u(x, y) x u(x, y) y 2 = 0 and 2 v(x, y) x v(x, y) y 2 = 0. Such functions are known as harmonic functions. Thus if either u(x, y) or v(x, y) fail to be harmonic, the function f(z) is not differentiable. 6 Although many important complex functions are holomorphic, including the functions z n, e z, ln(z), sin(z), and cos(z), and hence differentiable in the standard complex variables sense, there are commonly encountered useful functions which are not: The function f(z) = z, where z denotes complex conjugation, fails to satisfy the Cauchy- Riemann conditions. The functions f(z) = Re(z) = z+ z = x and g(z) = Im(z) = z z 2 2j Riemann conditions. = y fail the Cauchy- The function f(z) = z 2 = zz = x 2 + y 2 is not harmonic. 5 A function is analytic on some domain if it can be expanded in a convergent power series on that domain. Although this condition implies that the function has derivatives of all orders, analyticity is a stronger condition than infinite differentiability as there exist functions which have derivatives of all orders but which cannot be expressed as a power series. For a complex-valued function of a complex variable, the term (complex) analytic has been replaced in modern mathematics by the entirely synonymous term holomorphic. Thus real-valued power-series-representable functions of a real-variable are analytic (real-analytic), while complex-valued power-series-representable functions of a complexvariable are holomorphic (complex-analytic). We can now appreciate the merit of distinguishing between holomorphic and (real) analytic functions a function can be nonholomorphic (i.e. non-complex-analytic) in the complex variable z = x + j y yet still be (real) analytic in the real variables x and y. 6 Because a harmonic function on R 2 satisfies the partial differential equation known as Laplace s equation, by existence and uniqueness of the solution to this partial differential equation its value is completely determined at a point in the interior of any simply connected region which contains that point once the values on the boundary (boundary conditions) of that region are specified. This is the reason that contour integration of a complex-analytic (holomorphic) function works and that we have the freedom to select that contour to make the integration as easy as possible. On the other hand, there is, in general, no equivalent to contour integration for an arbitrary function on R 2. See the excellent discussion in Flanigan [6].

5 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v1.0 5 Any nonconstant purely real-valued function f(z) (for which it must be the case that v(z, y) 0) fails the Cauchy-Riemann condition. In particular the real function f(z) = z = zz = x2 + y 2 is not differentiable. 7 Note in particular, the implication of the above for the problem of minimizing the real-valued squared-error loss functional l(a) = E { η k āξ k 2} { } = E (η k āξ k )(η k āξ k ) E {ē k e k } (2) for finite second-order moments stationary scalar complex random variables ξ k and η k, and unknown complex constant a = a x + ja y. Using the theory of optimization in Hilbert spaces, the minimization can be done by invoking the projection theorem (which is equivalent to the orthogonality principle) [34]. Alternatively, the minimization can be performed by completing the square. Either procedure will result in the Wiener-Hopf equations, which can then be solved for the optimal complex coefficient variable a. However, if a gradient procedure for determining the optimum is desired, we are immediately stymied by the fact that the purely real nonconstant function l(a) is not complex-analytic (holomorphic) and therefore its derivative with respect to a does not exist in the conventional sense of a complex derivative [3]-[6], which applies only to holomorphic functions of a. A way to break this impasse will be discussed in the next section. Meanwhile note that all of the real-valued nonholomorphic functions shown above can be viewed as functions of both z and its complex conjugate z, as this fact will be of significance in the following discussion. 3 Extensions of the Complex Derivative The CR-Calculus In this section we continue to focus on functions of a single complex variable z. The primary references for the material developed here are Nehari [11], Remmert [12], and Brandwood [14]. 3.1 A Possible Extension of the Complex Derivative. As we have seen, in order for the complex derivative of a function of z = x + j y, f(z) = u(x, y) + j v(x, y), to exist in the standard holomorphic sense, the real partial derivatives of u(x, y) and v(x, y) must not only exist, they must also satisfy the Cauchy-Riemann conditions (1). As noted by Flanigan [6]: This is much stronger than the mere existence of the partial derivatives. However, the mere existence of the (real) partial derivatives is necessary and sufficient for a stationary point 7 Thus we have the classic result that the only holomorphic real-valued functions are the constant real-valued functions.

6 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v1.0 6 of a (necessarily nonholomorphic) non-constant real-valued functional f(z) to exist when f(z) is viewed as a differentiable function of the real and imaginary parts of z, i.e., as a function over R 2, f(z) = f(x, y) : R 2 R. (3) Thus the trick is to exploit the real R 2 vector space structure which underlies C when performing gradient-based optimization. In essence, the remainder of this note is concerned with a thorough discussion of this trick. Towards this end, it is convenient to define a generalization or extension of the standard partial derivative to nonholomorphic functions of z = x + j y that are nonetheless differentiable with respect to x and y and which incorporates the real gradient information directly within the complex variables framework. After Remmert [12], we will call this the real-derivative, or R-derivative, of a possibly nonholomorphic function in order to avoid confusion with the standard complexderivative, or C-derivative, of a holomorphic function which was presented and discussed in the previous section. Furthermore, we would like the real-derivative to reduce to the standard complex derivative when applied to holomorphic functions. Note that if one rewrites the real-valued loss function (2) in terms of purely real quantities, one obtains (temporarily suppressing the time dependence, k) l(a) = l(a x, a y ) = E { e 2 x + e2 y} = E { (ηx a x ξ x a y ξ y ) 2 + (η y + a y ξ x a x ξ y ) 2}. (4) From this we can easily determine that and l(a x, a y ) a x = 2 E {e x ξ x + e y ξ y }, l(a x, a y ) a y = 2 E {e x ξ y e y ξ x }. Together these can be written as ( + j ) l(a) = l(a x, a y ) + j l(a x, a y ) = 2 E {ξ k ē k } (5) a x a y a x a y which looks very similar to the standard result for the real case. Indeed, equation (5) is the definition of the generalized complex partial derivative often given in engineering textbooks, including references [7]-[9]. However, this is not the definition used in this note, which instead follows the formulation presented in [10]-[20]. We do not use the definition (5) because it does not reduce to the standard C-derivative for the case when a function f(a) is a holomorphic function of the complex variable a. For example, take the simplest case of f(a) = a, for which the standard derivative yields d f(a) = 1. In this case, the definition (5) applied to da f(a) unfortunately results in the value 0. Thus we will not view the definition (5) as an admissible generalization of the standard complex partial derivative, although it does allow the determination of the stationary points of l(a). 8 8 In fact, it is a scaled version of the conjugate R-derivative discussed in the next subsection.

7 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v The R-Derivative and Conjugate R-Derivative. There are a variety of ways to develop the formalism discussed below (see [11]-[14]). Here, we roughly follow the development given in Remmert [12] with additional material drawn from Brandwood [14] and Nehari [11]. Note that the nonholomorphic (nonanalytic in the complex variable z) functions given as examples in the previous section can all be written in the form f(z, z), where they are holomorphic in z = x + j y for fixed z and holomorphic in z = x j y for fixed z. 9 It can be shown that this fact is true in general for any complex- or real-valued function f(z) = f(z, z) = f(x, y) = u(x, y) + j v(x, y) (6) of a complex variable for which the real-valued functions u and v are differentiable as functions of the real variables x and y. This fact underlies the development of the so-called Wirtinger calculus [12] (or, as we shall refer to it later, the CR-calculus.) In essence, the so-called conjugate coordinates, Conjugate Coordinates: c (z, z) T C C, z = x + j y and z = x j y (7) can serve as a formal substitute for the real r = (x, y) T representation of the point z = x+j y C [12]. 10 According to Remmert [12], the calculus of complex variables utilizing this perspective was initiated by Henri Poincaré (over 100 years ago!) and further developed by Wilhelm Wirtinger in the 1920 s [10]. Although this methodology has been fruitfully exploited by the German-speaking engineering community (see, e.g., references [13] or [31]), it has not generally been appreciated by the English speaking engineering community until relatively recently. 11 For a general complex- or real-valued function f(c) = f(z, z) consider the pair of partial derivatives of f(c) formally 12 defined by R-Derivative of f(c) f(z, z) and Conjugate R-Derivative of f(c) z= const. f(z, z) (8) z= const. 9 That is, if we make the substitution w = z, they are analytic in w for fixed z, and analytic in z for fixed w. This simple insight underlies the development given in Brandwood [14] and Remmert [12]. 10 Warning! The interchangeable use of the various notational forms of f implicit in the statement f(z) = f(z, z) can lead to confusion. To minimize this possibility we define the term f(z) (z-only) to mean that f(z) is independent of z (and hence is holomorphic) and the term f( z) ( z only) to mean that f(z) is a function of z only. Otherwise there are no restrictions on f(z) = f(z, z). 11 An important exception is Brandwood [14] and the work that it has recently influenced such as [1, 15, 16]. However, these latter references do not seem to fully appreciate the clarity and ease of computation that the Wirtinger calculus (CR-calculus) can provide to the problem of differentiating nonholomorphic function and optimizing realvalued functions of complex variables. Perhaps this is do to the fact that [14] did not reference the Wirtinger calculus as such, nor cite the rich body of work which had already existed in the mathematics community ([11, 18, 12]). 12 These statements are formal because one cannot truly vary z = x + j y while keeping z = x j y constant, and vice versa.

8 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v1.0 8 where the formal partial derivatives are taken to be standard complex partial derivatives (C-derivatives) taken with respect to z in the first case and with respect to z in the second. 13 For example, with f(z, z) = z 2 z we have f = 2z z and f = z2. As denoted in (8), we call the first expression the R-derivative (the real-derivative) and the second expression the conjugate R-derivative (or R-derivative). It is proved in [11, 14, 12] that the R-derivative and R-derivative formally defined by (8) can be equivalently written as 14 f = 1 2 ( ) f x j f y and f = 1 2 ( ) f x + j f y where the partial derivatives with respect to x and y are true (i.e., non-formal) partial derivatives of the function f(z) = f(x, y), which is always assumed in this note to be differentiable with respect to x and y (i.e., to be R-differentiable). Thus it is the right-hand-sides of the expressions given in (9) which make rigorous the formal definitions of (8). Note that from equation (9) that we immediately have the properties Comments: = = 1 and = = 0. (10) 1. The condition f = 0 is true for an R-differentiable function f if and only the Cauchy- Riemann conditions are satisfied (see [11, 14, 12]). Thus a function f is holomorphic (complex-analytic in z) if and only if it does not depend on the complex conjugated variable z. I.e., if and only if f(z) = f(z) (z only) The R-derivative, f, of an R-differentiable function f is equal to the standard C-derivative, f (z), when f(z, z) is independent of z, i.e., when f(z) = f(z) (z only). 13 A careful and rigorous analysis of these formal partial derivatives can be found in Remmert [12]. In [12], a differentiable complex function f is called C-differentiable while if f is differentiable as a mapping from R 2 R 2, it is said to be real-differentiable (R-differentiable) (See Footnote 3). It is shown in [12] that the partial derivatives (8) exist if and only if f is R-differentiable. As discussed further below, throughout this note we assume that all functions are globally real-analytic (R-analytic), which is a sufficient condition for a function to be globally R-differentiable. 14 Recall the representation f = f(x, y) = u(x, y) + j v(x, y). Note that the relationships (9) make it clear why the partial derivatives (8) exist if and only if f is R-differentiable. (See footnotes 3 and 13). 15 This obviously provides a simple and powerful characterization of holomorphic and nonholomorphic functions and shows the elegance of the Wirtinger calculus formulation based on the use of conjugate coordinates (z, z). Note that the two Cauchy-Riemann conditions are replaced by the single condition f = 0. The reader should reexamine the nonholomorphic (nonanalytic in z) functions discussed in the previous section in the light of this condition. (9)

9 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v An R-differentiable function f is holomorphic in z (complex-analytic in z) if and only if it does not depend on the variable z, f(z, z) = f( z) ( z only), which is true if and only if f = 0. To summarize, an R-differentiable function f is holomorphic (complex-analytic in z) if and only if f(z) = f(z) (z only), which is true if and only if f = 0, in which case the R-derivative coincides with the standard C-derivative, f = f (z). We call the single condition f = 0 the Cauchy-Riemann condition for f to be holomorphic: Cauchy Riemann Condition: f = 0 (11) Real-Analytic Complex Functions. Throughout the discussion given above we have been making the assumption that a complex function f is real differentiable (R-differentiable). We henceforth make the stronger assumption that complex functions over C are globally real-analytic (Ranalytic) over R 2. As discussed above, and rigorously proved in Remmert [12], R-analytic functions are R-differentiable and R-differentiable. A function f(z) has a power series expansion in the complex variable z, f(z) = f(z 0 ) + f (z 0 )(z z 0 ) f (z 0 )(z z 0 ) n! f(n) (z 0 )(z z 0 ) n + where the complex coefficient f (n) (z 0 ) denotes an n-times C-derivative of f(z) evaluated at the point z 0, if and only if it is holomorphic in an open neighborhood of z 0. If the function f(z) is not holomorphic over C, so that the above expansion does not exist, but is nonetheless still R-analytic as a mapping from R 2 to R 2, then the real and imaginary parts of f(z) = u(x, y) + j v(x, y), z = x + j y, can be expanded in terms of the real variables r = (x, y) T, u(r) = u(r 0 ) + u(r 0) (r r 0 ) + (r r 0 ) T r r v(r) = v(r 0 ) + v(r 0) (r r 0 ) + (r r 0 ) T r r ( ) T u(r0 ) (r r 0 ) + r ( ) T v(r0 ) (r r 0 ) + Note that if the R-analytic function is purely real, then f(z) = u(x, y) and we have f(r) = f(r 0 ) + f(r 0) (r r 0 ) + (r r 0 ) T r r r ( ) T f(r0 ) (r r 0 ) + r

10 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v Properties of the R- and R-Derivatives. The R-derivative and R-derivative are both linear operators which obey the product rule of differentiation. The following important and useful properties also hold (see references [11, 12]). 16 Complex Derivative Identities: f ( ) f = (12) f ( ) f = (13) df = f f dz + d z Differential Rule (14) h(g) = h g g + h ḡ ḡ Chain Rule (15) h(g) = h g g + h ḡ ḡ Chain Rule (16) As a simple consequence of the above, note that if f(z) is real-valued then f(z) = f(z) so that we have the additional very important identity that ( ) f f(z) R = f (17) As a simple first application of the above, note that the R-derivative of l(a) can be easily computed from the definition (2) and the above properties to be { } l(a) ā = E {ē ēk ke k } = E ā e e k k + ē k = E {0 e k ē k ξ k } = E {ξ k ē k }. (18) ā which is the same result obtained from the brute force method based on deriving expanding the loss function in terms of the real and imaginary parts of a, followed by computing (5) and then using the result (9). Similarly, it can be easily shown that the R-derivative of l(a) is given by l(a) a = E { ξk e k }. (19) Note that the results (18) and (19) are the complex conjugates of each other, which is consistent with the identity (17). We view the pair of formal partial derivatives for a possibly nonholomorphic function defined by (8) as the natural generalization of the single complex derivative (C-derivative) of a holomorphic 16 In the following for z = x + j y we define dz = dx + j dy and d z = dx j dy, while h(g) = h g denotes the composition of the two function h and g.

11 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v function. The fact that there are two derivatives under general consideration does not need to be developed in elementary standard complex analysis courses where it is usually assumed that f is always holomorphic (complex-analytic in z). In the case when f is holomorphic then f is independent of z and the conjugate partial derivative is zero, while the extended derivative reduces to the standard complex derivative. First-Order Optimality Conditions. As mentioned in the introduction, we are often interested in optimizing a scalar function with respect to the real and imaginary parts r = (x, y) T of a complex number z = x + j y. It is a standard result from elementary calculus that a first-order necessary condition for a point r 0 = (x 0, y 0 ) T to be an optimum is that this point be a stationary point of the loss function. Assuming differentiability, stationarity is equivalent to the condition that the partial derivatives of the loss function with respect the parameters r = (x, y) T vanish at the point r = (x 0, y 0 ) T. The following fact is an easy consequence of the definitions (8) and is discussed in [14]: A necessary and sufficient condition for a real-valued function, f(z) = f(x, y), z = x +j y, to have a stationary point with respect to the real parameters r = (x, y) T R 2 is that its R- derivative vanishes. Equivalently, a necessary and sufficient condition for f(z) = f(x, y) to have a stationary point with respect to r = (x, y) T R 2 is that its R-derivative vanishes. For example, setting either of the derivatives (18) or (19) to zero results in the so-called Wiener- Hopf equations for the optimal MMSE estimate of a. This result can be readily extended to the multivariate case, as will be discussed later in this note. The Univariate CR-Calculus. As noted in [12], the approach we have been describing is known as the Wirtinger calculus in the German speaking countries, after the pioneering work of Wilhelm Wirtinger in the 1920 s [10]. Because this approach is based on being able to apply the calculus of real variables to make statements about functions of complex variables, in this note we use the term CR-calculus interchangeable with Wirtinger calculus. Despite the important insights and ease of computation that it can provide, it is the case that the use of conjugate coordinates z and z (which underlies the CR-calculus) is not needed when developing the classical univariate theory of holomorphic (complex-analytic in z) functions. 17 It is only in the multivariate and/or nonholonomic case that the tools of the CR-calculus begin to be indispensable. Therefore it is not developed in the standard courses taught to undergraduate engineering and science students in this country [3]-[6] which have changed little in mode of presentation from the earliest textbooks The differential calculus of these operations... [is]... largely irrelevant for classical function theory... R. Remmert [12], page For instance, the widely used textbook by Churchill [3] adheres closely to the format and topics of its first edition which was published in The latest edition (the 7th at the time of this writing) does appear to have one brief homework problem on differentiating nonholomorphic functions.

12 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v Ironically, the elementary textbook by Nehari [11] was an attempt made in 1961 (almost 50 years ago!) to integrate at least some aspects of the CR-calculus into the elementary treatment of functions of a single complex variable. 19 However, because the vast majority of textbooks treat the univariate case, as long as the mathematics community, and most of the engineering community, was able to avoid dealing with nonholomorphic functions, there was no real need to bring the ideas of the CR-calculus into the mainstream univariate textbooks. Fortunately, an excellent, sophisticated and extensive introduction to univariate complex variables theory and the CR-calculus is available in the textbook by Remmert [12], which is a translation from the 1989 German edition. This book also details the historical development of complex analysis. The highly recommended Remmert and Nehari texts have been used as primary references for this note (in addition to the papers by Brandwood [14] and, most importantly for the second-order analysis given below, van den Bos [25]). The Multivariate CR-Calculus. Although one can forgo the tools of the CR-calculus in the case of univariate holomorphic functions, this is not the situation in the multivariate holomorphic case where mathematicians have long utilized these tools [17]-[20]. 20 Unfortunately, multivariate complex analysis is highly specialized and technically abstruse, and therefore virtually all of the standard textbooks are accessible only to the specialist or to the aspiring specialist. It is commonly assumed in these textbooks that the reader has great facility with differential geometry, topology, calculus on manifolds, and differential forms, in addition to a good grasp of advanced univariate complex variables theory. Moreover, because the focus of the theory of multivariate complex functions is primarily on holomorphic functions, whereas our concern is the essentially ignored (in this literature) case of nonholomorphic real-valued functionals, it appears to be true that only a very small part of the material presented in these references is directly useful for our purposes (and primarily for creating a rigorous and self-consistent multivariate CR-calculus framework based on the results given in the papers by Brandwood [14] and van den Bos [25]). The clear presentation by Brandwood [14] provides a highly accessible aspect of the first-order multivariate CR-calculus as applied to the problem of finding stationary values for real-valued functionals of complex variables. 21 As this is the primary interest of many engineers, this pithy paper is a very useful presentation of just those very few theoretical and practical issues which are needed to get a clear grasp of the problem. Unfortunately, even twenty years after its publication, this paper still is not as widely known as it should be. However, the recent utilization of the Brandwood results in [1, 13, 15, 16] seems to indicate a standardization of the Brandwood presentation of the complex gradient into the mainstream textbooks. The results given in the Brandwood paper [14] are particulary useful when coupled with with the significant extension of Brandwood s 19 This is still an excellent textbook that is highly recommended for an accessible introduction to the use of derivatives based on the conjugate coordinates z and z. 20 [The CR-calculus] is quite indispensable in the function theory of several variables. R. Remmert [12], page Although, as mentioned in an earlier footnote, Brandwood for some reason did not cite or mention any prior work relating to the use of conjugate coordinates or the Wirtinger calculus.

13 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v results to the problem of computing complex Hessians which has been provided by van den Bos s paper [25]. At this still relatively early stage in the development of a widely accepted framework for dealing with real-valued (nonholomorphic) functions of several complex variables, presumably even the increasingly widely used formalism of Brandwood [14] and van den Bos [25] potentially has some room for improvement and/or clarification (though this is admittedly a matter of taste). In this spirit, and mindful of the increasing acceptance of the approach in [14] and [25], in the remainder of this note we develop a multivariate CR-calculus framework that is only slightly different than that of [14] and [25], incorporating insights available from the literature on the calculus of multivariate complex functions and complex differential manifolds [17]-[20] Multivariate CR-Calculus The remaining sections of this note will provide an expanded discussion and generalized presentation of the multivariate CR-calculus as presented in Brandwood [14] and van den Bos [25]. The discussion given below also utilizes insights gained from references [17, 18, 19, 20, 21, 22]. 4.1 The Space Z = C n. We define the n-dimensional column vector z by z = ( z 1 z n ) T Z = C n where z i = x i + j y i, i = 1,, n, or, equivalently, z = x + j y with x = (x 1 x n ) T and y = (y 1 y n ) T. The space Z = C n is a vector space over the field of complex numbers with the standard component-wise definitions of vector addition and scalar multiplication. Noting the one-to-one correspondence ( x z C n r = R R y) 2n = R n R n it is evident that there exists a natural isomorphism between Z = C n and R = R 2n. The conjugate coordinates of z C n are defined by z = ( z 1 z n ) T Z = C n 22 Realistically, one must admit that many, and likely most, practicing engineers will be unlikely to make the move from the perspective and tools provided by [14] and [25] (which already enable the engineer to solve most problems of practical interest) to that developed in this note, primarily because of the requirement of some familiarity of (or willingness to learn) concepts of differential geometry at the level of the earlier chapters of [21] and [22]).

14 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v We denote the pair of conjugate coordinate vectors (z, z) by ( ) z c C 2n = C n C n z Noting that c, (z, z), z, (x, y), and r are alternative ways to denote the same point z = x + j y in Z = C n, for a function f : C n C m throughout this note we will use the convenient (albeit abusive) notation f(c) = f(z, z) = f(z) = f(x,y) = f(r) C m where z = x + j y Z = C n. We will have more to say about the relationships between these representations later on in Section 6 below. We further assume that Z = C n is a Riemannian manifold with a hermitian, positive-definite n n metric tensor Ω z = Ω H z > 0. This assumption makes every tangent space23 T z Z = C n z a Hilbert space with inner product v 1,v 2 = v H 1 Ω z v 2 v 1,v 2 C n z. 4.2 The Cogradient Operator and the Jacobian Matrix The Cogradient and Conjugate Cogradient. operators respectively as the row operators 24 Define the cogradient and conjugate cogradient Cogradient Operator: ( 1 n ) (20) Conjugate cogradient Operator: ( 1 n ) where (z i, z i ), i = 1,, n are conjugate coordinates as discussed earlier and the component operators are R-derivatives and R-derivatives defined according to equations (8) and (9), = 1 ( j ) and = 1 ( + j ), (22) i 2 x i y i i 2 x i y i 23 A tangent space at the point z is the space of all differential displacements, dz, at the point z or, alternatively, the space of all velocity vectors v = dz dt at the point z. These are equivalent statements because dz and v are scaled version of each other, dz = vdt. The tangent space T z Z = C n z is a linear variety in the space Z = C n. Specifically it is a copy of C n affinely translated to the point z, C n z = {z} + Cn. 24 The cogradient is a covariant operator [22]. It is not itself a gradient, but is the co mpanion to the gradient operator defined below. (21)

15 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v for i = 1,, n. 25 Equivalently, we have = 1 ( 2 x j ) y and = 1 2 ( x + j ), (23) y When applying the cogradient operator, z is formally treated as a constant, and when applying the conjugate cogradient operator, z is formally treated as a constant. For example, consider the scalar-valued function f(c) = f(z, z) = z 1 z 2 + z 1 z 2. For this function we can readily determine by partial differentiation on the z i and z i components that f(c) = ( z ) f(c) 2 z 1 and = ( ) z 2 z 1. The Jacobian Matrix. Let f(c) = f(z, z) C m be a mapping 26 f : Z = C n C m. The generalization of the identity (14) yields the vector form of the differential rule, 27 df(c) = f(c) f(c) f(c) dc = dz + d z, Differential Rule (24) where the m n matrix f is called the Jacobian, or Jacobian matrix, of the mapping f, and the m n matrix f the conjugate Jacobian of f. The Jacobian of f is often denoted by J f and is computed by applying the cogradient operator component-wise to f, f 1 (c) f 1 (c) f J f (c) = f(c) 1 1 (c) n =. =..... C m n, (25) f n(c) f n(c) 1 f n(c) n and similarly the conjugate Jacobian, denoted by Jf c is computing by applying the conjugate cogradient operator component-wise to f, f 1 (c) f 1 (c) f Jf c f(c) 1 1 (c) n (c) = =. =..... C m n. (26) f n(c) f n(c) 1 f n(c) n 25 As before the left-hand-sides of (22) and (23) are formal partial derivatives, while the right-hand-sides are actual partial derivatives. 26 It will always be assumed that the components of vector-valued functions are R-differentiable as discussed in footnotes (3) and (13). 27 At this point in our development, the expression f(c) f(c) dc only has meaning as a shorthand expression for dz+ f(c) d z, each term of which must be interpreted formally as z and z cannot be varied independently of each other. (Later, we will examine the very special sense in which the a derivative with respect to c itself can make sense.) Also note that, unlike the real case discussed in [22], the mapping dz df(c) is not linear in dz. Even when interpreted formally, the mapping is affine in dz, not linear.

16 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v With this notation we can write the differential rule as df(c) = J f (c) dz + J c f (c) d z. Differential Rule (27) Applying properties (12) and (13) component-wise yields the identities f(c) = ( ) f(c) = J f (c) and f(c) = ( ) f(c) = J f c (c). (28) Note from (28) that, ( ) f(c) J f (c) = = f(c) f(c) Jc f (c) =. (29) However, in the important special case that f(c) is real-valued (in which case f(c) = f(c)) we have f(c) R m J f (c) = f(c) = f(c) = Jc f (c). (30) With (27) this yields the following important fact which holds for real-valued functions f(c), 28 f(c) R m df(c) = J f (c) dz + J f (c) dz = 2 Re {J f (c) dz}. (31) Consider the composition of two mappings h : C m C r and g : C n C m, h g = h(g) : C n C r. The vector extensions of the chain rule identities (15) and (16) to h g are h(g) h(g) = h g = h g g + h ḡ ḡ g + h ḡ ḡ Chain Rule (32) Chain Rule (33) which can be written as J h g = J h J g + Jh c J g c (34) Jh g c = J h Jg c + Jc J h g (35) 28 The real part of a vector (or matrix) is the vector (or matrix) of the real parts. Note that the mapping dz df(c) is not linear.

17 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v Holomorphic Vector-valued Functions. By definition the vector-valued function f(z) is holomorphic (analytic in the complex vector z) if and only if each of its components f i (c) = f i (z, z) = f i (z 1,, z n, z 1,, z n ) i = 1,, m is holomorphic separately with respect to each of the components z j, j = 1,, n. In the references [17, 18, 19, 20] it is shown that f(z) is holomorphic on a domain if and only if it satisfies a matrix Cauchy Riemann condition everywhere on the domain: Cauchy Riemann Condition: J c f = f = 0 (36) This shows that a vector-valued function which is holomorphic on C n must be a function of z only, f(c) = f(z, z) = f(z) (z only). Stationary Points of Real-Valued Functionals. Suppose that f is a scalar real-valued function from C n to R, 29 f : C n R ; z f(z). As discussed in [14], the first-order differential condition for a real-valued functional f to be optimized with respect to the real and imaginary parts of z at the point z 0 is Condition I for a Stationary Point: f(z 0, z 0 ) = 0 (37) That this fact is true is straightforward to ascertain from equations (20) and (23). An equivalent first-order condition for a real-valued functional f to be stationary at the point z 0 is given by Condition II for a Stationary Point: f(z 0, z 0 ) = 0 (38) The equivalence of the two conditions (37) and (38) is a direct consequence of (28) and the fact that f is real-valued. Differentiation of Conjugate Coordinates? Note that the use of the notation f(c) as shorthand for f(z, z) appears to suggest that it is permissible to take the complex cogradient of f(c) with respect to the conjugate coordinates vector c by treating the complex vector c itself as the variable of differentiation. This is not correct. Only complex differentiation with respect to the complex vectors z and z is well-defined. Thus, from the definition c col(z, z) C 2n, for c viewed as a complex 2n-dimensional vector, the correct interpretation of f(c) is given by [ ] f(c) = f(z, z), f(z, z) 29 The function f is unbolded to indicate its scalar-value status.

18 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v Thus, for example, we have that ch Ωc c H Ω which would be true if it were permissible to take the complex cogradient with respect to the complex vector c (which it isn t). Remarkably, however, below we will show that the 2n-dimensional complex vector c is an element of an n-dimensional real vector space and that, as a consequence, it is permissible to take the real cogradient with respect to the real vector c! Comments. With the machinery developed up to this point, one can solve optimization problems which have closed-form solutions to the first-order stationarity conditions. However, to solve general nonlinear problems one must often resort to gradient-based iterative methods. Furthermore, to verify that the solutions are optimal, one needs to check second order conditions which require the construction of the hessian matrix. Therefore, the remainder of this note is primarily concerned with the development of the machinery required to construct the gradient and hessian of a scalarvalued functional of complex parameters. 4.3 Biholomorphic Mappings and Change of Coordinates. Holomorphic and Biholomorphic Mappings. A vector-valued function f is holomorphic (complexanalytic) if its components are holomorphic. In this case the function does not depend on the conjugate coordinate z, f(c) = f(z) (z-only), and satisfies the Cauchy-Riemann Condition, As a consequence (see (27)), J c f = f = 0. f(z) holomorphic df(z) = J f (z) dz = f(z) dz. (39) Note that when f is holomorphic, the mapping dz df(z) is linear, exactly as in the real case. Consider the composition of two mappings h : C m C r and g : C n C m, h g = h(g) : C n C r, which are both holomorphic. In this case, as a consequence of the Cauchy-Riemann condition (36), the second chain rule condition (35) vanishes, Jh g c = 0, and the first chain rule condition (34) simplifies to f and g holomorphic J h g = J h J g. (40) Now consider the holomorphic mapping ξ = f(z), dξ = df(z) = J f (z) dz (41)

19 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v and assume that it is invertible, z = g(ξ) = f 1 (ξ). (42) If the invertible function f and its inverse g = f 1 are both holomorphic, then f (equivalently, g) is said to be biholomorphic. In this case, we have that showing that dz = g(ξ) ξ dξ = J g(ξ) dξ = J 1 f (z) dξ, ξ = f(z), (43) J g (ξ) = J 1 f (z), ξ = f(z). (44) Coordinate Transformations. Admissible coordinates on a space defined over a space of complex numbers are related via biholomorphic transformations [17, 18, 19, 20]. Thus if z and ξ are admissible coordinates on Z = C n, there must exist a biholomorphic mapping relating the two coordinates, ξ = f(z). This relationship is often denoted in the following (potentially confusing) manner, ξ = ξ(z), dξ = ξ(z) dz = J ξ(z) dz, ξ(z) ( ) 1 (ξ) = J ξ (z) = Jz 1 (ξ) = (45) ξ z = z(ξ), dz = (ξ) ξ dξ = J z(ξ) dξ, (ξ) ξ ( ) 1 ξ(z) = J z (ξ) = J 1 ξ (z) =, (46) These equations tell us how vectors (elements of any particular tangent space C n z ) properly transform under a change of coordinates. In particular under the change of coordinates ξ = ξ(z), a vector v C n z must transform to its new representation w C n ξ(z) according to the Vector Transformation Law: w = ξ v = J ξ v (47) For the composite coordinate transformation η(ξ(z)), the chain rule yields Transformation Chain Rule: η = η ξ ξ or J η ξ = J η J ξ (48) Finally, applying the chain rule to the cogradient, f, of a an arbitrary holomorphic function f we obtain f ξ = f ξ for ξ = ξ(z). This shows that the cogradient, as an operator on holomorphic functions, transforms like Cogradient Transformation Law: ( ) ξ = ( ) ξ = ( ) J z = ( ) J 1 ξ (49)

20 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v Note that generally the cogradient transforms quite differently than does a vector. Finally the transformation law for the metric tensor under a change of coordinates can be determined from the requirement that the inner product must be invariant under a change of coordinates. For arbitrary vectors v 1,v 2 C n z transformed as we have w i = J ξ v i C n ξ(z) i = 1, 2, w 1,w 2 = w H 1 Ω ξ w 2 = v H 1 J H ξ Ω ξ J ξ v 2 = v H 1 J H z Ω ξ J z v 2 = v H 1 Ω z v 2 = v 1,v 2. This results in the Metric Tensor Transformation Law: Ω ξ = J H ξ Ω z J 1 ξ = J H z Ω z J z (50) 5 The Gradient Operator z 1 st -Order Approximation of a Real-Valued Function. Let f(c) be a real-valued scalar 30 functional to be optimized with respect to the real and imaginary parts of the vector z Z = C n, f : C n R. As a real-valued function, f(c) does not satisfy the Cauchy-Riemann condition (36) and is therefore not holomorphic. From (31) we have (with f(z) = f(z, z) = f(c)) that df(z) = 2 Re {J f (z) dz} = 2 Re { } f(z) dz. (51) This yields the first order relationship f(z + dz) = f(z) + 2 Re { } f(z) dz and the corresponding first-order power series approximation { } f(z) f(z + z) f(z) + 2 Re z (52) (53) which will be rederived by other means in Section 6 below. 30 And therefore unbolded.

21 K. Kreutz-Delgado Copyright c , All Rights Reserved Version UCSD-ECE275CG-S2009v The Complex Gradient of a Real-Valued Function. The relationship (51) defines a nonlinear functional, df c ( ), on the tangent space C n z,31 { } f(c) df c (v) = 2 Re v, v C n z, c = (z, z). (54) Assuming the existence of a metric tensor Ω z we can write [ ( ) ] H H f f v = Ω 1 z Ω z v = ( z f Ω z v = z f, v, (55) where z f is the gradient of f, defined as Gradient of f: ( f z f Ω 1 z (56) Consistent with this definition, the gradient operator is defined as Gradient Operator: z ( ) Ω 1 z ( ( ) (57) Note the relationships between the gradients and the cogradients. One can show from the coordinate transformation laws for cogradients and metric tensors that the gradient z f transforms like a vector and therefore is a vector, z f C n z. Equations (54) and (55) yield, df c (v) = 2 Re { z f, v }. Keeping v = 1 we want to find the directions v of steepest increase in the value of df c (v). We have as a consequence of the Cauchy-Schwarz inequality that for all unit vectors v C n z, df c (v) = 2 Re { z f, v } 2 z f, v 2 z f v = 2 z f. This upper bound is attained if and only if v z f, showing that the gradient gives the directions of steepest increase, with + z f giving the direction of steepest ascent and z f giving the direction of steepest descent. The result (57) is derived in [14] for the special case that the metric is Euclidean Ω z = I. 32 Note that the first-order necessary conditions for a stationary point to exist is given by z f = 0, but that it is much easier to apply the simpler condition f = 0 which does not require knowledge of the metric tensor. Of course this distinction vanishes when Ω z = I as is the case in [14]. 31 Because this operator is nonlinear in dz, unlike the real vector-space case [22], we will avoid calling it a differential operator.. 32 Therefore one must be careful to ascertain when a result derived in [14] holds in the general case. Also note the corresponding notational difference between this note and [14]. We have z denoting the gradient operator for the general case Ω z I while [14] denotes the gradient operator as z for the special case Ω z = I.