Tutorial on Stochastic Differential Equations

Transcription

1 Please cite as J. R. Movellan (211 Tutorial on Stochastic Differential Equations, MPLab Tutorials Version 6.1. Tutorial on Stochastic Differential Equations Javier R. Movellan Copyright c 23, 24, 25, 26, Javier R. Movellan

2 This document is being reorganized. Expect redundancy, inconsistencies, disorganized presentation... 1 Motivation There is a wide range of interesting processes in robotics, control, economics, that can be described as a differential equations with non-deterministic dynamics. Suppose the original processes is described by the following differential equation dx t = a(x t (1 dt with initial condition X, which could be random. We wish to construct a mathematical model of how the may behave in the presence of noise. We wish for this noise source to be stationary and independent of the current state of the system. We also want for the resulting paths to be continuous. As it turns out building such a model is tricky. An elegant mathematical solution to this problem may be found by considering a discrete time versions of the process and then taking limits in some meaningful way. Let π = { = t t 1 t n = t} be a partition of the interval [, t]. Let π t k = t k+1 t k. For each partition π we can construct a continuous time process X π defined as follows X π t = X (2 X π t k+1 = X π t k + a(x π t k π t k + c(x π t k (N tk+1 N tk (3 where N is a noise process whose properties remain to be determined and b is a function that allows us to have the amount of noise be a function of time and of the state. To make the process be continuous in time, we make it piecewise constant between the intervals defined by the partition, i.e. X π t = X π t k for t [t k, t k+1 (4 We want for the noise N t to be continuous and for the increments N tk+1 N tk to have zero mean, and to be independently and identically distributed. It turns out that the only noise source that satisfies these requirements is Brownian motion. Thus we get Xt π = X + a(xt π k t k + c(xt π k B k (5 k= where t k = t k+1 t k, and B k = B tk+1 B tk where B is Brownian Motion. Let π = max{ t k } be the norm of the partition π. It can be shown that as π the X π processes converge in probability to a stochastic process X. It follows that and that lim π k= a(xt π k k = k= a(x s ds (6 c(xt π k B k (7 k= converges to a process I t I t = lim π k= c(xt π k B k (8

3 Note I t looks like an integral where the integrand is a random variable c(x s and the integrator B k is also a random variable. As we will see later, I t turns out to be an Ito Stochastic Integral. We can now express the limit process X as a process satisfying the following equation X t = X + a(x s ds + I t (9 Sketch of Proof of Convergence: Construct a sequence of partitions π 1, π 2, each one being a refinement of the previous one. Show that the corresponding Xt π i form a Cauchy sequence in L 2 and therefore converge to a limit. Call that process X. In order to get a better understanding of the limit process X there are two things we need to do: (1 To study the properties of Brownian motion and (2 to study the properties of the Ito stochastic integral. 2 Standard Brownian motion By Brownian motion we refer to a mathematical model of the random movement of particles suspended in a fluid. This type of motion was named after Robert Brown that observed it in pollens of grains in water. The processes was described mathematically by Norbert Wiener, and is is thus also called a Wiener Processes. Mathematically a standard Brownian motion (or Wiener Process is defined by the following properties: 1. The process starts at zero with probability 1, i.e., P (B = = 1 2. The probability that a randomly generated Brownian path be continuous is The path increments are independent Gaussian, zero mean, with variance equal to the temporal extension of the increment. Specifically for s 1 t 1 s 2, t 2 B t1 B s1 N (, s 1 t 1 (1 B t2 B s2 N (, s 2 t 2 (11 and B t2 B s2 is independent of B t1 B s1. Wiener showed that such a process exists, i.e., there is a stochastic process that does not violate the axioms of probability theory and that satisfies the 3 aforementioned properties.

4 2.1 Properties of Brownian Motion Statistics From the properties of Gaussian random variables, E(B t B s = (12 Var(B t B s = E[(B t B s 2 ] = t s (13 E((B t B s 4 ] = 3(t s (14 Var[(B t B s 2 ] = E[(B t B s 4 ] E[(B t B s 2 ] 2 = 2(t s 2 (15 Cov(B s, B t = s, for t > s (16 s Corr(B s, B t =, for t > s. (17 t Proof: For the variance of (B t B s 2 we used the that for a standard random variable Z E(Z 4 = 3 (18 Note Var(B T = Var(B T B = T (19 since P (B = and for all t Moreover, Var(B t+ t B t = t (2 Cov(B s, B t = Cov(B s, B s + (B t B s = Cov(B s, B s + Cov(B s, (B t B s = Var(B s = s (21 since B s and B t B s are uncorrelated Distributional Properties Let B represent a standard Brownian motion (SBM process. Self-similarity: For any c, X t = 1 c B ct is SBM. We can use this property to simulate SBM in any given interval [,T] if we know how to simulate in the interval [, 1]: If B is SBM in [,1], c = 1 T then X t = T B 1 T t is SBM in [,T]. Time Inversion: X t = tb 1 is SBM t Time Reversal: X t = B T B T t is SBM in the interval [, T ] Symmetry: X t = B t is SBM Pathwise Properties Brownian motion sample paths are non-differentiable with probability 1 This is the basic why we need to develop a generalization of ordinary calculus to handle stochastic differential equations. If we were to define such equations simply as dx t = a(x t + c(x t db t (22 dt dt

5 we would have the obvious problem that the derivative of Brownian motion does not exist. Proof: Let X be a real valued stochastic process. For a fixed t let π = { = t t 1, t n = t} be a partition of the interval [, t]. Let π be the norm of the partition. The quadratic variation of X at t is a random variable represented as < X, X > 2 t and defined as follows n < X, X > 2 t = lim X tk+1 X tk 2 (23 π k=1 We will show that the quadratic variation of SBM is larger than zero with probability one, and therefore the quadratic paths are not differentiable with probability 1. Let B be a Standard Brownian Motion. For a partition π = { = t t 1, t n = t} let Bk π be defined as follows Bk π = B tk (24 Let n S π = ( Bk π 2 (25 Note and Thus k=1 E(S π = t k+1 t k = t (26 k= Var(S π = Var [ ( Bk π 2] k= = 2 (t k+1 t k 2 k= 2 π (t k+1 t k = 2 π t (27 k= ( lim π Var(Sπ = lim E 2 ( Bk π 2 t = (28 π This shows mean square convergence, which implies convergence in probability, of S π to t. (I think almost sure convergence can also be shown. Comments: k= If we were to define the stochastic integral (db s 2 as Then s (db s 2 = (db s 2 = lim π Sπ (29 d s = t (3

6 If a path X t (ω were differentiable almost everywhere in the interval [, T ] then < X, X > 2 t (ω lim = ( max t [,T ] X t(ω 2 lim n t k= ( t X t k (ω 2 (31 2 t (32 = ( max t [,T ] X t(ω 2 lim n (n(t/n2 = (33 where X = dx/dt. Since Brownian paths have non-zero quadratic variation with probability one, they are also non-differentiable with probability one. 2.2 Simulating Brownian Motion Let π = { = t t 1 t n = t} be a partition of the interval [, t]. Let {Z 1,, Z n }; be i.i.d Gaussian random variables E(Z i = ; Var(Z i = 1. Let the stochastic process B π as follows, Moreover, B π t = (34 B π t 1 = B π t + t 1 t Z 1 (35. (36 B π t k = B π t k 1 + t k t k 1 Z k (37 B π t = B π t k 1 for t [t k 1, t k (38 For each partition π this defines a continuous time process. It can be shown that as π the process B π converges in distribution to Standard Brownian Motion Exercise Simulate Brownian motion and verify numerically the following properties E(B t = (39 Var(B t = t (4 db 2 s = 3 The Ito Stochastic Integral We want to give meaning to the expression s ds = t (41 Y s db s (42 where B is standard Brownian Motion and Y is a process that does not anticipate the future of Brownian motion. For example, Y t = B t+2 would not be a valid

7 integrand. A random process Y is simply a set of functions f(t, from an outcome space Ω to the real numbers, i.e for each ω Ω Y t (ω = f(t, ω (43 We will first study the case in which f is piece-wise constant. In such case there is a partition π = { = t t 1 t n = t} of the interval [,.t] such that where f n (t, ω = C k (ωξ k (t (44 k= { 1 if t [tk, t ξ k (t = k+1 else where C k is a non-anticipatory random variable, i.e., a function of X and the Brownian noise up to time t k. For such a piece-wise constant process Y t (ω = f n (t, ω we define the stochastic integral as follows. For each outcome ω Ω More succinctly ( Y s (ωdb s (ω = C k (ω B tk+1 (ω B tk (ω k= ( Y s db s = C k B tk+1 B tk k= This leads us to the more general definition of the Ito integral Definition of the Ito Integral Let f(t, be a non-anticipatory function from an outcome space Ω to the real numbers. Let {f 1, f 2, } be a sequence of elementary non-anticipatory functions such that ( 2ds] lim E[ f(s, ω f n (s, ω = (48 n Let the random process Y be defined as follows: Y t (ω = f(t, ω Then the Ito integral (45 (46 (47 Y s db s (49 is a random variable defined as follows. For each outcome ω Ω f(s, ωdb s (ω = lim n f n (t, ωdb s (ω (5 where the limit is in L 2 (P. It can be shown that an approximating sequence f 1, f 2 satisfying (48 exists. Moreover the limit in (5 also exists and is independent of the choice of the approximating sequence. Comment Strictly speaking we need for f to be measurable, i.e., induce a proper random variable. We also need for f(t, to be F t adapted. This basically means that Y t must be a function of Y and the Brownian motion up to time t It cannot be a function of future values of B. Moreover we need E[ f(t, 2 dt].

8 3.1 Properties of the Ito Integral T Var(I t = E(I 2 t = (X s + Y s db s = X s db s = E(I t = (51 The Ito integral is a Martingale process X s db s + E(X 2 s ds (52 Y s db s (53 T X s db s + X s db s for t (, T (54 t E(I t F s = I s lfor all t > s (55 where E(I t F s is the least squares prediction of I t based on all the information available up to time s. 4 Stochastic Differential Equations In the introduction we defined a limit process X which was the limit process of a dynamical system expressed as a differential equation plus Brownian noise perturbation in the system dynamics. The process was a solution to the following equation where X t = X + I t = a(x s ds + I t (56 lim π c(xπ t k B k (57 It should now be clear that I t is in fact an Ito Stochastic Integral I t = c(x s db s (58 and thus X can be expressed as the solution of the following stochastic integral equation X t = X + a(x s ds + c(x s db s (59 It is convenient to express the integral equation above using differential notation dx t = a(x t dt + c(x t db t (6 with given initial condition X. We call this an Ito Stochastic Differential Equation (SDE. The differential notation is simply a pointer, and thus acquires its meaning from, the corresponding integral equation.

9 4.1 Second order differentials The following rules are useful X t (dt 2 = (61 X t db t dt = (62 X t db t dw t = if B, W are independent Brownian Motions (63 X t (db t 2 = Symbolically this is commonly expressed as follows X t dt (64 (65 dt 2 = (66 db t dt = (67 db t dw t = (68 (db t 2 = dt (69 Sketch of proof: Let π = { = t t 1 t n = t} a partition of the [, t] with equal intervals, i.e. t k+1 t k = t. Regarding dt 2 = note lim t k= Regarding db t d t = note lim t k= Regarding db 2 t = dt note k= X tk t 2 = lim t t X tk t B k = lim t t k= X s ds = (7 X s db s = (71 E [( 2 ] X tk Bk 2 X tk t = E [( 2] X tk ( Bk 2 t = k= k = k= E[X tk X t k ( Bk 2 t( Bk 2 t] (72 If k > k then ( Bk 2 t is independent of X t k X tk ( Bk 2 t, and therefore E[X tk X tk ( B 2 k t( B 2 k t] = E[X tk X tk ( B 2 k t]e[ B2 k t] = (73 Equivalently, if k > k then ( B 2 k t is independent of X t k X tk ( B 2 k t and therefore E[X tk X tk ( B 2 k t( B 2 k t] = E[X tk X tk ( Bk 2 t]e[ Bk 2 t] = (74

10 Thus k= k = E[X tk X tk ( Bk 2 t( Bk 2 t] = E[X 2 t k ( Bk 2 t 2 ] Note since B k is independent of X tk then Thus k= (75 E[X 2 t k ( B 2 k t 2 = E[X 2 t k ]E[( B 2 k t 2 ] (76 E [( = E[X 2 t k ]Var( B 2 k = 2E[X 2 t k ] t 2 (77 k= X tk Bk 2 k= 2 ] X tk t = k= which goes to zero as t. Thus, in the limit as t t k= t k= E[X 2 t k ] 2 t (78 lim X tk Bk 2 = lim X tk t (79 where the limit is taken in the mean square sense. Thus Regarding db t dw t = note E [( 2 ] X tk B k W k k= X s db 2 s = = k= k = X s ds (8 E[X tk X tk B k W k B k W k ] (81 If k > k then B k, W k are independent of X tk X tk B k W k and therefore E[X tk X tk B k W k B k W k ] = E[X tk X tk B k W k ]E[ B k ]E[ W k ] = (82 Equivalently, if k > k then B k, W k are independent of X tk X tk B k W k and therefore E[X tk X tk B k W k B k W k ] = E[X tk X tk B k W k ]E[ B k ]E[ W k ] = (83 Finally, for k = k, B k, W k and X tk are independent, thus Thus E[X 2 t k B 2 k W 2 k ] = E[X 2 t k ]E[ B 2 k]e[ W 2 k ] = E[X 2 t k ] t 2 (84 E [( 2 ] X tk B k W k = E[Xt 2 k ] t 2 (85 k= which converges to as t. Thus k= X s db s dw s = (86

11 4.2 Vector Stochastic Differential Equations The form dx t = a(x t dt + c(x t db t (87 is also used to represent mutlivariate equations. In this case X t represents an n- dimensional random vector, B t an m-dimensional vector of m independent standard Brownian motions, and c(x t is an n m matrix. a is commonly known as the drift vector and b the dispersion matrix. 5 Ito s Rule Let X t be governed by an SDE dx t = a(x t, tdt + c(x t, tdb t (88 Let Y t = f(x t, t. Ito s rule tells us that Y t is governed by the following SDE dy t def = t f(t, X t dt + x f(t, x T dx t dxt t 2 xf(t, X t dx t (89 where Equivalently db i,t db j,t def = δ(i, j dt (9 dxdt def = (91 dt 2 def = (92 dy t = t f(x t, td t + x f(x t, t T a(x t, td t + x f(x t, t T c(x t, tdb t (c(x trace t, tc(x t, t T 2 xf(x t, t dt (93 where x f(x, t T a(x, t = f(x, t a i (x, t x i i (94 ( trace c(x, tc(x, t T 2 xf(x, t = (c(x, tc(x, t T 2 f(x, t ij x i j i x j (95 Note b is a matrix. Sketch of Proof: To second order Y t = f(x t+ t, t + t f(x t, t = t f(x t, t t + x f(x t, t T X t t2 2 t f(x t, t XT t 2 xf(x t, t X t + t( x t f(x t, t T X t t where t, x are the gradients with respect to time and state, and 2 t is the second derivative with respect to time, 2 x the Hessian with respect to time and x t the (96

12 gradient with respect to state of the gradient with respect to time. Integrating over time and taking limits Y t =Y In differential form Expanding dx t dy s = Y + Y t = Y + 2 t f(x s, s(ds k= Y tk (97 t f(x s, sd s + x f(x s, s T dx s dx T s 2 xf(x s, sdx s ( x t f(x s, s T dx s ds (98 dy t = t f(x t, td t + x f(x t, t T dx t t f(x t, t(dt dxt t 2 xf(x t, tdx t + ( x t f(x t, t T dx t dt (99 ( x t f(x t, t T dx t dt = ( x t f(x t, t T a(x t, t(dt 2 + ( x t f(x t, t T c(x t, tdb t dt = (1 where we used the standard rules for second order differentials Moreover dx T t 2 xf(x t, tdx t (dt 2 = (11 (db t dt = (12 (13 = (a(x t, tdt + c(x t, tdb t T 2 xf(x t, t(a(x t, tdt + c(x t, tdb t = a(x t, t T 2 xf(x t, ta(x t, t(dt 2 + 2a(X t, t T 2 xf(x t, tc(x t, t(db t dt + db T t c(x t, t T 2 xf(x t, tc(x t, t(db t (14 Using the rules for second order differentials where Thus (dt 2 = (15 (db t dt = (16 dbt T K(X t, tdb t = K i,j (X t, tdb i,t db j,t = K i,i dt (17 i j i K(X t, t = c(x t, t T 2 xf(x t, tc(x t, t (18 dy t = t f(x t, td t + x f(x t, t T a(x t, tdt + x f(x t, t T c(x t, tdb t + 1 (c(x 2 trace t, tc(x t, t T 2 xf(x t, t dt (19

13 where we used the fact that K ii (X t, tdt = trace(kdt i ( = trace c(x t, t T 2 xf(x t, tc(x t, t ( = trace c(x t, tc(x t, t T 2 xf(x t, t ( Product Rule Let X, Y be Ito processes then d(x t Y t = X t dy t + Y t dx t + dx t dy t (111 Proof: Consider X, Y as a joint Ito process and take f(x, y, t = xy. Then Applying Ito s rule, the Product Rule follows. Exercise: Solve T B tdb t symbolically. Let a(x t, t =, c(x t, t = 1, f(x, t = x 2. Thus and Apllying Ito s rule we get df(x t, t = f(x t, t t f t = (112 f x = y (113 f y = x (114 2 f x y = 1 (115 2 f x 2 = 2 f y 2 = (116 dx t = db t (117 X t = B t (118 f(t, x = t (119 f(t, x = 2x x (12 2 f(t, x x 2 = 2 (121 dt + f(x t, t x + 1 ( 2 trace c(x t, tc(x t, t T 2 f(x, t x 2 a(x t, tdt + f(x t, t c(x t, tdb t x (122 db 2 t = 2B t db t + dt (123

14 Equivalently Therefore B 2 t = 2 db 2 s = 2 NOTE: db 2 t is different from (db t 2. Exercise: Get E[e β B t ] B s db s + ds (124 B s db s + t (125 B s db s = 1 2 B2 t 1 2 t (126 Let Let a(x t, t =, c(x t, t = 1, i.e., dx t = db t. Let Y t = f(x t, t = e βbt, and dx t = db t. Using Ito s rule Taking expected values Y t = Y + β dy t = βe βbt db t β2 e βbt dt (127 e βbs db s + β2 2 E[Y t ] = E[Y ] + β2 2 e βbs ds (128 E[Y s ]ds (129 where we used the fact that E[ eβbs db s ] = because for any non anticipatory random variable Y t, we know that E[ Y sdb s ] =. Thus and since E[Y ] = 1 de[y t ] dt = β2 2 E[Y t] (13 E[e βbt ] = e β2 2 t (131 Exercise: Solve the following SDE dx t = αx t dt + βx t db t (132 In this case a(x t, t = αx t, c(x t, t = βx t. Using Ito s formula for f(x, t = log(x Thus f(t, x = t (133 f(t, x = 1 x x (134 2 f(t, x x 2 = 1 x 2 (135 d log(x t = 1 αx t dt + 1 βx t db t 1 X t X t 2Xt 2 β 2 Xt 2 dt = (α β2 2 dt + βdb t (136

15 Integrating over time Note log(x t = log(x + (α β2 2 t + βb t (137 X t = X exp((α β2 2 t exp(βb t (138 β2 (α E[X t ] = E[X ]e 2 t E[exp(αB t ] = E[X ]e αt (139 6 Generator of an Ito Diffusion The generator G t of the Ito diffusion dx t = a(x t, tdt + c(x t, tdb t (14 is a second order partial differential operator. For any funcion f it provides the directional derivative of f averaged accross the paths generated by the diffusion. In particular given the function f, the function G t [f] is defined as follows G t [f](x = de[f(x t X t = x] dt = E[df(X t X t = x] dt Note using Ito s rule Taking expected values E[f(X t+ t X t = x] f(x = lim t t (141 d f(x t = x f(x t, t T a(x t, tdt + x c(x t, t T db t + 1 (c(x 2 trace t, tc(x t, t T 2 xf(x t, t dt (142 G t [f](x = E[df(X t X t = x] dt = x f(x T a(x, t + 1 (c(x, 2 trace tc(x, t T 2 xf(x (143 In other words G t [ ] = i a i (x, t x i [ ] (c(x, tc(x, t T 2 i,j i j x i x j [ ] (144 7 Adjoints Every linear operator G on a Hilbert space H with inner product <, > has a correspoinding adjoint operator G such that < Gx, y >=< x, G y > for all x, y H (145 In our case the elements of the Hilbert space are functions f, g and the inner product will be of the form < f, g >= f(x g(x dx (146 Using partial integrations it can be shown that if

16 then G[f](x = i G [f](x = i NOTe: Confirm b or bsquare f(x a i (x, t + 1 ( x i 2 trace c(x, tc(x, t T 2 xf(x (147 (148 [f(xa(x, t] [b i,j (x, tf(x] (149 x i 2 x i,j i x j 8 The Feynman-Kac Formula (Terminal Condition Version Let X be an Ito diffusion with generator G t G t [v](x = i dx t = a(x t, tdt + c(x t, tdb t (15 v(x, t a i (x, t + 1 x i 2 Let v be the solution to the following pde v(x, t t i j (c(x, tc(x, t T i,j 2 v(x, t x i x j (151 = G t [v](x, t v(x, tf(x, t (152 with a known terminal condition v(x, T, and function f. It can be shown that the solution to the pde (152 is as follows v(x, s = E [ v(x T, T exp ( T s f(x t dt ] X s = x (153 We can think of v(x T, T as a terminal reward and of T s f(x tdt as a discount factor. Informal Proof: Let s t T let Y t = v(x t, t, Z t = exp( s f(x τ dτ, U t = Y t Z t. It can be shown (see Lemma below that Usint Ito s product rule dz t = Z t f(x t dt (154 du t = d(y t Z t = Z t dy t + Y t dz t + dy t dz t (155 Since dz t has a dt term, it follows that dy t dz t =. Thus Using Ito s rule on dv we get du t = Z t dv(x t, t v(x t, tz t f(x t dt (156 dv(x t, t = t v(x t, tdt + ( x v(x t, t T a(x t, tdt + ( x v(x t, t T c(x t, tdb t + 1 (c(x 2 trace t, tc(x t, t T 2 xv(x t, t dt (157

17 Thus du t =Z t [ t v(x t, t + ( x v(x t, t T a(x t, t + 1 (c(x 2 trace t, tc(x t, t T 2 xv(x t, t ] v(x t, tf(x t dt + Z t ( x v(x t, t T c(x t, tdb t (158 and since v is the solution to (152 then Integrating taking expected values U T U s = du t = ( x v(x t, t T c(x t, tdb t (159 T s Y t ( x v(x t, t T c(x t, tdb t (16 E[U T X s = x] E[U s X s = x] = (161 where we used the fact that the expected values of integrals with respect to Brownian motion is zero. Thus, since U s = Y Z = v(x s, s Using the definition of U T we get We end the proof by showing that First let Y t = s f(x τ dτ and note E[U T X s = x] = E[U s X s = x] = v(x, s (162 v(x, s = E[v(X T, T e R T s f(xtdt X s = x] (163 Y t = dz t = Z t f(x t dt (164 + t Let Z t = exp( Y t. Using Ito s rule t f(x τ dτ f(x t t (165 dy t = f(x t dt (166 dz t = e Yt dy t e Yt (dy t 2 = e Yt f(x t dt = Z t f(x t dt (167 where we used the fact that (dy t 2 = Z 2 t f(x t 2 (dt 2 = (168 9 Kolmogorov Backward equation The Kolmogorov backward equation tells us at time s whether at a future time t the system will be in the target set A. We let ξ be the indicator function of A, i.e, ξ(x = 1 if x A, otherwise it is zero. We want to know for every state x at time s < T what is the probability of ending up in the target set A at time T. This is call the the hit probablity.

18 Let X be an Ito diffusion dx t = a(x t, tdt + c(x t, tdb t (169 X = x (17 The hit probability p(x, t satisfies the Kolmogorov backward pde i.e., p(x, t = G t [p](x, t (171 t p(x,t t = i a i(x, t p(x,t x i i,j (c(x, tc(x, tt 2 p(x,t ij x i x j (172 subject to the final condition p(x, T = ξ(x. The equation can be derived from the Feynman-Kac formula, noting that the hit probability is an expected value over paths that originate at x at time s T, and setting f(x =, q(x = ξ(x for all x p(x, t = p(x T A X t = x] = E[ξ(X T X t = x] = E[q(X T e R T t f(xsds ] (173 1 The Kolmogorov Forward equation Let X be an Ito diffusion dx t = a(x t, tdt + c(x t, tdb t (174 X = x (175 with generator G. Let p(x, t represent the probability density of X t evaluated at x given the initial state x. Then where G is the adjoint of G, i.e., p(x,t t = i p(x, t t = G [p](x, t (176 x i [p(x, ta(x, t] i,j x i x j [(c(x, tc(x, t T ij p(x, t] ( Girsanov s Theorem Let X a process defined by the following SDE where c( c( is an invertible matrix. Let where ( T Z def = exp a(x t k(x t dx t 1 2 dx t = a(x t dt + c(x t db t (178 T a(x t k(x t a(x t dt (179 k(x t def = (c(x t c(x t 1 (18

19 and B is a Brownian process under the probability measure P. Let the probability measure Q be defined as follows dp dq = Z (181 Then the process X 1:T is Brownian under Q. Heuristic Proof: For a given path x the ratio of the probability density of x under P and Q can be approximated as follows dp(x dq(x k p(x tk+1 x tk x tk q(x tk+1 x tk x tk were π = { = t < t 1 < t n = T } is a partition of [, T ] and (182 p(x tk+1 x tk = G( x tk a(x tk t k, t k k(x t 1 (183 q(x tk+1 x tk = G( x tk, t k k(x t 1 (184 where g(, µ, σ is the multivariate Gaussian distribution with mean µ and covariance matrix c. Thus log dp(x dq(x 1 ( ( x tk a(x tk t k k(x t ( x tk a(x tk t k 2 t k k= x t k k(x t x tk = a(x tk k(x t x tk 1 2 a(x t k k(x tk a(x tk t k (185 k= taking limits as π log dp(x dq(x = T a(x t k(x t dx t 1 2 T a(x t k(x t a(x t dt (186 Notes dp dq = exp ( T T a(x t k(x t a(x t dt a(x t k(x t dx t 1 2 ( T = exp a(x t k(x t (a(x t, tdt + c(x t, tdb t 1 T 2 ( T = exp a(x t k(x t c(x t, tdb t + 1 T a(x t k(x t a(x t dt 2 a(x t k(x t a(x t dt (187 And if c(x t is invertible k(x t c(x t = c(x t 1 and dp dq = exp ( T a(x t c 1 (X t, tdb t This form is popular in some textbooks. T a(x t c 1 (X t 2 dt (188

20 12 Solving Stochastic Differential Equations Let dx t = a(t, X t dt + c(t, X t db t. (189 Conceptually, this is related to dxt dt = a(t, X t + c(t, X t dbt dt where dbt dt is white noise. However, dbt dt does not exist in the usual sense, since Brownian motion is nowhere differentiable with probability one. We interpret solving for (189, as finding a process X t that satisfies X t = M + a(s, X s ds + c(s, X s dbs. (19 for a given standard Brownian process B. Here X t is an Ito process with a(s, X s = K s and c(s, X s = H s. a(t, X t is called the drift function. c(t, X t is called the dispersion function (also called diffusion or volatility function. Setting b = gives an ordinary differential equation. Example 1: Geometric Brownian Motion Using Ito s rule on log X t we get dx t = ax t dt + bx t db t (191 X = ξ > (192 d log X t = 1 X t dx t ( 1 2 Xt 2 (dx t 2 (193 = dx t 1 X t 2 b2 dt (194 = (ax t 12 b2 dt + αdb t (195 Thus log X t = log X + (a 12 b2 t + bb t (196 and Processes of the form X t = X e (a 1 2 b2 t+bb t (197 Y t = Y e αt+βbt (198 where α and β are constant, are called Geometric Brownian Motions. Geometric Brownian motion X t is characterized by the fact that the log of the process is Brownian motion. Thus, at each point in time, the distribution of the process is log-normal. Let s study the dynamics of the average path. First let Y t = e bbt (199

21 Using Ito s rule Thus dy t = be bbt db t b2 e bbt (db t 2 (2 Y t = Y + b Y s db s b2 + t Y s ds (21 E(Y t = E(Y b2 E(Y s ds (22 de(y t d t = 1 2 b2 E(Y t (23 E(Y t = E(Y e 1 2 b2t = e 1 2 b2 t E(X t = E(X e (a 1 2 b2 t E(Y t = E(X e (a 1 2 b2 t (24 (25 Thus the average path has the same dynamics as the noiseless system. Note the result above is somewhat trivial considering E(dX t = de(x t = E(a(X t dt + E(c(X t db t = E(a(X t dt + E(c(X t E(dB t (26 (27 de(x t = E(a(X t dt (28 and in the linear case E(a(X t dt = E(a t X t + u t dt = a t E(X t + u t dt (29 These symbolic operations on differentials trace back to the corresponding integral operations they refer to. 13 Linear SDEs 13.1 The Deterministic Case (Linear ODEs Let x t R n be defined by the following ode dx t dt = a tx t + u t (21 x o = ξ (211 where u t is known as the driving, or input, signal. The solution takes the following form: x t = Φ t (x + Φ 1 s u s ds (212 where Φ t is an n n matrix, known as the fundamental solution, defined by the following ODE dφ t = a t Φ t dt (213 Φ = I n (214

22 Example: Thus Let x t be a scalar such that dx t dt = α (x t u t (215 Φ t = e αt (216 x t = e t x + ( 1 e αt u ( The Stochastic Case Linear SDEs have the following form dx t = (a t X t + u t dt + m (c i,t X t + v i,t db i,t (218 i=1 = (a t X t + u t dt + v t db t + m c i,t X t db i,t (219 i=1 X = ξ (22 where X t is an n dimensional random vector, B t = (B 1,, B m, b i,t are n n matrices, and v i,t are the n-dimensional column vectors of the n m matrix v t. If b i,t t = for all i, t we say that the SDE is linear in the narrow sense. If v t = for all t we say that the SDE is homogeneous. The solution has the following form ( X t = Φ t (X + Φ 1 s u s m b i,s v i,s ds + i=1 Φ 1 s m v i,s db i,s i=1 (221 where Φ t is an n n matrix satisfying the following matrix differential equation m dφ t = a t Φ t dt + b i,s Φ s db i,s (222 i=1 Φ = I n (223 One property of the linear Ito SDEs is that the trajectory of the expected value equals the trajectory of the associated deterministic system with zero noise. This is due to the fact that in the Ito integral the integrand is independent of the integrator db t : E(dX t = de(x t = E(a(X t dt + E(c(X t db t = E(a(X t dt + E(c(X t E(dB t (224 (225 de(x t = E(a(X t dt (226 and in the linear case E(a(X t dt = E(a t X t + u t dt = a t (E(X t + u t dt ( Solution to the Linear-in-Narrow-Sense SDEs In this case dx t = (a t X t + u t dt + v t db t (228 X = ξ (229

23 where v 1,, v m are the columns of the n m matrix v, and db t is an m-dimensional Brownian motion. In this case the solution has the following form X t = Φ t (X + Φ 1 s u s ds + Φ 1 s v s db s (23 where Φ is defined as in the ODE case This solution can be interpreted using a sym- White Noise Interpretation bolic view of white noise as dφ t = a t dt (231 Φ = I n (232 W t = db t d t (233 and thinking of the SDE as an ordinary ODE with a driving term given by u t +v t W t. We will see later that this interpretation breaks down for the more general linear case with b t. Moment Equations Let Then ρ r,s def = E ((X r m r (X s m s (234 ρ 2 t def = ρ t,t = Var(X t (235 de(x t = a t de(x t + u t (236 dt ( E(X t = Φ t E(X + Φ 1 s u s ds (237 ρ 2 t = Φ t (ρ 2 + Φ 1 ( s v s Φ 1 T s v s ds Φ T t (238 dρ 2 t dt = a tρ 2 t + ρ 2 t a T t + v t v T (239 ρ r,s = Φ r (ρ 2 + r s Φ 1 ( t v t Φ 1 T t v t dt Φ T s (24 where r s = min{r, s}. Note the mean evolves according to the equivalent ODE with no driving noise. Constant Coefficients: In this case dx t = ax t + u + vdb t (241 Φ t = e at (242 E(X t = e at( E(X + e as uds (243 Var(X t = ρ 2 t = e at {ρ 2 + e as vv T ( e as T ds } (e at T (244

24 Example: Linear Boltzmann Machines (Multidimensional OU Process where θ is symmetric and τ >. Thus in this case and Thus X t = e tθ (X + dx t = θ(γ X t dt + 2τdB t (245 a = θ, (246 u = θγ, (247 Φ t = e tθ (248 e sθ dsθγ + 2τ =e tθ ( X + θ 1 [ e sθ] t θγ + 2τ =e tθ (X + (e tθ Iγ + 2τ e sθ db s e sθ db s e as db s (249 (25 (251 E(X t = e tθ E(X + ( I e tθ γ (252 lim E(X t = γ (253 t Var(X t = ρ 2 t = e {ρ tθ 2 + 2τ { = e 2tθ ρ 2 + 2τ θ 1 [ e 2sθ ] } t 2 } e 2sθ ds e tθ (254 (255 = e 2ts { ρ 2 + τθ 1 ( e 2tθ I } (256 = e 2tθ ρ 2 + τθ 1 ( I e 2tθ (257 = τθ 1 + e 2tθ ( ρ 2 τθ 1 (258 lim Var(X t = τθ 1 (259 t where we used the fact that a, e at are symmetric matrices, and e at dt = a 1 e at. If the distribution of X is Gaussian, the distribution continues being Gaussian at all times. Example: Harmonic Oscillator Let X t = (X 1,t, X 2,t T represent the location and velocity of an oscillator ( ( ( 1 dx t = ax t + db b t = dx α β t + db b t (26 thus X t = e at (X + e as ( b db s 13.4 Solution to the General Scalar Linear Case (261 Here we will sketch the proof for the general solution to the scalar linear case. We have X t R defined by the following SDE dx t = (a t X t + u t dt + (b t X t + v t db t (262

25 In this case the solution takes the following form X t = Φ t (X + Sketch of the proof Φ 1 s (u s v s b s ds + Φ 1 s v s dbs (263 dφ t = a t Φ t dt + b t Φ t db t (264 Φ = 1 (265 Use Ito s rule to show dφ 1 t = (b 2 a t Φ 1 t dt b t Φ 1 t db t (266 Use Ito s rule to show that if Y 1,t, Y 2,t are scalar processes defined as follows then dy 1,t = a 1 (t, Y 1,t dt + b 1 (t, Y 1,t db t, for i = 1, 2 (267 (268 d(y 1,t Y 2,t = Y 1,t dy 2,t + Y 2,t dy 1,t + b 1 (t, Y 1,t b 2 (t, Y 2,t dt (269 Use the property above to show that Integrate above to get (263 Use Ito s rule to get d(x t Φ 1 t = Φ 1 t ((u t v t b t dt + v t db t (27 d log Φ 1 t = log Φ t = log Φ 1 t = ( 1 2 b2 t a t d t b t db t (271 White Noise Interpretation Does not Work of the general linear case would be (a s 1 2 b sds + b s db s. (272 The white noise interpretation dx t = (a t X t + u t d t + (b t X t + v t W t dt (273 = (a t + b t W t X t d t + (u t + v t W t dt (274 If we interpret this as an ODE with noisy driving terms and coefficients, we would have a solution of the form X t = Φ t (X + Φ 1 s (u s + v t W t dt (275 = Φ t (X + Φ 1 s (u s v s b s ds + Φ 1 s v s dbs (276 with (277 dφ t = (a t + b t W t Φ t dt = a t Φ t dt + b t Φ t db t (278 Φ = 1 (279 The ODE solution to Φ would be of the form log Φ t = (a s + b s W s ds = a s ds + b s db s (28 which differs from the Ito SDE solution in (272 by the term b sd s /2.

26 13.5 Solution to the General Vectorial Linear Case Linear SDEs have the following form dx t = (a t X t + u t dt + m (b i,t X t + v i,t db i,t (281 i=1 = (a t X t + u t dt + vdb t + m b i,t X t db i,t (282 i=1 X = ξ (283 where X t is an n dimensional random vector, B t = (B 1,, B m, b i,t are n n matrices, and v i,t are the n-dimensional column vectors of the n m matrix v t. The solution has the following form ( X t = Φ t (X + Φ 1 s u s m b i,s v i,s ds + i=1 Φ 1 s m v i,s db i,s where Φ t is an n n matrix satisfying the following matrix differential equation dφ t = a t Φ t dt + i=1 (284 m b i,s Φ s db i,s Φ = I n (285 i=1 An explicit solution for Φ cannot be found in general even when a, b i are constant. However if in addition of being constant they pairwise commute, ag i = g i a, g i g j = g j g i for all i, j then {( } Φ t = exp a 1 m m b 2 i t + b i B i,t (286 2 i=1 i=1 Moment Equations Let m t = E(X t, s t = E(X t X T t, then dm t = a t m t + u t, dt (287 ds t m dt = a ts t + s t a T t + b i,t m t b T i,t + u t m T t + m t u T t (288 + m i=1 i=1 ( bi,t m t vi,t T + v i,t m T t b T i,t + v i,t vi,t T, (289 The first moment equation can be obtained by taking expected values on (281. Note it is equivalent to the differential equation one would obtain for the deterministic part of the original system. For the second moment equation apply Ito s rule dx t X T = X t dx T t + (dx t X T t + m [ (bi,t X t + v i,t ( Xt T b T ] i,t + v i,t dt (29 substitute the dx t for its value in (281 and take expected values. i=1

27 Example: Multidimensional Geometric Brownian Motion dx i,t = a i X i,t dt + X i,t n j=1 b i,j db j,t (291 for i = 1,, n. Using Ito s rule d log X i,t = dx i,t + 1 ( 1 2 (dx i,t 2 = X i,t 2 X i, t (292 a i 1 2 b2 i,j + b i,j db i,j j j (293 Thus X i,t = X i, exp a i 1 n 2 j=1 log X i,t = log(x i, + a i 1 n 2 j=1 and thus X i,t has a log-normal distribution. 14 Important SDE Models b 2 i,j b 2 i,j n t + b i,j B j,t j=1 (294 n t + log b i,j B j,t (295 j=1 Stock Prices: Exponential Brownian Motion with Drift Vasiceck( 1977 Interest rate model: OU Process Cox-ingersol-Ross (1985 Interest rate model dx t = ax t dt + bx t db t (296 dx t = α(θ X t dt + bdb t (297 dx t = α(θ X t dt + b X t db t (298 Generalized Cox Ingersoll Ross Model for short term interest rate, proposed by Chan et al (1992. Let dx t = (θ θ 1 X t dt + γx Ψ t db t, for Ψ, γ > (299 Thus where X s = X1 Ψ s γ(1 Ψ (3 d X s = a( X s ds + db s (31 a(x = (θ + θ 1ˆxˆx Ψ γ Ψγ 2 ˆxΨ 1 (32 where ˆx = (γ(1 Ψx (1 Ψ 1 (33 A special case is when Ψ =.5. In this case the process increments are known to have a non-central chi-squared distribution (Cox, Ingersoll, Ross, 1985

28 Logistic Growth Model I The solution is Model II Model III X t = dx t = ax t (1 X t /kd t + bx t db t (34 X exp { (a b 2 /2t + bb t } 1 + X k a exp {(a b2 /2s + bb s } ds (35 dx t = ax t (1 X t /kd t + bx t (1 X t /kdb t (36 dx t = rx t (1 X t /kd t srx 2 t db t (37 In all the models r is the Maltusian growth constant and k the carrying capacity of the environment. In model II, k is unattainable. In the other models X t can have arbitrarily large values with nonzero probability. Gompertz Growth ( k dx t = αx t d t rx t log X t d t + bx t db t (38 where r is the Maltusian growth constant and k the carrying capacity. For α = we get the Skiadas version of the model, for r = we get the lognormal model. Using Ito s rule on Y t = e βt log X t we can get expected value follows the following equation E(X t = exp { log(x e rt} { γ } { } b 2 exp r (1 e rt exp (1 e 2rt (39 4r where γ =α b2 2 (31 Something fishy in expected value formula. Try α = b =! 15 Stratonovitch and Ito SDEs Stochastic differential equations are convenient pointers to their corresponding stochastic integral equations. The two most popular stochastic integrals are the Ito and the Stratonovitch versions. The advantage of the Ito integral is that the integrand is independent of the integrator and thus the integral is a Martingale. The advantage of the Stratonovitch definition is that it does not require changing the rules of standard calculus. The Ito interpretation of m dx t = f(t, X t dt + g j (t, X t db j,t (311 is equivalent to the Stratonovitch equation dx t = f(t, X t dt 1 m m [ ] gi,j m g i j (t, X t dt + g j (t, X t db j,t (312 2 x i i=1 j=1 j=1 j=1

29 and the Stratonovitch interpreation of m dx t = f(t, X t dt + g j (t, X t db j,t (313 j=1 is equivalent to the Ito equation dx t = f(t, X t dt m m i=1 j=1 [ ] gi,j g i j x i m (t, X t dt + g j (t, X t db j,t (314 j=1 16 SDEs and Diffusions Diffusions are rocesses governed by the Fokker-Planck-Kolmogorov equation. All Ito SDEs are diffusions, i.e., they follow the FPK equation. There are diffusions that are not Ito diffusions, i.e., they cannot be described by an Ito SDE. Example: diffusions with reflection boundaries. 17 Appendix I: Numerical methods 17.1 Simulating Brownian Motion Infinitesimal piecewise linear path segments Get n independent standard Gaussian variables {Z 1,, Z n }; E(Z i = ; Var(Z i = 1. Define the stochastic process ˆB as follows, ˆB t = (315 ˆB t1 = ˆB t + t 1 t Z 1 (316. (317 ˆB tk = ˆB tk 1 + t k t k 1 Z k (318 Moreover, ˆB t = ˆB tk 1 for t [t k 1, t k (319 This defines a continuous time process that converges in distribution to Brownian motion as n Linear Interpolation Same as above but linearly interpolating the starting points of path segments. ˆB t = ˆB tk 1 + (t t k ( ˆB tk 1 ˆB tk /(t k t k 1 for t [t k 1, t k (32 Note this approach is non-causal, in that it looks into the future. I believe it is inconsistent with Ito s interpretation and converges to Stratonovitch solutions

30 Fourier sine synthesis ˆB t (ω = k= Z k(ωφ k (t where Z k (ω are same random variables as in previous approach, and φ k (t = 2 ( 2T (2k+1R sin (2k+1Rt 2T As n B converges to BM in distribution. Note this approach is non-causal, in that it looks into the future. I believe it is inconsistent with Ito s interpretation and converges to Stratonovitch solutions 17.2 Simulating SDEs Our goal is to simulate Order of Convergencce dx t = a(x t dt + c(x t db t, t T X = ξ (321 Let = t 1 < t 2 < t k = T A method is said to have strong oder of convergence α if there is aconstant K such that supe Xtk ˆX k < K( tk α (322 t k A method is said to have week oder of convergence α if there is a constant K such that E[X sup tk ] E[ ˆX k ] < K( tk α (323 t k Euler-Maruyama Method ˆX k = ˆX k 1 + a( ˆX k 1 (t k t k 1 + c( ˆX k 1 (B k B k 1 (324 B k = B k 1 + t k t k 1 Z k (325 where Z 1,, Z n are independent standard Gaussian random variables. The Euler-Maruyama method has strong convergence of order α = 1/2, which is poorer of the convergence for the Euler method in the deterministic case, which is order α = 1. The Euler-Maruyama method has week convergence of order α = 1. Milstein s Higher Order Method: the Ito-Taylor expansion It is based on a higher order truncation of ˆX k = ˆX k 1 + a( ˆX k 1 (t k t k 1 + c( ˆX k 1 (B k B k 1 ( c(x k 1b (X k 1 ( (B k B k 1 2 (t k t k 1 (327 B k = B k 1 + t k t k 1 Z k (328 This method has strong convergence of order α = History The first version of this document, which was 17 pages long, was written by Javier R. Movellan in The document was made open source under the GNU Free Documentation License 1.1 on August 12 22, as part of the Kolmogorov Project.

31 October 9, 23. Javier R. Movellan changed the license to GFDL 1.2 and included an endorsement section. March 8, 24: Javier added Optimal Stochastic Control section based on Oksendals book. September 18, 25: Javier. Some cleaning up of the Ito Rule Section. Got rid of the solving symbolically section. January 15, 26: Javier added new stuff to the Ito rule. Added the Linear SDE sections. Added Boltzmann Machine Sections. January/Feb 211. Major reorganization of the tutorial.