Other Optimization Techniques. Similar to steepest descent, but slightly different way of choosing direction of next step: x r +1 = x.

Size: px

Start display at page:

Download "Other Optimization Techniques. Similar to steepest descent, but slightly different way of choosing direction of next step: x r +1 = x."

Lynne Howard
5 years ago
Views:

1 Conjugate Gradient Other Optimization Techniques Similar to steepest descent, but slightly different way of choosing direction of next step: x r +1 = x r + r s r s 0 = g 0 s r +1 = g r +1 + r +1 s r { new term r is chosen to minimize h( x r +1 ). This yields g t r +1 g r = 0 Here we allow a further step in the s r direction. One choice (Fletcher - Reeves) for r +1 is r +1 = g r +1 g r Winter Semester 006/7 Computational Physics I Lecture 1 1

2 Newton-Raphson Assume the function that we want to minimize is twice differentiable. Then, a Taylor expansion gives h( x + ) a + b t + 1 t C where Now a = h( x ), h( x + b = ) b + C h( x ) = g ( x ), C = h x i x j = H Because C is symmetric (check) For an extremum, we have b + C = 0 = C 1 b or x r +1 = x r H( x r ) 1 g ( x r ) Winter Semester 006/7 Computational Physics I Lecture 1

3 Newton-Raphson x r +1 = x r H( x r ) 1 g ( x r ) i.e., the search direction is s = H( x r ) 1 g ( x r ) and =1 This converges quickly (if you start with a good guess), but the penalty is that the Hessian needs to be calculated (usually numerically) Again, convergence is when s is sufficiently small How would we calculate the Hessian numerically? Use Lagrange polynomial in several dimensions and work it out Winter Semester 006/7 Computational Physics I Lecture 1 3

4 Bounded Regions The standard tool for minimization in particle physics is the MINUIT program (CERN library). It has also made its way well outside the particle physics community. Author: Fred James Here is how MINUIT handles bounded search regions - it transforms the parameter to be optimized as follows: a = arcsin b a 1 = a + b a is the exernal (user) parameter is the internal parameter MINUIT is available within PAW, ROOT ( sin +1) Winter Semester 006/7 Computational Physics I Lecture 1 4

5 MINUIT MINUIT uses a (variable metric) conjugate gradient search algorithm (along with others). Basic idea: assume that the function to minimize can be approximated by a quadratic form near the minimum build up iteratively an approximation for the inverse of the Hessian matrix. Recall h( x + ) h( x ) + h( x ) + 1 t H the approximation for the Hessian is updated as follows: H i +1 = H i + ( x i +1 x i ) ( x i +1 x i ) ( x i +1 x i ) ( h i +1 h i ) H [ i ( h i +1 h i )] H i ( h i +1 h i ) ( h i +1 h i ) H i ( h i +1 h i ) [ ] where the symbol represents an outer product of two vectors (a matrix) a b ( ) ij = a i b j Winter Semester 006/7 Computational Physics I Lecture 1 5

6 Fourier Transforms Fourier transforms are very important as a way of summarizing the data with a few parameters because the transform of the data is itself very interesting (e.g., power spectrum, momentumcoordinate space representation, ) H( f ) = h(t) e ift dt H( f ) frequency domain representation h(t) = H( f ) e ift dt h(t) time domain representation Warning: there is no unanimity on factors in front of the integral. Often the angular frequency is used = f H() = h(t) e it dt h(t) = 1 H() e it d Winter Semester 006/7 Computational Physics I Lecture 1 6

7 Fourier Transform Fourier Transform is a linear operation: transform of the sum of two functions is the sum of the transforms the transform of a constant times a function is constant times the transform h(t) real H( f ) = [ H( f )] * h(t) imaginary H( f ) = [ H( f )] * h(t) even H( f ) = H( f ) h(t) odd H( f ) = H( f ) h(t) real,even H( f ) real, even h(t) real,odd H( f ) imaginary, odd h(t) imaginary,even H( f ) imaginary, even h(t) imaginary,odd H( f ) real, odd Winter Semester 006/7 Computational Physics I Lecture 1 7

8 Fourier Transform Further properties: h(at) 1 a H 1 b h t b f a Hbf ( ) h(t t 0 ) H( f )e ift 0 h(t)e if 0 t H( f f 0 ) We are typically interested in the Fourier analysis of a discretely sampled data set. Define the time step (taken to be constant here) as. The sampling rate (frequency) is 1/. Define the samples as h n = h(n) n =,3,,1,0,1,,3, Winter Semester 006/7 Computational Physics I Lecture 1 8

9 Nyquist frequency f c 1 Nyquist frequency This is the highest frequency which can be resolved with a sampling frequency f=1/. If a continuous function h(t) is limited in frequency components to frequencies less than f c, then h(t) is completely determined by its samples h n. It can then be written as follows: h(t) = n = sin[ f h c (t n) ] n (t n) However, if there are frequency components which are higher than f c, then they will be spuriously moved in the range f<f c (aliasing). Winter Semester 006/7 Computational Physics I Lecture 1 9

10 Data Example Fit Aliasing 10 Hz sampling 5 Hz sampling Conditions are: Sine wave with f=4 Hz, phase offset =0.1 i.e., h(t) = sin( + ft) = sin( t) Winter Semester 006/7 Computational Physics I Lecture 1 10

11 H( f ) = sin( t)e ift dt Example = sin(0.1)cos(8t)e ift dt + cos(0.1)sin(8t)e ift dt e 8it + e 8it = sin(0.1) e ift dt + cos(0.1) Recall the relation: eifx df so we have H( f ) = sin(0.1) = sin(0.1) = sin(0.1) e 8it e 8it Winter Semester 006/7 Computational Physics I Lecture 1 11 e ift dt = (x) where (x) is the Dirac Delta function e 8it + e 8it e ift dt eit( f + 4) + e it( f 4) dt + cos(0.1) + cos(0.1) e 8it e 8it e ift dt eit( f + 4) e it( f 4) dt [ ( f + 4) + ( f 4) ] + cos(0.1) [ ( f + 4) ( f 4) ]

12 Discrete Fourier Transform Suppose we have N consecutive sampled points h k h(t k ), t k k, k = 0,1,,,N 1 We can extract the amplitude for N frequency components since we have N data points. Define the frequency components as f n n 1 n = N N,..., N (take N even) Note: there are N+1 frequencies, but we will find that the two at the ends are equal, so only N independent. Negative frequencies allows us to include sine and cosine terms. So N 1 H( f n ) = h(t)e ifnt dt h k e if nt k = h k e if nk = h k e ikn / N H n N 1 k = 0 h k e ikn / N k = 0 Winter Semester 006/7 Computational Physics I Lecture 1 1 N 1 k = 0 Discrete Fourier Transform N 1 k = 0

13 Discrete Fourier Transform The discrete fourier transform does not depend on any dimensional parameters. Note H n = H N n [ e ik(n n)/n = e ik e ikn / N = e ikn / N ] In particular H N / = H N / We can therefore rewrite the sum as follows N 1 H n h k e ikn / N n = 0,,N 1 k = 0 Discrete inverse Fourier transform h k = 1 N N 1 n = 0 H n e ikn / N k = 0,,N 1 Winter Semester 006/7 Computational Physics I Lecture 1 13

14 Discrete Fourier Transform * * Get the discrete Fourier components * Do n=0,63 Hn(n,1)=0.D0 Hn(n,)=0.D0 Do k=0,63 Hn(n,1)=Hn(n,1) & +amplitude(k,1)*dcos(twopi*k*n/64.) & -amplitude(k,)*dsin(twopi*k*n/64.) Hn(n,)=Hn(n,) & +amplitude(k,1)*dsin(twopi*k*n/64.) & +amplitude(k,)*dcos(twopi*k*n/64.) Enddo Write (11,*) N,Hn(N,1),Hn(N,) Enddo amplitude(k,1) real components amplitude(k,) imaginary components Winter Semester 006/7 Computational Physics I Lecture 1 14

15 Discrete Fourier Transform Let s try it out on our sine wave data: Recall, signal f=4 Hz Here s the result (64 points): Large components are: f 5 = = 3.9, f 6 = = 4.06 f 38 = f 64 6 = f 6 f 39 = f 64 5 = f 5 cos and sin needed because of phase offset. Winter Semester 006/7 Computational Physics I Lecture 1 15

16 Discrete Fourier Transform Here s the inverse transform * * Now we try the inverse transform * Do n=0,63 amplitude(n,1)=0.d0 amplitude(n,)=0.d0 Do k=0,63 amplitude(n,1)=amplitude(n,1) & +Hn(k,1)*dcos(twopi*k*n/64.) & +Hn(k,)*dsin(twopi*k*n/64.) amplitude(n,)=amplitude(n,) & -Hn(k,1)*dsin(twopi*k*n/64.) & +Hn(k,)*dcos(twopi*k*n/64.) Enddo Write (1,*) N,amplitude(n,1)/64.,amplitude(n,)/64. Enddo * Winter Semester 006/7 Computational Physics I Lecture 1 16

17 Fast Fourier Transform The discrete Fourier transforms as we described it requires a sum over N terms for each of the N components. I.e., the number of operations scales as N. A large part of the success of Fourier transforms for analysis of electronic signals, optical images, x-ray tomography,, results from the fact that a numerical algorithm was found which requires of order Nlog N operations - the socalled Fast Fourier Transform (FFT). Here is how it works: F k = N 1 e ijk / N f j = e ik( j )/N f j + e ik( j +1)/ N f j +1 N /1 N /1 N /1 Winter Semester 006/7 Computational Physics I Lecture 1 17 = e ikj /(N /) f j + W k e ikj /(N /) f j +1 = F k e + W k F k o N /1 where W e i / N What is won? The sums in the individual terms in the last line only have 1/ as many terms, and the same factor appears

18 Fast Fourier Transform To see how this works in detail, we take an explicit example of having 8 data points (taking a power of is important! If you don t have enough data, pad with zeroes). F k = F k e + W k F k o where W e i / N, F k e 3 W kj f j, F o k W kj f j +1 Now use a binary representation for the index k=4k +k 1 +k 0 where the k i s are 0,1. Then, W kj = ( i /8 e ) (4k + k + k ) j 1 0 = e i(k + k 1 /+ k /4) 0 = e i(k /+ k /4) 1 0 = W j(k + k 1 0 ) i.e., the k bit is irrelevant. So, e F k = F (k1,k 0 ) o + W k F (k1,k 0 ) 3 e F (k1,k 0 ) 3 W j(k + k 1 ) 0 f j F (k1,k 0 ) o 3 W j(k + k 1 ) 0 f j +1 Winter Semester 006/7 Computational Physics I Lecture 1 18

19 Let s try again: F k e = 1 3 Fast Fourier Transform W j(k + k 1 ) 0 f j = W ( j )(k + k 1 ) 0 f ( j ) + W ( j +1)(k + k 1 ) 0 f ( j +1) 1 = W ( j )(k + k 1 ) 0 f ( j ) + W (k + k 1 0 ) F k e = F k ee + W (k 1 + k 0 ) F k eo 1 1 W ( j )(k 1 + k 0 ) f ( j +1) and W 4 j(k + k ) 1 0 = ( i /8 e ) (8k + 4 k 1 0 ) j = W jk 0 F k o = F k oe + W (k 1 + k 0 ) F k oo Can perform one more step: F ee k = F eee + W 4k 0 F eeo F eee = f 0 F eeo = f 4 The sums have disappeared! Winter Semester 006/7 Computational Physics I Lecture 1 19

20 Fast Fourier Transform The final pieces are: F ee k = F eee + W 4k 0 F eeo = f 0 + W 4k 0 f 4 F eo k = F eoe + W 4k 0 F eoo = f + W 4k 0 f 6 F oe k = F oee + W 4k 0 F oeo = f 1 + W 4k 0 f 5 F oo k = F ooe + W 4k 0 F ooo = f 3 + W 4k 0 f 7 Then, F e k = F ee + W (k + k ) 1 0 F eo F o k = F oe + W (k + k ) 1 0 F oo Note k 0 =0,1 So, need 8 multiplications and 8 additions for this step Here k 0 =0,1 k 1 =0,1 So, again 8 multiplications and 8 additions for this step Finally F k = F e + W k F o again 8 multiplications and 8 additions for this step Winter Semester 006/7 Computational Physics I Lecture 1 0

21 Fast Fourier Transform So we need N operations per level, and there are log N levels. The scaling of the computational time is therefore Nlog N rather than N. E.g., N=1000 Nlog N1000*10=10 4 N =10 6 How to implement in practice. Note that the trick is to find out which value of n corresponds to which pattern of e,o in F eoeooeoe... = f n Answer: reverse pattern of e,o. Assign e=0, o=1, and the binary value gives n. eee eee examples oeo eoe 010 Winter Semester 006/7 Computational Physics I Lecture 1 1

22 Some examples. Agric. Food Chem., 5 (0), , /jf04940e S (04) Web Release Date: September 9, 004 Copyright 004 American Chemical Society Discrimination of Olives According to Fruit Quality Using Fourier Transform Raman Spectroscopy and Pattern Recognition Techniques Barbara Muik, Bernhard Lendl, Antonio Molina-Díaz, Domingo Ortega-Calderón,# and María José Ayora-Cañada* Department of Physical and Analytical Chemistry, University of Jaén, Paraje las Lagunillas s/n, E-3071 Jaén, Spain; Institute of Chemical Technologies and Analytics, Vienna University of Technology, Getreidemarkt 9/164, A-1060 Wien, Austria; and CIFA Venta del Llano, IFAPA, Ctra. Bailén-Motril km 18.5, E-360 Mengíbar, Jaén, Spain Multiplication of large integers The fastest known algorithms for the multiplication of large integers or polynomials are based on the discrete Fourier transform: the sequences of digits or coefficients are interpreted as vectors whose convolution needs to be computed; in order to do this, they are first Fourier-transformed, then multiplied component-wise, then transformed back. Winter Semester 006/7 Computational Physics I Lecture 1

23 Power Spectrum The autocorrelation of a function is Corr[y]() = y(t) * y(t + ) dt and the power spectrum is defined as the Fourier transform of the autocorrelation PS[y]( f ) = Corr[y]()e if d For a periodic function, the correlation is often defined as the expectation value. There is no convention on the normalization, so be careful about the values. Best to see which frequencies dominate a given spectrum. Here is a practical approach for a discretely sampled function: C k = N 1 c j e ijk / N k = 0,1,,N 1 Winter Semester 006/7 Computational Physics I Lecture 1 3

24 P(0) = P( f 0 ) = 1 N C 0 Power Spectrum P( f k ) = 1 [ N C k + C ] N N k k = 1,,, 1 P( f c ) = P( f N / ) = 1 N C N / where only positive frequencies are considered: f k k N = f c k N k = 0,1,, N Let s try it out on our example: The 4Hz frequency is picked out. Winter Semester 006/7 Computational Physics I Lecture 1 4

25 Noise We now add some noise to our spectrum (Gaussian smearing with =0.5) and see what happens: Winter Semester 006/7 Computational Physics I Lecture 1 5

26 Exercizes 1. Solve the minimization problem from last lecture with MINUIT.. Generate 64 data points using f (t) = cos( /4 + f 1 t) + cos(f t) f 1 = 0.5, f = 1 = 0. and fit with a discrete Fourier transform. Extract the power spectrum. Winter Semester 006/7 Computational Physics I Lecture 1 6

Lecture 7. Fourier Analysis

Lecture 7. Fourier Analysis Lecture 7 Fourier Analysis Summary Lecture 6 Minima and maxima 1 dimension : Bracket minima : 3 values of f(x) : f(2) < f(1) and f(2)