CS 6604: Data Mining Large Networks and Time-series. B. Aditya Prakash Lecture #12: Time Series Mining

Size: px

Start display at page:

Download "CS 6604: Data Mining Large Networks and Time-series. B. Aditya Prakash Lecture #12: Time Series Mining"

Allen Lane
6 years ago
Views:

1 CS 6604: Data Mining Large Networks and Time-series B. Aditya Prakash Lecture #12: Time Series Mining

2 Pa<erns Y X 2

3 Pa<erns Y More Data X 3

4 Pa<erns Y Anomaly? X 4

5 Pa<erns Y Anomaly? ExtrapolaFon X 5

6 Y Pa<erns ImputaFon Anomaly ExtrapolaFon X 6

7 Pa<erns ImputaFon Anomaly Compression ExtrapolaFon 7

8 FOURIER TRANSFORM Based on slides by Christos Faloutsos. 8

9 DSP - Detailed outline DFT what why how Arithmetic examples properties / observations DCT 2-d DFT Fast Fourier Transform (FFT) 9

10 Introduction Goal: given a signal (eg., sales over time and/ or space) Find: patterns and/or compress count lynx caught per year year 10

11 What does DFT do? A: highlights the periodicities 11

12 Why should we care? A: several real sequences are periodic Q: Such as? 12

13 Why should we care? A: several real sequences are periodic Q: Such as? A: sales patterns follow seasons; economy follows 50-year cycle temperature follows daily and yearly cycles Many real signals follow (multiple) cycles 13

14 Why should we care? For example: human voice! Frequency analyzer speaker identification impulses/noise -> flat spectrum high pitch -> high frequency Freq.exe 14

15 DFT and stocks Dow Jones Industrial index, 6/18/ /21/

16 DFT and stocks Log(ampl) Dow Jones Industrial index, 6/18/ /21/2001 just 3 DFT coefficients give very good approximation freq 16

17 17 DFT: definition Discrete Fourier Transform (n-point): inverse DFT = = + = = = ) / 2 *exp( 1/ 1) ( ) / 2 *exp( 1/ n f f t n t t f n t f j X n x j n t f j x n X π π

18 How does it work? Skip alue Decomposes signal to a sum of sine (and cosine) waves. Q:How to assess similarity of x with a wave? x ={x 0, x 1,... x n-1 } 0 1 n-1 time 18

19 How does it work? Skip A: consider the waves with frequency 0, 1,...; use the inner-product (~cosine similarity) alue freq. f=0 value freq. f=1 (sin(t * 2 π/n) ) 0 1 n-1 time 0 1 n-1 time 19

20 How does it work? Skip A: consider the waves with frequency 0, 1,...; use the inner-product (~cosine similarity) value freq. f=2 0 1 n-1 time 20

21 How does it work? Skip basis functions 0 1 n-1 sine, freq =1 0 1 n n-1 cosine, f=1 sine, freq = n n-1 cosine, f=2 21

22 How does it work? Skip Basis functions are actually n-dim vectors, orthogonal to each other similarity of x with each of them: inner product DFT: ~ all the similarities of x with the basis functions 22

23 How does it work? Skip Since e jf = cos(f) + j sin(f) (j=sqrt(-1)), we finally have: 23

24 24 DFT: definition Discrete Fourier Transform (n-point): inverse DFT = = + = = = ) / 2 *exp( 1/ 1) ( ) / 2 *exp( 1/ n f f t n t t f n t f j X n x j n t f j x n X π π

25 DFT: definition Good news: Available in all symbolic math packages, eg., in mathematica x = [1,2,1,2]; X = Fourier[x]; Plot[ Abs[X] ]; 25

26 DFT: definition (variations: 1/n instead of 1/sqrt(n) exp(-...) instead of exp(+...) ) 26

27 DFT: definition Observations: X f : are complex numbers except X 0, who is real Im (X f ): ~ amplitude of sine wave of frequency f Re (X f ): ~ amplitude of cosine wave of frequency f x: is the sum of the above sine/cosine waves 27

28 DFT: definition Skip Observation - SYMMETRY property: X f = (X n-f )* ( * : complex conjugate: (a + b j)* = a - b j ) 28

29 DFT: definition Definitions A f = X f : amplitude of frequency f X f 2 = Re(X f ) 2 + Im(X f ) 2 = energy of frequency f phase φ f at frequency f Im A f X f φ f Re 29

30 DFT: definition Skip Amplitude spectrum: X f vs f (f=0, 1,... n-1) SYMMETRIC (Thus, we plot the first half only) 0 1 n-1 30

31 DFT: definition Skip Phase spectrum φ f vs f (f=0, 1,... n-1): Anti-symmetric (Rarely used) 31

32 DFT: examples flat Amplitude time freq 32

33 DFT: examples Low frequency sinusoid time freq 33

34 DFT: examples Sinusoid - symmetry property: X f = X * n-f time freq 34

35 DFT: examples Higher freq. sinusoid time freq 35

36 DFT: examples examples =

37 DFT: examples examples Ampl Freq. 37

38 DFT: Amplitude spectrum Amplitude: A = Re ( X ) + f 2 2 f 2 Im ( X f ) count actual mean mean+f req12 Ampl. freq=0 freq=12 year Freq. 38

39 DFT: Amplitude spectrum count actual mean mean+f req12 Ampl. freq=0 freq=12 year Freq. 39

40 DFT: Amplitude spectrum count actual mean mean+f req12 Ampl. freq=0 freq=12 year Freq. 40

41 DFT: Amplitude spectrum excellent approximation, with only 2 frequencies! so what? actual mean mean+f req Freq

42 DFT: Amplitude spectrum excellent approximation, with only 2 frequencies! so what? A1: compression A2: pattern discovery (A3: forecasting) 42

43 DFT: Amplitude spectrum excellent approximation, with only 2 frequencies! so what? A1: (lossy) compression A2: pattern discovery

44 DFT: Amplitude spectrum excellent approximation, with only 2 frequencies! so what? A1: (lossy) compression A2: pattern discovery actual mean mean+f req12 44

45 DFT: Amplitude spectrum Let s see it in action! ( FFTSpectrumAnalyser.html) plain sine phase shift two sine waves the chirp function 45

46 Plain sine 46

47 Plain sine 47

48 Plain sine phase shift 48

49 Plain sine phase shift 49

50 Plain sine 50

51 Two sines 51

52 Two sines 52

53 Chirp 53

54 Chirp 54

55 DFT: Parseval s theorem sum( x t 2 ) = sum ( X f 2 ) Ie., DFT preserves the energy or, alternatively: it does an axis rotation: x1 x = {x0, x1} x0 55

56 DFT: Parseval s theorem sum( x t 2 ) = sum ( X f 2 ) Ie., DFT preserves the energy or, alternatively: it does an axis rotation: x1 x = {x0, x1} x0 56

57 DSP - Detailed outline DFT what why how Arithmetic examples properties / observations DCT 2-d DFT Fast Fourier Transform (FFT) 57

58 Arithmetic examples Skip Impulse function: x = { 0, 1, 0, 0} (n = 4) X 0 =? value time 58

59 Arithmetic examples Skip Impulse function: x = { 0, 1, 0, 0} (n = 4) X 0 =? A: X 0 = 1/sqrt(4) * 1* exp(-j 2 π 0 / n ) = 1/2 X 1 =? X 2 =? X 3 =? 59

60 Arithmetic examples Skip Impulse function: x = { 0, 1, 0, 0} (n = 4) X 0 =? A: X 0 = 1/sqrt(4) * 1* exp(-j 2 π 0 / n ) = 1/2 X 1 = -1/2 j X 2 = - 1/2 X 3 = + 1/2 j Q: does the symmetry property hold? 60

61 Arithmetic examples Skip Impulse function: x = { 0, 1, 0, 0} (n = 4) X 0 =? A: X 0 = 1/sqrt(4) * 1* exp(-j 2 π 0 / n ) = 1/2 X 1 = -1/2 j X 2 = - 1/2 X 3 = + 1/2 j Q: does the symmetry property hold? A: Yes (of course) 61

62 Arithmetic examples Skip Impulse function: x = { 0, 1, 0, 0} (n = 4) X 0 =? A: X 0 = 1/sqrt(4) * 1* exp(-j 2 π 0 / n ) = 1/2 X 1 = -1/2 j X 2 = - 1/2 X 3 = + 1/2 j Q: check Parseval s theorem 62

63 Arithmetic examples Skip Impulse function: x = { 0, 1, 0, 0} (n = 4) X 0 =? A: X 0 = 1/sqrt(4) * 1* exp(-j 2 π 0 / n ) = 1/2 X 1 = -1/2 j X 2 = - 1/2 X 3 = + 1/2 j Q: (Amplitude) spectrum? 63

64 Arithmetic examples Impulse function: x = { 0, 1, 0, 0} (n = 4) X 0 =? A: X 0 = 1/sqrt(4) * 1* exp(-j 2 π 0 / n ) = 1/2 X 1 = -1/2 j X 2 = - 1/2 X 3 = + 1/2 j Q: (Amplitude) spectrum? A: FLAT! 64

65 Arithmetic examples Q: What does this mean? 65

66 Arithmetic examples Q: What does this mean? A: All frequencies are equally important -> we need n numbers in the frequency domain to represent just one non-zero number in the time domain! frequency leak 66

67 DSP - Detailed outline DFT what why how Arithmetic examples properties / observations DCT 2-d DFT Fast Fourier Transform (FFT) 67

68 Observations DFT of step function: x = { 0, 0,..., 0, 1, 1,... 1} x t t 68

69 Observations DFT of step function: x = { 0, 0,..., 0, 1, 1,... 1} x t f = 0 t 69

70 Observations DFT of step function: x = { 0, 0,..., 0, 1, 1,... 1} x t f=1 f = 0 t 70

71 Observations DFT of step function: x = { 0, 0,..., 0, 1, 1,... 1} x t f=1 t the more frequencies, the better the approx. ringing becomes worse f = 0 reason: discontinuities; trends 71

72 Observations Ringing for trends: because DFT subconsciously replicates the signal x t t 72

73 Observations Ringing for trends: because DFT subconsciously replicates the signal x t t 73

74 Observations Ringing for trends: because DFT subconsciously replicates the signal x t t 74

75 original 75

76 DC and 1st 76

77 DC and 1st And 2nd 77

78 DC and 1st And 2nd And 3rd 78

79 DC and 1st And 2nd And 3rd And 4th 79

80 Observations Q: DFT of a sinusoid, eg. x t = 3 sin( 2 π / 4 t) (t = 0,..., 3) Q: X 0 =? Q: X 1 =? Q: X 2 =? Q: X 3 =? 80

81 Observations Q: DFT of a sinusoid, eg. x t = 3 sin( 2 π / 4 t) (t = 0,..., 3) Q: X 0 = 0 Q: X 1 = -3 j Q: X 2 = 0 Q: X 3 = 3j check symmetry check Parseval 81

82 Observations Q: DFT of a sinusoid, eg. x t = 3 sin( 2 π / 4 t) (t = 0,..., 3) Q: X 0 = 0 Q: X 1 = -3 j A f Q: X 2 = 0 Q: X 3 = 3j Does this make sense? f 82

83 Property Shifting x in time does NOT change the amplitude spectrum eg., x = { } and x = { }: same (flat) amplitude spectrum (only the phase spectrum changes) Useful property when we search for patterns that may slide 83

84 DSP - Detailed outline DFT what why how Arithmetic examples properties / observations DCT 2-d DFT Fast Fourier Transform (FFT) 84

85 DSP - Detailed outline DFT what why how Arithmetic examples properties / observations DCT 2-d DFT Fast Fourier Transform (FFT) 85

86 FFT Skip What is the complexity of DFT? X f n = 1/ n t= 1 0 x t * exp( j2π tf / n) 86

87 FFT Skip What is the complexity of DFT? X f n = 1/ n t= 1 0 x t * exp( j2π tf / n) A: Naively, O(n 2 ) 87

88 FFT Skip However, if n is a power of 2 (or a number with many divisors), we can make it O(n log n) Main idea: if we know the DFT of the odd time-ticks, and of the even time-ticks, we can quickly compute the whole DFT Details: in Num. Recipes 88

89 DFT - Conclusions It spots periodicities (with the amplitude spectrum ) can be quickly computed (O( n log n)), thanks to the FFT algorithm. standard tool in signal processing (speech, image etc signals) actual mean mean+f req Freq

90 WAVELET TRANSFORM Based on material by Christos Faloutsos. 90

91 Reminder: Problem: Goal: given a signal (eg., #packets over time) Find: patterns, periodicities, and/or compress count lynx caught per year (packets per day; virus infections per month) year 91

92 Wavelets - DWT DFT is great - but, how about compressing a spike? value time 92

93 Wavelets - DWT DFT is great - but, how about compressing a spike? A: Terrible - all DFT coefficients needed! value time Ampl Freq. 93

94 Wavelets - DWT DFT is great - but, how about compressing a spike? A: Terrible - all DFT coefficients needed! value time

95 Wavelets - DWT Similarly, DFT suffers on short-duration waves (eg., baritone, silence, soprano) value time 95

96 Wavelets - DWT freq Solution#1: Short window Fourier transform (SWFT) But: how short should be the window? value time time 96

97 Wavelets - DWT freq Answer: multiple window sizes! -> DWT Time domain DFT SWFT DWT time 97

98 Haar Wavelets subtract sum of left half from right half repeat recursively for quarters, eight-ths,... 98

99 Wavelets - construction x0 x1 x2 x3 x4 x5 x6 x7 99

100 Wavelets - construction level 1 d1,0 s1,0 d1,1 s1, x0 x1 x2 x3 x4 x5 x6 x7 100

101 Wavelets - construction level 2 d2,0 s2,0 d1,0 s1,0 d1,1 s1, x0 x1 x2 x3 x4 x5 x6 x7 101

102 Wavelets - construction etc... d2,0 s2,0 d1,0 s1,0 d1,1 s1, x0 x1 x2 x3 x4 x5 x6 x7 102

103 Wavelets - construction Q: map each coefficient on the time-freq. plane d2,0 s2,0 f d1,0 s1,0 d1,1 s1, x0 x1 x2 x3 x4 x5 x6 x7 t 103

104 Wavelets - construction Q: map each coefficient on the time-freq. plane d2,0 s2,0 f d1,0 s1,0 d1,1 s1, x0 x1 x2 x3 x4 x5 x6 x7 t 104

105 Haar wavelets - code #!/usr/bin/perl5 # expects a file with numbers # and prints the dwt transform # The number of time-ticks should be a power of 2 # USAGE # haar.pl <fname> # the smooth component of the signal # the high-freq. component # collect the values into the = split ); } my $len = scalar(@vals); my $half = int($len/2); while($half >= 1 ){ for(my $i=0; $i< $half; $i++){ $diff [$i] = ($vals[2*$i] - $vals[2*$i + 1] )/ sqrt(2); print "\t", $diff[$i]; $smooth [$i] = ($vals[2*$i] + $vals[2*$i + 1] )/ sqrt(2); } print $half = int($half/2); } print "\t", $vals[0], "\n" ; # the final, smooth component 105

106 Wavelets - construction Observation1: + can be some weighted addition - is the corresponding weighted difference ( Quadrature mirror filters ) Observation2: unlike DFT/DCT, there are *many* wavelet bases: Haar, Daubechies-4, Daubechies-6, Coifman, Morlet, Gabor,

107 Wavelets - how do they look like? E.g., Daubechies-4 107

108 Wavelets - how do they look like? E.g., Daubechies-4?? 108

109 Wavelets - how do they look like? E.g., Daubechies-4 109

110 Wavelets - Drill#1: Q: baritone/silence/soprano - DWT? f value t time 110

111 Wavelets - Drill#1: Q: baritone/silence/soprano - DWT? f value t time 111

112 Wavelets - Drill#2: Q: spike - DWT? f t

113 Wavelets - Drill#2: Q: spike - DWT? f t

114 Wavelets - Drill#3: Q: weekly + daily periodicity, + spike - DWT? f t 114

115 Wavelets - Drill#3: Q: weekly + daily periodicity, + spike - DWT? f t 115

116 Wavelets - Drill#3: Q: weekly + daily periodicity, + spike - DWT? f t 116

117 Wavelets - Drill#3: Q: weekly + daily periodicity, + spike - DWT? f t 117

118 Wavelets - Drill#3: Q: weekly + daily periodicity, + spike - DWT? f t 118

119 Wavelets - Drill#3: Q: DFT? DWT DFT f f t t 119

120 Delta? x(0)=1; x(t)= 0 elsewhere f t 120

121 Delta? x(0)=1; x(t)= 0 elsewhere f t 121

122 2 cosines? x(t) = cos( 2 * pi * 4 * t / 1024 ) + 5 * cos( 2 * pi * 8 * t / 1024 ) f t 122

123 2 cosines? x(t) = cos( 2 * pi * 4 * t / 1024 ) + 5 * cos( 2 * pi * 8 * t / 1024 ) f t Which one is for freq.=4? 123

124 2 cosines? x(t) = cos( 2 * pi * 4 * t / 1024 ) + 5 * cos( 2 * pi * 8 * t / 1024 ) f~8 f~4 f t Which one is for freq.=4? 124

125 Chirp? x(t) = cos( 2 * pi * t * t / 1024 ) f t 125

126 Chirp? x(t) = cos( 2 * pi * t * t / 1024 ) f t 126

127 Chirp? x(t) = cos( 2 * pi * t * t / 1024 ) SWFT? f t 127

128 Chirp? x(t) = cos( 2 * pi * t * t / 1024 ) SWFT f t 128

129 More examples (BGP updates) [Prakash+, 2009] #updates time 129

130 More examples (BGP updates) Low freq.: omitted freq. f t 130

131 More examples (BGP updates) freq.?? 131

132 More examples (BGP updates) Prolonged spike freq. f t 132

133 More examples (BGP updates) 15K msgs, for several hours: 6pm-4am freq. 133

134 Wavelets - Drill Or use R, octave or matlab R: install.packages("wavelets") library("wavelets") X1<-c(1,2,3,4,5,6,7,8) dwt(x1, n.levels=3, filter= d4 ) mra(x1, n.levels=3, filter= d4 ) 134

135 Wavelets - k-dimensions? easily defined for any dimensionality (like DFT, DCT) 135

136 Wavelets - example Wavelets achieve *great* compression: ,000 # coefficients 136

137 Wavelets - intuition Edges (horizontal; vertical; diagonal) 137

138 Wavelets - intuition Edges (horizontal; vertical; diagonal) recurse 138

139 Wavelets - intuition Edges (horizontal; vertical; diagonal) 139

140 Advantages of Wavelets Better compression (better RMSE with same number of coefficients) closely related to the processing of the mammalian eye and ear Good for progressive transmission handle spikes well usually, fast to compute (O(n)!) 140

141 Overall Conclusions DFT spot periodicities DWT : multi-resolution - matches processing of mammalian ear/eye better Both: powerful tools for compression, pattern detection in real signals Both: included in math packages (matlab, R, mathematica,... ) 141

Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Chapter 12 Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted for noncommercial,