Symbolic Dynamics of Digital Signal Processing Systems Dr. Bingo Wing-Kuen Ling School of Engineering, University of Lincoln. Brayford Pool, Lincoln, Lincolnshire, LN6 7TS, United Kingdom. Email: wling@lincoln.ac.uk
My research interests Symbolic dynamics Limit cycle, fractal and chaotic behaviors of digital filters with two s, sigma delta modulators and perceptron training algorithms Optimization theory Semi-infinite programming, nonconvex optimization and nonsmooth optimization with applications to filter, filter bank, wavelet kernel and pulse designs Time frequency analysis Nonuniform filter banks, filter banks with block samplers to multimedia and biomedical signal processing Control theory Impulsive control, optimal control, fuzzy control and chaos control for HIV model and avian influenza model
Acknowledgements Prof. Xinghuo Yu (Royal Melbourne of Institute of Technology) Dr. Joshua D. Reiss (Queen Mary University of London) Dr. Herbert Ho-Ching Iu (University of Western Australia) Dr. Hak-Keung Lam (King s College London) 3
Introduction Outline complement arithmetic Sigma delta modulators Perceptron training algorithms Conclusions Q&A Session 4
Introduction Definition Symbolic dynamics is a kind of system dynamics which involves multi-level signals. Motivations Many practical signal processing systems, such as digital filters with two s complement arithmetic, sigma delta modulators and perceptron training algorithms, are symbolic dynamical systems. They are found in almost everywhere. 5
Challenges Introduction Symbolic dynamical systems could lead to chaotic, fractals and divergent behaviors. Stability conditions are generally unknown. Admissibility conditions are generally unknown. 6
What are digital filters? U Digital filters are systems that are characterized in the frequency domain. Y ( ) ( ω) H ω = U ( ω) ( k) u h(k) y( k) ( ) jωk ω = u( k) e ( ) jωk H ω = h( k) e Y ( ω) = y( k) k k k e jωk 7
Types of digital filters Lowpass filters Allow a low frequency band of a signal to pass through and attenuate a high frequency band. 8
Types of digital filters Bandpass filters Allow intermediate frequency bands of a signal to pass through and attenuate both low and high frequency bands. 9
Types of digital filters Highpass filters Allow a high frequency band of a signal to pass through and attenuate a low frequency band.
Types of digital filters Band reject filters Allow both low and high frequency bands of a signal to pass through and attenuate intermediate frequency bands.
Types of digital filters Notch filters Allow almost all frequency components to pass through but attenuate particular frequencies.
Types of digital filters Oscillators Allow particular frequency components to pass through and attenuate almost all frequency components. 3
Types of digital filters Allpass filters Allow all frequency components to pass through. (g) Magnitude response H(ω).5.5 3 4 Frequency ω 4
Hardware schematic u( k) Accumulator f( ) y( k) a z - z - b 5
When no input is present, the filter can be described by the following nonlinear state space difference equation: x ( k + ) x( k + ) = ( ( )) ( + ) F x k x k = f x( k) ( b x ( k) + a x ( k) ) Direct form 6
where f( ) is the nonlinear function associated with the two s complement arithmetic f(v) Straight lines -3-3 v Overflow No - overflow Overflow 7
8 where and m is the minimum integer satisfying ( ) ( ) ( ) { } ( ) { } m m k s x x x x I k x k x a b,,,,,,, :, L L < < A = ( ) ( ) ( ) ( ) k s k x k x k + = + A x + + m x a x b m
9 is called symbolic sequences. ( ) k s ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 3 3 < + = < + = < + = k x a k x b k s k x a k x b k s k x a k x b k s
T For example: x( ) = [.635.635], b = and a =. 5 b x( ) + a x( ) =.93 s( ) = b x () + a x () =.534 s() = M b x b x b x b x b x M s = ( 4) + a x( 4) =.8 s( 4) = and x( 5) ( 5) + a x( 5) =.7597 s( 5) = ( 6) + a x( 6) =.684 s( 6) = ( 7) + a x( 7) =.689 s( 7) = and x( 8) ( 8) + a x ( 8) =.839 s( 8) = and x ( 9) ( L L) Two's complement value =.998 =.93 =.96
Stability analysis of the corresponding linear system Eigenvalues of A λ = a ± a + 4 b
b R R R 3 - R 5 a - R R 3 R 4
R 5 :Magnitudes of both eigenvalues <. The corresponding linear system is stable. R and R 3 : Either one of the magnitudes of the eigenvalues <. The corresponding linear system is unstable. R and R 4 :Magnitudes of both eigenvalues are greater than. The corresponding linear system is unstable. 3
Effects of different initial conditions when the eigenvalues of system matrix are on the unit circle Type I trajectory There is a single rotated and translated ellipse in the phase portrait. a =.5, b =, x.6.6 ( ) = 4
5
Type II trajectory There are some rotated and translated ellipses in the phase portrait..66 a =.5, b =, x( ) =.66 6
This corresponds to the limit cycle behavior. 7
Type III trajectory There is a fractal pattern on the phase portrait. a =.5, b =, x.635.635 ( ) = 8
Conclusion: Very sensitive to initial conditions 9
3 Step response of second order digital filters with two s The system can be represented as: where and DC offset on symbolic sequences. ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) k s k u k x k x k u k x a k x b f k x k + + = + + = + B A x B = = a b A
.3.3 For a =.5, b =, c =, x( ) = 3
For a =.5, b = and c = s(k)= s(k)= s(k)=- s(k)=- 3
.7.7 For a =.5, b =, c =, x( ) =, s = (,,,,L) 33
For a =.5, b = and c = 34
.99.99 For a =.5, b =, c =, x( ) = 35
For a =.5, b = and c = 36
Sinusoidal response of second order digital filters with two s 37
38
Autonomous case when the eigenvalues of A are inside the unit circle 39
.8.6.4. x..4.6.8.8.6.4...4.6.8 x 4
.8.6.4 x. -. -.4 -.6 -.8 - - -.8 -.6 -.4 -. x..4.6.8 4
Autonomous case for third order digital filters with two s realized in cascade form System block diagram: f () y ( k ) f ( ) Delay Delay c x 3 ( k ) a Delay x ( k ) b x ( k ) y( k ) 4
43
44
Autonomous case for third-order digital filters with two s realized in parallel form f ( ) System block diagram: a Delay x ( k) b Delay x ( k) y ( k) f ( ) y( k) c Delay x 3 ( k) y ( k) 45
46
47
Admissibility of second order digital filters with two s Admissible set of periodic symbolic sequences is defined as a set of periodic symbolic sequences such that there exists an initial condition that produces the symbolic sequences. Set of initial conditions Admissible set of periodic symbolic sequences x() s 48
For example, when a =.5, b =, c =, M Admissible s Not admissible s s = (,L) s = (,L) s = (,L) s = (,,L ) s = (,,L) s = (,,L) s = (,,L ) s = (,,L) s = (, 3 s = (,,,L ) s = (,,,L) s = (,,,L ) s = (,,,L) s = (,,,L) s = (,,,L ) s = (,,,L ) s =,,,L s = (,,,L) s = (,,,L) s = (,,,L ) s = (,,,L) s = (,,,L) s = (,,,L) 5 Not found L ( ) 49
A periodic sequence s with period M is admissible if and only if for i =,, L, M M j= s ( mod( i + j, M )) sin M cos M θ sinθ j θ < 5
Sigma delta modulators What is sigma delta modulators? Sigma delta modulators are devices implementing sigma delta modulations and are widely used in analogue-to-digital conversions. The input signals are first oversampled to obtain the inputs of the sigma delta modulators. The loop filters are to separate the quantization noises and the input signals so that very high signal-to-noise ratios could be achieved at very coarse quantization schemes. 5
Sigma delta modulators Block diagram u(k) F(z) y(k) Q s(k) S U S N ( z) F( z) = ( z) + F( z) ( z) = ( z) + F( z) 5
Sigma delta modulators Nonlinear state space dynamical model: u cosθ z z F( z) = cosθ z + z y( k) cosθ y( k ) + y( k ) = cosθ ( u( k ) Q( y( k ) )) ( y( k ) Q( y( k ) )) y( k) = cosθ y( k ) y( k ) + cosθ ( u( k ) Q( y( k ) )) ( y( k ) Q( y( k ) )) ( k ) [ u( k ) u( k ) ] T x( k) [ y( k ) y( k ) ] T s( k) [ Q( y( k ) ) Q( y( k ) )] T 53
Sigma delta modulators A x Q cos θ B ( k +) = Ax( k) + B( u( k) s( k) ) ( y) y otherwise cosθ There are only finite numbers of possibilities of s(k). Hence, s(k) can be viewed as symbol and s(k) is called a symbolic sequence. 54
Sigma delta modulators Second order marginally stable bandpass sigma delta modulators.5 x.5 -.5 - - -.5.5.5 55 x
Sigma delta modulators 56
Sigma delta modulators Second order strictly stable bandpass sigma delta modulators.5 x(k).5 -.5 57 -.5.5.5 Clock cycle 3 3.5 4 4.5 5 4 x
Sigma delta modulators 58
Sigma delta modulators 59
Sigma delta modulators General second order bandpass sigma delta modulators 6
6 Admissibility of periodic sequence M M M M M M M D D D D D D D D D D D D D L L O M M O O O M O L ( ) ( ) ( ) ( ) [ ] T M s M s s s,,, L ζ ( ) ( ) + + θ θ θ θ sin sin sin sin j c j d j c j d D j Sigma delta modulators
K υ diag ( ) ( ) ) M M I A, L, I A [ υ, L υ] T ( s( ), s( ), L, s( M ) ) τ, s M ( d + c) u ( d c) + sin jθ, sinθ j= sin u θ jθ Σ be the admissible set of periodic output sequences with period M Σ = : Sigma delta modulators { s Q( K( Dζ + τ) ) = ζ} M j= sin 6
Perceptron training algorithms What is a perceptron training algorithm? A perceptron is a single neuron that employs a single bit quantization function as its activation function. x x d y ( k) ( k) where M w ( k) w d ( k) Q y( k) w ( k) ( ) ( T k = Q w ( k) x( k) ) x( k ) [, x ( k ),, x ( k )] T L d w( k ) [ w ( k ) w ( k ),, w ( k )] T d Q ( z), L z z < 63
Perceptron training algorithms What is a perceptron training algorithm? The weights of a perceptron is usually found by the perceptron training algorithm. w ( k + ) = w( k) + t ( k) y( k) x ( k) where t(k) is the desired output corresponding to x(k). If w(k) converges, then the steady state value of w(k) could be employed as the weights of the perceptron. 64
Perceptron training algorithms Example w Iteration index k w old (k) x(k) t(k) y(k) w new (k) [ ] T [ 5 ] T - [- -5 -] T [- -5 -] T [ ] T - - [- -5 -] T [- -5 -] T [ ] T - [ -4 ] T 3 [ -4 ] T [ 3 3] T - [ - 3] T 4 [ - 3] T [ 4 ] T - [ -5 ] T 5 [ -5 ] T [ 3] T - [ -3 4] T 6 [ -3 4] T [ 5 ] T - - [ -3 4] T 7 [ -3 4] T [ ] T - - [ -3 4] T 8 [ -3 4] T [ ] T [ -3 4] T Converge ( k + ) = w( k) ( k) y( k) x ( k) 9 [ -3 4] T [ 3 3] T [ -3 4] T [ -3 4] T [ 4 ] T - - [ -3 4] T [ -3 4] T [ 3] T [ -3 4] T + t 65
Perceptron training algorithms Example 4 x 3 Linearly Separable 4 6 x 66
Perceptron training algorithms Example w ( k + ) = w( k) Iteration index k w old (k) x(k) t(k) y(k) w new (k) [ ] T [ ] T - [- ] T [- ] T [ ] T - [ ] T [ ] T [ ] T [ ] T 3 [ ] T [ ] T - [- - ] T 4 [- - ] T [ ] T - - [- - ] T 5 [- - ] T [ ] T - [ - ] T 6 [ - ] T [ ] T - [ ] T 7 [ ] T [ ] T - [ - ] T 8 [ - ] T [ ] T - [- - ] T Limit Cycle + t ( k) y( k) x ( k) 9 [- - ] T [ ] T - [ - ] T [ - ] T [ ] T - [ ] T [ ] T [ ] T - [ - ] T 67
x Perceptron training algorithms Example Nonlinearly Separable - - x 68
Perceptron training algorithms Example 3 Class Class - 3.8.8.8.8.6.6.6.6.4.4.4.4.... w (k) w (k) w (k) -. -..8 -. -.4 -.4.6 -.4 -.6 -.6.4 -.6 -.8 -.8. -.8 - - -.5.5 x - 3 4 Time index k 3 4 Time index k - 3 4 69 Time index k
7 Perceptron training algorithms Example 4 The set of training feature vectors Desirable output Initial weight Result: Set of weights of the perceptron: Invariant set of weights of the perceptron: Invariant map: Note: is not bijective because,,, { },,, ( ) [ ] T,, = w ( ) [ ] T k,, = w Z k [ ] { } T,, = [ ] { } [ ] { } T T F,,,, : I 3 3 : ~ R R I F ( ) ( ) ( ) k k k T x w x > k Z,,,
Perceptron training algorithms Class Class -.8 -. -..8.6 -. -.4.6.4 -.3 -.6.4. -.4 -.8. x w (k) -.5 w (k) - w (k) -. -.6 -. -. -.4 -.7 -.4 -.4 -.6 -.8 -.6 -.6 -.8 -.9 -.8 -.8 - - -.5.5 x - 3 4 Time index k - 3 4 Time index k - 3 4 Time index k 7
Perceptron training algorithms Example 5 The set of training feature vectors.746,.867 Desirable output {,,, } Initial weight w =,.793,.33 Invariant set of the weights of the perceptron consists of three hyperplanes. ~ 3 3 Note: I F : R R is not bijective because k Z T x k > w k x k ( ) ( ) ( ) ( ) [ ] T.758,.83,.39.5883.364. 668 7
Perceptron training algorithms Class Class -..5.8.5 w.6 x.4 -.5. - -.5 4 -. 3 -.4 -.6 -.5-3 x w -.5-73 - w
Conclusions Many digital signal processing systems are symbolic dynamical systems. These symbolic dynamical systems could exhibit fractal, chaotic and divergent behaviors. By investigating the properties of these symbolic dynamical systems, unwanted behaviors could be avoided. 74
Q&A Session Thank you! Let me think Bingo 75