Mathematical Foundations of Signal Processing

Mathematical Foundations of Signal Processing Module 4: Continuous-Time Systems and Signals Benjamín Béjar Haro Mihailo Kolundžija Reza Parhizkar Adam Scholefield October 24, 2016

Continuous Time Signals 1 Modeling Continuous Time Signals Functions Continuous Time Systems 2 Analyzing Continuous Time Signals Why do we need Transforms? Fourier Transforms Fourier Series

Modeling Continuous Time Signals

Functions Any observed signal can be considered as a function (the values of a function) Harmonic Signal For instance, if we play a tone with a violin, the signal we observe (we hear) can be modeled by a harmonic signal s(t) = K α k e j2πfkt, k=1 that is, linear combination of sinusoids with frequencies f 1,...,f K (see courses Statistical Signal Processing and Signal Processing for Audio & Acoustics) 4.1 4

Functions Harmonic SIgnal (ct d) s(t) = 5e i2πν0t +10e i2π4ν0t +2e i2π20ν0t +0.7e i2π200ν0t. 4.1 5

Functions: Spaces Space of integrable functions L 1 (R) (required to apply integral tools) { } L 1 (R) = x(t) s.t. x(t) dt <. Note: f L 1 (R) is a sufficient condition to define R f(t)dt. Finite energy functions L 2 (R) (square integrable functions, a Hilbert space) { } L 2 (R) = x(t) s.t. x(t) 2 dt <. Note: Being in a Hilbert space is equivalent to observing quantities that make sense from a physical point of view signals of finite energy In case of periodic functions of period T, we rather consider L 1 ([ T/2,T/2)) and L 2 ([ T/2,T/2)) (absolutely and square absolutely integrable over the period) R R 4.1 6

Functions: Spaces Bounded functions L (R) (the magnitude of the signal is bounded) Recall that s(t) = esssup x(t). Notice that any measured quantity is t R bounded by the saturation level of the measuring instrument Space of smooth functions C p (continuously-differentiable functions), with p = 1,2,... Functions that are smooth, that is continuously differentiable up to the order p. Such notion is related to the bandwidth of the function (how fast the function varies) We should also mention piecewise smooth functions, i.e. functions that are smooth only on intervals 4.1 7

Piecewise smooth function: Linear interpolator Consider a finite sequences of (real) bounded values x 0,x 1,...,x K corresponding to the time instants 0,T,...,KT. For simplicity, we assume x 0 = x K = 0. The function that interpolates with lines such values is given by { x n +(t nt)(x n+1 x n ) nt t < (n+1)t, n = 0,...,K 1 x(t) = 0 t / [0,KT) Notice that, by construction, x(t) L 1 (R) L 2 (R) L (R). 4.1 8

Special Functions Dirac delta (generalized function) The Dirac delta is formally defined as δ(t) = 0, t 0 and f(t)δ(t)dt = f(0), with Why it is not a function? A function is defined point-wise; R R δ(t)dt = 1. If a function is 0 for every t 0 (i.e., zero almost everywhere), then its integral is 0, not 1. Ergo, δ(t) is a generalized function. Within the framework of this course, we can keep treating it as a function with not much risk, but for more complex signal models, a more precise definition is required (see course Mathematical Principles of Signal Processing). δ(t) is very handy to model: Events localized in time (spikes, impulses) Impulse response of a LSI system 4.1 9

Special Functions Heaviside function Another important function, particularly used in system theory, for instance to measure the response time of a system. The Heaviside function is defined as { 1 t 0; u(t) = 0 t < 0. Notice that the Dirac delta generalized function is commonly considered to be the derivative of the Heaviside function. Such a result is defined within a specific theory (distribution theory) and it is definitively not trivial 4.1 10

Special Functions Gaussian The Gaussian function is widely used: It has nice mathematical properties (Gaussian kernel) It models important physical phenomena such as diffusion It provides a good approximation, via the central limit theorem, of many other phenomena The Gaussian is a symmetric function with general form given by f(t) = γe α(t µ)2, where µ defines the symmetry point. By choosing α = 1/2σ 2, γ = 1/σ 2π the function is L 1 -normalized, i.e., f(t) L 1 = 1 and it can play the role of a probability density function 4.1 11

Deterministic correlation Deterministic autocorrelation a(τ) = Properties x(t)x (t τ)dt = x(t), x(t τ) t, τ R a(τ) = a ( τ), a(0) = Deterministic crosscorrelation c(τ) = x(t) 2 dt = x 2 x(t)y (t τ)dt = x(t), y(t τ) t, τ R Properties: ( ( ) c x,y(τ) = y(t τ)x (t)dt) = y(t )x (t +τ)dt = cy,x( τ) 4.1 12

Continuous Time Systems We are used to see a system as a box that outputs a certain function y(t) when fed by a function x(t) x T y We can actually elegantly formalize a system by considering it as an operator mapping an input function x V into an output function y V y = T(x) The function space V is typically L 2 (R) or L (R) Notice that a priori T is time-dependent 4.1 13

Types of Systems Definition (Linear system) A continuous-time system T is called linear when, for any inputs x and y and any α, β C, T(αx +βy) = αt(x)+βt(y) Definition (Memoryless system) A continuous-time system T is called memoryless when, for any real τ and inputs x and x, 1 {τ} x = 1 {τ} x 1 {τ} T(x) = 1 {τ} T(x ) Definition (Causal system) A continuous-time system T is called causal when, for any real τ and inputs x and x, 1 (,τ] x = 1 (,τ] x 1 (,τ] T(x) = 1 (,τ] T(x ) 4.1 14

Types of Systems Definition (Shift-invariant system) A continuous-time system T is called shift invariant when, for any real τ and input x, y = T(x) y = T(x ), where x (t) = x(t τ) and y (t) = y(t τ) Definition (BIBO stable system) A continuous-time system T is called bounded-input bounded-output stable when a bounded input x produces a bounded output y = T(x): x L (R) y L (R) 4.1 15

Basic Systems Shift Modulator Integrator Averaging operators y(t) = x(t t 0 ) y(t) = e jω0t x(t) y(t) = t x(τ)dτ y(t) = 1 T t+t/2 t T/2 x(τ)dτ, t R 4.1 16

Systems: Differential Equation Linear constant-coefficient differential equations describe LSI systems and are of the form: M d k x(t) N d k y(t) y(t) = b k dt k a k dt k k=0 Solution y (h) (t) to the homogeneous equation obtained by setting the input x(t) to zero Particular solution y (p) (t) found typically by assuming the output to be of the same form as the input k=1 Complete solution formed by superposition of the solution to the homogeneous equation and the particular solution 4.1 17

Linear Shift-Invariant System Systems that are linear shift invariant (LSI) are the most important ones: Several physical systems (real world) can be modeled as such They have very nice properties, the salient one being that they can be be described via the impulse response h(t) Definition (Impulse response) δ(t) T h(t) A function h is called the impulse response of LSI continuous-time system T when input δ produces output h Remark: We are used to consider the impulse response as the output when the input is a Dirac impulse. We shall see that the definition of the impulse response of an LSI system finds it natural and most rigorous context in the transform domain 4.1 18

Linear Shift-Invariant System Convolution We can express an arbitrary input to LSI system T as x(t) = x(τ)δ(t τ)dτ The output resulting from this input is y = T x = T = x(τ)δ(t τ)dτ = x(τ)h(t τ)dτ = h x x(τ)tδ(t τ)dτ 4.1 19

Linear Shift-Invariant System Convolution Definition (Convolution) The convolution between functions h and x is defined as (Hx)(t) = (h x)(t) = x(τ)h(t τ)dτ = where H is called the convolution operator associated with h. x(t τ)h(τ)dτ, In signal processing framework, convolution operator is commonly called filtering and the impulse response filter. 4.1 20

Linear Shift-Invariant System Properties 1) Convolution and BIBO stability Theorem An LSI system is BIBO stable if and only if its impulse response h(t) is in L 1 (R) (h(t) is absolutely integrable). Proof. a) BIBO stable if h(t) L 1 (R): A good exercise, try to do it yourself b) h(t) L 1 (R) if BIBO stable: We proceed by contradiction... Theorem If the input to a BIBO-stable system is in L p (R), the output is in L p (R) as well. 4.1 21

Linear Shift-Invariant System Linear Convolution with Circularly-Extended Signal Let a T-periodic signal x(t) L 1 ([0,T]) or L 2 ([0,T]) be the input of a convolutional system (filter) with impulse response h(t) L 1 (R) Call h T (t) the periodized version of h(t) h T (t) = n Zh(t nt) Then the output y(t) of the convolutional filter is given (h x)(t) = (h T x)(t) Definition (Circular convolution) The circular convolution between T-periodic functions h and x is defined as (Hx)(t) = (h x)(t) = T/2 T/2 x(τ)h(t τ)dτ = T/2 T/2 x(t τ)h(τ)dτ, where H is called the circular convolution operator associated with h 4.1 22

Analyzing Continuous Time Signals

Why do we need Transforms? Transforming a function (here a signal or the impulse response of a system) is motivated by one or both of the two following reasons: View from a different perspective the characteristics of the function (for example, the energy of the function) For instance, the Fourier transform enables to compute the power spectrum and therefore to see the distribution of the energy in the frequency domain Move to a space where certain computations and manipulations are simplified For instance, a convolution in the time domain becomes a product in the Fourier domain Notice that here we need the transform to be invertible Remark: For a deterministic function, Fourier and Laplace transforms provide a different perspective of its characteristics and a space where certain computations are simplified (the two above motivations are strictly linked). For a stochastic process this is not the case! (see course Mathematical Principles of Signal Processing) 4.2 24

Fourier transform Eigenfunctions of the convolution operator LSI systems have all unit-modulus complex exponential functions as eigenfunctions v(t) = e jωt, t R (H v)(t) = (h v)(t) = = h(τ)e jωτ dτ } {{ } λ ω v(t τ)h(τ)dτ = e jωt }{{} v(t) e jω(t τ) h(τ)dτ v is an eigenfunction of H with corresponding eigenvalue λ ω that we call the frequency response of the system H(ω) H e jωt = h e jωt = H(ω)e jωt 4.2 25

Fourier Transform It is straightforwardly defined for L 1 (R) signals, for L 2 (R) signal one needs to exploit Hilbert space tools It provides a representation of the signal in the frequency domain Under certain conditions, it is invertible 4.2 26

Fourier Transform Definition (Fourier transform) The Fourier transform of a function x is X(ω) = x(t)e jωt dt, ω R. (1a) It exists when (1a) is defined and is finite for all ω R; we then call it the spectrum of x. The inverse Fourier transform of X is x(t) = 1 2π X(ω)e jωt dω, t R. (1b) It exists when (1b) is defined and is finite for all t R. When the Fourier transform and inverse Fourier transform exist, we denote the Fourier transform pair as FT x(t) X(ω). We discuss two cases: x(t) L 1 (R) x(t) L 2 (R) allows the use of all the tools from Hilbert space theory 4.2 27

Fourier Transform Functions in L 1 (R) Existence: Existence of the Fourier transform straightforwardly follows from the existence of the integral for L 1 (R) functions, that is, x(t)e jωt dt = x(t) dt < Riemann-Lebesgue Theorem: Theorem R Let x(t) L 1 (R) and call X(ω) its Fourier transform. Then X(ω) is continuous and bounded (proof requires the Dominated Convergence Thm.) lim ω ± X(ω) = 0 The Fourier transform is a mapping from L 1 (R) to the space of continuous and bounded functions R 4.2 28

Fourier Transform Functions in L 1 (R) Fourier transform of the box function Let x be the box function, which is in L 1 (R) x(t) = { 1/ t0, for t t 0 /2; 0, otherwise Its Fourier transform is given by X(ω) = 1 t0/2 t0 t0/2 = ejωt0/2 e jωt0/2 jω t 0 e jωt dt = 1 jω t 0 e jωt t0/2 t0/2 = t 0 sinc ( ωt0 ) 2 which is bounded and continuous, but in L 2 (R), and not in L 1 (R) 1 x(t) 1 X(ω) 1 2 1 2 t 6Π 2Π 2Π 6Π ω 4.2 29

Fourier Transform Functions in L 1 (R) Inverse transform: x(t) L 1 (R) does not guarantee that X(ω) can be inverse-transformed In order to be inverse-transformed X(ω) must be in L 1 (R), and there is no subspace of x(t) L 1 (R) that implies this. This is one of the main drawbacks of the Fourier transform of L 1 (R) functions! If X(ω) L 1 (R) then x(t) = R X(ω)e jωt dω (the proof makes use of Gaussian kernel and standard Calculus arguments. Nevertheless, it is a bit tedious) 4.2 30

Fourier Transform Functions in L 2 (R) If x L 2 (R), the Fourier transform integral may or may not be defined for every ω R. The extension of the Fourier transform and its inverse from L 1 (R) to L 2 (R) is technically nontrivial Existence: The Fourier transform gives for any x in L 2 (R) a Fourier transform X itself in L 2 (R) (the Fourier transform in L 2 (R) is a mapping from a Hilbert space to a Hilbert space) Inverse transform: The inverse Fourier transform can be defined similarly for any X in L 2 (R), and it indeed provides an inversion, so x can be recovered from X (Going to the Fourier domain for performing computations really makes sense. Knowing that the Fourier transform can be extended rigorously to all of L 2 (R), we can put the technicalities aside) 4.2 31

Fourier Transform Functions in L 2 (R) Fourier transform of the sinc function ω0 ( x(t) = 2π sinc ω0 t ). 2 x(t) is in L 2 (R), but not in L 1 (R), and we cannot find the Fourier transform from the defining integral We find the Fourier transform by recognizing that the sinc function and the box function are a Fourier transform pair, and verify that X(ω) = { 2π/ω0, for ω ω 0 /2; 0, otherwise x(t) 1 X(ω) 1 1 1 t π π ω 4.2 32

Fourier Transform Properties Linearity Shift in time Shift in frequency Differentiation in time Integration Convolution in time αx(t) + βy(t) x(t t 0 ) e jω0t x(t) t d n x(t) dt n x(τ) dτ FT αx(ω)+βy(ω) FT e jωt0 X(ω) FT X(ω ω 0 ) FT (jω) n X(ω) FT 1 jω X(ω) (h x)(t) FT H(ω)X(ω) Convolution in frequency h(t) x(t) FT 1 (H X)(ω) 2π 4.2 33

Fourier Transform Properties Deterministic autocorrelation a(t) = x(τ)x (τ t)dτ FT A(ω) = X(ω) 2 Parseval s equality: unitary transform up to a scale factor and x 2 = 1 2π X 2 x, y = 1 X, Y 2π Adjoint The adjoint of the Fourier transform, F : L 2 (R) L 2 (R), is determined uniquely by Fx, y = x, F y for every x,y L 2 (R) Since Parseval s equality shows that F/ 2π is a unitary operator, we have Thus, ( 1 2π F) = ( 1 2π F) 1 = 2πF 1 F = 2πF 1 4.2 34

Fourier Transform Transform pairs Dirac delta Box Function { 1/ t0, for t t x(t) = 0 /2; 0, otherwise, Sinc function 1 FT 2π δ(ω) FT X(ω) = t 0 sinc(ωt 0 /2) x(t) = ω0 2π sinc(ω 0t/2) { FT 2π/ω0, for ω ω X(ω) = 0 /2; 0, otherwise. Heaviside function u(t) = { 1, for t 0; 0, otherwise. FT U(ω) = πδ(ω)+ 1 jω Gaussian function g(t) = γe αt2 FT π /(4α) G(ω) = γ α e ω2 4.2 35

Fourier Transform Regularity and spectral decay If X(ω) γ 1+ ω 1+ε, for all ω R, for some positive constants γ and ε, then X(ω) is integrable, or, X L 1 (R). Therefore, the inverse Fourier transform x is bounded and continuous, or, x C 0 If X(ω) γ 1+ ω p+1+ε for all ω R, for some nonnegative integer p and positive constants γ and ε, then x C p The Fourier transform of any x in C p is bounded by X(ω) for some positive constant γ γ 1+ ω p+1, for all ω R, Fourier transform of any continuous function is bounded by X(ω) for some positive constant γ γ, for all ω R, 1+ ω 4.2 36

Fourier Series Eigenfunctions of the Circular Convolution Operator The Fourier series arises from identifying the eigenfunctions of the circular convolution operator, which have the following form v(t) = e j(2π/t)kt, t R (H v)(t) = (h v)(t) = = = T/2 T/2 T/2 T/2 T/2 T/2 e j(2π/t)k(t τ) h(τ)dτ h(τ)e j(2π/t)kτ dτ } {{ } λ k v(t τ)h(τ)dτ e j(2π/t)kt }{{} v(t) v is an eigenfunction of H with corresponding eigenvalue λ k that we call the frequency response H k H e j(2π/t)kt = h e j(2π/t)kt = H k e j(2π/t)kt 4.2 37

Fourier Series Projecting onto the subspaces generated by each of the eigenfunctions Definition (Fourier series) The Fourier series coefficients of a periodic function x with period T are X k = 1 T T/2 T/2 x(t)e j(2π/t)kt dt, k Z. (4a) They exist when (4a) is defined and finite for all k Z; we then call them the spectrum of x. The Fourier series reconstruction from X is x(t) = k ZX k e j(2π/t)kt, t [ T/2,T/2). When the Fourier series coefficients exist and the reconstruction converges, we denote the Fourier series pair as x(t) FS X k. 4.2 38

Fourier Series Spectrum in l 1 (Z) Let the T-periodic function x have spectrum X l 1 (Z). Then Fourier series converges absolutely for every t since Xk e j(2π/t)kt = X k e j(2π/t)kt = X k = X 1 <. k Z k Z Moreover, the Fourier series converges to a continuous function x(t) When x is continuous and X l 1 (Z), k Z x(t) = x(t), for all t R 4.2 39

Fourier series Fourier Series as an Orthonormal Basis Expansion Theorem (Orthonormal basis from Fourier series) The set {ϕ k } k Z with ϕ k (t) = 1 T e j(2π/t)kt, t [ T/2,T/2), forms an orthonormal basis for L 2 ([ T/2,T/2)). Up to scaling by the constant T, the Fourier series is an orthonormal basis expansion. The orthonormality leads to geometrically-intuitive properties using Hilbert space tools 4.2 40

Fourier series Spectrum in l 2 (Z) Theorem (Fourier series on L 2 ([ T/2,T/2))) Let x L 2 ([ T/2, T/2)) have Fourier series coefficients X = Fx. Then the following hold: L 2 inversion: Let x be the Fourier series reconstruction from X. Then using the L 2 ([ T/2,T/2)) norm. x x = 0 Norm conservation: The linear operator TF : L 2 ([ T/2,T/2)) l 2 (Z) is unitary. This yields the Parseval s equalities x 2 = x, y = T/2 T/2 x(t) 2 dt = T k Z T/2 X k 2, T/2x(t)y (t)dt = T k Z X k Y k, where T-periodic function y has Fourier series coefficients Y. Least-squares approximation: The function x N (t) = N X k e j(2π/t)kt k= N is the least-squares approximation of x on the subspace spanned by {ϕ k (t)} N k= N. 4.2 41

Fourier series Properties Linearity Shift in time Shift in frequency Differentiation Integration t T/2 αx(t) + βy(t) x(t t 0) e j(2π/t)k0t x(t) d n x(t) dt n x(τ) dτ Circular convolution in time (h x)(t) Circular convolution in frequency FS αx k +βy k FS e j(2π/t)kt0 X k FS FS FS X k k0 ( j 2π ) n T k X k T Xk, for k 0 j2πk FS T H k X k h(t) x(t) FS (H X) k Circular deterministic autocorrelation a(t) = T/2 T/2 x(τ)x (τ t)dτ FS A k = T X k 2 4.2 42

Fourier series Transform pairs Square wave x(t) = { 1, for 1/2 t < 0; 1, for 0 t < 1/2; X k = { 2j/(πk), for k odd; 0, for k even x(t) = l Z 2j π(2l+1) ej2π(2l+1)t = 1 x(t) l=0 4 π(2l+1) sin(2π(2l+1)t). 2 Π X k 1 1 2 1 2 1 t 1 10 10 k 4.2 43

Fourier series Transform pairs Triangle wave y(t) = 1 2 t, for t [ 1/2,1/2) Y 0 = 0 { 1/(πk) Y k = 2, for k odd; 0, for k nonzero and even. x(t) X k 1 2 1 4 1 1 2 1 2 1 t 10 10 k 4.2 44

Fourier series Transform pairs Dirac comb s T (t) = n Zδ(t nt) S T,k = = = 1 T/2 T T/2 n Z δ(t nt)e j(2π/t)kt dt 1 T/2 δ(t)e j(2π/t)kt dt T T/2 1 T, k Z 1 T k Ze j(2π/t)kt = δ(t nt) n Z 4.2 45

Fourier series Relation of the Fourier series coefficients to the Fourier transform For a function that is identically zero outside of [ T/2,T/2), the Fourier series integral is the same as the Fourier transform integral for frequency ω = (2π/T)k An arbitrary T-periodic function x can be restricted to [ T/2,T/2) by defining x = 1 [ T/2,T/2) x It is then easy to verify that X k = 1 T X where X is the Fourier transform of x ( ) 2π T k, k Z, 4.2 46

Fourier series Fourier transform of periodic functions Let x be a T-periodic function, and let x be the restriction of x to [ T/2,T/2): x(t) = 1 [ T/2,T/2) x(t), t R Using the Dirac comb, we can write x(t) = (s T x)(t), t R Assuming we may apply the convolution theorem, we have X(ω) = S T (ω) X(ω), ω R We can write the Fourier transform of the Dirac comb as S T (ω) = n Z e jωnt The Dirac delta function S T can be expressed as a scaled Dirac comb with spacing 2π/T: S T (ω) = n Ze jωnt = 2π T δ ( ω 2π T k) = 2π T s 2π/T(ω). Finally, we may write the Fourier transform of X(ω) using the Fourier series coefficients of x: X(ω) = 2π X k δ ( ω 2π T k) k Z k Z 4.2 47

Fourier series Poisson sum formula Theorem (Poisson sum formula) Let s T be the Dirac comb and let x be a function with decay sufficient for the periodization (s T x)(t) = n Zx(t nt) to converge absolutely for all t. Then x(t nt) = 1 T n Z k Z X ( 2π T k) e j(2π/t)kt, where X is the Fourier transform of x. Specializing to T = 1 and t = 0, = n Zx(n) X(2πk). k Z 4.2 48

Fourier series Regularity and spectral decay Note A discontinuous function such as the square wave has Fourier series coefficients decaying only as O(1/k) A continuous function such as the triangle wave leads to a decay of O(1/k 2 ) Generally X k γ 1+ k q+1+ε for all k Z for some nonnegative integer q and positive constants γ and ε, implies x has q continuous derivatives If the periodic function x has q continuous derivatives, then its Fourier series coefficients satisfy X k γ 1+ k q+1 for all k Z 4.2 49