Reduced Complexity Models in the Identification of Dynamical Networks: links with sparsification problems

Transcription

1 Reduced Complexity Models in the Identification of Dynamical Networks: links with sparsification problems Donatello Materassi, Giacomo Innocenti, Laura Giarré December 15, 2009

2 Summary 1 Reconstruction of a topology Motivation Problem Formulation 2 Links with Sparsification problems Sparse representation and Compressive sensing Greedy algorithms Cycling Orthogonal Least Squares 3 Examples 4 Conclusions

3 Networks and their applications Networks are everywhere Communications Systems (Internet, wireless communications) Transportations (traffic control) Automatic Learning (neural networks) Parallel Computing (cellular automata) Formation Control (unmanned vehicles) Process Scheduling (Petri nets)

4 Networks and their applications Design Advantages Redundancy Modularity Distributed algorithms Technological applications the network topology is observable it is the objective of the design

5 Topology Identification: Motivations Reconstruction Genetics (Phylogenetic Trees) Cognitive Science (Computational Theory of Mind) Biochemistry (Gene Regulatory Networks) Complexity Reduction Cognitive Science (CTM and brain modularity) Biochemistry (gene regulatory networks) Economics (stock market, currency exchange) Social Systems There are not many techniques to identify a topology

6 Problem formulation N scalar time series {X i } i=1,...,n are observed. Assume zero-mean and wide-sense stationarity Linear smoothing via Wiener Filtering along all X j s: N 2 N min E Q j (z) X j W (j) i (z)x i (z) W (j) i j=1 j=1,j i (1) Draw an arc (i, j) if W (j) i 0 Leads to a complete graph

7 Problem formulation Consider to choose only m j > 0 arcs for X j [ ( m j )] 2 min E Q W (j) (z) 0 m j j (z) X j W (j) i (z)x i k=1 (2) where W (j) (z) 0 = # of non-zero entries of W (j) (z) Q j are frequency weights Draw an arc (i, j) if W ji 0 Leads to reduced complexity

8 Sparse representation Consider a full-rank matrix Ψ R p q with q >> p (Ψ is called dictionary ) Define the problem min x 0 Ψw (3) w 0 m We are looking for the best representation of x 0 with at most m non-zero entries This is a formulation of the Compressive Sensing problem

9 Links with Sparsification problems The two problems look very similar [ ( m j )] 2 min E Q W (j) (z) 0 m j j (z) X j W (j) i (z)x i k=1 (4) min x 0 Ψw (5) w 0 m Minimization of a norm induced by a scalar product Sparsity constraint of the solution Non-convex problems Common greedy algorithms

10 Greedy algorithms Selection Find the atom that has the highest scalar product with the residual ψ i = arg max ψ Ψ < r i 1, ψ > and add it to the the selected atoms Ψ i = Ψ i 1 ψi Update Update the coefficients w i and the residual r i = x0 Ψ i w i minimizing the new error. MP: optimize the coefficients on the last selected atom; OLS: optimize the minimum error over the set of selected atoms

11 Orthogonal Least Squares OLS projects a vector on the elements (atoms) of a redundant base (dictionary) and chooses the element that provides the largest cost reduction x 3 x 2 x 1

12 Orthogonal Least Squares OLS projects a vector on the elements (atoms) of a redundant base (dictionary) and chooses the element that provides the largest cost reduction x 3 x 2 x 1

13 Orthogonal Least Squares OLS projects a vector on the elements (atoms) of a redundant base (dictionary) and chooses the element that provides the largest cost reduction x 3 x 2 x 1

14 Orthogonal Least Squares OLS projects a vector on the elements (atoms) of a redundant base (dictionary) and chooses the element that provides the largest cost reduction x 3 x 2 x 1

15 Orthogonal Least Squares OLS projects a vector on the elements (atoms) of a redundant base (dictionary) and chooses the element that provides the largest cost reduction x 3 x 2 x 1

16 Orthogonal Projection What we mean by orthogonal projection in our space of stochastic processes X j? By orthogonal projection of a stochastic process X j on a set of stochastic processes {X αi } i=1,...,mj, we mean the best estimate of X j which can be given, in the sense of the least squares, as a linear dynamic combination of {X αji } i=1,...,mj. Such an estimate is given by the well-known Wiener Filter.

17 Cycling Orthogonal Least Squares (COLS) Orthogonal Least Squares greedy algorithm similar to Matching Pursuit (but slower) terminates in m steps may not stop in a local minimum Cycling Orthogonal Least Squares (COLS) first m steps as in OLS any chosen atom can be replaced by another one if there is cost reduction slower that OLS (makes more steps) terminates in a local minimum

18 Cycling Orthogonal Least Squares (COLS) Cycling Orthogonal Least Squares: 0. define X 0 := 0 (null time series) and c = initialize the m j -ple S = X 0, X 0..., X 0 2. while c m j 2a. for i = 1,..., m, i j define S i as the m j ple where X i replaces the i th element of S 0 Define r (i) k 2b. α = arg max i r (i) k as the projection of r on to S i 2c. if X alpha = S[j] then c = c + 1 2d. else S[j] = X alpha, c = 1, j = j + 1 mod m j 3. return S

19 Network reconstruction The transfer functions were randomly generated third order FIR models, simulation steps, white additive and mutually not correlated noises acting on every single node. Q j = 1, non-causal Wiener Filters. Thus it can be said that we are considering a smoothing scenario for our modeling. The Wiener filters have been computed estimating the cross-spectral densities of the signals X i, under the assumption of ergodicity.

20 Reconstruction via COLS (a) (b) (c) (d) Figure: The actual structure (a), and the reconstruction obtained using the Cycling OLS algorithm with m j = 1 (b), m j = 2 (c) and m j = 3 (d).

21 Application to Currency Exchange Rates SEK NOK CAD GBP DKK USD EU CNY KRW JPY MXN INR THB HKD TWD LKR MYR BRL SGD AUD ZAR NZD

22 Application to Currency Exchange Rates CAD USD MXN BRL CAD USD MXN BRL CAD USD MXN BRL EU INR EU INR EU INR GBP LKR GBP LKR GBP LKR DKK SGD DKK SGD DKK SGD NOK MYR NOK MYR NOK MYR SEK THB SEK THB SEK THB ZAR KRW ZAR KRW ZAR KRW AUD JPY AUD JPY AUD JPY NZD CNY HKD TWD NZD CNY HKD TWD NZD CNY HKD TWD (a) (b) (c) Figure: The reconstructed topologies obtaining applying the Cycling OLS to the exchange rate time series of the 22 selected currencies.

23 Conclusions We derive a topological structure from a set of time series. Every time series is represented as a node in a graph. No a priori info on the network. In order to modulate the complexity of the final graph a maximum number m j of arcs pointing at X j is assumed. The network complexity reduction is strictly related to the problem of sparsification Suboptimal greedy algorithms with a modification of the Orthogonal Least Squares. Applications to real data: currencies (daily data of the past 10 years normalized to the swiss frank)

24 Bibliography E. Candès, M. Wakin, and S. Boyd, Enhancing sparsity by reweighted l 1 minimization, Journal of Fourier Analysis and Applications, M. Timme, Revealing network connectivity from response dynamics, Phys. Rev. Lett., J. A. Tropp, Greed is good: Algorithmic results for sparse approximation, IEEE Trans. Inform. Theory, E. J. Candès and T. Tao, Decoding by linear programming, IEEE Transactions on Information Theory, S. Mallat and Z. Zhang, Matching pursuits with time-frequency dictionaries, IEEE Transactions on Signal Processing, 1993.

25 Suboptimal Modeling Approach The transfer functions {W ji (z)} describe the linear dynamical component of the dependencies among the processes. Hence, the corresponding errors {e j } are responsible for the remaining unmodeled components. Such sets provide respectively a qualitative and quantitative description of the dynamics among the time series. Suboptimal strategy For any given X j, find the process X k and the corresponding transfer function W jk, which provide the best contribution to X j in term of the realted modeling error e j, i.e. such that [ E (Q(z)(X j W jk (z)x k )) 2] = min E j i [ (Q(z)(X j W ji (z)x i )) 2] (6)

26 Frequency Domain Approach Given two stochastic processes X i, X j and a transfer function W (z), consider the quadratic cost E [ (ε Q ) 2] (7) where ε Q := Q(z) (X j W (z)x i ), (8) being Q(z) an arbitrary stable and causally invertible function weighting the modeling error X j W (z)x i. The frequency domain solution The problem of evaluating the transfer function Ŵ (z) such that the quadratic cost (7) is minimized is solved by the Wiener filter.

27 Wiener Filter Proposition (Wiener filter) The Wiener filter modeling X j by X i is the linear stable filter Ŵ ji minimizing the filtered quantity (7). Its expression is given by Ŵ ji (z) = Φ X i X j (z) Φ Xi (z), (9) Φ Xi (z) and Φ Xi X j (z) being the power spectral density and the cross-power spectral density respectively. The Wiener filter does not depend upon Q(z).

28 Coherence Function Since the weighting function Q(z) does not affect the Wiener filter we choose Q(z) equal to F j (z), the inverse of the spectral factor of Φ Xj (z), that is Φ Xj (z) = F 1 j (z)(f 1 j (z)) (10) with F j (z) stable and causally invertible. Then, the minimum cost assumes the form min E[ε 2 F j ] = π π ( 1 Φ ) X j X i (ω) 2 dω (11) Φ Xi (ω)φ Xj (ω) and it turns out to depend explicitly on the coherence function: C Xi X j (ω) := Φ X j X i (ω) 2 Φ Xi (ω)φ Xj (ω). (12)

29 Coherence-based Distance Define on the field Θ of the discrete zero mean and wide sense stationary stochastic processes the binary function [ 1 π ( d(x i, X j ) := 1 CXi X 2π j (ω) ) ] 1/2 dω X i, X j Θ. (13) Proposition π The function d(, ) as defined in (13) is a metric on Θ, that is d(x 1, X 2 ) 0 d(x 1, X 2 ) = 0 X 1 = X 2 d(x 1, X 2 ) = d(x 2, X 1 ) d(x 1, X 3 ) d(x 1, X 2 ) + d(x 2, X 3 ) for all X 1, X 2, X 3 Θ.