Applications of algorithmic complexity to network science and synthetic biology

Transcription

1 Applications of algorithmic complexity to network science and synthetic biology Hector Zenil Unit of Computational Medicine, KI Conference on Computability, Complexity and Randomness CCR 2014 Institute for Mathematical Sciences, National University of Singapore, June 9, 2014 Hector Zenil Applications of algorithmic complexity 1 / 35

2 Outline Outline: 1 The challenge of application of AIT 2 Estimating a Universal Distribution (CTM and BDM) 3 2-dimensional Kolmogorov complexity 4 Kolmogorov complexity of a graph 5 Application of BDM to network and synthetic biology Material mostly from the following papers: 1 Zenil et al. Physica A (2014). [4] 2 joint with Soler et al. Computability (2013). [2] 3 joint with Soler et al. PLoS ONE (2014). [7] 4 joint with Terrazas et al. Natural Computing (2013) [1] 5 Zenil and Tegnér, Wiley Book Series on Bioinformatics (to appear) [6] 6 Two forthcoming (Unit of Computational Medicine) Hector Zenil Applications of algorithmic complexity 2 / 35

3 Outline Information theory in medicine Figure : Used in MRI for mapping images with different coordinate systems. Images A and B are aligned by maximizing the Mutual Information of A and B. Hector Zenil Applications of algorithmic complexity 3 / 35

4 Motivation Sequences and networks are today fundamental objects of study in current molecular biology. Phylogenetics (R. Cilibrasi and P.M.B. Vitányi, Clustering by Compression (2005)): NID(x, y) = max{k(x y),k(y x)} max{k(x),k(y)} Figure : Sensitivity of information complexity measures to a simulation of GC-content variability. Each species specific average GC-content lies somewhere on each curve determining its place in a phylogenetic space. Hector Zenil Applications of algorithmic complexity 4 / 35

5 Molecular computation: programming porphyrin molecules Porphyrin molecules give the red color to blood. They are ideal for drug transportation. Computer simulation with Wang tiles. We produced predictions of behavioural self-organization base on a guided search of the conformational space with the use of AIT tools, in particular NCD and algorithmic mutual information. Configurations correspond to what is observed in the lab. [joint work with ICOS, U. of Nottingham] [Terrazas et al. Natural Computing (2013)] Hector Zenil Applications of algorithmic complexity 5 / 35

6 Programming porphyrin molecules (cont) Mapping the parameter space Conformational space: [joint work with ICOS, U. of Nottingham] [Terrazas et al. Natural Computing (2013)] Hector Zenil Applications of algorithmic complexity 6 / 35

7 Network biology The Human Genome Project completed in Network biology (Barabási group, 2004) is the realization that complexity in molecular biology (the cell) is in its interactions: Genes regulating other genes Proteins physically interacting Involved metabolic pathways Signaling networks connecting all levels Figure : Source: The NY Academy of Sciences (the DREAM project ebriefings) Hector Zenil Applications of algorithmic complexity 7 / 35

8 Introduction A typical biological network Figure : Pathway Projector: Web-Based Zoomable Pathway Browser Using KEGG Atlas and Google Maps API (Kono et al. PLoS ONE, 2009) Hector Zenil Applications of algorithmic complexity 8 / 35

9 [Source: Zenil and Tegnér, Wiley Book Series on Bioinformatics, to appear] Hector Zenil Applications of algorithmic [Data complexity source: Jeong et al. (2000) Nature 407, 651] 9 / 35 Biological networks are neither simple nor random Comparing the compressibility of biological networks C(g) vs. randomized C(r V(g) ) and complete versions C(c V(g) ): Network (g) V(g) E(g) C(c V(g) ) C(g) C(r V(g) ) Chlamydia Pneumoniae Mycoplasma Pneumoniae Chlamydia Trachomatis Rickettsia Prowazekii Mycoplasma Genitalium Treponema Pallidum Aeropyrum Pernix Oryza Sativa Arabidopsis Thaliana Pyrococcus Furiosus Pyrococcus Horikoshii Helicobacter Pylori Enterococcus Faecalis Streptococcus Pyogenes

10 Information network biology Node/edge deletion (in-silico knockout experiments) can send a network towards or away from randomness: Hector Zenil Applications of algorithmic complexity 10 / 35

11 Algorithmic information content of a network complete graph: K log( N ) random graph: K E The Kolmogorov complexity of hand picked (complete and Erdös-Rényi random) networks Complete and disconnected graphs with N nodes have low (algorithmic) information content. In a random graph every edge e E requires some information to be described. Hector Zenil Applications of algorithmic complexity 11 / 35

12 Compression has its limits Figure : Distributions plot for all 2 n strings of length < 15. Checksums produce stepped compressed values even if smooth values were expected for any string length. Notice the overlapping of values for short strings (low sensitivity). [Soler et al., Computability (2013)] Hector Zenil Applications of algorithmic complexity 12 / 35

13 The challenge of applicability of AIT Attempts to apply AIT faces 2 main challenges: 1 Uncomputability, and 2 Instability (dependency). Nevertheless: 1 K is semi-computable 2 We have the invariance theorem. K U1 (s) K U2 (s) < c U1,U 2 (1) Yet in practice the constant involved can be arbitrarily large and the invariance theorem tells nothing about the convergence. Hector Zenil Applications of algorithmic complexity 13 / 35

14 Rate of convergence of K and the behaviour of c with respect to s We can try to quantify c U1,U 2 for many U i s. No value can be guaranteed to be a bound, only statistical evidence. How do we expect it to behave? A natural universal Turing machine? Is there any condition on U i to achieve a well-behaved smooth behaviour of c in its convergence of values? Some have suggested natural programming languages, other have suggested very small Turing machines, both approaches seem to make arbitrary choices. We take the natural programming language approach and suggest that there is a common behaviour for c for natural choices of programming languages or universal Turing machines (we are formalizing this). Hector Zenil Applications of algorithmic complexity 14 / 35

15 Algorithmic Probability Definition Let U be a (prefix-free) universal Turing machine and p a program that produces s running on U, then m(s) = 1/2 p < 1 (2) p:u(p)=s (m is called Levin s semi-measure, or Levin s Universal Distribution). Connection to K! The greatest contributor in the def. of m(s) is the shortest program p, i.e. K(s). [Solomonoff (1964); Levin (1974)] Hector Zenil Applications of algorithmic complexity 15 / 35

16 Algorithmic Probability (modern version of the typing monkey theorem) [Inspired by a sketch by C. Bennett] Hector Zenil Applications of algorithmic complexity 16 / 35

17 The Coding theorem The algorithmic Coding theorem describes the reverse connection between K(s) and m(s): Theorem K(s) = log 2 (m(s)) + O(1) (3) Frequency and complexity are related If a string s is produced by many programs then there is also a short program that produces s (Thomas & Cover (1991)). [Levin (1974), Chaitin (1976)] Hector Zenil Applications of algorithmic complexity 17 / 35

18 Numerical approximation of m(s) Let (n, m) be the space of all n-state m-symbol Turing machines, with n, m > 1, we define a probability distribution: Definition D(n, m)(s) = the function that assigns to every finite binary string s the quotient: {T (n, m) : T produces s} 0 < < 1 (4) {T (n, m)} D(n, m)(s) complies with the subunarity condition as a semi-measure. Definition K m (s) = log 2 (D(n, m)(s)) (5) The more frequently a string is produced the lowest its Kolmogorov complexity, and the converse. [Delahaye & Zenil, App. Math and Computation (2011)] Hector Zenil Applications of algorithmic complexity 18 / 35

19 [Soler, Zenil et al, Computability (2013)] [Soler, Zenil et al, PLoS ONE (2014)] Hector Zenil Applications of algorithmic complexity 19 / 35 Coding Theorem method flow chart Enumerate & run every TM (n, m) for increasing n and m (Busy Beaver values to determine halting time, otherwise informed runtime cutoff value (see e.g. Calude & Stay, Most programs stop quickly or never halt, 2006).

20 Calculating D(n, m) Theorem For n, m, D(n, m) is uncomputable (by reduction to Rado s Busy Beaver problem). F means exhaustive calculation, F means exhaustive calculation with informed runtime cutoff and S means sample. D(n, m) m = n = 2 F 3 F 4 F, F 2D S S S S S 5 F, S 2D [Soler et al. PLoS ONE (2014)] [Zenil et al. Methods and Applications of K Complexity, Springer (forthcoming)] Hector Zenil Applications of algorithmic complexity 20 / 35

21 Complexity Tables Table : The 22 bit-strings in D(2, 2) from (2, 2)-Turing machines that halt (normalized by total number of machines that halt) (as a number, not as a fraction, values are lower bound of D for n, m ) Online Algorithmic Complexity Calculator: Hector Zenil Applications of algorithmic complexity 21 / 35

22 Output distributions Figure : Left: Halting probability of all Turing machines with 2 symbols and 4 states. Right: Output frequency distribution of all Turing machines with 2 symbols and 4 states. Hector Zenil Applications of algorithmic complexity 22 / 35

23 The Online Algorithmic Complexity Calculator (OACC) [ Hector Zenil Applications of algorithmic complexity 23 / 35

24 Resources needed for calculation of D Table : Letter code: F full space, S sample, R(n, m) reduced enumeration. Time is given in seconds (s), hours (h) and days (d). (n,m) Calculation Number of Machines Time (2,2) F (6 steps) R(2, 2) = s (3,2) F (21) R(3, 2) = s (4,2) F (107) R(4, 2) = h (4,2) 2D F 2D (1500) R(4, 2) 2D = d (4,4) S (2000) d (4,5) S (2000) d (4,6) S (2000) d (4,9) S (4000) d (4,10) S (4000) d (5,2) F (500) R(5, 2) = d (5,2) 2D S 2D (2000) d Calculated differences of D(s) for s occurring in each (n, m) space: estimation of the c in the invariance theorem looks smooth and decaying with respect to increasing n, m. [Soler et al. PLoS ONE (2014)] Hector Zenil Applications of algorithmic complexity 24 / 35

25 Complementary methods for different string lengths The methods to approximate K coexist and complement each other for different string lengths. short strings long strings scalability < 100 bits > 100 bits Lossless compression method Coding Theorem method (CTM) CTM + Block Decomposition method (BDM) Hector Zenil Applications of algorithmic complexity 25 / 35

26 2-dimensional Kolmogorov complexity and Graph Kolmogorov complexity What is the Kolmogorov complexity of an adjacency matrix? Figure : Two-dimensional Turing machines, also known as Turmites (Langton, Physica D, 1986), e.g. Langton s ant is a 2D universal TM (Gajardo et al. Discrete App. Math. 2002). [Zenil et al. Physica A (2014)] Hector Zenil Applications of algorithmic complexity 26 / 35

27 Block Decomposition method (BDM) The Block Decomposition method uses the Coding Theorem method. Formally, we will say that an object c has (2D) Kolmogorov complexity: K 2Dd d (c) = K 2D (r u ) + log 2 (n u ) (6) (r u,n u ) c d d where c d d represents the set with elements (r u, n u ), obtained from decomposing the object into (possibly overlapping) blocks of d d with boundary conditions. In each (r u, n u ) pair, r u is one of such squares and n u its multiplicity. [H. Zenil, F. Soler-Toscano, J.-P. Delahaye and N. Gauvrit, Two-Dimensional Kolmogorov Complexity and Validation of the Coding Theorem Method by Compressibility (2012)] Hector Zenil Applications of algorithmic complexity 27 / 35

28 Proof of concept: one-dimensional cellular automata An elementary cellular automaton (ECA) is defined by a local function f : {0, 1} 3 {0, 1}, Figure : Space-time evolution of a cellular automaton (ECA rule 30). f maps the state of a cell and its two immediate neighbours (range = 1) to a new cell state: f t : r 1, r 0, r +1 r 0. Cells are updated synchronously according to f over all cells in a row. Hector Zenil Applications of algorithmic complexity 28 / 35

29 Classification of ECA by BDM [H. Zenil, F. Soler-Toscano, J.-P. Delahaye and N. Gauvrit, Two-Dimensional Kolmogorov Complexity and Validation of the Coding Theorem Method by Compressibility (2012)] Hector Zenil Applications of algorithmic complexity 29 / 35

30 Network complexity calculation (BDM) Block Decomposition method (BDM) from 2D Turing machines Related to network motif analysis (subgraph decomposition). BDM is a finer grained measure than motif analysis alone: also sensitive to occurrence frequency and diversity but also internal complexity of the motifs. [Zenil et al. Physica A (2014)] Hector Zenil Applications of algorithmic complexity 30 / 35

31 K and graph isomorphisms Figure : Left: An adjacency matrix is not a graph invariant yet isomorphic graphs have similar K. Right: Graphs with large automorphism group size (group symmetry) have lower K (BDM c.f. BDM slides) (and are more likely to be produced by a random process due to their multiple symmetries). Let A(G) be the adjacency matrix of G and Aut(G) its automorphism group, then, K(G) = min{k(a(g)) A(G) A(Aut(G))}, where A(Aut(G)) is the set of adjacency matrices for each G Aut(G). (The problem is likely in NP but not in NP-complete). [Zenil et al. Physica A (2014)] Hector Zenil Applications of algorithmic complexity 31 / 35

32 Graph automorphisms and algorithmic complexity by BDM Classifying (and clustering) 250 graphs (no Aut(G) correction) with different topological properties by K (BDM): Hector Zenil Applications of algorithmic complexity 32 / 35

33 Testing BDM on dual and cospectral graphs [Source: Zenil et al. Physica A (2014), and Zenil, Kiani, Tegnér (submitted)] Hector Zenil Applications of algorithmic complexity 33 / 35

34 Why BDM? I want to do something a bit more clever than applying some other s lossless compression algorithm. I need better resolution than compression even if limited to small objects (small perturbation detection). There is a lot of room for improvements on BDM, specifically to equation 6, but 6 can be implemented as a lookup table and runs in O(n) time (same as compression). Another improvement (also with negative time complexity impact) is overlapping decomposition, that increases the resolution of BDM (forthcoming paper). Just as compression algorithms, CTM+BDM is proving to deliver results. The CTM+BDM as it is designed seems pragmatically different to compression. Compression finds long-range patterns (the longer a patterned string the better compression), while CTM+BDM focus is on small patterns (by limitation!). I use lossless compression too (and Block Entropy, etc), so I join efforts strengthening each other s method. Hector Zenil Applications of algorithmic complexity 34 / 35

35 Algorithmic probability One question is whether 2 non-isomorphic graphs can have exactly the same algorithmic information content: Answer is provided by algorithmic probability (AP) and the algorithmic Coding theorem. This is called The Universal Distribution. The more complex the (exponentially) less likely to happen (this is not true for traditional information theory!). Many simple networks have the same complexity, but very few complex networks will have the same complexity. [Source: Solomonoff (1964), Levin (1974), Chaitin (1975)] Hector Zenil Applications of algorithmic complexity 35 / 35

36 G. Terrazas, H. Zenil and N. Krasnogor, Exploring Programmable Self-Assembly in Non DNA-based Computing, Natural Computing, vol 12(4): , F. Soler-Toscano, H. Zenil, J.-P. Delahaye and N. Gauvrit, Correspondence and Independence of Numerical Evaluations of Algorithmic Information Measures, Computability, vol. 2, no. 2, pp , H. Zenil and E. Villarreal-Zapata, Asymptotic Behaviour and Ratios of Complexity in Cellular Automata Rule Spaces, International Journal of Bifurcation and Chaos, vol. 13, no. 9, H. Zenil, F. Soler-Toscano, K. Dingle and A. Louis, Correlation of Automorphism Group Size and Topological Properties with Program-size Complexity Evaluations of Graphs and Complex Networks, Physica A: Statistical Mechanics and its Applications, vol. 404, pp , J.-P. Delahaye and H. Zenil, Numerical Evaluation of the Complexity of Short Strings, Applied Mathematics and Computation, H. Zenil and J. Tegnér, Methods of information theory and algorithmic complexity for network biology. In M. Elloumi et al. (eds.), Pattern Recognition in Computational Molecular Biology, Wiley Book Series on Bioinformatics, John Wiley & Sons Ltd., (to appear) Hector Zenil Applications of algorithmic complexity 35 / 35

37 F. Soler-Toscano, H. Zenil, J.-P. Delahaye and N. Gauvrit, Calculating Kolmogorov Complexity from the Output Frequency Distributions of Small Turing Machines, PLoS ONE, 9(5): e96223, J.P. Delahaye and H. Zenil, On the Kolmogorov-Chaitin complexity for short sequences, in Cristian Calude (eds), Complexity and Randomness: From Leibniz to Chaitin, World Scientific, G.J. Chaitin A Theory of Program Size Formally Identical to Information Theory, J. Assoc. Comput. Mach. 22, , R. Cilibrasi and P. Vitányi, Clustering by compression, IEEE Trans. on Information Theory, 51(4), A.N. Kolmogorov, Three approaches to the quantitative definition of information Problems of Information and Transmission, 1(1):1 7, L. Levin, Laws of information conservation (non-growth) and aspects of the foundation of probability theory, Problems of Information Transmission, 10(3): , R.J. Solomonoff. A formal theory of inductive inference: Parts 1 and 2, Information and Control, 7:1 22 and , S. Wolfram, A New Kind of Science, Wolfram Media, Hector Zenil Applications of algorithmic complexity 35 / 35