Evolution of N-gram Frequencies under Duplication and Substitution Mutations

Transcription

1 208 IEEE International Symposim on Information Theory (ISIT) Evoltion of N-gram Freqencies nder Dplication and Sstittion Mtations Hao Lo Electrical and Compter Engineering University of Virginia Charlottesville, VA 22904, USA Moshe Schwartz Electrical and Compter Engineering Ben-Grion University of the Negev Beer Sheva 84050, Israel Farzad Farnod (Hassanzadeh) Electrical and Compter Engineering University of Virginia Charlottesville, VA 22904, USA Astract The driving force ehind the generation of iological seqences are genomic mtations that shape these seqences throghot their evoltionary history. An nderstanding of the statistical properties that reslt from mtation processes is of vale in a variety of tass related to iological seqence data, e.g., estimation of model parameters and compression. At the same time, de to the complexity of these processes, designing tractale stochastic models and analyzing them are challenging. In this paper, we stdy two types of mtations, tandem dplication and sstittion. These play a critical role in forming tandem repeat regions, which are common featres of the genome of many organisms. We provide a stochastic model and, via stochastic approximation, stdy the ehavior of the freqencies of N- grams in reslting seqences. Specifically, we show that N-gram freqencies converge almost srely to a set which we identify as a fnction of model parameters. From these freqencies, other statistics can e derived. In particlar, we present a method for finding pper onds on entropy. I. INTRODUCTION Genomic seqences are formed over illions of years y iological mtation processes, inclding insertions, deletions, dplications, and sstittions. These processes can e viewed as stochastic string editing operations that shape the statistical properties of seqence data. For any given set of sch processes and the associated proailities, however, it is difficlt to characterize the statistical properties that will arise. At the same time, nderstanding these properties is eneficial in oth analysis and storage of the vast amont of iological data that is availale nowadays. In this paper, or goal is to provide a etter nderstanding of the ehavior of tandem dplication and sstittion mtations y stdying the evoltion of N-gram freqencies (a..a. - mer freqencies) as sstrings of a seqence ndergoing these mtations. N-gram freqencies are of interest since they allow s to determine the sstrings that are liely to e generated nder different modeling assmptions. In addition, they can e sed to learn other properties of the seqence, e.g., onds on entropy, which provides a limit of compression. Given seqence data, they can also e sed to estimate model parameters, i.e., mtation rates. Dplication refers to copying a segment of the seqence (called the template) and inserting it into the seqence. The two main types of dplication are interspersed dplication and tandem dplication. In the former, there is generally no relationship etween where the template is located and where the copy is inserted. In the latter, which is the type of dplication stdied here, the copy is inserted immediately after the template. This process is generally thoght to e cased y slipped-strand mispairings [], where dring DNA synthesis, one strand in a DNA dplex ecomes misaligned with the other. A sstittion refers to changing a symol in the seqence. Tandem dplications and sstittions, along with other mtations, lead to tandem repeats, i.e., stretches of DNA in which the same pattern is repeated many times. Depending on the length of the pattern, these repeats are referred to as microsatellites or minisatellites. Tandem repeats are nown to case important phenomena sch as chromosome fragility [2]. In or model, a seqence evolves only throgh tandem dplications of different lengths and sstittions. In reality other mtations, sch as deletions, are also present in tandem repeat regions. However, for the sae of simplicity they are not inclded in or model. The analysis of more complete models is left to ftre wor. Frthermore, the term evoltion refers to changes reslting from random mtations. The significantly more complex effect of natral selection is not considered. Or stdy starts y considering how N-gram freqencies (nmer of occrrences divided y the length of the evolving string) change as a reslt of different mtations. To analyze sch freqencies, we se the stochastic approximation method, which enales modeling a discrete dynamic system y a corresponding continos model descried y an ordinary differential eqation (ODE We show that the reslting ODE is stale and prove that N-gram freqencies converge almost srely to a set determined y the ODE. Or approach allows s to compte the limit for the freqency of any N-gram as a fnction of model parameters. We will then se these reslts to provide onds on the entropy of seqences generated y the aforementioned mtation processes. In previos wor, the related prolem of finding the cominatorial capacity of tandem dplication systems has een stdied [3], [4]. Systems with oth tandem dplication and sstittion, again from a cominatorial point of view, were stdied in [5]. The stochastic approximation framewor has een sed for stdying interspersed dplications [6] and estimation of model parameters in tandem dplication systems [7]. Estimating the entropy of DNA seqences has een stdied in [8], [9]. However neither of these are ased on stochastic se /8/$ IEEE 2246

2 208 IEEE International Symposim on Information Theory (ISIT) qence evoltion models nor are they capale of characterizing the asymptotic freqencies of N-grams. The rest of the paper is organized as follows. Notation and preliminaries are given in the next section. In Section III, we derive the expected ehavior of the N-gram freqencies nder tandem dplication and sstittion. The proof of convergence and the limits are given in Section IV. Bonds on entropy are presented in Sections V. Section VI provides the conclding remars. De to space limitations, some of the proofs are omitted. II. PRELIMINARIES AND NOTATION For a positive integer m, let [m] = {,..., m} and Z m = {0,..., m }. For an alphaet Σ, the set of all finite strings over Σ is denoted Σ. The elements in strings are indexed starting from 0, e.g., s = s 0 s m, where s = m is the length of s. For 0 i, j m, s j i denotes s is i+ s j. A j-(s)string is a (s)string of length j. For, v Σ, their concatenation is denoted y v. For a positive integer j, j is a concatenation of j copies of, where is a string or a single symol. Note that the sperscript j in j i and j has different meanings. Consider an initial string s 0 and a process where in each step a random transform, or mtation, is applied to s n, reslting in s n+. To avoid the complications arising from ondaries, we assme the strings s n are circlar, with a given origin and direction. The indexing is modlo the length s n of s n. Or attention is focsed on tandem dplication and sstittion mtations. As an example of a tandem dplication, from s = we may otain s = , where the template, is overlined and the copy is nderlined. The length of a dplication is the length of the template (3 in the preceding example A sstittion changes one of the elements of s n to a different symol from the alphaet, e.g., We assign proaility q 0 to sstittions, where a position is chosen niformly at random and the crrent symol is changed with eqal proaility to one of the other alphaet symols. Frthermore, to a dplication of length we assign proaility q. The position of the template is chosen at random among the s n possile options. We assme there exists K sch that q = 0 for > K. Hence, K =0 q =. Stochastic Approximation: Let U denote the set of N- grams, that is, U = Σ N, and whenever an ordering of those strings is reqired we shall assme a lexicographic ordering. For U, let µ n denote the nmer of occrrences of sstring in s n, and µ n = (µ n) U. Let x n = µ n s, we n are interested in the asymptotic ehavior of x n = µ n s n = (x n) U, the vector of the freqencies of the sstrings. Let l n+ = s n+ s n and define E [ ] to e the expected vale conditioned on l n+ =. We also let {F n } e the filtration generated y the random variales {x n, s n }. Define δ (x) = δ (x n ) = E [µ n+ µ n F n ] h (x) = δ (x) x h(x) = q h (x () The vector δ represents the expected change in the vector of the freqencies of N-grams assming a sstittion ( = 0) or a dplication of length has occrred. Here, we have assmed that E [µ n+ µ n F n ] depends only on x n, and given x n, it is independent from µ n and n. The correctness of this assmption will e evident from Theorem 2. Becase of independence from n, we write δ (x) = δ (x n Frthermore, the element of δ that corresponds to is denoted y δ (x More precisely, δ (x n) = E [µ n+ µ n F n ]. This notation also extends to h. The vector h determines the overall expected ehavior of the system. If a certain set of conditions are satisfied, then Theorem elow can e sed to relate the discrete system whose expected ehavior is descried y h(x n ) to a continos system descried y an ordinary differential eqation (ODE The reqired conditions are similar to those in [6] and hold in or setp. An additional condition reqires n / s n = and n / s n 2 <, which can e proven sing the Borel- Cantelli lemma if q 0 <. Theorem. (See [0, Theorem 2]) The seqence {x n } converges almost srely to a compact connected internally chain transitive invariant set of the ODE dx t /dt = h(x t To find the aforementioned differential eqation, we need to find δ (x) for all with q > 0 and U. In finding δ (x), the following definitions will e sefl. For Σ and > 0, define φ () to e a vector of length whose ith element determines if the symol in position i of eqals the symol in position i. More specifically, the ith element of φ () is X i, i = 0,, 2,... φ () i = 0, i, i = i B, i, i i, where the X i and B are dmmy variales. Only the positions of 0 s in φ () are of importance to s. Let the lengths of the maximal rns of 0 s immediately after X and at the end of φ () e denoted y l and r, respectively. Note that either of these may e eqal to 0. If φ () = X0 0 N, then l = r = N. Otherwise, we have φ () = X0 0 l Y 0 r, for some Y that starts and ends with B. For example, for = 00000, we have φ 3 () = X0 2 00B0000, l = 2, and r = 4. To enale s to sccinctly represent the reslts, we define several fnctions. These fnctions relate to the freqencies of other sstrings that can generate via appropriate dplication events. First, for N, let G (x) = x z 0 N z+, z where the sm is over all z sch that (φ ()) z+ z = 0. For = 00000, G 3 (x) = 2x Frthermore, for > 0 and N +, let min(l, ) min(r, ) F,l(x) = x N i, F,r(x) = i= i= x N i 0, 2247

3 208 IEEE International Symposim on Information Theory (ISIT) M x 0, if φ () = X0 0 N (x) = =N + 0, else We se B () to denote set of strings at Hamming distance from. Also for, v Σ, the indicator fnction I(, v) eqals if = v and eqals 0 otherwise. III. EVOLUTION OF SUBSTRING FREQUENCIES In this section, we first find δ (x) = (δ (x)) U for > 0 (dplication) and then for = 0 (sstittion This will enale s to show that the sstring freqencies converge almost srely and find the limit set. We only consider sstrings of length N > for each since the freqencies of shorter sstrings can e otained from these. So for the whole model, we can only consider of length N > K. Theorem 2. For an integer > 0 and a string = 0 N, if + N < 2, then δ (x) = F,l(x) + F,r(x) + M (x) (N )x, and if N 2, δ (x) = F,l(x) + F,r(x) + G (x) (N )x. Before proving the theorem, we present two examples for = 3 and Σ = [3]: δ 23 3 (x) = x 23 + x 23 + x 32 δ (x) = 3x x x x x x Proof: Sppose a dplication of length occrs in s n, reslting in s n+. The nmer of occrrences of may change de to the dplication event. To stdy this change, we consider the N-sstrings of s n that are eliminated (do not exist in s n+ ) and the N-sstrings of s n+ that are new (do not exist in s n Any new N-sstring mst intersects with oth the template and the copy in s n+. Liewise, an eliminated N-sstring mst inclde symols on oth sides of the template in s n, i.e., the template mst e a strict sstring of the N-sstring that incldes neither its leftmost symol nor its rightmost symol. As an example, sppose s n = v w, s n+ = v w, where = 3, the (new) copy is nderlined, and v, w. Let N = 5. The new 5-sstrings are 34564, 45645, 56456, and the eliminated sstring is Formally, let s n = a 0 a i a i+ a i+ a i++ a sn, s n+ = a 0 a i a i+... a i+ a i+... a i+ a i++... a sn, where the sstring a i+ a i+ is dplicated. The new N- sstrings created in s n+ are y = a i++ a i a i+ a i+ a i+2... a i+n, for N. Note that we have defined sch that the first element of the copy, a i+, is at position in y. The Case 3 Case 2 Case s n+ Case Case 2 Case 3 Figre. Possile cases for new occrrences of in s n+. Cases aove and elow s n+ correspond to + N < 2 and N 2, respectively. The hatched oxes, from left to right, are the template and the copy. N-sstrings eliminated from s n are a i c+ a i+n c, for c N. For a given, let Y denote the indicator random variale for the event that y =, that is, the dplication creates a new occrrence of in s n+ in which the first symol of the copy is in position. In the example aove, if = 45645, then y 3 = and ths Y 3 =. Frthermore, let W denotes the nmer of occrrences of that are eliminated. We have δ (x) = N = N E [Y F n ] E [W F n ] = E [Y F n ] (N )x, = where the second eqality follows from the fact that each of the N eliminated N-sstrings are eqal to with proaility x. To find δ, it sffices to find E [Y F n ], or eqivalently, Pr(Y = F n, l = We consider different cases ased on the vale of, which determines how overlaps with the template and the copy. These cases are illstrated in Figre and are considered in Lemmas 3 5. Smming over the expressions provided y these lemmas provides the desired reslt. We omit the details, as well as the proofs of Lemmas 4 and 5 de to similarity to that of Lemma 3. Lemma 3 (Case For < min(, N + ), E [Y F n ] = x N I( 0, + Proof: For < min(, N +) (regardless of N 2 or N < 2), the new occrrences of always contains some (t not all) of the template and all of the new copy. This scenario is laeled as Case in Figre. Sppose Y =. Since the copy and the template are identical, elements of that coincide with the same positions in these two sstrings mst also e identical. So a necessary condition for Y = is 0 = +. Assme this condition is satisfied. Then Y = if and only if the seqence starting at the eginning of the template in s n is eqal to N, which has proaility x N. 2248

4 208 IEEE International Symposim on Information Theory (ISIT) Lemma 4 (Case 2 Sppose min(, N + ) < max(n +, If N 2, then E [Y F n ] = x 0 N and if + N 2 2, then I(, + ), E [Y F n ] = x 0 I( N 0, N Note that this case cannot occr if N = 2. Lemma 5 (Case 3 For max (N +, ) N, E [Y F n ] = x 0 I( N, N We now trn or attention to sstittions ( = 0 Theorem 6. For a string of length N, we have δ0 = x v Nx. Σ v B () Before proving the theorem, we give an example for Σ = {, 2, 3}: δ 23 0 = 2 (x223 + x x 3 + x 33 + x 2 + x 22 ) 3x 23 Proof: A new occrrence of reslts from an appropriate sstittion in some v B (), which has proaility x v /( Σ On the other hand, an occrrence of is eliminated if a sstittion occrs in any of its N positions. So the expected nmer occrrences that vanish is Nx. IV. ODE AND THE LIMITS OF SUBSTRING FREQUENCIES Theorems 2 and 6 provide expressions for δ for 0 K. With these reslts in hand, we can formlate an ordinary differential eqation (ODE) whose limits are the same as those of the sstring freqencies of interest, x = (x ) U, where U is the set of strings of length N +. The ODE is of the form dxt dt = Ax t, where A is determined sing Theorems 2 and 6 as descried in ( On the right side in expressions for δ, terms of the form x v appear where v / U. However, we have v < N, so we replace x v with w xw, where the smmation is over all strings w of length N sch that v is a prefix of w. For example, consider q 0 = α, q = α, and Σ = {0, }. From Theorems 2 and 6, for x = (x 00, x 0, x 0, x ), we have dx/dt = Ax, where A = 2α α 0 α ( + α) 0 α α 0 ( + α) α 0 α 2α. (2) Theorem 7. Consider a tandem dplication and sstittion system with distrition q = (q ) over these mtations sch that q = 0 for > K and q 0 <. The freqencies of sstrings of length N K + converges almost srely to the nll space of the matrix A, descried aove. Proof: We first show that the reslting ODE is stale. This is done y applying the Gershgorin circle theorem to the colmns of A (see e.g., (2) In each colmn, the diagonal element is the only element that can e negative. We show that each colmn of A sms to 0, which implies that the rightmost point of each circle is the origin. Ths, each eigenvale of A is either 0 or has a negative real part. Define A to e the matrix satisfying h (x) = A x so that h(x) = Ax = q h (x) = q A x. For the example of (2), the matrices A 0 and A are A 0 = , A = We show that each colmn of A sms to zero for each, which implies the desired reslt. Fix v U and consider the colmn in A that corresponds to x v. To identify the elements in this colmn, we mst consider expressions for h (x) = δ (x) x and chec if xv appears on the right side. For > 0, the only negative term corresponds to h v, where the coefficient is (N Inspecting the proofs of Lemmas 3 5 shows that for each vale of [N ], there is only one sch that x v appears in h with a nonnegative coefficient, and the coefficient is. For example, for =, from Lemma 3, this is eqal to v v N. Since there are N possile choices for, the sm of every colmn in A is 0, as desired. For = 0, we have h (x) = δ (x), where δ (x) is given in Theorem 6. The colmn corresponding to x v has a negative term eqal to N and N( Σ ) positive terms, where each of the positive terms is eqal to Σ, so the sm is again 0. We have shown that all eigenvales are either 0 or have negative real parts. For any valid initial point x 0, the sm of the elements mst e. Frthermore, each element mst e nonnegative. The fact that the colmns of A sm to 0 shows that the sm of the elements of any soltion x t also eqals. Frthermore, since only diagonal terms in A can e negative, each element of x t is also nonnegative. Ths x t is onded. From the Jordan canonical form theorem, we can write A = P JP, for an invertile matrix of generalized eigenvectors P. Let y t = P x t, let C e any compact internally chain transitive set of the ODE ẏ t = Jy t. From the ondedness of x t, and ths y t, and the fact that all eigenvales of A except for 0 have negative real parts, we can prove that all flows initiated in C are constant. The same mst hold for all flows in D, for any D that is an internally chain transitive invariant set of the ODE ẋ t = Ax t. Hence, any point in x D mst e in the nll space of A, that is, Ax = 0. For example, for the matrix A of (2), the vector in the nll space whose elements sm to, and ths the limit of x n, is 2( + 3α) (α +, 2α, 2α, α + )T. (3) As α, all for 2-sstrings ecome eqally liely, each with proaility /4. Note however that or analysis is not applicale to q 0 = α = since the condition n / s n 2 < is not satisfied. On the other hand, for a small proaility of sstittion, 0 < α, almost all 2-sstrings are either 2249

5 208 IEEE International Symposim on Information Theory (ISIT) 00 or, as expected. For α = 0, the nll space is spanned y z = (, 0, 0, 0) T and z 2 = (0, 0, 0, ) T and the limit set is {az + ( a)z 2 : 0 a }. V. BOUNDS ON ENTROPY The entropy of this process may e pper onded sing techniqes from semiconstrained systems [] [3]. We first formally define the entropy, and then arge that it is pper onded y an appropriately defined semiconstrained system. Consider the string s n, otained from s 0 y n ronds of mtations, as descried previosly. Its length is s n = s 0 + n i= l i, and its expected length is E[ s n ] = s 0 + n K i= iq i. We define the entropy after n ronds as H n = Pr(s n = w) log E[ s n ] Σ Pr(s n = w), w Σ as well as H = lim sp n H n. Let s recall some definitions concerning semiconstrained systems (see [3] Let P(Σ N ) denote the set of all proaility measres on Σ N. A semiconstrained system is defined y Γ P(Σ N The set of admissile words of the semiconstrained system, denoted B(Γ), contains exactly all finite words over the alphaet Σ, whose N-gram distrition is in Γ, and B l (Γ) = B(Γ) Σ l. An expansion of Γ y ɛ > 0 is defined as B ɛ (Γ) = { ξ P(Σ N ) : } inf ν ξ TV ɛ, ν Γ where TV denotes the total-variation norm. The capacity of Γ is then defined as cap(γ) = lim lim sp ɛ 0 + n n log Σ B n (B ɛ (Γ)), which intitively measres the information per symol in strings whose N-gram distrition is almost in Γ. Theorem 8. For the mtation process descried aove, H cap(γ), where Γ is the compact connected internally chain transitive invariant set that x n converges almost srely to y Theorem. Remar 9. We comment that if Γ = {ξ}, i.e., Γ contains a single shift-invariant measre, then cap(γ) has a nice form (see [], [3]): cap(γ) = a...a N Σ N ξ a...an log Σ a ξ...a N ξ a...a, N where ξ is the marginal of ξ on the first N coordinates, i.e., ξ a...a N = Σ ξa...a N. We se the preceding remar to find an pper ond on the system whose limit is given y (3 We have ( ξ 0 = ξ ) = /2. 2α It then follows that for this system H H 2 +3α, where H 2 is the inary entropy fnction. A shift-invariant measre ξ P(Σ N ) is a measre that satisfies a Σ ξaw = a Σ ξwa for all w Σ N. The N-gram distritions of cyclic strings are always shift invariant, and ths a converging seqence of sch measres also converges to a shift-invariant measre. VI. CONCLUSION In this paper, we provided a method for determining the limits of N-gram freqencies (as sstrings of the evolving seqence) for tandem dplications and sstittions. We also presented a method for finding pper onds on the entropy of these systems. One direction for extending the crrent wor is inclding other mtation types that are present in tandem repeat regions sch as deletions and insertions. Open prolems inclde qantifying the finite-time ehavior in these systems, determining the rate of convergence to the limits, and lower onds on entropy. REFERENCES [] N. Mndy and A. J. Helig, Origin and Evoltion of Tandem Repeats in the Mitochondrial DNA Control Region of Shries (Lanis spp.), Jornal of Moleclar Evoltion, vol. 59, no. 2, pp , [2] K. Usdin, The iological effects of simple tandem repeats: Lessons from the repeat expansion diseases, Genome Research, vol. 8, no. 7, pp. 0 09, Jl [3] F. Farnod, M. Schwartz, and J. Brc, The Capacity of String-Dplication Systems, IEEE Trans. Information Theory, vol. 62, no. 2, pp , Fe [4] S. Jain, F. Farnod, and J. Brc, Capacity and Expressiveness of Genomic Tandem Dplication, IEEE Trans. Information Theory, vol. 63, no. 0, Oct [5] S. Jain, F. Farnod, M. Schwartz, and J. Brc, Noise and Uncertainty in String-Dplication Systems, in IEEE Int. Symp. Information Theory (ISIT), Aachen, Germany, Jn [6] F. Farnod, M. Schwartz, and J. Brc, A stochastic model for genomic interspersed dplication, in IEEE Int. Symp. Information Theory (ISIT), Jn. 205, pp [7], Estimating Mtation Rates nder a Stochastic Model for Tandem Dplication and Sstittion, in In Preparation, smtd.pdf. [8] A. O. SCHMITT and H. HERZEL, Estimating the Entropy of DNA Seqence, Jornal of Theoretical Biology, vol. 88, no. 3, pp , Oct [9] D. Loewenstern and P. N. Yianilos, Significantly Lower Entropy Estimates for Natral DNA Seqences, Jornal of Comptational Biology, vol. 6, no., pp , 999. [0] V. S. Borar, Stochastic approximation, Camridge Boos, [] O. Elishco, T. Meyerovitch, and M. Schwartz, Semiconstrained systems, IEEE Transactions on Information Theory, vol. 62, no. 4, pp , 206. [2], On encoding semiconstrained systems, IEEE Transactions on Information Theory, 207, To appear. [3], On independence and capacity of mltidimensional semiconstrained systems, arxiv preprint arxiv: ,