16 Modeling a Language by a Markov Process

Transcription

1 K. Pommeening, Language Statistics Modeling a Language by a Makov Pocess Fo deiving theoetical esults a common model of language is the intepetation of texts as esults of Makov pocesses. This model was intoduced by Shannon in his fundamental papes published afte Wold Wa II. If we look at lette fequencies only, we define a Makov pocess of ode 0. If we also incopoate bigam fequencies into ou model, it becomes a Makov pocess of ode 1, if we include tigam fequencies, of ode 2, and so on. In this section we want to deive theoetical expectation values fo κ, ϕ, and χ. Fo this the ode of the Makov model is ielevant. Message Souces Let the alphabet Σ be equipped with a pobability distibution, assigning the pobability p s to the lette s Σ. In paticula p s = 1. We call (Σ,p) amessage souce and conside andom vaiables X in Σ, that is mappings X: Ω Σ whee Ω is a finite pobability space with pobability measue P, such that P (X 1 s)=p s fo all s Σ. Picking a lette of Σ at andom fom the message souce is modeled as evaluating X(ω) fo some ω Ω. We calculate the expectation values of the Konecke symbols fo andom vaiables X, Y: Ω Σ and lettes s Σ whee Y may belong to a message souce (Σ,q) with a possibly diffeent pobability distibution q =(q s ) : δ sx (ω) = 1 if X(ω) =s 0 othewise δ XY (ω) = 1 if X(ω) =Y (ω) 0 othewise Lemma 4 (i) E(δ sx )=p s fo all s Σ. (ii) If X and Y ae independent, then E(δ XY )= p sq s. (ii) If X and Y ae independent, then δ sx and δ sy ae independent. Poof. (i) Since δ takes only the values 0 and 1, we have E(δ sx )=1 P (X 1 s)+0 P (Ω X 1 s)=p (X 1 s)=p s. (ii) In the same way, using the independence of X and Y, E(δ X,Y ) = 1 P (ω X(ω) =Y (ω)) + 0 P (ω X(ω) = Y (ω)) = P (X = Y )= P (X 1 s Y 1 s) = P (X 1 s) P (Y 1 s)= p s q s

2 K. Pommeening, Language Statistics 81 (iii) δ 1 sx (1) = {ω X(ω) =s} = X 1 s, and δ 1 sx (0) = Ω X 1 s. The same fo Y. The assetion follows because P (X 1 s Y 1 s)=p (X 1 s) P (Y 1 s). Picking a andom text of length is modeled by evaluating an -tuple of andom vaiables at some ω. This leads to the following definition: Definition. A message of length fom the message souce (Σ,p)is a sequence X =(X 1,...,X ) of andom vaiables X 1,...,X :Ω Σ such that P (X 1 i s) =p s fo all i =1,..., and all s Σ. Note. In paticula the X i ae identically distibuted. They ae not necessaily independent. The Coincidence Index of Message Souces Definition. Let Y =(Y 1,...,Y ) be anothe message of length fom a possibly diffeent message souce (Σ,q). Then the coincidence index of X and Y is the andom vaiable defined by K XY :Ω R K XY (ω) := 1 #{i =1,..., X i(ω) =Y i (ω)} = 1 δ Xi (ω),y i (ω) We calculate its expectation unde the assumption that each pai of X i and Y i is independent. Fom Lemma 4, using the additivity of E, we get E(K XY )= 1 E(δ Xi,Y i )= 1 p s q s = p s q s independently of the length. Theefoe it is adequate to call this expectation the coincidence index κ LM of the two message souces L, M. We have poven: Theoem 2 The coincidence index of two message souces L =(Σ,p) and M =(Σ,q) is κ LM = p s q s Now we ae eady to calculate theoetical values fo the typical coincidence indices of languages unde the assumption that the model message souce fits thei eal behaviou:

3 K. Pommeening, Language Statistics 82 Example 1, andom texts vesus any language M: Hee all p s = 1/n, theefoe κ Σ = n 1/n q s =1/n. Example 2, English texts vesus English: Fom Table 39 we get the value Example 3, Geman texts vesus Geman: The table gives Example 4, English vesus Geman: The table gives Note that these theoetical values fo the eal languages diffe slightly fom the fome empiical values. This is due to two facts: The model as evey mathematical model is an appoximation to the tuth. The empiical values undely statistical vaiations and depend on the kind of texts that wee evaluated. The Coss-Poduct Sum of Message Souces Fo a message X =(X 1,...,X ) fom a message souce (Σ,p)wedefinethe (elative) lette fequencies as andom vaiables o moe explicitly, M sx :Ω R, M sx = 1 δ sxi, M sx (ω) = 1 #{i X i(ω) =s} fo all ω Ω. We immediately get the expectation E(M sx )= 1 E(δ sxi )=p s. Definition. Let X =(X 1,...,X ) be a message fom the souce (Σ,p), and Y =(Y 1,...,Y t ), a message fom the souce (Σ,q). Then the cosspoduct sum of X and Y is the andom vaiable X XY :Ω R, X XY := 1 t M sx M sy.

4 K. Pommeening, Language Statistics 83 Table 39: Calculating theoetical values fo coincidence indices Lette s English Geman Squae Squae Poduct p s q s p 2 s qs 2 p s q s A B C D E F G H I J K L M N O P Q R S T U V W X Y Z Sum

5 K. Pommeening, Language Statistics 84 To calculate its expectation we assume that each X i is independent of all Y j, and each Y j is independent of all X i. Unde this assumption let us call the messages X and Y independent. Then fom Lemma 4 and the fomula we get X XY := 1 t E(X XY )= 1 t j=1 t t δ sxi δ syj j=1 E(δ sxi )E(δ syj )= p s q s again independently of the length. Theefoe we call this expectation the coss-poduct sum χ LM of the two message souces L, M. Wehave poven: Theoem 3 The coss-poduct sum of two message souces L =(Σ,p) and M =(Σ,q) is χ LM = p s q s. The Inne Coincidence Index of a Message Souce Let X =(X 1,...,X ) be a message fom a souce (Σ,p). In analogy with Sections 10 and 14 we define the andom vaiables by the fomulas Ψ X := Ψ X, Φ X :Ω R M 2 sx, Φ X := 1 Ψ x 1 1. We ty to calculate the expectation of Ψ X fist: Ψ X = δ sxi = 1 2 δ sxi + δ sxi δ sxj j=i since δ 2 sx i = δ sxi. Taking the expectation value we obseve that fo a sensible esult we need the assumption that X i and X j ae independent fo i = j. In the language of Makov chains this means that we assume a Makov chain of ode 0: The single lettes of the messages fom the souce ae independent fom each othe.

6 K. Pommeening, Language Statistics 85 Unde this assumption we get E(Ψ X ) = 1 2 p s + E(δ sxi )E(δ sxj ) j=i = 1 2 p s + p 2 s 1 j=i 1 ( 1) = p 2 s. Fo Φ X the fomula becomes a bit moe elegant: E(Φ X )= 1 1 p 2 s = p 2 s. Let us call this expectation E(Φ X )the(inne) coincidence index of the message souce (Σ,p), and let us call (by abuse of language) the message souce of ode 0 if its output messages ae Makov chains of ode 0 only. (Note that fo a mathematically coect definition we should have included the tansition pobabilities into ou definition of message souce.) Then we have poved Theoem 4 The coincidence index of a message souce L =(Σ,p) of ode 0 is ϕ L = p 2 s. The assumption of ode 0 is elevant fo small text lengths and negligeable fo lage texts, because fo natual languages dependencies between lettes affect small distances only. Reconsideing the tables in Section 11 we note in fact that the values fo texts of lengths 100 coespond to the theoetical values, wheeas fo texts of lengths 26 the values ae suspiciously smalle. An explanation could be that epeated lettes, such as ee, oo,, ae elatively ae and contibute pooly to the numbe of coincidences. This affects the powe of the ϕ-test in an unfiendly way. On the othe hand consideing Sinkov s test fo the peiod in Section 13 we note that the columns of a polyalphabetic ciphetext ae decimated excepts fom natual texts whee the dependencies between lettes ae ielevant: The assumption of ode 0 is justified fo Sinkov s test.