The distribution of characters, bi- and trigrams in the Uppsala 70 million words Swedish newspaper corpus

Size: px

Start display at page:

Download "The distribution of characters, bi- and trigrams in the Uppsala 70 million words Swedish newspaper corpus"

Gordon Allison
5 years ago
Views:

1 Uppsala University Department of Linguistics The distribution of characters, bi- and trigrams in the Uppsala 70 million words Swedish newspaper corpus Bengt Dahlqvist Abstract The paper describes some of the characteristics of words and single character distribution as well as the distribution of bi- and trigrams in the Uppsala Newspaper Corpus. This corpus consists of more that 70 million words collected from all published articles in two major Swedish daily newspapers between the years 1995 and 1996.

2 1 The text material The Uppsala newspaper corpus consists of somewhat more than 70 million words. It was collected from all the published articles in the Swedsish newspapers Svenska Dagbladet (SvD) and Upsala Nya Tidning (UNT) between the years 1995 and In total it comprises over single articles. Articles Tokens Types Characters SvD UNT Table 1. frequency information for the texts in the newspaper corpus. The text material was delivered from the papers in the form of ANSI coded textfiles, including specific format and other information codes used by the papers. After the extraction of the plain text in this material, the files contained texts as shown in table 1 above. A token is defined as any character string delimited by whitespace (spaces etc.). The types are the unique tokens in the corpus. ANSI CHAR FREQUENCY PERCENT e a r n t s i l d o m k g v f h ä p u å ö b c , j ^ y x " : w ) ( ' z ? é / q ! < > ; & ] [ Table 2. The 60 most frequent characters in the corpus, sorted by descending frequency. Listed in table 2 above are the 60 most frequent characters in the corpus. This table differs from the corresponding table in [2] by the ordering and the symbol representation, which here includes only lower case letters (i.e. upper case letters are counted together with their lower case equivalents). Thus, the line for character e represents both the character e (occurring 2

3 with a frequency of , ANSI-code 100) and the character E (frequency , ANSI-code 69). The table demonstrates that the single most frequent character in the corpus is the space character, which comprises per cent of the corpus text. The most frequent letter is e, comprising 7.81 per cent of the corpus. A more detailed description of the text material and the text extracting methods employed while constructing the corpus is to be found in [2]. There is presented a list of all the 256 ANSI characters in the corpus, sorted by code value and occurrence. In total, 191 different ANSI characters are present in the corpus with a total frequency of , which is the number of the characters in the corpus as a whole. 2 The sets of alfabetic and non-alfabetic characters Alfabetic symbols, letters and delimiters in the corpusen are defined as follows: letters (56) = { abcdefghijklmnopqrstuvwxyzåäöàáâãæçèéêëìíîïñòóôõøùúûüýÿß } delimiters (14) = space, CR, formfeed, tab and the set {.,?!"()/_= } Of course, the letters also include the upper case ones. So, 53 upper case letters are present in the corpus ( ýÿß are only to be found in the lower case). All the other characters aside from these two sets (191 2x56 14 = 61) are symbols of the type +, -, : etc. See [2] for a full description of the occurrence of these characters. a 7.42 b 1.16 c 1.02 d 3.42 e 7.81 f 1.61 g 2.52 h 1.58 i 4.51 j 0.57 k 2.58 l 4.16 m 2.72 n 6.87 o 3.40 p 1.49 q 0.01 r 6.90 s 5.25 t 6.81 u 1.43 v 1.80 w 0.06 x 0.09 y 0.50 z 0.03 å 1.27 ä 1.55 ö 1.18 à 0.00 (1621) á 0.00 (3018) â 0.00 (259) ã 0.00 (81) æ 0.00 (1145) ç 0.00 (1190) è 0.00 (3982) é 0.02 ê 0.00 (366) ë 0.00 (863) ì 0.00 (30) í 0.00 (668) î 0.00 (159) ï 0.00 (115) ñ 0.00 (189) ò 0.00 (70) ó 0.00 (1360) ô 0.00 (475) õ 0.00 (61) ø 0.00 (2014) ù 0.00 (35) ú 0.00 (219) û 0.00 (55) ü 0.00 (10641) ý 0.00 (59) ÿ 0.00 (27) ß 0.00 (30) Table 3. Distribution of letters (percentage) in the corpus. Frequencies for low frequency characters are given in parenthesis. In table 3 part of the information in table 2 is repeated, this time given only for letters and listed in alphabetic order. The letters stated represent both the upper case and lower case occurrences. Percentages rounded to 0.00 (two significant decimals) are pair with the 3

4 corresponding frequency in parenthesis. For example, the low occurrence of the letter ü equals with five significant decimals * 100 / = Probabilities In terms of probability, statements like the following can be made: selecting, with uniform probability, a character in the corpus, the probability of getting the letter e (or E ) is This can be deduced from table 3. The probability of getting any other character is then of course Also, the probability of getting the space character is Further, one can divide the set of letter into consonants: { bcdfghjklmnpqrstvwxzçñß }, and vowels: { ieäyöuaåoâîûêôàáãæèéëìíïòóõøùúüýÿ }. The probability to find at random a vowel is then P(vowel) = P(i)+P(e) = and finding a consonant P(consonant) = P(b)+P(c) = also, for all other chars More, if one assumes that the character randomly drawn is a letter, the probability of getting a vowel is expressed as: P (vowel) = P(vowel letter) = And, in the same way, for a consonant: P (consonant) = P(consonant letter) = The two probabilities above are examples of a conditional probability. It further shows that the sum of these probabilities still becomes one: P (vowel) + P (consonant) = P(vowel letter) + P(consonant letter) = 1 To postulate the letter property of the characters to be studied (i.e. only regard this subset) is a natural assumption when one wants to work with well-formed words only, suitable for lexicon work. Further, one can look at the bigrams and compute the conditional probability that a consonant follows after a vowel (e.g. the bigram ab ) in a word. This can be expressed as the probability P(consonant vowel) = In this way all probabilities for any combination of vowels/consonants can be computed. See the table below for an overview.. Vowel follows Consonant follows Vowel first % % Consonant first % % Table 4. Conditional probabilities for pair of vowels/consonants in words. 4 Bi- and trigrams In many cases, it is desirable to study the occurrence of tuplets and triplets of characters belonging to the letter set, {abcdefghijklmnopqrstuvwxyzåäöàáâãæçèéêëìíîïñòóôõøùúûüýÿß}. Such n-tuples are usually denoted bi- and trigrams, respectively. Exemples of such are ll, ck, abc and att. 4

5 Before starting an analysis of the bi- and trigrams in our newspaper corpus, we make the same restrictions regarding the tokens as in the previous paper [2]. That is, excluded from the set of tokens to be studied are strings consisting only of 1) numerals, 2) numerals in combinations with non-letters eller 3) non-letters proper. Remaining for further study are the tokens described in [2] as belonging to the data file fx.fre (containing frequency sorted, and in the sense described earlier, only well-formed words), in total tokens and types. Regarding all the tokens in the data file fx.fre as our fundamental data source, some basic facts can be established. The corpus then gives a total of bigrams, of them unique and with a total frequency of In the same manner we have trigrams, unique and a total frequency of All this regarding only transitions from letter to letter or word ending (i.e. multiple whitespaces not counted). 4.1 Bigrams In table 5 below, the 30 most frequent bigrams (from a letter to another letter) from the corpus are listed. The first column lists the bigrams themselves, followed by a frequency column, and a column (%tot) stating the percentage of occurrence based on the proportion of the bigram over all the tokens (as defined in the data file fx.fre). Finally, a column (%letter) giving the percentage regarding only transitions between letters is included. This figure gives a more resonable estimate of the expectancy for a specific character when regarding well-formed words. Expressed in statistical terms, every bigram can be regarded as occurring with a conditional probability. For the bigram en this means that the probability for an e to be followed by a n in a word equals 0.281, i.e. P(n e) = If the possibility for the following character to be a word delimiter is also include, i.e. a word ending, the bigram can be a e (the letter e followed by a space) the probability then will be modified and expressed as P (n e) = Bigram Frequency %tot %letter en er de ar an et in st te tt at ra ll om re ti ör ta nd ng na la ka sk

6 fö oc ge är me li Table 5. The 30 most frequent bigrams. 4.2 Trigrams In the same way as table 5 shows the frequency (absolute and in percent) for the bigram occurences in the corpus, table 6 shows this for the trigrams. A trigram is defined as a threeletter character sequence. Here, as before for the bigrams, the first numeric column in the table shows the absolute frequency, followed by the same frequency in percent, named %tot, over the whole set of characters in the corpus and finally in percentage of letters and word endings only, denoted %letter. The most frequent trigram is för with the occurrence 1.05%. The conditional probability of getting a r given that one already has the sequence fö and still expects a letter is then, as seen in the table, P(r fö) = Trigram Frequency %tot %letter för att ing och ter det and nde som den ill gen ska ade til med nin var rna der lig sta nte era ver nge are ett int han Table 6. The 30 most frequent trigrams. 6

7 5 Further data Some more interesting figures from the corpus are to be found in the appendix. In table A1 is shown the distribution of letters for word start and endings. Table A2 shows the transition probability in percentage between different letters. Table A3 and A4 show the bigram word beginnings and endings in a way analog with the previous tables 5 and 6. Table A5 and A6 show the distribution and some statistics for the word lengths in the corpus. Again, this is displayed in a graphical form in fig. A1. 6 Conclusion This paper has in a descriptive manner presented a number of fundamental characteristics of the newspaper corpus from the lowest level, involving the distribution of single characters, bigrams, trigrams and single words. Further, some things have been said about probabilities and conditional such to be used for computing expectancies of character sequences. While it is of value as such to have a thorough description of a corpus as such, this knowledge has a number of useful applications. For example one can mention proof-reading, where the knowledge of bi- and trigram distribution can be utilised for making spelling corrections. Also, the transition probabilities between letters can used for text generation, using stochastic models of different orders. Another classic example for the use of bi- and trigram probabilities is in cryptanalysis to solve ciphers. 7

8 7 References 1. Dahlqvist, B, Word Frequency Lists for the Uppsala Newspaper Corpus. Collection of Swedish word lists from the 70 million word Uppsala Newspaper Corpus, Dept. of Linguistics, Uppsala University, Dahlqvist, Bengt, A Swedish Text Corpus for Generating Dictionaries, Project Report 3.1.3, EC-project Scarrie, Gaines, H. F., Cryptanalysis, a Study of Ciphers and Their Solutions, Dover Publications, ISBN , Oakes, Michael P., Statistics for Corpus Linguistics, Edinburgh University Press, ISBN ,

9 Appendix %First %Last 6.97 a b c d e f g h i j k l m n o p q r s t u v w x y z å ä ö 0.08 Table A1. First and last letter for words, distribution in percent. 9

10 E 1 I 1 F 2 J 2 B 2 P 3 S 5 M 6 D 6 V 6 G 6 H 7 K 10 L 10 N 10 T 10 R 12 A R 23 N 21 T 14 L 8 D 7 G 5 S 5 V 5 M 5 K 2 P 1 F 1 B 1 C 1 Å 1 Ö 1 V 1 Y 1 K 1 G 2 D 3 I 3 N 5 T 5 L 5 U 5 M 7 B 7 S 8 O 8 E 9 A 9 R 19 B E 26 A 14 L 13 O 10 R 9 I 8 Y 4 U 4 Ö 3 B 3 Ä 3 Å 2 J 1 S 1 C 1 R 1 T 1 U 1 N 2 Ä 2 S 4 A 5 E 6 Y 7 I 9 O 60 C H 48 K 29 E 9 I 5 A 3 O 2 L 1 U 1 C 1 Y 1 G 1 V 1 Y 1 S 1 Ä 1 Ö 2 D 2 U 2 Å 2 O 3 L 5 R 7 I 9 A 16 E 17 N 27 D E 54 A 13 R 5 I 5 S 4 O 3 L 2 Ä 2 D 2 N 2 U 1 Å 1 Ö 1 J 1 V 1 G 1 B 1 F 1 J 1 C 1 I 2 P 3 H 3 K 4 B 4 N 4 V 5 S 5 L 6 M 7 G 8 R 10 T 13 D 22 E N 27 R 25 T 16 L 7 D 6 S 5 M 3 K 2 F 2 G 2 V 1 C 1 X 1 B 1 P 1 U 1 V 1 Ö 1 G 1 K 1 Y 1 P 1 U 1 Ä 2 D 2 T 4 M 4 L 5 S 6 R 7 N 8 O 8 F 8 I 9 A 11 E 18 F Ö 35 R 13 A 8 T 8 I 8 O 7 L 4 Å 4 E 4 F 3 Ä 2 U 2 Y 1 S 1 P 1 L 1 D 1 T 1 U 1 Ö 1 Y 2 G 3 O 3 E 5 Å 5 R 5 Ä 5 A 15 I 21 N 30 G E 27 A 19 R 7 S 7 T 7 Å 4 I 4 O 4 G 3 N 3 Ö 2 H 2 L 2 Ä 2 U 2 D 1 J 1 Y 1 D 1 I 1 P 1 L 1 A 2 O 2 M 3 N 3 K 4 E 4 S 4 R 4 G 5 T 5 C 58 H A 37 E 20 O 11 U 6 Ä 6 Ö 6 Å 4 I 4 J 1 L 1 Y 1 R 1 N 1 E 1 P 1 H 1 C 1 K 2 B 2 G 2 F 3 D 4 M 6 S 8 V 9 N 10 R 14 L 15 T 19 I N 27 L 12 G 12 S 10 T 7 K 6 D 5 O 4 V 4 E 3 C 2 A 2 R 2 F 1 M 1 P 1 P 1 I 1 V 1 M 1 K 2 F 2 A 3 O 3 N 3 E 4 B 4 H 4 G 5 D 5 T 7 R 8 Ö 8 S 15 L 24 J A 25 U 17 O 15 Ä 14 E 13 Ö 6 L 3 D 2 N 1 K 1 S 1 Y 1 Å 1 Ä 2 U 2 O 3 L 3 Ö 3 N 5 E 6 A 7 R 8 I 12 C 15 S 30 K A 26 O 14 E 12 T 11 R 7 L 5 U 5 S 4 N 4 I 3 V 2 Ä 2 Ö 2 H 1 Y 1 Y 1 Ö 1 M 1 G 1 T 1 N 2 D 2 F 2 P 2 Å 2 R 3 U 3 K 3 S 4 B 4 Ä 5 O 7 E 11 I 12 A 13 L 21 L L 21 A 17 I 14 E 12 S 4 T 4 Ä 4 D 4 O 3 J 3 U 3 M 2 V 2 Å 1 K 1 Y 1 Ö 1 N 1 N1T 1 Y 1 Ö 2 I 2 U 3 L 4 Ä 4 S 4 R 4 E 9 M 10 A 15 O 39 M E 25 A 17 I 10 M 9 O 6 Å 4 Ä 3 U 3 S 3 P 2 T 2 N 2 Ö 2 Y 2 B 2 L 2 F 1 H 1 R 1 G 1 Ö 1 L 1 M 1 D 1 G 1 S 1 T 1 L 1 Å 3 N 4 U 4 Ä 4 R 5 O 8 I 16 A 19 E 28 N D 14 G 14 A 14 S 11 T 9 I 8 E 7 N 6 O 4 Ä 2 K 2 U 1 Y 1 Å 1 L 1 F 1 V 1 H 1 B 1 C 1 V 1 D 3 G 3 J 3 F 4 P 4 L 4 B 4 H 5 M 5 I 6 N 7 T 9 R 11 K 12 S 17 O M 21 C 17 N 15 R 15 L 8 T 6 S 3 P 2 D 2 G 2 K 2 V 1 F 1 B 1 U 1 H 1 K 1 D 1 Ä 1 Y 1 T 1 N 1 X 1 L 2 R 2 I 3 Ö 3 E 4 M 6 O 9 A 9 S 15 U 16 P 24 P Å 19 E 15 P 15 R 13 A 12 O 8 L 5 S 3 I 3 T 2 U 2 G 1 N 1 H 1 F 1 Ö 1 K 1 U 1 Y 1 O 1 E 4 S 4 A 5 R 6 G 6 C 7 L 8 I 9 M 12 N 13 D 19 Q U 54 V 42 I 1 A 1 Y 1 I 1 R 1 B 2 U 2 G 2 D 2 K 3 P 3 F 3 Å 3 T 4 O 7 Ä 8 Ö 11 A 20 E 25 R A 17 E 15 I 11 S 7 N 7 O 6 T 5 D 4 K 4 Ä 3 Å 3 U 2 G 2 L 2 B 2 M 2 R 2 Ö 1 Y 1 V 1 F 1 H 1 J 1 Å 1 V 1 P 1 Y 1 Ö 2 M 2 Ä 2 K 3 O 3 D 3 G 4 U 4 L 4 T 7 S 7 A 8 R 9 E 9 I 12 N 15 S T 21 K 13 O 10 E 8 A 8 I 7 S 6 V 3 L 3 Ä 3 P 3 Å 3 M 2 J 1 U 1 N 1 Ö 1 Y 1 B 1 C 1 H 1 D 1 F 1 R 1 Ö 1 Y 1 M 1 Å 1 Ä 2 F 2 G 2 L 2 O 3 R 4 K 4 U 4 I 4 N 7 A 13 T 15 S 15 E 18 T E 20 T 19 I 15 A 14 R 6 S 5 O 5 Ä 2 U 2 Y 2 V 2 N 1 Å 1 L 1 H 1 Ö 1 J 1 B 1 F 1 C 1 I 1 V 2 O 2 A 2 P 2 F 3 G 3 E 3 D 4 B 4 S 5 M 6 N 6 H 7 L 9 J 9 T 9 R 11 K 11 U N 21 T 19 R 10 P 10 S 10 L 9 D 4 M 4 K 4 B 2 G 1 V 1 E 1 A 1 C 1 F 1 I 1 G 1 Å 1 D 2 U 2 N 3 O 4 Ä 4 R 4 K 5 L 5 E 6 T 6 Ö 7 I 12 S 13 A 25 V A 25 I 22 E 22 Ä 11 Å 6 S 3 D 2 O 1 T 1 U 1 L 1 R 1 N 1 G 1 U 1 M 1 G 1 N 1 Ö 1 L 1 K 2 Y 2 W 2 I 3 D 3 T 4 R 6 H 6 A 7 S 10 E 22 O 25 W A 29 E 23 I 22 O 8 H 4 S 4 N 1 R 1 W 1 Ä 1 T 1 L 1 B 1 Y 1 O 4 I 4 U 5 A 12 Ä 13 E 59 X E 24 T 23 P 13 A 11 I 10 L 3 U 3 N 3 O 2 K 1 C 1 F 1 J 1 H 1 B 1 D 1 O 1 C 1 G 1 A 1 E 2 D 2 H 2 F 4 K 5 M 8 S 9 B 9 L 11 R 12 N 14 T 16 Y C 14 R 12 S 12 G 11 T 10 N 7 D 7 A 6 L 5 M 4 K 3 P 2 F 2 O 1 H 1 E 1 B 1 V 1 K 1 Y 1 H 1 C 1 L 2 D 3 S 3 U 4 R 4 O 6 E 7 N 9 Z 10 I 13 T 14 A 21 Z A 19 E 18 I 17 O 12 Z 10 U 2 Y 2 N 2 H 2 L 2 J 2 K 1 B 1 Ü 1 É 1 M 1 T 1 S 1 D 1 K 1 B 2 D 3 H 4 T 4 L 5 N 5 F 5 V 7 G 8 M 8 S 11 R 13 P 24 Å R 27 N 24 G 12 T 10 L 9 D 7 S 5 K 3 V 1 E 1 G 2 B 3 F 3 K 4 D 4 M 5 H 6 J 6 N 8 T 9 L 11 S 11 R 12 V 14 Ä R 35 N 19 L 12 T 7 G 7 S 4 M 4 V 3 K 3 D 2 C 1 X 1 F 1 N 1 D 2 J 3 B 3 K 4 M 4 T 4 G 5 S 5 L 5 R 6 H 6 F 52 Ö R 60 V 7 K 5 S 4 D 4 N 3 T 3 M 3 G 3 J 3 P 2 L 2 Table A2. Transition probabilities for letters preceeding and succeeding a given letter. 10

11 2gram Frequens %tot %letter %cumul de i_ oc fö at me ha in en so ti på st av vi är va sk ko fr si be an ma om ut sa ka pr et re mi se up bl al vä sv tr un sä he kr ba li fi mo nä lä ja Table A3. Bigram word beginnings, the 50 most frequent ones. 11

12 2gram Frequency %tot %letter %cumul en er et tt ar de om _i an ch är na ll ör ra på ka te av ed re ng ta on ns la as ga ig år or ag gt st at in ts så nd es rt da ad ån kt id nt lt el ot Table A4. Bigram word endings, the 50 most frequent ones. 12

13 Lengh Frequency Percent Cumul.% , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,97619 > , Table A5. Distribution of word lengths in the corpus. Word lengh: Mean = Median = Standard deviation = Table A6. Statistics for the word length. 13

14 Ordlängd Procent Tecken Fig. A1. Word lengths (no of characters) and percentage of occurrence. 14

A L A BA M A L A W R E V IE W

A L A BA M A L A W R E V IE W Volume 52 Fall 2000 Number 1 B E F O R E D I S A B I L I T Y C I V I L R I G HT S : C I V I L W A R P E N S I O N S A N D TH E P O L I T I C S O F D I S A B I L I T Y I N