Counting Crosswords. April 19, 2006 Draft 2.5

Transcription

1 ountng rosswords avd J.. ackay Jeremy horpe pr 19, 2006 raft 2.5 bstract hannon s cacuaton of the number of crosswords assumed that the rows and coumns of crosswords contan typca strngs from the anguage. owever, n most anguages, most crosswords w have rows and coumns that are atypca. hs atypcaty modfes the way n whch we count the number of crosswords. 1 ntroducton hannon (1948) observed that arge numbers of arge crosswords can be constructed f a anguage has suffcenty ow redundancy: arge two dmensona crosswords are possbe, he sad, f the entropy per character W of the anguage consstng of words separated by spaces satsfes W > 1 2 max, (1) where max s the maxmum achevabe entropy per character. hs observaton of hannon s was eaborated by Wof and ege (1998). hey nterpreted hannon s asserton that arge two dmensona crosswords are possbe as meanng that the number of vad crosswords grows exponentay wth the number of squares n the grd. hey counted the number of vad crosswords n a anguage by countng the number of typca ways of fng n the rows of a grd, then evauatng the probabty that one such fng-n aso has vad and typca coumns. hey reproduced hannon s resuts, and augmented them by suppyng a tghter condton, appcabe to anguages n whch etters are not used wth equa frequency. he probem wth hannon s resut (1), you see, s that f we swtch anguage from ngsh (n whch we know two-dmensona crosswords are possbe) to ngsh pus a few foregn words that use new characters not ncuded n the orgna ngsh aphabet, then certany a the crosswords we made n ngsh are st vad crosswords n the new anguage, but the nequaty (1) may we be voated, snce addng extra characters to the aphabet ncreases max on the rght hand sde. 1

2 G F F F V G V Y K Y J V V G W F K X K K Y Fgure 1. rosswords of types (mercan) and (rtsh). n a type (or mercan) crossword, every row and coumn conssts of a successon of words of ength 2 or more separated by one or more spaces. n a type (or rtsh) crossword, each row and coumn conssts of a mxture of words and snge characters, separated by one or more spaces, and every character es n at east one word (horzonta or vertca). Whereas n a type crossword every etter es n a horzonta word and a vertca word, n a typca type crossword ony about haf of the etters do so; the other haf e n one word ony. ype crosswords are harder to create than type because of the constrant that no snge characters are permtted. ype crosswords are generay harder to sove because there are fewer constrants per character. o Wof and ege derved a tghter condton for arge two-dmensona crosswords to be possbe, W > 1 2 0, (2) where 0 s the entropy of the monogram dstrbuton of the anguage. Wof and ege s cacuatons were modfed n (ackay, 2003, ecton 18.1) so as to gve condtons not ony for the type of crossword but aso for the type (fgure 1). ackay weakened Wof and ege s cacuatons by assumng the anguage conssted of W words a of the same ength. sng typcaty arguments smar to those of Wof and ege, the condtons for arge crosswords of the two types to be possbe were found to be as foows, where 0 s the entropy of the monogram dstrbuton for non-space characters, and the entropy of the anguage consstng of arbtrary strngs of words s W og 2 W + 1. (3) rossword type ondton for crosswords W > W > f we set 0 = 4.2 bts and = 5 then we can estmate how bg a vocabuary 2

3 s requred for crosswords of the two types to be numerous: for type, W 2 0/2 1500; and for type, W 2 0/4 40. hese fgures seem consstent wth experence: we can easy wrte chdren s crosswords of type, but most crosswords of type contan more obscure words. owever, these cacuatons of the number of vad crosswords underestmate ths number by countng ony typca crosswords. t s key that atypca fngs-n of the crossword domnate the true count. We now gve an exampe ustratng the naccuracy of the condton (2), then gve a new cacuaton of the number of crosswords, assumng a smpe anguage mode. We w use the foowng notaton: W p f 1 f w 1 2 umber of words n dctonary typca word ength monogram dstrbuton for non-space characters (etters) number of squares n crossword number of etter squares number of words n the crossword number of etter squares whose etters appear n one word ony number of etter squares whose etters appear n two words ote f 1 = n crosswords of type n whch the typca word ength s, typca vaues of f w, f 1, 1, and 2 are as foows: 2 1 f w f counterexampe to condton (2) onsder a anguage wth 514 characters, of whch two, 0 and 1, have probabty 1/4 each, and the other 512 characters have probabty 1/1024 each. he entropy of ths monogram dstrbuton s 0 = 6 bts. f the dctonary of the anguage contans W = words a of ength = 5 then the entropy per character of the anguage s W = 2 bts. Gven ths etter entropy 0 and anguage entropy W, our condtons for crosswords expect nether type of crossword to be possbe. ut n fact, the Wengsh dctonary w amost 3

4 certany contan amost a the thrty-two possbe fve-etter bnary strngs, 00000, 00001, 00010,..., (snce each had a probabty of 2 10 of occurrng when a new word was added to the dctonary, and there are 2 12 words). o a the 2 f1 atypca crosswords that contan excusvey the characters 0 and 1 are amost certany vad croswords. hus an exponentay arge number of crosswords do exst for ths anguage, abet atypca crosswords domnated by just 32 crossword-frendy words from the dctonary. ow, hannon coud defend hs cacuaton by sayng m not nterested n atypca crosswords, ony want to count crosswords n whch the rows and coumns are typca of the anguage here, crosswords that use the fu dctonary n a baanced manner. owever, we suspect that n rea fe, crosswords are ndeed popuated by an atypca dstrbuton that favours crossword-frendy words. We therefore offer a new cacuaton that aows the words that succeed n makng crosswords to be atypca. 3 new cacuaton magne that a anguage s made by creatng a dctonary of W words, a of ength, from a monogram dstrbuton p. We count the number of twodmensona crosswords of squares by assumng that an approprate grd of back and whte squares has been made, and evauatng the probabty that each possbe n-fng s vad. (n contrast to the cacuaton n (ackay, 2003, ecton 18.1), we do not restrct attenton to typca n-fngs.) We denote the dctonary by {d (w) } W w=1 ; w runs over the words n the order that they were created; the th etter of the wth word s d (w). We denote a canddate crossword by the vector X whose components are x s, wth s runnng from 1 to f 1. From the vector X we can extract the tentatve words x (n), where n runs from 1 to = f w. ach word x (n) conssts of components from X. o understand our cacuaton, magne that a possbe n-fngs X are created before the dctonary s generated; we then ask, what s the probabty that n-fng X w turn out to be vad? o hep us answer ths queston, we ntroduce a key, w(n), whch s a putatve mappng from words n the grd n to words n the dctonary w. f an n-fng s vad, then there exsts a key w(n) such that, for each n x (n) = d (w(n)), for a. (4) We denote the key by W. he number of possbe keys for a grd contanng words s W. We can then count the expected number of vad crosswords, Ω, as foows. Ω (d (w(n)) =x (n) ) (5) X W n 4

5 hs count s approxmate because we are overcountng n cases where there are mutpe vad keys (e.g., a partcuar word appears twce n the dctonary) and the cacuaton s naccurate f any partcuar dctonary word s used twce n the grd, because t treats as ndependent events that are not. he probabty (d(w(n)) =x (n) ) s the probabty, when the dctonary comes to be generated, that the word d (w(n)) w exacty match the word x (n) defned by the n-fng, X. K so far? nce the dctonary s beng generated from a monogram dstrbuton p, (d (w(n)) =x (n) ) = p (n) x. (6) ow, n the monster sum-product (5), each component x n of X s mentoned n an expresson of the form (d (w(n)) =x (n) ) ether twce or once, f t appears n two words or just one, respectvey. ur sum-product s about to smpfy. For components that appear n just one word, the smpfcaton nvoves a factorzaton ( p x (n) (n) x ) = p = 1. For components that appear n two words, the two events of the form (d (w(n)) = ) must nvove the random dctonary producng the same character n two dstnct words; we thus obtan a factor p 2. (7) We sum over X frst, for fxed W. et s ca a key W non-codng f a n map to dstnct w under w(n). We focus attenton on non-codng keys. ecause the dctonary words are generated ndependenty and dentcay, a non-codng keys W yed exacty the same answer for the quantty ( 2 (d (w(n)) =x (n) ) = p) 2, (8) X n where 2 s the number of ntersecton characters. ssumng W, the number of non-codng keys s approxmatey the number of keys, W, so the expected number of vad crosswords s: ) 2 Ω W ( p 2 = W fw ( p 2 ) 2. (trcty, we want to be free to send to nfnty, so the assumpton that W ooks ke t produces probems; what we coud do here s try to ntroduce a better defnton of non-codng.) et s massage ths nto a form 5

6 that we can compare wth the Wof ege expresson for the number of vad crosswords, Ω W 2 (2W 0). (9) We ( see that the factor W fw s anaogous to 2 2W, and the curous factor ) p2 s anaogous to 2 0. efore we carry out the massage, et s focus on the curous factor. he sum of squares ( ) p2 s masqueradng as a repacement for the nverse-exponentaentropy, pp ; ths approxmaton, p p p 2, (10) s a surprsngy good one for any 20 dmensona probabty vector, for exampe, the approxmaton s accurate to wthn a factor of 2.5! o teachers of nformaton theory, ths approxmaton may be famar, because t arses from a common error made when sovng entropy nequates. onsder a student tasked wth provng that the entropy (p) of an dmensona vector p s bounded above by og. e uses Jensen s nequaty: p og 1 = ( ) p og p = og p og p = og p 2. (11) p ops! e has not proved that the entropy s bounded above by anythng; nstead he has proved that t s bounded beow by a quantty known as the order-two ény entropy, ( ) (2) (p) og. (12) [he order-r ény entropy s ( (r) (p) = 1 1 r og p 2 p r ) ]. (13) o, to concude our cacuaton, et s defne the ény monogram entropy per character (ncudng spaces) by (2) ono + 1 (2) (p). (14) hen, usng W og 2 W +1 and the fgures from the tabe at the end of secton 1, we can rewrte our resut (9) as two resuts, one for each type of crossword: (2W (2) Ω 2 ono ) (15) Ω 2 (W 1 4 (2) ono ). (16) hus the condtons for there to be exponentay many crosswords become: 6

7 rossword type ondton for crosswords W > 1 2 (2) ono W > 1 4 (2) ono. hese condtons are peasngy smar to Wof and ege s, wth the smpe repacement of the entropy by the ény entropy. 3.1 hree-dmensona crosswords he condton for d-dmensona mercan-stye crosswords (n whch every etter partcpates n d words) can be derved n the same way. he number of crosswords s ) d Ω W ( p d = W fw(d) ( p d ) d. We assume f w (d) = d +1 and d = +1. We note that the order d ény entropy has appeared: ( ) og = (d 1) (d) (p) (17) p d We defne the order d ény monogram entropy per character (ncudng spaces) by (d) ono + 1 (d) (p). (18) hen (dw (d 1)(d) Ω 2 ono ) (19) hus the condton for there to be exponentay many d-dmensona type- crosswords s: ondton for crosswords W > d 1 d (d) ono. hs resut s dentca to hannon s ( the redundancy must be ess than 1/d ) f the ény monogram entropy s equa to the entropy of the unform dstrbuton. 4 predcton spn-off of our cacuaton s a predcton about atypcaty of words n crosswords, assumng crosswords are seected at random from the set of a vad crosswords, wthout human nterventon. he predcton ony appes to anguages modeed by our random dctonary mode, and t s best tested n 7

8 type crosswords n whch there are characters of both types, ntersectng and non-ntersectng. he predcton s that whereas the etters n non-ntersectng squares are expected to have the same dstrbuton as the source p that generated the dctonary, the etters at ntersectons of words are expected to come from the dstrbuton q p2. (20) p2 t woud be nce to test ths predcton on a corpus of rea crosswords; professona crossword authors, however, are famar wth ths tendency of the more common etters to appear n the ntersectons of azy-constructed crosswords, and take speca effort to counteract t. 5 ppcaton to two-dmensona (d, k) constrants Wof (2001) dscusses the capacty 2 (d, k) of two-dmensona bnary arrays whose rows and coumns satsfy the (d, k) constrant: after every 1 there must be at east d and at most k 0s. e observes that hannon s condton for crosswords fas to predct correcty n a cases whether (d,k) 2 > 0. n partcuar, t s known that (2,4) 2 > 0 and (1,2) 2 = 0; but hannon s condton predcts that crosswords wth nether (d, k) = (2, 4) nor (1, 2) exst. (hannon s condton makes dentca predctons for these two cases, because hs condton depends ony on a anguage s onedmensona capacty, (d,k) 1 ; these two capactes happen to satsfy (2,4) 1 = (1,2) ) hs conundrum s not soved by Wof s anayss. o, does our observaton that most crosswords w have rows and coumns that are atypca hep resove ths conundrum? he answer s amost. For each one-dmensona channe, we can ntroduce parameters p that contro the transton probabtes between states. We can compute how the fracton of 1s emtted depends on p, p 1 (p); and compute the dependence of the one-dmensona capacty (d,k) 1 on p. [We here extend the defnton of capacty to aow dependence on p; the true capacty s the maxmum over p of (d,k) 1 (p).] ow, magne generatng grds randomy fed wth the fracton of 1s n the grd beng p 1. he number of such grds scaes as 2 2(p1), (21) where 2 s the bnary entropy functon. he probabty that a the rows of the grd are (a) vad accordng to the (d, k) constrant, and (b) typca of the parameters p that we ntroduced above, s 2(d,k) 1 (p) f =. (22) 22(p1(p)) 8

9 (2,4)(p2,p3) (1,2)(p) Fgure 2. Graphs of the exponents (2,4) 2 (p 2, p 3) and (1,2) 2 (p) as the arguments p, p 2, and p 3 vary. For (2,4) 2 (p 2, p 3), the horzonta axs s p 2 and p 3 takes the vaues 0.375, 0.5, 0.535, 0.625, 0.75, whch bracket the optmum. he maxmum vaue of the exponent (2,4) 2 (p 2, p 3) s For (1,2) 2 (p), the horzonta axs s p, the probabty of emttng a 1 after a run of one zero. he probabty that a the coumns are aso vad and typca s aso f. o, negectng nter-coumn correaton (n the sprt of hannon), we obtan the number of crosswords wth parameters p by mutpyng (21) by f 2. ( 22(d,k) 1 (p) 2 2 1(p) ) Ω(p) 2 = (23) 2(p1(p)) 2 2(p1(p)) whch s an ncreasng or decreasng functon of wth exponent (d,k) 2 (p) = 2 (d,k) 1 (p) 2 (p 1 (p)). (24) [erhaps for carty we shoud name ths exponent (d,k) 2 (p) rather than (d,k) 2 (p), snce t seems reasonabe to reserve (d,k) 2 for the true, unknown two-dmensona capacty? Furthermore, ths exponent s not even a genune bound on the capacty, snce t depends on the assumpton that nter-coumn correatons can be negected.] When we compute the maxmum vaue of ths exponent, we fnd competey dfferent vaues for the cases (d, k) = (1, 2) and (2, 4), and the exponent (d,k) 2 (p) s greater n the atter case. o we amost sove the conundrum. ut ony amost, because the maxmum exponent found for (d, k) = (2, 4) s st negatve. 5.1 etas We parameterze the one-dmensona (2, 4) sequence generator by parameters p 2 and p 3, whch are the probabtes of emttng a 1 after a run of two zeroes 9

10 and three zeroes respectvey. t equbrum, the fracton of 1s emtted s he one-dmensona capacty s p 1 (p 2, p 3 ) = 1.0/(4 p 2 + (1 p 2 )(1 p 3 )). (25) (2,4) 1 (p 2, p 3 ) = p 1 (p 2, p 3 ) 2 (p 2 ) + f 3 (p 2, p 3 )(p 3 ) (26) where he exponent f 3 (p 2, p 3 ) = (1 p 2 )/(4 p 2 + (1 p 2 )(1 p 3 )) (27) (2,4) 2 (p 2, p 3 ) = 2 (2,4) 1 (p 2, p 3 ) 2 (p 1 (p 2, p 3 )) (28) s potted n fgure 2 for a range of vaues of p 2 and p 3, aongsde the correspondng exponent (1,2) 2 (p) for the (1, 2) channe. vdenty, to compute the true capacty (2,4) 2, whch s known to be at east 1/8 (appendx ), we w have to take nto account correatons between neghbourng rows and coumns. 5.2 akng nto account correatons We can use the exact same method on a set of two-by-two tes that obey the rues of the (2,4) constrant. We ntroduce 15 free parameters, sove for the prncpa egenvector, and fnd the entropy of the te dstrbuton and the capacty of the one-dmensona channe. We then maxmze the exponent wth respect to the free parameters. nfortunatey, whe we do fnay obtan a postve bound on the capacty of the two-dmensona (2,4) array, the resuts of ths effort do not mprove on the known bounds on the capacty. ur optmzed two-by-two tes (descrbed n appendx ) estabshed that the capacty of the two-dmensona (2,4) array s t s smpe to estabsh, usng four 4-by-4 tes, a better bound of 1/8 = (appendx ) (Kato and Zeger, 1999). ore sophstcated methods are needed to obtan tght bounds on the capacty of (d, k) arrays. eferences Kato,., and Zeger, K. (1999) n the capacty of two-dmensona run ength constraned channes. rans. nfo. heory 45 (5): ackay,. J.. (2003) nformaton heory, nference, and earnng gorthms. ambrdge nversty ress. vaabe from 10

11 hannon,.. (1948) mathematca theory of communcaton. e ys. ech. J. 27: , Wof, J. K. (2001) 2001 hannon ecture. onstraned sequences, crossword puzzes and hannon. nformaton heory ocety ewsetter 51 (3): 1 7. Wof, J. K., and ege,. (1998) n two-dmensona arrays and crossword puzzes. n roceedngs of the 36th erton onference on ommuncaton, ontro, and omputng, ept. 1998, pp erton ouse. wo-by-two tes ere are the detas for the bound on capacty of he one-dmensona transton matrx maps from runength states [r 1, r 2 ] to runength states [r 1, r 2], where r counts the ength of the current run of zeros n row. When we optmzed ths transton matrx, we found an equbrum dstrbuton that emtted many the tes wth weght 1. he dstrbuton over tes was (0.000, 0.251, 0.247, 0.251, 0.002, 0.247, 0.002) he non-zero entres n the transton matrx were: [1,0] [0,2] = 1, [0,1] [2,0] = 1, [2,0] [1,2] = 0.535, [0,2] [2,1] = 0.535, [2,0] [0,2] = 0.465, [0,2] [2,0] = 0.465, [2,0] [4,2] = , [0,2] [2,4] = , [3,0] [1,2] = 0.535, [0,3] [2,1] = 0.535, [3,0] [0,2] = 0.465, [0,3] [2,0] = 0.465, [4,0] [1,2] = 1, [0,4] [2,1] = 1, [2,1] [1,3] = 0.375, [1,2] [3,1] = 0.375, [2,1] [0,3] = 0.407, [1,2] [3,0] = 0.407, [2,1] [4,0] = 0.218, [1,2] [0,4] = 0.218, [3,1] [1,0] = , [1,3] [0,1] = , [3,1] [1,3] = 0.468, [1,3] [3,1] = 0.468, [3,1] [0,3] = 0.508, [1,3] [3,0] = 0.508, [4,1] [1,3] = 1, [1,4] [3,1] = 1, [3,2] [0,1] = 0.33, [2,3] [1,0] = 0.33, [3,2] [1,0] = 0.33, [2,3] [0,1] = 0.33, [3,2] [0,4] = 0.34, [2,3] [4,0] = 0.34, [4,2] [1,4] = 1, [2,4] [4,1] = 1, [4,3] [1,0] = 1, [3,4] [0,1] = 1. he onedmensona capacty s bts, and the entropy of te dstrbuton s bts. Four-by-four tes he capacty of the (d, k) = (2, 4) array s at east (og 2 4)/16 = 1/8, because the foowng four 4-by-4 tes may be freey ntermnged (Kato and Zeger, 1999)