On Longest Increasing Subsequences and Random Young Tableaux: Experimental Results and Recent Theorems

Transcription

1 On Longest Increasing Subsequences and Random Young Tableaux: Experimental Results and Recent Theorems Jon A. Wellner University of Washington April 9, 2002 Abstract Ulam (1961) apparently first posed the following question: what is the average (or distribution of) the length L n of the longest increasing subsequence of a random permutation of the first n integers? Experimental (Monte-Carlo) evidence has played an important role in the study of L n begining with Ulam (1961), and continuing with Baer and Brock (1969), and Odlyzko and Rains (2000). We present experimental evidence concerning the distribution of the length L n of the longest increasing increasing subsequence of a random permutation of length n. In particular, the experimental data confirm the known result E(L n ) 2 n and strongly suggest that Var(L n ) cn 1/3 for a constant c This supports and complements the recent results of Baik, Deift, and Johansson (1999), who apparently knew of the monte-carlo results of Odlyzko and Rains (2000). In the last section we also combine our experimental results with those of Odlyzko and Rains (2000). AMS 2000 subject classifications. Primary: 60G15, 60G99; secondary 60E05. Key words and phrases. limiting distributions, longest increasing subsequence, mean, monte-carlo method, random permutation, random Young Tableaux, random matrices, Ulam s metric, variance. 1

2 1 Introduction. For a recent survey of this problem with connections to patience sorting, see Aldous and Diaconis (1999). 2 History, part 1. Let Π n =(Π n1,...,π nn )bearandom permutation of the first n integers {1,...,n}. Thus P (Π n = π n )= 1 n! for all permutations π n of {1,...,n}. Let L n = L n (Π n ) be the length of the longest increasing subsequence in Π n. For example, if Π Π 9 = (3, 2, 1, 4, 5, 9, 6, 7, 8), then L 9 = 6 for this outcome since (Π 3, Π 4, Π 5, Π 7, Π 8, Π 9 )=(1, 4, 5, 6, 7, 8) is an increasing subsequence in Π of length 6. Note that this subsequence is not unique: in the present example (Π 2, Π 4, Π 5, Π 7, Π 8, Π 9 )=(2, 4, 5, 6, 7, 8) and (Π 1, Π 4, Π 5, Π 7, Π 8, Π 9 )=(3, 4, 5, 6, 7, 8) are also increasing subsequences in Π of length 6. The length of the longest increasing subsequence is also the basis of a metric between permutations: if π = π n and σ = σ n are two permutations of {1,...,n}, then Ulam (1972) defined U(π, σ) =n L n (σ π 1 ); see also Ulam (1981), Diaconis (1982) pages , and Critchlow (1985). Ulam (1961) posed the question: Question 1: How fast does E(L n ) grow with n? Ulam (1961) gave a preliminary monte-carlo analysis of this question, but the first steps towards a definitive answer were first found by Baer and Brock (1968). and by Hammersley (1972). Baer and Brock (1968) tabulated the distributions of L n and of max{l n,k n }, where K n is the length of the longest decreasing subsequence, for n between 1 and 36, and then computed estimates of E(L n ) for n up to 10, 000; see their Figure 1, and note the plotted line 2 n. The exact calculations of Baer and Brock (1968) were extended to n 75 by MacKay (1976), and to n 120 by Odlyzko and Rains (2000). Hammersley (1972) used Kingman s subadditive ergodic theorem and a clever embedding of the problem in a bivariate Poisson process to show that L n n p c and E(L n ) n c for a constant c (π/2,e), but he was unable to prove that the constant c =2as suggested by the calculations of Baer and Brock (1968). 3 History, part 2: Young tableaux. Schensted (1961) showed that there is a one-to-one correspondence between a random permuation Π n and a pair of Young tableaux of the same shape For our example, Π Π 9 =(3, 2, 1, 4, 5, 9, 6, 7, 8), one of the corresponding Young tableau, Λ 9, is given (in the English style ) by 2

3 or, rotating the axes by 90 (in the French style ), The hook lengths are the numbers of elements in the Young tableau above and to the right of each cell in this version of the tableau: thus the hook lengths {h ij } for the above tableau are hook product probability The corresponding hook product is (i,j) Λ n h ij = =3456. Frame, Robinson, and Thrall (1954) showed that the number of (standard) tableaux of a given shape λ n is n! n! (i,j) Λ n h ij h(λ n ). As a consequence of this together with the Schensted correspondence, for any given shape λ n, P (Λ n = λ n )= n! h 2 (λ n ) = 1 n! n! n! h(λ n ) h(λ n ). For our example with n = 9, n! = , the shape corresponding to our given permutation π has probability P (Λ 9 = λ) = (3456) 2 = Summing over all shapes λ with a fixed height of the first column, yields the distribution of L n : P (L n = k) = λ(0)=k P (Λ n = λ) = 3 λ(0)=k n! h 2 (λ).

4 For our example, P (L 9 =6)= 3 P (Λ 9 = λ i )= =.04537, i=1 in agreement with the tabled probability in Baer and Brock (1968), Table 1a, page

5 Figure 1: Scaled Young tableau with limiting shape. The corresponding cumulative tableau is given by Because the tableaux Λ n have total area n, itisuseful to rescale the picture to have total area 1. Thus if Λ n (t) isthe (random) function giving the upper boundary of the tableaux in the plane (with total area n), set f n (t) =n 1/2 Λ n (tn 1/2 ), t [0, ). Figure 1 shows an example of a scaled random Young tableaux for n = 625 together with the limit function of Logan and Shepp (1977) Note that the area under f n is 1 for every n so that f n can be viewed as a random density function on [0, ). 5

6 Similarly, corresponding to the cumulative tableau displayed above, set F n (t) = t 0 f n (s)ds. Thus F n is a (random) distribution function on [0, ) with F n (0) = 0 and F n ( ) =1. Logan and Shepp (1977) found the shape λ which maximizes P (Λ n = λ) = n! h 2 (λ) ; equivalently, minimize log h(λ) = log h ij (i,j) Λ n = S Λ n log{λ(x S ) y S + λ 1 (y S ) x S }. After rescaling by n, and passing to the limit this becomes equivalent to finding the shape (a non-increasing function from [0, ) to[0, ) )f which minimizes subject to H(f) f(x) 0 0 log{f(x) y + f 1 (y) x}dydx. (1) Theorem 1. (Logan and Shepp, 1977). For t (0, 2), 0 f(x)dx =1. (2) f n (t) p f 0 (t) (with respect to a complicated metric d) where f 0, given by 0 θ π, minimizes H in (1) subject to (2); i.e. f 0 (t) = 2 (sin θ θ cos θ), π (3) t = 2 (sin θ θ cos θ)+2cos θ, π (4) H(f 0 ) H(f) for all f with 0 f(x)dx =1. Logan and Shepp (1977) used this to show that the constant c in Hammersley s theorem satisfies c 2. Now define g n (t) byrotating the coordinate system for f n by 45 and scaling both new axes by 2. Theorem 2. (Vershik and Kerov, 1977). For t (0, 2), g n (t) p g 0 (t), 6

7 Figure 2: Limit function with coordinate system rotated by 45. where g 0 is given by g 0 (t) = { 2 π (t arcsin(t)+ 1 t 2 ), t 1 t, t 1. Note that g 0 is just the function f 0 in the new coordinate system with both axes scaled by 2. Vershik and Kerov (1977) succeeded in using this to show that c 2 and in fact that c =2. Figure 2 gives a plot of the limiting shape g 0 in the rotated coordinate system. By now there are several other proofs that c =2: Aldous and Diaconis (1995) prove it again via a limit theorem for an interacting particle system process, the Hammersley process. Seppäläinen (1996) attacks the problem by way of stick breaking and a related inviscid Burgers equation. 4 History, part 3: how big is Var(L n )? The next questions about L n are, quite naturally: Question 2: How fast does Var(L n ) grow with n? Question 3: Does n α (L n / n 2) converge in distribution for some α? For some time there was little progress on the theoretical front, but partial results were obtained by Pilpel (1990), Talagrand (1995), Bollobás and Janson (1997), and Kim (1996). Meanwhile Odlyzko and Rains (2000), inaneffort apparently dating back to about 1994, carried out an extensive monte-carlo study of L n = L (1) n and L (2) n, the height of the second row of the Young tableaux, with n ranging up to [Note: I did not see the paper Odlyzko and Rains (2000) until April 2002.] Meanwhile, because of some analogies with the Grenander estimator of a monotone density (Grenander (1956), Groeneboom (1985), Groeneboom (1989)), I made the following conjecture in 1996 or 1997, and started the monte-carlo study reported here, finally getting up to n = with a C program running on a cluster of Pentium machines under Linux. 7

8 Conjecture 1. n 1/3 ( L n n 2) d Z for some (tight) random variable Z with EZ < 0 and Var(Z) <. Assuming Conjecture 1 is true, then it would follow that E { L n 2 n } n 1/6 E(Z) and and hence both and Var ( L n 2 n ) n 1/3 Var(Z) log( E { L n 2 n } ) log( EZ)+ 1 6 log n log ( L n 2 n ) log(var(z)) + 1 log n. 3 Thus, if Conjecture 1 is correct, plots of means and variances of L n 2 n respectively would yield slopes of 1/6 and 1/3 inlog log plots, and the intercepts in these plots will yield log( E(Z)) and log(var(z)). 5 The Monte-Carlo Data and Plots. To obtain experimental evidence in favor of Conjecture 1, I carried out a Monte-Carlo study. The C-program for calculating random permutations Π n and their lengths L n was obtained from David Eppstein, University of California at Irvine, and put into operation by Greg Warnes. 8

9 Table 1. Summary Data from the Monte-Carlo Experiment; Means and Variances of L n ; 10 4 monte-carlo replications for each n n mean variance n mean variance

10 All the data generated in the experiments are available at Figures 3 and 4 display the empirical means and variances of (2 n L n ) plotted versus n. The straight lines were obtained by varying the intercept and keeping the slopes fixed at 1/6 and 1/3 respectively. Figure 5 displays the empirical means of L n / n Figure 3: Empirical means of (L n 2 n) and line with slope 1/ Figure 4: Empirical variances of (L n 2 n) and line with slope 1/3. 10

11 Figure 5: Empirical means of L n / n Figure 6: Empirical variances of n 1/3 (L n / n 2). 11

12 Table 2. Estimated constants (Wellner, Odlyzko and Rains); and theoretical results from Baik, Deift, Johansson (1999) and Tracy and Widom (1994) Estimated Estimated n E(Z) Var(Z) fitted lines estimates (OR) theory (BDJ) History, part 4: Conjecture 1 is true! In fact Conjecture 1 is true, as has been proved by Baik, Deift, and Johansson (1999). Theorem 3. (Baik, Deift, and Johansson, 1999). where ( P (Z z) =exp n 1/3 (L n / n 2) d Z and the function u satisfies the Painlevé II equation subject to the boundary condition z ) (x z)u 2 (x) dx F (z) u (x) =xu(x)+2u 3 (x) u(x) Ai(x) as x. By earlier results of Tracy and Widom (1994), the solution u exists, is unique, ( ) e (4/3)x3/2 u(x) = Ai(x)+O x 1/4 x, and x ( u(x) = 1+O(x 2 ) ) as x. 2 By numerical calculations, Tracy and Widom (1994) and Baik, Deift, and Johansson (1999) find that for Z F, E(Z) = and Var(Z) = For discussions of further related results, see Aldous and Diaconis (1999), Deift (2000), and the lecture notes of Tracy (2001). 12

13 Figure 7: Histogram of 10 4 simulated values of n 1/3 (L n / n 2) with n = Figure 8: The Tracy-Widom distribution F together with Histogram of 10 4 simulated values of n 1/3 (L n / n 2) with n = Part 5: Combining the Monte-Carlo results of Section 3 with Odlyzko and Rains Given the Monte-Carlo results of Odlyzko and Rains (2000) posted at it seems reasonable to combine with the results presented here in Section 3. Plots below parallel the plots 3-6 in Section 3, but with the mean and variance summaries from the experiments 13

14 of Odlyzko and Rains (2000) added. Note that all the black dots (Wellner) are based on 10 4 Monte- Carlo replications; the red dots (Odlyzko-Rains) are based on 10 7 replications for n =10 4, replications for n =10 5,10 5 replications for n =10 6 and n =10 7,10 4 replications for n =10 8, replications for n =10 9, and replications for n = Figure 9: Empirical means of (2 n L n ) and line with slope 1/ Figure 10: Empirical variances of (2 n L n ) and line with slope 1/3. 14

15 Figure 11: Empirical means of L n / n Figure 12: Empirical variances of n 1/3 (L n / n 2). 15

16 Figure 13: Empirical means of n 1/3 (2 L n / n). Acknowledgements: Thanks are due to Persi Diaconis for a long conversation in August 1997 about this problem. I owe thanks to Greg Warnes for obtaining the C program from David Eppstein, for getting it working, and for time on the machines at the Fred Hutchinson Cancer Research Center. I also owe thanks to Craig Tracy for providing the Mathematica program used to calculate the Tracy-Widom distribution shown in Figure 8. References Abhyankar, S. S. (1988). Enumerative Combinatorics of Young Tableaux. Marcel Dekker, New York. Aldous, D. and Diaconis, P. (1995). Hammersley s interacting particle process and longest increasing subsequences. Probability Theory and Related Fields 103, Aldous, D. and Diaconis, P. (1999). Longest increasing subsequences: from patience sorting to the Baik-Deift-Johansson theorem. Bull. (New Series) Amer. Math. Soc. 36, Baer, R. M. and Brock, P. (1968). Natural sorting over permutation spaces. Math. Comp. 22, Baik, J., Deift, P., and Johansson, K. (1998). On the distribution of the length of the longest increasing subsequence of random permutations. J. Amer. Math. Soc. 12, Baik, J., Deift, P., and Johansson, K. (2000). On the distribution of the length of the second row of a Young diagram under Plancherel measure. Geom. Funct. Anal. 10, Bollobás, B. and Janson, S. (1997). On the length of the longest increasing subsequence in a random permutation. In Combinatorics, geometry and probability (Cambridge, 1993), , Cambridge Univ. Press, Cambridge, Critchlow, D. (1985). Metric Methods for Analyzing Partially Ranked Data. Lect. Notes Statist. 34, Springer-Verlag, New York. 16

17 Deift, P. (2000). Integrable systems and combinatorial theory. Notices Amer. Math. Soc. 47, Diaconis, P. (1982). Group Theory in Statistics. IMS Lecture Notes - Monograph Series 11, IMS, Hayward. Frame, J. S., Robinson, G. de V., and Thrall, R. M. (1954). The hook graphs of the symmetric group. Canad. J. Math. 6, Fredman, M.L. (1975). On computing the length of longest increasing subsequences. Discrete Mathematics 11, Frieze, A. (1991). On the length of the longest monotone sequence in a random permutation. Ann. Appl. Probab. 1, Fulton, W. F. (1997). Young Tableaux With Applications to Representation Theory and Geometry. London Mathematical Society, Student Text 35, Cambridge. Greene, C. (1974). An extension of Schensted s theorem. Advances in Math. 14, Grenander, U. (1956). On the theory of mortality measurement, part II. Skandinavisk Aktuarietidskrift 39, Groeneboom, P. (1985). Estimating a monotone density. Proceedings of the Berkeley Conference in Honor of Jerzy Neyman and Jack Kiefer, Vol. II, Lucien M. LeCam and Richard A. Olshen eds. Groeneboom, P. (1989). Brownian motion with a parabolic drift and Airy functions. Probability Theory and Related Fields 81, Hammersley, J. M. (1972). A few seedlings of research. In Proc. Sixth Berk. Symp. Math. Stat. and Prob., Vol. 1, Johansson, K. (2001). Discrete orthogonal polynomial ensembles and the Plancherel measure. Ann. of Math. (2) 153, Kerov, S. (1993). Gaussian limit for the Plancherel measure of the symmetric group. C. R. Acad. Sci. Paris 316, Kim, J. H. (1996). On increasing subsequences of random permutations. J. Comb. Theor. A 76, Kingman, J. (1973). Subadditive ergodic theory. Ann. Probability 1, Knuth, D. E. (1973). The Art of Computer Programming, III. Addison-Wesley, New York. Krattenthaler, C. (1995). The Major Counting of Nonintersecting Lattice Paths and Generating Functions for Tableaux. Memoirs of the Amer. Math. Soc. 552, American Mathematical Society, Providence. Logan, B. F. and Shepp, L. A. (1977). A variational problem for random Young tableaux. Advances in Mathematics 26, MacKay, J. (1975). The largest degrees of irreducible characters of the symmetric group. Math. Comp. 30, Narayana, T. V. (1979). Lattice Path Combinatorics with Statistical Applications. University of Toronto Press, Toronto. 17

18 Odlyzko, A.M. and Rains, E.M. (2000). On longest increasing subsequences in random permutations. In Analysis, geometry, number theory: the mathematics of Leon Ehrenpreis (Philadelphia, PA, 1998), , Contemp. Math. 251, Amer. Math. Soc., Providence, RI. On longest increasing subsequences in random permutations. Technical Report, AT&T Labs. Pilpel, S. (1990). Descending subsequences of random permutations. J. Comb. Theor. A 53, Rains, E. M. (1998). Increasing subsequences and the classical groups. Electronic J. of Combinatorics 5, R12. Sagan, B. (1991). The Symmetric Group: Representations, Combinatorial Algorithms, and Symmetric Functions. Wadsworth, Pacific Grove. Schensted, C. (1961). Longest increasing and decreasing subsequences. Canad. J. Math. 13, Semba, I. (1984). On generating and counting all the longest increasing subsequences. J. Inform. Proc. 7, 1-4. Seppalainen, T. (1996). A Microscopic model for the Burgers equation and longest increasing subsequences. Electronic Journal of Probability 1, paper 5. Seppalainen, T. (1997). Increasing sequences of independent points on the planar lattice. Ann. Appl. Prob. 7, Skiena, S. (1990). Implementing Discrete Mathematics. Addison-Wesley, Reading. Talagrand, M. (1995). Concentration of measure and isoperimetric inequalities in product space. Publications Mathématiques IHES 81, Tracy, C. A. (2001). Univerality of the distribution functions of random matrix theory. Lecture Notes for the Midwest Probability Symposium, October Available at: lalley/. Tracy, C. A., and Widom, H. (1994). Level-spacing distribution and the Airy kernel. Commun. Math. Phys. 159, Ulam, S. M. (1961). Monte Carlo calculations in problems of mathematical physics. Modern Mathematics for the Engineer: Second Series. E. F. Beckenbach, Editor. McGraw-Hill, New York. Ulam, S. M. (1972). Some ideas and prospects in biomathematics. Ann. Rev. Biophys. Bioeng. 1, Ulam, S. M. (1981). Future applications of mathematics in the natural sciences. American Mathematical Heritage: Algebra and Applied Mathematics. Texas Tech. University, Mathematics Series, 13, Verskik, A. M. and Kerov, S.V. (1977). Asymptotics of the Plancherel measure of the symmetric group and the limiting form of Young tables. Soviet Math. Dokl. 18, Translation of Dokl. Acad. Nauk. SSSR 233, Wolfram, S. (1996). The Mathematica Book. Wolfram Media, Champaign. 18

19 University of Washington Statistics Box Seattle, Washington U.S.A. 19