Compression in the Space of Permutations

Transcription

1 Compression in the Space of Permutations Da Wang Arya Mazumdar Gregory Wornell EECS Dept. ECE Dept. Massachusetts Inst. of Technology Cambridge, MA 02139, USA University of Minnesota Minneapolis, MN 55455, USA Abstract We investigate the lossy compression of permutations by analyzing the trade-off between the size of a source code and the distortion with respect to l 1 distance of inversion vectors, Kendall tau distance, Spearman s footrule and Chebyshev distance. We show that given two permutations, Kendall tau distance upper bounds the l 1 distance of inversion vectors and a scaled version of Kendall tau distance lower bounds the l 1 distance of inversion vectors with high probability, which indicates an equivalence of the source code designs under these two distortion measures. Similar equivalence is established for all the above distortion measures, every one of which has different operational significance and applications in ranking and sorting. These findings show that an optimal coding scheme for one distortion measure is effectively optimal for other distortion measures above. 1 Introduction In this paper we consider the lossy compression (source coding) of permutations, which is motivated by storing ranking data, and lower bounding the complexity of approximate sorting. In a variety of applications such as admission and recommendation systems (such as Yelp.com and IMDB.com), ranking, or the relative ordering of data, is the key object of interest. In many databases, the principle queries ask for the top k-items or the median object, etc. Noting that a ranking of n items can be represented as a permutation on n elements 1 to n, storing a ranking is equivalent to storing a permutation. In general, to store a permutation of n elements, we need log 2 () n log 2 n O(n) bits. However, if we can tolerate certain error (instead of the top restaurant, say the query returns one of the top five), then how many bits are necessary for storage? In addition to application on compression, source coding of the permutation space is also related to the analysis of comparison-based sorting algorithms. Given a group of elements of distinct values, comparison-based sorting can be viewed as the process of finding a true permutation by pairwise comparisons, and since each comparison in sorting provides at most 1 bit of information, the log-size of the permutation set S n provides a lower bound to the required number of comparisons, i.e., log = n log n + O (n). Similarly, the lossy source coding of permutations provides a lower bound to the problem of comparison-based approximate sorting, which can be seen as searching a true permutation subject to certain distortion. Again, the log-size of the code indicates the amount of information (in terms of bit) needed to specify the 1

2 true permutation subject to certain distortion, which in turn provides a lower bound on the number of pairwise comparisons needed. The problem of approximate sorting has been investigated in [1], where results for the moderate distortion regime are derived with respect to the Spearman s footrule metric [2] (see below for definition). On the other hand, every comparison-based sorting algorithm corresponds to a compression scheme of the permutation space, as we can treat the outcome of each comparison as 1 bit. This string of bits is a (lossy) representation of the permutation that is being (approximately) sorted. However, reconstructing the permutation from the compressed representation may not be straightforward. In our earlier work [3], a rate-distortion theory for permutation space is developed, with the worst-case distortion as the parameter, and the rate rate-distortion functions and source code designs for two different distortion measures, Kendall tau distance and the l 1 distance of the inversion vectors, are derived. In Section 3 of this paper we show that under average-case distortion, the rate-distortion problem under Kendall tau distance and l 1 distance of the inversion vectors are equivalent and hence the code design could be used interchangeably, leading to simpler coding schemes for the Kendall tau distance case (than developed in [3]), as discussed in Section 4. Moreover, the rate-distortion problem under Chebyshev distance is also considered and its equivalence to the cases above is established. Operational meaning and importance of all these distance measures is discussed in Section 2. While these distance measures usually have different intended applications, our findings show that an optimal coding scheme for one distortion measure is effectively optimal for other distortion measures. 2 Problem formulation In this section we discuss aspects of the problem formulation. We provide a mathematical formulation the rate-distortion problem on a permutation space in Section 2.2, introduce the distortions of interest in Section 2.3, and discuss their operation meaning in Section Notation Let S n denote the symmetric group of n elements. We write the elements of S n as arrays of natural numbers with values ranging from 1,..., n and every value occurring only once in the array. For example, [3, 4, 1, 2, 5] S 5. This is also known as the vector notation for permutations. Given a metric d : S n S n R + {0}, we define a permutation space X (S n, d). Throughout the paper, we denote the set {1,..., n} as [n], and let [a : b] {a, a + 1,..., b 1, b} for any two integers a and b. 2.2 Rate-distortion problem Given a permutation space, we define the following rate-distortion problem. Definition 1 (Codebook for permutations under average-case distortion). A (n, D n ) source code C n S n for X (S n, d) is a set of M n = C n permutations such that for 2

3 a σ that is drawn uniformly from S n, there exists π C n that E [d(π, σ)] D n. Let A(n, D n ) be minimum size of the (n, D n ) source codes in X (S n, d). We define the minimal rate for distortion D n as R(D n ) log A(n, D n). log As to the classical rate-distortion setup, we are interested in deriving the trade-off between distortion level D n and the rate R(D n ) as n. In this work we show that for the distortions d(, ) of interest, lim n R(D n ) exists. Instead of requiring E [d(π, σ)] D n in the above definition, we may require P [d(π, σ) > D n ] 0, (1) as n. As we will see, this stronger requirement does not change the asymptotic rate-distortion trade-off. 2.3 Distortion measures There are several distortion measures possible in permutations. Some of the most natural measures include Kendall-tau distance and l p distances, where p {1, 2,..., }. In this section we introduce the distortion measures of interest, Spearman s footrule (l 1 distance between two permutation vectors), Chebyshev distance (l distance between two permutation vectors), the l 1 distance of inversion vectors and Kendall tau distance. Definition 2 (Spearman s footrule). Given two permutations σ 1 and σ 2, the Spearman s footrule between σ 1 and σ 2 is n d l1 (σ 1, σ 2 ) σ 1 σ 2 1 = σ 1 (i) σ 2 (i) Definition 3 (Chebyshev distance). Given two permutations σ 1 and σ 2, the Chebyshev distance between σ 1 and σ 2 is i=1 d l (σ 1, σ 2 ) σ 1 σ 2 = max 1 i n σ 1(i) σ 2 (i) Inversion vector is a representation of a permutation, building upon the notion of inversions. Definition 4 (inversion, inversion vector). An inversion in a permutation σ S n is a pair (σ(i), σ(j)) such that i < j and σ(i) > σ(j). We use I n (σ) to denote the total number of inversions in σ S n, and K n (k) {σ S n : I n (σ) = k} (2) to denote the number of permutations with k inversions. A permutation σ S n is associated with an inversion vector x σ G n [0 : 1] [0 : 2] [0 : n 1], where for i = 1,..., n 1, x σ (i) = { j [n] : j < i + 1, σ 1 (j) > σ 1 (i + 1) }. In words, x σ (i) is the number of inversions in σ in which i + 1 is the first element. 3

4 It is well known that mapping from S n to G n is one-to-one and straightforward [4]. Definition 5 (l 1 distance of inversion vectors). Given two permutations σ 1 and σ 2, we define the l 1 distance of two inversion vectors as n 1 d x,l1 (σ 1, σ 2 ) x σ1 (i) x σ2 (i). (3) i=1 Now we introduce another distortion measure of interest, Kendall tau distance. Definition 6 (Kendall tau distance). The Kendall tau distance d τ (σ 1, σ 2 ) from one permutation σ 1 to another permutation σ 2 is defined as the minimum number of transpositions of pairwise adjacent elements required to change σ 1 into σ 2. The Kendall tau distance is upper bounded by ( n 2). Remark 1 (Bubble-sort). Let e [1, 2,..., n] be the identify permutation, and given a input sequence of σ, d τ (σ, e) is the number of swaps needed in a bubble-sort algorithm [4]. 2.4 Operational meaning of the distortion measures Spearman s footrule is a very well-known measure of disarray (see, [2]), where the sum of the deviations at different positions is measured. The Chebyshev distance measures maximum of the deviations at different positions. Therefore, Spearman s footrule (l 1 distance) measures the total deviation, while Chebyshev distance (l distance) measures the worst-case deviation. In approximate sorting, we may care one measure more than the other, or both. In addition, inversion vector provides a measure of disorder in a sequence. Given a sequence v 1, v 2,..., v n and permutation π such that v π(1) < v π(2) <... < v π(n), then π(n + 1 k) is the index of the k-th largest element, and x π (n k) indicates the number of elements that are smaller than and have indices larger than of the top-k element. In particular, the position of the top-1 element is n x π (n 1). Apart from the natural sorting algorithms, Kendall tau distance has recently attracted a lot of interest in the area of error-correcting codes for Flash memory [5, 6], Kendall tau distance is also a global measure of disarray that is very popular in statistics. It is closely related to the Spearman s footrule, as we will see next. 3 Relationships between distortion measures In this section we show all four distortion measures define in Section 2.3 are closely related to each other. 3.1 Spearman s footrule and Kendall tau distance Theorem 1 (Relationship of Kendall tau distance and l 1 distance of permutation vectors [2]). Let σ 1 and σ 2 be any permutations in S n, then where σ 1 is the permutation inverse of σ. d l1 (σ 1, σ 2 )/2 d τ (σ 1 1, σ 1 2 ) d l1 (σ 1, σ 2 ), (4) 4

5 3.2 l 1 distance of inverse vectors and Kendall tau distance We show that the l 1 distance of inversion vectors and the Kendall tau distance are closely related in Theorems 2 and 3, which helps to establish the equivalence of the rate-distortion problem later. The Kendall tau distance between two permutation vectors provides upper and lower bounds to the l 1 distance between the inversion vectors of the corresponding permutations, as indicated by the following theorem. Theorem 2. Let π 1 and π 2 be any permutations in S n, then 1 n 1 d τ (π 1, π 2 ) d x,l1 (x π1, x π2 ) d τ (π 1, π 2 ) (5) Proof. See Section 5.1. Remark 2. The bound in Theorem 2 is tight as there exists π 1 and π 2 such that the equality is satisfied. For example, when n = 2m, let σ 1 = [1, 3, 5,..., 2m 3, 2m 1, 2m, 2m 2,..., 6, 4, 2], σ 2 = [2, 4, 6,..., 2m 2, 2m, 2m 1, 2m 3,..., 5, 3, 1], then d τ (σ 1, σ 2 ) = n(n 1)/2 and d x,l1 (σ 1, σ 2 ) = n/2. For another instance, let σ 1 = [1, 2,..., n 2, n 1, n], σ 2 = [2, 3,..., n 1, n, 1] then d τ (σ 1, σ 2 ) = n 1 and d x,l1 (σ 1, σ 2 ) = 1. While Theorem 2 shows in general knowing d τ (σ 1, σ 2 ) may not lead a tight lower bound to d x,l1 (σ 1, σ 2 ) due to the 1/(n 1) factor, Theorem 3 shows that it provides a tight lower bound with high probability. Theorem 3. For any π S n, let σ be a permutation chosen uniformly from S n, then for any constant c 1 < 1/2. P [c 1 d τ (π, σ) d x,l1 (π, σ)] = 1 O (1/n) (6) Proof. See Section Spearman s footrule and Chebyshev distance Let σ 1 and σ 2 be any permutations in S n, then d l1 (σ 1, σ 2 ) n d l (σ 1, σ 2 ), (7) and additionally, the scaled Chebyshev distance lower bounds the Spearman s footrule with high probability. Theorem 4. For any π S n, let σ be a permutation chosen uniformly from S n, then for any constant c 2 < 1/3. P [c 2 n d l (π, σ) d l1 (π, σ)] = 1 O (1/n) (8) Proof. See Section

6 4 Rate distortion function Theorem 5 (Equivalence of lossy source codes under different distortion measures). Below, any of the source codes of the left hand side, implies a source code of the right. 1. (n, D n ) source code for X (S n, d τ ) (n, D n ) source code for X (S n, d x,l1 ), 2. (n, D n ) source code for X (S n, d x,l1 ) (n, D n /c 1 + O (n)) source code for X (S n, d τ ) for any c 1 < 1/2, 3. (n, D n ) source code for X (S n, d l1 ) (n, D n ) source code for X (S n, d τ ). 4. (n, D n ) source code for X (S n, d τ ) (n, 2D n ) source code for X (S n, d l1 ), 5. (n, D n /n) source code for X (S n, d l ) (n, D n ) source code for X (S n, d l1 ), 6. (n, D n ) source code for X (S n, d l1 ) (n, D n /n/c 2 + O (1)) source code for X (S n, d l ) for any c 1 < 1/3. Proof. Statement 1 follow directly from (5). For statement 2, let A n {σ : c 1 d τ (σ, π) d x,l1 (σ, π), π S n }, then Theorem 3 indicates that A n = (1 O (1/n)). Let C n be the (n, D n ) source code for X (S n, d x,l1 ) and σ be a permutation chosen uniformly from S n, then for let π σ be the codeword for σ in C n, E [d τ (π, σ)] = 1 d τ (σ, π σ ) = 1 d τ (σ, π σ ) + d τ (σ, π σ ) σ S n σ A n σ S n\a n 1 d x,l1 (σ, π σ ) /c 1 + n 2 /2 σ A n σ S n\a n D n /c 1 + O (1/n) (n 2 ) = D n /c 1 + O (n). Statement 3 and 4 follow directly from Theorem 1. Statement 5 follows from (7). For statement 6, similar to the proof for state 2, define B n {σ : c 2 n d l (σ, π) d l1 (σ, π), π S n }, then by Theorem 4, E [d l (π, σ)] = 1 d l (σ, π σ ) = 1 d l (σ, π σ ) + d l (σ, π σ ) σ S n σ Bn σ S n\b n 1 d l1 (σ, π σ ) + σ Bn σ S n\b n n D n /n/c 2 + O (1/n) n = D n /n/c 2 + O (1). 6

7 We obtain Theorem 6 as a direct consequence of Theorem 5. Theorem 6 (Rate distortion function for all distortion measures). For permutation spaces X (S n, d x,l1 ), X (S n, d τ ), and X (S n, d l1 ), { 1 if D = O (n) R(D) = 1 δ if D = Θ ( n 1+δ), 0 < δ 1. (9) For permutation space X (S n, d l ), { 1 if D = O (1) R(D) = 1 δ if D = Θ ( n δ), 0 < δ 1. (10) Proof. See [3] for the rate-distortion functions for X (S n, d x,l1 ) and X (S n, d τ ). The rest follows from Theorem Code design implications Theorem 5 indicates that the lossy compression scheme for all the distortion measures in this paper can be used interchangeably, after scaling the distortion properly. In particular, under average-case distortion, one can use the code for permutation space X (S n, d x,l1 ) to permutation spaces X (S n, d τ ), X (S n, d l1 ) and X (S n, d l ). This is beneficial due to the simplicity of the compression algorithm in X (S n, d x,l1 ), which is simply component-wise scalar quantization, as shown in [3]. In this scheme, for the (k 1)-th component, (k = 2,, n), we quantize k points in [0 : k 1], uniformly with m = k/(2d k + 1) points, leading to maximal distortion D k = kd/(n + 2) 2, overall distortion and log of codebook size 5 Proofs log M n = D n = n D k = k=2 (n 1)(n + 2)D (n + 2) 2 D, n (n + 2) 2 log m k n log = (1 δ)n log n + O (n). 2D k=2 5.1 Proofs for Theorem 2 Given two permutations π 1 and π 2, we can find their corresponding inversion vector x π1 and x π2. It is known that [7, Lemma 4] d l1 (x π1, x π2 ) d τ (π 1, π 2 ). Below we first show that in the worst case d l1 (x π1, x π2 ) cannot be tightly lower bounded by d τ (π 1, π 2 ) in Theorem 2 via Lemma 7, but then with high probability the two are with the same order in Theorem 3. 7

8 Lemma 7. For any two permutations π, σ in S n such that d x,l1 (π, σ) = 1, d τ (π, σ) n 1. Proof. Let x π = [a 2, a 3,..., a n ] and x σ = [b 2, b 3,..., b n ], then without loss of generality, we have for a certain 2 k n, { b i i k a i = b i + 1 i = k. Let π and σ be permutations in S n 1 with element k removed from π and σ correspondingly, then x π = x σ, and hence π = σ. Therefore, the Kendall tau distance between σ and π is determined only by the location of element k in σ and π, which is at most n 1. Proof. The proof of [7, Lemma 4] indicates that for any two permutation π 1 and π 2 with k = d x,l1 (π 1, π 2 ), let σ 0 π 1 and σ k π 2, then there exists a sequence of permutations σ 1, σ 2,..., σ k 1 such that d x,l1 (σ i, σ i+1 ) = 1, 0 i k 1. Then k 1 d τ (π 1, π 2 ) = d τ (σ i, σ i 1 ) (a) i=0 k 1 (n 1) = (n 1)d x,l1 (π 1, π 2 ), i=0 where (a) is due to Lemma Proofs for Theorem 3 To prove Theorem 3, we analyze the mean and variance of the Kendall tau distance and l 1 distance of inversion vectors between a permutation in S n and a randomly selected permutation, in Lemma 9 and Lemma 10 respectively. We first show an obvious fact. Lemma 8. Let σ be a permutation chosen uniformly from S n, then x σ (i) Unif ([0 : i]), 1 i n 1. Proof. Obvious. Lemma 9. For any π S n, let σ be a permutation chosen uniformly from S n, and X τ d τ (π, σ), then E [X τ ] = n(n 1), Var [X τ ] = 4 n(2n + 5)(n 1). 72 Proof. Let σ be another permutation chosen independently and uniformly from S n, then we have both πσ 1 and σ σ 1 are uniformly distributed over S n. Note that Kendall tau distance is right-invariant [8], then d τ (π, σ) = d τ (πσ 1, e) and d τ (σ, σ) = d τ (σ σ 1, e) are identically distributed, and hence the result follows [2, Table 1] and [4, Section 5.1.1]. 8

9 Lemma 10. For any π S n, let σ be a permutation chosen uniformly from S n, and X x,l1 d x,l1 (π, σ), then E [X x,l1 ] > n(n 1), Var [X x,l1 ] < 8 (n + 1)(n + 2)(2n + 3). 6 Proof. By Lemma 8, we have X x,l1 = n 1 i=1 a i U i, where U i Unif ([0 : i]) and a i x π (i). Let V i = a i U i, m 1 = min {i a i, a i } and m 2 = max {i a i, a i }, then 1/(i + 1) d = 0 2/(i + 1) 1 d m 1 P [V i = d] = 1/(i + 1) m d m 2 0 otherwise. Hence m 1 E [V i ] = d 2 i d=1 m2 d=m 1 +1 d 1 i = 2(i + 1) [2(1 + m 1)m 1 + (m 2 + m 1 + 1)(m 2 m 1 )] = ( ) 1 (m1 + m 2 ) 2 i(i + 2) + i = 2(i + 1) 2 4(i + 1) > i 4, Var [V i ] E [ ] Vi 2 2 i d 2 (i + 1) 2. i + 1 Then d=0 n 1 E [X x,l1 ] = E [V i ] > i=1 n 1 Var [X x,l1 ] = Var [V i ] < i=1 n(n 1), 8 (n + 1)(n + 2)(2n + 3) (i + 1) (m2 1 + m i) With Lemma 9 and Lemma 10, now we show that the event that a scaled version of the Kendall tau distance is larger than the l 1 distance of inversion vectors is unlikely. Proof for Theorem 3. Let c 1 = 1/3, let t = n2, then noting 7 by Chebyshev s inequality, t = E [c X τ ] + Θ ( n ) Std [Xτ ] = E [X x,l1 ] Θ ( n ) Std [Xx,l1 ], P [c X τ > X x,l1 ] P [c X τ > t] + P [X x,l1 < t] O (1/n) + O (1/n) = O (1/n). The general case of c 1 < 1/2 can be proved similarly. 9

10 5.3 Proof for Theorem 4 Lemma 11. For any π S n, let σ be a permutation chosen uniformly from S n, and X l1 d l1 (π, σ), then E [X l1 ] = n2 3 + O (n), Var [X l 1 ] = 2n O ( n 2). Proof. See [2, Table 1]. Proof for Theorem 4. For any c > 0, cn d l (π, σ) cn(n 1), and for any c 2 < 1/3, Lemma 11 and Chebyshev s inequality indicate Therefore, P [d l1 (π, σ) < c 2 n(n 1)] = O(1/n). P [d l1 (π, σ) c 2 n d l (π, σ)] P [d l1 (π, σ) c 2 n(n 1)] = 1 P [d l1 (π, σ) < c 2 n(n 1)] = 1 O (1/n). References [1] Joachim Giesen, Eva Schuberth, and Milo Stojakovi. Approximate sorting. In Jos R. Correa, Alejandro Hevia, and Marcos Kiwi, editors, LATIN 2006: Theoretical Informatics, volume 3887, pages Springer, Berlin, Heidelberg, [2] Persi Diaconis and R. L. Graham. Spearman s footrule as a measure of disarray. Journal of the Royal Statistical Society. Series B (Methodological), 39(2): , [3] Da Wang, Arya Mazumdar, and Gregory W. Wornell. A rate-distortion theory for permutation spaces. In Proc. IEEE Int. Symp. Inform. Th. (ISIT), pages , [4] Donald E. Knuth. Art of Computer Programming, Volume 3: Sorting and Searching. Addison-Wesley Professional, 2 edition, [5] Anxiao Jiang, M. Schwartz, and J. Bruck. Error-correcting codes for rank modulation. In Proc. IEEE Int. Symp. Inform. Th. (ISIT), pages , [6] A. Barg and A. Mazumdar. Codes in permutations and error correction for rank modulation. IEEE Trans. Inf. Theory, 56(7): , [7] A. Mazumdar, A. Barg, and G. Zemor. Constructions of rank modulation codes. IEEE Trans. Inf. Theory, 59(2): , [8] Michael Deza and Tayuan Huang. Metrics on permutations, a survey. Journal of Combinatorics, Information and System Sciences, 23: ,