SYMMETRIC SUBGROUP ACTIONS ON ISOTROPIC GRASSMANNIANS

Transcription

1 1 SYMMETRIC SUBGROUP ACTIONS ON ISOTROPIC GRASSMANNIANS HUAJUN HUANG AND HONGYU HE Abstract. Let G be the group preserving a nondegenerate sesquilinear form B on a vector space V, and H a symmetric subgroup of G of the type G 1 G. We explicitly parameterize the H-orbits in Gr G (r, the Grassmannian of r-dimensional isotropic subspaces of V, by a complete set of H-invariants. We describe the Bruhat order in terms of the majorization relationship over a diagram of these H-invariants. The inclusion order, the stabilizer, the orbit dimension, the open H-orbits, the decompositions of an H orbit into H G 0 and H 0 orbits are also explicitly described. 1. Introduction The symmetric subgroup orbits in flag manifolds have been extensively studied. Their parametrization, in the most general form, is due to Matsuki [10, 11, 1, 13] and Springer [18]. There are finitely many such orbits. In addition, there is a natural topological ordering, called the Bruhat order, among the orbits: Orbits O O if the Zariski closure of O contains O. The Bruhat order can be described purely algebraically in terms of the Matsuki-Springer parameter [11, 16, 17, 6]. In this paper, we thoroughly characterize one family of symmetric subgroup actions on the Grassmannians of isotropic subspaces by complete sets of invariants, and we describe their Bruhat orders by majorization relationships over diagrams of these invariants. Consider the following symmetric pairs. Table 1. Symmetric pairs (G, H and representation spaces V G H V Conditions Sp n (F Sp m (F Sp n m (F F m F n m 0 < m < n O n (F O m (F O n m (F F m F n m 0 < m < n O(p, q O(p 1, q 1 O(p p 1, q q 1 R p 1+q 1 R p p 1+q q 1 0 < p 1 < p, 0 < q 1 < q U(p, q U(p 1, q 1 U(p p 1, q q 1 C p 1+q 1 C p p 1+q q 1 0 < p 1 < p, 0 < q 1 < q Note: F is an infinite field; F is the algebraic closure of F. 1 This paper is partially supported by an NSF grant. Keywords: Grassmannian, symmetric subgroup, sesquilinear form, invariant, stabilizer, Bruhat order, inclusion order 1

2 HUAJUN HUANG AND HONGYU HE In Table 1, V is a vector space equipped with a nondegenerate sesquilinear form B; G is the subgroup of GL(V preserving the form B; H is a symmetric subgroup of G stabilizing two subspaces U and W of V, where V = U W and U W with respect to form B. A subspace S of V is said to be isotropic if S S with respect to form B. Let Gr G (r denote the projective variety of r-dimensional isotropic subspaces of V. Gr G (r is called an isotropic Grassmannian or simply Grassmannian. This paper focuses on the H-action in Gr G (r for the triples (G, H, V in Table 1. The symplectic case (G, H = (Sp n (F, Sp m (F Sp n m (F has been carefully investigated [14, 15]. Let S be an isotropic subspace of V = U W. In [14], P. Rabau and D. S. Kim construct an integral 4-tuple (1.1 (dim(s U, dim(s W, dim S (P US P W S (S U (S W, 1 dim S S (P U S P W S to parameterize the H-orbit of S and to determine the partial order induced by the inclusion of isotropic subspaces. In [15], P. Rabau uses the 4-tuple to describe the stabilizer of S under the H-action and the codimension of an H-orbit. In this paper, we remove the fraction 1 from the last term of (1.1 and denote the resulting 4-tuple by (r U (S, r W (S, a(s, b(s. Then we determine the Bruhat order in terms of the majorization relationship over a diagram of r U (S, r W (S, a(s and b(s. In Section, we give a general discussion of H\Gr G (r for the triples (G, H, V in Table 1. The first main result is a parametrization of the H-orbits by a finite convex subset of an integral lattice. Theorem 1.1. Let (G, H, V be given as in Table 1. (1 If G = Sp n (F or O n (F, then the H-orbits in Gr G (r can be parameterized by an integral 4-tuple of H-invariants (r U, r W, a, b defined by (.7. ( If G = O(p, q or U(p, q, then the H-orbits in Gr G (r can be parameterized by an integral 5-tuple of H-invariants (r U, r W, a, b U, b W, where b U and b W are defined in (5. and (6.. In general, the H-orbit of S Gr G (r is isomorphic to H/H S, where H S is the stabilizer of S in the H-action. The structure of H S is determined by Theorem.3. The dimension of H S is given by Corollary.4. The second main result we carry out is an explicit description of the Bruhat order in terms of our H-invariants (r U, r W, a, b and (r U, r W, a, b U, b W. The Bruhat order follows a simple majorization relation over diagrams of these H-invariants. Theorem 1.. Consider the Bruhat order over H\Gr G (r for (G, H, V in Table 1.

3 SYMMETRIC SUBROUP ACTIONS ON ISOTROPIC GRASSMANNIANS 3 (1 When G = Sp n (F or O n (F, we make the following diagram: (1. b a r U r W The H-orbit parameterized by (r U, r W, a, b is greater than the H-orbit parameterized by (r U, r W, a, b in the Bruhat order if and only if: b b, a + b a + b, r U + a + b r U + a + b, r W + a + b r W + a + b. Note that each of b, a + b, r U + a + b and r W + a + b is the sum of all nodes connected to a given node via a descending path in diagram (1.. The inequality a + b a + b is redundant. ( When G = O(p, q or U(p, q, we make the following diagram: (1.3 b U a r U b W r W The H-orbit parameterized by (r U, r W, a, b U, b W is greater than the H-orbit parameterized by (r U, r W, a, b U, b W in the Bruhat order if and only if: b U b U, b W b W, a + b U + b W a + b U + b W, r U + a + b U + b W r U + a + b U + b W, r W + a + b U + b W r W + a + b U + b W. Note that each of b U, b W, a + b U + b W, r U + a + b U + b W and r W + a + b U + b W is the sum of all nodes connected to a given node via a descending path in diagram (1.3. The inequality a + b U + b W a + b U + b W is redundant. Therefore, in terms of the Bruhat order, the H-invariants we use provide the most natural way to describe the H-action in Gr G (r. We also determine the inclusion order over the H-orbits of all isotropic subspaces. Two H-orbits O and O have the inclusion order O O if there exist S O and S O such that S S. The third main result of this paper is an extension of P. Rabau and D. S. Kim s work on the inclusion order of symplectic case [14, Theorem 4.3] to the inclusion orders of the other cases. See Theorems 4.6, 5.6 and 6.5. When G = O(C or G = O(p, q, both G and H are disconnected. Let G 0 be the identity component of G. We illustrate in Sections 4 and 5 how an H-orbit in Gr G (r decomposes into (H G 0 and H 0 orbits. Our view point is purely algebraic. We derive our theorem by analyzing the simultaneous isometry of a set of subspaces using the tools presented in [9, Theorem 5.3]. It is unclear yet how our parametrization should be identified with the Matsuki-Springer parametrization [1, 10, 11, 18].

4 4 HUAJUN HUANG AND HONGYU HE The H-orbits on isotropic Grassmannians play an important role in explicit construction of automorphic L-functions []. The main motivation of this paper comes from representation theory. Recall that functions on isotropic Grassmannian Gr G (r can be used to define certain degenerate principal series I P (v. The representation I P (v is one of the most intensively studied series of representations. In case G/P is the Lagrangian Grassmannian and G the symplectic group, a preliminary investigation by the second author gives a branching law for the unitary I P (v H [4, 5]. This branching law is multiplicity free and yields a Howe type L -correspondence [7, 8] between certain unitary representations of G 1 and certain unitary representations of G. So the remaining question is to see if the degenerate principal series in other cases will decompose in a similar fashion when restricted to H. A first step is to understand how H acts in Gr G (r, in particular how H acts on the open orbits in Gr G (r. The question of the structure of the open orbits is answered in Corollaries 3.4, 4.4, 5.4, 6.4. Isotropic Grassmanian is a special case of partial flag variety. Another interesting example of symmetric group action on flag variety is the real semisimple group action on complex flag variety [0]. In this case, one often gets a finite number of open orbits and the structure of these open orbits has broad implications in representation theory..1. Settings. Let. Preliminary (.1 N 0 := {0} N = {0, 1,, 3, }. Let F be an infinite field, and V a vector space over F equipped with a nondegenerate sesquilinear form B. Denote the orthogonal direct sum V = U W if V = U W as vector spaces and U W with respect to the form B, that is, B(u, w = 0 for any u U and w W. Suppose dim V = n. Let G(V or G(n denote the isometry group of V that preserves B: G(V = G(n := {g GL F (V B(g(v, g(v = B(v, v for v, v V }. A symmetric pair (G, H in Table 1 has the form (G(V, G(U G(W for certain sesquilinear form B and certain decomposition V = U W. For a fixed form B, let Gr G (r be the Grassmannian of r-dimensional isotropic subspaces of V... H-invariants in Gr G (r. Define the radical of a subspace S of V by (. Rad(S := S S = {v v S, B(v, v = 0 for any v S}. So S is isotropic if and only if Rad(S = S. A flag of a vector space is a nested sequence of subspaces.

5 SYMMETRIC SUBROUP ACTIONS ON ISOTROPIC GRASSMANNIANS 5 Now suppose S is an r-dimensional isotropic subspace of V, namely S Gr G (r. It induces a flag F U (S of U and a flag F W (S of W, respectively: (.3 (.4 F U (S : F W (S : {0} S U P U S U (S U U U. {0} S W Rad(P W S P W S Rad(P W S W (S W W W. Here P U S := (S + W U is the projection of S onto the U-component with respect to the decomposition V = U W. Likewise for P W S. The symmetric subgroup H = G(U G(W of G consists of elements of GL(V that preserve the form B and stabilize the subspaces U and W. If h(s = S for h H and S, S Gr G (r, then h sends each subspace in F U (S (resp. F W (S bijectively to its counterpart in F U (S (resp. F W (S. So the dimensions of subspaces in the flags F U ( and F W ( are H-invariants. For u P U S, let ū denote the element u + in has a nondegenerate sesquilinear form B induced by B: (.5 B(ū, ū := B(u, u, for ū, ū P U S. Then P U S. P U S P Similarly, W S Rad(P W has a nondegenerate sesquilinear form (also denoted by B induced S by B. Two vector spaces equipped with sesquilinear forms, L 1 with form B 1 and L with form B, are called isometric, if there exists a linear bijection φ : L 1 L, called an isometry, such that B 1 (u, u ( = B (φ(u, φ(u for any u, u L 1. With this P notation, the isometry class of U S, B Rad(P U is H-invariant since H preserves B. S If u + w, u + w S such that u, u U and w, w W, then (.6 B(u, u = B(w, w. It immediately implies the following lemma. ( P Lemma.1. The isometry class of W S, B Rad(P W is the additive inverse of the isometry class of U S S ( P Rad(P U., B S ( For example, if G P U S ( O(p, q, then G P W S Rad(P W S O(q, p.

6 6 HUAJUN HUANG AND HONGYU HE We are now ready to present a complete set of H-invariants in Gr G (r for symmetric pairs (G, H in Table 1. Denote (.7a (.7b (.7c (.7d r U (S := dim(s U, r W (S := dim(s W, a(s := dim Rad(P US S U b(s := dim P US = dim = dim Rad(P W S S W P W S Rad(P W S = dim = dim S (P US P W S (S U (S W, Obviously, r U (S, r W (S, a(s and b(s are nonnegative integers and (.8 r U (S + r W (S + a(s + b(s = dim S = r. S S (P U S P W S. Theorem.. Let (G, H, V be a triple in Table 1. Let r be an integer such that 0 r 1 dim V. For S Gr G(r, the ( integral 4-tuple (r U (S, r W (S, a(s, b(s defined P in (.7 and the isometry class of U S, B Rad(P U defined in (.5 form a complete set S of H-invariants that uniquely determines the H-orbit of S in Gr G (r. Proof. The H-invariant part is clear. Let us show that these H-invariants uniquely determine the H-orbit of S in Gr G (r. Let S Gr G (r satisfy that (1 ((r U (S, r W (S, a(s, b(s = ((r U (S, r W (S, a(s, b(s; P ( U S, B P Rad(P U is isometric to U S S Rad(P U., B S We explicitly construct an element of H that sends S to S. For a subspace P 1 of a vector space P, let P P 1 denote the set of subspaces P of P such that P = P 1 P. (1 According to r U (S = r U (S, select a linear bijection φ 0 : S U S U. ( Select U (S U and U Rad(P U S (S U. According to a(s = a(s, select a linear bijection φ 1 : U U. ( P (3 Select U 3 P U S. Then (U 3, B is isometric to U S Rad(P U., B S Likewise, ( select U 3 ( P U S Rad(P U S (. Then (U 3, B is isometric to P U S, B P Rad(P U. Since U S, B P S Rad(P U and U S, B S Rad(P U are isometric, we can S select an isometry φ : U 3 U 3 with respect to the form B. (4 Now according to P U S = (S U U U 3, the linear map φ := φ 0 φ 1 φ is an isometry from P U S to P U S. By Witt s extension theorem, φ can be extended to an isometry h U of U, that is, h U G(U. (5 Next, according to r W (S = r W (S, select a linear bijection ψ 0 : S W S W.

7 SYMMETRIC SUBROUP ACTIONS ON ISOTROPIC GRASSMANNIANS 7 (6 Select W 1, P W S (S W. Define a linear map ψ 1, : W 1, P W S as follow: Choose a basis {w 1,, w k } of W 1, ; for each w i, choose u i P U S such that u i + w i S; then φ(u i P U S ; choose w i P W S such that φ(u i + w i S ; define ψ 1, (w i := w i. It is routine to check that ψ 1, is a well-defined linear injection and W 1, := Im ψ 1, P W S (S W. Let ψ 1, : W 1, W 1, be the linear bijection defined by ψ 1, (w := ψ 1, (w. Then ψ 1, is an isometry by (.6. (7 Now according to P W (S = (S W W 1,, the linear map ψ := ψ 0 ψ 1, is an isometry from P W (S to P W (S. By Witt s extension theorem, ψ can be extended to an isometry h W G(W. (8 Finally, h := h U h W is an element of H that sends S to S..3. The stabilizer of S Gr G (r in the H-action. Let H S denote the stabilizer of S Gr G (r in the H-action. Then every h H S is of the form h = h U h W, where h U G(U stabilizes the subspaces in the flag F U (S defined in (.3, and h W G(W stabilizes the subspaces in the flag F W (S defined in (.4. Choose a basis B U := {û 1,, û dim U } of U such that: (1 Each of the nontrivial subspaces in F U (S, namely S U,, P U S, U, (S U U, and U, is spanned by the first few vectors of B U. P ( Note that U S and U Rad(P U are nondegenerate with respect to their S forms induced from B. We may further assume that û i P U S for i = r U + a + b + 1,, dim U r U a. Then U = P U S dim U r U a i=r U +a+b+1 Fû i. With respect to the basis B U, A 11 A A 3 A (.9 h U = A 44 A 55 A 66 Here A 11 GL ru (S(F and A 66 GL ru (S(F uniquely determine each other; A GL a(s (F and A 55 GL a(s (F uniquely determine each other;

8 8 HUAJUN HUANG AND HONGYU HE ( A 33 G A 44 G P U S ( (PU S U ;. Note that h U is in the parabolic subgroup H(S, U of G(U that preserves the flag {0} (S U U (S U U U. The G and has the Levi factor GL ru (S(F GL a(s (F G [ ] A33 0 factor of h U in H(S, U is. 0 A 44 Next we consider h W G(W. In the basis B U of U, ( Rad(PU S U a+b (S U Fû ru +i = P U S. i=1 ( Rad(PU S U For each û ru +i with i = 1,, a + b, we select a vector in P W S, denoted ŵ rw +i, such that û ru +i + ŵ rw +i S. It is easy to see that a+b (S W Fŵ rw +i = P W S. i=1 The set {ŵ rw +1,, ŵ rw +a+b} can be extended to a basis B W := {ŵ 1,, ŵ dim W } of W, such that: (1 Each of the subspaces in F W (S, namely S W, Rad(P W S, P W S, Rad(P W S W, (S W W, and W, is spanned by the first few vectors of B W. ( ŵ i P W S for i = r W + a + b + 1,, dim W r W a, so that Rad(P W S W = P W S dim W r W a i=r W +a+b+1 Then with respect to the basis B W, B 11 (.10 B B 3 B h W = B 44 B 55 B 66 Here Fŵ i. B 11 GL rw (S(F and B 66 GL rw (S(F uniquely determine each other; B GL a(s (F and B 55 GL a(s (F uniquely determine each other;

9 SYMMETRIC SUBROUP ACTIONS ON ISOTROPIC GRASSMANNIANS 9 ( B 33 G ; P W S Rad(P W S ( (PW S W Rad(P W S B 44 G ; The conditions û ru +i + ŵ rw +i S for i = 1,, a + b and h U h W imply that [ ] B B 3 = B 33 [ ] A A 3. A 33 The h W is in the parabolic subgroup H(S, W of G(W that preserves the flag {0} (S W Rad(P W S Rad(P W S W (S W W W and has the Levi factor GL rw (S(F GL a(s (F G. The G factor of h W is [ ] B B 44 ( [Rad(PW S] W Rad(P W S H S ( [Rad(PW S] W Rad(P W S Theorem.3. Let (G, H, V be a triple in Table 1 and S Gr G (r. Then h H S if and only if h = h U h W, where h U G(U and h W G(W satisfy that: (1 h U is of the form (.9 with respect to the basis B U of U. ( h W is of the form (.10 with respect to the basis B W of W. (3 h U in (.9 and h W in (.10 are subjected to the constraint: (.11 [ ] A A 3 A 33 = [ ] B B 3. B 33 Corollary.4. The dimension of the H-orbit O S of S is (.1 dim O S = dim G(U + dim G(W dim H S, and dim H S equals to [ dim G(U + dim G(W dim G 1 + dim G ( (PU S U + dim G ( Rad(PU S U ( (PW S W Rad(P W S + 1 [dim GL(S U + dim GL(S W ] dim Hom F ( Rad(PW S ] W dim G Rad(P W S ( PU S + dim G ( Rad(PU S S U, P U S Proof. It suffices to find the dimension of Lie algebra of H S. Let dim[a 3 ] denote the dimension of block A 3 when h = h U h W goes through all elements of H S. Then ( Rad(PU S dim[a 11 ] = dim GL(S U, dim[a ] = dim GL, S U ( Rad(PU S dim[a 3 ] = dim Hom F S U, P U S, ( ( PU S (PU S U dim[a 33 ] = dim G, dim[a 44 ] = dim G..

10 10 HUAJUN HUANG AND HONGYU HE The other terms can be obtained easily. By the Levi decompositions of H(S, U and H(S, W, and the constraints of h = h U h W given by Theorem.3, it is straightforward to find dim H S. 3. Symplectic Groups Let B be a nondegenerate symplectic form over V F n. Let V = U W where dim U = m and dim W = n m. Then (3.1 G = G(V Sp n (F, H = G(U G(W Sp m (F Sp n m (F. We first recall some results in [14, 15] regarding the H-invariants and the stabilizer in Gr G (r for integer r with 0 r n. Then we shall study the Bruhat order of H-orbits in Gr G (r The H-invariants in Gr G (r. For S Gr G (r, let r U (S, r W (S, a(s and b(s be defined in (.7. Theorem. verifies the following result of D. S. Kim and P. Rabau: Theorem 3.1. [14, Theorem 4.3] The Sp-type ( r U (S, r W (S, a(s, 1 b(s is a complete set of H-invariants that uniquely determines the H-orbit of S in Gr G (r. Let (r U (S, r W (S, a(s, b(s parameterize the H-orbit of S. Denote the H-orbit by O (r U (S, r W (S, a(s, b(s. The range of (r U (S, r W (S, a(s, b(s is as follow: Theorem 3.. [14, Theorem 4.3] A 4-tuple (r U, r W, a, b N 4 0 parameterizes an H-orbit in Gr G (r if and only if b is even, r U + r W + a + b = r, and (3.a (3.b r U + a + b m, r W + a + b n m. 3.. The Bruhat order of H\Gr G (r. The Bruhat order of the H-orbits in Gr G (r can be described by elementary linear algebra method. The idea is that if O Gr G (r and S is in the Zariski closure O, then for any subspace decomposition V = R L, (3.3 lim S O dim P R S dim P R S, lim dim P RS S O Rad (P R S dim P RS Rad (P R S. We make the following diagram of H-invariants for S Gr G (r: (3.4 b(s a(s r U (S r W (S

11 SYMMETRIC SUBROUP ACTIONS ON ISOTROPIC GRASSMANNIANS 11 The Bruhat order of H\Gr G (r is characterized by a majorization relationship over diagram (3.4. Define a partial order on the diagram, where nodes A A if and only if there exists a descending path from A to A. For each node A, we add the values of all nodes no less than node A. The resulting quantities are: b(s = dim of a maximal nondegenerate subspace of P U S = dim of a maximal nondegenerate subspace of P W S, a(s + b(s = dim P US P W S = dim P US S S U = dim P W S S W, r U (S + a(s + b(s = dim P U S, r W (S + a(s + b(s = dim P W S. Theorem 3.3. The Bruhat order O(r U, r W, a, b O(r U, r W, a, b holds in H\Gr G (r if and only if the following inequalities hold: (3.5a (3.5b (3.5c (3.5d b b, a + b a + b, r U + a + b r U + a + b, r W + a + b r W + a + b. The inequality (3.5b is implied by r U + r W + a + b = r = r U + r W + a + b, (3.5c and (3.5d. Proof. By (3.3, inequalities (3.5 are necessary for O(r U, r W, a, b O(r U, r W, a, b. To show that they are sufficient, we will prove several claims associate to some basic operations over the node values of diagram (3.4 that preserve the inequalities in (3.5. These basic operations allow us to change from (r U, r W, a, b to (r U, r W, a, b whenever inequalities (3.5 hold. Fix a basis {u 1,, u m } of U and a basis {w 1,, w n m } of W such that: [ B(ui, u j ] m m = [ ] m, [ B(wi, w j ] (n m (n m = [ ] (n m. Let O(r U, r W, a, b be an H-orbit. The following vectors form a basis B S of an element S of O(r U, r W, a, b: u i 1 + w i and u i + w i 1, for i = 1,, b/, u b+i + w b+i, for i = 1,, a, u b+a+i, for i = 1,, r U, w b+a+i, for i = 1,, r W. In the following arguments, let S x for x F be the subspace of V spanned by all vectors in B S but a few vectors being replaced. It will be easy to verify that: (1 The entries of the basis vectors of S x given above are polynomials of x. ( S x O(r U, r W, a, b for every x F {0}.

12 1 HUAJUN HUANG AND HONGYU HE Then S 0 is in the Zariski closure of O(r U, r W, a, b since F is an infinite field. In particular, S 0 Gr G (r. If S 0 O(r U, r W, a, b, then O(r U, r W, a, b O(r U, r W, a, b in the Bruhat order. We assume that the related H-orbits in the following claims always exist. (1 Claim: O(r U, r W, a, b O(r U, r W, a +, b. Applying (3. to O(r U, r W, a +, b, r U + a + b m 1, r W + a + b n m 1. Let S x for x F be constructed as follow: Vector in B S being replaced u b + w b 1 Replaced by vector x(u b + w b 1 + u m + w n m Then S x O(r U, r W, a, b for x F {0} and S 0 O(r U, r W, a +, b. ( Claim: O(r U, r W, a, b O(r U +, r W, a, b. Applying (3.a to O(r U +, r W, a, b, r U + a + b m 1. Let S x for x F be constructed as follow: Vectors in B S being replaced u b 1 + w b u b + w b 1 Replaced by vectors u b 1 + xw b x u b + xw b 1 + u m Then S x O(r U, r W, a, b for x F {0} and S 0 O(r U +, r W, a, b. (3 Claim: O(r U, r W, a, b O(r U, r W +, a, b. The proof is similar. (4 Claim: O(r U, r W, a, b O(r U + 1, r W + 1, a, b. Let S x for x F be constructed as follow: Vectors in B S being replaced u b 1 + w b u b + w b 1 Replaced by vectors xu b 1 + w b u b + xw b 1 Then S x O(r U, r W, a, b for x F {0} and S 0 O(r U + 1, r W + 1, a, b. (5 Claim: O(r U, r W, a, b O(r U + 1, r W, a + 1, b. Applying (3.a to O(r U + 1, r W, a + 1, b, r U + a + b m 1. Let S x for x F be constructed as follow: Vector in B S being replaced u b + w b 1 Replaced by vector x(u b + w b 1 + u m Then S x O(r U, r W, a, b for x F {0} and S 0 O(r U + 1, r W, a + 1, b. (6 Claim: O(r U, r W, a, b O(r U, r W + 1, a + 1, b. The proof is similar.

13 SYMMETRIC SUBROUP ACTIONS ON ISOTROPIC GRASSMANNIANS 13 (7 Claim: O(r U, r W, a, b O(r U + 1, r W, a 1, b. Let S x for x F be constructed as follow: Vector in B S being replaced u b+a + w b+a Replaced by vector u b+a + xw b+a Then S x O(r U, r W, a, b for x F {0} and S 0 O(r U + 1, r W, a 1, b. (8 Claim: O(r U, r W, a, b O(r U, r W + 1, a 1, b. The proof is similar. By Theorem 3., the set of 4-tuples (r U, r W, a, b that parameterize H-orbits in Gr G (r consists of the integer points in a convex set. If (r U, r W, a, b and (r U, r W, a, b parameterize two H-orbits in Gr G (r and they satisfy the inequalities in (3.5, we can find a sequence of 4-tuples (r U, r W, a, b = (r (0 U, r(0 W, a(0, b (0, (r (1 U, r(1 W, a(1, b (1, such that O(r (i 1 U, r (i 1 W, (r (d U, r(d W, a(d, b (d = (r U, r W, a, b,, a(i 1, b (i 1 O(r (i U, r(i W, a(i, b (i by one of the above claims for i = 1,, d. Then O(r U, r W, a, b O(r U, r W, a, b and the sufficient part is proved. Corollary 3.4. Let (G, H be the symplectic symmetric pair in Table 1. (1 When r < min(m, n m, the unique open H-orbit in Gr G (r is { O(0, 0, 0, r if r is even; O(0, 0, 1, r 1 if r is odd. ( When min(m, n m r n, the unique open H-orbit in Gr G (r is { O(0, r m, 0, m if m n m; O(r n + m, 0, 0, n m if m n m. Example 3.5. Let G = Sp 4m+8 (F and H = Sp m (F Sp m+8 (F. We describe the Bruhat order of the H-orbits in the maximal isotropic Grassmannian Gr G (m + 4. Here n = m + 4 and r = m + 4. By Theorem 3., (r U, r W, a, b N 4 0 parameterizes an H-orbit in Gr G (m + 4 if and only if the following constraints hold: b is even; b {0,, 4,, m}; r U + r W + a + b = m + 4; a = 0; r U + a + b m; = r U = m r W + a + b m + 4. b ; r W = m + 4 b. By Theorem 3.3, the Bruhat order of H\Gr G (m + 4 is: O(0, 4, 0, m > O(1, 5, 0, m > O(, 6, 0, m 4 > > O(m, m + 4, 0, 0.

14 14 HUAJUN HUANG AND HONGYU HE Example 3.6. Let G = Sp 8 (F and H = Sp 4 (F Sp 4 (F. Then n = 4 and m =. Consider the Bruhat order of H-orbits in Gr G (3, where r = 3. By Theorem 3., (r U, r W, a, b N 0 4 parameterizes an H-orbit in Gr G (3 if and only if b is even; r U + r W + a + b = 3; r U + a + b ; r W + a + b. So b = 0 or b =. There are 6 H-orbits in Gr G (3 parameterized by: (r U, r W, a, b {(1, 1, 1, 0, (1,, 0, 0, (, 1, 0, 0, (1, 0, 0,, (0, 1, 0,, (0, 0, 1, }. By Theorem 3.3, the Bruhat order of H\Gr G (3 is given by the following diagram: O(1, 0, 0, O(0, 0, 1, O(0, 1, 0, O(1, 1, 1, 0 O(, 1, 0, 0 O(1,, 0, The inclusion order of H-orbits. In [14], P. Rabau and D. S. Kim discuss an inclusion order on the H-orbits of isotropic subspaces in all possible dimensions, that is, on ( n H\ Gr G (r. The order is defined as follow: r=0 O(r U, r W, a, b O(r U, r W, a, b if there exist S O(r U, r W, a, b and S O(r U, r W, a, b such that S S. Obviously, this order is different from the Bruhat order, as any two distinct H-orbits on a given Gr G (r have no relation. Theorem 3.7. [14, Theorem 4.3] For symplectic pair (G, H in Table 1, two H-orbits O(r U, r W, a, b O(r U, r W, a, b if and only if r U r U, r W r W, b b, r U +a+ b r U +a + b, r W +a+ b r W +a + b. For S O(r U, r W, a, b, r U = dim(s U, b is the dimension of a maximal nondegenerate subspace of P U S, and r U + a + b is the dimension of a maximal nilpotent subspace of P U S. Similarly for r W and r W + a + b.

15 SYMMETRIC SUBROUP ACTIONS ON ISOTROPIC GRASSMANNIANS Dimensions of orbit and stabilizer. Let S O(r U, r W, a, b Gr G (r. The stabilizer H S of S under the H-action is discussed in [15, Section 5.II]. The results are verified by Theorem.3. Theorem 3.8. [15, Theorem 5.] The codimension of O(r U, r W, a, b is codim O(r U, r W, a, b = r U r W + a(r U + r W + r U (n m r W a b + r W (m r U a b ( a + ( a = 3r U r W (r U + r W (a + b + r U (n m + r W m +. By dim Gr G (r = nr 3 r + 1 r, we get the dimensions of O(r U, r W, a, b and H S : dim O(r U, r W, a, b = dim Gr G (r codim O(r U, r W, a, b, dim H S = dim H dim O(r U, r W, a, b ( ( m + 1 n m + 1 = + dim O(r U, r W, a, b. The results coincide with those of Corollary Orthogonal Groups on an Algebraically Closed Field Let F be an infinite algebraically closed field with char(f. Let B be a nondegenerate symmetric form over V F n. Suppose V = U W where dim U = m and dim W = n m. Then (4.1 G = G(V O n (F, H = G(U G(W O m (F O n m (F. We consider the H-action on the isotropic Grassmannian Gr G (r for 0 r n/ The H-invariants in Gr G (r. Let S Gr G (r. Then P U S has a nondegenerate symmetric form B defined in (.5. The isometry class of ( P U S unique as F is algebraically closed. Theorem. implies the following result., B is Theorem 4.1. The 4-tuple (r U (S, r W (S, a(s, b(s defined in (.7 is a complete set of H-invariants that determines the H-orbit of S in Gr G (r. Let (r U (S, r W (S, a(s, b(s parameterize the H-orbit of S in Gr G (r, and denote the H-orbit by O (r U (S, r W (S, a(s, b(s. The next theorem determines the range of this 4-tuple. It is similar to Theorem 3.. The proof is skipped. Theorem 4.. A 4-tuple (r U, r W, a, b N 0 4 parameterizes an H-orbit in Gr G (r if and only if: (1 r U + r W + a + b = r.

16 16 HUAJUN HUANG AND HONGYU HE (4.a (4.b ( The following inequalities hold: r U + a + b m, r W + a + b n m. Let {u 1,, u m } and {w 1,, w n m } be an orthonormal basis of U and W, respectively. Choose i F such that i + 1 = 0. The following vectors span a subspace of O(r U, r W, a, b provided that (r U, r W, a, b satisfies the conditions in Theorem 4.: u j + iw j, j = 1,, b; u b+j 1 + iu b+j + w b+j 1 + iw b+j, j = 1,, a; (4.3 u b+a+j 1 + iu b+a+j j = 1,, r U ; w b+a+j 1 + iw b+a+j j = 1,, r W. 4.. The Bruhat order of H\Gr G (r. The Bruhat order of the H-orbits in Gr G (r is similar to that of the symplectic case. We make the following diagram of H-invariants: (4.4 b(s a(s r U (S r W (S The Bruhat order of H\Gr G (r is characterized by the majorization relationship over the diagram. For each node A in the diagram, we define a quantity by adding the values of all nodes connected to node A via descending paths. So we get b(s, a(s + b(s, r U (S + a(s + b(s, and r W (S + a(s + b(s. Theorem 4.3. The Bruhat order O(r U, r W, a, b O(r U, r W, a, b holds in Gr G (r if and only if they satisfy the following inequalities: (4.5a (4.5b (4.5c (4.5d b b, a + b a + b, r U + a + b r U + a + b, r W + a + b r W + a + b. The inequality (4.5b is implied by (4.5c and (4.5d. The proof is similar to that of Theorem 3.3 and we omit it here. Corollary 4.4. (1 If r min{m, n m}, the unique open H-orbit in Gr G (r is O(0, 0, 0, r. ( If min{m, n m} r n/, the unique open H-orbit in Gr G (r is { O(0, r m, 0, m if dim U dim W, O(r n + m, 0, 0, n m if dim U > dim W.

17 SYMMETRIC SUBROUP ACTIONS ON ISOTROPIC GRASSMANNIANS 17 Example 4.5. Let G = O 8 (F and H = O 4 (F O 4 (F. Then n = 8 and m = 4. Consider the Bruhat order of H\Gr G (3, where r = 3. By Theorem 4., (r U, r W, a, b N 0 4 parameterizes an H-orbit in Gr G (3 if and only if r U + r W + a + b = 3; r U + a + b 4; r W + a + b 4. Adding the last two inequalities and using the first equality, we have a 1. possible 4-tuples (r U, r W, a, b are: (, 1, 0, 0, (1,, 0, 0, (1, 1, 0, 1, (1, 0, 0,, (0, 1, 0,, (1, 1, 1, 0, (0, 0, 1,. By Theorem 4.3, the Bruhat order of H\Gr G (3 is given by the following diagram: O(0, 0, 1, O(1, 0, 0, O(0, 1, 0, O(1, 1, 0, 1 O(1, 1, 1, 0 O(, 1, 0, 0 O(1,, 0, 0 Comparing with the case G = Sp 8 (F, H = Sp 4 (F Sp 4 (F and r = 3 in Example 3.6, there is one additional orbit in the orthogonal case The inclusion order of H-orbits. The inclusion order O(r U, r W, a, b O(r U, r W, a, b holds if there exist S O(r U, r W, a, b and S O(r U, r W, a, b such that S S. This order for the case (G, H = (O n (F, O m (F O n m (F is determined by the same inequalities as in the symplectic case (cf. Theorem 3.7. Theorem 4.6. The inclusion order of two H-orbits O(r U, r W, a, b O(r U, r W, a, b holds if and only if (4.6 r U r U, r W r W, b b, r U + a + b r U + a + b, r W + a + b r W + a + b. Proof. Suppose O(r U, r W, a, b O(r U, r W, a, b. Let S O(r U, r W, a, b and S O(r U, r W, a, b satisfy that S S. Then S U S U; Every maximal nondegenerate subspace of P U S is contained in a maximal nondegenerate subspace of P U S. Every minimal nondegenerate subspace of U that contains P U S contains a minimal nondegenerate subspace of U that contains P U S ; Similar arguments hold on the W component. The

18 18 HUAJUN HUANG AND HONGYU HE By homogeneity property of orbits, we may assume that S is spanned by the vectors in (4.3. Taking dimensions, we get inequalities (4.6. Conversely, suppose (r U, r W, a, b and (r U, r W, a, b satisfy inequalities (4.6. Let S O(r U, r W, a, b be spanned by the vectors in (4.3. If a a, it is easy to find a subspace S S such that S O(r U, r W, a, b. Otherwise, a < a. By a reduction process, we may assume that r U = 0, r W = 0, b = 0 and a = 0. Then (4.6 implies that min{r U, r W } + b a. Again, it is easy to find a subspace S of S such that S O(r U, r W, a, b = O(0, 0, a, Dimensions of orbit and stabilizer. Given S O(r U, r W, a, b, Theorem.3 and Corollary.4 provide the structure of H S as well as the dimensions of O(r U, r W, a, b and H S. Theorem 4.7. For S O(r U, r W, a, b, dim O(r U, r W, a, b = dim H dim H S = ( m + ( n m dim H S, and dim H S equals to [( ( ( ( ] 1 m n m m + + ru + rw ru a n m rw a (4.7 ( ( ( b m ru a b n m rw a b + ab Decompose an H-orbit into H G 0 and H 0 orbits. When F = C, the identity component of G and H is, respectively: G 0 = SO n (F, H 0 = SO m (F SO n m (F. Let I n be the n n identity matrix. Denote the matrices (4.8 I + n := I n and I n := I n 1 ( I 1. The group G has two connected components, namely G 0 and In G 0. For t 1, t {+, }, let H t 1 t denote the connected component of I t 1 m I t n m in H. Then H decomposes into two (H G 0 -cosets and four H 0 -cosets as follow: H H G 0 H+(H G 0 H + + H H+ H + Theorem 4.8. An H-orbit O(r U, r W, a, b in Gr G (r always decomposes into 1 or (H G 0 -orbits. Moreover, O(r U, r W, a, b decomposes into (H G 0 -orbits if and only if both equalities in (4.a and (4.b hold, if and only if r + a = n/.

19 SYMMETRIC SUBROUP ACTIONS ON ISOTROPIC GRASSMANNIANS 19 Proof. The number of (H G 0 -orbits in the H-orbit of S O(r U, r W, a, b equals to [H : (H G 0 H S ] {1, }. So [H : (H G 0 H S ] = if and only if (H G 0 H S = H G 0, if and only if H S G 0. Theorem.3 implies that the Levi factor of H S is isomorphic to GL ru GL rw GL a G(b G(m r U a b G(n m r W a b, where for K {GL ru, GL rw, G(b}, the K-component of H is diagonally embedded in a matrix group K K, and the GL a -component is diagonally embedded in a matrix group GL a GL a GL a GL a. So H S G 0 = SO n (F if and only if (4.9 m r U a b = 0, n m r W a b = 0. These are equivalent to the equalities in (4.a and (4.b. It remains to prove the last statement. If both equalities in (4.a and (4.b hold, the sum of these two equalities produces r + a = n/. Conversely, if r + a = n/, then both equalities in (4.a and (4.b must hold. Corollary 4.9. An open H-orbit of Gr G (r decomposes into open (H G 0 -orbits if and only if r = n/, in which n is even and Gr G (r consists of maximal isotropic subspaces of V. Theorem The (H G 0 -orbit of S O(r U, r W, a, b decomposes into 1 or H 0 -orbits. Moreover, it decomposes into H 0 -orbits if and only if b = 0 and at least one of the equalities in (4.a and (4.b holds. Proof. The number of H 0 -orbits in the (H G 0 -orbit of S equals to [H G 0 : H 0 (H G 0 S ] = [H G 0 : H 0 (H S G 0 ] {1, }. Moreover, [H G 0 : H 0 (H S G 0 ] = if and only if H 0 (H S G 0 = H 0, if and only if H S G 0 H 0. By the structure of H S described in Theorem.3, H S G 0 H 0 if and only if b = 0, and at least one of the equalities in (4. holds. Corollary An H-orbit O(r U, r W, a, b in Gr G (r decomposes into 1,, or 4 H 0 -orbits. It decomposes into 4 H 0 -orbits if and only if b = 0 and r + a = n/. 5. Real Orthogonal Groups Let V := R p+q be equipped with a nondegenerate symmetric bilinear form B of the type I p ( I q. Let V = U W be an orthogonal decomposition such that B U is of the type I p1 ( I q1 and B W is of the type I p p1 ( I q q1. Denote (5.1 G = G(V O(p, q, H = G(U G(W O(p 1, q 1 O(p p 1, q q 1. We shall discuss the H-orbits in Gr G (r for 0 r min{p, q}.

20 0 HUAJUN HUANG AND HONGYU HE 5.1. The H-invariants in Gr G (r. By Theorem., the H-orbit of S Gr G (r is determined by the H-invariants r U (S, r W (S, a(s and b(s defined in (.7 and by P the isometry class of ( U S, B. According to (.6, we can denote (5.a b U (S := the dimension of a maximal positive definite subspace of P U S = the dimension of a maximal negative definite subspace of P W S, (5.bb W (S := the dimension of a maximal negative definite subspace of P U S = the dimension of a maximal positive definite subspace of P W S. Then b(s = b U (S + b W (S and ( ( PU S PW S G = O (b U (S, b W (S, G = O (b W (S, b U (S. Rad(P W S The following result is obvious. Theorem 5.1. The 5-tuple (r U (S, r W (S, a(s, b U (S, b W (S is a complete set of H-invariants that determines the H-orbit of S in Gr G (r. We say that (r U (S, r W (S, a(s, b U (S, b W (S parameterizes the H-orbit of S, and the H-orbit is denoted by O (r U (S, r W (S, a(s, b U (S, b W (S. Theorem 5.. A 5-tuple (r U, r W, a, b U, b W N 0 5 parameterizes an H-orbit in Gr G (r if and only if the 5-tuple satisfies all constraints below: (5.3a (5.3b (5.3c (5.3d (1 r U + r W + a + b U + b W = r. ( The following conditions hold: r U + a + b U p 1, r U + a + b W q 1, r W + a + b W p p 1, r W + a + b U q q 1. Proof. First we prove the necessary part. Suppose that (r U, r W, a, b U, b W parameterizes an H-orbit in Gr G (r. Obviously, r U +r W +a+b U +b W = r. Let S + be a maximal positive definite subspace of P U S. Then every vector v S 1 := S + satisfies that (v, v 0. Let U be a maximal negative definite subspace of U. Then S 1 U = {0} and S 1 + U U. So (r U + a + b U + q 1 = dim S 1 + dim U = dim(s 1 U dim U = p 1 + q 1. This leads to (5.3a. Similarly for (5.3b, (5.3c, and (5.3d. Next we prove the sufficient part. If a 5-tuple (r U, r W, a, b U, b W meets all the constraints in Theorem 5., we find an isotropic subspace S Gr G (r whose H- orbit is parameterized by (r U, r W, a, b U, b W. Let {u + 1,, u + p 1, u 1,, u q 1 } and

21 SYMMETRIC SUBROUP ACTIONS ON ISOTROPIC GRASSMANNIANS 1 {w + 1,, w + p p 1, w 1,, w q q 1 } be an orthogonal basis of U and W, respectively, such that (5.4 B(u + i, u+ i = 1, B(u j, u j = 1, B(w+ t, w + t = 1, B(w l, w l = 1. Let S be the subspace of V spanned by the following basis vectors: { } u + i + w bu i { } u i=1 i + w + bw i { u + i=1 b U +i + u b W +i + w+ b W +i + a (5.5 w b U +i} i=1 { u + b U +a+i + } ru u b W +a+i { w + i=1 b W +a+i + } rw w b U +a+i. i=1 Then S Gr G (r and the H-orbit of S is parameterized by (r U, r W, a, b U, b W. 5.. The Bruhat order of H\Gr G (r. Construct the following diagram of H-invariants for S Gr G (r: (5.6 b U (S b W (S a(s r U (S r W (S The Bruhat order of H\Gr G (r can be characterized by the majorization relationship over this diagram. For each node A in the diagram, we define a quantity by adding the values of all nodes connected to node A via descending paths. Then we get b U (S = dim of a maximal positive definite subspace of P U S, b W (S = dim of a maximal negative definite subspace of P U S, a(s + b U (S + b W (S = dim P US P W S = dim P US S S U = dim P W S S W, r U (S + a(s + b U (S + b W (S = dim P U (S, r W (S + a(s + b U (S + b W (S = dim P W (S. Theorem 5.3. The Bruhat order O(r U, r W, a, b U, b W O(r U, r W, a, b U, b W holds in Gr G (r if and only if the following inequalities hold: (5.7a (5.7b (5.7c (5.7d (5.7e b U b U, b W b W, a + b U + b W a + b U + b W, r U + a + b U + b W r U + a + b U + b W, r W + a + b U + b W r W + a + b U + b W. Inequality (5.7c is implied by (5.7d and (5.7e. Proof. The proof is similar to that of Theorem 3.3. The necessary part is obvious. It remains to prove the sufficient part. We will prove some claims associate to several basic operations on the node values of diagram (5.6 that preserve the order defined

22 HUAJUN HUANG AND HONGYU HE by inequalities (5.7. These operations allow us to change from (r U, r W, a, b U, b W to (r U, r W, a, b U, b W whenever inequalities (5.7 hold. Fix an orthogonal basis {u + 1,, u + p 1, u 1,, u q 1 } of U and an orthogonal basis {w 1 +,, w p p + 1, w1,, wq q 1 } of W that satisfy (5.4. Let (r U, r W, a, b U, b W be a 5-tuple that meets the constraints in Theorem 5.. In the following argument, let S x for x R be the subspace of V spanned by all vectors in (5.5 but one vector being replaced. It will be obvious that: (1 The entries of the basis vectors of S x given above are polynomials of x. ( S x O(r U, r W, a, b U, b W for every x R +. Then S 0 is in the Zariski closure of O(r U, r W, a, b U, b W. If S 0 O(r U, r W, a, b U, b W, then O(r U, r W, a, b U, b W O(r U, r W, a, b U, b W in the Bruhat order. (1 Claim: O(r U, r W, a, b U, b W O(r U, r W, a + 1, b U 1, b W. Applying (5.3b and (5.3c to (r U, r W, a + 1, b U 1, b W, r U + a + b W q 1 1, r W + a + b W p p 1 1. Let S x be constructed as follow: Vector in (5.5 being replaced u + b U + w b U Replaced by the vector (1 + xu + b U + u q 1 + w p p (1 + xw b U Then S x O(r U, r W, a, b U, b W for x R + and S 0 O(r U, r W, a + 1, b U 1, b W. ( Claim: O(r U, r W, a, b U, b W O(r U + 1, r W, a, b U 1, b W. Applying (5.3b to (r U + 1, r W, a, b U 1, b W, Let S x be constructed as follow: Vector in (5.5 being replaced u + b U + w b U r U + a + b W q 1 1. Replaced by the vector (1 + x u + b U + (1 x u q 1 + xw b U Then S x O(r U, r W, a, b U, b W for x R + and S 0 O(r U + 1, r W, a, b U 1, b W. (3 Claim: O(r U, r W, a, b U, b W O(r U, r W +1, a, b U 1, b W. The proof is similar. The next three claims are similar to the above three: (4 Claim: O(r U, r W, a, b U, b W O(r U, r W, a + 1, b U, b W 1. (5 Claim: O(r U, r W, a, b U, b W O(r U + 1, r W, a, b U, b W 1. (6 Claim: O(r U, r W, a, b U, b W O(r U, r W + 1, a, b U, b W 1. (7 Claim: O(r U, r W, a, b U, b W O(r U + 1, r W, a 1, b U, b W. Let S x be constructed as follows: Vector in (5.5 being replaced Replaced by the vector u + b U +a + u b W +a + w+ b W +a + w b U +a u + b U +a + u b W +a + x(w+ b W +a + w b U +a

23 SYMMETRIC SUBROUP ACTIONS ON ISOTROPIC GRASSMANNIANS 3 Then S x O(r U, r W, a, b U, b W for x R + and S 0 O(r U + 1, r W, a 1, b U, b W. (8 Claim: O(r U, r W, a, b U, b W O(r U, r W +1, a 1, b U, b W. The proof is similar. These claims associate to some basic operations on the node values of diagram (5.6 that preserve the order defined by inequalities (5.7. The last part is similar to that of the proof of Theorem 3.3. Theorem 5.3 together with Corollary 5.8 implies the following result: Corollary 5.4. (1 When r < min{p 1, q q 1 } + min{q 1, p p 1 }, the open H-orbits in Gr G (r are not unique, and they are given by: O(0, 0, 0, b U, b W, where b U + b W = r, 0 b U min{p 1, q q 1 } and 0 b W min{q 1, p p 1 }. ( When min{p 1, q q 1 } + min{q 1, p p 1 } r min{p, q}, there is a unique open H-orbit in Gr G (r given by: { O(r b U b W, 0, 0, b U, b W if dim U > dim W, O(0, r b U b W, 0, b U, b W if dim U dim W, where b U = min{p 1, q q 1 } and b W = min{q 1, p p 1 }. Example 5.5. Consider the case G = O(5, 5 and H = O(, 3 O(3,. Then p = 5, q = 5, p 1 =, q 1 = 3, p p 1 = 3, q q 1 =. Let r = 4. We consider the Bruhat order of H-orbits in Gr G (4. By Theorem 5., an H-orbit O(r U, r W, a, b U, b W satisfies that: r U + r W + a + b U + b W = 4, r U + a + b U, r U + a + b W 3, r W + a + b W 3, r W + a + b U. Then (r U, r W, a, b U, b W could be one of the following 5-tuples: (0, 0, 0, 1, 3, (0, 0, 0,,, (0, 1, 0, 1,, (1, 0, 0, 1,, (1, 1, 0, 0,, (1,, 0, 0, 1, (, 1, 0, 0, 1, (1, 1, 0, 1, 1, (,, 0, 0, 0.

24 4 HUAJUN HUANG AND HONGYU HE By Theorem 5.3, we obtain the following Bruhat order of H\Gr G (4: O(0, 0, 0,, O(1, 0, 0, 1, O(0, 0, 0, 1, 3 O(0, 1, 0, 1, O(1, 1, 0, 0, O(1, 1, 0, 1, 1 O(, 1, 0, 0, 1 O(1,, 0, 0, 1 O(,, 0, 0, 0 There are two open H-orbits in this case The inclusion order of H-orbits. The inclusion partial order for real orthogonal case is determined as follow: Theorem 5.6. There exist S O(r U, r W, a, b U, b W and S O(r U, r W, a, b U, b W such that S S if and only if the following inequalities hold: r U r U, r W r W, b U b U, b W b W, r U + a + b U r U + a + b U, r U + a + b W r U + a + b W, r W + a + b U r W + a + b U, r W + a + b W r W + a + b W. The theorem can be proved by a similar reduction process as in the proof of Theorem Dimensions of orbit and stabilizer. Theorem.3 characterizes the structure of the stabilizer H S of S O(r U, r W, a, b U, b W Gr G (r. Corollary.4 implies the following result. Theorem 5.7. For S O(r U, r W, a, b U, b W, ( p1 + q 1 dim O(r U, r W, a, b U, b W = dim H dim H S = + ( p p1 + q q 1 dim H S, and dim H S equals to ( 1 p1 + q ( p + q p1 q 1 + r U + ( r W bu + b W + a(b U + b W 1 ( p1 + q 1 r U a 1 ( p + q p1 q 1 r W a (5.8 ( ( p1 + q 1 r U a b U b W p + q p1 q 1 r W a b U b W + +.

25 SYMMETRIC SUBROUP ACTIONS ON ISOTROPIC GRASSMANNIANS 5 Formula (5.8 is similar to formula (4.7, because dim R O(p, q = dim C O p+q (C and dim R GL a (R = dim C GL a (C. A direct consequence of (5.8 is the following corollary. Corollary 5.8. If both O(r U, r W, a, b U, b W and O(r U, r W, a, b U, b W exist and b U + b W = b U + b W, then dim O(r U, r W, a, b U, b W = dim O(r U, r W, a, b U, b W Decompose an H-orbit into H G 0 and H 0 orbits. We assume that p, q, p 1, q 1, p p 1, q q 1 > 0 for simplicity. Denote the matrices I + n := I n and I n := I n 1 ( I 1. Then G = O(p, q has 4 connected components, namely the G 0 -cosets of I t 1 p I t q for t 1, t {+, }. In particular, SO(p, q = G 0 (I p I q G 0. Similarly, H = O(p 1, q 1 O(p p 1, q q 1 has 16 connected components, denoted by H t 1t t 3 t 4 := ( I t 1 p 1 I t q 1 I t 3 p p 1 I t 4 q q 1 H0 for t 1, t, t 3, t 4 {+, }. Obviously, H = H 0. Then H and H G 0 decompose into cosets as follow: (5.9 H H G 0 H ++(H G 0 H ++(H + G 0 H++(H + G 0 H 0 H H + + H + + Fix an orthogonal basis {u + 1,, u + p 1, u 1,, u q 1 } {w + 1,, w + p p 1, w 1,, w q q 1 } of V that satisfies (5.4. Let S O(r U, r W, a, b U, b W be the canonical subspace spanned by the vectors in (5.5. The number of (H G 0 -orbits in the H-orbit of S equals to [H : (H G 0 H S ] {1,, 4}. In fact, [H : (H G 0 H S ] = 4/m, where m is the number of (H G 0 -cosets of H that intersect H S. The number m can be determined by (r U, r W, a, b U, b W as follows. Recall the constraints (5.3: r U + a + b U p 1, r U + a + b W q 1, r W + a + b W p p 1, r W + a + b U q q 1. Lemma 5.9. The following statements hold: (1 H S intersects H + ++(H G 0 if and only if r U + a + b U < p 1 or r W + a + b W < p p 1. ( H S intersects H + ++(H G 0 if and only if r U + a + b W < q 1 or r W + a + b U < q q 1. (3 H S intersects H ++(H G 0 if and only if a < r.

26 6 HUAJUN HUANG AND HONGYU HE Proof. The necessary part can be done by investigating the possible sign combinations in the Levi factor of h H S. The sufficient part can be proved by the following explicit construction: (1 If r W + a + b W < p p 1, then w + p p 1 is not a component in the basis (5.5 of S. So w + p p 1 S and I p1 I q1 Ip p 1 I q q1 H S H + ++ H S [H++(H + G 0 ]. Similarly, r U + a + b U < p 1. ( The argument is similar to the preceding one. (3 If a < r, then at least one of b U, b W, r U, r W is greater than 0. Suppose b U > 0. Then S has a basis vector u + 1 +w1 in (5.5. Let L GL(V have -1 eigenspace span{u + 1, w1 } and +1 eigenspace span{u + 1, w1 }. Then L H S H+ + H S [H++(H G 0 ] since H+ + = H++H +. + Similar for the other cases. Lemma 5.9 leads to the following result. Theorem The number N of (H G 0 -orbits in an H-orbit is given below: N G H H-orbit Conditions 4 O(a, a O(a, a O(a, a O(0, 0, a, 0, 0 O(p, a O(p 1, a O(p p 1, a O(0, 0, a, 0, 0 p 1 a, p p 1 a, p > a O(a, q O(a, q 1 O(a, q q 1 O(0, 0, a, 0, 0 q 1 a, q q 1 a, q > a r U + a + b U = p 1, r W + a + b W = p p 1, O(p, q O(p 1, q 1 O(p p 1, q q 1 O(r U, r W, a, b U, b W r U + a + b W = q 1, (p = q r W + a + b U = q q 1, r U + r W + b U + b W > 0 1 all the other situations. Theorem 5.10 and Corollary 5.4 imply the following decompositions of open H- orbits into open (H G 0 -orbits. Corollary An open H-orbit in Gr G (r always decomposes into 1 open (H G 0 - orbit except for the case p = q = r, in which the unique open H-orbit { O(0, r p 1 q 1, 0, p 1, q 1 if p 1 + q 1 r, O(p 1 + q 1 r, 0, r q 1, r p 1 if p 1 + q 1 > r, decomposes into open (H G 0 -orbits. Proof. Let O(r U, r W, a, b U, b W be an open H-orbit in Gr G (r. By Corollary 5.4, we have a = 0 and b U + b W > 0. According to Theorem 5.10, O(r U, r W, a, b U, b W decomposes into 1 or open (H G 0 -orbits, and it decomposes into (H G 0 -orbits if and only if r U + b U = p 1, r W + b W = p p 1, r U + b W = q 1, r W + b U = q q 1.

27 SYMMETRIC SUBROUP ACTIONS ON ISOTROPIC GRASSMANNIANS 7 These together with r = r U + r W + b U + b W Corollary 5.4 gives the unique open H-orbit. imply that p = q = r. In such case, Similarly, the number of H 0 -orbits in the (H G 0 -orbit of S Gr G (r equals to [H G 0 : H 0 (H G 0 S ] = [H G 0 : H 0 (H S G 0 ] {1,, 4}. According to (5.9, [H G 0 : H 0 (H S G 0 ] = 4/l where l is the number of cosets in (H G 0 /H 0 = {H 0, H, H +, + H +} that intersect H S. This can be determined by the 5-tuple (r U, r W, a, b U, b W and the sign combinations of the Levi factor of h H S. We omit the details here as there are many cases involved. 6. Unitary Groups Suppose that V := C p+q is equipped with a nondegenerate Hermitian form B of the type I p ( I q. Let V = U W be an orthogonal decomposition such that B U is of the type I p1 ( I q1 and B W is of the type I p p1 ( I q q1. Denote (6.1 G := G(V U(p, q, H := G(U G(W U(p 1, q 1 U(p p 1, q q 1. We consider the H-orbits in Gr G (r for 0 r min{p, q}. Most results in this section are similar to those in Section 5. Their proofs are hence skipped. In counting the dimensions, the subspaces of V and the related quotient spaces refer to complex vector spaces, but all groups and orbits refer to real ones The H-invariants in Gr G (r. Define (6.a b U (S := the dimension of a maximal positive definite subspace of P U S = the dimension of a maximal negative definite subspace of P W S, (6.bb W (S := the dimension of a maximal negative definite subspace of P U S = the dimension of a maximal positive definite subspace of P W S. Then b(s = b U (S + b W (S, and the following result is true: Theorem 6.1. The 5-tuple (r U (S, r W (S, a(s, b U (S, b W (S is a complete set of H-invariants that determines the H-orbit of S in Gr G (r. Denote the H-orbit of S Gr G (r by O (r U (S, r W (S, a(s, b U (S, b W (S, where (r U (S, r W (S, a(s, b U (S, b W (S parameterizes the H-orbit. Theorem 6.. A 5-tuple (r U, r W, a, b U, b W N 0 5 parameterizes an H-orbit in Gr G (r if and only if the 5-tuple satisfies all constraints below: (1 r U + r W + a + b U + b W = r.

28 8 HUAJUN HUANG AND HONGYU HE ( The following conditions hold: (6.3a (6.3b (6.3c (6.3d r U + a + b U p 1, r U + a + b W q 1, r W + a + b W p p 1, r W + a + b U q q 1. Let G := O(p, q, H := O(p 1, q 1 O(p p 1, q q 1. Let V with a real symmetric form B be the natural representation space of (G, H. Let Gr G (r be the r-dimensional isotropic Grassmannian of V. Apparently, there is a one-to-one correspondence between H\Gr G (r and H \Gr G (r for every 0 r min{p, q}. So the Bruhat order and the inclusion order here are the same as those in Section The Bruhat order of H\Gr G (r. Construct the following diagram of H-invariants for S Gr G (r: (6.4 b U (S b W (S a(s r U (S r W (S The Bruhat order of H\Gr G (r can be characterized by a majorization relationship over this diagram, as in the real orthogonal case. Theorem 6.3. The Bruhat order O(r U, r W, a, b U, b W O(r U, r W, a, b U, b W holds in Gr G (r if and only if the following inequalities hold: (6.5a (6.5b (6.5c (6.5d (6.5e b U b U, b W b W, a + b U + b W a + b U + b W, r U + a + b U + b W r U + a + b U + b W, r W + a + b U + b W r W + a + b U + b W. Moreover, inequality (6.5c is implied by inequalities (6.5d and (6.5e. Corollary 6.4. (1 When r < min{p 1, q q 1 } + min{q 1, p p 1 }, the open H-orbits in Gr G (r are not unique, and they are given by O(0, 0, 0, b U, b W, where b U + b W = r, 0 b U min{p 1, q q 1 } and 0 b W min{q 1, p p 1 }.