The commutator subgroup and the index formula of the Hecke group H 5

J. Group Theory 18 (2015), 75 92 DOI 10.1515/jgth-2014-0040 de Gruyter 2015 The commutator subgroup and the index formula of the Hecke group H 5 Cheng Lien Lang and Mong Lung Lang Communicated by James Howie Abstract. Let A be an ideal of ZŒ2 cos.=5/ and let H.A/ be the principal congruence subgroup of level A of the Hecke group H 5. The present article gives explicit formulas for ŒH 5 W H.A/. As a byproduct, we prove that the commutator subgroup of H 5 is not congruence. Consequently, the fact that the commutator subgroup of the modular group is congruence cannot be carried over to the Hecke groups. 1 Introduction 1.1 Hecke groups Let q 3 be a fixed integer. The (homogeneous) Hecke group H q is defined to be the maximal discrete subgroup of SL.2; R/ generated by S and T, where S D ; T D 1 q (1.1) 1 0 with q D 2 cos.=q/. Let A be an ideal of ZŒ q. We define three congruence subgroups of H q as follows: H 0.A/ D ¹.a ij / 2 H q W a 21 2 Aº; (1.2) H 1.A/ D ¹.a ij / 2 H q W a 11 1; a 22 1; a 21 2 Aº; (1.3) H.A/ D ¹.a ij / 2 H q W a 11 1; a 22 1; a 12 ; a 21 2 Aº: (1.4) Let Z 2 D h I i. The (inhomogeneous) Hecke group and its congruence subgroups are defined as G q D H q =Z 2 ; G 0.A/ D H 0.A/=Z 2 ; G 1.A/ D H 1.A/Z 2 =Z 2 ; G.A/ D H.A/Z 2 =Z 2 : The present article determines the index ŒH 5 W H.A/. Note that ZŒ 5 is a principal ideal domain and that the minimal polynomial of 5 over Z is x 2 x 1.

76 C. L. Lang and M. L. Lang 1.2 The obstruction Observe that H.A/ is the kernel of the homomorphism f W H 5 SL.2; ZŒ =A/. In the special case A D.2/, it is known that f W H 5 SL.2; ZŒ =2/ Š A 5 and that ŒH 5 W H.2/ D 10 (see [9]). Consequently, the map f is not always surjective and the finding of the index ŒH q W H./ is more involved. As a matter of fact, one will see in Section 6 that f is surjective if and only if the norm of A is prime to 6. Note that our result is in line with the modular group case only if gcd.n.a/; 6/ D 1 (see (1.5)). 1.3 The plan and the main results We consider the decomposition A D Q A i, where N.A i / is a power of some rational prime p i and gcd.n.a i /; N.A j // D 1. To determine the index, we will prove in the following sections that (i) ŒH 5 W H.A/ D Q i ŒH 5 W H.A i / (Section 4), (ii) ŒH 5 W H.A i / can be expressed in terms of the norm N.A i / (Section 5). Statement (i) is achieved by studying the normal subgroup H.A i /H.A j /. Statement (ii) can be done by studying the group structure of SL.2; ZŒ =A i /. Our main result shows that if gcd.n./; 6/ D 1, then ŒH 5 W H.2 a 3 b / D I a J b N./ Y 3.1 N.P / 2 /; (1.5) P j./ where the product is over the set of all prime ideals P that divide, I 0 D 1, I 1 D 10, I a D 52 6.a 1/ if a 2, J 0 D 1, J b D 1203 6.b 1/ if b 1. Our proof is elementary but involves some curious construction of matrices (see Sections 3.3 and 3.4). Note that our strategy works for other groups as well (see [5] for elementary matrix groups of P. M. Cohn [3]). 1.4 First application Subgroups of H q are called congruence if they contain some principal congruence subgroups. It is well known that the commutator subgroup of the modular group SL.2; Z/ is congruence of level 12. As an application of the index formula given in (1.5), we prove that (unlike the modular group case) the commutator subgroup of H 5 is not congruence (Section 7). Two subgroups A and B of SL.2; R/ are commensurable with each other if A \ B is of finite index in both A and B. As the commutator subgroups of SL.2; Z/ D H 3 ; H 4 and H 6 are congruence, our study suggests the following conjecture.

Commutator subgroup and index formula of the Hecke group H 5 77 Conjecture. The commutator subgroup of H q is congruence if and only if H q is commensurable with the modular group. A weaker version of the above conjecture is: Conjecture. Let q 5 be a prime. Then the commutator subgroup of H q is not congruence. 1.5 Second application Let f be a function (form) defined on H=.n/, where H is the union of the upper half plane and Q [ ¹1º and where.n/ is the principal congruence subgroup of the modular group D PSL.2; Z/. Then P f j is a function (form) of H=, where the sum is over the set of a complete set of coset representatives of =.n//. With the help of our proof of the index formula (1.5), a complete set of coset representatives of G 5 =G./ can be determined. As a consequence, the above construction for functions (forms) of H= can be extended to G 5 as well. Note that the space of forms for G q has positive dimension (Ogg [12, Theorem 3]). 1.6 Organisation of the paper The remainder of this article is organised as follows. In Section 2, we give results related to SL.2; ZŒ q / and ŒH 5 W H.A/ where A is a prime ideal. Section 3 is devoted to the study of some technical lemmas. In particular, we give a set of generators for G.2/ (see Lemma 3.1) which will be used in the determination of the index ŒH.2/ W H.4/. Section 4 investigates the index formula for H.A/. It is shown that the index formula for the principal congruence subgroup (1.4) is multiplicative. Section 5 gives the closed form of the index ŒH 5 W H.A i /. The main result of this article can be found in Section 6. Throughout the sections, we set D 2 cos : (1.6) 5 2 Known results of SL.2; ZŒ q / and ŒH 5 W H.A/ For A an ideal of ZŒ q, we may define the principal congruence subgroup L.A/ of L q D SL.2; ZŒ q / analogously. The formula for the index of the principal congruence subgroup in L q is easily calculated as the modular group case (see [14]): ŒL q W L.A/ D N.A/ 3 Y P ja.1 N.P / 2 /; (2.1)

78 C. L. Lang and M. L. Lang where N.A/ denotes the absolute norm of A in ZŒ q and the product is over the set of all prime ideals P that divide A. Let 2 ZŒ q. It follows easily from (2.1) that ŒH. n / W H. nc1 / N./ 3 : (2.2) Recall that D 2 cos =5. Let be a prime ideal of ZŒ. Denoted by p the smallest positive rational prime in. Applying our results in [9], one has (i) ŒH 5 W H./ D 10 if p D 2, (ii) ŒH 5 W H./ D 120 if p D 3 or 5, (iii) ŒH 5 W H./ D.p 1/p.p C 1/ if p D 10k 1, (iv) ŒH 5 W H./ D.p 2 1/p 2.p 2 C 1/ if 3 p D 10k 3. 3 Technical lemmas 3.1 Results of Kulkarni In [4], Kulkarni applied a combination of geometric and arithmetic methods to show that one can produce a set of independent generators in the sense of Rademacher for the congruence subgroups of the modular group, in fact for all subgroups of finite indices. His method can be generalised to all subgroups of finite index of the Hecke groups G q D H q =Z 2, where q is a prime. See [8, Propositions 8 10 and Section 3] for details. By [8, Section 3], we have the following lemma. Lemma 3.1. The principal congruence subgroup G.2/ D H.2/Z 2 =Z 2 has index 10 in G 5 D H 5 =Z 2. A set of generators is given by µ 1 2 1 C 2 2 C 2 1 C 2 2 2 D ; ; ; : 2 1 2 1 C 2 2 C 2 1 C 2 3.2 Reduced form a Observe that a=b is in reduced form if b is a column vector of some 2 H5. For any a; b 2 ZŒ such that the greatest common divisor of a and b is a unit, applying results of Leutbecher ([10, 11]), there exists a unique n 2 Z such that a n =b n is in reduced form. We shall now give an algorithm that enables us to determine the reduced factor n D e.a=b/.

Commutator subgroup and index formula of the Hecke group H 5 79 Let a; b 2 ZŒ n ¹0º be given such that the greatest common divisor of a and b is a unit. Then there exists a unique rational integer q such that (i) a D.q/b C r, jbj (ii) 2 < r jbj 2. We call such a division algorithm pseudo-euclidean (see [13] for more details). In terms of matrices, the above can be written as 1 q a D r : (3.1) b b Note that 1 q 2 H 5 : Applying the pseudo-euclidean algorithm repeatedly, one has a D.q 1 /b C r 1 ; b D.q 2 /r 1 C r 2 ; r 1 D.q 2 /r 2 C r 3 ; : r kc1 D.q kc2 /r kc2 C 0: The finiteness of the algorithm is governed by the fact that the set of cusps of H 5 is QŒ [ ¹1º. Note that, in terms of matrices, the above can be written as a D A r kc2 ; (3.2) b 0 where A 2 H 5. It is clear that gcd.a; b/ D gcd.b; r 1 / D D gcd.r kc1 ; r kc2 / D r kc2 is a unit (since a and b are coprime). As is a primitive unit, there exists some e.a=b/ 2 Z[¹0º such that r kc2 D e.a=b/. Multiplying (3.2) by e.a=b/, one has a e.a=b// b e.a=b/ D A 1 : (3.3) 0 Since A 2 H 5 and 1=0 is a reduced form, we conclude that a e.a=b/ =b e.a=b/ is the reduced form of a=b. We state without proof the following lemma ([8, Proposition 6]).

80 C. L. Lang and M. L. Lang Lemma 3.2. One has in reduced form. a b c d 2 H5 if and only if ad bc D 1 and a=c, b=d are Lemma 3.3. Let p 2 N be an odd prime and let a D 2 2, c D p 3. For every m 2 Z, S D 1 acm a2 m c 2 H.m/: (3.4) m 1 C acm Proof. The reduced form of 2=p is 2 2 =p 3. By the definition of reduced form, there exist some u; v 2 ZŒ such that X D 22 u p 3 2 H 5 : (3.5) v As a consequence, XT m X 1 2 H 5 for all m 2 Z. Note that the matrix form of XT m X 1 is given by (3.4). 3.3 Some matrices of H 5 By Lemma 3.2, one can show easily that H 5 contains the following matrices which will be used for our study: 1 C 2 2 C 2 C 2 3 C 2 2 3 ; ; ; : 2 1 C 2 2 C 1 2 2 C 1 4 C 3 4 2 3.4 Remarks A key proposition (Proposition 4.3) in our study of the index formula requires the existence of D.a ij / 2 H.p/ such that 1 p (mod p 2 ) (equivalently, that a 11 1; a 22 1; a 12 p; a 21 2.p 2 / ZŒ /. The matrices we introduced in Section 3.3 will be used in Section 5 for that purpose. It is perhaps worthwhile to point out that we did not obtain the above matrices simply by chance but rather by a study of the fundamental domains of the subgroups G.p/. See [8] for more detail. 4 The index ŒH 5 W H./ is multiplicative In this section, we investigate the index formula for the principal congruence subgroup (1.4) and prove that the index formula is multiplicative (Lemma 4.1). Note that the index formula for the inhomogeneous Hecke group G q is not multiplicative.

Commutator subgroup and index formula of the Hecke group H 5 81 Lemma 4.1. Let, 2 ZŒ and let a and b be the smallest positive rational integers in the ideals./ and./ respectively. Suppose that gcd.a; b/ D 1. Then H 5 D H./H./ and ŒH 5 W H./ D ŒH 5 W H./ ŒH 5 W H./ : Proof. Let K D H./H./. It is clear that 1 a 1 b ; 2 K: (4.1) Since gcd.a; b/ D 1, there exist some m; n 2 Z such that am C bn D 1. As a consequence, we have m n T D 1 a 1 b D 1 2 K: (4.2) Since H 5 is generated by S and T, every element in H 5 is a word w.s; T /. Since T 2 K, it follows that SK D KS (K is normal) and w.s; T /K is S i K for some i (S has order 4 in H 5 ). Hence the index of K in H 5 is either 1, 2 or 4. This implies that.st / 4 2 K. Since the order of ST is 5, we conclude that ST 2 K. It follows that ST; T 2 K. As a consequence, we have S 2 K. Hence K D H 5. The lemma can now be proved by the Second Isomorphism Theorem and the fact that H./ \ H./ D H./. Remark 4.2. In the case q D 5, ZŒ 5 is a principal ideal domain. As a consequence, every ideal of ZŒ 5 takes the form./ for some. Proposition 4.3. Let p 2 N be an odd prime. Suppose that H.p/ contains an element such that.mod p 2 /. Then 1 p ŒH.p n / W H.p nc1 / D p 6 for all n 2 N and ŒH 5 W H.p n / D p 6.n 1/ ŒH 5 W H.p/ : Proof. Let X D pn 1 and A D T pn 2 H.p n /. It follows that X 1 pn.mod p nc1 / and A D 1 pn :

82 C. L. Lang and M. L. Lang As a consequence, H.p n / contains A, SAS 1, TSAS 1 T 1, and X, SXS 1 and TSXS 1 T 1. The above matrices modulo p nc1 are given by 1 p n 1 p n 2 p n 3 ; p n ; 1 p n 1 C p n 2 ; 1 p n 1 p n p n 2 ; p n ; 1 p n 1 C p n : Applying Lemma A.1 in Appendix A, they generate an elementary abelian group of order p 6. Hence ŒH.p n / W H.p nc1 / p 6 : By (2.2), ŒH.p n / W H.p nc1 / D p 6. Corollary 4.4. Suppose that H.4/ contains some such that 1 4.mod 8/. Then ŒH.2 n / W H.2 nc1 / D 2 6 for n 2 and ŒH 5 W H.2 n / D 2 6.n 2/ ŒH 5 W H.4/ : Remark. In the case p D 2, H.2/ possesses no such that 1 2.mod 4/. Consequently, Corollary 4.4 cannot be improved to H.2/. 5 The index ŒH 5 W H./ where N./ is a power of a prime Throughout this section, N./ is a power of a rational prime p. There are five cases to consider: (i) p D 5, (ii) p D 2, (iii) p D 3, (iv) p 3 is of the form 10k 3, (v) p is of the form 10k 1. 5.1 N./ is a power of 5 Let D 2 C. The case p D 5 is slightly different as 5 D 2 2 ramified totally in ZŒ. As a consequence, D n for some n. We shall first determine the order

Commutator subgroup and index formula of the Hecke group H 5 83 of H.5 n /=H.5 nc1 /. Lemma 3.3 (p D 5, m D 5) implies that S 2 H.5/, where 1 12 5 1 205 S.mod 25/: By Proposition 4.3, ŒH.5 m / W H.5 mc1 / D 5 6 if m 1. By (2.2), ŒH. m / W H. mc1 / 5 3. Hence ŒH. m / W H. mc1 / D 5 3 for m 2: In the case m D 1, applying our results in Section 3.3, H. C 2/ contains the following matrix: 11 6 10 C 5 a D D T 2 3 C 2 2 3 2 H. C 2/: 4 C 3 4 2 4 C 3 4 2 Let J D 1 0. Then J 2 Aut H5. Let D ¹a; b D SaS 1 ; c D JaJ 1 º H./: The matrices in modulo 5 are given by 4 0 a I C. C 2/ 4 1 1 1 b I C. C 2/ 0 4 1 4 c I C. C 2/ 0 4.mod 5/;.mod 5/;.mod 5/: The set generates a group of order 5 3 modulo 5. By (2.2), H./=H.5/ Š h i and jh./=h.5/j D 5 3. In summary, one has ŒH. m / W H. mc1 / D 5 3 for m 1: (5.1) Recall that ŒH 5 W H./ D 120 and that N.x/ is the absolute norm of x. It follows from (5.1) that ŒH 5 W H. m / D 120 5 3.m 1/ D N. m / 3.1 N./ 2 / for m 1: (5.2)

84 C. L. Lang and M. L. Lang 5.2 N./ is a power of 2 Since 2 is a prime in ZŒ, we have D 2 m. Note first that ŒH 5 W H.2/ D 10. We shall now determine the index ŒG.2/ W G.4/. Applying Lemma 3.1, a set of generators of G.2/ is given by µ 1 2 1 C 2 2 C 2 1 C 2 2 2 D ; ; ; : 2 1 2 1 C 2 2 C 2 1 C 2 Since 2 generates G.2/, one can show by direct calculation that G.2/=G.4/ is elementary abelian of order 2 4. Since I 2 H.2/ H.4/, one has ŒH.2/ W H.4/ D 2 5 : We now study ŒH.2 n / W H.2 nc1 / for n 2. By (2.2), ŒH.2 n / W H.2 nc1 2 6 : By our results in Section 3.3, 1 4 1 0 4 1 1 4 2 1 C 2 2 C 2 2 H.4/=H.8/: 2 1 C 2 By Corollary 4.4, ŒH 5 W H.2/ D 10; ŒH 5 W H.2 n / D 5 2 6.n 1/ for n 2: (5.3) 5.3 N./ is a power of 3 Since 3 2 ZŒ is a prime, D 3 m. Note first that ŒH 3 W H.3/ D 120. Applying our results in Section 3.3, 1 A D C 2 1 9 C 4 2 3 D 2 H 5 : 2 C 1 2 2 C 1 9 C 6 3 2 Direct calculation shows that E 3 D A 3 1 3.mod 9/: Applying Proposition 4.3, we have ŒH 5 W H.3 m / D 120 3 6.m 1/ for m 1: (5.4)

Commutator subgroup and index formula of the Hecke group H 5 85 5.4 N./ is a power of p, where 3 p takes the form 10k 3 It follows that p is a prime in ZŒ and D p n. Note first that ŒH 5 W H.p/ D jsl.2; p 2 /j: By Lemma 3.3 (m D p), 1 12p T 20p S 2 H.p/=H.p 2 /: By Proposition 4.3, ŒH 5 W H.p n / D p 6.n 1/ jsl.2; p 2 /j D N.p n / 3.1 N.p/ 2 /: (5.5) 5.5 N./ is a power of p, where p takes the form 10k 1 Observe that p is not a prime in ZŒ and p D, where and are primes in ZŒ. Consequently, D p r s. By Lemma 3.3 (m D p), 1 12p T 20p S 2 H.p/=H.p 2 /: By Proposition 4.3, It follows immediately from (2.2) that In summary, ŒH.p m / W H.5; p mc1 / D p 6 : ŒH. m p n / W H.5; mc1 p n / D p 3 for m 1: ŒH.p m / W H.p mc1 / D p 6 ; ŒH. m p n / W H. mc1 p n / D p 3 for m 1: (5.6) Lemma 5.1. Let p D 10k 1 2 N be a rational prime and let p D, where and are primes in ZŒ. Then ŒH 5 W H.p/ D ŒH 5 W H./ ŒH 5 W H./ D..p 1/p.p C 1// 2 : Proof. Let H 0.p/ be given as in (1.2). Applying results of [2] and [9], (i) ŒH 5 W H 0.p/ D.p C 1/ 2 (see [2, Lemma 1]), (ii) H./ \ H./ D H.p/, ŒH 5 W H./ D ŒH 5 W H./ D.p 1/p.p C 1/, (iii) G 5 =G./ Š G 5 =G./ Š PSL.2; p/ is simple (see [9, Corollary 2]). By (i) (ii) of the above, H./ H./. By (iii) of the above, H./H./ D H 5. Applying the Second Isomorphism Theorem, one has ŒH 5 W H.p/ D ŒH 5 W H./ ŒH 5 W H./ : Hence ŒH 5 W H.p/ D..p 1/p.p C 1// 2.

86 C. L. Lang and M. L. Lang The index ŒH 5 W H.p r s / can be determined by applying (5.6) and Lemma 5.1. In summary, ŒH 5 W H./ D N./ Y 3.1 N.P / 2 /: (5.7) 6 The main results P j./ Let X 2 ZŒ. Consider the decomposition X D Q i x i, where N.x i / is the power of a rational prime p i such that gcd.p i ; p j / D 1 for all i j. By Lemma 4.1, ŒH 5 W H.X/ D Q i ŒH 5 W H.x i /. The index ŒH 5 W H.x i / can be determined by applying (5.2), (5.3), (5.4), (5.5), and (5.7). In short, we have, if gcd.n./; 6/ D 1, then ŒH 5 W H.2 a 3 b / D I a J b N./ Y 3.1 N.P / 2 /; (6.1) P j./ where the product is over the set of all prime ideals that divide, I 0 D 1, I 1 D 10, I a D 5 2 6.a 1/ if a 2, J 0 D 1, J b D 120 3 6.b 1/ if b 1. As a corollary of formula (6.1), one has the following. Corollary 6.1. Let A be an ideal. Then the natural homomorphism f W H 5 SL.2; ZŒ =A/ is surjective if and only if the norm of A is prime to 6. 7 Application: The commutator subgroup of H 5 is not congruence Applying our results in Section 5.1, we have the following: H 5 =H.5/ Š H. C 2/=H.5/ H 5 =H. C 2/ Š E 5 3SL.2; 5/; where H. C 2/=H.5/ Š E 3 5 Š Z 5 Z 5 Z 5 is the elementary abelian group of order 5 3 and H 5 =H. C 2/ Š SL.2; 5/, H. C 2/=H.5/ Š ha; b; ci D h i, where 4 0 a I C. C 2/ ; 4 1 1 1 b I C. C 2/ ; (7.1) 0 4 1 4 c I C. C 2/ : 0 4

Commutator subgroup and index formula of the Hecke group H 5 87 7.1 H 5 5 and H 0 are not congruence 5 Denote by H 5 5 the subgroup of H 5 generated by all the elements of the forms x 5, where x 2 H 5. Then H 5 5 is known as the power subgroup of H 5 (see [1]). It is clear that H 5 5 is normal and that S; T 5 2 H 5 5. Further, the index of H 5 5 in H 5 is 5. Since H 5 =H 5 5 is abelian, H 5 5 contains the commutator subgroup H 0 5. Lemma 7.1. Suppose that H 5 5 is congruence. Then H.5/ H 5 5. Proof. Suppose that H5 5 is congruence. Then H.5m r/ H5 5 for some 5m r 2 N, where m 1 and gcd.5; r/ D 1. Let K be the smallest normal subgroup that contains H.5 m r/ and T 5m. It is clear that K H.5 m / \ H5 5. Since gcd.5; r/ D 1, T 5m 2 K and T r 2 H.r/, it follows that T 2 KH.r/ H.5 m /H.r/: Note that KH.r/ is a normal subgroup of H 5 and that T 2 KH.r/. Applying the proof of Lemma 4.1, one has H 5 D KH.r/. Note that H.5 m /H.r/ D H 5 and that H.5 m / \ H.r/ D H.5 m r/. By the Second Isomorphism Theorem, K=H.5 m r/ H.r/=H.5 m r/ D KH.r/=H.5 m r/ D H 5 =H.5 m r/ D H.5 m /=H.5 m r/ H.r/=H.5 m r/: Hence K D H.5 m /. As a consequence, H.5 m / D K H5 5. Let m be the smallest positive integer such that H.5 m / H5 5. Suppose that m 2. Applying our main result (6.1), H 5 =H.5 m / Š SL.2; ZŒ =5 m / Š SL.2; ZŒ /=L.5 m / (see Section 2 for notation). Hence H 5 =H.5 m / and SL.2; ZŒ /=L.5 m / have the same coset representatives. Since 1 1 is a coset representative of the quotient SL.2; ZŒ /=L.5 m /, there exists some 2 H 5 such that 1 1.mod 5 m /: Since m 2, one has 5m 1 2 H5 5. As a consequence, H 5 5=H.5m / contains the elements T 5m 1 D 1 5m 1 and 1 5 m 1 5m 1 2 H.5 m 1 /=H.5 m / \ H5 5 =H.5m /:

88 C. L. Lang and M. L. Lang Now we consider the subgroup V of H5 5=H.5m / generated by T 5m 1 and 5m 1. Applying the proof of Proposition 4.3, we obtain that V D H.5 m 1 /=H.5 m /. In particular, we have H.5 m 1 / H5 5. This contradicts the minimality of m. Hence m D 1 and H.5/ H5 5. Proposition 7.2. The subgroups H5 5 and H 5 0 are not congruence. Proof. Since H5 0 H 5 5, it suffices to show that H 5 5 is not congruence. Suppose that H5 5 is congruence. By Lemma 7.1, H.5/ H 5 5. As H 5=H.C2/ Š SL.2; 5/ has no normal subgroup of index 5 and H5 5 has index 5 in H 5, it follows that H. C 2/ is not a subgroup of H5 5. This implies that H 5 5H. C 2/ D H 5. By the Second Isomorphism Theorem, jh. C 2/=ŒH 5 5 \ H. C 2/ j D 5 and jœh 5 5 \ H. C 2/ =H.5/j D 52 : Note that E 5 3SL.2; 5/ Š H 5 =H.5/ acts on D D ŒH 5 5 \ H. C 2/ =H.5/ Š Z 5 Z 5 by conjugation. Note also that D is a subgroup of h i (see (7.1)). Recall that J D 2 Aut H 5 : 1 0 The subgroup D is invariant under the conjugation of J and every element of H 5 (in particular, S and T ). However, one sees by direct calculation that the only nontrivial subgroup of h i invariant under J, S, and T is h i itself (see Appendix B). A contradiction. Hence H5 5 is not congruence. 7.2 Discussion Denote by M r the number of subgroups of G 5 D H 5 =Z 2 of index r. Applying the results of [7], we get M 2 D 1, M 3 D M 4 D 0 and M 5 D 26. The only subgroup V of index 2 is generated by ST and TS (see Section 1.1 for notations). Since G 5 =G.2/ Š D 10 has a subgroup of index 2, V contains G.2/. Hence V is congruence of level 2. It follows from Proposition 7.2 that G 5 5 D hx5 W x 2 G 5 i is non-congruence of index 5, level 5 (see [6]). Hence, the smallest index of a non-congruence subgroup of G 5 is 5. As a consequence, the smallest index of a non-congruence subgroup of H 5 is also 5. Note that the smallest index of a noncongruence subgroup of SL.2; Z/ is 7.

Commutator subgroup and index formula of the Hecke group H 5 89 A Appendix A Lemma A.1. Let p 2 N be a prime. Then modulo p nc1 generates a group of order p 6, where is given as ± D 1 p n i 1 0 ; p n i 1 ; 1 p n ic1 p n ic2 W i D 0; 1 : (A.1) p n i 1Cp n ic1 Proof. Put the matrices in (A.1) into the forms I C p n U and I C p n V. One sees easily that.i C p n U /.I C p n V / I C p n.u C V /.mod p nc1 /: (A.2) Hence modulo p nc1 generates an abelian group. Note that (A.2) makes the multiplication of I C p n U and I C p n V into the addition of U and V. In order to show generates a group of order p 6 modulo p nc1, we consider the groups * + M D X i D 1 pn i 1 0 ; Y i D p n i ; 1 N D * Z i D 1 pn ic1 p n ic2 p n i 1 C p n ic1 It is easy to see that M and N are abelian groups of order p 4 and p 2 respectively. Applying (A.2) and the fact that N is abelian, the elements in N take the following simple form: Z c 0 0 Zc 1 + 1 1 P pn 1 id0 c i ic1 p n P 1 id0 c i ic2 p n P 1 id0 c i i 1 C p n P 1 id0 c i ic1 Note that we may assume that 0 c i p take the form X a 0 0 X a 1 1 Y b 0 0 Y b 1 1 : : (A.3) 1. Similar to (A.3), the elements in M 1 p n P 1 id0 a i i p n P 1 id0 b : (A.4) i i 1 Suppose that we have Z c 0 0 Zc 1 1 X a 0 0 X a 1 1 Y b 0 0 Y b 1 1.mod p nc1 /. An easy study of the.22/-entries of (A.3) and (A.4) implies that 1X 1 C p n c i ic1 1.mod p nc1 /: id0 Hence c 0 D c 1 D 0. As a consequence, M \ N D ¹1º. Hence jhij D jm jjn j D p 6 :

90 C. L. Lang and M. L. Lang B Appendix B Lemma B.1. Let D C 2 and let D ¹a; b; cº, where a; b; c are given as in (7.1). Then the only nontrivial subgroup of h i invariant under the action of S, T and J is h i. Proof. Since.I C U /.I C V / I C.U C V /.mod 5/; multiplication of.i CU /.I CV / can be transformed into addition of U and V. This makes the multiplication of matrices a, b, and c easy. Consequently, one has r D.ac/.ab/ I C 0 0 ; 3 0 s D.ac/.ab/ 1 I C 0 3 ; 0 0 3 0 t D bc I C : 0 3 It is clear that h i D ha; b; ci D hr; s; ti: Let A; B 2 G 5. Set A B D BAB 1. Direct calculation shows that r S D s 1 ; r T D rs 1 t 2 ; r J D s; s S D r 1 ; s T D s; s J D r; t S D t 1 ; t T D st; t J D t 1 : (B.1) Denote by M a nontrivial subgroup of hr; s; ti that is invariant under the conjugation of J, S and T. Let 1 D r i s j t k 2 M. One sees easily that: (i) If k 6 0 (mod 5), without loss of generality, we may assume that k D 1. Then J S D t 2 2 M. It follows that t 2 M. Hence Consequently, s 2 M. This implies In summary, r; s; t 2 M. t T D st 2 M : s S D r 1 2 M :

Commutator subgroup and index formula of the Hecke group H 5 91 (ii) If k 0 (mod 5), then takes the form r i s j. Suppose that i 0 (mod 5). Then 1 s j 2 M. It follows that s 2 M. Consequently, Hence r D s T 2 M : rs 1 t 2 D r T 2 M : As a consequence, t 2 M. In summary, r; s; t 2 M. In the case i 6 0 (mod 5), we may assume that i D 1. Hence rs j 2 M. It follows that.rs j / T.rs j / 1 D s 1 t 2 2 M : Consequently,.s 1 t 2 / T D st 2 2 M. This implies that.s 1 t 2 /.st 2 / D t 4 2 M. Hence t 2 M. One now sees easily that r; s; t 2 M. Hence the only nontrivial subgroup of h i invariant under J, S and T is h i itself. Acknowledgments. We would like to thank the anonymous referee for pointing out a mistake in a previous version of the proof of Lemma 7.1. Bibliography [1] I. N. Cangul, R. Sahin, S. Ikikardes and O. Koruoglu, Power subgroups of some Hecke groups II, Houston J. Math. 33 (2007), no. 1, 33 42. [2] S. P. Chan, M. L. Lang, C. H. Lim and S. P. Tan, The invariants of the congruence subgroups G 0.P / of the Hecke group, Illinois J. Math. 38 (1994), 636 652. [3] P. M. Cohn, On the structure of GL 2 of a ring, Publ. Math. Inst. Hautes Études Sci. 30 (1966), 5 53. [4] R. S. Kulkarni An arithmetic-geometric method in the study of subgroups of the modular group, Amer. J Math. 113 (1991), 1053 1134. [5] C. L. Lang and M. L. Lang, Wohlfahrt s theorem and index formula for elementary matrix groups and SL.2; O/, preprint (2014), http://arxiv.org/abs/1405. 6280. [6] C. L. Lang and M. L. Lang, Wohlfahrt s theorem for the Hecke group G 5, preprint (2014), http://arxiv.org/abs/1401.0776. [7] M. L. Lang, C. H. Lim and S. P. Tan, Subgroups of the Hecke groups with small index, Linear Multilinear Algebra 35 (1993), 75 77. [8] M. L. Lang, C. H. Lim and S. P. Tan, Independent generators for congruence subgroups of Hecke groups, Math. Z. 220 (1995), 569 594.

92 C. L. Lang and M. L. Lang [9] M. L. Lang, C. H. Lim and S. P. Tan, Principal congruence subgroups of the Hecke groups, J. Number Theory 85 (2000), 220 230. [10] A. Leutbecher, Über die Heckeschen Gruppen G./, Abh. Math. Semin. Univ. Hambg. 31 (1967), 199 205. [11] A. Leutbecher, Über die Heckeschen Gruppen G./, II, Math. Ann. 211 (1974), 63 68. [12] A. Ogg, Modular Forms and Dirichlet Series, W. A. Benjamin, New York, 1969. [13] D. Rosen, A class of continued fractions associated with certain properly discontinuous groups, Duke Math. J. 21 (1954), 549 563. [14] G. Shimura, Introduction to the Arithmetic Theory of Automorphic Functions, Princeton University Press, Princeton, 1971. Received May 19, 2014; revised September 16, 2014. Author information Cheng Lien Lang, Department of Mathematics, I-Shou University, Kaohsiung, Taiwan. E-mail: cllang@isu.edu.tw Mong Lung Lang, Singapore 669608, Republic of Singapore. E-mail: lang2to46@gmail.com