Group theoretical methods in Machine Learning

Transcription

1 Group theoretical methods in Machine Learning Risi Kondor Columbia University Tutorial at ICML 2007

2 Tiger, tiger, burning bright In the forests of the night, What immortal hand or eye Dare frame thy fearful symmetry? William Blake ( )

3 Evariste Galois ( ) Niels Henrik Abel ( )

4 g : X X x g(x)

5 x x g 1 g 1 (x) g 2 g 1 g 2 g 2 (g 1 (x)) g 2 (g 1 (x))

6 1. g 1 g 2 g 1 g 2 For any g 1, g 2 G, g 2 g 1 G.

7 2. g 1 g 1 g 2 g 1 g 2 g 3 g 2 g 3 g 3 For any g 1, g 2, g 3 G, g 3 (g 2 g 1 ) = (g 3 g 2 )g 1.

8 3. x e x There exists an identity e G such that eg = ge = g for any g G.

9 4. x g g 1 x For any g G there is a g 1 G such that g 1 g = e.

10 A set G endowed with multiplication is a group if 1. (closure); g 1 g 2 G 2. (associativity); g 3 (g 2 g 1 ) = (g 3 g 2 )g 1 3. e G such that eg = ge = g (identity); 4. there exists a g 1 G such that g 1 g = g g 1 = e (inverses). G is a commutative (Abelian) group if g 1 g 2 = g 2 g 1

11 Example 1 The cyclic group Z n = {0, 1, 2,..., n 1} xy = x + y mod n

12 Example 2 Klein s Viergruppe V = {1, i, j, k} 1 i j k 1 1 i j k i i 1 k j j j k 1 i k k j i 1 V = Z 2 Z 2

13 Example 3 The quaternion group Q = {1, i, j, k, 1, i, j, k} 1 i j k 1 i j k 1 1 i j k 1 i j k i i 1 k j i 1 k j j j k 1 i j k 1 i k k j i 1 k j i i j k 1 i j k i i 1 k j i 1 k j j j k 1 i j k 1 i k k j i 1 k j i = 1 ( 1)a = a( 1) = a i 2 = j 2 = k 2 = 1 ij = k

14 Example 4 The icosahedron group I h = A 5

15 Example 5 The symmetric groups S n group of bijections σ : {1, 2,..., n} {1, 2,..., n} i.e., permutations of n objects

16 Example 6 The integers Z xy = x + y

17 Example 7 The reals R and the Euclidean vector spaces R n xy = x + y

18 Example 8 The rotation groups SO(n) group of orthogonal matrices of det 1 n n

19 Example 9 The Euclidean group ISO(n) and group of rigid body motions ISO + (n) Erlangen program (1872): geometry is the study of properties invariant under a group

20 Example 10 The special unitary groups SU(n) group of n n unitary matrices of determinant 1

21 Example 11 The general linear group GL(n) group of n n invertible matrices

22 Example 12 ( G = Z 7 3 Z 11 ) 2 ((A8 A 12 ) Z 2 )

23 Example 13 The Monster group M M = What immortal hand or eye Dare frame thy fearful symmetry?

24 Finite groups Infinite groups Z n V Q M S n Countable groups Z Continuous groups Lie groups compact non-compact SO(n) SU(n) R n ISO + (n)

25 Closed fields Fields Rings Groups commutative non-commutative Semigroups

26 Representations

27 The idea is to model groups by ρ: G C d d such that ρ(g 2 g 1 ) = ρ(g 2 ) ρ(g 1 )

28 Example S 3 2 ρ(e) = ( ) 3 ρ((12)) = ( ) 1 ρ((123)) = ( ) 1/2 3/2 3/2 1/2

29 Example A representation of Q = {1, i, j, k, 1, i, j, k} : ρ(1) = ( ( i 0 ρ(i) = 0 i ( 0 1 ρ(j) = 1 0 ( 0 i ρ(k) = i 0 ρ( x) = ρ(x) ) ) ) ) ρ(i) 2 = ρ(j) 2 = ρ(k) 2 = ρ( 1) ρ( 1) ρ(x) = ρ( x) ρ(i) ρ(j) = ρ(k)

30 What are the fundamental representations?

31 1. Two representations are equivalent if ρ 1 (x) = T 1 ρ 2 (x) T x G 2. A representation is reducible if T 1 ρ(x) T = ( ρ1 (x) 0 0 ρ 2 (x) ) x G A complete set of inequivalent irreducible representations we denote R. For any compact group the irreducibles ρ R ρ(x) 1 = ρ(x) can always be chosen to be unitary,.

32 Maschke/Wedderburn theorem Any representation ρ of a compact group reduces essentially uniquely into a direct sum ρ(x) = T 1[ ρ ] ρ (x) T of irreducible representations.

33 The Clebsch-Gordan decomposition ρ 1 (x) ρ 2 (x) = C 1 ρ 1,ρ 2 [ ρ ] ρ (x) C ρ1,ρ 2

34 Harmonic analysis

35 f(x) = k= f k e ikx f k = 1 2π π π e ikx f(x) dx f(k) = e i2πkx f(x) dx Joseph Fourier

36 "An algorithm for the machine calculation of complex Fourier series,", James W. Cooley and John W. Tukey, Math. Comput. 19, (1965) John Wilder Tukey

37 Let CG be the space of functions f : G C. The left-translate of f is defined f z (x) = f(z 1 x). Are there any subspaces of f f z under? CG which are invariant

38 Translation is a linear operator, so in vector form it acts by v ρ(z) v. In particular, f z = ρ r (z) f, where [ρ r (z)] y,x = { 1 if zx = y 0 otherwise is the regular representation.

39 If we define f(ρ) = x G f(x) ρ(x) ρ R then f z (ρ) = ρ(z) f(ρ) So the columns of f(ρ) are projections onto minimal left-translation invariant subspaces.

40 The Fourier transform on a finite group is f(ρ) = x G f(x)ρ(x) ρ R Note similarity to f(k) = e i2πkx f(x) dx

41 The Fourier transform on a group is f(ρ) = x G f(x)ρ(x) ρ R The inverse transform is f(x) = 1 G ρ R [ ] d ρ tr f(ρ) ρ(x 1 )

42 f(ρ) = x G f(x) ρ(x) f(x) = 1 G ρ R [ ] d ρ tr f(ρ) ρ(x 1 ) 1. Linearity: f +g = f + ĝ 2. Unitarity: f, g = f, ĝ 3. Left-translation: f z (ρ) = ρ(z) f(ρ) 4. Convolution: f g(ρ) = f(ρ) ĝ(ρ)

43 How about right translation f (z) (x) = f(xz 1 )?

44 f (z) (ρ) = f(ρ) ρ(z) so right-translation preserves f(ρ) the subspaces spanned by the rows of. The isotypal decomposition is CG = ρ R V ρ where the V ρ are both left- and right-invariant. This decomposition is independent of the choice of R!

45 The Fourier transform F : f f is an isomorphism F : CG ρ R C d ρ d ρ.

46 The Laplacian can quantify the complexity of functions: = 2 x x x 2 D e ikx = k 2 e ikx [ ] i,j = 1 if i j d i if i = j 0 otherwise f f = 1 2 (f(i) f(j)) 2 i j

47 For example, in regularized risk minimzation f = arg min [ m i=1 L(f(x i ), y i ) + f, f H ] where H is the RKHS induced by a kernel k. My favorite kernel on graphs is the diffusion kernel k(x, x ) = [ e β ] x,x, giving rise to the regularizer f, f = e β /2 f, e β /2 f

48 So what would the Laplacian on a group look like?

49 The Laplacian must be a self-adjoint, invariant operator: g, f = g, f g, f = (g(x)) f(x) dµ G f z = ( f) z f (z) = ( f) (z)

50 Theorem If is a left- and right-invariant operator then = ρ R α ρ 1 ρ In particular, f(ρ) = α f(ρ) ρ.

51 f = l f where l(ρ) = α ρ I l z = l (z) l(z 1 x z) = l(x) i.e., l is a class function.

52 Example: data on Sn

53 ranking votes ranking votes ranking votes ranking votes ranking votes American Psychological Association voting data [Diaconis]

54 The kernels generated by transpositions, adjacent transpositions and cyclically adjacent transpositions

55 Invariance in Physics

56 The invariance group of classical Physics is ISO(3) R Galileo Galilei ( )

57 To preserve ( x) 2 + ( y) 2 + ( z) 2 c 2 ( t 2 ) relativity adopted the Lorentz group SO + (1, 3) ct x y z = cosh(β) sinh(β) 0 0 sinh(β) cosh(β) ( ) 1 + v/c β = log 1 v2 /c 2 ct x y z Albert Einstein ( )

58 Symmetry implies conservation: (roughly) dj dt = H θ time space rotation energy momentum angular mom. Emmy Noether ( )

59 symmetries unitary op. observables generators pure states dimensions of irreps Eugene Wigner ( )

60 Quantum Chromodynamics SU(3), SU(6),... lead to quarks and even stranger animals... Murray Gell-Man 1929-