HomeWork Assignment 7 ECE661

Transcription

1 HomeWork Assignment 7 ECE661 Muhammad Ahmad Ali PUID: November 23,

2 1 Problem Solution 1.1 PCA For doing face recognition through PCA we proceed as follows. First, all the images are vectorized so that i-th image is represented as X i vector. Then a matrix X is formed as follows, Where, X = [X 1 m, X 2 m,.., X N m] m = 1 N Then, we find the eigenvector matrix U of X T X. As explained in class, the eigenvectors W of XX T are found as given below, N i=1 W = XU We then retain p eigenvectors from the matrix W with largest eigenvalues to get the PCA basis W p. Then the projection of all training images on W p is obtained, y i = W T p (X i m) Similarly, each test image is projected on to the PCA basis. X i y j = W T p (X j m) For each j-th test image and i-th training image the distance d ij is calculated, d ij = y T i y j The i for which d ij is minimum for j-th test image, gives the index of best matching training image. If the class label of that training image matches the class label of the test image, a correct classification is registered. This is essentially a nearest neighbor classification in the reduced dimensionality in the form of PCA basis. The accuracy of the classification is obtained by, accuracy(p) = No. of images correctly classified Total No. of images Where, p is the number of eigenvectors used. 2

3 1.2 LDA For LDA we need to find directions w that maximize, J(w) = wt S B w w T S W w Where, S B is between class variance matrix and S W is between class variance matrix given by, S B = 1 N c (m i m)(m i m) T N c S W = 1 N c N c i=1 1 N i N i=1 i j=1 (X ij m i )(X ij m i ) T To solve this problem Yu and Yang algorithm is used. First of all, we need to find the eigenvectors of S B. These are found using the same trick as used in PCA. A matrix M is formed as follows, M = [m 1 m, m 2 m,.., m Nc m] Where m i is the mean of the i-th class. Then obviously, S B = MM T (ignoring the scale factor 1/N c ). So first the eigenvector matrix V of M T M is obtained and the eigenvectors V B of MM T are found as given below, V B = MV The eigenvectors obtained above are sorted in descing order of their eigenvalues and those with eigenvalues close to zero were discarded. I obtained 29 eigenvectors by this method. Let us call Y the eigenvector matrix left behind after discarding. Then a matrix Z is obtained, Z = YD 1/2 B Where, D B = Y T S B Y. Finally we obtain the eigenvector matrix U of Z T S W Z and they are sorted in ascing order of their eigenvalues. I do not discard the eigenvectors corresponding to largest eigenvalues at this point. So the LDA basis is obtained by, W = ZU Once LDA basis is obtained, the rest of the procedure is same as explained for PCA. We retain first p eigenvectors from W as a matrix W p and then project both the training and test images onto the basis and perform nearest neighbor classification in this basis as described in previous section. The results of applying both algorithms to the data are shown in next section. The answer to the question about space dimensionality p is also given in next section. 3

4 2 Results 2.1 PCA Using the training data set we first compute the PCA basis as described in section 1.1. This gives us a matrix, whose columns contain the desired eigenvectors. If we analyze the eigenvalues we see that the bulk of the energy is concentrated in the initial directions. Specifically we found out that the first 75 vectors span the 95% of the energy. From this we can conclude that 75 eigenvectors are sufficient for good result of the PCA method. The plot of the energy as a function of the number of the eigenvectors is given below, Energy % No. of PCA Eigenvectors Figure 1: Cumulative Energy in eigenvectors The plot of the PCA classification accuracy as a function of the number of eigenvectors is given below. We noticed that we achieved 100% accuracy by using 17 eigenvectors: 1 Accuracy of PCA No. of Eigenvectors Figure 2: PCA accuracy as a function of No. of Eigenvectors 4

5 2.2 LDA The LDA basis was obtained by following the procedure described in section 1.2. This gives a sized matrix W containing 29 basis vectors. Also, as explained in class, the number of basis vectors in LDA cannot be greater than (No. of classes - 1). In our case (No. of classes = 30), so at most 29 basis vectors can be obtained. The plot of the LDA classification accuracy as a function of the number of eigenvectors is given below. We noticed that we achieved 100% accuracy by using 10 eigenvectors: 1 Accuracy of LDA No. of Eigenvectors Figure 3: LDA accuracy as a function of No. of Eigenvectors 5

6 2.3 Comparison of PCA and LDA The comparison of the two methods is shown in the figure below. We can see that PCA achieves 100% accuracy at 17 eigenvectors while LDA achieves 100% accuracy for just 10 eigenvectors PCA LDA Accuracy No. of Eigenvectors Figure 4: Comparison of PCA and LDA accuracy as a function of No. of Eigenvectors 6

7 3 Matlab Code The code is given below, % readdata.m function [X,labelVector] = readdata srcdir = D:\CourseWork\ECE661\HW7\pca-lda\ECE661_hw7_images\hw7_images\train ; d = dir([srcdir, /*.png ]); % N = # of examples N = length(d); im = imread([srcdir, /, d(1).name ]); [h,w,k] = size( im ); % M = feature dimensionality M = h*w; X = zeros( M, N ); for( i = 1 : N ) dispstr = [ >>>>>> Reading file :, num2str(i), of, num2str(n) ]; disp( dispstr ); filename = d(i).name; im = imread( [srcdir, /, d(i).name] ); [h,w,k] = size( im ); if( k == 3 ) im = rgb2gray( im ); X( :, i ) = double( im(:) ); labelvector(i) = getsubjectid( filename ); function sub_id = getsubjectid( filename ) usindices = find( filename == _ ); sub_id = str2num( filename( 1 : usindices(1)-1) ); 7

8 % pca.m function [W,mn] = pca(x) [M,N] = size(x); % subtract off the mean for each dimension mn = mean(x,2); X = X - repmat(mn,1,n); G = X *X; % find the eigenvectors and eigenvalues [EV, V] = eig(g); size(v) [ V, idx ] = sort( diag(v) ); idx = idx( : -1 : 1 ); V = V( idx ); EV = EV( :, idx ); energy = cumsum(v)./ sum(v); save( Eigen-Value-Energy.mat, energy ); cutoff = find( energy >= 1 ); cutoff = cutoff(1); W = X * EV; W = normalizevectors(w); % runpcaclassification.m function accuracy = runpcaclassification load Train-Data; X_train = X; labels_train = labelvector; load Test-Data; 8

9 X_test = X; labels_test = labelvector; V_pca = pca(x_train); nbasis = size( V_pca, 2 ); for( i = 1 : nbasis ) dispstr = [ >>>>> Testing using the first, num2str(i), eigen vectors ]; disp( dispstr ); V_subset = V_pca( :, 1 : i ); X_train_sub = V_subset * X_train; X_test_sub = V_subset * X_test; accuracy(i) = runnearestneighborclassification( X_train_sub, labels_train, X_test_sub, labels_test ); % lda.m function W = lda( X, labelvector ) [ featurevectorsize, datasetsize ] = size(x); uniquelabels = unique( labelvector ); nclasses = length( uniquelabels ); globalmean = mean( X, 2 ); phi_b = zeros( featurevectorsize, nclasses ); phi_w = zeros( featurevectorsize, datasetsize); for( i = 1 : nclasses ) thisclassindices = find( labelvector == uniquelabels( i ) ); thisclassdata = X( :, thisclassindices ); thisclassmean = mean( thisclassdata, 2 ); M = thisclassmean - globalmean; phi_b(:,i) = M; thisclasscount = length( thisclassindices ); Y = thisclassdata - repmat( thisclassmean, 1, thisclasscount ); 9

10 phi_w( :, thisclassindices) = Y; Sb_trick = phi_b * phi_b; [V,D] = eigs( Sb_trick, nclasses ); V = phi_b * V; retained = length( find( diag(d)>0.05 ) ); Y = V(:,1:retained); Db = D(1:retained,1:retained); Z = Y * diag( (diag(db)).^(-0.5)); phi_w_z = phi_w *Z; Z_Sw_Z = phi_w_z *phi_w_z; [U,Dw] = eigs( Z_Sw_Z, size(z_sw_z, 1) ); U = U(:,:-1:1); W = Z*U; W = normalizevectors(w); % runldaclassification.m function accuracy = runldaclassification load Train-Data; X_train = X; labels_train = labelvector; load Test-Data ; X_test = X; labels_test = labelvector; V_lda = lda( X_train, labels_train ) ; nbasis = size( V_lda, 2 ) ; for( i = 1 : nbasis ) dispstr = [ >>>>> Testing using the first, num2str(i), of, num2str(nbasis), eigen vectors ]; disp( dispstr ) ; V_subset = V_lda( :, 1 : i ) ; X_train_sub = V_subset * X_train ; X_test_sub = V_subset * X_test ; 10

11 accuracy(i) = runnearestneighborclassification( X_train_sub, labels_train, X_test_sub, labels_test ); % runnearestneighborclassification.m function p = runnearestneighborclassification( X_train, labels_train, X_test, labels_test ) testcount = size( X_test, 2 ); traincount = size( X_train, 2 ); correct = 0; for( i = 1 : testcount ) thisexample = X_test( :, i ); for( j = 1 : traincount ) ssd(j) = sum( (thisexample(:) - X_train(:,j)).^2 ); [mindiff, minidx ] = min( ssd ); predicted_label( i ) = labels_train( minidx ); if( predicted_label(i) == labels_test(i) ) correct = correct + 1; p = correct / testcount; % normalizevectors.m function K = normalizevectors(v) sizev = size(v); K = V; numberofvectors = sizev(2); V2 = V.^2; for i = 1 : numberofvectors denom = sum(v2(:,i)); denom = sqrt(denom); 11

12 K(:,i) = V(:,i)/denom; 12