Singular Value Decomposition and Digital Image Compression

Transcription

1 Singular Value Decomposition and Digital Image Compression Chris Bingham December 1, 016 Page 1 of

2 Abstract The purpose of this document is to be a very basic introduction to the singular value decomposition (SVD) process. There will be an explanation of how the compute the SVD of an m n matrix A, followed by an example problem. SVD has many applications but this document will focus on it s application to digital image compression with example images. Page 1 of

3 As a digital photographer I am always dealing with storage of photos on hard drives and being concerned about the space required. The photographs that come straight out of camera are massive files that aren t easily transferred to clients. When I saw a project retaining to digital image compression by way of singular value decomposition (SVD), it immediately stood out as something I would like to learn. Through this document I will be exploring the mathematical processes taken to compress a digital image using linear algebra. I will be using one of my own photographs to give a visual demonstration of how SVD affects a photograph. SVD is a method used to diagonalize any non symmetric matrix A. Through this diagonalization, matrix A is decomposed into three matrices. One of those matrices is made up of a diagonal that contains the singular values of A, which will be very important in the application of image compression. The other two matrices are made up of the orthogonal unit vectors that correspond to the singular values found. When multiplied, these three matrices equal matrix A. This is useful in image compressions because you can choose how many of the singular values and their corresponding orthogonal unit vectors to obtain a compressed version of matrix A. Before jumping into image compession, let s go over how to calculate the SVD of any m n matrix. Singular Value Decomposition Theorem Let A be an m n matrix with rank r. Then there exists an m n matrix Σ for which the diagonal entries in D are the first r singular values of A, σ 1 σ... σ r > 0, and there exist an m m orthogonal matrix U and an n n orthogonal matrix V such that A = UΣV T Page 1 of

4 Process of SVD SVD can be achieved by hand calculations if you re attempting to decompose a small matrix, such as a matrix. The calculations become much more time consuming as the matrices get larger. Computer programs such as Mathematica or Matlab can be used to calculate the SVD of a matrix A quickly. In the following section I will outline and explain each step to determine the SVD of matrix A. I will follow that up by giving an example matrix. Step One: Calculate A T A and find its eigenvalues The first step is to take A and find its product with the transpose of A, or A T. Once you have the new matrix A T A, you will need to find the eigenvalues and their corresponding eigenvectors. Finding the product of A T A is necessary to be able to diagonalize as you will obtain a symmetric matrix. You need a symmetric matrix to be able to determine the eigenvalues of matrix A. Assume A to be an m n matrix: Step Two: Create matrices V and Σ A T A = A T A n m m n n n To create matrix V, take the eigenvalues obtained from A T A and rearrange them from highest magnitude to lowest. The eigenvectors corresponding to those eigenvalues will make up the columns of V. The last step for matrix V is to turn the columns into unit vectors. The columns of matrix V are known as the right singular vectors of A. V = [ v 1 v... v n ] To create Σ you will start by temporarily creating a matrix D that consists of the first r singular values of A. The singular values of A are obtained by taking the square root of the eigenvalues found from A T A. Then you will want to rearrange them from highest magnitude Page of

5 to lowest. σ D = σ r Now that you have a diagonal matrix D, you can create Σ. Sigma will have the same dimensions as your original matrix A. It is a partitioned matrix with the matrix D in the top left corner and and the rest is filled in with zeroes. Here is what Σ should look like: [ r n r] Σ mxn = D 0 r 0 0 m r Step Three: Create Matrix U The last matrix to construct is matrix U. To construct the columns of U, find the product of the original matrix A and the columns of V. Then multiply by 1/σ r. Remember that matrix U will be an m m matrix. The columns of U are known as the left singular vectors of A.Below is the formula used to find the columns of U. u k = 1 σ k (Av k ) (1) Letting k range from 1 to the rank of A. You should end up with something like this U = [ u 1 u u m ] You may run into a problem here where you have used all of the non-zero singular values to find the columns that make up matrix U, but do not have a square orthogonal matrix. In this case, you will need to use the Gram-Schmidt process to find a basis for the subspace spanned by the existing columns of U. To do so you want to start by finding a vector y that is not in the plane spanned by the existing column vectors of U. Then you can plug it into this formula to find the remaining column vectors of U to produce a square orthogonal matrix needed for the SVD to be complete. Page of

6 u = y y.u 1 u 1.u 1 u 1 y.u a u a.u a u a Letting a be the number of column vectors in U you already have. The last thing to remember is the convert any column vectors found using the Gram-Schmidt process into unit vectors. Final Step We have now created all three matrices required for the SVD of A. They should have the form; Method to Check SVD Calculation A = U Σ V T m n m m m n n n The method to check whether you performed SVD correctly on a matrix requires a few steps involving partitioning matrices and column-row multiplication. I will outline how to go about doing this below. Let r be the rank of the original matrix A. v T σ A = [ ]. u 1 u u r u m v T r 0 0 σ r vr+1 T 0 0. Start by multiplying the matrix U by the columns of Σ. v T n Page 4 of

7 v A = [ σ 1 u 1 σ u σ r u r 0 0 ] 1. T vr T. v T r+1 For the next step you will want to partition the two remaining matrices. A = [ σ 1 u 1 σ u σ r u r 0 0 ] By performing column-row multiplication we get v T n v T 1. v T r v T r+1 A = σ 1 u 1 v T 1 + σ u v T σ r u r v T r + 0v T r+1 + 0v T n. v T n The values in Σ and V T after σ r and v T r will all be zeros, resulting in Page 5 of A = σ 1 u 1 v1 T + σ u v T σ r u r vr T () Which can be seen as k A k = σ i u i vi T () i=1 With k being the number of singular values you want to use. To check whether the original matrix is achieved you will use all of the singular values. However this is also the method through which you find an approximation matrix A k of the original matrix A.

8 Example of SVD Using a Matrix Given the matrix A below, compute A = UΣV T. 1 1 A = Step One A T A = [ ] = 0 1 [ ] 1 1 Now that we have a square diagonal matrix, we can easily find the eigenvalues and corresponding eigenvectors. Since this is a x matrix, we can use the shortcut formula to find the characteristic polynomial. Find the trace and determinant of A T A and substitute into the following formula: p(λ) =λ T λ + D p(λ) =λ 4λ + p(λ) =(λ )(λ 1) The eigenvalues, arranged from highest magnitude to lowest are and 1. To find the corresponding eigenvectors calculate A I and A I. Through elementary row operations and determining the basis for each eigenvalue, we find the eigenvectors. [ ] 1 λ 1 = v 1 = 1 [ ] 1 λ = 1 v = 1 Page 6 of

9 Step Two Since we have the eigenvectors of A T A, we can use them as the columns of matrix V. [ ] 1 1 V = 1 1 We want matrix V to be an orthogonal matrix, so divide each column by it s length to make them unit vectors. [ ] 1 V = 1 We can also construct Σ at this point because we know the eigenvalues. Take the square root of each eigenvalue and place them on the diagonal of the matrix, as arranged from highest magnitude to lowest. At this point we need to make sure that the dimensions of Σ are the same as the dimensions of the original matrix A. 0 Step Three Σ = The last matrix to construct is matrix U. To do so, use formula (1). u 1 = [ ] 1 0 = u = [ ] = Page 7 of

10 Therefore: U = Here we run into a problem. Following the theorem for SVD, this matrix needs to be a square orthogonal matrix. We only have two singular values to plug into the formula though, resulting in a x matrix. To fix this we use the Gram-Schmidt Process to find the third vector. To do this, we need to choose a vector that is in the same plane as u 1 and u. Then we have everything we need to plug into the formula to find the third vector. I chose the vector 1 u = y y.u 1 u 1.u 1 u 1 y.u u.u u y = Plugging what we have for u 1 and u and y into the formula we achieve we get our third vector. The last step is to convert it to a unit vector and we have completed matrix U. 1 u = 1 u 1 u = 1 1 We now have the correct matrix U and we can complete the SVD of matrix A. U = Page 8 of

11 Final Step We now have all three matrices needed for the SVD of matrix A. Check Answer A = UΣV T [ = ] T Using the method described, we can check our SVD calculation. Our calculated SVD has the form: A = [ ] u 1 u u σ 1 0 [ ] 0 σ v T 1 v 0 0 T Plugging columns and singular values and rows we have into forumla (), we get A = [ ] [ ] A = A = (4) The two matrices above are both rank one matrices. The first of which is actually the rank one approximation, or A 1, of the original matrix A. When you add the two rank one matrices together, you will see that they are equal to the original matrix A. Page 9 of

12 Approximation of the Original Matrix We discovered in the method to check that our calculations equaled the original matrix A, that a formula () can be found to calculate various approximated matrices of the A. Using said formula, you can choose how many singular values to use in order to find an approximation of A. The number of singular values directly determines the rank of the approximated matrix. This is because the summation formula is adding k rank one matrices together, with the final result being a matrix of rank k. If we once again look at our example matrix A, we can see how this formula works. We can see in (4) that if we were looking to find a rank one approximation of A, or A 1, then 1 1 A 1 = σ 1 u 1 v1 T = 1 1 When trying to achieve a rank one approximation we omit all other ranks of the original matrix A. For this specific example we started with a rank two matrix. Therefore, if we add only one more rank, we end up with our original matrix A. For matrices with a much larger rank, there will be more options for approximated matrices. The example image that will be shown below will begin as a rank 480 matrix. We will see that more of the pertinent information regarding the original information corresponds to the higher singular values. Approximated matrices are a great way to remove some of the redundant information contained in a matrix and still have very useful information retained. This will be seen in the application to digital images. Application of SVD to Image Compression We now know how SVD can be used to find an approximation of an m n matrix. When applied to digital images, this can be used to compress an image to achieve a smaller file size for easier storage or transfer. This is a unique opportunity to see a visual representation of how SVD affects a matrix. In order to perform this, I used Mathematica. This is a powerful program that 1 1 Page 10 of

13 not only enabled me to upload and extract data from the image, but also compute its SVD and approximated data matrices. Digital images are stored as data in the form of pixels. Each pixel is represented by a cell in a matrix. The value of the pixel is the intensity of that pixel. A truly grayscale image consists of one matrix while color images are made up of three matrices storing the intensity for red, blue and green. For my demonstration of the affect that SVD can have on an image, I will be showing a grayscale image of the New York skyline that I captured. The image size is 480 pixels by 640 pixels. Data is extracted from the image to create a matrix with 480 rows and 640 columns, containing 07,00 entries. The rank of the image matrix is 480. This gives us plenty of possible approximations to try. Through this we will see a gradual change in the image quality as we increase the rank of the approximation. Visual Representation of SVD You will see in the images below that the number of singular values used has a big impact on the image compression. By only using one singular value in Figure, the image is unrecognizable. Therefore it is not a very useful approximation. When 80 singular values are used in Figure 7, there is still evidence of some lost data in the approximation but is very much recognizable. When using 160 singular values in Figure 8, we get an image that is closer to the original quality of the image. Depending on its application, this image could be used in place of the original, and has a compressed file size. By the time we ve added 0 singular values in Figure 9, there is almost no visual change from the original image. We can assume that from that point forward, any additional rank approximated added would have very minimal change, if any. This is a clear example that most of the pertinent information in the original image is stored with the higher singular values. Therefore for this image exclusively we can say that somewhere around a rank 160 approximation could be a desirable image. This of course depends on just how clear you desire the image to be and what it will be used for. Page 11 of

14 Page 1 of Figure 1: The original image that data was extracted from to create matrix A.

15 Page 1 of Figure : Approximation using 1 singular value. The image is unrecognizable. The highest compression of the image, but not useful at all.

16 Page 14 of Figure : With a rank 5 approximation, we begin to see more information being added to the image.

17 Page 15 of Figure 4: Rank 10 approximation.

18 Page 16 of Figure 5: Rank 0 approximation, the image continues to get sharper. This is still too compressed and an image of this quality is not desirable.

19 Page 17 of Figure 6: Rank 40 approximation.

20 Page 18 of Figure 7: Rank 80 approximation.

21 Page 19 of Figure 8: Rank 160 approximation.

22 Page 0 of Figure 9: Rank 0 approximation. You can see at this point that the image is almost identical to the original.

23 Page 1 of Figure 10: The approximated matrix with the same rank as the original. Looks identical to original image.

24 Through this document, I broke down the process to calculate the SVD of an m n matrix A. My hope is that this will be a useful tool for students to be able to understand the process and be able replicate the calculations with ease. SVD is an extremely powerful process that be used to manipulate matrices. I barely scratched the surface of the applications for SVD. As a visual learner, this was the most intriguing application as it gave me an opportunity to see a visual representation of an approximated matrix. References [1] A. Schultz. 006 SWEET Applications of SVD aschultz/summer06/math10/coursenotes and handouts/mathematica / [] C. Long. Visualization of Matrix Singular Value Decomposition [] D. Kalman The Singularly Valuable Decomposition: The SVD of a Matrix [4] D. Lay. 01 Linear Algebra and its Applications [5] J. Chen. Image Compression with SVD Page of