Introduction to Computational Data Analysis CX4240, 2019 Spring Lecture 02 Linear Algebra Basics Chao Zhang College of Computing Georgia Tech These slides are based on slides from Le Song and Andres Mendez-Vazquez.
Outline Linear Algebra Basics Norms Multiplications Matrix Inversion Trace and Determinant Eigen Values and Eigen Vectors Singular Value Decomposition Matrix Calculus 2
Why Linear Algebra?
Linear Algebra Basics 4
Outline Linear Algebra Basics Norms Multiplications Matrix Inversion Trace and Determinant Eigen Values and Eigen Vectors Singular Value Decomposition Matrix Calculus 5
Norms 6
Norms 7
Vector Norm Examples 8
Special Matrices 9
Outline Linear Algebra Basics Norms Multiplications Matrix Inversion Trace and Determinant Eigen Values and Eigen Vectors Singular Value Decomposition Matrix Calculus 10
Multiplications 11
Multiplications 12
Inner Product Properties 13
Inner Product Properties 14
Inner Product Properties 15
Outline Linear Algebra Basics Norms Multiplications Matrix Inversion Trace and Determinant Eigen Values and Eigen Vectors Matrix Decomposition Matrix Calculus 16
Linear Independence and Matrix Rank 17
Range and Null Space 18
Row and Column Space 19
Matrix Rank: Examples What are the ranks for the following matrices? 20
Matrix Inverse 21
Outline Linear Algebra Basics Norms Multiplications Matrix Inversion Trace and Determinant Eigen Values and Eigen Vectors Singular Value Decomposition Matrix Calculus 22
Matrix Trace 23
Matrix Determinant 24
Properties of Matrix Determinant 25
Outline Linear Algebra Basics Norms Multiplications Matrix Inversion Trace and Determinant Eigen Values and Eigen Vectors Singular Value Decomposition Matrix Calculus 26
Eigenvalues and Eigenvectors 27
Computing Eigenvalues and Eigenvectors 28
Eigenvalue Example Slide credit: Shubham Kumbhar 29
Matrix Eigen Decomposition 30
Properties of Eigendecomposition 31
Outline Linear Algebra Basics Norms Multiplications Matrix Inversion Trace and Determinant Eigen Values and Eigen Vectors Singular Value Decomposition Matrix Calculus 32
Singular Value Decomposition 33
Singular Value Decomposition 34
Geometric Meaning of SVD Image Credit: Kevin Binz 35
M<latexit sha1_base64="fsxrvhuf+uha5zzl6jy675pf76s=">aaab8xicbvbns8nafhypx7v+vt16wsycp5kiomeify9cbvulbsib7uu7dlmjuxuhhp4llx4u8eq/8ea/cdpmok0dc8pme+y8crlbtxhdb6e0srq2vlhergxt7+zuvfcp2jpofcmwi0wsoghvkljelufgycdrskna4emwvs79hydumsfy3kws9cm6ldzkjborpfyiakzbmn1o+9waw3dnimvek0gncjt71a/eigzphniwqbxuem5i/iwqw5naaawxakwog9mhdi2vneltz7peu3jilqejy2wfngsm/t7iakt1jarszj5ql3q5+j/xtu146wdcjqlbyeyfhakgjib5+wtaftijjpzqprjnstiiksqmlalis/awt14m7bo659a9u/na46qoowxhcayn4mefnoagmtacbhke4rxeho28oo/ox3y05bq7h/ahzucpu+kq8q==</latexit> <latexit sha1_base64="fsxrvhuf+uha5zzl6jy675pf76s=">aaab8xicbvbns8nafhypx7v+vt16wsycp5kiomeify9cbvulbsib7uu7dlmjuxuhhp4llx4u8eq/8ea/cdpmok0dc8pme+y8crlbtxhdb6e0srq2vlhergxt7+zuvfcp2jpofcmwi0wsoghvkljelufgycdrskna4emwvs79hydumsfy3kws9cm6ldzkjborpfyiakzbmn1o+9waw3dnimvek0gncjt71a/eigzphniwqbxuem5i/iwqw5naaawxakwog9mhdi2vneltz7peu3jilqejy2wfngsm/t7iakt1jarszj5ql3q5+j/xtu146wdcjqlbyeyfhakgjib5+wtaftijjpzqprjnstiiksqmlalis/awt14m7bo659a9u/na46qoowxhcayn4mefnoagmtacbhke4rxeho28oo/ox3y05bq7h/ahzucpu+kq8q==</latexit> <latexit sha1_base64="fsxrvhuf+uha5zzl6jy675pf76s=">aaab8xicbvbns8nafhypx7v+vt16wsycp5kiomeify9cbvulbsib7uu7dlmjuxuhhp4llx4u8eq/8ea/cdpmok0dc8pme+y8crlbtxhdb6e0srq2vlhergxt7+zuvfcp2jpofcmwi0wsoghvkljelufgycdrskna4emwvs79hydumsfy3kws9cm6ldzkjborpfyiakzbmn1o+9waw3dnimvek0gncjt71a/eigzphniwqbxuem5i/iwqw5naaawxakwog9mhdi2vneltz7peu3jilqejy2wfngsm/t7iakt1jarszj5ql3q5+j/xtu146wdcjqlbyeyfhakgjib5+wtaftijjpzqprjnstiiksqmlalis/awt14m7bo659a9u/na46qoowxhcayn4mefnoagmtacbhke4rxeho28oo/ox3y05bq7h/ahzucpu+kq8q==</latexit> <latexit sha1_base64="fsxrvhuf+uha5zzl6jy675pf76s=">aaab8xicbvbns8nafhypx7v+vt16wsycp5kiomeify9cbvulbsib7uu7dlmjuxuhhp4llx4u8eq/8ea/cdpmok0dc8pme+y8crlbtxhdb6e0srq2vlhergxt7+zuvfcp2jpofcmwi0wsoghvkljelufgycdrskna4emwvs79hydumsfy3kws9cm6ldzkjborpfyiakzbmn1o+9waw3dnimvek0gncjt71a/eigzphniwqbxuem5i/iwqw5naaawxakwog9mhdi2vneltz7peu3jilqejy2wfngsm/t7iakt1jarszj5ql3q5+j/xtu146wdcjqlbyeyfhakgjib5+wtaftijjpzqprjnstiiksqmlalis/awt14m7bo659a9u/na46qoowxhcayn4mefnoagmtacbhke4rxeho28oo/ox3y05bq7h/ahzucpu+kq8q==</latexit> U<latexit sha1_base64="i69u2bbvulryn7n5sm3b9sq3yd8=">aaab8xicbvbns8nafhypx7v+vt16wsycp5kiomeif48vtftss9lsn+3szsbsvggl9f948aciv/+nn/+nmzyhbr1yggbey+dnkehh0hw/ndla+sbmvnm7sro7t39qptxqmtjvjpsslrhubnrwkrt3uadknurzggwst4pjbe63n7g2ilypoe14p6ijjulbkfrpsrdrhadh5s8g1zpbd+cgq8qrsa0knafvr94wzmneftjjjel6bol9jgoutpjzpzcanla2ospetvtrijt+nk88i2dwgziw1vypjhp190zgi2omuwan84rm2cvf/7xuiuf1pxmqszertvgotcxbmotnk6hqnkgcwkkzfjyrywoqkunbusww4c2fvepaf3xprxv3l7xgtvfhgu7gfm7bgytowb00wqcgcp7hfd4c47w4787hyrtkfdvh8afo5w/icpd5</latexit> <latexit sha1_base64="i69u2bbvulryn7n5sm3b9sq3yd8=">aaab8xicbvbns8nafhypx7v+vt16wsycp5kiomeif48vtftss9lsn+3szsbsvggl9f948aciv/+nn/+nmzyhbr1yggbey+dnkehh0hw/ndla+sbmvnm7sro7t39qptxqmtjvjpsslrhubnrwkrt3uadknurzggwst4pjbe63n7g2ilypoe14p6ijjulbkfrpsrdrhadh5s8g1zpbd+cgq8qrsa0knafvr94wzmneftjjjel6bol9jgoutpjzpzcanla2ospetvtrijt+nk88i2dwgziw1vypjhp190zgi2omuwan84rm2cvf/7xuiuf1pxmqszertvgotcxbmotnk6hqnkgcwkkzfjyrywoqkunbusww4c2fvepaf3xprxv3l7xgtvfhgu7gfm7bgytowb00wqcgcp7hfd4c47w4787hyrtkfdvh8afo5w/icpd5</latexit> <latexit sha1_base64="i69u2bbvulryn7n5sm3b9sq3yd8=">aaab8xicbvbns8nafhypx7v+vt16wsycp5kiomeif48vtftss9lsn+3szsbsvggl9f948aciv/+nn/+nmzyhbr1yggbey+dnkehh0hw/ndla+sbmvnm7sro7t39qptxqmtjvjpsslrhubnrwkrt3uadknurzggwst4pjbe63n7g2ilypoe14p6ijjulbkfrpsrdrhadh5s8g1zpbd+cgq8qrsa0knafvr94wzmneftjjjel6bol9jgoutpjzpzcanla2ospetvtrijt+nk88i2dwgziw1vypjhp190zgi2omuwan84rm2cvf/7xuiuf1pxmqszertvgotcxbmotnk6hqnkgcwkkzfjyrywoqkunbusww4c2fvepaf3xprxv3l7xgtvfhgu7gfm7bgytowb00wqcgcp7hfd4c47w4787hyrtkfdvh8afo5w/icpd5</latexit> <latexit sha1_base64="i69u2bbvulryn7n5sm3b9sq3yd8=">aaab8xicbvbns8nafhypx7v+vt16wsycp5kiomeif48vtftss9lsn+3szsbsvggl9f948aciv/+nn/+nmzyhbr1yggbey+dnkehh0hw/ndla+sbmvnm7sro7t39qptxqmtjvjpsslrhubnrwkrt3uadknurzggwst4pjbe63n7g2ilypoe14p6ijjulbkfrpsrdrhadh5s8g1zpbd+cgq8qrsa0knafvr94wzmneftjjjel6bol9jgoutpjzpzcanla2ospetvtrijt+nk88i2dwgziw1vypjhp190zgi2omuwan84rm2cvf/7xuiuf1pxmqszertvgotcxbmotnk6hqnkgcwkkzfjyrywoqkunbusww4c2fvepaf3xprxv3l7xgtvfhgu7gfm7bgytowb00wqcgcp7hfd4c47w4787hyrtkfdvh8afo5w/icpd5</latexit> <latexit sha1_base64="uqhygdb8ht/vcx5pjpqswfwf/lk=">aaab+hicbvdlsgmxfl1tx7u+wnxpjlgev2vgbf0w3bisab/qguomzbshswzimkid+ivuxcji1k9x59+yawehrqcch3pu5z6cmofmg9f9dkpr6xubw+xtys7u3n61dndy0xgqcg2tmmeqf2jnozo0bzjhtjcoikxiatec3or+95eqzwl5ykyjdqqesryxgo2vbrwql7azh1hm37orwlnbre423dnqkvekuoccruhtyx/gjbvugskx1n3ptuyqywuy4xrw8vnne0wmeet7lkosqa6yefazorxkeewxsk8anfd/b2ryad0voz3my+pllxf/8/qpia6cjmkknvssxaeo5cjekg8bdzmixpcpjzgozrmimsyke2o7qtgsvouvr5loecnzg97drb15xdrrhmm4gtpw4bkacastaaobfj7hfd6cj+ffexc+fqmlp9g5gj9wpn8adl+tva==</latexit> <latexit sha1_base64="uqhygdb8ht/vcx5pjpqswfwf/lk=">aaab+hicbvdlsgmxfl1tx7u+wnxpjlgev2vgbf0w3bisab/qguomzbshswzimkid+ivuxcji1k9x59+yawehrqcch3pu5z6cmofmg9f9dkpr6xubw+xtys7u3n61dndy0xgqcg2tmmeqf2jnozo0bzjhtjcoikxiatec3or+95eqzwl5ykyjdqqesryxgo2vbrwql7azh1hm37orwlnbre423dnqkvekuoccruhtyx/gjbvugskx1n3ptuyqywuy4xrw8vnne0wmeet7lkosqa6yefazorxkeewxsk8anfd/b2ryad0voz3my+pllxf/8/qpia6cjmkknvssxaeo5cjekg8bdzmixpcpjzgozrmimsyke2o7qtgsvouvr5loecnzg97drb15xdrrhmm4gtpw4bkacastaaobfj7hfd6cj+ffexc+fqmlp9g5gj9wpn8adl+tva==</latexit> <latexit sha1_base64="uqhygdb8ht/vcx5pjpqswfwf/lk=">aaab+hicbvdlsgmxfl1tx7u+wnxpjlgev2vgbf0w3bisab/qguomzbshswzimkid+ivuxcji1k9x59+yawehrqcch3pu5z6cmofmg9f9dkpr6xubw+xtys7u3n61dndy0xgqcg2tmmeqf2jnozo0bzjhtjcoikxiatec3or+95eqzwl5ykyjdqqesryxgo2vbrwql7azh1hm37orwlnbre423dnqkvekuoccruhtyx/gjbvugskx1n3ptuyqywuy4xrw8vnne0wmeet7lkosqa6yefazorxkeewxsk8anfd/b2ryad0voz3my+pllxf/8/qpia6cjmkknvssxaeo5cjekg8bdzmixpcpjzgozrmimsyke2o7qtgsvouvr5loecnzg97drb15xdrrhmm4gtpw4bkacastaaobfj7hfd6cj+ffexc+fqmlp9g5gj9wpn8adl+tva==</latexit> <latexit sha1_base64="uqhygdb8ht/vcx5pjpqswfwf/lk=">aaab+hicbvdlsgmxfl1tx7u+wnxpjlgev2vgbf0w3bisab/qguomzbshswzimkid+ivuxcji1k9x59+yawehrqcch3pu5z6cmofmg9f9dkpr6xubw+xtys7u3n61dndy0xgqcg2tmmeqf2jnozo0bzjhtjcoikxiatec3or+95eqzwl5ykyjdqqesryxgo2vbrwql7azh1hm37orwlnbre423dnqkvekuoccruhtyx/gjbvugskx1n3ptuyqywuy4xrw8vnne0wmeet7lkosqa6yefazorxkeewxsk8anfd/b2ryad0voz3my+pllxf/8/qpia6cjmkknvssxaeo5cjekg8bdzmixpcpjzgozrmimsyke2o7qtgsvouvr5loecnzg97drb15xdrrhmm4gtpw4bkacastaaobfj7hfd6cj+ffexc+fqmlp9g5gj9wpn8adl+tva==</latexit> <latexit sha1_base64="2ehdqpzqsme6l8h0i0lcyad4h94=">aaab/hicbvdlssnafj3uv62vajdugkvwvrirdfl047jcx9demplo2qgtszi5euoiv+lghsju/rb3/o2tngttptbwoode7pnjx5wpso1vo7k2vrg5vd2u7ezu7r+yh0c9fsws0c6jecqhplaum0g7widtqswpdn1o+/7spvd7d1qqfokopdh1qjwrlgaeg5zgzt0nmuz9iovl9y7qr8g6+chs2e17dmuvocvpobltkfnljioshfqa4vipowph4gvyaioc5ju3uttgziyndkipwcfvxjypn1unwhlbqst1e2dn1d8bgq6vsknftxzr1bjxip95wwscky9jik6acri4fctcgsgqmrdgtficpnuee8l0votmscqedf81xykz/ovv0jtvonbtubtotk7loqrogj2gm+sgs9rct6inuoigfd2jv/rmpbkvxrvxsritgovohf2b8fkdgnmvua==</latexit> <latexit sha1_base64="2ehdqpzqsme6l8h0i0lcyad4h94=">aaab/hicbvdlssnafj3uv62vajdugkvwvrirdfl047jcx9demplo2qgtszi5euoiv+lghsju/rb3/o2tngttptbwoode7pnjx5wpso1vo7k2vrg5vd2u7ezu7r+yh0c9fsws0c6jecqhplaum0g7widtqswpdn1o+/7spvd7d1qqfokopdh1qjwrlgaeg5zgzt0nmuz9iovl9y7qr8g6+chs2e17dmuvocvpobltkfnljioshfqa4vipowph4gvyaioc5ju3uttgziyndkipwcfvxjypn1unwhlbqst1e2dn1d8bgq6vsknftxzr1bjxip95wwscky9jik6acri4fctcgsgqmrdgtficpnuee8l0votmscqedf81xykz/ovv0jtvonbtubtotk7loqrogj2gm+sgs9rct6inuoigfd2jv/rmpbkvxrvxsritgovohf2b8fkdgnmvua==</latexit> <latexit sha1_base64="2ehdqpzqsme6l8h0i0lcyad4h94=">aaab/hicbvdlssnafj3uv62vajdugkvwvrirdfl047jcx9demplo2qgtszi5euoiv+lghsju/rb3/o2tngttptbwoode7pnjx5wpso1vo7k2vrg5vd2u7ezu7r+yh0c9fsws0c6jecqhplaum0g7widtqswpdn1o+/7spvd7d1qqfokopdh1qjwrlgaeg5zgzt0nmuz9iovl9y7qr8g6+chs2e17dmuvocvpobltkfnljioshfqa4vipowph4gvyaioc5ju3uttgziyndkipwcfvxjypn1unwhlbqst1e2dn1d8bgq6vsknftxzr1bjxip95wwscky9jik6acri4fctcgsgqmrdgtficpnuee8l0votmscqedf81xykz/ovv0jtvonbtubtotk7loqrogj2gm+sgs9rct6inuoigfd2jv/rmpbkvxrvxsritgovohf2b8fkdgnmvua==</latexit> <latexit sha1_base64="2ehdqpzqsme6l8h0i0lcyad4h94=">aaab/hicbvdlssnafj3uv62vajdugkvwvrirdfl047jcx9demplo2qgtszi5euoiv+lghsju/rb3/o2tngttptbwoode7pnjx5wpso1vo7k2vrg5vd2u7ezu7r+yh0c9fsws0c6jecqhplaum0g7widtqswpdn1o+/7spvd7d1qqfokopdh1qjwrlgaeg5zgzt0nmuz9iovl9y7qr8g6+chs2e17dmuvocvpobltkfnljioshfqa4vipowph4gvyaioc5ju3uttgziyndkipwcfvxjypn1unwhlbqst1e2dn1d8bgq6vsknftxzr1bjxip95wwscky9jik6acri4fctcgsgqmrdgtficpnuee8l0votmscqedf81xykz/ovv0jtvonbtubtotk7loqrogj2gm+sgs9rct6inuoigfd2jv/rmpbkvxrvxsritgovohf2b8fkdgnmvua==</latexit> SVD Example V T From [Strang] 36
Outline Linear Algebra Basics Norms Multiplications Matrix Inversion Trace and Determinant Eigen Values and Eigen Vectors Singular Value Decomposition Matrix Calculus 37
<latexit sha1_base64="11y8g8fa9znqeubrie41aw9zolk=">aaab6nicbvbns8naej3ur1q/qh69lbbbu0leqmeif48v7qe0owy2m3bpzhn2j2ij/qlepcji1v/kzx/jts1bwx8mpn6bywzekehh0hw/ncla+sbmvng7tlo7t39qpjxqmtjvjddzlgpdcajhuijerigsdxlnarri3g7gnzo//ci1ebf6wenc/ygolqgfo2il+6f+uf+uufv3drjkvjxuieejx/7qdwkwrlwhk9syrucm6gduo2cst0u91pcesjed8q6likbc+nn81ck5s8qahlg2pzdm1d8tgy2mmusb7ywojsyynxp/87ophld+jlssildssshmjcgyzp4ma6e5qzmxhdit7k2ejaimdg06jruct/zykmldvd236t1dvurxerxfoiftoacpalchw2haexgm4rle4c2rzovz7nwswgtopnmmf+b8/gbjmi3a</latexit> <latexit sha1_base64="11y8g8fa9znqeubrie41aw9zolk=">aaab6nicbvbns8naej3ur1q/qh69lbbbu0leqmeif48v7qe0owy2m3bpzhn2j2ij/qlepcji1v/kzx/jts1bwx8mpn6bywzekehh0hw/ncla+sbmvng7tlo7t39qpjxqmtjvjddzlgpdcajhuijerigsdxlnarri3g7gnzo//ci1ebf6wenc/ygolqgfo2il+6f+uf+uufv3drjkvjxuieejx/7qdwkwrlwhk9syrucm6gduo2cst0u91pcesjed8q6likbc+nn81ck5s8qahlg2pzdm1d8tgy2mmusb7ywojsyynxp/87ophld+jlssildssshmjcgyzp4ma6e5qzmxhdit7k2ejaimdg06jruct/zykmldvd236t1dvurxerxfoiftoacpalchw2haexgm4rle4c2rzovz7nwswgtopnmmf+b8/gbjmi3a</latexit> <latexit sha1_base64="11y8g8fa9znqeubrie41aw9zolk=">aaab6nicbvbns8naej3ur1q/qh69lbbbu0leqmeif48v7qe0owy2m3bpzhn2j2ij/qlepcji1v/kzx/jts1bwx8mpn6bywzekehh0hw/ncla+sbmvng7tlo7t39qpjxqmtjvjddzlgpdcajhuijerigsdxlnarri3g7gnzo//ci1ebf6wenc/ygolqgfo2il+6f+uf+uufv3drjkvjxuieejx/7qdwkwrlwhk9syrucm6gduo2cst0u91pcesjed8q6likbc+nn81ck5s8qahlg2pzdm1d8tgy2mmusb7ywojsyynxp/87ophld+jlssildssshmjcgyzp4ma6e5qzmxhdit7k2ejaimdg06jruct/zykmldvd236t1dvurxerxfoiftoacpalchw2haexgm4rle4c2rzovz7nwswgtopnmmf+b8/gbjmi3a</latexit> <latexit sha1_base64="11y8g8fa9znqeubrie41aw9zolk=">aaab6nicbvbns8naej3ur1q/qh69lbbbu0leqmeif48v7qe0owy2m3bpzhn2j2ij/qlepcji1v/kzx/jts1bwx8mpn6bywzekehh0hw/ncla+sbmvng7tlo7t39qpjxqmtjvjddzlgpdcajhuijerigsdxlnarri3g7gnzo//ci1ebf6wenc/ygolqgfo2il+6f+uf+uufv3drjkvjxuieejx/7qdwkwrlwhk9syrucm6gduo2cst0u91pcesjed8q6likbc+nn81ck5s8qahlg2pzdm1d8tgy2mmusb7ywojsyynxp/87ophld+jlssildssshmjcgyzp4ma6e5qzmxhdit7k2ejaimdg06jruct/zykmldvd236t1dvurxerxfoiftoacpalchw2haexgm4rle4c2rzovz7nwswgtopnmmf+b8/gbjmi3a</latexit> Matrix Calculus x k 38
Summary Linear Algebra Basics Norms Multiplications Matrix Inversion Trace and Determinant Eigen Values and Eigen Vectors Singular Value Decomposition Matrix Calculus 39