Matrix Multiplicatio Data Structures ad Algorithms Adrei Bulatov
Algorithms Matrix Multiplicatio 7- Matrix Multiplicatio Matrix multiplicatio. Give two -by- matrices A ad B, compute A B. k kj ik ij b a c Brute force. arithmetic operatios. Fudametal questio. a we improve upo brute force? b b b b b b b b b a a a a a a a a a c c c c c c c c c L M O M M L L L M O M M L L L M O M M L L ( 3 ) Θ
Algorithms Matrix Multiplicatio 7-3 Matrix Multiplicatio: Divide ad oquer Divide: partitio A ad B ito ½-by-½ blocks. oquer: multiply 8 ½-by-½ recursively. ombie: add appropriate products usig 4 matrix additios. A A B B A B A B ( A B ) + ( A B) ( A B ) + ( A B ) ( A B ) + ( A B) ( A B ) + ( A B ) T( ) 3 ( / ) + Θ( ) T( ) Θ( ) 8T 443 recursive calls 3 add, form submatrices
Algorithms Matrix Multiplicatio 7-4 Better Divide ad oquer Key idea: multiply -by- block matrices with oly 7 multiplicatios. A A A A B B B B 3 4 5 6 7 A ( A ( A A ( A ( A ( A ( B ( B + A + A + A A A B ) B ) B B ) ( B ) ( B ) ( B ) ) + + + B B B ) ) ) 5 3 5 + 4 + + + 4 3 + 7 6
Algorithms Matrix Multiplicatio 7-5 Fast Matrix Multiplicatio Fast matrix multiplicatio. (Strasse, 969) Divide: partitio A ad B ito ½-by-½ blocks. ompute: 4 ½-by-½ matrices via 0 matrix additios. oquer: multiply 7 ½-by-½ matrices recursively. ombie: 7 products ito 4 terms usig 8 matrix additios. Aalysis. Assume is a power of. T() # arithmetic operatios. ( / ) T( ) 7T + Θ( ) T( ) Θ( ) O( 443 3 recursive calls add, subtract log 7.8 )
Algorithms Matrix Multiplicatio 7-6 Matrix Multiplicatio: ractice Implemetatio issues. Sparsity. achig effects. Numerical stability. Odd matrix dimesios. rossover to classical algorithm aroud 8. ommo misperceptio: "Strasse is oly a theoretical curiosity." Advaced omputatio Group at Apple omputer reports 8x speedup o G4 Velocity Egie whe ~,500. Rage of istaces where it's useful is a subject of cotroversy. Remark. a "Strasseize" Axb, determiat, eigevalues, ad other matrix operatios.
Algorithms Matrix Multiplicatio 7-7 Matrix Multiplicatio: Theory Q. Multiply two -by- matrices with oly 7 scalar multiplicatios? log 7 A. Yes! [Strasse, 969] Θ( ) O( Q. Multiply two -by- matrices with oly 6 scalar multiplicatios? log 6 A. Impossible. [Hopcroft ad Kerr, 97] Θ( ) O( Q. Two 3-by-3 matrices with oly scalar multiplicatios? log A. Also impossible. Θ( 3 ) O( Q. Two 70-by-70 matrices with oly 43,640 scalar multiplicatios? A. Yes! [a, 980] Decimal wars. December, 979: O(.583 ). Jauary, 980: O(.580 ). log 70 43640 Θ( ) O(.8.59.77.80 ) ) ) )
Algorithms Matrix Multiplicatio 7-8 Matrix Multiplicatio: Theory (ctd) Best kow. O(.376 ) [oppersmith-wiograd, 987.] ojecture. O( +ε ) for ay ε > 0. aveat: Theoretical improvemets to Strasse are progressively less practical.
Dyamic rogrammig Data Structures ad Algorithms Adrei Bulatov
Algorithms Dyamic rogrammig 7-0 Algorithmic aradigms Greed. Build up a solutio icremetally, myopically optimizig some local criterio. Divide-ad-coquer. Break up a problem ito two sub-problems, solve each sub-problem idepedetly, ad combie solutio to sub-problems to form solutio to origial problem. Dyamic programmig. Break up a problem ito a series of overlappig sub-problems, ad build up solutios to larger ad larger sub-problems.
Algorithms Dyamic rogrammig 7- Algorithmic aradigms Bellma. ioeered the systematic study of dyamic programmig i the 950s. Etymology. Dyamic programmig plaig over time. Secretary of Defese was hostile to mathematical research. Bellma sought a impressive ame to avoid cofrotatio. - "it's impossible to use dyamic i a pejorative sese" - "somethig ot eve a ogressma could object to"
Algorithms Dyamic rogrammig 7- Algorithmic aradigms Bellma. ioeered the systematic study of dyamic programmig i the 950s. Etymology. Dyamic programmig plaig over time. Secretary of Defese was hostile to mathematical research. Bellma sought a impressive ame to avoid cofrotatio. - "it's impossible to use dyamic i a pejorative sese" - "somethig ot eve a ogressma could object to"
Algorithms Dyamic rogrammig 7-3 Weighted Iterval Schedulig Weighted iterval schedulig problem. Istace A set of jobs. Job j starts at s j, fiishes at f j, ad has weight or value v j. Two jobs compatible if they do't overlap. Objective Fid maximum weight subset of mutually compatible jobs.
Algorithms Dyamic rogrammig 7-4 Uweighted Iterval Schedulig: Review Recall: Greedy algorithm works if all weights are. osider jobs i ascedig order of fiish time. Add job to subset if it is compatible with previously chose jobs. Observatio. Greedy algorithm ca fail spectacularly if arbitrary weights are allowed. weight 999 weight a b 0 3 4 5 6 7 8 9 0 Time
Algorithms Dyamic rogrammig 7-5 Weighted Iterval Schedulig Notatio: Label jobs by fiishig time: f f... f. Let p(j) be the largest idex i < j such that job i is compatible with j. Example. p(8) 5, p(7) 3, p() 0. 3 4 5 6 7 0 3 4 5 6 7 8 9 0 8 Time
Algorithms Dyamic rogrammig 7-6 Dyamic rogrammig: Biary hoice Let OT(j) deote the value of a optimal solutio to the problem cosistig of job requests,,..., j. ase : OT selects job j. caot use icompatible jobs { p(j) +, p(j) +,..., j - } must iclude optimal solutio to problem cosistig of remaiig compatible jobs,,..., p(j) ase : OT does ot select job j. optimal substructure must iclude optimal solutio to problem cosistig of remaiig compatible jobs,,..., j- OT ( j) 0 max { v + OT ( p( j)), OT ( j ) } j if j 0 otherwise
Algorithms Dyamic rogrammig 7-7 Weighted Iterval Schedulig: Brute Force Iput:, s,,s, f,,f, v,,v sort jobs by fiish times so that f f f compute p(), p(),, p() retur ompute-opt() ompute-opt(j) if (j 0) retur 0 else retur max(v j +ompute-opt(p(j)),ompute-opt(j-))
Algorithms Dyamic rogrammig 7-8 Weighted Iterval Schedulig: Brute Force Observatio. Recursive algorithm fails spectacularly because of redudat sub-problems expoetial algorithms. Example Number of recursive calls for family of "layered" istaces grows like Fiboacci sequece. 5 3 4 5 4 3 3 0 0 p() 0, p(j) j - 0
Algorithms Dyamic rogrammig 7-9 Weighted Iterval Schedulig: Memoizatio Memoizatio: Store results of each sub-problem i a cache; lookup as eeded. Iput:, s,,s, f,,f, v,,v sort jobs by fiish times so that f f f compute p(), p(),, p() set OT[0]:0 for j to do set OT[j]:max(v j +OT[p(j)],OT[j-]) edfor retur OT[]
Algorithms Dyamic rogrammig 7-0 Weighted Iterval Schedulig: Ruig Time Theorem Memoized versio of algorithm takes O( log ) time. roof Sort by fiish time: O( log ). omputig p( ) : O() after sortig by fiish time Each iteratio of the for loop: O() Overall time is O( log ) Remark. O() if jobs are pre-sorted by fiish times QED
Algorithms Dyamic rogrammig 7- Automated Memoizatio Automated memoizatio. May fuctioal programmig laguages (e.g., Lisp) have built-i support for memoizatio. Q. Why ot i imperative laguages (e.g., Java)? (defu F () (if (< ) (+ (F (- )) (F (- ))))) Lisp (efficiet) static it F(it ) { if ( < ) retur ; else retur F(-) + F(-); } Java (expoetial) F(40) F(39) F(38) F(38) F(37) F(37) F(36) F(37) F(36) F(36) F(35) F(36) F(35) F(35) F(34)
Algorithms Dyamic rogrammig 7- Fidig a Solutio Dyamic programmig algorithm computes optimal value. What if we wat the solutio itself? Do some post-processig Fid-Solutio(j) if j 0 the output othig else if v j +M[p(j)]>M[j-] the do prit j Fid-Solutio(p(j)) edif else Fid-Solutio(j-) edif