An Efficient Evolutionary Algorithm for Solving Incrementally Structured Problems
|
|
|
- Cecil Johnson
- 5 years ago
- Views:
Transcription
1 An Efficient Evolutionary Algorithm for Solving Incrementally Structured Problems Jason Ansel Maciej Pacula Saman Amarasinghe Una-May O Reilly MIT - CSAIL July 14, 2011 Jason Ansel (MIT) PetaBricks July 14, / 30
2 Who are we? I do research in programming languages (PL) and compilers The PetaBricks language is a collaboration between: A PL / compiler research group A evolutionary algorithms research group A applied mathematics research group Jason Ansel (MIT) PetaBricks July 14, / 30
3 Who are we? I do research in programming languages (PL) and compilers The PetaBricks language is a collaboration between: A PL / compiler research group A evolutionary algorithms research group A applied mathematics research group Our goal is to make programs run faster We use evolutionary algorithms to search for faster programs Jason Ansel (MIT) PetaBricks July 14, / 30
4 Who are we? I do research in programming languages (PL) and compilers The PetaBricks language is a collaboration between: A PL / compiler research group A evolutionary algorithms research group A applied mathematics research group Our goal is to make programs run faster We use evolutionary algorithms to search for faster programs The PetaBricks language defines search spaces of algorithmic choices Jason Ansel (MIT) PetaBricks July 14, / 30
5 A motivating example How would you write a fast sorting algorithm? Jason Ansel (MIT) PetaBricks July 14, / 30
6 A motivating example How would you write a fast sorting algorithm? Insertion sort Quick sort Merge sort Radix sort Jason Ansel (MIT) PetaBricks July 14, / 30
7 A motivating example How would you write a fast sorting algorithm? Insertion sort Quick sort Merge sort Radix sort Binary tree sort, Bitonic sort, Bubble sort, Bucket sort, Burstsort, Cocktail sort, Comb sort, Counting Sort, Distribution sort, Flashsort, Heapsort, Introsort, Library sort, Odd-even sort, Postman sort, Samplesort, Selection sort, Shell sort, Stooge sort, Strand sort, Timsort? Jason Ansel (MIT) PetaBricks July 14, / 30
8 A motivating example How would you write a fast sorting algorithm? Insertion sort Quick sort Merge sort Radix sort Binary tree sort, Bitonic sort, Bubble sort, Bucket sort, Burstsort, Cocktail sort, Comb sort, Counting Sort, Distribution sort, Flashsort, Heapsort, Introsort, Library sort, Odd-even sort, Postman sort, Samplesort, Selection sort, Shell sort, Stooge sort, Strand sort, Timsort? Poly-algorithms Jason Ansel (MIT) PetaBricks July 14, / 30
9 std::stable sort /usr/include/c++/4.5.2/bits/stl algo.h lines Jason Ansel (MIT) PetaBricks July 14, / 30
10 std::stable sort /usr/include/c++/4.5.2/bits/stl algo.h lines Jason Ansel (MIT) PetaBricks July 14, / 30
11 Is 15 the right number? The best cutoff (CO) changes Depends on competing costs: Cost of computation (< operator, call overhead, etc) Cost of communication (swaps) Cache behavior (misses, prefetcher, locality) Jason Ansel (MIT) PetaBricks July 14, / 30
12 Is 15 the right number? The best cutoff (CO) changes Depends on competing costs: Cost of computation (< operator, call overhead, etc) Cost of communication (swaps) Cache behavior (misses, prefetcher, locality) Sorting doubles with std::stable sort: CO 200 optimal on a Phenom 905e (15% speedup over CO = 15) CO 400 optimal on a Opteron 6168 (15% speedup over CO = 15) CO 500 optimal on a Xeon E5320 (34% speedup over CO = 15) CO 700 optimal on a Xeon X5460 (25% speedup over CO = 15) Jason Ansel (MIT) PetaBricks July 14, / 30
13 Is 15 the right number? The best cutoff (CO) changes Depends on competing costs: Cost of computation (< operator, call overhead, etc) Cost of communication (swaps) Cache behavior (misses, prefetcher, locality) Sorting doubles with std::stable sort: CO 200 optimal on a Phenom 905e (15% speedup over CO = 15) CO 400 optimal on a Opteron 6168 (15% speedup over CO = 15) CO 500 optimal on a Xeon E5320 (34% speedup over CO = 15) CO 700 optimal on a Xeon X5460 (25% speedup over CO = 15) If the best cutoff has changed, perhaps best algorithm has also changed Jason Ansel (MIT) PetaBricks July 14, / 30
14 Algorithmic choices Language e i t h e r { I n s e r t i o n S o r t ( out, i n ) ; } or { QuickSort ( out, i n ) ; } or { MergeSort ( out, i n ) ; } or { R a d i x S o r t ( out, i n ) ; } Jason Ansel (MIT) PetaBricks July 14, / 30
15 Algorithmic choices Language e i t h e r { I n s e r t i o n S o r t ( out, i n ) ; } or { QuickSort ( out, i n ) ; } or { MergeSort ( out, i n ) ; } or { R a d i x S o r t ( out, i n ) ; } Representation Decision tree synthesized by our evolutionary algorithm Jason Ansel (MIT) PetaBricks July 14, / 30
16 Decision trees Optimized for a Xeon E7340 (8 cores): N < 600 Insertion Sort N < 1420 Quick Sort Merge Sort (2-way) Text notation (will be used later): I 600 Q 1420 M 2 Jason Ansel (MIT) PetaBricks July 14, / 30
17 Decision trees Optimized for Sun Fire T200 Niagara (8 cores): N < 1461 N < 75 N < 2400 Merge Sort (8-way) Merge Sort (4-way) Merge Sort (2-way) Text notation: M M M M 2 Jason Ansel (MIT) PetaBricks July 14, / 30
18 The configuration encoded by the genome Decision trees Algorithm parameters (integers, floats) Parallel scheduling / blocking parameters (integers) Synthesized scalar functions (not used in the benchmarks shown) The average PetaBricks benchmark s genome has: 1.9 decision trees 10.1 algorithm/parallelism/blocking parameters 0.6 synthesized scalar functions possible configurations Jason Ansel (MIT) PetaBricks July 14, / 30
19 Outline 1 PetaBricks Language 2 Autotuning Problem 3 INCREA 4 Evaluation 5 Conclusions Jason Ansel (MIT) PetaBricks July 14, / 30
20 PetaBricks programs at runtime Request Program Response Jason Ansel (MIT) PetaBricks July 14, / 30
21 PetaBricks programs at runtime Request Program Response Configuration: - point in ~100D space Measurement: - performance - accuracy (QoS) Jason Ansel (MIT) PetaBricks July 14, / 30
22 PetaBricks programs at runtime Request Program Response Configuration: - point in ~100D space Measurement: - performance - accuracy (QoS) Offline Autotuning Jason Ansel (MIT) PetaBricks July 14, / 30
23 The challenges Evaluating objective function is expensive Must run the program (at least once) More expensive for unfit solutions Scales poorly with larger problem sizes Fitness is noisy Randomness from parallel races and system noise Testing each candidate only once often produces an worse algorithm Running many trials is expensive Decision tree structures are complex Theoretically infinite size We artificially bound them to bits (23 ints) each Jason Ansel (MIT) PetaBricks July 14, / 30
24 Contrast two evolutionary approaches GPEA: General Purpose Evolutionary Algorithm Used as a baseline INCREA: Incremental Evolutionary Algorithm Bottom-up approach Noisy fitness evaluation strategy Domain informed mutation operators Jason Ansel (MIT) PetaBricks July 14, / 30
25 General purpose evolution algorithm (GPEA) Initial population???? Cost = 0 Jason Ansel (MIT) PetaBricks July 14, / 30
26 General purpose evolution algorithm (GPEA) Initial population 72.7s??? Cost = 72.7 Jason Ansel (MIT) PetaBricks July 14, / 30
27 General purpose evolution algorithm (GPEA) Initial population 72.7s 10.5s?? Cost = 83.2 Jason Ansel (MIT) PetaBricks July 14, / 30
28 General purpose evolution algorithm (GPEA) Initial population 72.7s 10.5s 4.1s? Cost = 87.3 Jason Ansel (MIT) PetaBricks July 14, / 30
29 General purpose evolution algorithm (GPEA) Initial population 72.7s 10.5s 4.1s 31.2s Cost = Jason Ansel (MIT) PetaBricks July 14, / 30
30 General purpose evolution algorithm (GPEA) Initial population 72.7s 10.5s 4.1s 31.2s Cost = Generation 2???? Cost = 0 Jason Ansel (MIT) PetaBricks July 14, / 30
31 General purpose evolution algorithm (GPEA) Initial population 72.7s 10.5s 4.1s 31.2s Cost = Generation 2 4.2s 5.1s 2.6s 13.2s Cost = 25.1 Jason Ansel (MIT) PetaBricks July 14, / 30
32 General purpose evolution algorithm (GPEA) Initial population 72.7s 10.5s 4.1s 31.2s Cost = Generation 2 4.2s 5.1s 2.6s 13.2s Cost = 25.1 Generation 3???? Cost = 0 Jason Ansel (MIT) PetaBricks July 14, / 30
33 General purpose evolution algorithm (GPEA) Initial population 72.7s 10.5s 4.1s 31.2s Cost = Generation 2 4.2s 5.1s 2.6s 13.2s Cost = 25.1 Generation 3 2.8s 0.1s 3.8s 2.3s Cost = 9.0 Jason Ansel (MIT) PetaBricks July 14, / 30
34 General purpose evolution algorithm (GPEA) Initial population 72.7s 10.5s 4.1s 31.2s Cost = Generation 2 4.2s 5.1s 2.6s 13.2s Cost = 25.1 Generation 3 2.8s 0.1s 3.8s 2.3s Cost = 9.0 Generation 4???? Cost = 0 Jason Ansel (MIT) PetaBricks July 14, / 30
35 General purpose evolution algorithm (GPEA) Initial population 72.7s 10.5s 4.1s 31.2s Cost = Generation 2 4.2s 5.1s 2.6s 13.2s Cost = 25.1 Generation 3 2.8s 0.1s 3.8s 2.3s Cost = 9.0 Generation 4 0.3s 0.1s 0.4s 2.4s Cost = 3.2 Jason Ansel (MIT) PetaBricks July 14, / 30
36 General purpose evolution algorithm (GPEA) Initial population 72.7s 10.5s 4.1s 31.2s Cost = Generation 2 4.2s 5.1s 2.6s 13.2s Cost = 25.1 Generation 3 2.8s 0.1s 3.8s 2.3s Cost = 9.0 Generation 4 0.3s 0.1s 0.4s 2.4s Cost = 3.2 Cost of autotuning front-loaded in initial (unfit) population We could speed up tuning if we start with a faster initial population Jason Ansel (MIT) PetaBricks July 14, / 30
37 General purpose evolution algorithm (GPEA) Initial population 72.7s 10.5s 4.1s 31.2s Cost = Generation 2 4.2s 5.1s 2.6s 13.2s Cost = 25.1 Generation 3 2.8s 0.1s 3.8s 2.3s Cost = 9.0 Generation 4 0.3s 0.1s 0.4s 2.4s Cost = 3.2 Cost of autotuning front-loaded in initial (unfit) population We could speed up tuning if we start with a faster initial population Key insight Smaller input sizes can be used to form better initial population Jason Ansel (MIT) PetaBricks July 14, / 30
38 Bottom-up evolutionary algorithm Train on input size 64 Jason Ansel (MIT) PetaBricks July 14, / 30
39 Bottom-up evolutionary algorithm Train on input size 32, to form initial population for: Train on input size 64 Jason Ansel (MIT) PetaBricks July 14, / 30
40 Bottom-up evolutionary algorithm Train on input size 16, to form initial population for: Train on input size 32, to form initial population for: Train on input size 64 Jason Ansel (MIT) PetaBricks July 14, / 30
41 Bottom-up evolutionary algorithm Train on input size 1, to form initial population for: Train on input size 2, to form initial population for: Train on input size 8, to form initial population for: Train on input size 16, to form initial population for: Train on input size 32, to form initial population for: Train on input size 64 Naturally exploits optimal substructure of problems Jason Ansel (MIT) PetaBricks July 14, / 30
42 Noisy fitness evaluation Both strategies terminate slow tests early GPEA uses 1 trial per candidate algorithm INCREA adaptively changes the number of trials Represents fitness as a probability distribution Runs a single tailed t-test to get confidence in differences Runs more trails if confidence is low Jason Ansel (MIT) PetaBricks July 14, / 30
43 Domain informed mutation operators Mutation operators deal with larger structures in the genome Add algorithm Y to the top of decision tree X Scale cutoff X using a lognormal distribution Generated fully automatically by our compiler Jason Ansel (MIT) PetaBricks July 14, / 30
44 Outline 1 PetaBricks Language 2 Autotuning Problem 3 INCREA 4 Evaluation 5 Conclusions Jason Ansel (MIT) PetaBricks July 14, / 30
45 Experimental Setup Measuring convergence time Important to both program users and developers Vital in online autotuning Three fixed-accuracy PetaBricks programs: Sort 2 20 (small input size) Sort 2 23 (large input size) Matrix multiply Eigenvector solve Representative runs Average of 30 runs, with tests for statistical significance in paper Run an 8-core Xeon running Debian 5.0 Jason Ansel (MIT) PetaBricks July 14, / 30
46 Sort 2 20 : training input size 1e+07 1e+06 Training Input Size INCREA GPEA Training Time Jason Ansel (MIT) PetaBricks July 14, / 30
47 Sort 2 20 : candidates tested INCREA GPEA Tests Conducted Training Time Jason Ansel (MIT) PetaBricks July 14, / 30
48 Sort 2 20 : performance Best Candidate (s) INCREA GPEA Training Time Jason Ansel (MIT) PetaBricks July 14, / 30
49 Sort 2 23 : performance Best Candidate (s) INCREA GPEA Training Time Jason Ansel (MIT) PetaBricks July 14, / 30
50 INCREA: Sort, best algorithm at each generation Input size Training time (s) Genome Q 64 Q p Q 64 Q p I I I 270 R 1310 R p I 270 Q 1310 Q p I 270 Q 1310 Q p I 270 Q 3343 Q p I 189 R R p I 189 R R p I 189 R R p I 189 R R p I 189 R M I 189 Q 5585 Q p I 189 Q 5585 Q p I 456 Q 5585 Q p I 456 Q 5585 Q p I = insertion-sort Q = quick-sort R = radix-sort M x = x-way merge-sort p indicates run in parallel Jason Ansel (MIT) PetaBricks July 14, / 30
51 INCREA: Sort, best algorithm at each generation Input size Training time (s) Genome Q 64 Q p Q 64 Q p I I I 270 R 1310 R p I 270 Q 1310 Q p I 270 Q 1310 Q p I 270 Q 3343 Q p I 189 R R p I 189 R R p I 189 R R p I 189 R R p I 189 R M I 189 Q 5585 Q p I 189 Q 5585 Q p I 456 Q 5585 Q p I 456 Q 5585 Q p I = insertion-sort Q = quick-sort R = radix-sort M x = x-way merge-sort p indicates run in parallel Jason Ansel (MIT) PetaBricks July 14, / 30
52 INCREA: Sort, best algorithm at each generation Input size Training time (s) Genome Q 64 Q p Q 64 Q p I I I 270 R 1310 R p I 270 Q 1310 Q p I 270 Q 1310 Q p I 270 Q 3343 Q p I 189 R R p I 189 R R p I 189 R R p I 189 R R p I 189 R M I 189 Q 5585 Q p I 189 Q 5585 Q p I 456 Q 5585 Q p I 456 Q 5585 Q p I = insertion-sort Q = quick-sort R = radix-sort M x = x-way merge-sort p indicates run in parallel Jason Ansel (MIT) PetaBricks July 14, / 30
53 INCREA: Sort, best algorithm at each generation Input size Training time (s) Genome Q 64 Q p Q 64 Q p I I I 270 R 1310 R p I 270 Q 1310 Q p I 270 Q 1310 Q p I 270 Q 3343 Q p I 189 R R p I 189 R R p I 189 R R p I 189 R R p I 189 R M I 189 Q 5585 Q p I 189 Q 5585 Q p I 456 Q 5585 Q p I 456 Q 5585 Q p I = insertion-sort Q = quick-sort R = radix-sort M x = x-way merge-sort p indicates run in parallel Jason Ansel (MIT) PetaBricks July 14, / 30
54 INCREA: Sort, best algorithm at each generation Input size Training time (s) Genome Q 64 Q p Q 64 Q p I I I 270 R 1310 R p I 270 Q 1310 Q p I 270 Q 1310 Q p I 270 Q 3343 Q p I 189 R R p I 189 R R p I 189 R R p I 189 R R p I 189 R M I 189 Q 5585 Q p I 189 Q 5585 Q p I 456 Q 5585 Q p I 456 Q 5585 Q p I = insertion-sort Q = quick-sort R = radix-sort M x = x-way merge-sort p indicates run in parallel Jason Ansel (MIT) PetaBricks July 14, / 30
55 INCREA: Sort, best algorithm at each generation Input size Training time (s) Genome Q 64 Q p Q 64 Q p I I I 270 R 1310 R p I 270 Q 1310 Q p I 270 Q 1310 Q p I 270 Q 3343 Q p I 189 R R p I 189 R R p I 189 R R p I 189 R R p I 189 R M I 189 Q 5585 Q p I 189 Q 5585 Q p I 456 Q 5585 Q p I 456 Q 5585 Q p I = insertion-sort Q = quick-sort R = radix-sort M x = x-way merge-sort p indicates run in parallel Jason Ansel (MIT) PetaBricks July 14, / 30
56 GPEA: Sort, best algorithm at each generation Generation Training time (s) Genome I 448 R I 413 R I 448 R I 448 Q I 448 Q I 448 Q I 448 Q 4096 Q p I 448 Q 4096 Q p I 448 Q 4096 Q p I 448 Q 4096 Q p I 371 Q 4096 Q p I 272 Q 4096 Q p I 272 Q 4096 Q p I 272 Q 4096 Q p I 272 Q 4096 Q p I 370 Q 8192 Q p I 370 Q 8192 Q p... I = insertion-sort Q = quick-sort R = radix-sort M x = x-way merge-sort p indicates run in parallel Jason Ansel (MIT) PetaBricks July 14, / 30
57 Matrix Multiply (input size 1024x1024) Best Candidate (s) INCREA GPEA Training Time Jason Ansel (MIT) PetaBricks July 14, / 30
58 Eigenvector Solve (input size 1024x1024) Best Candidate (s) INCREA GPEA Training Time Jason Ansel (MIT) PetaBricks July 14, / 30
59 Outline 1 PetaBricks Language 2 Autotuning Problem 3 INCREA 4 Evaluation 5 Conclusions Jason Ansel (MIT) PetaBricks July 14, / 30
60 Conclusions Take away The technique of solving incrementally structured problems by exploiting knowledge from smaller problem instances may be more broadly applicable. Take away PetaBricks is a useful framework for comparing techniques for autotuning programs. Jason Ansel (MIT) PetaBricks July 14, / 30
61 Thanks! Questions? Jason Ansel (MIT) PetaBricks July 14, / 30
Language and Compiler Support for Auto-Tuning Variable-Accuracy Algorithms
Language and Compiler Support for Auto-Tuning Variable-Accuracy Algorithms Jason Ansel Yee Lok Wong Cy Chan Marek Olszewski Alan Edelman Saman Amarasinghe MIT - CSAIL April 4, 2011 Jason Ansel (MIT) PetaBricks
PetaBricks: Variable Accuracy and Online Learning
PetaBricks: Variable Accuracy and Online Learning Jason Ansel MIT - CSAIL May 4, 2011 Jason Ansel (MIT) PetaBricks May 4, 2011 1 / 40 Outline 1 Motivating Example 2 PetaBricks Language Overview 3 Variable
Data Structures and Algorithms
Data Structures and Algorithms Spring 2017-2018 Outline 1 Sorting Algorithms (contd.) Outline Sorting Algorithms (contd.) 1 Sorting Algorithms (contd.) Analysis of Quicksort Time to sort array of length
Sorting. CMPS 2200 Fall Carola Wenk Slides courtesy of Charles Leiserson with changes and additions by Carola Wenk
CMPS 2200 Fall 2017 Sorting Carola Wenk Slides courtesy of Charles Leiserson with changes and additions by Carola Wenk 11/17/17 CMPS 2200 Intro. to Algorithms 1 How fast can we sort? All the sorting algorithms
Linear Selection and Linear Sorting
Analysis of Algorithms Linear Selection and Linear Sorting If more than one question appears correct, choose the more specific answer, unless otherwise instructed. Concept: linear selection 1. Suppose
Genetic Algorithm: introduction
1 Genetic Algorithm: introduction 2 The Metaphor EVOLUTION Individual Fitness Environment PROBLEM SOLVING Candidate Solution Quality Problem 3 The Ingredients t reproduction t + 1 selection mutation recombination
Analytical Modeling of Parallel Systems
Analytical Modeling of Parallel Systems Chieh-Sen (Jason) Huang Department of Applied Mathematics National Sun Yat-sen University Thank Ananth Grama, Anshul Gupta, George Karypis, and Vipin Kumar for providing
Algorithms. Algorithms 2.2 MERGESORT. mergesort bottom-up mergesort sorting complexity divide-and-conquer ROBERT SEDGEWICK KEVIN WAYNE
Algorithms ROBERT SEDGEWICK KEVIN WAYNE 2.2 MERGESORT Algorithms F O U R T H E D I T I O N mergesort bottom-up mergesort sorting complexity divide-and-conquer ROBERT SEDGEWICK KEVIN WAYNE http://algs4.cs.princeton.edu
Sorting Algorithms. We have already seen: Selection-sort Insertion-sort Heap-sort. We will see: Bubble-sort Merge-sort Quick-sort
Sorting Algorithms We have already seen: Selection-sort Insertion-sort Heap-sort We will see: Bubble-sort Merge-sort Quick-sort We will show that: O(n log n) is optimal for comparison based sorting. Bubble-Sort
Data Structures and Algorithms
Data Structures and Algorithms Autumn 2018-2019 Outline 1 Algorithm Analysis (contd.) Outline Algorithm Analysis (contd.) 1 Algorithm Analysis (contd.) Growth Rates of Some Commonly Occurring Functions
Sorting. CS 3343 Fall Carola Wenk Slides courtesy of Charles Leiserson with small changes by Carola Wenk
CS 3343 Fall 2010 Sorting Carola Wenk Slides courtesy of Charles Leiserson with small changes by Carola Wenk 10/7/10 CS 3343 Analysis of Algorithms 1 How fast can we sort? All the sorting algorithms we
Lecture 1: Asymptotics, Recurrences, Elementary Sorting
Lecture 1: Asymptotics, Recurrences, Elementary Sorting Instructor: Outline 1 Introduction to Asymptotic Analysis Rate of growth of functions Comparing and bounding functions: O, Θ, Ω Specifying running
Sorting. Chapter 11. CSE 2011 Prof. J. Elder Last Updated: :11 AM
Sorting Chapter 11-1 - Sorting Ø We have seen the advantage of sorted data representations for a number of applications q Sparse vectors q Maps q Dictionaries Ø Here we consider the problem of how to efficiently
Sorting algorithms. Sorting algorithms
Properties of sorting algorithms A sorting algorithm is Comparison based If it works by pairwise key comparisons. In place If only a constant number of elements of the input array are ever stored outside
Fundamental Algorithms
Chapter 2: Sorting, Winter 2018/19 1 Fundamental Algorithms Chapter 2: Sorting Jan Křetínský Winter 2018/19 Chapter 2: Sorting, Winter 2018/19 2 Part I Simple Sorts Chapter 2: Sorting, Winter 2018/19 3
Fundamental Algorithms
Fundamental Algorithms Chapter 2: Sorting Harald Räcke Winter 2015/16 Chapter 2: Sorting, Winter 2015/16 1 Part I Simple Sorts Chapter 2: Sorting, Winter 2015/16 2 The Sorting Problem Definition Sorting
Algorithms and Their Complexity
CSCE 222 Discrete Structures for Computing David Kebo Houngninou Algorithms and Their Complexity Chapter 3 Algorithm An algorithm is a finite sequence of steps that solves a problem. Computational complexity
Divide and Conquer Algorithms. CSE 101: Design and Analysis of Algorithms Lecture 14
Divide and Conquer Algorithms CSE 101: Design and Analysis of Algorithms Lecture 14 CSE 101: Design and analysis of algorithms Divide and conquer algorithms Reading: Sections 2.3 and 2.4 Homework 6 will
Insertion Sort. We take the first element: 34 is sorted
Insertion Sort Idea: by Ex Given the following sequence to be sorted 34 8 64 51 32 21 When the elements 1, p are sorted, then the next element, p+1 is inserted within these to the right place after some
COMP 382: Reasoning about algorithms
Fall 2014 Unit 4: Basics of complexity analysis Correctness and efficiency So far, we have talked about correctness and termination of algorithms What about efficiency? Running time of an algorithm For
Fast Algorithms for Segmented Regression
Fast Algorithms for Segmented Regression Jayadev Acharya 1 Ilias Diakonikolas 2 Jerry Li 1 Ludwig Schmidt 1 1 MIT 2 USC June 21, 2016 1 / 21 Statistical vs computational tradeoffs? General Motivating Question
CS 700: Quantitative Methods & Experimental Design in Computer Science
CS 700: Quantitative Methods & Experimental Design in Computer Science Sanjeev Setia Dept of Computer Science George Mason University Logistics Grade: 35% project, 25% Homework assignments 20% midterm,
Accelerating Decoupled Look-ahead to Exploit Implicit Parallelism
Accelerating Decoupled Look-ahead to Exploit Implicit Parallelism Raj Parihar Advisor: Prof. Michael C. Huang March 22, 2013 Raj Parihar Accelerating Decoupled Look-ahead to Exploit Implicit Parallelism
Chapter 8: Introduction to Evolutionary Computation
Computational Intelligence: Second Edition Contents Some Theories about Evolution Evolution is an optimization process: the aim is to improve the ability of an organism to survive in dynamically changing
Outline. 1 Merging. 2 Merge Sort. 3 Complexity of Sorting. 4 Merge Sort and Other Sorts 2 / 10
Merge Sort 1 / 10 Outline 1 Merging 2 Merge Sort 3 Complexity of Sorting 4 Merge Sort and Other Sorts 2 / 10 Merging Merge sort is based on a simple operation known as merging: combining two ordered arrays
Central Algorithmic Techniques. Iterative Algorithms
Central Algorithmic Techniques Iterative Algorithms Code Representation of an Algorithm class InsertionSortAlgorithm extends SortAlgorithm { void sort(int a[]) throws Exception { for (int i = 1; i < a.length;
COL 730: Parallel Programming
COL 730: Parallel Programming PARALLEL SORTING Bitonic Merge and Sort Bitonic sequence: {a 0, a 1,, a n-1 }: A sequence with a monotonically increasing part and a monotonically decreasing part For some
Data selection. Lower complexity bound for sorting
Data selection. Lower complexity bound for sorting Lecturer: Georgy Gimel farb COMPSCI 220 Algorithms and Data Structures 1 / 12 1 Data selection: Quickselect 2 Lower complexity bound for sorting 3 The
Sorting and Searching. Tony Wong
Tony Wong 2017-03-04 Sorting Sorting is the reordering of array elements into a specific order (usually ascending) For example, array a[0..8] consists of the final exam scores of the n = 9 students in
What s the best data structure for multivariate polynomials in a world of 64 bit multicore computers?
What s the best data structure for multivariate polynomials in a world of 64 bit multicore computers? Michael Monagan Center for Experimental and Constructive Mathematics Simon Fraser University British
Data Compression Techniques
Data Compression Techniques Part 2: Text Compression Lecture 5: Context-Based Compression Juha Kärkkäinen 14.11.2017 1 / 19 Text Compression We will now look at techniques for text compression. These techniques
Chapter 5. Divide and Conquer CLRS 4.3. Slides by Kevin Wayne. Copyright 2005 Pearson-Addison Wesley. All rights reserved.
Chapter 5 Divide and Conquer CLRS 4.3 Slides by Kevin Wayne. Copyright 2005 Pearson-Addison Wesley. All rights reserved. 1 Divide-and-Conquer Divide-and-conquer. Break up problem into several parts. Solve
Introduction to Randomized Algorithms III
Introduction to Randomized Algorithms III Joaquim Madeira Version 0.1 November 2017 U. Aveiro, November 2017 1 Overview Probabilistic counters Counting with probability 1 / 2 Counting with probability
Information-Based Similarity Index
Information-Based Similarity Index Albert C.-C. Yang, Ary L Goldberger, C.-K. Peng This document can also be read on-line at http://physionet.org/physiotools/ibs/doc/. The most recent version of the software
MIDTERM I CMPS Winter 2013 Warmuth
test I 1 MIDTERM I CMPS 101 - Winter 2013 Warmuth With Solutions NAME: Student ID: This exam is closed book, notes, computer and cell phone. Show partial solutions to get partial credit. Make sure you
CS 361, Lecture 14. Outline
Bucket Sort CS 361, Lecture 14 Jared Saia University of New Mexico Bucket sort assumes that the input is drawn from a uniform distribution over the range [0, 1) Basic idea is to divide the interval [0,
POLY : A new polynomial data structure for Maple 17 that improves parallel speedup.
: A new polynomial data structure for Maple 7 that improves parallel speedup. Department of Mathematics, Simon Fraser University British Columbia, CANADA Parallel Computer Algebra Applications ACA 22,
Mat Week 6. Fall Mat Week 6. Algorithms. Properties. Examples. Searching. Sorting. Time Complexity. Example. Properties.
Fall 2013 Student Responsibilities Reading: Textbook, Section 3.1 3.2 Assignments: 1. for sections 3.1 and 3.2 2. Worksheet #4 on Execution s 3. Worksheet #5 on Growth Rates Attendance: Strongly Encouraged
HYCOM and Navy ESPC Future High Performance Computing Needs. Alan J. Wallcraft. COAPS Short Seminar November 6, 2017
HYCOM and Navy ESPC Future High Performance Computing Needs Alan J. Wallcraft COAPS Short Seminar November 6, 2017 Forecasting Architectural Trends 3 NAVY OPERATIONAL GLOBAL OCEAN PREDICTION Trend is higher
Bin Sort. Sorting integers in Range [1,...,n] Add all elements to table and then
![Bin Sort. Sorting integers in Range [1,...,n] Add all elements to table and then](/thumbs/87/95070515.jpg "Bin Sort. Sorting integers in Range [1,...,n] Add all elements to table and then")
Sorting1 Bin Sort Sorting integers in Range [1,...,n] Add all elements to table and then Retrieve in order 1,2,3,...,n Stable Sorting Method (repeated elements will end up in their original order) Numbers
Algorithms. Jordi Planes. Escola Politècnica Superior Universitat de Lleida
Algorithms Jordi Planes Escola Politècnica Superior Universitat de Lleida 2016 Syllabus What s been done Formal specification Computational Cost Transformation recursion iteration Divide and conquer Sorting
Student Responsibilities Week 6. Mat Properties of Algorithms. 3.1 Algorithms. Finding the Maximum Value in a Finite Sequence Pseudocode
Student Responsibilities Week 6 Mat 345 Week 6 Reading: Textbook, Section 3.1 3. Assignments: 1. for sections 3.1 and 3.. Worksheet #4 on Execution Times 3. Worksheet #5 on Growth Rates Attendance: Strongly
Lecture 17: Trees and Merge Sort 10:00 AM, Oct 15, 2018
CS17 Integrated Introduction to Computer Science Klein Contents Lecture 17: Trees and Merge Sort 10:00 AM, Oct 15, 2018 1 Tree definitions 1 2 Analysis of mergesort using a binary tree 1 3 Analysis of
Parallel Transposition of Sparse Data Structures
Parallel Transposition of Sparse Data Structures Hao Wang, Weifeng Liu, Kaixi Hou, Wu-chun Feng Department of Computer Science, Virginia Tech Niels Bohr Institute, University of Copenhagen Scientific Computing
Gecco 2007 Tutorial / Grammatical Evolution
Gecco 2007 Grammatical Evolution Tutorial Conor Ryan Biocomputing and Developmental Systems Group Department of Computer Science and Information Systems University of Limerick Copyright is held by the
POLY : A new polynomial data structure for Maple.
: A new polynomial data structure for Maple. Center for Experimental and Constructive Mathematics Simon Fraser University British Columbia CANADA ASCM 2012, Beijing, October 26-28, 2012 This is joint work
Lecture 2: MergeSort. CS 341: Algorithms. Thu, Jan 10 th 2019
Lecture 2: MergeSort CS 341: Algorithms Thu, Jan 10 th 2019 Outline For Today 1. Example 1: Sorting-Merge Sort-Divide & Conquer 2 Sorting u Input: An array of integers in arbitrary order 10 2 37 5 9 55
Minisymposia 9 and 34: Avoiding Communication in Linear Algebra. Jim Demmel UC Berkeley bebop.cs.berkeley.edu
Minisymposia 9 and 34: Avoiding Communication in Linear Algebra Jim Demmel UC Berkeley bebop.cs.berkeley.edu Motivation (1) Increasing parallelism to exploit From Top500 to multicores in your laptop Exponentially
Sparse Polynomial Multiplication and Division in Maple 14
Sparse Polynomial Multiplication and Division in Maple 4 Michael Monagan and Roman Pearce Department of Mathematics, Simon Fraser University Burnaby B.C. V5A S6, Canada October 5, 9 Abstract We report
ITEC2620 Introduction to Data Structures
ITEC2620 Introduction to Data Structures Lecture 6a Complexity Analysis Recursive Algorithms Complexity Analysis Determine how the processing time of an algorithm grows with input size What if the algorithm
CPS 616 DIVIDE-AND-CONQUER 6-1
CPS 616 DIVIDE-AND-CONQUER 6-1 DIVIDE-AND-CONQUER Approach 1. Divide instance of problem into two or more smaller instances 2. Solve smaller instances recursively 3. Obtain solution to original (larger)
Mathematical Induction
Mathematical Induction MAT231 Transition to Higher Mathematics Fall 2014 MAT231 (Transition to Higher Math) Mathematical Induction Fall 2014 1 / 21 Outline 1 Mathematical Induction 2 Strong Mathematical
4: Some Model Programming Tasks
4: Some Model Programming Tasks John Burkardt Information Technology Department Virginia Tech... FDI Summer Track V: Parallel Programming... http://people.sc.fsu.edu/ jburkardt/presentations/... parallel
Lecture 4. Quicksort
Lecture 4. Quicksort T. H. Cormen, C. E. Leiserson and R. L. Rivest Introduction to Algorithms, 3rd Edition, MIT Press, 2009 Sungkyunkwan University Hyunseung Choo choo@skku.edu Copyright 2000-2018 Networking
CMPSCI611: Three Divide-and-Conquer Examples Lecture 2
CMPSCI611: Three Divide-and-Conquer Examples Lecture 2 Last lecture we presented and analyzed Mergesort, a simple divide-and-conquer algorithm. We then stated and proved the Master Theorem, which gives
Algorithms and Data Structures
Algorithms and Data Structures, Divide and Conquer Albert-Ludwigs-Universität Freiburg Prof. Dr. Rolf Backofen Bioinformatics Group / Department of Computer Science Algorithms and Data Structures, December
Algorithms, Design and Analysis. Order of growth. Table 2.1. Big-oh. Asymptotic growth rate. Types of formulas for basic operation count
Types of formulas for basic operation count Exact formula e.g., C(n) = n(n-1)/2 Algorithms, Design and Analysis Big-Oh analysis, Brute Force, Divide and conquer intro Formula indicating order of growth
CSCE 222 Discrete Structures for Computing
CSCE 222 Discrete Structures for Computing Algorithms Dr. Philip C. Ritchey Introduction An algorithm is a finite sequence of precise instructions for performing a computation or for solving a problem.
Review Of Topics. Review: Induction
Review Of Topics Asymptotic notation Solving recurrences Sorting algorithms Insertion sort Merge sort Heap sort Quick sort Counting sort Radix sort Medians/order statistics Randomized algorithm Worst-case
Hydra: Generation and Tuning of parallel solutions for linear algebra equations. Alexandre X. Duchâteau University of Illinois at Urbana Champaign
Hydra: Generation and Tuning of parallel solutions for linear algebra equations Alexandre X. Duchâteau University of Illinois at Urbana Champaign Collaborators Thesis Advisors Denis Barthou (Labri/INRIA
Chapter 5. Divide and Conquer CLRS 4.3. Slides by Kevin Wayne. Copyright 2005 Pearson-Addison Wesley. All rights reserved.
Chapter 5 Divide and Conquer CLRS 4.3 Slides by Kevin Wayne. Copyright 25 Pearson-Addison Wesley. All rights reserved. Divide-and-Conquer Divide-and-conquer. Break up problem into several parts. Solve
A Brief Introduction to Multiobjective Optimization Techniques
Università di Catania Dipartimento di Ingegneria Informatica e delle Telecomunicazioni A Brief Introduction to Multiobjective Optimization Techniques Maurizio Palesi Maurizio Palesi [mpalesi@diit.unict.it]
Lower and Upper Bound Theory
Lower and Upper Bound Theory Adnan YAZICI Dept. of Computer Engineering Middle East Technical Univ. Ankara - TURKEY 1 Lower and Upper Bound Theory How fast can we sort? Most of the sorting algorithms are
Dense Arithmetic over Finite Fields with CUMODP
Dense Arithmetic over Finite Fields with CUMODP Sardar Anisul Haque 1 Xin Li 2 Farnam Mansouri 1 Marc Moreno Maza 1 Wei Pan 3 Ning Xie 1 1 University of Western Ontario, Canada 2 Universidad Carlos III,
Modeling and Tuning Parallel Performance in Dense Linear Algebra
Modeling and Tuning Parallel Performance in Dense Linear Algebra Initial Experiences with the Tile QR Factorization on a Multi Core System CScADS Workshop on Automatic Tuning for Petascale Systems Snowbird,
On Adaptive Integer Sorting
On Adaptive Integer Sorting Anna Pagh 1, Rasmus Pagh 1, and Mikkel Thorup 2 1 IT University of Copenhagen, Rued Langgaardsvej 7, 2300 København S, Denmark {annao, pagh}@itu.dk 2 AT&T Labs Research, Shannon
ELEMENTARY NUMBER THEORY AND METHODS OF PROOF
CHAPTER 4 ELEMENTARY NUMBER THEORY AND METHODS OF PROOF Copyright Cengage Learning. All rights reserved. SECTION 4.5 Direct Proof and Counterexample V: Floor and Ceiling Copyright Cengage Learning. All
MA008/MIIZ01 Design and Analysis of Algorithms Lecture Notes 3
MA008 p.1/37 MA008/MIIZ01 Design and Analysis of Algorithms Lecture Notes 3 Dr. Markus Hagenbuchner markus@uow.edu.au. MA008 p.2/37 Exercise 1 (from LN 2) Asymptotic Notation When constants appear in exponents
NUMBERS AND CODES CHAPTER Numbers
CHAPTER 2 NUMBERS AND CODES 2.1 Numbers When a number such as 101 is given, it is impossible to determine its numerical value. Some may say it is five. Others may say it is one hundred and one. Could it
Efficient Application of Countermeasures for Elliptic Curve Cryptography
Efficient Application of Countermeasures for Elliptic Curve Cryptography Vladimir Soukharev, Ph.D. Basil Hess, Ph.D. InfoSec Global Inc. May 19, 2017 Outline Introduction Brief Summary of ECC Arithmetic
Computer Algorithms CISC4080 CIS, Fordham Univ. Outline. Last class. Instructor: X. Zhang Lecture 2
Computer Algorithms CISC4080 CIS, Fordham Univ. Instructor: X. Zhang Lecture 2 Outline Introduction to algorithm analysis: fibonacci seq calculation counting number of computer steps recursive formula
Computer Algorithms CISC4080 CIS, Fordham Univ. Instructor: X. Zhang Lecture 2
Computer Algorithms CISC4080 CIS, Fordham Univ. Instructor: X. Zhang Lecture 2 Outline Introduction to algorithm analysis: fibonacci seq calculation counting number of computer steps recursive formula
Strategies for Stable Merge Sorting
Strategies for Stable Merge Sorting Preliminary version. Comments appreciated. Sam Buss sbuss@ucsd.edu Department of Mathematics University of California, San Diego La Jolla, CA, USA Alexander Knop aknop@ucsd.edu
When to Use Bit-Wise Neutrality
When to Use Bit-Wise Neutrality Tobias Friedrich Department 1: Algorithms and Complexity Max-Planck-Institut für Informatik Saarbrücken, Germany Frank Neumann Department 1: Algorithms and Complexity Max-Planck-Institut
POLY : A new polynomial data structure for Maple 17 that improves parallel speedup.
: A new polynomial data structure for Maple 17 that improves parallel speedup. Centre for Experimental and Constructive Mathematics Simon Fraser University. Maplesoft presentation, August 14th, 2012 This
Copyright 2000, Kevin Wayne 1
Divide-and-Conquer Chapter 5 Divide and Conquer Divide-and-conquer. Break up problem into several parts. Solve each part recursively. Combine solutions to sub-problems into overall solution. Most common
ECE260: Fundamentals of Computer Engineering
Data Representation & 2 s Complement James Moscola Dept. of Engineering & Computer Science York College of Pennsylvania Based on Computer Organization and Design, 5th Edition by Patterson & Hennessy Data
Applying Machine Learning for Gravitational-wave Burst Data Analysis
Applying Machine Learning for Gravitational-wave Burst Data Analysis Junwei Cao LIGO Scientific Collaboration Research Group Research Institute of Information Technology Tsinghua University June 29, 2016
An analogy from Calculus: limits
COMP 250 Fall 2018 35 - big O Nov. 30, 2018 We have seen several algorithms in the course, and we have loosely characterized their runtimes in terms of the size n of the input. We say that the algorithm
Auto-Tuning Complex Array Layouts for GPUs - Supplemental Material
BIN COUNT EGPGV,. This is the author version of the work. It is posted here by permission of Eurographics for your personal use. Not for redistribution. The definitive version is available at http://diglib.eg.org/.
The Viterbi Algorithm EECS 869: Error Control Coding Fall 2009
1 Bacground Material 1.1 Organization of the Trellis The Viterbi Algorithm EECS 869: Error Control Coding Fall 2009 The Viterbi algorithm (VA) processes the (noisy) output sequence from a state machine
! Break up problem into several parts. ! Solve each part recursively. ! Combine solutions to sub-problems into overall solution.
Divide-and-Conquer Chapter 5 Divide and Conquer Divide-and-conquer.! Break up problem into several parts.! Solve each part recursively.! Combine solutions to sub-problems into overall solution. Most common
Comp 11 Lectures. Mike Shah. July 26, Tufts University. Mike Shah (Tufts University) Comp 11 Lectures July 26, / 45
Comp 11 Lectures Mike Shah Tufts University July 26, 2017 Mike Shah (Tufts University) Comp 11 Lectures July 26, 2017 1 / 45 Please do not distribute or host these slides without prior permission. Mike
Quiz 1 Solutions. (a) f 1 (n) = 8 n, f 2 (n) = , f 3 (n) = ( 3) lg n. f 2 (n), f 1 (n), f 3 (n) Solution: (b)
Introduction to Algorithms October 14, 2009 Massachusetts Institute of Technology 6.006 Spring 2009 Professors Srini Devadas and Constantinos (Costis) Daskalakis Quiz 1 Solutions Quiz 1 Solutions Problem
Evolutionary Design I
Evolutionary Design I Jason Noble jasonn@comp.leeds.ac.uk Biosystems group, School of Computing Evolutionary Design I p.1/29 This lecture Harnessing evolution in a computer program How to construct a genetic
Exploring the Potential of Instruction-Level Parallelism of Exposed Datapath Architectures with Buffered Processing Units
Exploring the Potential of Instruction-Level Parallelism of Exposed Datapath Architectures with Buffered Processing Units Anoop Bhagyanath and Klaus Schneider Embedded Systems Chair University of Kaiserslautern
CSE101: Design and Analysis of Algorithms. Ragesh Jaiswal, CSE, UCSD
Greedy s Greedy s Shortest path Claim 2: Let S be a subset of vertices containing s such that we know the shortest path length l(s, u) from s to any vertex in u S. Let e = (u, v) be an edge such that 1
Fast Sorting and Selection. A Lower Bound for Worst Case
Presentation for use with the textbook, Algorithm Design and Applications, by M. T. Goodrich and R. Tamassia, Wiley, 0 Fast Sorting and Selection USGS NEIC. Public domain government image. A Lower Bound
Multimedia Communications. Scalar Quantization
Multimedia Communications Scalar Quantization Scalar Quantization In many lossy compression applications we want to represent source outputs using a small number of code words. Process of representing
Sorting DS 2017/2018
Sorting DS 2017/2018 Content Sorting based on comparisons Bubble sort Insertion sort Selection sort Merge sort Quick sort Counting sort Distribution sort FII, UAIC Lecture 8 DS 2017/2018 2 / 44 The sorting
A Quantum Chemistry Domain-Specific Language for Heterogeneous Clusters
A Quantum Chemistry Domain-Specific Language for Heterogeneous Clusters ANTONINO TUMEO, ORESTE VILLA Collaborators: Karol Kowalski, Sriram Krishnamoorthy, Wenjing Ma, Simone Secchi May 15, 2012 1 Outline!
! Insert. ! Remove largest. ! Copy. ! Create. ! Destroy. ! Test if empty. ! Fraud detection: isolate $$ transactions.
Priority Queues Priority Queues Data. Items that can be compared. Basic operations.! Insert.! Remove largest. defining ops! Copy.! Create.! Destroy.! Test if empty. generic ops Reference: Chapter 6, Algorithms
CMPT 307 : Divide-and-Conqer (Study Guide) Should be read in conjunction with the text June 2, 2015
CMPT 307 : Divide-and-Conqer (Study Guide) Should be read in conjunction with the text June 2, 2015 1 Introduction The divide-and-conquer strategy is a general paradigm for algorithm design. This strategy
Highly-scalable branch and bound for maximum monomial agreement
Highly-scalable branch and bound for maximum monomial agreement Jonathan Eckstein (Rutgers) William Hart Cynthia A. Phillips Sandia National Laboratories Sandia National Laboratories is a multi-program
Design and Analysis of Algorithms
CSE 101, Winter 2018 Design and Analysis of Algorithms Lecture 4: Divide and Conquer (I) Class URL: http://vlsicad.ucsd.edu/courses/cse101-w18/ Divide and Conquer ( DQ ) First paradigm or framework DQ(S)
Randomized Sorting Algorithms Quick sort can be converted to a randomized algorithm by picking the pivot element randomly. In this case we can show th
CSE 3500 Algorithms and Complexity Fall 2016 Lecture 10: September 29, 2016 Quick sort: Average Run Time In the last lecture we started analyzing the expected run time of quick sort. Let X = k 1, k 2,...,
Embedded probabilistic programming
Embedded probabilistic programming Oleg Kiselyov FNMOC oleg@pobox.com Chung-chieh Shan Rutgers University ccshan@cs.rutgers.edu 17 July 2009 Probabilistic inference Model (what) Pr(Reality) Pr(Obs j Reality)
Measurement & Performance
Measurement & Performance Timers Performance measures Time-based metrics Rate-based metrics Benchmarking Amdahl s law Topics 2 Page The Nature of Time real (i.e. wall clock) time = User Time: time spent
Measurement & Performance
Measurement & Performance Topics Timers Performance measures Time-based metrics Rate-based metrics Benchmarking Amdahl s law 2 The Nature of Time real (i.e. wall clock) time = User Time: time spent executing
MA/CSSE 473 Day 05. Factors and Primes Recursive division algorithm
MA/CSSE 473 Day 05 actors and Primes Recursive division algorithm MA/CSSE 473 Day 05 HW 2 due tonight, 3 is due Monday Student Questions Asymptotic Analysis example: summation Review topics I don t plan