Automatic Differentiation Algorithms and Data Structures. Chen Fang PhD Candidate Joined: January 2013 FuRSST Inaugural Annual Meeting May 8 th, 2014

Size: px

Start display at page:

Download "Automatic Differentiation Algorithms and Data Structures. Chen Fang PhD Candidate Joined: January 2013 FuRSST Inaugural Annual Meeting May 8 th, 2014"

Stephen Morrison
5 years ago
Views:

1 Automatic Differentiation Algorithms and Data Structures Chen Fang PhD Candidate Joined: January 2013 FuRSST Inaugural Annual Meeting May 8 th,

2 About 2 3 of the code for physics in simulators computes derivatives 2

3 Automatic Differentiation (AD) 1. AD data type Value Gradient x 1 x 1, x 1 x 1, x 1 x 2 2. Operator overloading x y = x y + y x x 1 x 2 = x 1 x 2, x 1 x 2 x 1, x 2 x 2 + x 2 x 1 x 1, x 1 x 2 3

4 Example of Automatic Differentiation f x 1, x 2 = cos x 1 + x 1 exp x 2 Procedures: x 1 = x 1, 1, 0 x 2 = x 2, 0, 1 cos x 1 = cos x 1, sin x 1, 0 exp x 2 = exp x 2, 0, exp x 2 x 1 exp x 2 = x 1 exp x 2, exp x 2, x 1 exp x 2 f x 1, x 2 = cos x 1 + x 1 exp x 2, sin x 1 + exp x 2, x 1 exp x 2 f x 1 f x 2 4

5 Popular AD Packages There are many AD packages available: OpenAD ADOL-C Various implementation methods Source Transformation Operator Overloading Various implementation language Fortran C/C++ MATLAB Python Applications that use AD: Ocean circulation Optimization of dynamic systems governed by PDEs Nuclear reactor applications 5

6 Challenges for AD in Reservoir Simulation 1. Variable sparsity pattern Diagonal Block diagonal Stenciling Evaluation 6

7 Challenges for AD in Reservoir Simulation 2. Variable switching 7

8 Automatically Differentiable Expression Templates Library (ADETL) Developed by Rami Younis at Stanford University Initial prototype to prove viability of AD for reservoir simulation ADETL solves some reservoir simulation chanllenges: Block sparse gradient data structure Two algorithms to compute derivative operations Variable switching and adaptive implicit problems Builds runtime expression for derivatives 8

9 Sparse Algorithms in ADETL Algorithm 1: axpy Phase 1 Phase 2 9

10 Sparse Algorithms in ADETL Algorithm 2: running pointers 10

11 Variable Switching in ADETL Independent variable set: Components in liquid phase Components in gas phase [ Po, Pg, Pw, So, Sg, x1, x2, x3, x4, y1, y2, y3, y4] Gas phase disappears: Auto (de)activation [ Po, Pg, Pw, So, Sg, x1, x2, x3, x4, y1, y2, y3, y4] 11

12 Application of ADETL ADETL has been successfully applied in the development of reservoir simulators: AD-GPRS (Stanford University) Unconventional shale gas reservoir simulator (FuRSST) We are improving ADETL towards ADETL

13 So what s next? 1. Can we deal with various degrees of sparsity patterns? 2. Can we avoid the cost of runtime sparsity detection? 13

14 Non-zero Call Stack 14

15 Univariate Dense Multivariate Sparse Multivariate ADETL

16 Stage 1 Univariate terms Test case: product of sequence N F x = f i x F x = i=1 N j=1 df j x dx i j f i x Case # # of arguments (N)

17 N Univariate Test 1 F x = f i x 1. Hand differentiation F x = i=1 N j=1 df j x dx i j f i x N Value & Derivative Expressions 1 F x = f 1 x F x = f 1 x 2 F x = f 1 x f 2 x F x = f 1 x f 2 x + f 1 x f 2 x 3 F x = f 1 x f 2 x f 3 x F x = f 1 x f 2 x f 3 x + f 1 x f 2 x f 3 x + f 1 x f 2 x f 3 x 17

18 Univariate Test 2 2. ADunivariate (new datatype) N ADunivariate Expression 1 R = a 2 R = a * b 3 R = a * b * c 18

19 Univariate Test 3 3. ADETL (block sparse gradient) N ADETL Expression 1 R = a 2 R = a * b 3 R = a * b * c 19

20 N Univariate Test Result F x = f i x F x = i=1 N j=1 df j x dx i j f i x 20

21 Stage 2 Dense Multivariate Test case: summation F x 1, x 2 x n = f i x 1, x 2 x n 4 i=1 Case # # of independent variables F x 1, x 2 x n x 1 = F x 1, x 2 x n x n = 4 fi x 1, x 2 x n i=1 i=1 x 1 4 fi x 1, x 2 x n x n 21

22 Dense Multivariate Test 1 1. Manual implementation Case 1 Case 2 Case

23 Dense Multivariate Test 2 2. Expression Templates with dense gradient Vector1 Vector2 Vector3 Vector4 Vector1 + Vector2 Vector1 + Vector2 + Vector3 Vector1 + Vector2 + Vector3+Vector4 23

24 Dense Multivariate Test 3 3. ADETL (block sparse gradient) Case 1 Case 2 Case

25 Time(microseconds) Problem 2 - Dense Multivariate Manual 2. Expression Template 3. ADETL F x 1, x 2 x n = N i=1 f i x 1, x 2 x n 4.1X X 7.5X 1X X 1X Number of elements in dense vector Manual Expression Template ADETL 25

26 Avoiding sparse operations sparse dense d1 d2 d x x 1 x x 2 x x x 1 x x x 3 x x x 1 x x x 4 x x x 1 x x x x = 5 x x x 1 x x x 6 x x x 1 x x x 7 x x x 1 x x x 8 x x x 1 x x x 9 x x 1 x x Sparse Jacobian Seed matrix Compressed Jacobian 26

27 Compressed AD 1. Variable activation (Dense Jacobian) 2. Evaluation U = U S x x x x J = x x x x x x x x x x x x recover J = x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x 27

28 Challenge with Compressed AD 1. Sparsity pattern is unknown 2. Variable switching d1 d2 d x x 1 x x 2 x x x 1 x x x 3 x x x 1 x x x 4 x x x 1 x x x x = 5 x x x 1 x x x 6 x x x 1 x x x 7 x x x 1 x x x 8 x x x 1 x x x 9 x x 1 x x Sparse Jacobian Seed matrix Compressed Jacobian 28

29 ADETL 2.0 Full range of data types and efficient algorithms Ability to convert from one type to the other Compressed AD Automatic variable set Seed matrix will be incorporated Benchmark kernel to test AD packages for reservoir simulation calculations 29

30 Automatic Differentiation Algorithms and Data Structures Chen Fang PhD Candidate Joined: January 2013 FuRSST Inaugural Annual Meeting May 8 th,

Generating pseudo- random numbers

Generating pseudo- random numbers What are pseudo-random numbers? Numbers exhibiting statistical randomness while being generated by a deterministic process. Easier to generate than true random numbers?