Comparing the processing efficiency of the parallel FDTD method on the Intel Phi processor and IBM s Bluegene/Q

Transcription

1 Comparig the processig efficiecy of the parallel FDTD method o the Itel Phi processor ad IBM s Bluegee/Q R. Ilger, D. B. Davidso Computatioal Electromagetics Group Departmet of Electrical ad Electroic Egieerig Uiversity of Stellebosch

2 Outlie CHPC cof, Dec FDTD i the cotext of HPC 2 Parallelisatio of the FDTD 3 Efficiecy of the FDTD o the Itel Phi 4 Efficiecy of the FDTD o the Blue Gee/Q 5 Coclusios ad Recommedatios Compare FDTD o Phi ad BGQ 1/28

3 FDTD i the cotext of HPC Electromagetic modellig, 3 parts to applicatio: 1) Data iput facility such as Autocad or other GUI,Meshig 2) Computatioal model processig egie FDTD 3) Visualisatio egie & save results big data Remove variatio owig to differet algorithms, programmig styles, data models. Performace i terms of efficiecy Compare FDTD o Phi ad BGQ 2/ 28

4 FDTD as a process CHPC cof, Dec 2013 Total ru time of FDTD program 1520s 100% Time cosumed by FDTD differece equatios 1499s 98.62% Time cosumed by calculatig FDTD boudaries 20s 1.32% Start-up time before FDTD iteratios 1s 0.06% Embarrassigly parallel? Not really! Memory boud Compare FDTD o Phi ad BGQ 3/28

5 Maxwell s equatios ad FDTD Yee grid. cell sizig boudary coditios, source ad reflectors. FDTD computatioally sparse. eeds low latecy data supply. subject to data orderig ad coalescece. Compare FDTD o Phi ad BGQ 4/28

6 FDTD equatios CHPC cof, Dec H x(i,j,k) = D ax (i,j,k) H x i,j,k D bx (i,j,k) E z i,j,k + D bx (i,j,k) E y i,j,k E z(i,j 1,k) E y (i,j,k 1) H y (i,j,k) = D ay (i,j,k) H y i,j,k D by (i,j,k) E x i,j,k + D by (i,j,k) E z i,j,k E x(i,j,k 1) E y (i 1,j,k) H z(i,j,k) = D az (i,j,k) H z i,j,k D bz (i,j,k) E y i,j,k + D bz (i,j,k) E x i,j,k E y (i 1,j,k) E x(i,j 1,k) E x(i,j,k) = C ax (i,j,k) E x i,j,k + C bx (i,j,k) H z i,j +1,k H z(i,j,k) C bx (i,j,k) H y i,j,k H y i,j,k J source (x) E y (i,j,k) = C ay (i,j,k) E y i,j,k + C by (i,j,k) H x i,j,k+1 H x(i,j,k) C by (i,j,k) H z i+1,j, H z(i,j,k) J source (y ) Courat coditios: +1 E z(i,j,k) = C az (i,j,k) E z i,j,k C bz (i,j,k) H y i+1,j,k C bz (i,j,k) H x i,j +1,k H y (i,j,k) H x(i,j,k) J source (z) Δt 1 c 1 x y z 2 Compare FDTD o Phi ad BGQ 5/28

7 Outlie CHPC cof, Dec FDTD i the cotext of HPC 2 Parallelisatio of the FDTD 3 Efficiecy of the FDTD o the Itel Phi 4 Efficiecy of the FDTD o the Blue Gee/Q 5 Coclusios ad Recommedatios Compare FDTD o Phi ad BGQ 6/28

8 Programmig methods, task parallel MPI_Recv(Receive colum of magetic frige values from thread B to replace i thread A) MPI_Sed(Sed colum of electric frige values from thread A to replace i thread B) #pragma omp parallel for private(i,j) for (i = 0; i < ie; i++) { for (j = 1; j < je; j++) { ex[i][j] = caex[i][j]*ex[i][j]+cbex[i][j] * ( hz[i][j]- hz[i][j-1]); } } Compare FDTD o Phi ad BGQ 7/28

9 Programmig, vector registers Vector registers iheret i some cotemporary cores: Sigle float 32 Double float 64 SSE 128 AVX or QPX 256 AVX2 512 AVX specificatio up to 1024 bits Compare FDTD o Phi ad BGQ 8/28

10 DTD with AVX or QPX usig compiler itrisics AVX with 8 floats per operatio for (i = 1; i < ie; i++) { for (j = 0; j < je; j+=8) { aoh0=( m256 *)&ex1[i][j]; aoh1=( m256 *)&caex[i][j]; aoh2=( m256 *)&cbex[i][j]; aoh3=( m256 *)&hz1[i][j]; Outlie aoh5=( m256 *)&ey1[i][j]; aoh6=( m256 *)&caey[i][j]; aoh7=( m256 *)&cbey[i][j]; aoh8=( m256 *)&hz1[i-1][j]; hz_shift=&hz1[i][j-1]; amdst=( m256*)hz_shift amtar=_mm256_loadu_ps(amdst); am1 = _mm256_mul_ps(*aoh0,*aoh1); am2 = _mm256_sub_ps(*aoh3,amtar); am3 = _mm256_mul_ps(*aoh2,am2); am4 = _mm256_add_ps(am1,am3); am5 = _mm256_mul_ps(*aoh5,*aoh6); am6 = _mm256_sub_ps(*aoh8,*aoh3); am7 = _mm256_mul_ps(*aoh6,am6); am8 = _mm256_add_ps(am5,am7); _mm256_store_ps(&ey1[i][j],am8); _mm256_store_ps(&ex1[i][j],am4); QPX with 4 doubles per operatio for (i = 1; i < ie; i++) { for (j = 0; j < je; j+=4) { am1=vec_ld(0l,&hz1[i][j]); am2=vec_ld(0l,&dahz[i][j]); am3=vec_ld(0l,&dbhz[i][j]); am4=vec_ld(0l,&ex1[i][j]); am5=vec_ld(0l,&ey1[i][j]); am6=vec_ld(0l,&ex1[i][j+1]); am7=vec_ld(0l,&ey1[i+1][j]); am8=vec_mul(am1,am2); am9=vec_sub(am6,am4); am10=vec_sub(am7,am5); am11=vec_add(am9,am10); am12=vec_mul(am11,am3); am13=vec_add(am12,am8); vec_st(am13, 0, &hz1[i][j]);

11 Throughput scale with SSE or AVX Compare FDTD o Phi ad BGQ 10/28

12 Outlie CHPC cof, Dec FDTD i the cotext of HPC 2 Parallelisatio of the FDTD 3 Efficiecy of the FDTD o the Itel Phi 4 Efficiecy of the FDTD o the Blue Gee/Q 5 Coclusios ad Recommedatios Compare FDTD o Phi ad BGQ 11/28

13 Schematic of Itel Phi architecture 60 cores 4 hardware threads per core 8GB Ru i ative mode GDDR5 memory Fully coheret L2 cache bit VPU per core Symmetric desig, MC receive evely distributed & Compare FDTD o Phi ad BGQ 12/28

14 Efficiecy of FDTD o the Itel Phi Compare FDTD o Phi ad BGQ 13/28

15 DTD with loop combiatio Two loop structures: CHPC cof, Dec 2013 Loop i from 0 to m { Loop j from 0 to { ex[i][j] = caex[i][j] * ex[i][j] + cbex[i][j] * ( hz[i][j] - hz[i][j-1] ); } } Loop i from 0 to m { Loop j from 0 to { ey[i][j] = caey[i][j] * ey[i][j] + cbey[i][j] * ( hz[i-1][j] - hz[i][j] ); } } Sigle loop structure results i more commo data residet i cache: Loop i from 0 to m { Loop j from 0 to { ex[i][j] = caex[i][j] * ex[i][j] + cbex[i][j] * ( hz[i][j] - hz[i][j-1] ); ey[i][j] = caey[i][j] * ey[i][j] + cbey[i][j] * ( hz[i-1][j] - hz[i][j] ); } }

16 Code re-arragemet, NAG Compare FDTD o Phi ad BGQ 15/28

17 Outlie CHPC cof, Dec FDTD i the cotext of HPC 2 Parallelisatio of the FDTD 3 Efficiecy of the FDTD o the Itel Phi 4 Efficiecy of the FDTD o the Blue Gee/Q 5 Coclusios ad Recommedatios Compare FDTD o Phi ad BGQ 16/28

18 Blue Gee/Q A2 processor compoets CHPC cof, Dec PowerPC-A2 processor cores ruig at 1.6GHz. 16 of them are User cores, 1 is for system maagemet (hadles iterrupts message passig, etc) ad the 18th is a spare, for icreased fault tolerace. 4 chaels to 16GB DDR3 memory 55w per ode Compare FDTD o Phi ad BGQ 17/28

19 Blue Gee/Q CHPC cof, Dec 2013 Compare FDTD o Phi ad BGQ 18/28

20 FDTD throughput o Itel Phi, A2 processors Compare FDTD o Phi ad BGQ 19/28

21 Multiple threads per ode CHPC cof, Dec 2013 Compare FDTD o Phi ad BGQ 20/ 28

22 FDTD o previous versios of the Bluegee Compare FDTD o Phi ad BGQ 21/28

23 FDTD o GPU cluster ad Blue Gee/P FDTD computatioal performace o small GPU cluster ad Blue Gee/P is of similar order. Compare FDTD o Phi ad BGQ 22/28

24 Evolutio of FDTD processig o the Blue Gee Per rack BG/P 4096 cores BG/Q cores Compare FDTD o Phi ad BGQ 23/28

25 QPX o Blue Gee/Q CHPC cof, Dec 2013 Compare FDTD o Phi ad BGQ 24/28

26 Blue Gee/Q immese dimesios, 24 racks Compare FDTD o Phi ad BGQ 25/ 28

27 Outlie CHPC cof, Dec FDTD i the cotext of HPC 2 Parallelisatio of the FDTD 3 Efficiecy of the FDTD o the Itel Phi 4 Efficiecy of the FDTD o the Blue Gee/Q 5 Coclusios ad Recommedatios Compare FDTD o Phi ad BGQ 26/28

28 Icomplete developmet cycle To fully uderstad the parallel FDTD implemetatios: Imperative stage i HPC developmet cycle is the tuig/performace aalysis phase so as to uderstad ad optimise the icreasigly sophisticated hardware. Low level uderstadig resultig from aalysis will lead to higher level gais. Compare FDTD o Phi ad BGQ 27/28

29 Coclusios & Recommedatios Costatly chagig hardware evolutio makes compariso a time widowed evet. Best Performace from hybrid architecture ad hybrid parallelisatio methods Processor & Memory, latecy, computatioal sparsity, memory eeds to be related to processig core for FDTD efficiecy? Lessos leared from GPU data supply. Eve faster with curret hardware, ot always obvious. Parallel performace tools vital to uderstad process o more sophisticated hardware. Ackowledgmets NRF,CHPC,SKA, Itel User Forums, Juelich Computig cetre, NAG(UK), Chris Armstrog FDTD optimisatio. Compare FDTD o Phi ad BGQ 28/ 28