Dense Matrix Algorithms (Chapter 8) Alexandre David B2-206

Átírás

1 Dense Matrix Algorithms (Chapter 8) Alexandre David B2-206

2 Dense Matrix Algorithm Dense or full matrices: few known zeros. Other algorithms for sparse matrix. Square matrices for pedagogical purposes only can be generalized. Natural to have data decomposition input/output/intermediate data mapping schemes based on data partitioning Alexandre David, MVP'06 2

3 Today Matrix*Vector Matrix*Matrix Solving systems of linear equations Alexandre David, MVP'06 3

4 Matrix*Vector Recall A x y y i = n k = 1 a ik x k Serial algorithm: n 2 multiplications and addition. W = n Alexandre David, MVP'06 4

5 Matrix*Matrix Recall A B C c ij = n k = 1 a ik b kj Serial algorithm: n 3 multiplications and addition. W = n Alexandre David, MVP'06 5

6 Matrix*Vector Serial Algorithm y i = n k = 1 a ik x k procedure MAT_Vec(A,x,y) for i := 0 to n-1 do y[i] := 0 for k := 0 to n-1 do y[i] := y[i] + A[i,k]*x[k] done done endproc How to parallelize? Alexandre David, MVP'06 6

7 Matrix*Matrix Serial Algorithm c ij = n k = 1 a ik b kj procedure MAT_MULT(A,B,C) for i := 0 to n-1 do for j := 0 to n-1 do C[i,j] := 0 for k := 0 to n-1 do C[i,j] := C[i,j] + A[i,k]*B[k,j] done done done endproc How to parallelize? Alexandre David, MVP'06 7

8 Matrix*Vector Row-wise 1-D Partitioning Initial distribution: Each process has a row of the n*n matrix. Each process has an element of the n*1 vector. Each process is responsible for computing one element of the result Alexandre David, MVP'06 8

9 Matrix*Vector 1-D A x y n processes But every process needs the entire vector all-to-all broadcast Alexandre David, MVP'06 9

10 All-to-All Broadcast x Alexandre David, MVP'06 10

11 Parallel Computation A x y n processes n y i = k = 1 a ik x k in parallel on the n processes Alexandre David, MVP'06 11

12 Example Matrix*Vector (Program 6.4) Allgather (All-to-all broadcast) Partition on rows. Multiply Alexandre David, MVP'06 12

13 Analysis All-to-all broadcast & multiplications in Θ(n). For n processes W=n 2 =nt P. The algorithm is cost-optimal. A parallel system is cost-optimal iff pt P =Θ(W) Alexandre David, MVP'06 13

14 Performance Metrics Efficiency E=S/p. Measure time spent in doing useful work. Previous sum example: E = Θ(1/logn). Cost C=pT P. A.k.a. work or processor-time product. Note: E=T S /C. Cost optimal if E is a constant Alexandre David, MVP'06 14

15 Using Fewer Processes Brent s scheduling principle: It s possible. Using p processes: n/p rows per process. Communication time = t s logp+t w (n/p)(p-1) ~ t s logp+t w n = Θ(n). Computation: n*n/p. pt P = Θ(n 2 ) = W It is cost optimal Alexandre David, MVP'06 15

16 Scalability Recall Efficiency increases with the size of the problem. Efficiency decreases with the number of processors. Scalability measures the ability to increase speedup in function of p Alexandre David, MVP'06 16

17 Isoefficiency Function For scalable systems efficiency can be kept constant if T 0 /W is kept constant. For a target E Keep this constant Isoefficiency function W=KT 0 (W,p) Alexandre David, MVP'06 17

18 Is Our Algorithm Scalable? T 0 =pt P -W T 0 =t s p logp+t w np. We want to determine W=KT 0. Try with both terms separately: W=Kt s p logp. W=Kt w np=n 2 W=(Kt w p) 2. Bound from concurrency: p=o(n) W=Ω(p 2 ). W=Θ(p 2 ) : asymptotic isoefficiency function. Rate to increase the problem size (in function of p) to maintain a fixed efficiency: p=θ(n) Alexandre David, MVP'06 18

19 Matrix*Vector 2-D Matrix n*n partitioned on n*n processes. Vector n*1 distributed in the last (or 1 st column). Similarly we want fewer processes: blocks of (n/ p) 2 elements Alexandre David, MVP'06 19

20 Matrix*Vector 2-D A x y Processes in column i need element of the vector in row i. 1. Distribute on diagonal. 2. One-to-all broadcast on columns. 3. Multiplication. 4. All-to-one reduction (+) Alexandre David, MVP'06 20

21 Example Matrix*Vector (Program 6.8) Partition. Distribute vector. Sum reduce on rows. Row sub-topology. Colum sub-topology. Local multiplication Alexandre David, MVP'06 21

22 Which one is better? 1-D or 2-D? Alexandre David, MVP'06 22

23 Analysis Communications: one-to-one Θ(1) + one-to-all broadcast Θ( logn) + all-to-one reduction Θ( logn). + multiplications Θ(1). T P =Θ(n 2 logn) not cost-optimal. Brent s scheduling principle? Alexandre David, MVP'06 23

24 Using Fewer Processes Blocks of (n/ p) 2 elements. Costs: one to one in t s +t w n/ p + one-to-all broadcast in (t s +t w n/ p) log p + all-to-one reduction in (t s +t w n/ p) log p + computations in (n/ p) 2. Total ~ n 2 /p+t s logp+(t w n/ p) logp. pt P =Θ(n 2 ) cost-optimal Alexandre David, MVP'06 24

25 Scalability Analysis T 0 =pt P -W=t s logp+t w n p logp. As before, isoefficiency analysis: W=Kt s p logp. W=Kt w n p logp=n 2 W=(Kt w p logp) 2. Bound from concurrency: p=o(n 2 ) W=Ω(p). W=Θ(p log 2 p). p=f(n)? p log 2 p=θ(n 2 ) p=θ(n 2 / log 2 n) Alexandre David, MVP'06 25

26 Which One Is Better? 1-D: T P ~ n 2 /p+t s logp+t w n. 2-D: T P ~ n 2 /p+t s logp+(t w n/ p) logp. 1-D: W=Θ(p 2 ). 2-D: W=Θ(p log 2 p). Degree of concurrency Alexandre David, MVP'06 26

27 Block Matrix*Matrix procedure BLOCK_ MAT_MULT(A,B,C) for i := 0 to qn-1 do for j := 0 to n-1 q do C[i,j] := 0 for k := 0 to qn-1 do C[i,j] := C[i,j] + A[i,k]*B[k,j] done done done endproc q*q blocks of (n/q)*(n/q) submatrices. Still n 3 additions & multiplications Alexandre David, MVP'06 27

28 A Simple Parallel Algorithm Map the algorithm to p=q 2 processes. We need all A[i,k] and B[k,j] to compute the C[i,j]. Steps: All-to-all broadcast of A[i,k] on rows. All-to-all broadcast of B[k,j] on columns. Local multiplications Alexandre David, MVP'06 28

29 Analysis Costs: all-to-all p broadcasts of n 2 /p elements = t s log p+t w (n 2 /p)( p-1) *2 + computations = p multiplications of (n/ p)*(n/ p) matrices cost n 3 /p. pt P =Θ(n 3 ) for p=o(n 2 ) cost-optimal. Isoefficiency W=Θ(p 3/2 ). Drawback: memory requirement in n 2 p. Better? Alexandre David, MVP'06 29

30 Cannon s Algorithm Idea: re-schedule computations to avoid contention. Processes on rows i hold a different A[i,k]. Processes on columns j hold a different B[k,j]. Rotate the matrices we need only 2 submatrices per process at any time. memory efficient in O(n 2 ) Alexandre David, MVP'06 30

31 Align A & B Alexandre David, MVP'06 31

32 Alexandre David, MVP'06 32

33 Alexandre David, MVP'06 33

34 Analysis Costs: 2* (A & B) p-single step shifts = 2(t s +t w n 2 /p) p + p multiplications of (n/ p)*(n/ p) submatrices = n 3 /p. Cost-optimal, same isoefficiency function as previously Alexandre David, MVP'06 34

35 The DNS Algorithm 3-D partitioning! Cube with faces corresponding to A, B, C. Internal nodes correspond to multiply operations P i,j,k. Multiplications in time Θ(1). Additions in time Θ(logn). Communication Can use up to n 3 processes better concurrency Alexandre David, MVP'06 35

36 Alexandre David, MVP'06 36

37 k j i Alexandre David, MVP'06 37

38 Communication Steps Move the columns of A & rows of B. One-to-all broadcast along j & i axis. All-to-one reduction (+) along k axis. Communication on groups of n processes, in time Θ(logn). Not cost optimal for n 3 processes Alexandre David, MVP'06 38

39 Brent s Scheduling Principle Theorem If a parallel computation consists of k phases taking time t 1,t 2,,t k using a 1,a 2,,a k processors in phases 1,2,,k then the computation can be done in time O(a/p+t) using p processors where t =sum(t ), i a =sum(a i t ). i Alexandre David, MVP'06 39

40 Look At One Dimension k phases = logn. t i = constant time. a i = n/2,n/4,,1 processors. With q processors we can use time O( logn+n/p). Choose q=o(n/ logn) time O( logn) and this is optimal! 3-D: use p=o(n 3 / log 3 n) p=q Alexandre David, MVP'06 40

41 Systems of Linear Equations A x b a 0,0 x 0 +a 0,1 x 1 + +a 0,n-1 x n-1 =b 0, a n-1,0 x 0 +a n-1,1 x 1 + +a n-1,n-1 x n-1 =b n Alexandre David, MVP'06 41

42 Solving Systems of Linear Equations Step 1: Reduce the original system to U x y Step2: Solve & back-substitute from x n-1 to x Alexandre David, MVP'06 42

43 Technical Issues Non singular matrices. Numerical precision (is the solution numerically stable) permute columns. In particular no division by zero, thanks. Procedure known as Gaussian elimination with partial pivoting Alexandre David, MVP'06 43

44 Gaussian Elimination W=2n 3 / Alexandre David, MVP'06 44

45 Parallel Gaussian Elimination 1-D partitioning: 1 process/row. Process j computes A[*,j]. Cost (+communication) = Θ(n 3 logn) not cost optimal. All processes work on the same iteration. k+1 iteration starts when k th iteration is complete. Improve: pipelined/asynchronous version Alexandre David, MVP'06 45

46 Pipelined Version P k forwards & does not wait. P j s forward & do not wait Alexandre David, MVP'06 46

47 Pipelined Gaussian Elimination No logn for communication (no broadcast) and the rest of the computations are the same. The pipelined version is cost-optimal. Fewer processes: Block 1-D partitioning, loss of efficiency due to idle processes (load balance problem). Cyclic 1-D partitioning better Alexandre David, MVP'06 47

48 Gaussian Elimination 2-D Partitioning Similar as before. Pipelined version cost-optimal. More scalable than 1-D Alexandre David, MVP'06 48

49 Finally Back-Substitution Intrinsically serial algorithm. Pipelined parallel version not cost optimal. Does not matter because of lower order of magnitude Alexandre David, MVP'06 49