Modeling Parallel and Distributed Systems with Finite Workloads

Transcription

1 Modeling Parallel and Distributed Systems with Finite Workloads Ahmed M. Mohamed, Lester Lipsky and Reda Ammar {ahmed, lester, Dept. of Computer Science and Engineering University of Connecticut, Storrs, CT 6269, USA Abstract In studying or designing parallel and distributed systems one should have available a robust analytical model that includes the major parameters that determine the system performance. Jackson networks have been very successful in modeling parallel and distributed systems. However, they have their limitations. In particular, the product-form solution of Jackson networks assumes steady state and exponential service centers or certain specialized queueing disciplines. In this paper, we use a transient model studying distributed systems with finite workload (no new arrivals). Using some nonexponential distributions we show to what extent the exponential distribution can be used to approximate other distributions. When the number of tasks to be executed is large enough, the model approaches the product-form solution in those cases where the Jackson networks can be applied. We also study some cases where Jackson networks can t be applied (the nonexponential servers have queueing). The model can be used for reliability analysis of systems that allow failures without repair (fail-stop). Key words: Analytical Modeling, Performance Prediction, Queueing Models, Jackson Networks Reliability Analysis and Transient Analysis.. Introduction It is often assumed that tasks in parallel systems are independent and run independently using separate hardware (e.g. Fork/Join type applications). In this case the problem is reduced to order statistics [22,23,24,25]. However, in many applications tasks must interact through shared resources (e.g. shared data, communication channels). Hence, order statistics analysis is not adequate and one must apply more general models. We used the product-form solution for Jackson networks in [,2] to model clusters of workstations. This model is satisfactory if the number of tasks is much greater than the degree of multi-tasking and the shared resources do not have high variance for task services. However, if this is not the case, the transient model discussed here should be employed. The steady state solution of Jackson networks assumes exponential service times. It has been shown by Leland and Ott [5] that the distribution of the CPU times at BELLCORE are power tailed (PT). Also, Lipsky and Crovela[6,9], Hatem[3] and others, found that file sizes stored on disks are PT. If these observations are true generally, then performance modeling based on exponential distributions are not adequate. In a previous work, we developed a transient model for Jackson networks for exponential distributions [3]. In [5] we presented a transient model in which the dedicated servers (i.e., local CPU s and Disks) have nonexponential service times. In this case, the steady state value of the model is the same as the steady state value of the product form solution of Jackson networks. In this paper, we present the case when shared servers (i.e., shared disks and the communication network) have nonexponential service times (Jackson networks can t be applied to such systems). The model we present includes the several performance parameters that affect the performance of parallel and distributed systems. More parameters always can be added to the basic model (e.g., scheduling overhead, multitasking,.etc) if needed. Also, the model can be used in reliability analysis for systems that allow failures without repair (fail-stop). In our analysis, we use the Linear Algebraic Queueing Theory (LAQT) approach. All the necessary background needed can be found in [7]. The rest of the paper is organized as follows: In Section 2, we give a brief background. In Section 3, we introduce our transient model. In Section 4, we show how to use the model to analyze the performance of different configurations of parallel and distributed systems. In Section 5, we show how to use the model in reliability analysis. In Section 6, we show some of our results.

2 2. Background Since the early 97s, networks of queues have been studied and applied to numerous areas in computer science and engineering with a high degree of success. General exponential queueing network models were first solved by Jackson [4] and by Gordon and Newell [2] who showed that certain classes of steady state queueing networks with any number of service centers could be solved using a product-form solution. A substantial contribution was made by Buzen [7,8] who showed that the ominouslooking formulas were computationally manageable. Basket et al [2] summarized under what conditions the product form solution could be used (e.g., processor sharing, multiple classes, etc). Thereafter, the analysis of queueing networks began to be considered as a research field of its own. We used the product form solution of Jackson networks to analyze the performance of clusters of workstations []. The model then was used to develop efficient data allocation algorithms [2]. Then we [3] developed a transient model for Jackson networks and showed to what extent the steady state model can be used. However, there is a computational problem (state space explosion) in attempting to use this model for large systems. An approximation to the transient model was presented in [4]. 3. The Transient model First we introduce some definitions that are important for our analysis. A complete description can be found in [6]. - Objects used if only one customer in the system S is a system consisting of a set of service centers. Ξ is the set of all internal states of S. p is the entrance row vector where p i is the probability that upon entering S, a customer will go to server i. q is the exit vector where q i is the probability of leaving the system when service completed at server i. M is the completion rate matrix whose diagonal elements are the completion rates of the individual servers. The rest of the elements are zeros. P is the transition matrix where P ij is the probability that a customer will go from server i to server j when service is completed at i. B is the service rate matrix, B = M (I P). τ is a column vector where τ i is the mean time until a customer leaves S, given that he started at server i. V is the service time matrix where V ij is the mean time a customer spends at j from the time it first visits i until it leaves the system. V = B - ε is a column vector all of whose components are ones. - Objects used if k > tasks in the system Ξ k is the set of all internal states of S when there are k active customers there. There are D(k) such states. M k is the completion rate matrix where [M k ] ii the service rate of leaving state i. The rest of the elements are zeros. P k is the transition matrix where [P k ] ij i,j Ξ k, is the probability that the system will go from state i to state j when service is completed while the system is in state i. Q k is the exit matrix where [Q k ] ij is the probability of a customer leaving S when the system was in state i Ξ k, leaves the system in state j Ξ k-. R k entrance matrix where [R k ] ij is the probability that a customer upon entering S finding it in state i Ξ k- will go to server that puts the system in state j Ξ k. τ k is a column vector of dimension D(k) where [τ k ] i is the mean time until a customer leaves S, given that the system started in state i Ξ k. Suppose we have a computer system made up of K processors or workstations and we wish to compute a job made of N tasks where N > K. The first K tasks are assigned to the system and the rest are queued up waiting for service. Once a task finishes it is immediately replaced by another task from the execution queue. Assume that the system initially opens up and K tasks flow in. The first task enters putting the system in state p Ξ.The second task enters and takes the system from that state to state pr 2 Ξ 2 and so on. The state of the system after the K th task enters is: p K = pr 2 R 3 R K It can be shown that the mean time until the first task finishes and leaves the system is [4], τ K = M k - ε k + P k τ k τ K = (I k P k) - M k - ε k = V k ε k The mean time is given by, t k = p k V k ε k How long does it take for the next tasks to finish?. There are two possibilities, either N = K or N > K. 3. Case. (N = K) First, define matrix Y k where [Y k ] ij is the probability that S will be in state j Ξ k- immediately after a departure, given that the system was in state i Ξ k and no other customers have entered. Y k can be obtained from the following argument. When an event occurs in S, either someone leaves, [Q k ], or the internal state of the system changes, [P k ], and eventually somebody leaves, [Y k ]. Y k = Q k + P k Y k Y k = (I k P k ) - Q k = (I k P k ) - M k - M k Q k =V k M k Q k k K

3 We then consider how long it takes for the second task to finish after the first one left. p K Y K (V K- ε K- ) = p K Y K (τ K- ) where, [p k ] i is the probability that the system was in state i when epoch began (epoch is the time between two successive departure). This means, after the first task leaves, the system is in state p k Y k with k, tasks. The second task takes (τ k- ) to leave next. The time between the second and third departures is, p K Y K (Y K- ) (V K-2 ε K-2 ) = p K Y K (Y K- ) (τ K-2 ) In general the mean time to finish all tasks is given by, E(T) = p K[τ K + Y K τ K Y K Y K - Y τ ] 3.2 Case 2. (N > K) The first K tasks are assigned to the system and the rest are queued up waiting for service. When a task leaves the system, another one immediately takes its place, putting the system in state Y K R K = V K M K Q K R K where, Y K R K ε K = ε K The mean time until the second task finishes is: p K Y K R K (V K ε K ) = p K Y K R K (τ K) Now, another tasks enters the system, putting the system in state, Y K R K Y K R K = (V K M K Q K R K ) 2 The mean time until the third task finishes is given by, p K (Y K R K ) 2 (V K ε K) = p K (Y K R K ) 2 (τ K) Eventually we will reach case one again, where there are only K tasks remaining, but with initial state p K (Y K R K ) N-K. In general, the mean time to finish executing all tasks is, N K E(T) = p K [ (Y K R K ) i ] (τ K) i= + p K (Y K R K ) N-K Y K [τ K + Y K τ K- + Y K Y K- τ K Y K Y K- Y τ ] Each term of the above equation helps describe the transient behavior. When i is small, the term (Y K R K ) i is different for different values of i which gives different departure times for different epochs. Once i becomes large the term approaches ε K p ss (p SS Y K R K = p SS )gives the steady state solution for Jackson networks where applicable. Once the number of tasks remaining becomes less than the number of processors, we have different values of k (k < K) for different system sizes, which leads to the other transient region (draining region). 4. Modeling approach The success of a performance model is dependent on how accurately the model matches the system and more importantly what insights does it provide for performance analysis. The major parameters we are modeling include communication contention, geometry configurations, time needed to access different resources and data distribution. Such a model is useful to get a basic understanding of how the system performs. More details can always be added to the basic model (e.g., scheduler overheads, multitasking, task dependencies, etc) but with increase in the state space. 4. Application model The target parallel application can be considered to be a set of independent, identically distributed (iid) tasks, {t, t 2... t N }, The tasks are queued up (if N > K), and the first K tasks are assigned to the system. When a task is finished, it is immediately replaced by another task in the queue. The active tasks can communicate with each other by exchanging data through each other's disks. The tasks run in parallel, but they must queue up for service when they wish to access the same device. Each task consists of a finite number of I/O and CPU instructions. Therefore, the execution of the task consists of phases of computation, then I/O then computation, etc, until finished. We assume that during an I/O phase the task cannot start a new computational phase (no CPU-I/O overlap). Assume that T is the random variable that represents the running time of a task if it is alone in the system. Then, the mean execution time E(T) for a task can be divided into three components: T, T 2 and T 3, where, E(T ) is the expected time needed to execute non-i/o instructions locally (local CPU time). E(T 2 ) is the expected time needed to execute I/O instructions locally (local disk time) E(T 3 ) is the expected time needed to execute I/O instructions else where(remote disk time) As in [,2], we use the following parameters to represent the above components. X = E(T ) + E(T 2 ), C * X = E(T ), ( C) * X = E(T 2 ), Y = E(T 3 ). C is the fraction of local time spent at the local CPU. So based on the above parameters we can write T as: E(T) = C * X + ( C ) * X + Y. All of this is for the case when there is only one task in the system (therefore no contention). The performance model uses these parameters to calculate the effect of contention when more than one task is running. 4.2 System model It is assumed that the computer system consists of a network of workstations under the control of a single scheduling mechanism with a centralized data storage disk.

4 All workstations contact this central disk when they request global data. 4.3 Task activity Each task spends some time in its local CPU doing computation then, assuming it is not finished, with probability (-q) goes to some disk for I/O. Tasks may need access a remote disk(s) through a communication channel with probability (-q)p 2 and their local disk with probability (-q)p After finishing its I/O the task returns to its local workstation. The task finishes its execution and leave the system with probability q. Thus, the number of computational cycles is geometrically distributed with mean of (/q) cycles. 4.4 Modeling central storage systems The system consists of K workstations and a central server. A workstation can be modeled as one server or more based on the assumptions made. Here, we assume that each workstation consists of two servers (CPU and a disk). The K workstations are connected by a shared network channel (one server). Therefore the number of servers needed to model K workstations is 2*K + 2. The Kronecker-product formulation of the system would require the following number of states: D(K) = (2*K + 2) K Since we assumed that the tasks are iid, we can use a reduced product representation. Even after using the reduced product space the number of states can still be huge. We still need to reduce the number of states. Since the tasks never compete for CPU s or local I/O, an equivalent model to the above description is considered. We can consider the first server in the system to be a load dependent server representing all of the CPUs. Then another load dependent server can represent all of the local disks. Finally two load independent servers represent the communication channel and the central disk. Therefore, the number of servers is four, which reduces the state space to: D RP ( K ) K + = K 3 Assume we are modeling a system with five workstations and all servers have exponential service times. The following are the basic matrices (when only one task is running in the cluster). p = [,,, ], as we assume that a task starts its execution from the CPU. q = [q,,, ], as we assume that a task always leaves from the CPU. CPU Disk Comm. R.Disk P p( q) p ( q ) 2 = M µ cpu = µ d µ com where, p + p 2 = From V = M (I P) - it follows that: pv = [t cpu /q, t d * p (-q)/q,t com *p 2 (-q)/q, t rd *p 2 (-q)/q] where, t cpu = /µ cpu, t d = /µ d, t com = /µ com, t rd = /µ rd µ rd pv is the time components vector that represents the total time spent by a task in each stage of the system (for the case k = ). For example the task will spend (t cpu /q) units of time in its CPU. The M matrix is known from the architecture of the cluster but the reformation of the P matrix to the parameters given in the application model described earlier is still to be specified. The application model specifies the total time spent by a task in the cluster (with no contention) as [CX, (- C)X, BY, Y]. By using these time components we obtain the parameters in the P matrix as follows, q = t cpu / CX p = q * ( C) X / t d * ( q) p 2 = q * Y / t rd * ( q) If we wish to use non-exponential distribution as the service distribution for the CPU, we only need to change the matrices as shown in the next subsection Erlangian distributions The Erlangian-m distribution describes the time it takes for a customer to be served by m identical exponential servers in series. Therefore, the Erlangian- is the exponential distribution and its pdf is f(t) = µ exp(-µ t) where µ is the service rate, t cpui = /µ cpui The pdf of the Erlangian-2 distribution is f(t) = µ (µ t) exp(-µ t) If we wish to use the Erlangian-m as the service distribution of the CPU, we replace the CPU server by m identical servers. For example if we use the Erlangian-2 instead of the exponential distribution in the previous example, then our basic matrices are: CPU CPU2 Disk Comm R.disk p( p2( P =,

5 M µ cpu = µ cpu 2 µ D µ com µ RD p = [,,,, ], q = [, q,,, ] The value of t cpu is still the same as before and t cpu, t cpu2 are t cpu = t cpu2 = t cpu / 2 We can use the same approach to the local disk, remote disk or communication channel Hyperexponential Distributions The other class is the family of Hyperexponential distributions whose pdf is of the form f Hm (t) :=p µ exp(-µ t) + p 2 µ 2 exp(-µ 2 t) m p m µ m exp(-µ m t) = pi[ µ i exp( µ it )] i= where, p i and µ i are real and p +.p m =. Considering the same example but with the Hyperexponential-2(H 2 ) as the service distribution of the CPU. The basic matrices are: p = [p H2, -p H2,,, ] q = [q, q,,, ] P CPU CPU2 Disk Comm R.disk = ph2 ph2 µ cpu = M µ cpu -p -p 2 H2 H2 µ D p( p( µ com p2( p2( µ RD We need to calculate three parameters µ, µ 2 and p H2. Once E(T H2 ) = t cpu and the variance (σ 2 ) are chosen, one more parameter is still needed. One possibility is to fix the third parameter based on the physical system (i.e. specific value for p H2 ) or use the third moment. Another possibility is to fit the value of the pdf at. That is, let f H2 () = p H2 µ + ( - p H2 ) µ 2 Once the basic matrices are constructed, we can construct the other matrices that are to be used to represent a cluster of K servers (P k, M k, Q k, R k ) and K active tasks. See [7] for more details. Ξ k := { i = (α, α 2, α 3, α k ) α j k, K and α j = k } k K j= Ξ k is the set of all internal states of the system when there are k active customers there. Each m-tuple represents a state where α j is the number of tasks at server j. [M k ] ii = α µ + α 2 µ 2 + α 3 µ 3 + α k µ k = α j µ j j= [P k ] ii :=, unless [(i) (j)] has exactly two nonzero elements, one with the value and the other is. This means that only one task can move at a time. Let α a be the number of customers in the server where the task left and α b is the number of customers in the server where the task went. Then, [P k ] ij = [P] ab α aµ a [M ] k ii [R k ] ij =, unless [(j) (i)] has exactly one nonzero element and that element would have the value. Let a be the component that is not zero. [R k ] ij = p a [Q k ] ij =, unless [(i) (j)] has exactly one nonzero element and that element would have the value. Let a be the component that is not zero. [Q k ] ij = α aµ a q [ M ] a k ii 5. Reliability analysis The model presented in this paper can be used in reliability analysis of systems with failure without repair as follows: - First, Identify the system and application parameters. - Apply the model and get the initial expected running time for the target application. - Assume a failure distribution. - After the first epoch, calculate the failure probability of each server in the system. - Modify the entrance vector then recalculate the expected running time for the remaining tasks. - Repeat the previous two steps after each epoch. - Add more resources if the expected running time does not satisfy the application demand. A reliability model for simple systems with failures and repair has introduced in []. We are presently working on combining the two models. K

6 6. Results In this section, we show some of the results that can be obtained from the model. First, we show the different performance regions (transient, steady state and draining). Then, we demonstrate the effect of the distribution on the steady state value of the system. Finally, we study the effect of the service distribution and the performance region on some of the performance metrics (speedup and execution time prediction). We calculated two sets of examples one for K=5 and one for K=8. The average execution time per task is 2 units of time (E(T)=2). The application is running on a 5-node and 8-node cluster. To show the effect of the performance region on the accuracy of performance measures, we set N = 3 and. The results we show are for a central cluster. 6. Shared servers with non-exponential service times We assume that the shared server (remote disk) is the non-exponential server while the dedicated servers are exponential servers. In this case Jackson networks models can t be applied. 6.. Performance behavior In Figures,2, we show how the performance behavior of an application changes if we assume different service distributions. As mentioned earlier, there are three different regions in the performance characteristic of any system. At the beginning, the system is idle and a set of tasks start to execute. The system begins its transient region and stays there for some time. Exp H2,C2= H2,C2=5 execution queue. Once the number of tasks remaining in the execution queue becomes less than the degree of multitasking, the draining region starts. The draining continues until the system finishes all of the tasks. In all the cases that we show in this paper, the region where the interdeparture time is appears to be constant is the steady state region. Time 4 Exp H2,C2= H2,C2= Task Order Fig task application running on 8-node cluster. In Figures and 2, we compare exponential (C 2 = ), with Hyperexponential-2 with (C 2 = ) and (C 2 = 5). C 2 is the coefficient of variation. In all the cases we show, the time axis (the interdeparture time) is in log scale, the interdeparture time is the time between two successive task completions. Note that the steady state values in Figures and 2 did change that much even though we increased the K from 5 to 8 workstations. This happens because the contention at the central disk Performance prediction One of the main objectives of having a performance model is to be able to predict the running time of the target application. In most of the cases, the exponential distribution is assumed. As we mentioned earlier, it has been shown that the exponential assumption is not recommended for accurate estimations. Since parallel applications tend to have a wide range of execution times (large C 2 ). Time 5 K = 5 4 Task Order Fig. 3-task application running on 5-node cluster. After a number of tasks are processed the system approaches the steady state region. The system continues to execute tasks as they are available in the E% 3 2 N=3 N= C 2 Fig. 3. Prediction Error

7 Here, we give some examples to show the effect of assuming the exponential assumption where it is not valid. In Figures 3 and 4, we calculate the percentage error of the application, if the exponential distribution is assumed. The percentage error is calculated as : E = E E( Tact) ( Tact) app E( T exp app ) app *, where, E(T act ) app is the total execution time of the parallel application with the right distribution. E(T exp ) app is the total execution time of the parallel application if the exponential distribution is assumed. E% K = 8 N=3 N= C 2 Fig.4. Prediction Error Our results indicate that the exponential distribution fails to approximate distributions with large C 2. In Fig 3,4 we see that the error exceeds 2 % if C 2 =. The percentage error always increases with increasing C Speedup It always important to know the value of the speedup that your computing resources can provide. We found that one of the common problems is that the actual speedup is less than the expected speedup from the available resources. Our results show that there are three reasons. Obviously the first reason is contention. The second reason is the operating region (i.e., steady state or transient). In Figures 5 and 6, we use the same application but with different number of tasks (3, ) and calculate the speed up for different C 2. In case of 3 tasks, the transient region is dominating. Then we increased the number of tasks to in order for the steady state region to dominate. It is clear that, if the system is working in the transient region, the speed up is much less than if the system is working in the steady state region. This behavior can be explained if we return to Figures to 2 and notice the effect of the transient and draining regions explained earlier. Speedup Speedup K = C 2 Fig 5. Speedup K = 8 N=3 N= C 2 Fig. 6. Speedup N=3 N= The third reason for not getting the expected speed up is using the incorrect service distribution. It is clear that, the exponential distribution overestimates the speedup that we get if we use Hyperexponential distributions. This explains why we do not get the expected speedup from the system. We believe that both system operating region and the service distribution are equally important in analyzing the performance of parallel and distributed systems. 6.2 Dedicated Servers with Non-Exponential Service Times Here we assume that the dedicated servers (e.g. CPU) are the non-exponential server while the shared servers are exponential servers, reported in [5]. In this case Jackson networks models can be applied and our model is considered as a transient model for Jackson networks. Here, the steady state value of the system is the same for all distributions and equals the steady state value of the PF (steady state) solution. For results and analysis, see [5].

8 Conclusion In this paper, we presented an analytical performance model that can be used as a transient model for Jackson networks. Also, it can be used in cases where Jackson networks can t be applied. The model can be applied to systems with population size constraints. The model can use any service distribution and is not limited to exponential distributions. We also used the model to analyze the performance of parallel and distributed systems. Our results indicate that system operating region (steady state or transient) and service distribution can have a significant impact on the performance behavior of parallel and distributed systems. We showed how performance metrics like speedup and execution time prediction significantly depend on system operating region and service distribution. We then showed to what extent the exponential distribution can be used to approximate other service distributions. Finally, we utilized the model to be used in reliability analysis. REFERENCES []Ahmed Mohamed, Lester Lipsky and Reda Ammar, Performance Model for a Cluster of Workstations. The Fourth International Conference on Communications in Computing (CIC 23). Las Vegas, NV, Jun. 23. [2] Ahmed Mohamed, Lester Lipsky and Reda Ammar, Efficient Data Allocation for a Cluster of Workstations. ISCA 6th International Conference on Parallel and Distributed Computing Systems, Reno, NV, Aug 3. [3] Ahmed Mohamed, Lester Lipsky and Reda Ammar, Transient Model for Jackson Networks and its Application in Cluster Computing, Submitted to Journal of cluster Computing, Nov. 23. [4] Ahmed Mohamed, Lester Lipsky and Reda Ammar, Transient Model for Jackson Networks and its Approximation, 7 th International Conference on Principles of Distributed Systems (OPODIS3), Dec. 23. [5] Ahmed Mohamed, Lester Lipsky and Reda Ammar, Jackson Networks as Approximations to Parallel and Distributed Systems, Submitted to Sigmatrix, Nov. 23. [6] R. Buyya, High Performance Cluster Computing: Architecture and Systems, Prentice Hall PTR, NJ, 999. [7]J.Buzen, Queueing Network Models of Multiprogramming, Ph.D. Thesis, Harvard University, 97. [8] J. Buzen, Computational Algorithms for Closed Queueing, Comm. ACM, Vol 6, No. 9, Sep 973 [9] R. Chen, A Hybrid Solution of Fork/Join Synchronization in Parallel Queues, IEEE Transactions on Parallel and Distributed Systems, vol. 2, no. 8, pp , Aug. 2. [] P. Fiorini and C. Campbell and Lester Lipsky, An Analytic Approach to Assess the Performability of Parallel and Distributed Systems 5th International Conference on Parallel and Distributed Computing Systems, Louisville, Ky, Sep. 9-2, 22. [] I. Foster and Kesselman, The Grid: Blueprint for a New Computing Infrastructure, Morgan-Kaufmann, 998 [2] W. Gordon, G. Newell, Closed Queueing Systems, JORSA, Vol. 5, pp , 967. [3] J. Hatem and Lester Lipsky, Buffer Problems on Telecommunications Networks, 5 th International Conference on Telecommunication Systems, Nashville, TN, 997. [4] J. Jackson, Jopshop-Like Queueing Systems, J. TIMS, Vol., pp. 3-42, 963. [5] Leland, T. Ott, Analysis of CPU times on 6 VAX /78 at BELLCORE, Proceeding of the International Conference on Measurements and Modeling, April 986. [6] Lester Lipsyk, The Importance of Power-Tail Distributions for Modeling Queueing Systems,, Operations Research, Vol 47, No. 2, (March-April 999). [7] Lester Lipsky, "QUEUEING THEORY: A Linear Algebraic Approach", McMillan, New York, 992. [8] Lester Lipsky and J. Church, Applications of Queueing Network Models," Comp. Surveys, 9, pp , Sep [9] Lester Lipsky and Mark E. Crovela, "Long-Lasting Transient Conditions in Simulations with Heavy-Tailed Workloads," in Proceedings of the 997 Winter Simulation Conference, December 997. [2] F. Basket, R. Muntz, K. Chandy, Open, closed and Mixed Networks of Queues with Different Classes of Customers, JACM, Vol. 22, pp , Apr 975. [2] A. Tehranipour,, The Generalized M/G/C//N-Queue as a Model for Time-Sharing Systems, ACM-IEEE Joint Symposium on Applied Computing, Fayetteville, Apr, 99. [22] K. S. Trividi, Probability & Statistics with Reliability, Queueing and Computer Science Applications, Prentice-Hall, Inc., New Jersey, 982. [23] Qin, Sholl, Ammar, Micro Time Cost Analysis of Parallel Computations, IEEE Trans. Computers, vol. 4, no. 5, pp , May 99. [24] Yan, Zhang, Song, An Effective and Practical Performance Prediction Model for Parallel Computing on Nondedicated Heterogeneous Networks of Workstations, J. Para Dist Computing, vol.38, No., pp. 63-8, 996. [25] T. Zhang, S. Kang and Lester Lipsky,, On The Performance of Parallel Computers: Order Statistics and Amdahl s Law, International Journal Of Computers And Their Applications, Vol 3, No. 2, August 996.