Stochastic Analysis and File Availability Enhancement for BT-like File Sharing Systems. John.C.S.Lui The Chinese University of Hong Kong

Transcription

1 Stochastic Analysis and File Availability Enhancement for BT-like File Sharing Systems John.C.S.Lui The Chinese University of Hong Kong 1

2 Outline 2

3 Outline Introduction 2

4 Outline Introduction Modeling the Performance 2

5 Outline Introduction Modeling the Performance Extension on heterogeneous Network 2

6 Outline Introduction Modeling the Performance Extension on heterogeneous Network Modeling the Availability 2

7 Outline Introduction Modeling the Performance Extension on heterogeneous Network Modeling the Availability Availability Improvement Algorithms 2

8 Outline Introduction Modeling the Performance Extension on heterogeneous Network Modeling the Availability Availability Improvement Algorithms Conclusion 2

9 The Problem with C/S Publishing 3

10 The Problem with C/S Publishing 3

11 The Problem with C/S Publishing More customers require more bandwidth 3

12 BitTorrent Solution 4

13 BitTorrent Solution 4

14 BitTorrent Solution Users cooperate in the distribution 4

15 Publishing Content Web Server 5

16 Publishing Content Web Server 5

17 Publishing Content AAA.torrent Web Server 5

18 Publishing Content Tracker 5

19 Publishing Content Tracker 5

20 Publishing Content 5

21 Motivation 6

22 Motivation To understand the BT protocol 6

23 Motivation To understand the BT protocol Why is it good? 6

24 Motivation To understand the BT protocol Why is it good? Can we do better? 6

25 Motivation To understand the BT protocol Why is it good? Can we do better? Contribution: 6

26 Motivation To understand the BT protocol Why is it good? Can we do better? Contribution: Analytical metrics 6

27 Motivation To understand the BT protocol Why is it good? Can we do better? Contribution: Analytical metrics Insights for Protocol-designers 6

28 Related Work X. Yang and et al, Infocom 2004: Simple Markov Model Numerical calculation needed L. Massoulie and et al, Sigmetrics 2004 Detailed Markov Model D. Qiu and et al, Sigcomm 2004 Simple Fluid Model Some analytical results are obtained 7

29 BitTorrent Model 8

30 BitTorrent Model 8

31 N peers BitTorrent Model 8

32 N peers BitTorrent Model The File is divided into M chunks 8

33 N peers BitTorrent Model Peer i The File is divided into M chunks I have Ci chunks 8

34 N peers BitTorrent Model Peer i The File is divided into M chunks I have Ci chunks Peer j I have Cj chunks 8

35 after some time collects afigure small n 1: are 2M states ineven the model), then theitstate space is extr collects a small number of chunks, the effectiveness of this new peer is veryas ha In [18, 21], all peers arelarge considered different from peers with number of and distinguish the states of peers by the number of However it[18, is not true inpeers reality because when h large number of chunks. On the other hand, if we consider all combination of F Inchunks 21], all are considered different [16] (i.e. peers with only BitTorrent and peers with onlymodel F, F are of the same type so w 3 some 4ofisdifferent timetrue it collects asosmall not in reality because. peers with only F1, F2 and peers with only Even F3However,MFafter types therenumw 4 are it are 2 states in the model), then the state sp N peers different from peersactually with large number of chu distributed among peers,after which could be ensu Even some time it collects a small l), then the state space is extremely large. In and this distinguish paper, we use different approach the astates of peers by the n different chunks [16] (i.e. peers with only F, F 1 different from peers with large number o i and peer j have c and c chunks respectively, where i j The File is divided of peers by the number of chunks they areand holding (i.e. peers with only F, F 1 2 peers with only F, F are of the same 3 4then the state space are 2M states in the model), different chunks [16] (i.e. peers with only into M chunks i at thatneed peermi can obtain least one useful chunk from F are of the same type so we + 1 Peer states). Assume chunks are uniformly distributed among which cou and distinguish thepeers, states of peersactually by the num M are 2 states in the model), then the state which actually could be ensured by the rarest-first chunk policy. peer clear that Pi,j =i and 1. When ccijfand,3we peers with only,f are oflet therespectiv same typ and peer j chave c4jhave: chunks iselection and distinguish theci states of peers by the I have chunks distributed among peers, which actually couldchb cj chunks respectively, where ci, cj {0, 1, that...,peer M }.i Let derive can us obtain at the leastprobability one useful andppeers with the sam 3, F4 are of = cionly 1and cfp[chunks in peer i,jj have i and peer chunks respectively j least one useful chunk from peer j, which we denote as P. When <cjc,jactually,we it have: is c clear that P = 1. When cciwhich i,j i,j i distributed among peers,! ci "one useful chunk that peer i can obtain at least n ci cj, we have: ci cj cj chunks respec i and cp and i i,j = 1 P[chun clear thatpeer Pi,j j= =have 1. 1 When c c, we have:!m i " j= 1 "M that peer i can obtain at least one!useful c c j i = 1 P[chunks in peer j are subset of chunks in peer i] P i i,j = 1 P[chunks cj clear that P = 1. When c c, we hav =i 1 = i,j! ci "!jci!"m " given the peer i and peercjj choldi j cso number (ci cj of + chunks 1) cj i (c i 1) = 1=!1M "(1) = 1 Peer j = 1!M " = 1. Pi,j P[chu Min (M 1)1). From (M Eq. cj +(1), 1) Iwe have Cj Figure need use Mpeer + c1ji and varia cj! ci "p So given the number of chunks chunks 8 4 cj the number of all states Mnumber is still large number, can So given the ofachunks peer i1and =need use!peer " in Figure 1). From Eq. (1), we hunks peer i and peer j holding, we can estimate the probability P (as illustratedmm

36 Simplified Peer States 9

37 Simplified Peer States I have little 9

38 Simplified Peer States I have little I have a lot 9

39 Simplified Peer States I have little I have a lot I am a seeder 9

40 Simplified Peer States I have little I have a lot I am a seeder 9

41 Simplified Peer States I have little I have a lot I am a seeder 9

42 Simplified Peer States I have little I have a lot I am a seeder 9

43 Simplified Peer States I have little I have a lot I am a seeder 9

44 Simplified Peer States I have little I have a lot I am a seeder 9

45 Simplified Peer States I have little I have a lot I am a seeder 9

46 Simplified Peer States X1 I have little I have a lot I am a seeder 9

47 Simplified Peer States X1 X2 I have little I have a lot I am a seeder 9

48 Simplified Peer States X1 X2 I have little I have a lot Y I am a seeder 9

49 Dynamics 10

50 Dynamics 10

51 Dynamics Transfer Rate from State 1 to State 2 10

52 Dynamics Transfer Rate from State 1 to State 2 Transfer Rate from State 2 to State 3 10

53 Dynamics Transfer Rate from State 1 to State 2 Transfer Rate from State 2 to State 3 10

54 Dynamics Transfer Rate from State 1 to State 2 Transfer Rate from State 2 to State 3 Downloading rate per connection 10

55 Dynamics Transfer Rate from State 1 to State 2 Transfer Rate from State 2 to State 3 10

56 Dynamics Transfer Rate from State 1 to State 2 Transfer Rate from State 2 to State 3 Connectivity 10

57 Dynamics Transfer Rate from State 1 to State 2 Transfer Rate from State 2 to State 3 10

58 Dynamics Transfer Rate from State 1 to State 2 Transfer Rate from State 2 to State 3 Bandwidth Constraint 10

59 Dynamics Transfer Rate from State 1 to State 2 Transfer Rate from State 2 to State 3 10

60 Dynamics Transfer Rate from State 1 to State 2 Transfer Rate from State 2 to State 3 Chunk Size 10

61 Dynamics Transfer Rate from State 1 to State 2 Transfer Rate from State 2 to State 3 10

62 Dynamics Transfer Rate from State 1 to State 2 Transfer Rate from State 2 to State 3 Chunk Number 10

63 Dynamics Transfer Rate from State 1 to State 2 Transfer Rate from State 2 to State 3 10

64 Compare the Dynamics with Simulation: 11

65 Compare the Dynamics with Simulation: 70 Number of Leechers/Seeders Leechers Seeders Our Model Qiu s[18] Model Simulation Number of Leechers/Seeders time t 11

66 Compare the Dynamics with Simulation: 11

67 Compare the Dynamics with Simulation: Number of Leechers/Seeders Seeders Leechers Our Model Qiu s[18] Model Simulation Number of Leechers/Seeders time t 11

68 Compare the Dynamics with Simulation: 11

69 Compare the Dynamics with Simulation: 140 Number of Leechers/Seeders Leechers Seeders Our Model Qiu s[18] Model Simulation time t 11

70 Compare the Dynamics with Simulation: 11

71 Steady State T d = X 1 + X 2 λ T p = { O( N 2 ) Case 1, O( N) Case 2 or 3. 12

72 Steady State By Little s Law, average downloading time in the steady state is derived by: T d = X 1 + X 2 λ T p = { O( N 2 ) Case 1, O( N) Case 2 or 3. 12

73 Steady State By Little s Law, average downloading time in the steady state is derived by: T d = X 1 + X 2 λ T p = { O( N 2 ) Case 1, O( N) Case 2 or 3. 12

74 Steady State By Little s Law, average downloading time in the steady state is derived by: T d = X 1 + X 2 λ T p = { O( N 2 ) Case 1, O( N) Case 2 or 3. 12

75 Steady State By Little s Law, average downloading time in the steady state is derived by: The system throughput in steady state is derived by: T p = T d = X 1 + X 2 λ { O( N 2 ) Case 1, O( N) Case 2 or 3. 12

76 The system shows good scalability: Insights: Scalability Tp=O(N^2) in Case 1 Tp=O(N) in Case 2 & 3 13

77 (b) Insights: Scalability Figure 2: Comparing dynamics of peer evolutions for our The system 220 shows good scalability: Tp=O(N^2) in Case 1 Tp=O(N) in Case 2 & 3 Average Downloading Time T d Our Model Qiu s[18] Model Simulation System Throughput T p Number of peers (a) T d as the function of N Figure 3: Comparing System Scalability 13

78 The system shows good scalability: Insights: Scalability Tp=O(N^2) in Case 1 Tp=O(N) in Case 2 & 3 13

79 (b) (c) Insights: Scalability of peer evolutions for our model and Qiu s model under three d The system shows good Our Model Qiu s[18] Model scalability: Simulation Tp=O(N^2) in Case 1 Tp=O(N) in Case 2 & 3 System Throughput T p Our Model Qiu s[18] Model Simulation Number of peers s the function of N Number of peers (b) T p as the function of N 13

80 The system shows good scalability: Insights: Scalability Tp=O(N^2) in Case 1 Tp=O(N) in Case 2 & 3 13

81 Insights: Popularity The arrival rate represents the popularity of the served file. T d λ = λ α λ 3/2 Case 1, 1 2 α λ 3/2 Case 2, 0 Case 3. More popular the file is, less downloading time, in Case 1 and 2. The downloading time keeps the same in Case 3. 14

82 Insights: Seeding Let T s = 1/γ be the average seeding time Increase seeding time T s : less downloading time T d in case 1 and 2; same downloading time T d in case 3. Extreme situation: T s =0: Downloading time T d won t be infinity 15

83 Insights: Topology The average degree of a peer in overlay: This degree is affected by the list returned by tracker (30-60 by default) Larger : reduce T d in case 1 and 2 won t help in case 3, only burden the network 16

84 Insights: Bandwidth Larger B: reduce T d in case 2 and 3 won t help in case 1 17

85 Where you are Introduction Modeling the Performance Extension on heterogeneous Network Modeling the Availability Availability Improvement Algorithms Conclusion 18

86 Impact of Firewall 19

87 Dynamics 20

88 Dynamics 20

89 Dynamics Arrival Rate of non-firewalled peers 20

90 Dynamics Arrival Rate of non-firewalled peers Arrival Rate of firewalled peers 20

91 Dynamics Arrival Rate of non-firewalled peers Number of nonfirewalled peers Arrival Rate of firewalled peers 20

92 Dynamics Arrival Rate of non-firewalled peers Number of nonfirewalled peers Arrival Rate of firewalled peers Number of firewalled peers 20

93 Insights: 21

94 Insights: Without non-firewalled peers, the peers behind firewalls can not finish downloading 21

95 Insights: Without non-firewalled peers, the peers behind firewalls can not finish downloading Non-firewalled peers perform better 21

96 Insights: Without non-firewalled peers, the peers behind firewalls can not finish downloading Non-firewalled peers perform better The performance gap is related to the arrival rate 21

97 Insights: Without non-firewalled peers, the peers behind firewalls can not finish downloading Non-firewalled peers perform better The performance gap is related to the arrival rate This gap can be very large even the two arrival rates are very close 21

98 Where you are Introduction Modeling the Performance Extension on heterogeneous Network Modeling the Availability Availability Improvement Algorithms Conclusion 22

99 File Availability 23

100 File Availability The probability that a peer can get ith chunk from its neighborhood: 23

101 File Availability The probability that a peer can get ith chunk from its neighborhood: 23

102 File Availability The probability that a peer can get ith chunk from its neighborhood: Number of copies of Chunk i 23

103 File Availability The probability that a peer can get ith chunk from its neighborhood: Number of copies of Chunk i Average number of neighbors 23

104 File Availability The probability that a peer can get ith chunk from its neighborhood: Number of copies of Chunk i Average number of neighbors 23

105 File Availability The probability that a peer can get ith chunk from its neighborhood: Number of copies of Chunk i Average number of neighbors 23

106 File Availability The probability that a peer can get ith chunk from its neighborhood: Number of copies of Chunk i The probability that a peer can finish its download: Average number of neighbors 23

107 File Availability The probability that a peer can get ith chunk from its neighborhood: Number of copies of Chunk i The probability that a peer can finish its download: Average number of neighbors 23

108 Optimal Chunk Distribution 24

109 Optimal Chunk Distribution Total Storage capacity 24

110 Optimal Chunk Distribution Maximize the probability of downloading all chunks: Total Storage capacity 24

111 Optimal Chunk Distribution Maximize the probability of downloading all chunks: Total Storage capacity 24

112 Optimal Chunk Distribution Maximize the probability of downloading all chunks: Total Storage capacity 24

113 Optimal Chunk Distribution Maximize the probability of downloading all chunks: Total Storage capacity 24

114 Optimal Chunk Distribution Maximize the probability of downloading all chunks: Total Storage capacity 24

115 Optimal Chunk Distribution Maximize the probability of downloading all chunks: Total Storage capacity The optimal solution: h1=h2= hm=c/m 24

116 Optimal Chunk Distribution Maximize the probability of downloading all chunks: Total Storage capacity Chunks should be distributed as equally as possible The optimal solution: h1=h2= hm=c/m 24

117 Define the Measurement 25

118 Define the Measurement 25

119 Define the Measurement Optimization solution to this unconstrained l1- norm problem: only increase hi when chunk i has the fewest copies 25

120 Define the Measurement Optimization solution to this unconstrained l1- norm problem: only increase hi when chunk i has the fewest copies Use Rarest First Policy 25

121 Define the Measurement Optimization solution to this unconstrained l1- norm problem: only increase hi when chunk i has the fewest copies Use Rarest First Policy Synchronization problem occurs in high connectivity system 25

122 Where you are Introduction Modeling the Performance Extension on heterogeneous Network Modeling the Availability Availability Improvement Algorithms Conclusion 26

123 { File Enhancement Algorithm Choose chunk i probabilistically F according to: σ i = h i h j >0 h j. h i = { V h i = 2( h h i ) M if h i h 0 otherwise. 27

124 Experiments Low bandwidth: vn GRF LRF GFAE LFAE RD Average Downloading Time !=0.2 GRF LRF GFAE LFAE!=0.3 RD!=0.4!=0.5!=0.6!=0.7!= Connection probability! (a) Availability, B = (b) Average downloading time, B =

125 Experiments High bandwidth: vn GRF LRF GFAE LFAE RD Average Downloading Time !=0.2!=0.3!=0.4!=0.5!=0.6 GRF LRF GFAE LFAE RD!=0.7!= Connection probability! (a) Availability, B = (b) Average downloading time, B = 12 29

126 Conclusion 30

127 Conclusion Propose the analytical model to understand BT 30

128 Conclusion Propose the analytical model to understand BT On Throughput, 30

129 Conclusion Propose the analytical model to understand BT On Throughput, On Availability, 30

130 Conclusion Propose the analytical model to understand BT On Throughput, On Availability, Sensitivity analysis on different system parameters. 30

131 Conclusion Propose the analytical model to understand BT On Throughput, On Availability, Sensitivity analysis on different system parameters. Extend the model to consider peers 30

132 Conclusion (Cont.) Validate the analytical result with extensive simulation (our model is more accurate than the Qiu s model) Propose new approach on chunk selection algorithm to enhance file availability 31