Anycast atency How many sites are enough? Presented by Ricardo de Oliveira Schmidt October 25, 2016 Madrid, Spain Presentation copyright 2016 by Ricardo de Oliveira Schmidt
Reference: Anycast atency: How Many Sites Are Enough? Ricardo de O. Schmidt, John Heidemann and Jan Harm uipers Technical Report ISI-TR-2016-708, USC/Information Sciences Institute, May 2016 http://www.isi.edu/~johnh/papers/schmidt16a.pdf
Anycast Definition Multiple copies of a service at different locations configured to answer with the same IP address Purpose Distribution, redundancy, resilience, performance Where Content Delivery Networks (CDNs) DNS at the root and TD levels
Anycast
Anycast
Anycast
Anycast Anycast uses BGP to map users to sites But BGP only approximates closest match
RQ's and Approach If BGP only approximates closest in anycast: 1. How good (or bad) is this approximation? 2. What is the impact of location of anycast sites? 3. What is the impact of routing policy? Can we determine a minimum number of anycast sites as optimal? To answer these questions we...... study the relationship between latency and number of anycast sites... look at the Root DNS as case of study
Our case of study were four letters of the Root DNS: C: 8 sites : 58 sites : 33 sites : 144 sites We DO NOT focus on: What a letter should or not do Root DNS (or a part of it) C-Root -Root -Root -Root CC C C C C C C
RIPE Atlas Our vantage points (VPs) are RIPE Atlas probes 7,900 of them 174 countries 2927 ASes The bias towards Europe does not impact our qualitative results and conclusions
Measurements Step 1: determine catchment CHAOS queries to hostname.bind to anycast prefix...... returns a string that identifies a server CHAOS query anycast service S1 S2 AMS VP S1SYD.anycast-service.org S1 S2 SYD S1 S2 AX
Measurements Step 2: determine (optimal) latency ICMP (ping) to all sites of the anycast...... as given by the identifier from CHAOS anycast service RTT to AMS S1 S2 AMS VP RTT to SYD S1 S2 SYD RTT to AX Is RTT to SYD the lowest? S1 S2 AX
What Performance do we see? Distribution of RTT for all letters: 1 0.8 0.6 CD 0.4 0.2 0 median RTT ~30ms 0 50 100 150 200 250 300 350 RTT (ms) C-Root actual -Root actual -Root actual -Root actual We do not determine a threshold for good or bad performance...... but we do believe that 30ms is very good
What RTT is possible? 1 Comparing actual to optimal possible: C-Root with 8 sites is at its optimal -Root with 144 sites has plenty of room But their median is roughly the same CD 0.8 0.6 0.4 0.2 median RTT = 32ms 0 0 50 100 150 200 250 300 350 RTT (ms) C-Root C-Root actual C-Root optimal Note: Comparing anycast and unicast routing CD 1 0.8 0.6 0.4 -Root -Root actual -Root optimal 0.2 median RTT = 30ms 0 0 50 100 150 200 250 300 350 RTT (ms)
What about location? ocation matters! Simulation using C-Root sites 1 0.9 0.8 from west to east coast Note: CD 0.7 0.6 0.5 0.4 0.3 C-Root optimal AX+ORD+IAD+J AX+ORD+IAD AX+ORD AX (only) C-Root has sites in Europe Bias from RIPE Atlas is visible 0.2 0.1 0 0 50 100 150 200 250 300 350 RTT (ms)
What about location? ocation matters! Simulation using C-Root sites from US and Europe 1 0.9 0.8 0.7 0.6 CD 0.5 Note: C-Root has sites in Europe Bias from RIPE Atlas is visible 0.4 0.3 0.2 0.1 0 C-Root optimal CDG CDG + AX CDG + AX + J CDG + AX + J + RA 0 50 100 150 200 250 300 350 RTT (ms)
What about location? long distribution tail ocation matters! Simulation using C-Root sites from US and Europe 1 0.9 0.8 0.7 0.6 CD 0.5 Note: C-Root has sites in Europe Bias from RIPE Atlas is visible 0.4 0.3 0.2 0.1 0 C-Root optimal CDG CDG + AX CDG + AX + J CDG + AX + J + RA 0 50 100 150 200 250 300 350 RTT (ms)
Sites vs ocation So, at the end many sites do help those at the end of the distribution tail...... but it depends where and how connected RTT (ms) 300 200 100 A A A A A A A A A A A A A A A A A A A A A A A E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E S S S S N N N N N O O O O Africa () Asia (A) Europe (E) South America (S) North America (N) Oceania (O) C-Root 0 MY (20) SC (6) RE (5) MU (5) UG (5) TZ (5) ID (33) BD (13) NP (8) AU (69) CN (14) ZA (52) BT (5) IN (15) SG (45) PH (26) UY (10) C (16) E (10) NC (5) AR (18) P (7) H (22) SN (6) BR (49) NZ (53) GU (5) TW (5) (22) R (10) AE (8) JP (72) B (5) BJ (6) IR (28) AD (9) Z (21) CR (5) TN (7) I (18) CY (10) PR (6) GR (46) AM (17) GE (8) TR (20) IS (8) RU (307) BY (15) M (9) A (15) IE (75) BA (7) I (59) NO (83) T (10) V (15) EE (18) PT (48) RO (28) UA (128) SE (96) BG (66) IT (153) MD (8) P (92) RS (29) CA (132) HR (26) ES (93) BE (140) CZ (166) HU (36) DE (612) US (682) D (78) GB (347) CH (163) R (477) SI (28) AT (106) N (330) S (32) U (25) country code (# of VPs)
Sites vs ocation So, at the end many sites do help those at the end of the distribution tail...... but it depends where and how connected RTT (ms) 300 200 100 A A A A A A A A A A A A A A A A A A A A A A A E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E S S S S N N N N N O O O O Africa () Asia (A) Europe (E) South America (S) North America (N) Oceania (O) C-Root 0 MY (20) SC (6) RE (5) MU (5) UG (5) TZ (5) ID (33) BD (13) NP (8) AU (69) CN (14) ZA (52) BT (5) IN (15) SG (45) PH (26) UY (10) C (16) E (10) NC (5) AR (18) P (7) H (22) SN (6) BR (49) NZ (53) GU (5) TW (5) (22) R (10) AE (8) JP (72) B (5) BJ (6) IR (28) AD (9) Z (21) CR (5) TN (7) I (18) CY (10) PR (6) GR (46) AM (17) GE (8) TR (20) IS (8) RU (307) BY (15) M (9) A (15) IE (75) BA (7) I (59) NO (83) T (10) V (15) EE (18) PT (48) RO (28) UA (128) SE (96) BG (66) IT (153) MD (8) P (92) RS (29) CA (132) HR (26) ES (93) BE (140) CZ (166) HU (36) DE (612) US (682) D (78) GB (347) CH (163) R (477) SI (28) AT (106) N (330) S (32) U (25) country code (# of VPs) RTT (ms) 300 200 100 A A A A A A A A A A A A A A A A A A A A A A E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E S S S S S N N N N N N O O O Africa () Asia (A) Europe (E) South America (S) North America (N) Oceania (O) -Root 0 BD (14) NP (9) SC (6) IR (26) PH (20) PR (5) BT (6) SG (48) PT (52) SN (6) GE (11) JP (88) H (25) (20) BY (13) I (20) HR (37) BA (6) IQ (7) BJ (6) ES (112) A (17) IS (10) MY (20) MT (5) E (15) CY (13) DO (6) IT (189) M (13) EE (24) T (15) MD (9) RE (5) MU (5) B (6) PE (5) BG (70) N (407) HU (56) CA (150) IN (13) SI (34) P (120) S (36) DE (730) AT (147) RO (34) US (812) AD (8) MX (5) CZ (214) I (84) IE (88) U (37) GB (464) V (20) RU (371) ZA (64) P (7) AU (81) Z (21) BE (171) GR (62) UA (172) R (572) CH (191) TH (5) AR (20) MZ (5) D (98) ID (30) C (11) NZ (58) BR (42) GU (5) NO (107) AE (6) SE (135) R (14) UY (13) RS (34) TR (24) country code (# of VPs)
Sites vs ocation So, at the end many sites do help those at the end of the distribution tail...... but it depends where and how connected distribution tail RTT (ms) RTT (ms) 300 200 100 0 300 200 100 A A A A A A A A A A A A A A A A A A A A A A A E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E S S S S N N N N N O O O O MY (20) SC (6) RE (5) MU (5) UG (5) TZ (5) ID (33) BD (13) NP (8) AU (69) CN (14) ZA (52) BT (5) IN (15) SG (45) PH (26) UY (10) C (16) E (10) NC (5) AR (18) P (7) H (22) SN (6) BR (49) NZ (53) GU (5) TW (5) (22) R (10) AE (8) JP (72) B (5) BJ (6) IR (28) AD (9) Z (21) CR (5) TN (7) I (18) CY (10) PR (6) GR (46) AM (17) GE (8) TR (20) IS (8) RU (307) BY (15) M (9) A (15) IE (75) BA (7) I (59) NO (83) T (10) V (15) EE (18) PT (48) RO (28) UA (128) SE (96) BG (66) IT (153) MD (8) P (92) RS (29) CA (132) HR (26) ES (93) BE (140) CZ (166) HU (36) DE (612) US (682) D (78) GB (347) CH (163) R (477) SI (28) AT (106) N (330) S (32) U (25) country code (# of VPs) Africa () Asia (A) Africa () Asia (A) Europe (E) South America (S) Europe (E) South America (S) North America (N) Oceania (O) A A A A A A A A A A A A A A A A A A A A A A E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E S S S S S N N N N N N O O O North America (N) Oceania (O) C-Root -Root 0 BD (14) NP (9) SC (6) IR (26) PH (20) PR (5) BT (6) SG (48) PT (52) SN (6) GE (11) JP (88) H (25) (20) BY (13) I (20) HR (37) BA (6) IQ (7) BJ (6) ES (112) A (17) IS (10) MY (20) MT (5) E (15) CY (13) DO (6) IT (189) M (13) EE (24) T (15) MD (9) RE (5) MU (5) B (6) PE (5) BG (70) N (407) HU (56) CA (150) IN (13) SI (34) P (120) S (36) DE (730) AT (147) RO (34) US (812) AD (8) MX (5) CZ (214) I (84) IE (88) U (37) GB (464) V (20) RU (371) ZA (64) P (7) AU (81) Z (21) BE (171) GR (62) UA (172) R (572) CH (191) TH (5) AR (20) MZ (5) D (98) ID (30) C (11) NZ (58) BR (42) GU (5) NO (107) AE (6) SE (135) R (14) UY (13) RS (34) TR (24) country code (# of VPs)
inal Considerations ooking at real-world anycast deployments we learned that...... number of sites can make a difference on performance, but...... location and connectivity of sites have higher impact... we believe that ~12 well-connected sites is "enough" for performance or the future, we will focus on other purposes of anycast: Resilience to Denial of Service attacks oad balancing
inal Considerations ooking at real-world anycast deployments we learned that...... number of sites can make a difference on performance, but...... location and connectivity of sites have higher impact... we believe that ~12 well-connected sites is "enough" for performance or the future, we will focus on other purposes of anycast: Resilience to Denial of Service attacks oad balancing in about 20 minutes (depending on Wouter)
r.schmidt@utwente.nl http://www.ricardoschmidt.com/ Acknowledgements: Geoff Huston, George Michaelson, Ray Bellis, Cristian Hesselman, Benno Overeinder, Jaap Akkerhuis, Duane Wessels, Paul Vixie, Romeo Zwart, Anand Buddhdev, and C-Root operators. SIDN abs, Nnet abs and SURnet Self-managing Anycast Networks for the DNS (SAND) project http://www.sand-project.nl/ DNS Anycast Security (DAS) project http://www.das-project.nl/