The Analysis of Microburst (Burstiness) on Virtual Switch

Transcription

1 The Analysis of Microburst (Burstiness) on Virtual Switch Chunghan Lee Fujitsu Laboratories Copyright 2016 FUJITSU LABORATORIES LIMITED

2 Background What is Network Function Virtualization (NFV)? NFV virtualizes network functions (e.g. Firewall, Load balancer) on IA servers The function is called as virtualized network function (VNF) Virtual switch is one of major components for virtual network on NFV Infrastructure (NFVI) QoS property is also important for NFV Virtual network* *Virtual switch is included in virtual network An image is from 1 Copyright 2014 FUJITSU LABORATORIES LIMITED

3 Microburst (1) What is microburst? Spikes in shot time period, but causing decreased performance Impact of microburst on network Although traffic with QoS property is generated, the spikes can be occurred by massive traffic in very short time period This phenomenon cannot catch the ordinary monitoring (SNMP) When packet drop is occurred by the spike, it is changed as the microburst SNMP (Coarse-grained) Spike Packet (Find-grained) Packet drop!! Sudden spikes in throughput 2 Copyright 2016 FUJITSU LABORATORIES LIMITED

4 Microburst (2) Impact of microburst on NFVI Although traffic with QoS property is generated on VNF, the sudden spikes would be occurred due to resource state Packet loss can be occurred by the spikes (microbursts) Traffic with QoS Sudden spikes!! NFVI (tx) NFVI (rx) 3 Copyright 2016 FUJITSU LABORATORIES LIMITED

5 Problem Microburst on NFVI Ordinary application measurement cannot catch the microbursts on NFVI There are queues and buffers between NFVIs A deep understanding of packet processing issued by virtual switch and Linux kernel is required to clarify the cause of microburst Traffic with QoS Sudden spikes!! NFVI (tx) NFVI (rx) 4 Copyright 2016 FUJITSU LABORATORIES LIMITED

6 GOAL Investigate the occurrence of microburst on NFVI Find the cause of microbursts on NFVI 5 Copyright 2014 FUJITSU LABORATORIES LIMITED

7 Approach Prepare two types of UDP traffic to observe the microbursts on an OvS bridge Foreground : Target with QoS property Background : Occur a lack of CPU resource at kernel Generate fore/background traffic and capture their packets with tcpdump Analyze them in packet-level to observe the microburst Profile kernel functions to clarify the cause of microbursts 6 Copyright 2016 FUJITSU LABORATORIES LIMITED

8 Overview of testbed Foreground traffic (UDP) Sending rate : 2Gbps [Datagram size : 1400 bytes (No fragmentation)] vport tx queue : default value (0), UDP buffer : default value (200 KBytes) Background traffic (UDP) 8 vports are used on the same OVS bridge iperf using UDP mode with 10 parallel flows per vport (iperf option p) Sending rate of UDP flow : 1Gbps (configured bandwidth by iperf) 7 Copyright 2016 FUJITSU LABORATORIES LIMITED

9 Server and switch spec. Overview of spec. All servers have the same spec. Server type : Fujitsu RX100S7 CPU (4 cores) Intel(R) Xeon(R) CPU 3.10GHz Memory 16GB (Speed: 1333 MHz) OS CentOS 7.2 Kernel version NIC 10G:Intel X710 iperf Open vswitch (release version) vport queue Default value (txqueuelen : 0) UDP buffer Default value (200 Kbytes) Switch spec. Fujitsu SR-X 526 (10G switch) 8 Copyright 2016 FUJITSU LABORATORIES LIMITED

10 Investigation of microburst 9 Copyright 2016 FUJITSU LABORATORIES LIMITED

11 Measurement points 6 throughput measurement points Sender : iperf, Entering OVS, Leaving OVS Receiver : iperf, Entering OVS, Leaving OVS iperf (TX) iperf Entering OVS (tcpdump) OVS (bridge) iperf (RX) iperf User Kernel Leaving OVS (tcpdump) OVS (bridge) Leaving OVS (tcpdump) NIC driver Network NIC driver Entering OVS (tcpdump) < Sender > < Receiver > Throughput measurement points 10 Copyright 2016 FUJITSU LABORATORIES LIMITED

12 Throughput at 6 measurement points Throughput of foreground per second Leaving OVS (sender), the throughput is decreased At receiver, the throughput is also decreased at iperf (RX) < Sender > Decreased throughput < Receiver > Decreased throughput 11 Copyright 2016 FUJITSU LABORATORIES LIMITED

13 Throughput of foreground at sender Throughput of foreground per 1 millisecond Entering OVS, the sending rate is fluctuated due to a lack of CPU Leaving OVS, sudden spikes in throughput are found Throughput with QoS property is changed Unit time : 1 millisecond < Entering OVS > < Sender > iperf < Leaving OVS > Entering OVS OVS (bridge) Leaving OVS NIC driver Network 12 Copyright 2016 FUJITSU LABORATORIES LIMITED

14 Packet drop and throughput at receiver A relation between throughput and packet drop The timing of packet drop is similar to the timing of sudden spikes in throughput at receiver The overflow of socket buffer is frequently occurred by the sudden spikes (spikes microbursts) < Receiver > < Packet drop at socket > iperf Leaving OVS Socket OVS (bridge) < Leaving OVS > NIC driver Network 13 Copyright 2016 FUJITSU LABORATORIES LIMITED

15 Packet drop and spacing at receiver A relation between packet drop and packet spacing The packet spacing with moving average (MA)* is decreased while the number of drop packets on socket buffer is increased Microburst Packet spacing Burstiness Packet loss < Receiver > iperf Leaving OVS Socket OVS (bridge) NIC driver *MA leg : 1000 Network 14 Copyright 2016 FUJITSU LABORATORIES LIMITED

16 Cause of microburst 15 Copyright 2016 FUJITSU LABORATORIES LIMITED

17 Find the cause of microburst Profiling kernel functions using perf Profiling rate (sampling rate) is 1 millisecond Focus on process (iperf with 2Gbps) only for the profiling Profiling Kernel by perf 16 Copyright 2016 FUJITSU LABORATORIES LIMITED

18 Common function call graph Packet processing at Linux kernel Net I/F Enqueue (qdisc, tx-ring) From OVS to qdisc OVS (bridge) From vport to OVS IP layer (TX) TCP/UDP layer (TX) Copy data iperf Kernel User 17 Copyright 2016 FUJITSU LABORATORIES LIMITED

19 Measurement points with TCP/IP stack Leaving OVS, the packet capture is occurred after qdisc Entering OVS Leaving OVS Entering OVS Leaving OVS Throughput measurement points 18 Copyright 2016 FUJITSU LABORATORIES LIMITED

20 Summary Throughput of foreground per second Leaving OVS (sender), the throughput is decreased At receiver, the throughput is also decreased at iperf (RX) < Sender > Packet drop at qdisc < Receiver > Packet drop at socket buffer by microbursts 19 Copyright 2016 FUJITSU LABORATORIES LIMITED

21 Conclusion We investigated the occurrence of microbursts on NFVI A major cause of microburst is packet queuing on qdisc, and the packet loss at socket buffer on receiver is occurred by the microbursts At qdisc, the throughput is decreased to 63% and the queue size is not enough to absorb the packets At socket buffer (rx), the throughput is decreased to 41% and the buffer capacity is also not enough We found the cause of microbursts using kernel profiling Although the total sending rate is 10 Gbps, qdisc at the sender is frequently full 20 Copyright 2016 FUJITSU LABORATORIES LIMITED

22 Future work Clarify the cause why qdisc is frequently full although the total sending rate is 10 Gbps Analyze the profiling results with kernel trace Modify Linux kernel to change the packet capture point Extend the experiments Without OVS (Clarify the overhead of OVS) With TCP, DPDK OVS, VMs based on vhost 21 Copyright 2016 FUJITSU LABORATORIES LIMITED

23 22 Copyright 2015 FUJITSU LABORATORIES LIMITED