CCIE 400-101: Network Principles - General Network Challenges

on

Asymmetric routing

  • Quite simply, asymmetric routing is when a packet takes one path to the destination and returns using another path.
  • Asymmetric routing in itself is not a problem from a TCP/IP communication perspective but it does create trouble with certain firewall setups.
  • With stateful inspection, the firewall keeps track of active connections in its state table. Packets that do not have an entry in the state table or are not explicitly permitted by an access-list, will be discarded. Due to this behavior, asymmetric routing can cause packets to be dropped.


  • Assume that the Client sends an HTTP Request to the Web Server. The packet exits the network through ASA1, which builds and records the connection state entry. 
  • Now, the HTTP Response is forwarded to ASA2 but it does not have any connection information for the packet or an access-list that would allow the packet to pass, so the HTTP Response is dropped.

Unicast flooding

  • Unicast flooding is the unintentional behavior of a switch treating a unicast packet like a broadcast. This occurs when the destination MAC address is not found in the CAM table. The switch then floods the packet (broadcast). 
  • This can happen when a device wants to send a packet to a host for which it has a valid ARP entry but the MAC address has timed out from the switch's CAM table.  
  • The primary causes for unicast flooding are asymmetric routing, STP topology changes, and CAM table overflow.

Asymmetric Routing

  • Consider the traffic flows in the diagram below:
  1. S1--VLAN 1--Switch A--Router A--VLAN 2--Switch B--VLAN 2--S2 (blue line)
  2. S2--VLAN 2--Switch B--Router B--VLAN 1--Switch A--VLAN 1--S1 (red line)


  • Let's think about what happens when traffic flows from S1 to S2. The servers are in different VLANs. Router A is the default gateway for S1, and Router B is the default gateway for S2. 
  • S1 sends an ARP request for its default gateway (Router A) and receive a reply. Meanwhile, Switch A tracks both MAC addresses (S1 and Router A) in its CAM table.
  • Router A then pushes out an ARP request and saves the MAC address of S2 in its ARP cache. Switch A also records the MAC address of S2.
  • But what happens after this point? Router A will not receive packets from S2 directly as they will traverse through Router B. Therefore, the Switch A will actually learn the MAC address of Router B, which rewrites the frame after receiving it from S2. 
  • Note that Router A still has a valid ARP entry for S2 (4 hours by default) but the MAC address will time out after 300 seconds (default). So after 5 minutes, Switch A will have no record of S2 in its CAM table but Router A will still build the frame correctly due to the valid ARP entry. This causes Switch A to receive a packet to a destination MAC address to which it has no entry. Switch A must then flood the packet.
  • The same problem will occur in the reverse direction as well.

 

STP Topology Changes

  • TCN BPDUS shorten the CAM aging time to the ForwardDelay (15 seconds by default). If the MAC address is not relearned within that interval, the entry is deleted. 
  • Flooding will occur as the switch rebuilds its CAM table.
  • Again, the end hosts likely still have valid ARP entries to build the Layer 2 packet but the switch does not know how to reach the destination MAC addresses.
  • Ports with the STP PortFast feature enabled will not cause TCNs when going to or from the forwarding state.

CAM Table Overflow

  • The CAM table is full and new MAC addresses cannot be learned. 
  • Packets destined to such MAC addresses are flooded until  space becomes available in the CAM table.
  • This is possible but rare, since most modern switches have large enough forwarding tables to accommodate MAC addresses for most designs. 
  • CAM table exhaustion can also be caused by an attack where one host starts generating frames each sourced with different MAC address - eating up all CAM table resources. Once the CAM table becomes saturated, other traffic will be flooded because new learning cannot occur. This kind of attack can be prevented by limiting the number of MAC addresses learned on untrusted ports by using the port security feature.  


Out of Order Packets

  • Out of order delivery simply means packets arriving to destination in a different order from which they were sent. 
  • Out-of-order packets can be caused by a per-packet load balancing using multiple different speed paths to reach the target, poorly configured queuing along a path or asymmetric routing.
  • In the case of queuing, out-of-order packets can occur when the packets are not forwarded in a first-in/first-out (FIFO) order. 
  • Packet reordering can affect performance and efficiency of packet in the following ways:
    1. Cause unnecessary retransmission: When the TCP receiver gets packets out of order, it sends duplicate ACKs to trigger fast retransmit algorithm at the sender. These ACKs makes the TCP sender infer a packet has been lost and retransmit it.
    2. Limit transmission speed: When fast retransmission is triggered by duplicate ACKs, the TCP sender assumes it is an indication of network congestion. It reduces  its congestion window to limit the transmission speed, which needs to grow larger from a “slow start” again.
    3. Reduce the recipient's efficiency: TCP receiver has to hand in data to the upper layer in order. When reordering happens, TCP has to buffer all the out-of-order packets until getting all packets in order. Meanwhile, the upper layer gets data in burst rather than smoothly, which also reduce the system efficiency as a whole. 

Impact of Microburst

  • Microbursting is a phenomenon where rapid bursts of data traffic are sent in quick succession, leading to periods of full line-rate transmission that can overflow packet buffers.
  • Interface ignores and overruns typically point to microbursts. This is indicative of the Rx ring and the corresponding packet buffer being overwhelmed due to data bursts over extremely short periods of time (microseconds).
  • Microbursts are difficult to detect because the interface counters typically average the traffic over a 5-minute interval by default.
  • Example: The interface utilization is only 20-30% of the available bandwidth but the output drop count is high. 

Switch#show interface Fa0/1 | include duplex|output drops|rate
  Full-duplex, 10Mb/s, media type is 10/100BaseTX
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 7228
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 2921000 bits/sec, 477 packets/sec

  • If the output drops increase with interface utilization closing on line speed, more bandwidth is needed.
  • If the output drops increment, but the overall traffic utilization of that interface is otherwise low, most likely the network is experiencing some type of bursty traffic. 
  • To troubleshoot microbursts, a packet sniffer that can capture traffic over a long period of time and allow analyzing of the results in a graph with very short time intervals is needed.

Comments

Post a Comment