Study Case: Cisco Modeling Labs 1.3 / Nexus 9000v / vPC

Background

The latest release Cisco Modeling Labs 1.3 added support for Nexus 9000v. I've been tasked to research the features and functionality to see if our NX-OS hardware labs could be run on CML. I've been given a lab guide with instructions and commands to test, which actually turned out to be outdated, incomplete, or just plain wrong. So what was supposed to be a simple copy/paste and verify job became an in-depth investigation. (Note: I have little to no prior experience with Nexus.) This post looks at my progress so far.

See: Cisco Modeling Labs 1.3 Release Notes

Virtual Port Channel (vPC) Overview

The first issue occurred when configuring vPC. When enabling the vPC feature and defining a vPC domain using the command vpc domain 34, I received the following log message.

N9K-1# configure

Enter configuration commands, one per line. End with CNTL/Z.

N9K-1(config)# feature vpc

N9K-1(config)# vpc domain 34

N9K-1 %$ VDC-1 %$ %STP-2-VPC_PEERSWITCH_CONFIG_DISABLED: vPC peer-switch configuration is disabled. Please make sure to change spanning tree "bridge" priority as per the recommended guidelines.

To this, Cisco documentation says:

Error Message STP-2-VPC_PEERSWITCH_CONFIG_DISABLED: vPC peer-switch configuration is disabled. Please make sure to change spanning tree "bridge" priority as per the recommended guidelines.

Explanation

Recommended Action

See: Cisco Nexus 9000 Series NX-OS System Messages Reference

What exactly are these vPC peer-switch recommended guidelines? There does not appear to be a short and clear answer, so figured I need to study vPC at least on a foundational level to be able to understand the root cause for this log message. Let's learn more about vPC!

First things first, vPC allows two physical Nexus switches to appear as one logical device. vPC belongs to the Multichassis EtherChannel (MEC) family of technologies. It is very similar to Virtual Switching System (VSS), which is available on high-end Catalyst switches. There are some differences between vPC and VSS, but I might leave those for another post.

The main benefit of vPC is eliminating STP blocked ports by providing a logical loop-free topology, and as a result, enabling full use of available bandwidth (active-active). vPC also provides fast convergence upon link or device failure. It can take over 30 seconds to move a STP blocking port to the forwarding state.

In the diagram below, the leftmost topology reflects a "normal" situation, where the access switch S3 has redundant paths to the distribution switches S1 and S2, and STP is blocking one of the links. The middle topology shows the physical layout of the vPC configuration. The rightmost topology illustrates how the devices operate logically - S3 sees S1 and S2 as one device, no ports are blocked.

See: BRKDCN-2378 - VPC Best Practices and Design on NX-OS

To correctly design and configure vPC you must have a sound knowledge of the vPC architecture components (there are many!), and follow the recommended design guidelines. The diagram below summarizes the terminology in one image, but we'll look at each component in more detail.

See: THE COMPLETE CISCO NEXUS VPC GUIDE

vPC domain: The common domain configured across the pair of vPC peer devices. The numerical value identifies the vPC and must match on both vPC peer devices. Only one domain ID per device is permitted.

vPC peer: The adjacent device connected via the vPC peer-link. One is acting as primary, the other as secondary.

vPC peer-link: Used to synchronize the state between the vPC peer devices by sending vPC control packets, which creates the illusion of a single control plane.

vPC peer keepalive link: A Layer 3 communications path used to test if the remote peer is operating properly by sending periodic keepalive messages (heart beat). The vPC peer keepalive link can be a management interface or switched virtual interface (SVI). No data or synchronization traffic moves over the vPC peer keepalive link; the only traffic on this link is a message that indicates that the originating switch is operating and running vPC.

vPC member port: A port who is a member of one of the vPCs configured on the vPC peers.

Cisco Fabric Services (CFS): Underlying protocol running on top of vPC peer-link providing reliable synchronization and consistency check mechanisms between the two peer devices.

vPC VLAN: A VLAN carried over the vPC peer-link and used to communicate via vPC with a third device. As soon as a VLAN is defined on vPC peer-link, it becomes a vPC VLAN.

Non-vPC VLAN: A VLAN that is not part of any vPC and not present on vPC peer-link.

Orphan device: A device that is on a vPC VLAN but only connected to one vPC peer and not to both.

Orphan port: A port that connects to an orphan device in vPC VLAN.

vPC Configuration

The order of the vPC configuration is important, and a basic vPC setup is established using the following four steps. (The random values used are straight from the lab guide.) Below is my lab topology.

1. Enable the vPC feature and configure the domain ID.

feature vpc

vpc domain 34

In this step, I got the log message that inspired me to study vPC and write this blog in the first place, but I ignored it for now.

2. Establish peer keepalive connectivity.

vrf context keepalive

interface eth1/5
no switchport
vrf member keepalive
ip address 192.168.251.93/24
no shutdown

The mgmt0 interfaces in my virtual lab are not connected, but I wanted a dedicated link for the keepalive process. I configured interface eth1/5 as a Layer 3 port and defined it in its own VRF.

A quick ping test to verify the link is up and operational.

N9K-1# ping 192.168.251.94
PING 192.168.251.94 (192.168.251.94): 56 data bytes
64 bytes from 192.168.251.94: icmp_seq=0 ttl=254 time=57.111 ms
64 bytes from 192.168.251.94: icmp_seq=1 ttl=254 time=32.483 ms
64 bytes from 192.168.251.94: icmp_seq=2 ttl=254 time=30.425 ms
64 bytes from 192.168.251.94: icmp_seq=3 ttl=254 time=37.146 ms
64 bytes from 192.168.251.94: icmp_seq=4 ttl=254 time=55.237 ms

--- 192.168.251.94 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 30.425/42.48/57.111 ms

Next, the remote end of the vPC peer-keepalive link is configured.

vpc domain 34
peer-keepalive destination 192.168.251.94 source 192.168.251.93 vrf keepalive

Verify peer-keepalive connectivity is established.

N9K-1# show vpc peer-keepalive

vPC keep-alive status : peer is alive
--Peer is alive for : (14) seconds, (168) msec
--Send status : Success
--Last send at : 2017.08.25 08:15:25 900 ms
--Sent on interface : Eth1/5
--Receive status : Success
--Last receive at : 2017.08.25 08:15:25 849 ms
--Received on interface : Eth1/5
--Last update from peer : (0) seconds, (166) msec

vPC Keep-alive parameters
--Destination : 192.168.251.94
--Keepalive interval : 1000 msec
--Keepalive timeout : 5 seconds
--Keepalive hold timeout : 3 seconds
--Keepalive vrf : keepalive
--Keepalive udp port : 3200
--Keepalive tos : 192

3. Create the vPC peer link.

Configure a port-channel between the switches as the peer link.

feature lacp

interface eth1/3-4
switchport mode trunk
channel-group 171 mode active

interface po171
switchport mode trunk
vpc peer-link

After using the vpc peer-link command, there is another log message.

Please note that spanning tree port type is changed to "network" port type on vPC peer-link. This will enable spanning tree Bridge Assurance on vPC peer-link provided the STP Bridge Assurance(which is enabled by default) is not disabled.

Ignore it. Basically, it tells you that the operating system automatically changed the port to STP port type network and added the command spanning-tree port type network command (see the running configuration).

Verify the vPC domain is up.

N9K-1# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 34
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 0
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Disabled
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po171 up 1,171

4. Configure vPCs with downstream switches or end devices.

From the downstream switch's perspective (N9K-Pod), the vPC will look like a normal port-channel.

interface eth1/2-3
switchport
channel-group 172 mode active

interface po172
switchport mode trunk

For vPC, the configuration is slighty different. First, the interface is added to the port-channel, and then the port-channel is moved to the vPC. The vPC number and the port-channel number do not need to match, but it makes life easier if they do. However, the vPC numbers must match between the peer switches.

interface eth1/2
switchport mode trunk
channel-group 172 mode active

interface po172
switchport mode trunk
vpc 172

Verify the vPC, port-channel, STP statuses.

N9K-1# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 34
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 1
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Disabled
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po171 up 1,171

vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
172 Po172 up success success 1,171

N9K-Pod# show port-channel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
b - BFD Session Wait
S - Switched R - Routed
U - Up (port-channel)
p - Up in delay-lacp mode (member)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
172 Po172(SU) Eth LACP Eth1/2(P) Eth1/3(P)

N9K-Pod# show spanning-tree

VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 4097
Address 5e00.0000.0007
Cost 3
Port 4267 (port-channel172)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32769 (priority 32768 sys-id-ext 1)
Address 5e00.0001.0007
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po172 Root FWD 3 128.4267 P2p

VLAN0171
Spanning tree enabled protocol rstp
Root ID Priority 4267
Address 5e00.0000.0007
Cost 3
Port 4267 (port-channel172)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32939 (priority 32768 sys-id-ext 171)
Address 5e00.0001.0007
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po172 Root FWD 3 128.4267 P2p

Well, there you have it. The configuration is probably ugly, but it's something I managed to put together in a short time and at least it somehow works. From what I can tell, the vPC subject is huge, and I've barely even scratched the surface. Looking forward to learning more! Next: Design and Configuration Guide: Best Practices for Virtual Port Channels (vPC).

I didn't manage to find out what the log message from the beginning of this post fully meant. I was able to configure vPC without paying attention to it. I came across the peer-switch command, which enables the vPC switch pair to appear as a single STP root in the Layer 2 topology. From the show spanning-tree output on N9K-Pod, it would seem as if this is already the case even without the peer-switch command. This is something to investigate further...

Network Playroom

Search This Blog