UCS vNIC Placement & vSphere vmnic E numeration on B460M4 servers

October 19th, 2016
UCS vNIC Placement & vSphere vmnic E numeration on B460M4 se...

I had the opportunity to work with the UCS B460 M4 blade servers last week and ran across some interesting challenges. The B460M4 servers are the dual, full-width form-factor blade servers for Cisco's Unified Computing System, and two of these blade servers completely fill up a single UCS 5108 chassis.

The B460M4 servers has (2) LAN on Motherboard (mLOM) and (4) PCIe mezzanine slots to provide network I/O redundancy and scalability. This allows for (2) UCS Virtual Interface Card (VIC) 1340s combined with (2) VIC port-expander cards, along with (2) additional VIC 1380 cards. All of these cards combine to provide 80 Gbps of bandwidth per fabric, and 160 Gbps of total bandwidth for the B460 M4 Blade server.

In order to provide traffic load-sharing and redundancy across these adapters, the placement of the virtual NICs (vNICs) that are presented to the OS must be carefully planned out. The OS is VMWare's vSphere 6.0 and because the VMNICs are actively nic-teamed, we want to be sure that the VMNICs team-pair are placed across each adapter in order to provide the best redundancy and traffic load-share.

There are essentially 4 physical adapters (the VIC 1340 and port expander both counts as a single adapter), that we need to place the vNICs on.  The B460 M4 server comprises of (2) B260 M4 servers connected together with a UCS scalability connector, which presents both blades as a single B460 M4 blade to UCSM.

 
 

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/hw/blade-servers/b_B260_M4/Upgrading_to_a_Cisco_UCS_B460_M4_Blade_Server.html#task_F6381A7575F347188EAF75E7D9E7EA97

In order to tie UCS virtual NICs (vNICs) to the physical adapters, there is a concept of virtual connection (vCON), where we map these vCONs to a specific adapter. There are two mapping schemes (Round-Robin & Linear Ordered) but for our scenario with 4 adapters - it really doesn't matter.  vCON1 will always map to Adapter 1, vCON2 will always map to Adapter 2, etc.

 

Here is a picture of the vCON to vNIC mapping that I came up with. Essentially each VMNIC pair (ex. vmnic0 and vmnic1) is mapped across the adapters within the same B260 server, so that in the event of an adapter failure, only ½ of the vmnic team is lost.  This is accomplished by mapping the VMNIC team across adapters within the same B260 server.  vCon1 and vCon2 maps to adapter 1 and adapter 3 respectively within B260M4 blade server #1, and vCon3 and vCon4 maps to adapter 4 and adapter 6 within B260M4 blade server #2.

 
 
 

Within UCS Central (there is an option to set this in UCS Manager as well), there is an option for manual vNIC placement within the service-profile template configuration detail, as indicated in the screenshot below:

 
 

As seen in the image above, the VMNICs are manually placed across the adapters, however when the OS (vSphere ESX) is installed, the OS enumerates the VMNICs based on the PCIe order presented.  

The BIG challenge here is that VMNICs will not line up with the vNIC interfaces within UCS, especially since there are 4 physical adapters and the actual & desired do not match up to the PCI express order presented in VMWare. For example, eth1 will not map to VMNIC1, which then becomes a huge support nightmare.  See image below:

 

To fix this issue, we'll have to re-number the VMNICs within ESX.  Here are the steps:

Step #1: Locate the management VMNIC (based on the MAC address), enable that VMNIC. Apply an IP address/mask/gateway to management vmkernel and enable SSH.  In our case, the MGMT VMNIC is VMNIC4.

 
 

Step #2: SSH to the host then run command “esxcfg-nics –l”, and pair up the mac address shown in the console window to the UCSM.

 

[root@localhost:~] esxcfg-nics -l
Name    PCI          Driver      Link Speed     Duplex MAC Address       MTU    Description
vmnic0  0000:8e:00.0 enic        Up   40000Mbps Full   00:25:b5:11:a0:6b 1500   Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic1  0000:cc:00.0 enic        Up   40000Mbps Full   00:25:b5:11:b0:6b 1500   Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic2  0000:4c:00.0 enic        Up   40000Mbps Full   00:25:b5:11:b0:68 1500   Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic3  0000:0e:00.0 enic        Up   40000Mbps Full   00:25:b5:11:a0:68 1500   Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic4  0000:08:00.0 enic        Up   40000Mbps Full   00:25:b5:11:a0:67 1500   Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic5  0000:46:00.0 enic        Up   40000Mbps Full   00:25:b5:11:b0:67 1500   Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic6  0000:88:00.0 enic        Up   40000Mbps Full   00:25:b5:11:a0:69 1500   Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic7  0000:89:00.0 enic        Up   40000Mbps Full   00:25:b5:11:a0:6a 1500   Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic8  0000:c6:00.0 enic        Up   40000Mbps Full   00:25:b5:11:b0:69 1500   Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic9  0000:c7:00.0 enic        Up   40000Mbps Full   00:25:b5:11:b0:6a 1500   Cisco Systems Inc Cisco VIC Ethernet NIC 

 

Step #3: Run the following command on the host to obtain the PCI bus address " localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias list | grep vmnic"

 

[root@localhost:~] localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias list | grep vmnic
pci       p0000:8e:00.0    vmnic0
pci       s00000009:04.00  vmnic7
pci       s0000000b:04.00  vmnic9
pci       s00000002:03.00  vmnic4
pci       s00000000:03.00  vmnic2
pci       s00000006:03.00  vmnic5
pci       s00000001:03.00  vmnic3
pci       s00000009:03.00  vmnic6
pci       p0000:cc:00.0    vmnic1
pci       s0000000b:03.00  vmnic8
[root@localhost:~]

 

Step #4: Open up UCSM again and map the PCI bus address, copy from the bus address output in Step #3 above. The output needs to be from Cisco UCS's perspective.

 
 

vmnic0 (eth0) -> vmnic6  s00000002:03.00
vmnic1 (eth1) -> vmnic8  s00000006:03.00
vmnic2 (eth2) -> vmnic2  s00000001:03.00
vmnic3 (eth3) -> vmnic3  s00000000:03.00
vmnic4 (eth4) -> vmnic7  s00000009:03.00 
vmnic5 (eth5) -> vmnic0  s0000000b:03.00
vmnic6 (eth6) -> vmnic9  s00000009:04.00
vmnic7 (eth7) -> vmnic1  s0000000b:04.00 
vmnic8 (eth8) -> vmnic4  p0000:8e:00.0
vmnic9 (eth9) -> vmnic5  p0000:cc:00.0

 

Step #5: Use the mapping notes that you’ve done between ESX and UCS and determine the bus address and paste them in this format.

 

localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias <vmnic number> ---bus-address <bus address> --bus-type pci

 

The bus address can be obtained from Step #4 above (marked in red)

 

localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic0 --bus-address s00000002:03.00 --bus-type pci
localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic1 --bus-address s00000006:03.00 --bus-type pci
localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic2 --bus-address s00000001:03.00 --bus-type pci
localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic3 --bus-address s00000000:03.00 --bus-type pci
localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic4 --bus-address s00000009:03.00 --bus-type pci
localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic5 --bus-address s0000000b:03.00 --bus-type pci
localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic6 --bus-address s00000009:04.00 --bus-type pci
localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic7 --bus-address s0000000b:04.00 --bus-type pci
localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic8 --bus-address p0000:8e:00.0 --bus-type pci
localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic9 --bus-address p0000:cc:00.0 --bus-type pci
 

 

Step #6: Reboot the host by running the “reboot” command in ESXi shell.

 

Step #7: Step #7:  Once the host reboots, change the MGMT VMNIC and compare the MAC addresses listed in VMWare to UCS again by running the "esxcfg-nics -l" command.

 
 

Voila! The VMNICs should match up with the vNIC order in UCS. If for some reason it doesn't match up, repeat the entire re-numbering process again. The vNIC placement issue isn't just isolated to VMWare's vSphere, it will also occur on other operating systems, like Microsoft Server 2012R2, especially with a 4-adapter setup on the B460M4 servers. However, re-naming NICs in Server 2012R2 is much easier. 

 

 

Join the High Availability, Inc. Mailing List

Subscribe