Over the last few months I had to work with Cisco and VMware support to solve an issue with Nexus 1000v in my customer environment. Almost one year ago! we deployed Cisco Nexus 1000v with L3 control (I wrote a post about Cisco Nexus 1000v L3 Capability so I recommend that you make yourself familiar with this post) and we had some problems with working Nexus 1000v in VMware infrastructure. Now, just for a quick reminder, L3 Control interface is used for VSM-VEM traffic. During design L3 Capability in my customer environment (I had some design considerations such as DMZ limitations) I chose the following scenario:
The above scenario is supported by Cisco where mgmt0 interface is used not only for management but also control (L3 capability) purpose. So what was the problem?
Our problem was that when we shutdown all the components i.e. vCenter (5.5U2), VSM (5.2(1)SV3(2.1)), ESXi (5.5U2) and rebooted the physical servers, the ESXi VEM module lost the programming for the port-profiles and VMs connected to Cisco Nexus 1000v profiles did not have network access. What we had to do was moving the VSM mgmt interface to the vStandard Switch. Fortunately it was possible because still we kept the same VLAN in our vSS. Of course, it is not a correct behavior and after many months Cisco wrote the following bug info (CSCuw26327) 2 days ago:
L3 Capability control has to be used only for vmk interfaces.
If we added "L3 capability control" on a veth interface, the system VLAN will be in-effective for that veth.
This should be explicitly mentioned in the configuration guide.
It means that at least now... and to be realistic.. for over the next few months, we need to move mgmt0/control0 from L3 Capability Port-Profile to vSphere Standard Switch or another Nexus 1000v Port-Profile without L3 capability mode. We have to add only vmk1 to the L3 Capability Port-Profile.
I hope that Cisco will update a Cisco Nexus 1000v documentation shortly.
