Best Practices for Virtual Networking: VMware, Inc.

Best Practices for Virtual Networking
Karim Elatov
Technical Support Engineer, GSS
© 2009 VMware Inc. All rights reserved
Agenda
Best Practices for Virtual Networking
Virtual Network Overview
vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks
What’s New in vSphere 5.0
Network Design Considerations
2
Virtual Network Overview - Physical to Virtual
Virtual
Physical
Physical
Physical
Switch
3
Virtual Switch
Physical
Switch
Conventional access, distribution, core design
Design with redundancy for enhanced availability
Under the covers, virtual network same as physical
Access layer implemented as virtual switches
Virtual Switch Options
Virtual Switch
Model
Details
vNetwork Standard
Switch
Host based:
1 or more per
ESX host
- Same as vSwitch in VI3
vNetwork Distributed
Switch
Distributed:
1 or more per
“Datacenter”
- Expanded feature set
- Private VLANs
- Bi-directional traffic shaping
- Network vMotion
- Simplified management
Cisco Nexus 1000V
Distributed:
1 or more per
“Datacenter”
- Cisco Catalyst/Nexus feature set
- Cisco NXOS cli
- Supports LACP
Virtual networking concepts similar with all virtual switches
4
ESX Virtual Switch: Capabilities
 NIC Teaming of Physical NIC(s) [uplink(s)] associated
MAC
address
assigned to
vnic
with vSwitches
VM0
MAC a
 Layer 2 - only forward frames VM <-> VM and VM <-
VM1
MAC b
MAC c
vSwitch
vSwitch
> Uplink; No vSwitch <-> vSwitch or Uplink <-> Uplink
 vSwitch will not create loops affecting Spanning
Tree in the physical network
 Can terminate VLAN trunks (VST mode) or pass
Physical
Switches
5
trunk through to VM (VGT mode)
Distributed Virtual Switch
Standard vSwitch
vNetwork & dvSwitch
vCenter
vCenter
Exist across 2 or more clustered hosts
•Provide similar functionality to vSwitches
•Reside on top of hidden vSwitches
vCenter owns the configuration of the dvSwitch
•Consistent host network configurations
6
Port Groups
 Template for one or more ports with a common
configuration
• VLAN Assignment
• Security
• Traffic Shaping (limit egress traffic from VM)
• Failover & Load Balancing
 Distributed Virtual Port Group (Distributed Virtual Switch)
• Bidirectional traffic shaping (ingress and egress)
• Network VMotion—network port state migrated upon
VMotion
7
NIC Teaming for Availability and Load Sharing
 NIC Teaming aggregates multiple physical uplinks:
VM0
VM1
• Availability—reduce exposure to single points of
failure (NIC, uplink, physical switch)
• Load Sharing—distribute load over multiple
uplinks (according to selected NIC teaming
vSwitch
NIC Team
algorithm)
 Requirements:
• Two or more NICs on same vSwitch
• Teamed NICs must have same VLAN configurations
KB - NIC teaming in ESXi and ESX (1004088)
8
NIC Teaming Options
Name
Algorithm—vmnic
chosen based upon:
Physical Network Considerations
Originating
Virtual Port ID
vnic port
Teamed ports in same L2 domain
(BP: team over two physical
switches)
Source MAC
Address
MAC seen on vnic
Teamed ports in same L2 domain
(BP: team over two physical
switches)
IP Hash*
Hash(SrcIP, DstIP)
Explicit Failover
Order
Highest order uplink
from active list
Teamed ports configured in static
802.3ad “Etherchannel”
- no LACP (Nexus 1000v for LACP)
- Needs MEC to span 2 switches
Teamed ports in same L2 domain
(BP: team over two physical
switches)
Best Practices:
•Originating Virtual PortID for VMs is the default, no extra configuration needed
•IP Hash, ensure that physical switch is properly configured for Etherchannel
*KB - ESX/ESXi host requirements for link aggregation (1001938)
*KB - Sample configuration of EtherChannel / Link aggregation with ESX/ESXi and Cisco/HP switches (1004048)
9
Cisco Nexus 1000v Overview
 Cisco Nexus 1000v is a software switch for vNetwork Distributed
Switches (vDS):
• Virtual Supervisor Module (VSM)
• Virtual Ethernet Module (VEM)
Things to remember:
• Virtual Ethernet Module (VEM)VSM uses external network fabric to
communicate with VEMs
• VSM does not take part in forwarding packets
• VEM does not switch traffic to other VEM without an uplink
10
Cisco Nexus 1000v Modules
Server 1
VM
#1
VM
#2
VM
#3
Server 2
VM
#4
VM
#5
VM
#6
VM
#7
Server 3
VM
#8
VEM
VMware
vSwitch
Nexus VEM
1000V
vDS
VMware
vSwitch
VMware ESX
VMware ESX
VM
#9
VM
#10
VM
#11
VM
#12
VEM
VMware
vSwitch
VMware ESX
Nexus 1000V
VSM
Virtual Supervisor Module (VSM)
• Virtual or Physical appliance running
Cisco OS (supports HA)
• Performs management, monitoring, &
configuration
• Tight integration with VMware Virtual
Center
11
vCenter Server
Virtual Ethernet Module (VEM)
• Enables advanced networking
capability on the hypervisor
• Provides each VM with dedicated
“switch port”
• Collection of VEMs = 1 DVS
Cisco Nexus 1000V Enables:
• Policy Based VM Connectivity
• Mobility of Network & Security
Properties
• Non-Disruptive Operational Model
vSwitch Configurations
Best Practices for Virtual Networking
Virtual Network Overview
vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks
What’s New in vSphere 5.0
Network Design Considerations
12
Cisco ‘show run’ and ‘show tech-support’
Obtain configuration of a Cisco router or switch
•Run commands in priviliged EXEC mode
•’show run’
•‘show tech-support’
The following is a Cisco EtherChannel sample configuration:
interface Port-channel1
switchport
switchport access vlan 100
switchport mode access
no ip address
!
interface GigabitEthernet1/1
switchport
switchport access vlan 100
switchport mode access
no ip address
channel-group 1 mode on
!
KB - Troubleshooting network issues with the Cisco show tech-support command (1015437)
13
Traffic Types on a Virtual Network
Virtual Machine Traffic
• Traffic sourced and received from virtual machine(s)
• Isolate from each other based on service level
vMotion Traffic
• Traffic sent when moving a virtual machine from one ESX host to
another
• Should be isolated
Management Traffic
• Should be isolated from VM traffic (one or two Service Consoles)
• If VMware HA is enabled, includes heartbeats
IP Storage Traffic—NFS and/or iSCSI via vmkernel interface
• Should be isolated from other traffic types
Fault Tolerance (FT) Logging Traffic
• Low latency, high bandwidth
• Should be isolated from other traffic types
How do we maintain traffic isolation without proliferating NICs? VLANs
14
Traffic Types on a Virtual Network, cont.
 Port groups in dedicated VLANs on a management-only virtual
switch.
Service console/VMK Interface
virtual machines
production
virtual switch
vMotion
106
storage
107
production
management
virtual switch
management
vMotion
15
mgmt
108
storage
VLAN Tagging Options
EST – External Switch Tagging
VGT – Virtual Guest Tagging
VST – Virtual Switch Tagging
VLAN
assigned in
Port Group
policy
vSwitch
vSwitch
VLAN Tags
applied in
Guest
vSwitch
PortGroup
set to VLAN
“4095”
Physical Switch
Physical Switch
External Physical
switch applies
VLAN tags
switchport access vlan
16
VLAN Tags
applied in
vSwitch
Physical Switch
VST is the best practice and
most common method
switchport trunk
switchport trunk
DVS Support for Private VLAN (PVLAN)
 Enable users to restrict communications
DMZ network
• Between VMs on the same VLAN or network
Web
email
database
document
application
segment
server
server
server
server the same
Allow
devices to share
IP subnet while server
being Layer 2 Isolated
 PVLAN Types
• Community
Benefits:
• VMs can communicate with VMs on
isolated
isolated
community
PVLAN
•Employ
Larger
subnets
(advantageous to hosting
Community
and Promiscuous
PVLAN
PVLANenvironments)
• Isolated
•Reduce Management Overhead
• VMs can only communicate with VMs on
the Promiscuous
• Promiscuous
• VMs can communicate with all VMs
router in promiscuous PVLAN
KB - Private VLAN (PVLAN) on vNetwork Distributed Switch - Concept Overview (1010691)
17
PVLAN Cost Benefit
W2003EE-32-A
PG
W2003EE-32-B
PG
W2003EE-32-A
PG
W2003EE-32-B
PG
W2003EE-32-A
PG
W2003EE-32-B
PG
W2003EE-32-A
PG
W2003EE-32-B
PG
W2003EE-32-A
PG
W2003EE-32-B
PG
W2003EE-32-A
W2003EE-32-B
PG
PG
Distributed Virtual Switch
TOTAL COST: 12 VLANs (one per VM)
W2003EE-32-A
W2003EE-32-B
W2003EE-32-A
W2003EE-32-B
W2003EE-32-A
W2003EE-32-B
W2003EE-32-A
W2003EE-32-B
W2003EE-32-A
W2003EE-32-B
W2003EE-32-A
W2003EE-32-B
PG (with Isolated PVLAN)
Distributed Virtual Switch
TOTAL COST: 1 PVLAN (over 90% savings…)
18
Link Aggregation
EtherChannel
•Port trunking between two to eight
•Active Fast Ethernet, Gigabit Ethernet, or 10 Gigabit Ethernet ports
EtherChannel vs. 802.3ad
•EtherChannel is Cisco proprietary and 802.3ad is an open standard
Note: ESX implements 802.3ad Static Mode Link Aggregation
LACP (one of the implementations included in IEEE 802.3ad)
•Link Aggregation Control Protocol (LACP)
•Control the bundling of several physical ports into a single logical channel
•Only supported on Nexus 1000v
KB ESX/ESXi host requirements for link aggregation (1001938)
19
Sample Link Aggregation Configuration
Supported switch Aggregation algorithm: IP-SRC-DST
Supported Virtual Switch NIC Teaming mode: IP HASH
KB - Sample configuration of EtherChannel / Link aggregation with ESX/ESXi andCisco/HP switches (1004048)
20
Failover Configurations
Link Status relies solely on the network adapter link state
•Cannot detect configuration errors
•Spanning Tree Blocking
•Incorrect VLAN
•Physical switch cable pulls
Beacon Probing sends out and listens for beacon probes
•Broadcast frames (ethertype 0x05ff)
Beacon Probing Best Practice
•Use at least 3 NICs for triangulation
•If only 2 NICs in team, can’t determine link failed
•Leads to shotgun mode results
KB - What is beacon probing? (1005577)
21
Figure — Using beacons to detect upstream
network connection failures.
Spanning Tree Protocol (STP) Considerations
 Spanning Tree Protocol creates loop-free L2 tree
VM0
topologies in the physical network
• Physical links put in “blocking” state to construct
loop-free tree
VM1
MAC b
MAC a
 ESX vSwitch does not participate in Spanning Tree
vSwitch
and will not create loops with uplinks
• ESX Uplinks will not block, always active (full use
vSwitch drops
BPDUs
Physical
Switches
Blocked link
Switches sending
BPDUs every 2s to
construct and
maintain Spanning
Tree Topology
of all links)
Recommendations for Physical Network Config:
1. Leave Spanning Tree enabled on physical network
and ESX facing ports (i.e. leave it as is!)
2. Use “portfast” or “portfast trunk” on ESX facing
ports (puts ports in forwarding state immediately)
3. Use “bpduguard” to enforce STP boundary
KB - STP may cause temporary loss of network connectivity when a failover or failback event occurs (1003804)
22
Tips & Tricks
Best Practices for Virtual Networking
Virtual Network Overview
vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks
What’s New in vSphere 5.0
Network Design Considerations
23
Tips & Tricks
 Load-Based Teaming (LBT)
• Dynamically balance network load over available uplinks
• Triggered by ingress or egress congestion at 75% mean utilization over a 30
second period
• Configure on DVS via “Route based on physical NIC load”
*LBT is not available on the Standard vSwitch (DVS feature for ingress/egress traffic shaping)
 Network I/O Control (NetIOC)
• DVS software scheduler to isolate and prioritize specific traffic types
contending for bandwidth on the uplinks connecting ESX/ESXi 4.1 hosts with
the physical network.
24
Tips & Tricks
Tip #1 – After physical to virtual migration, the VM MAC address can be
changed for Licensed Applications relying on physical MAC address. (KB
1008473)
Tip #2 – NLB Multicast needs physical switch Manual ARP resolution of NLB
cluster. (KB 1006525)
Tip #3 – Cisco Discovery Protocol (CDP) gives switchport configuration
information useful for troubleshooting (KB 1007069)
Tip #4 - Beacon Probing and IP Hash DO NOT MIX (duplicate packets and port
flapping) (KB 1017612 & KB 1012819)
Tip #5 – Link aggregation is never supported on disparate trunked switches – Use
VSS with MEC. (KB 1001938 & KB 1027731)
25
Tips & Tricks
Using 10GigE
Ingress (into switch)
traffic shaping policy
control on Port Group
Variable/high
b/w 2Gbps+
iSCSI
1-2G
NFS
VMotion
High
b/w
FT
Low b/w
SC
 2x 10GigE common/expected
• 10GigE CNAs or NICs
SC#2
 Possible Deployment Method
• Active/Standby on all Portgroups
vSwitch
FCoE
10GE
10GE
FCoE
Gbps
• VMs “sticky” to one vmnic
10
• SC/vmk ports sticky to other
• Use Ingress Traffic Shaping
FCoE
FCoE Priority Group
bandwidth reservation
(in CNA config utility)
to control traffic type per
Port Group
Best Practice: Ensure Drivers and Firmware are compatible
forPriority
success
• If FCoE, use
Group
bandwidth
(on CNA
vSphere 4.1 supports up to (4) 10GigE NICs; 5.0 supports
(8) reservation
10GigE NICs
utility)
26
Troubleshooting Virtual Networks
Best Practices for Virtual Networking
Virtual Network Overview
vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks
What’s New in vSphere 5.0
Network Design Considerations
27
Network Troubleshooting Tips
 Troubleshoot one component at a time
• Physical NICs
• Virtual Switch
• Virtual NICs
• Physical Network
 Tools for Troubleshooting
• vSphere Client
• Command Line Utilities
• ESXTOP
• Third party tools
• Ping and Traceroute
• Traffic sniffers & Protocol
Analyzers
• Wireshark
• Logs
28
Capturing Traffic
Best Practice: create a new management interface for this purpose
vSwitch must be in Promiscuous Mode (KBs 1004099 & 1002934)
ESXi uses tcpdump-uw (KB 1031186)
29
What’s New in vSphere 5.0
Best Practices for Virtual Networking
Virtual Network Overview
vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks
What’s New in vSphere 5.0
Network Design Considerations
30
What’s New in vSphere 5?
Monitor and troubleshoot virtual infrastructure traffic
• NetFlow V5
• Port mirror (SPAN)
• LLDP (standard based link layer discovery protocol) support simplifies the
network configuration and management in non-Cisco switch environment.
Enhancements to the network I/O control (NIOC)
• Ability to create User-defined resource pool
• Support for vSphere replication traffic type; a new system traffic type that
carries replication traffic from one host to another.
• Support for IEEE 802.1p tagging
What’s New in VMware vSphere 5.0 Networking Technical Whitepaper
31
Network Design Considerations
Best Practices for Virtual Networking
Virtual Network Overview
vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks
What’s New in vSphere 5.0
Network Design Considerations
32
Network Design Considerations
How do you design the virtual network for
performance and availability but maintain isolation
between the various traffic types
(e.g. VM traffic, VMotion, and Management)?
• Starting point depends on:
• Number of available physical ports on server
• Required traffic types
• 2 NIC minimum for availability, 4+ NICs
per server preferred
• 802.1Q VLAN trunking highly recommended for logical scaling
(particularly with low NIC port servers)
• Examples are meant as guidance and do not represent strict
requirements in terms of design
• Understand your requirements and resultant traffic types and
design accordingly
33
Example 1: Blade Server with 2 NIC Ports
 Candidate Design:
SC
vmkernel
• Team both NIC ports
• Create one virtual switch
Portgroup3
VLAN 30
Portgroup1
VLAN 10
Portgroup2
VLAN 20
vSwitch
• Create three port groups:
• Use Active/Standby policy
for each portgroup
vmnic0
vmnic1
VLAN Trunks
(VLANs 10, 20, 30)
• Portgroup1: Service Console (SC)
• Portgroup2: VMotion
• Portgroup3: VM traffic
• Use VLAN trunking
Active
Standby
Note: Team over dvUplinks with vDS
34
• Trunk VLANs 10, 20,
30 on each uplink
Example 2: Server with 4 NIC Ports
 Candidate Design:
• Create two virtual switches
Portgroup4
VLAN 40
Portgroup3
VLAN 30
SC
Portgroup1
VLAN 10
Portgroup2
VLAN 20
vSwitch1
vmnic0
• Team two NICs to each vSwitch
vmkernel
• vSwitch0 (use active/standby
vSwitch0
vmnic2
vmnic1
vmnic3
for each portgroup):
• Portgroup1: Service Console (SC)
• Portgroup2: VMotion
• vSwitch1 (use Originating Virtual
VLANs
30, 40
VLANs
10, 20
PortID)
• Portgroup3: VM traffic #1
• Portgroup4: VM traffic #2
Active
Standby
Note: Team over dvUplinks with vDS
• Use VLAN trunking
• vmnic1 and vmnic3: Trunk VLANs 10, 20
• vmnic0 and vmnic2: Trunk VLANs 30, 40
35
Example 3: Server with 4 NIC Ports (Slight Variation)
 Candidate Design:
• Create one virtual switch
Portgroup4
VLAN 40
Portgroup3
VLAN 30
SC
vmkernel
Portgroup1
VLAN 10
Portgroup2
VLAN 20
• Create two NIC teams
• vSwitch0 (use active/standby
for portgroups 1 & 2):
• Portgroup1: Service Console (SC)
vSwitch0
• Portgroup2: Vmotion
vmnic0
vmnic2
vmnic1
vmnic3
• Use Originating Virtual PortID
for Portgroups 3 & 4
VLANs
30, 40
VLANs
10, 20
• Portgroup3: VM traffic #1
• Portgroup4: VM traffic #2
Active
Standby
Note: Team over dvUplinks with vDS
• Use VLAN trunking
• vmnic1 and vmnic3: Trunk VLANs 10, 20
• vmnic0 and vmnic2: Trunk VLANs 30, 40
36
Questions
37