Dell PowerEdge FX2 – I/O Aggregator – FCoE (FSB

Transcription

Dell PowerEdge FX2 – I/O Aggregator –
FCoE (FSB) Deployment Guide
Deployment Guide for the FN410S IOA configured for FSB (FIP Snooping Bridge)
FCoE deployment environments
Dell Networking Solutions Engineering
December 2014
A Dell Deployment and Configuration Guide
Revisions
Date
Description
Authors
Decemeber 2014
Initial release
Curtis Bunch, Neal Beard, Mike Matthews, and
Kevin Locklear
©2014 Dell Inc., All rights reserved.
Except as stated below, no part of this document may be reproduced, distributed or transmitted in any form or by any
means, without express permission of Dell.
You may distribute this document within your company or organization only, without alteration of its contents.
THIS DOCUMENT IS PROVIDED “AS-IS”, AND WITHOUT ANY WARRANTY, EXPRESS OR IMPLIED. IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE SPECIFICALLY DISCLAIMED.
PRODUCT WARRANTIES APPLICABLE TO THE DELL PRODUCTS DESCRIBED IN THIS DOCUMENT MAY BE FOUND
AT: http://www.dell.com/learn/us/en/19/terms-of-sale-commercial-and-public-sector Performance of network
reference architectures discussed in this document may vary with differing deployment conditions, network loads, and
the like. Third party products may be included in reference architectures for the convenience of the reader. Inclusion
of such third party products does not necessarily constitute Dell’s recommendation of those products. Please consult
your Dell representative for additional information.
Trademarks used in this text:
Dell™, the Dell logo, Dell Networking, Dell Boomi™, Dell Precision™ ,OptiPlex™, Latitude™, PowerEdge™,
PowerVault™, PowerConnect™, OpenManage™, EqualLogic™, Compellent™, KACE™, FlexAddress™, Force10™ and
Vostro™ are trademarks of Dell Inc. Other Dell trademarks may be used in this document. Cisco Nexus®, Cisco
MDS®, Cisco NX-0S®, and other Cisco Catalyst® are registered trademarks of Cisco System Inc. EMC VNX®, and
EMC Unisphere® are registered trademarks of EMC Corporation. Intel®, Pentium®, Xeon®, Core® and Celeron® are
registered trademarks of Intel Corporation in the U.S. and other countries. AMD®is a registered trademark and AMD
Opteron™, AMD Phenom™ and AMD Sempron™ are trademarks of Advanced Micro Devices, Inc. Microsoft®,
Windows®, Windows Server®, Internet Explorer®, MS-DOS®, Windows Vista® and Active Directory® are either
trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Red Hat®
and Red Hat® Enterprise Linux® are registered trademarks of Red Hat, Inc. in the United States and/or other countries.
Novell® and SUSE® are registered trademarks of Novell Inc. in the United States and other countries. Oracle® is a
registered trademark of Oracle Corporation and/or its affiliates. Citrix®, Xen®, XenServer® and XenMotion® are either
registered trademarks or trademarks of Citrix Systems, Inc. in the United States and/or other countries. VMware®,
Virtual SMP®, vMotion®, vCenter® and vSphere® are registered trademarks or trademarks of VMware, Inc. in the
United States or other countries. IBM® is a registered trademark of International Business Machines Corporation.
Broadcom® and NetXtreme® are registered trademarks of Broadcom Corporation. Qlogic is a registered trademark of
QLogic Corporation. Other trademarks and trade names may be used in this document to refer to either the entities
claiming the marks and/or names or their products and are the property of their respective owners. Dell disclaims
proprietary interest in the marks and names of others.
2
Dell PowerEdge FX2 – I/O Aggregator – FCoE (FSB) Deployment Guide | Version 1.0
Table of contents
Revisions ..................................................................................................................................................................................................2
Executive Summary .............................................................................................................................................................................. 6
1
2
Introduction ..................................................................................................................................................................................... 7
1.1
Audience................................................................................................................................................................................ 7
1.2
Objectives ............................................................................................................................................................................ 8
1.3
Typographical Conventions ............................................................................................................................................. 8
Solution Overview ......................................................................................................................................................................... 9
2.1
PowerEdge FX2 Blade Server Chassis ............................................................................................................................ 9
2.1.1 Enabling FlexAddressing ................................................................................................................................................... 9
2.2
PowerEdge FN410S IOA ................................................................................................................................................... 9
2.2.1 IOA FIP-Snooping ............................................................................................................................................................ 10
2.2.2 IOA Port Mapping ............................................................................................................................................................. 10
2.2.3 IOA Operational Modes ................................................................................................................................................... 13
2.2.4 Factory Default Settings ................................................................................................................................................... 13
2.3
Fibre Channel Forwarder (FCF) Converged Switches ...............................................................................................14
2.3.1 Dell Networking S5000 ....................................................................................................................................................14
2.3.2 Cisco Nexus 5000 Series Switch ...................................................................................................................................14
2.3.3 Microsoft Windows Server 2012 R2 ..............................................................................................................................14
2.4
Configuration Prerequisites ............................................................................................................................................14
2.4.1 Documenting WWN/MAC Addresses ........................................................................................................................... 15
2.4.2 Planning Virtual SAN and Virtual Fibre Channel ......................................................................................................... 15
2.4.3 Configure the Converged Network Adapters (CNA) .................................................................................................16
2.5
3
4
3
Component Information .................................................................................................................................................16
Environment One – Dell FN410S in Standalone IOA Mode in a Dell S5000 Fabric Mode Environment ................. 17
3.1
Configuring both IOAs in Standalone Mode .............................................................................................................. 18
3.2
Configuring the Dell S5000 ............................................................................................................................................19
3.3
Configure DCB Map Options ......................................................................................................................................... 20
3.4
Configuring S5000 FCoE General Switch Options ....................................................................................................21
3.5
Configure S5000 VLT settings ....................................................................................................................................... 22
3.6
Configure S5000 Upstream Port-Channel................................................................................................................. 23
3.7
Physical Connectivity ...................................................................................................................................................... 24
Environment Two — Dell FN410S in Standalone IOA Mode in a Nexus Fabric Mode Environment ........................ 25
4.1
Configuring Both IOAs in Standalone Mode .............................................................................................................. 26
4.2
Configuring Nexus 5548—Initial Configuration and vPC ........................................................................................ 26
5
6
4.3
Configuring Nexus FCoE General Switch Options ................................................................................................... 29
4.4
Configuring Nexus Links to the IOA Including Essential FCoE Settings............................................................... 30
4.5
Configuring Cisco Nexus 5548UP Upstream Port Channel ................................................................................... 32
4.6
Environment Three — Dell FN410S in Programmable MUX IOA Mode in a Nexus Fabric Mode Environment ..... 34
5.1
Configuring the IOAs for PMUX Mode ........................................................................................................................ 35
5.2
Enabling FIP Snooping, Configuring the FCoE VLAN and Server-Facing Ports ................................................. 35
5.3
Configuring Upstream Port Channels and Membership Ports for the IOAs ....................................................... 37
5.4
5.5
5.6
Configuring Nexus links to the IOA including essential FCoE Settings .................................................................41
5.7
Configuring Cisco Nexus 5548UP Upstream Port-Channel .................................................................................. 43
5.8
Environment Four — Advanced Configuration using VLT in a FSB FCoE Environment with a Dell S5000 ............ 45
6.1
Configuring the IOAs for PMUX Mode ........................................................................................................................ 47
6.2
Enable FIP Snooping, Configure the FCoE VLAN and Server-Facing Ports ........................................................ 48
6.3
Configure Downstream and Upstream Ports for FIP Snooping............................................................................. 49
6.4
Configure IOAs VLTi Peer-Link ..................................................................................................................................... 50
6.5
External Facing Port Configurations ............................................................................................................................ 50
6.6
Factory Default Dell Networking S5000 ..................................................................................................................... 52
6.7
Dell Networking S5000 FCoE General Setup ............................................................................................................ 53
6.8
S5000 FCoE Map and FC Interface Configuration ................................................................................................... 53
6.9
S5000 Fibre Channel Zone Configuration.................................................................................................................. 54
6.10 Dell S5000 VLT Configuration....................................................................................................................................... 55
6.11 Configure S5000 Downstream and Upstream Port-Channels .............................................................................. 56
6.12 Physical Connectivity ...................................................................................................................................................... 58
7
4
Environment Five — Advanced Configuration using VLT in a FSB FCoE Environment with a Cisco Nexus ........... 59
7.1
Configuring the IOAs for PMUX Mode .........................................................................................................................61
7.2
Enabling FIP Snooping, Configuring the FCoE VLAN and Server-Facing Ports ..................................................61
7.3
Configuring IOAs FCoE Ports ........................................................................................................................................ 63
7.4
Configuring IOAs VLTi Peer-Link .................................................................................................................................. 64
7.5
External Facing Port Configurations ............................................................................................................................ 65
7.6
7.7
Configuring Cisco Nexus 5548UP vPC Member Links ............................................................................................ 68
7.8
7.9
Configuring Cisco Nexus 5548UP FCoE Links .......................................................................................................... 70
7.10 Physical Connectivity ....................................................................................................................................................... 71
8
Validation....................................................................................................................................................................................... 72
8.1
Validation — FN410S IOA (FSB) ..................................................................................................................................... 72
8.2
Validation — Cisco Nexus 5548UP ............................................................................................................................... 83
8.3
Validation — Dell S5000 ................................................................................................................................................. 86
A
Basic Terminology....................................................................................................................................................................... 88
B
Additional Resources and References .................................................................................................................................... 90
C
Attachments ..................................................................................................................................................................................91
Support and Feedback ........................................................................................................................................................................91
5
Executive Summary
Converging Ethernet and Storage networks is key to cutting costs, wiring and complexity in the growing
Enterprise infrastructure. The new Dell PowerEdge FX2 chassis, in conjunction with the PowerEdge
FN410S I/O Aggregator (IOA) and blade servers can be used to build the ideal Fibre Channel over Ethernet
(FCoE) solution for today’s converged storage environment. Instead of having separate fabric switches
providing Ethernet and Fibre Channel (FC) connectivity, this converged environment brings the network
devices together through one FCoE environment.
By default the PowerEdge FN410S IOA, is configured to enable the FIP (FCoE Initialization Protocol)
Snooping Bridge (FSB) protocol, and will transport converged FCoE to Top of Rack (ToR) or End of Row
(EoR) converged infrastructure switches. In this guide, Dell Networking S5000 and Cisco Nexus 5000 Fibre
Channel Forwarder (FCF) converged switches are used. Other vendors’ converged switches can be used
as long as they provide all essential Fibre Channel fabric services. An essential element of the high
availability component of the FCOE configuration is the Multi-path I/O (MPIO) configuration (Figure 1),
which allows for quick failover redundancy in the event a path becomes unavailable.
Fibre Channel
Infrastructure
Converged Infrastructure
Ethernet
Infrastructure
Active PATH
Secondary PATH
FIP Snooping Bridge (FSB) Capabilities of the FX2 – IOA, with MPIO Providing a Failover Path
6
1
Introduction
This deployment guide provides the necessary configuration details to install the FN410S IOA (Figure 2) in
an FCoE (FIP Snooping Bridge) topology. This document will focus on a select number of the multiple
network configurations where FCoE topologies exist.
PowerEdge FX2 Chassis with a FN410S IOA
Note: Quick-step pamphlets, which include only the steps necessary to configure each environment
have been attached to this guide. Attachments are also included, which provide the steps necessary to
configure the Broadcom 57810-K and Emulex OCm14102 CNAs for FCoE, and instructions for enabling
FlexAddressing on the FX2. For more information, please refer to Appendix C.
1.1
Audience
The intended audiences for this guide are network administrators and engineers who have a thorough
working knowledge of network connectivity using fiber optic and copper cabling, IP addressing, and
configuring switches via CLI. Specifically, the audience should understand and have experience using Link
Aggregation (LAG), port channels and VLANs. A basic knowledge of servers and PCs is required for testing,
validation, and troubleshooting.
7
The audience should have, at a minimum, a level of access permissions to allow programming of switches,
as well as the ability to change IP addresses and security settings of the servers in the provided examples.
It is helpful, but not necessary, to have an understanding of blade servers in a PowerEdge FX2 chassis and
their internal connectivity and port mapping to IOA modules.
1.2
Objectives
In Scope
This configuration guide covers the configuration of FCoE using the FN410S IOA as a FSB in several
environments, which contain either Dell S5000 or Cisco Nexus 55UP switches.
Out of Scope
Outside the scope of these objectives are server configurations, NIC teaming, configuring switch features
other than VLT and vPC, and troubleshooting beyond basic connectivity.
1.3
Typographical Conventions
The command line examples in this document use the following conventions:
8
Monospace Text
Identify CLI examples
Italic Monospace Text
Variables in CLI examples
Bold monospace Text
Identify commands entered at the CLI prompt
2
Solution Overview
This deployment guide presents five environments. In the first two environments, the Dell FN410S IOAs are
configured in Standalone IOA mode acting as FSBs to a pair of ToR switches. In the first environment, the
ToR switches are a pair of Dell Networking S5000 switches and in the second, a pair of Cisco Nexus
5548UPs. The third environment is the same as the second, except the IOAs are in Programmable Mux
(PMUX) mode. The final two environments demonstrate an advanced configuration using VLT in a FSB
environment; pair of Dell Networking S5000 switch are used in Environment 4 and pair of Cisco Nexus
5548UPs are used in Environment 5.
2.1
PowerEdge FX2 Blade Server Chassis
Dell’s FX architecture is a revolutionary approach to shared infrastructure for Enterprise Computing. It
combines the density and innovative agility of hyper-scale computing with the ease of use and efficiency
of advanced management technologies. The FX2 provides a straightforward, modular approach to
infrastructure, and introduces a more practical way to manage the complex IT needs of businesses.
FX2 Enclosure with two PowerEdge FC620 Servers
2.1.1
Enabling FlexAddressing
The FX2 supports the FlexAddressing feature, which allows virtual addresses (such as virtual MAC
addresses or virtual WWPNs) to be assigned to the slot a server is occupying in the chassis. These virtual
addresses are substituted for the factory assigned addresses on Ethernet and Fibre Channel devices. In the
event a device, such as a Converged Network Adapters (CNA), must be replaced, zoning and other
configurations on upstream switches do not need to be modified. This feature can be turned on and off in
the CMC. For instructions on enabling FlexAddressing, please refer to the attachment:
Enabling_FlexAddressing.pdf.
2.2
PowerEdge FN410S IOA
The FN410S IOA is a multi-layer, 12x10GbE switch that is installed in a Dell PowerEdge FX2 server chassis.
The FX2 chassis supports two IOAs, which in turn support up to eight internal server connections and four
uplink interfaces each. The IOA acts very similar to a pass-through module providing connectivity between
internal network adapters and external upstream network devices.
The IOA also provides an internal facing out-of-band (OOB) interface, which is a dedicated switch
management port that is segregated from operational network traffic on the switch ports.
9
2.2.1
IOA FIP-Snooping
This deployment guide focuses on deploying the Dell FX2 enclosure in a data center environment using
fiber channel storage via FCoE Initialization Protocol (FIP) Snooping. This provides the proper architecture
so that Data Center Bridging (DCB) values including Quality of Service (QoS), Access Control Lists (ACLs),
Priority Flow Control (PFC), Enhanced Transmission Selection (ETS), etc., are properly configured for FCoE
to function optimally. The FN410S can be described as a FIP snooping bridge (FSB) when used in this
manner.
2.2.2
IOA Port Mapping
The ports on the FN410S IOA are numbered from left to right on the switch. Ports 1 to 8 are internal
server-facing ports, and ports 9 to 12 are external ports.
2.2.2.1
IOA Port Mapping Half-Height Slots
Dual Port CNAs — Half-Height Slots
In half-height slots with dual port CNAs installed in the server blades, the servers map to a single port on
each of the two IOAs. Figure 4 and Table 1 present the port mapping for half-height slots with dual port
CNAs.
Note: Ports 2, 4, 6 and 8 are not used when using half-height blades with dual port adapters.
Slot 1
Slot 2
Slot 3
Slot 4
FN410S IOA A1 (Top)
Internal Ports
1
2
3
4
5
6
7
8
FN410S IOA A2 (Bottom)
Internal Ports
1
2
3
4
5
6
7
8
IOA Port Mapping Half-Height Slots with Dual Port CNAs
10
Table 1
Half-Height Slots with Dual Port CNAs
Slot
IOA A1 Port Numbers
(Top)
IOA A2 Port Numbers
(Bottom)
1
1
1
2
3
3
3
5
5
4
7
7
Quad Port CNAs — Half-Height Slots
In half-height slots with quad port CNAs installed in the server blades, the slots map to two ports on each
IOA. Figure 5 and Table 2 present the port mapping for half-height slots with quad port CNAs.
Slot 1
Slot 2
Slot 3
Slot 4
FN410S IOA A1 (Top)
Internal Ports
1
2
3
4
5
6
7
8
Internal Ports
1
2
3
4
5
6
7
8
Half-Height Slots with Quad Port CNAs
Table 2
11
Half-Height Slots with Quad Port CNAs
Slot
IOA A1 Port Numbers
(Top)
IOA A2 Port Numbers
(Bottom)
1
1,2
1,2
2
3,4
3,4
3
5,6
5,6
4
7,8
7,8
2.2.2.2
IOA Port Mapping – Quarter-Height Slots
In quarter-height slots configured with dual port CNAs, the servers map to a single port on each of the
two IOAs.
The naming convention for quarter-height slots are different from the previous configurations. As shown
in Figure 6, the first row of the quarter-height slots are designated 1a through 1d, and the second row 3a
through 3d. Figure 6 and Table 3 present the port mapping for quarter-height slots with dual port CNAs.
Note: Quad port CNA’s are not available for quarter-height servers.
1a
1b
1c
1d
3a
3b
3c
3d
FN410S IOA A1 (Top)
Internal Ports
1
2
3
4
5
6
7
8
Internal Ports
1
2
3
4
5
6
7
8
Quarter-Height Slots with Dual Port CNAs
Table 3
12
Half-Height Slots with Dual Port CNAs
Slot
IOA A1 Port Numbers
(Top)
IOA A2 Port Numbers
(Bottom)
1a
1
1
1b
2
2
1c
3
3
1d
4
4
3a
5
5
3b
6
6
3c
7
7
3d
8
8
2.2.3
IOA Operational Modes
The IOA supports three operational modes: Standalone (SMUX), VLT and Programmable MUX (PMUX). See
Table 4 for detailed descriptions of each mode. To enable a new operational mode the command
stack-unit 0 iom-mode <IOA_Mode> is issued in configuration mode. After enabling a new
operational mode, the switch must be reloaded.
By default, in Standalone and VLT modes all external ports are added to a single port channel (128) and all
VLANs (1-4094) are tagged on this port channel. Additionally any DCB protocol options are allowed as
well as iSCSI or FCoE settings. In these modes, there is no need for the Spanning Tree protocol (STP), so
STP services and settings cannot be modified through the GUI or CLI. This paper uses SMUX and PMUX
modes for all configurations.
Table 4
IOA Modes and Descriptions
IOA Mode
Description
Standalone mode (SMUX)
This is the factory default mode for the IOA. SMUX is a fully automated,
zero-touch mode that allows VLAN memberships to be defined on the
server-facing ports while all upstream ports are configured in port
channel 128 and cannot be modified.
VLT mode
In this low-touch mode, all configurations except VLAN membership are
automated. Port 9 is dedicated to the VLT interconnect in this mode.
Programmable MUX mode
(PMUX)
PMUX mode provides operational flexibility by allowing the administrator
to create multiple LAGs, configure VLANs on uplinks and to configure
DCB parameters on the server side.
Note: Virtual Link Trunking (VLT) allows physical links between two chassis appear as a single virtual link
to the network core or other switches (Edge, Access or ToR). VLT reduces the role of Spanning Tree
protocols by allowing LAG terminations on two separate distribution or core switches, and by supporting
a loop free topology. VLT provides Layer 2 multipathing, creating redundancy through increased
bandwidth, enabling multiple parallel paths between nodes and load-balancing traffic where alternative
paths exist.
2.2.4
Factory Default Settings
When the IOA mode is changed, the startup configuration is not deleted, this may lead to unexpected
problems. Throughout this guide, instead of simply changing IOA modes the Dell IOA is returned to its
factory defaults prior to configuring each environment.
Note: Returning the switch to its factory default settings places the switch in Standalone mode.
To return the Dell FN410S IOA to its factory default settings, issue the following command from the CLI:
restore factory-defaults stack-unit 0 clear-all. Figure 7 shows the output when restoring
the factory defaults on a Dell FN410S IOA.
13
FN410S_IOA_A1#restore factory-defaults stack-unit 0 clear-all
***********************************************************************
* Warning - Restoring factory defaults will delete the existing
*
* startup-config and all persistent settings (stacking, fanout, etc.)*
* After restoration the unit(s) will be powercycled immediately.
*
* Proceed with caution !
*
***********************************************************************
Proceed with factory settings? Confirm [yes/no]:yes
-- Restore status -Unit
Nvram
Config
-----------------------0
Success
Success
Power-cycling
Power-cycling
Power-cycling
Power-cycling
the
the
the
the
unit(s) in 3 seconds...
unit(s).
Restoring the Factory Defaults on the Dell FN410S IOA
2.3
Fibre Channel Forwarder (FCF) Converged Switches
The complexity involved with FCoE can be quite challenging. Much of the configuration relies on properly
configuring the FCF converged switches, as these switches are the primary configuration point for the rest
of the solution. The FN410S IOAs in the FX2 enclosure will pass DCB information from the ToR switches
down to the server’s CNAs and vice versa. In most cases, the server’s CNA will adjust its values so that they
match the DCB values presented by the ToR switch. This matching allows the FCoE to function optimally.
In this deployment guide, the Dell Networking S5000 and Cisco Nexus 5548UP are the ToR switches that
provide the FCoE/FC services.
2.3.1
Dell Networking S5000
The Dell Networking S5000 ToR switch offers innovative modular, converged networking capabilities. The
switch converges LAN and SAN traffic over a single 10GbE connection to help optimize server and storage
connectivity in data centers deploying separate networks based on different networking protocols.
2.3.2
Cisco Nexus 5000 Series Switch
The Cisco Nexus 5000 series is a family of datacenter Top of Rack ToR switch/fabric network devices.
2.3.3
Microsoft Windows Server 2012 R2
In this guide, the LUNs are presented to Microsoft Windows Server 2012 R2.
2.4
Configuration Prerequisites
This section details the prerequisite steps that should be completed before performing the steps in the
different environments covered in this guide.
14
These prerequisites include documenting the WWN and MAC addresses, which are used to add the CNAs
to Fibre Channel zones and configure the FCF switches, planning and documenting the Virtual SAN and
Virtual Fibre Channels and configuring the CNA to enable its FCoE capabilities.
2.4.1
Documenting WWN/MAC Addresses
To document the MAC/WWN addresses from the CNAs in the blade servers.
1. Login to the Chassis Management Controller (CMC).
2. In the left windowpane, select Server Overview.
3. Once the Server Overview page is populated, select WWW / MAC in the top pane (Figure 8).
The screen will show the MAC addresses for all the servers in the chassis. For our configuration,
we need the MAC address from Slot 1.
4. Scroll down to Slot 1 and set the Filters. In the Filter drop down, select Fabric A and in the next
drop down select FibreChannel.
5. Record the WWPN MAC addresses (Server-Assigned or Chassis-Assigned) highlighted in red. In our
example, the first A1 WWPN MAC address is 20:01:18:A9:9B:D6:7A:83 and the first A2 WWPN MAC
address is 20:01:18:A9:9B:D6:7B:54.
FX2 Chassis Management Controller (CMC) — WWN/MAC Screen (Slot 1)
2.4.2
Planning Virtual SAN and Virtual Fibre Channel
Plan the Virtual SAN (VSAN) and document the VSAN and vFC numbers. Be sure to note which side of the
fabric (A or B) each VSAN and vFC number will be used. When planning, keep the following in mind:
 The selected VSAN numbers should be easy to manage and facilitate troubleshooting.
 The VSAN numbers can range from 1 to 4094.
 SAN A and SAN B require different VSAN numbers.
Table 5 presents the settings used in the environments presented in this Deployment Guide.
15
Table 5
2.4.3
Settings Used in the Environments Presented in this Guide
Device
Type
SAN A
SAN B
Server 1
VSAN Number
2
3
FCoE VLAN
1000
1001
vFC Number
101
201
Binding Method
MAC
MAC
IOA Switch
A1 (Top)
A2 (Bottom)
Physical Port
Te0/1
Te0/1
WWPN
20:01:18:A9:9B:D6:7A:83 20:01:18:A9:9B:D6:7B:54
Configure the Converged Network Adapters (CNA)
The solutions in this document utilize a Broadcom 57810-K CNA. For this document, the Broadcom
Advanced Control Suite (BACS) was used to enable FCoE capabilities on the CNA. The attachment
CNA_Broadcom_57810-K_FCoE.pdf contains instructions on configuring the CNA using both the Ctrl-S
Configuration tool and BACS.
2.5
Component Information
The components used to configure and validate the solutions presented in this guide are listed in Table 6.
Table 6
Component Information
Component
Version
FX2 Chassis Management Controller (CMC)
1.05.X05
PowerEdge FN410S IOA
9.4.0.0
PowerEdge FC620 Blade Server
BIOS Revision 2.2.7
Lifecycle Controller (LC)
1.4.0.128
Broadcom 10GbE 2P 58810-k Mezzanine Card
7.8.16
Storage
EMC VNX 5300
05.32.000.5.008
Network
Cisco Nexus 5548UP
Cables
SFP+ Optical Transceivers (SR) with Fibre Cables
Chassis/Server
16
7.0(2)N1(1) NX-OS
5 meter cables
3
Environment One – Dell FN410S in Standalone IOA Mode
in a Dell S5000 Fabric Mode Environment
In this configuration, a two-port connection is made between two IOAs acting as FSBs and a pair of Dell
Networking S5000s acting as ToR switches (Figure 9 and Figure 10). The S5000 switches utilize a VLT
connection between themselves allowing a multi-chassis connection to the core. In a typical production
environment, most configurations will include several connections between servers, networking and
storage devices.
Environment One — Dell FN410S in Standalone IOA Mode in a Dell S5000 Fabric Mode
Environment
Note: A quickstep pamphlet, Environment-one.pdf, which includes only the steps necessary to configure this
environment has been attached to this guide.
17
PowerEdge FX2
Chassis
Te0/1
NIC1
Internal
Connections
NIC2
FN IOA Slot A1
Converged links
Te0/1
FN IOA Slot A2
Blade Server 1
Blade Server Internal Ports to IOA and Converged Uplinks
3.1
Configuring both IOAs in Standalone Mode
In this environment, the IOAs are used in SMUX mode which means minimal configuration is required for
FIP snooping. In the steps below (Figure 11), the IOAs are restored to factory defaults (SMUX is the factory
default mode) and the hostnames are set.
FN410S_IOA_A1
Factory default the IOA to place the switch in
Standalone mode.
FN410S_IOA_A2
Standalone mode.
restore factory-defaults stack-unit 0
clear-all
restore factory-defaults stack-unit 0
clear-all
Set the hostname for the switch.
configure
hostname FN410S_IOA_A1
end
configure
end
Configuring Standalone Mode on the IOAs
18
3.2
Configuring the Dell S5000
The following steps (Figure 12), enable the FC features needed to use the S5000 as a ToR switch operating
as a FCF and FC services provider, and create the downstream port channel used by the IOAs.
Dell S5000_1
Dell S5000_2
Enable the switch features needed for FCF
mode.
Enable Fibre Channel feature.
Enable full fabric switch mode.
Enable the Fibre Channel port to storage.
Enable the switch features needed for FCF
mode.
Enable Fibre Channel feature.
Enable full fabric switch mode.
Enable the Fibre Channel port to storage.
configure
hostname S5000_1
feature fc
fc switch-mode fabric-services
interface fi 0/0
no shutdown
end
configure
hostname S5000_2
feature fc
interface fi 0/0
no shutdown
end
Configure downstream connectivity.
Add both downstream ports to the IOA to port
channel 128.
Configure port channel 128.
Configure downstream connectivity.
Add both downstream ports to the IOA to port
channel 128.
Configure port channel 128.
configure
interface range te 0/12 - 13
port-channel-protocol lacp
port-channel 128 mode active
no shutdown
exit
interface po128
portmode hybrid
switchport
no shutdown
end
configure
no shutdown
exit
interface po128
portmode hybrid
switchport
no shutdown
end
Enabling FC Features and Configuring the Downstream Port Channel
19
3.3
Configure DCB Map Options
In the steps below (Figure 13), the Data Center Bridging (DCB) map is configured. The mapping specifies
that Ethernet traffic will receive 60% of available bandwidth while FCoE traffic is guaranteed 40%. The DCB
map is then applied to the physical interfaces connecting to the IOAs. As a final step, the FCoE VLAN is
defined on both switches.
Note: For additional information on Priority Flow Control (PFC), Data Center Bridging (DCB) and
Enhanced Transmission Selection (ETS) see the Dell Networking S5000 Switch Configuration Guide on
Dell.com.
Dell S5000_1
Dell S5000_2
Create DCB Map PFC and ETS settings for LAN
and SAN traffic.
Allocate 60% of the bandwidth to Priority
group 0 (LAN).
group 1 (SAN).
Assign FCoE priority 3 to priority group 1.
Create DCB Map PFC and ETS settings for LAN
and SAN traffic.
group 0 (LAN).
group 1 (SAN).
Assign FCoE priority 3 to priority group 1.
configure
dcb-map SAN_DCB_MAP
priority-group 0 bandwidth 60 pfc
off
on
priority-pgid 0 0 0 1 0 0 0 0
end
configure
dcb-map SAN_DCB_MAP
off
on
end
Apply DCB MAP to the downstream physical
interfaces.
Create the VLAN designated for FCoE.
Apply DCB MAP to the downstream physical
interfaces.
Create the VLAN designated for FCoE.
configure
dcb-map SAN_DCB_MAP
exit
interface vlan 1000
end
configure
dcb-map SAN_DCB_MAP
exit
interface vlan 1001
end
S5000 DCB Map Configuration
20
3.4
Configuring S5000 FCoE General Switch Options
The initial work necessary to create the Fibre Channel fabrics is performed in this section (Figure 14). First,
the default FCoE map is modified. A new fabric ID is associated with the FCoE VLAN, which was created
earlier, and a new FC Map value is assigned. Then a Fibre Channel zone is created and the appropriate
MAC addresses are added to the zone.
Note: Since a new FCoE map can be created, modifying the default FCoE map is not required.
Note: To obtain the port WWNs for the FC zone use the command show fc ns fabric to show all the
connected devices.
Finally, the zone is added to a zoneset and activated. Note that the WWN and FIP MAC addresses in the
following commands are just examples and should be changed to reflect the server CNAs being used.
21
Dell S5000_1
Dell S5000_2
Edit the default FCoE map and link to the
interfaces.
Modify the FCoE map, adding FCoE VLAN to
fabric ID.
Apply the FC map priority value.
Bind the FC map to the downstream interface.
Edit the default FCoE map and link to the
interfaces.
Modify the FCoE map, adding FCoE VLAN to
fabric ID.
Apply the FC map priority value.
Bind the FC map to the downstream interface.
configure
fcoe-map default_full_fabric
fabric-id 1000 vlan 1000
fc-map 0efc06
exit
interface port-channel 128
end
configure
fc-map 0efc07
exit
end
Create and apply the zoning configuration.
Create a zone.
Add members to the zone.
Create a zoneset.
Add the zone to the zoneset.
Activate the zoneset.
Create and apply the zoning configuration.
Create a zone.
Add members to the zone.
Create a zoneset.
Add the zone to the zoneset.
Activate the zoneset.
configure
fc zone S5000_1
member 20:01:18:a9:9b:d6:7a:83
member 50:06:01:61:3e:e0:18:70
exit
fc zoneset S5000_Storage_1
member S5000_1
exit
fc-fabric
active-zoneset S5000_Storage_1
end
configure
fc zone S5000_2
member 20:01:18:a9:9b:d6:7b:54
member 50:06:01:68:3e:e0:18:70
exit
member S5000_2
exit
fc-fabric
end
S5000 FCoE configuration
3.5
Configure S5000 VLT settings
In this section, a VLT is created between the two S5000 switches. This allows redundant upstream
connectivity to the core of the network. In this environment, the VLT topology is not connected to the
IOAs but connects to the infrastructure links upstream. In this case, the upstream VLT members connect
to a pair of Dell Networking S6000 switches. For more information on VLT with the S5000, please see the
Dell Networking S5000 Switch Configuration Guide on Dell.com.
Note: The Rapid Spanning Tree Protocol (RSTP) is supported in a VLT domain. It is recommended to
configure any STP options before VLT is configured on the peer switches. RSTP is required for initial loop
prevention during the VLT startup phase.
In the following steps (Figure 15), the ports used to connect the two S5000s are configured in a statically
configured port channel, the port channel is then brought out of administrative shutdown mode. A VLT
domain is then created and configured. In this environment, the same port channel ID is used on both
switches and the backup destination is the management IP of the peer switch. The Primary Priority setting
is used to configure which switch will function as lead in the VLT configuration.
Note: The Unit-ID is required for the VLTi peer-link to be established and cannot be the same on both
switches.
Dell S5000_1
Configure the interfaces for VLT domain.
Add VLTi peer-link interfaces to a port
channel.
Create the port channel.
Dell S5000_2
Configure the interfaces for VLT domain.
Add VLTi peer-link interfaces to a port
channel.
configure
interface range fo 0/48 - 52
no shutdown
exit
no shutdown
end
configure
no shutdown
exit
no shutdown
end
Create the VLT domain.
Set peer link backup interface.
Set backup destination for heartbeat link.
Set primary priority.
Set the Unit ID.
Create the VLT domain.
Set peer link backup interface.
Set backup destination for heartbeat link.
Set primary priority.
Set the Unit ID.
configure
vlt domain 55
peer-link port-channel 55
back-up destination 172.25.189.15
primary-priority 1
unit-id 0
end
configure
vlt domain 55
primary-priority 2
unit-id 1
end
S5000 VLTi Connection Configuration
22
3.6
Configure S5000 Upstream Port-Channel
The final step (Figure 16) on the S5000 switches is to create a port channel group for upstream
connectivity to the core network. In this example, the S5000 switches connect to a pair of Dell networking
S6000 series switches.
Dell S5000_1
Dell S5000_2
Configure upstream to S6000
Enable upstream switch port settings and add
them to a port channel.
Configure the port channel to handle tagged
and untagged traffic, and set it as a L2
interface.
Configure VLAN 8, which will carry LAN traffic,
tagging on appropriate upstream and
downstream port channels.
Configure upstream to S6000
Enable upstream switch port settings and add
them to a port channel.
Configure the port channel to handle tagged
and untagged traffic, and set it as a L2
interface.
Configure VLAN 8, which will carry LAN traffic,
tagging on appropriate upstream and
downstream port channels.
configure
no shutdown
exit
portmode hybrid
switchport
no shutdown
exit
interface vlan 8
tagged po8
tagged po128
end
configure
no shutdown
exit
portmode hybrid
switchport
no shutdown
exit
interface vlan 8
tagged po8
tagged po128
end
S5000 Upstream Port Channel Configuration
23
3.7
Physical Connectivity
At this point, Environment One has been configured and physical connectivity between all six switches can
be established. The following table (Table 7) serves as a guide for cabling the switches.
Refer to the Validation section for steps on verifying proper communication.
Table 7
FN410S IOA
From Switch / Port
24
S5000
To Switch / Port
From Switch / Port
To Switch / Port
☐
IOA-A1 / te0/10
S5000_1 / te0/12
☐
S5000_1 / te0/12
IOA-A1 / te0/10
☐
IOA-A1 / te0/11
S5000_1 / te0/13
☐
S5000_1 / te0/13
IOA-A1 / te0/11
☐
IOA-A2 / te0/10
S5000_2 / te0/12
☐
S5000_2 / te0/12
IOA-A2 / te0/10
☐
IOA-A2 / te0/11
S5000_2 / te0/13
☐
S5000_2 / te0/13
IOA-A2 / te0/11
☐
S5000_1 / te0/16
To S6000-1
☐
S5000_1 / te0/17
To S6000-2
☐
S5000_2 / te0/16
To S6000-1
☐
S5000_2 / te0/17
To S6000-2
☐
S5000_1/ fi0/0
To FC Storage Array1
☐
S5000_2/ fi0/0
☐
S5000_1 / te0/48
S5000_2 / te0/48
☐
S5000_1 / te0/52
S5000_2 / te0/52
☐
S5000_2 / te0/48
S5000_1 / te0/48
☐
S5000_2 / te0/52
S5000_1 / te0/52
4
Environment Two — Dell FN410S in Standalone IOA Mode
in a Nexus Fabric Mode Environment
In this configuration, a two-port connection is made between two IOAs acting as FSBs and a pair of Cisco
Nexus 5548UPs acting as ToR switches (Figure 17 and Figure 18). The Nexus switches utilize a vPC (Virtual
Port Channel) between the switches allowing a multi-chassis connection to the core. In a typical
production environment, most configurations will include several connections between servers,
networking and storage devices.
Ethernet Infrastructure
SAN
SAN
Environment Two– Dell FN410S in Standalone IOA Mode in a Nexus Fabric Mode Environment
Note: A quickstep pamphlet, Environment-Two.pdf, which includes only the steps necessary to configure this
environment has been attached to this guide.
25
PowerEdge FX2
Chassis
Te0/1
NIC1
FN IOA Slot A1
Internal
Connections
NIC2
Converged links
Te0/1
FN IOA Slot A2
Blade Server 1
Blade Server Internal Ports to IOA and Converged Uplinks
4.1
Configuring Both IOAs in Standalone Mode
In this environment, the IOAs are in SMUX mode which means minimal configuration is required for FIP
snooping. In the steps below (Figure 19), the IOAs are restored to factory defaults (SMUX is the factory
default mode) and the hostnames are set.
FN410S_IOA_A1
In this environment, the IOA is used in the
default mode, SMUX mode.
Standalone mode.
FN410S_IOA_A2
In this environment, the IOA is used in the
default mode, SMUX mode.
Standalone mode.
restore factory-defaults stack-unit
0
clear-all
0
clear-all
configure
end
configure
end
Configuring Standalone Mode on the IOAs
4.2
Configuring Nexus 5548—Initial Configuration and vPC
The following steps enables the features necessary to use the Cisco Nexus 5000 series as a FCF and FC
services provider and utilize vPC. In this environment, the vPC topology does not connect to the IOAs but
connects to the infrastructure links upstream. In this case, the upstream vPC members connect to a pair of
Cisco Nexus 7004 switches.
The Cisco proprietary vPC protocol allows a LAG to physically split and terminate into the two Nexus 5000
series switches while logically the LAG, from the point of view of the Nexus 7004, sees a single switch. For
more information on vPC, please refer to the Basic Terminology section.
26
The following steps (Figure 20), highlight configuring a vPC peer-link between the two Nexus 5000 series
switches, this includes enabling the features required for vPC, configuring the management interface and
configuring the vPC domain.
Nexus_5548_1
Enable the required features and
management interface for vPC.
Enable the LACP, vPC and NPIV features.
Nexus_5548_2
feature lacp
feature vpc
feature npiv
feature lacp
feature vpc
feature npiv
Configure the hostname and assign an IP
address to the management interface.
configure
hostname Nexus_5548-1
interface mgmt 0
ip address 172.25.188.60
255.255.0.0
no shutdown
end
configure
interface mgmt 0
ip address 172.25.189.60
255.255.0.0
no shutdown
end
Create a VPC domain.
Assign role priority.
Assign the keepalive management IP address
of 5548-2.
of 5548-1.
configure
vpc domain 1
role priority 1
peer-keepalive dest 172.25.189.60
end
configure
vpc domain 1
role priority 65535
end
vPC Domain and Initial Configuration
27
In the following steps (Figure 21), the port channel and port channel members for the vPC peer-link are
created and configured. On the trunk, all VLANs are allowed except VLANs 1000 and 1001, which are
configured and designated as FCoE VLANs in a later step. These VLANs need to be excluded from the peer
link to ensure proper FC communication.
Nexus_5548-1
Configure the port channel and port channel
members for the vPC peer-link.
Enable the switchport mode trunk.
Assign as a vPC peer-link.
Nexus_5548-2
Enabling switchport mode trunk.
configure
description “vPC Peer-Link”
switchport mode trunk
switchport trunk allowed vlan
except 1000-1001
no shutdown
vpc peer-link
end
configure
except 1000-1001
no shutdown
vpc peer-link
end
Assign interfaces to the port channel and
enable LACP.
enable LACP.
configure
interface ethernet 1/17-18
channel-group 55 mode active
no shutdown
end
configure
no shutdown
end
vPC Peer Link Setup
28
4.3
Configuring Nexus FCoE General Switch Options
The following set of commands (Figure 22) create the fibre fabrics. First, the FCoE VLAN is configured and
the separate FCoE VSANs are defined. Then, a single virtual Fibre Channel (vFC) interface is created and
the appropriate MAC address, which was documented earlier, is inserted. Finally, the vFC is associated with
the appropriate VSAN in the VSAN database.
It should be noted that multiple vFCs can been configured for redundancy. The WWN and FIP MAC
addresses used in these commands are just examples and should be based on the CNAs that are used.
Nexus_5548-1
Create a VLAN for the appropriate VSAN and
add the VSAN to the database.
Create a VLAN
Add a VSAN ID to the VLAN.
Nexus_5548-2
Create a VLAN
configure
interface vlan 1000
description “VSAN 2 VLAN”
fcoe vsan 2
no shutdown
end
configure
interface vlan 1001
fcoe vsan 3
no shutdown
end
Add the VSAN ID to the VSAN database.
Create the VFC interface and bind the FCoE
FIP-MAC address of the CNA recorded earlier.
Enable the designated Fibre Channel interface.
Add all of the interfaces to the VSAN database
and bind them to the VSAN ID created earlier.
Add all of the interfaces to the VSAN database
configure
vsan database
vsan 2
exit
interface vfc101
bind mac-address 5CF9.DD16.EF07
no shutdown
exit
interface fc2/1
no shutdown
exit
vsan database
vsan 2 interface vfc101
vsan 2 interface fc2/1
end
configure
vsan database
vsan 3
exit
interface vfc101
bind mac-address 5CF9.DD16.F010
no shutdown
exit
interface fc2/1
no shutdown
exit
vsan database
end
Nexus 5548UP FCoE Configuration
29
4.4
Configuring Nexus Links to the IOA Including Essential FCoE
Settings
In this section, the port channel that connects the 5548 switches and the IOAs is created and the
interfaces are associated with the port channel (Figure 23). The default VLAN for this trunk is set to 20. This
is not required but considered a best practice when deploying switches into production environments.
Finally, a zone name is created, tagged to a zone set, and the zone set is activated.
Note: Multiple VLANs are allowed through the port channel. VLANs 5 and 8 serve as examples and do not
necessarily reflect a production environment.
30
Nexus_5548-1
Nexus_5548-1
Configure the downstream port channels and
interface members.
Configure the links, add a description and the
port channel, of which they will be a part. The
rest of the configuration is done on the port
channel.
interface members.
rest of the configuration is done on the port
channel.
configure
description portchannel_eth1+2_to_IOA
switchport trunk native vlan 20
5,8,1000
end
configure
5,8,1001
end
Enable the two interfaces as trunks.
Assign the two interfaces facing the IOA to the
port channel.
port channel.
configure
interface Ethernet1/1-2
description
Downlink_to_FN4010S_IOA_A1
end
configure
description
end
Create the zone name and add the interfaces.
Create a zone set; place the zone name inside
of it.
Activate the zone set.
Create a zone set; place the zone name inside
of it.
configure
zone name zone1SAN_A vsan 2
member pwwn 20:01:18:A9:9B:D6:7A:83
member interface fc2/1
exit
zoneset name set1SAN_A vsan 2
member zone1SAN_A
exit
zoneset activate name set1SAN_A
vsan 2
end
configure
zone name zone1SAN_B vsan 3
member pwwn 20:01:18:A9:9B:D6:7B:54
exit
zoneset name set1SAN_B vsan 3
member zone1SAN_B
exit
zoneset activate name set1SAN_B
vsan 3
end
Nexus 5548UP Downstream Port Channel and FC Zone set Configuration
31
4.5
Configuring Cisco Nexus 5548UP Upstream Port Channel
The last step on the Nexus switches is to create a port channel group for upstream connectivity to the
core network (Figure 24). In this example, the Nexus 5548 connects to a pair of Cisco Nexus 7000 series
switches and all non-FCoE designated VLANs are allowed to transverse this trunk.
Note: When the IOAs are in Standalone mode, all VLAN IDs are passed to the Nexus switches. Typically,
the trunk connecting to the IOAs would be filtered to accept only the required VLANs.
Nexus_5548-1
Configure the upstream port channels and
interface members.
Set the description and the channel group for
the uplink ports. The rest of the configuration
is done on the port channel
Nexus_5548-2
interface members.
is done on the port channel
configure
description Ethernet_uplink_to_7k
no shutdown
end
configure
no shutdown
end
Assign the appropriate interfaces to the port
channel that was just created.
configure
description portchannel_eth9+10_to_7k
switchport trunk allowed vlan 5,8
no shutdown
end
configure
no shutdown
end
Nexus 5548UP Upstream Port Channel Configuration
32
4.6
At this point, Environment Two is configured and physical connectivity between all six switches can be
established. The following table (Table 8) serves as a guide for cabling the switches.
Table 8
FN410S IOA
From Switch / Port
33
Nexus 5548UP
To Switch / Port
From Switch / Port
To Switch / Port
☐
IOA-A1 / te0/9
5548-1 / eth1/1
☐
5548-1 / fc2/1
☐
IOA-A1 / te0/10
5548-1 / eth1/2
☐
5548-2 / fc2/1
☐
IOA-A2 / te0/9
5548-2 / eth1/1
☐
5548-1 / te0/17
5548-2 / te0/17
☐
IOA-A2 / te0/10
5548-2 / eth1/2
☐
5548-1 / te0/18
5548-2 / te0/17
☐
5548-1 / eth1/9
To Nexus 7000-1 / te0/1
☐
5548-1 / eth1/10
To Nexus 7000-2 / te0/1
☐
5548-2 / eth1/9
To Nexus 7000-1 / te0/2
☐
5548-2 / eth1/10
To Nexus 7000-2 / te0/2
5
Environment Three — Dell FN410S in Programmable MUX
IOA Mode in a Nexus Fabric Mode Environment
In this configuration (Figure 25), a two-port connection is made between two IOAs acting as FSBs to a pair
of Cisco Nexus 5548UP acting as ToR switches. Both IOAs are configured in PMUX mode allowing full
control over the configuration. The Nexus switches are configured to use vPC between the switches
allowing a multi-chassis connection to the core.
SAN
SAN
Environment Three – Dell FN410S in Programmable MUX IOA Mode in a Nexus Fabric Mode
Environment
Note: A quickstep pamphlet, Environment-Three.pdf, which includes only the steps necessary to
configure this environment has been attached to this guide.
34
5.1
Configuring the IOAs for PMUX Mode
In this environment, the IOAs are in PMUX mode. To ensure no configuration artifacts remain from an
earlier environment, the switch is set to its factory defaults before setting the IOAs to PMUX mode (Figure
26).
FN410S_IOA_A1
In this environment, the IOA is used in PMUX
mode.
Factory default the IOA.
FN410S_IOA_A2
mode.
Factory default the IOA.
0
clear-all
0
clear-all
Configure the IOA for PMUX mode.
Configure the IOA for PMUX mode.
configure
stack-unit 0 iom-mode programmablemux
end
reload
configure
stack-unit 0 iom-mode programmablemux
end
reload
Programmable MUX Configuration
5.2
Enabling FIP Snooping, Configuring the FCoE VLAN and ServerFacing Ports
In this section, the management interface is configured and the FIP snooping feature is enabled for use by
the IOAs (Figure 27). A default VLAN ID is set. Note that this is not required, since the default VLAN ID 1 can
be used for FIP initialization sessions. The internal, server facing interface is then configured by tagging
traffic for the designated FCoE VLAN. Finally the FCoE VLAN is created and FIP snooping is enabled on the
VLAN.
35
FN410S_IOA_A1
Management configuration
Set the hostname.
Set the IP for management.
Set the default route for management.
Enable LLDP hostname advertisement globally.
FN410S_IOA_A2
Set the hostname.
enable
configure
interface managementethernet 0/0
ip address 172.25.189.29 /16
yes
!
exit
manage route 0.0.0.0/0
172.25.189.254
protocol lldp
advertise management-tlv
managementaddress
system-name
end
enable
configure
ip address 172.25.189.30 /16
yes
!
exit
172.25.189.254
protocol lldp
managementaddress
system-name
end
Turn on the fip-snooping feature.
Enable fip-snooping globally.
Set the default VLAN
Set the default VLAN
configure
feature fip-snooping
fip-snooping enable
default vlan-id 20
end
configure
fip-snooping enable
default vlan-id 20
end
Create the FCoE VLAN.
Enable fip-snooping on the VLAN.
Set FC map to match the FCF switch.
Set FC map to match the FCF switch.
configure
interface vlan 1000
fip-snooping enable
no shut
exit
interface te 0/1
portmode hybrid
switchport
vlan tagged 1000
protocol lldp
management-address
system-name
dcbx port-role auto-downstream
dcbx version auto
no shutdown
end
configure
interface vlan 1001
fip-snooping enable
no shut
exit
interface te 0/1
portmode hybrid
switchport
vlan tagged 1001
protocol lldp
management-address
system-name
dcbx port-role auto-downstream
dcbx version auto
no shutdown
end
FSB Feature Enablement and Setup
36
5.3
Configuring Upstream Port Channels and Membership Ports for
the IOAs
In the steps below (Figure 28), a port channel is created for upstream traffic and the interface is configured
to connect to a FCF capable switch (in this case the Nexus 5548UP). The port-mode is then set to FCF to
anticipate FCoE traffic. In the second section, external facing ports 10 and 11 are added to the port
channel and configured.
FN410S_IOA_A1
FN410S_IOA_A2
Configure the upstream interfaces and port
channel.
Create a port channel for upstream
communication.
Enable FCF port-mode to allow the upstream
switch to initiate FC communication.
Configure the upstream interfaces and port
channel.
Create a port channel for upstream
communication.
Enable FCF port-mode to allow the upstream
switch to initiate FC communication.
configure
portmode hybrid
switchport
fip-snooping port-mode fcf
no shutdown
end
configure
portmode hybrid
switchport
no shutdown
end
Configure upstream interfaces to participate in
the port channel.
Configure upstream interfaces to participate in
the port channel.
configure
interface range te0/10-11
port-channel-protocol LACP
exit
protocol lldp
management-address
system-name
no advertise dcbx-tlv ets-reco
dcbx port-role auto-upstream
no shutdown
end
configure
interface range te0/10-11
port-channel-protocol LACP
exit
protocol lldp
management-address
system-name
no shutdown
end
Upstream Port Channels and Membership Ports
37
5.4
The following steps enable the features necessary to use the Cisco Nexus 5548UP as a FCF and FC
services provider and utilize vPC. In this environment, the Cisco vPC topology does not connect to the
IOAs but connects to the infrastructure links upstream. In this case, the upstream vPC members connect
to a pair of Cisco Nexus 7004 switches.
The Cisco proprietary vPC protocol allows a LAG to physically split and terminate into the two Nexus
5548UP switches, while logically the LAG, from the point of view of the Nexus 7004, sees a single switch.
For more information on vPC, please refer to the Basic Terminology section.
The following steps (Figure 29), highlight configuring a vPC peer-link between the two Nexus 5548UP
switches; this includes enabling the features required for vPC, configuring the management interface and
Nexus_5548-1
Nexus_5548-2
feature lacp
feature vpc
feature npiv
feature lacp
feature vpc
feature npiv
configure
interface mgmt 0
ip address 172.25.188.60
255.255.0.0
no shutdown
end
configure
interface mgmt 0
ip address 172.25.189.60
255.255.0.0
no shutdown
end
Assign keepalive management IP address of
5548-2.
Assign keepalive management IP address of
5548-1.
configure
vpc domain 1
role priority 1
end
configure
vpc domain 1
role priority 65535
end
38
Nexus_5548-1
Nexus_5548-2
configure
except 1000-1001
no shutdown
vpc peer-link
end
configure
Switchport trunk allowed vlan
except 1000-1001
no shutdown
vpc peer-link
end
enable LACP.
enable LACP.
configure
no shutdown
end
configure
no shutdown
end
vPC Peer Link Setup
39
5.5
The following set of commands (Figure 31) create the Fibre fabrics. First, the FCoE VLAN is configured and
the separate FCoE VSANs are defined. Then, a single virtual Fibre Channel (vFC) interface is created and
the appropriate MAC address, which was documented earlier, is inserted. Finally, the vFC is associated with
the appropriate VSAN in the VSAN database.
It should be noted that multiple vFCs can been configured for redundancy. The WWN and FIP MAC
addresses used in these commands are just examples and should be based on the CNAs that are used.
Nexus_5548-1
Create a VLAN
Nexus_5548-2
Create a VLAN
configure
interface vlan 1000
fcoe vsan 2
no shutdown
end
configure
interface vlan 1001
fcoe vsan 3
no shutdown
end
Add all the interfaces to the VSAN database
Add all the interfaces to the VSAN database
configure
vsan database
vsan 2
exit
interface vfc101
no shutdown
exit
interface fc2/1
no shutdown
exit
vsan database
end
configure
vsan database
vsan 3
exit
interface vfc101
no shutdown
exit
interface fc2/1
no shutdown
exit
vsan database
end
40
5.6
Configuring Nexus links to the IOA including essential FCoE
Settings
In this section, the port channel that connects the 5548 switches and the IOAs is created and the
interfaces are associated with the port channel (Figure 32). The default VLAN for this trunk is set to 20. This
is not required but considered a best practice when deploying switches into production environments.
Finally, a zone name is created, tagged to a zone set, and the zone set is activated.
Note: Multiple VLANs are allowed through the port channel. VLANs 5 and 8 serve as examples and do not
necessarily reflect a production environment.
41
Nexus_5548-1
Nexus_5548-2
interface members.
rest of the configuration is done on the portchannel.
interface members
configure
5,8,1000
end
configure
5,8,1001
end
port channel.
port channel.
configure
description
end
configure
description
end
Create a zone set, place the zone name inside
of it.
of it.
configure
exit
member zone1SAN_A
exit
vsan 2
end
configure
exit
member zone1SAN_B
exit
vsan 3
end
Nexus 5548UP Downstream Port Channel and FC Zone set Configuration
42
5.7
Configuring Cisco Nexus 5548UP Upstream Port-Channel
The last step on the Nexus switches is to create a port channel group for upstream connectivity to the
core network (Figure 33). In this example, the Nexus 5548 connects to a pair of Cisco Nexus 7000 series
switches and all non-FCoE designated VLANs are allowed to traverse this trunk.
Note: When the IOAs are in Standalone mode, all VLAN IDs are passed to the Nexus. Typically, the trunk
connecting to the IOAs would be filtered to accept only the required VLANs.
Nexus_5548-1
interface members.
is done on the port channel.
Nexus_5548-2
interface members.
is done on the port channel.
configure
no shutdown
end
configure
no shutdown
end
configure
no shutdown
end
configure
no shutdown
end
Nexus 5548UP Upstream Port Channel Configuration
43
5.8
At this point, Environment 3 has been configured and physical connectivity between all six switches can be
established. The following table (Table 9) serves as a guide for cabling the switches.
Table 9
FN410S IOA
From Switch / Port
44
Nexus 5548UP
To Switch / Port
From Switch / Port
To Switch / Port
☐
IOA-A1 / te0/10
5548-1 / eth1/1
☐
5548-1 / fc2/1
☐
IOA-A1 / te0/11
5548-1 / eth1/2
☐
5548-2 / fc2/1
☐
IOA-A2 / te0/10
5548-2 / eth1/1
☐
5548-1 / te0/17
5548-2 / te0/17
☐
IOA-A2 / te0/11
5548-2 / eth1/2
☐
5548-1 / te0/18
5548-2 / te0/17
☐
5548-1 / eth1/9
To Nexus 7000-1 / te0/1
☐
5548-1 / eth1/10
To Nexus 7000-2 / te0/1
☐
5548-2 / eth1/9
To Nexus 7000-1 / te0/2
☐
5548-2 / eth1/10
To Nexus 7000-2 / te0/2
6
Environment Four — Advanced Configuration using VLT in
a FSB FCoE Environment with a Dell S5000
In this configuration (Figure 34 and Figure 35), the two IOAs acting as FSBs have a VLT connection
upstream and a separate single link for FCoE traffic outside of the VLT. The S5000 switches are configured
to use VLT between the switches allowing a multi-chassis connection to the core.
Note: This configuration is limited by the number of available ports and it should only be used with
proper planning and understanding of these limitations.
FC Storage
Environment 4 — Advanced Configuration using VLT in a FSB FCoE Environment with a Dell
S5000
45
Switch A1 / SAN A
connection
PowerEdge FX2
Chassis
NIC Port 1
Te0/1
NIC Port 2
FN IOA Slot A1
VLTi
Internal
Connections
VLT members
Te0/1
FN IOA Slot A2
Blade Server 1
Switch A2 / SAN B
connection
Environment 4 Connections
Note: A quickstep pamphlet, Environment-4.pdf, which includes only the steps necessary to configure
this environment has been attached to this guide.
Once the configuration is complete, it will be necessary to determine if MPIO is configured. Ideally, the
two ports on the CNA would be connected to the same LUN from the FC/FCoE storage array. In the event
one path goes down the other path would be connected.
The ports labeled as VLT members will be connected to each of the two S5000 ToR switches. The VLT
configuration running on the S5000 will allow the IOAs to appear connected to one logical switch instead
of the two physical switches. The same is true for the S5000s, from these switches the IOAs will appear as
a single logical switch.
46
6.1
36).
FN410S_IOA_A1
mode.
Standalone mode.
FN410S_IOA_A2
mode.
Standalone mode.
0
clear-all
0
clear-all
Once the switch is reloaded, configure for
PMUX mode.
PMUX mode.
configure
stack-unit 0 iom-mode progammablemux
end
reload
configure
end
reload
IOA Management and Features Configuration
47
6.2
Enable FIP Snooping, Configure the FCoE VLAN and Server-Facing
Ports
In this section, the management interface is configured and FIP snooping is enabled for use by the IOAs
(Figure 37). The designated VLAN used to carry FCoE traffic is created and enabled for FIP Snooping.
Additionally, a FC Map ID is associated with the VLAN. The FC Map ID can be set globally with the same
results.
Note: In the configuration below, the spacing and the character y are required to copy and paste the
code into a console and achieve the expected configuration results.
FN410S_IOA_A1
Set the hostname.
FN410S_IOA_A2
Set the hostname.
enable
configure
ip address 172.25.189.29 /16
yes
!
exit
172.25.189.254
protocol lldp
advertise manage manage system-name
end
enable
configure
ip address 172.25.189.30 /16
yes
!
exit
172.25.189.254
protocol lldp
end
configure
fip-snooping enable
end
configure
fip-snooping enable
end
Set the FC map to match the FCF switch.
Set the FC map to match the FCF switch.
configure
interface vlan 1000
fip-snooping enable
no shut
fip-snooping fc-map 0EFC06
y
end
configure
interface vlan 1001
fip-snooping enable
no shut
fip-snooping fc-map 0EFC07
y
end
48
6.3
Configure Downstream and Upstream Ports for FIP Snooping
Next, FIP snooping is configured on the upstream and downstream ports (Figure 38), in this environment
ports te0/12 and te0/1. To enable multiple VLAN tagging, hybrid port mode is required. To apply this
mode, the port must first be removed from L2 mode (switch port). The interfaces are then tagged with the
defined FCoE VLAN, and the LLDP options and DCB settings are configured.
FN410S_IOA_A1
FN410S_IOA_A2
Configure the upstream port for FCoE.
Remove the switchport before enabling the
port as hybrid.
Tag the interface with the FCoE VLAN.
Set the fip-snooping port mode to FCF.
Set the DCB port role.
port as hybrid.
configure
interface te 0/12
no switchport
portmode hybrid
switchport
vlan tagged 1000
protocol lldp
no shut
end
configure
interface te 0/12
no switchport
portmode hybrid
switchport
vlan tagged 1001
protocol lldp
no shut
end
Configure the internal port facing the server.
port as hybrid.
port as hybrid.
configure
interface te0/1
no switchport
portmode hybrid
switchport
vlan tagged 1000
protocol lldp
end
configure
interface te0/1
no switchport
portmode hybrid
switchport
vlan tagged 1001
protocol lldp
end
FIP Snooping Interface Configuration
49
6.4
Configure IOAs VLTi Peer-Link
In this section, the interfaces used for LAN traffic and the VLTi are created (Figure 39). Then the VLTi
peer-link is built. This link connects the two IOAs, allowing VLT to function. A port channel is created even
though only a single interface is added.
FN410S_IOA_A1
FN410S_IOA_A2
Enable VLT and configure the VLTi peer-link.
Create the VLAN for VLT traffic.
Create the VLAN for Ethernet traffic.
Create the port channel interface for the VLT
peer-link.
Create the VLT domain and set the back-up
destination.
Create the port channel interface for the VLT
peer-link.
destination.
configure
interface vlan 55
no shut
exit
interface vlan 8
no shut
exit
interface po55
no shut
exit
vlt domain 55
unit-id 0
end
configure
interface vlan 55
no shut
exit
interface vlan 8
no shut
exit
interface po55
no shut
exit
vlt domain 55
unit-id 1
end
Tag the VLTi port channel with the VLT VLAN.
Add te0/9 to the VLTi port channel.
Add te0/9 to the VLTi port channel
configure
interface po55
vlan tagged 55
exit
interface te0/9
no shut
end
configure
interface po55
vlan tagged 55
exit
interface te0/9
no shut
end
VLT Configuration
6.5
External Facing Port Configurations
In the following steps (Figure 40), the external facing interfaces are configured. Each IOA offers two
upstream links, one to each ToR, allowing Ethernet path redundancy. The two external interfaces are
added to a port channel and the vlt-peer-lag command is used to allow both VLT peer switches to be
aware of the LAG termination. Finally, the downstream interface is tagged with the VLAN designated to
carry LAN traffic in this environment.
50
FN410S_IOA_A1
Enable the upstream facing VLT member
ports.
Add ports to po128.
Tag po128 with the LAN VLAN.
Specify the VLT peer interface.
FN410S_IOA_A2
Enable the upstream facing VLT member
ports.
Add ports to po128.
configure
no shut
exit
interface po128
portmode hybrid
switchport
vlan tagged 8
vlt-peer-lag po128
no shut
end
configure
no shut
exit
interface po128
portmode hybrid
switchport
vlan tagged 8
vlt-peer-lag po128
no shut
end
Tag the server facing interface with the LAN
VLAN.
Save the configuration.
Tag the server facing interface with the LAN
VLAN.
configure
interface te0/1
vlan tagged 8
end
copy run start
configure
interface te0/1
vlan tagged 8
end
copy run start
Upstream Port Channel and VLAN Configuration
51
6.6
Factory Default Dell Networking S5000
In the following steps (Figure 41), the startup configuration file of the S5000 switches is deleted to ensure
consistent deployment. Once this is done, a hostname is set. If SSH or telnet is being used for
management access, remote credentials are set. Then a management IP, which is used for the VLTi
heartbeat link, is set. Finally, the global LLDP features are set.
Dell S5000_1
Dell S5000_2
Factory default settings
Delete the startup-config file to return the
switch to the factory defaults.
Factory default settings
Delete the startup-config file to return the
switch to the factory defaults.
enable
delete flash://startup-config
yes
reload
enable
delete flash://startup-config
yes
reload
Configure the global switch settings.
Create a username for SSH access.
Set the management IP.
Set management route.
Configure LLDP settings.
Configure the global switch settings.
Create a username for SSH access.
Set the management IP.
Set management route.
Configure LLDP settings.
configure
hostname S5000_1
user admin secret 0 Dell1234 priv
15
enable secret 0 Dell1234
ip address 172.25.189.16 /16
172.25.189.254
protocol lldp
end
configure
hostname S5000_2
user admin secret 0 Dell1234 priv
15
enable secret 0 Dell1234
ip address 172.25.189.15 /16
172.25.189.254
protocol lldp
end
Dell Networking S5000 Basic Switch Configuration
52
6.7
Dell Networking S5000 FCoE General Setup
Once the general configuration of the switch is complete, the fabric services features are enabled globally
(Figure 42). Then, the designated FCoE VLAN is created and brought out of the administrative shutdown
state. A DCB map is created. Priority Flow Control (PFC) is disabled for Ethernet and enabled for FCoE.
Ethernet traffic is allocated 60% of available bandwidth, minimum, and FCoE traffic is given 40%,
minimum.
Dell S5000_1
Enable the FC feature and fabric-services.
Bring out of administrative shutdown.
Dell S5000_2
Enable the FC feature and fabric-services.
Bring out of administrative shutdown.
configure
feature fc
exit
interface vlan 1000
no shutdown
end
configure
feature fc
exit
interface vlan 1001
no shutdown
end
Create a DCB map.
Set 0 to 60% bandwidth with PFC turned off.
Set 1 to 40% bandwidth with PFC turned on.
Assign 0 to all priorities.
Assign 1 to priority group 3 (FCoE Default).
Create a DCB map.
Set 0 to 60% with PFC turned off.
Set 1 to 40% with PFC turned on.
Assign 0 to all priorities.
Assign 1 to priority group 3 (FCoE Default).
configure
dcb-map SAN_DCB_MAP
off
on
end
configure
dcb-map SAN_DCB_MAP
off
on
end
S5000 FCoE Features and DCB Map Configuration
6.8
S5000 FCoE Map and FC Interface Configuration
In this section, the default full fabric FCoE map is modified to associate the designated FCoE VLAN with a
fabric ID using the same number (Figure 43). A specific FC map is set that mirrors the map used on the
respective IOAs.
Note: If the FC-MAP values do not match, FCoE frames are dropped. A session between an ENode and a
FCF is established by the switch-bridge only when the FC-MAP value on the FCF matches the FC-MAP
value on the FIP snooping bridge.
Next, the interfaces connecting the storage array are brought out of administrative shutdown. The DCB
map and FCoE map are then applied to the interface carrying FCoE traffic to the IOA. This interface is set
as a RSTP edge port to ensure quicker convergence during switch reloads. The LLDP settings and DCB
port role settings are then set. For more information on LLDP and DCB interface settings, please see the
Dell Networking S5000 Switch Configuration Guide on Dell.com.
53
Dell S5000_1
Dell S5000_2
Modify the default FCoE map.
Associate the FCoE VLAN with a fabric ID.
Assign a FC map ID.
Enable the Fibre Channel interface to storage.
Modify the default FCoE map.
Associate the FCoE VLAN with a fabric ID.
Assign a FC map ID.
Enable the Fibre Channel interface to storage.
configure
fc-map 0efc06
exit
interface fi 0/0
no shutdown
end
configure
fc-map 0efc07
exit
interface fi 0/0
no shutdown
end
Configure the downstream port for FCoE.
Associate a port with the DCB map.
Associate a port with the FCoE map.
Set the port as a STP edge port.
Configure the downstream port for FCoE.
Associate a port with the DCB map.
Associate a port with the FCoE map.
Set the port as a STP edge port.
configure
interface te0/12
dcb-map SAN_DCB_MAP
spanning-tree rstp edge-port
protocol lldp
no shut
end
configure
interface te0/12
dcb-map SAN_DCB_MAP
spanning-tree rstp edge-port
protocol lldp
no shut
end
FCoE Map and Downstream Configuration
6.9
S5000 Fibre Channel Zone Configuration
Using the following commands (Figure 44), a Fibre Channel zone is created and the appropriate MAC
addresses are inserted. Then, the zone is added to a zoneset and activated.
Note: To identify the port PWWNs for the FC zone, the show fc ns fabric command can be used to
show all the connected devices.
54
Dell S5000_1
Create and activate the FC zone.
Assign members to the FC zone.
The show fc ns fabric command can be
used to obtain PWWN addresses.
Assign the zone to a zone set.
Dell S5000_2
Create and activate the FC zone.
Assign members to the FC zone.
The show fc ns fabric command can be
used to obtain PWWN addresses.
Assign the zone to a zone set.
configure
fc zone S5000_1
member 20:01:18:a9:9b:d6:7a:83
member 50:06:01:61:3e:e0:18:70
exit
member S5000_1
exit
fc-fabric
end
configure
fc zone S5000_2
member 20:01:18:a9:9b:d6:7b:54
member 50:06:01:68:3e:e0:18:70
exit
member S5000_2
exit
fc-fabric
end
S5000 Fibre Channel Zone Configuration
At this point, a FIP snooping session should be established between the IOAs and the S5000 switches. The
figure below (Figure 45) shows the console output from both an IOA and the S5000 showing the FIP
Snooping Session being created.
FN410S_IOA_A1#19:53:27: %STKUNIT0-M:CP %FCOE-6-SESSION_UPDATE: FIP Snooping
Session Created in VLAN: 1000 FCOE MAC : 0e:fc:08:01:35:00
S5000_1#20:21:23: %STKUNIT0-M:CP %FCOE-6-SESSION_UPDATE: FIP Snooping Session
Created in VLAN: 1000 FCOE MAC : 0e:fc:08:01:38:00
FIP Snooping Session Console Message from IOA and S5000
6.10
Dell S5000 VLT Configuration
In this section, a VLT is created between the two S5000 switches. This allows redundant upstream
connectivity to the core of the network. In this environment, the VLT topology is not connected to the
IOAs but connects to the core infrastructure upstream. While not shown in this deployment guide, the
upstream VLT members connect to a pair of Dell Networking S6000 switches. For more information on
VLT with the S5000, please see the Dell Networking S5000 Switch Configuration Guide on Dell.com.
Note: The RSTP Spanning Tree protocol is supported in a VLT domain. If Rapid Spanning Tree Protocol
(RSTP) will be included in this configuration, it must be configured before VLT is configured on the peer
switches. RSTP is required for initial loop prevention during the VLT startup phase.
In the following steps (Figure 46), the ports used to connect the two S5000s are configured in a statically
configured port channel and the port channel is brought out of administrative shutdown mode. A VLT
domain is then created and configured. In this environment, the same port channel ID is used on both
switches and the backup destination is the management IP of the peer switch. The Primary Priority setting
is used to configure which switch will function as lead in the VLT configuration.
55
Note: The Unit-ID is required for the VLTi peer-link to be established and cannot be the same on both
switches.
Dell S5000_1
Create the VLTi peer link LAG.
Place fo48 and fo52 into po 55.
Enable po 55.
Tag po 55 with VLAN 55.
Dell S5000_2
Create the VLTi peer link LAG.
Place fo48 and fo52 into po 55.
Enable po 55.
Tag po 55 with VLAN 55.
configure
no shut
exit
no shut
protocol lldp
exit
interface vlan 55
no shut
tagged po55
end
configure
no shut
exit
no shut
protocol lldp
exit
interface vlan 55
no shut
tagged po55
end
Configure the VLT domain.
Set the peer link.
Set the backup destination.
Configure priority and unit ID.
Configure the VLT domain.
Set the peer link.
Set the backup destination.
Configure priority and unit ID.
configure
vlt domain 55
primary-priority 1
unit-id 0
end
configure
vlt domain 55
primary-priority 2
unit-id 1
end
Dell S5000 VLT Configuration
6.11
Configure S5000 Downstream and Upstream Port-Channels
The last steps on the S5000 switches is to create a port channel group for downstream and upstream
connectivity to the IOAs and the core network. In this example, the S5000s connect to a pair of Dell
networking S6000 series switches upstream.
First, the interfaces used for downstream connectivity are added to the appropriate port channel, and the
port channel is configured as a layer 2 interface and capable of multiple VLAN tags. Similar steps are then
repeated on the upstream channel to the S6000. Finally, both the upstream and downstream connections
are tagged for VLAN 8.
56
Dell S5000_1
Create a port channel for downstream to
IOAs.
Place te24 and te25 into po 128.
Enable PO 128.
Dell S5000_2
Create a port channel for downstream to
IOAs.
Place te24 and te25 into po 128.
Enable PO 128.
configure
no shut
exit
interface po128
portmode hybrid
switchport
vlt-peer-lag po128
no shut
end
configure
no shut
exit
interface po128
portmode hybrid
switchport
vlt-peer-lag po128
no shut
end
Create a port channel for upstream to LAN
core.
Place te37 and te38 into po8.
Enable po8 to be be tagged with multiple
VLANs.
Create a port channel for upstream to LAN
core.
Place te37 and te38 into po8.
Enable po8 to be be tagged with multiple
VLANs.
configure
no shut
exit
configure
interface po8
portmode hybrid
switchport
vlt-peer-lag po8
no shut
end
configure
no shut
exit
configure
interface po8
portmode hybrid
switchport
vlt-peer-lag po8
no shut
end
Tag upstream and downstream with VLAN 8.
Save the running configuration.
Tag upstream and downstream with VLAN 8.
Save the running configuration.
configure
interface vlan 8
tagged po8
tagged po128
end
copy running-config startup-config
configure
interface vlan 8
tagged po8
tagged po128
end
copy running-config startup-config
S5000 VLT Member Configuration
57
6.12
At this point, Environment 4 has been configured and physical connectivity between all six switches can
Table 10
FN410S IOA
58
Dell Networking S5000
From Switch / Port
To Switch / Port
From Switch / Port
To Switch / Port
☐
IOA-A1 / te0/9
IOA-A2 / te0/9
☐
S5000_1 / te0/24
IOA-A1 / te0/10
☐
IOA-A1 / te0/10
S5000_1 / te0/24
☐
S5000_1 / te0/25
IOA-A2 / te0/10
☐
IOA-A1 / te0/11
S5000_2 / te0/24
☐
S5000_2 / te0/24
IOA-A1 / te0/11
☐
IOA-A1 / te0/12
S5000_1 / te0/12
☐
S5000_2 / te0/25
IOA-A2 / te0/11
☐
IOA-A2 / te0/9
IOA-A1 / te0/9
☐
S5000_1 / te0/37
To S6000-1
☐
IOA-A2 / te0/10
S5000_1 / te0/25
☐
S5000_1 / te0/38
To S6000-2
☐
IOA-A2 / te0/11
S5000_2 / te0/25
☐
S5000_2 / te0/37
To S6000-1
☐
IOA-A2 / te0/12
S5000_2 / te0/12
☐
S5000_2 / te0/38
To S6000-2
☐
S5000_1/ fi0/0
☐
S5000_2/ fi0/0
☐
S5000_1 / te0/48
S5000_2 / te0/48
☐
S5000_1 / te0/52
S5000_2 / te0/52
☐
S5000_2 / te0/48
S5000_1 / te0/48
☐
S5000_2 / te0/52
S5000_1 / te0/52
7
Environment Five — Advanced Configuration using VLT in
a FSB FCoE Environment with a Cisco Nexus
In this configuration, the two IOAs acting as FSBs have a VLT connection upstream and a separate single
link for FCoE traffic outside of this VLT. The Nexus switches are configured to use vPC between the
switches allowing a multi-chassis connection to the core. Both IOAs are configured to be running in
Programmable MUX mode.
Note: This configuration is limited by the number of available ports and it should only be used with
proper planning and understanding of these limitations.
SAN
FCoE link outside
of VLT and vPC
domains
SAN
FCoE link outside
of VLT and vPC
domains
Environment 5 – Advanced Configuration of IOA using VLT in an FSB FCoE Environment
59
Switch A1 / SAN A
connection
PowerEdge FX2
Chassis
NIC Port 1
Te0/1
NIC Port 2
FN IOA Slot A1
VLTi
Internal
Connections
VLT members
Te0/1
FN IOA Slot A2
Blade Server 1
Switch A2 / SAN B
connection
Environment 5 Connections
Note: A quickstep pamphlet, Environment-5.pdf, which includes only the steps necessary to configure
this environment has been attached to this guide.
Once the configuration is complete, it will be necessary to determine if MPIO is configured. Ideally, the
two ports on the CNA would be connected to the same LUN from the FC/FCoE storage array. In the event
one path goes down the other path would be connected.
60
7.1
50).
FN410S_IOA_A1
mode.
Standalone mode.
FN410S_IOA_A2
In this environment, the IOA is used in
PMUX mode.
Standalone mode.
0
clear-all
0
clear-all
PMUX mode.
PMUX mode.
configure
end
reload
configure
end
reload
7.2
Enabling FIP Snooping, Configuring the FCoE VLAN and ServerFacing Ports
In this section, the management interface is configured and FIP snooping is enabled for use by the IOAs
(Figure 51). In this environment, a default VLAN ID is set, even though this is not required since the default
VLAN ID 1 can be used for FIP initializing sessions. The internal, server facing interface is configured by
tagging traffic for the designated FCoE VLAN. Additional FCoE settings are then configured. Finally the
FCoE VLAN is created and FIP snooping is enabled on the VLAN.
61
FN410S_IOA_A1
Management Configuration
Set the hostname.
FN410S_IOA_A2
Management Configuration
Set the hostname.
enable
configure
ip address 172.25.189.29 /16
yes
!
exit
172.25.189.254
protocol lldp
managementaddress
system-name
end
enable
configure
ip address 172.25.189.30 /16
yes
!
exit
172.25.189.254
protocol lldp
managementaddress
system-name
end
configure
fip-snooping enable
default vlan-id 20
end
configure
fip-snooping enable
default vlan-id 20
end
Set the FC map to match FCF switch.
Set the FC map to match FCF switch.
configure
interface vlan 1000
fip-snooping enable
no shut
end
configure
interface vlan 1000
fip-snooping enable
no shut
end
FSB Feature Enablement and Setup
62
7.3
Configuring IOAs FCoE Ports
Next, the dedicated FCoE ports, both upstream and downstream, are configured (Figure 52). Notice that
the upstream port (te0/12) is set to port-mode FCF as well as tagged with the designated VLAN for FCoE
traffic.
FN410S_IOA_A1
FN410S_IOA_A2
port as hybrid.
port as hybrid.
Set DCB port role.
configure
interface te 0/12
no switchport
portmode hybrid
switchport
vlan tagged 1000
protocol lldp
managementaddress
system-name
no shut
end
configure
interface te 0/12
no switchport
portmode hybrid
switchport
vlan tagged 1001
protocol lldp
managementaddress
system-name
no shut
end
port as hybrid.
Remove switchport before enabling the port
as hybrid.
configure
interface te0/1
no switchport
portmode hybrid
switchport
vlan tagged 1000
protocol lldp
managementaddress
system-name
end
configure
interface te0/1
no switchport
portmode hybrid
switchport
vlan tagged 1001
protocol lldp
managementaddress
system-name
end
FCoE Enabled Upstream and Downstream Port Configuration
63
7.4
Configuring IOAs VLTi Peer-Link
In this section, the interfaces used for LAN traffic and the VLTi are created (Figure 53). Then the VLTi
peer-link is built. This link connects the two IOAs, allowing VLT to function. A port channel is created even
though only a single interface is added.
Note: As stated previously STP must be configured. No version of STP runs on the IOAs and those steps
are not required when dealing with a VLTi peer link and IOAs.
FN410S_IOA_A1
FN410S_IOA_A2
Create a port channel interface for the VLT
peer link.
Create the VLT domain and the set back-up
destination.
Create a port channel interface for the VLT
peer link.
destination.
configure
interface vlan 55
no shut
exit
interface vlan 8
no shut
exit
interface po55
no shut
exit
vlt domain 55
unit-id 0
end
configure
interface vlan 55
no shut
exit
interface vlan 8
no shut
exit
interface po55
no shut
exit
vlt domain 55
unit-id 1
end
Add te0/9 to the VLTi port channel.
Add te0/9 to VLTi port channel.
configure
interface po55
vlan tagged 55
exit
interface te0/9
no shut
end
configure
interface po55
vlan tagged 55
exit
interface te0/9
no shut
end
VLT Configuration
64
7.5
External Facing Port Configurations
In this step (Figure 54), the external facing interfaces are configured. Each IOA offers two upstream links,
one to each ToR, allowing Ethernet path redundancy. These two external interfaces are added to a port
channel. The vlt-peer-lag command is used to allow both VLT peer switches to be aware of the LAG
termination. Finally, the designated VLAN for LAN traffic is tagged on the downstream, server-facing,
interface.
FN410S_IOA_A1
Enable the VLT member ports facing
upstream.
Add ports to po128.
FN410S_IOA_A2
Enable the VLT member ports facing
upstream.
Add ports to po128.
configure
no shut
exit
interface po128
portmode hybrid
switchport
vlan tagged 8
vlt-peer-lag po128
no shut
end
configure
no shut
exit
interface po128
portmode hybrid
switchport
vlan tagged 8
vlt-peer-lag po128
no shut
end
Tag the server facing interface with LAN
VLAN.
Tag the server facing interface with LAN
VLAN.
configure
interface te0/1
vlan tagged 8
end
copy run start
configure
interface te0/1
vlan tagged 8
end
copy run start
External Facing VLT Enabled Port Channel
65
7.6
The following steps enable the features necessary to use the Cisco Nexus 5548UP as a Fibre Channel
Forwarder (FCF) and FC services provider. In this environment, the Cisco vPC topology does not connect
to the IOAs but connects to the infrastructure links upstream. In this case, the upstream vPC members
connect to a pair of Cisco Nexus 7004 switches.
The Cisco proprietary vPC protocol allows a LAG to physically split and terminate into the two Nexus
5548UP switches, while logically the LAG, from the point of view of the Nexus 7004, sees a single switch.
For more information on vPC, please refer to the Basic Terminology section.
The following steps (Figure 55), highlight configuring a vPC peer-link between the two Nexus 5548UP
switches; this includes enabling the features required for vPC, configuring the management interface and
Nexus_5548-1
Nexus_5548-2
feature lacp
feature vpc
feature npiv
feature lacp
feature vpc
feature npiv
Configure the hostname and an assign IP to
management.
Configure the hostname and assign an IP to
management.
configure
interface mgmt 0
ip address 172.25.188.60
255.255.0.0
no shutdown
end
configure
interface mgmt 0
ip address 172.25.189.60
255.255.0.0
no shutdown
end
of 5548-2.
of 5548-1.
configure
vpc domain 1
role priority 1
end
configure
vpc domain 1
role priority 65535
end
66
Nexus_5548-1
Configure port channel and port channel
Create a port channel.
Enable switchport mode trunk.
Assign as a vpc peer-link.
Nexus_5548-2
Configure port channel and port channel
Create a port channel.
Assign as a vpc peer-link.
configure
except 1000-1001
no shutdown
vpc peer-link
end
configure
Switchport trunk allowed vlan
except 1000-1001
no shutdown
vpc peer-link
end
Assign the interfaces to the port channel and
enable LACP.
Assign the interfaces to the port channel and
enable LACP.
configure
no shutdown
end
configure
no shutdown
end
vPC Peer Link Setup
67
7.7
Configuring Cisco Nexus 5548UP vPC Member Links
In these steps (Figure 57), the port channel and port channel members are configured on the Cisco Nexus
5548P.
Nexus_5548-1
members for IOA connectivity.
Specify the vPC ID.
Nexus_5548-2
members for IOA connectivity.
Specify the vPC ID.
configure
description “Port Channel to IOA”
switchport trunk allowed vlan all
vpc 1
no shutdown
end
configure
description “Port Channel to IOA”
Switchport trunk allowed vlan all
vpc 1
no shutdown
end
enable LACP.
enable LACP.
configure
description “IOA Link Member”
no shutdown
end
configure
description “IOA Link Member”
no shutdown
end
Configure vPC Member Links
68
7.8
The following set of commands (Figure 58), create the fibre fabrics. First, the FCoE VLAN is configured,
and the separate FCoE VLANs are defined.
Then, a single virtual Fibre Channel (vFC) interface is created and the appropriate MAC address, which was
documented earlier, is inserted. Finally, the vFC is associated with the appropriate VSAN in the VSAN
database.
It should be noted that multiple vFCs can been configured for redundancy. The WWN and FIP MAC in this
table are just examples and should be based on the CNAs that are used.
Nexus_5548-1
Create a VLAN.
Nexus_5548-2
Create a VLAN.
configure
vlan 1000
fcoe vsan 2
no shutdown
end
configure
vlan 1001
fcoe vsan 3
no shutdown
end
Add the VFC and FC interfaces to the VSAN
database and bind them to the VSAN ID
created earlier.
Add the VFC and FC interfaces to the VSAN
database and bind them to the VSAN ID
created earlier.
configure
vsan database
vsan 2
exit
interface vfc101
no shutdown
exit
interface fc2/1
no shutdown
exit
vsan database
end
configure
vsan database
vsan 3
exit
interface vfc101
no shutdown
exit
interface fc2/1
no shutdown
exit
vsan database
end
69
7.9
Configuring Cisco Nexus 5548UP FCoE Links
At the time of this writing, the FCoE protocol cannot utilize these redundant paths due to the
non-deterministic nature of how path selection is made through a VLT and vPC domain. To address this
issue, dedicated ports on the IOAs are used for FCoE traffic and are connected to the upstream Nexus
switches acting as a ToR.
The port is configured as a trunk so it will carry both native VLAN traffic as well as the appropriate FCoE
VLAN tagged traffic. Then the standard zone name and members are setup, placed in the zone set and the
zone set is activated (Figure 59).
Nexus_5548-1
Nexus_5548-2
interface members.
port-channel, of which they will be a part. The
interface members.
Configure the links add a description and the
port-channel, of which they will be a part. The
configure
interface Ethernet1/1
description FCoE_downlink_to_IOA_A1
20,1000
end
configure
interface Ethernet1/1
description FCoE_downlink_to_IOA_A2
21,1001
end
of it.
of it.
configure
exit
member zone1SAN_A
exit
vsan 2
end
configure
exit
member zone1SAN_B
exit
vsan 3
end
Downstream FCoE Link
70
7.10
At this point, Environment 5 has been configured and physical connectivity between all four switches can
Table 11
FN410S IOA
From Switch / Port
71
Nexus 5548UP
To Switch / Port
From Switch / Port
To Switch / Port
☐
IOA-A1 / te0/9
IOA-A2 / te0/9
☐
5548-1 / fc2/1
☐
IOA-A1 / te0/10
5548-1 / eth1/11
☐
5548-2 / fc2/1
☐
IOA-A1 / te0/11
5548-2 / eth1/11
☐
5548-1 / te0/17
5548-2 / te0/17
☐
IOA-A1 / te0/12
5548-1 / eth1/1
☐
5548-1 / te0/18
5548-2 / te0/18
☐
IOA-A2 / te0/9
IOA-A1 / te0/9
☐
5548-1 / eth1/9
To Nexus 7000-1 / te0/1
☐
IOA-A2 / te0/10
5548-1 / eth1/12
☐
5548-1 / eth1/10
To Nexus 7000-2 / te0/1
☐
IOA-A2 / te0/11
5548-2 / eth1/12
☐
5548-2 / eth1/9
To Nexus 7000-1 / te0/2
☐
IOA-A2 / te0/12
5548-2 / eth1/1
☐
5548-2 / eth1/10
To Nexus 7000-2 / te0/2
8
Validation
This section covers the commands used to validate the topology is correctly configured and functioning
as expected. Before using these commands, all cables should be in place between the servers, the
FC/FCoE SAN configured, and all switch settings configured.
8.1
Validation — FN410S IOA (FSB)
All of the commands listed in this section can be run against the FN410S IOA in either SMUX or PMUX
modes and can be used to validate all the environments. The output examples shown below are from
Environment 2 — Dell FN410S in Standalone IOA Mode in a Nexus Fabric Mode Environment.
show interface status
Check the general status of the ports, and links by using the show interfaces status command
(Figure 60).
FN410S_IOA_A1#show interfaces status
Port
Te
Te
Te
Te
Te
Te
Te
Te
Te
Te
Te
Te
0/1
0/2
0/3
0/4
0/5
0/6
0/7
0/8
0/9
0/10
0/11
0/12
Description
Status Speed
Up
Down
Down
Down
Down
Down
Down
Down
Down
Up
Up
Down
Duplex Vlan
10000 Mbit Full
Auto
Auto
Auto
Auto
Auto
Auto
Auto
Auto
Auto
Auto
Auto
Auto
Auto
Auto
Auto
Auto
10000 Mbit Full
10000 Mbit Full
Auto
Auto
1-4094
1-4094
1-4094
1-4094
1-4094
1-4094
1-4094
1-4094
-----
Output of the show interfaces status Command
show fip-snooping config
On the IOA, the FIP-Snooping configuration can be checked by using the show fip-snooping config
command (Figure 61). This command shows if the FIP-Snooping feature is enabled and the global
configuration is set. It will also display the current default of the FC-MAP value.
FN410S_IOA_A1#show fip-snooping config
FIP Snooping Feature enabled Status: Enabled
FIP Snooping Global enabled Status: Enabled
Global FC-MAP Value: 0X0EFC00
Maximum Sessions Per Enode Mac: 32
Output of the show fip-snooping config Command
72
show fip-snooping fcf
Check the IOA - FIP-snooping FCF connection information using the show fip-snooping fcf
command (Figure 62). This command displays the FCF’s MAC address, the connecting interface and
applicable VLAN configured for the connection.
FN410S_IOA_A1#show fip-snooping fcf
FCF MAC
------54:7f:ee:53:3e:a8
FCF Interface
------------Po 128
VLAN
---1000
FC-MAP
-----0e:fc:00
FKA_ADV_PERIOD
-------------8000
No. of Enodes
------------1
Output of the show fip-snooping fcf Command
show fip-snooping enode
Check the FCoE enabled connection to an enode or network adapter in a host server by using the show
fip-snooping enode command (Figure 63). This command displays the Enode MAC address, the port
to which the Enode is connected, the FCF’s MAC address, the applicable VLAN configured for the
connection and the FC-ID.
FN410S_IOA_A1#show fip-snooping enode
Enode MAC
--------18:a9:9b:d6:7a:83
Enode Interface
--------------Te 0/1
FCF MAC
------54:7f:ee:53:3e:a8
VLAN
---1000
FC-ID
----ed:00:00
Output of the show fip-snooping enode Command
show fip-snooping sessions
Using the IOA, the fip-snooping sessions can be displayed using: show fip-snooping sessions
(Figure 64). This command displays the following items that are that are used in an FCoE session: Enode
MAC address, Enode interface, and FCF MAC address, FCF connecting interface, FCoE VLAN being used,
FCoE MAC address, FC-ID, Port WWPN, and Port WWNN being used for the session.
Note: In order to fit on the page, the output below is split onto two lines.
FN410S_IOA_A1#show fip-snooping sessions
Enode MAC
--------18:a9:9b:d6:7a:83
FC-ID
----ed:00:00
Enode Intf
---------------Te 0/1
Port WWPN
--------20:01:18:a9:9b:d6:7a:83
FCF MAC
-----54:7f:ee:53:3e:a8
FCF Intf
------Po 128
VLAN
-----1000
FCoE MAC
-------0e:fc:00:ed:00:00
Port WWNN
--------20:00:18:a9:9b:d6:7a:83
Output of the show fip-snooping sessions Command
show fip-snooping statistics
Using the IOA, FIP-Snooping statistics can be validated using the show fip-snooping statistic
(Figure 65) command. This command displays the various counters applicable to the FIP-Snooping
feature. Specifics like vlan ids and ports can be specified.
73
FN410S_IOA_A1#show fip-snooping statistics interface vlan 1000
Number
Number
Number
Number
Number
Number
Number
Number
Number
Number
Number
Number
Number
Number
Number
Number
Number
Number
Number
Number
Number
Number
of
of
of
of
of
of
of
of
of
of
of
of
of
of
of
of
of
of
of
of
of
of
Vlan Requests
Vlan Notifications
Multicast Discovery Solicits
Unicast Discovery Solicits
FLOGI
FDISC
FLOGO
Enode Keep Alive
VN Port Keep Alive
Multicast Discovery Advertisement
Unicast Discovery Advertisement
FLOGI Accepts
FLOGI Rejects
FDISC Accepts
FDISC Rejects
FLOGO Accepts
FLOGO Rejects
CVL
FCF Discovery Timeouts
Enode Mac Timeouts
VN Port Session Timeouts
Session failures due to Hardware Config
:0
:0
:7
:0
:1
:0
:0
:31
:2
:31
:1
:2
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
FN410S_IOA_A1#show fip-snooping statistics interface te0/1
Number of Vlan Requests
:1
Number of Vlan Notifications
:0
Number of Multicast Discovery Solicits
:7
Number of Unicast Discovery Solicits
:0
Number of FLOGI
:1
Number of FDISC
:0
Number of FLOGO
:0
Number of Enode Keep Alive
:41
Number of VN Port Keep Alive
:3
Number of Multicast Discovery Advertisement
:0
Number of Unicast Discovery Advertisement
:0
Number of FLOGI Accepts
:0
Number of FLOGI Rejects
:0
Number of FDISC Accepts
:0
Number of FDISC Rejects
:0
Number of FLOGO Accepts
:0
Number of FLOGO Rejects
:0
Number of CVL
:0
Number of FCF Discovery Timeouts
:0
Number of Enode Mac Timeouts
:0
Number of VN Port Session Timeouts
:0
Number of Session failures due to Hardware Config
:0
Output of the show fip-snooping statistics Command
74
Show interfaces dcbx detail
The show interfaces dcbx detail command (Figure 66) shows vital information about the DCBX
configuration. This command shows specific detailed information about the configuration that has been
negotiated between the devices. In this example, interface TenGigabitEthernet 0/1 is used as a downlink to
a host server and network adapter installed in that system. Interface TenGigabitEthernet 0/9 and 0/10 are
the uplinks into the top-of-rack FCF switch.
Some key items in these results will be TRUE or False for Is Configuration Source?, Local DCBX
Compatibility mode is, and Peer Operating version is. These results are important in understanding the
negotiations between the switches and network adapter.
75
FN410S_IOA_A1#show interfaces dcbx detail
E-ETS Configuration TLV enabled
e-ETS Configuration TLV disabled
R-ETS Recommendation TLV enabled
r-ETS Recommendation TLV disabled
P-PFC Configuration TLV enabled
p-PFC Configuration TLV disabled
F-Application priority for FCOE enabled f-Application Priority for FCOE disabled
I-App priority for iSCSI enabled
i-App Priority for iSCSI disabled
-------------------------------------------------------------------------------Interface TenGigabitEthernet 0/1
Remote Mac Address 18:a9:9b:d6:7a:81
Port Role is Auto-Downstream
DCBX Operational Status is Enabled
Is Configuration Source? FALSE
Local DCBX Compatibility mode is CEE
Local DCBX Configured mode is AUTO
Peer Operating version is CEE
Local DCBX TLVs Transmitted: ErPFi
Interface TenGigabitEthernet 0/9
Remote Mac Address 54:7f:ee:53:3e:88
Port Role is Auto-Upstream
Is Configuration Source? FALSE
Local DCBX Compatibility mode is CIN
Peer Operating version is CIN
Remote Mac Address 54:7f:ee:53:3e:89
Port Role is Auto-Upstream
Is Configuration Source? TRUE
Local DCBX Compatibility mode is CIN
Peer Operating version is CIN
Output of the show interfaces dcbx detail Command
show interfaces pfc detail
The show interfaces pfc detail command (Figure 67) shows vital information about the DCBX
established PFC configuration. This command shows detailed information about the configuration that is
negotiated between the devices. In this example, interfac, TenGigabitEthernet 0/1 is used as a downlink to
a host server and network adapter installed in that system. Interfaces TenGigabitEthernet 0/9 and 0/10 are
the uplinks into the top-of-rack FCF switch.
Some key items in these results will be Priority list is 2, Remote Willing status (this should be willing is
enabled for the Host network adapter and willing is disabled for FCF Nexus switch). Other important items
are PFC DCBX Oper status, and FCOE TLV TX Status is enabled.
FN410S_IOA_A1#show interfaces pfc detail
76
Admin mode is on
Admin is enabled, Priority list is 4
Remote is enabled, Priority list is 3
Remote Willing Status is enabled
Local is enabled, Priority list is 3
Oper status is internally propagated
PFC DCBX Oper status is Up
State Machine Type is Feature
TLV Tx Status is enabled
PFC Link Delay 65535 pause quntams
Application Priority TLV Parameters :
-------------------------------------FCOE TLV Tx Status is enabled
ISCSI TLV Tx Status is disabled
Local FCOE PriorityMap is 0x8
Local ISCSI PriorityMap is 0x0
Remote FCOE PriorityMap is 0x8
2
0
2
0
Input
Pause
Input
Error
TLV pkts, 5 Output TLV pkts, 0 Error pkts, 0 Pause Tx pkts,
Rx pkts
Appln Priority TLV pkts, 4 Output Appln Priority TLV pkts,
Appln Priority TLV Pkts
Admin mode is on
Remote Willing Status is disabled
2 Input TLV pkts, 5 Output TLV pkts, 0 Error pkts, 0 Pause Tx pkts,
0 Pause Rx pkts
2 Input Appln Priority TLV pkts, 4 Output Appln Priority TLV pkts,
0 Error Appln Priority TLV Pkts
Admin mode is on
Oper status is recommended
77
2
,
2
,
Input TLV pkts, 5 Output TLV pkts, 0 Error pkts, 0 Pause Tx pkts
0 Pause Rx pkts
Input Appln Priority TLV pkts, 4 Output Appln Priority TLV pkts
0 Error Appln Priority TLV Pkts
Output of the show interfaces pfc detail Command
show interfaces ets detail
The show interfaces ets detail command shows vital information on the ETS settings in the
configuration (Figure 68). This command will show detailed information about the ETS configuration that
has been negotiated between the devices. In this example, interface TenGigabitEthernet 0/1 is used as a
downlink to a host server and network adapter installed in that system. Interfaces TenGigabitEthernet 0/9
and 0/10 are the uplinks into the top-of-rack FCF switch.
Some key items in these results will be TC-grp, Priority#, Bandwidth, and TSA. Note that typically, only
these two interfaces are seen, this command will list them all.
FN410S_IOA_A1#show interfaces ets detail
Max Supported PG is 4
Number of Traffic Classes is 8
Admin mode is on
Admin Parameters :
-----------------Admin is enabled
PG-grp
Priority#
Bandwidth
TSA
-----------------------------------------------0
0,1,2,3,5,6,7
50 %
ETS
1
4
50 %
ETS
2
3
4
5
6
7
15
Remote Parameters :
------------------Remote is enabled
78
PG-grp
Priority#
Bandwidth
TSA
-----------------------------------------------0
0,1,2,4,5,6,7
50 %
ETS
1
3
50 %
ETS
2
3
4
5
6
7
15
Remote Willing Status is enabled
Local Parameters :
-----------------Local is enabled
PG-grp
Priority#
Bandwidth
TSA
-----------------------------------------------0
0,1,2,4,5,6,7
50 %
ETS
1
3
50 %
ETS
2
3
4
5
6
7
15
-
ETS DCBX Oper status is Up
Conf TLV Tx Status is enabled
3 Input Conf TLV Pkts, 6 Output Conf TLV Pkts, 0 Error Conf TLV Pkts
Admin mode is on
Admin Parameters :
PG-grp
Priority#
Bandwidth
TSA
-----------------------------------------------79
0
1
2
3
4
5
6
7
15
0,1,2,3,5,6,7
4
50 %
50 %
-
ETS
ETS
-
Remote Parameters :
PG-grp
Priority#
Bandwidth
TSA
-----------------------------------------------0
0,1,2,4,5,6,7
50 %
ETS
1
3
50 %
ETS
2
3
4
5
6
7
15
-
Local Parameters :
PG-grp
Priority#
Bandwidth
TSA
-----------------------------------------------0
0,1,2,4,5,6,7
50 %
ETS
1
3
50 %
ETS
2
3
4
5
6
7
15
80
Admin mode is on
Admin Parameters :
PG-grp
Priority#
Bandwidth
TSA
-----------------------------------------------0
0,1,2,3,5,6,7
50 %
ETS
1
4
50 %
ETS
2
3
4
5
6
7
15
Remote Parameters :
PG-grp
Priority#
Bandwidth
TSA
-----------------------------------------------0
0,1,2,4,5,6,7
50 %
ETS
1
3
50 %
ETS
2
3
4
5
6
7
15
Local Parameters :
PG-grp
Priority#
Bandwidth
TSA
-----------------------------------------------0
0,1,2,4,5,6,7
50 %
ETS
1
3
50 %
ETS
2
3
4
5
6
-
81
7
15
-
-
Oper status is recommended
Oper status is internally propagatedETS DCBX Oper status is Up
Output of the show interfaces ets detail Command
After these validation steps, go into the disk management interface of the server and verify the SAN is
configured appropriately. The server will have a LUN available to use for storage.
82
8.2
Validation — Cisco Nexus 5548UP
The following commands can be used to validate the environments using the Cisco Nexus (Environments
2, 3 and 5).
show interface brief
Use the show interface brief command (Figure 69) to display the results of active ports and vFC
interfaces. Verify that the ports, which are expected to have links, show up correctly.
N5K-R188U37# show interface brief
------------------------------------------------------------------------------Interface Vsan
Admin Admin
Status
SFP
Oper Oper
Port
Mode
Trunk
Mode Speed Channel
Mode
(Gbps)
------------------------------------------------------------------------------fc2/1
2
auto
on
up
swl
F
8
--------------------------------------------------------------------------------Ethernet
VLAN
Type Mode
Status Reason
Speed
Port
Interface
Ch#
-------------------------------------------------------------------------------Eth1/1
1
eth trunk up
none
10G(D) 20
Eth1/2
1
eth trunk up
none
10G(D) 20
Eth1/17
1
eth
trunk
up
none
10G(D) 55
Eth1/18
1
eth
trunk
up
none
10G(D) 55
------------------------------------------------------------------------------Port-channel VLAN
Type Mode
Status Reason
Speed
Protocol
Interface
-------------------------------------------------------------------------------Po1
1
eth trunk down
No operational members
auto(D)lacp
Po20
1
eth trunk up
none
a-10G(D)lacp
Po55
1
eth trunk up
none
a-10G(D)lacp
------------------------------------------------------------------------------Interface Vsan
Admin Admin
Status
Bind
Oper
Oper
Mode
Trunk
Info
Mode
Speed
Mode
(Gbps)
------------------------------------------------------------------------------vfc101
2
F
on
trunking
18:a9:9b:d6:7a:83
TF
auto
Output of the show interface brief Command
show interface vfc 101
To show information about a virtual Fibre Channel Interfaces use the show interface vfc <vfc-id>
command (Figure 70). Multiple vfc (101 and 102) were used on multiple switches in the environments
detailed in this guide. Below is one example for vfc 101.
83
N5K-R188U37#show interface vfc 101
vfc101 is trunking
Bound MAC is 18:a9:9b:d6:7a:83
Hardware is Ethernet
Port WWN is 20:64:54:7f:ee:53:3e:bf
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 2
Trunk vsans (admin allowed and active) (1-2)
Trunk vsans (up)
(2)
Trunk vsans (isolated)
()
Trunk vsans (initializing)
(1)
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
0 frames input, 0 bytes
0 discards, 0 errors
0 frames output, 0 bytes
0 discards, 0 errors
last clearing of "show interface" counters never
interface last changed at Thu Jun 12 08:18:32 2014
Output of the show interface vfc101 Command
show fcns database
The devices, server interfaces and their NP uplinks are displayed with the show fcns database
command (Figure 71). The output below shows that VSAN 2 has two servers logged in (scsi-fcp: init).
N5K-R188U37# show fcns database
VSAN 2:
-------------------------------------------------------------------------FCID
TYPE PWWN
(VENDOR)
FC4-TYPE:FEATURE
-------------------------------------------------------------------------0xed0000
N
20:01:18:a9:9b:d6:7a:83
scsi-fcp:init
0xed00ef
N
50:06:01:61:3e:e0:18:70 (Clariion)
scsi-fcp:target
Total number of entries = 2
Output of the show fcns database Command
84
show flogi database
Use the show flogi database command (Figure 72) to verify that negotiations have happened properly
between the FCF and end devices. In this case, the Cisco Nexus 5548 is the FCF.
N5K-R188U37# show flogi database
-------------------------------------------------------------------------------INTERFACE
VSAN
FCID
PORT NAME
NODE NAME
-------------------------------------------------------------------------------fc2/1
2
0xed00ef 50:06:01:61:3e:e0:18:70 50:06:01:60:be:e0:18:70
vfc101
2
0xed0000 20:01:18:a9:9b:d6:7a:83 20:00:18:a9:9b:d6:7a:83
Total number of flogi = 2.
Output of the show flogi database Command
At this point, the vFC and FC interfaces should be populated in the FLOGI (fabric login) database. This
command is showing the devices that have done a valid FLOGI to the Cisco Nexus switch. The vFC should
show the expected port and node WWN of the network adapter being used in the server.
show zoneset active
The show zoneset active command (Figure 73) shows the status of the zones that have been created.
This command shows the current activated zone set and all the participating zones with their individual
members, the asterisk next to a FCID represents currently active sessions.
N5K-R188U37# show zoneset active
* fcid 0xed0000 [pwwn 20:01:18:a9:9b:d6:7a:83]
* fcid 0xed00ef [interface fc2/1 swwn 20:00:54:7f:ee:53:3e:80]
Output of the show zoneset active Command
show spanning-tree summary
The spanning tree configuration can be checked using the show spanning-tree summary command
(Figure 74). The location of blocking ports should be noted and the number of blocking ports should be
the number expected for this configuration. Otherwise it could be that an unintentional cable loop has
been created that will need to be resolved.
N5K-R188U37# show spanning-tree summary
Switch is in rapid-pvst mode
Root bridge for: VLAN0001, , VLAN0005, VLAN0008, VLAN1000
Port Type Default
is disable
Edge Port [PortFast] BPDU Guard Default is disabled
Edge Port [PortFast] BPDU Filter Default is disabled
Bridge Assurance
is enabled
Loopguard Default
is disabled
Pathcost method used
is short
STP-Lite
is enabled
Name
Blocking Listening Learning Forwarding STP Active
---------------------- -------- --------- -------- ---------- ---------VLAN0001
0
0
0
5
5
85
VLAN0005
0
0
0
2
2
VLAN0008
0
0
0
2
2
VLAN1000
0
0
0
1
1
---------------------- -------- --------- -------- ---------- ---------3 vlans
0
0
0
12
12
Output of the show spanning-tree summary Command
8.3
Validation — Dell S5000
The following commands can be used to validate the environments using the Dell Networking S5000
(Environments 1 and 4).
Show Interface Status
Use the show interface status command to display the active ports and verify that the ports, which
are expected to have links, show up correctly.
S5000_1#show interfaces status
Port
Description Status Speed
Duplex Vlan
Fc 0/0
Up
8000 Mbit Auto
-Fc 0/1
Down
Auto
Auto
-Te 0/12
Up
10000 Mbit Full
1,1000
Te 0/24
Up
10000 Mbit Full
-Te 0/25
Up
10000 Mbit Full
-Fo 0/48
Up
40000 Mbit Full
-Fo 0/52
Up
40000 Mbit Full
-Fo 0/56
Down
40000 Mbit Auto
-Fo 0/60
Down
40000 Mbit Auto
-“show interface status” command
Note: Interfaces not used in Environments 1 and 4 have been removed from Figure 75 for brevity.
Show FC NS Switch
The devices, server interfaces and their NP uplinks can be displayed with the show fc ns switch
command. The command shows that the S5000_1 switch has two devices logged into it.
86
S5000_1#show fc ns switch
Total number of devices =
Switch Name
Domain Id
Switch Port
FC-Id
Port Name
Node Name
Class of Service
Symbolic Port Name
Symbolic Node Name
Port Type
2
10:00:5c:f9:dd:ef:3b:00
1
Fc 0/0
01:00:00
50:06:01:61:3e:e0:18:70
50:06:01:60:be:e0:18:70
8
CLARiiON::::SPA23::FC::::::
CLARiiON::::SPA::FC::::::
N_Port
Switch Name
Domain Id
10:00:5c:f9:dd:ef:3b:00
1
Switch Port
FC-Id
Port Name
Node Name
Class of Service
Symbolic Port Name
Symbolic Node Name
Port Type
Te 0/12
01:38:00
20:01:18:a9:9b:d6:7a:83
20:00:18:a9:9b:d6:7a:83
8
Broadcom Port0 pWWN 20:01:18:a9:9b:d6:7a:83
Broadcom BCM57810 FCoE 7.8.75.0 7.8.81.0 WIN-12KR2
N_Port
“show fc ns switch” command
Show FC Topology
Use the show flogi database command to verify that negotiations have occurred properly between
the FCF and end devices. In this case, the Cisco Nexus 5548 is the FCF.
S5000_1#show fc topology
Port
Number
______
Port
Type
____
Local
PortWWN
_______
Remote
PortWWN
_______
Remote
NodeWWN
_______
FCID/
Domain ID
_________
Fc 0/0
F
20:00:5c:f9:dd:ef:3b:00
50:06:01:61:3e:e0:18:70
50:06:01:60:be:e0:18:70
01:00:00
Not Applicable
20:01:18:a9:9b:d6:7a:83
20:00:18:a9:9b:d6:7a:83
01:38:00
Te 0/12 VF
“show fc topology” command
At this point, the FC interfaces should be populated in the FLOGI database. This command is showing the
devices that have done a valid FLOGI (fabric login) to the S5000 switch. The VFC should show the
expected port and node WWN of the NETWORK ADAPTER being used in the server.
Show Zoneset Active
The show fc zoneset active command will show the status of the zones that have been put into
place. This command will show the current activated zoneset and all the participating zones with their
individual members, the asterisk next to a FCID represents a currently active sessions.
S5000_1#show fc zoneset active
Active Zoneset: S5000_Storage_1
ZoneName
ZoneMember
=======================================================
S5000_1
*20:01:18:a9:9b:d6:7a:83
*50:06:01:61:3e:e0:18:70
Example of “show fc zoneset active” command
87
A
Basic Terminology
CLI (Command Line Interface): Text-based telnet, secure shell (SSH), or serial type interface that is used
for issuing commands to a device.
CMC (Chassis Management Controller): A systems management component designed to manage one or
more Dell PowerEdge Blade Chassis.
ETS (Enhanced Transmission Selection): ETS allows a switch to allocate bandwidth to traffic classes and
share unused bandwidth with lower-priority traffic classes while coexisting with strict-priority traffic
classes. ETS is defined in the IEEE 802.1Qaz standard (IEEE, 2011).
FIP-Snooping (FCoE Initialization Protocol-Snooping): FIP-Snooping monitors FIP logins, solicitations,
and advertisements. The monitoring or snooping process allows the switch to gather information
pertaining to the ENode (FCoE Node) and FCF(FCoE forwarder) addresses and place filters that only allow
access to ENode devices that have logged-in successfully. This enables the FCoE VLAN to deny all other
traffic except this lossless FCoE storage traffic. The filtering process also secures the end-to-end path
between the ENode device and the FCF. The ENode will only be able to talk with the FCF in which it has
logged into.
FIP-Snooping bridge (FSB): A switch configured to perform FIP-Snooping.
FCF: FCoE forwarders (FCFs) act as an Ethernet and FC switch combined. All typical termination functions
that would occur on a FC switch occur on the FCF. FCFs give VF_Ports and VE_Ports for their virtual FC
interfaces.
FlexAddress: A virtual, unique, chassis assigned, slot persistent address that is substituted for the factory
programmed protocol specific address on Ethernet and Fibre Channel devices within a FlexAddress
enabled chassis.
IOM (IO module): IOM refers to the modules on the rear of the Dell PowerEdge chassis that will receive
and transmit I/O from the blade servers.
MAC Address (Media Access Control Address): A hardware specific address that uniquely identifies each
node of a network. MAC addresses are typically assigned by the vendors of network interface controllers
(NIC) and are stored in the hardware.
NPIV (N_Port ID Virtualization): NPIV allows multiple N_Port IDs to share a single physical N_Port. This
allows multiple Fibre Channel initiators to occupy a single physical port.
NPV ( N-port Virtualizer): NPV is a FC aggregation method, which passes traffic through to end devices,
while eliminating the need to use a domain ID for the device.
PFC (Priority Flow Control): PFC is flow control based on priority settings, which adds additional
information to the standard pause frame. These additional fields allow devices to pause traffic on a specific
priority instead of pausing all traffic.
ToR: Top of Rack (ToR) is a term for a switch that is positioned at the top of a server rack in a data center.
VLAN (Virtual Local Area Network): VLAN is a single layer-2 network (also called a broadcast domain, as
broadcast traffic does not escape a VLAN on its own). Multiple VLANs can be passed between switches
using switchport trunk interfaces. When passed across trunk links, frames in a VLAN are prefixed
88
with the number of the VLAN that they belong to—a twelve-bit value that allows just over 4000 differently
numbered VLANs.
VLT (Virtual Link Trunking): VLT allows physical links between two chassis to appear as a single virtual link
to the network core or other switches such as Edge, Access or ToR. VLT reduces the role of Spanning Tree
protocols by allowing LAG terminations on two separate distribution or core switches, and by supporting a
loop free topology. VLT provides Layer 2 multipathing, creating redundancy through increased bandwidth,
enabling multiple parallel paths between nodes and load-balancing traffic where alternative paths exist.
vPC: vPC refers to the combined PortChannel between the vPC peer devices and the downstream device.
vPC peer switch: The vPC peer switch is one of a pair of switches that are connected to the special
PortChannel known as the vPC peer link. One device will be selected as the primary device, and the other
will be the secondary device.
vPC peer link: The vPC peer link is the link used to synchronize states between the vPC peer devices. The
vPC peer link carries control traffic between two vPC switches and multicast, broadcast data traffic. In
some link failure scenarios, it also carries unicast traffic. You should have at least two 10 Gigabit Ethernet
interfaces for peer links.
vPC domain: This domain includes both vPC peer devices, the vPC peer keepalive link, and all the
PortChannels in the vPC connected to the downstream devices. It is also associated with the configuration
mode that you must use to assign vPC global parameters.
vPC peer keepalive link: The peer keepalive link monitors the vitality of a vPC peer switch. The peer
keepalive link sends periodic keepalive messages between vPC peer devices. The vPC peer keepalive link
can be a management interface or switched virtual interface (SVI). No data or synchronization traffic
moves over the vPC peer keepalive link; the only traffic on this link is a message that indicates that the
originating switch is operating and running vPC.
vPC member port: vPC member ports are interfaces that belong to the vPCs.
VSAN (Virtual SAN or Virtual Stroage Area Network): A VSAN is a logical partitioning of physical
connections to provide for fabric or SAN separation.
WWNN (World Wide Node Name): A unique identifier that is assigned to a manufacturer and hard-coded
into a Fibre Channel (FC) switch. The WWNN is used to identify a switch.
WWPN (World Wide Port Name): A unique identifier that is assigned to a manufacturer and hard-coded
into a Fibre Channel (FC) switch. The WWPN is used to identify an individual port on a switch.
89
B
Additional Resources and References
Cisco Systems, Inc. (2011). Cisco Nexus 5000 Series NX-OS SAN Switching Configuration Guide, Release
5.1(3)N1(1). San Jose: Cisco Systems, Inc
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/san_switching/513_n1_1/b_C
isco_n5k_nxos_sanswitching_config_guide_rel513_n1_1.html
Cisco Systems, Inc. (2010, 01). Cisco 4000 Series Design Guide. Retrieved 06 19, 2012, from Cisco.com:
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps10596/deployment_guide_c07574724.html
Fibre Channel over Ethernet Initialization Protocol, Cisco,
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11560403.html
IEEE. (2008, November 24). 802.1Qaz/D0.2. Standard for Local and Metropolitan Area Networks Virtual.
IEEE. (2009, Feb 9). 802.1Qbb/D1.0. Draft Standard for Local and Metropolitan Area Networks – Virtual.
IEEE. (2011, Sep 15). 802.1Qaz - Enhanced Transmission Selection. Retrieved May 30, 2012, from
IEEE802.org
http://www.ieee802.org/1/pages/802.1az.html
T11/08-264v0 FCoE: Increasing FCoE Robustness using FIP-Snooping and FPMA, T11,
http://www.t11.org/ftp/t11/pub/fc/bb-5/08-264v0.pdf
T11/09-291v0 FIP VLAN discovery updates, T11,
http://www.t11.org/ftp/t11/pub/fc/bb-5/09-291v1.pdf
Cisco Systems Inc. (2009). Configurating SAN Port Channel. Retrieved 7 15, 2013, from Cisco.com – Cisco
Nexus 5000 Series NX-OS SAN Switching Configuration Guide:
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/san_switching/Cisco_Nexus_
5000_Series_NX-OS_SAN_Switching_Configuration_Guide_chapter7.html
Cisco Systems, Inc. (2009, 07). Virtual PortChannel Quick Configuration Guide. Retrieved Jun 18, 2012,
from Cisco.com:
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/configuration_guide_c07543563.html
IEEE. (2011, Sep 15). 802.1Qaz - Enhanced Transmission Selection. Retrieved May 30, 2012, from
IEEE802.org: http://www.ieee802.org/1/pages/802.1az.html
Cisco Nexus 5000 Series NX-OS SAN Switching Configuration Guide, Release 5.2(1)N1(1)
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/san_switching/521n11/b_5k_
SAN_Switching_Config_521N11.html
90
C
Attachments
This deployment guide includes the following attachments.



CNA_Broadcom_57810-K_FCoE.pdf
This attachment includes the steps to configure the Broadcom 57810-K CNA for FCoE.
CNA_Emulex_OCm14102_FCoE.pdf
This attachment includes the steps to configure the Emulex OCm14102 CNA for FCoE.
Enabling_FlexAddressing.pdf
The following attachments include the steps necessary to configure each of the environments presented
in this deployment guide. These documents only include the configuration steps and are intended to
provide a handout that focuses specifically on the configuration steps of each environment.





Environment-1.pdf
Environment-2.pdf
Environment-3.pdf
Environment-4.pdf
Environment-5.pdf
Support and Feedback
Contacting Technical Support
Support Contact Information
Web: http://Support.Dell.com/
Telephone: USA: 1-800-945-3355
Feedback for this document
We encourage readers of this publication to provide feedback on the quality and usefulness of this
deployment guide by sending an email to [email protected]
91

Dell PowerEdge FX2 – I/O Aggregator – FCoE (FSB

Transcription

Similar documents

Napoleon Container Terminal

Subcutaneous Port Catheter (Port, porta

January 17, 2011 - Village of Port Clements

PDF 1mb - Marina Adelaide

TRACe-NET xM8P2G-1

#104 300 Klahanie Drive • Port Moody

The project incorporates two types of new architecture

PA NEWS - Port Authority Retirees Association, Inc.

Dog Meet and Greet! Saturday June 6th

Proceeds go to support NJROTC scholarships and the Port