XgOS CLI and Linux Host Software Guide

Transcription

Release V1.5
Xsigo Systems
70 West Plumeria Drive
San Jose, CA 95134
USA
http://www.xsigo.com
Tel: +1.408.329.5600
Part number: 650-20023-01 Rev A
Published: 01 2008
Regulatory Compliance Statements
EMI Statement (Class A)
“NOTE: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to part
15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the
equipment is operated in a commercial environment. This equipment generates, uses, and can radiate radio frequency
energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio
communications. Operation of this equipment in a residential area is likely to cause harmful interference in which case the
user will be required to correct the interference at his own expense.”
CANADA
This Class A digital apparatus complies with Canadian ICES-003.
Cet appareil numérique de la classe A est conforme à la norme NMB-003 du Canada.
Lithium battery-internal fuse replacement and disposal
CAUTION
Danger of explosion if the lithium battery is incorrectly replaced. Replace only with the same or equivalent type
recommended by the manufacturer. Dispose of used batteries according to the manufacturer's instructions.
Internal fuses are not intended to be replaced in the field. Return equipment with blown fuses to the manufacturer.
Laser Caution for I/O Cards (CDRH-US)
USE OF CONTROLS OR ADJUSTMENTS OR PERFORMANCE OF PROCEDURES OTHER THAN THOSE
SPECIFIED HEREIN MAY RESULT IN HAZARDOUS RADIATION EXPOSURE.
Complies with 21 CFR Chapter 1, Subchapter J, Part 1040.10.
IEC 60825-1: 1993, A1: 1997, A2: 2001; IEC 60825-2: 2000
Replacement Laser Transceiver Modules
For continued compliance with the above laser safety Standards, only approved Class 1 modules from our approved
vendors should be installed in the product. Contact Xsigo Customer Support (+1-408-736-3013) for approved-vendor
contact information.
Power Cord Set Requirements – General
The requirements listed below are applicable to all countries:
The length of the power cord set must be at least 6.00 feet (1.8 m) and a maximum of 9.75 feet (3.0 m).
All power cord sets must be approved by an acceptable accredited agency responsible for evaluation in the country where
the power cord set will be used.
The power cord set must have a minimum current capacity of 3A and a nominal voltage rating of 125 or 250 V ac~, as
required by each country's power system.
The appliance coupler on the power cord must meet the mechanical configuration of an EN 60320 / IEC 60320 Standard
Sheet C13 connector, for suitable mating with the appliance inlet on the product.
Power Cord Set Requirements – Specifics By Country
United States (UL), Canada (CSA)
The flexible power cord set must be UL Listed and CSA Certified, minimum Type SVT or equivalent, minimum No. 18
AWG, with 3-conductors that includes a ground conductor. The wall plug must be a three-pin grounding type, such as a
NEMA Type 5-15P (rated 15A, 120V) or Type 6-15P (rated 15A, 250V).
Europe (Austria (OVE), Belgium (CEBEC), Denmark (DEMKO), Finland (SETI), France (UTE), Germany (VDE), Italy
(IMQ), Netherlands (KEMA), Norway (NEMKO), Sweden (SEMKO), Switzerland (SEV), U.K. (BSI/ASTA)
The flexible power cord set must be <HAR> Type H03VV-F, 3-conductor, minimum 0.75mm2 conductor size. Power
cord set fittings, particularly the wall plug, must bear the certification mark of the agency responsible for evaluation in the
country where it is being used, with examples listed above.
Australia (DFT/SAA)
Cord is as described under “Japan (PSE)” immediately below. Pins in the power plug must be with the sheathed, insulated
type, in accordance with AS/NZS 3112:2000.
Japan (PSE)
The appliance coupler, flexible cord, and wall plug must bear a “PSE” Mark in accordance with the Japanese Denan Law.
The flexible cord must be Type VCT or VCTF, 3-conductor, 0.75 mm2 conductor size. The wall plug must be a grounding
type with a Japanese Industrial Standard C8303 (15A, 125V) configuration.
© 2008 Xsigo Systems, All Rights Reserved.
Preface
i
Purpose
This guide describes how to configure the Xsigo Operating System (XgOS) using the CLI. It also includes installation and
configuration information for the Xsigo host software. The XgOS software environment runs on the Xsigo Systems
VP780 I/O Director. The Xsigo host software runs on a Linux server.
Audience
This guide is intended for data center network administrators, and it assumes that its readers have knowledge and
familiarity with common configuration and management tasks related to administering a data center. Although this guide
does present some conceptual material about topics and technologies, it is not intended as a complete and exhaustive
reference on those topics.
Conventions
Table 1 shows the typographical conventions used in this guide.
Table 1 Syntax Usage
Convention
Description
Example
courier bold
Commands and keywords that must be
entered exactly as shown. It also highlights
significant lines in the screen output display.
show vnic
courier plain
Actual display output that has been copied
from the device. Also used for variable names
shown in command syntax.
resourceUnavailable
“”
Quotes reference specific fields taken from
the screen display on the device.
See the “state” field.
<>
Angle brackets indicate variables for user
input. Replace the angle brackets and variable
name with information that is indicative of
your setup.
add vnic <vnicname>.<server-profile>
<slot>/<port>
{}
Curly braces indicate a choice of required
keywords or variables. You must enter at least
one of the enclosed parameters.
set vnic {* | <vnic-name>}
[ ]
Square brackets indicate a choice of optional
keywords or variables.
show system version [-all]
|
A pipe operator indicates a choice. You can
enter one of the parameters on either side of
the pipe.
set vnic {* | <vnic-name>}
Xsigo Systems VP780
XgOS CLI and Linux Host Software
ii
Preface
Related Documentation
This document is one part of the Xsigo Systems documentation set:
•
•
XMS Web User Guide
•
VP780 XgView User Guide
•
IS24 Deployment Guide
Release notes are also available with each major hardware or software release.
Revision History
Table 2 shows the revision history for this document.
Table 2 Revision History
Document Title
Document Number
Revision Level
Revision Date
650-20023-01
A
01/2008
650-20009-02
B
12/2007
XgOS CLI Configuration Guide
650-20009-02
A
07/2007
Contacting Technical Support
Xsigo Technical Support is available for resolving technical questions, filing customer cases, and providing additional
information about the VP780. Xsigo Technical Support is available through the Web at this URL
http://www.xsigo.com/support
Please direct technical questions and requests for replacement components to:
Xsigo Customer Support +1-408-736-3013
Xsigo Systems VP780
Contents
iii
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .i
Chapter 1
New Features and Enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1
Chapter 2
XgOS CLI Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3
Login Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4
File System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6
Scripting Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11
Configuration Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15
Virtual Resources Quick Start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18
Set CLI Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22
Display CLI Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23
System Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24
Slot/Port Numbering Scheme. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26
InfiniBand Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27
I/O Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33
I/O Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34
Hardware Status and Environmentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35
If and if-state. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37
Configuration Save and Restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38
Software Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38
System Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41
Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44
Recovery CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46
Online Help. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46
Chapter 3
Server Profiles and Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47
Server Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47
Templates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .48
Default Gateway . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .50
Chapter 4
vNICs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53
Basic vNIC Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53
vNIC Counters and Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55
HA vNIC Pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55
Auto Switchover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60
Admin State Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61
vNIC Priority . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62
MTU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63
VLANs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64
Xsigo Systems VP780
iv
Contents
Chapter 5
vHBAs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .67
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .67
Basic vHBA Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .68
Persistent Binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70
LUNs Per Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .73
Target Prescan and Rescan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .74
Set FC Port Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77
vHBA Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79
FC Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .80
Admin State Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81
LUN Masking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82
Optional LUN Masking: No Report LUN Interception . . . . . . . . . . . . . . . . . . . . . . . . . .85
Chapter 6
vSSLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87
Configuration Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87
Chapter 7
VMware ESX Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89
CLI Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89
Configuration Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90
Caveats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .93
Chapter 8
Xen Hypervisor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .95
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .95
CLI Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .96
Example 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .97
Example 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .99
Chapter 9
Network QoS for vNICs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .101
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .101
Network QoS Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .101
QoS Feature Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .102
Information Rates and Burst Sizes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .102
QoS Operations Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .103
Default Set Profiles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .104
QoS Custom Sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .106
ACLs with QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .109
Disable QoS on a vNIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .111
10GigE Ingress 802.1p and IP Precedence Mapping . . . . . . . . . . . . . . . . . . . . . . . . .112
DSCP Mapping on 10GigE Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112
Xsigo Systems VP780
v
Contents
Chapter 10
SAN QoS for vHBAs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115
SAN QoS Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115
Command Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .116
Chapter 11
ACLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .119
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .119
Example: Deny Egress Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .119
Setting Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .121
Setting Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .122
Removing ACLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .123
Chapter 12
SAN Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125
Boot Sequence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125
Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .127
SAN Boot Control: Two Ways to Go. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .127
CLI Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .128
SAN Boot Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .130
Example 1: RHEL5 SAN Boot Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .131
Example 2: Other Mount Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .132
Example 3: show san boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .133
Example 4: show serer-profile san-boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .134
Example 5: Loadmount. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .134
Best Practices and Gotchas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .134
Chapter 13
Xsigo Initrd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .137
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .137
xsigo-boot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .137
Using the Xsigo initrd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .138
SAN Booting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .139
NFS Booting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .140
PXE Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .141
Boot Menu Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .143
Chapter 14
PXE Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .145
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .145
MAC Addresses for DHCP Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .146
DHCP Server Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .147
TFTP Server Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .148
Xsigo HCA Firmware Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .149
PXE Boot Virtual NIC Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .149
Xsigo Systems VP780
vi
Contents
Chapter 15
Clusters and Termination Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .151
Virtual I/O Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .151
OpenSM Decoupling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .157
Termination Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .159
Chapter 16
System Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .163
System Image Upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .163
System Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .165
User Accounts, Role Groups, and Domains. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .167
Identity Management System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .171
CLI Wrapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .180
CLI Display Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .180
Terminal Rows and Columns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .182
CLI History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .182
Syslog. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .183
SNMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .183
Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .185
NTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .186
Diagnostics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .186
Root Login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .187
10GE I/O Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .187
Chapter 17
Linux Host Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .189
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .189
Supported Linux Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .189
Installed Linux Version . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .190
Installation Procedure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .191
HCA Firmware and Option ROM Upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .193
vHBA Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .198
Debug Tool: xsigo-support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .198
Chapter 18
Source RPM: Building Xsigo Host Drivers . . . . . . . . . . . . . . . . . . . . . . . .203
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .203
Compatibility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .203
Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .203
SRC RPM File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .204
Basic rpmbuild Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .204
The SPEC File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .204
Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .206
Build Option 1: Stock Kernels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .206
Build Option 2: Custom Kernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .207
Build Option 3: Kernel with Upgraded OFED Stack . . . . . . . . . . . . . . . . . . . . . . . . . . .207
Build Option 4: Combination of Customer Kernel and Upgraded OFED Stack . . . . . .208
Non-RPM Builds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .208
OFED Patch Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .209
Xsigo Systems VP780
vii
Contents
Release Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .209
Customer Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .209
Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .211
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .215
Xsigo Systems VP780
viii
Contents
Xsigo Systems VP780
New Features and Enhancements
1
This chapter describes the new features, enhancements, and documentation improvements added to this guide since the
last publishing.
Release 1.5 New Features and Enhancements
Table 1 Release 1.5 Additions and Modifications
Feature Description
VMware ESX Servers on page 89.
Xen Hypervisor on page 95.
Termination Groups on page 159.
Virtual I/O Fabric on page 151.
SAN Boot on page 125.
Xsigo Initrd on page 137.
OpenSM Decoupling on page 157.
Usability enhancements:
•
New command for displaying servers. See show phys-server on page 27.
•
V* admin state up/down control. For vNICs, see Admin State Control on page 61. For vHBAs,
see Admin State Control on page 81.
•
A phys-con is no longer exposed to the end user. The command set server-profile -add
-phys-con was changed to set server-profile connect=<phys-server>. The
command remove server-profile <phys-server> phys-con was removed.
•
The syntax for add server-profile changed. The “@<vp780>:” was introduced.
For example, the physical connection changed from ceasar,ServerPort24 to
ceasar@iowa:ServerPort24.
•
The show vnic command now displays QoS information.
•
The set snmp -add-trap-dest command was changed to add snmp trap-dest.
The set snmp -remove-trap-dest command was changed to remove snmp trapdest. See SNMP on page 183.
NFS booting for vNICs. New kernel arguments are introduced. See NFS Booting on page 140.
LUN Masking on page 82.
Source RPM: Building Xsigo Host Drivers on page 203.
A modified HCA firmware upgrade procedure. See HCA Firmware and Option ROM Upgrades on
page 193.
The Linux host driver packages are renamed to “xsigo-hostdrivers-kmod” (note the kmod added to the
RPM name). There is also a new host driver installation command chkconfig xsigo on.
See Installation Procedure on page 191.
Optional LUN Masking: No Report LUN Interception on page 85.
Default Gateway on page 50.
Xsigo Systems VP780
2
Chapter 1: New Features and Enhancements
Table 1 Release 1.5 Additions and Modifications (continued)
Feature Description
Identity Management System on page 171.
The shaper service was removed from the feature Network QoS for vNICs on page 101.
Statistics: Realtime vs Historical. See Statistics on page 44.
XgOS CLI Overview on page 3.
Configuration Save and Restore on page 38.
Xsigo Systems VP780
XgOS CLI Overview
3
Introduction
The Xsigo Operating System (XgOS) is installed from a Xsigo Package File (XPF) file stored in compact flash on the
System Controller board. The XgOS provides a Command Line Interface (CLI) ecosystem containing a set of CLI
processes on which users are logged in through the console or SSH:
Console
SSH
CLI
Processes
Servers
I/O Cards
Logs
Scripting
Engine
XgOS
Configuration
Database
File
System
Figure 1 CLI Ecosystem
The CLI enables you to access and influence the following elements:
File System—A file storage system. See “File System” on page 6.
Scripting Engine—Enables you to run scripts within the CLI for each I/O card. The engine also enables you to
define new commands. The user CLI is a skin running on top of the CLI process. See “Scripting Engine” on
page 11.
Hardware—Servers, I/O cards, and system logs. See “InfiniBand Ports” on page 27, “I/O Cards” on page 33, “I/O
Ports” on page 34.
Configuration Database—The information model in the system. See “Configuration Database” on page 15.
See “Virtual Resources Quick Start” on page 18 for getting started with virtual resources.
Xsigo Systems VP780
4
Chapter 2: XgOS CLI Overview
Login Methods
There are two ways to log into the CLI: Console and SSH. Telnet is not supported. Up to 20 concurrent CLI sessions can
be established on the chassis (limited by the number of instances available in the address object).
Console
The console port is the Serial 1 port (top) on the Management module. The Serial 2 port (bottom) is used for engineering
debug purposes only.
Here are the default console serial port settings:
Baud rate: 115200 bps
Data bits: 8
Stop bits: 1
Parity: none
Flow control: none
The default username is “admin”. The default password is “admin”. The XgOS places you directly into a CLI session with
full administrative privileges:
iowa login: admin
Password: <deleted>
Welcome to XgOS
Copyright (c) 2007 Xsigo Systems, Inc.
All rights reserved.
Enter "help" for information on available commands.
admin@iowa[xsigo]
admin@iowa[xsigo] pwd
/home/admin
admin@iowa[xsigo] show user
name
descr roles
role-group
domains domain-group
------------------------------------------------------------------------admin
administrator administrator_group root
root
SSH
Use SSH to log into the CLI remotely. Telnet is not supported:
-bash-3.00$ ssh [email protected]
Password: admin
Welcome to XgOS
See also Root Login on page 187.
Xsigo Systems VP780
5
Login Methods
Display Login Information
Use show login and show users to display details about the active CLI sessions and configured user accounts.
Use set cli idle-timeout 0 to configure an infinite CLI timeout (no timeout). See Create a User on page 167.
Syntax
show login <session-id>
show users
Example
show login
----------------------------------------------------------------session
1
time
2007-08-20 21:28:20
name
admin
descr
roles
administrator
interface
cli
type
local
logged-in-from 172.16.48.120
----------------------------------------------------------------1 record displayed
show users
----------------------------------------------------------------name
admin
descr
roles
administrator
role-group
administrator_group
domains
root
domain-group root
Xsigo Systems VP780
6
File System
The XgOS provides a basic unix-like file system:
admin@iowa[xsigo] ls /
usb
etc
home
skins
config
log
sbin
bin
/usb is the USB port on the Management module.
/bin contains binary files.
/home contains users’ home directories.
/sbin contains system binaries not available to users.
/skins contains skin definitions for the CLI commands. The default skin is the “xsigo” skin. For example, see cat /etc/
skin, /etc/xsigorc, and the XML definition for show cli command show.
Default Login
The default login home directory is /home/admin:
admin@iowa[xsigo] pwd
/home/admin
All user data is stored in the “User data” partition on the hardrive:
admin@iowa[xsigo] show system
...
DISK STATUS
Partition
Size Available
Base OS
253.967M
59.694M
XgOS
1.192G
96.996M
System logs
9.169G
8.300G
Database
8.249G
7.651G
Temporary data
6.040G
5.701G
User data
2.752G
2.581G
Volatile data
184.901M
175.341M
Config data
44.292M
41.967M
Used %used
181.159M 71% |########################|
1.036G 86% |############------|
412.570M
4%
|#-----------------------|
182.848M
2% |------------------------|
32.062M
0% |------------------------|
32.234M
1% |------------------------|
0.014M
0% |------------------------|
0.038M
0% |------------------------|
Xsigo Systems VP780
7
File System
File Operations
The file command enables you to perform a variety of file operations.
Syntax
file copy <from-url> <to-url> [-force][-query][-recursive]
file
file
file
file
archive <dest-file> <src-file1> <src-file2> ...
compress <file>
unarchive <file>
uncompress <file>
file
file
file
file
file
file
file
file
diff <file1> <file2>
edit <filename>
find <filename>
hash <filename>
list
remove <filename>
search [<searchpattern>][-except][-ignorecase][-linenumbers][-recursive]
show <filename>
Parameter Description
file copy <from-url> <to-url> [-force][-query][-recursive]—Copies a file from a source
location to a destination. Replace <from-url> with a URL containing the source location from which the file
will be copied. Replace <to-url> with a URL containing the file-path destination.
All copy schemes have the following general syntax:
scheme://user@host/image-path.xpf
You can omit the “user@” bit if the same user name is available on the server from which you are loading the
XPF file. If the scheme is file:, you can omit the host.
http://<file-path>—Copies using HTTP.
https://<file-path>—Copies using HTTPS.
scp://<file-path>—Copies using SCP.
file://<file-path>—For copying from a file stored locally on the VP780. For example from disk,
USB (a mounted /usb device), or a /home directory.
ftp://<file-path>—Upgrades using FTP.
Note: These schemes are used also by the system upgrade command. See System Image Upgrades on
page 163.
file archive <dest-file> <src-file1> <src-file2> ...—Creates a file archive.
file compress <file>—Compresses a file archive.
file unarchive <file>—Unpacks a file archive.
file uncompress <file>—Uncompresses a file archive.
file diff <file1> <file2>—Displays the difference between two files.
file edit <filename>—Modifies a file.
Xsigo Systems VP780
8
file find <filename>—Finds a file.
file hash <filename>—Calculates the MD5 hash of the file contents.
file list—Displays the list of files.
file remove <filename>—Deletes a file.
file search—Searches files for regular expressions. The following parameters are supported:
<searchpattern>—Regular expression to search for.
-except—Finds everything except the regular expression.
-ignorecase—Ignores case in search.
-linenumbers—Shows line numbers for matching lines.
-recursive—Searches sub-directories.
show <filename>—Displays the contents of a file.
Example 1
To collect debug data for Xsigo customer support by using the redirect function (>):
show tech-support > mydebug
file copy mydebug scp://[email protected]/homes/joeuser/
mydebug.txt
[email protected]'s password:
Copying...
##############################################################################
############ [100%]
Example 2
To create an archive then compress it:
file archive foo.tar file1 file2
file compress foo.tar
Example 3
To find the text “foobar” in the file “myfile” and include the line number:
file search foobar -linenumbers myfile
15:foobarq
Logging
Log files are stored in /log.
admin@iowa[xsigo] ls /log
lost+found
coredumps
btmp
ulog
apache2
wtmp
postgresql
Xsigo Systems VP780
9
File System
news
ntpstats
ulog-acctd
ksymoops
xml
dmesg
user.log
user-debug.log
daemon.log
lastlog
kern.log
ib.log
postgresql.log
createdb.log
osm.log
install.log
apache2.pid
dumpster.log
osinstall.out
osinstall.err
user.log.2.gz
user.log.3.gz
user-debug.log.2.gz
user-debug.log.3.gz
user-debug.log.4.gz
user-debug.log.5.gz
user-debug.log.6.gz
user-debug.log.7.gz
user-debug.log.8.gz
user-debug.log.9.gz
user-debug.log.10.gz
user.log.7.gz
user.log.8.gz
user.log.4.gz
user.log.5.gz
user.log.6.gz
user.log.9.gz
osm.log.2.gz
user.log.10.gz
user.log.1.gz
osm.log.1.gz
user-debug.log.1.gz
The last bootup data of the chassis is stored in “dmesg”:
cat /log/dmesg
Standard syslog goes to “user.log”, where log rotation and auto-archive occurs for up to 10 gziped files:
user.log
user.log.1.gz
user.log.2.gz
user.log.3.gz
user.log.4.gz
user.log.5.gz
user.log.6.gz
user.log.7.gz
Xsigo Systems VP780
10
user.log.8.gz
user.log.9.gz
user.log.10.gz
The format of a log message is:
<date> <time> <hostname> <module>[<process-id>]: [<mssg-level>]
<object>::<text-message>
Example:
Jun 6 00:00:01 iowa vnicmanager[12532]: [ERR] VNIC::VNICManager
process_simm_message: ENTRY
User debugging goes to “user-debug.log” where log rotation also occurs automatically:
user-debug.log
user-debug.log.2.gz
user-debug.log.3.gz
user-debug.log.4.gz
user-debug.log.5.gz
user-debug.log.6.gz
user-debug.log.7.gz
user-debug.log.8.gz
user-debug.log.9.gz
user-debug.log.10.gz
Core dump files are stored in /log/coredumps:
systemcontrolle.8210.core
Crash dump files are text files that contain information from things that have crashed on the different processors (fpp, iop,
scp):
chassisCtr.449-fpp-XF1ZK142LZ-1.1.crash
Configuration Files
The /config directory contains the XML versions of the configuration:
admin@iowa[xsigo] ls /config
config-imported.xml
config.xml
config-save0.xml
config-save1.xml
config-save2.xml
config-save3.xml
config-save4.xml
config-save5.xml
config-save6.xml
config-save7.xml
config-save8.xml
Xsigo Systems VP780
11
Scripting Engine
config-save9.xml
The current configuration is “config.xml”, which is a symbolic link to “config-save1.xml”:
admin@iowa[xsigo] ls -l /config/
-rw-r--r-- xsigo 90174 Mon Jul 9 19:52:20 GMT 2007 config-imported.xml
-rwxrwxrwx xsigo 16 Fri Aug 3 21:16:11 GMT 2007 config.xml -> config-save1.xml
These configuration files can be imported using system import -xml <filename>.
Scripting Engine
The XgOS provides many scripts in /bin working as simplified unix commands:
admin@iowa[xsigo] ls /bin
pwd
grep
testsuite
ls
printevents
showlog
stress
cd
cat
chmod
sedit
mkdir
rm
mv
Aikido
All onboard scripts were created using the Aikido Language System. Aikido is an interpreted, dynamically typed
language that can be used for general purpose programming but is best suited for prototyping and scripting. It has been
derived from the ideas present in a large number of languages including Pascal, Ada, C, C++, JavaTM, JavaScript, and
Verilog.
See help scripts for more information about the use of Xsigo scripts.
See the following sites for more information on Aikido. Specifically, the Aikido Programming Language Reference
Manual:
http://sourceforge.net
http://en.wikipedia.org/wiki/Aikido_(programming_language)
Example 1: Creating 10 vNICs Using Aikido
Foo@bar[xsigo] foreach I 10
> add vnic vnic${I}.beach 5/2
> end
Using the Aikido scripting language, this example creates 10 vNICs called vnic0..vnic9 on the server-profile beach.
Example 2: mv
cat /bin/mv
Xsigo Systems VP780
12
#> Rename files
/*
* (C) 2004,2005 XSIGO SYSTEMS Inc. All rights reserved. This material may not be
* reproduced, displayed, modified or distributed without the express prior
written
* permission of the copyright holder.
*
* Author: David Allison
* Email: [email protected]
*
* $Id$
* $Date$
* $Revision$
* $Author$
*
* Description :
*/
if (args.size() < 2) {
throw "usage: mv file... dest"
}
var allfiles = []
for (var i = 0 ; i < args.size() - 1; i++) {
var files = glob (args[i])
foreach file files {
allfiles.append (file)
}
}
var dest = args[args.size() - 1]
var s = System.stat (dest)
var movetodir = false
if (s != null) {
if (s.S_ISDIR()) {
movetodir = true
}
}
if (allfiles.size() != 1 && !movetodir) {
throw "mv: Cannot move multiple files to a non-directory"
}
foreach file allfiles {
println ("moving " + file + " to " + dest)
if (movetodir) {
var destname = dest + "/" + Filename.filename (file)
System.rename (file, destname)
} else {
System.rename (file, dest)
}
Xsigo Systems VP780
13
Scripting Engine
}
Example 3: Need fun?
Play the onboard Xsigo games using the play command:
play [invaders | snake | sudoku]
They runs as Aikido scripts under /bin/stress:
admin@iowa[xsigo] ls /bin/stress/
sudoku.aikido
invaders.aikido
snake.aikido
admin@iowa[xsigo] /bin/stress/snake.aikido
SEDIT Script Editor
The Script Editor (SEDIT) is a simple but powerful onboard text editor that runs from within the CLI.
Syntax
There are two ways to start SEDIT and open a file:
sedit <filename>
file edit <filename>
Example
This example redirects (>) the output of show system to a file named “foo”, then uses file edit <filename> to
start the editor and open the file:
admin@iowa[xsigo] show system > foo
admin@iowa[xsigo] sedit foo
Command summary:
^w
^d
^f
^g
^p
...
write file (save)
quit editor
find regular expression
find next
for help
SEDIT runs as a script named sedit:
admin@iowa[xsigo] file edit /bin/sedit
See help sedit for documentation:
admin@iowa[xsigo] help sedit
Xsigo Systems VP780
14
Create Your Own Commands
Use the Xsigo Script Editor to create your own commands (scripts) and aliases.
Example:
Step 1
Use file edit to create and open a file:
file edit who
The Xsigo Script Editor starts.
Step 2
Define the behavior:
1
Step 3
show user
Save the file and exit the editor:
ctrl-w
ctrl-d
Step 4
Set the file access permissions and make the file executable:
chmod +x who
Step 5
Test the command:
admin@iowa[xsigo] who
----------------------------------------------------------------name
admin
descr
roles
administrator
role-group
administrator_group
domains
root
domain-group root
Xsigo Systems VP780
15
Configuration Database
The configuration database (config-db) is the information object model in the system:
Information Model
add
set
remove
show
system import
system export
Hard Disk
Figure 2 Information Model
The config-db’s behavior is influenced by the commands add, remove, set, and show. These commands modify the
configuration of the system. In contrast other commands, such as the system command, are not part of the information
model because they use objects (not modify) inside the information model. For example, system import and system
export.
See Virtual Resources Quick Start on page 18 for getting started with virtual resources.
Objects
An object is something you add use in the system. In general, no objects are added by default. You must add them
manually. The add command adds a new object with its default settings applied.
In user CLI mode, issue the add ? command to display an abstraction of the objects in the config-db. In expert CLI mode
(set cli mode expert), the actual object names are exposed:
admin@iowa[xsigo] add ?
Possible completions:
acl
Add an ACL
domain
Add a domain
domain-group
Add a domain group
group
Add a server group
iocard
Provision an IO card
pool
Add a server pool
pool-member
Add a pool member
qos
Add a QoS profile
san
Add SAN information
server-profile
Add a server profile
template
Add a server template
Xsigo Systems VP780
16
user
vhba
vlan
vm
vnic
vssl
vssl-application
vswitch
Add
Add
Add
Add
Add
Add
Add
Add
a
a
a
a
a
a
a
a
local user
virtual HBA
VLAN to a VNIC
Virtual machine
virtual NIC
virtual SSL
virtual SSL application
Virtual Switch for VM use
Common virtual resources to add are server-profile, vnic, vhba, and vssl:
add server-profile smoothb ceasar@iowa:ServerPort24
The same object cannot be added twice. If you do, an error is generated:
add server-profile smoothb ceasar@iowa:ServerPort24
server-profile smoothb already exists
In general, no objects are added by default. The one exception is QoS with its default profiles (see “Default Set Profiles”
on page 104).
Use show commands to display configuration settings in table format (default):
show server-profile smoothb
name
state descr phys-cons
hypervisor vnics vhbas vssls
--------------------------------------------------------------------------smoothb up/up
ceasar@iowa:ServerPort24 none
0
0
0
1 record displayed
Use the -list parameter to display the output in list format:
show -list server-profile smoothb
--------------------------------------------------------------------------name
smoothb
state
up/up
descr
phys-cons
ceasar,ServerPort24
hypervisor none
vnics
0
vhbas
0
vssls
0
--------------------------------------------------------------------------1 record displayed
Use the -xml parameter to use display the output in XML format:
show -xml server-profile smoothb
<table>
<row number="0">
<cell name="name" value="smoothb"/>
<cell name="state" value="up/up"/>
<cell name="descr" value=""/>
<cell name="phys-cons" value="ceasar,ServerPort24"/>
<cell name="hypervisor" value="none"/>
<cell name="vnics" value="0"/>
<cell name="vhbas" value="0"/>
<cell name="vssls" value="0"/>
</row>
</table>
Xsigo Systems VP780
17
Use a set command to change the default value of an object that has been added to the system. The set command is
used for setting attributes and performing actions. Optional qualifiers are prefixed by a dash (-) and end with an equal
sign (=);
set server-profile smoothb -descr="My first server profile”
show -list server-profile smoothb
--------------------------------------------------------------------------name
smoothb
state
up/up
descr
My first server profile
phys-cons
ceasar,ServerPort24
hypervisor none
vnics
0
vhbas
0
vssls
0
--------------------------------------------------------------------------1 record displayed
If an object has not been added (add), it cannot be set:
set server-profile yoyo -descr="My first server profile"
server-profile yoyo does not exist
System Configuration
Issue the show config command to display the running configuration in table format. There is also an XML version of
the configuration file in /config/config.xml.
The config.xml file is large and not very parseable onboard the chassis. Use file copy to copy config.xml to some
remote location and read the file with an XML reader.
Syntax
show config
cat /config/config.xml
Example 1
show config
#
# Xsigo System Configuration
# Model: VP780
# Serial: 050610240
#
# Date: Mon Aug 20 23:49:32 GMT 2007
# User: admin
...
Example 2
cat /config/config.xml
## VHBA Targets
############################################################################
<top:System xmlns:top="http://www.xsigo.com/services/xmlapi/top"
xmlns:xsigo="http://www.xsigo.com/services/xmlapi/xsigo"
xsigo:version="Migrated to Version 1.0.1 Build unknown" displayedName="iowa">
Xsigo Systems VP780
18
<application:Manager xmlns:application="http://www.xsigo.com/services/
xmlapi/application"/>
<config:Manager xmlns:config="http://www.xsigo.com/services/xmlapi/
config">
<config:BackupPolicy xmlns:config="http://www.xsigo.com/services/
xmlapi/config" displayedName="config-save" exportPassword="$2$~">
<trigger:TriggerMember xmlns:trigger="http://www.xsigo.com/
services/xmlapi/trigger" displayedName="commit-count-trigger-default"
triggerDn="system-local:trigger:commit-count-trigger-default"/>
...
Virtual Resources Quick Start
This section provides a brief introduction to the commands used to configure and monitor virtual resources on the system.
Basic Commands
There are several fundamental commands that influence the configuration database and perform basic system functions:
Create and delete virtual resources:
add
remove
Modify and display properties of virtual resources:
set
show
Set chassis related properties:
system
Server Profile Commands
Server profiles are containers that hold vNICs/vHBAs and are assigned to physical servers. Profiles provide the flexibility
to move an I/O personality from one physical server to another.
Examples:
Create a server profile for xserver1 and assign it to the physical server:
add server-profile xserver1 xserver1@iowa:ServerPort7
Display the properties of all server profiles:
show server-profile
Delete the server profile:
remove server-profile xserver1
Assign an existing server profile to a server:
set server-profile xserver1 connect xserver1@iowa:ServerPort7
See “Server Profiles and Templates” on page 47 for more information.
Xsigo Systems VP780
19
vNIC Commands
vNICs are given a name and assigned to a server profile and an Ethernet module port.
Examples:
Create a new vNIC for xserver1 and assign it to port 2 on the Ethernet module in slot 8:
add vnic vnic0.xserver1 8/2
Give vnic0 on xserver1 the IP address of 11.0.0.1 with netmask 255.255.255.0:
set vnic vnic0.xserver1 -addr-type=static -ip-addr=11.0.0.1/24
Display the properties of all vNICs:
show vnic
Change the netmask on vnic0.xserver1 to 255.0.0.0:
set vnic vnic0.xserver1 –netmask=255.0.0.0
Set vnic0.xserver1 to DHCP:
set vnic vnic0.xserver1 –addr-type=dhcp
Change the termination port of a vNIC:
set vnic vnic0.xserver1 –if=8/4
Create an HA vNIC with primary port 8/1 and secondary port 8/2:
add vnic vnic0.xserver1 8/1 ha 8/2
Delete a vNIC:
remove vnic vnic0.xserver1
See “vNICs” on page 53 for more information.
vHBA Commands
vHBAs are given a name and assigned to a server profile and a Fibre Channel (FC) module port.
Examples:
Create a new vHBA for xserver1 and assign it to port 1 on the FC module in slot 15:
add vhba vhba0.xserver1 15/1
Display the targets and LUN IDs the vHBA can detect:
show vhba vhba0.xserver1 targets
Display the properties of all vHBAs (WWNN/WWPN):
show vhba
Xsigo Systems VP780
20
Request a vHBA to rescan the SAN fabric:
set vhba vhba0.xserver1 rescan
You would do this if you changed LUN masking on an array, for example.
Note
vHBA prescan commands allow an “unbound” vHBA to perform an NPIV login and “see” the available targets and
LUNs. You can only perform these commands on a vHBA and server-profile that is not assigned to a physical server.
You can check this by typing show server-profile and make sure the state is “up/unassigned”.
Examples:
Create a server profile and vHBA to scan the fabric:
add server-profile testserver
add vhba vhba0.testserver 15/1
show vhba vhba0.testserver (view WWPN to provision LUNs)
Request an unbound vHBA to perform an NPIV login:
set vhba vhba0.testserver prescan
If you change LUN masking or if the fabric changes without an RSCN, you must logout/login to “rescan”:
set vhba vhba0.testserver remove-prescan
set vhba vhba0.testserver prescan
Request an unbound vHBA to logout of the SAN fabric:
set vhba vhba0.testserver remove-prescan
See “vHBAs” on page 67 for more information.
I/O Card Commands
The I/O modules and ports are the termination points of vNICs and enable vNICs to access network resources.
Examples:
Display all I/O cards in the chassis and their status:
show iocard
Display the port status of all I/O cards in the chassis:
show ioport
Change the MTU of an I/O port to support jumbo frames:
set ethernet-port 8/4 –mtu=9194
Note: You can only change the MTU of a port when no vNICs are assigned:
Display the MTU of an I/O port:
show ioport 8/4
Xsigo Systems VP780
21
Misc Show Commands
Display the XgOS version:
show system version
Display management Ethernet info:
show system interfaces
Display all logged in users:
show login
Display environmental information:
show hardware
Display information for supporting an issue:
show tech-support
Display discovered physical servers:
show physical-server
Troubleshooting Vstars
Vstars are virtual resources, such as vNICs, vHBAs, and vSSLs.
On the chassis:
show diagnostics vstar
The “Vinstall” column indicates when a vstar has been successfully installed (“Inst_ok”) on the host. When a
vstar is not installed, the field “Inst_pending” is displayed.
On the host server:
/opt/xsigo/bin/xsigo-support
Xsigo created a debug tool called xsigo-support. This script collects and dumps information for monitoring and
troubleshooting activity on the host server. See also Debug Tool: xsigo-support on page 198.
Xsigo Systems VP780
22
Set CLI Attributes
The set cli command configures different attributes of the CLI itself.
Syntax
set
set
set
set
set
set
set
set
set
set
set
set
set
cli
cli
cli
cli
cli
cli
cli
cli
cli
cli
cli
cli
cli
autocommit {off | on} [-noconfirm]
block-entry {off | on}
cols <number>
rows <number>
confirm {off | on}
echo {off | on}
idle-timeout <minutes>
mode {expert | user | xml}
paging {off | on}
progress-bar {off | on}
prompt {custom <value> | normal}
space-completion {off | on}
wrap {off | on}
autocommit {off | on} [-noconfirm]—The default is on. When a CLI command is complete, the system
automatically commits the changes to the configuration database. When set to off, any changes must be
manually written to the database using the commit command. The off option is useful for creating a set of
changes and then committing them as a group. Autocommit is disabled for ACLs on 10GigE cards (see add
acl).
block-entry {off | on}—Controls whether the CLI prompts for the entry of scripting blocks such as
“foreach”, etc.
cols <number>—Sets the number of columns on the screen. The default is 106 columns.
rows <number>—Sets the number of rows on the screen. The default is 54 rows.
confirm {off | on}—Sets the CLI confirmation mode. If the mode is set to on, the CLI confirms dangerous
commands.
echo {off | on}—Displays all CLI communication. The on option will echo all commands to the terminal
screen. The default is off.
idle-timeout <minutes>—After this many idle minutes, your CLI session will timeout. Configure a value
of “0” to configure an infinite CLI timeout (no timeout).
mode {expert | user | xml}—Controls the CLI mode. The default is user. See show cli mode.
paging {off | on}—Sets the CLI paging mode. When on, the display output stops when the screen is full.
When paging mode is off, the output does not stop at the end of the page.
progress-bar {off | on}—Determines if a progress bar is displayed on the screen for commands that are
expected to take a long time to execute.
prompt {custom <value> | normal}—Controls the current CLI prompt mode. The custom keyword sets
the prompt to be an arbitrary CLI expression. The normal keyword sets the prompt to be the full name of the
current object, such as “admin@chassis[xsigo]”.
Xsigo Systems VP780
23
Display CLI Attributes
space-completion {off | on}—Controls whether the CLI will complete commands when the space-bar is
pressed or not. The default is on.
wrap {off | on}—Controls whether the CLI will wrap text at the end of line or not. The default is on.
Example
set cli echo on
add server-profile foo
add server-profile foo
add server virtual "foo"
// if a template was specified, apply it now
top
commit noconfirm
set cli echo off
set cli echo off
add server-profile gogo
Display CLI Attributes
Use the show cli command to display different attributes of the CLI itself.
Syntax
show
show
show
show
show
show
show
show
show
show
show
show
show
show
show
show
show
show
cli
cli
cli
cli
cli
cli
cli
cli
cli
cli
cli
cli
cli
cli
cli
cli
cli
cli
autocommit
block-entry
cols
rows
confirm
command [<name>]
echo
history
idle-timeout
keys
loaded-commands
mode
paging
progress-bar
prompt
space-completion
user
wrap
Example 1
show cli mode
user
show cli autocommit
on
Xsigo Systems VP780
24
User mode is the default CLI mode on the system. All CLI commands are auto committed by default.
Example 2
set cli idle-timeout 0
show cli idle-timeout
The idle timeout is disabled
System Control
Use the system command to control various system attributes.
Syntax
system
system
system
system
system
system
system
system
system
system
system
system
system
broadcast <message>
cancel {restart | shutdown}
clear {config | garbage | logs}
cold-restart <message> [-delay=<sec>][-force][-noconfirm][-now]
warm-restart <message> [-delay=<sec>][-force][-noconfirm][-now]
downgrade [<args>][-noconfirm]
flush ims
install [license <key>][ssh-key <key>]
logout <session> <message>
shutdown <message> [-delay=<sec>][-force][-noconfirm][-now]
unmount usb
upgrade <url> [-noconfirm] [<args>]
verify
broadcast <message>—Sends a message to all CLI users who are logged in.
cancel {restart | shutdown}—Cancels a pending operation.
clear {config | garbage | logs}—The garbage option removes garbage, such as failed image installs,
from the disk. Prior to Release 1.0, Xsigo recommended that users clear the config and perform a cold-restart after
an upgrade. After Release 1.0, this procedure is not required.
cold-restart <message> [-delay=<sec>][-force][-noconfirm][-now]—Restarts the system and
removes power from the I/O cards. Prior to Release 1.0, Xsigo recommended that users clear the config and
perform a cold-restart after an upgrade. After Release 1.0, this procedure is not required.
warm-restart <message> [-delay=<sec>][-force][-noconfirm][-now]—Restarts the system
without removing power from the I/O cards. A warm-restart completes faster than a cold-restart.
downgrade [<args>][-noconfirm]—Downgrades to the previously installed image (will destroy current
image).
flush ims—Flushes the Identity Management System (IMS) data. See Identity Management System on
page 171.
install [license <key>][ssh-key <key>]—Install software on the system.
logout <session> <message>—Forces a user to logout (administrator only).
Xsigo Systems VP780
25
System Control
shutdown <message> [-delay=<sec>][-force][-noconfirm][-now]—Shuts down the system.
unmount usb—Unmounts a USB token. Under normal conditions, the system can mount and unmount a USB
file system without requiring this command.
upgrade <url> [-noconfirm] [<args>]—Upgrades the XgOS image. See “System Image Upgrades” on
page 163 for more information.
verify—Verifies the integrity of the installation.
Example 1
system broadcast The Red Sox beat the Yanks
Message received from admin at Mon Aug 13 21:51:02 GMT 2007
Broadcast message
The Red Sox beat the Yanks
Example 2
To perform an immediate cold restart of the system:
system cold-restart
Are you sure you want to restart the system (y/n)?y
***********************************
Xsigo system is being shut down now
***********************************
Connection to iowa closed.
Xsigo Systems VP780
26
Slot/Port Numbering Scheme
InfiniBand
Ports
I/O Ports
1
2
3
4
13
14
15
16
17
18
19
20
21
22
23
24
1
2
3
4
5
6
7
8
9
10
11
12
1
MGT
AUX
SERIAL-1
SERIAL-2
USB
1
2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
MGT
I/O Slots
Figure 3 VP780 Slot/Port Numbers
For example, you must specify a specific slot and port to add a vNIC:
admin@iowa[xsigo] add vnic foo.bar ?
4/1
nwEthernet1GbPort in
4/2
4/3
4/4
Repeat '?' for detailed help.
admin@iowa[xsigo] add vnic foo.bar 4/1
slot
slot
slot
slot
4
4
4
4
port
port
port
port
1
2
3
4
(up) used by 3 resources
(down) unused
(down) unused
(down) unused
Xsigo Systems VP780
27
InfiniBand Ports
InfiniBand Ports
InfiniBand (IB) is a channel based, switched-fabric interconnect for servers. IB interconnects processor nodes and I/O
nodes to a system area network. The architecture is independent of the host operating system and processor platform.
The Xsigo VP780 contains several internal 24-port IB switches (Mellanox). One switch attaches to an internal HCA
(IOCPort16). Each external IB port connects to a external HCA installed on a remote host server. An external Xsigo IS24
IB switch can be connected to the VP780 to extend the number of IB ports:
Xsigo IS24
IB
IB
IB
Blade 3
Blade 2
Blade 1
24 Host IB Ports
HCA HCA
I/O Modules
HCA HCA
... ... ...
Xsigo VP780
HCA HCA
Internal IB Switch
HCA HCA
Blade Server
Host Servers
Figure 4 Sample InfiniBand Topology
The VP780 contains an embedded Subnet Manager (SM) that manages the switching and pathing tables within the IB
fabric. When there are multiple SMs on a subnet, one SM will be the master SM through an election algorithm.
The remaining SMs become standby SMs. There is only one master SM per subnet.
The master SM is a key element in initializing and configuring an IB subnet. The master SM is elected as part of the
initialization process for the subnet and is responsible for the following:
•
Discovering the physical topology of the subnet
•
Assigning Local Identifiers (LIDs) to the end nodes, switches, and routers
•
Establishing possible paths among the end nodes
•
Sweeping the subnet, discovering topology changes and managing changes as nodes are added and deleted.
The communication between the master SM and the SM agents, and among the SMs, is performed with subnet
management packets.
If you prefer to use a 3rd-party SM (not the Xsigo chassis), see OpenSM Decoupling on page 157 for information on how
to disable the SM.
The IB specification is posted at http://www.infinibandta.org/specs/register/publicspec/
Note
Xsigo Systems VP780
28
Syntax
Use the following CLI commands to display and manage Infiniband port information:
set fabric-port <port> [counters | down | up] [-noconfirm]
set diagnostics opensm-param [log-level <value>][priority <value>][resweep]
show
show
show
show
show
show
show
fabric-port
ib ports
diagnostics opensm-param
diagnostics ib-topo
diagnostics sm-info
diagnostics switch-version
physical-server [<name>][*]
Example 1
The Xsigo host drivers communicate with Xsigo’s OpenSM by default. When an IB connected host server boots up, the
installed Xsigo host driver advertises the server’s host name to the VP780.
Issue show physical-server to display the list of IB connected servers:
show physical-server
name
guid
state descr port
cap server-profile pool-name
--------------------------------------------------------------------------------alexander 2c90200204cbd up
iowa:ServerPort8 efsdefault
The alexander server is connected to the VP780 named “iowa” on IB port 8 (iowa:ServerPort8).
When you issue add server-profile <name>, you will see the reported host server names for which command
completion can configure:
add server-profile myprofile ?
alexander,iowa:ServerPort8 Connection to host alexander (up)
Example 2
show fabric-port
-----------------------------------------------------------------------------name
teton
type
hcaPort
descr
chassis-port ServerPort19
id
2c90200204929
admin_state
N/A
oper-state
up
m-key
0
lid
4
sm-lid
61
link-width
4x
link-speed
2_5_Gbps
-----------------------------------------------------------------------------...
Xsigo Systems VP780
29
InfiniBand Ports
-----------------------------------------------------------------------------name
south-dakota
type
hcaPort
descr
chassis-port IOCPort16
id
1397020100013d
admin_state
N/A
oper-state
up
m-key
0
lid
61
sm-lid
61
link-width
4x
link-speed
2_5_Gbps
-----------------------------------------------------------------------------36 records displayed
Table 1 Field Descriptions for show fabric-port
Field
Description
name
Displayed host name of the server.
type
Type of port.
name
Port GUID name.
descr
User defined port description.
chassis-port
Local IB chassis port used for the connection.
The VP780 itself has an internal HCA on the SCP used to communicate with
the IB fabric. This internal HCA switch port is IOCPort16. This port is the
VP780’s representation in the IB framework.
id
Globally Unique Identifier (GUID). A persistent number that uniquely
identifies a device or component. An HCA is assigned a node GUID that is
stored in flash memory. Each port on an HCA is assigned a port GUID.
Xsigo’s IB vendor ID is 1397.
admin-state
Administrative state of the local IB port on the chassis.
oper-state
Operational state of the local IB port on the chassis.
m-key
Management key. A construct that is contained in Infiniband Architecture
(IBA) management datagrams to authenticate the sender to the receiver.
Xsigo Systems VP780
30
Table 1 Field Descriptions for show fabric-port
Field
Description
lid
Local Identifier. An address assigned to a port by the IB Subnet Manager
(SM), unique within the subnet, used for forwarding packets within the subnet.
The SM manages the switching and routing tables with the IB fabric.
The Source and Destination LIDs are present in the Local Route Header. A
Local Identifier is formed by the sum of the Base LID and the value of the
Path Bits. Unlike a fixed GUID, a LID can change from time-to-time.
sm-lid
The LID where the master SM is located. It is not the SM priority value.
link-width
link-speed
Link-width is the number of physical lanes (1, 4, 8, or 12) whereas link-speed
is the speed of the physical lanes (e.g. 2.5 Gbps (SDR), 5 Gbps (DDR), or 10
Gbps (QDR). If the link-width field is not 4x, there is something wrong.
The InfiniBand Architecture (IBA) defines a number of different link bit rates.
The lowest bit rate of 2.5 Gb/sec is referred to as a 1x (times one) link. Other
link rates are 10Gb/sec (4x) and 30 Gb/sec (1x2).
Example 3
show diagnostics opensm-param
OpenSM $ Current log level is 0x3
OpenSM $ Current sm-priority is 0
OpenSM $
OpenSM Version
: OpenSM Rev:openib-3.0.14-xsigo2
SM State/Mgr State : Master/Idle
SA State
: Ready
Routing Engine
: null (min-hop)
MAD stats
--------QP0 MADS outstanding
: 0
QP0 MADS outstanding (on wire) : 0
QP0 MADS rcvd
: 112540
QP0 MADS sent
: 112540
QP0 unicasts sent
: 8617
QP1 MADS outstanding
: 0
QP1 MADS rcvd
: 985875
QP1 MADS sent
: 0
Subnet flags
-----------Ignore existing lfts
Subnet Init errors
In sweep hop 0
Moved to master state
First time master sweep
Coming out of standby
:
:
:
:
:
:
0
0
0
0
0
0
Xsigo Systems VP780
31
InfiniBand Ports
Example 4
show diagnostics ib-topo
IB Subnet Topology: 20 HCAs, 12 TCAs 3 switches
Type Name
Node Guid
Port Port Guid
Lid OperState portName
--------------------------------------------------------------------------------HCA
massive 2C902002048FC 1
2C902002048FD 83
ACTIVE
foo:ServerPort17
HCA
teton
2C90200204928 1
2C90200204929 4
ACTIVE
foo:ServerPort19
HCA
bar
2C90200204930 1
2C90200204931 35
ACTIVE
foo:ServerPort20
...
TCA XsigoTCA:13970301 13970301000080 1 13970301000081 101 ACTIVE foo:IOCPort14
...
Switch Xsigo Core Switch
13970101000134 1
13970101000134 58 ACTIVE
Example 5
To determine the Mellanox firmware version installed on the VP780:
show diagnostics switch-version
Query:
FW Version:
Node GUID:
System Image GUID:
Node Description:
Board Serial Number:
PSID:
1.0.0
0x0013970101000123
0x0013970101000123
Xsigo Core: INI Ver: 1.0.0.2
050610240
XG_WHSK_CR_01
Query:
FW Version:
Node GUID:
System Image GUID:
Node Description:
PSID:
1.0.0
0x0013970102000123
0x0013970102000123
Xsigo Leaf 1: INI Ver: 1.0.0.2
050610240
XG_WHSK_L1_01
Query:
FW Version:
Node GUID:
System Image GUID:
Node Description:
PSID:
1.0.0
0x0013970103000123
0x0013970103000123
050610240
XG_WHSK_L2_01
To login as root and discover the IB topology:
-bash-3.00$ ssh root@iowa
Password:
iowa:~# ibnetdiscover | grep Switch
Switch 24 "S-0013970102000123"
port 0 lid 3 lmc 0
Xsigo Systems VP780
# "Xsigo Leaf 1: INI Ver: 1.0.0.2" base
32
Switch 24 "S-0013970101000123"
port 0 lid 2 lmc 0
Switch 24 "S-0013970103000123"
port 0 lid 4 lmc 0
# "Xsigo Core: INI Ver: 1.0.0.2" base
# "Xsigo Leaf 2: INI Ver: 1.0.0.2" base
To display the firmware versions installed:
iowa:~# ibsadmin
----------------------------Leaf1 switch firmware version
----------------------------Query:
FW Version:
1.0.0
Node GUID:
0x0013970102000123
System Image GUID:
0x0013970102000123
Node Description:
050610240
PSID:
XG_WHSK_L1_01
----------------------------Leaf2 switch firmware version
----------------------------Query:
FW Version:
1.0.0
Node GUID:
0x0013970103000123
System Image GUID:
0x0013970103000123
Node Description:
050610240
PSID:
XG_WHSK_L2_01
----------------------------Core switch firmware version
----------------------------Query:
FW Version:
Node GUID:
System Image GUID:
Node Description:
PSID:
1.0.0
0x0013970101000123
0x0013970101000123
Xsigo Core: INI Ver: 1.0.0.2
050610240
XG_WHSK_CR_01
To use the binary that calls from the CLI:
iowa:~# ibsadmin -b
Core IB switch firmware is up to date
Leaf2 IB switch firmware is up to date
Leaf1 IB switch firmware is up to date
Xsigo Systems VP780
33
I/O Cards
I/O Cards
Use show iocard to display available I/O line card information in the system.
There are feature differences and capability nuances between the 10GigE and QuadGigE I/O hardware modules. For more
details, see “QoS Feature Matrix” on page 102 , “ACLs” on page 119, and VLANs on page 64.
Syntax
show
show
show
show
show
show
show
show
show
show
show
show
show
show
show
show
show
iocard
iocard
iocard
iocard
iocard
iocard
iocard
iocard
iocard
iocard
iocard
iocard
iocard
iocard
iocard
iocard
iocard
*
<slot>
<slot> acl-stats [-timefilter=[<hours> | all | lastday | lasthour]]
<slot> alarms [-syslog]
<slot> dmesg [-timefilter=[<hours> | all | lastday | lasthour]]
<slot> errors [-syslog]
<slot> fabric-port [-counters]
<slot> ioport [<port>]
<slot> ioports [<port>][-timefilter=[<hours>|all|lastday|lasthour]]
<slot> qos [-timefilter=[<hours> | all | lastday | lasthour]]
<slot> scheduler [-timefilter=[<hours> | all | lastday | lasthour]]
<slot> stats [-timefilter=[<hours> | all | lastday | lasthour]]
<slot> vhbas [-timefilter=[<hours> | all | lastday | lasthour]]
<slot> vnics [-timefilter=[<hours> | all | lastday | lasthour]]
<slot> vssls [-timefilter=[<hours> | all | lastday | lasthour]]
<slot> warnings [-syslog]
Example
show iocard
slot
state
descr
type
v-resources
-----------------------------------------------------------------------------1
up/up
sanFc2Port4GbCard
2
4
up/up
nwEthernet4Port1GbCard
7
8
up/up
sanFc2Port4GbCard
0
13
up/up
sslNonProxy
0
4 records displayed
The field “v-resources” indicates the number of virtual resources (vNICs, vHBAs, vSSLs) that are associated with this
card. vNICs can be bound only to network Ethernet cards. vHBAs can be bound only to SAN FC cards.
Xsigo Systems VP780
34
I/O Ports
Use show ioport to display I/O port information on an I/O card.
Syntax
show
show
show
show
show
show
show
show
show
show
ioport
ioport
ioport
ioport
ioport
ioport
ioport
ioport
ioport
ioport
*
<slot/port>
<slot/port>
<slot/port>
<slot/port>
<slot/port>
<slot/port>
<slot/port>
<slot/port>
alarms
errors
qos
stats
vhbas
vnics
warnings
Example
show ioport
name
type
state
descr
v-resources
------------------------------------------------------------------------1/1
sanFc1GbPort
up/down
0
1/2
sanFc1GbPort
up/down
0
4/1
nwEthernet1GbPort
up/up
0
4/2
nwEthernet1GbPort
up/up
3
4/3
nwEthernet1GbPort
up/up
0
4/4
nwEthernet1GbPort
up/up
0
8/1
sanFc1GbPort
up/up
3
8/2
sanFc1GbPort
up/down
0
8 records displayed
show ioport 8/1
------------------------------------------------------------------------name
8/1
type
sanFc1GbPort
state
up/up
descr
wwnn
50:01:39:71:00:00:80:47
wwpn
50:01:39:70:00:00:80:47
rate
AutoNegotiate/2
frame-size
2048/2048
exec-throttle 65535
int-delay
1000
link-down
0
login-retry
8
login-timeout 4
port-down
8
topo
F
loop-delay
5
tape-support
true
vhbas
3
------------------------------------------------------------------------1 record displayed
Xsigo Systems VP780
35
Hardware Status and Environmentals
Hardware Status and Environmentals
Issue the show hardware command to display hardware information and environmental statistics.
Syntax
show hardware
Example
show hardware
#
# Xsigo System Hardware Status
# Model: VP780-CH-SDR
# Serial: 050610240
#
# Date: Wed Nov 14 23:02:30 GMT 2007
# User: admin
#
## IO Card Version status ########################################
slot type
model
serial
vchip-ver xt-ver
primaryboot-ver secondary-boot-ver diag-ver
------------------------------------------------------------------1 sanFc2Port4GbCard
VP780-4GFC-2P 080610379 1.0.18895 1.0.19841 2.00.07
2.00.05
1.15
4 nwEthernet4Port1GbCard VP780-1GE-4P 040711066 1.0.19623 1.0.19841 2.00.07
2.00.05
1.15
8 sanFc2Port4GbCard
VP780-4GFC-2P 160610506 1.0.18895 1.0.19841 2.00.07
2.00.05
1.15
13
sslNonProxy
VP780-SSL
080610294 1.0.18056 1.0.19
2.00.07
2.00.05
1.15
4 records displayed
## IO Card Environment status ####################################
slot type
state temperatures voltages
------------------------------------------------------------------1
sanFc2Port4GbCard
up
in=30 out=36 1v2=1.19 1v5=1.50
1v8=1.80 2v5=2.50
2v6=2.59 3v3=3.29
3v3sb=3.29
4
nwEthernet4Port1GbCard up
in=30 out=42 1v2=1.20 1v5=1.48
1v8=1.80 2v5=2.49
2v6=2.58 3v3=3.29
3v3sb=3.29
8
sanFc2Port4GbCard
up
in=31 out=37 1v2=1.19 1v5=1.50
1v8=1.80 2v5=2.50
2v6=2.59 3v3=3.29
3v3sb=3.29
Xsigo Systems VP780
36
13
sslNonProxy
up
in=31 out=45 1v2=1.19 1v5=1.48
1v8=1.78 2v5=2.45
2v6=2.57 3v3=3.27
3v3sb=3.29
4 records displayed
## Front Panel Version status ####################################
model
serial
xt-ver primary-boot-ver secondary-boot-ver diag-ver
------------------------------------------------------------------VP780-FRU-FP XG1AA0277
2.00.07
2.00.05
1.15
1 record displayed
## Front Panel Environment status ################################
state temperatures voltages
------------------------------------------------------------------up
in=29 out=27 1v2=1.19 1v5=1.48 1v8=1.79 2v5=2.48 2v6=2.59
3v3=3.25 3v3sb=3.29 5_d2=4.92
1 record displayed
## Fabric Card status ############################################
name model
serial
state temperatures
voltages
------------------------------------------------------------------1
VP780-FRU-FB XG1AA0291 up
in=32 mid=33 out=33 1v2_1=1.19
1v2_2=1.19
1v2_3=1.19
1v41_ddr=1.
41
1v41_sdr=1.
41 1v8=1.79
3v3=3.29
3v3sb=3.29
1 record displayed
## System Control Processor status ###############################
serial
cpu-usage mem-usage temperatures
voltages
------------------------------------------------------------------133100021 1.86027
32.7062
hd_temp_current=28
hd_temp_maximum=32
hd_temp_minimum=16
1 record displayed
## Power supply status ###########################################
id
descr
state
model
serial
vendor-model
-------------------------------------------------------------------
Xsigo Systems VP780
37
If and if-state
1
up/up
2
up/up
2 records displayed
VP780-FRU-PS
VP780-FRU-PS
RJ3804600
SB2542300
CAR1212FPC0000000
CAR1212FPCXXXX-4A
## Fan status ####################################################
name
descr
state
actual
expected
deviation
------------------------------------------------------------------1/1
up
3840
3900
-60
1/2
up
4080
3900
180
2/1
up
3840
3900
-60
2/2
up
3960
3900
60
3/1
up
3840
3900
-60
3/2
up
3960
3900
60
4/1
up
3720
3900
-180
4/2
up
3960
3900
60
If and if-state
Each slot/port has its own interface (if) with state information (if-state):
show vnic
-----------------------------------------------------------------------------name
myvinc.myserver
state
up/up
mac-addr 00:13:97:01:80:0B
ipaddr
if
4/2
term-grp
if-state up
type
vlans
none
vm
qos
-show vhba
-----------------------------------------------------------------------------name
myvhba.myserver
state
up/up
descr
if
8/1
wwnn
50:01:39:71:00:00:81:02
wwpn
50:01:39:70:00:00:81:02
map
lun-mask
See also the show diagnostics iop command:
show diagnostics iop <slot> <option-keyword>
Xsigo Systems VP780
38
Configuration Save and Restore
Before you perform an XgOS firmware upgrade, Xsigo recommends you export your system configuration to a file.
If your running-config gets lost during an upgrade, at least you can import a saved config. If you import a
configuration, the system migrates the old config to the new.
See System Image Upgrades on page 163 for details on how to upgrade a software image.
Syntax
system export <filename> [-cli -defaults][-xml]
system import <filename> [-cli][-xml]
The -xml is in XML format. The -cli is in expert CLI command format (not user mode format). The default is -xml.
export <filename> [-cli -defaults][-xml]—Exports the running-config to a file. The file can be
saved as CLI format or XML format.
import <filename> [-cli][-xml]—Loads a configuration file into the system. If you import a
configuration, the system migrates the old config to the new. The file can be imported as CLI format or XML
format.
Example
system export myconfig.xml -xml
system import myconfig.xml -xml
Software Information
Use the show software command to display software information.
Syntax
show software
Example
show software
## System status
##############################################################################
Booted on: Fri Aug 10 21:05:28 GMT 2007
uptime: 13 days, 21 hours, 5 minutes, 55 seconds
RECENT UPGRADES AND
Thu Aug 16 22:08:05
Wed Aug 15 21:22:25
Mon Jul 9 19:53:26
DOWNGRADES
GMT 2007: Upgraded to xgos-1.0.1.xpf
GMT 2007: Upgraded to xgos-1.0.1.xpf
GMT 2007: Upgraded to xsigo-1.0.0.xpf
Xsigo Systems VP780
39
Software Information
Thu Jul 5 18:07:42 GMT 2007: Upgraded to xsigo-1.0.0-RC17D.xpf
Fri Jun 29 18:11:15 GMT 2007: Upgraded to xsigo-1.0.0-RC17C.xpf
Current Base OS Version Information
ReleaseNumber: 144
CompatOS:
49
ReleasePerson: root
ReleaseDate:
2007/06/17 15:46:41
ReleaseHost:
chaos
KernelVersion: 2.6.16.24-xsigo-2007061501
Alternative Base OS Version Information
*** No information available
INSTALLED XgOS VERSIONS
Current: xsigos-1.0.1-0
Previous: xsigos-1.0.1
MEMORY INFORMATION
Total memory: 995.504M
Used memory: 363.730M
Free memory: 631.773M
Swap space used: 1.738M
DISK STATUS
Partition
Size Available
Used %used
Base OS
253.998M
60.842M
180.041M 70%
|############################------------|
XgOS
1.192G
470.137M
688.164M 56% |######################-----------------|
System logs
9.169G
8.516G
191.484M
2% |---------------------------------------|
Database
8.249G
7.634G
200.582M
2% |---------------------------------------|
Temporary data
6.040G
5.701G
32.062M
0% |---------------------------------------|
User data
2.752G
2.581G
32.324M
1% |---------------------------------------|
Volatile data
184.901M
175.341M
0.014M
0% |---------------------------------------|
Config data
44.292M
41.969M
0.036M
0% |---------------------------------------|
## Processes
##############################################################################
name
processor slot memory cpu-time num-restarts time-started
-----------------------------------------------------------------------------chassisCtr
fpp
1
19.4648
00:00:14 0
2007-0816 22:09:38.497
vhbaagent
iop
1
4.58203
00:00:08 0
2007-0816 22:10:28.501
Xsigo Systems VP780
40
chassisAgt
16 22:10:28.501
vnicagent
16 22:10:28.501
chassisAgt
16 22:10:28.501
vhbaagent
16 22:10:28.501
chassisAgt
16 22:10:28.501
vssl_agent
16 22:10:54.655
chassisAgt
16 22:10:54.655
xtctrl
16 22:13:23.419
xtctrl
16 22:13:23.419
apache2_prerun.sh
16 22:13:23.419
reap_db
16 22:13:23.419
resurrect_db
16 22:13:23.419
resurrect_sysctl
16 22:13:23.419
in.tftpd
16 22:09:18.479
xtctrl
16 22:10:28.501
opensm
16 22:09:18.479
postmaster
16 22:09:18.479
apache2
16 22:09:18.479
snmpagent
16 22:09:18.479
imagemanager
16 22:09:18.479
xc_xsmp
16 22:09:18.479
xc_xsm
16 22:09:18.479
healthmonitor
16 22:09:18.479
scd
16 22:09:18.479
chassisMgr
16 22:09:18.479
xc_manager
16 22:09:18.479
systemcontroller
16 22:09:18.479
iop
1
14.7305
00:00:07
0
2007-08-
iop
4
6.32422
00:00:06
0
2007-08-
iop
4
14.7344
00:00:07
0
2007-08-
iop
8
4.51953
00:00:07
0
2007-08-
iop
8
14.7773
00:00:07
0
2007-08-
iop
13
4.60547
00:00:05
0
2007-08-
iop
13
14.7383
00:00:06
0
2007-08-
scp
0
00:00:00
0
2007-08-
scp
0
00:00:00
0
2007-08-
scp
0
00:00:00
0
2007-08-
scp
0
00:00:00
0
2007-08-
scp
0
00:00:00
0
2007-08-
scp
0
00:00:00
0
2007-08-
scp
0.589844
00:00:00
0
2007-08-
scp
0.925781
00:00:00
0
2007-08-
scp
2.72656
00:00:22
0
2007-08-
scp
2.85547
00:00:00
0
2007-08-
scp
15.2773
00:00:26
0
2007-08-
scp
16.0898
00:01:04
0
2007-08-
scp
1
4.10938
00:00:04
0
2007-08-
scp
1
12.7422
00:00:05
0
2007-08-
scp
1
15.1719
00:01:12
0
2007-08-
scp
1
15.2031
00:04:33
0
2007-08-
scp
1
15.6016
00:02:53
0
2007-08-
scp
1
16.4609
00:04:06
0
2007-08-
scp
1
16.9062
00:05:07
0
2007-08-
scp
1
18.1641
00:11:12
0
2007-08-
Xsigo Systems VP780
41
System Monitoring
scriptsvc
scp
16 22:09:18.479
vssl_manager
scp
16 22:09:18.479
vhbamanager
scp
16 22:09:18.479
vnicmanager
scp
16 22:09:18.479
mimm
scp
16 22:09:18.479
34 records displayed
1
34.293
00:00:04
0
2007-08-
1
37.8906
00:01:50
0
2007-08-
1
38.0703
00:02:08
0
2007-08-
1
38.3242
00:02:15
0
2007-08-
1
43.2031
00:06:42
0
2007-08-
## Core dumps (in /log/coredumps/)
##############################################################
-rw------- root
124547072
-rw------- root
124018688
-rw------- root
123940864
-rw------- root
199634944
-rw------- root
129167360
admin@iowa[xsigo]
Tue Jul 10 23:03:03 GMT 2007
Mon Jul 16 04:46:36 GMT 2007
Mon Jul 16 07:07:29 GMT 2007
Sun Jul 15 00:05:15 GMT 2007
Tue Jul 10 19:43:32 GMT 2007
System Monitoring
Use the following commands to display various system attributes.
See “Diagnostics” on page 186 for more troubleshooting details.
Syntax
watch {ioports | vnics}
show alarms
show system
show system copyright
show system credits
show system date
show system dmesg
show system errors [-timefilter=[<hours> | all | lastday | lasthour]]
show system info
show system interfaces
show system license
show system log [debug | syslog]
show system loglevel <level>
show system next-boot
show system processes
show system server-connection
show system status
show system syslog
Xsigo Systems VP780
42
show
show
show
show
system
system
system
system
syslog-server
user
version [-all]
warnings [-timefilter=[<hours> | all | lastday | lasthour]]
watch {ioports | vnics}—A dynamic window that displays the realtime performance counters of I/O ports
and vNICs.
show system—Displays a summary of the system attributes: Last boot time, uptime, recent upgrades and
downgrades, current base OS (Linux) version information, installed XgOS versions, memory information, and
hard disk status.
copyright—Copyright information.
credits—Shows those responsible for this product.
date—Shows the current system local date and time.
dmesg—Base OS messages.
errors [-timefilter=[<hours> | all | lastday | lasthour]]—Syslog errors.
info—System information.
interfaces—Shows all the network interfaces in the system.
license—End User License Agreement.
log [debug | syslog]—Shows logs.
loglevel <level>—Shows the syslog level of each individual service.
next-boot—Shows the location from which the system will boot next time.
processes—Shows process information.
server-connection—Server connection information.
status—Shows information on the status of the system.
syslog—Syslog entries.
syslog-server—The syslog server.
user—Shows internal information about the current user.
version [-all]—Shows version information for the system.
warnings [-timefilter=[<hours> | all | lastday | lasthour]]—Syslog warnings
Example 1
watch ioports
IOPORTS measured in mpbs
Wed Nov 14 03:43:59 GMT 2007
name
type
state
v-res
in
in-rate
out
out-rate
-----------------------------------------------------------------------------1/1
sanFc1GbPort
down
0
0
0
0
0
Xsigo Systems VP780
43
System Monitoring
1/2
sanFc1GbPort
4/1
nwEthernet1GbPort
4/2
nwEthernet1GbPort
4/3
nwEthernet1GbPort
4/4
nwEthernet1GbPort
8/1
sanFc1GbPort
8/2
sanFc1GbPort
8 records displayed
down
up
down
down
down
down
down
0
2
3
0
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
q - quit, b - bytes, p - pkts, % - percent, m - mpbs, c - clear, u - up, d - down
Example 2
show system
Booted on: Fri Jun 22 23:09:55 GMT 2007
uptime: 20 days, 21 hours, 2 minutes, 44 seconds
RECENT UPGRADES AND
Mon Jul 9 19:53:26
Thu Jul 5 18:07:42
Fri Jun 29 18:11:15
Wed Jun 27 18:51:21
Fri Jun 22 23:09:23
DOWNGRADES
GMT 2007: Upgraded
GMT 2007: Upgraded
GMT 2007: Upgraded
GMT 2007: Upgraded
GMT 2007: Upgraded
to
to
to
to
to
xsigo-1.0.0.xpf
xsigo-1.0.0-RC17D.xpf
xsigo-1.0.0-RC17C.xpf
xsigo-branch-1.0.0-16980.xpf
xsigo-1.0.0-RC17A.xpf
Current Base OS Version Information
ReleaseNumber: 144
CompatOS:
49
ReleasePerson: root
ReleaseDate:
2007/06/17 15:46:41
ReleaseHost:
chaos
KernelVersion: 2.6.16.24-xsigo-2007061501
Alternative Base OS Version Information
*** No information available
INSTALLED XgOS VERSIONS
Current: xsigos-1.0.0
Previous: xsigos-1.0.0-RC17D
MEMORY INFORMATION
Total memory: 995.504M
Used memory: 399.367M
Free memory: 596.137M
Swap space used: 1.750M
DISK STATUS
Partition
Size Available
Used %used
Base OS
253.998M
60.842M
180.041M 70%
|###################################################-----------------------|
XgOS
1.192G
470.688M
687.613M 56%
|#########################################---------------------------------|
System logs
9.169G
8.556G
150.500M
1% |-------------------------------------------------------------------------|
Xsigo Systems VP780
44
Database
8.249G
7.631G
203.406M
-------------------------------------------------|
Temporary data
6.040G
5.701G
32.062M
-------------------------------------------------|
User data
2.752G
2.581G
32.152M
-------------------------------------------------|
Volatile data
184.901M
175.341M
0.014M
-------------------------------------------------|
Config data
44.292M
41.969M
0.036M
-------------------------------------------------|
2%
|#------------------------
0%
|-------------------------
1%
|-------------------------
0%
|-------------------------
0%
|-------------------------
Statistics
The system collects two types of statistics:
•
Real time statistics—Displayed any time you issue a show <xyz> stats command.
•
History statistics—A log of statistics collected over time. Use a stats-policy to set a polling time
interval for collecting historical stats. By default, stats are gathered and stored in the local database on the
hard disk every 15 minutes. The XMS web management software uses this information to graph historical
statistics.
Only real-time statistics can be cleared (not historical).
Syntax
Real-time stats:
show vnic <name> [vnic-stats | queue-stats | igmp-stats]
show vhba <name> stats
set vnic <name> clear [vnic-stats | queue-stats | igmp-stats]
set vhba <name> clear stats
Historical stats. There is a stats-policy for each type of statistic object:
set stats-policy acl -interval=<time> {on | off}
set stats-policy domain -interval=<time> {on | off}
set stats-policy domain-group -interval=<time> {on | off}
set stats-policy group -interval=<time> {on | off}
set stats-policy iocard -interval=<time> {on | off}
set stats-policy pool -interval=<time> {on | off}
set stats-policy pool-member -interval=<time> {on | off}
set stats-policy qos -interval=<time> {on | off}
set stats-policy san -interval=<time> {on | off}
set stats-policy server-profile -interval=<time> {on | off}
set stats-policy template -interval=<time> {on | off}
set stats-policy user -interval=<time> {on | off}
set stats-policy vhba -interval=<time> {on | off}
set stats-policy vlan -interval=<time> {on | off}
set stats-policy vm -interval=<time> {on | off}
set stats-policy vnic -interval=<time> {on | off}
set stats-policy vssl -interval=<time> {on | off}
set stats-policy vssl-application -interval=<time> {on | off}
set stats-policy vswitch -interval=<time> {on | off}
show stats-policies
Xsigo Systems VP780
45
Statistics
where -interval=<time> can be any of the following polling intervals:
1minute
15minutes
30minutes
45minutes
60minutes
default
By default, historical stats are turned off and the default polling interval is 15minutes.
Example 1
If a statistic is not available on a specific I/O hardware card, the question mark (?) field is displayed:
show vnic vn0.sp2 vnic-stats
-----------------------------------------------------------------------------name
vn0.sp2
vlan-id-or-none
300
rcv-pkt
0
ingress-octets
0
trans-pkt
5
egress-octets
376
invalid-ip-checksum
?
invalid-l4-checksum
?
flow-control-counter ?
mtu-err
?
ipchecksum-pkt
?
tcp-checksum-pkt
?
udp-checksum-pkt
?
tcpseg-pkt
?
green-pkt
?
yellow-pkt
?
red-pkt
?
-----------------------------------------------------------------------------1 record displayed
The 10GigE I/O card supports more statistics than the 1GigE I/O card.
Note
Example 2
By default, historical statistics are disabled (off) but collected every 15-minute interval when enabled (on):
set stats-policy acl -interval=45minutes on
show stats-policies
type
state
interval
--------------------------------------------------------------------acl
on
45minutes
fabric-switch
off
15minutes
fabric-switch-rx
off
15minutes
fabric-switch-tx
off
15minutes
hba-error
off
15minutes
Xsigo Systems VP780
46
ib-port
layer2
multicast
queue-qos
ssl-card
vhba
vnic
...
off
off
off
off
off
off
off
15minutes
15minutes
15minutes
15minutes
15minutes
15minutes
15minutes
Recovery CLI
Use the Recovery CLI to perform critical system tasks, such as password recovery or restoring factory defaults.
To access the Recovery CLI, you need to log in as user “RCLI” and use the password configured during the first boot.
The first boot config wizard has you set the RCLI password and the admin password.
XSIGO Recovery CLI (1.6.1)
[Status display disabled - expand screen to see it]
-- Main menu ---------------------------------------------------------------x. XsigOS control
c. Configuration maintenance
f. Core file maintenance
n. Network config
h. Refresh screen
b.
P. Reset system password
r. Chassis control i. Installation maintenance
l. Log file maintenance
d. Database maintenance
k. Disk maintenance
t. Terminal settings
Blow away ssh keys
F. Factory default
q. Quit
Selection:
If you see the recovery CLI press “q” to quit, and try again in about a minute (after the system fully boots up). If the
system persists in running the recovery CLI on login attempts, call technical support.
Online Help
At any point, you can issue the question mark (?) command to display context-sensitive help:
add vnic ?
<name> Virtual NIC name
Use help <command> to display a more detailed help.
help add vnic
Add a new virtual Network Interface Card (vNIC) to the system. You must provide
a hierarchical name for the vNIC at the time that it is added. A 'hierarchical'
name includes the name of the vNIC, plus the name of the server profile or
template to which the vNIC is assigned. The two names are separated by the dot
'.' character. For example: 'add vnic <vNIC_name>.<server_profile_name>'.
A second (optional) parameter of the 'add' command specifies the termination for
the vNIC. A vNIC can be terminated on an I/O port. For example: 'add vnic
<vNIC_name>.<server_profile_name> slot/port'.
Xsigo Systems VP780
Server Profiles and Templates
47
Server Profiles
A server profile is a logical representation of a physical host server’s I/O configuration. This entity can be assigned to one
or more physical servers. When the profile is assigned, the physical server assumes all of the I/O characteristics of a server
profile.
Syntax
add server-profile <name> <physical-server>
add server-profile <name> <physical-server> -hypervisor
[vmware][xen][none][virtualServer]
add server-profile <name> <physical-server> -template <temp-name>
set
set
set
set
set
set
set
server-profile
server-profile
server-profile
server-profile
server-profile
server-profile
server-profile
<name>
<name>
<name>
<name>
<name>
<name>
<name>
[connection <physical-server>][down][up][san-boot]
connect=<physical-server>
disconnect
-descr=”<text>”
-domain=<name>
-hypervisor-type=[vmware][xen][none][virtualServer]
-template=<name>
remove server-profile <name>
remove server-profile <name> [vhbas][vnics][vssls][-noconfirm]
show server-profile <name>
show server-profile <name> [vhbas][vnics][vssls][virtualServer][vswitches]
[-noconfirm][alarms][errors][phys-cons][pxe-boot][san-boot][warnings]
Example
Take the following steps to create a server profile:
Step 1
Add a server profile construct, named mytest, in the root domain:
add server-profile mytest ?
alexander,iowa:ServerPort8
ceasar,iowa:ServerPort24
Connection to host alexander (up)
Connection to host ceasar (up)
All the physical servers connected to the VP780 are displayed. The two servers listed (alexander and
caesar) were automatically discovered by the VP780.
For host driver documentation, see Linux Host Software on page 189.
Step 2
Assign the profile to a physical server:
add server-profile mytest alexander@iowa:ServerPort8
Step 3
Verify the profile was created correctly:
show server-profile mytest
name
state descr connection
hypervisor vnics vhbas vssls
-----------------------------------------------------------------------mytest up/up
alexander@iowa:ServerPort8 none
0
0
0
Xsigo Systems VP780
48
Chapter 3: Server Profiles and Templates
1 record displayed
Note that no I/O resources (vNICs, vHBAs, vSSLs) have been assigned to the new server profile. Resources will be
assigned to the profile in the following sections (See vNICs on page 53, vHBAs on page 67, and vSSLs on page 87.)
If the state displays “unassigned”, then the profile is created but not yet assigned to an actual host server. Use set
server-profile <name> connect=<actual-hostserver> for the assignment.
Templates
Use a template to clone a set of defined attributes onto another entity. For example, create a vNIC template to replicate
properties onto 100s of vNICs. Use a template as a configuration shortcut (convenience).
Syntax
add template <name>
set template {<name> | *} [-descr=”<text>”][-domain=<name>]
remove template {<name> | *}
show template {<name> | *} [vhbas][vnics][vssls]
add template <name>—Creates a named template.
set template {<name> | *} [-descr=”<text>”][-domain=<name>]—Modifies the default attributes of
a template.
remove template {<name> | *}—Delete a template.
show template {<name> | *} [vhbas][vnics][vssls]—Displays information for a configured template.
You can filter the output by V* information.
Example 1
This example shows how to use a vNIC template with a termination group. When you instantiate the vNICs on the
template, the system will automatically pick one the I/O ports from the pool and start using it on a vNIC.
See “Termination Groups” on page 159 for more information.
Step 1
Create a named vNIC template:
add template myvnictemplate
Step 2
Add a vNIC to the template and associate it with the termination group:
add vnic myvnic.myvnictemplate mytermgroup
show -list template myvnictemplate vnics
-----------------------------------------------------------------------name
myvnic.myvnictemplate
state
/
mac-addr
ipaddr
if
term-grp mytermgroup
if-state -
Xsigo Systems VP780
49
Templates
type
vlans
none
vm
qos
Step 3
To instantiate the vNIC template, create a server profile based on the vNIC template. The vNICs
associated with the termination group will now be allocated ports:
add server-profile myserver ceasar,iowa:ServerPort24 -template=myvnictemplate
Example 2
set template * -descr="This is a test"
show template
name
descr
vnics
vhbas
vssls
-----------------------------------------------------------------------------mytemplate
This is a test
0
0
0
mytemplate2
This is a test
0
0
0
2 records displayed
Xsigo Systems VP780
50
Default Gateway
Define a default gateway on a server profile to enable IP communication with hosts on different IP subnets. This feature
enables centralized IP address administration from the VP780. Given this feature, a default gateway need not be
configured directly on a host.
Note
In Release 1.5, the default gateway feature is not yet supported for a Windows server 64 bit platform host.
However, a Windows 32 bit host does support the default gateway feature.
Syntax
add gateway <gw-name> <default-gw-addr> <dns-addr> <domain-name>
set gateway <gw-name> [-dns=<dns-addr>][-domain-name=<name>][-ipaddr=<addr>]
set server-profile <name> -default-gateway=[<gw-name>][none]
show gateway [<name>]
Example 1
10.1.10.112/24
MGT
AUX
SER-1
SER-2
USB
e8/1
10.1.11.112/24
IB
10.1.10.111/24
10.1.11.111/24
Thorne
Figure 1 Default Gateway Topology
Take the following steps to configure a default gateway:
Step 1
From the hostserver, confirm the following entities are not reachable: default gateway address, DNS
server address, and domain name.
cat /etc/resolv.conf
route
ping 10.1.11.112
Issue the route command to confirm the server cannot reach the outside network because you have not yet
configured a default gateway. Likewise ping 10.1.11.112 will fail in this example because the route is
not yet installed in the routing table.
Step 2
On the VP780, add a server profile and vNIC:
add server-profile s23 thorne@connecticut:ServerPort22
add vnic test_1.s23 8/1
set vnic test_1.s23 -addr-type=static -ip-addr=10.1.10.111/24
Xsigo Systems VP780
51
Default Gateway
Step 3
Create a default-gateway profile. Specify the gateway-profile name, default gateway IP address,
DNS server IP address, and domain name:
add gateway test 10.1.10.112 1.1.1.1 testorg
show gateway test
name
descr
addr
dns-addr
domain-name
-----------------------------------------------------------------------test
10.1.10.112
1.1.1.1
testorg
Tip: The gateway’s IP addr must be on the same subnet as the vNIC’s address.
Step 4
Associate the default-gateway profile with the server profile:
set server-profile s23 -default-gateway=test
show server-profile s23
name state descr connection
def-gw vnics vhbas vssls
-----------------------------------------------------------------------s23 up/up
thorne@connecticut:ServerPort22 test
1
0
0
Step 5
On the hostserver, verify the default gateway and DNS server were pushed to the hostserver and
installed properly:
cat /etc/resolv.conf
route
ping 10.1.11.112
Example 2
To modify an existing default-gateway profile:
Step 1
Use the none option to disassociate the default-gateway profile with the server profile:
set server-profile s23 -default-gateway=none
Step 2
Note all the gateway options you can change:
set gateway test ?
[Optional qualifiers]
-descr
Description
-dns
IP address of DNS server
-domain
Set the domain for a gateway
-domain-name Internet domain name
-ipaddr
IP address of default gateway
Step 3
This example changes the DNS to 2.2.2.2. After the change is made, the default-gateway profile must be
reassociated back to the server profile:
set gateway test -dns=2.2.2.2
set server-profile s23 -default-gateway=test
show gateway test
name
descr
addr
dns-addr
domain-name
-------------------------------------------------------------------test
10.1.10.112
2.2.2.2
testorg
Xsigo Systems VP780
52
Xsigo Systems VP780
vNICs
53
Introduction
A virtual Network Interface Card (vNIC) virtualizes NIC connectivity. A vNIC is a virtual NIC that appears to the OS as
a physical NIC and enables a server to have a Ethernet network attachment without having a physical NIC present.
Instead of the client server using an NIC, an Infiniband (IB) HCA is used and then virtualizes the NIC allowing for
Ethernet connectivity.
To enable vNICs for VMware and Xen environments, see VMware ESX Servers on page 89 and Xen Hypervisor on
page 95.
To enable vNICs for QoS, see Network QoS for vNICs on page 101.
Basic vNIC Configuration
A vNIC involves the following bringup procedure:
•
Adding a server profile
•
Creating a named vNIC
•
Associating the vNIC to a server profile and physical I/O card
•
Setting IP address information
•
Verifying the configuration and state
•
Confirming the vNIC was installed as a device on the host server
Syntax
add server-profile <server-name> <server>@<vp780>:<ib-port>
add vnic <vnic-name>.<server-name> <slot>/<port>
set vnic <vnic-name>.<server-name> -addr-type=[static|dhcp] -ip-addr=<addr/
mask>
set vnic <vnic-name>.<server-old> move <vnic-name>.<server-new>
set vnic <vnic-name-old>.<server-name> rename <vnic-name-new>.<server-name>
remove vnic [*] [<vnic-name>][-noconfirm]
show vnic [*] [<vnic-name>][-detail]
show vnic <vnic-name>.<server-name> vnic-stats
add server-profile <server-name> <actual-physcon>—Creates a named server <servername> and associates it with the actual hostname (<actual-physcon>) associated with the resource.
This hostname is also known as the physical connection (phys-con). Once a server-profile is added, you
can add subsequent vNICs (add vnic) to it.
add vnic <vnic-name>.<server-name> <slot>/<port>—Creates a named vNIC, associates it with a
server name, and specifies a physical slot/port on the chassis. A 10GigE I/O card can support 128 vNICs.
A QuadGigE card can support 62 vNICs.
Xsigo Systems VP780
54
Chapter 4: vNICs
set vnic <vnic-name>.<server-name> -addr-type=[static|dhcp] -ip-addr=<address/
mask>—Configures an IP address on the named vNIC. The address type can be static or dhcp assigned.
The default is dhcp.
Note
The Xsigo chassis automatically assigns MAC addresses to vNICs from a pool of internal-sequential
addresses.
Example
add server-profile myserver alexander@iowa:ServerPort8
add vnic myvinc.myserver 4/2
set vnic myvinc.myserver -addr-type=static -ip-addr=10.1.1.1/32
show vnic myvinc.myserver
-----------------------------------------------------------------------------name
myvinc.myserver
state
up
mac-addr 00:13:97:01:80:08
ipaddr
10.1.1.1/32
descr
if
4/2
if-state up
type
static
vlans
none
vm
show ioport 4/2
------------------------------------------------------------------name
4/2
type
nwEthernet1GbPort
state
up/up
descr
rate
auto/1Gbps
mtu
1500
avail-in-cir
0Kbps
avail-out-cir 1Gbps
mode
access
flowcontrol
false
vnics
3
vlans
none
------------------------------------------------------------------1 record displayed
show vnic myvinc.myserver vnic-stats
------------------------------------------------------------------name
myvinc.myserver
vlan-id-or-none
0
rcv-pkt
321188606
ingress-octets
1339034086
trans-pkt
517245121
egress-octets
2996294724
invalid-ip-checksum
0
invalid-l4-checksum
0
flow-control-counter 0
mtu-err
0
ipchecksum-pkt
?
Xsigo Systems VP780
55
vNIC Counters and Statistics
tcp-checksum-pkt
0
udp-checksum-pkt
0
tcpseg-pkt
0
green-pkt
?
yellow-pkt
?
red-pkt
?
------------------------------------------------------------------1 record displayed
vNIC Counters and Statistics
There are several ways to gather vNIC counters and statistics.
On the host server:
ifconfig <vnic-name>—Displays statistics as collected by the OS through the network layer.
cat /proc/driver/vnic/devices/<vnic-name>—Shows stats as collected by the vNIC driver.
/opt/xsigo/bin/xsigo-support—Collects and dumps information for monitoring and troubleshooting
your host-software installation. See Debug Tool: xsigo-support on page 198.
On the VP780:
show vnic <vnic-name> vnic-stats
set vnic <vnic-name>.<server-name> clear [vnic-stats][queue-stats]
Use these commands to display and clear statistics as collected by the vnic-stats model in the chassis.
HA vNIC Pairs
High Availability (HA) vNIC pairs can be configured for a single VP780 chassis, or for two separate VP780s.
The system does not support the dynamic re configuration of vNIC failover characteristics. Once you create an HA
enabled vNIC, the system does not allow you to change its failover characteristics. You must delete the vNIC then create
a new one from scratch.
Single-Chassis Configuration
This section documents an example of configuring HA within a single VP780. In this example, you will:
•
Create a vNIC
•
Give the vNIC a name
•
Assign the vNIC to a server profile
•
Bind the vNIC to a physical interface card (slot/port)
•
Configure the new vNIC as the primary vNIC of a high-availability vNIC pair
•
Bind the secondary vNIC (created automatically) to a second physical
interface card (slot/port)
Xsigo Systems VP780
56
Chapter 4: vNICs
Step 1
Create a vNIC called “haNIC1” and assign it to a server profile “vserver1”:
add vnic haNIC1.vserver1 ?
6/1 nwEthernet1GbPort in
slot
slot
slot
slot
slot
slot
6
6
6
6
8
8
port
port
port
port
port
port
1
2
3
4
1
2
All of the available physical Ethernet cards are displayed.
Step 2
Bind the vNIC to a physical Ethernet card. Select the slot/port that you want to link to the vNIC (in this
example, “6/1”):
add vnic haNIC1.vserver1 6/1 ?
ha Specify High Availability characteristics
Step 3
Specify the primary vNIC of the high-availability pair by selecting ha. The first vNIC created and
designated as ha automatically becomes the primary vNIC of the pair:
add vnic haNIC1.vserver1 6/1 ha ?
6/1
nwEthernet1GbPort in slot 6 port 1 (down)
6/2
6/3
6/4
-mac
Secondary HA group MAC address
-primary
This is a primary HA VNIC
-secondary This is a secondary HA VNIC (need to
specify group MAC address)
Step 4
Bind the secondary vNIC to a physical Ethernet card. Select the slot/port that you want to link to the
secondary vNIC (in this example, “6/3”), then press Enter. Do not select the same slot/port that was
assigned to the primary vNIC.
add vnic haNIC1.vserver1 6/1 ha 6/3
This command set created a high-availability vNIC pair on a single chassis. The primary vNIC is named “haNIC1”.
The secondary vNIC was created automatically and named “haNIC1S”. (Note the “S” appended to the end of the name.)
The full name of the primary vNIC was automatically assigned as the high-availability group’s name.
Multiple-Chassis Configuration
This section documents an example of configuring HA across multiple VP780s. In this example, you will:
1.
Log into the first VP780
— Create a vNIC
— Give the vNIC a name
— Assign the vNIC to a server profile
— Bind the vNIC to a physical interface card (slot/port)
Xsigo Systems VP780
57
HA vNIC Pairs
— Configure the new vNIC as the primary vNIC in a high-availability vNIC pair
— Retrieve the MAC address of the primary vNIC
2.
Log into the second VP780
— Create a vNIC
— Give the vNIC a name
— Assign the vNIC to a server profile
— Bind the vNIC to a physical interface card (slot/port)
— Configure the new vNIC as the secondary NIC in a high-availability vNIC pair
— Enter the primary vNIC’s MAC address
Take the following steps:
Step 1
Log into the first VP780 chassis.
Step 2
Create a vNIC. Add a vNIC, called “haNIC1”, and assign it to a server profile (“vserver1”):
Step 3
slot
slot
slot
slot
slot
slot
6
6
6
6
8
8
port
port
port
port
port
port
1
2
3
4
1
2
Bind the vNIC to a physical Ethernet card. Select the slot/port that you want to link to the vNIC (in this
example, “6/1”):
A single option is displayed which enables you to configure the new vNIC as half of a high-availability
vNIC pair.
Step 4
Configure the vNIC as half of a high-availability pair. Select “ha”:
6/1
nwEthernet1GbPort in slot
6/2
6/3
6/4
8/1
8/2
6
6
6
6
8
8
port
port
port
port
port
port
1
2
3
4
1
2
(down)
(down)
(down)
(down)
-mac Secondary HA group MAC address
-primaryThis is a primary HA VNIC
-secondaryThis is a secondary HA VNIC (need to
Xsigo Systems VP780
58
Chapter 4: vNICs
Step 5
Configure the vNIC as the primary vNIC of the high-availability pair. Select “-primary”, then press
Enter.
add vnic haNIC1.vserver1 6/1 ha -primary
This command set created a vNIC (haNIC1), assigned it to a server profile (vserver1), bound it to a physical slot/port (6/
1), and specified the vNIC as the primary vNIC in a high-availability vNIC pair.
Step 6
Retrieve the MAC address of the primary vNIC.
show vnic haNIC1.vserver
-----------------------------------------name
haNIC1.vserver1
state
resourceUnavailable
mac-addr
00:13:97:01:80:01
ipaddr
descr
if
6/1
mcast-group
type
mtu
1500
group
haNIC1.vserver1
group-pref
primary
flags
vlans
none
------------------------------------------
Step 7
Log into the second VP780 chassis.
Step 8
Create a second vNIC. Add a second vNIC, give it the same name as the primary vNIC (“haNIC1”), and
assign it to the same server profile as the primary vNIC (“vserver1”):
Step 9
slot
slot
slot
slot
slot
slot
6
6
6
6
8
8
port
port
port
port
port
port
1
2
3
4
1
2
Bind the second vNIC to a physical Ethernet card on the second chassis. Select the slot/port that you
want to link to the secondary vNIC (in this example, “8/2”):
A single option is displayed which enables you to configure the new vNIC as one half of a high-availability
vNIC pair.
Step 10 Configure the second vNIC as the second half of a high-availability pair. Select “ha”:
6/1nwEthernet1GbPort in slot 6 port 1 (down)
Xsigo Systems VP780
59
HA vNIC Pairs
8/1 nwEthernet1GbPort in slot 8 port 1
-macSecondary HA group MAC address
Step 11 Configure the second vNIC as the secondary vNIC of the high-availability pair. Select “-secondary”:
add vnic haNIC1.vserver1 8/2 ha -secondary ?
8/1nwEthernet1GbPort in slot 8 port 1
-macSecondary HA group MAC address
Step 12 Insert the primary vNIC’s MAC address. Select “-mac”. Type ‘<space>’, enter the MAC address
retrieved in Step 6, then press Enter.
add vnic haNIC1.vserver1 8/2 ha -secondary -mac=00:13:97:01:80:01
This command set created a high-availability vNIC pair across two VP780s. The HA group’s name was automatically set
to haNIC1.vserver1. Both the primary and secondary vNICs are named haNIC1.
Xsigo Systems VP780
60
Chapter 4: vNICs
Auto Switchover
Auto switchover enables a vNIC to revert back to a primary path after it’s restored (comes back online). When autoswitchover is not configured, a vNIC remains on the secondary path and never reverts back to primary (default).
Note
Auto switchover is appropriate for cases where traffic engineering requires that a specific vNIC always be
used for network communication.
Syntax
add vnic <name>.<profile> <pri-s/p> -auto-switchover=true ha <sec-s/p>
show vnic <name>.<profile> -detail
Default: auto switchover is disabled.
Example
Card 1/1 is the primary link for a vNIC named test_1.01bardeen. The secondary link connects to card 2/1:
Host Server
v1p
v1s
Infiniband
MGT
AUX
SER-1
SER-2
1/1
Ethernet
2/1
Ethernet Switch
Figure 1 vNIC Reverts Back to 1/1
When 1/1 goes down, traffic fails over to path 2/1. When 1/1 comes back online, the vNIC reverts back to using 1/1
automatically. Any failure along the path (Ethernet or Infiniband) of the vNIC will force traffic flow to the other side.
Note that show vnic -detail displays “flags” is set to “A” once -auto-switchover is enabled:
add vnic test_1.01bardeen 1/1 -auto-switchover=true ha 2/1
show vnic test_1.01bardeen -detail
-----------------------------------------------------------------------------name
test_1.01bardeen
state
up
mac-addr
00:13:97:01:80:09
ipaddr
descr
Xsigo Systems VP780
61
Admin State Control
if
1/1
if-state
up
mcast-group
type
mtu
1500
group
test_1.01bardeen
group-pref
primary
flags
A
vlans
none
vm
uvi
1
queue-map-type disabled
----------------------------------------------------------------------------1 record displayed
Admin State Control
Use set vnic up | down to control the administrative state of a configured vNIC.
Syntax
set vnic <vnic-name>.<server-name> up
set vnic <vnic-name>.<server-name> down
up—Activates a vNIC (default)
down—Deactivates a vNIC
Example
show vnic myvnic.myserver
------------------------------------------------------------------name
myvnic.myserver
state
up/up
mac-addr 00:13:97:01:80:06
ipaddr
if
4/2
term-grp
if-state up
type
vlans
none
vm
qos
-------------------------------------------------------------------1 record displayed
set vnic myvnic.myserver down
Are you sure you want to deactivate VNIC myvnic.myserver (y/n)?y
------------------------------------------------------------------name
myvnic.myserver
Xsigo Systems VP780
62
Chapter 4: vNICs
state
down/down
mac-addr 00:13:97:01:80:06
ipaddr
if
4/2
term-grp
if-state up
type
vlans
none
vm
qos
-------------------------------------------------------------------1 record displayed
vNIC Priority
Depending on your card type, there are up to 8 priorities that can be set for a named vNIC. A 10GigE I/O card has 4
priorities. A QuadGigE card supports 8 different priorities.
Syntax
set vnic <vnic-name> ingress-qos -priority=<value>
set vnic <vnic-name> egress-qos -priority=<value>
Example
show vnic foo.bar qos
--------------------------------------------name
foo.bar
level
queue
queue
0
direction ingress
descr
template
policer
shaper
wred
wfq
priority
1
enables
----set vnic foo.bar ingress-qos -priority=?
[Available values]
0
1
2
3
4
5
6
7
default (default 1)
set vnic foo.bar ingress-qos -priority=4
Xsigo Systems VP780
63
MTU
MTU
The Maximum Transmission Unit (MTU) is the largest physical packet size (in bytes) that a network can transmit. MTU
values are only applicable to Ethernet ports, and the MTU of the I/O port must match the MTU of the neighboring
switch.
Only jumbo setting or normal setting on ports is supported. Per vNIC MTU is not supported.
Syntax
set ethernet-port <slot>/<port> -mtu=<value>
show ethernet-port <slot>/<port>
The default MTU <value> is 1500.
Example
Step 1
Select the I/O port and set the new MTU value:
set ethernet-port 4/1 -mtu=9194
Step 2
Confirm the new MTU setting:
show ethernet-port 4/1
-----------------------------------------------------------------------name
4/1
type
nwEthernet1GbPort
state
up/up
descr
rate
auto/1Gbps
mtu
9194
avail-in-cir
0Kbps
avail-out-cir 1Gbps
mode
access
flowcontrol
false
vnics
2
vlans
none
-----------------------------------------------------------------------1 record displayed
Xsigo Systems VP780
64
Chapter 4: vNICs
VLANs
A Virtual LAN (VLAN) is a private, independent, logical network that is created within a physical network. A VLAN
behaves like an ordinary LAN, but connected devices do not have to be physically connected to the same network
segment.
Syntax
add server-profile <profile-name> <server>@<vp780>:<ib-port>
add vnic <vnic>.<profile-name> <slot>/<port>
add vlan <vlan-id>.<vnic>.<profile-name>
set ethernet-port <slot>/<port> -mode=trunk (mode is for 10GigE only)
set vlan <vlan-id>.<vnic>.<profile-name> -ip-addr=<addr/mask>
set vnic <vnic>.<name> -addr-type=<type> -ip-addr=<addr> -untagged-vlan-id=<value>
show vlan
show vnic <vnic>.<profile-name> vlans
and on the host side when using a QuadGigE I/O card:
ifconfig <vnic>.<vlan-id> <ipaddr/mask>
Example 1: 10GigE Untagged VLANs
This example works only on the 10GigE I/O card (not QuadGigE):
vnic test_1.s1
VLAN Domain 1
s1
s2
MGT
AUX
SER-1
SER-2
USB
e15/1
10GigE
vnic test_1.s2
VLAN Tags 5, 10
vnic test_1.s3
VLAN Domain 2
s3
s4
vnic test_1.s4
VP780
Figure 2 Example Topology for Untagged VLAN IDs
The 10GigE card enables you to configure VLANs without the host server having any information about the VLANs.
Use the set vnic <name> ... -untagged-vlan-id=<id> command to enable this feature. All the packets that
leave the host server on this vNIC exit without VLAN tags. The VLAN tag is inserted when the packet is processed by the
VP780’s 10GigE card. The same holds true for the reverse direction. When packets with VLAN tags arrive from the
upstream router, the VP780 strips those tags before forwarding the packets onto the host server.
Xsigo Systems VP780
65
VLANs
The VP780 (not the host) assumes the CPU overhead of adding and removing VLAN tags. Using untagged VLANs
reduces the host server’s CPU utilization by up to 30% during heavy traffic loads. Offloading tag insertions and removals
to the VP780 increase a host’s performance.
VMware does not work with VLAN tags. As a workaround, use the VP780 to add/remove the tags.
Note
Even through VLAN 5 and VLAN 10 both terminate on the same 10GigE card, the VLAN domains on the host servers
are isolated and can communicate with one another.
Take the following steps to configure untagged VLANs for a 10GigE card:
Step 1
By default, the 10GigE card runs in access mode. This mode assumes multiple VLAN tags are not
present, nor is interface sharing of VLANs. Therefore, you need to configure the 10GigE card for trunk
mode:
set ethernet-port 15/1 -mode=trunk
show ethernet-port 15/1
-----------------------------------------------------------------------------name
15/1
type
nwEthernet10GbPort
state
up/up
descr
rate
auto/10Gbps
mtu
1500
avail-in-cir
10Gbps
avail-out-cir 10Gbps
mode
trunk
flags
-vnics
0
vlans
none
Note: On a QuadGigE card, there is no concept of access mode vs trunk mode. These modes apply only to
10GigE.
Step 2
Create four vNICs then set them to participate in an untagged VLAN. The host servers will not be aware
that VLAN tagging is being performed on the 10GigE card. Note the -untagged-vlan-id option at
the end of the set command:
set vnic test_1.s1 -addr-type=static -ip-addr=1.1.1.1/24 -untagged-vlan-id=5
Tip: vNICs using the same VLAN IDs must be on the same IP subnet.
Xsigo Systems VP780
66
Chapter 4: vNICs
Step 3
Confirm each VLAN state is untagged and the proper VLAN ID was applied:
show vnic test_1.s1 vlans
name
state
descr
ipaddr
type
------------------------------------------------------------5.test_1.s1
untagged
Untagged VLAN
Example 2: QuadGigE
Take the following steps to configure a VLAN on a QuadGigE card:
Step 1
Create a vNIC with a VLAN ID of 5:
add server-profile s1 alexander@iowa:ServerPort23
add vlan 5.test_1.s1
Step 2
Assign addressing to the VLAN:
set vlan 5.test_1.s1 -ip-addr=10.1.1.1/24
When using VLANs, IP addresses should be bound only to VLANs, not the vNIC.
Step 3
Verify the VLAN’s state is “up”:
show vlan
name
state
descr
ipaddr
type
------------------------------------------------------------5.test_1.s1
up
10.1.1.1/24
static
Xsigo Systems VP780
vHBAs
67
Introduction
A virtual Host Bus Adapter (vHBA) virtualizes HBA connectivity. A vHBA is a virtual HBA that appears to the OS as a
physical HBA and enables a server to have a Fibre Channel (FC) SAN attachment without having a physical HBA
present. Instead of the client server using an HBA, an Infiniband (IB) HCA is used and then virtualizes the HBA allowing
for SAN connectivity.
Figure 1 displays a typical vHBA topology
HCA HCA
vHBA Host Drivers
HCA HCA
IB
WWPN 1
vHBA Host Drivers
24
FC
FC Switch
Fabric
WWPN 2
Storage
Array
VP780
HCA HCA
Installed with a 2-Port FC I/O Card
vHBA Host Drivers
FC
(Initiator)
SAN
(Target)
Host Servers
Installed with vHBA Host Drivers
(Initiator)
Figure 1 vHBA Topology
An IB connection exists between the Xsigo VP780 and host servers supporting the Xsigo vHBA host software stack.
Up to 24 IB ports are supported. A 2-port FC I/O card connects to a Storage Area Network (SAN) FC switch fabric. All
the host server vHBAs multiplex through the FC ports on the I/O card. A storage array is attached to the switch fabric.
Initiators are host servers that request I/O processing and actively seek out and interact with target devices on the SAN.
Targets are passive storage devices (arrays, JBODs, RAIDs, etc) that respond to requests sent by initiators. The VP780
itself is an I/O initiator that provides a conduit for host-server initiators to send commands to the fabric.
Note
Some target devices function also as data replicators. In this case, these targets function also as I/O initiators
replicating data (sync) to other locations.
The vHBA host software defines how the FC protocol will be transported (in/out) over IB. Without this software and the
details of the transport, the vHBA will not function and the payload cannot be sent over IB.
Both initiators and targets have a World Wide Node Name (WWNN) and a World Wide Port Name (WWPN). A 2-port FC
card itself has one WWNN, and each port has its own WWPN. These IDs register with one another to establish
communication.
N_Port ID Virtualization (NPIV) enables multiple fibre channel initiators (WWNs) to login and occupy a single physical
port. Your switch device (between the VP780 and the storage device) must be running a version of switch OS that
Xsigo Systems VP780
68
Chapter 5: vHBAs
supports NPIV, and it must be turned on. Without NPIV, a vHBA will not be able to log into the fabric. Note that some
switches require configuring the max number of NPIV logins.
Tip
Reset the VP780’s FC I/O module whenever the firmware is changed on the upstream FC switch. The I/O
module needs to rediscover the FC setting attributes. Do this by using the set fc-card <slot> reset
command.
See “SAN QoS for vHBAs” on page 115 for information about using vHBAs with QoS.
Basic vHBA Configuration
The following command syntax and example are provided for a basic vHBA configuration.
Syntax
add server-profile <server-name> <actual-physcon>
add vhba <vhba-name>.<server-name> <slot>/<port>
show vhba <vhba-name>.<server-name>
Example
Take the following steps to enable a minimum vHBA configuration:
Step 1
Inspect the IB port connectivity. Note the hostname and chassis-port information:
show ib ports
Step 2
Create a named server profile and bind it to a physical-server connection (phys-con):
add server-profile myserver ceasar@iowa:ServerPort24
Step 3
Find an FC card (sanFc2Port4GbCard) on which you can terminate a vHBA:
show iocard
slot
state
descr
type
v-resources
----------------------------------------------------------------------1
up/up
sanFc2Port4GbCard
0
2
up/up
sanFc2Port4GbCard
0
3
up/up
sanFc2Port4GbCard
0
4
up/up
sanFc2Port4GbCard
0
4 records displayed
Step 4
Find an FC slot/port to which you will assign a vHBA. In this example, 2/1 will be used:
show ioport
name
type
state
descr
vnics
vhbas
----------------------------------------------------------------------1/1
sanFc1GbPort
up/up
0
0
1/2
sanFc1GbPort
up/up
0
0
2/1
sanFc1GbPort
up/up
0
0
2/2
sanFc1GbPort
up/up
0
0
3/1
sanFc1GbPort
up/up
0
0
3/2
sanFc1GbPort
up/up
0
0
Xsigo Systems VP780
69
Basic vHBA Configuration
4/1
sanFc1GbPort
4/2
sanFc1GbPort
8 records displayed
up/up
up/down
0
0
0
0
The FC port (sanFc1GbPort) must be connected to a fibre-channel switch. In this case, the show iport
state will be “up/up”. If you see “up/down”, the cable might be disconnected from the port or the port is
disabled on the remote switch. A fibre-channel port can auto negotiate its speed up to 1, 2, and 4 Gbps.
Step 5
Create a vHBA, bind it to the server profile, and specify a slot/port on which to terminate the vHBA:
add vhba vhba1.myserver 2/1
In this example, the vHBA is “vhba1” and the server profile is “myserver”. The FC slot is “2”, and the FC
port is “1”. When you add a vHBA and specify a termination point, a vHBA is created on the server
automatically (assuming the correct host software is installed). If devices connect through that port, the hosts
will begin to discover the targets.
To define target order, see “Persistent Binding” on page 70.
If you receive the error message “Invalid vhba name - parent does not exist”, then the server profile was not
created successfully. Repeat the steps again.
Note
vHBas must be distinct when created on distinct chassis. For example, you can not have VH1.SP1 on two
different chassis that connect to one or more common servers.
Step 6
Verify the vHBA was created and its state is “up”:
show -list vhba vhba1.myserver
----------------------------------------------------------------------name
vhba1.myserver
state up
descr
if
2/1
wwnn
50:01:39:71:00:02:D1:1E
wwpn
50:01:39:70:00:02:D1:1E
map
----------------------------------------------------------------------1 record displayed
The state is “up” when the FC port is connected to a reachable FC switch.
If the state is “resourceUnavailable”, there is no FC connection. This field displays also for the case when
the server profile has no phys-con, or the host cannot communicate.
Note there are three-levels of oper-status on the VP780: card, port, vhba.
Tip
The access-control zoning on the switch and LUN masking must be set up properly in advance. Go to the
switch and verify the WWNs have logged in properly. Otherwise, you not see the appropriate devices via the
vHBA in the CLI. When set up properly, the prescan feature enables an unbound vHBA to display the
discovered targets and LUNs in the network environment. At this point, an unbound vHBA can be bound
to a server profile. See Target Prescan and Rescan on page 74 for more information.
Xsigo Systems VP780
70
Chapter 5: vHBAs
Note
By default, each vHBA has a queue depth of 16 queues. See show vhba <vhba-name>.<profilename> -detail
Persistent Binding
A target is a storage device on a SAN. A target can be a single disk, or it can have many devices (LUNs or volumes)
within it.
Users who bind targets to specific devices tend to also specify the scope and search order (persistent binding) of those
devices. In Xsigo’s application, persistent binding occurs within a vHBA. When a vHBA becomes active, it is working
with many devices in the network (i.e., switch communication, fabric login, device discovery). The vHBA then presents
this information to the remote OS. In order to preserve the remote OS’ device-to-drive binding across each bring-up, the
persistent binding setting is required.
Default: No persistent binding is assigned to a vHBA. When persistent binding is not configured, all the targets found for
the vHBA are reported to the remote OS in a random order (first come first serve). Persistent binding specifies the exact
order of the targets found.
Syntax
add san map <map-name> entry <order> <wwpn>
add vhba <vhba-name> <card>/<port> -map=<map-name>
set vhba <vhba-name> -map=<map-name>
remove san map <map-name>
show san map [<map-name>]
show vhba <vhba-name> map
add san map—Creates an ordered mapping of devices identified by World Wide Port Names (WWPN).
The vHBA uses these SAN map device IDs in this order. All devices discovered by the XgOS are subject to this
binding filter. Missing devices are skipped and no substitutes are made.
<map-name>—User-defined name for a map to add or set. A SAN map is the order in which the target disks
come up (become active).
entry <order>—Order number in the remote OS. The order range is from 0 to 255.
<wwpn>—World Wide Port Name. A 64-bit global address, where each number is delimited by colons (:).
Tip
The persistent binding can only apply to the target’s level but not to the Logical Unit Numbers (LUNs) level.
Therefore, an array-ordering problem could arise in the network when a new LUN is added to the topology.
In this case, the persistent binding would need to be redone.
Xsigo Systems VP780
71
Persistent Binding
Example 1
Take the following steps to configure a persistent map (binding) for an undeployed vHBA.
A vHBA is considered deployed when it has been assigned to a slot/port and a server ID (a server profile with phys-cons
set). The remote OS has already detected a specific target order. When a vHBa has already been deployed, the XgOS does
not allow users to change (set) this target order dynamically (on-the-fly). Likewise, when a persistent mapping is already
assigned to a vHBA, the XgOS does not allow users to modify that persistent mapping. You cannot add, delete, or modify
specific entries. In summary, mapping can be specified only at vHBA creation time (when the add vhba command is
issued).
Step 1
Add a named SAN map and specify its fixed WWPN target order. This example creates a SAN map with
8 targets:
add
add
add
add
add
add
add
add
san
san
san
san
san
san
san
san
map
map
map
map
map
map
map
map
mymap
mymap
mymap
mymap
mymap
mymap
mymap
mymap
entry
entry
entry
entry
entry
entry
entry
entry
0
1
2
3
4
5
6
7
21:00:00:20:37:C9:1D:C2
21:00:00:20:37:D5:45:FD
21:00:00:20:37:B3:F0:5C
21:00:00:20:37:90:88:90
21:00:00:20:37:C6:5E:B4
21:00:00:20:37:CC:EB:30
21:00:00:20:37:D5:37:18
21:00:00:20:37:8D:03:7D
Consider starting the entry order from 0 instead of 1 because the host OS uses 0 as the 1st order.
Step 2
Verify the persistent map was configured correctly:
show san map mymap
name
descr entries
-----------------------------------------------------------------mymap
0=21:00:00:20:37:C9:1D:C2,
1=21:00:00:20:37:D5:45:FD,
2=21:00:00:20:37:B3:F0:5C,
3=21:00:00:20:37:90:88:90,
4=21:00:00:20:37:C6:5E:B4,
5=21:00:00:20:37:CC:EB:30,
6=21:00:00:20:37:D5:37:18, 7=21:00:00:20:37:8D:03:7D
1 record displayed
You can omit the <map-name> to display information of all configured SAN maps.
Step 3
Create a server profile, a vHBA (not yet deployed), and bind them together with a persistent map:
add server-profile myserver
add vhba vhba101.myserver
set vhba vhba101.myserver -map=mymap
show vhba vhba101.myserver map
vhba
name
descr entries
-----------------------------------------------------------------------vhba101.myserver
mymap
0=21:00:00:20:37:C9:1D:C2,
1=21:00:00:20:37:D5:45:FD,
2=21:00:00:20:37:B3:F0:5C,
3=21:00:00:20:37:90:88:90,
4=21:00:00:20:37:C6:5E:B4,
5=21:00:00:20:37:CC:EB:30,
6=21:00:00:20:37:D5:37:18,
Xsigo Systems VP780
72
Chapter 5: vHBAs
7=21:00:00:20:37:8D:03:7D
1 record displayed
Step 4
Bind the named server profile to a physical connection:
set server-profile myserver -add-phys-con=myserver,ServerPort13
Step 5
Bind the vHBA to a physical slot/port:
set vhba vhba101.myserver -if=1/1
At this point, the vHBA is bound to the persistent map named “mymap”. When this vHBA finds its targets,
the vHBA sends target information to the host along with the target order. The hostdriver receives the target
information and propagates it up to the OS based on entry order in the map.
Step 6
Check the targets of the newly bound vHBA:
show vhba vhba101.myserver targets
vhba
name
wwnn
wwpn
luns
--------------------------------------------------------------------------------vhba101.myserver
20:00:00:20:37:8D:03:7D 21:00:00:20:37:8D:03:7D 0
vhba101.myserver
20:00:00:20:37:D5:37:18 21:00:00:20:37:D5:37:18 0
vhba101.myserver
20:00:00:20:37:CC:EB:30 21:00:00:20:37:CC:EB:30 0
vhba101.myserver
20:00:00:20:37:C6:5E:B4 21:00:00:20:37:C6:5E:B4 0
vhba101.myserver
20:00:00:20:37:90:88:90 21:00:00:20:37:90:88:90 0
vhba101.myserver
20:00:00:20:37:B3:F0:5C 21:00:00:20:37:B3:F0:5C 0
vhba101.myserver
20:00:00:20:37:D5:45:FD 21:00:00:20:37:D5:45:FD 0
vhba101.myserver
20:00:00:20:37:C9:1D:C2 21:00:00:20:37:C9:1D:C2 0
8 records displayed
This show targets command will not list the targets by the order specified in the persistent mapping. If you
want to verify this order, you need to check the host side.
Example 2
The persistent binding can be assigned while creating a vHBA, which is provided to you as a configuration convenience:
add server-profile myserver myserver,ServerPort13
add vhba vhba999.myserver 4/1 -map=mymap
Example 3
To remove a vHBA, server profile, and SAN map in the correct order:
remove -noconfirm vhba vhba101.myserver
remove -noconfirm server-profile myserver
remove -noconfirm san map mymap
Option: If you only want to remove “mymap”, you need to remove the associated vHBA. Skip the 2nd step (removal of
myserver) as shown above.
To check if any SAN map is remaining:
show san map
Nothing to display
Xsigo Systems VP780
73
LUNs Per Target
Note: Expect an error if you remove a SAN map without first unbinding the vHBA:
remove -noconfirm san map mymap
Commit failed: Cannot delete Persistent Mapping Set :mymap. Currently in use by
Vhba: vhba101 (error 111)
Example 4
Do not bind a map to a vHBA that has been already deployed. If you do, you will get an error:
set vhba vhba100.myserver -map=mymap
Commit failed: Cannot change Persistent Mapping reference systemlocal:san:persistent-mapping-Set-mymap while VHBA is deployed. Please make sure
that VHBA is not deployed. (error 118)
LUNs Per Target
A Logical Unit Number (LUN) is a unique ID used to identify a device (i.e., disk array) in a fibre channel network.
Syntax
set vhba <vhba-name> -luns-per-target=<value>
show vhba <vhba-name> targets
show -list <vhba-name> -detail
-luns-per-target=<value>—For the Xsigo hostdrivers, sets the number of LUNs to use per given target.
By default, the XgOS supports a maximum of 256 LUNs. See show vhba <vhba-name> targets for the
number of LUNs discovered by the XgOS.
Example
Take the following steps to configure -luns-per-target for a vHBA:
Step 1
Define a target value for a vHBA:
set vhba foo.finance -luns-per-target=128
Step 2
Use the -detail keyword to display the number of configured LUNs-per-target:
show -list vhba foo.finance -detail
----------------------------------------------------------------------name
foo.finance
state
resourceUnavailable
descr
if
8/2
wwnn
50:01:39:71:00:00:81:01
wwpn
50:01:39:70:00:00:81:01
luns-per-target 128
cmds-per-lun
8
map
flags
Xsigo Systems VP780
74
Chapter 5: vHBAs
----------------------------------------------------------------------1 record displayed
Target Prescan and Rescan
Target prescan and rescan enables you to discover the available target and LUN information on the network without
requiring a host server to be bound to the VP780. Use this feature to determine if the list of targets and LUNs are
satisfactory, or require any removals or additions, before committing them (binding) to a host-server profile. The XgOS
then supports binding the server profile with the phys-con after a prescan is complete.
The VP780 relies on fibre channel’s Registered State Change Notification (RSCN) to send target-state updates from the
remote switch to the VP780. The VP780’s IOP learns the update and notifies the host server of any changes. However
note that RSCN is turned off by default on some fibre-channel switches.
RSCN does not support reporting LUN state changes (add or remove). As a workaround for this RSCN limitation, you
must manually run rescan for a vHBA to detect any LUN level changes.
Syntax
set vhba <vhba-name>.<server-profile> prescan
set vhba <vhba-name>.<server-profile> remove-prescan
set vhba <vhba-name>.<server-profile> rescan
show vhba <vhba-name>.<server-profile> targets
prescan—Configures prescan state for an unbound vHBA.
remove-prescan—Removes a prior configured prescan state, which is required in order to re-issue a new
prescan state. Once you issue a prescan, the configuration resides on the I/O card. The system is incapable of
receiving any LUN changes through RSCN. You can issue prescan several times. However to detect LUN changes,
the prior prescan state must be removed (remove-prescan) from the vHBA before you can re-issue prescan
again.
rescan—Configures rescan state for a bound vHBA. RSCN does not support reporting LUN state changes. As a
workaround for this RSCN limitation, you must manually run rescan for a vHBA to detect LUN changes.
Example 1: prescan
To enable prescan for an unbound vHBA:
Step 1
Create an unbound server profile, where the state is “unassigned”:
add server-profile III
show server-profile III
------------------------------------------------------------name
III
state
up/unassigned
...
Step 2
Create a vHBA under this unbound server:
Xsigo Systems VP780
75
Target Prescan and Rescan
add vhba vhbaiii.III 4/1
At this point, show vhba <vhba-name>.<server-profile> will report the state as
“resourceUnavailable”, which is expected. The vHBA is not bound to a server.
Step 3
Set this vHBA to prescan state, which propagates target discovery to the FC I/O card
(sanFc2Port4GbCard) on the VP780:
set vhba vhbaiii.III prescan
Step 4
Display the discovered targets and LUNs in the network environment. If you add or remove a target on
the array side, those changes will be reflected accordingly on the VP780 through RSCN:
show vhba vhbaiii.III targets
vhba
name wwnn
wwpn
lun-ids
-----------------------------------------------------------------------vhbaiii.III
2F:9F:00:06:2B:10:C3:BA 2F:9F:00:06:2B:10:C3:BA 3,2,1,0
vhbaiii.III
2F:BF:00:06:2B:10:C3:BA 2F:BF:00:06:2B:10:C3:BA 3,2,1,0
vhbaiii.III
2F:DF:00:06:2B:10:C3:BA 2F:DF:00:06:2B:10:C3:BA 3,2,1,0
vhbaiii.III
2F:FF:00:06:2B:10:C3:BA 2F:FF:00:06:2B:10:C3:BA 3,2,1,0
4 records displayed
show vhba vhbaiii.III
-----------------------------------------------------------------------name
vhbaiii.III
state resourceUnavailable
descr
if
4/1
wwnn
50:01:39:71:00:00:F1:02
wwpn
50:01:39:70:00:00:F1:02
map
Example 2: Bind After Prescan
The ideal scenario is to bind the prescan-discovery results to a host server. The XgOS supports binding the server profile
with the phys-con after a prescan is complete, as long as you follow the correct configuration order.
Follow these steps to perform a prescan then bind the server profile:
Step 1
Create an unbound server profile:
add server-profile III
Step 2
Create a vHBA under this unbound server:
add vhba vhbaiii.III 4/1
Step 3
Set this vHBA to prescan state:
Step 4
Display the targets:
From now on if there are any RSCN changes, the targets will also be updated accordingly.
Note: At this point, you can also specify the target order by integrating persistent mapping with prescan.
See Persistent Binding on page 70. If you do, be sure to issue remove-prescan before binding.
Step 5
If you are satisfied with the results, you can bind the server-profile now:
Xsigo Systems VP780
76
Chapter 5: vHBAs
set server-profile III -add-phys-con=titan,ServerPort23
Step 6
From now on, this vHBA has become a normal vHBA. You can run rescan against it:
set vhba vhbaiii.III rescan
but then you cannot run prescan against this normal vHBA any longer.
Example 3: remove-prescan
You can issue prescan several times. However to detect LUN changes, the prior prescan state must be removed
(remove-prescan) from the vHBA before you can re-issue prescan again:
set vhba vhbaiii.III remove-prescan
Example 4: rescan
RSCN does not support reporting LUN state changes. For the VP780 to detect LUN changes, you must manually run
rescan for a vHBA. This action is a workaround for the RSCN limitation.
To detect LUN changes for a bound (normal) vHBA:
Step 1
Create a bound server profile:
add server-profile titan titan,ServerPort23
Step 2
Create a vHBA under this bound server:
add vhba vhba888.titan 4/1
Step 3
Display the targets:
show vhba vhba888.titan targets
Step 4
Configure this vHBA to rediscover (rescan state) the available LUN information. If there are any LUN
changes, they will be reflected after this rescan operation:
set vhba vhba888.titan rescan
Step 5
Display any new target and LUN information:
show vhba vhba888.titan targets
Xsigo Systems VP780
77
Set FC Port Attributes
Set FC Port Attributes
Each FC port is controlled by a backend logic chip. There is a set of attributes and properties that can be controlled from
the command line:
-----------------------------------------------------------------------------name
8/2
type
sanFc1GbPort
state
up/down
descr
wwnn
50:01:39:71:00:00:80:49
wwpn
50:01:39:70:00:00:80:49
rate
AutoNegotiate/0
frame-size
2048/2048
exec-throttle 65535
int-delay
1000
link-down
0
login-retry
8
login-timeout 4
port-down
8
topo
F
loop-delay
5
tape-support
true
vhbas
0
-----------------------------------------------------------------------------The most commonly used fibre-channel controls are rate, topology, frame-size, and execution-throttle. However, note that
modified attributes do not take effect until you reset the I/O card. See the example that follows.
Syntax
set
set
set
set
set
set
set
set
set
set
set
set
fc-port
fc-port
fc-port
fc-port
fc-port
fc-port
fc-port
fc-port
fc-port
fc-port
fc-port
fc-port
{*
{*
{*
{*
{*
{*
{*
{*
{*
{*
{*
{*
|
|
|
|
|
|
|
|
|
|
|
|
<slot>/<port>}
<slot>/<port>}
<slot>/<port>}
<slot>/<port>}
<slot>/<port>}
<slot>/<port>}
<slot>/<port>}
<slot>/<port>}
<slot>/<port>}
<slot>/<port>}
<slot>/<port>}
<slot>/<port>}
-descr=<text>
-execution-throttle={<number> | default}
-frame-size={512 | 1024 | 2048 | default}
-interrupt-delay-timer={<number> | default}
-link-down-timeout={<number> | default}
-login-retry-limit={<number> | default}
-login-timeout={<number> | default}
-loop-reset-delay={<number> | default}
-port-down-retry-limit={<number> | default}
-rate={1Gb|2Gb|4Gb|AutoNegotiate|default}
-tape-support={false | true | default}
-topology={F | L | default}
show ioport {* | <slot>/<port>}
-descr=<text>—Applies a text description to the FC port. Quotes are required around multiple words
containing spaces in between.
-execution-throttle={<number> | default}—The maximum number of FC I/O commands that can
be processed concurrently by a specific physical FC port at any time. This throttle is the number of I/O
commands pending per target/LUN. Enter a <number> between 1 and 65535. The default is 65,535.
Xsigo Systems VP780
78
Chapter 5: vHBAs
-frame-size={512 | 1024 | 2048 | default}—The size of the FC frame that traverses the port.
The default is 2048. Note that a lower frame size reduces the available bandwidth.
-interrupt-delay-timer={<number> | default}—An FC port processes large numbers of frames.
The interrupt-delay-timer is the amount of time the FC port must wait before interrupting the software to inform
it that a frame has not been processed/serviced (an error condition). Specify a <number> from 0 to 100,000
where each number (unit) is 100 micro seconds. The default is 1000 (100,000 micro seconds).
Xsigo recommends you not change the default. Setting a value that is too small could result in a barrage of
unwanted interrupts.
-link-down-timeout={<number> | default}—When a fibre link goes down, the FC port will wait
(delay) this <number> of seconds before declaring the fibre link down. Specify a value between 0 and 255
seconds. The default is 0.
-login-retry-limit={<number> | default}—Before any transactions can be started with a remote
device, the VP 780’s FC protocol must login to that device and establish a session. When that login does not
succeed, the login-retry-limit will be performed this <number> of times before declaring the login
attempt a complete failure. The default is 8.
-login-timeout={<number> | default}—A resource allocation timeout. When the FC port sends
control frames, it expects an acknowledgement (reply) from the remote side within this <number> of seconds
before declaring a communication error. The default is 4 seconds.
-loop-reset-delay={<number> | default}—Enter a <number> between 0 and 255. The default is
8 seconds.
-port-down-retry-limit={<number> | default}—Enter a <number> between 0 and 60.
The default is 5 seconds.
-rate={1Gb | 2Gb | 4Gb | AutoNegotiate | default}—The bandwidth rate of the FC port.
The default is AutoNegotiate. Based on the supported rate of the remote switch, the local FC port will
auto negotiate to that capable speed.
-tape-support={false | true | default}—Support for tape devices and cartridges, such as those
used for random access data storage and archiving. The default is true. Tape support is enabled by default.
-topology={F | L | FL | default}—The configured topology type. It can be switched fabric (F), arbitrated
loop (L), or both (FL). The default is F. An auto-negotiated topology is not supported for HBA virtualization.
show ioport—Displays the configured settings of an FC port.
* | <slot>/<port>—The physical slot and port coordinate to be configured. An asterisk (*) specifies all
available FC cards.
Example
Note that modified settings do not become effective until you reset the I/O card. To adopt new settings, the card must be
brought down, rebooted, and re initialized using the set iocard command:
show ioport
name
type
state
descr
vnics
vhbas
--------------------------------------------------------------------------4/1
sanFc1GbPort
up/up
0
1
4/2
sanFc1GbPort
up/down
0
0
5/1
sanFc1GbPort
up/up
0
1
5/2
sanFc1GbPort
up/up
0
0
Xsigo Systems VP780
79
vHBA Statistics
9/1
nwEthernet1GbPort
up/up
0
0
9/2
nwEthernet1GbPort
up/down
0
0
9/3
nwEthernet1GbPort
up/down
0
0
9/4
nwEthernet1GbPort
up/down
0
0
8 records displayed
set fc-port 4/1 -login-timeout=10
set iocard 4 down
Are you sure you want to shutdown the IO card in slot 4 (y/n)? y
set iocard 4 up
show ioport 1/1
-----------------------------------------------------------------------------name
4/1
type
sanFc1GbPort
state
up/up
descr
wwnn
50:01:39:71:00:00:80:01
wwpn
50:01:39:70:00:00:80:01
rate
AutoNegotiate/0
frame-size
2048/2048
exec-throttle 65535
int-delay
1000
link-down
0
login-retry
8
login-timeout 10
port-down
8
topo
F
loop-delay
5
tape-support
true
vhbas
1
vHBA Statistics
show vhba vhba1.crawford stats
-----------------------------------------------------------------------------name
vhba1.crawford
total-io
27136
read-byte-count
3380540138
write-byte-count
0
outstanding-request-count
0
io-request-count
27136
read-request-count
27042
write-request-count
0
task-management-request-count
94
target-count
36
lun-count
0
xsmp-xt-down-count
3
xsmp-xt-oper-state-request-count
4
map-fmr-count
27042
ummap-fmr-count
27042
used-map-fmr-count
0
abort-command-count
0
reset-lun-command-count
0
Xsigo Systems VP780
80
Chapter 5: vHBAs
reset-target-command-count
0
reset-bus-command-count
0
link-down-count
1
disc-info-update-count
3
target-lost-count
0
target-found-count
0
cqp-disconnect-count
4
dqp-disconnect-count
4
cqp-ib-snd-err-count
1
dqp-ib-snd-err-count
0
cqp-ib-rcv-err-count
0
dqp-ib-rcv-err-count
0
cqp-ib-remote-disconnect-err-count 0
dqp-ib-remote-disconnect-err-count 0
FC Monitoring
Use show fc-port to display fibre channel information.
Use set fc-port to control the FC settings. See Set FC Port Attributes on page 77.
Syntax
show fc-port
show fc-port {* | <slot>/<port>}
show fc-port {* | <slot>/<port>} [alarms][errors][qos][stats][vhbas][warnings]
Examples
show fc-port
name
type
state
descr
v-resources
---------------------------------------------------------------------1/1
sanFc1GbPort
up/down
0
1/2
sanFc1GbPort
up/down
1
8/1
sanFc1GbPort
up/up
0
8/2
sanFc1GbPort
up/down
0
4 records displayed
show fc-port 8/1
---------------------------------------------------------------------name
8/1
type
sanFc1GbPort
state
up/up
descr
wwnn
50:01:39:71:00:00:80:47
wwpn
50:01:39:70:00:00:80:47
rate
AutoNegotiate/2
frame-size
2048/2048
exec-throttle 65535
int-delay
1000
link-down
0
Xsigo Systems VP780
81
Admin State Control
login-retry
8
login-timeout 4
port-down
8
topo
F
loop-delay
5
tape-support
true
vhbas
0
---------------------------------------------------------------------1 record displayed
show fc-port 8/1 stats
---------------------------------------------------------------------name
8/1
controller-errs
0
device-errs
0
link-fails
0
loss-of-syncs
1
loss-of-signals
0
primitive-seq-protocol-errs 0
transmission-word-errs
0
crc-errs
0
---------------------------------------------------------------------1 record displayed
Admin State Control
Use set vhba up | down to control the administrative state of a configured vHBA.
Syntax
set vhba <vhba-name>.<server-name> up
set vhba <vhba-name>.<server-name> down
up—Activates a vHBA (default)
down—Deactivates a vHBA
Example
show vhba myvhba.myserver
set vhba myvhba.myserver down
Are you sure you want to deactivate VHBA myvhba.myserver (y/n)?y
show vhba myvhba.myserver
Xsigo Systems VP780
82
Chapter 5: vHBAs
LUN Masking
Logical Unit Number (LUN) Masking is an authorization feature that makes LUNs available to some vHBAs but not to
others. When you apply a LUN mask to a vHBA, only that one vHBA on the host can detect the LUNs.
The standard location to configure LUN masking is on the disk array itself. In Xsigo’s implementation, the VP780
configures LUN masking in a centralized SAN location—the vHBA (not the disk array):
Windows A, Customer 1
Disk Array
SAN Network
vHBA-A
Windows A
LUNs
Linux B
LUNs
vHBA-B
Linux B, Customer 2
add san lun-mask <mask-name> target <wwpn> <luns>
add vhba <vhba> <slot>/<port> -lun-mask=<mask-name>
set vhba <vhba>.<server-profile> -lun-mask=[<mask-name> | none]
Figure 2 LUN Masking Applied to vHBAs
In Figure 2, the VP780 controls which LUNs can be seen by the vHBAs. To accomplish this, the VP780 deploys different
vHBA policies (vHBA-A, vHBA-B) to maintain LUN security. When a vHBA is created, a different LUN mask is
assigned.
RSCN does not report LUN state changes. Whenever the LUN masking changes on an existing vHBA, you must also
issue a rescan on the VP780 to send an RSCN update. See Parameter Description on page 83 for details.
When LUN Masking is enabled, the SCSI “report luns” command will be intercepted and processed by the vHBA host
software and VP780. For more details, see Optional LUN Masking: No Report LUN Interception on page 85.
If a storage controller fails to register its new LUN settings with the fibre channel fabric name server, you might have to
trigger an RSCN in addition to the rescan on the VP780.
Caution
Windows-based servers attempt to write volume labels to all available LUNs. This action can render the
LUNs unusable by other operating systems and can result in data loss.
Syntax
add san lun-mask <mask-name> target <wwpn> <luns>
add vhba <vhba>.<server-profile> <slot>/<port> -lun-mask=<mask-name>
set vhba <vhba>.<server-profile> -lun-mask=[<mask-name> | none]
show vhba <vhba>.<server-profile> lun-mask [-detail]
show vhba <vhba>.<server-profile> targets
Xsigo Systems VP780
83
LUN Masking
By default LUN masking is not applied to a vHBA. All LUNs are visable by default.
add san lun-mask <mask-name> target <wwpn> <lun-range>—A named SAN LUN mask to create.
A tuple of target WWPN and LUN IDs is required. A <lun-range> can be a single LUN ID or a range of LUN
IDs. The range may contain multiple LUN IDs separated by commas or continuous IDs separated by a colon. For
example 1,5,6:9,34 means LUN IDs 1,5,6,7,8,9,34. A set vhba rescan is required each time LUN IDs
change.
add vhba <vhba>.<server-profile> <slot>/<port> -lun-mask=<mask-name>—Creates a
vHBA and specifies a LUN mask to be seen. Only these LUNs are allowed to be discovered over this vHBA.
set vhba <vhba>.<server-profile> -lun-mask=[<mask-name> | none]—Changes the LUN
masking on an existing vHBA. However note a set vhba rescan is required following a set vhba -lunmask. The none option removes an applied LUN masking from a vHBA.
show vhba <vhba>.<server-profile> lun-mask—Displays configured LUN mask information.
show vhba <vhba>.<server-profile> targets—Verifies if your LUN masking is working.
Example
Create a LUN Mask named "oracle-mask" with target WWPN "20:70:00:C0:FF:0A:81:30" and LUN ID "11":
add san lun-mask oracle-mask target 20:70:00:C0:FF:0A:81:30 11
Create a server profile and bind it to a physical connection:
add server-profile testlin2 testlin2@washington:ServerPort13
Create a vhba and bind the LUN Mask "oracle-mask" to it:
add vhba oracle-vhba1.testlin2 1/1 -lun-mask=oracle-mask
Now check to see the mask is correct. From the below output, we can see the target is masked with LUN 11. LUN 0 is
always shown. In case no physical LUN 0 was created, it will be a synonym of storage controller:
show vhba oracle-vhba1.testlin2 targets
vhba
name wwnn
wwpn
lun-ids
-------------------------------------------------------------------------------oracle-vhba1.testlin2 20:70:00:C0:FF:0A:81:30
20:70:00:C0:FF:0A:81:30
11,0
1 record displayed
In the case the storage device has two targets and each target has multiple LUNs, we will see:
vhba
name
wwnn
wwpn
lun-ids
--------------------------------------------------------------------------------oracle-vhba1.testlin2
20:70:00:C0:FF:0A:81:30 20:70:00:C0:FF:0A:81:30
11,0
oraclevhba1.testlin2
20:78:00:C0:FF:0A:81:30 21:78:00:C0:FF:0A:81:30 9,8,7,6,5,4,3,
0
2 records displayed
Next, we choose to include LUN 9 of the second target to the mask "oracle-mask":
add san lun-mask oracle-mask target 21:78:00:C0:FF:0A:81:30 9
Xsigo Systems VP780
84
Chapter 5: vHBAs
Display the settings of the LUN Mask "oracle-mask":
show san lun-mask oracle-mask
name
descr
targets
-------------------------------------------------------------------------------------------------------------oracle-mask
21:78:00:C0:FF:0A:81:30(0,9),
20:70:00:C0:FF:0A:81:30(0,11)
1 record displayed
Display the LUNs that vHBA "oracle-vhba1" is allowed to see:
show vhba oracle-vhba1.testlin2 lun-mask
vhba
name
descr
targets
--------------------------------------------------------------------------------oracle-vhba1.testlin2
oracle-mask
21:78:00:C0:FF:0A:81:30(0,9),
20:70:00:C0:FF:0A:81:30(0,11)
1 record displayed
However, before the rescan, the change will not take effect:
vhba
name
wwnn
wwpn
lun-ids
--------------------------------------------------------------------------------oracle-vhba1.testlin2 20:70:00:C0:FF:0A:81:30
20:70:00:C0:FF:0A:81:30
11,0
oracle-vhba1.testlin2
20:78:00:C0:FF:0A:81:30
21:78:00:C0:FF:0A:81:30
9,8,7,6,5,4,3,0
2 records displayed
Issue the rescan command:
set vhba oracle-vhba1.testlin2 rescan
After rescan, display the settings of the LUN Mask "oracle-mask" on vHBA "oracle-vhba1"; nothing changes here:
show vhba oracle-vhba1.testlin2 lun-mask
vhba
name
descr
targets
--------------------------------------------------------------------------------oracle-vhba1.testlin2 oracle-mask
21:78:00:C0:FF:0A:81:30(0,9),
20:70:00:C0:FF:0A:81:30(0,11)
1 record displayed
After rescan, display the LUNs that vHBA "oracle-vhba1" can see. Now the mask takes effect:
vhba
name
wwnn
wwpn
lun-ids
-------------------------------------------------------------------------------oracle-vhba1.testlin2
20:70:00:C0:FF:0A:81:30 20:70:00:C0:FF:0A:81:30
11,0
oracle-vhba1.testlin2
20:78:00:C0:FF:0A:81:30 21:78:00:C0:FF:0A:81:30
9,0
2 records displayed
Xsigo Systems VP780
85
Optional LUN Masking: No Report LUN Interception
Optional LUN Masking: No Report LUN Interception
Starting in Release 1.5, LUN Masking is mandatory for vHBAs by default. When a host (Linux or Windows) issues a
SCSI report LUNs, the chassis filters the response based on what is in the VP780 database. If LUN masking changes in an
array and a host issues a report LUNs, the new LUN will not be available to the host until a set vhba rescan
command is run on the VP780. In some cases, this approach goes against customer expectations and breaks the existing
model.
Use the -no-lun-masking feature to disable the LUN masking so that if you choose to do LUN masking on arrays,
rescans on the VP780 are not required. Specifically the -no-lun-masking feature disables the “report luns”
interception and allows all new LUN/target information to pass through directly to SCSI. When SCSI issues the “report
luns” command, the request will pass through the VP780’s IOP and discover the disk array’s new LUN/target information.
Host Server
SCSI
IB
VP780
FC card
-no-lun-masking
FC
IOP
SAN Network
report-luns pass through,
no interception: control-queue pair, data-queue pair
Figure 3 Report LUNs Pass Through
When a vHBA is created, LUN Masking is enabled by default. An administrator must use –no-lun-masking to
disable it. The –no-lun-masking flag can be specified only during the creation of a vHBA and cannot be changed
throughout the lifetime of this vHBA. After specifying this flag while creating a vHBA, the CLI will also prevent you
from assigning any lun-mask to this vHBA.
Syntax
add vhba <name>.<server> <slot>/<port> -no-lun-masking
Examples
To determine if LUN Masking is enabled for a vHBA, see the “l” value under “flags”. This filed means LUN Masking is
enabled:
add vhba bar.myserver 1/2
show vhba bar.myserver -detail
-----------------------------------------------------------------------------name
bar.myserver
state
up/resourceUnavailable
fabric-state
indeterminate
descr
if
1/2
if-state
down
Xsigo Systems VP780
86
Chapter 5: vHBAs
wwnn
wwpn
luns-per-target
cmds-per-lun
map
lun-mask
flags
local-id
50:01:39:71:00:00:81:01
50:01:39:70:00:00:81:01
256
8
--l
0
Use -no-lun-masking to disable LUN Masking on a newly added vHBA:
add vhba bar.myserver 1/1 -no-lun-masking
When LUN Masking is disabled, the CLI prevents you from assigning any LUN masking setting:
add vhba vhba888.titan 4/1 -no-lun-masking
set vhba vhba888.titan -lun-mask=oneida1
Commit failed: Please enable Lun Mask before setting Lun Mask (error 118)
Xsigo Systems VP780
vSSLs
87
Introduction
This section documents an example of creating a virtual Secure Socket Layer (vSSL) object. In this example, you will:
•
Create a vSSL card
•
Give the vSSL a name
•
Assign the vSSL to a server profile
•
Bind the vSSL to a physical SSL module
•
Verify the configuration
Configuration Procedure
Take the following steps:
Step 1
Create a vSSL object named “vssl_1” and assign it to the server profile “vserver1”:
add vssl vssl_1.vserver1 ?
13 sslNonProxy in slot 13
Step 2
Bind the vSSL card to a physical SSL card. Select the number of the slot containing the physical SSL
card that you want to link to the vSSL (in this example, “13”):
add vssl vssl_1.vserver1 13
Step 3
Verify the vSSL card configuration was correctly created correctly:
show vssl
------------------------------------name
vssl_1.vserver1
state
resourceUnavailable
descr
slot
13
bandwidth
1000
max-cons
10000
group
group-pref none
num-apps
0
--------------------------------------
Xsigo Systems VP780
88
Chapter 6: vSSLs
Xsigo Systems VP780
VMware ESX Servers
89
Introduction
From the VP780’s viewpoint, a VMware ESX server appears and works similar to a standard server. Simply add a server
profile and a vNIC or vHBA. All the configuration for vswitches and attaching virtual machines to network resources
occurs within the VMware Infrastructure Client (provided by VMware as part of the ESX server package).
The following comes into play when configuring the system:
•
Local ID—The identity of a vNIC on the ESX server. A Local ID also applies to vHBAs but it’s not as
significant. The mapping of network interfaces (vNIC to vswitch) has security implications whereas the
direct-mapping order of vHBAs is still present but of lesser concern.
•
Pre Defined vNICs—The Local ID maps vNICs into 32 pre defined vNIC names (vnic 1 through vnic 32) on
the ESX server. Unlike on standard Linux servers, you cannot pick your own vNIC name. A Local ID allows
you to specify which of those 32 pre installed vNICs you are going to use. Issue ifconfig after you install
the hostdrivers to see 32 vNICs not added or attached to anything. These are placeholders for when the
interfaces are associated to Virtual Machine Networks.
•
Pre Defined vHBAs—In the configuration section of VMware Infrastructure Client, a list of 12 virtual storage
adaptors are pre installed as soon as you load the Xsigo hostdrivers. A WWN appears next to the adaptors
that are configured for the VP780.
•
HA vNICs—High Availability (HA) vNIC support is handled through NIC Teaming. Use the VMware
Infrastructure Client to configure a teamed pair of vNICs. These two network interfaces attach to the same
vswitch.
Check the XgOS release notes for VMware limitations and workarounds.
Note
CLI Support
Create a server profile:
add server-profile <profile-name> <server-name>@<vp780-hostname>:<ib-port>
then add a vNIC or vHBA with a local-id value:
add vnic <vnic>.<profile-name> <slot>/<port> -local-id=<value>
add vhba <vhba>.<profile-name> <slot>/<port>
A local-id maps a vNIC into 32 pre defined vNIC names (vnic1 through vnic32) on the ESX server. A local-id for a
vHBA is rarely used. See Introduction on page 89.
On the ESX, there are several useful CLI commands:
esxcfg-xgmap
esxcfg-vswitch
esxcfg-vmhbadevs
Xsigo created esxcfg-xgmap but not the others. See the following configuration example for more details.
Xsigo Systems VP780
90
Chapter 7: VMware ESX Servers
Configuration Example
The ESX server in Figure 1 has four virtual machines (Service Console, bob, fred, joe). Each VM has Ethernet interfaces
(eth0 ... ethN), a vswitch, and belongs to a Virtual Machine Network. VNICs will appear as “vnic1”, “vnic2”, ... “vnic32”.
You can have any number of vswitches (vswitchN), and any given vswitch can associate with any number of vNICs:
Service eth0
Console
vswitch1
vnic1
Virtual Machine Network 1
eth1
bob
vswitch2
eth2
vnic2
Virtual Machine Network 2
eth3
fred
..
.
vswitchN
ethN
joe
vnic32
Interfaces
Vswitches
vNICs
Virtual
Machines
Figure 1 VMs, Vswitches, Virtual Machine Networks, and vNICs on an ESX Server
Take the following steps to enable vNIC communication between the ESX server and VP780:
Step 1
Install the InfiniBand RPM on the ESX server:
rpm -ivh VMware-esx-commsrc-infiniband-release-3.5.0-1.09.60.rev401.i386.rpm
Linux ships with its own IB drivers, but the ESX server does not. This IB RPM file must be installed
before the Xsigo ESX Commsrc file (next step).
Step 2
Install the Xsigo VMware hostdrivers on the ESX server:
rpm -ivh VMware-esx-commsrc-xsigo-release-3.5.0-v99x1.5.0_RC4E.i386.rpm
reboot
Xsigo Systems VP780
91
Configuration Example
Step 3
On the VP780, create a vNIC to use with the ESX server:
add server-profile myserver vmware@iowa:ServerPort23
add vnic myvinc.myserver 4/1 -local-id=4
If you do not specify a local-id when adding a vNIC, the ESX will assign one for you.
The vNIC’s addressing is not added on the VP780 side. VMware configures and manages the addressing.
Note: Release 1.5 has a limitation. You must add and attach a server profile first. If you add vNICs and
vHBAs to a server profile before you attach it (physconn) to the ESX server, the server profile will not work
properly. See Caveats on page 93 for more details and the workaround.
Step 4
Create a Virtual Machine Network using the VMware Infrastructure Client. The network Ethernet name
on the ESX server corresponds to the vNIC local-id configuration on the VP780. For example, local-id 1
corresponds to “vnic1”. Local-id 2 is “vnic2” and so on:
Figure 2 VMware Infrastructure Client Used with the VP780
Xsigo Systems VP780
92
Step 5
Xsigo created an XMS plugin for VMware Virtual Center. The plugin runs the XMS web interface. It
enables you to display and manage your virtual I/O via a VMware Infrastructure Client connection to
VMware Virtual Center.
To use the plugin, install the Xsigo ISO or zip file to VMware Virtual Center:
xms-plugin4vc-1.5.0_RC5D.iso
xms-plugin4vc-1.5.0_RC5D.zip
Once installed, add the Xsigo Virtual I/O plugin under the plugin tab. Select the “Virtual I/O” tab or click the
“Xsigo” icon to use the tool.
Xsigo XMS plugin
Xsigo Systems VP780
93
Caveats
Step 6
From the VP780, monitor the health of the vNICs:
show vnic <vnic>.<server> -detail
All configuration can be done via the VMware Infrastructure Client. However on the ESX server, there are
many useful CLI commands available to you.
To find the device mapping between the pre installed virtual resources and the ones that are attached into the
VP780:
esxcfg-xgmap
vh0 -> vmhba32
vh1 -> vmhba34
vn10 -> vnic10
vn11 -> vnic11
vn12 -> vnic12
....
The vNIC will need to be connected to a vswitch either through the ESX’s GUI or through the esxcfgvswitch command to uplink the vNIC and list it:
esxcfg-vswitch –L vnic1 vSwitch1
esxcfg-vswitch –l
The esxcfg-vswitch command provides an interface for adding, removing, and modifying virtual
switches and their settings. By default, there is a single virtual switch called “vSwitch0”.
The esxcfg-vmhbadevs command provides information about the LUNs available on the ESX server.
By default, the command will print a mapping of vmhbaX:X:X names to console /dev/ names:
esxcfg-vmhbadevs
vmhba0:0:0
/dev/sda
vmhba32:2:1
/dev/sdd
vmhba32:2:2
/dev/sde
vmhba32:2:3
/dev/sdf
vmhba32:2:4
/dev/sdg
...
Caveats
You must explicitly set the local-id on vnics and vhbas that are added to an unattached or administratively down server
profile. The local-id will be set automatically on resources that are added to an active server profile only.
Example:
add
add
add
set
server-profile server1
vnic vnic1.server1 1/1 -local-id=1
vhba vmhba34.server1 1/1 -local-id=3
server-profile server1 connect foo@bar:ServerPort1
Resources added to down or unconnected server-profiles without the local-id set will remain in the resourceUnavailable
state, and must be removed and re-added.
Xsigo Systems VP780
94
Xsigo Systems VP780
Xen Hypervisor
95
Introduction
A hypervisor is a virtualization platform that allows multiple guest operating systems to operate. Xen is a para virtualizing
hypervisor—a Virtual Machine Monitor (VMM). Xen can host multiple guest operating systems, each of which is
executed within a secure VM (in Xen terminology, a domain). Domains are scheduled by Xen to make effective use of the
available physical CPUs.
For Xen deployments, the VP780 delivers vNICs into the hypervisor layer and connects them into VMs via virtual
switches (vswitches, also known as Xen bridges). Ultimately, the vNICs appear as devices within the VMs:
VM1
VM2
vswitch
VM3
Xen Hypervisor Layer
vNIC
Figure 1 Configuration Model for Xen Hypervisor
The XgOS supports managing Xen VMs from the VP780. The following capabilities are supported:
•
Attaching vNICs and vHBAs to the DomUs
•
Discovering existing VMs
•
Controlling the bridging configuration via vswitches
•
VM MAC-address memory
vNIC
vNICs can be added to a virtual server running Xen. The XgOS can also attach configured vNICs to Xen VMs by means
of the native linux bridging support.
vNICs can be attached to a VM that is configured on the server by first configuring a vswitch. In the case of Xen VMs, the
vswitch should be named xenbrN (where N is an integer 1 or greater). The bridge will be visible on the server with the
brctl show command.
To attach a vNIC to a vswitch:
set vswitch <vswitch-name>.<server-name> attach vnic <vnic-name>
Xsigo Systems VP780
96
Chapter 8: Xen Hypervisor
To attach a vswitch to a VM:
set vswitch <vswitch-name>.<server-name> attach vm <vm-name>
For more information on bridged network interfaces and Xen VMs, please consult the Xen documentation included in
your Linux distribution.
vHBA
vHBAs can be added to a virtual server running Xen. Block devices discovered by the system can be attached to DomUs
normally (via the configuration file, or xm block-attach).
All block devices discovered over a particular vHBA can be attached to a DomU via the xgxen block-attach
command. The xgxen utility provides server-side support for managing Xen DomUs and Xsigo virtual resources.
CLI Support
Use the following vm and vswitch commands to configure and manage the Xen system.
Syntax
add vswitch <vswitch-name>
set vswitch <vswitch-name>.<server-name> attach {vm <name>|vnic <name>}
set vswitch <vswitch-name>.<server-name> detach {vm <name>|vnic <name>}
set vnic <name> attach vswitch
set vnic <name> dettach vswitch
remove vswitch <name> [-noconfirm]
show vswitch <name> [vms][vnics]
add vm <name> -type= [vmware | xen | virtualServer | none]
add vm <name> -virt-type= [application | default | full | none | os | para]
set vm <name> attach vswitch <name>
set vm <name> detach vswitch
set vm <name> down
set vm <name> up
set vm <name> migrate [* | <server-profile>] <savedir>
set vm <name> restart
set vm <name> restore <path>
set vm <name> save <path>
set vm <name> resume
set vm <name> suspend
remove vm <name> [-noconfirm]
show vm [<name>]
Optional qualifiers: [-descr=<text>][-managed=[default|false|true]][-nowait=][-virt-type=]
add vswitch <vswitch-name>—Adds a named bridge (virtual switch), where <name> is in the format
vswitch.server-profile.domain. A vswitch name is limited to 15 characters long.
-virt-type—The virtualization type. The para option specifies a para virtualized system. The full option
specifies a fully virtualized system. The default is para.
Xsigo Systems VP780
97
Example 1
migrate [* | <server-profile>] <savedir>—Migrates the VM to a different server profile (and server).
-managed=—Enables the VP780 to manage the vswitch or VM entity. When not enabled, the VP780 only
discovers and displays the entities.
-virt-type=—Virtualization type to configure. The default is para.
save <path>—Saves the state of a VM to a file. Use restore <path> to restore the state of a VM.
suspend—Pause the VM.
resume—Start the VM.
Example 1
This example applies to Linux systems running para virtualized hypervisors (not fully virtualized) under Linux systems.
Xen virtual interfaces connect to Xen bridges. The Dom0 virtual machine manages all the network interfaces, including
the ones added by the VP780. There is a direct correlation between the vswitches configured on the VP780 and the
bridges configured on the Linux host.
The Xen server shown in Figure 2 has four virtual machines (Dom0, bob, fred, DomU). Each VM has virtual interfaces
(vif).
Dom0
vif0.0
vif0.1
bob
vif1.0
vif1.1
xenbr0
eth0
xenbr1
vn0
xenbrU
vnU
vif2.0
fred
..
.
DomU
Virtual
Interfaces
Bridges
vNICs
Virtual
Machines
Figure 2 VMs, Bridges, and vNICs on a Xen Server
Xsigo Systems VP780
98
Note
Xen Windows DomU systems (fully virtualized) cannot have Ethernet interfaces added to bridges on the fly.
That is, Ethernet interfaces cannot attach to bridges automatically while Xen is running. This is a known
Xen problem.
The Xen server itself has a physical Ethernet interface (eth0). By default, there is a Layer 2 bridge connection (xenbr0)
between vif0.0 on Dom0 and the physical hardware eth0. It is very common for the other domains on the Xen server to
reside on the xenbr0 bridge also. When a host server is already configured to be a Xen machine, the VP780 will discover
xenbr0 automatically.
As a best practice, Xsigo recommends you pre configure all the network interfaces then start the VM from the Xsigo CLI.
Take the following steps to configure the VP780 to create bridges and attach the appropriate interfaces. The interfaces will
appear accordingly in Dom0 on the Xen server:
Step 1
Add the vswitch and create a new Xen bridge (i.e., xenbr1):
add server-profile server1 alexander@iowa:ServerPort23
add vswitch xenbr1.server1
show vswitch
name
descr
managed
vnics
vms
-----------------------------------------------------------------------xenbr1.server1
true
1 record displayed
As of Release 1.5, Xsigo supports only vswitch names that work properly within Xen. Those names follow
the syntax “xenbr<number>”. The Xen networking sub system makes this assumption about the bridge
name.
Step 2
Add a vNIC (i.e., vn0) for attachment into the VMs:
add vnic vn0.server1 4/1
Step 3
Add the VMs:
add vm bob.server1
add vm fred.server1
Step 4
Set the vswitch to build the connections between the vNIC and the VMs:
set vswitch xenbr1.server1 attach vnic vn0 attach vm fred
set vswitch xenbr1.server1 attach vnic vn0
set vswitch xenbr1.server1 attach vm fred
set vswitch xenbr1.server1 attach vm bob
show vswitch
name
descr
managed
vnics
vms
-----------------------------------------------------------------------xenbr1.server1
true
vn0
bob,fred
1 record displayed
Step 5
Start the VMs:
set
Are
set
Are
vm fred.server1 up
you sure you want to start virtual machine fred.server1 (y/n)?y
vm bob.server1 up
you sure you want to start virtual machine bob.server1 (y/n)?y
Xsigo Systems VP780
99
Example 2
Note: If Xen is not properly configured, you will see the error message “VM <name> failed to start”.
Step 6
When a VM comes up, the VP780 adds an Ethernet interface:
show vm
name
descr state
managed virt-type vnics
---------------------------------------------------------------------bob.server1
indeterminate true
para
vn0(xenbr1)
fred.server1
indeterminate true
para
vn0(xenbr1)
2 records displayed
Step 7
You can have multiple Ethernet interfaces on a VM:
add vswitch xenbr2.server1
add vnic vn1.server1
set vswitch xenbr2.server1 attach vnic vn1
set vswitch xenbr2.server1 attach vm fred
show vm fred.server1
name
descr state
-----------------------------------------------------------------------fred.server1
indeterminate true
para
vn1(xenbr2),vn0(xenbr1)
1 record displayed
show vm
name
descr state
-----------------------------------------------------------------------bob.server1
indeterminate true
para
vn0(xenbr1)
fred.server1
indeterminate true
para
vn1(xenbr2),vn0(xenbr1)
2 records displayed
The vNIC ordering is important to note. The first vNIC you configured appears on the end of the list.
Example 2
To set a VM to be fully virtualized and boot up correctly:
set cli mode expert
set server virtual server1 vm bob virtualization-type full
commit
Are you sure you want to commit these changes (y/n)?y
admin@iowa[top] set cli mode user
admin@iowa[xsigo]
Xsigo Systems VP780
100
Xsigo Systems VP780
Network QoS for vNICs
101
Introduction
Xsigo’s network Quality of Service (QoS) provides data-center operators the ability to treat packets differently, based on
the type of traffic, to satisfy different requirements.
Requirements can be expressed in terms of committed/peak information rate, committed/peak burst size, application
flows, traffic direction, network delay, and low latency incurred by an I/O module. QoS ensures traffic differentiation
during congestion periods. The behavior of one type of traffic should not affect the observable characteristics of another
type of traffic.
Note
The SAN QoS feature set uses vHBAs (not vNICs) and is very different from network QoS. See “SAN QoS
for vHBAs” on page 115.
Network QoS Services
The XgOS provides two Network QoS services:
Policing—Rate limits traffic to a designated rate.
WRED—Weighted Random Early Detection.
There are two ways to configure network QoS:
Default Sets—Use the default set profiles (recommended). See Default Set Profiles on page 104.
Custom Sets—Create your own custom set. See QoS Custom Sets on page 106.
Both approaches following the same QoS Operations Model on page 103.
Xsigo Systems VP780
102
Chapter 9: Network QoS for vNICs
QoS Feature Matrix
The following table describes the network QoS features supported by the 10GigE and QuadGigE I/O hardware modules.
Table 1 10GigE vs QuadGigE
10GigE
Supported
Feature
QuadGigE
Supported
Ingress and egress policing
WRED
4 global priorities that can be assigned to queues.
—
802.1p mappinga
—
IP TOS mapping
—
DSCP mapping
—
vNIC level profile attachment
Queue level profile attachment
—
Port level profile attachment
—
Assigning sets to a card
—
WFQ
a. See the “mark” option in “Setting
—
Actions” on page 121.
Information Rates and Burst Sizes
Network QoS assigns the amount of bandwidth and burst size to a given vNIC. The burst size is the amount of buffering
retained for when traffic arrives in bursts during congestion.
Bandwidth:
•
CIR—Committed Information Rate. The amount of bandwidth guaranteed to the vNIC. By default the CIR is
best effort. There is no rate restriction (imposed limit) over the bandwidth usage.
•
PIR—Peak Information Rate. The amount of best effort bandwidth (not guaranteed) for the vNIC to consume
as resources become available. By default, the PIR is the maximum-possible limit of the physical I/O card.
Burst size:
•
CBS—Committed Burst Size. The amount of data committed to be sent in one transaction.
Xsigo Systems VP780
103
QoS Operations Model
•
PBS—Peak Burst Size. The amount of best-effort data that can be sent in one transaction.
See also “Auto Calculation” on page 106.
Note
QoS Operations Model
QoS Set
P
P
P
Port Profiles
Q
Q
Q
Queue Profiles
V
V
V
vNIC Profiles
P
Q
V
8 queues within
each vNIC
IOcard 1
IOcard 2
IOcard 3
Figure 1 Hierarchy of Associated Relationships
In Xsigo’s implementation, a QoS set is a consolidated group of predefined profiles (policer, WRED) that can be
associated with different objects (vNICs, queues) at different levels in the configuration:
Table 2 A QoS Set and Its Profiles
Level
Policer Profiles
WRED Profiles
Port
0
0
vNIC
16
8
Queue
0
32
One 10GigE I/O port (P) can be associated with up to 128 vNICs (V) where each vNIC can be associated with several
different vNIC profiles. Each vNIC contains 8 queues (a common buffer pool), and each of them can be associated with a
different queue profile. Any traffic flowing in or out of the vNIC will pass only through the associated queues in the
Xsigo Systems VP780
104
global resource pool. The total bandwidth of all the vNICs associated with a port cannot exceed the cumulative capacity
of that port.
The general QoS configuration approach is as follows:
1.
Define a QoS default set (recommended) or custom set
2.
Specify a profile within the set
3.
Enable the QoS set and assign it to an I/O card (for custom sets only, not default)
4.
Associate the profile to a vNIC or queue, and specify a traffic direction (ingress or egress)
Tip
If you have multiple 10GigE cards and want to deploy the same QoS policy to all the cards irrespective of
vNIC movement, then use the same tested QoS set for all the cards. Each time a vNIC moves across line
cards, it will be treated with the same QoS behavior. Applying different QoS sets to different cards does not
guarantee QoS for vNIC movement.
Default Set Profiles
The XgOS provides a default set of QoS profiles as a configuration convenience to you. Issue the following commands to
display the default profile names and settings for the policer and WRED:
show qos network policer
show qos network wred
Sample output (see commentary after screen shots):
show qos network policer
name
level descr
cir
pir
cbs
pbs
--------------------------------------------------------------------------------default/100m_1g
global 100m_1g
100Mbps
1Gbps
7.86781MB 54.3594MB
default/10m_100m
global 10m_100m
10Mbps
100Mbps
805.664KB 7.86781MB
default/1g_10g
global 1g_10g
1Gbps
9.9297Gbps 54.3594MB 122.07MB
default/1m_10m
global 1m_10m
1Mbps
10Mbps
73.2422KB 805.664KB
default/2g_10g
global 2g_10g
2Gbps
9.9297Gbps 95.3674MB 122.07MB
default/3g_10g
global 3g_10g
3.00293Gbps 9.9297Gbps 108.719MB 122.07MB
default/4g_10g
global 4g_10g
4Gbps
9.9297Gbps 108.719MB 122.07MB
default/5g_10g
global 5g_10g
5.00122Gbps 9.9297Gbps 108.719MB 122.07MB
default/64k_1m
global 64k_1m
66Kbps
1Mbps
62.5KB
73.2422KB
default/6g_10g
global 6g_10g
6.00587Gbps 9.9297Gbps 108.719MB 122.07MB
default/7g_10g
global 7g_10g
7.00171Gbps 9.9297Gbps 108.719MB 122.07MB
show qos network wred
-------------------------------------------------------------------------------name
default/default
level
vnic
descr
default
red-min-thr
0
red-max-thr
0
red-drop-behavior
low
green-min-thr
0
green-max-thr
4112
green-drop-behavior
low
Xsigo Systems VP780
105
Default Set Profiles
yellow-min-thr
0
yellow-max-thr
6160
yellow-drop-behavior medium
mov-avg-fact
guar-thr
2064
--------------------------------------------------------------------------------name
default/wred_16k
level
vnic
descr
wred_16k
red-min-thr
0
red-max-thr
0
red-drop-behavior
low
green-min-thr
0
green-max-thr
2096
green-drop-behavior
low
yellow-min-thr
0
yellow-max-thr
4144
yellow-drop-behavior medium
mov-avg-fact
guar-thr
48
...
Note the default profile names, bandwidth sizes, and levels. By default, these default profiles are already applied to the I/
O networking cards (no set ethernet-card required). Simply choose a default profile (i.e., default/7g_10g), specify
a traffic direction (ingress or egress), and assign it to a vNIC. See “Example” on page 105.
Levels are categories that group profiles. A global level profile can be applied to multiple types of objects (I/O port,
queue, vNIC). For all else, specific categories of levels are applied to specific objects.
You can use these default profiles (recommended) or create your own custom profiles (see “QoS Custom Sets” on
page 106). The system also allows users to modify a default set and its behavior, then apply the new values to one or more
I/O cards.
Syntax
set vnic <name> {ingress-qos | egress-qos} -policer=default/<name> [enable |
disable] -policer -qos
show vnic
show qos network policer [* | <set/name>]
show qos network wred [ioport][queue][vnic [* | <set/name>]
A profile itself has no direction (ingress or egress). You must explicitly apply two profiles (one for each direction) to each
object. No QoS is available for a traffic direction that is not specified.
The system allows you to disable QoS on a specific vNIC. The default is enable.
Example
Choose a default profile (default/2g_10g), specify a traffic direction (ingress-qos, egress-qos), and assign
it to a vNIC (t1.foo) in both the ingress and egress direction:
set vnic t1.foo ingress-qos -policer=default/2g_10g -policer -qos
set vnic t1.foo egress-qos -policer=default/2g_10g -policer -qos
Xsigo Systems VP780
106
In this example, a policer was applied to the vNIC. During congestion, 2G is guaranteed (CIR). During non congestion
periods, maximum bandwidth is allowed.
To define your own QoS custom set (not use default/<name>), see the next sections.
QoS Custom Sets
The XgOS enables you to create your own QoS custom set (profile) and apply it to vNICs and queues. By default, a new
custom set is empty. However because the XgOS does not support disabling WRED, a new custom set inherits the WRED
default set and its associated characteristics. See show qos network wred.
A custom set must first be applied (set ethernet-card) to an I/O card before being applied to an object. After being
applied, a set becomes available for objects to use.
A policer restricts the amount of bandwidth to a set rate. All traffic received above a defined threshold is dropped.
Use add and set commands to control the policer’s behavior for vNICs. You can configure a QoS policer in the ingress
direction, egress direction, or both. The configurations can be asymmetrical over the same virtual object. For example, the
ingress policer can be set to 100 Mbps while the egress direction is 200 Mbps. After a policer has been added, you can
change its profile values dynamically (on-the-fly) by issuing set commands.
Syntax
add qos network policer <set/name> [-cir=<value>][-cbs=<value>][-pir=<value>][pbs=<value>]
set ethernet-card <slot> qos -set=<name>
set vnic <name> {ingress-qos | egress-qos -policer=<set>/<subset> -policer
show vnic <name> qos
where a policy name is in the form of <set>/<subset>.
Warning
Xsigo recommends you use the default WRED settings. WRED is very complex and difficult to
troubleshoot. Changing these well-tuned defaults could result is unexpected behavior for individual
applications. Use show qos network wred to display the defaults. To restore default WRED settings to a
vNIC, delete the vNIC then re-create the vNIC.
Auto Calculation
Auto calculation ensures that the optimal linear-function settings based on Xsigo test results are configured. Starting in
Release 1.0, the XgOS supports the auto calculation of CBS and PBS. When you specify the CIR and PIR (not CBS and
PBS), the system will auto calculate the equivalent CBS and PBS values for you.
Example:
add qos network policer aa/bb -cir=10m -pir=10m
show qos network policer aa/bb
name
level
descr
cir
pir
cbs
pbs
----------------------------------------------------------------------------aa/bb
global
10Mbps
10Mbps
805.664KB
805.664KB
1 record displayed
Xsigo Systems VP780
107
QoS Custom Sets
Note
In most cases, Xsigo recommends you do not modify the CBS or PBS. Use the auto calculated defaults by
specifying only the CIR and PIR.
Example: vNIC Custom Policer with 10G
test_1.whitney
MGT
AUX
SER-2
USB
Server1
SER-1
vnic
Xsigo VP780
10GE
10GE Attached Host
100 Mbps (Ingress)
100 Mbps (Egress)
Figure 2 Test Lab Topology
In this example, Server1 attaches to a Xsigo VP780 over a vNIC link. The VP780 is fitted with one 10GE I/O card that in
turn connects to a 10GE attached host. The VP780 sends traffic to Server1 over a vNIC named “test_1.whitney”. The QoS
policer restricts the amount of ingress traffic (from network to server) arriving on Server1 to 100 Mbps. The egress traffic
(from server to network) is also policed to 100 Mbps.
The following steps were taken to create a policer for one vNIC. Use the same approach for multiple vNICs:
Step 1
Create a named policer policy:
add qos network policer foo/2 -cir=100m -pir=100m
In this example, the name of the set (policer policy) is “foo”. The “/2” is the second subset policy (profile)
within the policy called “foo”. To differentiate your configurations, the system enables you to assign
different vNICs to different subset policies.
By default, the -cir and -pir are in kbits per second. The -cbs and -pbs are in bytes. However, you
can also use unit shortcuts: Mbps (m), Gbps (g), and Kbps (k). For example, -cir=100m -pir=100m.
The auto calculation of CBS and PBS is also available. See “Auto Calculation” on page 106.
Step 2
If you are using a 10GE I/O module (not QuadGigE), enable the QoS set and assign it to the appropriate
I/O card (“4” in this example):
set ethernet-card 4 qos -set=foo
Each card can be set with only one main policy, but that policy can contain many subset policies (i.e., /1, /2,
...).
The following command can also enable and disable the QoS set: set ethernet-card <num>
[enable | disable] -qos=<set-name>. Additionally, the command set ethernet-card <num>
qos -set=<set-name> -disable will send the QoS set to the card but not enable it.
Step 3
On a vNIC, enable policing for the ingress direction (network to server):
set vnic test_1.whitney ingress-qos -policer=foo/2 -policer
Use the -policer=none option to remove a policy.
Xsigo Systems VP780
108
Step 4
On the same vNIC, enable policing in the egress direction (server to network):
set vnic test_1.whitney egress-qos -policer=foo/2 -policer
A profile itself has no direction. You must explicitly apply two profiles (one for each direction) to each
object. No QoS is available for a traffic direction that is not specified.
Step 5
Verify the policer policy was assigned to the vNIC. The letter “p” in the “enables” column means the
policer policy was enabled on the vNIC correctly. The letter “q” in the “enables” column means QoS was
enabled on the vNIC correctly.
show -list vnic test_1.whitney qos
-----------------------------------------------------------------------name
test_1.whitney
level
queue
queue
0
direction ingress
descr
template
policer
wred
priority
1
enables
---------------------------------------------------------------------------name
test_1.whitney
level
queue
queue
0
direction egress
descr
template
policer
wred
priority
1
enables
---------------------------------------------------------------------------name
test_1.whitney
level
vnic
queue
direction ingress
descr
template
policer
foo/2
wred
default/default
priority
1
enables
qp-w-----------------------------------------------------------------------name
test_1.whitney
level
vnic
queue
direction egress
descr
template
policer
foo/2
wred
default/default
priority
1
enables
qp-w-
Xsigo Systems VP780
109
ACLs with QoS
-----------------------------------------------------------------------4 records displayed
Step 6
Display the information rate and burst-size values applied to the policy:
show -list qos network policer foo/2
-----------------------------------------------------------------------name
foo/2
level global
descr
cir
10Mbps
pir
10Mbps
cbs
805.664KB
pbs
805.664KB
-----------------------------------------------------------------------1 record displayed
Step 7
Display IOP diagnostics related to QoS:
show diagnostics iop <slot> qos-profile <name>
ACLs with QoS
ACL rule configurations can be used along side QoS. Specify an action for each matched condition. A condition
identifies the application flow to be chosen. An action specifies what to do with that flow:
QoS Set
P
P
P
Port Profiles
Q
Q
Q
Queue Profiles
V
V
V
vNIC Profiles
ACL rule set
ACL applied
to flow
Condition
Action
mark dscp <val>
enqueue <num>
learn {ingress | egress}
P
Q
V
8 queues within
each vNIC
Server
(Egress)
IOcard 1
= application flow
Network
(Ingress)
Figure 3 Applications Flow Controlled by an ACL
From an ingress viewpoint traffic flows from the network, into a port, into a vNIC, into 1 of 8 queues, and onto a server.
Each of the packets are evaluated against the defined ACL rules. Similarly, egress traffic (from server to network) is
evaluated against the defined ACL conditions.
Xsigo Systems VP780
110
After you create ACL rules, specify when to use the ACL rule set for a specific I/O card. Consider the following action
use cases (see “Setting Actions” on page 121 for more details):
•
Marking each packet in the flow with a DSCP value (mark dscp <val>).
•
Placing matched packets into a specific queue number (enqueue <num>).
•
Counting packets and collecting statistics for a flow that satisfies a condition ( learn ingress | egress)
The 10G I/O card supports application QoS, where specific traffic flows can be sent to different queues. Each vNIC
supports 8-prioritized queues (0 to 7). CLI is provided for you to control how those queues are used, such as setting QoS
preferential treatment (bandwidth limiting) features for each queue. Specific packets can be sent into different queues.
By default, a vNIC has one queue configured (queue 0).
Example: ACL-Based Policer for 10GigE I/O Cards
An ACL-based policer sets up an ACL that matches a particular flow, then polices that flow using QoS. For example, you
can police communication between two IP endpoints down to a specific rate. Or, you can police based on traffic type port
number (i.e., HTTP 80).
In the following example, server 1 (S1) is vNIC attached to the VP780. Server 2 (S2) is Ethernet attached. The following
configuration restricts (limits) all HTTP traffic headed in the egress direction (server to network) to 100 Mbps. All traffic
that is non HTTP traffic (no ACL match) gets max bandwidth.
MGT
vNIC
Ethernet
AUX
SER-1
SER-2
USB
S1
VP780
S2
Port 80, 100 Mbps
Figure 4 Limiting Egress HTTP Traffic to 100 MB
Note
Unlike a standard policer configuration (see Example: vNIC Custom Policer with 10G on page 107), ACLbased policing does not require QoS to be manually assigned to a vNIC.
The following example creates an ACL-based policer matching any HTTP traffic, then rate limits that traffic down to
100 Mbps:
Step 1
Create a named QoS policer to limit traffic to 100 Mbps:
add qos network policer test/100mhttp -cir=100m -pir=100m
Step 2
Enable the QoS set and assign it to the appropriate I/O card number (“1” in this example):
set ethernet-card 1 qos -set=test
Step 3
Create an ACL and assign it a name:
add acl web100m
Warning: ACLs are not autocommitted. You will need to enter 'commit' when
the ACL is complete
No auto commits exist for ACLs. Therefore, you must issue commit (see Step 5) once the ACL is
defined completely.
Xsigo Systems VP780
111
Disable QoS on a vNIC
Step 4
Define the ACL condition and action. The ACL names and rule numbers must match. All matched port
80 traffic in the egress direction will be restricted down to 100 Mbps by the QoS policer (test/100mhttp)
configured in the earlier step:
set acl web100m rule 1 action police test/100mhttp egress
set acl web100m rule 1 condition dest port exactly 80
Step 5
Issue the commit after you are finished creating the ACL, setting the action, and setting the condition:
commit
Step 6
Assign the ACL to the I/O card:
set ethernet-card 1 acl -set=web100m
Step 7
Inspect the applied ACL settings. If the destination port matches 80, the traffic is allowed to pass
through but it will be policed based on the policy test/100mhttp:
show acl
name
rule rank descr conditions
action
--------------------------------------------------------------------------------web100m 1
0
dest port exactly 80 allow, forget, police=test/100mhttp
egress
1 record displayed
Step 8
Inspect the applied I/O card settings. The “a” in the “enables” row means an ACL is assigned to the I/O
card. A “q” means a QoS policy is assigned to the card:
show iocard 1
-------------------------------------slot
1
state
up/up
descr
type
v-resources
1
acl
web100m
enables qa
-------------------------------------1 record displayed
Disable QoS on a vNIC
The XgOS allows you to disable QoS on a per vNIC basis.
Syntax
set vnic <name> ingress-qos {disable | enable} {-policer | -qos}
set vnic <name> egress-qos {disable | enable} {-policer | -qos}
Examples
To disable QoS for the ingress direction only on a vNIC named foo:
set vnic foo.bar ingress-qos disable -qos
Xsigo Systems VP780
112
To disable only the policer on a vNIC named foo:
set vnic foo.bar ingress-qos disable -policer
To enable the policer on a vNIC named foo:
set vnic foo.bar ingress-qos enable -policer
10GigE Ingress 802.1p and IP Precedence Mapping
The following table defines the mapping of 802.1p and IP precedence/TOS values to queues on 10GigE cards. The queue
numbers in the table are relative to vNICs.
Table 3 802.1p and IP Precedence/TOS Queue Mapping
802.1p user priority
IP Precedence/TOS
Queue Number
Network Control, 7
Control, 7
7
Voice, 6
6
6
Video, 5
5
5
Control load, 4
4
4
Excellent Effort, 3
3
3
Best Effort, 2
2
2
Spare, 1
1
1
Background, 0
Normal, 0
0
See the ACL mark option in “Setting Actions” on page 121.
DSCP Mapping on 10GigE Cards
DiffServ (RFC 2474) redefines the TOS byte by taking the top 6 bits of the top byte as a Differentiated Services Code
Point (DSCP). Hardware sets up a DSCP mapping table to map DSCP values to queues. All undefined values are mapped
to the queue corresponding to the DF.
Table 4 DSCP Name to Queue Mapping
DSCP Name
Value (Binary)
Queue Number
CS7
111000
7
CS8
110000
6
EF
101110
5
CS5
101000
5
AF43
100110
4
AF42
100100
4
Xsigo Systems VP780
113
DSCP Mapping on 10GigE Cards
Table 4 DSCP Name to Queue Mapping (continued)
Xsigo Systems VP780
DSCP Name
Value (Binary)
Queue Number
AF41
100010
4
CS4
100000
4
AF33
011110
3
AF32
011100
3
AF31
011010
3
CS3
011000
3
AF23
010110
2
AF22
010100
2
AF21
010010
2
CS2
010000
2
AF13
001110
1
AF12
001100
1
AF11
001010
1
CS1
001000
1
DF (Other)
000000
0
114
Xsigo Systems VP780
SAN QoS for vHBAs
115
Introduction
A vHBA supports QoS where the bandwidth is rate limited with shaping (not dropped). There are no queues, Policers, nor
WRED associated with FC traffic—only shapers.
See vHBAs on page 67 for information about non QoS vHBA features.
Note
SAN QoS Features
Supported features:
•
Shaping
•
CIR and CBS control
•
PIR and PBS control
•
vHBA service only
Not supported:
•
Policing
•
WRED
•
WFQ
•
Default set profiles (i.e., default/<name>). There is no default configuration created for SAN QoS.
•
Custom set profiles: I/O port, vNIC, queue
•
Using ACLs with SAN QoS
•
Auto calculation on SAN QoS for CBS and PBS
•
Ingress vs egress direction control
Xsigo Systems VP780
116
Chapter 10: SAN QoS for vHBAs
Command Support
QoS shaping services can be applied to FC cards by using add qos san and set qos san.
Syntax
add qos san <policy-name>
set qos san <policy-name> -cir=<value> -pir=<value> -cbs=<value> -pbs=<value>
show qos san <policy-name>
add qos san <policy-name>—Creates a named QoS shaping policy.
set qos san <policy-name> -cir=<value> -pir=<value> -cbs=<value> -pbs=<value>—
Configures shaping-policy values.
set vhba <vhba-name> qos <policy-name>—Binds the policy to a vHBA.
show qos san <policy-name>—Displays the configured SAN shaping service that is bound to a vHBA.
Example: vHBA with Shaping
Take the following steps to create a SAN QoS shaping policy and apply it to a vHBA:
Step 1
Create a named QoS shaping policy. The policy name is “test” in this example:
add qos san test
Step 2
Configure the shaping-policy values. SAN QoS only limits bandwidth (no drops):
set qos san test ?
-cbs
Committed burst size
-cir
Committed information rate (optional K,M,G suffix)
-descr Description
-pbs
Peak burst size
-pir
Peak information rate (optional K,M,G suffix)
set qos san test -cir=250 -pir=500 -cbs=15 -pbs=250
Step 3
Bind the policy to a vHBA. Whichever maximum bandwidth you defined will be applied to this vHBA.
The vHBA’s traffic will never exceed the defined policy values:
set vhba vhha1.finance qos test
Step 4
Verify the QoS shaping service is bound to the vHBA:
show vhba vhha1.finance qos
-------------------------------------------vhba
vhha1.finance
name
test
descr
cir
250Kbps
pir
500Kbps
Xsigo Systems VP780
117
Command Support
cbs
15
pbs
250
-------------------------------------------1 record displayed
Xsigo Systems VP780
118
Chapter 10: SAN QoS for vHBAs
Xsigo Systems VP780
ACLs
119
Introduction
Access Control Lists (ACLs) classify packets. The classification result can be applied to QoS application flows (mark,
police) or to network-access control (deny, allow).
There are many use cases for ACLs. Consider the following examples:
•
Prioritizing outbound traffic by marking fields in the IP header. Thereby, enabling upstream routers to handle
this marked (set) traffic in a specific way. For example, any RTP VoIP traffic within a certain port range
could have its IP TOS bit set to a value of 5. Any packet that satisfies these conditions will have its IP
header field set by the I/O card.
•
Intentionally dropping packets when a Denial of Service (DoS) attack is detected. All traffic must be blocked
from specific IP or MAC addresses.
ACLs are supported on the 10GigE I/O cards only.
Note
Example: Deny Egress Traffic
This example creates an ACL that blocks any traffic heading in an egress direction (server to network) where the
destination IP address is equal to 10.2.16.5.
Switch
vNIC
3/1
MGT
AUX
SER-1
SER-2
USB
Server1
10.2.16.4
VP780
Egress
Ingress
Server2
10.2.16.5
Server3
10.2.16.6
Figure 1 Block All Egress Traffic Destined for 16.5
Xsigo Systems VP780
120
Chapter 11: ACLs
Take the following steps to deny egress traffic:
Step 1
Create a named policy set (empty by default). No implicit assumptions or rules are made in this empty
set. The set in this example is named “block16_5”:
add acl block16_5
Warning: ACLs are not autocommitted. You will need to enter 'commit' when
the ACL is complete
Caution
Step 2
As indicated by the display message, the commit command must be issued after you define the condition
and action. See Step 3.
Add a rule to the named set, then specify an action and condition. Rule numbers must be between 1 and
1024:
set acl block16_5 rule 1 action deny egress
set acl block16_5 rule 1 condition dest ipaddr = 10.2.5.16 mask
255.255.255.255
In this example, any traffic that exits the Xsigo I/O card is considered the egress direction (server to
network). The condition matches on destination IP address 10.2.5.16 with a 32-bit mask length. All
other traffic is permitted to pass through except that destined for 10.2.5.16.
For a list of condition definitions, see “Setting Conditions” on page 122.
Step 3
Issue a commit after the ACL is defined:
commit
This command collects all the multiple configuration steps of your policy and stores them into the chassis’
database.
Step 4
Specify the I/O card and apply the named ACL:
set ethernet-card 3 acl -set=block16_5
The same set can be attached to multiple cards (one at a time). Once attached, the policy is downloaded and
programmed into the card. The defined conditions and actions will be applied to each packet passing
through the card and its ACL rule set.
Step 5
Verify the ACL was assigned to the I/O card. Look for the “a” field next to the “enables” In this
example, QoS (q) is also enabled:
show -list iocard 3
-------------------------------------------------------------------slot
3
state
up/up
descr
type
vnics
12
qos
acl
block16_5
enables qa-------------------------------------------------------------------1 record displayed
Xsigo Systems VP780
121
Setting Actions
Step 6
Display the contents of the ACL policy:
show -list acl
----------------------------------------------------------------------name
block16_5
rule
1
rank
0
descr
conditions dest ipaddr = 10.2.5.16 mask 255.255.255.255
action
deny, forget egress
Step 7
Display ACL statistics. In this example, the “acl-deny-pkt-counter” is equal to “6”, which indicates
packets are being dropped (as expected):
show iocard 3 acl-stats
name acl-deny-pkt-counter acl-enqued-pkt-counter acl-learned-flowscounter acl-mark-dot1p-pkt-counter acl-mark-tos-counter acl-rule aclrule-set
----------------------------------------------------------------------3 6 0 0 0 0 1 block16_5
1 record displayed
Step 8
Enable or disable an ACL:
set ethernet-card 3 disable -acl
set ethernet-card 3 enable –acl
Step 9
Disable the ACL set on the I/O card:
set ethernet-card 3 acl -set=none
Step 10 Remove the ACL:
remove acl block16_5
Setting Actions
An action is taken on a corresponding matched condition. An action is either allow or deny to be applied for a specific
traffic direction (ingress or egress).
Syntax
set acl <set-name> rule <num> action <def>
where <def> can be any of the following:
allow {both | egress | ingress}
deny {both | egress | ingress}
enqueue <num>
forget {both | egress | ingress}
learn {both | egress | ingress}
mark {disable|dot1p <val>|dscp <val>|iptos <val>}
police {*|<set/name>|none}
The default is allow both.
Xsigo Systems VP780
122
Chapter 11: ACLs
enqueue <num>—Each vNIC uses only one queue by default (queue 0). If the condition matches, the system
puts the packet into this queue number (from 0 to 7). Thereafter, a policy (i.e., a shaper) can be applied to the
queue.
learn—The system starts counting the number of packets that matched the condition.
forget—The system clears (forgets) the learned statistics.
mark—The result of an ACL classification rule can specify marking a packet. This option applies priority
marking to the packet using a supported marking algorithm:
— 802.1p marking
— IP precedence marking
— DSCP marking
Only one of three marking mechanisms can be specified at a time. Setting one of them negates the other two.
When the queue number (offsets 0 - 7) is specified, the marked packet is placed on the specified queue.
See 10GigE Ingress 802.1p and IP Precedence Mapping on page 112.
policer—Applies a QoS policer to the matched packet. The bandwidth can be limited to a specific level.
Example
set acl foo rule 3 action allow both
Setting Conditions
An ACL condition is a match-test rule to perform on a packet. A condition defines rules for fields the system checks
during packet processing. Operators are available to match strings in those fields that follow a specific pattern.
Note
Release 1.0 does not support “rank” rule configuration. All rules are evaluated with equal priority. There is
no evaluation-sequence order (rank).
Syntax
set acl <set-name> rule <num> condition <def>
A condition <def> encompasses the following general form:
<field-name><operator><value>
where any of the following are supported:
dest {ipaddr<oper><val>|mac<oper><val>|port <oper><val>}
src {ipaddr<oper><val>|mac<oper><val>|port <oper><val>}
dot1p <oper> <number-or-range>
dscp <oper> <number-or-range>
ioport <oper> <number-or-range>
protocol {tcp | udp}
tos <oper> <number-or-range>
Xsigo Systems VP780
123
Removing ACLs
vlan <oper> <number-or-range>
Operators match strings following a specific pattern. Use an operator in the following table to define how a field should be
checked, where <oper> can be any of the following.
Table 1 Operators Supported
Operator
Description
=
Equal to (including masks if appropriate). Value of the field is equal
to a single specified value (no wildcards)
exactly
Exactly equal to
Example
set acl test rule 1 condition dest ipaddr = 10.1.1.1 mask 255.255.255.255
show -list acl test
-----------------------------------------------------------------------------name
test
rule
1
rank
0
descr
conditions dest ipaddr = 10.1.1.1 mask 255.255.255.255
action
Removing ACLs
Use the remove acl command to delete configured ACLs on the system.
Syntax
remove acl <acl-name>
remove acl *
<acl-name>—Removes a single ACL.
*—Removes all ACLs.
Example
remove acl *
Remove all ACLs (y/n)?y
Xsigo Systems VP780
124
Chapter 11: ACLs
Xsigo Systems VP780
SAN Boot
125
Introduction
SAN Boot allows you to boot a host server from a SAN disk accessed through a vHBA. The remote disk to boot from is
identified by a target World Wide Port Name (WWPN) and Logical Unit Number (LUN) ID on a storage disk array
device:
Host Server to Boot
HCA HCA
IB
Interro
VP780
FC Port
w/vHBA
gate t
he Ha
rd Driv
es
Kernel Path
vol 1 /5
FC Network
vol 2 /7
Root FS Path
Disk Array
(WWPN, LUN ID)
Figure 1 SAN Boot Topology
SAN Boot in Release 1.5 beta supports RHEL4 and RHEL5 only.
Note
Boot Sequence
All computers boot from local boot devices. However to boot from a “remote” device, you need to make the device appear
to be local. Xsigo implemented ROM BIOS extensions for HCA card executes that follow these boot-sequence steps:
Step 1
On power up, the server’s BIOS performs basic hardware initialization.
Step 2
The OS loader is installed, which is specific to each OS type:
• For Linux, the loader is GRand Unified Bootloader (GRUB). This loader resides in the /boot sector
of a bootable disk. The software responsible for loading the GRUB loader is held in the
Option ROM of the HCA and runs in the context of BIOS.
• For Windows, the loader is NTLoader and is loaded by the same Option ROM in the HCA.
When the loader runs, it typically reads a configuration file from the disk and allows the user to boot the OS
in a number of configurations. For GRUB, this file is at /boot/grub/grub.conf.
For Linux, GRUB will load the kernel and the initial RAM disk (initrd) into memory and begin running the
kernel. The root file system (rootfs) given to the kernel will be the initrd.
The kernel runs a program in the initrd in the location /init. Typically this program is a shell script that loads
the kernel modules and mounts the real root file system. See Xsigo Initrd on page 137 for more details.
Xsigo Systems VP780
126
Chapter 12: SAN Boot
Step 3
The host server establishes a connection to the VP780, where the system determines the vHBA to use
and how to set up the communication path from the host server to the hard disk in the storage array.
Step 4
Interrogate the hard disk to determine if it is present and if the disk is bootable (or not). If bootable,
the Xsigo HCA functions as a hard-disk controller. BIOS begins to treat the Xsigo HCA as the local ID
controller for the local hard disk.
Step 5
BIOS reads the boot sequence, where the XgBoot for the Xsigo HCA must be the first boot device
(highest priority) in the list:
Figure 2 Dell 1950 BIOS, Slot 1 XgBoot HCA
Step 6
The data (Linux kernel or Windows) is sent from the hard disk and loaded into memory. The kernel
begins to work and loads all the necessary drivers into the host server.
Note the below boot messages as the server is connects to the WWN SAN storage device:
Figure 3 Typical SAN Bootup
Xsigo Systems VP780
127
Roles
You should also look for the message “XgBoot detected PnP BIOS” that indicates the Xsigo Option ROM is installed on
the HCA and the proper device-failover mechanisms are supported. If there are two Xsigo HCAs in the system with two
Option ROMs, two “XgBoot detected PnP BIOS” entries will appear. Thereafter, the system boots up into GRUB.
Roles
SAN Boot operates in three different roles (phases):
•
Load—Loads the operating system and initrd (in the case of Linux) from a SAN disk. The kernel and initrd
are located and installed into memory.
•
Mount—Mounts the root file system on a SAN disk. It may or may not be the same disk as the operating
system was loaded from. There are three mount types available: static, Logical Volume Management (LVM),
and direct. In the mount role, initrd starts and mounts the device file system corresponding to the vHBA
specified. The target disk and its root file system inside the OS are mounted.
•
Loadmount—Both roles are performed, and all files come from the same location. That is, the same target
disk contains the boot image and the root file system. This common use case loads the kernel and initrd into
memory, where initrd starts and mounts the device file system corresponding to the vHBA specified.
SAN Boot Control: Two Ways to Go
There are two ways to send instructions to initrd and control mounting the root file system:
Method 1—Use the information model in the VP780 (entered via the CLI), which communicates with the host’s
vHBA driver (/proc/driver/vhba/san-info). See CLI Support on page 128.
Method 2—Provide kernel command line arguments in the GRUB configuration file. See Kernel Command Line
Arguments on page 139.
A GRUB configuration takes precedence over a XgOS CLI configuration.
Note
Xsigo Systems VP780
128
CLI Support
The XgOS provides a robust set of CLI commands to control and monitor SAN Boot.
In the object model, SAN Boot objects are children of the server profile object. A San Boot object refers to a vHBA.
There can be multiple target disks under a SAN Boot object, where each disk is identified by a WWPN and LUN ID:
Server Profile
vHBA
SAN Boot
SAN Boot
Disk
Disk
Figure 4 SAN Boot Object Model
Syntax
Use the set server-profile san-boot command to set up the SAN Boot facilities for a server profile.
The simplest setup is to use the same disk for loading the operating system and mounting the root file system.
This approach uses the loadmount syntax and is achieved by the following command:
set server-profile <name> san-boot <vhba> <wwpn> <lun>
This command defines a SAN Boot object, loads the OS, and mounts a file system at a specific WWPN and LUN ID over
a named vHBA. The simple use case is to load and mount the same disk.
Additionally, SAN Boot provides the flexibility of configuring multiple paths through the FC fabric. For example,
the same disk could appear in two different targets and LUNs for redundancy and network-path protection. In this case,
set the mount type using the following options:
set server-profile <name> san-boot <vhba> <wwpn> <lun> mount [-vhba=<name>]
[direct </dev/node>][lvm <group-name> <volume-id>][static <wwpn> <lun>]
Using the set server-profile command is the simplest way to control SAN Boot objects. However complex setups
exist where there are multiple paths through a SAN to the same disk (multi pathing). There are also scenarios where you
may need to modify the existing facility incrementally for working with multiple disks. For fine-grain control of the SAN
Boot facility, Xsigo provides commands that control the SAN Boot objects directly (more complex but more powerful):
add
add
add
add
san
san
san
san
boot
boot
boot
boot
loader <vhba>.<server> <wwpn> <lun> [-mount-also]
mount <vhba>.<server> lvm <group-name> <volume-id>
mount <vhba>.<server> static <wwpn> <lun>
mount <vhba>.<server> direct </dev/node>
Xsigo Systems VP780
129
CLI Support
set san boot <vhba>.<server> lvm <group-name> <volume-id>
set san boot <vhba>.<server> static <wwpn> <lun>
set san boot <vhba>.<server> direct </dev/node>
remove san boot <vhba>.<server>
remove san boot [* | *.<server> | <vhba>.<server>] disk <wwpn> <lun>
Use these commands to verify your configuration:
show
show
show
show
san boot
san boot <vhba>.<server>
san boot [* | *.<server>]
server-profile <server> san-boot
Parameter Descriptions
set server-profile <name>—Identifies a named server profile to boot from.
san-boot <vhba>—Creates a boot object and associates it with a named vHBA. The SAN Boot object can use
only the vHBAs that are available to the server profile. You must have a previously created vHBA and scanned for
available LUNs. See Target Prescan and Rescan on page 74.
<wwpn>—Available target WWPN, as used for a physical hard disk.
<lun>—Available LUN ID on the target, as used for a logical hard disk.
mount—Optional. Specifies a mount type to use:
-vhba=<name>—Mounts an additional vHBA.
lvm <group-name> <volume-id>—Logical Volume Manager. LVM is a feature present in Linux that
handles a layer of logical disks over physical disks. For example, you can load from a static disk but mount
a logical volume. LVM is identified by two parameters: a <group-name> and <volume-id>.
The <volume-id> must be a name.
direct </dev/node>—If you know the exact device name on your host server, you can specify that
string to be mounted directly. For example, /dev/sda.
static <wwpn> <lun>—Specifies a static mount for a different disk. The use case is for a different
WWPN/LUN pair from that used to load the operating system.
remove san boot—Removes an entire SAN Boot object or only specific disks below it.
show san boot—Displays configured SAN Boot information on server profiles and vHBAs. In the output,
there is one line listed for each disk in the facility. Use the asterisk wildcard (*) for different display combinations.
show server-profile <server> san-boot—Displays SAN Boot information for a server profile.
Note
For more information on LUNs and targets, see the following sections: LUNs Per Target on page 73, Target
Prescan and Rescan on page 74, LUN Masking on page 82.
Xsigo Systems VP780
130
SAN Boot Install
There are two common Linux installers:
•
Anaconda (RedHat)
•
YaST/linuxrc (SuSE)
Xsigo does not modify these installers. Instead Xsigo uses the versions supplied by the OS provider and adds in Xsigo’s
driver disks. A Xsigo driver disk is created for a specific kernel for a specific OS.
The xsigo-boot tar archive:
xsigo-boot-<kversion>-<xgrelease>-<arch>.tar
contains img files and scripts compiled for specific kernel versions to support booting using the Xsigo drivers.
There are two files:
1.
xsigo-initrd-<kversion>-<arch>.img (A Linux initrd that uses the Xsigo drivers)
2.
xsigo-rhdd-<kversion>-<arch>.img (a Red Hat Anaconda installer driver disk)
or:
xsigo-dud-<kversion>-<arch>.img (a SuSE linuxrc driver update disk)
There are three ways you can add the Xsigo driver disk into the installer:
1.
Boot from a CD-ROM (recommended). Create an ISO image that provides the required drivers to the installer
during runtime.
2.
Boot from a USB token.
3.
Boot from the initrd itself. Xsigo provides a script that takes a standard initrd (for example the Anaconda initrd
from the RedHat installation disk) and then inserts the Xsigo drivers (as a driver disk) into that initrd. Thereafter,
enter the Xsigo Anaconda initrd into the PXE boot system as the initrd. As the kernel boots, it will load the Xsigo
drivers.
On the VP780, set a vHBA to load using a specific WWPN and LUN ID. The simplest setup is to use the same disk for
loading the operating system and mounting the root file system—the loadmount syntax:
set server-profile <name> san-boot <vhba-name> <wwpn> <lun>
The initrd loads from the mounted SAN Boot object. Once installed, the SAN disk will appear as any other disk in the
system.
More details for SAN Boot install are provided in PXE Install on page 141.
Xsigo Systems VP780
131
Example 1: RHEL5 SAN Boot Install
Example 1: RHEL5 SAN Boot Install
To install RHEL5 over vHBA to SAN storage:
Step 1
On the host server, install an HCA containing Xsigo’s SAN-Boot Option ROM. All HCAs shipped from
Xsigo manufacturing should have the Option ROM installed already.
If not, use the script xg_config to install the Option ROM:
/opt/xsigo/bin/xg_config
Note: Many SAN Boot capable servers exist. However only certain BIOSes support a certain numbers of
HCAs with the Xsigo Option ROM. Consult Xsigo Support for the hardware compatibility matrix.
See HCA Firmware and Option ROM Upgrades on page 193 for more details.
Step 2
In preparation for SAN boot, you will need to burn a CD image of Xsigo’s vHBA drivers used during
RHEL5 installation.
Download the appropriate xsigo-boot-<kernel>.tar file unique to your distribution and architecture. RHEL5
32-bit is used in this test.
Untar xsigo-boot-<kernel>.tar file and create a cd image from xsigo-rhdd-<kernel>.img
From Linux:
# tar xvf xsigo-boot-<kernel>.tar
# cdrecord -v dev=0,1,0 xsigo-rhdd-<kernel>.img
Step 3
From the VP780, create a server profile for the server performing the RHEL5 install to SAN storage.
For the physical connection, use the port GUID which is the HCA’s GUID plus 1.
For example, if an HCA’s GUID is 2c902000232138, then the port GUID would be 2c902000232139:
add server-profile pokemon 2c90200232139
Step 4
Add the vHBA to the server profile:
add vhba vhba0.pokemon 3/1
Step 5
Confirm that LUNs are visible to the server-profile by running a prescan and showing target luns. At this
point you will need to have already configured the storage side (i.e., zoning, disk allocation, etc).
set vhba vhba0.pokemon remove-prescan
set vhba vhba0.pokemon prescan
show vhba vhba0.pokemon targets
Step 6
(Optional). Remove all physical hard drives from server.
Step 7
Set up the server profile to do SAN boot. The set server-profile <name> san-boot provides
the server with all the information that it needs to connect to SAN storage upon boot up (i.e., vHBA to
use, WWPN and LUN to connect to, volume group, logical volume to use, etc).
set server-profile pokemon san-boot vhba0 50:06:01:60:3A:20:1E:83 0 mount
lvm VolGroup00 0
Step 8
Boot server with RHEL5 server disk 1.
At the initial Redhat prompt, type linux dd to inform Redhat that you would like to install custom drivers.
Insert Xsigo’s vHBA driver disk when prompted.
Xsigo Systems VP780
132
Re-insert RHEL5 server disk 1 cd when prompted.
Choose typical RHEL5 setup options and the complete install.
Step 9
After installation is complete, you will need to copy over the xsigo-initrd-<kernel>.img file
into the SAN storage /boot dir and modify the GRUB configuration for this initrd file. The file can be
copied over by moving the server profile with its vHBA to another server, mounting the boot partition,
and then copying the file over to it.
set server-profile pokemon disconnect
set server-profile pokemon connect popo@timbuktu:ServerPort2
## ON ANOTHER SERVER CONNECTED TO THE SAME VHBA LUND ##
mount /dev/sdb1 /mnt
cp xsigo-initrd-2.6.18-8.el5-i386.img /mnt
vi /boot/grub/grub.conf
and edit the initrd line to read the following:
initrd /xsigo-initrd-2.6.18-8.el5-i386.img
umount /mnt
Step 10 After you have copied the Xsigo initrd file to the SAN storage and edited the GRUB configuration file,
you will need to move the server profile with the vHBA back to the original server that is booting from
SAN:
set server-profile pokemon disconnect
set server-profile pokemon connect 2c90200232139
Example 2: Other Mount Types
To boot from one partition, but mount from a different partition on the same disk. This example boots from LUN 42 and
mounts LUN 41:
set server-profile foobar san-boot vh1 1:2:3:4:5:6:7:8 42 mount static
1:2:3:4:5:6:7:8 41
To boot from one disk but mount from another one. This example shows two different disks (with different WWPNs).
The server boots from one and mounts from the other:
set server-profile foobar san-boot vh1 1:2:3:4:5:6:7:8 42 mount static
8:7:6:5:4:3:2:1 99
To boot from a disk on one vHBA, and mount another disk on a different vHBA. In this scenario, the server boots from
one disk and then use a different vHBA to mount another disk:
set server-profile foobar san-boot vh1 1:2:3:4:5:6:7:8 42 mount -vhba=vh2 static
8:7:6:5:4:3:2:1 99
To load from a static disk but mount from a known device:
set server-profile myserver san-boot myvhba 1:2:3:4:5:6:7:8 42 mount direct /dev/
sda
To boot from a disk that uses LVM. The LVM group name is VolGroup00. The volume name is LogVol00:
set server-profile myserver san-boot myvhba 1:2:3:4:5:6:7:8 42 mount lvm
VolGroup00 LogVol00
Xsigo Systems VP780
133
Example 3: show san boot
Example 3: show san boot
show san boot
-------------------------------------------------------server
myserver
role
loadmount
vhba
myvhba
mnt-type lvm
lvm-grp
rhel5_group0
lvm-vol
bar
dev
mnt-opts
disks
01:02:03:04:05:06:07:08(42/LM)
-------------------------------------------------------1 record displayed
Table 1 Field Descriptions for show san boot
Field
Description
server
The name of the server profile.
role
The role of the disk. SAN Boot can operate in three different roles:
• load—The OS and initrd are installed.
• mount—The system mounts a drive and its root file system.
• loadmount—Both roles are performed (most common) and all files come
from the same location. The same target drive contains the boot image and
the root file system. This simple case loads the kernel and initrd into memory,
where initrd starts and mounts the device file system corresponding to the
vHBA specified.
• none—Disabled. No role is specified.
vhba
The vHBA used to access the SAN.
target
The target WWPN.
lun
The target LUN ID.
mnt-type
The type of mount to use. The choices are:
1.
lvm
2.
static
3.
direct
lvm-grp
LVM group name if the mount type is lvm.
lvm-vol
LVM volume number if the mount type is lvm.
dev
Device name if the mount type is direct.
mnt-opts
Additional mount options to use (for Linux mount).
Xsigo Systems VP780
134
Example 4: show serer-profile san-boot
To display the SAN Boot facilities on a VP780:
show server-profile fernwood san-boot
server
role
vhba mnt-type lvm-grp
lvm-vol dev mnt-opts disks
-----------------------------------------------------------------------------fernwood loadmount vhba1 lvm
VolGroup00 0 20:78:00:C0:FF:0A:78:AF(42/LM)
1 record displayed
Note the following information:
•
The server name is fernwood.
•
The vHBA used for SAN Booting is vhba1.
•
The role is loadmount with a mount type of LVM.
•
The disk to be mounted is LUN 42 on 20:78:00:C0:FF:0A:78:AF
Example 5: Loadmount
To perform a loadmount, where the host server boots and mounts the root file system from the same disk location:
set server-profile myserver san-boot myvhba 20:05:00:A0:B8:17:37:3F 3
This simple case loads the kernel and initrd into memory, where initrd starts and mounts the device file system
corresponding to the vHBA defined.
Best Practices and Gotchas
SAN Boot is complex. There are some gotchas to be aware of:
•
Upgrading the Xsigo drivers on a SAN booted server
The environment on a SAN booted server is different from a standard server with respect to the Xsigo
drivers. In a standard server, the drivers are installed in an RPM. On a SAN booted server, the drivers are
part of the initrd and are therefore already loaded and running when the server boots.
To upgrade the drivers on a SAN booted server, it is simply a matter of replacing the initrd in the /boot
partition with the new initrd containing the updated drivers. Once the file initrd has been copied to /boot,
reboot the server for the drivers to take effect.
•
Avoid multiple volume group names with the same name visible on a server
Avoid this by naming all the volume groups something different if they can be seen by more than one server.
The default is VolGroup00. If this is unavoidable, you can add a suffix to the group name in the SAN Boot
options. This suffix is appended by a “/” and is used to determine a filter for the LVM scan. The suffix can
be the LUN ID or a device name, such as “sdc”. Beware of using “/” characters in the suffix.
•
Avoid multiple /boot partitions with the same label
If a server sees more than one /boot partition, it will attempt to mount one of them. The mount point in
the /etc/fstab specifies the label /boot. Even when ambiguous, the system will mount something. This issue
is a problem only if you write to the /boot partition (to update the initrd or grub config file). To avoid it,
modify the label on the partition using the tune2fs command and use the same label in /etc/fstab.
Xsigo Systems VP780
135
Best Practices and Gotchas
For example, assume /boot is mounted as /dev/sdb1:
# tune2fs /dev/sdb1 -L /boot20
modify the /etc/fstab and replace /boot by /boot20
Getting Initrd onto the SAN
When you first install an OS on a SAN disk, the system will install the standard initrd from the OS. This action will not
work because the Xsigo drivers will not be loaded and will therefore be unable to mount the disk. You need to replace the
standard initrd with Xsigo’s special one.
The best way is to have a server booted from local disk and connected to the Xsigo chassis. This server should be able to
see all the SAN targets upon which you want to install the OS. When the installation is complete, shutdown the server that
you just installed, mount the SAN disk on the special server, and copy the initrd onto the /boot partition of the disk.
In this example, the special installation server is named “install”. Assume there is a vHBA FC card in slot 11, and port 1 is
connected to the SAN switch in the correct zone (so that it can see the targets).
foo@bar[xsigo] add server-profile install install@bar:ServerPort5
foo@bar[xsigo] add vhba vh1.install 11/1
On the “install” server, you can see all the disks using the sg_map -x command:
[root@install
/dev/sg0 0 0
/dev/sg1 0 0
/dev/sg2 0 0
/dev/sg3 0 0
/dev/sg4 0 0
/dev/sg5 0 0
/dev/sg6 2 0
/dev/sg7 1 0
~]# sg_map -x
0 0 13
0 10 0 /dev/sda
0 11 0 /dev/sdb
0 40 0 /dev/sdc
0 41 0 /dev/sdd
0 42 0 /dev/sde
0 0 0 /dev/sdf
0 0 5 /dev/scd0
The 5th column is the LUN ID.
If you are installing Red Hat with LVM, you will probably use the same logical volume group name and volume number
for each installation. By default this name is VolGroup00. The special installation server will balk if it sees two logical
volume groups with the same name, so you must restrict what the server can see.
The best way to configure the restriction is with a LUN mask on the VP780 so that the server does not see the other LUNs.
Assume you are installing on LUN 42:
foo@bar[xsigo] add san lun-mask sanboot target 20:78:00:C0:FF:0A:78:AF 42
Where 20:78:00:C0:FF:0A:78:AF is the WWPN of the disk and 42 is the LUN ID.
Ideally you should be able to set the lun-mask of the vHBA, but that does not cause all the existing disks on the host to
go away. For now, you have to do this:
foo@bar[xsigo] remove vhba vh1.install
foo@bar[xsgio] add vhba vh1.install 11/1 -lun-mask=sanboot
On the install server:
[root@install ~]# lvm vgscan
[root@install ~]# lvm vgchange -ay VolGroup00
[root@install ~]# sg_map -x
Xsigo Systems VP780
136
Find the /dev/ node corresponding to the LUN. Assume it is /dev/sdb:
[root@install ~]# mount /dev/sdb1 /mnt
[root@install ~]# cp xsigo-initrd-2.6.18-8.el5-i386.img /mnt
Now you need to modify grub.conf to point the initrd to the new initrd you installed. The GRUB config file is in /mnt/
grub/menu.lst. Once it has all been done, unmount the disk:
[root@install ~]# umoumt /mnt
Xsigo Systems VP780
Xsigo Initrd
137
Introduction
An initial RAM disk (initrd) is a ram-based file system provided to the kernel at boot time. The initrd’s purpose is to
mount the root file system. The format of the initrd is typically a compressed Copy Input Output (CPIO) archive.
The Xsigo initrd is a customized initrd for SAN Boot, NFS Boot, and PXE Boot. It loads the virtual drivers and mounts
the root file system using a vHBA or a vNIC (NFS mount).
The initrd contains a bash script called “init” plus a number of Aikido scripts that perform higher level functions.
The Aikido interpreter is supplied as part of the initrd in order to run the high level scripts. The init script performs the
following duties:
1.
Make device nodes (/dev/tty, etc)
2.
Load kernel modules and Xsigo drivers
3.
Wait for the Xsigo drivers to settle. Settling means to wait for the IB link to come up and to wait for vNICs and
vHBAs to appear.
4.
Mount the root file system (/sbin/init). Information on where to mount the root file system can be obtained from
two locations:
— From kernel arguments (see Kernel Command Line Arguments on page 139)
— From /proc/driver/vhba/san-info
5.
If all else fails, the initrd will drop to a text-based menu system that can be used for troubleshooting. See Boot
Menu Troubleshooting on page 143.
xsigo-boot
There are two common Linux installers:
•
Anaconda (Red Hat)
•
YaST/linuxrc (SuSE)
Xsigo does not modify these installers. Instead Xsigo uses the versions supplied by the OS provider and adds in Xsigo’s
driver disks. A Xsigo driver disk is created for a specific kernel for a specific OS.
The xsigo-boot tar archive contains img files and scripts used to support booting the Xsigo drivers.
Table 1 Files and Scripts Used for the Xsigo Initrd
File Name
Purpose
xsigo-boot-<kver>-<build>-<arch>.tar
Boot file compiled for a specific kernel version.
xsigo-initrd-<kver>-<arch>.img
A Linux initrd that uses the Xsigo drivers.
xsigo-rhdd-<kver>-<arch>.img
A Red Hat Anaconda installer driver disk.
Xsigo Systems VP780
138
Chapter 13: Xsigo Initrd
Table 1 Files and Scripts Used for the Xsigo Initrd (continued)
File Name
Purpose
xsigo-dud-<kver>-<arch>.img
A SuSE linuxrc driver update disk.
xg-insert-dd
If you want to avoid using a CD to load the driver
disk, you can insert the disk into the Anaconda
initrd using the script xg-insert-dd.
It creates a new initrd prefixed with the string
“xsigo-”.
xg-insert-dd-ext2
For RHEL4, inserts the driver disk into the
Anaconda initrd.
See also PXE Install on page 141 and SAN Boot Install on page 130.
Using the Xsigo initrd
The Xsigo initrd can be used to provide diskless operation of a server using the Xsigo vNIC interface or from a SAN disk
using a vHBA. The file is a standard format Linux initrd that can be supplied by the PXE server or GRand Unified
Bootloader (GRUB) along with a Linux kernel. When the kernel boots with the initrd, the Xsigo drivers will be loaded
and the Linux root file system can be mounted from a remote NFS server or SAN disk.
In order to specify which server to mount, the initrd reads the kernel command line and looks for Xsigo-specific options.
These options are used to specify which vNIC to use as the network interface and where to mount the root file system
from.
For SAN Boot the initrd can read the kernel command line, or it can use information supplied from the Xsigo chassis.
To see the contents of an initrd:
zcat xsigo-initrd-2.6.9-42.EL-i386.img | cpio -t
To extract the contents into a directory:
cd destdir
zcat xsigo-initrd-2.6.9-42.EL-i386.img | cpio -uid
See also SAN Boot Install on page 130.
Xsigo Systems VP780
139
SAN Booting
SAN Booting
SAN booting enables you to mount a file system that is located on a disk accessed using a vHBA. There are two ways to
specify where to mount the file system from:
•
XgOS command line (see CLI Support on page 128)
•
Kernel arguments specified in grub.conf (see below)
A GRUB configuration takes precedence over a XgOS CLI configuration.
See also SAN Boot on page 125.
Kernel Command Line Arguments
Sometimes users have an existing system for SAN Boot and prefer not to use the Xsigo chassis for SAN booting. Instead,
kernel arguments in grub.conf are used to specify where devices should be mounted.
The following mount information (not boot) overrides the information provided by the vHBA driver. The sanboot=
element supports different syntax combinations:
sanboot=<vhba>:<target>:<lun>
sanboot=lvm:<group>:<volume>
sanboot=<device-path>
where:
<vhba> is a vhba name
<target> is a target number (a SCSI ID)
<lun> is a LUN number
<group> is a LVM group name
<volume> is a LVM volume number
<device-path> is a path such as /dev/sdb
Multiple devices are supported, where multiple sanboot= arguments are included on the command line. Some kernel
arguments control aspects of the settle facility in the initrd. Settling is used to wait for Xsigo’s drivers to initialize.
The options are:
sanwait[=<secs>]
This option waits for vHBAs for <secs> seconds. A value of 0 means no wait. If no value is specified, then the
default is used (30 seconds).
netwait[=<secs>]
This option waits for vNIC for <secs> seconds. A value of 0 means no wait. If no value is specified, then the
default is used (30 seconds).
If both of these arguments are not present, then both are deemed to be enabled by default (wait for net and SAN).
When booting from SAN, the default behavior is to wait for both vNICs and vHBAs to settle. Because there may be no
vNICs present in the server profile, time is wasted by waiting for them. You can specify that the initrd waits for only the
vHBA by adding sanwait to the kernel command line.
Xsigo Systems VP780
140
You can have multiple arguments on the kernel command line.
And some for troubleshooting:
bootdebug
Turns on debugging detail in the initrd.
bootmenu
Runs the bootmenu instead of automatically booting. See Boot Menu Troubleshooting on page 143 for more
details.
Chassis Configuration
If you build a SAN boot configuration for a server-profile on a chassis, the vHBA driver will create a file called /proc/
driver/vhba/san-info containing information similar to the kernel arguments. The initrd can read this file and determine
the root file system location from it.
See CLI Support on page 128 for information on setting up the chassis configuration.
NFS Booting
Network File System (NFS) booting refers to the ability to use a vNIC to mount a root file system using NFS.
Mounting NFS Root using DHCP
The simplest way to use the boot system is using DHCP to get a vNIC running and mount the root file system. This is
specified by using the following kernel parameters:
nfsboot=<vnic-name>
nfsroot=<server-ip-address>:/<root-mount-point>
Where:
<vnic-name> is the name of the vNIC interface to use
<server-ip-address> is the IP address of the NFS server (not the hostname)
<root-mount-point> is the location on the server containing the root file system
The IP address, netmask and default gateway for the vNIC interface will be obtained from the DHCP server.
Mounting NFS Root Using Static Networking
If DHCP is not available or desired, you can also use static networking to configure the vNIC. The kernel parameters for
this are:
nfsboot=<vnic>:<vnic-ip>:<netmask>:<gateway-ip>
nfsroot=<server-ip-address>:/<root-mount-point>
Where:
<vnic> is the name of a vNIC interface
<vnic-ip> is the IP address to be given to the vNIC
Xsigo Systems VP780
141
PXE Install
<netmask> is the full format netmask (usually 255.255.255.0) for the vNIC
<gateway-ip> is the IP address of the default router or gateway
<server-ip-address> is the IP address of the NFS server (not hostname)
<root-mount-point> is the location on the server containing the root file system
Specifying Alternate Mount options
By default, the NFS file system is mounted using the options: “ro,nolock”. You can modify the “nolock” option and
provide additional options using the kernel parameter:
nfsmountopts=<options>
These options replace the “nolock” option in the default set. The “ro” option is always present.
Examples
Example 1: Mount the root file system using vnic0 from server 192.168.9.132:/nfsroot. Use DHCP to get the network
running.
Kernel args:
nfsboot=vnic0 nfsroot=192.168.9.132:/nfsroot
Example 2: Mount the root file system using vnic3 from 192.168.229.10:/nfsroot. Use a static network setup. The IP
address of the vNIC is 192.168.229.11, netmask is 255.255.255.0 (24 bits), default gateway is 192.168.229.1
Kernel args:
nfsboot=vnic3:192.168.229.11:255.255.255.0:192.168.229.1
nfsroot=192.168.229.10:/nfsroot
Example 3: Run in troubleshooting mode (boot menu).
Simply omit the kernel args
PXE Install
Xsigo’s PXE solution does not modify the Linux installer (Red Hat’s Anaconda installer, or SuSE’s linuxrc/YaST).
Instead, Xsigo uses the versions supplied directly by the OS provider and adds in Xsigo’s driver disks. A driver disk is an
ISO image that provides the installer with the required drivers. In Xsigo’s case, the vNIC and vHBA drivers are needed to
use them for installation.
To use a driver disk, you have to make it available to the installer during runtime. The typical way to do this is using a CD
or floppy. To do this, an ISO image called xsigo-rhdd-<kversion>-<arch>.img is provided. This file is an
Anaconda driver disk.
To use it, add the kernel argument dd to the kernel on boot up. Or at the boot: prompt, type linux dd. Anaconda will
prompt you for a driver disk when it runs. The easiest way is to burn the ISO image onto a CD and put it in the CD drive.
Anaconda will read the CD and install the Xsigo drivers.
Xsigo Systems VP780
142
When the drivers are loaded (which may take up to 30 seconds), Anaconda will then detect new vNICs along with any
physical Ethernet interfaces present on the server. The vNICs appear with the same name as was given on the Xsigo
chassis (vnic0, etc).
Note
Due to a limitation in the Anaconda installer, the vNICs may appear with the description “Unknown device:
199d:8202” instead of a more legible string.
Xsigo recommends that you modify the stock Anaconda initrd by inserting our driver disk into it. This can be done using
the script xg-insert-dd. If you do this, the “Unknown device” error will not be present and the vNICs will appear as
Xsigo vNIC
Inserting the Driver Disk into the Anaconda initrd
If you want to avoid using a CD to load the driver disk, you can insert the disk into the Anaconda initrd using the script
xg-insert-dd. To do this, you must have a built driver disk and an Anaconda initrd (obtained from your Red Hat
installation disks). Running the xg-insert-dd script creates a new initrd prefixed with the string “xsigo-”. If you use
this in place of the default Anaconda initrd, the Xsigo drivers will be loaded at boot time without the need to specify “dd”
on the kernel command line or having to insert a CD.
Special Consideration for Red Hat 4
Red Hat 4 is a mature operating system now. It uses an older kernel (2.6.9) and an older Anaconda installer (version 10).
This causes us some problems resulting in different behavior when using Anaconda. Here are some special rules that need
to be followed in order to use RHEL4:
1.
The script xg-insert-dd-ext2 must be used to insert the driver disk into the Anaconda initrd.
2.
You need to add the kernel arg: dd=path:/xsigo-dd.img (literally).
3.
You will see 31 additional ethernet interfaces numbered sequentially from the last physical port (eth2,
eth3...eth31).
4.
You will have to call your bootable vNIC “eth<x>” where <x> is a number from 2 to 31.
The reasons for these are that the kernel is too old to support proper addition of PCI devices with given names.
It automatically creates a full PCI bus and you cannot change the name given to the device.
The Anaconda supplied with rhel4 has a bug that causes it to crash if you supply a 'dd.img' in the initrd (this should cause
the driver disk to be automatically loaded). That is why you have to supply the dd=path:/xsigo-dd.img as a kernel arg.
Using Xsigo Driver Update Disks on a SuSE Operating System
Similar to the Red Hat driver disk, the SuSE Driver Update Disk is provided as an ISO image. The easiest way to use this
is to burn it onto CD and place it in the CD drive before you run the SuSE installer. It will be automatically loaded and the
Xsigo drivers will be initialized (this may take up to 30 seconds). The SuSE installer (linuxrc) will then see the Xsigo
vNICs as standard network devices.
Note
Due to a limitation in the linuxrc program, you must be careful in naming the vNICs on the Xsigo system.
The following prefixes are the only ones recognized as network devices by the installer: eth, veth, plip, tr,
arc, fddi, hip, ctc, escon, ci, iucv, hsi.
Xsigo Systems VP780
143
Boot Menu Troubleshooting
Xsigo suggests using “veth” as a prefix but care is needed as “veth” is also used by systems using Xen. It is best to avoid
names such as “veth0”, “veth1”, etc.
Inserting the Driver Update Disk into the Linuxrc initrd
The “xg-insert-dud” script inserts the Xsigo drivers into the SuSE “linuxrc” initrd (obtained from the SuSE Linux
installation disks). The script generates a new initrd with the prefix “xsigo-”. By replacing the default linuxrc initrd with
this new one, the Xsigo drivers will be loaded at boot time without requiring to insert a CD.
Boot Menu Troubleshooting
If the root file system cannot be mounted automatically, the initrd enters a mode called bootmenu. This mode is a
screen-based menu used to troubleshoot problems and try alternatives.
You can force the bootmenu to be run by adding the kernel argument bootmenu to command line.
The menu looks like this:
Xsigo Systems Virtual Boot (1.0.0)
-- Boot configuration -------------------------------------------------------[No boot configuration]
-- Main menu ----------------------------------------------------------------d. Boot from SAN Disk n. Boot from network
r. Refresh screen
c. Reset terminal
E. Exit to shell
R. Reboot
t. Terminal settings
s. Spawn shell
P. Power off
Selection:
The top part of the screen display allows you to enter and display the current boot configuration. Press a key to invoke a
menu option (no return is required). When entering information, TAB moves to the next field. Pressing ENTER moves to
the next field and terminates the entry at the last field. You can also use the UP and DOWN arrow keys to move between
fields.
The d option allows you to enter SAN boot information:
-- Boot configuration -------------------------------------------------------[No boot configuration]
-- SAN boot menu ------------------------------------------------------------v. Show VHBAs
l. Enter LVM configuration
d. Enter SAN disk configuration b. Boot operating system
r. Refresh screen
q. Return to previous menu
Selection:
To enter LVM information, use the l option:
-- SAN LVM configuration ----------------------------------------------------Group
Xsigo Systems VP780
VolGroup00______
144
Volume
Mount opts
0_______________
________________
r. Refresh screen
Selection:
For SAN disk information (target and lun), use the d option:
-- SAN disk configuration ---------------------------------------------------VHBA
Target
LUN
Mount opts
vhba1____________
3________________
42_______________
_________________
r. Refresh screen
Selection:
Xsigo Systems VP780
PXE Boot
145
Introduction
Preboot Execution Environment (PXE boot) enables a host server to boot up over the network. Using the PXE boot
protocol, the host server’s HCA Option ROM obtains the kernel boot image and initial RAM disk (initrd) from the
network (not local disk):
HCA
Host Server
Firmware
HCA HCA
Option
ROM
Kernel, initrd
VP780
Hard Disk
PXE/TFTP
Server
Ethernet port
w/vNIC
Data path
Ethernet network
IP address
DHCP
Server
add vnic <mypxeboot>.<myserver> <slot/port>
set vnic <mypxeboot.myserver> -addr-type=dhcp -boot-capable=yes -ip-addr=<myipaddress>
Figure 1 Network Boot Capable Host Server
During the OS boot sequence, the host server’s PXE agent (Option ROM) scans the network for a PXE image and reads
its boot up instructions from a boot file stored on a boot server.
The following PXE boot services are supported:
Booting to disk—The kernel and initrd come from the network, then the host server mounts modules that enable
its local disk to be visible on the network.
A network install—The host server runs an OS installer program over the network and chooses any of the
available options.
NFS mount—A disk-less mount. The host server obtains all its binaries over the network and advertises its disk
over NFS. Xsigo has created kernel arguments for NFS mount.
Configure a vNIC on the VP780 to support IP connectivity to the boot server. Two servers must be reachable: DHCP,
PXE/TFTP.
DHCP and TFTP servers can be any freeware or commercially available product. Most Linux server distributions include
a freeware DHCP server. For example, a distribution can include Internet Systems Consortium DHCP Server which is
freeware available through the ISC website at http://www.isc.org/index.pl?/sw/dhcp/.
For configuring PXE boot, you will need to configure an instance of a DHCP server and an instance of a TFTP server. In
the TFTP server, you will edit the default file (pxelinux.cfg/default) to point to the location where the Linux distribution
and a kick start process are. The kickstart process is used for configuring installing options. When the kick start process
runs, the boot process brings up the servers that are connected to the VP780 through one or more vNICs.
Xsigo Systems VP780
146
Chapter 14: PXE Boot
The following section documents:
•
getting MAC address information from the VP780
•
configuring a DCHP server and starting DHCP services through the /etc/dhcpd.conf file
•
configuring the TFTP server
•
configuring a TFTP boot file
•
configuring a VP780 vNIC to support PXE booting
Once the kernel and initrd are loaded from the PXE server, initrd runs and you can specify where to mount the hard disk
from over NFS by using kernel arguments. See Xsigo Initrd on page 137.
MAC Addresses for DHCP Requests
This section documents how to retrieve MAC address information from the VP780. You will use this information when
you configure the DHCP server.
Determine a range of MAC addresses that will be allowed to receive DHCP packets for PXE boot by issuing the show
chassis command. For example:
show chassis
--------------------------------------------------dn
chassis
descr
id
1
type
chassisIb15Slot
mac-addr
00:13:97:01:80:00
mac-addr-mask 12
wwn
50:01:39:70:00:00:80:00
--------------------------------------------------1 record displayed
In this example, the starting MAC address in the pool of MAC addresses is allowed to receive DHCP requests.
Note
The VP780’s MAC address consists of hard-coded bits and variable bits. The MAC Address Mask
determines the variable bits of the address. For example, the MAC Address and MAC Address Mask fields
in the displayed dialog are 00:1397:02:40:00 and 12, respectively. The MAC address mask of 12 indicates
that the last 12 bits of the address are variable. As a result, the address can be interpreted as
00:13:97:02:4x:xx where x is a configurable part of the MAC address.
Xsigo Systems VP780
147
DHCP Server Configuration
DHCP Server Configuration
To configure PXE boot through DHCP, first configure the DHCP server. Configuring the DHCP server is accomplished
by configuring DHCP lease variables and network and subnet addresses in the /etc/dhcpd.conf file. The minimum
parameters to provide DHCP services are:
•
default DHCP lease time
•
maximum DHCP lease time
•
DDNS update style
•
subnet address, netmask, and address and netmask ranges that can be used by booting servers
•
PXE boot Linux kernel
•
IP interface address of the vNIC where the TFTP boot server will be configured
•
path to the Linux distributions
Configure and start the DCHP server:
Step 1
Edit /etc/dhcpd.conf and add the following lines:
default-lease-time 600;
max-lease-time 7200;
ddns-update-style interim;
The lines above set the DHCP lease and renew parameters.
subnet 1.22.0.0 netmask 255.255.0.0{
range 1.22.0.1 1.255.254.254
The lines above set the addresses that can be assigned to the booting servers.
filename ‘pxelinux.0’;
The line above identifies the Linux kernel used for PXE booting. The kernel is part of the syslinux RPM.
next-server 1.255.255.249;
The line above should match the vNIC or physical NIC interface address that will host the TFTP boot server.
This address is usually the same as the DHCP server host.
option root-path “/tftpboot/”;
}
The line above identifies the locations of the Linux ramdisk.
Step 2
Start the DHCP services by issuing the following command:
# service dhcpd start
This step activates the DHCP daemon.
Tip
DHCP servers post all error and warning messages to a log file. When you enable DHCP, if errors occur, you
can check the log file at /var/log/messages
Xsigo Systems VP780
148
TFTP Server Configuration
Configuring a TFTP server consists of creating a directory for the Linux distributions, editing the tftp file and reloading
the xinetd file.
To configure a TFTP server, a TFTP RPM must be installed on the PXE server. You can verify that a TFTP RPM is
installed by issuing the following command:
# rpm -qa | grep tftp
If a TFTP server is not installed on the PXE server, you can download and install it by issuing the following command:
# rpm ivh tftp-server-*.rpm
Note
The following installation and configuration files are required for PXE boot up in a Red Hat Linux
environment. Unless otherwise indicated, these files are supplied by you, the customer:
- pxelinux.0 which is the network bootable file.
- vmlinuz-2.6.9-42.EL which is the bootable compressed Linux kernel.
- initrd-rehl4u4_xsigo-i386.img, which is the Xsigo ramdisk image file. This file is provided by Xsigo.
- pxelinux.cfg, which is the directory that contains a configuration file named “default”.
When a TFTP server is installed, proceed with configuring the TFTP server for PXE booting.
Step 1
Create a directory for the Linux distributions:
mkdir /tftpboot
Step 2
Set ownership of the directory to “nobody”:
# chown nobody:nobody /tftpboot
Also, make sure that /tftpboot directory has read-write permissions set for any accounts that are using this
directory.
Step 3
Edit the /etc/xinetd.d/tftp file to change the following line to “no”
# disable = no
This step is mandatory.
Step 4
As an option, you can add the following new lines: to the /etc/xinetd.d/tftp file:
# log_type
=FILE /var/log/ftplog
# log-on-success+=HOST EXIT DURATION
# log-on-failure+=HOST USERID ATTEMPT
Step 5
Load the xinetd file onto the server:
# service xinetd reload
Xsigo Systems VP780
149
Xsigo HCA Firmware Configuration
Xsigo HCA Firmware Configuration
To support PXE boot, you must flash the Xsigo HCA firmware with the new PXE boot programming options. See HCA
Firmware and Option ROM Upgrades on page 193.
PXE Boot Virtual NIC Configuration
The next steps in configuring a vNIC to support PXE boot, is to create a server profile and populate it with a vNIC.
Step 1
Locate a physical server on which you can configure a server profile:
show ib ports
--------------------------------------------------dn
ib:hca-2c90200204cb0:port-2c90200204cb1
name
port-2c90200204cb1
descr
id
2c90200204cb1
oper-state
up
type
hcaPort
host-name
WILSON
chassis-port ServerPort24
lid
7
--------------------------------------------------In this example, the red text highlights the host name of the physical server. Make a note of this string. You
will use it in the next step. If host names are not assigned to the servers, you can use the server’s GUID
which is indicated in the id field in the output of the show ib ports command.
Step 2
Add a server profile and specify the host name or GUID of the physical connection. For example, to add
a server profile called “pxeboot” to the physical server “wilson”, you would enter the following
command:
add server-profile pxeboot wilson,ServerPort4
Step 3
Add a vNIC, specify its port and interface number. For example, to add a vNIC called vn1.pxeboot to
slot 6, port 2:
add vnic vn1.pxeboot 6/2
Note
When you configure the vNIC’s IP address, make sure that the IP address you assign to the vNIC is from the
address range you configured on the DHCP server. The vNIC that you create will be assigned to the template
so that each time you use the template to create a server profile, a pre-configured vNIC is added to the server
profile. You can then right-click on the vNIC and use the Properties menu to change parameters as needed
for the new vNIC (for example, set a unique IP address).
Xsigo Systems VP780
150
Step 4
On the vNIC, set the following parameters:
• set the address type to dhcp
• set -boot-capable=yes
• configure an IP address and netmask
For example, to configure vn1.pubstest for PXE boot through DHCP, and configure IP address
192.168.91.99, you would enter:
set vnic vn1.pubstest2 -addr-type=dhcp -boot-capable=yes -ipaddr=192.168.91.99/24
Step 5
Reboot the host, and while the host is booting, enter the BIOS setup. When you see the following Xsigo
vNIC Boot Menu Option, save this bootable option:
vnic-memfree.zrom 5.4.0 (GPL) etherboot.org
Step 6
When the host boots from the pxelinux.cfg directory, two boot options are offered: linux and local.
• Select linux for performing a PXE boot again.
• Select local for booting from hard drive.
Xsigo Systems VP780
Clusters and Termination Groups
151
Virtual I/O Fabric
Virtual I/O Fabric enables you to expand the size of your virtual I/O capabilities by interconnecting multiple Xsigo chassis
together. From an IB perspective, a multi-chassis configuration appears as a single IB subnet:
Server Ports
IB
Server Ports
MGT
MGT
AUX
AUX
SER-1
SER-1
SER-2
SER-2
Server 2
USB
USB
Server 1
Server 3
Server 4
Figure 1 Server IB Ports Clustered Together
Limitation. Release 1.5 does not support cluster I/O, where all the I/O ports on multiple chassis function as a single logical
resource. The following is not supported:
1.
Termination group members across multiple chassis
2.
Moving a vNIC interface between chassis
The VP780 also supports a decoupled Subnet Manager (SM), which is part of a cluster environment. OFED 1.1 and 1.2
are supported on external IB attached servers that run SM functions. See OpenSM Decoupling on page 157 for more
information.
Xsigo Directory Service
The Xsigo Directory Service Daemon (XDSD) maintains a database of all the reachable chassis and host servers in the
cluster. XDSD runs as an instance on each VP780 and is enabled by default.
XDSD’ core functionality is to do the following:
1.
Accept XCM records from each chassis XCM.
2.
Accept requests from servers for XCM records.
3.
NodeName registration and query.
4.
XDS election process.
Xsigo Systems VP780
152
Chapter 15: Clusters and Termination Groups
When a Xsigo host driver starts up, it has no information on where XDS is running. However the driver does detect where
the SM is running:
Xsigo
Host
Drivers
VP780-1
VP780-2
VP780-3
XDS
XDS
XDS
SM
Xsigo
Host
Drivers
Figure 2 XDS and Host Drivers All Communicating via SM
The SM can run on the VP780 chassis or any external host server. There is no requirement to run SM on a Xsigo VP780
chassis. See OpenSM Decoupling on page 157.
Xsigo Systems VP780
153
Virtual I/O Fabric
XDS Registration Process
On initial boot up, the VP780 starts an XDS registration process to determine which chassis becomes the master XDS and
which chassis becomes the standby XDS. The VP780 that registers first with SM becomes the master. The registration
algorithm is first-come-first-serve.
Figure 3 describes the XDS registration process:
VP780 boots up
XDS starts,
contact OpenSM
Does a standby
XDS already exist?
No
Register with OpenSM
to become the standby XDS
Enter standby XDS mode
Yes
Enter polling state
Does a master
XDS already exist?
Yes
No
Register with OpenSM
as the master XDS
Deregister as the standby XDS
Refresh the master XDS registration given
the associated timeout-lease period, else SM
with remove its records.
Figure 3 XDS Registration Flow Diagram
A chassis first becomes a standby XDS, then a master. Only a standby XDS can become a master. This approach enables
the system to always have backup information, which avoids conditions where SM or a master XDS dies. In these cases,
all state information would be lost.
Xsigo Systems VP780
154
Adding Server Profiles
Regardless of the number of chassis in your network, there is only one designated master and one designated standby:
VP780-1
VP780-2
VP780-3
XDS
XDS
XDS
Master XDS
Standby XDS
Figure 4 Each VP780 Has Master and Standby XDS Information
After the master and standby XDS are identified, each cluster member can participate in server-profile creation.
When you issue the add server-profile command:
admin@iowa[xsigo] add server-profile <name> <phys-con>
The VP780 sends this server record to both the master and standby XDS. This record is retransmitted at periodic intervals.
To ensure database synchronization, each cluster member sends periodic updates to both the master and standby XDS.
If the master XDS fails, the standby will become the master and another VP780 in the cluster will become the new
standby.
Xsigo Systems VP780
155
Virtual I/O Fabric
Each host server knows the address for SM, which in turn sends the master XDS address to the host server. However, the
host server has no knowledge of a standby XDS. The master XDS provides a list of chassis-cluster members to the host
server:
SM
Host queries SM for XDS
information
XDS registers with SM
Host server
Xsigo
Host
Drivers
Master XDS
Host queries XDS for chassis list (cluster list)
Periodic updates
add server-profile server1 <phys-con>
add vnic myvnic.server1 <slot/port>
vNICs
VP780-1
add vhba myvhba.server2 <slot/port>
vHBAs
VP780-2
add vssl myvssl.server3 <slot>
vSSLs
VP780-3
Figure 5 Chassis Cluster Members Connect List
In the above figure, different virtual resources (vNICs, vHBAs, vSSLs) have been configured on server profiles on three
different chassis (VP780-1, VP780-2, VP780-3).
The flow of operations is as follows:
1.
The XDS registers with SM.
2.
The Xsigo host drivers query SM for XDS location information.
3.
The Xsigo host drivers query XDS for the cluster (chassis) list. This list information is used by the host server to
install virtual resources accordingly.
Xsigo Systems VP780
156
Diagnostics Command Support
Use the following diagnostics commands to control and monitor XDS.
Xsigo recommends you do not modify the XDS parameters (xds-param). Use the defaults.
Caution
Syntax
set
set
set
set
diagnostics
diagnostics
diagnostics
diagnostics
xds-param
xds-param
xds-param
xds-param
client-interval <seconds>
reg-interval <seconds>
service-lease <seconds>
topo-interval <seconds>
show diagnostics xds-info
show diagnostics sm-info
client-interval <seconds>—XDS interval for pulling client information (in seconds).
reg-interval <seconds>—XDS register interval (in seconds).
service-lease <seconds>—XDS service lease timer (in seconds).
topo-interval <seconds>—The XDS topology sweep interval. The topology service layer queries SA for
all NodeRecords, PortRecords, LinkRecords and generates an IB topology. This topology is created periodically.
The topology from the previous sweep is compared against to generate Node/Port add and deletes.
xds-info—XDS information.
sm-info—Subnet manager information.
Example
show diagnostics xds-info
XDS Information:
XDS Service Registration Interval : 15 secs
XDS Service Lease Time : 45 secs
XDS Client Poll Interval : 10 secs
XDS Topology Poll Interval : 60 secs
Master XDS Server Name: iowa, Lid: 0x1, Guid: 0x139702010000a1
StdBy XDS Server Name: Unknown, Lid: 0x0, Guid: 0x0
My XDS state: MASTER
show diagnostics sm-info
SM information:
- SM Lid
1
- SM Guid
0x139702010000a1
- SM key
0x0
- SM priority
0
- SM State
MASTER
When a chassis boots up, the XDS daemon running on the chassis contacts SM for the SA service record. If no record is
found, it tries to register itself with SA as the master XDS using the ServiceRecord creation. The ServiceRecord is a
standard database record supported in SA. The ServiceRecord consists of an XDS LID and XDS GUID. SA serializes all
Xsigo Systems VP780
157
OpenSM Decoupling
creation and guarantees only one XDS can create the record with a particular service ID and name. This information
enables an automatic XDS server election process.
When an XDS halts (or a chassis vanishes), there is a lease period associated with the service record. SA will remove this
record from the database when this record is not refreshed by the XDS server.
Each chassis also has a topology service that provides the IB topology to the management layer. This topology is retrieved
using standard SA queries: NodeRecords, PortInfoRecoreds, LinkRecords. At regular intervals the topology is created,
and changes in the topology is notified to the upper layers.
OpenSM Decoupling
Xsigo’s OpenSM can be disabled and replaced by a third party Subnet Manager (SM). Some customers prefer to use their
own version of InfiniBand SM because it includes custom extensions and can be managed externally to the Xsigo chassis.
Note
Certain SMs are qualified to work with the Xsigo VP780. Contact Xsigo customer support for more
information.
Use the set system is-subnet-manager command to control the OpenSM process running on the chassis.
By default, the OpenSM process starts automatically.
For more information about OpenSM, see InfiniBand Ports on page 27.
Syntax
set system is-subnet-manager {true | false | default}
show system info
set system is-subnet-manager—Controls the OpenSM process. There are three keyword options. The
true option enables OpenSM. The false option disable OpenSM. The default returns OpenSM to its
factory default setting, which is true.
show system info—Displays OpenSM state information. See the “is-sm” flag.
Example
The show system info command displays the “is-sm” flag, reflecting the current state of OpenSM:
show system info
----------------------------------------------------------------------------hostname
iowa
domain
lab.xsigo.com
address
192.168.8.133
netmask
255.255.252.0
model-num
VP780
serial-num
050610240
ipconfig
dhcp
default-gateway 192.168.8.1
Xsigo Systems VP780
158
timezone
GMT
domain-search
is-sm
true
----------------------------------------------------------------------------1 record displayed
set system is-subnet-manager false
Are you sure you want to relinquish subnet manager authority? If there are no
other subnet managersavailable, your subnet may become unmanaged (y/n)?y
show system info
-----------------------------------------------------------------------------hostname
iowa
domain
lab.xsigo.com
address
192.168.8.133
netmask
255.255.252.0
model-num
VP780
serial-num
050610240
ipconfig
dhcp
default-gateway 192.168.8.1
timezone
GMT
domain-search
is-sm
false
set system is-subnet-manager true
Are you sure you want to become a subnet manager? This may cause this Xsigo
system to grab ownership of the subnet from another manager (y/n)?y
Xsigo Systems VP780
159
Termination Groups
Termination Groups
A termination group is a collection of ports that allow for large-scale inclusion of vNICs. Once you create a pool of ports
under a termination group, you can add vNICs dynamically to the pool just as if you were adding it to a single port.
A termination group provides the versatility of adding large numbers of vNICs without regard to which I/O card and port
the traffic flows through.
A typical vNIC is conceptually inside a server profile and terminates on an I/O card’s port. In comparison, a termination
group contains member port elements. The vNICs terminate on a named termination group based on a round robin
algorithm. The order in which the ports are added to the termination group sets the round-robin order:
Server Profile
vNIC
Termination
Group Name
vNIC
vNIC
vNIC
add net-term-group <tgname> port <slot>/<port>
add vnic <myvnic>.<myserver> <tgname>
I/O Ports
I/O Cards
Figure 6 Termination Group Used for vNIC Termination
Note
In a future release, Link Aggregation Groups (LAGs) will be supported in hardware on the 10 X 1GigE card.
After it is declared which ports are members of the LAG, a vNIC can be terminated on a LAG.
Xsigo Systems VP780
160
Membership Rules
•
A termination group consists of 1 to n ports.
•
Only ports of the same type may be assigned to a termination group—Quad 4x1GigE ports vs 10x1GigE
ports vs 10GigE ports. For example, the system does not support a group containing 1 10G port and 4G
ports.
•
A port may belong to more than 1 termination group.
•
For XgOS Release 1.5, the following restrictions are imposed:
— All of the I/O cards in a termination group must have exactly the same of QoS and ACL settings:
The VP780 will check to make sure these settings are the same when the termination group is created
and any time a QoS or ACL change is made.
If a user tries to change a QoS or ACL setting used by a termination group, the VP780 will display an
error and not let the user make the change, unless the user selects to change all of the I/O cards impacted
by the termination group.
If there are overlapping termination groups on the same card, the change will impact all the termination
groups and all I/O cards associated with those termination groups.
— All the ports in a termination group must belong to the same chassis (cross chassis termination groups
are not supported in Release 1.5.)
— Users must ensure that all ports in the same termination group are terminated on the same L2/L3
network.
— Termination groups apply to Ethernet ports only.
Syntax
add net-term-group <tgname> port [* | <slot>/<port> | <slot>/*]
set net-term-group [<tgname> | *] -algorithm={roundRobin | default}
set net-term-group <tgname> reassign
set net-term-group [<tgname> | *] -descr=<text>
remove net-term-group [<tgname> | * | port <slot>/<port>]
show net-term-group [<tgname> | *]
add vnic myvnic.myserver <tgname>
set vnic <vnic-name> -term-group=[<tgname> | none]
show vnic
add net-term-group <tgname>—Creates a termination group and includes one or more port elements.
Replace <tgname> with a user-defined name. Specify the slot and port information for port elements to include
in the group. You can use the asterisk (*) for all slots/ports in the system, or just one slot (<slot>/*).
set net-term-group [<tgname> | *] -algorithm—Sets the port-selection algorithm of one or more
termination groups. The algorithm is roundRobin by default.
Xsigo Systems VP780
161
Termination Groups
set net-term-group <tgname> reassign—Reassigns ports to a termination group after they have been
removed using remove net-term-group port.
set net-term-group [<tgname> | *] -descr=—Sets the description text for one or more termination
groups. Replace <text> with your own description.
remove net-term-group [<tgname> | *]—Deletes one or more termination groups from the system.
show net-term-group [<tgname> | *]—Displays configuration details for one or more termination groups
on the system.
add vnic <vnic-name> <tgname>—References a vNIC at add time to a termination group.
set vnic <vnic-name> -term-group=[<tgname> | none]—References a pre existing vNIC to a
termination group. Use the none option to remove a vNIC from a termination group.
Example 1
Perform the following steps:
Step 1
Create a named termination group and add I/O port elements to it. These elements form a pool of
member I/O ports. The default port-selection algorithm is round robin.
add net-term-group mytermgroup port 4/1
add net-term-group mytermgroup port 4/2
Step 2
Display the configured settings:
show net-term-group mytermgroup
name
descr
algorithm
ports
-----------------------------------------------------------------------mytermgroup
roundRobin
4/2, 4/1
1 record displayed
Step 3
Reference the vNICs that will terminate on the named termination group:
add vnic myvnic.myserver mytermgroup
show -list vnic myvnic.myserver
-----------------------------------------------------------------------name
myvnic.myserver
state
up/resourceUnavailable
mac-addr 00:13:97:01:80:05
ipaddr
if
4/1
if-state up
type
vlans
none
vm
qos
Note that a pre existing vNIC can be associated with a termination group using the -term-group= option:
set vnic myvnic.myserver -term-group=mytermgroup
Xsigo Systems VP780
162
Example 2
Use the -term-group=none set option to remove a vNIC from a termination group:
set vnic myvnic.myserver -term-group=none
Example 3
remove net-term-group *
Remove all network termination groups (y/n)?y
Example 4
As a best practice, Xsigo recommends using vNIC templates in conjunction with termination groups. Create a vNIC
template then point it at the termination group. When you instantiate the vNICs on the template, the system will
automatically pick one the I/O ports from the pool and start using it on the vNIC.
This example shows how to use a vNIC template with a termination group:
Step 1
Create a named vNIC template:
add template myvnictemplate
Step 2
Add a vNIC to the template and associate it with the termination group:
add vnic myvnic.myvnictemplate mytermgroup
show -list template myvnictemplate vnics
-----------------------------------------------------------------------name
myvnic.myvnictemplate
state
/
mac-addr
ipaddr
if
if-state type
vlans
none
vm
qos
Step 3
To instantiate the vNIC template, create a server profile based on the vNIC template. The vNICs
associated with the termination group will now be allocated ports:
add server-profile myserver ceasar,iowa:ServerPort24 -template=myvnictemplate
Xsigo Systems VP780
System Management
163
System Image Upgrades
The XgOS software image is a Xsigo Package File (XPF) file. Use the system upgrade command to upgrade the
XgOS by supplying a URL for the path of the XPF file.
The XgOS upgrade procedure supports the following upgrade schemes:
•
Hypertext Transfer Protocol (HTTP)
•
HTTP over Secure Socket Layer (HTTPS)
•
Secure Copy (SCP)
•
File Transfer Protocol (FTP)
•
Local file
TFTP system upgrades are not supported.
See Configuration Save and Restore on page 38.
Syntax
system
system
system
system
system
upgrade
upgrade
upgrade
upgrade
upgrade
[-noconfirm]
[-noconfirm]
[-noconfirm]
[-noconfirm]
[-noconfirm]
http://<image-path.xpf> [debug]
https://<image-path.xpf> [debug]
scp://<image-path.xpf> [debug]
file://<image-path.xpf> [debug]
ftp://<image-path.xpf> [debug]
system export <filename> [-cli -defaults][-xml]
system import <filename> [-cli][-xml]
All schemes have the following general syntax:
scheme://user@host/image-path.xpf
You can omit the “user@” bit if the same user name is available on the server from which you are loading the XPF file.
If the scheme is file:, you can omit the host.
http://<image-path.xpf>—Upgrade using HTTP.
https://<image-path.xpf>—Upgrade using HTTPS.
scp://<image-path.xpf>—Upgrade using SCP.
file://<image-path.xpf>—For upgrading from a file stored locally on the VP780. For example from
disk, USB (a mounted /usb device), or a /home directory. In cases where you are using local upgrade through the
file command, you can copy the XPF file into the VP780 by using the file copy command.
ftp://<image-path.xpf>—Upgrade using FTP.
debug—The optional debug argument can be placed at the end of the command. The argument is passed through
to the image management software on the chassis.
-noconfirm—You can perform upgrades in confirmation or non-confirmation mode. The -noconfirm
argument is optional, and the behavior of prompts is different depending on whether you use this argument:
Xsigo Systems VP780
164
Chapter 16: System Management
— When you do specify -noconfirm, the upgrade completes without prompting you for confirmation.
The argument automatically answers yes to any prompts.
— When you do not specify -noconfirm, you will be prompted for a yes or no answer as needed during
the upgrade.
system export | import—Before you upgrade the software, Xsigo recommends you export your system
configuration to a file. If your running-config gets lost during an upgrade, at least you can import the old one.
Example 1
To upgrade the XgOS system image, perform the following steps:
Step 1
Ensure your permissions role is administrator:
show user
name
descr
roles
role-group
domains
domain-group
----------------------------------------------------------------------admin
administrator
administrator_group
root
root
Step 2
Issue the system upgrade command and supply the full path to the new system image. Here are
several examples for each of the supported upgrade types.
system
system
system
system
system
upgrade
upgrade
upgrade
upgrade
upgrade
http://cairo.xsigo.com/upgrades/xsigo-V1.0.00.xpf
https://cairo.xsigo.com/upgrades/xsigo-V1.0.00.xpf
scp://[email protected]/upgrades/xsigo-V1.0.00.xpf
file:///upgrades/xsigo-V1.0.00.xpf
ftp://[email protected]/upgrades/xsigo-V1.0.00.xpf
The CLI copies the XPF image to /var/tmp. The image manager deletes the image when it unpacks into /opt/
xsigo/xsigos. The image is hidden from the users logged into the system since the file is only a temporary
file copy.
While the upgrade occurs, status messages are displayed:
You have made changes to the configuration. Commit these before upgrading
(y/n)?y
Copying...
########################################################################
######################################## [100%]
The following software will be installed:
1. XgOS Operating System software including SCP Base OS
2. XgOS front-panel software
3. XgOS VNIC Manager and Agent software
4. XgOS VN10G Manager and Agent software
5. XgOS VHBA Manager and Agent software
6. XgOS VSSL Manager and Agent software
Are you sure you want to update the software (y/n)? y
Running preunpack scripts...
Installing...
########################################################################
######################################## [100%]
Verifying...
########################################################################
######################################## [100%]
Running preinstall scripts...
Installing package...
Xsigo Systems VP780
165
System Processes
Running postinstall scripts...
Installation successful. Please stand by for CLI restart.
admin@iowa[xsigo]
XgOS CLI is restarting - This might take a couple of minutes...
*01:00
System services are available again. Restarting the CLI now.
Welcome to XgOS
Copyright (c) 2007 Xsigo Systems, Inc. All rights reserved.
Note
If you get the following error during the upgrade:
Installation failed (Unable to unpack package file xsigo-<build-x>.xpf
where <build-x> is the system image, then issue the system clear garbage command to remove any
partial or failed installs.
Step 3
When the VP780 has completed its restart, issue the show system version command to verify that
the new software has been installed:
show system version
Build 1.0.0 - (root) Fri Jun
29 22:49:33 PDT 2007
Example 2
system clear config
This is a destructive operation. Your configuration will be cleared and the
system will be restarted. Please type 'confirm' to clear the configuration and
restart the system.
>confirm
system upgrade http://cairo.xsigo.com/upgrades/xsigo-V1.0.00.xpf
system cold-restart
Are you sure you want to restart the system (y/n)? y
Prior to Release 1.0, Xsigo recommended that users clear the config and perform a cold-restart after an upgrade.
After Release 1.0, this procedure is not required. The XgOS now ensures that the user’s config is not broken after an
upgrade (not the case before).
In general, a system clear config is not required before an upgrade. The only reason you might want to clear a
config is to completely wipe it out and start over again. This command resets all values in the VP780’s configuration
database to the factory defaults. When you issue the system clear config command, you are prompted for
confirmation before the configuration is cleared. When prompted, you must enter “confirm” to clear the configuration.
Any answer other than “confirm” aborts the system clear config command.
System Processes
Many processes are enabled on the VP780. Note the three types of processors (fpp, iop, scp) used in different card types
(front panel, I/O card, system controller). Each I/O card has its own I/O control Processor (IOP). Agents running at
Xsigo Systems VP780
166
various locations in the system communicate with managers. For example, the vhbaagent and vhbamanager work together
for vHBA creation, removal, RSCN, disk discovery tasks, and so on:
show system processes
name
processor slot memory cpu-time num-restarts time-started
-------------------------------------------------------------------------------chassisCtr
fpp
1
6.40625 00:00:03 0
2007-07-10 19:43:43.583
vhbaagent
iop
1
4.57031 00:00:01 0
2007-07-10 19:43:43.583
chassisAgt
iop
1
5.97656 00:00:01 0
2007-07-10 19:43:43.583
chassisAgt
iop
4
5.96484 00:00:01 0
2007-07-10 19:43:43.583
vnicagent
iop
4
6.3125 00:00:01 0
2007-07-10 19:43:43.583
vhbaagent
iop
8
4.50391 00:00:01 0
2007-07-10 19:43:43.583
chassisAgt
iop
8
6.00391 00:00:01 0
2007-07-10 19:43:43.583
vssl_agent
iop
13 4.58203 00:00:01 0
2007-07-10 19:43:43.583
chassisAgt
iop
13 5.98828 00:00:01 0
2007-07-10 19:43:43.583
xtctrl
scp
0
00:00:00 0
2007-07-10 19:44:29.555
xtctrl
scp
0
00:00:00 0
2007-07-10 19:44:29.555
apache2_prerun.sh scp
0
00:00:00 0
2007-07-10 19:44:29.555
reap_db
scp
0
00:00:00 0
2007-07-10 19:44:29.555
resurrect_db
scp
0
00:00:00 0
2007-07-10 19:44:29.555
resurrect_sysctl scp
0
00:00:00 0
2007-07-10 19:44:29.555
in.tftpd
scp
0.589844 00:00:00 0
2007-07-10 19:43:43.532
xtctrl
scp
0.925781 00:00:00 0
2007-07-10 19:43:43.583
opensm
scp
2.73047 00:00:03 0
2007-07-10 19:43:43.532
postmaster
scp
2.85547 00:00:00 0
2007-07-10 19:43:43.532
snmpagent
scp
16.0508 00:00:11 0
2007-07-10 19:43:43.532
apache2
scp
21.6328 00:00:51 0
2007-07-10 19:43:43.532
imagemanager
scp
1
4.10938 00:00:00 0
2007-07-10 19:43:43.532
xc_xsmp
scp
1
12.7344 00:00:00 0
2007-07-10 19:43:43.532
xc_xsm
scp
1
13.5742 00:00:10 0
2007-07-10 19:43:43.532
healthmonitor
scp
scd
scp
chassisMgr
scp
systemcontroller scp
xc_manager
scp
scriptsvc
scp
vssl_manager
scp
vhbamanager
scp
vnicmanager
scp
mimm
scp
34 records displayed
1
1
1
1
1
1
1
1
1
1
15.1836 00:00:36 0
15.5273 00:00:26 0
16.4609 00:00:33 0
16.7344 00:00:04 1
16.7812 00:00:43 0
34.2852 00:00:01 0
37.8203 00:00:18 0
38.043 00:00:23 0
38.3242 00:00:19 0
42.8555 00:01:03 0
2007-07-10
2007-07-10
2007-07-10
2007-07-10
2007-07-10
2007-07-10
2007-07-10
2007-07-10
2007-07-10
2007-07-10
19:43:43.532
19:43:43.532
19:43:43.532
19:43:43.532
19:43:43.532
19:43:43.532
19:43:43.532
19:43:43.532
19:43:43.532
19:43:43.532
The Main Information Model Manager (MIMM) process runs on the system controller card. The MIMM is responsible for
accepting all commands from the CLI. When an object is modified, the request goes to the MIMM process. In turn, the
configuration request is delegated onto a sub system model manager (i.e., vnicmanager, vhbamanager).
Each I/O card has a chassisAgt that reports hardware environmental event information. See show hardware.
Xsigo Systems VP780
167
User Accounts, Role Groups, and Domains
Users, role groups, and domain groups are all interrelated:
•
User accounts are created to grant people access to the chassis.
•
The role-group (privileges) that a user belongs to determines which objects the user can modify.
•
A Domain group contains a list of Domains. One Domain Group is assigned to each user. The user can write
to only the domains listed in his/her Domain Group but can read from any domain. The domain group the
user is assigned to specifies the domains in which the user can exercise role-group privileges.
Create a User
Step 1
Add a user:
add user frank
Step 2
Note the default role is operator with no role-group defined:
show user frank
name
descr
roles
role-group
domains
domain-group
-----------------------------------------------------------------------frank
operator
Step 3
Set a password for the user:
set user frank -password
New password:
New password again:
Step 4
Test the new user account:
quit
Connection to 192.168.8.133 closed.
Password:
Welcome to XgOS
frank@iowa[xsigo]
frank@iowa[xsigo] pwd
/home/frank
Step 5
(Optional) User privileges determine administrative abilities:
frank@iowa[xsigo] add user intruder
User not allowed to modify/create/delete system-local:security:userintruder due to insufficient privileges
frank@iowa[xsigo] remove user frank
Remove user frank (y/n)?y
Failed to remove security user frank
frank@iowa[xsigo] quit
Connection to 192.168.8.133 closed.
Xsigo Systems VP780
168
Password:
Welcome to XgOS
admin@iowa[xsigo] remove user frank
Remove user frank (y/n)?y
admin@iowa[xsigo]
Role Groups
A role group defines a user’s privileges with regards to modifying objects. A user can be optionally assigned to a role
group.
Example:
Step 1
Add a user named “newuser1”:
add user newuser1 ?
-domain-group domain group for user
-password
set password
-role-group
role group for user
At this point, you can attach the “user” object to a domain group, to a role group, or give it a password.
Step 2
Display the available role groups. The default role is operator with no role-group defined:
add user newuser1 -role-group=?
administrator_group
compute_group
equipment_group
fault_group
interface_security_group
network_group
operations_group
operator_group
policy_group
qos_group
stats_group
storage_group
transport_security_group
user_group
vhba_group
virtual_group
vnic_group
vssl_group
Xsigo Systems VP780
169
A role-group defines the write privileges assigned to a particular type of user. There are 18 pre defined
groups:
Table 1 Role Group Privileges
Step 3
Role Group Name
Privileges
administrator_group
Allows editing of all objects
compute_group
All operations related to a server’s physical
connection, all server-pool management tasks
equipment_group
All chassis-related, hardware-management tasks (e.g.
I/O cards, I/O ports)
fault_group
Can view alarms, set fault thresholds
interface_security_group
All vSSL configuration tasks
network_group
All objects related to vNIC configuration, QoS
parameters, ACLs, server-pools
operations_group
All operational tasks (e.g. add licenses, log
management, config import/export, DB collection)
operator_group
Allows read-only access and all ‘show’ commands
policy_group
Can edit backup policies, DB policies, and set
templates
qos_group
All QoS-related tasks
stats_group
All Statistics-related tasks
storage_group
Any vHBA configuration, or SAN-QoS tasks
transport_security_group
All XML (SOAP) operations, HTTPS
user_group
Allows editing of local users and role groups
vhba_group
All vHBA-related tasks
virtual_group
All virtual resources
vnic_group
All vNIC-related tasks
vssl_group
All vSSL-related tasks
Choose the role group to which the user will be assigned. Select a role group (“user_group”, in this
example).
add user newuser1 -role-group=user_group ?
-domain-group domain group for user
-password
set password
-role-group
role group for user
Xsigo Systems VP780
170
Step 4
Assign the user to a domain group:
add user newuser1 -role-group=user_group -domain-group=?
domain_group1
group1
root
Step 5
Choose the domain group. For example, select “domain_group1”:
add user newuser1 -role-group=user_group -domain-group=domain_group1
This command set created a user (newuser1). The user was assigned to both a role-group (user_group),
which defined the user’s privileges, and to a domain group (domain_group1), which specified the domains
in which the user could exercise those privileges.
Step 6
Verify the user configuration was correctly configured:
show user newuser1
name
descr
roles
role-group
domains
domain-group
-------------------------------------------------------------newuser1
user
user_group
domain1
domain_group1
Domain Groups
A domain group specifies the domain(s) in which the user may exercise role-group privileges.
Example:
Step 1
Create a domain group named “domain_group1”:
add domain-group domain_group1
Step 2
Choose a domain to attach to the domain group:
set domain-group domain_group1 add-domain domain1
This command set created a domain group (domain_group1), and attached it to a domain (domain1).
Xsigo Systems VP780
171
Identity Management System
The Identity Management System (IMS) service authenticates users and grants them suitable privileges according to
assigned user roles and domains when managing the VP780 through the CLI, GUI, or other applications. The actual IMS
server can be either Microsoft Active Directory (AD), local system, Remote Authentication Dial In User Service
(RADIUS), or Lightweight Directory Access Protocol (LDAP) depending on the user’s configuration. Once the
configuration is applied, the IMS service is completely transparent to the operator.
The IMS server functions as a central Authentication, Authorization, and Accounting (AAA) repository. Note that “local”
will always be present, even when AD/RADIUS/LDAP is chosen. This action also guarantees that a user can always log
on to a Xsigo chassis using a local admin account, even when all the connections to other IMS servers are down.
User
SSHd / PAM
GUI
Other
Apps
CLI
IPC
IMS Service
Configuration
internal
Database(MOs)
Caching
Config. files
IMS Server
Local
AD
RADIUS
LDAP
(IBM, SUN)
Figure 1 IMS Component Diagram
Xsigo Systems VP780
172
IMS supports AD and RADIUS. Each has two authentication methods:
1.
AD = Kerberos, simple (default)
2.
RADIUS = CHAP, PAP (default)
For AD, you can configure up to two servers: one primary, one secondary. For RADIUS, up to five servers can be
configured. Each RADIUS server has equal preference (no ranking).
Syntax
add
add
add
add
add
add
add
add
add
add
add
add
add
ims
ims
ims
ims
ims
ims
ims
ims
ims
ims
ims
ims
ims
ad-server
ad-server
ad-server
ad-server
ad-server
ad-server
ad-server
ad-server
ad-server
ad-server
ad-server
ad-server
ad-server
<name>
<name>
<name>
<name>
<name>
<name>
<name>
<name>
<name>
<name>
<name>
<name>
<name>
-authentication-type=[kerberos][simple][default]
-base-dn=<value>
-user-dn=<value>
-formal-user-dn=<value>
-port=[<number>][default]
-domain-represented-by=[dn][group][default]
-host-name=<name>
-password=<user-dn-password>
-server-mode=[primary][secondary][default]
kerberos -kdc-host-name=<value>
kerberos -kdc-port-num=[<number>][default]
kerberos -kerberos-default-domain=<value>
kerberos -kerberos-default-realm=<value>
add ims radius-server <name> <options>
add ims radius-user <name> -roles=<role>
set
set
set
set
set
ims
ims
ims
ims
ims
-cache-timeout=[<number> | default]
-maps-to-root=<value>
-search-order=[default][externalFirst][internalFirst]
-server-type=[default][ldap_ad][local_only][radius]
-token-timeout=[<number>][default]
set ims ad-server <name> <options>
set ims radius-server <name> <options>
set ims radius-user <name> <options>
remove ims ad-server <name>
remove ims radius-server <name>
remove ims radius-user [<name>]*]
show ims [-detail]
show ims ad-server [<name>|*][-detail]
show ims radius-server [<name>|*][-detail]
show ims radius-user [<name>|*]
show system user
system flush ims
Xsigo Systems VP780
173
Example 1
This example configures an AD server with simple (default) authentication. The example takes advantage of the following
default settings: –port (389), -domain-represented-by (group), -server-mode (primary) and -authentication-type (simple).
Only four fields need to be configured:
add ims ad-server AD -host-name=sfcorpdns1.xsigo.com [email protected] base-dn="DC=XSIGO,DC=COM" -password
New password:
New password again:
show ims ad-server AD
--------------------------------------------------------------------------name
AD
descr
host-name
sfcorpdns1.xsigo.com
admin-state
up
oper-state
up
authentication-type simple
server-mode
primary
--------------------------------------------------------------------------show ims ad-server AD -detail
--------------------------------------------------------------------------name
AD
descr
host-name
port
389
admin-state
up
oper-state
up
oper-state-qual
normal
err-message
user-dn
[email protected]
base-dn
DC=XSIGO,DC=COM
server-mode
primary
formal-user-dn
domain-represented-by
group
authentication-type
simple
kerberos-default-realm
kerberos-default-domain
kdc-host-name
kdc-port-num
---------------------------------------------------------------------------
Example 2
This example configures Kerberos as a secondary AD. This example takes advantage of the following default values: –
port (389), -domain-represented-by (group), and -kdc-port-num (88):
add ims ad-server AD2 -host-name=sfcorpdns2.xsigo.com [email protected] base-dn="DC=XSIGO,DC=COM" -server-mode=secondary -authentication-type=kerberos
-password -formal-user-dn="cn=JOE User,cn=Users,dc=xsigo,dc=com" kerberos -kdchost-name=sfcorpdns2.xsigo.com -kerberos-default-realm=XSIGO.COM -kerberosdefault-domain=xsigo.com
New password:
New password again:
Xsigo Systems VP780
174
show ims ad-server AD2
-----------------------------------------------------------------name
AD2
descr
host-name
admin-state
up
oper-state
up
authentication-type kerberos
server-mode
secondary
-----------------------------------------------------------------show ims ad-server AD2 -detail
-----------------------------------------------------------------name
AD2
descr
host-name
port
389
admin-state
up
oper-state
up
oper-state-qual
normal
err-message
user-dn
[email protected]
base-dn
DC=XSIGO,DC=COM
server-mode
secondary
formal-user-dn
cn=JOE User,cn=Users,dc=xsigo,dc=com
group
authentication-type
kerberos
XSIGO.COM
kerberos-default-domain xsigo.com
kdc-host-name
kdc-port-num
88
-----------------------------------------------------------------If the configuration is not correct, the oper-state will be “down”. An error message will show the corresponding warning
so the administrator will know how to use set ims ad-server <name> to resolve the problem. If the oper-state is
“indeterminate”, it means the server-type of IMS is not ldap_ad. The administrator can then use set ims –servertype to fix the problem:
set ims -server-type=ldap_ad
show ims
cache-timeout
token-timeout
server-type
-----------------------------------------------------------------240
5
ldap_ad
show ims -detail
-----------------------------------------------------------------cache-timeout
240
token-timeout
5
server-type
ldap_ad
search-order
internalFirst
maps-to-root
root
num-of-servers
3
num-of-ad-servers
2
num-of-sun-servers
0
num-of-ibm-servers
0
num-of-radius-servers 1
------------------------------------------------------------------
Xsigo Systems VP780
175
Example 3
A search-order of “internalFirst” means the local db is searched first. Use set ims -search-order= to change the
search order:
show ims -detail
-----------------------------------------------------------------------------cache-timeout
240
token-timeout
5
server-type
ldap_ad
search-order
internalFirst
maps-to-root
root
num-of-servers
2
num-of-ad-servers
1
num-of-sun-servers
0
num-of-ibm-servers
0
num-of-radius-servers 1
In the example above, two servers are configured: one AD server and one RADIUS server. Use the expert CLI (not user
CLI) to configure the SUN and IBM servers if needed.
Example 4
When you log in, the VP780 queries the specified IMS server for your active role and domain information. You can then
print that data to the terminal by using show system user command. In this example, the role is “OPERATOR”, and
this user belongs to 7 domains which is organized in a tree-like format:”
show system user
username: joeuser
session: 1
roles: 0x1 (OPERATOR)
domains:
domain-root:domain-SanJose-All
domain-root:domain-Team-linux-drivers
domain-root:domain-svn-all:domain-svn-sw
domain-root:domain-xsigli:domain-Team-Systest
domain-root:domain-eng:domain-Dept-Engineering
domain-root:domain-svn:domain-Dept-Engineering
domain-root:domain-Dept-All:domain-Dept-Engineering
...
show domain
name
descr
level
groups
--------------------------------------------------------root
0
root
“root” is the default top-most domain. Use add domain to create new domains on the system. Any user with the correct
privileges within the domain (or under the parent domain) can manage virtual resources within the domain.
Xsigo Systems VP780
176
Example 5
show domain
name
descr
level
groups
--------------------------------------------------------------------------root
0
root
1 record displayed
To add a domain Dept-All under root domain, you do not need to type root in the command as it’s assumed:
add domain Dept-All
show domain
name
descr
level
groups
--------------------------------------------------------------------------Dept-All
1
root
0
root
2 records displayed
To add a domain Dept-Engineering under Dept-All domain:
add domain Dept-Engineering.Dept-All
show domain
name
descr
level
groups
--------------------------------------------------------------------------Dept-All
1
Dept-Engineering.Dept-All
2
root
0
root
3 records displayed
set server-profile s23 -domain=?
Dept-All
Dept-Engineering.Dept-All
root
To set this server-profile to the domain Dept-Engineering.Dept-All:
set server-profile s23 -domain=Dept-Engineering.Dept-All
show server-profile
name
s23.Dept-Engineering.Dept-All
state
up/up
descr
connection titan@connecticut:ServerPort22
def-gw
test
vnics
1
vhbas
1
vssls
0
--------------------------------------------------------------------------1 records displayed
As you can see here, the vHBA name under this server-profile automatically has Dept-Engineering.Dept-All at the end
indicating that this vHBA is under this domain:
show vhba
--------------------------------------------------------------------------name
vhba7.s23.Dept-Engineering.Dept-All
Xsigo Systems VP780
177
state
up/up
fabric-state up
if
4/1
if-state
up
wwnn
50:01:39:71:00:00:F1:37
wwpn
50:01:39:70:00:00:F1:37
map
lun-mask
--------------------------------------------------------------------------23 records displayed
From now on, a user with the right role and under domain Dept-Engineering.Dept-All can manage resources such as
vHBAs under this server-profile. For example user foobar below belongs to domain-root:domain-Dept-All:domain-DeptEngineering so that he can manage this vHBA as long as he has the right roles:
show system user
username: foobar
session: 1
roles: 0x1 (OPERATOR)
domains:
domain-root:domain-SanJose-All
domain-root:domain-Team-linux-drivers
domain-root:domain-svn-all:domain-svn-sw
domain-root:domain-xsigli:domain-Team-Systest
domain-root:domain-eng:domain-Dept-Engineering
domain-root:domain-svn:domain-Dept-Engineering
domain-root:domain-Dept-All:domain-Dept-Engineering
...
Example 6
Normally IMS has to query the specified IMS server for roles/domains info when the user tries to log in as the first time.
This kind of queries could be resource intensive especially for external IMS servers (such as AD and RADIUS).
The chassis has a local cache to store role an domain information for 240 minutes by default. The next time you login with
that timeframe, IMS does not need to query AD again:
show ims -detail
-----------------------------------------------------------------------------cache-timeout
240
token-timeout
5
...
Configure set ims -cache-timeout=0 to disable the cache. AD will be queried every time someone logs in.
Additionally the system flush ims command is available for “ADMIN” users to flush the cache immediately:
system flush ims
Identity Management System cache flushed
Example 7
show ims ad-server * -detail
-----------------------------------------------------------------------------name
AD1
descr
host-name
ad1.xsigo.com
Xsigo Systems VP780
178
port
389
admin-state
up
oper-state
up
oper-state-qual
normal
err-message
user-dn
[email protected]
base-dn
DC=XSIGO,DC=COM
server-mode
primary
formal-user-dn
group
authentication-type
simple
kerberos-default-domain
kdc-host-name
kdc-port-num
The chassis maintains a connection between IMS and the remote AD server. The “user-dn” is the user that initiates and
maintains this connection. In the above example, the user is “[email protected]”. The user must have at least read
privileges since it queries all the role and domain information. The “base-dn” is the tree-search range. You can reduce the
search scope to increase the search speed, for example “DC=Users, DC=XSIGO, DC-COM”.
show ims radius-server * -detail
-----------------------------------------------------------------------------name
RAD1
descr
testtt
host-name
foo.xsigo.com
port
1812
admin-state
up
oper-state
indeterminate
oper-state-qual
normal
err-message
user-name
user1
authentication-type PAP
timeout
3
retries
3
Example 8
For an external AD server, configure the Roles for the users who want to log in to the VP780. Do this by creating groups
starting with the prefix “xg-” in AD then assign the people to the right group.
Currently Xsigo has 14 pre-defined roles:
- OPERATOR
/* Read --only Operator */
- OPERATIONS
/* Operations - view and adjust logs */
- FAULT
/* fault management related features */
- STATS
/* Statistics */
- EQUIPMENT
/* Equipment Provisioning */
- NETWORK
/* Network Connectivity Provisioning - allows provisioning
of vNICs and vNIC profiles etc */
- STORAGE
/* Storage Management */
Xsigo Systems VP780
179
-
USER
TRANSPORT_SECURITY
INTERFACE_SECURITY
COMPUTE
QOS
POLICY
ADMIN
/*
/*
/*
/*
/*
/*
/*
User Management */
SSL */
ACL */
COMPUTE */
QOS */
Policy Management */
Administrator - I am almighty */
Xsigo recommends you create all 14 groups at the beginning (prior to XgOS configuration). When you create groups by
using the interface provided by AD, remember to use Global for Group scope and Distribution for Group type as
illustrated by the picture below. These new groups all start with “xg-”, such as xg-operator, xg-operations, xg-fault, xginterface_security, etc. Afterwards if you want to assign a QOS and POLICY role to a user (i.e., Kent), simply add Kent to
group xg-qos and xg-policy. You can also assign roles to a whole group. For example if you want to give everyone under
software group the ADMIN role, simply assign the software group to xg-admin group.
Another approach is to create a group containing multiple roles. For example if you know everyone in the organization
requires COMPUTE, USER and STORAGE roles, instead of assigning everyone to these 3 groups, you can just create
one new role group xg-compute-user-storage then assign everyone to it.
Figure 2 AD Configuration
Xsigo Systems VP780
180
CLI Wrapping
When the terminal display output is too wide and unreadable across the screen, the system can capture the output and
display it in vertical mode.
Syntax
set cli wrap [off | on]
show cli wrap
Example
show iocard
--------------------------------------------------slot
1
state
up/up
descr
type
sanFc2Port4GbCard
v-resources 1
acl
enables
--
CLI Display Filters
Display output can be sent through different CLI display filters.
Syntax
show
show
show
show
-list <command>
-sortby <command>
-table <command>
-xml <command>
-list—Output in list format.
-sortby—Column to sort by. It changes the column upon which the table is sorted. Each time a table is
printed, there is a default sort column (or columns) by which it is sorted. This default is chosen to be the most
common.
-table—Output in table format.
-xml—Output in XML format.
Example 1
show -list vnic foo.bar
-------------------------------------------name
foo.bar
state
up
Xsigo Systems VP780
181
CLI Display Filters
mac-addr
ipaddr
descr
if
type
vlans
vm
00:13:97:01:80:06
6/1
none
Example 2
show -xml vnic foo.bar
<table>
<row number="0">
<cell name="name" value="foo.bar"/>
<cell name="state" value="up"/>
<cell name="mac-addr" value="00:13:97:01:80:06"/>
<cell name="ipaddr" value=""/>
<cell name="descr" value=""/>
<cell name="if" value="6/1"/>
<cell name="type" value=""/>
<cell name="vlans" value="none"/>
<cell name="vm" value=""/>
</row>
</table>
Example 3
To sort the vNIC output by the “if” column:
show -sortby=if vnics
To specify multiple columns:
show -sortby=name,if vnics
This command will use “name” as the primary sort and “if” as the secondary.
To perform a reverse sort:
show -sortby=!name,if vnics
Note: This command is one place in the CLI where command completion is not available.
Xsigo Systems VP780
182
Terminal Rows and Columns
The XgOS enables you to set and display the number of rows and columns for the terminal screen.
Syntax
set cli rows <number>
set cli cols <number>
show cli rows
show cli cols
rows—Number of rows on the terminal screen.
cols—Number of columns on the terminal screen.
Example
show cli rows
30
set cli rows 60
show cli rows
60
CLI History
Use the show cli history command to display the history of issued commands. The history log can be searched using
the up/down arrow keys and Ctrl-r command sequence.
Syntax
show cli history
show cli history <number>
where <number> is the number of saved history commands to display. The buffer limit size is 512 commands
per user. The log is persistent across CLI login sessions.
Example 1
show cli history
35
Wed Jul 4
502
Fri Aug 24
503
Fri Aug 24
504
Fri Aug 24
505
Fri Aug 24
506
Fri Aug 24
507
Fri Aug 24
...
01:44:01
01:48:58
17:57:04
18:11:23
18:14:25
18:26:19
18:33:47
GMT
GMT
GMT
GMT
GMT
GMT
GMT
2007
2007
2007
2007
2007
2007
2007
show fabric-port
show hardware
set cli idle-timeout 0
show software
show system
telnet fpp
show history
Xsigo Systems VP780
183
Syslog
Example 2
Step 1
Press Ctrl-r to initiate a history search:
():
Ctrl-c will interrupt the search. Repeated Ctrl-r will display the previous command.
Step 2
Enter the command text string to search on:
(gogo): add server-profile gogo
Step 3
Press the Enter key bring the command to the host prompt:
admin@iowa[xsigo] add server-profile gogo
Syslog
The following commands are available:
set system syslog <address> [-level=<value>]
show system syslog
SNMP
The XgOS supports SNMPv1 and v2. Get, getnext, and getbulk operations are all supported. Set operations are not
supported. Community strings are read only.
Syntax
add snmp trap-dest <ip-addr>[:<port>]
remove snmp trap-dest
set snmp -descr=<description>
set snmp -read-community=<string>
show snmp
The default read-community string is “public”.
Example
set snmp -add-trap-dest=10.1.1.1:162
set snmp -read-community=private
set snmp -descr="Xsigo Iowa"
show snmp
read-community
descr
trap-destinations
-----------------------------------------------------------------------------private
Xsigo Iowa
10.1.1.1:162
1 record displayed
Xsigo Systems VP780
184
MIB Support
The following MIBs are supported:
•
SNMP MIB-2 RFC-1213 system group and SNMP group
•
FC-MGMT-MIB
•
IF-INVERTED-STACK-MIB
•
IF-MIB
•
XSIGO-ALARM-MIB
•
Xsigo Enterprise MIB identifier is 24440
IF-INVERTED-STACK-MIB
Only the following tables return valid values for SNMP queries:
ifInvStackTable—For each VNIC/VHBA virtual resource there is an entry in the returning value. LowerLayer is
the type of service, either VNIC (512) or VHBA(768). The value of HigherLayer is the VID (Virtual Resource
Identification) of the instance.
IF-MIB
ifXTable—All stats related entries return value 0. Others show related VNIC/VHBA/IOPort information
ifStackTable—This is the reverse display of lowerLayer and higherLayer of ifInvStackTable
ifTable—All stats related entries return value 0. Others show related VNIC/VHBA/IOPort information
FC-MGMT-MIB
fcmInstanceTable—Use this table to get all information related to VHBAs. Also relate the index to the Xsigo
alarms generated when VHBA is created/removed. fcmInstanceStatus.index always returns value ‘ok’--bug 7085.
fcmPortTable—Use this table to get all information related to Fibre Channel IO ports on IO card. fcmPortWwn
and fcmPortNodeWwn have no return values—bug 7082.
XSIGO-ALARM-MIB
All tables are supported.
xgAlarmTable—Each row entry represents an active alarm in the system. Co-relate the
xgAlarmOptAffectedIfIndex with IF-MIB. Also xgAlarmAffectedName gives details what the alarm is about
which resource on Xsigo chassis.
xgFaultTypeTable—Although values are returned, this is only a laundry list of definitions of different fault
conditions. This table is intended as basis of future development.
xgFaultCauseTable—Same purpose as last xgFaultTypeTable.
xgAlarmTraps—Trap is sent when row entry is created/modified/destroyed.
active(1)—indicates that the conceptual row with all columns is available for use by the managed device.
Xsigo Systems VP780
185
Alarms
notInService(2)—indicates that the conceptual row exists in the agent, but is unavailable for use by the managed
device.
notReady(3)—indicates that the conceptual row exists in the agent, one or more required columns in the row are
not instantiated.
createAndGo(4)—supplied by a manager wishing to create a new instance of a conceptual row and make it
available for use.
createAndWait(5)—supplied by a manager wishing to create a new instance of a conceptual row but not making
it available for use.
destroy(6)—supplied by a manager wishing to delete all the instances associated with an existing conceptual row.
The way to interpret the trap is always use the last trap and use ifIndex to co-relate event with the V*. When it is deleted,
ifIndex is 0 but the xgAlarmName is indication of which virtual server and virtual resource is deleted.
Traps
All supported SNMP trap definitions are available in the XSIGO-ALARM MIB.
For sending out SNMP traps, you need to inform the chassis where to forward the traps:
set snmp -add-trap-dest=192.168.100.10:162
Note: Replace 192.168.100.10 with the IP address of the system where you are going to receive SNMP traps.
As of Release 1.0, the following activities on the chassis will generate an SNMP trap. Trap index is co-related with IFMIB ifIndex.
1.
Add/Remove VNIC/VHBA/VSSL
2.
IOCard down/up
3.
IOPort down/up
4.
Virtual server add/remove
5.
Power supply
Trap ID is not sequential since it is using vid. Removed V* will leave vid not sequential. A chassis reboot will create traps
for all events. Check the last event for related resource.
Alarms
Issue show alarms to display alarms in the system database.
Syntax
show alarms
Example
show alarms
time
type name
severity cause
descr
------------------------------------------------------------------------------
Xsigo Systems VP780
186
2007-08-16 22:09:54.439 server vserver1
physical
warning
terminationUnspecified no
compute
resource
provisioned.
NTP
The following commands are available:
set system ntp-server <address> [-prefer]
show system ntp-server
Diagnostics
Issue show diagnostics to display diagnostic system information:
Syntax
show
show
show
show
show
show
show
show
show
show
show
show
show
show
show
show
show
show
diagnostics
diagnostics
diagnostics
diagnostics
diagnostics
diagnostics
diagnostics
diagnostics
diagnostics
diagnostics
diagnostics
diagnostics
diagnostics
diagnostics
diagnostics
diagnostics
diagnostics
diagnostics
ib-topo
iop
ofed
opensm-param
physcon
sm-info
switch-version
vstar
xcm-rec
xcm-stats
xcm-sum
xds-info
xskt_stats
xsm-sum
xsmp-ctrl
xsmp-session
xsmp-stats
xsmp-sum
Example 1
The formatting of this next example is not correct due to a page-width limitation of this manual. However, notice the
excellent troubleshooting information. The “Vinstall” column indicates when a vstar has been successfully installed
(“Inst_ok”) on the host:
show diagnostics vstar
Name
Type
Global Id
Server Name
TCA Guid
Lid
Slot Adm
Oper
Vinstall
Xinstall
--------------------------------------------------------------------------------hood_sboot
VHBA 0x3970310103000012 hood.lab.xsigo.com 0x13970301000390 54
15
Up
Up
Inst_ok
Boot Info:
Xsigo Systems VP780
187
Root Login
eth12
61
3
Up
teton_sboot
54
15
Up
pn_1_1_1
48
1
Up
wwn : 0x5006016830230384,
lun:
0x0
Mount Info:
LVM Mount Group
: VolGroup00,
Volume : 0
VNIC
0x3970310102000725 localhost.localdomain 0x139703010000ba
Down
Inst_ok
Boot Info: bootable
VHBA 0x397031010300001a localhost.localdomain 0x13970301000390
Up
Inst_ok
VNIC 0x39703101020005b0 belford.lab.xsigo.com 0x13970301000164
Up
Inst_ok
When a vstar is not installed, the field “Inst_pending” is displayed.
Example 2
show diagnostics ib-topo
IB Subnet Topology: 3 HCAs, 4 TCAs 4 switches
Type
Name
Node Guid
Port Port Guid
Lid OperState
portName
-------------------------------------------------------------------------------HCA
ceasar 2C90200204CB0
2
2C90200204CB2
13 ACTIVE
ServerPort 24
HCA
alexander 2C90200204CBC
1
2C90200204CBD
8
ACTIVE
ServerPort 8
HCA
iowa xsigo SCP 139702010000A0 1
139702010000A1 1
ACTIVE
SCP port
TCA
Xsigo TCA:13970301 1397030100007E 1 1397030100007F 9 ACTIVE iopPort 8
TCA
Xsigo TCA:13970301 139703010000E9 1
139703010000EA 10 ACTIVE
Switch MT47396 Infiniscal
B8CFFFF00440C
1
B8CFFFF00440C
12
DOWN
...
Root Login
To log into the system as root, then su admin back into the user CLI:
-bash-3.00$ ssh root@iowa
Password:
iowa:~#
iowa:~# su admin
Welcome to XgOS
Copyright (c) 2007-2008 Xsigo Systems, Inc. All rights reserved.
admin@iowa[xsigo]
10GE I/O Modules
The 10GE I/O card can be installed in any slot on the chassis.
Supported Features
•
128 vNICs per card
•
Card-level High Availability (HA)
•
Access Control List (flow) policing
•
QoS on the vNICs configured on the card
Xsigo Systems VP780
188
•
MTU sizes from 1500 bytes to 9194 Kbytes
•
IPv4 TCP/UDP checksum offload
•
Untagged VLANs. Each vNIC can be assigned to a single untagged VLAN (between 1 - 4000)
•
8 traffic queues per vNIC
•
IGMP snooping. IGMP versions supported: v1, v2, v3 (partially supported)
•
Flow learning and statistics
•
512 multicast groups
•
802.1p, TOS, and DSCP mapping
Monitoring 10G
Use show ioport to inspect the state and configuration information on the card.
The following example displays a card installed in slot 8:
show ioport 8/1
-------------------------------------name
8/1
type
nwEthernet10GbPort
state
up/up
descr
speed
10000
rate
autoNegotiate
mtu
1500
avail-in-cir
0K
avail-out-cir 0K
mode
access
vnics
8
vlans
none
-------------------------------------1 record displayed
show iocard 8
-------------------------------------slot
8
state
up/up
descr
type
v-resources 8
acl
enables
--
Xsigo Systems VP780
Linux Host Software
189
Introduction
Xsigo created a host-software stack to be installed on supported GNU/Linux servers. The software is a collection of
Xsigo-specific kernel modules, daemons, hostdrivers, and other tools.
The following table provides a brief summary of the installed component modules and services:
Table 1 Installed Components and Services
Component
Description
xsigod
The Xsigo daemon. Performs user level configuration
periodically on the host on behalf of the Xsigo chassis.
vnic
The vNIC module. Provides virtual NIC services.
vhba
The vHBA module. Provides virtual HBA services.
kxsigod
The kxsigod module.
xsigoib
A simplified IB interface to all the Xsigo drivers.
ib_mthca
The InfiniBand Mellanox Technologies HCA module.
xcpm
The xcpm module. The session manager to the chassis that
obtains all the V* object information and forwards it to the
hostdrivers:
Supported Linux Distributions
The Xsigo VP780 supports host software for the following Linux distributions and processor architectures (i386-32, x8664):
•
RHEL4 update 3
•
RHEL4 update 4
•
RHEL4 update 5
•
RHEL5
•
RHEL5 XEN
•
SLES10
•
SLES10 XEN
This host software can be installed from the CD that accompanied the system, or from the Customer Support Portal (see
http://www.xsigo.com/support).
The CD has the following directory structure:
/xgos
/drivers/linux/redhat
/drivers/linux/utils
/drivers/linux/src
/drivers/linux/sles
/drivers/vmware
/drivers/windows
Xsigo Systems VP780
190
Chapter 17: Linux Host Software
The Linux host software files follow this naming convention:
xsigo-hostdrivers-kmod-<kernel-ver><patch-level>.<xsigo-build>.<bit-size>.rpm
Examples:
xsigo-hostdrivers-kmod-2.6.9_42.EL_1.5.0_RC1-1.i386.rpm
xsigo-hostdrivers-kmod-2.6.9_42.EL_1.5.0_RC1-1.x86_64.rpm
xsigo-hostdrivers-kmod-2.6.9_42.ELsmp_1.5.0_RC1-1.i386.rpm
xsigo-hostdrivers-kmod-2.6.9_42.ELsmp_1.5.0_RC1-1.x86_64.rpm
xsigo-hostdrivers-kmod-2.6.18_8.el5_1.5.0_RC1-1.i386.rpm
xsigo-hostdrivers-kmod-2.6.18_8.el5_1.5.0_RC1-1.x86_64.rpm
xsigo-hostdrivers-kmod-2.6.18_8.el5xen_1.5.0_RC1-1.i386.rpm
xsigo-hostdrivers-kmod-2.6.18_8.el5xen_1.5.0_RC1-1.x86_64.rpm
In these examples, the string “2.6.9_42” is for RHEL4 update 4 whereas “2.6.18_8” is for RHEL5. The 32-bit files end
with a “i386.rpm” suffix. The 64-bit files end with a “.x86_64.rpm” suffix.
Installed Linux Version
Depending on your platform, use the following commands to determine the installed Linux kernel, patch level (update
version), and OS version running on your server:
#
#
#
#
uname -a
cat /etc/redhat-release
cat /etc/SuSE-release
cat /etc/issue
Example 1:
The host is using Linux kernel 2.6.18-8.el5; the OS version is Red Hat Enterprise Linux Server Release 5:
# uname -a
Linux bona 2.6.18-8.el5 #1 SMP Fri Jan 26 14:15:21 EST 2007 i686 athlon i386
GNU/Linux
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5 (Tikanga)
Example 2:
The host is using Linux kernel 2.6.16.21-0.8; the OS version is SUSE Linux Enterprise Server 10:
# uname -a
Linux fremont 2.6.16.21-0.8-smp #1 SMP Mon Jul 3 18:25:39 UTC 2006 i686 i686
i386 GNU/Linux
# cat /etc/SuSE-release
SUSE Linux Enterprise Server 10 (i586)
VERSION = 10
If you require a newer Linux distribution, contact your Linux provider.
Xsigo Systems VP780
191
Installation Procedure
Installation Procedure
Take the following steps to install the Linux host software:
Step 1
Boot the server.
Step 2
Locate the Xsigo host software you want to copy.
Step 3
Install the host software using the command rpm -ivh <rpm-filename>. You must have root
permissions to perform an installation:
rpm -ivh xsigo-hostdrivers-kmod-2.6.9_42.ELsmp_1.5.0_RC1-1.i386.rpm
Preparing...
######################################### [100%]
1:xsigo-hostdrivers-kmod ###################################### [100%]
As of Release 1.5, only one version of the host software can be installed at a time. You cannot have one
installed for each matching kernel package.
If driver conflicts are encountered (i.e., the system declares the new drivers are “older” than the currentlyinstalled drivers), try using -Uvh:
rpm -Uvh <rpm-filename>
If problems persist, list any previous drivers you may have installed and uninstall them manually as shown
in this next example. In general, you should have the same version of all RPMs installed on the system:
rpm -qa | grep xsigo
xsigo-hca-firmware-1.5.0_RC1-1
xsigo-hostdrivers-kmod-2.6.9_42.ELsmp_trunk_2279-1
rpm –e xsigo-hostdrivers-kmod-2.6.9_42.ELsmp_trunk_2279-1
See man rpm for more information.
Step 4
Enable the new drivers to be loaded on reboot:
chkconfig xsigo on
Nothing prints to the terminal (normal behavior) This step enables the /etc/init.d/xsigo script to be invoked
on each boot of the system (for runlevels 3-5).
Step 5
Verify the Xsigo software is installed and enabled:
rpm -q xsigo-hostdrivers-kmod
xsigo-hostdrivers-kmod-2.6.9_42.ELsmp_1.5.0_RC1-1
chkconfig --list | grep xsigo
xsigo
0:off
1:off
2:on
3:on
4:on
5:on
6:off
If the drivers were loaded correctly, you will see “xsigo” and “on(s)” in the display.
You can also issue rpm -qV xsigo-hostdrivers-kmod to validate the package install.
Step 6
Reboot the server:
reboot
Note: In prior releases, Xsigo used service xsigo reloadall to start the Xsigo modules and unload
any old stock files (i.e, ib_mthca) that were still resident in kernel memory. However once the server
mounted a vNIC or vHBA file system, or started using an application such as multi-pathing software,
a service xsigo reloadall was unable to reload the busy drivers and thus generated many error
Xsigo Systems VP780
192
messages. As of Release 1.5, the commands service xsigo reloadall and service xsigo
start are deprecated.
Step 7
Confirm xsigod is running and verify the modules were loaded successfully into the kernel:
# service xsigo status
xsigod is running
Module vnic is loaded
Module vhba is loaded
Module kxsigod is loaded
Module ib_mthca is loaded
Module xcpm is loaded
Module xsigoib is loaded
Xsigod performs user level configuration on the host on behalf of the Xsigo chassis. This configuration
includes setting IP addresses (static or DHCP), VLANs, etc.
Step 8
View the module dependencies:
# lsmod | egrep 'vnic|vssl|xsigo'
kxsigod
12184 1
vssl
143208 0
vnic
64056 0
xcpm
37052 4 kxsigod,vhba,vssl,vnic
xsigoib
43348 4 vhba,vssl,vnic,xcpm
ib_sa
16981 5 xcpm,xsigoib,rdma_cm,ib_local_sa,ib_ipoib
ib_cm
38317 3 xsigoib,rdma_cm,ib_ucm
ib_mad
39129 6 xsigoib,ib_mthca,ib_local_sa,ib_umad,ib_sa,ib_cm
ib_core
49601 12 vssl,xsigoib,ib_mthca,rdma_cm,ib_local_sa,
Note: These dependencies are subject to change by Xsigo without notice.
Step 9
Verify the Infiniband port link connected to the Xsigo chassis is ACTIVE. Xsigo uses the Infiniband
port as transport to the server:
# cat /sys/class/infiniband/*/ports/*/state
1: DOWN
4: ACTIVE
In this example, port 1 has a state of down (1) and port 2 has a state of active (4). The IB port status can be
any of the following: 1 is down, 2 is init, 3 is armed, 4 is active, and 5 is active_defer.
Xsigo Systems VP780
193
HCA Firmware and Option ROM Upgrades
There are many types of Host Channel Adapter (HCA) cards, and each has its own firmware version. In most cases, the
firmware version is different for single port vs dual-port cards. By default, the host HCA’s firmware and Option ROM are
not upgraded during a hostdriver RPM install. See the Xsigo release notes for a list of supported card types and minimum
firmware versions.
Xsigo provides an HCA firmware RPM named “xsigo-hca-firmware-xxxx.rpm” (not to confuse with the hostdriver RPM)
where “xxxx” is the version string.
Xsigo provides a script called xg_config:
/opt/xsigo/bin/xg_config
The xg_config script has many uses:
•
To flash an HCA with new firmware and Option ROM.
•
To check the current running HCA version and Option ROM version.
•
To use scriptable options (non-interactive mode). This mode is useful for customers automating their
firmware upgrade. See xg_config --help for more information.
A server reboot is required after the HCA firmware is upgraded.
Note
Install Considerations
Consider the following issues when installing and programming multiple bootable HCAs:
•
Memory for loading the Option ROM from multiple PCI devices is limited.
•
Different vendors and devices consume different amounts of Option ROM space.
•
The following adapters are contributing to memory depletion: VGA adapter, SAS or RAID controller, NIC(s)
and HCA(s).
•
There is a possibility that BIOS will report an error on certain server models indicating inability to load the
Option ROM of some devices.
•
Consider using dual port HCA(s) for High Availability configuration instead of two single port HCA(s).
If enabling two single port HCA(s) is still preferred, it will be required to disable the Option ROM on some
of the other devices or devices themselves. Please contact Xsigo Customer Support to request best practice
information for a particular server model in terms of enabling two bootable HCA(s).
Procedure
Take these steps to upgrade the HCA firmware:
Step 1
Install the RPM “xsigo-hca-firmware-xxxx.rpm” (not to confuse with the hostdriver RPM):
rpm -ivh xsigo-hca-firmware-1.5.0_RC1-1.i386.rpm
Preparing...
######################################## [100%]
1:xsigo-hca-firmware
####################################### [100%]
Step 2
Run the HCA xg_config install script:
Xsigo Systems VP780
194
sudo /opt/xsigo/bin/xg_config
Follow the installation script and make the appropriate choices for your setup.
Step 3
Reboot the system. After an upgrade, a reboot of the system is required for the upgrade to take effect.
The following examples are provided next:
•
Example 1: Flash HCA Firmware on page 194
•
Example 2: Option ROM Upgrade on page 196
•
Example 3: Other Useful Commands on page 197
Example 1: Flash HCA Firmware
#> sudo /opt/xsigo/bin/xg_config
##############################################################################
# (C) 2007 XSIGO SYSTEMS Inc. All rights reserved. This material may not be
# reproduced, displayed, modified or distributed without the express prior
# written permission of the copyright holder.
##############################################################################
##############################################################################
# Main menu
##############################################################################
Selected card:
Node GUID
Board ID
CA type
Firmware version
Hardware version
Option ROM version
:
:
:
:
:
:
'0002:c902:0020:d5ac'
'MT_0230010002'
'MT25204'
'1.1.0'
'a0'
'XgBoot Version 0.2'
1) Flash HCA Firmware
2) Flash HCA Firmware + Option ROM
3) Flash Option ROM
4) Change selected card
0) Quit
Select option> 1
##############################################################################
# Flash HCA Firmware Menu
##############################################################################
Selected card:
Node GUID
Board ID
CA type
Firmware version
Hardware version
Option ROM version
:
:
:
:
:
:
'0002:c902:0020:d5ac'
'MT_0230010002'
'MT25204'
'1.1.0'
'a0'
1) 1.2.0
Xsigo Systems VP780
195
2) 1.1.0
0) Return to previous menu
Select firmware to use> 1
WARNING: Using this firmware will blank the existing Xg Option ROM.
Continue to flash device? (y/n) y
Upgrading HCA firmware, please do not interrupt the burn process or reboot the
machine ...
Current FW version on flash:
New FW version:
1.1.0
1.2.0
Burn image with the following GUIDs:
Node:
0002c9020020d5ac
Port1:
0002c9020020d5ad
Sys.Image: 0002c9020020d5af
Read and verify Invariant Sector
Read and verify PPS/SPS on flash
Repairing: Copy primary image to secondary
Burning first
FW image without signatures
Restoring first
signature
- OK
- OK
OK
- OK
- OK
The firmware on one or more of the HCAs has been upgraded.
It is recommended to reboot the machine in order for changes to take effect.
Press a key to continue
##############################################################################
# Main menu
##############################################################################
Selected card:
Node GUID
Board ID
CA type
Firmware version
Hardware version
Option ROM version
:
:
:
:
:
:
'0002:c902:0020:d5ac'
'MT_0230010002'
'MT25204'
'1.1.0'
'a0'
3) Flash Option ROM
0) Quit
Select option> 0
Exit...
#>
Xsigo Systems VP780
196
Example 2: Option ROM Upgrade
#> sudo /opt/xsigo/bin/xg_config
##############################################################################
# (C) 2007 XSIGO SYSTEMS Inc. All rights reserved. This material may not be
# reproduced, displayed, modified or distributed without the express prior
# written permission of the copyright holder.
##############################################################################
##############################################################################
# Main menu
##############################################################################
Selected card:
Node GUID
Board ID
CA type
Firmware version
Hardware version
Option ROM version
:
:
:
:
:
:
'0002:c902:0020:d5ac'
'MT_0230010002'
'MT25204'
'1.2.0'
'a0'
'unknown'
3) Flash Option ROM
0) Quit
Select option> 2
##############################################################################
# Flash HCA Firmware + Option ROM Menu
##############################################################################
Selected card:
Node GUID
Board ID
CA type
Firmware version
Hardware version
Option ROM version
:
:
:
:
:
:
'0002:c902:0020:d5ac'
'MT_0230010002'
'MT25204'
'1.2.0'
'a0'
'unknown'
1) 1.2.0 (XgBoot Version 0.2)
2) 1.1.0 (XgBoot Version 0.2)
0) Return to previous menu
Select firmware to use> 1
Continue to flash device? (y/n) y
Upgrading HCA firmware, please do not interrupt the burn process or reboot the
machine ...
Current FW version on flash:
New FW version:
1.1.0
1.2.0
Burn image with the following GUIDs:
Xsigo Systems VP780
197
Node:
0002c9020020d5ac
Port1:
0002c9020020d5ad
Sys.Image: 0002c9020020d5af
Read and verify Invariant Sector
Read and verify PPS/SPS on flash
Burning second
FW image without signatures
Restoring second
signature
-
OK
OK
OK
OK
The firmware on one or more of the HCAs has been upgraded.
It is recommended to reboot the machine in order for changes to take effect.
Press a key to continue
##############################################################################
# Main menu
##############################################################################
Selected card:
Node GUID
Board ID
CA type
Firmware version
Hardware version
Option ROM version
:
:
:
:
:
:
'0002:c902:0020:d5ac'
'MT_0230010002'
'MT25204'
'1.2.0'
'a0'
'unknown'
3) Flash Option ROM
0) Quit
Select option> 0
Exit...
Example 3: Other Useful Commands
Additional commands are available to check the HCA card and firmware versions:
cat /sys/class/infiniband/mthcaXX/hca_type
cat /sys/class/infiniband/mthcaXX/fw_ver
where XX is 0 or 1. The XX indicates a card number. For example, 0 means the first card. The following two examples
using mthca0, are on the first card.
To check the card type:
cat /sys/class/infiniband/mthca0/hca_type
MT25204
In this example, the MT25204 is a single port PCIe card.
To check the firmware version:
cat /sys/class/infiniband/mthca0/fw_ver
Xsigo Systems VP780
198
1.1.0
In this example, the MT25204 has firmware version 1.1.0 installed.
vHBA Support
Environments that use the Xsigo vHBA might need to have additional storage support packages installed. Specifically,
Linux vHBA users should be prepared to install the following rpm packages, in order to ensure proper functionality of
vHBA in both Xen and multipath storage environments:
•
sg3_utils
•
sg3_utils-libs
•
sg3_utils-devel
•
device-mapper-multipath
•
device-mapper
Debug Tool: xsigo-support
Xsigo created a debug tool called xsigo-support. This script collects and dumps information for monitoring and
troubleshooting your host-software installation. Xsigo engineers collect this information for debugging.
The script resides on the host server at this location:
/opt/xsigo/bin/xsigo-support
Use the --help suffix to display the keyword options:
./xsigo-support --help
Xsigo Linux host config and debug tool
Possible arguments:
-d, --showdebug
-s, --setdebug <0|1|2>
-o, --outputfile <filename>
-h, --help
show current debug loglevels
set the debug loglevels
optional file name for --dumpstate
print usage
Example:
cd /opt/xsigo/bin
./xsigo-support > dbg.output
vi dbg.output
Below are comments about the contents of the output file.
Notice the Xsigo-specific proc files stored in different directory locations. These files collect important driver-state
information:
/proc/driver/<module-name>/<input-filename> -->
The host server’s “heartbeat” to the chassis is described by the xcpm module. It is the session manager to the chassis:
The xcpm module controls the host server’s “heartbeat” to the chassis. Xcpm is the session manager to the chassis that
obtains all the V* object information and forwards it to the host drivers:
Xsigo Systems VP780
199
/proc/driver/xcpm/links/0 -->
State:
Hello interval (secs):
Session timeout (secs):
Number of session timeouts:
Hello recv count:
Hello send count:
Up
20
60
0
9352
9352
Datapath timeout (secs):
7200
Handle :
Local port:
Local lid:
Local guid:
Remote lid:
Remote guid:
0
1
8
0x2c90200204cbd
1
0x139702010000a1
There is robust vNIC and vHBA state information:
/proc/driver/vnic/devices/myvinc -->
Slot:
0
Admin state:
Up
HA Status:
Active
Config parameters:
guid:
0x139703010001a2
lid:
0xb
MAC addr:
0x139701800b
VID:
0x3970180102000007
mtu:
1500
bandwidth:
100
...
/proc/driver/vhba/devices/myvhba -->
VHBA Information
---------------Symbolic Name
: myvhba
Port WWN
: 0x500139708102
Host Number
: 2
VHBA flags
: 0
VHBA state
: VHBA_STATE_ACTIVE
Target count
: 1
Link State
: 1
Scan Required
: 0
...
Useful Infiniband information (GUID, LID, port state, ...) is dumped in /sys/class:
/sys/class/infiniband/mthca0/board_id -->
MT_0150000001
/sys/class/infiniband/mthca0/hca_type -->
MT25208
/sys/class/infiniband/mthca0/fw_ver -->
5.1.400
...
The output of lsmod displays a listing of installed modules:
Xsigo Systems VP780
200
Output of 'lsmod' -->
Module
Size Used by
kxsigod
12184 0
vhba
156260 0
vssl
143208 0
vnic
63928 0
xcpm
36928 4 kxsigod,vhba,vssl,vnic
xsigoib
43352 4 vhba,vssl,vnic,xcpm
ib_mthca
134184 0
ib_uverbs
43241 0
ib_umad
19697 0
ib_ucm
22085 0
ib_sa
16981 2 xcpm,xsigoib
ib_cm
38317 2 xsigoib,ib_ucm
ib_mad
39129 5 xsigoib,ib_mthca,ib_umad,ib_sa,ib_cm
ib_core
49601 9
vssl,xsigoib,ib_mthca,ib_uverbs,ib_umad,ib_ucm,ib_sa,ib_cm,ib_mad
...
See the dmesg kernel log showing the buffer of information being dumped by different drivers:
Output of 'dmesg' -->
hba>
vhba_ib_disconnect_qp: Disconnecting CQP 1
<vhba myvhba>
vhba_ib_disconnect_qp: Disconnecting DQP 2
flushing active ios for vhba 0
flushed 0 ios for vhba 0
cqp connect res id 100000301187039 slid=0x800 dlid=0xa00
dguid=0xea00000103971300
<vhba myvhba>
vhba_ib_connect_qp: CQP handle is 1
...
All the current proceses are displayed:
Output of 'ps aux' -->
USER
PID %CPU %MEM
root
1 0.0 0.0
root
2 0.0 0.0
root
3 0.0 0.0
root
4 0.0 0.0
root
5 0.0 0.0
root
6 0.0 0.0
...
VSZ
2488
0
0
0
0
0
RSS
576
0
0
0
0
0
TTY
?
?
?
?
?
?
STAT
S
S
SN
S
SN
S
START
Oct12
Oct12
Oct12
Oct12
Oct12
Oct12
TIME
0:00
0:00
0:00
0:00
0:00
0:00
COMMAND
init [5]
[migration/0]
[ksoftirqd/0]
[migration/1]
[ksoftirqd/1]
[migration/2]
Network device information:
Output of 'ifconfig -a' -->
eth0
Link encap:Ethernet HWaddr 00:13:72:55:5C:49
inet addr:192.168.10.175 Bcast:192.168.11.255 Mask:255.255.252.0
inet6 addr: fe80::213:72ff:fe55:5c49/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5388109 errors:6 dropped:0 overruns:0 frame:3
TX packets:6390 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:629584782 (600.4 MiB) TX bytes:656819 (641.4 KiB)
Base address:0xccc0 Memory:fe4e0000-fe500000
...
Xsigo Systems VP780
201
CPU and memory information:
Output of 'cat /proc/cpuinfo' -->
processor
: 0
vendor_id
: GenuineIntel
cpu family
: 15
model
: 4
model name
: Intel(R) Xeon(TM) CPU 2.80GHz
...
Output of 'cat /proc/meminfo' -->
MemTotal:
2074736 kB
MemFree:
778228 kB
Buffers:
215536 kB
Cached:
712404 kB
SwapCached:
0 kB
Active:
511400 kB
...
Currently loaded Xsigo hostdriver and its version:
Output of 'rpm -qi xsigo-hostdrivers-kmod' -->
Name
: xsigo-hostdrivers-kmod
Relocations: (not relocatable)
Version
: 2.6.9_42.ELsmp_trunk_2279
Vendor: (none)
Release
: 1
Build Date: Fri 12 Oct 2007 12:11:36 PM PDT
Install Date: Fri 12 Oct 2007 05:36:08 PM PDT
Build Host: xlbuild06.xsigo.com
Group
: System Environment/Kernel
Source RPM: xsigo-hostdrivers-kmod2.6.9_42.ELsmp_trunk_2279-1.src.rpm
Size
: 6668498
License: GPL/BSD
...
Xsigo service modules loaded and running:
Output
xsigod
Module
Module
Module
Module
Module
Module
of 'service xsigo status' -->
is not running
vnic is loaded
vhba is loaded
kxsigod is loaded
ib_mthca is loaded
xcpm is loaded
xsigoib is loaded
Xsigo Systems VP780
202
Xsigo Systems VP780
Source RPM: Building Xsigo Host Drivers
203
Introduction
Xsigo Systems provides source RPM Package Managers (RPMs) for advanced users and developers to help support a
wide array of Linux distributions. There are numerous requirements that must be satisfied in order to both compile and
produce a compatible driver. The utmost of care should be taken when preparing a driver from the available source, and
careful documentation should be kept in order to assist Xsigo Systems Support in understanding your environment.
Background: Xsigo distributes two types of hostdriver RPMs—binary and source. Binary RPMs are compiled for a
specific kernel and system architecture. Source RPMs contain the source code for building the binary package. Xsigo’s
hostdrivers are kernel modules. Since it is impossible for Xsigo to directly support every version of Linux distribution
(kernel and architecture), Xsigo provides its hostdrivers as source RPMs. You compile these kernel modules against
specific kernel distributions then install them as binary RPMs.
Compatibility
The source RPM has been compiled and tested with the following base Linux distributions or base kernels:
•
Redhat Enterprise Linux 4, Update 3
•
Redhat Enterprise Linux 5, Update 0
•
SUSE Enterprise Linux, version 10
•
Generic kernels starting at 2.6.11 thru 2.6.18
Optionally, Xsigo Systems has tested and shown compatibility with updated InfiniBand (IB) drivers based on
OpenFabrics Enterprise Distribution (OFED)-1.1, and OFED-1.2.X.
Xsigo has tested its drivers against x86 and x86_64 architectures only.
Xsigo is constantly updating its compatibility matrix to follow Open Fabrics, Kernel.org, and various Linux distributions.
If you need support for a platform or distribution that is not one of the listed kernels or architectures, please contact your
sales or support engineer for further information.
For the latest OFED release and install information, go to http://www.openfabrics.org
Note
Prerequisites
In addition to selecting a compatible base kernel, other requirements must be met. You should understand the origin of
each of the following requirements. Some of the requirements include a base C compiler, base C Library (libc), kernel
development headers, kernel symbol-files, kernel config (.config), additional patches, updates, and fixes. In some cases,
the Xsigo hostdrivers require updates or fixes in your base kernel, dependent drivers, or related tools/compilers.
One example of both updated features and fixes is the ib_mthca.ko from pre-OFED-1.2.
Users looking to build a driver on their system should consult the target distribution’s documentation on building drivers
to insure that they have installed all the necessary prerequisites of the target distribution.
Please also read thru the Source RPM Release Notes for an explanation of known issues, workarounds and other common
suggestions.
Xsigo Systems VP780
204
Chapter 18: Source RPM: Building Xsigo Host Drivers
SRC RPM File
Xsigo provides one generic source RPM for all supported kernel distributions:
xsigo-hostdrivers-kmod-<build>.src.rpm
The RPM itself is not specific to every supported Linux installation.
Basic rpmbuild Example
Using a basic example and all default values, the driver can be built as the root user on a Redhat Enterprise Linux 5
System:
root@rhel5# rpmbuild -–rebuild xsigo-hostdrivers-kmod-linux_1.5.0-1.src.rpm
<…extensive output…>
Wrote: /usr/src/redhat/RPMS/x86_64/xsigo-hostdrivers-kmod-2.6.18-53.el5_1.5.01.x86_64.rpm
Wrote: /usr/src/redhat/RPMS/x86_64/xsigo-hostdrivers-kmod-debuginfo- 2.6.1853.el5_1.5.0-1.x86_64.rpm
Note that two RPM files are built. The file containing the –debuginfo contains some of the debugging information for use
with a debugger such as gdb. The other file contains the drivers, management, and startup scripts.
Then install the binary RPM:
#
#
#
#
rpm –Uvh xsigo-hostdrivers-kmod-2.6.18-53.el5_1.5.0-1.x86_64.rpm
chkconfig xsigo on
reboot
service xsigo status
The SPEC File
Often, a user will find it necessary to customize some aspect of the driver build process. Many of these behaviors are set
through default environment variables, SPEC files at the top of the rpm-SPEC file, or through system scripts.
To make these customizations, you should first install the RPM source:
root@system# rpm -i xsigo-hostdrivers-kmod-linux_1.5.0-1.src.rpm
The source files will be installed at the appropriate location as configured in your RPM program. In Redhat, this location
prefix is /usr/src/redhat, while on SLES, this location is /usr/src/packages.
Inside this prefix directory, you will find several other directories including BUILD, RPMS, SOURCE, SPECS, and
SRPMS. In the SPECS directory, you will find a file named /usr/src/redhat/SPECS/xsigo-hostdrivers.spec. You will find
several SPEC variables that have initial values, and others dynamically set via scripts. You should consult the spec file for
specific documentation. See also the following table.
Xsigo Systems VP780
205
The SPEC File
Table 1 Spec File Variable Descriptions and Values
Automatically
Checked
Default
Value
Acceptable
Values
The Xsigo host drivers by default are written to
compile against the OFED 1.1 and earlier API.
By enabling this option, the drivers will be patched
appropriately to enable compiling against the OFED
1.2.X distribution given the slight differences in the
API. By default, this will automatically be enabled if
an OFED 1.2.X installation is found as part of the
kernel.
Yes
0
0 or 1
infer_ib_devel_he
aders
This option allows you to build the Xsigo host
drivers against updated OFED installations which
are not part of the kernel source tree. If multiple
OFED distributions are installed, then the kversion
environment variable will be used. By default, this
will automatically be checked and set accordingly if
an OFED installation is found outside the kernel
source tree.
Yes
0
0 or 1
fixup_module_sy
mvers
Enable this option if you are building against an
OFED installation which is installed outside the
kernel source tree. This option is needed for kernels
prior to 2.6.18 which supported finding the
Module.symvers file in the top level of kernel source
directory first. By default, there is no check done for
this so this option must be specified by the user
before building the binary RPM.
No
0
0 or 1
mthca_fix
Enable this if you would like to use the work around
for the rdb_per_qp issue in the ib_mthca.ko kernel
module. Otherwise, no updated ib_mthca.ko kernel
module will be built. Only certain kernel versions
support this since it requires the previously patched
ib_mthca kernel module source code to be in the
source RPM package. By default, this will be
enabled if patched ib_mthca kernl module sources
for the appropriate running kernel are found in the
source RPM.
Yes
0
0 or 1
fmr
Enable this option if you would like to use the
updated Fast Memory Registration (FMR) API.
Currently, only needed on the RHEL4u5 2.6.9-55
kernel. If the 2.6.9-55 kernel is found, this option
will be enabled.
Yes
0
0 or 1
Spec File Variable
Description
ofed1_2
Xsigo Systems VP780
206
Environment Variables
When building the drivers, you might need to override some default locations and values. These values are set through
environment variables.
Table 2 Variables to Set
Variable
Description
kversion
This environment variable can be set to specify the kernel version you would like
to build the Xsigo host drivers for. The default value for this is the kernel you are
currently running with (e.g. uname -r).
ksrc
This environment variable can be set to point to the directory of where the kernel
development headers and symbol files are located. The default directory is based
on where the kernel headers are for your running kernel (e.g. /lib/modules/
${kversion}/build).
XSIGOFLAGS
This environment variable can be set to specify additional flags to the compiler
such as additional include paths and build parameters. Typically used to specify
the additional include paths for OFED installations which are not part of the
kernel source tree (e.g. export XSIGOFLAGS=" -I/usr/src/ofa_kernel/include ").
Note that XSIGOFLAGS is automatically set thru one of the external scripts
when OFED is installed.
There are several build options:
•
Build Option 1: Stock Kernels
•
Build Option 2: Custom Kernels
•
Build Option 3: Kernel with Upgraded OFED Stack
•
Build Option 4: Combination of Customer Kernel and Upgraded OFED Stack
Build Option 1: Stock Kernels
•
Tested environments: RHEL4, RHEL5, SLES10
•
Dependencies: kernel-devel RPM
In this scenario, all of your kernel source and devel-headers/objects should be located inside the path /lib/modules/ùname
-r`/build. This symbolic link is the default location for the xsigo-hostdriver src-rpm to look for the kernel source directory.
Command sequence procedure:
# rpm -ivh xsigo-hostdrivers-kmod-linux_<#version>-1.src.rpm
# rpmbuild -bb /usr/src/redhat/SPECS/xsigo-hostdrivers.spec
Xsigo Systems VP780
207
•
Tested Environments: 2.6.16, 2.6.18.1 (mainline)
•
Dependencies: Complete compiled kernel tree
When compiling your own kernel and drivers, you will need to retain both the kernel source tree and some of the binary
files. Often, when you install your kernel, it will make the symbolic link /lib/modules/ùname -r`/build. If this is not the
case, you will need to export the location of the kernel prior to running rpmbuild.
# export ksrc=/root/linux-2.6.18.1
This procedure will override the default kernel location.
Build Option 3: Kernel with Upgraded OFED Stack
•
Tested Environments: 2.6.16.21 + OFED-1.2, 2.6.16.21 + OFED-1.1, RHEL4 + OFED-1.1
•
Dependencies: Compiled kernel source trees and updated OFED headers
Replacing the InfiniBand driver stack with an updated OFED stack can and likely will result in API changes for the
drivers. It is likely that you will need to modify the existing native InfiniBand calls to conform to the current headers.
In order to have kbuild look in the proper location for the InfiniBand stack, you will need to set the environment variable
XSIGOFLAGS. This modifies the search path when kbuild is compiling to look for the header files before looking in the
default kernel source directory.
# export XSIGOFLAGS=" -I /usr/src/ofa_kernel-1.2.5.1/include "
A suggestion to find the proper include path is to find the “include/rdma” directory in your build tree:
# find /root/ofed-1.2.5 -name rdma
/root/ofed-1.2.5/include/rdma
In this scenario, you want to set XSIGOFLAGS to this:
# export XSIGOFLAGS=” -I /root/ofed-1.2.5/include “
Xsigo Systems VP780
208
Build Option 4: Combination of Customer Kernel and
Upgraded OFED Stack
Often, users will have both a custom kernel and an upgraded OFED stack.
The key here is to make sure the following requirements are met:
1.
The symbolic link /lib/modules/ùname -r`/build correctly points to the kernel source tree.
Alternately, you can override the default kernel tree location by setting the ksrc environment variable.
2.
Set the XSIGOFLAGS environment variable to the appropriate path for the correct OFED header path.
3.
Make sure you work out the workqueues and C syntax (typically set by kernel version) and that the headers/API
match the IB-API of the Xsigo drivers. Some combinations are included with patches.
#
#
#
#
rpm -ivh xsigo-hostdrivers-kmod-linux_<#version>-1.src.rpm
export ksrc='/root/linux-2.6.18.1'
export kversion='2.6.18.1' (This value often matches uname –r)
rpmbuild -bb /usr/src/redhat/SPECS/xsigo-hostdrivers.spec
Non-RPM Builds
While Xsigo intends their drivers to be installed on a system which leverages the RPM (Redhat Package Manager), it is
still possible for advanced users to extract the source code and build each driver manually.
When you do this, you should also take care to include the appropriate xsigod userland configuration application and
startup scripts.
Here is a command sequence to build the 1.5 drivers manually from the src-RPM file:
#
#
#
#
#
#
#
#
#
#
rpm2cpio xsigo-hostdrivers-kmod-linux_1.5.0-1.src.rpm | cpio -iud
tar xzvf xsigo_branch_1.5.0.tar.gz
make -C/lib/modules/ùname -r`/build M=`pwd`/ksrc/xsigoib
make -C/lib/modules/ùname -r`/build M=`pwd`/ksrc/xcpm
make -C/lib/modules/ùname -r`/build M=`pwd`/ksrc/vnic
make -C/lib/modules/ùname -r`/build M=`pwd`/ksrc/vhba
mkdir –p /lib/modules/ùname –r`/updates/kernel/drivers/InfiniBand/ulp
cp ksrc/*/*.ko /lib/modules/ùname –r`/updates/kernel/ drivers/InfiniBand/ulp
depmod –a
cp scripts/xsigo /etc/init.d/xsigo
<You will still need to active the init.d script>
# make -C apps/xsigod
# cp apps/xsigod/xsigod /usr/bin/xsigod
Xsigo Systems VP780
209
OFED Patch Files
OFED Patch Files
The patch program takes a patch file containing a difference listing produced by the diff program and applies those
differences to one or more original files, producing patched versions.
Xsigo uses two patches:
1.
xsigo-linux-2.6.9-55.patch is used to handle a change in ib_fmr_pool_map_phys API in xsigoib/
xsigoib.c
2.
ofed-1.2.patch is used to handle changes in OFED 1.2 as compared with Xsigo’s source code base and
affects a number of files.
The patches are normally invoked as part of Xsigo’s spec file. If they need to be manually applied, invoke the patch
program.
Example:
patch <<ofed-1.2.patch>
Note the first < is part of the command and the <> denotes the real file name.
Release Notes
•
RHEL4 kernel-devel packages do not include all the requisite InfiniBand headers. Xsigo has include the
missing headers in the source-RPM file, which can be extracted and added to the compiler include path
through the XSIGOFLAGS variable. Or, you can copy them manually:
cp /usr/src/redhat/SOURCES/rhel4_headers.tar
cd /usr/src/kernels/<kernel>/
tar xvf rhel4_headers.tar
tar zxvf <kernel>.tgz
•
/usr/src/kernels/<kernel>/
If running on against a OFED-1.2.5.X IB stack, the following kernel log message (dmesg) is benign:
ib_cm: req timeout_ms 16896 > 8192, decreasing
ib_cm: req remote_cm_response_timeout 22 > 21, decreasing
ib_cm: req local_cm_response_timeout 22 > 21, decreasing It can be eliminated
by setting max_timeout ib_cm module parameter to 23.
Customer Support
Before contacting Xsigo customer support, gather the following information about how you are using/building the drivers:
•
The base kernel origin (is it a SLES/RHEL/kernel.org, compilers, the .config, etc)
•
Any modifications to the Xsigo drivers and specs.
•
A brief description of the build process you are using.
•
Any custom hardware, firmware, or custom loading of the drivers.
•
Any SAN-boot or related configuration/initrd/ information (how did you install the image to SAN, etc)
Xsigo Systems VP780
210
Xsigo Systems VP780
Glossary
211
Table 1 lists the terms used in this document and other Xsigo documentation. The definitions provided here should only
be used in the context of the Xsigo VP780 and the Xsigo Operating System (XgOS).
Table 1 Terms and Definitions
Term
Definition
Active
Directory
Active Directory (AD) is an implementation of LDAP directory services by Microsoft for use
primarily in Windows environments. Its main purpose is to provide central authentication and
authorization services for Windows based computers. Active Directory also allows administrators to
assign policies, deploy software, and apply critical updates to an organization.
CPIO
Copy Input Output. A binary file archiver and a file format. CPIO’s use by the RPM Package
Manager continues to make CPIO an important archive format. See man cpio.
Domain
A collection of virtual resources and physical resources (if using groups and pools) that are
associated with some type of access-controlled group. (i.e. HR, Finance, Engineering, or Web
Servers, Application Servers, dB Servers). Currently, Domains do not include I/O cards, I/O ports or
I/B ports.
Domain
Group
Contains a list of Domains. One Domain Group is assigned to each user. The user can write to only
the domains listed in the Domain Group but can read from any domain. In future releases, the user
will only be able to read from and write to the domains in their Domain Group.
FC
Fibre Channel. The American National Standards Institute (ANSI) began work on FC in 1988, and
since then the X3T11 Task Group (see www.t11.org) has developed 20+ standards. FC has its own
stack of protocol levels (layers), ranging from the physical connectors and media (FC-0) to upperlevel protocols (FC-4). Each of these levels defines a different and separate part of the how FC
equipment communicates. The different FC-4 protocols (FCP, IP, Virtual Interface, and others) are
tied directly to different kinds of applications (storage, networking, and clustering) for different
uses. For more background information, see www.fibrechannel.org
Group
A set of server profiles, created from pool members of the same pool, associated with a template.
Multiple groups may share the same pool. Once a server is assigned to a group, no other group can
use that server. A pool member (server profile) is assigned to a group by “growing” the group. A
server profile is removed from a group by “shrinking” the group, or by removing a specific server
profile (e.g. as the result of a failure).
HA vNICs
High Availability vNIC - A pair of virtual Ethernet interfaces, that are both assigned to the same
server profile, but bound to different physical interfaces.
HBA
Host Bus Adaptor. A Fibre Channel network interface card used in a SAN fabric. FC HBAs are
replacing SCSI HBAs.
HCA
Host Channel Adapter. An InfiniBand network interface card used in an InfiniBand network.
An HCA provides high-speed connectivity and virtual interfaces, based on the InfiniBand interface.
An HCA can have 1 or 2 ports.
hypervisor
A hypervisor is a virtualization platform that allows multiple guest operating systems to run at the
second level above the hardware. Xen is an example of a para virtualizing hypervisor—a Virtual
Machine Monitor (VMM). Xen can host multiple guest operating systems, each of which is
executed within a secure VM (in Xen terminology, a domain).
Xsigo Systems VP780
212
Glossary
Table 1 Terms and Definitions (continued)
Term
Definition
IB
InfiniBand. A switched fabric communications link primarily used in high-performance computing.
IB is the result of merging two competing designs, Future I/O, developed by Compaq, IBM, and
Hewlett-Packard, with Next Generation I/O (ngio), developed by Intel, Microsoft, and Sun. For
more information, see www.infinibandta.org
IDE
Integrated Drive Electronics. Throughout the 1980’s, a standard interface for connecting hosts to
direct-attached storage devices. Parallel SCSI was another approach.
I/O
Input/Output. In computer architecture, the combination of the CPU and main memory (i.e. memory
that the CPU can read and write to directly, with individual instructions) is considered the heart of a
computer. Any movement of information from or to that complex, for example to or from a disk
drive, is considered I/O.
I/O Module
A physical card that is installed in one of 15 slots in the chassis’ card bay. There are three types of
I/O module: Ethernet, Host Bus Adapter and SSL. The Ethernet and Host Bus Adapter modules
provide access to Ethernet and Fibre Channel networks, respectively. The SSL module provides
secure Ethernet communications.
I/O Port
A single port on an Ethernet module, a Host Bus Adapter module, or one of the 24 InfiniBand
server ports.
JBOD
Just A Bunch of Disks. Very large storage arrays, capable of storing terabytes and terabytes of data.
Farms of JBODs connect through an FC SAN. In a JBOD each disk is visible to the SAN, assigned
an address, and is treated as an autonomous device even though the physical disks are located in the
same enclosure.
jitter
For QoS the delta between packets on the receive side. Low jitter is guaranteed by having a lowlatency queue mechanism. In this way, a flow is guaranteed service and packets are not held up
(delayed) in buffers.
Kerberos
Kerberos is a network authentication protocol. It is designed to provide strong authentication for
client/server applications by using secret-key cryptography. Kerberos was developed in the Athena
Project at the Massachusetts Institute of Technology (MIT). The name is taken from Greek
mythology; Kerberos was a three-headed dog who guarded the gates of Hades. Kerberos lets a user
request an encrypted “ticket” from an authentication process that can then be used to request a
particular service from a server.
LDAP
The Lightweight Directory Access Protocol (LDAP) is an application protocol for querying and
modifying directory services running over TCP/IP. A client starts an LDAP session by connecting
to an LDAP server, by default on TCP port 389. The client then sends operation requests to the
server, and the server sends responses in turn.
Managed
Object
An object-oriented representation of a resource managed in a device. This can be a physical or
logical resource. One flow always get serviced.
NAS
Network Attached Storage. NAS use common client networks, such as Ethernet, to connect client
computers to a host-file server. Unlike SANs, the client does not directly communicate with the
storage. Data exchange occurs at the file level, unlike a SAN where data is operated at the block
level over FC.
OFED
OpenFabrics Enterprise Distribution. OFED is the driver stack for the InfiniBand Host Channel
Adaptor (HCA). For more information, see http://www.openfabrics.org/resources.htm
Xsigo Systems VP780
213
Glossary
Term
Definition
OpenSM
The default Subnet Manager running on the Xsigo VP780.
Phys-Con
Physical Connection - The InfiniBand port, on the VP780, that is physically attached to a particular
server.
Policy
Configuration of automatic system behavior (e.g. stats collection, dB cleanup, etc.).
Pool
A set of pool members. (For example, a “low_end_server” pool may contain single-CPU x86
servers with 512MB to 1GB of RAM.) A pool may be shared by multiple groups.
Pool
Member
A pool member is a physical server consisting of one or more phys-cons, that is assigned to a single
pool by the user, based on similar properties (e.g. CPU type, CPU #, CPU speed, Memory Size).
Pool members have assignable strings that can define the properties of the physical server. All
physical servers connected to a Xsigo system are, by default, assigned to the “default pool” under
the “root domain”.
Quality of
Service
The Quality of Service (QoS) object allows the data traffic of individual applications or interfaces
to be managed. The performance of a particular application can be guaranteed by raising the priority
of its dataflow, relative to the other applications.
RADIUS
Remote Authentication Dial In User Service (RADIUS) is an Authentication, Authorization, and
Accounting (AAA) protocol for controlling access to network resources. RADIUS is commonly
used by ISPs and corporations managing access to Internet or internal networks across an array of
access technologies including modem, DSL, wireless, and VPNs.
RAID
Redundant Array of Inexpensive Disks.
RDMA
Remote Direct Memory Access. One of the key problems with server I/O is the CPU overhead
associated with data movement between memory and I/O devices, such as LAN and SAN
interfaces. InfiniBand solves this problem by using RDMA to offload data movement from the
server CPU to the InfiniBand HCA. Using RDMA, the sending device either reads data from or
writes data to the target devices' user space memory, thereby avoiding CPU interrupts and multiple
data copies on the memory bus. This approach enables RDMA to significantly reduce the CPU
overhead associated with data movement between nodes.
Role
One of 14 fixed-privilege levels that a user may be assigned (e.g. Stats, Operators, Administrators,
Fault (see alarms), Equipment, Security, QoS, etc.).
Role Group
A list of Roles. Roles define the user’s read/write privileges. Only one Role Group is assigned to
each user. The user can write to only the objects controlled by the roles listed in the role group but
can read all other objects.
Root Domain
The root domain (domain-root) is the default top-level management domain. Other domains, and all
new managed objects, are created under the Root Domain (See “Domain”).
SAN
Fibre Channel Storage Area Network. A SAN is a network of storage and system components, all
communicating on a fibre-channel network, that can be used to consolidate and share storage,
provide high-performance links to data devices, add redundant links to storage systems, speed up
data backup, and support high-availability clustering systems. The advent of SANs has been driven
by today’s insatiable appetite for storage. See www.snia.org for more background information.
Xsigo Systems VP780
214
Glossary
Term
Definition
SCSI
Small Computer Systems Interface. In the early 1980’s, SCSI was the standard direct-attach storage
interface to SCSI-enabled disks. As computer systems increased in speed and data storage needs
increased, the parallel bus architecture of SCSI began hitting performance and distance limits. In
response to this need, FC was introduced to provide gigabit-speed serial networking capabilities for
storage.
Server
Profile
One instance of a server I/O configuration that is assignable to a single physical server via an IB
port (phys-con).
Sub-domain
A collection of managed objects under a common management domain. The default domain is
“domain-root”. Up to four levels of sub-domains can be created under the root domain.
Subdomains provide a way of partitioning the management of datacenter resources. Typically, each
domain would be owned and managed by different administrators.
System
An object that identifies a given system at the highest (“top”) level in the object model
hierarchy.
Template
A predefined configuration that can be applied to a server profile when the server profile is created.
A server profile can be an instance of a template.
User
An internal or external representation of a person. Users either exist locally or remotely via LDAP,
Active Directory, or Radius. By default, an “admin” user is created locally.
vHBA
Virtual Host Bus Adapter - A Fibre Channel Storage connection, provided without a physical HBA.
vLAN
Virtual Local Area Network - A private, independent, logical networks that are created within a
physical network. A vLAN behaves like an ordinary LAN, but connected devices don't have to be
physically connected to the same network segment.
VM
Virtual Machine. A VM is a software entity that runs its own operating systems and applications, as
if it were a physical computer. A VM behaves exactly like a physical computer and contains its own
virtual (software based) CPU, RAM, hard disk, and NIC. An operating systems installed on a VM
is called a guest operating system.
vNIC
Virtual Network Interface Card - An Ethernet interface, provided without a physical NIC.
vSSL
Virtual Secure Socket Layer link - An SSL connection, provided without an SSL appliance or
proxy.
WWNN
World Wide Node Name
WWPN
World Wide Port Name
Xsigo Systems VP780
Index
215
Symbols
B
/bin 6
/config 10
/home 6
/sbin 6
/skins 6
/usb 6
Baud rate 4
BIOS 126
boot-capable 150
bootdebug 140
bootmenu 140
A
AAA 171
access mode 65
accounts 167
ACLs 119
ACLs with QoS 109
action 121
AD 171
add acl 120
add domain-group 170
add gateway 50
add ims 172
add net-term-group 160
add qos network 106
add qos san 116
add san boot 128
add san map 70
add server-profile 47
add snmp 183
add template 48
add user 167
add vhba 68
add vlan 64
add vm 96
add vnic 53
add vssl 87
add-trap-dest 185
administravite state 61
alarms 185
Anaconda 130
auto calculation, for QoS 106
auto switchover 60
auto-switchover 60
C
CBS 102
CHAP 172
chkconfig xsigo on 191
CIR 102
commit 120
config.xml 11
configuration save and restore 38
console 4
conventions, document i
Copy Input Output 137
core dumps 10
crash dump 10
custom sets for QoS 106
D
Data bits 4
default gateway 50
default sets for QoS 104
dhcp 53
DiffServ 112
display filters 180
dmesg 9
domain group 170
DomU 95
DoS 119
DSCP 112
E
egress-qos 106
enqueue 122
environmentals 35
ESX 90
Xsigo Systems VP780
216
Index
esxcfg-vmhbadevs 89
esxcfg-vswitch 89
esxcfg-xgmap 89
execution-throttle 77
F
Fibre Channel 67
file archive 7
file compress 7
file copy 7
file diff 7
file edit 7
file find 7
file hash 7
file list 7
file remove 7
file search 7
file show 7
file system 6
file unarchive 7
file uncompress 7
Flow control 4
fstab 134
ftp 7
full 96
G
gateway 50
GRUB 125
GUID 29
H
ha 56
HA, vNICs 55
HCA, firmware and options ROM upgrades 193
help 46
High Availability 55
history statistics 44
hostdrivers 189
host-software stack 189
http 7
https 7
hypervisor 95
I
I/O cards 33
I/O ports 34
ib_mthca 189
IBA 29
Identity Management System 171
if-state 37
IMS 171
Infiniband 27, 67
ingress-qos 106
initial RAM disk 137
initiator 67
initrd 137
interfaces 37
interrupt-delay-timer 78
IOP 165
ip-addr 53
K
Kerberos 172
kernel arguments 139
ksrc 206
kversion 206
kxsigod 189
L
LAGs 159
LDAP 171
Linux hostdrivers 189
Load 127
Loadmount 127
Local ID 89
local-id 89
log into the CLI 4
logging 8
loop-reset-delay 78
LUN 73
LUNs per target 73
luns-per-target 73
LVM 127, 129
Xsigo Systems VP780
217
Index
M
MAC addresses 54
mark 122
MIBs 184
MIMM 166
Mount 127
MTU 63
N
netwait 139
network QoS 101
new features and enhancements 1
NFS 140
nfsboot 140
nfsmountopts 141
no-lun-masking 85
NPIV 67
NTLoader 125
NTP 186
ntp-server 186
O
OFED 203
OFED patch files 209
online help 46
OpenSM decoupling 157
Option ROM 125
P
PAP 172
para 96
Parity 4
PBS 103
persistent binding 70
phys-con 47
PIR 102
plugin for VMware 92
policing 101
ports 34
prescan 74
Xsigo Systems VP780
priority 62
processors 165
PXE boot 145
Q
QoS, for vHBAs 115
QoS, for vNICs 101
quit 167
R
RADIUS 171
real time statistics 44
recovery CLI 46
remove acl 121
remove ims 172
remove net-term-group 160
remove san boot 129
remove san map 70
remove server-profile 47
remove snmp 183
remove template 48
remove user 167
remove vm 96
remove vnic 53
remove-prescan 74
rescan 74
resolv.conf 50
resourceUnavailable 69
role group 168
root login 187
route 50
RPM drivers 191
rpmbuild 204
RSCN 74
rule 121
S
SAN 67
SAN Boot 125
SAN QoS 115
san-boot 128
sanboot 139
218
Index
sanwait 139
scp 7
server profile 47
server profiles 47
service xsigo status 192
set acl 111, 120
set cli cols 182
set cli rows 182
set cli wrap 180
set domain-group 170
set ethernet-port 63
set fc-port 77
set gateway 50
set ims 172
set net-term-group 160
set qos san 116
set san boot 129
set server-profile 47
set snmp 183
set stats-policy 44
set system syslog 183
set template 48
set user 167
set vlan 64
set vm 96
set vnic 53
set vswitch 95, 96
sg_map 135
shaping 115
show alarms 41, 185
show cli 182
show cli history 182
show cli wrap 180
show ethernet-port 63
show fc-port 80
show gateway 50
show hardware 35
show ims 172
show iocard 33
show ioport 34
show -list 180
show login 5
show net-term-group 160
show physical-server 28
show qos network 104
show qos san 116
show san boot 129
show san map 70
show server-profile 47
show snmp 183
show -sortby 180
show stats-policies 44
show system 6, 41
show system syslog 183
show -table 180
show template 48
show users 5
show vhba 68
show vlan 64
show vm 96
show vnic 53
show vssl 87
show -xml 180
sigo-initrd 130
SNMP 183
source RPM 203
SPEC file 204
SSH 4
static 53
statistics 44
statistics, for vNICs 55
stats-policy 44
Stop bit 4
Subnet Manager 30
subnets 50
syslog 183
system export 38
system flush ims 172
system import 38
system upgrade 163
T
tape-support 78
targets 67
Telnet 4
template 48
termination group 159
traps 185
trunk mode 64
U
unassigned 48
untagged-vlan-id 64
user accounts 167
Xsigo Systems VP780
219
Index
V
vHBA 67
virtual Host Bus Adapter 67
Virtual I/O Fabric 151
Virtual Machine 95
virtual Network Interface Card 53
virtualServer 96
VLANs 64
VMware 89
vNIC 53
vNIC counters and statistics. 55
vNIC priority 62
vNICs
HA 55
statistics 54
vSSLs 87
vswitches 95
xsigo-support 198
Y
YaST 130
W
watch 41
WRED 101
WWNN 67
WWPN 67
X
xcpm 189
XDSD 151
xds-info 156
xds-param 156
Xen 95
xenbrN 95
xg_config 193
xg-insert-dd 142
XgOS 3
xgxen 96
XMS plugin for VMware 92
XPF file 3
xsigo-boot 130, 137
xsigod 189
xsigo-dud 130
XSIGOFLAGS 206
xsigoib 189
xsigo-rhdd 130
Xsigo Systems VP780

XgOS CLI and Linux Host Software Guide

Transcription

Similar documents

Xsigo Presentation

1. Basic Concepts

#104 300 Klahanie Drive • Port Moody

The Suez Canal Area Development Project

Installing And Configuring Windows Deployment

Greater Houston Port Bureau, Capt. Bill Diehl

(041) 360 4491 Fax :0866873881 Email: [email protected]

- Pacific Disaster Net

MADE FOR WALKING

PDF First - Datalogics