Low Latency Tunings

LOW LATENCY SETTINGS:
1. TCP/UDP buffers ( we may already have standard settings for these )
net.ipv4.tcp_rmem
net.ipv4.tcp_wmem
net.ipv4.tcp_mem
net.core.wmem_max
net.core.rmem_max
net.core.wmem_default
net.core.rmem_default
eg.
net.ipv4.tcp_rmem =” 4096 16777216 33554432”
net.ipv4.tcp_wmem =” 4096 16777216 33554432”
net.ipv4.tcp_mem = “4096 16777216 33554432”
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.netdev_max_backlog = 30000
net.core.netdev_max_backlog = 30000
net.ipv4.tcp_low_latency=1
(optimize tcp stack for low latency over high throughput. Faster connection times, works better for smaller pkts)
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_ecn=0
(disable router explicit congestion notification, will help when we see large file transfer stalls in a small env with 1/2 routers not a good)
net.ipv4.tcp_app_win=0
(do not reserve space or handle differently for slower apps. Larger the value smaller the buffer for specific window)
net.ipv4.tcp_no_metrics_save=1
(tcp saving various connection metrics in route cache when connections are closed. These caches are consulted for future similar connections.
Will increase performance on the market access side or when we do have similar connections)
rhash_entries=1024
2. Initial congesstion window for the routes (default is 3. Can send initially 3 pkts without acks. Maybe required on the market access side ..
Order sending might be delayed or bundled)
Change it with:
ip route change default via n.n.n.n dev eth0 initcwnd #"
(here I did for the default route but can be done for any routes )
To check the value:
"ip route show"
3. Disable C-states, C1E State ( if exists ), HT, power mgmt from BIOS.
4. Disable SMIs ( when supported/agreed by HW vendors ) thru BIOS. Not all vendors support this by default … Will minimize jitters greatly.
5. Disable/remove irqbalance, cpuspeed and other not needed services
6. SFC ( Solarflare )
lro=0
( large receive offload, will not bundle recv pkts into larger chunks)
lso=0
( large send offload, will not bundle tx pkts into larger chunks )
rx_alloc_method=1 ( how memory is allocated for received pkts by the sfc driver. Value of 1 is better for smaller pkts. this is what we
have )
rss_cores=#cores
sfc_affinity
(create per core queues, helpful if dropping pkts consistently and also observing delays in receiving pkts in the application )
(to affinitize or steer pkt streams to cores/connections, Cannot use with onload simultaneously)
There are kernel parameters at the OS level to steer pkts as well. Which I have not tested.
7. Netdev buffers
net.core.netdev_max_backlog= # (higher than the default 2500 , for 10g should be much higher )
net.core.netdev_budget=300
(Depends. Default of 300 is usually ok. Max number of pkts taken from interfaces in each polling cycle)
(I have written a simple script to monitor the netdev buffers will send it soon )
8. Ring buffers, interrupt coalescing, moderation
adjust the ring buffers accordingly ( SFC might have limitation of 4096 for both RX and TX .. Other cards have different values )
no interrupt coalescing or moderation for any interfaces
9. Bind device ( nic, raid , disk, usb .. etc ) interrupts to certain cores
This depends on the application as per binding what to which cores .. We must achieve data locality for the application
See which cores are in which sockets.. Depending on the vendors with the same Intel/AMD processors numbers will vary
"numactl --hardware" will show for sure
10. ONLOAD (If and when used)
EF_POLL_USEC=1000
(onload spins at user space for 1ms. If not used, it'll block. Better to stay in the user space rather in kernel space)
Can check with:
onload_stackdump | grep poll
(will greatly increase the performance as it'll avoid interrupts and kernel CS. By default onload uses epoll() when it goes into polling mode )
EF_RXQ_SIZE
(need to match the ring buffer RX setting )
EF_TCP_INITIAL_CWND
(may need to adjust this value depending on the initial congestion window set mentioned above in this email )
There are other settings which all depends on how the application is performing using onload.
“onload_stackdump” is a good starting point
Onload can only be used with SFC cards, other cards have TOE which needs different settings
11. Using cpuset, tuna , isolcpus, taskset
It's a good practice to use isolcpus to isolate cores from general scheduling. I believe we are already doing this here cpuset, tuna
both almost do similar things. So whatever the appdev/unix teams are comfortable with.
I personally like tuna better as once I know what thread is doing what I can bind them to certain cores properly in order to receive
data locality. To do this we need to understand the apps better. Is the app running on a busy loop or not? Personally I prefer to bind
the thread to a certain core and let it be running there all the time. Using cpusets will have the thread migrate to cores in the same
set which sometimes may not be desirable. Setting thread priorities/class might also be a good idea. But all of these depend on the
application behavior.
12. Events kernel thread
When we see the application is observing delays in receiving pkts/messages or there's a longer than usual time spent from the time
the pkt came to the NIC and the message delivered to the application, we may need to bump up the priority of the kernel event
thread. These threads process pending events in the kernel generated by device drivers …This would increase the epoll(), poll(),
select() calls which I believe most of the NIC vendors use. When using ONLY onload this would not matter much
13. MRG 1.3
A whole new ball game and MUST be carefully planned when used. Maybe soon in a wholesale level :-) There are different things
that need to be done for this.
UDP PACKET LOSS:
1.
Poorly sized UDP receive (rx) buffer sizes / UDP buffer socket overflows
1.
Description: Oracle RAC Global cache block processing is ‘bursty’ in nature and, consequently, the OS may
need to buffer receive(rx) packets while waiting for CPU. Unavailable buffer space may lead to silent packet
loss and global cache block loss. `netstat –s` or `netstat –su` on most UNIX will help determine
UDPInOverflows, packet receive errors, dropped frames, or packets dropped due to buffer full errors.
Action: Packet loss is often attributed to inadequate( rx) UDP buffer sizing on the recipient server, resulting in
buffer overflows and global cache block loss. The UDP receive (rx) buffer size for a socket is set to 128k when
Oracle opens the socket when the OS setting is less than 128k. If the OS setting is larger than 128k Oracle
respects the value and leaves it unchanged. The UDP receive buffer size will automatically increase according
to database block sizes greater than 8k, but will not increase beyond the OS dependent limit. UDP buffer
overflows, packet loss and lost blocks may be observed in environments where there are excessive timeouts
on ‘global cache cr requests’ due to inadequate buffer setting when DB_FILE_MULTIBLOCK_READ_COUNT is
greater than 4. To alleviate this problem, increase the UDP buffer size and decrease the
DB_FILE_MULTIBLOCK_READ_COUNT for the system or active session.
2.
3.
2.
To determine if you are experiencing UDP socket buffer overflow and packet loss, on most UNIX platforms,
execute
3.
`netstat –s` or `netstat –su` and look for either “udpInOverflows”, “packet receive errors” or “fragments
dropped” depending on the platform.
4.
NOTE: UDP packet loss usually results in increased latencies, decreased bandwidth, increased cpu utilization
(kernel and user), and memory consumption to deal with packet retransmission.
Poor interconnect performance and high cpu utilization. `netstat –s` reports packet reassembly failures
1.
Description: Large UDP datagrams may be fragmented and sent in multiple frames based on Medium
Transmission Unit (MTU) size. These fragmented packets need to be reassembled on the receiving node. High
CPU utilization (sustained or frequent spikes), inadequate reassembly buffers and UDP buffer space can cause
packet reassembly failures. `netstat -s` reports a large number of Internet Protocol (IP) ‘reassembles failed’
and ‘fragments dropped after timeout’ in the “IP Statistics” section of the output on the receiving node.
Fragmented packets have a time-to-live for reassembly. Packets that are not reassembled are dropped and
requested again. Fragments that arrive and there is no space for reassembly are silently dropped
2.
`netstat –a` IP stat counters:
3104582 fragments dropped after timeout
34550600 reassemblies required
8961342 packets reassembled ok
3104582 packet reassembles failed.
3.
Action: Increase fragment reassembly buffers, allocating more space for reassembly. Increase the time to
reassemble packet fragment., increase udp receive buffers to accommodate network processing latencies that
aggravate reassembly failures and identify CPU utilization that negatively impacts network stack processing.
4.
On LINUX:
To modify reassembly buffer space, change the following thresholds:
5.
/proc/sys/net/ipv4/ipfrag_low_thresh (default = 196608)
/proc/sys/net/ipv4/ipfrag_high_thresh (default = 262144)
6.
To modify packet fragment reassembly times, modify:
/proc/sys/net/ipv4/ipfrag_time (default = 30)
7.
see your OS for the equivalent command syntax.
Network packet corruption resulting from UDP checksum errors and/or send (tx) / receive (rx) transmission errors
1.
4.
Mismatched MTU sizes in the communication path
1.
5.
Description: UDP includes a checksum field in the packet header which is read on receipt. Any corruption of
the checksum results in silent dropped packets. Checksum corruptions result in packet retransmissions,
additional cpu overhead for the additional request and latencies in packet processing.
Action: Use tcpdump/snoop/network utilities sniffer utility to capture packet dumps to identify checksum
errors and confirm checksum corruption. Engage sysadmins and network engineers for root cause. Checksum
offloading on NICs have been known to create checksum errors. Consider disabling the NIC checksum
offloading, if configured, and test. On LINUX ethtool –K <IF> rx off tx off disables the checksum offloading.
Description: Mismatched MTU sizes cause ‘packet too big’ failures and silent packet loss resulting in global
cache block loss and excessive packet retransmission requests.
Action: The MTU is the “Maximum Transmission Unit” or frame size configured for the interconnect
interfaces. The default standard for most UNIX is 1500 bytes for Ethernet. MTU definitions should be identical
for all devices in the interconnect communication path. Identify and monitor all devices in the interconnect
communication path. Use large, non-default sized, ICMP probe packets for` ping`, `tracepath` or `traceroute`
to detect mismatched MTUs in the path. Use `ifconfig` or vendor recommended utilities to determine and set
MTU sizes for the server NICS. See Jumbo Frames #12 below. Note: Mismatched MTU sizes for the
interconnect will inhibit nodes joining the cluster in 10g and 11g.
Faulty or poorly seated cables/cards
1.
Description: Faulty network cable connections, the wrong cable, poorly constructed cables, excessive length
and wrong port assignments can result in inferior bit rates, corrupt frames, dropped packets and poor
performance.
Action: CAT 5 grade cables or better should be deployed for interconnect links. All cables should be securely
seated and labeled according to LAN/port and aggregation, if applicable. Cable lengths should conform to
vendor Ethernet specifics.
6.
Interconnect LAN non-dedicated
1.
7.
Lack of Server/Switch Adjacency
1.
8.
Description: Network devices are said to be ‘adjacent’ if they can reach each other with a single hop across a
link layer. Multiple hops add latency and introduce unnecessary complexity and risk when other network
devices are in the communication path.
Action: All GbE server interconnect links should be (OSI) layer 2 direct attach to the switch or switches (if
redundant switches are configured). There should be no intermediary network device, such as a router, in the
interconnect communication path. The unix command `traceroute` will help identify adjacency issues.
IPFILTER configured
1.
9.
Description: Shared public IP traffic and/or shared NAS IP traffic, configured on the interconnect LAN will
result in degraded application performance, network congestion and, in extreme cases, global cache block
loss.
Action: The interconnect/clusterware traffic should be on a dedicated LAN defined by a non-routed subnet.
Interconnect traffic should be isolated to the adjacent switch(es), e.g. interconnect traffic should not extend
beyond the access layer switch(es) to which the links are attached. The interconnect traffic should not be
shared with public or NAS traffic. If Virtual LANs (VLANS) are used, the interconnect should be on a single,
dedicated VLAN mapped to a dedicated, non-routed subnet, which is isolated from public or NAS traffic.
Description:IPFILTER (IPF) is a host based firewall or Network Address Translation (NAT) software package that
has been identified to create problems for interconnect traffic. IPF may contribute to severe application
performance degradation, packet loss and global cache block loss.
Action: disable IPFILTER
Outdated Network driver or NIC firmware
1.
Description: Outdated NIC drivers or firmware have been known to cause problems in packet processing
across the interconnect. Incompatible NIC drivers in inter-node communication may introduce packet
processing latencies, skewed latencies and packet loss.
Action: Server NICs should be the same make/model and have identical performance characteristics on all
nodes and should be symmetrical in slot id. Firmware and NIC drivers should be at the same (latest) rev. for all
server interconnect NICs in the cluster.
10. Proprietary interconnect link transport and network protocol
1.
Description: Non-standard, proprietary protocols, such as LLT, HMP,etc, have proven to be unreliable and
difficult to debug. Miss-configured proprietary protocols have caused application performance degradation,
dropped packets and node outages.
Action: Oracle has standardized on 1GbE UDP as the transport and protocol. This has proven stable, reliable
and performant. Proprietary protocols and substandard transports should be avoided. IP and RDS on
Inifiniband are available and supported for interconnect network deployment and 10GbE has been certified
for some platforms (see OTN for details) – certification in this area is ongoing.
11. Misconfigured bonding/link aggregation
1.
Description: Failure to correctly configure NIC Link Aggregation or Bonding on the servers or failure to
configure aggregation on the adjacent switch for interconnect communication can result in degraded
performance and block loss due to ‘port flapping’, interconnect ports on the switch forming an aggregated link
frequently change ‘UP’/’DOWN’ state.
Action: If using link aggregation on the clustered servers, the ports on the switch should also support and be
configured for link aggregation for the interconnect links. Failure to correctly configure aggregation for
interconnect ports on the switch would result in ‘port flapping’, switch ports randomly dropping, resulting in
packet loss.
Bonding/Aggregation should be correctly configured per driver documentation and tested under load. There
are a number of public domain utilities that help to test and measure link bandwidth and latency performance
(see iperf). OS, network and network driver statistics should be evaluated to determine efficiency of bonding.
12. Misconfigured Jumbo Frames
1.
Description: Misconfigured Jumbo Frames may create mismatched MTU sizes described above.
Action: Jumbo Frames are IEEE non-standard and as a consequence, care should be taken when configuring. A
Jumbo Frame is a frame size around 9000bytes. Frame size may vary depending on network device vendor and
may not be consistent between communicating devices. An identical maximum transport unit (MTU) size
should be configured for all devices in the communication path if the default is not 9000 bytes. All the network
devices , switches/NICS/line cards, in operation must be configured to support the same frame size (MTU
size). Mismatched MTU sizes, where the switch may be configured to be MTU:1500 but the server
interconnect interfaces are configured to be MTU:9000 will lead to packet loss, packet fragmentation and
reassembly errors which cause severe performance degradation and cluster node outages. The IP stats in
`netstat –s` on most platforms will identify frame fragmentation and reassembly errors. The command
`ifconfig -a`, on most platforms, will identify the frame size in use (MTU:1500). See the switch vendor’s
documentation to identify Jumbo Frames support.
13. NIC force full duplex and duplex mode mismatch
1.
Description: Duplex mode mismatch is when two nodes in a communication channel are operating at
half-duplex on one end and full duplex on the other. This may be manually misconfigured duplex modes or,
one end configured manually to be half-duplex while the communication partner is auto negotiate. Duplex
mode mismatch results in severely degraded interconnect communication.
Action: Duplex mode should be set to auto negotiate for all Server NICs in the cluster *and* line cards on the
switch(es) servicing the interconnect links. Gigabit Ethernet standards require auto negotiation set to “on” in
order to operate. Duplex mismatches can cause severe network degradation, collisions and dropped packets.
Auto negotiate duplex modes should be confirmed after every hardware/software upgrade affecting the
network interfaces. Auto negotiate on all interfaces will operate at 1000 full duplex.
14. Flow-control mismatch in the interconnect communication path
1.
Description: Flow control is the situation when a server may be transmitting data faster than a network peer
(or network device in the path) can accept it. The receiving device may send a PAUSE frame requesting the
sender to temporarily stop transmitting.
Action: Flow-control mismatches can result in lost packets and severe interconnect network performance
degradation.
2.
tx flow control should be turned off
rx flow control should be turned on
tx/rx flow control should be turned on for the switch(es)
3.
NOTE: flow control definitions may change after firmware/network driver upgrades. NIC settings should be
verified after any upgrade
15. Packet loss at the OS, NIC or switch layer
1.
Description: Any packet loss as reported by OS, NIC or switch should be thoroughly investigated and resolved.
Packet loss can result in degraded interconnect performance, cpu overhead and network/node outages.
Action: Specific tools will help identify which layer you are experiencing the packet/frame loss
(process/OS/Network/NIC/switch). netstat, ifconfig, ethtool, kstat (depending on the OS) and switch port stats
would be the first diagnostics to evaluate. You may need to use a network sniffer to trace end-to-end packet
communication to help isolate the problem (see public domain tools such as snoop/wireshare/ethereal). Note,
understanding packet loss at the lower layers may be essential to determining root cause. Under sized ring
buffers or receive queues on a network interface are known to cause silent packet loss, e.g. packet loss that is
not reported at any layer. See NIC Driver Issues and Kernel queue lengths below. Engage your systems
administrator and network engineers to determine root cause.
16. NIC Driver/Firmware Configuration
1.
Description: Misconfigured or inadequate default settings for tunable NIC public properties may result in
silent packet loss and increased retransmission requests.
Action: Default factory settings should be satisfactory for the majority of network deployments. However,
there have been issues with some vendor NICs and the nature of interconnect traffic that have required
modifying interrupt coalescence settings and the number of descriptors in the ring buffers associated with the
device. Interrupt coalescence is the CPU interrupt rate for send (tx) and receive (rx) packet processing. The
ring buffers hold rx packets for processing between CPU interrupts. Misconfiguration at this layer often results
in silent packet loss. Diagnostics at this layer require sysadmin and OS/Vendor intervention.
17. NIC send (tx) and receive (rx) queue lengths
1.
Description: Inadequately sized NIC tx/rx queue lengths may silently drop packets when queues are full. This
results in gc block loss, increased packet retransmission and degraded interconnect performance.
Action: As packets move between the kernel network subsystem and the network interface device driver,
send (tx) and receive (rx) queues are implemented to manage packet transport and processing. The size of
these queues are configurable. If these queues are under configured or misconfigured for the amount of
network traffic generated or MTU size configured, full queues will cause overflow and packet loss. Depending
on the driver and quality of statistics gathered for the device, this packet loss may not be easy to detect.
Diagnostics at this layer require sysadmin and OS/Vendor intervention. (c.f. iftxtqueue and
netdev_max_backlog on linux)
18. Limited capacity and over-saturated bandwidth
1.
Description: Oversubscribed network usage will result in interconnect performance degradation and packet
loss.
Action: An interconnect deployment best practice is to know your interconnect usage and bandwidth. This
should be monitored regularly to identify usage trends, transient or constant. Increasing demands on the
interconnect may be attributed to scaling the application or aberrant usage such a bad sql or unexpected
traffic skew. Assess the cause of bandwidth saturation and address it.
19. Over subscribed CPU and scheduling latencies
1.
Description: Sustained high load averages and network stack scheduling latencies can negatively affect
interconnect packet processing and result in interconnect performance degradation, packet loss, gc block loss
and potential node outages.
Action: Scheduling delays when the system is under high CPU utilization can cause delays in network packet
processing. Excessive, sustained latencies will cause severe performance degradation and may cause cluster
node failure. It is critical that sustained elevated CPU utilization be investigated. The `uptime` command will
display load average information on most platforms. Excessive CPU interrupts associated with network stack
processing may be mitigated through NIC interrupt coalescence and/or binding network interrupts to a single
CPU. Please work with NIC vendors for these types of optimizations. Scheduling latencies can result in
reassembly errors. See #2 above.
20. Switch related packet processing problems
1.
Description: Buffer overflows on the switch port, switch congestion and switch misconfiguration such as MTU
size, aggregation and Virtual Land definitions (VLANs) can lead to inefficiencies in packet processing resulting
in performance degradation or cluster node outage.
Action: The Oracle interconnect requires a switched Ethernet network. The switch is a critical component in
end-to-end packet communication of interconnect traffic. As a network device, the switch may be subject to
many factors or conditions that can negatively impact interconnect performance and availability. It is critical
that the switch be monitored for abnormal, packet processing events, temporary or sustained traffic
congestion and efficient throughput. Switch statistics should be evaluated at regular intervals to assess trends
in interconnect traffic and to identify anomalies.
21. QoS which negatively impacts the interconnect packet processing
1.
Description: Quality of Service definitions on the switch which shares interconnect traffic may negatively
impact interconnect network processing resulting in severe performance degradation
Action: If the interconnect is deployed on a shared switch segmented by VLANs, any QoS definitions on the
shared switch should be configured such that prioritization of service does not negatively impact interconnect
packet processing. Any QoS definitions should be evaluated prior to deployment and impact assessed.
22. Spanning tree brownouts during reconvergence.
1.
Description: Ethernet networks use a Spanning Tree Protocol (STP) to ensure a loop-free topology where
there are redundant routes to hosts. An outage of any network device participating in an STP topology is
subject to a reconvergence of the topology which recalculates routes to hosts. If STP is enabled in the LAN and
misconfigured or unoptimized, a network reconvergence event can take up to 1 minute or more to recalculate
(depending on size of network and participating devices). Such latencies can result in interconnect failure and
cluster wide outage.
Action: Many switch vendors provide optimized extensions to STP enabling faster network reconvergence
times. Optimizations such as Rapid Spanning Tree (RSTP), Per-VLAN Spanning Tree (PVST), and Multi-Spanning
Tree (MSTP) should be deployed to avoid a cluster wide outage.
23. sq_max_size inadequate for STREAMS queing
1.
Description: AWR reports high waits for “gc cr block lost” and/or “gc current block lost”. netstat output does
not reveal any packet processing errors. `kstat -p -s ‘*nocanput*` returns non-zero values. nocanput indicates
that queues for streaming messages are full and packets are dropped. Customer is running STREAMS in a RAC
in a Solaris env.
Action: Increasing the udp max buffer space and defining unlimited STREAMS queuing should relieve the
problem and eliminate ‘nocanput’ lost messages. The following are the Solaris commands to make these
changes:
2.
`ndd -set /dev/udp udp_max_buf <NUMERIC VALUE>`
set sq_max_size to 0 (unlimited) in /etc/system. Default = 2
3.
udp_max_buf controls how large send and receive buffers (in bytes) can be for a UDP socket. The default
setting, 262,144 bytes, may be inadequate for STREAMS applications. sq_max_size is the depth of the message
queue.
Troubleshooting InfiniBand connection issues using OFED tools
The Open Fabrics Enterprise Distribution (OFED) package has many debugging tools available as part of the standard release. This article
describes the use of those tools to troubleshoot the hardware and firmware of an InfiniBand fabric deployment.
First, the /sys/class sub-system should be checked to verify that the hardware is up and connected to the InfiniBand fabric. The following
command will show the InfiniBand hardware modules recognized by the system:
ls /sys/class/infiniband
This example will use the module mlx4_0, which is typical for Mellanox ConnectX* series of adapters. If this, or a similar module, is not found,
refer to the documentation that came with the OFED package on starting the OpenIB drivers.
Next, check the state of the InfiniBand port:
cat /sys/class/infiniband/mlx4_0/ports/1/state
This command should return “ACTIVE” if the hardware is initialized, and the subnet manager has found the port and added the port to the
InfiniBand fabric. If this command returns “INIT” the hardware is initialized, but the subnet manager has not added the port to the fabric yet.
If necessary, start the subnet manager:
/etc/init.d/opensmd start
Once the port on the head node is in the “ACTIVE” state, check the state of the InfiniBand port on all the compute nodes to ensure that all of
the Infiniband hardware on the compute nodes has been initialized, and the subnet manager has added all of the compute nodes ports on to
the fabric. This article will use the pdsh tool to run the command on all nodes:
pdsh –a cat /sys/class/infiniband/mlx4_0/ports/1/state
All nodes should report “ACTIVE”. If a node reports it cannot find the file, ensure the OpenIB drivers is loaded on that node. Refer to the
documentation that came with the OFED package on starting the OpenIB drivers.
Once all of the compute nodes report that port 1 is “ACTIVE”, verify the speed on each port using the following commands:
cat /sys/class/infiniband/mlx4_0/ports/1/rate
pdsh –a cat /sys/class/infiniband/mlx4_0/ports/1/rate
This is a good first check for a bad cable or connection. Each port should report the same speed. For example, the output for double data rate
(DDR) InfiniBand cards will be similar to “20 Gb/sec (4X DDR)”.
Once the above basic checks are complete, more in-depth troubleshooting can be performed. The main OFED tool for troubleshooting
performance and connection problems is ibdiagnet. This tool runs multiple tests, as specified on the command line during the run, to detect
errors related to the subnet, bad packets, and bad states. These errors are some of the more common seen during initial setup of Infiniband
fabrics.
Run ibdiagnet with the following command line options:
ibdiagnet –pc –c 1000
The output will be similar to this:
Loading IBDIAGNET from: /usr/lib64/ibdiagnet1.2
-W- Topology file is not specified.
Reports regarding cluster links will use direct routes.
Loading IBDM from: /usr/lib64/ibdm1.2
-W- A few ports of local device are up.
Since port-num was not specified (-p option), port 1 of device 1 will be
used as the local port.
-I- Discovering ... 17 nodes (1 Switches & 16 CA-s) discovered.
-I---------------------------------------------------I- Bad Guids/LIDs Info
-I---------------------------------------------------I- No bad Guids were found
-I---------------------------------------------------I- Links With Logical State = INIT
-I---------------------------------------------------I- No bad Links (with logical state = INIT) were found
-I---------------------------------------------------I- PM Counters Info
-I---------------------------------------------------I- No illegal PM counters values were found
-I---------------------------------------------------I- Fabric Partitions Report (see ibdiagnet.pkey for a full hosts list)
-I---------------------------------------------------I---------------------------------------------------I- IPoIB Subnets Check
-I---------------------------------------------------I- Subnet: IPv4 PKey:0x7fff QKey:0x00000b1b MTU:2048Byte rate:10Gbps SL:0x00
-W- No members found for group
-I---------------------------------------------------I- Bad Links Info
-I- Errors have occurred on the following links
(for errors details, look in log file /tmp/ibdiagnet.log):
-I--------------------------------------------------Link at the end of direct route "1,5"
----------------------------------------------------------------I- Stages Status Report:
STAGE Errors Warnings
Bad GUIDs/LIDs Check 0 0
Link State Active Check 0 0
Performance Counters Report 0 0
Partitions Check 0 0
IPoIB Subnets Check 0 1
Link Errors Check 0 0
Please see /tmp/ibdiagnet.log for complete log
----------------------------------------------------------------I- Done. Run time was 9 seconds.
The warning “No members found for group” can safely be ignored.
In this example, a bad link was found: “Link at the end of direct route “1,5”.” "1,5" refers to the LID numbers associated with the individual
ports. The following commands can be used to identify the LID numbers associated with each port:
cat /sys/class/infiniband/mlx4_0/ports/1/lid
pdsh –a /sys/class/infiniband/mlx4_0/ports/1/lid
This command generates a list of LIDs associated with nodes. In the output of the above command, locate the entries for 0x1 and 0x5. 0x1 is
likely the head node. For errors of this type, reseat or replace the InfiniBand cable connecting the node corresponding to LID 0x5.
Finally, run ibdiagnet once more time to verify there are no errors, and then to check the error state of each port. Each test should pass.
ibdiagnet –pc –c 1000
ibcheckerrors.
KICKSTART FOR DISKS LARGER THAN 2TB:
Some machines we build have large root disk arrays, bigger than 2TB. They exceed the capacity of msdos partition table and require GPT (GUID
Partition Table) to be installed in order to use the whole disk. Only RHEL 6.1 supports it now and it required significant customization of the
kickstart configuration file. Also, it only works on the machines equipped with UEFI Boot Manager. IBM x3650 M3 and x3755 M3 do have it but I
only tested the new setup on x3650 M3
In order to install RHEL 6.1 to a large disk machine, do the following:
o
In Opsware, pick OS Sequence called RBCCM-NY-RHEL-6.1_GPT. It is equivalent to RBCCM-NY-RHEL-6.1 except of GPT support.
o
This will cause regular installation to be executed. Once it is over, machine will reboot. On that reboot, go to the System Configuration and
Boot Management screen (a.k.a. BIOS config) and follow this set of menu options:
Boot Manager | Add Boot Option | Load File (first) | EFI | redhat | grub.efi
o
Provide some short but meaningful name to the new boot option through Input Description option.
o
Once this is done, press Commit Changes.
o
Exit the System Configuration screen, the machine should boot into the newly installed OS.
On a regular, non-Large Disk system, this OS Sequence should work as the regular one, no additional activity is required.
# Kickstart file automatically generated by anaconda.
#version=DEVEL
install
nfs --server=159.55.104.162 --dir=/vol/XRZ0/opsware/linux/redhat/x86_64/v6.1
lang en_US.UTF-8
keyboard us
network --onboot no --device eth0 --noipv4 --noipv6
network --onboot no --device eth1 --noipv4 --noipv6
network --onboot yes --device eth2 --mtu=1500 --bootproto dhcp
network --onboot no --device eth3 --noipv4 --noipv6
network --onboot no --device usb0 --noipv4 --noipv6
rootpw --iscrypted X4qmByxwKBQbY
# Reboot after installation
reboot
firewall --disabled
authconfig --useshadow --enablemd5
selinux --disabled
timezone --utc America/New_York
bootloader --location=mbr --driveorder=sda --append="crashkernel=auto rhgb quiet"
# The following is the partition information you requested
# Note that any partitions you deleted are not expressed
# here so unless you clear all partitions first, this is
# not guaranteed to work
#clearpart --none
#part /boot/efi --fstype=efi --grow --maxsize=200 --size=20
#part /boot --fstype=ext4 --size=500
#part pv.008003 --grow --asprimary --size=25600
#volgroup vg00 --pesize=32768 pv.008003
#logvol / --fstype=ext4 --name=root --vgname=vg00 --size=10240
#logvol swap --name=swap --vgname=vg00 --size=4096
#logvol /tmp --fstype=ext4 --name=tmp --vgname=vg00 --size=2048
#logvol /var --fstype=ext4 --name=var --vgname=vg00 --size=6144
repo --name="Red Hat Enterprise Linux" --baseurl=nfs:159.55.104.162:/vol/XRZ0/opsware/linux/redhat/x86_64/v6.1 --cost=100
%packages
@Base
@Core
@base
@core
OpenIPMI
OpenIPMI-libs
audit
audit-libs
autofs
busybox
compat-openldap
crash
dos2unix
e2fsprogs
grub
kexec-tools
krb5-libs
krb5-libs.i686
ksh
libcollection
libcollection.i686
libdhash
libdhash.i686
libini_config
libini_config.i686
lm_sensors
lvm2
mailx
ncompress
net-snmp
net-snmp-libs
net-snmp-utils
nfs-utils
nss-pam-ldapd
ntp
openldap
openldap-clients
openssl098e
pam.i686
parted
perl
perl-Compress-Zlib
perl-Convert-ASN1
perl-HTML-Parser
perl-HTML-Tagset
perl-IO-Socket-SSL
perl-LDAP
perl-Net-SSLeay
perl-URI
perl-XML-NamespaceSupport
perl-XML-SAX
perl-libwww-perl
psacct
sssd
sssd-client
strace
sudo
sysstat
tcpdump
tcsh
telnet
zsh
-NetworkManager
-NetworkManager-glib
-alsa-lib
-alsa-utils
-audiofile
-authconfig-gtk
-bluez-libs
-cdparanoia-libs
-cups
-cups-libs
-dbus-python
-desktop-file-utils
-elinks
-esound
-fetchmail
-foomatic
-ftp
-gimp-print
-gnome-keyring
-gnome-python2
-gnome-python2-bonobo
-gnome-python2-canvas
-gnome-python2-gtkhtml2
-gnome-vfs2
-gnuplot
-gphoto2
-gpm
-gstreamer
-gtkhtml2
-gtksourceview
-indexhtml
-isdn4k-utils
-jpackage-utils
-jwhois
-kdemultimedia
-kernel-devel
-lftp
-lha
-libbonobo
-libbonoboui
-libglade2
-libgnome
-libgnomecanvas
-libgnomeui
-libibverbs
-libjpeg
-libmthca
-libpng
-libtiff
-libvorbis
-libwnck
-libwvstreams
-libxml2
-libxslt
-logwatch
-lokkit
-lrzsz
-minicom
-mkbootdisk
-mt-st
-mtr
-mutt
-mysql
-nc
-nmap
-numactl
-openib
-openmotif
-opensm-libs
-pango
-pinfo
-ppp
-procmail
-pyOpenSSL
-pygtk2
-pyorbit
-pyxf86config
-rdate
-rdist
-redhat-logos
-redhat-menus
-rhn-check
-rhn-client-tools
-rhn-setup
-rhnsd
-rp-pppoe
-rsh
-samba-client
-samba-common
-setuptool
-sox
-star
-stunnel
-system-config-network-tui
-talk
-tmpwatch
-tog-pegasus
-usermode-gtk
-vnc-server
-wireless-tools
-wvdial
-yp-tools
-ypbind
%end
%pre
set -x
[ -d /opsw ] && exit
mkdir /opsw
mount /dev/sda1 /opsw
cp /opsw/phase2/* /tmp
umount /opsw
/tmp/miniagent --server 159.55.104.44 8017
%end
%pre --logfile /tmp/anaconda.pre_log
#!/bin/bash
echo "Running pre-install script..."
# find out the size of the boot disk
parted="/usr/sbin/parted"
grep="/bin/grep"
awk="/bin/awk"
sed="/bin/sed"
dd="/usr/bin/dd"
sleep="/bin/sleep"
# echo 'y' because parted asks for some confirmation when hits a raw disk without mbr
sda_size=`echo 'y' | $parted -l /dev/sda | $grep ^Disk| $grep sda | $awk -F':' '{print $2}' | $sed -e 's/^ *//' | $sed -e 's/[^0-9]*$//'`
echo "/dev/sda size == $sda_size GB"
echo ""
# clean up the MBR and partition table of the root disk
/usr/bin/dd bs=512 count=10 if=/dev/zero of=/dev/sda
echo ""
if [ $sda_size -gt 2048 ]
then
echo "Root disk is bigger than 2TB, installing gpt partition table!"
/usr/sbin/parted --script /dev/sda mklabel gpt
else
echo "Root disk is smaller than 2TB, installing msdos partition table."
/usr/sbin/parted --script /dev/sda mklabel msdos
fi
echo ""
/usr/sbin/parted -l /dev/sda
/bin/sleep 30
%end
%post --nochroot
echo "Running post-install scritp"
# prepare UEFI boot environment
# (don't know how to make RHEL installer to do it automatically)
/usr/bin/cp /mnt/sysimage/boot/grub/grub.conf /mnt/sysimage/boot/efi/EFI/redhat/grub.conf
# necessary for grubby to update the right file during the kernel upgrades
/usr/bin/rm /mnt/sysimage/etc/grub.conf
/usr/bin/ln -s /boot/efi/EFI/redhat/grub.conf /mnt/sysimage/etc/grub.conf
# make logs available after kickstart for debugging purpose
/usr/bin/cp /tmp/*log /mnt/sysimage/root
%end
%post
touch /var/tmp/agentInstallTrigger; while [ -f /var/tmp/agentInstallTrigger ]; if [ -f /var/tmp/doneAgentInstall ]; then break; fi; do sleep 5; done
%end
FUSION IO:
RAID1-SETUP
http://kb.fusionio.com/KB/a18/creating-a-raid1mirrored-set-using-the-iodrive-on-linux.aspx?KBSearchID=16919
Creating a RAID1 (Mirrored) Set using the ioDrive on Linux
To create a RAID1/Mirror set on Linux, enter the command:
$ mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/fioa /dev/fiob
This creates a mirrored set using the two ioDrives fioa and fiob. (Use the fio-status utility to view your specific device
names.)
Start up the RAID1 mirror
To start the RAID1 mirrored, set and mount it as md0, enter the command:
mdadm --assemble /dev/md0 /dev/fioa /dev/fiob
Stop the RAID1 mirror
To stop the RAID1 set, enter the command:
mdadm --stop /dev/md0
Make the mirror persistent (exists after a reboot)
Use the following commands to have your RAID1 mirror will appear after a reboot.
$ sudo sh -c 'echo "DEVICE /dev/fioa /dev/fiob" >/etc/mdadm.conf'
$ sudo sh -c 'mdadm --detail --scan >>/etc/mdadm.conf'
On some versions of Unix, the configuration file is in /etc/mdadm/mdadm.conf,
not /etc/mdadm.conf.
On most systems, the RAID1 mirror will be created automatically upon reboot. However, if you have problems
accessing /dev/md0after a reboot, run this command:
$ sudo mdadm --assemble --scan
DRIVER-SETUP
http://kb.fusionio.com/KB/a64/loading-the-driver-via-udev-or-init-script-for-md-and-lvm.aspx?KBSearchID=16922
-Loading the Driver via udev or Init Script for md and LVM There are two main methods for loading the Fusion-IO Driver: udev and the iodrive
script. The default method of loading is udev.
Using udev to Load the Driver Most modern linux distributions use udev to facilitate driver loading. Usually it just works, behind the scenes,
loading drivers. udev automatically loads drivers for installed hardware on the system. It does this by walking through the devices on the PCI
bus, and then loading any driver that has been properly configured to work with that device. This way, drivers can be loaded without having to
use scripts or modify configuration files. The Fusion-io drivers are properly configured, and udev will find the drivers and attempt to load them.
However, there are some cases where loading by udev is not appropriate, or can have issues.
Udev will wait 180 seconds for the driver to load, then it will exit. In most cases, this is plenty of time, even with multiple ioDrives installed. But
if the drives were shut down improperly, loading the driver and attaching the drives takes longer than the 180 seconds. In this case, udev will
exit. The driver will not exit, but will continue working on attaching the drives.
There is not always a problem when udev exits early. The drivers will eventually load, and then you will be able to use the attached block
devices. But, if the drivers do take too long to load, and udev does exit, and file systems are set to be mounted in the fstab, then the system file
system check (fsck) will fail, and the system will stop booting. In most distributions the user will drop into a single-user mode, or repair mode.
Again, this is normal behavior; after the driver finishes rescanning and attaching the drive, a reboot will fix things. For most users, this will not
happen often enough to be an issue.
But for installations with many devices, or for server installations where dropping into single-user mode is unacceptable, there is an alternative
method for driver loading that does not have these issues.
Using Init Scripts to Load the Driver The ioDriver packages provide init scripts that are installed on all supported Linux distributions. These
scripts typically reside in /etc/init.d/iodrive or /etc/rc.d/iodrive. The scripts are used to load and start the driver, and to mount filesystems,
after the system is up. This method completely avoids the udev behavior described above. (It will wait as long as it takes for drives to be
attached).
NOTE: These steps assume that the logical volumes /dev/md0 (md) and /dev/vg0/fio_lv (LVM) have been setup beforehand. There are other
knowledgebase articles that provide details for logical volume creation, including updating lvm.conf to work with ioDrives. Follow the
instructions based your driver version. (jump to: Using Init Scripts to Load the 1.2.x Driver)
Using Init Scripts to Load the 2.x.x Driver
Step 1: Modifying /etc/modprobe.d/iomemory-vsl.conf
Edit /etc/modprobe.d/iomemory-vsl.conf, and uncomment the blacklist line:
Before:
To keep ioDrive from auto loading at boot, uncomment below:
# blacklist iomemory-vsl
After:
To keep ioDrive from auto loading at boot, uncomment below:
blacklist iomemory-vsl.
This keeps udev from automatically loading the driver
Step 2: Modifying /etc/fstab Add noauto to the options in the appropriate line in the /etc/fstab file. This will keep the OS from trying to check
the drive for errors on boot.
Before:
LABEL=/ / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sda5 swap swap defaults 0 0
/dev/md0 /iodrive_mountpoint ext3 defaults 1 2
/dev/vg0/fio_lv /iodrive_mountpoint2 ext3 defaults 1 2
After:
LABEL=/ / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2 tmpfs
/dev/shm tmpfs defaults 0 0 devpts /dev/pts
devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sda5 swap swap defaults 0 0
/dev/md0 /iodrive_mountpoint ext3 defaults,noauto 0 0
/dev/vg0/fio_lv /iodrive_mountpoint2 ext3 defaults,noauto 0 0
Step 3: Modifying /etc/sysconfig/iomemory-vsl.

/etc/sysconfig/iomemory-vsl, and uncomment ENABLED=1 to enable the init script.
Before:
If ENABLED is not set (non-zero) then iomemory-vsl init script will not be used.
#ENABLED=1
After:
If ENABLED is not set (non-zero) then iomemory-vsl init script will not be used.
ENABLED=1
While editing /etc/sysconfig/iomemory-vsl, add the mountpoint to the MOUNTS variable so it will be automatically attached and mounted:
Before:
An IFS separated list of md arrays to start once the driver is loaded. Arrays should be configured in the mdadm.conf file.
Example: MD_ARRAYS="/dev/md0 /dev/md1"MD_ARRAYS=""
An IFS separated list of LVM volume groups to start once the driver is loaded. Volumes should be configured in lvm.conf.
Example: LVM_VGS="/dev/vg0 /dev/vg1"LVM_VGS=""
An IFS separated list of mount points to mount once the driver is loaded.
These mount points should be listed in /etc/fstab with
"noauto" as one of the mount options.
Example /etc/fstab:
/dev/fioa /mnt/fioa ext3 defaults,noauto 0 0
/dev/fiob /mnt/firehose ext3 defaults,noauto 0 0
Example:
MOUNTS="/mnt/fioa /mnt/firehose"MOUNTS=""
After:
11. An IFS separated list of md arrays to start once the driver is loaded. Arrays should be configured in the mdadm.conf file.
# Example: MD_ARRAYS="/dev/md0 /dev/md1"MD_ARRAYS="/dev/md0"
An IFS separated list of LVM volume groups to start once the driver is loaded. Volumes should be configured in lvm.conf.
# Example:
LVM_VGS="/dev/vg0 /dev/vg1"LVM_VGS="/dev/vg0"
An IFS separated list of mount points to mount once the driver is loaded. These mount points should be listed in /etc/fstab with
"noauto" as one of the mount options.
Example /etc/fstab:
/dev/fioa /mnt/fioa ext3 defaults,noauto 0 0
/dev/fiob /mnt/firehose ext3 defaults,noauto 0 0
Example:
MOUNTS="/mnt/fioa /mnt/firehose"MOUNTS="/iodrive_mountpoint /iodrive_mountpoint2"
Step 4: Verifying the Status of the Init Script Make sure the iodrive script loads at run levels 1 through 5 (runlevel 0 is shutdown and runlevel 6 is
reboot). Run the following commands:
$ chkconfig iomemory-vsl on$ chkconfig --list iomemory-vsl iomemory-vsl 0:off 1:on 2:on 3:on 4:on 5:on 6:off
Using Init Scripts to Load the 1.2.x Driver Step 1: Modifying /etc/modprobe.d/iodrive Edit /etc/modprobe.d/iodrive, and uncomment the
blacklist line:
Before:
1.
To keep ioDrive from auto loading at boot, uncomment below# blacklist fio-driverAfter:
2.
To keep ioDrive from auto loading at boot, uncomment belowblacklist fio-driverThis keeps udev from automatically loading the
driver
Step 2: Modifying /etc/fstab. Add noauto to the options in the appropriate line in the /etc/fstab file. This will keep the OS from trying to check
the drive for errors on boot.
Before:
LABEL=/ / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sda5 swap swap defaults 0 0
/dev/md0 /iodrive_mountpoint ext3 defaults 1 2
/dev/vg0/fio_lv /iodrive_mountpoint2 ext3 defaults 1 2
After:
LABEL=/ / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sda5 swap swap defaults 0 0
/dev/md0 /iodrive_mountpoint ext3 defaults,noauto 0 0
/dev/vg0/fio_lv /iodrive_mountpoint2 ext3 defaults,noauto 0 0
Step 3: Modifying /etc/sysconfig/iodrive
Edit /etc/sysconfig/iodrive, and add the mountpoint to the MOUNTS variable, so it will be automatically attached and mounted:
Before:
1.
An IFS separated list of md arrays to start once the driver is# loaded. Arrays should be configured in the mdadm.conf file.
2.
# Example: MD_ARRAYS="/dev/md0 /dev/md1"MD_ARRAYS=""
3.
# An IFS separated list of LVM volume groups to start once the driver is# loaded. Volumes should be configured in lvm.conf.#
Example: LVM_VGS="/dev/vg0 /dev/vg1"LVM_VGS=""# An IFS separated list of mount points to mount once the driver is# loaded.
These mount points should be listed in /etc/fstab with# "noauto" as one of the mount options.# Example /etc/fstab:#/dev/fioa
/mnt/fioa ext3 defaults,noauto 0 0#/dev/fiob /mnt/firehose ext3 defaults,noauto 0 0# Example: MOUNTS="/mnt/fioa
/mnt/firehose"MOUNTS=""After:
2.
An IFS separated list of md arrays to start once the driver is# loaded. Arrays should be configured in the mdadm.conf file.# Example:
MD_ARRAYS="/dev/md0 /dev/md1"MD_ARRAYS="/dev/md0"# An IFS separated list of LVM volume groups to start once the driver
is# loaded. Volumes should be configured in lvm.conf.# Example: LVM_VGS="/dev/vg0 /dev/vg1"LVM_VGS="/dev/vg0"# An IFS
separated list of mount points to mount once the driver is# loaded. These mount points should be listed in /etc/fstab with# "noauto"
as one of the mount options.# Example /etc/fstab:#/dev/fioa /mnt/fioa ext3 defaults,noauto 0 0#/dev/fiob /mnt/firehose ext3
defaults,noauto 0 0# Example: MOUNTS="/mnt/fioa /mnt/firehose"MOUNTS="/iodrive_mountpoint /iodrive_mountpoint2"Step 4:
Verifying the Status of the Init Script
Make sure the iodrive script loads at run levels 1 through 5 (runlevel 0 is shutdown and runlevel 6 is reboot). Run the following commands:
$ chkconfig iodrive on$ chkconfig --list iodrive iodrive 0:off 1:on 2:on 3:on 4:on 5:on 6:off
LVM-ENABLE
http://kb.fusionio.com/KB/a36/enabling-the-iodrive-for-lvm-use.aspx
Enabling the ioDrive for LVM Use The Logical Volume Manager (LVM) volume group management application handles mass storage devices like
the ioDrive if you add the ioDrive as a supported type:
1. Locate and edit the /etc/lvm/lvm.conf configuration file.
2. Add an entry similar to the following to that file:
types = [ "fio", 16 ]
The parameter “16” represents the maximum number of partitions supported by the drive. For the ioDrive, this can be any number from 1
upwards, with 16 as the recommended setting.
Do NOT set this parameter to 0.