Troubleshooting

Troubleshooting
If you have problems with a SolarWinds product, the causes are usually related to
an incorrect configuration or corrupted files. The suggestions listed in this section
can often clear up these problems.
Additionally, you can visit our knowledge base repository for articles to help
remedy specific issues. The address is: http://knowledgebase.solarwinds.com/kb/
You can also talk to other users as well as the SolarWinds staff by logging on to
thwack.com
Back Up Your Data
As a first step in any troubleshooting procedure, you should back up your
SolarWinds database. For more information, see “Creating Database Backups”
on page 903.
Verify Program Operation
SolarWinds runs many components at the same time to deliver a view of your
network status. Confirm that the following components are running on your
SolarWinds server:
Services:
l
l
l
l
l
l
l
l
l
l
l
l
l
Message Queuing
Net.Tcp Port Sharing Service
SNMP Trap Service
SolarWinds Alerting Engine service
SolarWinds Collector Data Processor, Management Agent, and Polling Controller services
SolarWinds Information Service
SolarWinds Job Engine and Job Engine v2
SolarWinds Job Scheduler
SolarWinds Network Performance Monitor Service. All SolarWinds products
use this service. It is not exclusive to SolarWinds SAM.
SolarWinds Product services
SolarWinds Information Service
SolarWinds Module Engine
SolarWinds Syslog and Trap Services
1319
Troubleshooting
SQL Server
Internet Information Service (IIS)
l
l
Stop and Restart
Many problems disappear when programs are restarted. Stopping and restarting
Internet Information Service (IIS) may eliminate web page problems. Problems
with polling or data gathering may be eliminated by stopping and restarting the
SolarWinds Network Performance Monitor Service using the available shutdown
tool that you can locate as follows:
1. Click Start > All Programs > SolarWinds > Advanced Features > Orion
Service Manager.
For a complete refresh of the system, reboot the computer.
Run the Configuration Wizard
Running the Configuration Wizard, which refreshes files on the web server and
performs checks on the structure of your database, may solve many problems.
Note: Before you run the Configuration Wizard, you should close all open
applications and stop the SolarWinds Network Performance Monitor Service in
the Windows Services Control Panel. It will be restarted by the Wizard at the end
of its process.
Using Full Variable Names
If you are having difficulty acquiring expected values from a selected variable,
using the ${VariableName}format, it may be helpful to include the database table
name within the variable name, as in ${Nodes.IP_Address}.
Why do my SAM WMI Monitors Show Status
Unknown?
To monitor SAM applications containing WMI component monitors, the
following conditions must be true:
l
l
WMI on the remote server is enabled and functioning properly.
The remote server is accessible through a RPC connection in order to run
the WMI queries.
If these conditions cannot be met, the WMI component monitors in SAM show an
Unknown status. Examples of some issues that can prevent these conditions from
being met include, but are not limited to:
1320
Working with Temporary Directories
l
l
l
l
Trying to connect to a remote computer where you do not have local Administrator rights.
A firewall blocking the WMI traffic.
An operating system that is not configured for WMI.
Mistyping the credential password in the SAM component monitor.
To help diagnose and fix these issues and others, we can test the WMI services,
the remote WMI connections, and the SolarWinds SAM component configuration
to discover and correct the issues that can prevent your WMI component monitors
from functioning correctly.
The topics in this guide are as follows:
l
l
l
l
WMI Troubleshooting Flowchart for SolarWinds SAM. Provides a flowchart
of the troubleshooting decisions described in this guide.
Testing Local WMI Services. Ensures WMI is running correctly on the target
computer.
Testing Remote WMI Connections. Ensures the WMI connection to the target computer is not being blocked, ignored, or rejected.
Testing SolarWinds SAM Component Configuration. Ensures you are properly configuring the WMI component credentials in SolarWinds SAM.
Working with Temporary Directories
The following sections provide procedures for moving Windows and SQL Server
temporary directories to optimize SolarWinds server performance and resources.
Moving the SQL Server Temporary Directory
The SQL Server temporary directory, tempdb, where temporary database objects
generated during table creation and sorting are stored, is typically created in the
same location on your SolarWinds database server as the master, model, and msdb
databases. Moving the tempdb database to a physical drive separate from your
SolarWinds database can significantly improve overall system performance.
l
l
For more information about moving the SQL Server 2005 temporary directory, tempdb, for see “Moving System Databases – Example A: Moving the
tempdb Database.”
For more information about moving the SQL Server 2008 temporary directory, tempdb, for see “Moving System Databases – Example A: Moving the
tempdb Database.”
1321
Troubleshooting
Redefining Windows System Temporary Directories
Following established Windows standards, the SolarWinds installer may use
Windows User and System TEMP and TMP variable directories as temporary
scratch spaces for file expansion and execution. If you do not have the required
scratch space available in the default User or System TEMP and TMP directories,
use the following procedure to redefine your default locations.
Note: Regardless of where you actually install SolarWinds, some common files
may be installed where the operating system of your SolarWinds server are
located.
To Redefine Default System Temporary Directories:
1.
2.
3.
4.
Log on to your SolarWinds server as a user with administrative rights.
Right-click My Computer, and then click Properties.
Click Advanced, and then click Environment Variables.
Select the variable name representing the directory you want to redefine,
click Edit, and then provide a new path as the Variable value for the selected temporary directory variable.
Slow Performance on Windows Server 2008
If SolarWinds is installed on Windows Server 2008 or Windows Vista and there
are any devices on your network, between your SolarWinds server and your
database server, that do not support RFC 1323, “TCP Extensions for High
Performance,” the TCP window size auto-tuning feature of Windows Server 2008
and Windows Vista may prevent your SolarWinds server from successfully
connecting with your SolarWinds database. This TCP auto-tuning feature is
intended as a network-sensitive restriction, applied by the receiver—your
SolarWinds server—on the amount of data it is allowed or able to receive. If any
devices along the network path between your SolarWinds server and your
SolarWinds database do not support the TCP window scaling detailed in RFC
1323, the allowed size of the TCP window in packets sent to your SolarWinds
server may not match the TCP window size reported by packets sent from your
SolarWinds server. This mismatch may lead to failed connections between your
SolarWinds server and your SolarWinds database. The following procedure
disables this TCP auto-tuning feature, resetting the TCP receive window to 64kB.
1322
To Disable TCP Auto-tuning:
To Disable TCP Auto-tuning:
1. Click Start > All Programs > Accessories.
2. Right-click Command Prompt, and then click Run as administrator.
3. If you are prompted by User Account Control, click Continue to open
the elevated command prompt.
Note: In some cases, having User Account Control (UAC) enabled on Windows Server 2008 and Windows Vista (evaluation-only) can lead to installation errors. For more information about disabling UAC, see the article,
“Disabling User Account Control (UAC)” in the SolarWinds Knowledge
Base.
4. At the prompt, enter the following: netsh interface tcp set global autotuninglevel=disabled
5. Close the command prompt window, and then restart your SolarWinds
server.
1323
Troubleshooting
WMI Troubleshooting Flowchart for SolarWinds
SAM
1324
Testing Local WMI Services
Note: The following Microsoft article describes how to set permissions in a
workgroup after an upgrade from Microsoft Windows 2000 Professional to
Microsoft Windows XP Professional. http://support.microsoft.com/?kbid=290403
Testing Local WMI Services
Testing the local WMI services helps us isolate any faults on the target server we
are trying to monitor. The testing program is a Microsoft program named
WBEMTest that comes already installed on Microsoft Windows operating
systems.
Test WMI on the Target Server
Complete the following procedure to check whether WMI on the target
server is functioning correctly:
1. Log on to the target server with an administrator account.
2. Click Start > Run, enter wbemtest.exe and then click OK.
3. Click Connect on the Windows Management Instrumentation Tester window.
1325
Troubleshooting
4. Enter root\cimv2 in the field at the top of the dialog box next to the Connect
button.
5. Click Connect.
6. Click Enum Classes.
1326
Test WMI on the Target Server
7. Select the Recursive radio button without entering a superclass name, and
then click OK.
8. If the WMI class you are querying appears in this list, local WMI services are functioning correctly. Skip to the next topic and test remote WMI.
9. If the list does not appear or does not contain the desired WMI class,
WMI is not functioning correctly. Continue reading this section for guidance
on repairing WMI services on the target server.
10. Click Close, and then click Exit.
1327
Troubleshooting
Reset the WMI Counters
At times, the WMI performance counters may not get transferred to WMI because
services were delayed or started out of order
(http://support.microsoft.com/kb/820847).
To manually reset the WMI counters:
1.
2.
3.
4.
5.
6.
Stop the Windows Management Instrumentation service.
Click Start, click Run, type cmd, and then click OK.
At the command prompt, type winmgmt /resyncperf, and then press Enter.
At the command prompt, type wmiadap.exe /f, and then press Enter.
Type exit, and then press Enter to close the command prompt.
Start the Windows Management Instrumentation service.
After resetting the WMI counters, retest WMI. If resetting the WMI counters did not
solve your problem, see “WMI is Still Not Working. Now What?”
Testing Remote WMI Connectivity
Testing the remote WMI connectivity of the target server helps us isolate faults
that could prevent the target server from receiving or responding to our remote
WMI requests. The testing program is a Microsoft program named WBEMTest that
comes already installed on Microsoft Windows operating systems.
Remotely Test WMI on the Target Server
Complete the following procedure to check whether the target server is
responding appropriately to remote WMI requests that originate from the
SolarWinds SAM server:
1. Log on to the SolarWinds SAM server with an administrator account.
2. Click Start > Run, enter wbemtest.exe and then click OK.
3. Click Connect on the Windows Management Instrumentation Tester window.
1328
Remotely Test WMI on the Target Server
4. Enter \\Target_Primary_IP_Address\root\cimv2 in the field at the top of the
dialog box. Replace Target_Primary_IP_Address in the above example
with the actual Hostname or Primary IP Address of the target server.
5. Enter the user name in the User field, the password in the Password field,
and NTLMDOMAIN:NameOfDomain in the Authority field. Replace
NameOfDomain with the domain of the user account specified in the User
field.
6. Click Connect.
7. Click Enum Classes.
1329
Troubleshooting
8. Select the Recursive radio button without entering a superclass name,
and then click OK.
9. If the WMI class list appears, remote WMI is functioning correctly. Skip to
the next topic and test your SAM credentials.
10. If the list does not appear, remote WMI is not functioning correctly.
Continue reading this topic for guidance on restoring remote WMI connections on the target server, and retest remote WMI after completing each
troubleshooting step.
11. Click Close, and then click Exit.
1330
Verify Administrator Credentials
Verify Administrator Credentials
Only a credential that has administrator rights on the target server has the
necessary permissions to access the target server’s WMI services. Make sure that
the username and password you are using belongs to an administrator on the
target server.
If the administrator credential is a domain member, be sure to specify both the
user name and the domain in the standard Microsoft syntax. For example:
DOMAIN\Administrator.
Enable Remote Procedure Call (RPC)
Remote WMI connections use RPC as a communications interface. If the RPC
service is disabled on the target server, remote WMI connections cannot be
established.
To enable the RPC service:
1.
2.
3.
4.
Log on to the target server with an administrator account.
Click Start, click Run, type services.msc, and then press Enter.
Scroll the list to Remote Procedure Call (RPC)
Right-click Remote Procedure Call (RPC), and then click Start on the
shortcut menu.
Configure Distributed Component Object Model (DCOM) and User
Account Control (UAC)
If the target computer is running Windows Vista or Windows Server 2008, you
may be required to make settings changes to allow remote WMI requests
(http://msdn.microsoft.com/en-us/library/aa822854(VS.85).aspx).
Item
Need
l
DCOM
l
l
l
l
l
Default and Limits permissions edited to allow the following actions:
Local launch (default permission)
Remote launch (default permission)
Local activation (limits permission)
Remote activation (limits permission)
For more information, see “Enabling DCOM”
1331
Troubleshooting
WMI
Namespaces
User
Account
Control
l
l
Modify the CIMV2 security to enable and remote enable
the account used to access the server or workstation
through WMI. You must ensure the security change
applies to the current namespace and sub-namespaces.
For more information, see “Enabling Account Privileges in
WMI”
Remote UAC access token filtering must be disabled
when monitoring within a workgroup environment. For
more information, see “Disabling Remote User Account
Control for Workgroups”
Enabling DCOM
WMI uses DCOM to communicate with monitored target computers. Therefore, for
Server & Application Monitor to use WMI, DCOM must be enabled and properly
configured.
To enable DCOM permissions for your Server & Application Monitor credentials:
1. Log on to the target server with an administrator account.
2. Navigate to Start > Control Panel > Administrative Tools > Component
Services. You need to switch to the Classic View of the Control Panel to
use this navigation path. You can also launch this console by
double-clicking comexp.msc in the /windows/system32 directory.
3. Expand Component Services > Computers.
4. Right-click My Computer, and then select Properties.
5. Select the COM Security tab, and then click Edit Limits in the Access Permissions grouping.
6. Ensure the user account you want to use to collect WMI statistics has Local
Access and Remote Access, and then click OK.
7. Click Edit Default, and then ensure the user account you want to use to
collect WMI statistics has Local Access and Remote Access,
8. Click OK.
9. Click Edit Limits in the Launch and Activation Permissions grouping.
10. Ensure the user account you want to use to collect WMI statistics has Local
Launch, Remote Launch, Local Activation, and Remote Activation, and then click
OK.
1332
Enabling Account Privileges in WMI
11. Click Edit Default, and then ensure the user account you want to use to
collect WMI statistics Local Launch, Remote Launch, Local Activation, and
Remote Activation.
12. Click OK.
Enabling Account Privileges in WMI
The account you specify in the Credentials Library must possess security access
to the namespace and sub-namespaces of the monitored target computer. To
enable these privileges, complete the following procedure.
To enable namespace and sub-namespaces privileges:
1. Log on to the computer you want to monitor with an administrator account.
2. Navigate to Start > Control Panel > Administrative Tools > Computer
Management > Services and Applications. You need to switch to the
Classic View of the Control Panel to use this navigation path.
3. Click WMI Control, and then right-click and select Properties.
4. Select the Security tab, and then expand Root and click CIMV2.
5. Click Security and then select the user account used to access this computer and ensure you grant the following permissions:
Enable Account
Remote Enable
6. Click Advanced, and then select the user account used to access this computer.
7. Click Edit, select This namespace and subnamespaces in the Apply to field,
and then click OK.
8. Click OK on the Advanced Security Settings for CIMV2 window.
9. Click OK on the Security for Root\CIMV2 window.
10. Click Services in the left navigation pane of Computer Management.
11. Select Windows Management Instrumentation in the Services result pane, and
then click Restart.
Disabling Remote User Account Control for Workgroups
If you are monitoring a target in a workgroup, you need to disable remote User
Account Control (UAC). This is not recommended but it is necessary when
monitoring a workgroup computer. Disabling remote user account control does
not disable local user account control functionality.
Warning: The following procedure requires the modification or creation of a
registry key. Changing the registry can have adverse effects on your computer
1333
Troubleshooting
and may result in an unbootable system. Consider backing up your registry before
making these changes.
To disable remote UAC for a workgroup computer:
1.
2.
3.
4.
Log on to the computer you want to monitor with an administrator account.
Click Start > Accessories > Command Prompt.
Enter regedit.
Expand HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System.
5. Locate or create a DWORD entry named LocalAccountTokenFilterPolicy and
provide a DWORD value of 1.
Note: To re-enable remote UAC, change this value to 0.
Add a Windows Firewall Exception for Remote WMI Connections
If the target computer has Windows Firewall enabled, it must have a Remote WMI
exception to allow remote WMI traffic through (http://msdn.microsoft.com/enus/library/aa389286(VS.85).aspx).
To add this exception:
1. Click Start, click Run, type cmd and then press Enter.
2. At the command prompt, type netsh firewall set service RemoteAdmin enable,
and then press Enter.
3. At the command prompt, type exit, and then press Enter.
If adding the firewall exception did not solve your problem, see “WMI is Still Not
Working. Now What?”
Do You Need an Additional Polling Engine?
Symptoms that may indicate you need an additional polling engine include, but
are not limited to, the following:
l
l
l
Polling completion rates are lower than expected which leads to gaps in
data.
Sporadic component monitor timeouts usually resulting in component monitors showing an "Unknown" status, reporting the error, "Connection
timeout. Job cancelled by scheduler."
Resource exhaustion on the SQL server.
For more information, see "Additional Polling Engine and Web Console" on page
930.
1334
Verify SAM Component Configuration
Verify SAM Component Configuration
Make sure that the credential you are using for remote WMI is the same credential
that you are using in the SAM component.
After you click Set Component Credentials to set the component credentials,
you must also click Submit at the bottom of the page.
Service reporting, "Invalid Class"
There are various possibilities for this to occur. Read the following section to help
troubleshoot this issue. In APM v4 and higher, services can be polled via RPC or
WMI. Ensure the correct service is running by navigating to Edit Application >
RPC or WMI.
Repair the WMI Repository according to this KB article:
http://knowledgebase.solarwinds.com/kb/questions/633/Error+message+%22Inva
lid+Class%22+when+using+WMI+monitors
1335
Troubleshooting
Note: Vista and Windows 2008 have a built-in method for repairing the WMI
repository. Open an Administrator command prompt and run: winmgmt
/salvagerepository
If you are using APM prior to version 4 and using WMI, try the following:
There are three required WMI queries for service monitoring to work:
SELECT * FROM Win32_Service
SELECT * FROM Win32_Process
SELECT * FROM Win32_PerfRawData_PerfProc_Process
You can test these by going to Start > Run > WBEMTEST on the SolarWinds
server.
1.
2.
3.
4.
5.
6.
7.
Click on Connect.
In the whitespace at the top, you should have \\ipaddress\root\CIMV2
For user and password, fill in the credentials used in APM.
For Authority, type in NTLMDOMAIN: and enter your domain.
Click Connect.
Click Query
Enter the above three queries, one at a time. More often than not, the culprit is the last query. If your problem still exists,
try the following:
1. Re-sync the counters by opening a command prompt and typing: winmgmt
/resyncperf
2. Check to make sure performance counters are not disabled: http://thwack.com/forums/68/application--server-management/21/application-performance-monitor/23124/win32perfrawdataperfprocpro/
Note: WMI is an operating system component. If the previous steps do no
work, you may need to contact Microsoft for further information.
WMI is Still Not Working. Now What?
This guide depicts only the most common scenarios that can cause WMI services
to fail. If you are unable to get WMI services to work at this point, it is time to
consult Microsoft on this topic.
l
“WMI Troubleshooting.” Microsoft Developer Network. http://msdn.microsoft.com/en-us/library/aa394603.aspx
1336
Troubleshooting Hardware Health
Troubleshooting Hardware Health
This section describes possible causes and solutions concerning hardware
resources either not being reported or being reported incorrectly.
Hardware Prerequisite Checklist
If the following conditions cannot be met, the Hardware Health resources will not
be displayed. To monitor hardware in SAM, the following must be true:
l
l
l
l
The monitored node must be HP Proliant, Dell PoweEdge, IBM X-Series,
HP C7000, HP C3000, or Dell M1000e.
The node must be monitored using one of the following protocols:
o SNMP
o WMI
o ICMP nodes are allowed for VMWare when the Poll for VMware
option is selected.
The Hardware Monitoring Agent software, (provided by the vendor), is
installed on the remote server. This applies for both SNMP and WMI.
Required software can be found at "Monitoring Hardware Health" on page
834.
For VMware, the minimum requirements are as follows: ESX server version 3.5, 4.0, 4.1, ESXi version 5.0, vCenter version 4.0, 4.1, 5.0.
The following systems have been verified to work properly with SAM's hardware
monitoring features. Note: Other systems may work as well.
l
l
l
l
l
Dell PowerEdge M610, R210, R610, R710, R900, 1950, 2850, 2950,
2970, 6850
HP ProLiant DL320 G4, DL360 G3, DL360 G4, DL380 G4, DL380 G6,
ML570 G3
IBM IBM System x3550, System x3550 M2, System x3550 M3, System
x3650, System x3650 M2, System x3650 M3, x3850, eServer 306m
HP C7000, HP C3000
Dell M1000e.
Note: IBM's ServeRAID Manager software must be installed on IBM X-Series
servers for storage hardware health information to be displayed in SolarWinds
SAM. HP’s WBEM providers are required for HP servers polled via WMI.
Installation instructions can be at "Monitoring Hardware Health" on page 834.
1337
Troubleshooting
Hardware Troubleshooting Flowchart
1338
Troubleshooting an SNMP Node
Note: Dell does not make array and hard disk health information visible from WMI
managed nodes. To monitor storage health on Dell servers, use SNMP.
Troubleshooting an SNMP Node
The most common issue customers face is that hardware information is not
available via SNMP because the Hardware Monitoring Agent software was
installed before SNMP was installed. This means MIBs were never installed
and/or configured correctly. The easiest solution is to uninstall and then re-install
the Hardware Monitoring Agent software after installing SNMP on the server. If
this is not the case, follow the troubleshooting steps as outlined below:
Verify the node was successfully added using SNMP:
1. Verify the polling method on the Node Details page as shown below:
2. Verify the Hardware Monitoring Agent software is installed on the remote
server and running. For verification instructions, see "Accessing Hardware
Monitoring Agent Software" on page 836.
3. Determine if SNMP responds for the proper OID. Below are the correct
OIDS for each vendor:
For HP:1.3.6.1.4.1.232.2.2.2.1.0
For Dell:1.3.6.1.4.1.674.10892.1.300.10.1.8.1
For IBM:1.3.6.1.4.1.2.6.159.1.1.60.3.1
l
To determine if the remote server responds to the correct OID, you can use
the MIB browser from SolarWinds Engineer’s Toolset, which can be downloaded from http://www.solarwinds.com/downloads/. Additionally, you can
use other applications capable of making SNMP requests.
If you do not have a tool for checking OIDs on the remote server, you can create
an SNMP walk by using the SNMPWalk.exe installed with SAM, normally located at
1339
Troubleshooting
C:\Program Files (x86)\SolarWinds\Orion\SnmpWalk.exe. SNMPWalk.exe will
be used in
this demonstration.
Using SNMPWalk.exe:
1. Start SNMPWalk.exe and type in the IP address of the remote server and the
community string for SNMP.
2. Click Scan.
3. After completing the scan, save the SNMP walk in a text file.
4. Open the text file and manually search for the OIDs.
If the Remote Server does not respond on this OID, the Hardware Monitoring
Agent software may not be properly configured. Check to see if the Hardware
Monitoring Agent software has imported the correct MIBs as outlined in the
following table.
For information on verifying Hardware Monitoring Agent software, see "Accessing
Hardware Monitoring Agent Software" on page 836.
HP
Dell
IBM
CPQSTDEQMIB
MIB-Dell-10892
IBM-SYSTEM-HEALTH-MIB
CPQSINFOMIB
StorageManagementMIB
IBM-SYSTEM-ASSETID-MIB
1340
Troubleshooting a WMI Node
CPQIDA-MIB
IBM-SYSTEM-LMSENSOR-MIB
CPQHLTHMIB
IBM-SYSTEM-MEMORY-MIB
CPQSTSYSMIB
IBM-SYSTEM-POWER-MIB
CPQIDE-MIB
IBM-SYSTEM-PROCESSOR-MIB
IBM-SYSTEM-RAID-MIB
ADAPTEC-UNIVERSALSTORAGE-MIB
Troubleshooting a WMI Node
The following conditions must be met before you can proceed troubleshooting
WMI nodes:
l
l
l
The node has successfully been added via WMI.
WMI is working properly on the remote server.
The Hardware Monitoring Agent software is installed on the remote server
and running.
Using Wbemtest.exe to troubleshoot WMI:
1. Open wbemtest.exe, usually located at C:\Windows\System32\wbem\wbemtest.exe.
2. Connect from the problematic node (either the SAM server or the additional
poller server) to the remote server using wbemtest.exe.
3. Click Connect.
4. In the Namespace field enter:
For IBM and HP enter: \\RemoteServerIpAddress\root
For Dell enter: \\RemoteServerIpAddress\root\cimv2
1341
Troubleshooting
5. Enter administrator credentials.
6. Click Connect.
7. Once connected, click Query… from the main screen. The Query dialog
appears.
1342
Troubleshooting a VMWare Node
8. Enter: select * from __Namespace
9. Replace Namespace with the following:
l For HP nodes, replace Namespace with HPQ
l For Dell nodes, replace Namespace with Dell
l For IBM nodes, replace Namespace with IBMSD
10. If the proper Namespace is found, connect to this Namespace.
l \\RemoteServerIpAddress\root\IBMSD for IBM.
l \\RemoteServerIpAddress\root\HPQ for HP.
l \\RemoteServerIpAddress\root\cimv2\Dell for Dell.
11. Run a Query for specific information:
Select Manufacturer, Model, SerialNumber from CIM_Chassis
12. If the test was not successful, re-install the platform or Hardware Monitoring
Agent software on the remote server with the latest release.
Troubleshooting a VMWare Node
VMWare nodes can be polled for Hardware information either through the
vCenter or directly by using the CIM protocol. Polling through the vCenter uses
1343
Troubleshooting
VMWare's native API interface. Polling the ESX server directly uses the CIM
protocol to get Hardware information.
To determine if a node is polled through the vCenter or directly:
1. From the web console, navigate to Settings > Virtualization Settings
2. Listed will be table of all the currently polled VMWare nodes. This table
contains the Polling Through column. Note: This column may be hidden. If
the column is hidden, unhide it by clicking the dropdown menu of an adjacent column and check the Polling Through option:
3. Use the illustration below to determine how your VMWare is being polled.
1344
Troubleshooting AppInsight for Exchagne
Troubleshooting AppInsight for Exchagne
The following sections will help you identify and correct AppInsight for Exchange
errors concerning configuration and performance counters:
l
l
Troubleshooting Error Codes in AppInsight for Exchange
Troubleshooting Exchange Performance Counters
Troubleshooting Permissions
The following information will help you troubleshoot issues with the following:
l
l
l
l
Non-Domain Accounts
Adding Local Administrative privileges to Active Directory Account
Exchange Access
Mailbox Exchange Access
Non-Domain Account
Local accounts (Non-Domain) cannot access Exchange Management interfaces
and therefore are not supported by AppInsight for Exchange. Please select an
Active Directory account or create a new one to use with AppInsight for
Exchange.
Add Local Administrative privileges to Active Directory Account
1. On the server where you want to grant local administrative privileges, open
a Computer Management console.
2. Navigate to System Tools > Local Users and Groups > Groups and then
double click the Administrators group.
3. Click Add and type in the Active Directory username of the account you
want to grant administrative privileges to, and then press Enter. (Ensure the
location is set to either the domain where the account is located or Entire
Directory.)
4. Click Apply and then click OK.
5. Alternatively, you can add an Active Directory group to the Local Administrators group and then add the AD user accounts to that group.
6. To verify the account and local group membership has been configured
properly, run the following in a PowerShell session:
$Recurse = $true
$GroupName = 'Administrators'
Add-Type -AssemblyName System.DirectoryServices.AccountManagement
$ct = [System.DirectoryServices.AccountManagement.ContextType]::Machine
1345
Troubleshooting
$group = [System.DirectoryServices.AccountManagement.GroupPrincipal]::FindByIdentity
($ct,$GroupName)
$LocalAdmin = $group.GetMembers($Recurse) | select @{N='Domain'; E=
{$_.Context.Name}}, samaccountName, @{N='ObjectType'; E={$_.StructuralObjectClass}} -Unique
$LocalAdmin = $LocalAdmin | Where-Object {$_.ObjectType -eq "user"}
Exchange Access
Granting Least Privilege access to the Exchange Organization can be
accomplished using Active Directory Users and Computers (ADUC)
1. From the Start Menu open ADUC and navigate to the Microsoft Exchange
Security Groups OU.
2. Double click on the View-Only Organization Management group.
3. After the window opens, click the Members tab and then click Add.
4. Type the username of the account you want to grant access to the
Exchange organization, and then click OK.
1346
Troubleshooting Permissions
5. Click Apply and then click OK .
6. Close the ADUC window.
Access can also be granted using the Exchange Management
Shell with the following command:
${USER} = Username of service account
Add-RoleGroupMember -Identity "View-Only Organization
Management" -Member "$USER"
To verify the management role is properly assigned, use the
following commands:
Get-RoleGroupMember -Identity "View-Only Organization
Management" | Where-Object {$_.SamAccountName -eq
"$USER"}
Get-RoleGroupMember -Identity "Organization Management"
| Where-Object {$_.SamAccountName -eq "$USER"}
or
Get-ManagementRoleAssignment -RoleAssignee “${USER}” |
Where-Object {$_.RoleAssigneeName -eq "View-Only
Organization Management" -or $_.RoleAssigneeName -eq
"Organization Management"}
Mailbox Search Access
Mailbox Search access is required to determine attachment counts and sizes. It
can be granted using the Exchange Management Shell (EMS).
1. From the Start Menu, open the EMS.
2. Type: New-ManagementRoleAssignment -Role "Mailbox Search" -User
"${USER}" and press Enter.
To verify the management role has been properly assigned, use
the following command:
Get-ManagementRoleAssignment -RoleAssignee “${USER}” Role "Mailbox Search" | Where-Object {$_.RoleAssignmentDelegationType -eq "Regular"}
Note: Exchange Management Roles can be assigned to role
assignees using either regular or delegating role assignments.
l
Regular role assignments enable the role assignee to access
the permissions provided by the management role entries on this
1347
Troubleshooting
l
role.
Delegating role assignments give the role assignee the ability to
assign this role to Role Groups, Users, or Universal Security
Groups.
For more information, see:
l
l
http://technet.microsoft.com/en-us/library/dd876958(v=exchg.150).aspx
http://www.powershellmagazine.com/2013/05/23/pstip-retrieve-group-membership-of-an-active-directory-group-recursively/
Troubleshooting Exchange Performance Counters
Occasionally, you may encounter an Exchange server which is missing some of
the expected performance counters. If this happens, you need to verify whether
the counters are simply disabled or if they are completely missing. The simplest
way to do this is through the registry.
1. Navigate to the service with the missing performance counters.
Note: Individual services are listed under HKLM\SYSTEM\CurrentControlSet\Services.
2. Expand the service and click on the Performance key. The important values we want to ensure exist are listed below and displayed in Figure 1.
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\<ServiceName>\Performance]
Example: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESE\Performance]
"Close"="ClosePerformanceData"
"Collect"="CollectPerformanceData"
"Library"="<Path to performance counter DLL file>"
Example: "Library"="C:\\Program Files\\Microsoft\\Exchange
Server\\V15\bin\\perf\\%PROCESSOR_ARCHITECTURE%\\eseperf.dll"
"Open"="OpenPerformanceData"
"PerfIniFile"="<Name of performance counter INI file>"
Example: "PerfIniFile"="eseperf.ini"
The "Library" file path is typically "C:\Program
Files\Microsoft\Exchange
1348
Troubleshooting Exchange Performance Counters
Server\%ExchangeVersion%\Bin\perf\%Processor_
Architecture%\%DLLFileName%"
Figure 1.
3. Verify the counters have not been disabled by expand the service and then
clicking on the "Performance" key.
4. Check for the value Disable Performance Counters (See Figure 2.) If this
value exists, ensure the data value is 0. Any other value will disable the
counters.
5. Once the value is confirmed to be set to 0, close all PerfMon windows and
then reopen them.
Note: The performance counters should be visible at this time. If the performance counters are not visible, proceed to the next step.
Figure 2.
1349
Troubleshooting
6. If the values First Counter, First Help, Last Counter, and Last Help are listed (See Figure 3), it is highly recommended to unload the performance
counters prior to reloading them.
Figure 3.
To Unload Performance Counters: (See Figure 4)
1. Close all PerfMon windows and stop any services which may be using
these counters.
2. Open the Exchange Management Shell (EMS) in the Run as Administrator
context.
3. Type: Add-PSSnapin Microsoft.Exchange.Management.PowerShell.Setup
and then press Enter.
4. Type: Remove-PerfCounters -DefinitionFileName "<Path to counter
definition XML file> and then press Enter.
a. The default location for Exchange counter definition files is: C:\Program Files\Microsoft\Exchange Server\%ExchangeVersion%\Setup\perf\%XMLFileName%
To Reload Performance Counters: (See Figure 4)
1. Close all PerfMon windows and stop any services which may be using
these counters.
2. Open the Exchange Management Shell (EMS) in the Run As Administrator
context.
3. Type: Add-PSSnapin Microsoft.Exchange.Management.PowerShell.Setup
and then press Enter.
4. Type: New-PerfCounters -DefinitionFileName "<Path to counter definition XML file> and then press Enter.
1350
Troubleshooting Error Codes in AppInsight for Exchange
a. The default location for Exchange counter definition files is: C:\Program Files\Microsoft\Exchange Server\%ExchangeVersion%\Setup\perf\%XMLFileName%
5. Check the application log to verify the counters were properly loaded and
no PerfLib errors exist. Reopen PerfMon to ensure the counters are available.
Figure 4.
Troubleshooting Error Codes in AppInsight for Exchange
Following are the possible error messages associated with AppInsight for
Exchange. Included are possible causes and solutions.
Configuration Errors
Error Message, Information, and Remediation
Error message: Remote configuration was unsuccessful due to the following:
"The deployment service executable was not found on the Orion server." For
details, see the log on the Orion server: ([ALLUSERSPROFILE]
\ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log).
Description: Internal error. The Remote Installation Service.exe file was not
found. (Default file location: C:\Program Files (x86)\SolarWinds\Orion\APM).
This can be caused by an incorrect SAM installation or Remote Installation
Service.exe was deleted or modified.
Remediation: Add the Remote Installation Service.exe file to the following
folder: C:\Program Files (x86)\SolarWinds\Orion\APM.
1351
Troubleshooting
Error message: Remote configuration was unsuccessful due to the following:
"The configurator executable was not found on the Orion server." For details,
see the log on the Orion server: ([ALLUSERSPROFILE]
\ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log).
Description: Internal error. SolarWinds.APM.RemoteWinRmConfiguratorFull.exe
file was not found (Default file location: C:\Program Files (x86)
\SolarWinds\Orion\APM). This can be caused by an incorrect SAM installation or
SolarWinds.APM.RemoteWinRmConfiguratorFull.exe was deleted or modified..
Remediation: Add SolarWinds.APM.RemoteWinRmConfiguratorFull.exe to the
following folder: C:\Program Files (x86)\SolarWinds\Orion\APM.
1352
Troubleshooting Error Codes in AppInsight for Exchange
Error message: Remote configuration was unsuccessful due to the following:
"Access denied." For details, see the log on the Orion server:
([ALLUSERSPROFILE]
\ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log).
Description: The provided user account does not have access to the
Administrator share on the remote Exchange server.
Remediation: Connect to the Administrator share with
\\ExchangeComputerName\<drive letter>$\ from the Orion server.
1353
Troubleshooting
Error message: Remote configuration was unsuccessful due to the following:
"The configurator executable has an invalid signature." For details, see the log
on the Orion server: ([ALLUSERSPROFILE]
\ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log).
Description: The configuration executable cannot be started on the remote
Exchange server due to any of the following:
n
n
n
The high level of User Account Control settings
The provided user account does not have privileges high
enough to run *.exe files
SolarWinds.APM.RemoteWinRmConfiguratorFull.exe is not
signed with a Solarwinds certificate.
Remediation: n
Reduce User Account Control settings on the remote
Exchange server;
1354
Troubleshooting Error Codes in AppInsight for Exchange
n
Ensure the user's account has privileges to run *.exe files
Error message: Remote configuration was unsuccessful due to the following:
"The network path was not found." For details, see the log on the Orion server:
([ALLUSERSPROFILE]
\ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log).
Description: This is a Windows error indicating that a networking component is
malfunctioning.
Remediation: l
l
Ensure you have access to the Shared Storage drive.
1. In Windows Explorer, navigate to My Network Places > Entire
Network > Microsoft Windows Network > (workgroup name) >
(Shared Storage drive name).
Firewalls can block traffic to the network which could generate this message. Try temporarily disabling any software or hardware firewalls to isolate the problem.
1355
Troubleshooting
l
Anti-spyware tools can block network traffic. Try temporarily disabling any
anti-spyware programs and restart the computer to isolate the problem.
Error message: Remote configuration was unsuccessful due to the following:
"Executable configuration file was not found on the Orion server." For details,
see the log on the Orion server: ([ALLUSERSPROFILE]
\ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log).
Description: Internal error.
was not found
(Default file location: C:\Program Files (x86)\SolarWinds\Orion\APM.) This
can be caused by an incorrect SAM installation or
SolarWinds.APM.RemoteWinRmConfiguratorFull.exe.config was deleted or
modified.
SolarWinds.APM.RemoteWinRmConfiguratorFull.exe.config
Remediation: Add SolarWinds.APM.RemoteWinRmConfiguratorFull.exe.config
to the following location: C:\Program Files (x86)\SolarWinds\Orion\APM.
1356
Troubleshooting Error Codes in AppInsight for Exchange
For information about Microsoft error codes, see: http://msdn.microsoft.com/enus/library/cc231199.aspx
1357
Troubleshooting
Error message: Remote configuration was unsuccessful due to the following:
"PowerShell 2.0 was not detected on the Exchange server." Learn how to
correct this. For details, see the log on the remote computer. ([ALLUSERSPROFILE]
\ProgramData\Solarwinds\Logs\APM\RunWinRMConfigurator.log).
Description: PowerShell 2.0 is not installed on the remote Exchange server.
Remediation: PowerShell 2.0 should be installed on Exchange server.
Installing PowerShell 2.0 on Server 2008:
For Server 2008, download the installation file for PowerShell 2.0 at
http://support.microsoft.com/kb/968929/en-us and install it on the
server.
Note: PowerShell 2.0 is automatically installed on Server 2008 R2
and therefore no additional installation is required.
Installing PowerShell 2.0 on Server 2012:
1. Open Server Manager.
1358
Troubleshooting Error Codes in AppInsight for Exchange
2. Click on the Manage menu, and the select Add Roles and
Features.
3. After the wizard opens, click Next until you get to the Installation Type page.
4. Select Role-based or feature-based installation.
5. Click Next until you reach the Features page.
6. Scroll down to Windows PowerShell. It will likely show itself
as partially installed (square inside box).
7. Check the box next to Windows PowerShell 2.0 Engine.
8. Click Next and then Install.
9. When the installation finishes, click Close.
Error message: Remote configuration was unsuccessful due to the following:
"This account must be an Active Directory account with local administrative
1359
Troubleshooting
privileges." Learn how to correct this. For details, see the log on the remote
computer. ([ALLUSERSPROFILE]
\ProgramData\Solarwinds\Logs\APM\RunWinRMConfigurator.log).
Description: The provided user account does not have Local Administrative
privileges.
Remediation:
Add Local Administrative privileges to Active Directory Account:
1. On the server where you wish to grant local administrative privileges,
open a Computer Management console.
Note: On Windows 2012, add this privilege using the Active Directory
console.
2. Navigate to System Tools > Local Users and Groups > Groups and
double click the Administrators group.
3. Click Add and type in the Active Directory username of the account you
want to grant administrative privileges, and then press Enter. (Ensure the
location is set to either the domain where the account is located or Entire
Directory.)
4. Click Apply and then click OK.
Note: Alternatively, you can add an Active Directory group to the local
administrators group and add the Active Directory user accounts to that
group.
Error message: Remote configuration was unsuccessful due to the following:
"This account must be an Active Directory account with organization wide
Exchange access (View-Only-Organization Management and Mailbox Search
Management Role)." Learn how to correct this. For details, see the log on the
remote computer. ([ALLUSERSPROFILE]
\ProgramData\Solarwinds\Logs\APM\RunWinRMConfigurator.log).
Description: The provided user account does not have View-Only-Organization
Management or Mailbox Search Management privileges.
Remediation:
Grant View-Only-Organization Management privileges:
Granting Least Privilege access to the Exchange Organization can be
accomplished using Active Directory Users and Computers (ADUC). To
accomplish this, take the following steps:
1360
Troubleshooting Error Codes in AppInsight for Exchange
1. From the Start Menu, open ADUC and navigate to the Microsoft
Exchange Security Groups OU.
2. Double click on the View-Only Organization Management group.
3. After the window opens, click the Members tab and then click Add.
4. Type the username of the account you want to grant access to the
Exchange organization, and then click OK.
5. Click Apply, and then click OK.
6. Close the ADUC window.
Grant Mailbox Search Access:
Mailbox Search access is required to determine attachment counts and sizes.
This can be granted using the Exchange Management Shell (EMS).
1. From the Start Menu, open the EMS.
2. Type: New-ManagementRoleAssignment -Role "Mailbox Search" User <Username of account being granted access> and then press
Enter.
3. To verify the management role has been properly assigned, enter the
following command:
Get-ManagementRoleAssignment -RoleAssignee <Username of
account>
1361
Troubleshooting
1362