Spotlight on SQL Server Getting Started Guide
Transcription
Spotlight on SQL Server Getting Started Guide
Spotlight® on SQL Server Enterprise 10.5 Getting Started Guide © 2014 Quest Software, Inc. ALL RIGHTS RESERVED. This guide contains proprietary information protected by copyright. The software described in this guide is furnished under a software license or nondisclosure agreement. This software may be used or copied only in accordance with the terms of the applicable agreement. No part of this guide may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose other than the purchaser’s personal use without the written permission of Quest Software, Inc. The information in this document is provided in connection with Quest products. No license, express or implied, by estoppel or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Quest products. EXCEPT AS SET FORTH IN QUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR INABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves the right to make changes to specifications and product descriptions at any time without notice. Quest does not make any commitment to update the information contained in this document. If you have any questions regarding your potential use of this material, contact: Quest Software World Headquarters LEGAL Dept 5 Polaris Way Aliso Viejo, CA 92656 email: [email protected] Refer to our Web site (www.quest.com) for regional and international office information. Patents Protected by U.S. Patent #6,901,582. Additional patents pending. Trademarks Quest, Quest Software, the Quest Software logo and Spotlight are trademarks and registered trademarks of Quest Software, Inc in the United States of America and other countries. For a complete list of Quest Software’s trademarks, please see http://www.quest.com/legal/trademark-information.aspx. Other trademarks and registered trademarks are property of their respective owners. Spotlight on SQL Server 10.5 Getting Started Guide Friday, 30 May 2014 Table of Contents Use Spotlight on SQL Server 5 Diagnose CPU 5 Diagnose IO 6 Diagnose Blocking 8 Monitor | Playback 10 Identify Expensive SQL 12 Turn on and configure SQL Analysis 12 Real-Time Data and Expensive SQL 14 Data Over a Selected Time Frame and Expensive SQL 15 Filter Displayed and Collected Data 16 Report on Database Growth 18 Analyze Wait Statistics 20 Find Long Running SQL 21 Configure the Virtualization Overhead Guage 22 VMware - Connection Properties | Details 23 Troubleshoot Spotlight on SQL Server 24 Installation Issues Remote installation of the diagnostic server fails Connection Issues Problems connecting to a SQL Server Instance 24 24 24 24 WMI Errors 25 Miscellaneous Issues 28 Monitor Multiple SQL Server Instances 28 Unable to perform certain operations 28 Spotlight on SQL Server - Missing Performance Counters 29 Spotlight on SQL Server - SQL Activity Drilldown | Sessions page 29 Spotlight on SQL Server Enterprise Getting Started Guide 4 Table of Contents Playback 30 Appendix: Contact Quest 31 Contact Quest Support 31 Spotlight Support 31 Contact Quest Software 32 About Quest Software 32 Index 33 1 Use Spotlight on SQL Server Diagnose CPU Spotlight continually monitors CPU usage on the target server and alarms if it crosses a threshold. The thresholds are: CPU Usage Alarm severity 0 – 67% Normal 67 – 80% Low 80 – 93% Medium 93 – 100% High A CPU alarm indicates that a bottleneck on CPU is likely. This bottleneck could exist for a number of reasons – expensive queries could be running, excessive compilation or recompilation could be occurring, or a process on the server, not even related to SQL Server, is ‘hogging’ the CPU. With Spotlight, you can investigate the causes of a bottleneck both in real-time and in the immediate past. When Spotlight detects that there is a bottleneck on the CPU, an alarm is raised. You can see alarms on the Spotlight home page, and in Spotlight Today. Spotlight on SQL Server Enterprise Getting Started Guide 6 Use Spotlight on SQL Server Our first order of business is to find out what process is using all of the CPU. From the Spotlight Home Page From Spotlight Today Click Diagnostics. Select the alarm and click Diagnose. This opens the Diagnostics drilldown. The Diagnostics drilldown helps you answer questions such as: l Why is CPU usage higher than usual? l Is the SQL Server causing high CPU usage? l If it is the SQL Server, is it optimizing queries or executing them? The first grid on the CPU page shows possible causes of CPU pressure and the likelihood that each is contributing to high CPU usage on the server being monitored. Click an item in the grid to display an explanation of the diagnosis and supporting information. Diagnose IO Sustained high I/O wait time rates are a good indicator that of a disk subsystem bottleneck and that I/O device service times will be degraded. When Spotlight detects unusually high disk activity on the SQL Server machine it raises an alarm. Spotlight looks at SQL Server I/O wait times to determine if there is an IO bottleneck. A value larger than 15 ms/IO indicates a possible bottleneck. The thresholds for the I/O wait time alarm are: Spotlight on SQL Server Enterprise Getting Started Guide 7 Use Spotlight on SQL Server I/O Wait Time Alarm severity 0 - 15 Normal 15 - 50 Medium 50 - infinity High After detecting a potential disk bottleneck, Spotlight displays an alarm on the home page and in Spotlight Today. The first thing we need to do is identify the process that is generating high amounts of I/O activity. From the Spotlight Home Page From Spotlight Today Click Diagnostics | IO. Select the alarm and click Diagnose. This opens the I/O Diagnostics drilldown. The I/O Diagnostics drilldown helps you answer questions such as: l Why is the I/O higher than usual? l Is the high I/O due to paging activity? l Is the SQL Server causing the high I/O? l If the SQL Server is causing the high I/O, is it through query executions, scan operators, PAGE_IO_LATCH or LOG waits? Spotlight on SQL Server Enterprise Getting Started Guide 8 Use Spotlight on SQL Server The first grid on the I/O page shows the possible causes of high I/O and the likelihood of each contributing to this instance of the alarm. Click an item in the grid to display an explanation of the diagnosis and supporting information. Diagnose Blocking Blocking in SQL Server occurs when a session that was performing some task is unable to progress because it must wait on a resource that is currently being used by another session. The resource that is being waited on can be either a physical structure like a lock on a table or an internal SQL Server resource such as a latch. Excessive blocking can be a major cause of poor application performance since it reduces the throughput of the system. Often a user of an application does not realize that they are waiting on a resource held by another user. From their point of view, it often seems like their application has stopped responding. When diagnosing blocking, you want to start your investigation by answering the following questions: l Who is waiting on what? l How long have they been waiting? l What SQL was running while they were waiting? Spotlight not only alerts you to blocking issues in your database but also helps you answer the above questions so you can quickly resolve the problem causing the blocking. Spotlight on SQL Server Enterprise Getting Started Guide 9 Use Spotlight on SQL Server When Spotlight detects a blocking issue, an alarm is displayed on the home page. The first thing we want to do is look at the Blocking drilldown. The Blocking drilldown shows details about the current blocks and in particular, answers the questions stated above. On the Spotlight homepage, click the Blocked Processes button, and then click the “Blocking” drilldown. The Blocking grid shows the sessions that are either blocked or are causing other sessions to be blocked. The hierarchy in the tree diagram makes it easy to see which session is causing the blocking. In addition, the resource on which the sessions are waiting is displayed. You can ask Spotlight to resolve these to actual object names by clicking on the ‘?’ button. The time spent waiting and the SQL last executed are also displayed. Spotlight on SQL Server Enterprise Getting Started Guide 10 Use Spotlight on SQL Server From here, you can view session details. 1. Select the entry in the grid. 2. Click View Session Details. Monitor | Playback You can view events and data collected in the recent past as though they were happening in real-time. You can do this via the Playback ribbon. To playback time 1. Select a connection from the Spotlight browser. 2. Select the Monitor ribbon tab. 3. Click to show: Alarms Description Playback Select an alarm or moment in time in the past to return to. Once the time is selected, the Spotlight home page, Spotlight Today and page drilldowns show details from that time identical in format to that displayed in "live" state, so you can view historical data in the same way as you view a live connection. Tip: Click Real Time to return to the present time. Note: Playback / Historical data is not displayed for SQL Azure database information. Spotlight on SQL Server Enterprise Getting Started Guide 11 Use Spotlight on SQL Server Alarms Description Control Description Select the date. Use the back and forward buttons to change the date. Click the time to playback to. The alarms list scrolls to show those alarms occurring at any given time. Select to see the behavior at the time when an important alarm was raised. Note: The time scale is colored according to the most severe alarm for the connection. Tip: Press and hold to magnify the time scale. The alarms occurring on the selected date. Click an alarm to playback to the time the alarm occurred. Note: The color indicates the severity of the alarm. Real Return to present time the Spotlight home page and page drilldowns. Time Rewind Go back in time on the Spotlight home page or page drilldown. Click the associate arrow to define how far back to travel: 1 minute, 5 minutes, 10 minutes, 1 hour or 1 day. Skip Starting from the past, skip forward in time on the Spotlight home page or page drilldown. Click the associate arrow to define how far forward to skip: 1 minute, 5 minutes, 10 minutes, 1 hour or 1 day. Spotlight on SQL Server Enterprise Getting Started Guide 12 Use Spotlight on SQL Server Alarms Description Play Starting from the past, step forward in time through the alarms in sequence. Show on the Spotlight home page or page drilldown. Click the associate arrow to define the speed of play. Tip: On the Spotlight Home Page you can view the recent history of a single component. Identify Expensive SQL Identifying expensive SQL can be a time consuming process. Use the SQL Analysis feature in Spotlight to identify SQL that is consuming resources on your server. With SQL Analysis, you can quickly answer questions such as: l What SQL has executed the most? l What SQL consumed the most CPU or I/O in total? l What SQL consumed the most Average CPU or I/O? l What SQL took to longest time to execute? Answering these questions will help you to direct your tuning efforts more productively. SQL Analysis uses either ROWSET or sampling Server Side trace to collect raw data on SQL statement execution. It then aggregates this data by similar SQL statements so that potentially thousands or even millions of rows of raw data are distilled down into an aggregate picture of workload. You can set filters at point of collection so that only the events that you are interested in are collected and when viewing the data, you can also filter the data to narrow the focus down even further. Turn on and configure SQL Analysis 1. Click Configure | SQL Analysis 2. Choose one of the following from the connection list: l l To change the setting for an individual connection, select the connection name. l Select Override the default settings. l Select Enable SQL Analysis. To change the setting for all connections, select Default Settings and select Override the default settings. Spotlight on SQL Server Enterprise Getting Started Guide 13 Use Spotlight on SQL Server 3. Click Data Collection. 4. Configure the following fields on the Data Collection page: Option Description SQL Analysis is scheduled to collect every n minutes Note: This feature is available only in Spotlight on SQL Server Enterprise. How often Spotlight collects SQL Analysis data. The default is six hours. To change how often Spotlight collects SQL Analysis data a. Click the link. b. Select Override the default settings for the collection 'SQL Analysis'. c. Under ‘Store reporting data in the Spotlight Statistics Repository’, click the Store data link. d. Set the collection schedule. Data can be collected at regular intervals or at a specific time on one or more days. Use continuous Rowset trace Select to use Rowset trace to collect SQL Analysis data. Use sampling Server Side trace Select to use sampling Server Side trace to collect SQL Analysis data. Rowset trace is convenient though may affect throughput on busy servers. It should be used with care. Server Side trace is more suitable for busy servers than Rowset trace. Server Side trace requires additional configuration. Run trace for or until trace file reaches Server Side trace will run until one of these conditions is met. Specify the maximum amount of time Server Side trace should run for. Specify the maximum size of the trace file. Server trace file location (on the SQL Server host) Type a location for the server trace file. The path is relative to the SQL Server instance. Retrieve data through SQL Server Select to process trace files on the SQL Server instance. This option has the potential to affect throughput on busy servers. For example, C:\Trace Files. Spotlight on SQL Server Enterprise Getting Started Guide 14 Use Spotlight on SQL Server Option Description Use this option when there are permission or firewall settings preventing the Spotlight Diagnostic Server machine from retrieving files from the SQL Server machine. This is the default option. Retrieve data from the file system Select to process trace files on the Spotlight Diagnostic Server. The Spotlight Diagnostic Server machine must have the appropriate permissions required to connect to the SQL Server instance. This option is not available when configuring default settings. Trace file location (from Diagnostic Server host) Type the location of the server trace files on the SQL Server instance. The location should be a shared folder the Spotlight Diagnostic Server can connect to. For example, \\computername\Trace Files. This option is not available when configuring default settings. Real-Time Data and Expensive SQL You can use the SQL Analysis grid when load testing in a testing environment. It shows SQL Analysis data in real time. If you are trying to answer the question “I am simulating application workload in my test environment and I want to look at a breakdown of SQL statements in real-time. I am interested in what SQL statements are consuming the most average CPU right now.”, you would want to look at an aggregate of SQL statement executions (workload) in real-time. The SQL Analysis grid in the SQL Activity drilldown allows you to do exactly that. 1. Select a SQL Server connection from the Spotlight browser. 2. Click Monitor | SQL Activity. 3. Select SQL Analysis. 4. Do the following: To identify Sort the grid by SQL that consumed the most CPU Total CPU Spotlight on SQL Server Enterprise Getting Started Guide 15 Use Spotlight on SQL Server To identify Sort the grid by The average CPU consumed by a particular statement Average CPU SQL that was executed the most Execution Count SQL with the highest logical I/O Average Reads Tips: l Use filters to refine the data shown in the grid. See "Filter Displayed and Collected Data" (page 16) for more information. l Some columns are hidden by default. To view hidden columns, right-click the grid header row and select Organize Columns. l Results are cleared when you leave the SQL Analysis grid. Click Clear Grid to clear the grid manually. l To wrap a column onto multiple lines, right-click the data content of the grid and select Properties | Options. Note: Word wrap may degrade the performance of Spotlight. l If Quest SQL Optimizer is installed, you can use it to tune the non-conforming SQL that has been identified. To do this, click Optimize SQL. Data Over a Selected Time Frame and Expensive SQL Note: This feature is available only in Spotlight on SQL Server Enterprise. The SQL Analysis - Workload view shows the aggregate workload over a selected time period. Use the SQL Analysis - Workload view when diagnosing slow response times. If you are trying to answer the question “What SQL statement consumed the most CPU over month end processing?”, you would want to look at an aggregate of SQL statement executions (workload) over a time range. The SQL Analysis – Workload view allows you to do exactly that. 1. Select Reporting and Trending | Views | SQL Server Performance | SQL Analysis Workload in the Spotlight Browser. Spotlight on SQL Server Enterprise Getting Started Guide 16 Use Spotlight on SQL Server 2. Click Report | Select Connections and select at least one connection. 3. Select a Time Range. 4. Click Refresh. 5. Do the following: To identify Sort the grid by SQL that consumed the most CPU Total CPU The average CPU consumed by a particular statement Avg CPU SQL that was executed the most Execution Count SQL with the highest logical I/O Avg Reads Tip: Use filters to refine the data shown in the grid. See "Filter Displayed and Collected Data" (page 16) for more information. Filter Displayed and Collected Data To filter the data displayed Start from the data displayed in the SQL Analysis grid or the SQL Analysis Workload view Spotlight on SQL Server Enterprise Getting Started Guide 17 Use Spotlight on SQL Server SQL Analysis grid SQL Analysis Workload view 1. Click Change Filter. 1. Click Filter Data. 2. Select Filter results. 2. Select Filter results. To add a filter to the list 1. Click Add. 2. From the Column list, select the column you want to base your filter on. 3. From the Condition list, select a condition to apply to the column. 4. In the Value field, type a value to filter events by. You can use the percent sign (%) wildcard with the is like/is not like condition. Tips: l You cannot use wildcard characters for the Database name when connected to a SQL Server 2000 instance. l To turn off filtering, clear the Filter results checkbox. l To edit or delete a filter select it in the filter list and click the appropriate button. To filter the data collected 1. Click Configure | SQL Analysis. 2. Choose one of the following from the connection list: l To change the setting for an individual connection, select the connection name. l Select Override the default settings. Select Enable SQL Analysis. l To change the setting for all connections, select Default Settings and select Override the default settings. 3. Ensure the Filters page is selected. l Click Add. Tips: Spotlight on SQL Server Enterprise Getting Started Guide 18 Use Spotlight on SQL Server l You can use the percent sign (%) and underscore (_) as wildcards. l You cannot use wildcard characters in the Database name field when connected to a SQL Server 2000 instance. l When filtering on a string field such as DatabaseName or ApplicationName, use only the "LIKE" and "NOT LIKE" operators. The other operators apply only to numeric values. Tips: l Use the Arrow buttons to change the order of the filters. l Edit or delete a filter by selecting it in the filter list and clicking the appropriate button. l Filtering affects the average, minimum, maximum, and last values displayed in the grid, as calculations are performed only on data retrieved from the SQL Server instance. l To turn off filtering, on the Filters page, click Clear. This removes all filters. 4. You can specify the number and type of SQL Executions retrieved from the SQL Server instance a. Select the Advanced page. b. In the Aggregate the top n SQL statements field, enter the maximum number of SQL executions to be retrieved from the SQL Server instance. c. From the Sorting by list, specify the criterion you want Spotlight to use to determine what the "top" SQL statements are. Choose from Average Duration, Average CPU, or Average IO. Spotlight aggregates the data collected and then stores only the aggregated data. Report on Database Growth Note: This feature is available only in Spotlight on SQL Server Enterprise. To report on database growth across multiple SQL servers, use the Large Databases view in Spotlight Reporting and Trending. The Large Databases view shows the largest databases on the selected SQL Servers. With Spotlight, you can view this data in a number of different ways. 1. Select Reporting and Trending | Views | SQL Server Environment | Large Databases in the Spotlight Browser. Spotlight on SQL Server Enterprise Getting Started Guide 19 Use Spotlight on SQL Server 2. Click Report | Select Connections and select at least one connection. 3. Select a Time Range. 4. Click Refresh. By default, the data is shown as a bar chart, listing each database on the selected servers in order of size. You can change the way the data is displayed using the Chart View button, located above the chart. For example, if you wanted to view the growth of your largest databases over time you can change the chart to a line graph. To do this: Spotlight on SQL Server Enterprise Getting Started Guide 20 Use Spotlight on SQL Server 1. Click the Chart View button and then select Line from the list. 2. Click Refresh. To look at the space consumption proportionally you could use a pie chart. Click Chart View | Pie. You can use the data and chart from the view in your own reports by exporting the view. To do this, click Export and choose a location to save the files. Spotlight saves the chart as a JPG file and the grid as a CSV file. Analyze Wait Statistics When troubleshooting SQL Server performance issues, one area you can focus your investigation on is waits. Long wait times can be an indicator of performance bottlenecks so identifying where waits are occurring may assist you in your tuning efforts. Spotlight not only provides an easy way to see if your SQL server has a problem with waits but also identifies on which particular resource waits are occurring. Spotlight takes the raw data from SQL Server and automatically calculates the rate of wait over time so you have a more current perspective. Spotlight on SQL Server Enterprise Getting Started Guide 21 Use Spotlight on SQL Server To start your investigation into waits, go to the SQL Activity drilldown and click the Wait Statistics tab. The Waits Detail grid shows detailed waits statistics for the SQL Server instance being monitored. It shows all wait types in SQL Server and breaks down the statistics into signal time and resource wait time. (SQL Server provides only wait time and signal wait time.) By default, the Waits Detail grid is sorted by Wait Time Rate (ms/s) which allows you to immediately see which wait types are being waited on right now. Find Long Running SQL The SQL - Long Running SQL alarm alerts you to SQL that has been running longer than a specified time frame. Because this time frame is unique to each environment, the SQL - Long Running SQL alarm, by default, is not configured. Scenario You work in an OLTP environment and often get complaints about delays in server processing. You want to be notified when SQL on a particular server is running longer than expected so you can investigate what is causing the delay. You configure Spotlight to raise a medium severity alarm when it detects SQL running for longer than five minutes. When the SQL - Long Running SQL alarm is raised, you use the Sessions drilldown to view the long running SQL. Spotlight on SQL Server Enterprise Getting Started Guide 22 Use Spotlight on SQL Server To configure the Alarm: SQL - Long Running SQL 1. Click Configure | Alarms. 2. Select a connection to configure. Select Default Settings to apply the configuration to all connections. 3. Select the alarm SQL - Long Running SQL. 4. Select Override the default settings for the alarm "SQL - Long Running SQL". 5. Click Add Severity and select a severity. Scenario: Click Add Severity and select Medium. 6. Select the check box of the new severity. 7. Copy and paste the text in the Description cell from the Normal severity to the new severity. 8. In the Start cell, type the duration in seconds, of how long the SQL should run for before the alarm is raised. Scenario: In the Start cell, type 300 . Tip: You can also configure alarm severities using keys. In the SQL - Long Running SQL alarm, the key is the SPID. See "Configure Keyed Alarms" in the online help for more information. Configure the Virtualization Overhead Guage The Virtualization Overhead gauge shows the percentage of CPU that is unavailable to this virtual machine because it is being consumed either by other virtual machines or by VMware itself. In a virtual environment, the physical CPU of a host is shared by virtual machines and the VMware Hypervisor. This means there can be multiple processes wanting to use the physical CPU of the host at the same time. The result is that virtual machines may be ready to run but have to wait to be scheduled on a CPU. Vmware call this “ready time”. The Virtualization Overhead gauge shows the amount of ready time as a percentage of the theoretical maximum CPU available to the virtual machine. To see virtualization data on Spotlight Home Pages 1. There needs to be a connection to the VMware Server. See Connections / Recent Connections for more information. 2. In the Spotlight Browser, right-click the Windows connection and select Properties. 3. In the Host field, select the connection to the VMware Server. 4. In the VM Name field, select the name of the virtual machine. Spotlight on SQL Server Enterprise Getting Started Guide 23 Use Spotlight on SQL Server The Virtualization Overhead gauge now displays data on the corresponding Windows Home Page and SQL Server Home Page. Tip: SQL Server connections automatically use the virtualization data from their corresponding Windows connections. You do not need to configure the SQL Server connection once the Windows connection is configured. VMware - Connection Properties | Details To create a connection or edit connection properties, click Configure | Connections. Select VMware - Connection Properties to create a VMware connection to an ESX or VirtualCenter server that hosts a Windows machine. Field Description Address The IP address or hostname of the ESX server or VirtualCenter server that hosts the Windows machine. Tip: If you have a VirtualCenter server then connect to that in preference to the ESX host. That way if the virtual machine migrates from one host to another the VirtualCenter server can still be interrogated for data for the Virtual Machine. User A valid user account on the target system. Note: For Spotlight to monitor a VMware server, the username specified must have at least a read-only role for the ESX server or VirtualCenter server that you want to connect to. Password The password required to connect the specified user to the server. 2 Troubleshoot Spotlight on SQL Server Installation Issues Remote installation of the diagnostic server fails Note: This feature is available only in Spotlight on SQL Server Enterprise. Remote installation of the Diagnostic Server fails if the user has both a domain account and a local account with the same user name. Additionally, if the user uses a domain account that does not have permission to access the remote machine (that is, the account does not have administrator privileges) then the remote installation will not complete. Connection Issues Spotlight on SQL Server retrieves its data from the SQL Server via the Spotlight Diagnostic Server. The Spotlight Diagnostic Server also uses WMI queries to retrieve performance counter information from the Windows server on which SQL Server is running. The Spotlight Diagnostic Server cannot connect to a server unless it can access all of this information. This means you need to have the appropriate permissions on the server. Also, the data must be retrievable over your network. Spotlight on SQL Server requires more than just a specific port TCP/IP connection to SQL Server to do this. The port on the Spotlight Diagnostic Server must be open. For more information, see the Spotlight on SQL Server Deployment Guide. This document is available in PDF from Start | Quest Software | Spotlight. Problems connecting to a SQL Server Instance If you have trouble connecting to a SQL Server instance, check that you can: l Connect to the SQL Server using another tool such as Microsoft's SQL Server Management Studio, or sqlcmd. l Connect to WMI using another tool such as Microsoft's WMI CIM Studio, or wmic. (Troubleshooting WMI) If you have problems with any of these checks, you will need to investigate and resolve these issues before you can connect to that server using Spotlight on SQL Server. Spotlight on SQL Server Enterprise Getting Started Guide 25 Troubleshoot Spotlight on SQL Server WMI Errors The following is a guide to troubleshooting WMI errors. Error Solution The RPC server is unavailable. Spotlight on SQL Server is not able to establish a connection with the target computer. Exception from HRESULT: 0x800706BA Verify the following: l The address of the server is entered correctly via the Spotlight Connection Manager. Verify correct host name or IP address (Configure | Connections). Verify the host is available (not currently shut down). l The Remote Procedure Call (RPC) service is running on the target computer. Verity that "Remote Procedure Call (RPC)" is running and set to auto start after restart. l TCP Port 135 is open to internal traffic on the Spotlight Diagnostic Server and the target computer. WMI opens an undetermined port in addition to port 135. This can be troublesome in a firewalled environment. Specifying a port range for WMI is recommended for this type of environment. See How to limit the number of ports used by WMI for more information. l The target computer is not blocked by the firewall. Either configure WMI to use a fixed ports range (How to limit the number of ports used by WMI) or enable remote administration exception. Follow these steps to enable remote administration exception. On the target computer: 1) Open the Group Policy Object Editor (gpedit.msc), open Computer Configuration | Administrative Templates | Network | Network Connections | Windows Firewall 2) Open either Domain Profile or Standard Profile, depending on which profile you want to configure. 3) Enable the following exceptions: "Allow Remote Administration Exception" and "Allow File and Printer Sharing Exception". http://msdn.microsoft.com/en-us/library/aa389286(VS.85).aspx Access is denied. Exception from HRESULT: l The "TCP/IP NetBIOS Helper" service is running. Verity that "TCP/IP NetBIOS Helper" is running and set to auto start after restart. l The "Windows Management Instrumentation" service is running on the target computer. Verity that "Windows Management Instrumentation" is running and set to auto start after restart. The Windows user specified to monitor the target computer (Configure | Connections) is unknown to the target or does not have administrator rights. See Windows Account Permissions for more information. Spotlight on SQL Server Enterprise Getting Started Guide 26 Troubleshoot Spotlight on SQL Server Error Solution 0x80070005 (E_ ACCESSDENIED) WMI connection time outs WMI query failed: Invalid class. The timeout value defaults to the value DCOM specifies (usually 60 seconds). You can adjust this value via dcomcnfg.exe The WMI class does not exist on the target machine Solution: Recreate the WMI classes. [0x80041010] Windows Connections Either of the following will recreate the WMI classes: l Open a command prompt window and run the following command: wmiadap /f l Use the Microsoft Extensible Counter List (Exctrlst) utility to enable the following counters: Win32_PerfDisk, Win32_PerfNet, Win32_PerfOS and Win32_PerfProc. http://www.microsoft.com/downloads/details.aspx?displaylang=en&familyid=7FF99683B7EC-4DA6-92AB-793193604BA4 Note: A reboot of the target machine is required. SQL Server Analysis Services Connections Either of the following will recreate the WMI classes: l Open a command prompt window on the machine hosting the SQL Server Analysis Services instance. Run the following command: wmiadap /f l Unregister and re-register the WMI classes. To do so: 1. On the problematic machine with SQL Server Analysis Services installed, find out the SQL installation path. See below for the default installation path. Your installation path may be different. SQL Server Analysis Services 2012 C:\Program Files\Microsoft SQL Server\MSAS11.MSSQLSERVER\OLAP\bin\Counters SQL Server Analysis Services 2008 C:\Program Files\Microsoft SQL Server\MSAS10.MSSQLSERVER\OLAP\bin\Counters SQL Server Analysis Services 2005 C:\Program Files\Microsoft SQL Server\MSSQL.2\OLAP\bin Spotlight on SQL Server Enterprise Getting Started Guide 27 Troubleshoot Spotlight on SQL Server Error Solution Note: You are looking for the folder that contains the following files: For unamed instances: MSSQLServerOLAPService perf-MSSQLServerOLAPServicemsmdctr.ini For named instances: MSOLAP$Your_SSAS_NamedInstanceName perf-Your_SSAS_NamedInstanceNamemsmdctr.ini If you need to check the named instance name, use services.msc. 2. At the command prompt, change folder to the installation path. For example, cd C:\Program Files\Microsoft SQL Server\MSAS10.MSSQLSERVER\OLAP\bin\Counters 3. Run the following commands to unload and load counters: unamed instances unlodctr MSSQLServerOLAPService named instances unlodctr MSOLAP$Your_SSAS_NamedInstanceName lodctr perfMSSQLServerOLAPServicemsmdctr.ini lodctr perf-Your_SSAS_ NamedInstanceNamemsmdctr.ini Tip: Replace Your_SSAS_NamedInstanceName with the SQL Server Analysis Services named instance name. If you need to check the named instance name, use services.msc. 4. Run the following command to parse all the performance libraries on the system and refresh the performance counter classes on the problematic machine: wmiadap /f 5. Use wbemtest.exe to verify the WMI Classes exist. WMI query failed: Invalid query. [0x80041017] The columns returned by the WMI class are incomplete or unexpected. Solution: Update the WMI classes by running the following command at the command prompt of the target computer. wmiadap.exe /f See http://msdn.microsoft.com/en-us/library/aa394528(VS.85).aspx for more information. 0x80014064 User credentials cannot be used for local connections This error occurs when wmic command is executed locally on the target machine and credentials specified Spotlight on SQL Server Enterprise Getting Started Guide 28 Troubleshoot Spotlight on SQL Server Error Solution [0x800705af] The paging file is too small for this operation to complete. You may find the following link useful: http://blogs.technet.com/b/kevinholman/archive/2010/06/09/wmi-leaks-memory-on-server-2008r2-monitored-agents.aspx WMI query failed: Out of memory. [0x80041006] 1. At the command prompt run "wbemtest" 2. Connect to the "root" namespace (not "root\default", just "root") 3. Click Open Instance. Specify "__ProviderHostQuotaConfiguration=@" 4. Select Local Only for easier readability. You will see the threshold values. 5. Increase the MemoryPerHost value to something greater. For example, double it (256 MB) 6. Save Property 7. Save Object 8. Click Exit. 9. Restart WMI services. Invalid verb The wmic command has attempted to access a WMI class that does not exist. Solution: Check the spelling of parameters on the wmic command. Invalid Global Switch The specified host, user or domain name contains special characters like '-' or '/'. Solution: Modify the command by adding quotation marks. wmic /node: 'MonHostFQDN' /user: 'DOMAIN\USER' path Win32_PerfRawData_ PerfDisk_LogicalDisk get FreeMegabytes Miscellaneous Issues Monitor Multiple SQL Server Instances Spotlight on SQL Server is designed to monitor a maximum of 100 SQL Servers, Analysis Services or Replication instances and 100 Windows servers. Exceeding this recommended limit on 32 bit environments may result in poor performance or product instability, due to the 1 Gb memory limit. On 64 bit environments theoretically more connections can be monitored as the 1 Gb limit no longer applies, however testing of this has been limited. Unable to perform certain operations If, whilst using the Spotlight on SQL Server client, a message is displayed stating: You are not connected to the Diagnostic Server as a member of the Spotlight Diagnostic Administrators group. Only members of that group can perform the requested operation. Spotlight on SQL Server Enterprise Getting Started Guide 29 Troubleshoot Spotlight on SQL Server and the user needs to be able to perform the operation that led to this message, the Spotlight on SQL Server client user must be added to the Spotlight Diagnostic Administrators group on the Diagnostic Server. See “Update the Spotlight Diagnostic user groups” in the Spotlight on SQL Server Deployment Guide, for more information. Spotlight on SQL Server - Missing Performance Counters Spotlight on SQL Server uses the SQL Server virtual table sysperfinfo (SQL Server 2000) and sys.dm_os_performance_counters (SQL Server 2005 or later) to retrieve data for many of its displays. In some rare cases, this table may not contain information. When this is the case, Spotlight on SQL Server will not be able to collect the data it requires, and will display "0" for many of its components. Most obvious will be the Memory icons on the home page, which will show 0 MB of memory used by SQL Server. Also, many of the flows on the home page will show no activity, and many drilldowns will show incomplete information. Spotlight on SQL Server raises an informational alarm shortly after connecting if it detects that the sysperfinfo or sys.dm_os_performance_counters table contains no data. You can confirm the existence of this problem by running the following SQL in Query Analyzer for SQL Server 2000: select * from master..sysperfinfo or by running the following SQL in SQL Server Management Studio for SQL Server 2005 or later: select * from sys.dm_os_performance_counters If this query returns no records, then your SQL Server performance counters are missing and Spotlight on SQL Server will not be able to operate correctly. Sometimes, for a variety of reasons, the SQL Server Performance Monitor counters will not show up as they should. Often, but not always, this problem can be fixed by following these steps: 1. At the command prompt, type the following: unlodctr.exe MSSQLServer 2. Then type: lodctr.exe <SQL Server path>\binn\sqlctr.ini 3. Reboot the server. More information can be found in the SQL Server Knowledge Base at msdn.microsoft.com. Spotlight on SQL Server - SQL Activity Drilldown | Sessions page If errors are displayed on the Session Trace page (SQL Activity drilldown | Sessions page), install the following on the Spotlight Diagnostic Server machine: Spotlight on SQL Server Enterprise Getting Started Guide 30 Troubleshoot Spotlight on SQL Server l SQL Server Management Tools The version of SQL Server Management Tools required is dependent on the latest version of SQL Server monitored. When monitoring SQL Server 2012, SQL Server 2012 Management Tools are required. l Windows XP Service Pack 2 Applies only if the Spotlight Diagnostic Server machine is running on Windows XP. Playback Playback data is collected at scheduled intervals, in response to some alarms, and also while viewing a drilldown using the Spotlight client. This means that history data may not be available for a drilldown in Playback mode. You can change the rate at which data is collected. See "Configure Scheduling" in the online help for more information. Appendix: Contact Quest Contact Quest Support Quest Support is available to customers who have a trial version of a Quest product or who have purchased a Quest product and have a valid maintenance contract. Quest Support provides unlimited 24x7 access to our Support Portal at www.quest.com/support. From our Support Portal, you can do the following: l Retrieve thousands of solutions from our Knowledge Base l Download the latest releases and service packs l Create, update and review Support cases View the Global Support Guide for a detailed explanation of support programs, online services, contact information, policies and procedures. The guide is available at: www.quest.com/support. Spotlight Support When contacting Quest Support about a Spotlight product, please have the version and build number handy. You can obtain both by choosing the About option from the Help menu in the relevant Spotlight application. If your question is about an error message, write the message down and have it available for the technical support representative. Whenever you have a question about Spotlight, click Support Bundle or Contact Support on the Help menu. This creates a file called SpotlightSupport.zip. This file contains a snapshot of your Spotlight installation. Send this file and your request for assistance to Quest Software ([email protected]). Note: You can find SpotlightSupport.zip in the following folder: l C:\Users\<user>\AppData\Roaming\Quest Software\Spotlight\Default\Support l For Windows XP / Windows Server 2003 the folder is C:\Documents and Settings\<user>\Application Data\Quest Software\Spotlight\Default\Support If you find that information in the Help system or printed user guides is incorrect, unclear, or incomplete, please report the problem to Quest Support. Spotlight on SQL Server Enterprise Getting Started Guide 32 Appendix: Contact Quest Contact Quest Software Email: [email protected] Mail: Quest Software, Inc. World Headquarters 5 Polaris Way Aliso Viejo, CA 92656 USA Web site: www.quest.com Refer to our Web site for regional and international office information. About Quest Software Now more than ever, organizations need to work smart and improve efficiency. Quest Software creates and supports smart systems management products—helping our customers solve everyday IT challenges easier and faster. Learn more at www.quest.com. P Index A about Quest Software 29 playback 10 Q 32 B blocking performance counters missing 8 Quest Software 32 Quest support 31 R C connection limit 28 connections record and playback 10 rowset trace 13 S issues 24 troubleshoot 24 server side trace 13 Contact Quest Software 32 session trace errors 29 counters 29 snapshots 10 Spotlight browser 10 CPU usage 5 Customer support 31 D database growth 18 SQL expensive 12 long running 21 SQL Analysis E error codes, WMI 25 configure 12 expensive SQL 12 filter data 16 SQL Server H history 10 connection limit 28 view 10 performance counters 29 history browser 10 Support T I installation issues 24 IO bottleneck 6 L live view 31 10 Technical support 31 troubleshoot connections 24 installation 24 Spotlight on SQL Server Enterprise Getting Started Guide Index performance counters 29 Playback 30 session trace 29 unable to perform certain operations 28 user groups 28 WMI 25 V view history 10 Virtualization Overhead gauge 22 VMware connection 23 W wait statistics 20 WMI issues 25 34