DataKeeper™ Tips, Tuning and Troubleshooting
Transcription
DataKeeper™ Tips, Tuning and Troubleshooting
Module 4: Tips, Tuning and Troubleshooting August 10 Learning Objectives At the end of this module, you will understand: Minimizing initial synchronization time DataKeeper™ registry settings The DataKeeper™ command line interface User-reported issues and resolutions Known restrictions Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 2 August 10 DataKeeper™ Tips, Tuning and Troubleshooting Local Initial Synchronization Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 3 August 10 DataKeeper™ Tips, Tuning and Troubleshooting Local Initial Synchronization Addresses issue of limited bandwidth Much faster than WAN-based resynchronization Not a panacea Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 4 August 10 DataKeeper™ Tips, Tuning and Troubleshooting: Local Initial Synchronization Setup Final IP Configuration on Source and Target Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 5 August 10 DataKeeper™ Tips, Tuning and Troubleshooting: Local Initial Synchronization Create Static Routes and Test via Ping Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 6 August 10 DataKeeper™ Tips, Tuning and Troubleshooting: Local Initial Synchronization Local synch, then relocate target Connect to all servers. Create job and mirror(s) with appropriate compression setting(s). Complete initial synchronization. Shutdown target. Delete route to target from source. Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 7 August 10 DataKeeper™ Tips, Tuning and Troubleshooting: Local Initial Synchronization Local synch, then relocate target Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 8 August 10 DataKeeper™ Tips, Tuning and Troubleshooting Performance Optimization Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 9 August 10 DataKeeper™ Tips, Tuning and Troubleshooting: Performance Optimization Compression Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 10 August 10 DataKeeper™ Tips, Tuning and Troubleshooting: Performance Optimization Replication Bandwidth Throttle Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 11 August 10 DataKeeper™ Tips, Tuning and Troubleshooting: Performance Optimization Registry Settings Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 12 DataKeeper™ Tips, Tuning and Troubleshooting: Performance Optimization Registry Settings HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ExtMirr\Parameters August 10 Parameter Default Value(s) DontFlushAsynchQueue none drive letter(s) of replicated volume(s) WriteQueueHighWater 9600 decimal WriteQueueLowWater 50 decimal MaxResyncPasses 200 decimal Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 13 DataKeeper™ Tips, Tuning and Troubleshooting: WAN Performance Optimization Registry Settings HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ExtMirr\Parameters August 10 Parameter Default (decimal) Recommended (Decimal) 3000 10000 (max = 40000) MaxPingMisses 1 12 NetworkRetries 1 10 NetworkRetryDelay 5000 10000 NetWriteTimeout 10000 30000 9 9 PingInterval ValidateTargetRetries Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 14 DataKeeper™ Tips, Tuning and Troubleshooting: WAN Performance Optimization Registry Settings Global Setting: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters GlobalMaxTcpWindowSize (REG_DWORD) = Window Size Tcp1323Opts (REG_DWORD) = 1 For NIC being used for replication: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters \Interfaces\{GUID} TcpWindowSize (REG_DWORD) = Window Size Where: Window Size = Bandwidth * Latency August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 15 August 10 DataKeeper™ Tips, Tuning and Troubleshooting Command Line Interface (EMCMD) Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 16 August 10 DataKeeper™ Tips, Tuning and Troubleshooting Command Line Interface (EMCMD) Volume Related Service Related STOPSERVICE GETSERVICEINFO Job Related CREATEJOB DELETEJOB UPDATEJOB GETJOBINFO GETJOBINFOFORVOL LOCKVOLUME UNLOCKVOLUME GETVOLUMEINFO GETMIRRORVOLINFO GETEXTENDEDVOLUMEINFO GETCOMPLETEVOLUMELIST UPDATEVOLUMEINFO ISPOTENTIALMIRRORVOL RESTARTVOLUMEPIPE GETVOLUMEDRVSTATE Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 17 DataKeeper™ Tips, Tuning and Troubleshooting Command Line Interface (EMCMD) Volume Related GETCONFIGURATION SETCONFIGURATION GETSOURCEMIRROREDVOLUMES GETTARGETMIRROREDVOLUMES ISBREAKUSERREQUESTED Mirror Related August 10 CREATEMIRROR RESYNCMIRROR BREAKMIRROR PAUSEMIRROR CONTINUEMIRROR Mirror Related DELETEMIRROR DELETELOCALMIRRORONLY GETMIRRORTYPE GETRESYNCSTATUS READREGISTRY CLEARSWITCHOVER SWITCHOVERVOLUME Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 18 August 10 DataKeeper™ Tips, Tuning and Troubleshooting: Command Line Interface EMCMD (used by LifeKeeper) Rewind Related Bitmap Related GETREMOTEBITMAP MERGETARGETBITMAP ENABLEREWIND DISABLEREWIND SUSPENDREWIND GETREWINDENABLED GETREWINDSUSPENDED GETREWINDLOGFILE REWINDTAG FLUSHREWINDLOGFILE Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 19 August 10 DataKeeper™ Tips, Tuning and Troubleshooting: Command Line Interface Usage: EMCMD <ip address> GETMIRRORVOLINFO <letter> Command: Output: emcmd 192.168.251.98 getmirrorvolinfo e E: 1 192.168.251.98 192.168.252.98 1 Parameter Description E Mirrored volume 1 Current ROLE of the system specified 0 = None 1 = Source 2 = Target 192.168.251.98 SOURCE system 192.168.252.98 TARGET system 1 Current STATE of the mirror 0 – None 1 – Mirroring 2 – Resyncing 3 – Broken 4 – Paused 5 – Resync Pending Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 20 August 10 DataKeeper™ Tips, Tuning and Troubleshooting: Command Line Interface Use Case: Manual Switchover, Source Out of Service EMCMD <target ip> SWITCHOVERVOLUME <letter> Use Case: Manual Switchover, Source Destroyed EMCMD <target ip> DELETELOCALMIRRORONLY <letter> EMCMD <target ip> CLEARSWITCHOVER <letter> Source must be off-line, otherwise a mirror split-brain condition may be created. Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 21 August 10 DataKeeper™ Tips, Tuning and Troubleshooting: Command Line Interface Use Case: Target as Tape Backup Source Tape backup tools may be used without application-aware agents Scripted/automated Description Command Pause the mirror on the source system specifying IP address of target EMCMD <source ip> PAUSEMIRROR <letter> <target ip> Unlock volume on target system. EMCMD <target ip> UNLOCKVOLUME <letter> Backup or copy data from target system. copy; xcopy; ntbackup; backup; … Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 22 August 10 DataKeeper™ Tips, Tuning and Troubleshooting: Command Line Interface Use Case: Target as Tape Backup Source Description Command Lock volume on target system. EMCMD <target ip> LOCKVOLUME <letter> Continue mirror on source system EMCMD <source ip> CONTINUEMIRROR <letter> <target ip> When mirror paused and unlocked, writes to source are flagged in bitmap. Sent to target when mirror continued (partial resynchronization). When target unlocked, it is writable. Any blocks written on target overwritten by source blocks when mirror continued (partial resynchronization). Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 23 August 10 DataKeeper™ Tips, Tuning and Troubleshooting: Command Line Interface Use Case: Disk-to-Disk Copy Mirror Source Replication Mirror Target Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 24 August 10 DataKeeper™ Tips, Tuning and Troubleshooting Troubleshooting, Known Issues and Restrictions Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 25 August 10 DataKeeper™ Tips, Tuning and Troubleshooting “Split Brain” Scenario Communications between systems lost Target volume brought on-line Communications between systems re-established Event logs indicate there are two sources Mirror paused on both servers Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 26 August 10 DataKeeper™ Tips, Tuning and Troubleshooting Split Brain Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 27 August 10 DataKeeper™ Tips, Tuning and Troubleshooting Split Brain Recovery Determine which server should be source Choose mirror source Click “OK” Source Server Mirror partially resynchronized Disconnect/Reconnect GUI from/to servers if status incorrect Target Server Resynch Volume X: Volume X: Mirror Source Mirror Target Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 28 August 10 DataKeeper™ Tips, Tuning and Troubleshooting Issues and Resolutions Component Message/Issue Cause Suggested Action User Interface Cannot connect to servers NetBIOS (137-139) and/or SMB (445) ports blocked by firewall Open ports User Interface Can connect to servers, create job and select volumes for source and target, but mirror creation fails DataKeeper dispatch port (9999) or mirror ports (10000-10025) blocked by firewall Open ports User Interface No volumes available for mirroring Only C: available or volume contains page file Create new volume or remove page file from volume Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 29 August 10 DataKeeper™ Tips, Tuning and Troubleshooting Issues and Resolutions Message/Issue Cause Suggested Action User Interface Failed to create mirror. Target System event log entry contains error code: 0xC0000055 Distributed Link Tracking Client service enabled and active Stop and disable service on both source and target User Interface Mirror created, but not stored in job. Event log entry contains “File: .\GuiThread.cpp Line 3099 …” Local security policy for “Network Access: Let Everyone permissions apply to anonymous users” is disabled. Enable policy on all servers. Create mirror failed. vmms.exe program has volume open preventing DataKeeper from obtaining volume lock Stop vmms.exe program Component System Event Log Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 30 August 10 DataKeeper™ Tips, Tuning and Troubleshooting Issues and Resolutions Component Message/Issue Cause Suggested Action Windows Clustering Newly created resource appears off-line but is unlocked Expected behavior Bring resource on-line Windows Clustering DataKeeper Volume resource not available for use DataKeeper installed before cluster name resource created Perform repair install of DataKeeper Windows Clustering DataKeeper Volume resource not available for use. Event log contains “DataKeeperVolume.dll, Error: 70” All cluster nodes were not on-line Bring all cluster nodes on-line Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 31 August 10 DataKeeper™ Tips, Tuning and Troubleshooting Known Issues and Workarounds Component Message/Issue Cause Suggested Action Performance Monitor on Windows Server 2008 64-bit No DataKeeper counter logs available in Performance Monitor DataKeeper counters not compatible with 64-bit Performance Monitor Use 32-bit Performance Monitor (under %WINDIR%\SysWow64\) Performance Monitor on Windows Server 2003 Counter logs do not work on Windows Server 2003 Known issue to be resolved in future release Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 32 August 10 DataKeeper™ Tips, Tuning and Troubleshooting Known Restrictions and Recommendations DataKeeper home directory path name must be 32 characters or less Pause mirror before running CHKDSK on source volume Pause mirror before running defragmentation on source volume Heavy write activity when using synchronous mirror negatively effects application performance Resizing mirrored Dynamic Disks not supported Create bitmap directory before relocating bitmap files Bitmaps on Dynamic Disks that are not “System” are not supported Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 33 August 10 DataKeeper™ Tips, Tuning and Troubleshooting Summary Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 34 August 10 DataKeeper™ Tips, Tuning and Troubleshooting Summary Minimizing initial synchronization time for WAN deployments DataKeeper™ registry keys and settings The DataKeeper™ command line interface and several use cases How to recover from a “split-brain” situation User configuration oversights and errors Known restrictions and explanations Copyright © 2000-2010 by SteelEye Technology, Inc. All Rights Reserved. 35 End Module 4 Lab: Tips, Tuning and Troubleshooting Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 36